IBM
UX Researcher
Start to sunset: building a new learning platform to optimize global team performance.
Long Term Project
SUMMARY
1
Background
2
The Product
3
Spotlight Study
4
Outcomes
The product
Unity is a learning platform where CIO teams learn two different essentials to the business.
Key responsibilities
Began as one of several UXRs, then as resources shifted I took on the role of leading UX Research efforts for Unity.
What made this unique
Built, promoted and launched this brand new platform then tested engagement and presented findings to our CIO VP.
THE PEOPLE (the best part)
CIO Unity Team
Alejandra Padilla
Jen Morgan
Gabby Hoefer
Maddy Barr
Christi Wilbert
Product Owner
Product Owner
UX Researcher
Visual Designer
Visual Designer
Stakeholders
Nicole Narcisco
Kathryn Guarini
VP, CIO Design
CIO, IBM
BACKGROUND
CIO Unity
Research on building, promoting and launching this brand new platform.
After two years on CIO Unity, a ton of research has been conducted. Below is a taste of some of the research I’ve lead, then I’ll spotlight a project I’m especially proud of in the upcoming sections.
Timeline of Unity's development.
Qualitative
User interviews to test my new feature design
Designed a new feature and ran 5 user interviews to understand expectations and actionability.
Mixed Methods
A/B testing the home page redesign during user interviews
Guided interview with A/B testing of two different home page design options for clarity and discovery.
Mixed Methods
Testing the success of outreach events with teams
Survey to follow up with event attendees and their direct reports to determine course adoption.
THE PRODUCT
Our new learning platform
Unity is a learning platform where you learn two different skills. It’s internal-only, so let’s pretend those skills are...
Emoji of a plant in a pot
Gardening
Understanding the gardener

Teams struggle to understand the best type of gardener to request for their project goals. Sometimes they staff gardeners, but are actually looking for lawn mowing tasks.

IBMers need a way of learning the basics of gardening so that they can make the right requests.
Emoji of a blue bicycle
Biking
Development teams don’t bike effectively

Development teams are essential to the business and helping them bike effectively is critical. Some teams are unsure of where to begin with their training wheels.

IBMers need a way of learning the basics of biking so that they can work effectively.
BUSINESS VALUE
Circle with 3 points representing cultural transformation, performance increases and cost time savings. In the middle of the circle is the goal of the project: Deliver the future of IT to propel IBMs success and growth.

"Individually, a single designer's impact may be limited, but fostering a cultural shift holds the power to scale that impact exponentially."

USER RESEARCH SPOTLIGHT: MEASURING PRODUCT IMPACT
Mixed Methods
Survey: What impact has CIO Unity made during it’s first 6 months of release?
Prior to this, there was no way to measure Unity’s business impact. This formed the baseline that we’ll refer to and build upon every 6 months.
Purpose
Understand the business impact of CIO Unity resources.

Approach
Survey invites were distributed via email and slack to a nonrandom sample of IBMers in March 2023.
  • x% response rate
  • ~300 responses
  • 6 minutes
Map of the world with percentages calling out each company, representing who the participants who filled out the survey.

*Numbers and specific details have been fictionalized or removed due to confidentiality.

OUTCOMES: 
DESIGN
BUSINESS
MARKETING
Qualitative
There is a gap between discovering knowledge and applying it to their teams.
Respondents provided open-text answers reflecting the impact of Unity on their team. I categorized their responses and a pattern of 3 stages emerged.
  • There is a disconnect between learning the course and applying it to their teams. The majority of users were still in the learning phase with Unity, so they were not seeing the benefits yet.

  • Recommendation: future efforts should explore ways to move the needle from discovering courses to applying them to their products.
“CIO Unity…helps us to discover how to apply biking techniques in our daily work. But we need to apply it.”
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
Quantitative
Overall, the design of Unity is well received and users prefer the biking over gardening.
We evaluated Unity usage trends were evaluated, revealing an excellent NPS and the most popular courses over time.
  • Unity had a high NPS score, revealing excellent user experience overall with the product.

  • Biking courses saw more traffic than gardening courses. The top biking courses are consistent across the last 2 quarters.

  • There were some nuances found in the specific courses, which are protected under NDA and cannot be disclosed publicly.

  • Signals of progress towards organic usage of Unity was found, as less course-specific events were conducted this quarter but traffic was not as impacted.
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
OUTCOMES:  
DESIGN
BUSINESS
MARKETING
We shifted our analysis efforts to case studies due to our small sample size.
Unity is still a new product, with only about 300 active users, so our sample size of those responding to the survey was small and difficult to establish statistical significance.
  • As a result, we did a deep dive into specific teams in order to understand the impact of using Unity.

  • This also enables us to track these teams over time. In future research, we plan to explore how the teams have continued to develop.
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
Mixed Methods
Gardening Case Study: IBM Seller Incentives
The IBM Seller Incentives (ISI) team attributes Unity to their +7 point increase in Plant Health, a popular metric used to measure gardening across the industry.
  • ISI was struggling with gardening across the last couple years, but began using Unity last quarter to refresh their skills and saw a 7-point increase in their metrics.

  • Even with a gardener embedded on their team, ISI reports high impact from Unity resources.
“We have Plant Health metrics [to show impact]... Unity enforced our need for additional training.”
-Manager
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
Mixed Methods
Biking Case Study: Pricing Systems Teams
A manager in the Pricing Systems group praised Unity and attributed a large metric increase to the Unity courses.
  • Although I am unable to disclose the metric, this is a popular measurement used across the industry, so increases show significant impact.

  • After Unity exposure, 22/45 Pricing Systems projects showed high or elite metrics in February 2023.
“We have a number of [measurements] to show significant impact.”
-Pricing Systems
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
OUTCOMES:  
DESIGN
BUSINESS
MARKETING
Qualitative
Marketing is a huge part of spreading awareness and Unity impact.
As we tried to reach as many Unity users as possible, we broke down the CIO organization domains that they stemmed from and how much awareness they had of Unity.
  • CIO Design (the organization I’m part of) had the highest awareness, as expected.

  • This calls for additional research into why the lowest group of domains showed less awareness and determine improvements.
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
Qualitative
Evaluating the impact of Unity labs: our team outreach program.
Our OCM team has been running labs over the past 3 months to help 12 leaders implement Unity courses on their teams.
  • All leaders reviewed courses and 7/12 have brought them back to their teams, resulting in 18 teams or 108 people applying Unity courses.

  • A case study with the Q2C team shows the impact of learning one of the biking practices on their team. The team is moving twice as fast due to learning to ride their bike.

  • Previous research has shown the impact of course-specific events to increase team member engagement. Unity Labs show promise for working with leaders to increase promotion and top-down team usage.
“Other leaders are missing out on this opportunity.”
-Domain leader
Graph showing Unity baseline in three stages: discover, apply and benefit. The majority of users fall into the discover stage, indicating a need for efforts to move them into apply and benefit.
Image with 6 different screens representing the different designs of the unity website