Passer au contenu principal

How to conduct UX benchmarking using the METRIC framework

Is your product team worried they’re not heading in the right direction? Maybe designers on your team are unsure if their design solutions are effectively solving user problems. Or product managers are wondering if the current product holds up against live products already on the market. If you’re finding it difficult to answer these questions, benchmarking could be your best next step to creating effective products that users actually want. 

I’ve used benchmarking in a variety of product settings, but the common thread across all of them was that I used the METRIC framework to help us get to the answer. I created and refined the METRIC benchmarking framework (which I’ll explain in more detail in this article) and rolled it out across a team of 40 UX practitioners, with each UX team member using it to benchmark the products they supported. The process is easily repeatable, and better yet, it gave everyone a consistent way to measure how well their products were performing and identify opportunities to improve experiences.

Here, I’ll go into detail about the benefits of benchmarking, where the METRIC framework comes in, and how you can implement the process with your own team.

Consider your team’s UX maturity and tolerance for mistakes

So why consider benchmarking in the first place? First, it’s best to define it for you and your team.

I define benchmarking as the act of tracking usability or other UX metrics over time. It can be especially helpful as users, features, or products change over time.

Benchmarking is often performed as a formal process that’s conducted at regular intervals to meet your team’s needs (like once or twice a year). Teams can benchmark against their own product historically, other products on the market, overall industry benchmarks, stakeholder-defined targets, and more.

How can it be helpful? Benchmarking can do some magical things—it can show a stakeholder how well the product is performing within a market, it can give a team confidence that they’re heading in the right direction, and it can do all of that relatively quickly if you pick the right measurement formats.

I’ve seen benchmarking be received in a variety of ways, and its reception usually comes down to the team’s UX maturity and the company’s culture towards mistakes. I tend to see two common scenarios: First, if a company punishes mistakes, teams tend to resist benchmarking out of fear. Second, if a company sees mistakes as part of an iterative process (and understands that those mistakes can be fixed), then teams are usually excited about benchmarking. Some examples:

  • Once a UX team and I pitched benchmarking to a product team that wholeheartedly rejected it. They had low UX maturity and had never worked with research before, yet had started building the product anyway. The company had a culture of fear around mistakes, and ultimately the product owner didn’t want to get a “bad grade” on their hard work. They refused to let UX run any benchmarking studies.
    ‍
  • Inside this same company, I ran benchmarking with a team that was a little braver. They had medium UX maturity and had been working with me as a researcher for a few years. During benchmarking, I found something surprising in our survey results—a core workflow was rated very low on ease of use. This was surprising to all of us, so I went to our Google Tag Manager data that we were also collecting for benchmarking and found that users were only completing that workflow 50% of the time. I even found one user who stayed inside that workflow and clicked the same button 11 times over a few hours. I ran an impromptu focus group and found that there was a single piece of data necessary for a separate task that could only be found in the middle of this workflow. Luckily, changing this was already on the roadmap. Benchmarking validated and de-risked the product owner’s roadmap.

This leads to an important gut check early on: when you are initially introducing benchmarking to your team, consider their UX maturity and how mistakes are perceived in your organization. This will help you pitch benchmarking to your team and have it be received well. As long as the product team is brave enough, the METRIC process works very well in teams with lower UX maturity.

The METRIC Framework

METRIC consists of six key steps:

  1. Mobilize your team
  2. Establish what’s important
  3. Target key metrics
  4. Ready your tools and setup
  5. Initiate the study and analyze
  6. Communicate your results and sync with stakeholders

Before you start the METRIC process, I recommend reviewing the HEART and CASTLE frameworks for ideas of what types of measurements you can use while benchmarking. Each one is essentially a list of different measurements you can use for benchmarking in a B2C or enterprise setting. Reviewing them will help you lay the groundwork before you dive in.

1. Mobilize your team

Who is involved: the entire product team

The first thing you need to do is gather your team. It’s often helpful to include:

  • Product Managers
  • Product Owners
  • UX Designers
  • Business Analysts
  • Project Managers
  • Optional but may be helpful: organizational change managers, data scientists, software engineer leads

Why this is important: getting the right people in the room means you don’t miss anything and all angles of successful benchmarking are covered.

2. Establish what’s important

Who is involved: the entire product team

Once you have your team together, you need to establish what’s important.

As your mobilized team meets, you can show them metrics from the HEART framework (for B2C products) and CASTLE framework (for enterprise products), noting that each one is broken out into sections. (Not all framework metrics will be relevant or important to your product.) With your team, determine:

  • What tasks are most important?
  • Which product metrics are most relevant?
  • What KPIs matter most to the users’ leadership or the client? (Pick 1-3)
  • If applicable, which HEART or CASTLE framework sections should we use? (Pick 2-4)

Each person on the team may have a different opinion or perspective on what metrics are important—that’s okay. You can collect more than one metric.

Why this is important: in enterprise product spaces, it can be difficult to narrow down which tasks and KPIs to focus on because the products are often quite involved. You likely aren’t going to have the time or budget to benchmark every single task in your product. Establishing what tasks and KPIs are most important will help you focus your benchmarking efforts and make them more effective. 

Only move forward to the next step if your team confidently knows what KPIs matter most to your client or users’ leadership in an enterprise setting. Tracking these metrics will make your benchmarking results more relevant to those key stakeholders.

3. Target key metrics

Who is involved: UX Researcher or the entire product team (depends on UX maturity)

Who is involved in this step often depends on your team’s UX maturity. If the team is particularly nervous about benchmarking, involve them in this step so they have more buy-in to the results. If your team has high UX maturity, they will likely trust you to take care of this step by yourself as they look forward to seeing the results.

If your team is new to this process, it may take some time to answer the questions in the “Establish what’s important” section; this may need to be a separate meeting.

Why this is important: you may have come up with a general idea of which KPIs to focus on, but have not matched them up to your benchmarking measurements yet. Taking this time to sit down and ensure you’re covering all your bases will mean more effective benchmarking that gives you a holistic, complete picture of your product. 

Select specific, important metrics that map to each framework section. They may already be listed in the HEART or CASTLE frameworks, or they may be other metrics that are important to your team. If applicable, try to select one measurement from each of these categories:

  • Attitudinal (e.g., ease-of-use rating, satisfaction rating) 
  • Behavioral (e.g., average time on task, task completion rate)
  • Performance (e.g., retention rate, activation rate) 

As you are defining the key metrics you will use, also consider:

  • The benefits of collecting data on both leading and lagging metrics
  • What data is already being collected by other teams
  • What defines success for each metric

4. Ready your tools and set up

Who is involved: UX Researcher, anyone else needed to set up tools

Now it’s time to pick your data collection tools and set them up. When benchmarking, it’s helpful to benchmark against outside sources as well as against past iterations of your product. Using survey methods like SUPR-Q or UEQ can help you quickly and easily compare your product to external products without needing to survey large sample sizes.

You may need some help setting up some tools, so don’t be afraid to reach out to other people in your organization.

Why this is important: you can’t collect any data without the right tools. Your organization may already have access to the tools you need—this is a time to get everything set up for clear data collection.

Some tools you may use:

  • Google Tag Manager
  • Survey tools
  • Data reports created by operations teams or other groups (in Power BI, Tableau, etc.)

5. Initiate the study and analyze

Who is involved: UX Researcher

It’s time to launch your study!

When launching in an enterprise setting, it’s important to focus on internal feedback and overcome a common barrier: employees using your product are busy and lack motivation to take your survey. To get the best survey completion rates, it’s a good idea to communicate the importance of benchmarking to leadership—and ask them to encourage their teams to take the survey. Your message is clear: “This survey is a chance for your voice to be heard. We take the results very seriously and will implement fixes based on your feedback.” 

Why this is important: if you don’t coordinate with enterprise leadership, you may not get the feedback you need during benchmarking and risk all of your planning going to waste. Partnering with enterprise leadership will ensure you get the highest response rate to your surveys possible.

When you analyze your results, look for surprising data and see if any of your other data sources can add clarity. Don’t be afraid to add a focus group to get additional clarity on quantitative data.

Some survey methodologies like SUPR-Q or UEQ come with their own analysis tool, which makes your life easier.

As you’re analyzing your data, you can compare results to:

  • Past benchmarks on your own product
  • Specific competitor products
  • Other products on the market (UEQ is one way to do this)
  • UX industry benchmarks (SUS, SUPR-Q, etc.)
  • Published industry benchmarks
  • Business targets defined by leadership or clients

6. Communicate results and sync with stakeholders

Who is involved: the entire product team

Compile all your data into one report, focusing on high-level findings and team action items. As you build a narrative around the findings and data, try not to overwhelm your team with too much information. Include detailed findings in the appendix of your report. For example: “We surveyed internal employees about their usage of X product, and found that all tasks except one were very easy to use. We reviewed Google Tag Manager data and also held a focus group, where we found that internal employees had to enter that workflow just to grab a piece of data for a separate one. A fix for this is already on the roadmap.”

Why this is important: giving your stakeholders the high-level summary of the benchmarking findings keeps them engaged and helps them act on the findings, if applicable. If you get too much into the weeds, they may begin to tune out.

After you present your findings to your team, determine together how often you want to benchmark going forward. You may want to schedule benchmark data collection around product updates, giving users a few weeks to adjust to the changes but not enough time to find workarounds.

Share your benchmarking data with as many people as you can—users’ enterprise leadership, development teams, data scientists, etc. Anybody I’ve presented this data to has been grateful to hear how things are going with the product they have devoted so much time to.

Leveraging your results

So, how do you know if your efforts worked? After you’ve collected this data, there are a number of ways to leverage your results:

  • You can use these findings as a way to show the ROI or impact of UX and UX research. For example, compare different benchmarks against the product releases where UX and UX research were heavily involved. This is a great way to numerically show how UX improves outcomes for users.
    ‍
  • You can share your results in your yearly review or on your resume. These numbers can help add some quantitative proof to the hard work you do year-round.

You can also have the clarity of knowing you’ve successfully benchmarked if you have clear, meaningful metrics you’re measuring, can repeat the process in the future for long-term comparisons, and you've turned findings into actionable recommendations.

About the author

Nicole Hunt is a Lead UX Researcher with 7+ years of experience helping organizations make better decisions, faster with human-driven insights. With a background spanning agency, enterprise, and nonprofit environments, Nicole combines strategic thinking with a practical, approachable research style to turn complex user needs into actionable insights that improve experiences and drive business outcomes.