Eat the frog: Why UX researchers should keep doing the hard part
If you asked a researcher how much they value critical thinking skills, the answer is probably something like “a helluva lot.”
Yet, as AI has worked its way into our everyday lives, and into our research workflows, many of us are spending less mental effort on study design, data analysis, and readout creation. In general, we’re creating less. AI does the initial creation work, while our job becomes pushing back, critiquing its output, and massaging it until we get something we like.
But what happens when researchers lose their ability to think critically? And what should we be worried about losing to AI in the process?
The concept of eating the frog, or “eat that frog” comes from the motivational speaker and author, Brian Tracy. The idea is that you should do the hardest thing first, then move onto the easier parts of your day. The desired result? You’ll accomplish more, and feel more accomplished. If you’re adept at using AI, you can make it so you never have to eat the frog again. I want to take this moment to argue: we need to eat that frog.
I wanted to take a closer look at how we got here, where I think it’s important to keep the effort, and what we can do about it.
Does it matter how we get there? Three questions to consider.
As humans, we love to avoid cognitive hardship. A meta-analysis of 170 studies found that mental effort feels unpleasant across nearly every kind of task and population researchers have studied, and in experiments people will even give up money to avoid a harder task. Sometimes we hate it so much, we put the task off until the very last minute. We’ve all been there, turning in the school report minutes before it’s due. So when AI offers to take the hard part off our plate, it’s tempting to let it. But does it matter if we do?
The answer is complicated. I’ve found that, as a community, we’re often more likely to be skeptical of AI, or at least to use it with more intention. You can now build or buy AI tooling trained for UX research, and as long as there’s a human in the loop, its outputs are generally reliable. It makes us faster. And it is great that it makes us faster.
Put more bluntly: if we can create an output that’s as good, on average, as that of a senior-level user researcher, why does it matter how we got there if the goal at the end of the day is to accurately inform some business goal?
I think that’s a valid argument. But I have questions about it, and they all come back to one thing: quality, skill, and an understanding of our users are all built through effortful thinking. Quality comes from knowing what it looks like, and that comes from experience. Skill comes from practice. And understanding users is at its best when we’ve had to think critically about data.
It’s also led me to three questions:
1. Is AI tooling really as good as a senior-level researcher?
You only know the answer if you’ve grappled with the data yourself. If you don’t understand how a finding was arrived at, you don’t know how solid it is. And “on average” can hide a lot. The finding that changes a roadmap is rarely the average one.
We’ve all seen what research made with little effort looks like:
- Sometimes it’s too long for a human to read
- It’s overly formal, with strange phrasing that’s hard to follow
- It includes far too many details, many of which aren’t relevant to the plot
- It has no narrative or a weak one
- It might feel cryptic, long, and overly detailed
I’ve seen in many cases, the tool is weak and so is the prompting, whether lazy or uninformed. AI literacy is still emerging, and without guardrails built in, AI slop is the standard.
2. Do better tools and better prompts really help?
Even with a great tool, someone has to know what a “good narrative” looks like, and that comes from experience. The original output from AI is never the final output. It requires constant massaging and editing before it gets to a place that’s good enough. To do that well, we need to remain critical thinkers, highly skilled at understanding what good looks like: what to throw out, and what to add in.
And if we stop grappling with the data, or with study design and question writing, how do we get better at any of it? Learning and improvement in this field are hard-earned, through practice, effort, and time.
For me, part of what matters is my skills staying sharp and improving, and my memory staying engaged through effortful thinking that builds links to new information. Psychologists call this the generation effect: we remember what we produce ourselves far better than what we only read.
3. How do we help our stakeholders, especially the product managers deciding which features users get and which improvements come first, to deeply understand our end users?
I truly don’t have an answer here, but it’s important to ask ourselves, “How are users thinking about their jobs and our tools?” and “How do we build for them unless we truly understand their working conditions and their needs?”
If that understanding can still come from quickly generated AI outputs, then that’s great. But we need to consider the question, because something that’s lost is the researcher’s own understanding of the user. If a researcher’s understanding of the user is shallow, how could a stakeholder’s be more meaningful?
Where do we keep the effort intact?
So what do we do? I would argue we can do two things:
- We can use AI tooling to augment our workflows rather than fully automate them
- We can continue creating
Each of us needs to decide where and when it’s important to create, with the understanding that there’s a whole spectrum to choose from.
At one end, some researchers use AI for smaller tasks, like an interview notes spreadsheet: each question asked, a row for each participant, and their answers to each question, pulled from the recordings and transcripts. It’s common for UXRs to synthesize from a spreadsheet like this, doing counts, theming, and so on. They’re augmenting their workflow by handing AI one time-consuming task, but they still take on the first job of synthesis themselves. From there, they can have AI check their analysis and help them put together the report.
Even the methodologists who designed a validated workflow for AI-assisted thematic analysis recommend that researchers analyze part of the data by hand first, before bringing in the AI. Further along, you can do that analysis in partnership with AI. At the far end, full automation is possible.
Personally, I’ve built my own network of agents, Magpie, that walks through a research workflow end to end. It helps with the work, but the core thinking is still mine. It asks what I’m trying to learn before it drafts anything, and it explains all of the decisions it makes.
Then, at key points, it hands the work back to me. I’m responsible for stating my team’s assumptions before Magpie plans a study. I look for patterns in the data before it does any thematic analysis or coding, and I decide which themes to keep, adjust, or cut. That’s often how I work today.
There’s evidence that this kind of setup matters. In a randomized study of nearly 1,000 high school math students, those who practiced with a standard ChatGPT-style assistant did better during practice but scored 17% lower on the exam they took without it. Students whose AI tutor gave hints instead of answers showed no significant drop.
Lately, though, I’m considering doing the occasional study fully by hand. Or using AI only to combine my notes and transcripts into a spreadsheet of responses to each question or hypothesis, which I can then work from myself, bringing AI in only after I’ve noted some counts, themes, and findings of my own.
So where should you land on that spectrum? It depends on your goal. If your goal is to ensure higher-quality outputs, then perhaps you want to do synthesis manually first, then bring in AI as a helper. If your goal is to keep high effort in your workflow, that’s where the frog comes in.
Is it time for researchers to eat the frog?
So, back to where we started: how much do you value your critical thinking skills? Here’s my challenge: What’s one thing in your research work that’s so difficult to do manually that you hate it? Perhaps that’s your signal to eat the frog.
Remember that school report you put off until the last minute? Eating the frog is the opposite. You do the hardest, least appealing task first. Do that thing. Then do it again every once in a while, and think about how what you’re able to do as a result of that hard work compares to what happens when you don’t.
Our brain works like a muscle. You need to use it.
