Usability test screener questions that recruit the right five people

Why do screener questions decide whether a usability test works?

A usability test can only find the problems its participants are able to hit, and the screener decides who those participants are. Five is the usual number per round because Jakob Nielsen's analysis in Why You Only Need to Test with 5 Users found that a first study with 5 users finds about 85% of the usability problems in a design. That arithmetic assumes the five are representative. Recruit two people who are too expert, or one who never does the task, and you are running a test with three.

The catch is that respondents want the incentive. Kathryn Whitenton of Nielsen Norman Group, in How to use Screening Questions to Select the Right Participants, warns that "screening questions that make the purpose of the study obvious can backfire", because people who can guess the study topic will exaggerate to qualify. A screener therefore has two jobs that pull against each other: learn specific things about the person, and give away nothing about the study.

This guide is the recruiting step in a four-part sequence for running your next real test. The session script template comes next, then moderation, then the report. For the overall method, start with how to run a usability test.

What should a usability test screener ask?

Ask about what the person has recently done, not who they are. Age, income and job title are proxies; the thing you actually need is someone who performs the task you are testing, in conditions close to your product's real use. Three behaviour questions do most of the filtering:

Instead of this demographic question	Ask this behaviour question	Why it filters better
"What is your age?"	"In the last 4 weeks, how many times did you order groceries online for delivery?"	Recency and frequency describe the exact task you will test
"Are you tech-savvy?"	"Which of these did you use for your most recent grocery order?" (list of services plus distractors)	Shows their current tool, so you know which habits they bring
"What is your job title?"	"Who in your household decides what goes in the weekly shop?"	Names the role that does the task, which a title cannot

Then add one open question, written so there is no right answer: "Describe the last time you ordered groceries online. What did you buy and how did it go?" Whitenton recommends open-ended questions for experience levels because a person who really does the task gives concrete, specific detail that is hard to fake, and one who is exaggerating usually cannot. Ask whoever reads the answers to judge them against a written rule (for example: names a service, a rough basket, and one thing that went well or badly) so two recruiters accept the same people.

How do you hide the purpose of the study in screener questions?

Use distractors on every multiple-choice question that would otherwise give the game away. Whitenton's rule is that the distractor answers should be realistic activities, plausible enough that someone guessing cannot spot the target. If you need people who order groceries online, list grocery delivery alongside takeaway delivery, meal kits and click-and-collect, not alongside something absurd.

Two more habits keep the screener honest:

• Screen out the "select all" respondent. On multi-select questions, Whitenton notes that "Selecting all choices is a warning sign that the person may be too eager to participate", and suggests preferring someone who picked fewer options.
• Put the qualifying question in the middle. A target question at the very start or end is easier to spot. Bury it among two or three neutral ones about related habits.

Who should a usability test screener disqualify?

Disqualify anyone whose knowledge would let them complete tasks that real users fail. In practice that is four groups:

1. Colleagues. The GOV.UK Service Manual on finding participants advises: "Avoid using your own staff for research into the public facing parts of your service. Their knowledge and experience can mean they will use the service in a very different way."
2. Friends and family of the team. They are polite, they guess what you hope to see, and they rarely admit confusion to someone they know.
3. Power users and beta testers. They know the workarounds. Recruit them only when the test is specifically about expert workflows.
4. People who work in your industry. Whitenton recommends excluding people "whose job relates to the design you are evaluating" unless the design is for experts, and suggests an open "What is your occupation?" question rather than a list of excluded jobs, because people self-identify more honestly when they do not know what is being screened out.

Add a research-history question too: "Have you taken part in a paid research study in the last 6 months?" Six months is a judgement call rather than a published rule. The aim is to avoid people who take part in research so often that they behave like reviewers instead of users.

What does a usability test screener template look like?

Copy the blank version below into your survey tool or recruitment brief. Each question has a decision rule next to it, so whoever reads the responses applies the same bar.

USABILITY TEST SCREENER
Study (internal name, never shown to respondents):
Flow being tested:
Target per group:            Spares:          Session length:
Format (remote or in person, moderated):

Q1 Recent frequency of the task
   "In the last [period], how many times did you [task, phrased neutrally]?"
   Options: 0 / 1 / 2 to 3 / 4 or more
   Accept: [range]      Reject: [range]

Q2 Current tool or service
   "Which of these did you use the last time you [broader activity]?"
   Options: [target option] + [3 plausible distractors] + Other (please say)
   Accept: [options]    Reject: selects every option

Q3 Role in the task
   "Who usually [does the task] in your [household or team]?"
   Options: Me / Shared / Someone else
   Accept: [options]

Q4 Open description (read by a person)
   "Describe the last time you [task]. What happened?"
   Accept if the answer names: [concrete detail 1], [detail 2], [detail 3]

Q5 Occupation (open text)
   "What is your occupation, and what industry do you work in?"
   Reject: [industries close to the product], anyone at [your company]

Q6 Research history
   "Have you taken part in a paid research study in the last [period]?"
   Reject: Yes, more than [number] times

Q7 Device and setup
   "Which device would you use for a 45 minute video call?"
   Options: Laptop or desktop with camera / Phone / Tablet / Not sure
   Flag "Not sure" for a setup call

Q8 Availability
   [Session slots]

Mix needed across accepted respondents:
   [e.g. at least 2 of 5 with characteristic X]

What does a filled consumer screener look like?

Here is the template filled for the running example used across this sequence: a grocery delivery app has redesigned its checkout, where customers pick a delivery slot, set substitution preferences and apply a voucher. The team wants five participants for remote, moderated, 45-minute sessions.

USABILITY TEST SCREENER
Study: Checkout redesign round 1
Flow being tested: basket to order confirmation
Target per group: 5          Spares: 1          Session length: 45 min
Format: remote, moderated, clickable prototype

Q1 "In the last 4 weeks, how many times did you order
   food or household items online for delivery?"
   Options: 0 / 1 / 2 to 3 / 4 or more
   Accept: 2 to 3, 4 or more     Reject: 0, 1

Q2 "Which of these did you use the last time you got
   food delivered or collected?"
   Options: Supermarket delivery website or app / Takeaway app /
   Meal kit subscription / Click and collect / Other
   Accept: Supermarket delivery     Reject: ticks all five

Q3 "Who usually does the main food shop in your household?"
   Options: Me / Shared / Someone else
   Accept: Me, Shared

Q4 "Describe the last time you ordered groceries online.
   What happened?"
   Accept if the answer names: a service, roughly what was
   ordered, and one thing that went well or badly

Q5 "What is your occupation, and what industry do you work in?"
   Reject: grocery retail, ecommerce, UX or product roles,
   anyone at the company running the study

Q6 "Have you taken part in a paid research study in the
   last 6 months?"
   Reject: Yes, more than twice

Q7 "Which device would you use for a 45 minute video call?"
   Flag "Phone", "Tablet" and "Not sure" for a setup call

Mix needed across the 5:
   at least 2 who have had an item substituted in the last month
   at least 1 who mostly shops on a phone

The mix line is where the screener earns its keep. Substitutions are part of the redesign, so a group that never receives substitutions would leave half the flow untested.

How do you screen when the product has distinct user types?

Split the recruit into groups, and use fewer people per group. Nielsen's guidance in the same 5-user article is that "You need to test additional users when a website has several highly distinct groups of users", and he recommends 3 to 4 users from each category when testing two groups, and 3 from each when testing three or more, because behaviour overlaps across groups.

B2B products almost always have at least two groups, because the person who configures the product is rarely the person who uses it daily. Take an invoice approval tool for small and mid-sized companies, where finance admins set up approval rules and department heads approve supplier invoices. The two groups need different screeners:

SCREENER A: FINANCE ADMINS (target 4, spare 1)
Q1 "In the last 3 months, which of these have you done
   yourself at work?"
   Options: Set up who approves supplier invoices / Ran payroll /
   Reconciled bank statements / Chose a new finance tool / None
   Accept: Set up approvals    Reject: ticks all, or None
Q2 "Which tool does your company use to pay supplier invoices?"
   Open text. Accept: names a tool or describes the process
Q3 "Roughly how many supplier invoices does your company
   pay per month?"
   Options: Under 20 / 20 to 100 / 101 to 500 / Over 500
   Accept: 20 to 100, 101 to 500
Q4 "What is your role and company size?" (open text)
   Reject: accountancy firms serving clients, finance software
   vendors, our customers' admins already in the beta

SCREENER B: APPROVERS (target 4, spare 1)
Q1 "In the last month, did you approve or sign off any of
   these at work?"
   Options: Supplier invoices / Expense claims / Holiday
   requests / Purchase orders / None
   Accept: Supplier invoices or Purchase orders
Q2 "Roughly how many did you approve in that month?"
   Options: 1 to 5 / 6 to 20 / Over 20
   Accept: any
Q3 "How do you usually approve them?"
   Options: Email reply / In a finance tool / In a chat
   message / On paper / Other
   Accept: any, but aim for a mix
Q4 "What is your role and which department do you lead?"
   Reject: finance department roles (they belong in Screener A)

Notice that Screener B accepts any approval method on Q3. The mix of people approving by email, in a tool and on paper is useful variety, because the new flow has to work for someone coming from each habit.

Which extra participants should you add to a round?

Two additions are worth their cost on most consumer tests. The first is a second distinct user type when one exists, as above. The second is people with access needs or lower confidence with technology. The GOV.UK guidance says to do research with people who "have limited digital skills or poor literacy" and with disabled people, and points out that "Some people with access needs do not identify as having a disability", so a screener should ask about specific difficulties such as reading small text, not about a diagnosis. It also recommends allowing up to a month to find disabled participants or assistive technology users through an agency, so start that part of the recruit first.

For participants who are less confident with technology, offer a ten-minute setup call a day or two before the session. Use it to check three things: they can join the video call, they can share their screen, and the prototype link opens on the device they will use. That call protects the first ten minutes of your real session, which is otherwise where a nervous participant's confidence drains away.

What goes on the incentive and reminder checklist?

Recruiting is not done when five people say yes. Nielsen's survey of 201 usability professionals, in Recruiting Test Participants for Usability Studies, found an average no-show rate of 11%, about one in nine participants. That survey dates from 2003, so treat the figure as a rough baseline rather than a current rate. It still supports the practical rule of booking one spare for every five sessions. Run through this checklist for every round:

• Incentive amount agreed and appropriate for the participant type and session length.
• Incentive given at the start of the session, as the GOV.UK Service Manual recommends.
• Rule agreed for poor fits: Hoa Loranger at Nielsen Norman Group advises that "it's best to pay the participant the entire incentive" even when someone turns out not to match the profile.
• One spare booked per five sessions, or a floater on standby.
• Confirmation sent immediately with date, time zone, length, link and what they will need.
• Information sheet and consent form sent before the session.
• Reminder sent the day before, asking for a reply to confirm.
• Reminder sent one hour before, with the link again.
• Setup call booked for anyone flagged on the device question.
• Screener answers stored separately from session notes and deleted once the round is over.

Nielsen's own explanation for paying properly is blunt: no-shows are "one of the main reasons I recommend paying fairly generous incentives to test participants."

What can a screener not tell you?

A screener is self-reported, so it filters out obvious mismatches and honest mistakes but cannot verify anything. Someone can still describe a grocery order they never placed. The defence is a warm-up question at the start of every session that asks for the same story in more detail. If the participant cannot describe their last order with any specifics, run the session anyway (you have paid for it), but mark it in your notes and weigh their results accordingly when you analyse.

The other limit is that a screener only finds people who answer screeners. Recruiting through your own product's users, a panel or an agency each over-represents someone. The GOV.UK guidance's answer is to vary recruitment channels across rounds, which is also the only practical fix for a team running small rounds every few weeks.

What happens after the screener?

Once five people are booked, write the session guide. The usability test script template continues this grocery checkout example with the introduction script, warm-up questions that double as the screener check above, and the task scenarios. After that, how to moderate a usability test covers the session itself, and how to write a usability test report covers what stakeholders receive. If you plan to code the recordings with an assistant afterwards, AI for usability test analysis explains the code list to prepare before session one.

Builders Camp runs live and self-paced bootcamps in product management and AI product building. See the Usability Testing for Product Managers bootcamp for the next cohort and the self-paced version.