-
Notifications
You must be signed in to change notification settings - Fork 3
User research study process
Planning research often takes longer than you expect and is a crucial step for making the most of users’ time once you do sit down with them. It’s not unusual for research planning to take a week or two of dedicated time, and it often includes many conversations about goals, expectations, and logistics. Your planning will include creating a study plan and script with your research team, recruiting participants, and preparing research materials.
Any GitHub issues associated with the planning, carrying out, or analyzing of a user research project should get the label "User research".
Designate the project manager. Unless there are extenuating circumstances, the project manager will be one of the UX designers.
Based on the project scope and timeline, the project manager will then figure out who will be participating in research, and work with them on the plan and script.
If any research-team participants are new to research, make time to walk them through the plan and script so they can ask questions and understand what to expect. Set expectations for what good research notetaking is and talk about how to work together and avoid stepping on toes during notetaking. The notetaking section of this page provides more details on this.
Project manager will select the bias reduction activity and make sure all team members involved in any aspect of the research participate.
- Decide on the goals of the study
- Decide on the right method and participants needed to meet the goals
- Write a study plan (examples in research wiki).
- Parts of the plan include: Test assets, Background & scope, Timeline, Goals, Method, Desired participants, User types, Recruitment strategy, Efforts towards inclusive research, Interview Guide
- This document should make these things clear:
- What sites, pages, or features are you asking about?
- What big questions are you trying to answer?
- What method are you using? Is it generative research? Stakeholder interviews? Usability testing?
- How many people are you hoping to talk to? This can be a range, but should help set expectations for scale. Most of our research rounds have included either 3-5 or 5-8 sessions. (Resource: Why You Only Need to Test with 5 Users). Keep in mind that more participants does not just mean more interviews, but more notes to collate and analyze. Consider this when setting the timeline.
- How will you find participants? What’s the plan for recruiting them? What connections can you use to recruit participants?
- When do you know you’re done with this round/sprint of research?
- What should this research help you do?
- What is everyone’s current workload? The study timeline can be shorter or longer, depending on the team members’ workload outside of the study.
- What will each member of the research team be responsible for? We generally assign one person to handle all aspects of each of the 3 major parts: recruitment, interviews, consolidation and analysis.
You can find sample scripts in each sprint folder in this Research branch. Most research scripts include these sections:
- Introduction
- Remind them that this is not a test of them, and there are no wrong answers.
- Remind them that this is voluntary, and they can call it at any point if they no longer want to participate.
- Consent to participate in research
- Ask if the participant has read the User Research Participant Guide and ensure that the participant can access the guide. If they haven’t, give them a few minutes to read it and make sure they don’t have any questions. Then get verbal consent to participate in the study.
- What questions to ask
- Potential follow-up questions
- Concluding thanks, questions, or invitations to follow up
- Find people using method decided in the plan
- Track people in the Research Participant Pool (People tab) and individual study recruitment spreadsheet (only direct team can access because of PII)
- Send emails asking people to participate
- Choose an interview time for people who accept
- If possible – give the participant a date range and let them choose their day/time. Avoid sending them specific times to choose from until they have narrowed down the day/time.
- Send a meeting request for interview and debrief
- Include this language in your meeting request: “Thank you so much for taking the time to talk with us. If this time doesn't work for you, feel free to propose a different time. Please read our User Research Participant Guide before speaking with us and we are happy to answer any questions. We will be using Teams for this meeting. If you have trouble getting in, feel free to call NAME HERE at XXX-XXX-XXXX, and she can walk you through it. Please plan to be at a computer to share your screen with us. Also, please let us know if you need any accommodations. We look forward to talking with you!"
- Send debrief invite to ODDD team members
- Schedule 30min at the end of each interview day for notes consolidation. This is a working meeting for interviewer and notetakers to enter their notes into the consolidated sheet while it’s fresh and have time to ask each other questions while doing so.
For some research methods, you’ll need to prepare something concrete to put in front of users. These materials should be identified in the research plan, so the team can be ready with the design and functionality you’re hoping to test. You’ll also need a plan to publish or share the prototype with users (in the past, we've used everything from paper print-outs to the preview site). When possible, we use the live site.
Research sessions can be energy vampires for the interviewers and note-takers. Try to avoid this by:
- Make sure the research team can set aside time before interviews to mentally prepare and time after to decompress.
- When possible, arrange for other ODDD members (not on the research team) to cover smaller tasks, time-sensitive tasks on interview days.
- Don’t under-assign the numeric rating to interview issues in GitHub – interviews include prep time and background work – they take far more work than that assigned interview time implies!
For each session, you’ll need:
- Interviewer
- Notetaker
- Location (physical or virtual)
- Participant
Resource: My Best Advice for Conducting User Interviews
- Spreadsheet of participants, with codenames and columns to track whether they’ve been contacted, interviewed, thanked, etc. Remember, this is Personally Identifiable Information (or PII): make sure this data is stored someplace appropriate (NOT GitHub), and only the core research team has access.
- Folder for research notes, with blank/template notes documents
- Never use real names in these documents
- Make sure permissions are restricted to core research team
- Don’t try to intersperse notes with the interview script, because it’s confusing (you can include the script for reference, but it’s much easier to have a blank “notes” section at the bottom of the doc for notetakers to work in)
- Invitation email (or other recruiting materials) explaining what the research is about and what to expect. Make sure your invitation email won't bias the research in unintended ways.
During research sessions, users are doing us a favor. Do whatever you can to ensure that they have a good experience.
- Schedule a little more time than you think you'll need — it can take a few minutes to get everyone on the line, or you may have an especially chatty participant. An extra 15 minutes can keep it from feeling rushed.
- Make sure calendar invites are clear, both to users and your fellow researchers (for PII-protection and clarity, it sometimes makes sense to use separate team-facing and user-facing invites).
- Direct notetakers to the correct docs ahead of time, and make sure they know their role.
- Ask any research team members to show up 2-5 minutes early.
- Test your call-in line, screen sharing tools, or video conference before your first session.
- Confirm with participants that they’ll be able to use conferencing, video, or screen sharing tools, and come up with a backup plan if they can’t.
Introduce everyone on the call and make clear who's taking notes or observing.
- Leave enough time for notetakers or observers to ask follow-up questions at the end.
- Leave time for the interviewee to ask questions.
- Adapt the script as needed to make the interviewee comfortable and be respectful of their time.
- Thank them for their time and be very clear about any follow-up promised.
- Notetakers avoid interrupting interview to give instructions – try to limit participation to asking questions during designated time to do so.
- Observers do not ask questions or verbally participate in the interview in any way – the more people we have speaking up in the interview, the more stress it can put on the interviewee. And it can disrupt the cadence of the interview.
We use detailed note taking in lieu of audio/video recording (mostly for logistical reasons). It's important to align about what kind of notes the team should be taking so that:
- Team members who couldn’t attend the session are able to look at the notes and get a very clear sense of what the user said, did, and felt
- Researchers capture all the details so that when our memories do eventually fade, we can refer to the notes
- Make sure summary is written in a way that can be copied and pasted into consolidated notes cells and others will know what they mean
- DO NOT LIMIT SUMMARY TO QUESTIONS FROM THE INTERVIEW. A lot of good feedback comes from users telling us about things we didn’t ask about. If we don’t put these in the summary, they don’t make it into the analysis
- Review notes again after the debrief to make sure everything made it into the summary. Think of the summary as a more cleaned up version of the notes – not just those deemed most important.
- Review notes after debrief to look for confirmation bias. Make sure the negative feedback makes it into the summary just as much, if not more, than the positive – including direct quotes if they are powerful.
Resources: Tips to capturing the best data from user interviews
- Immediately after the session, set up a few minutes with any research team members who were in the interview.
- If it’s not possible to debrief immediately, do this by the end of the day, or within 24 hours.
- During this debrief, discuss surprises, and reflect on what you heard.
- Identify a few key takeaways in writing, or by highlighting notes. It can be helpful to note these (often quotes from the user) at the top of the notes document.
- Identify the user type and their placement in the user type purpose triangle and expertise matrix. If there is not enough time for this, add it to the end of day note consolidation meeting.
- If you promised the user any follow-up communications, identify who will send them and when.
After you’ve completed all sessions, it’s time to rigorously synthesize what you heard and learned. Synthesis can involve different activities and is up to the person responsible for consolidation and analysis to decide what activities make sense for the study. The activities often include:
- Time for all research participants to collaboratively gather insights from each session into a single document/space (For instance, this might mean pulling each key takeaway onto a sticky note on a Miro board.)
- Some way to track either individual users (e.g. each user gets a different color sticky) or user groups (e.g. each major user group gets a color)
- Grouping, sorting, and discussion:
- You want the research team to work through the big themes, notice which things were heard in multiple sessions, and identify any outliers.
- This is important to do collaboratively for a few reasons: to make this work visible to stakeholders, to build consensus within the team, to harness the perspectives of different listeners, and to correct for biases that can arise (especially if not every research team member was in every interview).
- It helps to have a clear facilitator, and circle back multiple times to make sure the groupings make sense to everyone.
- Naming themes or insights; this forces more clarity in the affinity mapping. Statements or sentences (rather than words/phrases) can help work toward useful content that will become part of a report or summary.
- Avoid duplicating sticky notes or facts if they fit into more than one grouping. Instead, change the grouping to accommodate all the stickies/facts.
- Identify recommendations and/or opportunities based on the insights or themes. Recommendations are actions to take; opportunities are things to think about, monitor, or consider for more future research.
- Avoid including people in synthesis who didn’t participate in any research sessions as an interviewer or notetaker. If you do need to include team members who weren’t able to at least observe research sessions, make sure they understand that their role will be limited to listening and/or asking questions.
Resources:
After collaborative synthesis, a few team members can further boil down findings into a report, presentation, or summary document.
-
This document should cover not just what you learned, but why it matters. What is the impact?
-
Aim to make this document true and useful, rather than comprehensive.
-
All research team members should review this document to make sure it rings true and accurately represents the research as they understood it.
-
Put this document someplace where future team members can find it! (We usually post reports in this research branch, organized by sprint.)
-
Update the User Type diagram
-
Update the Research Participant Pool with new information from the study.
- After each round of research, the whole team — led by the product manager — should identify how the research changes the next sprint’s work. This could include new bugs identified, new features to explore, or a different design focus. Depending on the study, the entire team might also be involved in prioritizing the recommendations, or that might be done by the study team and presented to the rest of ODDD for input.
- Post study guide and findings on GitHub in the Research repository.
- Hold a Retro to reflect on how the study went and how we can best apply lessons learned. Save notes from the retro in the study folder so they can be reviewed prior to planning the next study.