Check best practices for a usability study

[mitchellevan]

Draft questions for participant interview

Hi [NAME],

Happy new year! May I schedule an hour with you in January? It could be any time during work hours or outside of work hours, up to you.

To help me prepare for the call, please answer these five questions, and optionally look at the further questions below.

1. What languages do you understand?

2. What do you consider your home language and dialect?

3. Which computer and mobile operating systems, browsers, and assistive technologies do you use? What are their language settings? If you’re not sure about the settings we can check on the call.

4. Before our call I will send you a web link, and I’ll ask you to browse the page during our call and discuss it with me. With this in mind, would prefer Zoom, Google Meet, Microsoft Teams, or a different platform for our call?

5. Is there anything else you’d like me to know, to help us communicate?

The remaining items below I would normally just share in writing or verbally as we go. However, since you’re an accessibility professional and one of my first participants, I would welcome your feedback on the content for my calls. Is there anything you would do differently?

When I schedule the call, I will send sample web pages like these:
España - Español
Antigua and Barbuda - English
(In these sample pages I’m striving for technically correct coding of realistic content, while testing the limits of what’s supported in user agents. If a participant tells me in advance that they know a less common language, before the call I might add a phrase or whole page in that language. This helps me uncover the technological limits in my controlled tests, and it might also reveal other kinds of problems that I had not thought of.)

The rest of these questions are for discussion on the call. If I talk to people with assistive technologies other than a screen reader, I will adapt the questions.

1. May I record this call?

2. I’m learning how people use various languages on their devices and websites, especially people with disabilities. Do the languages in the sample web pages work well for you? What problems do you notice? (My hypothesis: Assistive technology rendering of languages varies along several variables. The variables include: the human language; language switching; labels and other programmatic structures; the participant’s user agents and settings, especially which languages are installed in the OS. Problems include: pronunciation quality; applying the pronunciation rules of a different language; failures to identify the content language. Exploratory: behaviors with a braille display.)

3. Do you have any tricks for working around problems like these? (Hypotheses: For short phrases, some users slow it down and manage to understand it with the wrong pronunciation. For longer phrases and whole pages, when language detection or switching fails, some people manually force their screen reader into a different language.)

4. Let’s review and confirm your language settings. (I’ll ask the participant to confirm the UI language of their operating systems, browsers, and assistive technologies; their screen reader synthesizer languages or similar; their input methods, including keyboard languages and speech-to-text. I’ll ask them to tell me their preferred language(s) for web pages — I’ll detect and display this on a page, but I don’t want to log it automatically.)

(If there’s time on the call, I’ll repeat the above questions for desktop or mobile. Otherwise I’ll leave the second platform for a written follow-up.)

5. Do you sometimes send messages or posts in different languages? Are there problems? Do you have tricks for working around the problems? (Hypotheses: problems with spell checking; speech to text input; AT reading back the user’s entry; apps and websites not correctly handling the character sets.)

6. Do you use video subtitles or captions? Are there any language limitations? (This question is exploratory. Depending on the participant, I’ll steer toward the EN 301 549 requirement for spoken interlingual subtitles, or toward access for the Deaf and hard-of-hearing. This is principally about watching prerecorded and live videos, but I’m also open to learning about video content creation. Hypothesis: Screen readers don’t consistently announce subtitles. Exploratory: learning more about real-world use cases or unmet needs.)

7. Thinking about the techniques we’ve covered, how easy or difficult is it for you to use your languages online?

8. Can you introduce me to anybody else, who might use assistive technology with more than one language?

9. When I publish my results, may I give your real name as a participant? (Otherwise I’ll use just a code name, like “M1.”) Would you like to share a link or contact info?

10. Do you have any questions or suggestions for me?

Process questions

• Stipend
• Intellectual property: (avoiding) trademark; copyleft/copyright
• Participants who are Vispero-internal or not
• Academia

[mitchellevan]

Hi @sloandr @davidofyork, this is for CSUN 2025. I'm planning on one-on-one in-person and remote interviews with around five participants. Would you please take a look at this proposed interview script, and let me know if there's anything I might need to do differently? A particular concern is to make sure I'm respecting people's privacy.

[mitchellevan]

TPGi internal call schedued for 2025-01-07