Qubes Journalist Workstation Testing: 2nd Round (Q4 2018)
- 2/3 of participants were estimated to be between 36 and 75 years old.
- Most participants were white, with 1 European PoC, and 1 American PoC. Gender appears about half-and-half along the binary.
- Increased attention is desired for future studies to more aggressively recruit from marginalized populations, and from journalists working with organizations outside North America.
- 17% of participants work outside North America.
- 6 folks that have never used SecureDrop, 12 existing users.
- 4 freelance journalists, and 14 newsroom staffers from 7 newsrooms
- Among the 7 newsrooms using SecureDrop, each has trained an average of 4.3 journalists, but only an average of 1.5 are regularly "willing to use SecureDrop."
- 4 of the 7 newsrooms have journalists across multiple offices using or needing SecureDrop resources.
- Every-other-week appears to be the average rate at which participants check their current SDs.
- Two newsrooms check very infrequently, and the rest between every few days and every day.
- More About Testing Participants in more detail (see Participants tabs at bottom of GSheet).
-
Spreadsheet tracking flows, features, and functionality being looked at.
- Detailed findings and insights are in the Testing Insights Matrix tab.
- Insights collaboratively compiled between Nina and Erik, after each session—beginning with p12; earlier participants, ~75% of things put into spreadsheet.
- Fancy charts up above are from more detailed info on test participants to date, on the second tab of the spreadsheet
- Detailed findings and insights are in the Testing Insights Matrix tab.
2 participants (including last week's)
- Both +40
- Both self-described "experts"
- Both trained in security things, one existing SD user
- Both staff IJs w/ lengthy tenures in journalism
Top shared findings:
- Both participants expected a new window to pop-up to compose replies in
- Neither participants cited likeness to messaging paradigm, but both recognized email paradigm
- Both delighted by use of disposable VMs for viewing documents
- Both confused by "Detaching Drive" Qubes dialog, that followed "Attaching Drive" dialog—but then successfully continued in the export task with the automatically opened "Save As" window.
- Sidenote: Hypothesizing this could be very confusing for users when not in the middle of an export task; if simply attaching a drive, what will they be left thinking after seeing that "Detachment" message?
First participant highlights
.Top Findings:
- Participant seemed to be pressed for time and somewhat distracted
- "(current system kvetches)... if this is any faster, I'll have a party at my house!"
- Inquired about a "Download All" button for files
- "there’s never a situation where I don’t want to download (any files)."
- "Oh, another VM for the file. Exciting!"
- Confused by Qubes "Detatching Device" message that follows "Attaching Device," but moved on to save file via new save-as window, anyway.
Second participant highlights
.Top Findings:
- Lots of room for disorientation in the viewer applications w/in the disposable VM
- Encryption to self a nice to have
- Found #3 confusing in export instructions
- Didn't spot status bar; could be visibility issue, or just static prototype issue
- Was interested in scanning metadata to learn about file before opening
- Printing as secure sharing (so recipient could not re-forward digitally)
- Confused by Qubes "Detatching Device" message that follows "Attaching Device," but moved on to save file via new save-as window, anyway.
-
Discussion Guide w/ Prototype Links
- Studio prototype skipped, Craft prototype refactored accordingly:
- 1/3 size reduction
- Font sizes all increased to facilitate remote-testing needs (vs reflecting actual client design interests)
- All objects updated to match optimised library items, to improve performance
- Published to support in-browser scaling
- Debugged w/ help from Erik :)
- Studio prototype skipped, Craft prototype refactored accordingly:
- 1 Participant
NO FINDINGS: Prototype(s) buggy, participant offered to reschedule
- Participant had established repor w/ FPF folks via past trainings
- Prototype files had been fully refactored to address issues from prior week; despite that, one minor bug proved to be a roadblock.
- Tools capability to mimic Qubes dropdown/flyout in connecting a USB drive, proving very difficult to accurately reproduce in Nina's WYSIWYG prototyping tools (yep, plural).
- Nina/Erik delayed further scheduling for next week, to prioritize fully smoothing-out prototypes this week.
- Staff editor
- 10-15yrs as a journo
- Tech fluence: Expert: I'm comfortable writing code or taking things apart to see how they work
- Two-Factor Authentication, PGP/GPG encryption, Full-disk encryption, GitHub, Signal, Tor, Windows, mac OS Familiar with SD
- Discussion Guide w/ Prototype Links
- 4 Participants
- None, existing SecureDrop Users
-
First user
- From South Africa but currently in NA
- Freelancer
- Not familiar with VMs.
- Uses WhatsApp for encrypted comms.
- Only just heard about SD through survey.
-
Second user
- In South America
- Linux user
- Familiar with VMs, Encryption, etc.; Whonix
- Not highly technical
- Uses Mailvelope for encrypted comms.
- Familiar with SD and had previously spoken with Micah about possibly getting it for his newsroom.
-
Third user
- In North America
- Linux user
- Familiar with VMs, Mac user
- Moderately technical
- Reporter for a local corp-owned newsroom; not much IJ experience, it seemed (didn't ask).
- Prototype performance issues with this one.
Top Findings
.3rd Participant
- I was surprised at how little encryption seemed to be happening"
- "Interface felt sleek and tidy"
- Need to resize to fit prototype better in browser window (waaay shrink!)
- Overall strong concidence in system
- Positive iteration on attach wizard; an improvement. Possible TMI for a single screen?
- For later refinement of app: Messaging in "Save As" should be smarter, more caveating around this stuff; to revisit once braoder app more resolved.
2nd & 1st Participants
- Instructions— could refine wrt #2
- BACKUP PLAN!! Must always have one, be more careful to not over-write older versions of prototypes.
- Font sizing needs to be more easily legible for users for prototype, while full frame fitting into window better; bigger to-do
- Export flow remains troublesome; second user (ESL) REALLY struggled
- Paper airplane button (knockout on the SD qube) "is obtuse"
- How much are we evoking Email vs Chat?
- Findings roughed-out in "Testing Insights Matrix" tab of spreadsheet, yet to cut/paste quotes. Noteworthy moments are summarized in awful all-caps, though.
- Discussion Guide w/ Prototype Links (same as prior session; new prototype is still too buggy to test from).
- 2 Participants
- Both "Somewhat Comfortable," older users
- Neither, existing SecureDrop Users
Top Findings
-
- Export wizard remains stumbly
- Unchanged message at top, odd
- Absent functionality (in prototype) to execute pairing, confusing
- Both know about 2FA (via banking apps)
- Both interpreted "Safe USB" drive well
- Both were unfamiliar with encryption, but guessed what it was well enough
- Slight confusion with mixed messaging/email paradigm
- First user was entirely unable to find "Compose Reply to Benign Artichoke" block
- Graphic of paper airplane against 2-hue Qube, identified as 'unclear'
- No immediate action on this, tbd
- Good to include mention of "non-profit/foss" in intro-blurb to non-users
- Should develop a contingency plan for users with sloooooowwwwwww connections
- 2 users, 2 newsrooms
-
Invision prototype that demonstrates time-based activity messages and download/decryption
- Click desktop icon
- Click sign-in button
- Observe: activity messaging in top-left of screen, and contextually where relevant
- Click: Benign Artichoke's 6.4mb file, to see download/decrypt. Entire bar is supposed to be clickable, at the start.
-
Invision prototype that demonstrates view and export flows
- Benign Artichoke's 6.4mb file may be viewed. Window currently only close'able via VM window's "x" button
- Same file may be exported.
- Click on the graphic of the fly-out menu in the wizard, to fudge the system automatically detecting an attached drive.
- If you click "Next" that assumes the user failed at attaching their drive.
- 5 users, 3 newsrooms
- Focus on Online/Offline, Sign In, and overall comprehension of what's being looked at
- Export & Save functionality, introduced with last participant
-
Invision prototype
- Note: at end of discussion guide below, nav-paths are outlined.
- Primary updates to this prototype were addition of activity messaging and attach-USB wizard(ish)
- Updated discussion guide
- Prototype Updates
- Add toast messaging for opening disp-VMs and delete
- Refinements to Attach USB wizard(ish), including pings from OS wrt attached drive status
- Update ordering-by-chronology and mix-in responded-to messages in Beta screen
- Add "Seen By" in Beta screen
- Update available "Export" window functionality
- Add Refresh-bar messaging wrt downloading/decrypting, and grey bars over preview text while things decrypt
*Running Themes Addressed
- Problem: User expectations management remains incomplete
- Iterative Solution: Add more toast messages where appropriate—confirm delete action, and what was deleted; confirm downloaded files?; when opening disposable VMs (see above), etc.
- Problem: Users confused with regards to what is encrypted/decrypted and downloaded, and when.
- Iterative Solution: Update "Refresh" text throughout processes to better communicate to users what's going on and when
- When downloading messages from server
- When decrypting messages
- Possibly to tweak "Last Refresh" details to include decryption/download language
- Iterative Solution: To discuss with Jen, timing/lag wrt decrypting message text upon load—if it makes sense, will add grey-bars indicating activity where preview text should be in Sources List pane.
- Iterative Solution: Update "Refresh" text throughout processes to better communicate to users what's going on and when
- Problem: Users confused by many things upon entering prototype; "I see read/unread but don't remember having seen those Sources before," when do I take files to the SVS, etc.
- Iterative Solution: Update framing of prototype/session in script.
-
Findings Report (all testing to date, including from this sprint)
- 2 users, same newsroom
-
Invision prototype note: linked prototype includes file viewing/exporting functionality that was not shown or reviewed in research sessions
- Remove re-authentication when a user signs-in, goes offline, then goes back online
- Updated read/unread pattern to load first-view upon sign-in with selected message being the most recent "Read," with all unreads shown above
- Tweaked to language on files in all states
- Tweaked language on offline/online banner to improve comprehension and actionability
- Planning
- Research plan
- Discussion guide & nav path outline
- Invision prototype
- Online location (Erik's GMeet—links vary with participant invitations)
Findings & Action Items
-
Hi/lo observations from testing, below. Detailed findings are in the linked GDoc; fully anonymized, and open for comments.
- Not clear. Most significant actionable issue between both participants.
- Subtlety in colors, italicization, etc., totally washed-over both users' heads
- Combined with confusion around encryption/decryption status of shown content, both emerged in parallel as highest priorities for next iteration.
- Seeing unread messages in Offline mode, confusing—as workflows unlikely to reproduce in reality.
- Next Iteration Action(s):
- 2-state Offline/Online banner w/ "Go Online" and "Last Refresh" and a hard-refresh button, include
- Interstitial sign-in pane between clicking the app icon and initial client view; Erik's ASCII sketch for reference
- Remove "Unread" messages from initial view in Offline mode
- Need messaging to confirm successful action completion, and where actions took place (local, server, both)
- "Conversations" concept confusing (Note: bucket nested within Sources list, to separate newsroom-reciprocated correspondence from un-reciprocated submissions)
- Timestamp(s) in messaging pane too small/pale
- Discoverability issues with "Reply To" block
- Pointers on message bubbles, confusing
- Language: "Export" more functionally clear than "Save"
- Attached file things generally clear, some bumps to smooth-out
- Idea validated that "Pick things to delete" as a dropdown item on a single Source, would be useful.
- Next Iteration Action(s):
- Include toast (transient notification) messaging—initially, for "Delete" flow, and with explicit messaging
- Update "Delete Source" language to "Delete Account"
- Remove "Conversations" bucket (currently nested within Source's list)
- Simplify read/unread paradigm, with removed "offline" state and "Conversations"
- Change all uses of "Save" to "Export"
- Add Download icon, and omit word "Decrypt" in initial view—but note decryption happening in-progress.
- Include "Latest Action" verbiage on messaging pane timestamp, and improve visible discovery
- Include big paper-airplane-like button in "Reply To" field
- Kill the bubble pointers, make less bubbly-looking
- Research plan
- Discussion guide & Nav Path Outline
- Prototype
- Online location (Erik's GMeet—links vary with participant invitations)
- Existing customers, Erik handling
- Correspondence messaging
-
Kickoff
- Meet to revisit goals, methods, and timing
- Shift goals for this iteration?
- Things to do differently?
- Different users to recruit?
- Meet to revisit goals, methods, and timing
-
Preparation Communication Train
- New Moderator/Note-taker pairs? Anyone interested in observation?
- Schedule session blocks
- Outreach to prospective Participants
- Add new participants to spreadsheets
- GCal invites to Participants, Moderator(s), Note-taker(s), and optional Observer(s)
- Follow-up Moderator introduction email
-
Session Preparation
- Updates to Research Plan?
- New Session Outline
- Update Discussion Guide
- New Materials Assembly (prototype, cards, etc)
- Internal meeting to communicate prototype updates & new interest points w/ pairs
- Per-user, customized/password-protected prototypes generated?
- Conduct sessions
- Post-session synthesis
-
Wrapping-Up
- Complete findings/synthesis synopsis; post to wiki
- Compose dev/design action-items; share-out where appropriate (as GitHub Issue or in UxD wiki)
- Post Template Prototype to wiki
- Post Discussion Guide, Research Plan, Session Outline to wiki
- Everything above: final artifacts attached here as links?
- Write/Send thank you notes to Participants
- Get rid of this damn checklist
- Present/debrief findings to team
- Post findings to Gitter and Forums
-
Kickoff
- Create wiki page
- Meet to discuss goals, methods, and timing
- One-off study, or a series?
- Timing of synthesis/findings/iteration cycles
- How many sessions per iteration: where will the value be?
-
Preparation Communication Train
- Create Study Spreadsheet to track all participants
- Internal outreach to establish Moderator/Note-taker pairs, and to solicit interest in optional observers
- Schedule session blocks
- Initial outreach to prospective Participants
- GCal invites to Participants, Moderator(s), Note-taker(s), and optional Observer(s)
- Follow-up Moderator introduction email
-
Session Preparation
- Create wiki page
- Begin Research Plan
- Begin Session Outline
- Begin Discussion Guide
- Begin Materials Assembly (prototype, cards, etc)
- Complete Plan & Outline
- Internal session dry-run
- Complete Discussion Guide + Template Prototype
- Per-user, customized/password-protected prototypes generated?
- Conduct sessions
- Post-session synthesis
-
Wrapping-Up
- Complete findings/synthesis synopsis; post to wiki
- Compose dev/design action-items; share-out where appropriate (as GitHub Issue or in UxD wiki)
- Post Template Prototype to wiki
- Post Discussion Guide, Research Plan, Session Outline to wiki
- Everything above: final artifacts attached here as links?
- Write/Send thank you notes to Participants
- Get rid of this damn checklist
- Present/debrief findings to team
- Post findings to Gitter and Forums