Skip to content

Onboarding Design Experiment Setup #6525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Aug 11, 2025
Merged

Onboarding Design Experiment Setup #6525

merged 15 commits into from
Aug 11, 2025

Conversation

@mikescamell
Copy link
Contributor

@mikescamell mikescamell commented Aug 5, 2025 •
edited
Loading

Task/Issue URL: https://app.asana.com/1/137249556945/project/1207908166761516/task/1210983806890977?focus=true

Description

This PR refactors the onboarding design experiment implementation by introducing a new OnboardingDesignExperimentManager that centralizes experiment-related logic. The manager handles enrollment, cohort determination, and firing experiment-specific pixels for tracking user interactions throughout the onboarding flow.

Key changes include:

  • Added a data store to track site and SERP visit counts
  • Implemented comprehensive pixel tracking for onboarding interactions
  • Replaced direct toggle checks with manager methods for better abstraction
  • Added support for tracking user interactions with onboarding dialogs and options

Steps to test this PR

Onboarding Experience

  • Verify new users see the appropriate onboarding experience based on experiment cohort
  • Confirm that navigating through onboarding flows works correctly
  • Test that site suggestions and search options function properly
  • Verify fire animations display correctly based on experiment cohort

Pixels

Use Logcat to check for Pixels firing.

When there are multiple options you'll need to clear data and get back to the same position.

Variant – Modified Control

Filter logs using: experiment_metrics_onboardingDesignExperimentAug25_modifiedControl

  • Intro screen shown → introScreenDisplayed

  • Comparison screen shown → comparisonScreenDisplayed

  • Tap “Choose Browser” → chooseBrowser

  • Accept system Set as default dialog → setDefaultRate

  • Address-bar-position sheet displayed → setAddressBarDisplayed

    • Select Top → addressBarSetTop
    • Select Bottom → addressBarSetBottom

  • “Try a search” bubble appears → tryASearchDisplayed

    • Tap first suggestion → firstSearchSuggestion
    • Tap second suggestion → secondSearchSuggestion
    • Tap third suggestion → thirdSearchSuggestion
    • Enter any other query → searchOrNavCustom

  • SERP message dialog appears → messageOnSerpDisplayed

  • Site-suggestion bubble (adjacent) appears → visitSitePromptDisplayedAdjacent

    • Tap first site → firstSiteSuggestion
    • Tap second site → secondSiteSuggestion
    • Tap third site → thirdSiteSuggestion

  • (Clear data, get back to "Try a search" dialog, dismiss and then open a new tab) Site-suggestion bubble appears → visitSitePromptDisplayedNewTab

    • Tap first site → firstSiteSuggestion
    • Tap second site → secondSiteSuggestion
    • Tap third site → thirdSiteSuggestion

  • Trackers-blocked dialog shown → trackersBlockedMessageDisplayed

  • No-trackers dialog shown → noTrackersMessageDisplayed

  • Tracker-network dialog shown → trackerNetworkMessageDisplayed

  • From a blockers dialog, tap privacy-shield → privacyDashClickedFromOnboarding

  • Fire-button education prompt displayed → fireButtonPromptDisplayed

    • Trigger Fire action from that prompt → fireButtonClickedFromOnboarding

  • End-of-journey screen shown → finalOnboardingScreenDisplayed

  • Second non-DDG site loads → secondSiteVisit

  • Second DDG SERP loads → secondSerpVisit

Variant – Buck

Filter logs using: experiment_metrics_onboardingDesignExperimentAug25_buck

  • Intro screen shown → introScreenDisplayed

  • Comparison screen shown → comparisonScreenDisplayed

  • Tap “Choose Browser” → chooseBrowser

  • Accept system Set as default dialog → setDefaultRate

  • Address-bar-position sheet displayed → setAddressBarDisplayed

    • Select Top → addressBarSetTop
    • Select Bottom → addressBarSetBottom

  • “Try a search” bubble appears → tryASearchDisplayed

    • Tap first suggestion → firstSearchSuggestion
    • Tap second suggestion → secondSearchSuggestion
    • Tap third suggestion → thirdSearchSuggestion
    • Enter any other query → searchOrNavCustom

  • SERP message dialog appears → messageOnSerpDisplayed

  • Site-suggestion bubble (adjacent) appears → visitSitePromptDisplayedAdjacent

    • Tap first site → firstSiteSuggestion
    • Tap second site → secondSiteSuggestion
    • Tap third site → thirdSiteSuggestion

  • (Clear data, get back to "Try a search" dialog, dismiss and then open a new tab) Site-suggestion bubble appears → visitSitePromptDisplayedNewTab

    • Tap first site → firstSiteSuggestion
    • Tap second site → secondSiteSuggestion
    • Tap third site → thirdSiteSuggestion

  • Trackers-blocked dialog shown → trackersBlockedMessageDisplayed

  • No-trackers dialog shown → noTrackersMessageDisplayed

  • Tracker-network dialog shown → trackerNetworkMessageDisplayed

  • From a blockers dialog, tap privacy-shield → privacyDashClickedFromOnboarding

  • Fire-button education prompt displayed → fireButtonPromptDisplayed

    • Trigger Fire action from that prompt → fireButtonClickedFromOnboarding

  • End-of-journey screen shown → finalOnboardingScreenDisplayed

  • Second non-DDG site loads → secondSiteVisit

  • Second DDG SERP loads → secondSerpVisit

Variant – BB

Filter logs using: experiment_metrics_onboardingDesignExperimentAug25_bb

  • Intro screen shown → introScreenDisplayed

  • Comparison screen shown → comparisonScreenDisplayed

  • Tap “Choose Browser” → chooseBrowser

  • Accept system Set as default dialog → setDefaultRate

  • Address-bar-position sheet displayed → setAddressBarDisplayed

    • Select Top → addressBarSetTop
    • Select Bottom → addressBarSetBottom

  • “Try a search” bubble appears → tryASearchDisplayed

    • Tap first suggestion → firstSearchSuggestion
    • Tap second suggestion → secondSearchSuggestion
    • Tap third suggestion → thirdSearchSuggestion
    • Enter any other query → searchOrNavCustom

  • SERP message dialog appears → messageOnSerpDisplayed

  • Site-suggestion bubble (adjacent) appears → visitSitePromptDisplayedAdjacent

    • Tap first site → firstSiteSuggestion
    • Tap second site → secondSiteSuggestion
    • Tap third site → thirdSiteSuggestion

  • (Clear data, get back to "Try a search" dialog, dismiss and then open a new tab) Site-suggestion bubble appears → visitSitePromptDisplayedNewTab

    • Tap first site → firstSiteSuggestion
    • Tap second site → secondSiteSuggestion
    • Tap third site → thirdSiteSuggestion

  • Trackers-blocked dialog shown → trackersBlockedMessageDisplayed

  • No-trackers dialog shown → noTrackersMessageDisplayed

  • Tracker-network dialog shown → trackerNetworkMessageDisplayed

  • From a blockers dialog, tap privacy-shield → privacyDashClickedFromOnboarding

  • Fire-button education prompt displayed → fireButtonPromptDisplayed

    • Trigger Fire action from that prompt → fireButtonClickedFromOnboarding

  • End-of-journey screen shown → finalOnboardingScreenDisplayed

  • Second non-DDG site loads → secondSiteVisit

  • Second DDG SERP loads → secondSerpVisit

UI changes

No UI changes in this PR - this is an internal refactoring of the experiment implementation.

@mikescamell mikescamell force-pushed the feature/mike/onboarding-test/experiment_setup branch 2 times, most recently from 5df882e to 49ffdf3 Compare August 7, 2025 10:55
@mikescamell mikescamell force-pushed the feature/mike/onboarding-test/experiment_setup branch from 49ffdf3 to 39e4693 Compare August 7, 2025 11:00
@marcosholgado marcosholgado mentioned this pull request Aug 7, 2025
Merged
6 tasks
@mikescamell mikescamell force-pushed the feature/mike/onboarding-test/experiment_setup branch from 39e4693 to 4fc9c46 Compare August 7, 2025 13:14
@marcosholgado marcosholgado force-pushed the feature/mike/onboarding-test/experiment_setup branch from 4fc9c46 to b10b580 Compare August 7, 2025 13:41
@marcosholgado Graphite App
Copy link
Contributor

marcosholgado commented Aug 7, 2025 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

@mikescamell mikescamell force-pushed the feature/mike/onboarding-test/experiment_setup branch from b10b580 to 19c27c6 Compare August 8, 2025 10:15
@marcosholgado marcosholgado force-pushed the feature/mike/onboarding-test/experiment_setup branch from 19c27c6 to 647aaa7 Compare August 8, 2025 15:01
@marcosholgado marcosholgado mentioned this pull request Aug 8, 2025
Merged
2 tasks
@mikescamell
Expand conversion window for app use metrics
cf8bf7a
@mikescamell
Update conversion windows for search metrics to include additional ra…
08adf59 
…nges
@mikescamell
scouting: remove unused onPrivacyShieldSelected() function
2551b12 
This functionality was moved the omnibar last year but never removed from the BrowserTabViewModel, it actually caused me some confusion as I didn't see it fire when pressing the shield so removing this now so others don't have the same problem
@mikescamell
Setup experiment enrolment
cea61a6 
Caching isExperimentEnabled and onboardingDesignExperimentCohort means we don't have to litter UI building callsites with coroutine scopes and launches. It would be a pervasive change and calling this synchronously gives me confidence.

Using both app launch and onPrivacyConfigDownloaded should cover cases where we turn the experiment off.
@mikescamell
Update all callsites with new experiment check functions
3bd11fc
@mikescamell
Add metrics and site/serp count tracking store
23cb039
@mikescamell
Add metric pixel firing to pre-onboarding flow
5a3e06f
@mikescamell
add browser onboarding metrics
dd716de
@mikescamell
add fire animation test for modified control
8d4854b 
want to make sure we have this covered
@mikescamell
add tests for OnboardingDesignExperimentManagerTest
95c22bb
@mikescamell
Fix tests
2603136
@mikescamell
formatting
ce1b7aa
@mikescamell
add onboarding design experiment configuration
12d997c
@mikescamell mikescamell force-pushed the feature/mike/onboarding-test/experiment_setup branch from 647aaa7 to 12d997c Compare August 8, 2025 15:28
@mikescamell mikescamell changed the title [WIP] Onboarding Design Experiment Setup Onboarding Design Experiment Setup Aug 11, 2025
@mikescamell mikescamell marked this pull request as ready for review August 11, 2025 09:27
@mikescamell
fix missing imports
2adb6f5
@mikescamell
Switch to experiment search options
aed4ffb 
Looks like we switched to the experiment search options but forgot to update the code after.

This takes care of that and ensures that our pixel tracking is now correct and does not report as a user making a custom search on the "Try a search" onboarding dialog option
marcosholgado
marcosholgado approved these changes Aug 11, 2025
View reviewed changes
Copy link
Contributor

@marcosholgado marcosholgado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mikescamell mikescamell merged commit ea7f906 into develop Aug 11, 2025
18 of 19 checks passed
@mikescamell mikescamell deleted the feature/mike/onboarding-test/experiment_setup branch August 11, 2025 11:31
@mikescamell mikescamell mentioned this pull request Aug 15, 2025
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcosholgado marcosholgado marcosholgado approved these changes

Assignees

@marcosholgado marcosholgado

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mikescamell @marcosholgado