Skip to content

Add CTA experiments #675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jan 23, 2020
Merged

Add CTA experiments #675

merged 6 commits into from
Jan 23, 2020
+165 −111

Conversation

@subsymbolic
Copy link
Contributor

@subsymbolic subsymbolic commented Jan 23, 2020 •
edited
Loading

Task/Issue URL: https://app.asana.com/0/361428290920652/1158110040574203
Tech Design URL: N/A

Description:
Adds CTA retention experiments to help up determine effectiveness of CTAa

Steps to test this PR:

Check CTAs are shown for control:

  1. Open VariantManager.kt and comment out all variants other than mq control
  2. Run a fresh installation of the app and launch it to the welcome screen
  3. Ensure the m_odb_v pixel has NOT fired yet
  4. Press Continue
  5. Ensure the the Make DuckDuckGo Your Default onboarding is shown and the m_odb_v pixel now fires
  6. Select Maybe later to get to the home screen
  7. Ensure the widget CTA shows

Check that only widget CTA is shown on high end device and triggers correct pixels when default browser CTA feature is suppressed:

  1. Open VariantManager.kt and comment out all variants other than mr
  2. Launch the app on an API 26 above emulator or device that supports widget pinning
  3. Run a fresh installation of the app and launch it to the welcome screen, press Start Browsing
  4. Ensure the widget CTA shows and the m_wc_s pixel fires
  5. Press Add Widget and ensure the m_wc_l pixel fires
  6. Press Add Automatically and ensure the CTA is gone
    --
  7. Reset the app and run through the previous steps however this time select Dismiss at step 4
  8. Ensure the m_wc_d pixel fires
  9. Ensure the Add widget CTA is now gone

Check that only widget CTA is shown on low end device and triggers correct pixels when default browser CTA feature is suppressed:

  1. Open VariantManager.kt and comment out all variants other than mr
  2. Launch the app on an API 22-25 emulator or device
  3. Run a fresh installation of the app and launch it to the welcome screen, press Start Browsing
  4. Ensure the widget CTA shows and the m_wlc_s pixel fires
  5. Press Show Me and ensure the m_wlc_l pixel fires
  6. Press Go to home screen and add the widget to the home screen
  7. Relaunch the app, close the info screen and ensure the CTA is gone
    --
  8. Reset the app and run through previous steps 1-4 however this time select Dismiss at step 4
  9. Ensure the m_wlc_d pixel fires
  10. Ensure the CTA is now gone

Check only default browser CTA shown when widget CTA feature suppressed:

  1. Open VariantManager.kt and comment out all variants other than ms
  2. Run a fresh installation of the app and launch it to the welcome screen
  3. Ensure the m_odb_v pixel has NOT fired yet
  4. Press Continue
  5. Ensure the the Make DuckDuckGo your default onboarding is shown and the m_odb_v pixel now fires
  6. Select Maybe later to get to the home screen
  7. Ensure the widget CTA is not shown

Check that no CTA shown when they are suppressed:

  1. Open VariantManager.kt and comment out all variants other than mt
  2. Run a fresh installation of the app and launch it to the welcome screen and press press Start Browsing
  3. Note that no CTAs were displayed
Internal references:

Software Engineering Expectations
Technical Design Template

@subsymbolic
Add CTA experiment
b5f4050
@subsymbolic
Merge branch 'develop' into feature/mia/cta_experiment
9936bac 
# Conflicts:
#	app/src/androidTest/java/com/duckduckgo/app/cta/ui/CtaViewModelTest.kt
@subsymbolic
Update to new method name after merge
59c2ca0
marcosholgado
marcosholgado approved these changes Jan 23, 2020
View reviewed changes
Copy link
Contributor

@marcosholgado marcosholgado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! Really liked the way you managed the different variants/features and very good addition only sending the v pixel when the fragment is actually visible. On that front I left a comment but it doesn't block the PR. Nice one!

app/src/main/java/com/duckduckgo/app/onboarding/ui/page/DefaultBrowserPage.kt
super.onAttach(context)
}

override fun setUserVisibleHint(isVisibleToUser: Boolean) {
Copy link
Contributor

@marcosholgado marcosholgado Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are still not using the latest version of fragment-ktx but this method has been deprecated in 1.1.0 (https://developer.android.com/jetpack/androidx/releases/fragment). Probably not something for this PR but maybe create a LHF task?

Copy link
Contributor Author

@subsymbolic subsymbolic Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, internal task created here https://app.asana.com/0/414730916066338/1158589410650834

cmonfortep
cmonfortep reviewed Jan 23, 2020
View reviewed changes
app/src/androidTest/java/com/duckduckgo/app/cta/ui/CtaViewModelTest.kt Outdated
@Test
fun whenRefreshCtaOnNewTabThenReturnDaxIntroCta() = runBlockingTest {
setConceptTestVariant()
fun whenRefreshCtaOnNewTabAndConceptTestFeatureActiveThenReturnDaxIntroCta() = runBlockingTest {
Copy link
Contributor

@cmonfortep cmonfortep Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @subsymbolic, #673 gave you some conflicts here. Happy to help you in solving this later today.

I will still put some context here:

  • we renamed/mapped isNewTab to isBrowserShowing to show a better intention inside our ctaViewModel.refreshCta.
    isNewTab = false equals isBrowserShowing = true
    isNewTab = true equals isBrowserShowing = false
  • testcases names have been affected by the change too:
    whenRefreshCtaOnNewTab -> whenRefreshCtaOnHomeTab
    whenRefreshCtaOnExistingTab -> whenRefreshCtaWhileBrowsing

Copy link
Contributor Author

@subsymbolic subsymbolic Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@subsymbolic
Merge branch 'develop' into feature/mia/cta_experiment
58be872 
# Conflicts:
#	app/src/androidTest/java/com/duckduckgo/app/cta/ui/CtaViewModelTest.kt
@subsymbolic subsymbolic merged commit 0eb8e9e into develop Jan 23, 2020
@subsymbolic subsymbolic deleted the feature/mia/cta_experiment branch January 23, 2020 20:12
@subsymbolic
Copy link
Contributor Author

subsymbolic commented Jan 23, 2020

Thanks for the review @marcosholgado

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cmonfortep cmonfortep cmonfortep left review comments

@marcosholgado marcosholgado marcosholgado approved these changes

@CDRussell CDRussell Awaiting requested review from CDRussell

Assignees

@marcosholgado marcosholgado

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@subsymbolic @marcosholgado @cmonfortep