GF Signup : Experiment Allow Paid Domain redirect For all Domains #80647

ddc22 · 2023-08-15T22:30:09Z

Fixes Automattic/growth-foundations#159

Related to

Experiment : 21362-explat-experiment, 21363-explat-experiment

Proposed Changes

An evolution on the experiment: Add/gated blog domain available for free plan #78825. This adds a plan upsell modal when any custom paid domain is selected.
The .blog based experiment was removed and replaced with the above so control will not allow a plans purchase.
Loads the above experiments on the domain step
Fixes a previous bug related to the free/free experiment by adding a loader in place of the free plan button on onboarding and the free plan link on the onboarding-pm flows.

Typical Testing Path

Go through the main /start or /start/onboarding-pm flow, and pick ~~a .blog domain~~ ANY PAID DOMAIN in the domain step.
In the plans page, it should say " is a redirect"
Clicking the Free plan button, the modal should show the new copy of explaining why a paid plan is required for a custom primary domain.
Make sure both the CTAs work as expected.
Pick a free domain, and it the above copy modal should not be shown (Either the free/free modal or site creation step should be shown depending on the variant that you are assigned to in that experiment).
Make sure relevant loaders are working related to the Link loader int the subtitle and button loader in the grid

Comprehensive experiment testing plan including regression tests

Experiment `calypso_onboardingpm_plans_paid_domain_on_free_plan`

Preconditions

Experiment should be active: 21362-explat-experiment
Should be in onboarding-pm flow
Select any paid domain on the domain step

Event : Click on Start with a free plan link on the page subtitle

Control	Variant

Experiment `calypso_onboarding_plans_paid_domain_on_free_plan`

Preconditions

Experiment should be active: 21363-explat-experiment
Should be in onboarding flow
Select any paid domain on the domain step

Event : Click on the free plan button

Control	Variant

Experiment `calypso_gf_signup_onboarding_pm_free_free_dont_miss_out_modal_v3`

Preconditions

Experiment should be active: 21344-explat-experiment
Should be in onboarding-pm flow
Select the free subdomain on the domain step

Event : Click on Start with a free plan link on the page subtitle

Control	Variant
`Proceed to next step`

Experiment `calypso_gf_signup_onboarding_free_free_dont_miss_out_modal_v3`

Preconditions

Experiment should be active: 21343-explat-experiment
Should be in onboarding flow
Select the free subdomain on the domain step

Event : Click on the free plan button

Control	Variant
`Proceed to next step`

Pre-merge Checklist

Has the general commit checklist been followed? (PCYsg-hS-p2)
Have you written new tests for your changes?
Have you tested the feature in Simple (P9HQHe-k8-p2), Atomic (P9HQHe-jW-p2), and self-hosted Jetpack sites (PCYsg-g6b-p2)?
Have you checked for TypeScript, React or other console errors?
Have you used memoizing on expensive computations? More info in Memoizing with create-selector and Using memoizing selectors and Our Approach to Data
Have we added the "[Status] String Freeze" label as soon as any new strings were ready for translation (p4TIVU-5Jq-p2)?
For changes affecting Jetpack: Have we added the "[Status] Needs Privacy Updates" label if this pull request changes what data or activity we track or use (p4TIVU-ajp-p2)?

github-actions · 2023-08-15T22:30:21Z

Calypso Live (direct link)

	https://calypso.live?image=registry.a8c.com/calypso/app:build-78450

Jetpack Cloud live (direct link)

	https://calypso.live?image=registry.a8c.com/calypso/app:build-78450&env=jetpack

ddc22 · 2023-08-15T22:36:45Z

client/my-sites/plans-features-main/index.tsx

@@ -211,7 +211,7 @@ const PlansFeaturesMain = ( {
 	isSpotlightOnCurrentPlan,
 }: PlansFeaturesMainProps ) => {
 	const [ isFreePlanPaidDomainDialogOpen, setIsFreePlanPaidDomainDialogOpen ] = useState( false );
-	const [ isFreeFreeUpsellOpen, setIsFreeFreeUpsellOpen ] = useState( false );
+	const [ isFreePlanFreeDomainDialogOpen, setIsFreePlanFreeDomainDialogOpen ] = useState( false );


Unrelated: Renamed a confusing variable.

matticbot · 2023-08-15T22:47:35Z

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

Sections (~108 bytes added 📈 [gzipped])

name                  parsed_size           gzip_size
update-design-flow         +513 B  (+0.1%)     +108 B  (+0.0%)
plugins                    +513 B  (+0.0%)     +108 B  (+0.0%)
plans                      +513 B  (+0.0%)     +108 B  (+0.0%)
link-in-bio-tld-flow       +513 B  (+0.0%)     +108 B  (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~155 bytes added 📈 [gzipped])

name                                             parsed_size           gzip_size
async-load-signup-steps-plans-theme-preselected       +513 B  (+0.1%)     +108 B  (+0.1%)
async-load-signup-steps-plans                         +513 B  (+0.1%)     +108 B  (+0.1%)
async-load-signup-steps-domains                       +170 B  (+0.0%)      +47 B  (+0.0%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

aaronyan

Looks good so far and tests well for both onboarding and onboarding-pm.

We are splitting the tests so that the onboarding flow uses calypso_onboarding_plans_paid_domain_on_free_plan and the onboarding-pm flow uses calypso_onboardingpm_plans_paid_domain_on_free_plan.

aaronyan · 2023-08-18T19:58:11Z

client/my-sites/plans-features-main/hooks/use-is-custom-domain-allowed-on-free-plan.ts

 ): DataResponse< boolean > => {
-	const [ isLoadingAssignment, experimentAssignment ] = useExperiment(
-		'calypso_onboarding_plans_dotblog_on_free_plan_202307',
+	const [ isLoadingAnyDomainExperiment, assignmentAnyDomainExperiment ] = useExperiment(


Consider using the word "paid" in AnyPaidDomain here!

client/my-sites/plans-features-main/hooks/use-is-custom-domain-allowed-on-free-plan.ts

aaronyan

LGTM! Thanks for answering all of my questions, @jdc91.

southp · 2023-08-22T07:16:29Z

It looks like the loading placeholder in the modal is somehow broken in this PR, or has been broken at some points in the past 🤔

By clicking on either the Free plan button or the "start with a free plan" anchor with a paid domain picked, it will silently do nothing, until the experiment is loaded and another click on them:

2023-08-22.3.13.24.mov

From the modal code, the original idea should be that a loading placeholder is shown instead of nothing happening: https://github.com/Automattic/wp-calypso/blob/trunk/client/my-sites/plans-features-main/components/free-plan-paid-domain-dialog.tsx#L356 . I feel this is an important UX issue to fix before launching. @jdc91 could you look into it?

ddc22 · 2023-08-22T12:14:11Z

Good catch James I will work on this!

southp · 2023-08-23T02:59:22Z

@jdc91 Sorry that my previous comment wasn't clear enough. The new change has added a loading placeholder to the "Start with a free plan" anchor-like button. However, it's also happening on /start/:

2023-08-23.10.53.37.mov

I actually think your solution is more fine-tuned. However, it's in the experimental stage so I'm thinking we can just show a loading placeholder inside the modal when the experiments hasn't been loaded yet. That should be applicable for both /start/onboarding-pm and /start/, and that's what the original code is supposed to do. What do you think? I'm surely up for the other way as long as there is no unresponsive click to the Free button :)

southp · 2023-08-23T10:11:59Z

@jdc91 The unresponsive clicks on the free button issue has surely been resolved by 7bd5229. However, there are two things I'm not sure I understand:

Since what we need to confirm before showing the modal is the variation of the Free/Free modal experiment, shouldn't we checking isPlanUpsellEnabledOnFreeDomain.loading in https://github.com/Automattic/wp-calypso/pull/80647/files#diff-18214fd84316736ff6662f03fa201b6255f4c20f277c0eb683b5e36e9c9dea6dR601 instead?
f6442bf does introduce a loading place holder for the "Start with a free plan" button in /start/onboarding-pm flow, i.e. this button:
.
However, I'm not seeing the same happening for the Start with Free button in /start/:
.
If I understand it correctly that we need to load the Free/Free experiment to confirm whether clicking on the Free button needing to show the FOMO modal, shouldn't the Start with Free button in /start also has a similar loading placeholder?

Sorry if I review the changes too soon. Since our timezone is quite far apart now, I thought I'd just leave what's in my mind early :)

donlair · 2023-08-23T12:22:22Z

@jdc91 I think you could avoid the timing issues w/ test assignment by assigning folks to both tests on the Domains step and have logic in the modal to check for which variant to show and fire the appropriate exposure event. That should eliminate the need for a placeholder and would probably let you simplify the conditionals in plans-features-main CC @aaronyan

ddc22 · 2023-08-23T13:38:41Z

Thanks @donlair and this makes sense too 🤔 ... @aaronyan Is it ok to load the experiment at the end of the domain step?

jeyip · 2023-08-23T23:58:03Z

aaronyan · 2023-08-24T01:02:43Z

Is it ok to load the experiment at the end of the domain step?

@jdc91, yeah I think it's okay to load the experiment once the user selects a paid domain. Good idea @donlair!

southp

Since @jeyip has already done a comprehensive test, I've mostly focused on reviewing and checking for edge conditions. It works and LGTM 🚢

@donlair Thanks for proposing the idea of preloading the experiments in the domain step, which surely will greatly help the user experience. I just wanted to point out that preloading or not, we will still need to implement the blocking logics, since that's for guaranteeing the logical correctness; preloading is only for a better user experience. e.g. In my ~10 times of testing this PR in thie round of review, I still did see the loading placeholder shown once, so chances are that people on a not-so-good connection will need this guarantee :)

southp · 2023-08-24T02:00:08Z

client/my-sites/plan-features-2023-grid/types.ts

-		text: TranslateResult;
+		callback?: () => void;
+		text?: TranslateResult;
+		status?: 'blocked' | 'enabled';


Nit: it looks like the only value in use is 'blocked', which means it's likely can be just isBusy of boolean value, but I'm fine with leaving it this way :)

I actually was struggling to find a proper term but I think I just got it. How about, 'disabled' | 'enabled' | 'busy' ?

a8ci18n · 2023-08-24T15:18:47Z

This Pull Request is now available for translation here: https://translate.wordpress.com/deliverables/8804914

Thank you @jdc91 for including a screenshot in the description! This is really helpful for our translators.

a8ci18n · 2023-09-01T12:32:17Z

Translation for this Pull Request has now been finished.

ddc22 changed the title ~~Paid Free Domain For all Domains~~ GF Signup : Paid Free Domain For all Domains Aug 15, 2023

ddc22 self-assigned this Aug 15, 2023

ddc22 added the growth foundations label Aug 15, 2023

ddc22 requested review from a team and aneeshd16 August 15, 2023 22:35

matticbot added the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Aug 15, 2023

ddc22 commented Aug 15, 2023

View reviewed changes

ddc22 added 2 commits August 18, 2023 00:34

Paid Free Domain For all Domains

f32a659

Update experiment name

3249370

ddc22 force-pushed the add/all-domain-paid-free-modal branch from 39cd7cf to 3249370 Compare August 18, 2023 04:34

ddc22 changed the title ~~GF Signup : Paid Free Domain For all Domains~~ GF Signup : Experiment Allow Paid Domain redirect For all Domains Aug 18, 2023

Cleanup .blog experiment, expose other to onboarding flow too

87a95da

ddc22 requested review from southp and aaronyan August 18, 2023 15:22

ddc22 marked this pull request as ready for review August 18, 2023 15:22

aaronyan reviewed Aug 18, 2023

View reviewed changes

Use own experiment per flow

86213e9

ddc22 force-pushed the add/all-domain-paid-free-modal branch from c32a51f to 86213e9 Compare August 21, 2023 20:25

aaronyan reviewed Aug 22, 2023

View reviewed changes

client/my-sites/plans-features-main/hooks/use-is-custom-domain-allowed-on-free-plan.ts Show resolved Hide resolved

client/my-sites/plans-features-main/hooks/use-is-custom-domain-allowed-on-free-plan.ts Show resolved Hide resolved

aaronyan approved these changes Aug 22, 2023

View reviewed changes

ddc22 added 2 commits August 22, 2023 09:19

Loading placeholder until experiment is loaded

f6442bf

Props should be optional

11e020b

Fix upgrade block since we are now showing a loader'

7bd5229

ddc22 added 2 commits August 23, 2023 12:19

Complete loader for free plan button on onboarding flow

aa303c1

Load experiment assignment early

d92aa93

ddc22 requested a review from a team as a code owner August 23, 2023 16:55

southp approved these changes Aug 24, 2023

View reviewed changes

ddc22 merged commit 81bdd0c into trunk Aug 24, 2023

ddc22 deleted the add/all-domain-paid-free-modal branch August 24, 2023 13:56

github-actions bot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Aug 24, 2023

ddc22 mentioned this pull request Aug 25, 2023

GF Signup:Fix preloading conditions #81039

Merged

7 tasks

ddc22 mentioned this pull request Sep 5, 2023

Paid/Free Confidence check experiment name update #81408

Merged

7 tasks

ddc22 mentioned this pull request Oct 2, 2023

Attempt 2 - Ships free/free and partial paid/free experiment variants #82361

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GF Signup : Experiment Allow Paid Domain redirect For all Domains #80647

GF Signup : Experiment Allow Paid Domain redirect For all Domains #80647

ddc22 commented Aug 15, 2023 •

edited

Loading

github-actions bot commented Aug 15, 2023 •

edited by matticbot

Loading

ddc22 Aug 15, 2023

matticbot commented Aug 15, 2023 •

edited

Loading

aaronyan left a comment

aaronyan Aug 18, 2023

aaronyan left a comment

southp commented Aug 22, 2023

ddc22 commented Aug 22, 2023

southp commented Aug 23, 2023

southp commented Aug 23, 2023 •

edited

Loading

donlair commented Aug 23, 2023

ddc22 commented Aug 23, 2023

jeyip commented Aug 23, 2023 •

edited

Loading

aaronyan commented Aug 24, 2023

southp left a comment

southp Aug 24, 2023

ddc22 Aug 24, 2023

a8ci18n commented Aug 24, 2023

a8ci18n commented Sep 1, 2023

GF Signup : Experiment Allow Paid Domain redirect For all Domains #80647

GF Signup : Experiment Allow Paid Domain redirect For all Domains #80647

Conversation

ddc22 commented Aug 15, 2023 • edited Loading

Proposed Changes

Typical Testing Path

Comprehensive experiment testing plan including regression tests

Experiment calypso_onboardingpm_plans_paid_domain_on_free_plan

Preconditions

Experiment calypso_onboarding_plans_paid_domain_on_free_plan

Preconditions

Experiment calypso_gf_signup_onboarding_pm_free_free_dont_miss_out_modal_v3

Preconditions

Experiment calypso_gf_signup_onboarding_free_free_dont_miss_out_modal_v3

Preconditions

Pre-merge Checklist

github-actions bot commented Aug 15, 2023 • edited by matticbot Loading

ddc22 Aug 15, 2023

Choose a reason for hiding this comment

matticbot commented Aug 15, 2023 • edited Loading

aaronyan left a comment

Choose a reason for hiding this comment

aaronyan Aug 18, 2023

Choose a reason for hiding this comment

aaronyan left a comment

Choose a reason for hiding this comment

southp commented Aug 22, 2023

ddc22 commented Aug 22, 2023

southp commented Aug 23, 2023

southp commented Aug 23, 2023 • edited Loading

donlair commented Aug 23, 2023

ddc22 commented Aug 23, 2023

jeyip commented Aug 23, 2023 • edited Loading

Testing

Requirements

/onboarding-pm

calypso_onboardingpm_plans_paid_domain_on_free_plan

calypso_gf_signup_onboarding_pm_free_free_dont_miss_out_modal_v3

/onboarding

calypso_onboarding_plans_paid_domain_on_free_plan

calypso_gf_signup_onboarding_free_free_dont_miss_out_modal_v3

Browsers

Issues

aaronyan commented Aug 24, 2023

southp left a comment

Choose a reason for hiding this comment

southp Aug 24, 2023

Choose a reason for hiding this comment

ddc22 Aug 24, 2023

Choose a reason for hiding this comment

a8ci18n commented Aug 24, 2023

a8ci18n commented Sep 1, 2023

ddc22 commented Aug 15, 2023 •

edited

Loading

Experiment `calypso_onboardingpm_plans_paid_domain_on_free_plan`

Experiment `calypso_onboarding_plans_paid_domain_on_free_plan`

Experiment `calypso_gf_signup_onboarding_pm_free_free_dont_miss_out_modal_v3`

Experiment `calypso_gf_signup_onboarding_free_free_dont_miss_out_modal_v3`

github-actions bot commented Aug 15, 2023 •

edited by matticbot

Loading

matticbot commented Aug 15, 2023 •

edited

Loading

southp commented Aug 23, 2023 •

edited

Loading

jeyip commented Aug 23, 2023 •

edited

Loading