Ensure all newcomers receive Growth features (end the 20% control group)
Closed, ResolvedPublic

Description

As a Product Manager, I want to disable the Growth control group (in which some users don't receive Growth features on certain wikis), because I believe we have proven the value of the "core" Growth features.

We will continue to test Growth features and conduct experiments (with control groups) for all major Growth features released, but we will no longer have a 20% control group which does not receive Growth features at all.

Description

  • Ensure 100% of new accounts created receive Growth features.
  • Keep the code that allows for returning to an 80/20 test in the future if we decide an additional experiment is needed.

We love data... so why are we disabling the control group?

Impacted Wikis:

  • arwiki
  • bnwiki
  • cswiki
  • dewiki
  • eswiki
  • fawiki
  • ptwiki
  • trwiki

Note: German Wikipedia requested to have a 20% control group in the past, so we will keep this control group enabled if they prefer. @Trizek-WMF will reach out to determine if they want to keep the control group. If our contacts at German wikipedia agree on ending the control group by our release date, then they will be included in this change.

Acceptance Criteria

  • Given I create a new account on any Wikipedia (except German Wikipedia), when the account is created, then I receive the core newcomer Growth features automatically (welcome survey, newcomer homepage, suggested edits, help panel)
  • Update deployment table and any other related public Growth documentation (AKA check in with @Trizek-WMF )

Completion checklist

Functionality

  • The patches have been code reviewed and merged
  • The task passes its acceptance criteria

Engineering

  • There are existing and passing unit/integration tests
  • Tests for every involved patch should pass
  • Coverage for every involved project should have improved or stayed the same

Design & QA

  • If the task is UX/Design related: it must be reviewed and approved by the UX/Design team
  • Must be reviewed and approved by Quality Assurance.

Documentation

  • Related and updated documentation done where necessary

Event Timeline

@Trizek-WMF - Should we let impacted wikis know in advance? Or since communities have been asking for this, do you think it's OK to make the change and then let them know?
(If you want to update wikis in advance, what release date should we set?)

As I understand it, one consideration is that mentors will start to get ~20% more mentees (and thus more mentee questions) after this change.

I can work on this once @Trizek-WMF / @KStoller-WMF define a deployment date here. FTR, this would be a config change, which means we can deploy it at any day (except Fridays and weekends).

Note that most wikis are already at 100%. This change will affect the following wikis, which are at 80% now:

  • arwiki
  • bnwiki
  • cswiki
  • dewiki
  • eswiki
  • fawiki
  • ptwiki
  • trwiki

In addition to that, enwiki and eswiki don't have mentorship enabled for 100% of users, but only for 10% (enwiki) and 15% (eswiki). I don't think changing this in scope of this task, but mentioning this for completeness.

dewiki has a control group at their demand, as they want to figure out if our features improve retention locally.

dewiki has a control group at their demand, as they want to figure out if our features improve retention locally.

Thanks for pointing that out! The comments in the server configuration says control group at dewiki was made because it has more than 500 new users a month, I guess that's out of date information.

I'll follow up with them (@KStoller-WMF FYI).

Thank you! Do you want to wait until we hear from dewiki before setting a release date then?

We can either wait, or do the change without them and then catch-up.

Per meeting decision of the day, we will end control groups. The set date is November 2, 2022.

An exception will be made for German Wikipedia. But if our contacts there agree on ending the control group as well at this date, they will be included as well.

Urbanecm_WMF changed Due Date from Nov 2 2022, 5:00 AM to Nov 2 2022, 10:00 PM.Oct 31 2022, 4:26 PM
KStoller-WMF renamed this task from End Growth Control Group to Ensure all newcomers receive Growth features (end the 20% control group).Oct 31 2022, 7:10 PM
KStoller-WMF updated the task description. (Show Details)

Change 851604 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Deploy Growth features to 100% users at all wikis but dewiki

https://gerrit.wikimedia.org/r/851604

This is scheduled for tomorrow, 2022-11-02, UTC morning window (07:00 UTC).

Change 851604 merged by jenkins-bot:

[operations/mediawiki-config@master] Deploy GrowthExperiments to 100% users at all wikis but dewiki

https://gerrit.wikimedia.org/r/851604

Mentioned in SAL (#wikimedia-operations) [2022-11-02T07:01:56Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:851604|Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)]]

Mentioned in SAL (#wikimedia-operations) [2022-11-02T07:02:23Z] <urbanecm@deploy1002> urbanecm and urbanecm: Backport for [[gerrit:851604|Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-11-02T07:06:34Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:851604|Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)]] (duration: 04m 38s)

This is done for all wikis but dewiki. Moving to In Progress, pending clarification on how to proceed with dewiki (fyi @Trizek-WMF).

I think we can close when QA is done: German Wikipedia's working group will have a meeting about their control group in the coming months, and they will then have to decide on what to do.

T322234: Ensure all newcomers receive Growth features at German Wikipedia (end the 20% control group) will track it.

On my side, I updated the deployment table.

Etonkovidova updated the task description. (Show Details)