User story:
As the Growth team, I want to quickly determine if the new Positive Reinforcement features are having a positive impact on newcomers, because if we see any negative impact then we will want to revert changes and if we don't see any positive changes then we will need to make further improvements or consider sunsetting the features.

• https://www.mediawiki.org/wiki/Growth/Positive_reinforcement#Measurement_and_results
• https://phabricator.wikimedia.org/tag/growthexperiments-levelingup/

Hypotheses
The Positive Reinforcement features aim to provide or improve the tools available to newcomers and mentors in three specific areas that will be described in more detail below. Our hypothesis is that once a newcomer has made a contribution (say by making a structured task edit), these features will help create a positive feedback cycle that increases newcomer motivation.

Experiment plan
Similarly as we have done for previous Growth team projects, we want to test our hypotheses through controlled experiments (also called "A/B tests"). This will allow us to establish a causal relationship (e.g. "The Leveling Up features cause an increase in retention of xx%"), and it will allow us to detect smaller effects than if we were to give it to everyone and analyze the effects pre/post deployment.

In this controlled experiment, a randomly selected half of users will get access to Positive Reinforcement features (the "treatment" group), and the other randomly selected half will instead get the current (September 2022) Growth feature experience (the "control" group). In previous experiments, the control group has not gotten access to the Growth features. The team has decided to move away from that (T320876), which means that the current set of features is the new baseline for a control group.

These experiments will first run on the pilot wikis. We can extend this to additional wikis if we find a need to do that, but it would only happen after we have analyzed the leading indicators and found no concerns.

Each experiment will run for approximately one month, and for each experiment we will have an accompanying set of leading indicators that we will analyze two weeks after deployment.

Leveling up: treatment group gets both the updated Impact module and the Leveling up features.

Acceptance Criteria:

• Work with Product Manager and Growth team to establish leading indicators for Leveling Up work
• Document leading indicators on Positive Reinforcement project page
• Post MVP launch of Leveling Up, evaluate and record a summary of leading indicator results on Positive Reinforcement project page