In T374664#10267324, @Sgs wrote:

@nettrom_WMF, in our last meeting you mentioned you would define treatment and control group more precisely in terms of account ages. For now, we're not restricting any account by its age, all accounts including newly created are distributed 50/50 into control and the treatment group community-updates-module. Is this ok the rollout on Wed, October 30 to Spanish Wikipedia, French Wikipedia, and Egyptian Arabic?

Confirmed by generating events on testwiki and querying logged events in the Data Lake that performer.id is now captured correctly in the database table. Closing this as resolved, thanks everyone!

In T372840#10227052, @KStoller-WMF wrote:

I had assumed we would do a 50/50 split for both new accounts AND existing accounts as I imagine that will allow us to gather enough data to make a statistically significant conclusion the fastest. But I'm open to whatever @nettrom_WMF thinks is best.

In T374675#10158054, @nettrom_WMF wrote:

Moving this to "blocked" for now because I'll be waiting on or coordinating with folks in T371347 to verify the data in the Data Lake once we figure out where it is (currently there's only a few events from 2024-09-11).

I've verified that the impression and click events are firing as expected for the module on testwiki.

@KStoller-WMF : Sounds good to me! Ping me when the overview is on MW and I'll take a look?

The estimate has been done and the associated googledoc shared with stakeholders for review. In short, we're in need of about 10k visitors to the Homepage, which in a worst-case scenario we're expected to get in 1.5 months. We also have valid reasons to expect that worst-case scenario to not come true, but we've also defined leading indicators to tell us what is happening just in case.

I've taken a look at it and couldn't find anything to add or change. Moving it to "done". Maybe @mpopov wants to have the pleasure of closing this?

I checked out the data and the associated dashboard, and this looks good to me.

Awesome example dashboard, great work!

Confirmed that these click events are firing correctly on both desktop and mobile web, moving to "done".

When verifying these events, I don't see action_context set for these (but it's set for the subsequent click event). Moving it back to "blocked" for now, but that might not be the right column?

Looks good to me, moving to "Done"

In T374251#10140473, @Sgs wrote:

I have been highly discouraged from taking this approach by @mforns and @phuedx so I'm avoiding it for now, action_context=friendly_update_title. One caveat is we don't have anything that identifies uniquely a "community update". If the title is updated but it's an update on the same announcement, we'll get two different contexts.

While I know a decision has been made, I wanted to add a few notes about option 1 in case someone comes back to this in the future. From the current task description:

In T374464#10134507, @Sgs wrote:

Looping @nettrom_WMF to give us some guidance on which views would be more interesting to create in the context of the experiment.

In T374251#10126287, @Sgs wrote:

Thanks for providing the detailed specification @nettrom_WMF. I have a some questions:

1. In the instrument specification the field used to capture the friendly name is action_subtype, the documentation for that field says it is used to classify actions, I'm not sure if the update friendly name is a good candidate for such classification. I wonder how is it different from capturing it under action_context? I'm imaging we introduce a second link in the future (eg: wikitext is allowed in the description), how would we differentiate the source link that originated the event?

I think the last time we used this data to make changes was in 2019, where clicks on these links were one of the Help Panel Leading Indicators. My recollection, and @Trizek-WMF should correct me if I'm wrong, is that afterwards we had a fairly standard set of suggested links to go in the Help Panel (and subsequently the Help module on the Homepage) that reflected those stats.

I'm in favour of proposal A for two reasons:

I'm chiming in to make sure we're on the same page and don't spend efforts on possibly dead ends. Here's my understanding of some of the things we're discussing:

In T365889#10031417, @Sgs wrote:

In early conversations with @DMburugu and @KStoller-WMF, we agreed that aside from the new Metrics Platform instruments, we would also add "legacy" instrumentation through the EventPlatform as the rest of homepage modules so we could compare results. Does this make sense to you @nettrom_WMF ? If so, we need to add a new module, community-updates in the module enumeration. Is there any other data that would be relevant to capture here as action data? eg: the title of the community update

In T366627#10030134, @VirginiaPoundstone wrote:

What do you think about these latencies @nettrom_WMF @mpopov? Acceptable?

When a user rejects an image, we log the rejection reasons in the instrumentation. The schema definition describes this in more detail. I've confirmed using recent (July 2024) event data that we capture this. Event data is then kept for 90 days.

In T360829#9903872, @MNeisler wrote:

OPEN QUESTIONS/NOTES
[1] The current definition specifies that the edit be completed on a mobile device. I also limited the analysis to users that registered on a mobile device so we are reviewing users that complete a constructive edit on the same platform where they registered. This would exclude desktop registrations. Any concerns or suggested changes to this approach?

I've completed the initial investigation into the effects of how load times affect Homepage engagements.

Can confirm that this problem has re-emerged in HomepageVisit as well, running the same query that @phuedx
used, but modified to remove the API counts and changed to query event.homepagevisit gives:

We ran into this three years ago when two of the Growth team's schemas were migrated to Event Platform. Might be useful to check what was done there: T288853

In T354965#9914223, @VirginiaPoundstone wrote:

@nettrom_WMF are you using this epic for your work on [[ https://app.asana.com/0/1207174936768639/1207174812002724 | [Hypothesis] SDS2.5.7 Define a way to measure time-to-data-collection for instruments ]]? If not, please hang whatever task you are using off this epic. Merci!

I've reviewed the notebooks so they're in line with the Data Publication guidelines and submitted the publication form. The notebooks are now available on GitLab: https://gitlab.wikimedia.org/nettrom/2021-Growth-structured-tasks

Changing the priority of this task to medium, I don't think this is an UBN task since we're probably working on this slowly over this quarter and next.

The report from the analysis is now on-wiki: https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Add_an_image/Experiment_analysis,_March_2024

In T360829#9662018, @mpopov wrote:

Morten, can you please confirm if you're probably going to use that definition or if there's a good chance you change the time frame based on your experience over the past several years? (No need to commit to anything here, just a simple vibes check.)

I agree with what's being said here. The performer values would reflect what the current user is, and if that user successfully logs in it'll reflect their logged in status. If the login attempt fails, it'll reflect their either anon or temp user status.

In T346327#9455949, @Sgs wrote:

In T346327#9444225, @nettrom_WMF wrote:

2. I'm definitely in favour of not having duplicate events. Just so I understand the event generation here, we're instrumenting both account creation and login, but the auto-login doesn't trigger a login event through this instrumentation? It's instead that it creates a second account creation "success" event?

See also my comments above T346327#9446991, a way of solving this could be to have a different dedicated schema for login attempts (and impressions?). Would that be appropriate @nettrom_WMF ?

A written summary of our findings have been published on the Positive reinforcement project page in the section "Personalized praise experiment results". The notebooks associated with this analysis have been uploaded to this Gitlab repository and carry the name of this phab task. Closing as resolved.

A written summary of our findings have been published on the Positive reinforcement project page in the section "Personalized praise experiment results". The notebooks associated with this analysis have been uploaded to this Gitlab repository and carry the name of this phab task. Closing as resolved.

The notebooks for the previously mentioned preliminary analysis of the effects of mentorship have been added to this Gitlab repository using a naming convention that reflects that they're associated with this phab task. Resolving this parent task now, and I'll close the child tasks once their notebooks are published as well.

Out of curiosity, and in hoping to make this easier to fix, I grabbed daily aggregates for October 2023. Looks like something was deployed the week of Oct 23 as that's when the proportion increases from the expected 6–7%:

In T346327#9421209, @Sgs wrote:

@nettrom_WMF would you mind taking a second look to the analytics schema change 962569, I've added some questions. Mainly, are we instrumenting all account creations or just from temporary users? And how should the auto-login after account creation be logged? Currently it would produce a "success"-Special:CreateAccount event which duplicates the account creation success. cc @Cyndymediawiksim

Closing this task as declined for now. If the need for additional data analysis comes up, feel free to file another request.

I've collected data and calculated the statistics for both of the campaigns using the date/time ranges listed in the task description.

I've tested creating new environments on two stat machines with different OSes, and both with and without adding the conda-meta/pinned file. From what I can tell, the conda-forge R 4.2.3 package no longer results in a broken environment because it works just fine without the pin-file. Looking at the file listing on conda-forge, there have been several new versions of the package released since this bug was opened. Based on documentation I've found from googling it should be possible to install a specific build version, but it looks like the Mamba solver doesn't support it properly yet. It doesn't make sense to try this with the classic solver because that wasn't able to resolve the dependencies when I tried that back in August.

I've collected data to understand the effects of this campaign after about one week. During the data gathering for this update, I discovered a data quality issue in one of the underlying datasets that rendered us unable to correctly identify reverts (this has been filed as T352899). I was able to work around this by modifying the query that collects editing data.

I've run the numbers on the currently available data^1 to get a sense of where we're at. Overall, we get the following counts and proportions of page views, "unique visitors" (based on combinations of IPs and user-agents for page views of the account creation page), registrations, and activations.

In T350745#9361752, @spatton wrote:

@KStoller-WMF and/or @nettrom_WMF, could you please let us know if you'd be able to provide an update on account signups some time after the first 24 hour mark, and then occasionally in December? Thanks!

In T350745#9331846, @Sgs wrote:

Do we have any kind of instrumentation for this in place? Or should we add it to the requirements and prepare some experiment so we can analyze the impact of the TY pages cc @nettrom_WMF