Update: 10-Feb

The measurement plan is being drafted in this document (read only access for now): Talk pages project/Notifications/Measurement plan.

Immediate next steps

• 1. @ppelberg to finish drafting hypotheses
• 2. @ppelberg + @iamjessklein + @MNeisler to discuss
• 3. @MNeisler + @iamjessklein to propose quantitative and qualitative evaluation methods, respectively.

Guardrails
In this conversation on mediawiki.org, @Sdkb raised two potential consequences that could emerge when people are able to be more specific about the activity they are and are not made aware of:

1. People could see a conversation, deem it to be not relevant enough for them to subscribe to in the moment, , and subsequently, miss out on a conversation that would've been valuable for them to be aware of.
2. To evade "opponents," people editing in bad faith could start a new topic on a talk page to avoid the attention of people subscribed to a conversation ongoing elsewhere on the page.

I'm documenting these scenarios here so we can think about how we might monitor their prevalence.

24-Feb update
Per yesterday's conversation, @MNeisler is going to:

• 1a) Review the "Hypotheses" and provide feedback if she thinks edits should be made to them
• 1b) Propose quantitative metrics for each hypothesis, if applicable
• 2a) Review the "Guardrails" and provide feedback if she thinks edits should be made to them
• 2b) Propose quantitative metrics for each guardrail

Next steps:

• 1. Team to assign impact value for each "Pre-mortem" scenario
• 2. @ppelberg to prioritize scenarios
• 3. @MNeisler to draft quantitative leading indiactors for applicable "pre-mortem" scenarios
• 4. @MNeisler + @ppelberg' to finalize "Hypotheses" and KPIs," pending team review
• 5. @MNeisler + @ppelberg' to draft/finalized "Experiment" plan

In T274215#6910170, @ppelberg wrote:

Next steps:

• 1. Team to assign impact value for each "Pre-mortem" scenario
• 2. @ppelberg to prioritize scenarios

The drafted pre-mortem can be found here (limited access for now): Talk pages project/Notifications/Pre-mortem.

• 3. @MNeisler to draft quantitative leading indiactors for applicable "pre-mortem" scenarios

@MNeisler, for each of the "Scenarios" in this sheet [i] whose "Priority" has been marked as "HIGH" or "MEDIUM" can you please do the below?

• 1. Add quantitative "Leading indicator" to applicable scenarios?
• 2. Review the existing [Quant] "Leading indicator" I've suggested and comment any revisions you think should be made to them?

i. The contents of this spreadsheet will automatically be added to the Google Doc where we originally started documenting and evaluating the various pre-mortem scenarios.

@MNeisler, for each of the "Scenarios" in this sheet [i] whose "Priority" has been marked as "HIGH" or "MEDIUM" can you please do the below?

1. Add quantitative "Leading indicator" to applicable scenarios?
   1. Review the existing [Quant] "Leading indicator" I've suggested and comment any revisions you think should be made to them?

@ppelberg - I've completed reviewing existing and adding quantifiable Leading indicators for all the high and medium tasks in the document as comments/suggestions. Let me know if you have any questions about the changes.

In T274215#7024359, @MNeisler wrote:

@ppelberg - I've completed reviewing existing and adding quantifiable Leading indicators for all the high and medium tasks in the document as comments/suggestions. Let me know if you have any questions about the changes.

Excellent.

Per the conversation @MNeisler and I had earlier today, this task can now be "Resolved" considering we have done the following (copied from the task's ===Done section):

• Pre-mortem is complete (read: scenarios are documented and prioritized)

We've completed the pre-mortem and defined during which analysis (Adoption / Engagement / Impact) we will check on the quantitative leading indicators we've defined for the "MEDIUM" and "HIGH" priority scenarios. More in the Pre-mortem section of the measurement plan.

• Hypotheses and project KPI is finalized

We've finalized the hypotheses and identified the KPI we will use for this set of interventions: Response rate as defined as the average time that elapses between when someone posts on a talk page and when a Junior Contributor posts a new comment in response. More in the Hypotheses section of the measurement plan.

• Experiment plan is drafted

We've defined three analyses that will comprise the topic subscription experiment. These analyses can be found in the Experiment plan of the measurement plan.

Tasks representing each of these analyses will be created as sub-tasks of T280895.