Create Task
Maniphest T331702

Migrate mw_rc_irc servers to Bullseye
Closed, ResolvedPublic
Actions

Description

These are still on Buster. To upgrade the existing stack at least two things need to be resolved:

  • build the patched ratbox for bullseye-wikimedia
  • Migrate the Python parts of the broadcasting stack to Python 3

The alternative is to move forward with https://phabricator.wikimedia.org/T234234

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• MoritzMuehlenhoff created this task.Mar 10 2023, 8:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2023, 8:56 AM
• MoritzMuehlenhoff triaged this task as Medium priority.Mar 10 2023, 8:57 AM
akosiaris added a parent task: T291916: Tracking task for Bullseye migrations in production.Mar 14 2023, 10:17 AM
• MoritzMuehlenhoff added a project: SRE-Unowned.Mar 16 2023, 3:26 PM
andrea.denisse subscribed.Mar 21 2023, 3:13 AM
gerritbot added a comment.Mar 22 2023, 12:55 PM

Change 902077 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move udpmixircecho to Python 3 for Bullseye hosts

https://gerrit.wikimedia.org/r/902077

gerritbot added a project: Patch-For-Review.Mar 22 2023, 12:55 PM
gerritbot added a comment.Mar 23 2023, 8:07 AM

Change 902077 merged by Muehlenhoff:

[operations/puppet@production] Move udpmixircecho to Python 3 for Bullseye hosts

https://gerrit.wikimedia.org/r/902077

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2023, 8:10 AM
gerritbot added a comment.Mar 23 2023, 9:53 AM

Change 902309 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add irc2002

https://gerrit.wikimedia.org/r/902309

gerritbot added a project: Patch-For-Review.Mar 23 2023, 9:53 AM
gerritbot added a comment.Mar 23 2023, 10:26 AM

Change 902309 merged by Muehlenhoff:

[operations/puppet@production] Add irc2002

https://gerrit.wikimedia.org/r/902309

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2023, 10:30 AM
ops-monitoring-bot added a comment.Mar 23 2023, 10:44 AM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host irc2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2023, 11:16 AM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host irc2002.wikimedia.org with OS bullseye completed:

  • irc2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202303231044_jmm_947400_irc2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
gerritbot added a comment.Mar 23 2023, 11:39 AM

Change 902325 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign mw_rc_irc role to irc2002

https://gerrit.wikimedia.org/r/902325

gerritbot added a project: Patch-For-Review.Mar 23 2023, 11:39 AM
gerritbot added a comment.Mar 23 2023, 11:43 AM

Change 902325 merged by Muehlenhoff:

[operations/puppet@production] Assign mw_rc_irc role to irc2002

https://gerrit.wikimedia.org/r/902325

gerritbot added a comment.Mar 23 2023, 12:00 PM

Change 902349 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix typo in systemctl command

https://gerrit.wikimedia.org/r/902349

gerritbot added a comment.Mar 23 2023, 12:16 PM

Change 902349 merged by Muehlenhoff:

[operations/puppet@production] Fix typo in systemctl command

https://gerrit.wikimedia.org/r/902349

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2023, 12:30 PM
gerritbot added a comment.Mar 23 2023, 12:37 PM

Change 902362 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix NRPE check when running under Python 3

https://gerrit.wikimedia.org/r/902362

gerritbot added a project: Patch-For-Review.Mar 23 2023, 12:37 PM
gerritbot added a comment.Mar 23 2023, 2:10 PM

Change 902362 merged by Muehlenhoff:

[operations/puppet@production] Fix NRPE check when running under Python 3

https://gerrit.wikimedia.org/r/902362

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2023, 2:30 PM
ops-monitoring-bot added a comment.Mar 23 2023, 3:03 PM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host irc1002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2023, 3:36 PM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host irc1002.wikimedia.org with OS bullseye completed:

  • irc1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202303231503_jmm_1164999_irc1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
Peachey88 added a project: Wikimedia-IRC-RC-Server.Mar 25 2023, 1:46 PM
gerritbot added a comment.Apr 4 2023, 12:29 PM

Change 905626 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/mediawiki-config@master] Also broadcast RCFeed/IRC events to irc1002/irc2002

https://gerrit.wikimedia.org/r/905626

gerritbot added a project: Patch-For-Review.Apr 4 2023, 12:29 PM
gerritbot added a comment.Apr 17 2023, 9:03 AM

Change 905626 merged by jenkins-bot:

[operations/mediawiki-config@master] Also broadcast RCFeed/IRC events to irc1002/irc2002

https://gerrit.wikimedia.org/r/905626

Stashbot added a comment.Apr 17 2023, 9:04 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-17T09:04:26Z] <ladsgroup@deploy2002> Started scap: Backport for [[gerrit:905626|Also broadcast RCFeed/IRC events to irc1002/irc2002 (T331702)]]

Maintenance_bot removed a project: Patch-For-Review.Apr 17 2023, 9:10 AM
Stashbot added a comment.Apr 17 2023, 9:12 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-17T09:12:54Z] <ladsgroup@deploy2002> jmm and ladsgroup: Backport for [[gerrit:905626|Also broadcast RCFeed/IRC events to irc1002/irc2002 (T331702)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Stashbot added a comment.Apr 17 2023, 9:48 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-17T09:48:45Z] <ladsgroup@deploy2002> Finished scap: Backport for [[gerrit:905626|Also broadcast RCFeed/IRC events to irc1002/irc2002 (T331702)]] (duration: 44m 21s)

ops-monitoring-bot added a comment.Apr 25 2023, 3:22 PM

Icinga downtime and Alertmanager silence (ID=a3bb7d5d-06b6-47e0-986b-d299e4bb9639) set by jmm@cumin2002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Non-functional, WIP for Bullseye update

irc2002.wikimedia.org
gerritbot added a comment.Thu, Jun 13, 12:24 PM

Change #1043038 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] udpmxircecho: One more Python 2 -> Python 3 fix

https://gerrit.wikimedia.org/r/1043038

gerritbot added a project: Patch-For-Review.Thu, Jun 13, 12:24 PM
gerritbot added a comment.Thu, Jun 13, 2:06 PM

Change #1043038 merged by Muehlenhoff:

[operations/puppet@production] udpmxircecho: One more Python 2 -> Python 3 fix

https://gerrit.wikimedia.org/r/1043038

Maintenance_bot removed a project: Patch-For-Review.Thu, Jun 13, 2:32 PM
gerritbot added a comment.Mon, Jun 17, 12:35 PM

Change #1046659 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] irc.w.o: Add support for Bullseye

https://gerrit.wikimedia.org/r/1046659

gerritbot added a project: Patch-For-Review.Mon, Jun 17, 12:35 PM
Stashbot added a comment.Tue, Jun 18, 7:56 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-18T07:56:51Z] <moritzm> uploaded python-irc 8.5.3+dfsg-4+wmf1 to apt.wikimedia.org T331702

gerritbot added a comment.Tue, Jun 18, 8:44 AM

Change #1046659 merged by Muehlenhoff:

[operations/puppet@production] irc.w.o: Add support for Bullseye

https://gerrit.wikimedia.org/r/1046659

gerritbot added a comment.Tue, Jun 18, 9:03 AM

Change #1047021 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mw-irc: Fix installation of Prometheus Python client package

https://gerrit.wikimedia.org/r/1047021

gerritbot added a comment.Tue, Jun 18, 9:28 AM

Change #1047021 merged by Muehlenhoff:

[operations/puppet@production] mw-irc: Fix installation of Prometheus Python client package

https://gerrit.wikimedia.org/r/1047021

Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 18, 9:31 AM
• MoritzMuehlenhoff added a comment.Tue, Jun 18, 10:15 AM

Bullseye-based servers are up and running, one can connect to irc1002.wikimedia.org and irc2002.wikimedia.org the same way as for irc1001/irc2001.

However, when joining the #en.wikipedia channel on irc1001 (current) and irc1002 (new) and comparing the events, there's less output for the latter. So there seems to be some issue with broadcasted mediawiki events (or the way these are ingested on the new installations)

andrea.denisse unsubscribed.Tue, Jun 18, 4:13 PM
gerritbot added a comment.Wed, Jun 19, 7:41 AM

Change #1047430 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/deployment-charts@master] Add new IRC servers also to the k8s hosts

https://gerrit.wikimedia.org/r/1047430

gerritbot added a project: Patch-For-Review.Wed, Jun 19, 7:41 AM
• MoritzMuehlenhoff added a comment.Wed, Jun 19, 7:57 AM
In T331702#9902698, @MoritzMuehlenhoff wrote:

Bullseye-based servers are up and running, one can connect to irc1002.wikimedia.org and irc2002.wikimedia.org the same way as for irc1001/irc2001.

However, when joining the #en.wikipedia channel on irc1001 (current) and irc1002 (new) and comparing the events, there's less output for the latter. So there seems to be some issue with broadcasted mediawiki events (or the way these are ingested on the new installations)

The reason for this turned out to be unrelated to the ircecho changes/new OS: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/905626 added irc1002/irc2002 to mediawiki-config back in April 2023, but that never made it into the deployment-charts/wikikube and consequently there was only the remaining mw-on-baremetal traffic until it dried out fully yesterday evening.

gerritbot added a comment.Wed, Jun 19, 10:35 AM

Change #1047430 merged by Muehlenhoff:

[operations/deployment-charts@master] Add new IRC servers also to the k8s hosts

https://gerrit.wikimedia.org/r/1047430

Maintenance_bot removed a project: Patch-For-Review.Wed, Jun 19, 11:30 AM
gerritbot added a comment.Thu, Jun 20, 6:54 AM

Change #1047851 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Switch irc.wikimedia to one of the new Bullseye hosts

https://gerrit.wikimedia.org/r/1047851

gerritbot added a project: Patch-For-Review.Thu, Jun 20, 6:54 AM
gerritbot added a comment.Thu, Jun 20, 6:58 AM

Change #1047851 merged by Muehlenhoff:

[operations/dns@master] Switch irc.wikimedia to one of the new Bullseye hosts

https://gerrit.wikimedia.org/r/1047851

Stashbot added a comment.Thu, Jun 20, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T07:04:00Z] <moritzm> failover irc.wikimedia.org to the new Bullseye servers T331702

Maintenance_bot removed a project: Patch-For-Review.Thu, Jun 20, 7:30 AM
Stashbot added a comment.Thu, Jun 20, 8:08 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T08:08:14Z] <moritzm> reboot of irc1001 to nudge clients to re-connect to the new bullseye host T331702

• MoritzMuehlenhoff added a comment.Thu, Jun 20, 8:15 AM

I've pointed the irc.w.o CNAME to irc1002 and rebooted to irc1001 to force a failover to 1002. All bots which were connected to #de.wikipedia reconnected to the new node.

I'll decommission the old servers on Monday.

gerritbot added a comment.Mon, Jun 24, 12:03 PM

Change #1049137 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/deployment-charts@master] irc.wikimedia.org: Stop sending broadcast events to the old buster nodes

https://gerrit.wikimedia.org/r/1049137

gerritbot added a project: Patch-For-Review.Mon, Jun 24, 12:04 PM
gerritbot added a comment.Tue, Jun 25, 10:05 AM

Change #1049137 merged by Muehlenhoff:

[operations/deployment-charts@master] irc.wikimedia.org: Stop sending broadcast events to the old buster nodes

https://gerrit.wikimedia.org/r/1049137

gerritbot added a comment.Tue, Jun 25, 10:13 AM

Change #1049503 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch old irc hosts to insetup::buster

https://gerrit.wikimedia.org/r/1049503

gerritbot added a comment.Tue, Jun 25, 10:29 AM

Change #1049503 merged by Muehlenhoff:

[operations/puppet@production] Switch old irc hosts to insetup::buster

https://gerrit.wikimedia.org/r/1049503

Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 25, 10:31 AM
ops-monitoring-bot added a comment.Thu, Jun 27, 8:26 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: irc2001.wikimedia.org

  • irc2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
gerritbot added a comment.Thu, Jun 27, 8:28 AM

Change #1050260 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove irc1001/irc2001 from site.pp

https://gerrit.wikimedia.org/r/1050260

gerritbot added a project: Patch-For-Review.Thu, Jun 27, 8:28 AM
gerritbot added a comment.Thu, Jun 27, 8:30 AM

Change #1050261 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/mediawiki-config@master] Remove irc1001/irc2001 from mediawiki-config

https://gerrit.wikimedia.org/r/1050261

gerritbot added a comment.Thu, Jun 27, 8:35 AM

Change #1050260 merged by Muehlenhoff:

[operations/puppet@production] Remove irc1001/irc2001 from site.pp

https://gerrit.wikimedia.org/r/1050260

ops-monitoring-bot added a comment.Thu, Jun 27, 8:37 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: irc1001.wikimedia.org

  • irc1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
• MoritzMuehlenhoff closed this task as Resolved.Thu, Jun 27, 8:38 AM
• MoritzMuehlenhoff claimed this task.

The old nodes have been decommissioned, all done.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits