Decommission ganeti1019
Closed, ResolvedPublic

Description

The server had issues before (T367071) and after the repair the reimage still showed failing disks, so there's probably some issue with the controller/mainboard. Since the host is OOW and new replacements are coming in next quarter, we'll go ahead with the decom.

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member and site project (ops-sitename) depending on site of server

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

Related Objects

Event Timeline

Change #1050263 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti1019 from list of active Ganeti nodes in eqiad

https://gerrit.wikimedia.org/r/1050263

Change #1050263 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti1019 from list of active Ganeti nodes in eqiad

https://gerrit.wikimedia.org/r/1050263

MoritzMuehlenhoff updated the task description. (Show Details)

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti1019.eqiad.wmnet

  • ganeti1019.eqiad.wmnet (FAIL)
    • Host not found on Icinga, unable to downtime it
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

VRiley-WMF updated Other Assignee, added: Jclark-ctr.
VRiley-WMF added a subscriber: Jclark-ctr.
VRiley-WMF updated the task description. (Show Details)

Removed the server and ran the decom script.