db1246 fully repooled.

Cloning from db1182 completed successfully.

[+Kwaku]

Stalling until db1164 is back in service: T307198: db1164 fails to POST/boot/etc

I've set the host to 'failed' in netbox: https://netbox.wikimedia.org/dcim/devices/2999/

On the web console, i can see it get as far as this before resetting:

Success.

In T303603#7841068, @Milimetric wrote:

As @Zabe says, T233004 is stalled on this. Is there anything I can help with? This might be more of a "teach Dan how to do this" slowdown than a help, but gets you an extra person to help with these kinds of changes in the future? Anyway, I'm game.

The script dropped the connection after 2h15m.

As an example, i depooled a host (db1120) so i could reboot it. 1.5h later, i'm still waiting for refreshLinkRecommendations to drop it's connection so that i can actually do the reboot. I have no way of knowing when that will happen. I also don't know how bad/disruptive/etc it would be to just reboot it anyway.

In T305469#7843940, @Marostegui wrote:

For what is worth, es2029 and es2030 are scheduled to be done 14th, which is a bank holiday for me, so someone else would need to bring mysql up once that is done.

Note for next time: various references to the username also needed to be updated. E.g.:

• https://gerrit.wikimedia.org/r/c/operations/software/+/777760
• https://gerrit.wikimedia.org/r/c/operations/software/+/777756
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/777754

In T301315#7831250, @jcrespo wrote:

The only thing left is to check (by DBAs) that the backup available at (e.g.) : dbprov1002:/srv/backups/dumps/latest/dump.db_inventory.2022-04-05--03-13-31 looks as expected (content-wise).

In T301315#7821645, @jcrespo wrote:

One last thing- does orchestrator + zarcillo server have a name? Backups (jobs) have a completely arbitrary identifier for monitoring purposes- any suggestion (even if it is orchestrator OR zarcillo containing both)? Alternatively, it is possible to setup separate backup jobs, even for the same server.

In T301315#7821493, @jcrespo wrote:

@Kormat I can help setting up backups, but someone has to make clear first what to backup (what databases?), from which servers/dcs, and how frequently they are needed.

@jcrespo: Can i hand this task over to you? I think the dumps grants probably need some cleanup, if you can have a look at that, too.

In T301315#7811237, @Marostegui wrote:

Regarding the plan I would include the following steps before dropping the database in db2093:

• Check current grants on db2093 to make sure they are the same on db1115