Stuck global renames

Sometimes you'll find that a rename becomes stuck at a project (see for example phab:T365533). In that case, please follow these instructions:

• Wait at least 3 hours since the rename started. The system will automatically retry failed renames a few times.
• Please open a task in Phabricator (this link wil automatically add the right tags). In there please:
• post a link to the Special:GlobalRenameProgress link to the account whose rename has become stuck;
• mention the old and new user name.

A developer with production shell access should run the fixStuckGlobalRename.php script on the maintenance server to requeue the job:

mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=<...> --logwiki=metawiki '<old username>' '<new username>'

• wiki: the database name of the wiki where the rename job has become stuck, e.g. ptwiki. If it is stuck on multiple wikis, use the one that comes first alphabetically.
• logwiki: the database name of the wiki where the global rename was started. This will always be metawiki.
• --ignorestatus is a parameter that can be added to fix a global rename that's stuck in running state even though the job is not actually running (so trying to run the script without this switch gives skipping duplicate rename from...). This used to happen a lot in the past due to fragile error handling; shouldn't be the case anymore. Use with care; could make a mess if the job really is running.

Once you are done, log in the production SAL that you've run the script, by issuing an IRC command in #wikimedia-operations such as:

• !log T12345 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ptwiki --logwiki=metawiki 'OldName' 'NewName'

In case of doubt, ask the maintainers of CentralAuth.

We don't usually do this because it is time-consuming and these days stuck renames are pretty much one-off infrastructure issues (jobs getting lost, which cannot be proven or disproven via logs). But if there are more stuck renames than usual and you suspect something is wrong, or requeueing doesn't help, check Logstash:

• Searching for the old or new username in the CentralAuth log channel (dashboard) should catch the more obvious things and narrow down the time range (based on when the job ran on the preceding wikis).
• Searching for the job runner URL might reveal more elusive errors like exceptions and DB timeouts (dashboard; try to filter by wiki and/or time range).