Create Task
Maniphest T354855

ferm sometimes fails to restart on Kubernetes workers via xtables lock held by kube-proxy
Closed, ResolvedPublic
Actions

Description

On Kubernetes workers ferm sometimes fails to restart (these restarts are e.g. triggered by Puppet if a central Ferm macro gets updated). One example:

Jan 03 19:30:08 mw1465 systemd[1]: Stopped ferm firewall configuration.
Jan 03 19:30:08 mw1465 systemd[1]: Starting ferm firewall configuration...
Jan 03 19:30:08 mw1465 ferm[1868235]: Starting Firewall: ferm
Jan 03 19:30:08 mw1465 ferm[1868274]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Jan 03 19:30:08 mw1465 ferm[1868238]: Failed to run /usr/sbin/iptables-legacy-restore
Jan 03 19:30:08 mw1465 ferm[1868238]: Firewall rules rolled back.
Jan 03 19:30:08 mw1465 ferm[1868281]:  failed!
Jan 03 19:30:08 mw1465 systemd[1]: ferm.service: Main process exited, code=exited, status=1/FAILURE
Jan 03 19:30:08 mw1465 systemd[1]: ferm.service: Failed with result 'exit-code'.
Jan 03 19:30:08 mw1465 systemd[1]: Failed to start ferm firewall configuration.
Jan 08 15:18:08 mw1465 systemd[1]: Starting ferm firewall configuration...

These do not recover automatically with the subsequent Puppet run, apparently because this error condition does not get detected by ferm-status.

We could explore whether there's a way to pass -w to iptables-save/iptables-restore via Ferm (from a quick look that doesn't exist, needs a closer look at the sources)

Details

SubjectRepoBranchLines +/-
check_ferm: Add -w 5 to iptables checkoperations/puppetproduction+1 -1
P:kubernetes:node: Autorestart ferm.serviceoperations/puppetproduction+15 -0
kubernetes: Space out ferm icinga checkoperations/puppetproduction+27 -21
ferm: Check ferm.service status in ferm_status.pyoperations/puppetproduction+15 -4
Customize query in gerrit

Related Objects

Event Timeline

• MoritzMuehlenhoff created this task.Jan 11 2024, 1:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2024, 1:32 PM
• MoritzMuehlenhoff triaged this task as Medium priority.Jan 11 2024, 1:32 PM
Stashbot added a comment.Jan 23 2024, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T12:17:11Z] <claime> Restarting ferm.service on k8s node mw1495.eqiad.wmnet - T354855

Stashbot added a comment.Jan 25 2024, 11:26 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-25T11:26:43Z] <claime> Restarting ferm.service on k8s node kubernetes2036.codfw.wmnet - T354855

Stashbot added a comment.Jan 29 2024, 1:26 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-29T13:26:33Z] <claime> Restarting ferm.service on k8s node kubernetes2055 - T354855

Stashbot added a comment.Feb 2 2024, 12:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-02T12:16:00Z] <claime> Restarting ferm.service on k8s node mw1424 - T354855

Stashbot added a comment.Feb 21 2024, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-21T14:08:25Z] <claime> restarted ferm.service on kubernetes2055.codfw.wmnet mw2440.codfw.wmnet mw2297.codfw.wmnet kubernetes2016.codfw.wmnet - T354855

gerritbot added a comment.Feb 23 2024, 1:17 PM

Change 1005978 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] ferm: Check ferm.service status in ferm_status.py

https://gerrit.wikimedia.org/r/1005978

gerritbot added a project: Patch-For-Review.Feb 23 2024, 1:17 PM
Stashbot added a comment.Mar 4 2024, 11:47 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T11:47:54Z] <claime> Disabling puppet on C:profile::firewall::log::ferm to deploy 1005978 - T354855

Stashbot added a comment.Mar 4 2024, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T11:48:56Z] <claime> Disregard previous puppet disable message, waiting a bit T354855

Stashbot added a comment.Mar 4 2024, 12:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T12:22:51Z] <claime> Disabling puppet on C:profile::firewall::log::ferm to deploy new ferm_status.py - T354855

gerritbot added a comment.Mar 4 2024, 12:27 PM

Change 1005978 merged by Clément Goubert:

[operations/puppet@production] ferm: Check ferm.service status in ferm_status.py

https://gerrit.wikimedia.org/r/1005978

Stashbot added a comment.Mar 4 2024, 12:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T12:28:32Z] <claime> Enabling puppet on kubernetes2019 to test new ferm_status.py - T354855

Stashbot added a comment.Mar 4 2024, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T12:30:53Z] <claime> Enabling puppet on mw2322 to test new ferm_status.py - T354855

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2024, 12:31 PM
Stashbot added a comment.Mar 4 2024, 12:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T12:33:11Z] <claime> Enabling puppet on puppetboard2003 to test new ferm_status.py - T354855

Stashbot added a comment.Mar 4 2024, 12:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-04T12:38:06Z] <claime> Re-enabling puppet on C:profile::firewall::log::ferm to deploy new ferm_status.py - T354855

Clement_Goubert closed this task as Resolved.Mar 4 2024, 12:43 PM
Clement_Goubert claimed this task.
Clement_Goubert subscribed.

Deployed, puppet now restarts ferm.service if the systemd unit's status is failed.

gerritbot added a comment.May 14 2024, 1:17 PM

Change #1031440 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] kubernetes: Space out ferm icinga check

https://gerrit.wikimedia.org/r/1031440

gerritbot added a project: Patch-For-Review.May 14 2024, 1:17 PM
gerritbot added a comment.May 14 2024, 2:07 PM

Change #1031440 merged by Clément Goubert:

[operations/puppet@production] kubernetes: Space out ferm icinga check

https://gerrit.wikimedia.org/r/1031440

Maintenance_bot removed a project: Patch-For-Review.May 14 2024, 2:31 PM
Clement_Goubert reopened this task as Open.Thu, Jun 27, 4:32 PM
Clement_Goubert raised the priority of this task from Medium to High.

This issue is biting us again, the time between a puppet run that fails to start ferm.service and the subsequent run is enough to cause issues with eventgate, leading to flurries of Can't enqueue job errors in mediawiki. It is in particular triggered when we add new kubernetes nodes, because that causes all other kubernetes nodes in the cluster to update their ferm rules.

Would adding Restart=on-failure to ferm.service for kubernetes worker be a possible short-term solution @MoritzMuehlenhoff ?

• MoritzMuehlenhoff added a comment.Thu, Jun 27, 5:11 PM
In T354855#9931347, @Clement_Goubert wrote:

This issue is biting us again, the time between a puppet run that fails to start ferm.service and the subsequent run is enough to cause issues with eventgate, leading to flurries of Can't enqueue job errors in mediawiki. It is in particular triggered when we add new kubernetes nodes, because that causes all other kubernetes nodes in the cluster to update their ferm rules.

Would adding Restart=on-failure to ferm.service for kubernetes worker be a possible short-term solution @MoritzMuehlenhoff ?

Possibly, we'd have to try to find out if that works reliably. ferm is a sysvinit script after all and those are the greatest at passing though error states.

The alternative is to untangle the check from puppet runs and instead have it run as a systemd timer more often.

gerritbot added a comment.Tue, Jul 2, 2:07 PM

Change #1051378 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P:kubernetes:node: Autorestart ferm.service

https://gerrit.wikimedia.org/r/1051378

gerritbot added a project: Patch-For-Review.Tue, Jul 2, 2:07 PM
gerritbot added a comment.Tue, Jul 2, 3:10 PM

Change #1051378 merged by Clément Goubert:

[operations/puppet@production] P:kubernetes:node: Autorestart ferm.service

https://gerrit.wikimedia.org/r/1051378

Maintenance_bot removed a project: Patch-For-Review.Tue, Jul 2, 3:31 PM
gerritbot added a comment.Fri, Jul 5, 10:48 AM

Change #1052283 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] check_ferm: Add -w 5 to iptables check

https://gerrit.wikimedia.org/r/1052283

gerritbot added a project: Patch-For-Review.Fri, Jul 5, 10:48 AM
gerritbot added a comment.Fri, Jul 5, 12:01 PM

Change #1052283 merged by Clément Goubert:

[operations/puppet@production] check_ferm: Add -w 5 to iptables check

https://gerrit.wikimedia.org/r/1052283

Maintenance_bot removed a project: Patch-For-Review.Tue, Jul 9, 8:39 PM
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Wed, Jul 10, 2:38 PM
Clement_Goubert closed this task as Resolved.Wed, Jul 10, 2:41 PM

We've only had one spike of job enqueuing errors since merging Restart=on-failure, I think I'll call this resolved for now, and reopen if we see problems again.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits