I think the problem is the same we faced here: https://phabricator.wikimedia.org/T359306
RESTBase which proxies requests to parsoid uses lowercase GET:

"method":"get"

More specifically:
https://gerrit.wikimedia.org/g/mediawiki/services/change-propagation/deploy/+/464d3bcc59e68764be3e3765e5511e2e3eff4559/node_modules/hyperswitch/lib/hyperswitch.js#456
https://gerrit.wikimedia.org/g/mediawiki/services/change-propagation/deploy/+/464d3bcc59e68764be3e3765e5511e2e3eff4559/node_modules/hyperswitch/lib/hyperswitch.js#439

There is already a patch for this issue here:
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009237

@daniel 👋 Ok to go ahead with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009237?

Ah, @Jdforrester-WMF came to the rescue, thank you :)

Change 1009500 had a related patch set uploaded (by Jaime Nuche; author: Daniel Kinzler):

[mediawiki/core@wmf/1.42.0-wmf.21] REST: allow lower-case method names

https://gerrit.wikimedia.org/r/1009500

Change 1009493 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/alerts@master] mw-on-k8s: Add MediaWikiHTTPErrorRatio alert

https://gerrit.wikimedia.org/r/1009493

@Jgiannelos @Jdforrester-WMF I've created a backport patch to unblock the train: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009500. But it would still need a pair of eyes before I'm comfortable backporting it

Change 1009500 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.21] REST: allow lower-case method names

https://gerrit.wikimedia.org/r/1009500

https://gerrit.wikimedia.org/r/1009500 didn't solve the issue: https://grafana.wikimedia.org/goto/KZMbFoASz?orgId=1

Change 1009542 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009542

@jnuche Can you try this one? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009542

We can put it in mwdebug before merge and check it.

Change 1009544 had a related patch set uploaded (by Jaime Nuche; author: Daniel Kinzler):

[mediawiki/core@wmf/1.42.0-wmf.21] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009544

@daniel thanks, created a version for 1.42.0-wmf.21 too.

We can put it in mwdebug before merge and check it.

@Ladsgroup would you be up to do that? (EDIT: I think all parsoid is in K8s now, so I'm not sure how to go about doing that)

If someone gives me an example of a failure url (the one on the description works now, I guess rollback?) I can try.

This one currently fails: https://pl.wikiquote.org/api/rest_v1/page/html/Baldur's_Gate/565992?redirect=false

In T359509#9611730, @Ladsgroup wrote:

If someone gives me an example of a failure url (the one on the description works now, I guess rollback?) I can try.

Note that we can't try this on mwdebug, since the request goes through restbase.

I put the patch in mwdebug2001 and tested it and it doesn't fix it. We need to find a way to test this.

More debugging:

• I am testing on a local RESTBase instance connected to mediawiki beta (master)
• Without any changes to the codebase RESTBase will send a request to /w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/pagebundle
• The response fails with error:
  • 'The GET request method does not accept a request body'
  • It indicates that indeed MW patch makes the method name uppercase

• If I manually patch the underlying library that makes the request and replace get with GET i don't see any errors

Here are the raw debug requests from my env:

• Default with lowercase method names:
  • https://phabricator.wikimedia.org/P58645
• Patched library
  • https://phabricator.wikimedia.org/P58645

Body is null in both, the only difference I see that the underlying library gzips the lowercase version but doesn't gzip the uppercase version.

In T359509#9611803, @Ladsgroup wrote:

I put the patch in mwdebug2001 and tested it and it doesn't fix it. We need to find a way to test this.

restbase isn't going to hit mwdebug2001... would be nice if we had a way to tell restbase to hit a debug server.

Body is null in both, the only difference I see that the underlying library gzips the lowercase version but doesn't gzip the uppercase version.

Perhaps it disables the body for upper-case GET, but since we have been using lower-case get, we have been sending gezipped bodies with get requests?

This workaround looks like it fixes things on my local RESTBase instance:
https://github.com/wikimedia/restbase/pull/1343

Change 1009544 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.21] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009544

Mentioned in SAL (#wikimedia-operations) [2024-03-07T15:48:49Z] <jnuche@deploy2002> Started scap: Backport for [[gerrit:1009544|REST: ignore request body on GET requests (T359509)]]

Mentioned in SAL (#wikimedia-operations) [2024-03-07T15:50:22Z] <jnuche@deploy2002> jnuche: Backport for [[gerrit:1009544|REST: ignore request body on GET requests (T359509)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-03-07T15:59:56Z] <jnuche@deploy2002> Finished scap: Backport for [[gerrit:1009544|REST: ignore request body on GET requests (T359509)]] (duration: 11m 06s)

This looks like its working now if I query restbase:

curl -v  "http://restbase.svc.codfw.wmnet:7233/test.wikipedia.org/v1/page/summary/Dog"

From the parsoid health chart:
40Xs responses are down to the previous rate

Thank you all for investigating! And sorry for causing a mess...

We need a client patch. It's not OK to use lower-case method names.

In T359509#9613793, @tstarling wrote:

We need a client patch. It's not OK to use lower-case method names.

Yea, and it's also not OK to send content-length: 0 with a GET request.

But some clients do - in particular, RESTbase does.

Change 1010932 had a related patch set uploaded (by Jgiannelos; author: Daniel Kinzler):

[mediawiki/core@wmf/1.42.0-wmf.22] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1010932

Change 1009542 merged by jenkins-bot:

[mediawiki/core@master] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009542

Change 1010932 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.22] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1010932

Mentioned in SAL (#wikimedia-operations) [2024-03-14T13:36:30Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:1010932|REST: ignore request body on GET requests (T359509)]]

Mentioned in SAL (#wikimedia-operations) [2024-03-14T13:38:43Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 jgiannelos and lucaswerkmeister-wmde: Backport for [[gerrit:1010932|REST: ignore request body on GET requests (T359509)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-03-14T13:58:37Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:1010932|REST: ignore request body on GET requests (T359509)]] (duration: 22m 06s)

Metrics looks better now after the backport fix:

Also the URL from the original report:

jgiannelos@deploy2002:~$ curl -v -o /dev/null "http://restbase.svc.codfw.wmnet:7233/it.wikipedia.org/v1/page/html/Iremel%27"

< HTTP/1.1 200

jgiannelos@deploy2002:~$