Create Task
Maniphest T175706

Progressively switch Wikimedia wikis from Tidy to RemexHTML
Closed, ResolvedPublic
Actions

Description

As of 2018-06-14, the following groups remain to be converted:

  • 78 Wikipedias
  • 12 Wiktionaries
  • 2 Wikibookses
  • 0 Wikiquotes
  • 0 Wikivoyages
  • 10 Wikisources
  • 0 Wikinewses
  • 0 Wikiversities
  • 2 Shared and testing wikis
  • 0 Wikimedias: T188881
  • 0 Private wikis

On July 5, 2018, all wikis that are still using Tidy will be switched over to use RemexHtml.
https://quarry.wmflabs.org/query/27652 shows linter counts for ns0 for these remaining wikis (except commons).

Details

SubjectRepoBranchLines +/-
Replace deprecated use of $wgUseTidy in Special:ExpandTemplatesmediawiki/coremaster+1 -1
Remove $wgUseTidy and $wgTidyConfig from configurationoperations/mediawiki-configmaster+0 -11
Replace Tidy with RemexHtml everywhereoperations/mediawiki-configmaster+0 -115
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• ssastryT175706 Progressively switch Wikimedia wikis from Tidy to RemexHTML
 ResolvedJdforrester-WMFT175095 Enable RemexHTML on mediawiki.org and testwiki
 ResolvedKizuleT175971 Enable RemexHTML on wikitech.wikimedia.org
 DeclinedKizuleT175972 Enable RemexHTML on wikimediafoundation.org
 ResolvedKizuleT176088 Enable RemexHTML on eswikiversity
 ResolvedJdforrester-WMFT176150 Enable RemexHTML on fawiki
 ResolvedJdforrester-WMFT177989 Enable RemexHTML on nowiki
 Resolved• ssastryT181188 Enable RemexHTML on dewiki
 Resolved• ssastryT181190 Enable RemexHTML on itwiki
 Resolved• ssastryT182042 Enable RemexHTML on 170 additional wikis with zero high-priority linter errors (as of Mon, Dec 4, 2017)
 Resolved• ssastryT184656 Enable RemexHTML on additional wikis with < 10 high-priority linter errors in all linter categories
 Resolved• ssastryT185945 Enable RemexHTML on ruwiki, fiwiki, svwiki, hewiki
 ResolvedtstarlingT89331 Replace HTML4 Tidy in MW parser with an equivalent HTML5 based tool
 Resolved• ssastryT120345 Set up mass visual diff testing with a custom install of mediawiki
 Resolvedbd808T125882 Create labs project for parsing team to run experimental mediawiki installs
 Resolved• ssastryT134423 Deprecate nonstandard behavior of self-closed HTML tags in wikitext.
 ResolvedcscottT136652 Add tracking category to pages that use deprecated Tidy-specific self-closing tags (<b />, <div />, etc.)
 DeclinedtstarlingT136668 Puppetize HTML5Depurate
 ResolvedtstarlingT136669 Merge mediawiki.raggett.css into other CSS + remove conditional enabling for Tidy
 Resolved• ElitreT145530 Support the Parsing team with the "Remove Tidy dependency from MediaWiki output" goal
 DeclinedNoneT155634 Tidy strips whitespace after HTML tags AND adds newlines between HTML tags
 Resolved• ssastryT175099 PHP-parser + Remex combo output differs from PHP-parser + Tidy combo on some dl-dt wikitext snippets
 ResolvedJdforrester-WMFT185753 MediaWiki should default to using RemexHtml for tidy
 Resolved• ssastryT188167 Run parser tests with RemexHtml as the tidy implementation
 ResolvedJdforrester-WMFT191670 Determine future of MWTidy::checkErrors in a Remex world
 Resolved• ssastryT188000 Enable RemexHTML on all wikinews wikis
 Resolved• ssastryT188008 Enable RemexHTML on a bunch of miscellaneous wikis
 Resolved• ssastryT188009 Enable RemexHTML on all private wikis
 Resolved• ssastryT188010 Enable RemexHTML on additional wikis with < 25 errors in all high priority categories
 Resolved• ssastryT188352 Enable RemexHTML on all closed wikis
 Resolved• ssastryT188869 Enable RemexHTML on kowiki, mznwiki, warwiki, cebwiki, nlwiki
 Resolved• ssastryT188880 Enable RemexHTML on all wikiversity wikis
 Resolved• ssastryT188881 Enable RemexHTML on all wikimedia chapters/user groups wikis
 Resolved• ssastryT190726 Enable RemexHTML on all wikiquotes (except frwikiquote)
 Resolved• ssastryT190731 Enable RemexHTML on additional wikis with < 50 errors in all high priority categories
 Resolved• ssastryT192299 Enable RemexHTML on additional wikis with < 100 errors in all high priority categories
 Resolved• ssastryT192301 Enable RemexHTML on frwikiquote
 Resolved• ssastryT192386 Enable RemexHTML on metawiki
 ResolvedPcoombeT192480 Fix Lint errors for current fundraising banners
 Resolved• ssastryT192821 Enable RemexHTML on wikibook wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace)
 DuplicateNoneT192950 Find unbalanced HTML in current signature of users and inform the user and/or reset the signature
 Resolved• ssastryT193685 Enable RemexHTML on all wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace)
 Resolved• ssastryT195263 Do another round of Tidy replacement on May 30th and June 14th (last deploys before final switch)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
PeterBowman subscribed.Jan 22 2018, 8:10 PM
TTO subscribed.EditedJan 25 2018, 8:23 AM

What is going to happen on locked wikis that have linter errors? Will a staff member fix these errors themselves? (Not all locked wikis are abandoned and irrelevant - I'm thinking in particular of the Wikimania wikis, which contain archived content that people may wish to view.)

• Elitre added a comment.Jan 26 2018, 6:39 PM

Do stewards have the ability to edit locked wikis?

• ssastry closed subtask T185945: Enable RemexHTML on ruwiki, fiwiki, svwiki, hewiki as Resolved.Feb 6 2018, 2:37 PM
• ssastry closed subtask T184656: Enable RemexHTML on additional wikis with < 10 high-priority linter errors in all linter categories as Resolved.Feb 7 2018, 12:34 AM
Jdforrester-WMF removed a parent task: T89331: Replace HTML4 Tidy in MW parser with an equivalent HTML5 based tool.Feb 10 2018, 12:25 AM
Jdforrester-WMF added a subtask: T185753: MediaWiki should default to using RemexHtml for tidy.
Jdforrester-WMF edited subtasks, added: T89331: Replace HTML4 Tidy in MW parser with an equivalent HTML5 based tool; removed: T185753: MediaWiki should default to using RemexHtml for tidy.
Jdforrester-WMF closed subtask T188000: Enable RemexHTML on all wikinews wikis as Resolved.Feb 27 2018, 12:23 AM
Jdforrester-WMF closed subtask T188009: Enable RemexHTML on all private wikis as Resolved.
Jdforrester-WMF closed subtask T188008: Enable RemexHTML on a bunch of miscellaneous wikis as Resolved.
TTO added a comment.Feb 27 2018, 12:35 AM
In T175706#3925097, @Elitre wrote:

Do stewards have the ability to edit locked wikis?

I believe so. The issue is whether there are stewards with the required technical skill set and/or the time to deal with the problems, though.

Jdforrester-WMF updated the task description. (Show Details)Feb 27 2018, 1:29 AM
• Elitre added a comment.Feb 27 2018, 11:11 AM
In T175706#4004495, @TTO wrote:
In T175706#3925097, @Elitre wrote:

Do stewards have the ability to edit locked wikis?

I believe so. The issue is whether there are stewards with the required technical skill set and/or the time to deal with the problems, though.

If the errors are minor, then they probably aren't worth fixing. I asked the question in the related task.

Tgr mentioned this in T133410: Deploy TemplateStyles to WMF production.Feb 28 2018, 2:49 AM
• ssastry closed subtask T188352: Enable RemexHTML on all closed wikis as Resolved.Mar 4 2018, 6:31 PM
• MZMcBride subscribed.Mar 6 2018, 5:27 AM
• ssastry closed subtask T188010: Enable RemexHTML on additional wikis with < 25 errors in all high priority categories as Resolved.Mar 13 2018, 11:17 PM
• ssastry closed subtask T188869: Enable RemexHTML on kowiki, mznwiki, warwiki, cebwiki, nlwiki as Resolved.Mar 14 2018, 5:29 PM
• Deskana mentioned this in T189834: [Epic] Deploy TemplateStyles to Wikivoyages.Mar 15 2018, 11:02 PM
• ssastry closed subtask T188880: Enable RemexHTML on all wikiversity wikis as Resolved.Mar 21 2018, 5:13 PM
• ssastry updated the task description. (Show Details)Mar 21 2018, 5:16 PM
Jdforrester-WMF updated the task description. (Show Details)Mar 22 2018, 9:06 PM
BethNaught subscribed.Apr 2 2018, 7:39 PM
Alfa80 subscribed.Apr 3 2018, 2:32 AM
RazeSoldier subscribed.Apr 3 2018, 7:42 AM
• ssastry closed subtask T188881: Enable RemexHTML on all wikimedia chapters/user groups wikis as Resolved.Apr 4 2018, 5:19 PM
• ssastry closed subtask T190726: Enable RemexHTML on all wikiquotes (except frwikiquote) as Resolved.
eranroz added a subtask: T191516: Parser generates broken lists when closing table (td, th, tr, table) tags are on the same line as the list item.Apr 5 2018, 4:00 PM
• ssastry removed a subtask: T191516: Parser generates broken lists when closing table (td, th, tr, table) tags are on the same line as the list item.Apr 5 2018, 4:17 PM
He7d3r subscribed.Apr 6 2018, 7:37 PM
Louperivois subscribed.EditedApr 10 2018, 11:39 PM

It seems the VisualEditor is generating new Lint Errors. This should be fixed before deployment. T164053

• Elitre added a comment.Apr 11 2018, 4:19 PM
In T175706#4122029, @Louperivois wrote:

It seems the VisualEditor is generating new Lint Errors. This should be fixed before deployment. T164053

I see only one example in the linked task. Can you clear out user mistake/manipulation?

• ssastry closed subtask T190731: Enable RemexHTML on additional wikis with < 50 errors in all high priority categories as Resolved.Apr 12 2018, 3:14 AM
• ssastry updated the task description. (Show Details)Apr 12 2018, 3:17 AM
Jdforrester-WMF closed subtask T89331: Replace HTML4 Tidy in MW parser with an equivalent HTML5 based tool as Resolved.Apr 13 2018, 3:09 AM
• ssastry added a subtask: T192386: Enable RemexHTML on metawiki.Apr 17 2018, 5:49 PM
• ssastry added a subtask: T192821: Enable RemexHTML on wikibook wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace).Apr 23 2018, 5:29 PM
Aklapper mentioned this in T192855: Remex enabled on all wikis in MW 1.30-wmf.30 exposing corruption (signatures coloring unrelated follow-up sections, etc.) on unfixed content.Apr 24 2018, 4:32 PM
• ssastry closed subtask T192301: Enable RemexHTML on frwikiquote as Resolved.Apr 25 2018, 5:13 PM
• ssastry updated the task description. (Show Details)
• ssastry mentioned this in T193438: CL support for addressing linter errors on large wikis.Apr 30 2018, 8:17 PM
Jdforrester-WMF closed subtask T192386: Enable RemexHTML on metawiki as Resolved.May 2 2018, 5:35 PM
• ssastry closed subtask T192299: Enable RemexHTML on additional wikis with < 100 errors in all high priority categories as Resolved.May 2 2018, 6:00 PM
Jdforrester-WMF updated the task description. (Show Details)May 2 2018, 6:04 PM
StevenJ81 subscribed.May 2 2018, 10:41 PM

I'm not sure if incubatorwiki was switched yet. But Xaosflux and his bot are going to start working on the wiki this weekend. As of the moment, we have over 3,000 high-priority and over 10,000 medium-priority errors on incubatorwiki, and because of the nature of the wiki and its test communities, it has not been so easy to get all the errors corrected. So I'm inclined to ask, if this is still possible, for incubatorwiki to be in the absolute final set of wikis to be converted, so we have time to address this. Thank you.

• ssastry added a subtask: T193685: Enable RemexHTML on all wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace).May 2 2018, 10:42 PM
• ssastry subscribed.May 2 2018, 10:46 PM
In T175706#4176795, @StevenJ81 wrote:

I'm not sure if incubatorwiki was switched yet. But Xaosflux and his bot are going to start working on the wiki this weekend. As of the moment, we have over 3,000 high-priority and over 10,000 medium-priority errors on incubatorwiki, and because of the nature of the wiki and its test communities, it has not been so easy to get all the errors corrected. So I'm inclined to ask, if this is still possible, for incubatorwiki to be in the absolute final set of wikis to be converted, so we have time to address this. Thank you.

You don't need to address medium-priority issues. Only high-priority ones.

And, yes, if a wiki still has too many of those issues, we will switch end of June as planned.

StevenJ81 added a comment.May 3 2018, 1:53 PM

Thank you for your response, @ssastry. No, I understand that an update will happen then. I just want to give us the maximum time possible before then to clean up.

• ssastry mentioned this in T193414: Servers using tidy-html5 are rendering pages differently, especially with <bdi>.May 9 2018, 9:46 PM
• ssastry closed subtask T192821: Enable RemexHTML on wikibook wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace) as Resolved.May 9 2018, 11:15 PM
• ssastry updated the task description. (Show Details)
Jdforrester-WMF updated the task description. (Show Details)May 10 2018, 3:37 PM
• ssastry updated the task description. (Show Details)May 16 2018, 5:41 PM
• ssastry closed subtask T193685: Enable RemexHTML on all wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace) as Resolved.
Lofhi subscribed.May 23 2018, 5:50 AM
Jdforrester-WMF closed subtask T195263: Do another round of Tidy replacement on May 30th and June 14th (last deploys before final switch) as Resolved.May 30 2018, 6:32 PM
Jdforrester-WMF updated the task description. (Show Details)
• ssastry reopened subtask T195263: Do another round of Tidy replacement on May 30th and June 14th (last deploys before final switch) as Open.May 30 2018, 6:44 PM
• ssastry updated the task description. (Show Details)Jun 14 2018, 4:52 PM
• ssastry updated the task description. (Show Details)
• ssastry updated the task description. (Show Details)Jun 14 2018, 5:06 PM
• ssastry closed subtask T195263: Do another round of Tidy replacement on May 30th and June 14th (last deploys before final switch) as Resolved.Jun 14 2018, 6:12 PM
• ssastry updated the task description. (Show Details)Jun 15 2018, 7:15 PM
AfroThundr3007730 subscribed.Jun 18 2018, 3:34 PM
AfroThundr3007730 mentioned this in T197603: Deploy TemplateStyles to the English Wikipedia on 2018-07-19.Jun 18 2018, 3:39 PM
gerritbot subscribed.Jun 26 2018, 4:27 PM

Change 442142 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/mediawiki-config@master] Replace Tidy with RemexHtml everywhere

https://gerrit.wikimedia.org/r/442142

gerritbot added a project: Patch-For-Review.Jun 26 2018, 4:27 PM
• ssastry added a comment.Jun 30 2018, 8:42 AM
In T175706#4316182, @gerritbot wrote:

Change 442142 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/mediawiki-config@master] Replace Tidy with RemexHtml everywhere

https://gerrit.wikimedia.org/r/442142

This final switch is now scheduled to be deployed on July 5th during the European mid-day SWAT slot.

• Vvjjkkii reopened subtask T195263: Do another round of Tidy replacement on May 30th and June 14th (last deploys before final switch) as Open.Jul 1 2018, 1:08 AM
• Vvjjkkii reopened subtask T193685: Enable RemexHTML on all wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace) as Open.Jul 1 2018, 1:12 AM
• Vvjjkkii reopened subtask T192950: Find unbalanced HTML in current signature of users and inform the user and/or reset the signature as Open.
AfroThundr3007730 closed subtask T195263: Do another round of Tidy replacement on May 30th and June 14th (last deploys before final switch) as Resolved.Jul 1 2018, 5:08 AM
AfroThundr3007730 closed subtask T193685: Enable RemexHTML on all wikis with < 100 linter errors in all high priority linter categories in ns0 (main namespace) as Resolved.
gerritbot added a comment.Jul 3 2018, 3:41 PM

Change 443645 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/mediawiki-config@master] Remove $wgUseTidy and $wgTidyConfig from configuration

https://gerrit.wikimedia.org/r/443645

gerritbot added a comment.Jul 5 2018, 1:18 PM

Change 442142 merged by jenkins-bot:
[operations/mediawiki-config@master] Replace Tidy with RemexHtml everywhere

https://gerrit.wikimedia.org/r/442142

Stashbot subscribed.Jul 5 2018, 1:26 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-05T13:26:13Z] <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:442142| Replace Tidy with RemexHtml everywhere (T175706)]] (duration: 00m 50s)

• ssastry closed this task as Resolved.Jul 5 2018, 1:30 PM
• ssastry claimed this task.
RazeSoldier added a project: User-notice.Jul 5 2018, 2:23 PM

Already replace Tidy with RemexHtml on all wikis.

Johan moved this task from To Triage to Already announced/Archive on the User-notice board.Jul 5 2018, 8:54 PM
Johan moved this task from Already announced/Archive to In current Tech/News draft on the User-notice board.Jul 5 2018, 9:34 PM
Liuxinyu970226 unsubscribed.Jul 6 2018, 4:00 AM
Johan moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Jul 12 2018, 4:40 AM
Mainframe98 mentioned this in T201452: Second part of my talk page in Wikipedia has large fonts.Aug 7 2018, 8:23 PM
gerritbot added a comment.Sep 20 2018, 8:05 PM

Change 443645 merged by jenkins-bot:
[operations/mediawiki-config@master] Remove $wgUseTidy and $wgTidyConfig from configuration

https://gerrit.wikimedia.org/r/443645

gerritbot added a comment.Sep 27 2018, 5:36 PM

Change 463328 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Replace deprecated use of $wgUseTidy in Special:ExpandTemplates

https://gerrit.wikimedia.org/r/463328

gerritbot added a comment.Sep 28 2018, 10:36 PM

Change 463328 merged by jenkins-bot:
[mediawiki/core@master] Replace deprecated use of $wgUseTidy in Special:ExpandTemplates

https://gerrit.wikimedia.org/r/463328

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)).Sep 28 2018, 11:00 PM
Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:54 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 13 2022, 2:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits