- Affected components: All MediaWiki extensions that use a parser hook or a Parser.php method.
- Engineer(s) or team for initial implementation: Parsing Team.
- Code steward: Parsing Team.
Motivation
It is well-known that the Parsing Team is seeking to make Parsoid the default wikitext engine for MediaWiki. Given that Parsoid has an entirely different processing model and implementation and pipeline, Parsoid cannot support the exact parsing API (which turns out to just be whatever methods are public in Parser.php) and hooks that the core parser supports.
Given the above, the parsing team has been at work to define a Parsoid Extension API that extensions can use to hook into Parsoid and support the same functionality that they currently implement with the core parser.
Current Status
At this point, Parsoid has a late-draft proposal for such an API and is being elaborated in detail at mw:Parsoid/Extension_API. This page has been evolving since March 2020 and at this time, in August 2020, we consider this to be in a good enough shape to go through a TechCom RFC process. This extension API has been matched with corresponding implementation updates in Parsoid. All of Parsoid's extensions currently in production on Wikimedia wikis strictly follow this extension API. So, this proposal is not just an on-paper API but is actually a real functioning implementation.
Extensions using this API
- Gallery, Pre, Nowiki (Core extensions)
- Cite, Poem
- ImageMap (almost ready)
- RawHTML, StyleTag (Parser Tests)
Admittedly, this API doesn't yet capture the full diversity of use cases out there wrt how extensions interact with the core parser. While we hope to get there eventually, we aim to do that in stages.
Step 1a (DONE) : Extract an extension API out of Parsoid's extension implementations and demonstrate proof of concept. As part of this process, we have extensively refactored Parsoid extension implementations to refine the API to be coherent and consistent and not cheat by virtue of being in the Parsoid codebase.
Step 1b (IN PROGESS) : Consult as widely as possible to ensure adequate exposure to the upcoming changes and ensure developers have opportunities to provide feedback. See Exploration section below for a bit more detail.
Step 1c (IN PROGRESS) : Get explicit approval of the core design of Parsoid's Extension API from TechCom. All additional review and feedback is welcome. But, this approval ensures we can proceed with expand the API without having to back to the drawing board around basic ideas and principles
Step 2: Ensure Parsoid Extension API is suitably expanded to capture all the uses cases for extensions deployed on the Wikimedia cluster.
Step 3: Ensure all Wikimedia extensions are "Parsoid-compatible" (or have suitable workarounds to continue functioning when Parsoid replaces the core parser on Wikimedia wikis). The Parsing Team will rely on and expect engineering help from other teams and developers in achieving this goal. But, resolving this is outside the scope of the RFC and I am stating this here for reasons of completeness and providing a fuller picture.
Step 4: Ensure Parsoid Extension API is suitably expanded to capture the broader set of use cases outside Wikimedia wikis.
Requirements
I already covered this in some detail above when I laid out the steps. But, broadly, initially, any extension API that we develop for Parsoid should be able to support current functionality provided by extensions deployed on the Wikimedia cluster. For the longer term, the requirements expand to extension use cases beyond Wikimedia. But for the purposes of this RFC, we are restricting this to Wikimedia wikis only.
Exploration
The Parsing Team has started the process of consultation in different venues this year:
- Early request for review from a small set of Wikimedia engineers: Discussion on mw:Talk/Parsoid/Extension_API
- Early look at Parsoid Extension API presentation at EMWCon 2020 in April: https://www.mediawiki.org/wiki/EMWCon_Spring_2020/Program
- An WMF-only internal email to tech-all and product-all in July 2020
- Retargeting extensions for Parsoid: Tech Talk in August 2020; Video, Slides
- File a TechCom RFC and get additional feedback. Seek explicit approval of core design of the Parsoid Extension API
- Seek wider feedback via outreach on wikitech-l and mediawiki-l