The source code at https://github.com/wikimedia/research-recommendation-api has lot of legacy code, broken or unmaintained dependencies. The web frontend is with bower, jquery and such very old tooling. Recent updates by machine learning team got it somewhat functional to the extend it is integrated to liftwing. But adding new features require more fixups to get a smooth local development experience. We can ignore the web frontend part (AKA - gapfinder) for now as we are interested only in the API.

My preference is to enhance the "new" recommendation API at https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_content_translation_recommendation so that it can accept a topic(example: Chemisty, History, Africa, Music etc) and give recommendations. It should accept more than one topic. We can also see an intersection of topic and article in later stage.

This ticket proposes to adjust the tab that the user navigates to by default to by considering the previous selections, and the existence of previous contributions by the user:

Internally - in CX production and in our developer workflows we directly use cxserver APIs and not the RESTBase apis like https://en.wikipedia.org/api/rest_v1/#/Transforms/doMT.

dda + nukta forming the same ligature rendering of rra is a common issue in Gurmukhi fonts. For example Ektype's Mukta has this issue. And this practice of having same shape for nukta form and RRA is not adviced, yet many fonts has them. This is the reason why you see two different shapes as reported above. Common users not aware of this encoding difference, but focusing only in rendering, uses them interchangeably. This is wrong usage appears in corpus. For example, in many dravidian scripts I have seen people using 0(zero) in the place of ഠ, :(colon) instead of ഃ(visarga) and so on. A neuaral MT system learns them and the same issues appear in MT output. I have seen this issue in many other languages too.

@elukey Thanks for these details. Currently in our code, models are downloaded using a boostrap shell script(called via docker entrypoint mechanism) using simple wget. These models are then mounted to the docker volume. So our server code just assumes the models are present in a configurable file system location. Do you see any issue if we follow this approach? Does the caveats you mentioned complicate this approach?

Round trip technique like wikitext->html->wikitext is one way to achieve this. However it has limitations. For example, if wikitext has a template and one of the template parameter is nested wikitext, we will miss it in html rendering(For example i18n sentences with plural syntax etc). So translation will be incomplete.

I was able to reproduce and find out the pattern that cause this issue. Repeated references. Only the first one gets fixed in MT. Second one onwards, it appears plain text. A few months back I had addressed this by keeping a search start in look up logic, but it is not catching repeatations outside the sentence. I am exploring potential solutions.

The CX entrypoint is also duplicated if you click multiple times while language selector is loading:

After the migration to node fetch, the error is still there:

	TypeError: Cannot read properties of undefined (reading 'pages')
    at processResult (/srv/service/lib/mw/BatchedAPIRequest.js:85:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The issue is resolved and the root cause of bad requests from preq library is also resolved

Browsers natively support hyphenation(breaking the word at proper position) these days. No need to change the content for this. Following CSS example shows how to do this. I developed hyphenation system for Indian languages and that is what Chrome, Firefox, TeX, Libreoffice, Indesign etc using these days.

Both MinT and Apertium does not use proxy. They were working and then we added MT clients with proxy support . Then clients without proxy started showing this issue- This is not consistently reproducible, but happens very frequent.

Issue is resolved now as train is rolled back. Not closing as we need to monitor this when train is running with backported patch

It seems the backend issue is T359509: REST API calls suddenly all returning 400 and there is already a patch to be reviewed and merged:

mw-cli can help us to create many language wikis in a cloud instance.
So we can have http://en.mediawiki.mwdd.localhost:8080, http://ig.mediawiki.mwdd.localhost:8080 ..

Tangential note: https://ruralindiaonline.org/en/articles/in-2023-paribhasha-builds-a-peoples-archive-in-peoples-languages/ is a bad example because of broken rendering in the scripts used in title image - We should never do that.

There is a feature in superset where we can just embed any dashboards in any web page. That seems the easiest approach here. https://github.com/apache/superset/tree/master/superset-embedded-sdk

A screenshot illustrating reference misplacement with current prototype: From https://en.wikipedia.org/wiki/Polar_bear

The above patch is a quick run to identify the required efforts to migrate from servicerunner. It is not for merge. My proposal is to modernize various parts of cxserver, while using servicerunner as process manager. Do this migrations in iterations and at later stage when cxserver does not have a strong dependency on servicerunner other than a process manager, replace it. Doing everything in one go is too risky as cxserver is the backbone of our heavily used translation system.

The list of models for a language pair is provided in API output of https://translate.wmcloud.org/api/languages
This is linked in the UI - See bottom links - API Spec

Additional information: This issue happens with indictrans2-en-indic model. NLLB-200 gives correct output

this will allow the language team to use this model server

However, when inspecting the target language selector you can notice that Santali (sat) is not listed.

The actual failure can be reproduced by visiting https://cxserver.wikimedia.org/v2/page/sv/nn/Royal_Society_for_the_Protection_of_Birds

Page sv:Royal_Society_for_the_Protection_of_Birds could not be found. TypeError: item.dispose is not a function

Root cause is a regresssion from recent cxserver upgrade. Fix already in place https://gerrit.wikimedia.org/r/c/mediawiki/services/cxserver/+/978192 waiting for deployment

From our past observations, especiailly during translaiton campaigns, many users participate, potentially creating low quality articles. The review happens much later. Reviwers also had complained that they cannot review all these articles on time. When review happens, articles get deleted. So the deletion happens weeks later the translation activity. Considering this, the chances that a new user has a deleted translation while making intentional or unintentaionl low quality translation is rare.
Hence, the proposed strict limit if user has deletion in last 30 days might not have expected effect. However, I support keeping this in place. But the user should be clearly communicated why their translation limits are high.

The current logic in CX for CJK group of languages(including chinese) is follows. The tokens are characters instead of words, so 人口 has 2 tokens.

@elukey, What do you mean by 'reaching out to you by next time' ? Regarding the architecture of MinT and why it is not using LiftWing we had discussion in the past. I don't think it is not useful to repeat. There is a reason why we put the models in people.wikimedia.org - it was as per recommendation from SRE and this ticket was created to make it more reliable. We still need a public location for models download as MinT is not designed for WMF instrastructure alone.

We need 2TB scratch volume mounted too.

https://test.wikipedia.org/w/rest.php/coredev/v0/transform/wikitext/to/html/Oxygen looks good. If this can be exposed for all production wikis, we can definitely move to this endpoint.

It seems we need to continue with restbase for the time being till a stable, well documented API is known as replacement, right?

http://parsoid-external-ci-access.beta.wmflabs.org - Does this use actual production wiki? Or beta.wmflabs.org? If it is beta.wmflabs.org, then we will be limited by content and supported languages right?

If you need access to pagebundles or the transform endpoints, then we have to figure something out.

I think we have a serious problem here.
At https://phabricator.wikimedia.org/T350219#9298055, @daniel wrote:

"Parsoid endpoints are not expected to work for external requests. So this is "working" as expected."

The restbase endpoint is no longer working. What changed? @daniel, @MSantos

Fixed in sentencex version 0.5.1

@Sportzpikachu Thanks for the PR. Please note that jquery.i18n has a successor banana.i18n which is a framework agnostic js library. That is the library we are actively going to maintain. If your usecase can use that library, it would be much better.

The model expects sentences. That is how it is trained. For example, words like "Moon" can appear in many latin based languages as proper noun or reference to a title of a book etc. The prediction quality increase as more words are provided. Then it knows better about the context of the word.

Thank you @isarantopoulos and @elukey !

We have the service in production: https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_language_identification_prediction

@elukey If I understood that documentation correctly, if the service required oauth token, still Anonymous users can use it with the applicable ratelimiting. am I right?
There would be usecases where non-mediawiki static webpage using this API and this anonymous ratelimited option should be sufficient.

Yes, references are moved to the end of sentence. Also seen in this example below. The positioning of references after the correct position in translation is slightly complicated and need to be implemented.

In T340507#9242994, @isarantopoulos wrote:

@santhosh Thanks for creating the model card!
Is there a client/system that will use this at the moment? If yes, is there an estimate on the amount of traffic we should be expecting? Main reason I am asking is so that we know the scaling requirements (if any) and also can validate via load testing.