Full details are on MediaWiki.

Between Elastic 6.5 and 6.8, changes in Lucene have caused changes to tokenization for the standard tokenizer:

• A lot of "interesting" Unicode characters are now surviving tokenization.
• The tokenizer no longer splits on narrow no-break spaces (U+202F).

It also turns out that both of the above were already true for the ICU tokenizer in ES 6.5.

The Nori (Korean) tokenizer has changed the way it defines character sets (regular vs "extended"), while still breaking on clearly different character sets (Hangul, Cyrillic, Latin, Greek), leading to lots of small changes in Korean tokens: 6.4% fewer tokens in my Korean Wikipedia sample, 9.5% fewer tokens in my Korean Wiktionary sample.

Next Steps

To be discussed with the Search Team. Options include:

• Do nothing immediately. There are very few tokens affected, and some of the issues are already present in 6.5.
• Do nothing until ES 7.10. It's hard to keep up with all the small changes in every new version of Elastic and Lucene, so just wait until ES 7.10 and re-assess then. Who knows what may have been changed or fixed?
• Start fixing stuff, following something like the following list, roughly sorted by priority/complexity:
  • Add a NNBSP => space character filter to plain filters and unpacked text filters everywhere.
  • Look into tuning the aggressive_splitting filter for English and Italian to see if we can get what we want out of it without losing all the interesting rare characters coming out of the 6.8 tokenizers.
  • Look into harmonizing tokenization + aggressive_splitting across all (non-monolithic) languages so that similar inputs get similar outputs when possible (e.g., when not dependent on dictionaries or other language-specific processing). See T219550.
    • Consider upgrading all languages that use the standard tokenizer to the ICU tokenizer.
    • Consider using some form of aggressive_splitting everywhere possible. See T219108.
  • Look into patching our version of the Hebrew tokenizer to allow interesting Unicode characters to come through
  • If various issues still exist in ES 7.10:
    • Open an upstream ticket for the standard and ICU tokenizers to split on NNBSP characters.
    • Open an upstream ticket for the Thai tokenizer to allow interesting Unicode characters to come through.

The consensus today was that we should just let it ride for 6.8 and re-assess in 7.10. If these problems still exist, then we can start fixing them there. I'll update the 7.10 task (T301131).