Research:Newsletter/2024/May: Difference between revisions

However, like many other current concerns about AI, these have been speculative and anecdotal. Hence the value of a quantitative analysis that tries to identify the causal impact of ChatGPT on Wikipedia in a statistically rigorous manner. Without conducting experiments though, i.e. based on observational data alone, it is not easy to establish that particular change or external event caused persistent increases or decreases in Wikipedia usage overall (as opposed to one-time spikes from particular events, or [[m:Research:Newsletter/2019/November#Seasonality_in_pageviews_reflects_plants_blooming_and_birds_migrating|recurring seasonal changes]]). The paper's literature review section cites only one previous publication which achieved that for Wikipedia pageviews: a 2019 paper by three authors from the Wikimedia Foundation (see our earlier coverage: [[m:Research:Newsletter/2019/December#An_awareness_campaign_in_India_did_not_affect_Wikipedia_pageviews,_but_a_new_software_feature_did|"An awareness campaign in India did not affect Wikipedia pageviews, but a new software feature did"]]). They had used a fairly sophisticated statistical approach ([[Bayesian structural time series]]) to first create a counterfactual forecast of Wikipedia traffic in a world where the event in question did not happen, and then interpret the difference between that forecast and the actual traffic as related to the event's impact. Their method successfully estimated the impact of a software change (consistent with the results of a previous randomized experiment conducted by this reviewer), as highlighted by the authors of the present paper: "Technological changes can [...] have significant and pervasive changes in user behaviour as demonstrated by the significant and persistent drop in pageviews observed in 2014 [sic, actually 2018] when Wikipedia introduced a page preview feature allowing desktop users to explore Wikipedia content without following links." The WMF authors concluded their 2019 paper by expressing the hope that "it lays the groundwork for exploring more standardized methods of predicting trends such as page views on Wikipedia with the goal of understanding the effect of external events."

# The common crawl size of the [[GPT-3]] main training data as a proxy for the effectiveness of ChatGPT in that language.

"For page views, we first performed a two-sided [[Wilcoxon rank-sum test|Wilcoxon Rank Sum test]] to identify whether there was a difference between the two periods (regardless of

"While the [[Wilcoxon signed-rank test|Wilcoxon Signed-Rank test]]<!--sic--> provided weak evidence for changes among the languages before and after the release of ChatGPT, we note ambiguities in the findings and limited accounting for seasonality. To address this and better evaluate any impact, we performed a [[panel regression]] using data for each of the four metrics. Additionally, to account for longer-term trends, we expanded our sample period to cover a period of three years with data from the 1st of January in 2021 to the 1st of January 2024."

While this second method accounts for weekly and yearly seasonality, it too does not attempt to disentangle the impact of ChatGPT from ongoing longer term trends. (While the given regression formula includes a language-specific [[fixed effect]], it doesn't have one for the availability of ChatGPT in that language, and also no slope term.) The usage of Wikipedia might well have been decreasing or increasing steadily during those three years for other reasons (say the basic fact that every year, the number of Internet users worldwide [https://ourworldindata.org/grapher/number-of-internet-users?country=~OWID_WRL increases by hundreds of millions]). Indeed, a naive application of the method would yield the counter-intuitive conclusion that ChatGPT ''increased'' Wikipedia traffic in those languages where it was available:

To avoid this fallacy (and partially address the aforementioned lack of trend analysis), the authors apply the same method to their (so to speak) [[control group]], i.e. "the six language versions of Wikipedia where ChatGPT is was unavailable":

Our findings suggest an increase in page visits and visitor numbers [i.e. page views and unique devices] that occurred across languages regardless of whether ChatGPT was available or not, although the observed increase was generally smaller in languages from countries where it was available. Conversely,

Unfortunately this preprint does not adhere to research best practices about providing replication data or code (let alone a [[preregistration]]), making it impossible to e.g. check whether the analysis of pageviews included automated traffic by spiders etc. (the default setting in the Wikimedia Foundation's [https://wikimedia.org/api/rest_v1/#/Pageviews%20data/get_metrics_pageviews_aggregate__project___access___agent___granularity___start___end_ Pageviews API]), which would considerably impact the interpretations of the results. The paper itself notes that such an attempt was made for edits ("we tried to limit the impact of bots by requesting only contributions from users") but doesn't address the analogous question for pageviews.

An earlier version of the paper as uploaded to ArXiv had the title "'The Death of Wikipedia?' – Exploring the Impact of ChatGPT on Wikipedia Engagement", which was later shortened by removing the attention-grabbing "Death of Wikipedia". As explained in the paper itself, that term refers to "an anonymous Wikipedia editor's fears that generative AI tools may lead to the death of Wikipedia" – specifically, the essay [[User:Barkeep49/Death of Wikipedia]], via its mention in a ''New York Times'' article, see [[Wikipedia:Wikipedia Signpost/2023-08-01/In the media]]. While the paper's analysis conclusively disproves that Wikipedia has died as of May 2024, it is worth noting that Barkeep49 did not necessarily predict the kind of immediate, lasting drop that the paper's methodology was designed to measure. In fact, the aforementioned NYT article quoted him as saying (in July 2023) "It wouldn't surprise me if things are fine for the next three years [for Wikipedia] and then, all of a sudden, in Year 4 or 5, things drop off a cliff." Nevertheless, the paper's findings leave reason for doubt whether this will be the first of the many [[predictions of the end of Wikipedia]] to become true.

''Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, [[m:Research:Newsletter#How to contribute|are always welcome]].''

It should be noted that the authors' relieved conclusion ("thankfully") is somewhat in contrast with the result of a larger scale blind experiment published last year in preprint form (see our coverage: "[[m:Research:Newsletter/2023/September#In_blind_test%2C_readers_prefer_ChatGPT_output_over_Wikipedia_articles_in_terms_of_clarity%2C_and_see_both_as_equally_credible|In blind test, readers prefer ChatGPT output over Wikipedia articles in terms of clarity, and see both as equally credible]]").

"This paper presents the first few-shot LLM-based chatbot that almost never hallucinates and has high conversationality and low latency. WikiChat is grounded on the English Wikipedia, the largest curated free-text corpus. WikiChat generates a response from an LLM, retains only the grounded facts, and combines them with additional information it retrieves from the corpus to form factual and engaging responses. We distill WikiChat based on GPT-4 into a 7B-parameter [[LLaMA]] model with minimal loss of quality, to significantly improve its latency, cost and privacy, and facilitate research and deployment. [...] we show that our best system achieves 97.3% factual accuracy in simulated conversations. It significantly outperforms all retrieval-based and LLM-based baselines, and by 3.9%, 38.6% and 51.0% on head, tail and recent knowledge compared to GPT-4. Compared to previous state-of-the-art retrieval-based chatbots, WikiChat is also significantly more informative and engaging, just like an LLM. WikiChat achieves 97.9% factual accuracy in conversations with human users about recent topics, 55.0% better than GPT-4, while receiving significantly higher user ratings and more favorable comments."

An online demo is available at https://wikichat.genie.stanford.edu/ . The [https://github.com/stanford-oval/WikiChat code] underlying the paper has been released under an open source license, and [https://github.com/stanfod-oval/WikiChat?tab=readme-ov-file#run-a-distilled-model-for-lower-latency-and-cost two distilled models] (for running the chatbot locally without relying e.g. on OpenAI's API) have been published on Huggingface.

See also our review of a previous (preprint) version of this paper: "[[m:Research:Newsletter/2023/July#Wikipedia-based_LLM_chatbot_%22outperforms_all_baselines%22_regarding_factual_accuracy|Wikipedia-based LLM chatbot 'outperforms all baselines' regarding factual accuracy]]"

From the abstract:<ref>{{Cite journal| doi = 10.3390/fi15020067| issn = 1999-5903| volume = 15| issue = 2| pages = 67| last1 = Bagnoli| first1 = Franco| last2 = de Bonfioli Cavalcabo’| first2 = Guido| title = A Simple Model of Knowledge Scaffolding Applied to Wikipedia Growth| journal = Future Internet| date = February 2023| doi-access = free}}</ref>

From the abstract:<ref>{{Cite journal| doi = 10.1038/s41598-024-58305-2| issn = 2045-2322| volume = 14| issue = 1| pages = 8331| last1 = Zhu| first1 = Manran| last2 = Yasseri| first2 = Taha| last3 = Kertész| first3 = János| title = Individual differences in knowledge network navigation| journal = Scientific Reports| date = 2024-04-09| pmid = 38594309| arxiv = 2303.10036| bibcode = 2024NatSR..14.8331Z| url = https://www.nature.com/articles/s41598-024-58305-2}}</ref>

"In this thesis, we present studies to assess and improve the state of labels and languages in knowledge graphs and apply multilingual information. We propose ways to use multilingual knowledge graphs to reduce gaps in coverage between languages. We explore the current state of language distribution in knowledge graphs by developing a framework - based on existing standards, frameworks, and guidelines - to measure label and language distribution in knowledge graphs. We apply this framework to a dataset representing the web of data, and to Wikidata. [...] Due to its multilingual editors, Wikidata has a better distribution of languages in labels. [...] A way of overcoming the lack of multilingual information in knowledge graphs is to transliterate and translate knowledge graph labels and aliases. We propose the automatic classification of labels into transliteration or translation in order to train a model for each task. [...] A use case of multilingual labels is the generation of article placeholders for Wikipedia using neural text generation in lower-resourced languages. On the basis of surveys and semi-structured interviews, we show that Wikipedia community members find the placeholder pages, and especially the generated summaries, helpful, and are highly likely to accept and reuse the generated text."

''See also [[mw:Extension:ArticlePlaceholder]] and our coverage of a subsequent paper: [[m:Research:Newsletter/2023/November#"Using_natural_language_generation_to_bootstrap_missing_Wikipedia_articles:_A_human-centric_perspective"|"Using natural language generation to bootstrap missing Wikipedia articles: A human-centric perspective"]]''

"[[Entity Linking]] (EL) is the task of connecting already marked mentions in an utterance to their corresponding entities in a knowledge graph (KG) [...]. In the past, this task was tackled by using popular knowledge bases such as DBpedia [67], Freebase [11] or Wikipedia. While the popularity of those is still imminent, another alternative, named Wikidata [120], appeared."