View source for Objective Revision Evaluation Service/What

<div style="float:right; padding:1ex 1em;">__TOC__</div>

[[ORES]], the Objective Revision Evaluation Service, is a service that provides access to various prediction models.  A user of the service provides ORES with the name of a wiki, a model to apply and a revision ID to apply the model to.  The service responds with a "score" in the form of a [[:en:JSON|JSON]] document, a data format that is both machine- and human-readable.  These "scores" are generally predictions about some characteristic of the revision.  

== Using the service ==
<center>
{{inline figure
 | filename = ORES url structure.svg
 | size = 320px
 | id = ORES url structure
 | caption = This URL asks, "What is the probability that the edit that created revision ID 642215410 cause damage and needs to be reverted?" [https://ores.wmflabs.org/scores/enwiki/damaging/642215410/ try it!]
}}
{{inline figure
 | filename = 642215410 diff.png
 | size = 320px
 | id = Diff of 642215410
 | caption = A screenshot of the diff of rev_id 642215410 from English Wikipedia. This edit replaces a reference with "LLAMAS GROW ON TREES".
}}
<div style="display: inline-block;text-align: left; font-size: .9em;">
{{anchor|ORES response}}
<syntaxhighlight lang="javascript">
"642215410": {
  "prediction": true,
  "probability": {
    "false": 0.047408808815936794,
    "true": 0.9525911911840632
  }
}
</syntaxhighlight>
'''ORES response.''' ORES predicts that this edit <br/>
is damaging to the article.
</div>
</center>
ORES is a [[:en:web service|web service]].  You can access via your web browser or make it requests to it within a script, bot or tool.  [[#ORES url structure]] provides a labeled example of request to apply the ''damaging'' model to [[:en:Special:Diff/642215410|''642215410'']] in the English Wikipedia (''enwiki'').  [[#Diff of 642215410]] shows a diff of the revision itself.  [[#ORES response]] shows that ORES predicts (with ~95% certainty) that the edit is damaging. 

Currently the system serves models that predict the qualities of an edit ([[ORES/reverted|Will it need to be reverted?]], [[ORES/damaging|Is it damaging?]] and [[ORES/goodfaith|Was it made in goodfaith?]]) and the quality of an article as of a revision ([[ORES/wp10|Which article quality assessment rating is most likely right?]]).  Models for predicting [[Research:Automated_classification_of_edit_types|edit types]] are in development.

== Supervised machine learning -- in a nutshell ==
{{main|:en:Supervised learning}}
<center>
{{inline figure
 | filename = Supervised machine learning in a nutshell.svg
 | size = 600px
 | id = Supervised machine learning data flow
 | caption = A data-flow diagram of training and testing a machine learning model.
}}
</center>
ORES' prediction models are ''trained'' by showing them many ''observations'' that have been ''labeled'' by humans.  The prediction models are then used to make predictions about new ''observations''.  This process is commonly referred to as ''[[:en:supervised machine learning|supervised machine learning]]''.  [[#Supervised machine learning data flow]] shows the flow of data in a step-by-step process to "train" and "test" a "prediction model".  From the diagram:

# Humans provide judgements via labels (e.g. Damaging or not, FA-class/Stub-class article, etc.).  These judgements are collecting into a set of "labeled observations"
# Labeled observations are split into a "training set" and "test set".  The "training set" is provided to the machine learner to learn from.  The "test set" is held aside for future use.
# The machine learner uses the "training set" to learn how to predict the provided labels (i.e. learns to replicate human judgements) and a "prediction model" is produced.  This prediction model can be used to automatically label new observations. 
# The previously withheld "test set" is given to the "prediction model" and the "predicted labels" are compared to the "human labels" to generate statistics about the model's performance.

Note that both the "prediction model" and "stats" from testing are outlined.  This is because the two are kept together and use within the ORES system.  You can check a model's statistics by dropping the revision ID from the path.  E.g. https://ores.wmflabs.org/scores/enwiki/damaging/