OpenAI Five

This article, OpenAI Five, has recently been created via the Articles for creation process. Please check to see if the reviewer has accidentally left this template after accepting the draft and take appropriate action as necessary. Reviewer tools: Inform author

OpenAI Five is an artificial intelligence (AI) software system that plays the Dota 2 esport video game.^[1] It was developed by OpenAI, a non-profit artificial intelligence (AI) research lab founded by Elon Musk, Sam Altman and others with the mission to develop safe and friendly AI in a way that benefits humanity.^[2][3][4]

In August 2017, the first version of the OpenAI Dota 2 bot played in a 1v1 game against a top professional player, Dendi, where an AI system defeated him at The International 2017.^[5][6] Greg Brockman, co-founder and CTO of OpenAI, explained that the AI system had learned by playing against itself for two weeks of real time, and that the learning software was a step in the direction of creating software that can handle complex tasks "like being a surgeon".^[7][8]

During 2018, OpenAI has been developing a system to be able to play Dota against a team of five humans. In June 2018, the ability of OpenAI Five expanded to play together as a team of five and were able to defeat teams of amateur and semi-professional players.^[9][10] In August 2018, OpenAI Five beat former Dota professionals ranked in the 99.95th percentile at the Benchmark game winning two games and losing one.^[11][12]In August 2018, OpenAI Five played against a pro team of five people at The International 2018, where the AI system lost two games to the human team.^[13][14]

According to OpenAI, the lab uses Dota as a test system for general-purpose AI to capture the unpredictability and continuous nature of the real world, e.g. teamwork, long time horizons, and hidden patterns.^[15] The training system used for Dota is also applied to control a robotic hand.^[16][17][18]

Dota 2 is listed among the best and most complex video games^[19], with professionals who train full time for many years^[20]to gain a part of Dota’s annual $40M prize pool, the largest in esports gaming.^[21]

Previous AI vs human milestones have been in games such as Jeopardy!, Chess, and Go.^[22][23][24] Currently, AI labs are trying to develop AI systems to be able to compete in more complex video games that require strategic thinking, team play, reasoning, etc.^[25][26]

According to the OpenAI website, the work on the AI system for Dota 2 started at the end of 2016.^[27]In August 2017, the system beat top professional, Dendi, in a 1v1 game during The International 7.^[28][29]

In June 2018, OpenAI Five defeated an in-house OpenAI team in a fairly restricted game of 5v5 players.^[30][31]

In August 2018, during the Benchmark, OpenAI Five defeated former Dota professionals with a somewhat restricted game of 5v5 players with 5 invulnerable couriers and 18 heroes for selection in the AI system.^[32][33]

In August 2018, the OpenAI Five team played against top professional Dota 2 players at The International 8 and lost both games. The first game was against paiN Gaming, listed in the top 18 Dota 2 teams who won more than $450,000 in career earnings.^[34]The second game was with a team of Chinese super professional players. Three of them had played together in the same team with an average of $1 million tournament earnings for each of these players.^[35]The AI system lost the second game after 45 minutes handling a competitive back-and-forth game.^[36]After The International 8 match Garry Kasparov wrote in his tweet: “They'll get there, and sooner than expected. Machines will be superior in any closed system eventually.”^[37]

According to the OpenAI official blog post, unlike the Benchmark game the International one was played against significantly stronger human players, a pro team that has been training for the game full-time. Another difference was that hero lineups were provided by a third party rather than by OpenAI Five. Also, the AI system developers removed the last major restriction that most Dota professionals consider “Real Dota” gameplay: OpenAI heroes got a single mortal team courier where previously each hero had its own invulnerable courier to deliver items for them.^[38]

Each OpenAI Five network contains a single layer with a 1024-unit LSTM^[39] that observes the current game state extracted from the Dota developer’s API. The neural network conducts actions via numerous possible action heads (no human data involved), and every head has meaning. For instance, the number of ticks to delay an action, what action to select – the X or Y coordinate of this action in a grid around the unit. In addition, action heads are computed independently. The AI system observes the world as a list of 20,000 numbers and takes an action by conducting a list of 8 enumeration values. Also, it selects different actions and targets to understand how to encode every action and observes the world.^[40]

OpenAI Five has been developed as a general-purpose reinforcement learning training system on the Rapid infrastructure. Rapid consists of two layers: it spins up thousands of machines and helps them ‘talk’ to each other and a second layer runs software. OpenAI Five has played 180 years’ worth of games in a self-play technique (which is a part of the reinforcement learning approach) using a scaled-up version of Proximal Policy Optimization.^[41][42]running on 256 GPUs and 128,000 CPU cores.^[43]

Jeopardy!, Chess and Go are among milestones in AI vs human gaming. But unlike these, Dota 2 is a real-time strategy game played between two teams that consists of five players.^[44]

An AI system playing Dota 2 has the following features:

Long run view: Dota games run at 30 frames per second for an average of 45 minutes, which results in 80,000 ticks per game. Some discrete actions, like town portal running, can significantly impact the game. OpenAI Five observes every fourth frame, generating 20,000 moves. By comparison, chess usually ends before 40 moves, Go before 150 moves.^[45]

Partially observed state of the game: Elements can see only the area around them. The rest of the map is covered in a fog hiding enemies and their approaches. Professional play requires making inferences based on incomplete data, as well as recreating (prototyping) what one’s opponent might be up to. Chess and Go, by comparison, are full-information games.^[46][47]

Continuous action space: each hero can take dozens of actions, and many actions target either another unit or a position. The OpenAI Five developers discretize the space into 170,000 possible actions per hero. Without counting the perpetual aspects of the game, there are an average of ~1,000 valid actions each tick. By comparison, the average number of actions in chess is 35 and 250 in Go.^[48]

Continuous observation space: Dota is played on a large ceaseless map with ten heroes, dozens of buildings and NPC units, along with countless game features, like runes, trees and wards, and so on. The OpenAI system observes the state of a game though developers’ Bot API, as 20,000 numbers that constitute all information a human is allowed to get access to. A chess board is represented as about 70 lists (an 8x8 board of 6-piece types – here is historical info); where a Go board has about 400 enumerations (a 19x19 board of 2-piece types counting Ko).^[49]

OpenAI Five advancements have received acknowledgement from the AI, tech and Dota community. Bill Gates tweeted after the OpenAI team’s in-house game in June^[50][51]:

#AI bots just beat humans at the video game Dota 2. That’s a big deal, because their victory required teamwork and collaboration – a huge milestone in advancing artificial intelligence.

After The International 8 match Garry Kasparov his tweeted^[52] : “They'll get there, and sooner than expected. Machines will be superior in any closed system eventually.”

According The Verge^[53], Andreas Theodorou, an AI researcher at the University of Bath who uses computer games to study collaboration, says the latest research on 5v5 games is a big step forward, although he notes that perhaps the most “significant achievement” is OpenAI’s use of visualizations to debug their agents. “These techniques show how even reinforcement learning and machine learning systems, in general, can be transparent”.

In a conversation with MIT Technology Review^[54], AI experts said that the achievement of the OpenAI Five system is significant. “Dota 2 is an extremely complicated game, so even beating strong amateurs is truly impressive,” says Noam Brown, a researcher at Carnegie Mellon University in Pittsburgh. “In particular, dealing with hidden information in a game as large as Dota 2 is a major challenge.”

• Glossary of artificial intelligence

• Official website

This article, OpenAI Five, has recently been created via the Articles for creation process. Please check to see if the reviewer has accidentally left this template after accepting the draft and take appropriate action as necessary. Reviewer tools: Inform author