The RTS problem is a simplified real-time strategy game domain with two types of units: workers and marines. Workers help gather minerals by finding mineral patches, and minerals are used to train marines or more workers. Marines are used for combat. There is a single base controlled by each player. The goal is to destroy the opponent's base. The opponent will play a strategy chosen from a set of fixed strategies. The domain is continuous. At each step, the action taken by the learning agent is the composition of all actions taken by the workers and marines, so the dimensionality of the action space can grow or shrink over time. Reward is based purely on the outcome of the game.  In the case of a tie, scores are assigned based on the relative accomplishments of both players.

Creating good decision-making algorithms RTS agents is an exciting challenge. Modern commercial developers typically do not have the resources to implement proper AI in their RTS games. Instead they often rely on agents that that cheat, e.g. are given more knowledge than human players. Even with this extra information, general decision-making AI can be intractable, so RTS game agents are often controlled by predictable hard-coded scripts. Players want better AI in RTS games, which poses a great challenge for reinforcement learning.

So far there have been few  attempts to apply reinforcement learning to RTS games. Ponsen, Spronck, and Tuyls (2006) used a modified version of HSMQ, based on MAXQ, to show that hierarchical learning helps in a simple gathering task. Madeira, Corruble, Ramalho, and Ratitch (2004) used bootstrap learning, with several levels of abstraction, to learn parameters one level at a time, using a fixed policy at other levels. Ponsen, Munoz-Avila, Spronck, and Aha (2006) use abstract representations of states to form an MDP, and use dynamic scripting to update the values of each agent's policy. Finally, Sharma, Holmes, et al. (2007) used transfer learning to learn in a hybrid case-based reasoning / reinforcement learning framework.

The competition domain is based on ortslite, a staging platform for the ORTS game engine, and will be altered between training and testing. See the Rules Page for more information about the altered evaluation paradigm.

Getting Started

1. Download and unpack the latest RL competition software package from the download page .
   
   
2. Read and follow the build instructions which are part of the technical overview. The overview describes the starting points for a C++ agent and Java agent, how to test your agent using the RLViz app, and pointers to documentation on how to build an agent in a different language. This overview also contains information about the formulation of the problem and game mechanics.
   
   
3. When you execute your agent in a proving run, ie. after some testing, start the proving app. From the main RL competition package directory:

       cd proving 
       bash run.bash        

References  

[Ponsen et al., 2006] Ponsen, M.J.V., Spronck P., Tuyls, K. (2006)  Hierarchical Reinforcement Learning with Deictic Representation in a Computer Game , 18th Benelux Conference on Artificial Intelligence (BNAIC 2006), October 5-6, Namur, Belgium

[Madeira et al., 2004] Madeira, C., Corruble V., Ramalho G., and Ratitch B.: (2004) Bootstrapping the Learning Process for the Semi-automated Design of a Challenging Game AI. In Proceedings of the AAAI Workshop on Challenges in Game AI, San Jose, CA, USA.

[Ponsen et al., 2006] Ponsen, M.J.V., Muñoz-Avila, H., Spronck, P., and Aha, D.W. (2006). Automatically Generating Game Tactics via Evolutionary Learning. AI Magazine, Vol 27, pp. 75-84

[Sharma et al., 2007] Manu Sharma, Michael Holmes, Juan Carlos Santamaria, Arya Irani, Charles Lee Isbell Jr., Ashwin Ram: Transfer Learning in Real-Time Strategy Games Using Hybrid CBR/RL. IJCAI 2007: 1041-1046