About This Page

Below are the current details about the testing runs for the rl-competition.  The evaluation criteria is the same as in the proving runs, so I won't rehash that all in this document.  Check here for more details about proving runs and evaluation.

Here is the current plan for how to run the test phase:

Each team gets ONE test run per event

It looks like we will be allowing each team 1 test run per event in the month of June. This is the only way we have conceived to be sure the results represent what we're looking for. If people get more than 1 run, they might re-use data about the testing MDPs between runs which would circumvent the spirit of the test phase.

Extra test runs will be awarded if circumstances warrant it

We will have a special arbitration process to allow people a second run in special circumstances. We have no interest in our competitors working for months on their agent and then not being able to test it because of a technical glitch.

Test runs will happen through the proving application

An updated release of the competition package will feature an updated proving application with new events corresponding to the test versions in each domain.  For these events, the number of runs available will NOT reset every week.  Competitors will be able to continue doing proving runs throughout June, so there will be incentive to do testing at the last minute <wince>.

The results will not be published before the workshop

The results will not be published, and participants are be asked not to compare with each other before the ICML workshop.  Please. 

Teams should not share *any* information about test runs

Teams should not share *any* information about test runs, so that we can keep the playing field fair for everyone.

Any questions about this setup and these rules should be asked in the forums or through e-mail to This e-mail address is being protected from spam bots, you need JavaScript enabled to view it

Good luck.