Skip to content

Creating and executing a challenge

Michael Röder edited this page Apr 12, 2017 · 2 revisions

A Challenge is a collection of one or more challenge task. Each challenge task uses exactly one benchmark with a certain configuration to benchmark systems that participate in the challenge. Challenges have some features that separate them from "normal" experiments that can be executed on the platform:

  • The experiments of a challenge are executed on a particular date. From that time on, the experiments of the challenge have a higher priority than other experiments.
  • Challenge results have a publication date. Before this date is reached, the results are not publicly visible. Participants can see only the results of their system while the challenge owner can see all results.
  • Challenges and their tasks offer the possibility to add descriptions.

Note that you need to have a special role to be able to create a challenge. If you want to use the HOBBIT platform for a certain challenge, please contact the hobbit project team.

Challenge workflow

  1. Create a challenge
    • If the challenge can be already be visible, you can tick the checkbox. Otherwise, the challenge won't be visible for normal users.
  2. Create challenge tasks
    • The creation of the tasks work like the definition of an experiment but without choosing a system.
  3. Participant registration
    • After the creation and
    • if the challenge is visible, users can register their system for one or more challenge tasks if the system fits to the benchmark of the challenge task
  4. Close the challenge
    • Closing the challenge is necessary to fix the registrations and the challenge task configuration. Note that from that point on, no additional user will be able to register a system for the challenge.
    • Note that participants can still update their system images.
  5. Execution date reached
    • The single experiments are executed.
    • The challenge organiser can see the results in the challenge view. The participants can see only their result.
  6. Publication date is reached
    • The results are publicly available.