Create a Challenge on Codabench

This repository is for challenge organizers to organize a challenge on Codabench.

This repository has two competition bundles:

1. Results submission only competition (without ingestion program)
2. Code submission competition (with ingestion program)

Sample Competitions

Code Submission Challenge
Results Submission Challenge

---

Table of Contents

1. Understanding the Starting Kit
2. Understanding the Compatition Bundle
3. Instructions about datasets
4. How to create a challenge on Codabench?
5. How to upload datasets on Codabench?
6. How to create a task on Codabench?
7. How to change a task in a competition?

---

Understanding the Starting Kit

The starting kit consists of the following directories and files:

1. ingestion_program
   This directory consists of the following files:

   • data_io.py : contains functions to load the data
   • ingestion.py : responsible for loading, running the model with the input data and to save the results in the results directory
   • metadata : a file required to run the code on Codabench

2. scoring_program
   This directory consists of the following files:

   • metric.txt : a text file which consists of the name of the metric of evaluation
   • solution.py : to load only the solutions from the input data
   • score.py : responsible to compute score from the predictions made by ingestion.py and ground truths from solution.py
   • libscores.py : consists of pre-defined metrics
   • my_metric.py: a file in which you can define your own custom metric or use any metric from a library e.g. scikit-learn
   • metadata : a file required to run the code on Codabench

3. scoring_output
   This directory consists of scores.html and scores.txt files. These files are used to store scores from scoring program and to load scores to the leader board

4. sample_data
   This directory contains the sample data (a subset of Dataset 0) for the challenge.

5. sample_code_submission
   This directory consists of model.py file which has a Model Class with different functions. You have to change this file but the names of the functions should be the same. You can also keep a pre-trained model in this directory which could be used as a pre-trained model when you submit a code submission.

6. sample_result_submission
   This directory contains train.predict and test.predict files which are used to calculate the score.

7. logo.png
   This image is the logo for your competition on Codabench website.

8. README.ipynb
   This is python notebook which gives the participants a tour from data loading to code submission on Codabench.

⚠️ Check the example starting_kit in this repository

---

Understanding the Compatition Bundle

The competition_bundle consists of all the files and folders from starting_kit along with the following additional files.

1. competition.yaml
   This file consists of important details and settings of the competition which include:

• title
• description
• image
• end date
• documentation pages
• phases
• tasks
• leaderboard

See the example competition.yaml file in the competition_bundle

2. documentation pages
   There are several documentation pages to describe the competition:

• overview.md
• evaluation.md
• data.md
• terms.md

⚠️ Check the example competition_bundle in this repository

---

Instructions about datasets

The challenge consists of two phases:

• Phase 1 (Feedback Phase)
• Phase 2 (Final Phase)

This challenge is designed for a triplet of datasets:

• Dataset 0 (to be used as public_data)
• Dataset 1 (to be used in Phase 1)
• Dataset 2 (be used in Phase 2)

⚠️ Check the example datasets in this repository

---

How to create a challenge on Codabench?

Follow the steps below to create a challenge on Codabench.

Step 1 : Clone this repository

git clone https://github.com/ihsaan-ullah/create_a_codabench_challenge.git

Step 2 : Zip Competition Bundle

1. Go the repository

cd create_a_codabench_challenge

2. Zip the competition bundle

zip -r competition_bundle.zip competition_bundle/*

Step 3 : Sign Up on Codabench

1. Create an account on the Codabench Website: https://www.codabench.org/accounts/signup

Step 4 : Login on Codabench

1. Login into your account : https://www.codabench.org/accounts/login/

Step 2 : Upload Bundle

1. Click Management under Benchmarks in the Navigation bar

2. Click Upload button

3. Click the input file field
   

4. Choose the competition_bundle.zip created in Step 2

5. Click View to see the uploaded competition

⚠️ Note

You have to make changes according to the specifications and goals of your challenge before uploading it to Codabench.

---

How to upload datasets on Codabench?

Follow the steps below to upload Public Data (Dataset 0) and Input/Reference Data (Dataset 1 and Dataset 2) to Codabench

1. Click Resources in the top navigation bar

2. Click Add Dataset button

3. Fill the form by entering the following details:

   • Name
   • Description
   • Select type
     • Ingestion Program
     • Input Data
     • Public Data
     • Reference Data
     • Scoring Program
     • Starting Kit
     • Submission
   • Select the zipped dataset and click upload

4. Click Upload button to save changes.

⚠️ Note

• For Public Data (Dataset 0) choose the type Public Data
• For Input Data (Dataset 1, Dataset 2) choose Input Data
• Choose other types for scoring_program, ingestion_program or starting_kit.

---

How to create a task on Codabench?

1. Click Resources in the top navigation bar

2. Click Tasks tab

3. Click Create Task button

4. Fill the form by entering Name and Description in Details tab and selecting Scoring Program, Ingestion Program, Reference Data, and Input Data from the uploaded Datasets.

5. Click Create button to save changes.

---

How to change a task in a competition?

Follow the steps below to change a task in your competition.

1. Click Management under Benchmarks in the Navigation bar

2. Click the Edit button in front of the competition you want to edit

3. Click the Edit button in front of the phase for which you want to change the task

4. Click the tasks dropdown to select a new task. You can remove existing task by clickin the cross button

5. Click the Save button at the bottom of the page to save changes.

---

Contact

In case of any confusion, contact by email : ihsan2131@gmail.com

Bugs/Issues

Report any inconsistencies or bugs related to Codabench Platform on Github: https://github.com/codalab/codabench/issues