{ "id": 416589, "title": "Steps to follow to have a valid submission for Phase II ", "url": "/competitions/budgeted-model-training-iccv-2023-rcv-workshop/discussion/416589", "author_name": "", "post_date": "2023-06-12T08:32:05.915969200Z", "votes": 2, "comment_count": 34, "views": 0, "content": "

We received a lot of invalid submissions so here is the stepwise guide for a valid submission.

\n

The second Phase of the competition involves testing the code submissions on ImageNet100. The chosen classes are different from what is released on Kaggle. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.

\n
    \n
  1. Ensure that the code submitted runs on the environment provided in the yml file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.
    2. \n
  2. The submissions must be done in a zip file with the team name attached as in TEAM_NAME.zip. This TEAM_NAME will be displayed on the Phase 2 leaderboard.
    3. \n
  3. The zip file should only contain the training script with the name main.py. Participants are requested to hardcode the parameters inside the script as no config files will be accepted.
    4. \n
  4. The script will be tested against the data directory containing data for Phase 2. The data follows a similar format to what is put on Kaggle. The data directory contains 3 folders: train, val, and test along with 2 csv’s: train.csv and val.csv. Participants are requested to create dataloaders accordingly or use the ones provided in the trainer script.
    5. \n
  5. Participants are requested to include the mandatory arguments root_dir and output_dir inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump the models and submission.csv.
    6. \n
  6. The script is expected to fetch the training, validation, and test data from the data directory, train the model, complete inference on the test data provided, and dump the predictions as submission.csv inside the output_dir provided as the argument. Reference for data loaders is provided in the trainer script.
    7. \n
  7. The competition puts forward 2 constraints:
    \na. Training under 9 hours of the time limit.
    \nb. Training with a budgeted CUDA memory of 6GB.
    8. \n
  8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code.
    \ntup = torch.cuda.mem_get_info()
    \ntorch.cuda.set_per_process_memory_fraction((6*(1024)**3)/tup[1], 0)
    9. \n
  9. It is expected that the participants include the logic to break the training loop or set the number of epochs such that the train time constraint of 9 hrs is not violated. An example of how this can be done is presented in the trainer script using time.time(), which upon exceeding ~9hrs raises a TRAIN_END_FLAG and breaks out of the training loop and jumps straight to inference.
    10. \n
  10. So in summary:
    11. \n
\n", "messages": [ { "id": "2296901", "postDate": "06/12/2023 08:32:05", "content": "

We received a lot of invalid submissions so here is the stepwise guide for a valid submission.

\n

The second Phase of the competition involves testing the code submissions on ImageNet100. The chosen classes are different from what is released on Kaggle. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.

\n
    \n
  1. Ensure that the code submitted runs on the environment provided in the yml file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.
    2. \n
  2. The submissions must be done in a zip file with the team name attached as in TEAM_NAME.zip. This TEAM_NAME will be displayed on the Phase 2 leaderboard.
    3. \n
  3. The zip file should only contain the training script with the name main.py. Participants are requested to hardcode the parameters inside the script as no config files will be accepted.
    4. \n
  4. The script will be tested against the data directory containing data for Phase 2. The data follows a similar format to what is put on Kaggle. The data directory contains 3 folders: train, val, and test along with 2 csv’s: train.csv and val.csv. Participants are requested to create dataloaders accordingly or use the ones provided in the trainer script.
    5. \n
  5. Participants are requested to include the mandatory arguments root_dir and output_dir inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump the models and submission.csv.
    6. \n
  6. The script is expected to fetch the training, validation, and test data from the data directory, train the model, complete inference on the test data provided, and dump the predictions as submission.csv inside the output_dir provided as the argument. Reference for data loaders is provided in the trainer script.
    7. \n
  7. The competition puts forward 2 constraints:
    \na. Training under 9 hours of the time limit.
    \nb. Training with a budgeted CUDA memory of 6GB.
    8. \n
  8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code.
    \ntup = torch.cuda.mem_get_info()
    \ntorch.cuda.set_per_process_memory_fraction((6*(1024)**3)/tup[1], 0)
    9. \n
  9. It is expected that the participants include the logic to break the training loop or set the number of epochs such that the train time constraint of 9 hrs is not violated. An example of how this can be done is presented in the trainer script using time.time(), which upon exceeding ~9hrs raises a TRAIN_END_FLAG and breaks out of the training loop and jumps straight to inference.
    10. \n
  10. So in summary:
    11. \n
\n", "rawMarkdown": "We received a lot of invalid submissions so here is the stepwise guide for a valid submission. \n\nThe second Phase of the competition involves testing the code submissions on ImageNet100. The chosen classes are different from what is released on Kaggle. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.\n\n1. Ensure that the code submitted runs on the environment provided in the [yml](https://www.kaggle.com/competitions/budgeted-model-training-iccv-2023-rcv-workshop/data?select=rcv.yml) file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.\n2. The submissions must be done in a zip file with the team name attached as in **TEAM_NAME.zip**. This TEAM_NAME will be displayed on the Phase 2 leaderboard.\n3. The zip file should only contain the training script with the name **main.py**. Participants are requested to hardcode the parameters inside the script as **no config files will be accepted**. \n4. The script will be tested against the data directory containing data for Phase 2. The data follows a similar format to what is put on Kaggle. The data directory contains 3 folders: train, val, and test along with 2 csv’s: train.csv and val.csv. Participants are requested to create dataloaders accordingly or use the ones provided in the [trainer script](https://www.kaggle.com/code/mlnoob327/main-py).\n5. Participants are requested to include the mandatory arguments **root_dir** and **output_dir** inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump the models and submission.csv.\n6. The script is expected to fetch the training, validation, and test data from the data directory, train the model, complete inference on the test data provided, and dump the predictions as **submission.csv** inside the output_dir provided as the argument. Reference for data loaders is provided in the [trainer script](https://www.kaggle.com/code/mlnoob327/main-py).\n7. The competition puts forward 2 constraints:\na. Training under 9 hours of the time limit.\nb. Training with a budgeted CUDA memory of 6GB.\n8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code. \n`tup = torch.cuda.mem_get_info()`\n`torch.cuda.set_per_process_memory_fraction((6*(1024)**3)/tup[1], 0)`\n9. It is expected that the participants include the logic to break the training loop or set the number of epochs such that the train time constraint of 9 hrs is not violated. An example of how this can be done is presented in the [trainer script](https://www.kaggle.com/code/mlnoob327/main-py) using time.time(), which upon exceeding ~9hrs raises a TRAIN_END_FLAG and breaks out of the training loop and jumps straight to inference.\n10. So in summary:\n- Set up the environment.\n- Write the training script main.py which can accept root_dir and output_dir.\n- Name the output csv as submission.csv.\n- Include code snippets to ensure memory and time constraints.\n- Create a zip file with TEAM_NAME.zip.\n- Submit the zip file on the [submission link](https://forms.gle/YRHJRgcu29xsHUfL8).", "votes": null }, { "id": "2311318", "postDate": "06/21/2023 05:19:35", "content": "

Hello @rishabh2001 , we just submitted our solution a few days ago, would you please let us know if our solution is well received and being put on the leaderboard? by the way is there any way to get the training logs from your side?

\n

Thanks!

", "rawMarkdown": "Hello @rishabh2001 , we just submitted our solution a few days ago, would you please let us know if our solution is well received and being put on the leaderboard? by the way is there any way to get the training logs from your side?\n\nThanks!", "votes": null }, { "id": "2311340", "postDate": "06/21/2023 05:56:40", "content": "

Hi, @tuvovan211, Thanks for participating in the competition. We received your submission on 5/12/2023 at 12:12:35:09, but the submission is not compliant with the format expected. Your submission consists of the training script with config file which is to be trained using an ipynb notebook. For a uniform evaluation, please take note of instruction number 3 provided. The trainer_script has been provided for reference.

", "rawMarkdown": "Hi, @tuvovan211, Thanks for participating in the competition. We received your submission on 5/12/2023 at 12:12:35:09, but the submission is not compliant with the format expected. Your submission consists of the training script with config file which is to be trained using an ipynb notebook. For a uniform evaluation, please take note of instruction number 3 provided. The [trainer_script](https://www.kaggle.com/code/mlnoob327/main-py) has been provided for reference.", "votes": null }, { "id": "2311360", "postDate": "06/21/2023 06:08:45", "content": "

Hello, I submitted another version just few days ago (by following the guideline above), would you please check again?

", "rawMarkdown": "Hello, I submitted another version just few days ago (by following the guideline above), would you please check again?", "votes": null }, { "id": "2311362", "postDate": "06/21/2023 06:10:51", "content": "

\"\"

\n

This is the email that I got after submitting the code! @mlnoob327

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1400649%2F063b7628353bbfb05efa4d95ba9732d5%2FScreen%20Shot%202023-06-21%20at%203.09.52%20PM.png?generation=1687327824418926&alt=media)\n\nThis is the email that I got after submitting the code! @mlnoob327", "votes": null }, { "id": "2311432", "postDate": "06/21/2023 07:01:05", "content": "

Hi @tuvovan211,
\nThe screenshot suggests that submission has been made to the Budgeted Model Inference
\ncompetition. Please make use of the link provided under instruction number 10 to make the submission for this this competition which has the title: RCV Challenge 1 Code Submission. Ensure that no violations are made to reflect the position on the leaderboard. Thanks!

", "rawMarkdown": "Hi @tuvovan211, \nThe screenshot suggests that submission has been made to the [Budgeted Model Inference](https://www.kaggle.com/competitions/budgeted-model-inference-iccv-2023-rcv-workshop/overview)\ncompetition. Please make use of the link provided under instruction number 10 to make the submission for this this competition which has the title: **RCV Challenge 1 Code Submission**. Ensure that no violations are made to reflect the position on the leaderboard. Thanks!", "votes": null }, { "id": "2311506", "postDate": "06/21/2023 07:46:17", "content": "

@mlnoob327 hello, yeah thank you for pointing this out. I just submitted a new version, please check and let me know if there is any issue.

\n

thank you!

", "rawMarkdown": "mlnoob327 hello, yeah thank you for pointing this out. I just submitted a new version, please check and let me know if there is any issue.\n\nthank you!", "votes": null }, { "id": "2313042", "postDate": "06/22/2023 11:16:31", "content": "

@tuvovan211 We will run your submission for the next iteration of the leaderboard. If any issues pop up, we will inform you accordingly. Thanks!

", "rawMarkdown": "tuvovan211 We will run your submission for the next iteration of the leaderboard. If any issues pop up, we will inform you accordingly. Thanks!", "votes": null }, { "id": "2319555", "postDate": "06/27/2023 07:46:49", "content": "

Hi, for the instruction number 3 provided, if I have some other python files such as utils.py or other dependency python files, can I zip them all together to submit? Or should I move all the dependency code in these files to the main.py and then only submit man.py?

", "rawMarkdown": "Hi, for the instruction number 3 provided, if I have some other python files such as utils.py or other dependency python files, can I zip them all together to submit? Or should I move all the dependency code in these files to the main.py and then only submit man.py?", "votes": null }, { "id": "2319568", "postDate": "06/27/2023 07:57:15", "content": "

Please include all the code in main.py itself, thanks!

", "rawMarkdown": "Please include all the code in main.py itself, thanks!", "votes": null }, { "id": "2323747", "postDate": "06/30/2023 06:35:17", "content": "

Hi, after submitting, is there any feedback to tell us the code runs failed or successful?
\nAnd if the code has no problem, how long can we see the scores on the final leaderboard?

", "rawMarkdown": "Hi, after submitting, is there any feedback to tell us the code runs failed or successful? \nAnd if the code has no problem, how long can we see the scores on the final leaderboard?", "votes": null }, { "id": "2329406", "postDate": "07/04/2023 08:58:42", "content": "

Hi, @rishabh2001

\n

First of all, thank you for hosting such a wonderful competition.
\nFor a fair and competitive competition, we ask the following questions:

\n
    \n

  1. Model: In the case of Track 1 (Budgeted Model Training), the use of pretrained models is prohibited. Can't Track 2 also use pretrained models? If available, please tell us the available range(i.e. public available pretrain weights is ok, but private pretrain weights are not available)?

    2. \n

  2. Data: Can we use other data or generated data? or just use provided data? Also, is \bpseudo labeling possible for training? We expect both questions to be impossible, but we're asking because we want to be sure.

    3. \n

  3. Others: Like inference restrictions (gpu limit 16GB, RAM 32G…), are there any restrictions in the training stage (time, GPU limit)? Also, is there a plan for reproducibility verification of the learned model weights after the competition?

    4. \n
", "rawMarkdown": "Hi, @rishabh2001 \n\nFirst of all, thank you for hosting such a wonderful competition.\nFor a fair and competitive competition, we ask the following questions:\n\n1. Model: In the case of Track 1 (Budgeted Model Training), the use of pretrained models is prohibited. Can't Track 2 also use pretrained models? If available, please tell us the available range(i.e. public available pretrain weights is ok, but private pretrain weights are not available)?\n\n2. Data: Can we use other data or generated data? or just use provided data? Also, is \bpseudo labeling possible for training? We expect both questions to be impossible, but we're asking because we want to be sure.\n\n3. Others: Like inference restrictions (gpu limit 16GB, RAM 32G...), are there any restrictions in the training stage (time, GPU limit)? Also, is there a plan for reproducibility verification of the learned model weights after the competition?", "votes": null }, { "id": "2329448", "postDate": "07/04/2023 09:23:30", "content": "

Hi @shanweizsw, Thanks for taking part in the competition. We update the leaderboard periodically. You can have a look at the positions after submitting. Valid submissions with no error will find their positions here. If in case we encounter an error due to some bug, we will inform back via the email provided in the Google form.

", "rawMarkdown": "Hi @shanweizsw, Thanks for taking part in the competition. We update the [leaderboard](https://sites.google.com/view/rcv2023/leaderboard-workshop-challenges?authuser=0) periodically. You can have a look at the positions after submitting. Valid submissions with no error will find their positions here. If in case we encounter an error due to some bug, we will inform back via the email provided in the Google form.", "votes": null }, { "id": "2330044", "postDate": "07/04/2023 17:17:01", "content": "

Hi @sunghun, thanks for your kind words

\n
    \n
  1. For Track 2 pretrained weights (publicly available and trained on public dataset) are allowed.
    2. \n
  2. For track 2, although it is not prohibited, we would like to point out that the aim of the budgeted inference competition is more on efficient inference while maintaining decent accuracy. So we would recommend focussing more on increasing the inference speed.
    3. \n
  3. No there is no restriction on memory or time during training stage of inference track.
    \n3b. Track 1 inherently requires submitting training code and ensures reproducibility, for Track II we will ask for training code from the winners.
    4. \n
\n

Thanks for these questions, hope we are able to clarify everything (we will update the current instructions on the competition overview to include some of these clarifications). Please let us know if you have any other questions.

", "rawMarkdown": "Hi @sunghun, thanks for your kind words\n\n1. For Track 2 pretrained weights (publicly available and trained on public dataset) are allowed.\n2. For track 2, although it is not prohibited, we would like to point out that the aim of the budgeted inference competition is more on efficient inference while maintaining decent accuracy. So we would recommend focussing more on increasing the inference speed.\n3. No there is no restriction on memory or time during training stage of inference track. \n3b. Track 1 inherently requires submitting training code and ensures reproducibility, for Track II we will ask for training code from the winners.\n\nThanks for these questions, hope we are able to clarify everything (we will update the current instructions on the competition overview to include some of these clarifications). Please let us know if you have any other questions.", "votes": null }, { "id": "2330569", "postDate": "07/05/2023 04:55:04", "content": "

@rishabh2001
\nHello, I'm planning to submit a new version of the training code and I want to make sure if the result of the new code is lower than the current one, is the best one kept on the leaderboard or the most recent one?

\n

Thanks!

", "rawMarkdown": "rishabh2001 \nHello, I'm planning to submit a new version of the training code and I want to make sure if the result of the new code is lower than the current one, is the best one kept on the leaderboard or the most recent one?\n\nThanks!", "votes": null }, { "id": "2331164", "postDate": "07/05/2023 12:04:34", "content": "

Hi, We always update the leaderboard with the latest submission. You can choose to resubmit your best submission at the end to reflect your best score on the leaderboard.

", "rawMarkdown": "Hi, We always update the leaderboard with the latest submission. You can choose to resubmit your best submission at the end to reflect your best score on the leaderboard.", "votes": null }, { "id": "2336522", "postDate": "07/09/2023 12:15:35", "content": "

Thank you. @rishabh2001 ,
\nThank you for your kind and quick reply. Based on your answers, We were able to submit a submission.
\nWe have one additional question.

\n

When will the leaderboard for Phase 2 be updated?
\nAccording to the explanation you provided, I know that it is updated every 5 days, but if you look at the button at the bottom left of the Phase 2 leaderboard page, the last update time is 8 days ago.
\nThe competition is about 20 days away, and each team has only about 3-4 chances to submit.
\nIn particular, the team that submits for the first time, including our team, has no experience with issues such as memory and timeout, so the leaderboard update time is relatively important.

\n

Can you give us an approximate date and time for leaderboard updates (i.e. 7/15, 7/20, 7/30, 7/31 9:00 GMT-4) until the end of the competition? If possible, it will be a competition where more teams participate competitively(including our team).

", "rawMarkdown": "Thank you. @rishabh2001 , \nThank you for your kind and quick reply. Based on your answers, We were able to submit a submission.\nWe have one additional question.\n\nWhen will the leaderboard for Phase 2 be updated? \nAccording to the explanation you provided, I know that it is updated every 5 days, but if you look at the button at the bottom left of the Phase 2 leaderboard page, the last update time is 8 days ago.\nThe competition is about 20 days away, and each team has only about 3-4 chances to submit.\nIn particular, the team that submits for the first time, including our team, has no experience with issues such as memory and timeout, so the leaderboard update time is relatively important.\n\nCan you give us an approximate date and time for leaderboard updates (i.e. 7/15, 7/20, 7/30, 7/31 9:00 GMT-4) until the end of the competition? If possible, it will be a competition where more teams participate competitively(including our team).", "votes": null }, { "id": "2337278", "postDate": "07/10/2023 03:29:36", "content": "

Hi @sunghun,
\nWe are working to refresh and scale up our servers and would like to let you know that from today onwards we will try to update the leaderboard every 2 days (exact timing will depend on the total volume of new submissions).

\n

For Budgeted Inference challenge, we will be updating the leaderboard everyday, latest update being today morning.

", "rawMarkdown": "Hi @sunghun, \nWe are working to refresh and scale up our servers and would like to let you know that from today onwards we will try to update the leaderboard every 2 days (exact timing will depend on the total volume of new submissions). \n\nFor Budgeted Inference challenge, we will be updating the leaderboard everyday, latest update being today morning.", "votes": null }, { "id": "2337290", "postDate": "07/10/2023 03:38:06", "content": "

Hi @rishabh2001 , for the phase 2 of both training and inference track, is there any submission times limit for one team? Or we can always submit our updated version as we wish before the phase 2 ends?

", "rawMarkdown": "Hi @rishabh2001 , for the phase 2 of both training and inference track, is there any submission times limit for one team? Or we can always submit our updated version as we wish before the phase 2 ends?", "votes": null }, { "id": "2337576", "postDate": "07/10/2023 07:51:49", "content": "

There is no such hard limit on number of submissions but given running scripts takes time so you can expect us to evaluate and update leaderboard one submission (latest) per 2 days for training and 1 submission (latest) per day for inference track.

", "rawMarkdown": "There is no such hard limit on number of submissions but given running scripts takes time so you can expect us to evaluate and update leaderboard one submission (latest) per 2 days for training and 1 submission (latest) per day for inference track.", "votes": null }, { "id": "2337641", "postDate": "07/10/2023 09:00:57", "content": "

@rishabh2001
\nhello. A few days ago (July 5th) I submitted \"the phase 2 step\" training code in \"Track 1\". However, I have not received any feedback via email and cannot find my score on the leaderboard. I wonder if I should wait a few more days or not.

", "rawMarkdown": "rishabh2001\nhello. A few days ago (July 5th) I submitted \"the phase 2 step\" training code in \"Track 1\". However, I have not received any feedback via email and cannot find my score on the leaderboard. I wonder if I should wait a few more days or not.", "votes": null }, { "id": "2338196", "postDate": "07/10/2023 15:50:17", "content": "

Hi @helloimyjk, we are in the process of evaluating the current submissions, leaderboard should be updated by tomorrow. Thanks!

", "rawMarkdown": "Hi @helloimyjk, we are in the process of evaluating the current submissions, leaderboard should be updated by tomorrow. Thanks!", "votes": null }, { "id": "2341550", "postDate": "07/12/2023 06:54:16", "content": "

@rishabh2001 , Once in two days! Thank you so much.
\nMany teams seem to be able to use various strategies based on this.
\nIt seems that the organizers will go through a lot of time and effort due to the above decision, but we're so grateful that the organizers made this decision nonetheless.

", "rawMarkdown": "rishabh2001 , Once in two days! Thank you so much. \nMany teams seem to be able to use various strategies based on this. \nIt seems that the organizers will go through a lot of time and effort due to the above decision, but we're so grateful that the organizers made this decision nonetheless.", "votes": null }, { "id": "2345601", "postDate": "07/15/2023 14:19:58", "content": "

@rishabh2001
\nHello. I am still waiting for your updates. If you do not mind, I'd like to know the future schedule of leaderboard updates.

", "rawMarkdown": "rishabh2001 \nHello. I am still waiting for your updates. If you do not mind, I'd like to know the future schedule of leaderboard updates.", "votes": null }, { "id": "2345800", "postDate": "07/15/2023 18:00:30", "content": "

Hi @helloimyjk, the evaluation servers are up and running, we have also updated the leaderboard today. It will be updated on alternate days in the night.

", "rawMarkdown": "Hi @helloimyjk, the evaluation servers are up and running, we have also updated the leaderboard today. It will be updated on alternate days in the night.", "votes": null }, { "id": "2346359", "postDate": "07/16/2023 07:40:16", "content": "

Hi, for the training track, why my score on the Phase 2 leaderboard changed? For phase 2, I only submitted one time for now. The score was around 86.0 before. But now, it shows 83.48. What could be the reason for this?

", "rawMarkdown": "Hi, for the training track, why my score on the Phase 2 leaderboard changed? For phase 2, I only submitted one time for now. The score was around 86.0 before. But now, it shows 83.48. What could be the reason for this?", "votes": null }, { "id": "2346610", "postDate": "07/16/2023 11:40:26", "content": "

Dear @shanweizsw ,

\n

For now, we have put scores using single seeds and were internally testing for 5 seeds. The score update was an error from our side and we have corrected it now. However, after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds.

", "rawMarkdown": "Dear @shanweizsw ,\n\nFor now, we have put scores using single seeds and were internally testing for 5 seeds. The score update was an error from our side and we have corrected it now. However, after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds.", "votes": null }, { "id": "2353889", "postDate": "07/22/2023 05:29:30", "content": "

Hello, @rishabh2001. Thank you for your effort. I would like to know when to update the latest code. FYI, I submitted the new file on the 19th of July.

", "rawMarkdown": "Hello, @rishabh2001. Thank you for your effort. I would like to know when to update the latest code. FYI, I submitted the new file on the 19th of July.", "votes": null }, { "id": "2360771", "postDate": "07/27/2023 02:37:25", "content": "

I submitted the same code, 1st time was around 86 but now it is only 84.3. What could be the reason?

", "rawMarkdown": "I submitted the same code, 1st time was around 86 but now it is only 84.3. What could be the reason?", "votes": null }, { "id": "2360772", "postDate": "07/27/2023 02:37:44", "content": "

@dkgupta90

", "rawMarkdown": "dkgupta90", "votes": null }, { "id": "2360792", "postDate": "07/27/2023 02:56:56", "content": "

hi, I submitted the best submission (same as the first one) but it seems to decrease my score. could you please double check? @rishabh2001

", "rawMarkdown": "hi, I submitted the best submission (same as the first one) but it seems to decrease my score. could you please double check? @rishabh2001", "votes": null }, { "id": "2368101", "postDate": "08/01/2023 02:31:36", "content": "

Hi, thanks for your effort for this competition. Deepak Gupta mentioned before that \"after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds\". Since I see the top1 score 91.38 remained unchanged, so I would like to ensure that, is the score on the final leaderboard runned by 5 seeds for all the top methods? Or there will be an updated final version for the leaderboard score?

", "rawMarkdown": "Hi, thanks for your effort for this competition. Deepak Gupta mentioned before that \"after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds\". Since I see the top1 score 91.38 remained unchanged, so I would like to ensure that, is the score on the final leaderboard runned by 5 seeds for all the top methods? Or there will be an updated final version for the leaderboard score?", "votes": null }, { "id": "2370167", "postDate": "08/02/2023 08:28:09", "content": "

The current leaderboard is using only one seed, the leaderboard will be updated after running all the last submissions for multiple seeds. That will take few days from our side. On the side the candidate top teams will receive a mail by tomorrow regarding next steps to follow.
\nThanks for your patience.

", "rawMarkdown": "The current leaderboard is using only one seed, the leaderboard will be updated after running all the last submissions for multiple seeds. That will take few days from our side. On the side the candidate top teams will receive a mail by tomorrow regarding next steps to follow. \nThanks for your patience.", "votes": null }, { "id": "2374742", "postDate": "08/05/2023 07:45:06", "content": "

Hi, thanks for your reply. Since I have not received the email mentioned in your previous message, I would like to confirm that whether this is within normal process, or if I may missed any updated information?

\n

Also, for the final leaderboard which will be updated by running on multiple seeds, is there any updated information or time for that?

\n

Thanks again for your effort for this competition.

", "rawMarkdown": "Hi, thanks for your reply. Since I have not received the email mentioned in your previous message, I would like to confirm that whether this is within normal process, or if I may missed any updated information? \n\nAlso, for the final leaderboard which will be updated by running on multiple seeds, is there any updated information or time for that? \n\nThanks again for your effort for this competition.", "votes": null }, { "id": "2405517", "postDate": "08/24/2023 00:26:16", "content": "

@shanweizsw any updates?

", "rawMarkdown": "shanweizsw any updates?", "votes": null } ], "comments": [ { "id": 2311318, "author_name": "tuvovan211", "author_url": "", "post_date": "06/21/2023 05:19:35", "content": "

Hello @rishabh2001 , we just submitted our solution a few days ago, would you please let us know if our solution is well received and being put on the leaderboard? by the way is there any way to get the training logs from your side?

\n

Thanks!

", "votes": null, "replies": [ { "id": 2311340, "author_name": "mlnoob327", "author_url": "", "post_date": "06/21/2023 05:56:40", "content": "

Hi, @tuvovan211, Thanks for participating in the competition. We received your submission on 5/12/2023 at 12:12:35:09, but the submission is not compliant with the format expected. Your submission consists of the training script with config file which is to be trained using an ipynb notebook. For a uniform evaluation, please take note of instruction number 3 provided. The trainer_script has been provided for reference.

", "votes": null, "replies": [ { "id": 2311360, "author_name": "tuvovan211", "author_url": "", "post_date": "06/21/2023 06:08:45", "content": "

Hello, I submitted another version just few days ago (by following the guideline above), would you please check again?

", "votes": null, "replies": [] }, { "id": 2311362, "author_name": "tuvovan211", "author_url": "", "post_date": "06/21/2023 06:10:51", "content": "

\"\"

\n

This is the email that I got after submitting the code! @mlnoob327

", "votes": null, "replies": [ { "id": 2311432, "author_name": "mlnoob327", "author_url": "", "post_date": "06/21/2023 07:01:05", "content": "

Hi @tuvovan211,
\nThe screenshot suggests that submission has been made to the Budgeted Model Inference
\ncompetition. Please make use of the link provided under instruction number 10 to make the submission for this this competition which has the title: RCV Challenge 1 Code Submission. Ensure that no violations are made to reflect the position on the leaderboard. Thanks!

", "votes": null, "replies": [ { "id": 2311506, "author_name": "tuvovan211", "author_url": "", "post_date": "06/21/2023 07:46:17", "content": "

@mlnoob327 hello, yeah thank you for pointing this out. I just submitted a new version, please check and let me know if there is any issue.

\n

thank you!

", "votes": null, "replies": [ { "id": 2313042, "author_name": "mlnoob327", "author_url": "", "post_date": "06/22/2023 11:16:31", "content": "

@tuvovan211 We will run your submission for the next iteration of the leaderboard. If any issues pop up, we will inform you accordingly. Thanks!

", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2319555, "author_name": "shanweizsw", "author_url": "", "post_date": "06/27/2023 07:46:49", "content": "

Hi, for the instruction number 3 provided, if I have some other python files such as utils.py or other dependency python files, can I zip them all together to submit? Or should I move all the dependency code in these files to the main.py and then only submit man.py?

", "votes": null, "replies": [ { "id": 2319568, "author_name": "rishabh2001", "author_url": "", "post_date": "06/27/2023 07:57:15", "content": "

Please include all the code in main.py itself, thanks!

", "votes": null, "replies": [] } ] }, { "id": 2323747, "author_name": "shanweizsw", "author_url": "", "post_date": "06/30/2023 06:35:17", "content": "

Hi, after submitting, is there any feedback to tell us the code runs failed or successful?
\nAnd if the code has no problem, how long can we see the scores on the final leaderboard?

", "votes": null, "replies": [ { "id": 2329448, "author_name": "mlnoob327", "author_url": "", "post_date": "07/04/2023 09:23:30", "content": "

Hi @shanweizsw, Thanks for taking part in the competition. We update the leaderboard periodically. You can have a look at the positions after submitting. Valid submissions with no error will find their positions here. If in case we encounter an error due to some bug, we will inform back via the email provided in the Google form.

", "votes": null, "replies": [] } ] }, { "id": 2329406, "author_name": "sunghun", "author_url": "", "post_date": "07/04/2023 08:58:42", "content": "

Hi, @rishabh2001

\n

First of all, thank you for hosting such a wonderful competition.
\nFor a fair and competitive competition, we ask the following questions:

\n
    \n

  1. Model: In the case of Track 1 (Budgeted Model Training), the use of pretrained models is prohibited. Can't Track 2 also use pretrained models? If available, please tell us the available range(i.e. public available pretrain weights is ok, but private pretrain weights are not available)?

    2. \n

  2. Data: Can we use other data or generated data? or just use provided data? Also, is \bpseudo labeling possible for training? We expect both questions to be impossible, but we're asking because we want to be sure.

    3. \n

  3. Others: Like inference restrictions (gpu limit 16GB, RAM 32G…), are there any restrictions in the training stage (time, GPU limit)? Also, is there a plan for reproducibility verification of the learned model weights after the competition?

    4. \n
", "votes": null, "replies": [ { "id": 2330044, "author_name": "rishabh2001", "author_url": "", "post_date": "07/04/2023 17:17:01", "content": "

Hi @sunghun, thanks for your kind words

\n
    \n
  1. For Track 2 pretrained weights (publicly available and trained on public dataset) are allowed.
    2. \n
  2. For track 2, although it is not prohibited, we would like to point out that the aim of the budgeted inference competition is more on efficient inference while maintaining decent accuracy. So we would recommend focussing more on increasing the inference speed.
    3. \n
  3. No there is no restriction on memory or time during training stage of inference track.
    \n3b. Track 1 inherently requires submitting training code and ensures reproducibility, for Track II we will ask for training code from the winners.
    4. \n
\n

Thanks for these questions, hope we are able to clarify everything (we will update the current instructions on the competition overview to include some of these clarifications). Please let us know if you have any other questions.

", "votes": null, "replies": [ { "id": 2336522, "author_name": "sunghun", "author_url": "", "post_date": "07/09/2023 12:15:35", "content": "

Thank you. @rishabh2001 ,
\nThank you for your kind and quick reply. Based on your answers, We were able to submit a submission.
\nWe have one additional question.

\n

When will the leaderboard for Phase 2 be updated?
\nAccording to the explanation you provided, I know that it is updated every 5 days, but if you look at the button at the bottom left of the Phase 2 leaderboard page, the last update time is 8 days ago.
\nThe competition is about 20 days away, and each team has only about 3-4 chances to submit.
\nIn particular, the team that submits for the first time, including our team, has no experience with issues such as memory and timeout, so the leaderboard update time is relatively important.

\n

Can you give us an approximate date and time for leaderboard updates (i.e. 7/15, 7/20, 7/30, 7/31 9:00 GMT-4) until the end of the competition? If possible, it will be a competition where more teams participate competitively(including our team).

", "votes": null, "replies": [ { "id": 2337278, "author_name": "rishabh2001", "author_url": "", "post_date": "07/10/2023 03:29:36", "content": "

Hi @sunghun,
\nWe are working to refresh and scale up our servers and would like to let you know that from today onwards we will try to update the leaderboard every 2 days (exact timing will depend on the total volume of new submissions).

\n

For Budgeted Inference challenge, we will be updating the leaderboard everyday, latest update being today morning.

", "votes": null, "replies": [ { "id": 2337290, "author_name": "shanweizsw", "author_url": "", "post_date": "07/10/2023 03:38:06", "content": "

Hi @rishabh2001 , for the phase 2 of both training and inference track, is there any submission times limit for one team? Or we can always submit our updated version as we wish before the phase 2 ends?

", "votes": null, "replies": [ { "id": 2337576, "author_name": "rishabh2001", "author_url": "", "post_date": "07/10/2023 07:51:49", "content": "

There is no such hard limit on number of submissions but given running scripts takes time so you can expect us to evaluate and update leaderboard one submission (latest) per 2 days for training and 1 submission (latest) per day for inference track.

", "votes": null, "replies": [ { "id": 2341550, "author_name": "sunghun", "author_url": "", "post_date": "07/12/2023 06:54:16", "content": "

@rishabh2001 , Once in two days! Thank you so much.
\nMany teams seem to be able to use various strategies based on this.
\nIt seems that the organizers will go through a lot of time and effort due to the above decision, but we're so grateful that the organizers made this decision nonetheless.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2330569, "author_name": "tuvovan211", "author_url": "", "post_date": "07/05/2023 04:55:04", "content": "

@rishabh2001
\nHello, I'm planning to submit a new version of the training code and I want to make sure if the result of the new code is lower than the current one, is the best one kept on the leaderboard or the most recent one?

\n

Thanks!

", "votes": null, "replies": [ { "id": 2331164, "author_name": "rishabh2001", "author_url": "", "post_date": "07/05/2023 12:04:34", "content": "

Hi, We always update the leaderboard with the latest submission. You can choose to resubmit your best submission at the end to reflect your best score on the leaderboard.

", "votes": null, "replies": [ { "id": 2360792, "author_name": "tuvovan211", "author_url": "", "post_date": "07/27/2023 02:56:56", "content": "

hi, I submitted the best submission (same as the first one) but it seems to decrease my score. could you please double check? @rishabh2001

", "votes": null, "replies": [] } ] } ] }, { "id": 2337641, "author_name": "helloimyjk", "author_url": "", "post_date": "07/10/2023 09:00:57", "content": "

@rishabh2001
\nhello. A few days ago (July 5th) I submitted \"the phase 2 step\" training code in \"Track 1\". However, I have not received any feedback via email and cannot find my score on the leaderboard. I wonder if I should wait a few more days or not.

", "votes": null, "replies": [ { "id": 2338196, "author_name": "rishabh2001", "author_url": "", "post_date": "07/10/2023 15:50:17", "content": "

Hi @helloimyjk, we are in the process of evaluating the current submissions, leaderboard should be updated by tomorrow. Thanks!

", "votes": null, "replies": [ { "id": 2345601, "author_name": "helloimyjk", "author_url": "", "post_date": "07/15/2023 14:19:58", "content": "

@rishabh2001
\nHello. I am still waiting for your updates. If you do not mind, I'd like to know the future schedule of leaderboard updates.

", "votes": null, "replies": [ { "id": 2345800, "author_name": "rishabh2001", "author_url": "", "post_date": "07/15/2023 18:00:30", "content": "

Hi @helloimyjk, the evaluation servers are up and running, we have also updated the leaderboard today. It will be updated on alternate days in the night.

", "votes": null, "replies": [ { "id": 2353889, "author_name": "helloimyjk", "author_url": "", "post_date": "07/22/2023 05:29:30", "content": "

Hello, @rishabh2001. Thank you for your effort. I would like to know when to update the latest code. FYI, I submitted the new file on the 19th of July.

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2346359, "author_name": "shanweizsw", "author_url": "", "post_date": "07/16/2023 07:40:16", "content": "

Hi, for the training track, why my score on the Phase 2 leaderboard changed? For phase 2, I only submitted one time for now. The score was around 86.0 before. But now, it shows 83.48. What could be the reason for this?

", "votes": null, "replies": [ { "id": 2346610, "author_name": "dkgupta90", "author_url": "", "post_date": "07/16/2023 11:40:26", "content": "

Dear @shanweizsw ,

\n

For now, we have put scores using single seeds and were internally testing for 5 seeds. The score update was an error from our side and we have corrected it now. However, after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds.

", "votes": null, "replies": [ { "id": 2360771, "author_name": "tuvovan211", "author_url": "", "post_date": "07/27/2023 02:37:25", "content": "

I submitted the same code, 1st time was around 86 but now it is only 84.3. What could be the reason?

", "votes": null, "replies": [ { "id": 2360772, "author_name": "tuvovan211", "author_url": "", "post_date": "07/27/2023 02:37:44", "content": "

@dkgupta90

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2368101, "author_name": "shanweizsw", "author_url": "", "post_date": "08/01/2023 02:31:36", "content": "

Hi, thanks for your effort for this competition. Deepak Gupta mentioned before that \"after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds\". Since I see the top1 score 91.38 remained unchanged, so I would like to ensure that, is the score on the final leaderboard runned by 5 seeds for all the top methods? Or there will be an updated final version for the leaderboard score?

", "votes": null, "replies": [ { "id": 2370167, "author_name": "rishabh2001", "author_url": "", "post_date": "08/02/2023 08:28:09", "content": "

The current leaderboard is using only one seed, the leaderboard will be updated after running all the last submissions for multiple seeds. That will take few days from our side. On the side the candidate top teams will receive a mail by tomorrow regarding next steps to follow.
\nThanks for your patience.

", "votes": null, "replies": [ { "id": 2374742, "author_name": "shanweizsw", "author_url": "", "post_date": "08/05/2023 07:45:06", "content": "

Hi, thanks for your reply. Since I have not received the email mentioned in your previous message, I would like to confirm that whether this is within normal process, or if I may missed any updated information?

\n

Also, for the final leaderboard which will be updated by running on multiple seeds, is there any updated information or time for that?

\n

Thanks again for your effort for this competition.

", "votes": null, "replies": [ { "id": 2405517, "author_name": "tuvovan211", "author_url": "", "post_date": "08/24/2023 00:26:16", "content": "

@shanweizsw any updates?

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2296901": "We received a lot of invalid submissions so here is the stepwise guide for a valid submission. \n\nThe second Phase of the competition involves testing the code submissions on ImageNet100. The chosen classes are different from what is released on Kaggle. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.\n\n1. Ensure that the code submitted runs on the environment provided in the [yml](https://www.kaggle.com/competitions/budgeted-model-training-iccv-2023-rcv-workshop/data?select=rcv.yml) file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.\n2. The submissions must be done in a zip file with the team name attached as in **TEAM_NAME.zip**. This TEAM_NAME will be displayed on the Phase 2 leaderboard.\n3. The zip file should only contain the training script with the name **main.py**. Participants are requested to hardcode the parameters inside the script as **no config files will be accepted**. \n4. The script will be tested against the data directory containing data for Phase 2. The data follows a similar format to what is put on Kaggle. The data directory contains 3 folders: train, val, and test along with 2 csv’s: train.csv and val.csv. Participants are requested to create dataloaders accordingly or use the ones provided in the [trainer script](https://www.kaggle.com/code/mlnoob327/main-py).\n5. Participants are requested to include the mandatory arguments **root_dir** and **output_dir** inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump the models and submission.csv.\n6. The script is expected to fetch the training, validation, and test data from the data directory, train the model, complete inference on the test data provided, and dump the predictions as **submission.csv** inside the output_dir provided as the argument. Reference for data loaders is provided in the [trainer script](https://www.kaggle.com/code/mlnoob327/main-py).\n7. The competition puts forward 2 constraints:\na. Training under 9 hours of the time limit.\nb. Training with a budgeted CUDA memory of 6GB.\n8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code. \n`tup = torch.cuda.mem_get_info()`\n`torch.cuda.set_per_process_memory_fraction((6*(1024)**3)/tup[1], 0)`\n9. It is expected that the participants include the logic to break the training loop or set the number of epochs such that the train time constraint of 9 hrs is not violated. An example of how this can be done is presented in the [trainer script](https://www.kaggle.com/code/mlnoob327/main-py) using time.time(), which upon exceeding ~9hrs raises a TRAIN_END_FLAG and breaks out of the training loop and jumps straight to inference.\n10. So in summary:\n- Set up the environment.\n- Write the training script main.py which can accept root_dir and output_dir.\n- Name the output csv as submission.csv.\n- Include code snippets to ensure memory and time constraints.\n- Create a zip file with TEAM_NAME.zip.\n- Submit the zip file on the [submission link](https://forms.gle/YRHJRgcu29xsHUfL8).", "2311318": "Hello @rishabh2001 , we just submitted our solution a few days ago, would you please let us know if our solution is well received and being put on the leaderboard? by the way is there any way to get the training logs from your side?\n\nThanks!", "2311340": "Hi, @tuvovan211, Thanks for participating in the competition. We received your submission on 5/12/2023 at 12:12:35:09, but the submission is not compliant with the format expected. Your submission consists of the training script with config file which is to be trained using an ipynb notebook. For a uniform evaluation, please take note of instruction number 3 provided. The [trainer_script](https://www.kaggle.com/code/mlnoob327/main-py) has been provided for reference.", "2311360": "Hello, I submitted another version just few days ago (by following the guideline above), would you please check again?", "2311362": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1400649%2F063b7628353bbfb05efa4d95ba9732d5%2FScreen%20Shot%202023-06-21%20at%203.09.52%20PM.png?generation=1687327824418926&alt=media)\n\nThis is the email that I got after submitting the code! @mlnoob327", "2311432": "Hi @tuvovan211, \nThe screenshot suggests that submission has been made to the [Budgeted Model Inference](https://www.kaggle.com/competitions/budgeted-model-inference-iccv-2023-rcv-workshop/overview)\ncompetition. Please make use of the link provided under instruction number 10 to make the submission for this this competition which has the title: **RCV Challenge 1 Code Submission**. Ensure that no violations are made to reflect the position on the leaderboard. Thanks!", "2311506": "mlnoob327 hello, yeah thank you for pointing this out. I just submitted a new version, please check and let me know if there is any issue.\n\nthank you!", "2313042": "tuvovan211 We will run your submission for the next iteration of the leaderboard. If any issues pop up, we will inform you accordingly. Thanks!", "2319555": "Hi, for the instruction number 3 provided, if I have some other python files such as utils.py or other dependency python files, can I zip them all together to submit? Or should I move all the dependency code in these files to the main.py and then only submit man.py?", "2319568": "Please include all the code in main.py itself, thanks!", "2323747": "Hi, after submitting, is there any feedback to tell us the code runs failed or successful? \nAnd if the code has no problem, how long can we see the scores on the final leaderboard?", "2329406": "Hi, @rishabh2001 \n\nFirst of all, thank you for hosting such a wonderful competition.\nFor a fair and competitive competition, we ask the following questions:\n\n1. Model: In the case of Track 1 (Budgeted Model Training), the use of pretrained models is prohibited. Can't Track 2 also use pretrained models? If available, please tell us the available range(i.e. public available pretrain weights is ok, but private pretrain weights are not available)?\n\n2. Data: Can we use other data or generated data? or just use provided data? Also, is \bpseudo labeling possible for training? We expect both questions to be impossible, but we're asking because we want to be sure.\n\n3. Others: Like inference restrictions (gpu limit 16GB, RAM 32G...), are there any restrictions in the training stage (time, GPU limit)? Also, is there a plan for reproducibility verification of the learned model weights after the competition?", "2329448": "Hi @shanweizsw, Thanks for taking part in the competition. We update the [leaderboard](https://sites.google.com/view/rcv2023/leaderboard-workshop-challenges?authuser=0) periodically. You can have a look at the positions after submitting. Valid submissions with no error will find their positions here. If in case we encounter an error due to some bug, we will inform back via the email provided in the Google form.", "2330044": "Hi @sunghun, thanks for your kind words\n\n1. For Track 2 pretrained weights (publicly available and trained on public dataset) are allowed.\n2. For track 2, although it is not prohibited, we would like to point out that the aim of the budgeted inference competition is more on efficient inference while maintaining decent accuracy. So we would recommend focussing more on increasing the inference speed.\n3. No there is no restriction on memory or time during training stage of inference track. \n3b. Track 1 inherently requires submitting training code and ensures reproducibility, for Track II we will ask for training code from the winners.\n\nThanks for these questions, hope we are able to clarify everything (we will update the current instructions on the competition overview to include some of these clarifications). Please let us know if you have any other questions.", "2330569": "rishabh2001 \nHello, I'm planning to submit a new version of the training code and I want to make sure if the result of the new code is lower than the current one, is the best one kept on the leaderboard or the most recent one?\n\nThanks!", "2331164": "Hi, We always update the leaderboard with the latest submission. You can choose to resubmit your best submission at the end to reflect your best score on the leaderboard.", "2336522": "Thank you. @rishabh2001 , \nThank you for your kind and quick reply. Based on your answers, We were able to submit a submission.\nWe have one additional question.\n\nWhen will the leaderboard for Phase 2 be updated? \nAccording to the explanation you provided, I know that it is updated every 5 days, but if you look at the button at the bottom left of the Phase 2 leaderboard page, the last update time is 8 days ago.\nThe competition is about 20 days away, and each team has only about 3-4 chances to submit.\nIn particular, the team that submits for the first time, including our team, has no experience with issues such as memory and timeout, so the leaderboard update time is relatively important.\n\nCan you give us an approximate date and time for leaderboard updates (i.e. 7/15, 7/20, 7/30, 7/31 9:00 GMT-4) until the end of the competition? If possible, it will be a competition where more teams participate competitively(including our team).", "2337278": "Hi @sunghun, \nWe are working to refresh and scale up our servers and would like to let you know that from today onwards we will try to update the leaderboard every 2 days (exact timing will depend on the total volume of new submissions). \n\nFor Budgeted Inference challenge, we will be updating the leaderboard everyday, latest update being today morning.", "2337290": "Hi @rishabh2001 , for the phase 2 of both training and inference track, is there any submission times limit for one team? Or we can always submit our updated version as we wish before the phase 2 ends?", "2337576": "There is no such hard limit on number of submissions but given running scripts takes time so you can expect us to evaluate and update leaderboard one submission (latest) per 2 days for training and 1 submission (latest) per day for inference track.", "2337641": "rishabh2001\nhello. A few days ago (July 5th) I submitted \"the phase 2 step\" training code in \"Track 1\". However, I have not received any feedback via email and cannot find my score on the leaderboard. I wonder if I should wait a few more days or not.", "2338196": "Hi @helloimyjk, we are in the process of evaluating the current submissions, leaderboard should be updated by tomorrow. Thanks!", "2341550": "rishabh2001 , Once in two days! Thank you so much. \nMany teams seem to be able to use various strategies based on this. \nIt seems that the organizers will go through a lot of time and effort due to the above decision, but we're so grateful that the organizers made this decision nonetheless.", "2345601": "rishabh2001 \nHello. I am still waiting for your updates. If you do not mind, I'd like to know the future schedule of leaderboard updates.", "2345800": "Hi @helloimyjk, the evaluation servers are up and running, we have also updated the leaderboard today. It will be updated on alternate days in the night.", "2346359": "Hi, for the training track, why my score on the Phase 2 leaderboard changed? For phase 2, I only submitted one time for now. The score was around 86.0 before. But now, it shows 83.48. What could be the reason for this?", "2346610": "Dear @shanweizsw ,\n\nFor now, we have put scores using single seeds and were internally testing for 5 seeds. The score update was an error from our side and we have corrected it now. However, after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds.", "2353889": "Hello, @rishabh2001. Thank you for your effort. I would like to know when to update the latest code. FYI, I submitted the new file on the 19th of July.", "2360771": "I submitted the same code, 1st time was around 86 but now it is only 84.3. What could be the reason?", "2360772": "dkgupta90", "2360792": "hi, I submitted the best submission (same as the first one) but it seems to decrease my score. could you please double check? @rishabh2001", "2368101": "Hi, thanks for your effort for this competition. Deepak Gupta mentioned before that \"after the deadline passes, expect some shuffling in the leaderboard since we will run atleast the top 5 scores on the leaderboard using 5 seeds\". Since I see the top1 score 91.38 remained unchanged, so I would like to ensure that, is the score on the final leaderboard runned by 5 seeds for all the top methods? Or there will be an updated final version for the leaderboard score?", "2370167": "The current leaderboard is using only one seed, the leaderboard will be updated after running all the last submissions for multiple seeds. That will take few days from our side. On the side the candidate top teams will receive a mail by tomorrow regarding next steps to follow. \nThanks for your patience.", "2374742": "Hi, thanks for your reply. Since I have not received the email mentioned in your previous message, I would like to confirm that whether this is within normal process, or if I may missed any updated information? \n\nAlso, for the final leaderboard which will be updated by running on multiple seeds, is there any updated information or time for that? \n\nThanks again for your effort for this competition.", "2405517": "shanweizsw any updates?" }, "source": "meta" }