{
  "id": 58606,
  "title": "About the evaluation changes and their impacts on the results",
  "url": "/competitions/cvpr-2018-autonomous-driving/discussion/58606",
  "author_name": "",
  "post_date": "2018-06-11T05:47:02.186258300Z",
  "votes": 7,
  "comment_count": 1,
  "views": 0,
  "content": "<p>We respect and appreciate the Kaggle team and the competition organizer's effort and time on revising the evaluation code and hearing participants' voices once and again. </p>\n\n<p>However, the drastic changes and slowdown of the evaluation process during the last few days significantly impacted the competition results and might ruin lots of efforts from all hard-working teams.\nAs others complained on the discussion board, many not-so-large submissions were failed due to time-out even their size are much smaller than previous succeeded ones. Besides, lots of previous succeeded submissions failed to the new evaluations.</p>\n\n<p>Here are some records from our side:</p>\n\n<p>After the first re-evaluation finished (around Jun 10, 8:00am UTC time), our team had 88 valid entries with the best score of 0.23246.<img src=\"https://image.ibb.co/cLcCk8/Jun_10_8am_UTC.png\" alt=\"Jun_10_8am_UTC.png\"></p>\n\n<p>Without new submissions, a few hours later (around Jun 10, 4:00pm UTC time), we had fewer valid entries but higher scores (86 valid entries, best score of 0.23317). We believe that's from another re-evaluation.<img src=\"https://preview.ibb.co/ksFZXo/Jun_10_4pm_UTC.png\" alt=\"Jun_10_4pm_UTC.png\"></p>\n\n<p>Another few hours later (around Jul 11, 3:00am UTC time), we had even fewer valid entries (only 77 entries) and all our recent submissions within 3 days got an error.<img src=\"https://preview.ibb.co/djF1Co/Jun_11_3am_UTC.png\" alt=\"Jun_11_3am_UTC.png\"></p>\n\n<p>We believe all the errors are from time-out of the new evaluation code. Since our team, and possibly many other teams as well, optimized the submission file sizes and the number of instances based on the evaluation time of submissions, it is unfair and unreasonable to have such huge changes in a short period before the deadline of the competitions, especially without any anticipation or notification beforehand.</p>\n\n<p>We sincerely hope the organizers to announce necessary details about the changes in modifications to help all teams have better estimations on valid file size/ number of instances, and/or check and speed-up the evaluation code, and/or increase the evaluation time limit. We truly expect and believe the organizers will provide reasonable solutions or explanations.</p>",
  "messages": [
    {
      "id": "341162",
      "postDate": "06/11/2018 05:47:02",
      "content": "<p>We respect and appreciate the Kaggle team and the competition organizer's effort and time on revising the evaluation code and hearing participants' voices once and again. </p>\n\n<p>However, the drastic changes and slowdown of the evaluation process during the last few days significantly impacted the competition results and might ruin lots of efforts from all hard-working teams.\nAs others complained on the discussion board, many not-so-large submissions were failed due to time-out even their size are much smaller than previous succeeded ones. Besides, lots of previous succeeded submissions failed to the new evaluations.</p>\n\n<p>Here are some records from our side:</p>\n\n<p>After the first re-evaluation finished (around Jun 10, 8:00am UTC time), our team had 88 valid entries with the best score of 0.23246.<img src=\"https://image.ibb.co/cLcCk8/Jun_10_8am_UTC.png\" alt=\"Jun_10_8am_UTC.png\"></p>\n\n<p>Without new submissions, a few hours later (around Jun 10, 4:00pm UTC time), we had fewer valid entries but higher scores (86 valid entries, best score of 0.23317). We believe that's from another re-evaluation.<img src=\"https://preview.ibb.co/ksFZXo/Jun_10_4pm_UTC.png\" alt=\"Jun_10_4pm_UTC.png\"></p>\n\n<p>Another few hours later (around Jul 11, 3:00am UTC time), we had even fewer valid entries (only 77 entries) and all our recent submissions within 3 days got an error.<img src=\"https://preview.ibb.co/djF1Co/Jun_11_3am_UTC.png\" alt=\"Jun_11_3am_UTC.png\"></p>\n\n<p>We believe all the errors are from time-out of the new evaluation code. Since our team, and possibly many other teams as well, optimized the submission file sizes and the number of instances based on the evaluation time of submissions, it is unfair and unreasonable to have such huge changes in a short period before the deadline of the competitions, especially without any anticipation or notification beforehand.</p>\n\n<p>We sincerely hope the organizers to announce necessary details about the changes in modifications to help all teams have better estimations on valid file size/ number of instances, and/or check and speed-up the evaluation code, and/or increase the evaluation time limit. We truly expect and believe the organizers will provide reasonable solutions or explanations.</p>",
      "rawMarkdown": "We respect and appreciate the Kaggle team and the competition organizer's effort and time on revising the evaluation code and hearing participants' voices once and again. \n\nHowever, the drastic changes and slowdown of the evaluation process during the last few days significantly impacted the competition results and might ruin lots of efforts from all hard-working teams.\nAs others complained on the discussion board, many not-so-large submissions were failed due to time-out even their size are much smaller than previous succeeded ones. Besides, lots of previous succeeded submissions failed to the new evaluations.\n\nHere are some records from our side:\n\nAfter the first re-evaluation finished (around Jun 10, 8:00am UTC time), our team had 88 valid entries with the best score of 0.23246.![Jun_10_8am_UTC.png][1]\n\nWithout new submissions, a few hours later (around Jun 10, 4:00pm UTC time), we had fewer valid entries but higher scores (86 valid entries, best score of 0.23317). We believe that's from another re-evaluation.![Jun_10_4pm_UTC.png][2]\n\nAnother few hours later (around Jul 11, 3:00am UTC time), we had even fewer valid entries (only 77 entries) and all our recent submissions within 3 days got an error.![Jun_11_3am_UTC.png][3]\n\nWe believe all the errors are from time-out of the new evaluation code. Since our team, and possibly many other teams as well, optimized the submission file sizes and the number of instances based on the evaluation time of submissions, it is unfair and unreasonable to have such huge changes in a short period before the deadline of the competitions, especially without any anticipation or notification beforehand.\n\nWe sincerely hope the organizers to announce necessary details about the changes in modifications to help all teams have better estimations on valid file size/ number of instances, and/or check and speed-up the evaluation code, and/or increase the evaluation time limit. We truly expect and believe the organizers will provide reasonable solutions or explanations.\n\n\n  [1]: https://image.ibb.co/cLcCk8/Jun_10_8am_UTC.png\n  [2]: https://preview.ibb.co/ksFZXo/Jun_10_4pm_UTC.png\n  [3]: https://preview.ibb.co/djF1Co/Jun_11_3am_UTC.png",
      "votes": null
    },
    {
      "id": "341273",
      "postDate": "06/11/2018 10:37:18",
      "content": "<p>Hey, SaLaHaeiYou,\n    Not until your post did I realize that our team's valid submission has been flagged as invalid because of the time out error.  And we have totally not being properly informed! That hasty change indeed causes the goal of this challenged deviated drastically. Dismay is what I am feeling now (I was hoping to push the performance during the last hours, now the change of the rules almost negates that)...</p>",
      "rawMarkdown": "Hey, SaLaHaeiYou,\n    Not until your post did I realize that our team's valid submission has been flagged as invalid because of the time out error.  And we have totally not being properly informed! That hasty change indeed causes the goal of this challenged deviated drastically. Dismay is what I am feeling now (I was hoping to push the performance during the last hours, now the change of the rules almost negates that)...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 341273,
      "author_name": "stevenwudi",
      "author_url": "",
      "post_date": "06/11/2018 10:37:18",
      "content": "<p>Hey, SaLaHaeiYou,\n    Not until your post did I realize that our team's valid submission has been flagged as invalid because of the time out error.  And we have totally not being properly informed! That hasty change indeed causes the goal of this challenged deviated drastically. Dismay is what I am feeling now (I was hoping to push the performance during the last hours, now the change of the rules almost negates that)...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "341162": "We respect and appreciate the Kaggle team and the competition organizer's effort and time on revising the evaluation code and hearing participants' voices once and again. \n\nHowever, the drastic changes and slowdown of the evaluation process during the last few days significantly impacted the competition results and might ruin lots of efforts from all hard-working teams.\nAs others complained on the discussion board, many not-so-large submissions were failed due to time-out even their size are much smaller than previous succeeded ones. Besides, lots of previous succeeded submissions failed to the new evaluations.\n\nHere are some records from our side:\n\nAfter the first re-evaluation finished (around Jun 10, 8:00am UTC time), our team had 88 valid entries with the best score of 0.23246.![Jun_10_8am_UTC.png][1]\n\nWithout new submissions, a few hours later (around Jun 10, 4:00pm UTC time), we had fewer valid entries but higher scores (86 valid entries, best score of 0.23317). We believe that's from another re-evaluation.![Jun_10_4pm_UTC.png][2]\n\nAnother few hours later (around Jul 11, 3:00am UTC time), we had even fewer valid entries (only 77 entries) and all our recent submissions within 3 days got an error.![Jun_11_3am_UTC.png][3]\n\nWe believe all the errors are from time-out of the new evaluation code. Since our team, and possibly many other teams as well, optimized the submission file sizes and the number of instances based on the evaluation time of submissions, it is unfair and unreasonable to have such huge changes in a short period before the deadline of the competitions, especially without any anticipation or notification beforehand.\n\nWe sincerely hope the organizers to announce necessary details about the changes in modifications to help all teams have better estimations on valid file size/ number of instances, and/or check and speed-up the evaluation code, and/or increase the evaluation time limit. We truly expect and believe the organizers will provide reasonable solutions or explanations.\n\n\n  [1]: https://image.ibb.co/cLcCk8/Jun_10_8am_UTC.png\n  [2]: https://preview.ibb.co/ksFZXo/Jun_10_4pm_UTC.png\n  [3]: https://preview.ibb.co/djF1Co/Jun_11_3am_UTC.png",
    "341273": "Hey, SaLaHaeiYou,\n    Not until your post did I realize that our team's valid submission has been flagged as invalid because of the time out error.  And we have totally not being properly informed! That hasty change indeed causes the goal of this challenged deviated drastically. Dismay is what I am feeling now (I was hoping to push the performance during the last hours, now the change of the rules almost negates that)..."
  },
  "source": "meta"
}