{
  "id": 74709,
  "title": "Speed Prize results",
  "url": "/competitions/airbus-ship-detection/discussion/74709",
  "author_name": "",
  "post_date": "2018-12-14T18:15:55.176703600Z",
  "votes": 120,
  "comment_count": 24,
  "views": 0,
  "content": "<p>Hi Kaggle!</p>\n\n<p>[attention heads] decided to express thoughts on the Speed Prize results and submission review process.</p>\n\n<p>First of all, we wanted to thank organizers <a href=\"/inversion\">@inversion</a> <a href=\"/jeffaudi\">@jeffaudi</a> for their efforts to make this competition as much fair and transparent as possible and also to congratulate top-scoring teams: <a href=\"/marvelousninja\">@marvelousninja</a>, <a href=\"/seesee\">@seesee</a>, you've done well! </p>\n\n<p>However, the current situation makes us feel disappointed. </p>\n\n<p><strong>What happened</strong> </p>\n\n<p>It seems that Speed Prize submissions were NOT reviewed properly. If they were, first place solution would have been rejected since it had incorrectly implemented time measurement (<a href=\"https://www.kaggle.com/c/airbus-ship-detection/discussion/74443#439029\">https://www.kaggle.com/c/airbus-ship-detection/discussion/74443#439029</a>). Additionally, our submission has been lost somehow (accepted now),  which also raises questions about the review process (<a href=\"https://www.kaggle.com/c/airbus-ship-detection/discussion/69558#437305\">https://www.kaggle.com/c/airbus-ship-detection/discussion/69558#437305</a>). </p>\n\n<p><strong>What we propose</strong></p>\n\n<p>Our point is that Speed Prize submissions have to be reviewed again, thoroughly and transparently. To be consistent with the rules, only submissions submitted <strong>before the deadline</strong> (November 30, 2018) should be taken into account. </p>\n\n<p>Hope we will be heard.</p>\n\n<p>Dmitriy Danevskyi,</p>\n\n<p>[attention heads]</p>",
  "messages": [
    {
      "id": "439090",
      "postDate": "12/14/2018 18:15:55",
      "content": "<p>Hi Kaggle!</p>\n\n<p>[attention heads] decided to express thoughts on the Speed Prize results and submission review process.</p>\n\n<p>First of all, we wanted to thank organizers <a href=\"/inversion\">@inversion</a> <a href=\"/jeffaudi\">@jeffaudi</a> for their efforts to make this competition as much fair and transparent as possible and also to congratulate top-scoring teams: <a href=\"/marvelousninja\">@marvelousninja</a>, <a href=\"/seesee\">@seesee</a>, you've done well! </p>\n\n<p>However, the current situation makes us feel disappointed. </p>\n\n<p><strong>What happened</strong> </p>\n\n<p>It seems that Speed Prize submissions were NOT reviewed properly. If they were, first place solution would have been rejected since it had incorrectly implemented time measurement (<a href=\"https://www.kaggle.com/c/airbus-ship-detection/discussion/74443#439029\">https://www.kaggle.com/c/airbus-ship-detection/discussion/74443#439029</a>). Additionally, our submission has been lost somehow (accepted now),  which also raises questions about the review process (<a href=\"https://www.kaggle.com/c/airbus-ship-detection/discussion/69558#437305\">https://www.kaggle.com/c/airbus-ship-detection/discussion/69558#437305</a>). </p>\n\n<p><strong>What we propose</strong></p>\n\n<p>Our point is that Speed Prize submissions have to be reviewed again, thoroughly and transparently. To be consistent with the rules, only submissions submitted <strong>before the deadline</strong> (November 30, 2018) should be taken into account. </p>\n\n<p>Hope we will be heard.</p>\n\n<p>Dmitriy Danevskyi,</p>\n\n<p>[attention heads]</p>",
      "rawMarkdown": "Hi Kaggle!\n\n[attention heads] decided to express thoughts on the Speed Prize results and submission review process.\n\nFirst of all, we wanted to thank organizers @inversion @jeffaudi for their efforts to make this competition as much fair and transparent as possible and also to congratulate top-scoring teams: @marvelousninja, @seesee, you've done well! \n\nHowever, the current situation makes us feel disappointed. \n\n**What happened** \n\nIt seems that Speed Prize submissions were NOT reviewed properly. If they were, first place solution would have been rejected since it had incorrectly implemented time measurement (https://www.kaggle.com/c/airbus-ship-detection/discussion/74443#439029). Additionally, our submission has been lost somehow (accepted now),  which also raises questions about the review process (https://www.kaggle.com/c/airbus-ship-detection/discussion/69558#437305). \n\n**What we propose**\n\nOur point is that Speed Prize submissions have to be reviewed again, thoroughly and transparently. To be consistent with the rules, only submissions submitted **before the deadline** (November 30, 2018) should be taken into account. \n\nHope we will be heard.\n\nDmitriy Danevskyi,\n\n[attention heads]",
      "votes": null
    },
    {
      "id": "439101",
      "postDate": "12/14/2018 18:31:02",
      "content": "<p><strong>The only proper way to amend the situation is to do the following:</strong>\n- Dedicated local hardware, each contestant submits his own Dockerfile;\n- Inference time is measured together with IO. End-to-end. Just one <code>sh</code> script is evoked inside of the container;\n- Give all the contestants one final submission for their best / fastest model;</p>\n\n<p>Yeah, and with proper hardware and correct timing, I believe I can easily win, because my concurrency wrappers keep the GPUs loaded 100% of the time xD</p>",
      "rawMarkdown": "**The only proper way to amend the situation is to do the following:**\n- Dedicated local hardware, each contestant submits his own Dockerfile;\n- Inference time is measured together with IO. End-to-end. Just one `sh` script is evoked inside of the container;\n- Give all the contestants one final submission for their best / fastest model;\n\nYeah, and with proper hardware and correct timing, I believe I can easily win, because my concurrency wrappers keep the GPUs loaded 100% of the time xD",
      "votes": null
    },
    {
      "id": "439105",
      "postDate": "12/14/2018 18:37:44",
      "content": "<p>As for hardware, any modern machine with 1080 - 1080Ti - 2080 / 16-32GB or RAM / 8-12 core CPU will suffice.</p>",
      "rawMarkdown": "As for hardware, any modern machine with 1080 - 1080Ti - 2080 / 16-32GB or RAM / 8-12 core CPU will suffice.",
      "votes": null
    },
    {
      "id": "439107",
      "postDate": "12/14/2018 18:41:49",
      "content": "<p>I think that giving \"one final submission\" to each contestant is not a good idea given the fact the most of the top scoring solutions are publicly available with all the tricks and hints used. This would negate all the hard work done by those contestants to achieve their results.</p>",
      "rawMarkdown": "I think that giving \"one final submission\" to each contestant is not a good idea given the fact the most of the top scoring solutions are publicly available with all the tricks and hints used. This would negate all the hard work done by those contestants to achieve their results.",
      "votes": null
    },
    {
      "id": "439108",
      "postDate": "12/14/2018 18:45:30",
      "content": "<p>Given that there has been no proper code review by the challenge hosts ... it hardly matters</p>",
      "rawMarkdown": "Given that there has been no proper code review by the challenge hosts ... it hardly matters",
      "votes": null
    },
    {
      "id": "439383",
      "postDate": "12/15/2018 10:03:40",
      "content": "<p>Very shit situation, my opinion about kaggle just went down hardly.</p>",
      "rawMarkdown": "Very shit situation, my opinion about kaggle just went down hardly.",
      "votes": null
    },
    {
      "id": "439461",
      "postDate": "12/15/2018 14:14:02",
      "content": "<p>Kaggle play fair!</p>",
      "rawMarkdown": "Kaggle play fair!",
      "votes": null
    },
    {
      "id": "439464",
      "postDate": "12/15/2018 14:25:36",
      "content": "<p>This is not acceptable, Kaggle should make sure results are reviewed properly</p>",
      "rawMarkdown": "This is not acceptable, Kaggle should make sure results are reviewed properly",
      "votes": null
    },
    {
      "id": "439571",
      "postDate": "12/15/2018 20:16:59",
      "content": "<p>Kaggle, you should play fair and transparent!</p>",
      "rawMarkdown": "Kaggle, you should play fair and transparent!",
      "votes": null
    },
    {
      "id": "439596",
      "postDate": "12/15/2018 21:56:10",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "439597",
      "postDate": "12/15/2018 22:06:42",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "439724",
      "postDate": "12/16/2018 07:58:28",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "440040",
      "postDate": "12/16/2018 23:43:56",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "440204",
      "postDate": "12/17/2018 08:01:48",
      "content": "<p>Hi guys, first of all you did an amazing job, my congrats!</p>\n\n<p>I really hope Kaggle and especially Airbus will pay attention to this serious issue. One of the main reason why Kaggle is (was?) a great platform for data science competitions is the <em>transparency</em> of the evaluation. Unlike regular prizes, Speed Prize from Airbus Ship Detection is kind of a new one, so there is no surprise of having troubles with this very last part. However, it's not the case when hosts should turn a blind eye to participants' questions. There have been already similar disputes about prize eligibility and all of them were resolved, and this one is not an exception!</p>\n\n<p>Good luck Dmitriy and [attention heads]!</p>",
      "rawMarkdown": "Hi guys, first of all you did an amazing job, my congrats!\n\nI really hope Kaggle and especially Airbus will pay attention to this serious issue. One of the main reason why Kaggle is (was?) a great platform for data science competitions is the *transparency* of the evaluation. Unlike regular prizes, Speed Prize from Airbus Ship Detection is kind of a new one, so there is no surprise of having troubles with this very last part. However, it's not the case when hosts should turn a blind eye to participants' questions. There have been already similar disputes about prize eligibility and all of them were resolved, and this one is not an exception!\n\nGood luck Dmitriy and [attention heads]!",
      "votes": null
    },
    {
      "id": "440250",
      "postDate": "12/17/2018 09:39:32",
      "content": "<p>Thanks a lot for your support!</p>",
      "rawMarkdown": "Thanks a lot for your support!",
      "votes": null
    },
    {
      "id": "440327",
      "postDate": "12/17/2018 12:06:53",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "440594",
      "postDate": "12/17/2018 18:39:55",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "440615",
      "postDate": "12/17/2018 19:34:19",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "441006",
      "postDate": "12/18/2018 07:13:17",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "441113",
      "postDate": "12/18/2018 09:22:53",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "441435",
      "postDate": "12/18/2018 16:53:45",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "441671",
      "postDate": "12/18/2018 23:08:18",
      "content": "<p>Hi to everyone,</p>\n\n<p>Kaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors. We praise sharing and transparency but we also know that rules need to be applied in fairness. We have heard the community comments and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. </p>\n\n<p>Kind regards, \nJeff.</p>",
      "rawMarkdown": "Hi to everyone,\n\nKaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors. We praise sharing and transparency but we also know that rules need to be applied in fairness. We have heard the community comments and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. \n\nKind regards, \nJeff.",
      "votes": null
    },
    {
      "id": "457544",
      "postDate": "01/17/2019 17:00:21",
      "content": "<p>Can you share your code? @Dmitriy Danevskiy</p>",
      "rawMarkdown": "Can you share your code? @Dmitriy Danevskiy",
      "votes": null
    },
    {
      "id": "494142",
      "postDate": "03/19/2019 14:02:37",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "524406",
      "postDate": "04/28/2019 17:42:13",
      "content": "<p>To be fair, divide the prize between these two winning teams</p>",
      "rawMarkdown": "To be fair, divide the prize between these two winning teams",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 439101,
      "author_name": "snakers41",
      "author_url": "",
      "post_date": "12/14/2018 18:31:02",
      "content": "<p><strong>The only proper way to amend the situation is to do the following:</strong>\n- Dedicated local hardware, each contestant submits his own Dockerfile;\n- Inference time is measured together with IO. End-to-end. Just one <code>sh</code> script is evoked inside of the container;\n- Give all the contestants one final submission for their best / fastest model;</p>\n\n<p>Yeah, and with proper hardware and correct timing, I believe I can easily win, because my concurrency wrappers keep the GPUs loaded 100% of the time xD</p>",
      "votes": null,
      "replies": [
        {
          "id": 439107,
          "author_name": "vshmyhlo",
          "author_url": "",
          "post_date": "12/14/2018 18:41:49",
          "content": "<p>I think that giving \"one final submission\" to each contestant is not a good idea given the fact the most of the top scoring solutions are publicly available with all the tricks and hints used. This would negate all the hard work done by those contestants to achieve their results.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 439108,
          "author_name": "snakers41",
          "author_url": "",
          "post_date": "12/14/2018 18:45:30",
          "content": "<p>Given that there has been no proper code review by the challenge hosts ... it hardly matters</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 439105,
      "author_name": "snakers41",
      "author_url": "",
      "post_date": "12/14/2018 18:37:44",
      "content": "<p>As for hardware, any modern machine with 1080 - 1080Ti - 2080 / 16-32GB or RAM / 8-12 core CPU will suffice.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439383,
      "author_name": "anikishaev",
      "author_url": "",
      "post_date": "12/15/2018 10:03:40",
      "content": "<p>Very shit situation, my opinion about kaggle just went down hardly.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439461,
      "author_name": "azakharko",
      "author_url": "",
      "post_date": "12/15/2018 14:14:02",
      "content": "<p>Kaggle play fair!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439464,
      "author_name": "sergeyk0",
      "author_url": "",
      "post_date": "12/15/2018 14:25:36",
      "content": "<p>This is not acceptable, Kaggle should make sure results are reviewed properly</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439571,
      "author_name": "zverrr",
      "author_url": "",
      "post_date": "12/15/2018 20:16:59",
      "content": "<p>Kaggle, you should play fair and transparent!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439596,
      "author_name": "guzenko",
      "author_url": "",
      "post_date": "12/15/2018 21:56:10",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439597,
      "author_name": "ib151207",
      "author_url": "",
      "post_date": "12/15/2018 22:06:42",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439724,
      "author_name": "isawelephant",
      "author_url": "",
      "post_date": "12/16/2018 07:58:28",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 440040,
      "author_name": "kesjien",
      "author_url": "",
      "post_date": "12/16/2018 23:43:56",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 440204,
      "author_name": "merryhunter",
      "author_url": "",
      "post_date": "12/17/2018 08:01:48",
      "content": "<p>Hi guys, first of all you did an amazing job, my congrats!</p>\n\n<p>I really hope Kaggle and especially Airbus will pay attention to this serious issue. One of the main reason why Kaggle is (was?) a great platform for data science competitions is the <em>transparency</em> of the evaluation. Unlike regular prizes, Speed Prize from Airbus Ship Detection is kind of a new one, so there is no surprise of having troubles with this very last part. However, it's not the case when hosts should turn a blind eye to participants' questions. There have been already similar disputes about prize eligibility and all of them were resolved, and this one is not an exception!</p>\n\n<p>Good luck Dmitriy and [attention heads]!</p>",
      "votes": null,
      "replies": [
        {
          "id": 440250,
          "author_name": "ddanevskyi",
          "author_url": "",
          "post_date": "12/17/2018 09:39:32",
          "content": "<p>Thanks a lot for your support!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 440327,
      "author_name": "maksymx",
      "author_url": "",
      "post_date": "12/17/2018 12:06:53",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 440594,
      "author_name": "dashok",
      "author_url": "",
      "post_date": "12/17/2018 18:39:55",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 440615,
      "author_name": "olenapo",
      "author_url": "",
      "post_date": "12/17/2018 19:34:19",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 441006,
      "author_name": "pavelnagorny",
      "author_url": "",
      "post_date": "12/18/2018 07:13:17",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 441113,
      "author_name": "mykolak",
      "author_url": "",
      "post_date": "12/18/2018 09:22:53",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 441435,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "12/18/2018 16:53:45",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 441671,
      "author_name": "jeffaudi",
      "author_url": "",
      "post_date": "12/18/2018 23:08:18",
      "content": "<p>Hi to everyone,</p>\n\n<p>Kaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors. We praise sharing and transparency but we also know that rules need to be applied in fairness. We have heard the community comments and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. </p>\n\n<p>Kind regards, \nJeff.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 457544,
      "author_name": "xiaojidan",
      "author_url": "",
      "post_date": "01/17/2019 17:00:21",
      "content": "<p>Can you share your code? @Dmitriy Danevskiy</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 494142,
      "author_name": "kukoban",
      "author_url": "",
      "post_date": "03/19/2019 14:02:37",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 524406,
      "author_name": "alexworkplace",
      "author_url": "",
      "post_date": "04/28/2019 17:42:13",
      "content": "<p>To be fair, divide the prize between these two winning teams</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "439090": "Hi Kaggle!\n\n[attention heads] decided to express thoughts on the Speed Prize results and submission review process.\n\nFirst of all, we wanted to thank organizers @inversion @jeffaudi for their efforts to make this competition as much fair and transparent as possible and also to congratulate top-scoring teams: @marvelousninja, @seesee, you've done well! \n\nHowever, the current situation makes us feel disappointed. \n\n**What happened** \n\nIt seems that Speed Prize submissions were NOT reviewed properly. If they were, first place solution would have been rejected since it had incorrectly implemented time measurement (https://www.kaggle.com/c/airbus-ship-detection/discussion/74443#439029). Additionally, our submission has been lost somehow (accepted now),  which also raises questions about the review process (https://www.kaggle.com/c/airbus-ship-detection/discussion/69558#437305). \n\n**What we propose**\n\nOur point is that Speed Prize submissions have to be reviewed again, thoroughly and transparently. To be consistent with the rules, only submissions submitted **before the deadline** (November 30, 2018) should be taken into account. \n\nHope we will be heard.\n\nDmitriy Danevskyi,\n\n[attention heads]",
    "439101": "**The only proper way to amend the situation is to do the following:**\n- Dedicated local hardware, each contestant submits his own Dockerfile;\n- Inference time is measured together with IO. End-to-end. Just one `sh` script is evoked inside of the container;\n- Give all the contestants one final submission for their best / fastest model;\n\nYeah, and with proper hardware and correct timing, I believe I can easily win, because my concurrency wrappers keep the GPUs loaded 100% of the time xD",
    "439105": "As for hardware, any modern machine with 1080 - 1080Ti - 2080 / 16-32GB or RAM / 8-12 core CPU will suffice.",
    "439107": "I think that giving \"one final submission\" to each contestant is not a good idea given the fact the most of the top scoring solutions are publicly available with all the tricks and hints used. This would negate all the hard work done by those contestants to achieve their results.",
    "439108": "Given that there has been no proper code review by the challenge hosts ... it hardly matters",
    "439383": "Very shit situation, my opinion about kaggle just went down hardly.",
    "439461": "Kaggle play fair!",
    "439464": "This is not acceptable, Kaggle should make sure results are reviewed properly",
    "439571": "Kaggle, you should play fair and transparent!",
    "439596": "1",
    "439597": "1",
    "439724": "1",
    "440040": "1",
    "440204": "Hi guys, first of all you did an amazing job, my congrats!\n\nI really hope Kaggle and especially Airbus will pay attention to this serious issue. One of the main reason why Kaggle is (was?) a great platform for data science competitions is the *transparency* of the evaluation. Unlike regular prizes, Speed Prize from Airbus Ship Detection is kind of a new one, so there is no surprise of having troubles with this very last part. However, it's not the case when hosts should turn a blind eye to participants' questions. There have been already similar disputes about prize eligibility and all of them were resolved, and this one is not an exception!\n\nGood luck Dmitriy and [attention heads]!",
    "440250": "Thanks a lot for your support!",
    "440327": "1",
    "440594": "1",
    "440615": "1",
    "441006": "1",
    "441113": "1",
    "441435": "1",
    "441671": "Hi to everyone,\n\nKaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors. We praise sharing and transparency but we also know that rules need to be applied in fairness. We have heard the community comments and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. \n\nKind regards, \nJeff.",
    "457544": "Can you share your code? @Dmitriy Danevskiy",
    "494142": "1",
    "524406": "To be fair, divide the prize between these two winning teams"
  },
  "source": "meta"
}