Following the recent announcements from the competition organizers I am really thinking whether I should continue and submit stage 2 predictions. After distinctly specifying that labels of stage 1 test set will not be released and our models cannot change once we upload them, now all of a sudden the organizers are pointing us to a set of rules that somehow went missing from the site. Even if those rules had been present, a distinct clarification from the organizers about the current challenge obviously supersedes any generic non-competition specific rules.
\n\nAt the moment what our team uploaded doesn't retrain our model since that was what was required by the organizers. Had Julia said something else we would have submitted a completely different script which continues to train our models for a few more epochs on the released labels.
\n\nAfter investing almost two months of very hard and intensive work. The current way the organizers are treating the competition is very frustrating.
\n\nAs it stands our team placed fairly low in the ranking so I don't know if we had a realistic chance even if we could resubmit our script. I can't imagine what the top places of the LB are thinking after following the rules and making a submission that can now drop them much further than where they should be just because they were following the rules.
\n\nFor myself, I'm consulting with my team members, but my current state of mind is not to submit our predictions at all.
\n\nI would really appreciate the organizers response and hopefully a solution that will allow this competition to close with a more positive note.
", "messages": [ { "id": 410322, "postDate": "2018-10-25T20:26:03.213Z", "content": "Following the recent announcements from the competition organizers I am really thinking whether I should continue and submit stage 2 predictions. After distinctly specifying that labels of stage 1 test set will not be released and our models cannot change once we upload them, now all of a sudden the organizers are pointing us to a set of rules that somehow went missing from the site. Even if those rules had been present, a distinct clarification from the organizers about the current challenge obviously supersedes any generic non-competition specific rules.
\n\nAt the moment what our team uploaded doesn't retrain our model since that was what was required by the organizers. Had Julia said something else we would have submitted a completely different script which continues to train our models for a few more epochs on the released labels.
\n\nAfter investing almost two months of very hard and intensive work. The current way the organizers are treating the competition is very frustrating.
\n\nAs it stands our team placed fairly low in the ranking so I don't know if we had a realistic chance even if we could resubmit our script. I can't imagine what the top places of the LB are thinking after following the rules and making a submission that can now drop them much further than where they should be just because they were following the rules.
\n\nFor myself, I'm consulting with my team members, but my current state of mind is not to submit our predictions at all.
\n\nI would really appreciate the organizers response and hopefully a solution that will allow this competition to close with a more positive note.
", "rawMarkdown": "Following the recent announcements from the competition organizers I am really thinking whether I should continue and submit stage 2 predictions. After distinctly specifying that labels of stage 1 test set will not be released and our models cannot change once we upload them, now all of a sudden the organizers are pointing us to a set of rules that somehow went missing from the site. Even if those rules had been present, a distinct clarification from the organizers about the current challenge obviously supersedes any generic non-competition specific rules.\n\nAt the moment what our team uploaded doesn't retrain our model since that was what was required by the organizers. Had Julia said something else we would have submitted a completely different script which continues to train our models for a few more epochs on the released labels.\n\nAfter investing almost two months of very hard and intensive work. The current way the organizers are treating the competition is very frustrating.\n\nAs it stands our team placed fairly low in the ranking so I don't know if we had a realistic chance even if we could resubmit our script. I can't imagine what the top places of the LB are thinking after following the rules and making a submission that can now drop them much further than where they should be just because they were following the rules.\n\nFor myself, I'm consulting with my team members, but my current state of mind is not to submit our predictions at all.\n\nI would really appreciate the organizers response and hopefully a solution that will allow this competition to close with a more positive note.", "votes": 30 }, { "id": 411301, "postDate": "2018-10-27T20:07:49.547Z", "content": "It would be so much cleaner not to release the stage1 test labels and not to allow re-training models, otherwise it only encourages cheating.
\n\nAt least to keep the rules clearly stated ahead of competition, not clarified a few days before deadline.
", "rawMarkdown": "It would be so much cleaner not to release the stage1 test labels and not to allow re-training models, otherwise it only encourages cheating.\n\nAt least to keep the rules clearly stated ahead of competition, not clarified a few days before deadline.", "votes": 18 }, { "id": 410348, "postDate": "2018-10-25T22:07:17.217Z", "content": "I agree, this is a very frustrating experience and it is not being handled well at all. I, like you and many others, spent months working on this competition.
\n\nTo have no clarity until now - after the submission stage! - and even then through a link buried in a post that doesn't automatically notify competitors is absolutely insane.
\n\nMy understanding at submisson was that my model and its weights were to be frozen and used to predict the stage 2 test set. I did not set aside time to retrain using stage 1 labels this week (which can be huge when we're talking ensembles and object detection!), so feel like I will now be at a major disadvantage despite all my hard work.
", "rawMarkdown": "I agree, this is a very frustrating experience and it is not being handled well at all. I, like you and many others, spent months working on this competition. \n\nTo have no clarity until now - **after the submission stage!** - and even then through a **link buried in a post that doesn't automatically notify competitors** is absolutely insane.\n\nMy understanding at submisson was that my model and its weights were to be frozen and used to predict the stage 2 test set. I did not set aside time to retrain using stage 1 labels this week (which can be huge when we're talking ensembles and object detection!), so feel like I will now be at a major disadvantage despite all my hard work. ", "votes": 11, "replies": [ { "id": 410365, "postDate": "2018-10-25T23:22:39.823Z", "content": "Agreed. Both this and my previous experience with kaggle competition were marred by the host team unresponsiveness and cryptic messages. Several examples from this competition (other than the last few days):
\n\nNo official evaluation matrix
\n\nNo response about the training/test set problems
\n\nU turns (again, burried in posts) about the use of chexnet and nih data.
\n\nI love this site, that's why it frustrates me that much.
\n\nFor a start, important information should be sent to all teams, regardless of their subscriptions. Not by posts in the discussions. Also, some sort of stack overflow like system should be in place to see that no question to the hosts goes unanswered. I had two or 3 in this competition that were just left hanging.
\n\nJust my 2c. It's a free site, so i guess you get what you pay for, but if they wish to keep kagglers participating, they should up their game.
", "rawMarkdown": "Agreed. Both this and my previous experience with kaggle competition were marred by the host team unresponsiveness and cryptic messages. Several examples from this competition (other than the last few days):\n\nNo official evaluation matrix\n\nNo response about the training/test set problems\n\nU turns (again, burried in posts) about the use of chexnet and nih data.\n\nI love this site, that's why it frustrates me that much.\n\nFor a start, important information should be sent to all teams, regardless of their subscriptions. Not by posts in the discussions. Also, some sort of stack overflow like system should be in place to see that no question to the hosts goes unanswered. I had two or 3 in this competition that were just left hanging.\n\nJust my 2c. It's a free site, so i guess you get what you pay for, but if they wish to keep kagglers participating, they should up their game. ", "votes": 12 }, { "id": 410409, "postDate": "2018-10-26T02:08:17.123Z", "content": "@Meshul great points. Slack is a great platform to have Q&A regarding competition rules and/or addressing issues.
\n\n@ taindow You actually have two options. Rather its practical, that's up in the air. You can skip the training and just test against the new test data. The second option is to retrain and test. I know that some of the participants in DSB 2018 just tested versus restraining their models. Some had some good results.
\n\nHopefully the 3rd time is a charm and Kaggle get the verbage right.
", "rawMarkdown": "@Meshul great points. Slack is a great platform to have Q&A regarding competition rules and/or addressing issues. \n\n@ taindow You actually have two options. Rather its practical, that's up in the air. You can skip the training and just test against the new test data. The second option is to retrain and test. I know that some of the participants in DSB 2018 just tested versus restraining their models. Some had some good results. \n\nHopefully the 3rd time is a charm and Kaggle get the verbage right. ", "votes": 1 }, { "id": 410421, "postDate": "2018-10-26T03:11:32.893Z", "content": "Agree。Confusion now。TT
", "rawMarkdown": "Agree。Confusion now。TT", "votes": 2 } ] }, { "id": 410453, "postDate": "2018-10-26T05:00:58.937Z", "content": "There were similar issues in DSB 2018 competition. Not exactly the same, but issues due to some rule changes during competition. I don't have the details, but I am with you on the principle: rules should NOT change during a competition. Or, if they change, then the change must happen soon enough before end of stage 1 to have teams adapt to the new rules.
", "rawMarkdown": "There were similar issues in DSB 2018 competition. Not exactly the same, but issues due to some rule changes during competition. I don't have the details, but I am with you on the principle: rules should NOT change during a competition. Or, if they change, then the change must happen soon enough before end of stage 1 to have teams adapt to the new rules.", "votes": 10 }, { "id": 410442, "postDate": "2018-10-26T04:29:20.710Z", "content": "I agree and i should add, that together with the strange fact the the training set and the test set were labeled in different methods and hence have different statistics, something that goes against the basics of statistical machine learning, the release of stage 1 labels and the possibility to retrain is crucial and completely changes the game. The respectful organizers were not responsive at all to our questions regarding this issue, that's a pity.
", "rawMarkdown": "I agree and i should add, that together with the strange fact the the training set and the test set were labeled in different methods and hence have different statistics, something that goes against the basics of statistical machine learning, the release of stage 1 labels and the possibility to retrain is crucial and completely changes the game. The respectful organizers were not responsive at all to our questions regarding this issue, that's a pity.", "votes": 5, "replies": [ { "id": 410854, "postDate": "2018-10-26T19:03:03.333Z", "content": "Was it stated somewhere that the datasets were labeled with different methods?
", "rawMarkdown": "Was it stated somewhere that the datasets were labeled with different methods?" }, { "id": 410858, "postDate": "2018-10-26T19:13:30.073Z", "content": "https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/64723#380608
", "rawMarkdown": "https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/64723#380608" }, { "id": 411448, "postDate": "2018-10-28T07:00:16.510Z", "content": "Hi Fernando, yes it was stated: among ~25000 studies in stage 1 training set, only 1500 were triple labeled in the same way that stage 1 and stage 2 test sets were. The other 23500 were labeled by various radiologists, one per study. Moreover, see the comment of one of the organizers: https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/66323#391477
", "rawMarkdown": "Hi Fernando, yes it was stated: among ~25000 studies in stage 1 training set, only 1500 were triple labeled in the same way that stage 1 and stage 2 test sets were. The other 23500 were labeled by various radiologists, one per study. Moreover, see the comment of one of the organizers: https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/66323#391477", "votes": 1 } ] }, { "id": 411607, "postDate": "2018-10-28T14:55:45.183Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 411301, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2018-10-27T20:07:49.547000", "content": "It would be so much cleaner not to release the stage1 test labels and not to allow re-training models, otherwise it only encourages cheating.
\n\nAt least to keep the rules clearly stated ahead of competition, not clarified a few days before deadline.
", "votes": 18, "replies": [] }, { "id": 410348, "author_name": "Tom Aindow", "author_url": "", "post_date": "2018-10-25T22:07:17.217000", "content": "I agree, this is a very frustrating experience and it is not being handled well at all. I, like you and many others, spent months working on this competition.
\n\nTo have no clarity until now - after the submission stage! - and even then through a link buried in a post that doesn't automatically notify competitors is absolutely insane.
\n\nMy understanding at submisson was that my model and its weights were to be frozen and used to predict the stage 2 test set. I did not set aside time to retrain using stage 1 labels this week (which can be huge when we're talking ensembles and object detection!), so feel like I will now be at a major disadvantage despite all my hard work.
", "votes": 11, "replies": [ { "id": 410365, "author_name": "Moshel", "author_url": "", "post_date": "2018-10-25T23:22:39.823000", "content": "Agreed. Both this and my previous experience with kaggle competition were marred by the host team unresponsiveness and cryptic messages. Several examples from this competition (other than the last few days):
\n\nNo official evaluation matrix
\n\nNo response about the training/test set problems
\n\nU turns (again, burried in posts) about the use of chexnet and nih data.
\n\nI love this site, that's why it frustrates me that much.
\n\nFor a start, important information should be sent to all teams, regardless of their subscriptions. Not by posts in the discussions. Also, some sort of stack overflow like system should be in place to see that no question to the hosts goes unanswered. I had two or 3 in this competition that were just left hanging.
\n\nJust my 2c. It's a free site, so i guess you get what you pay for, but if they wish to keep kagglers participating, they should up their game.
", "votes": 12, "replies": [] }, { "id": 410409, "author_name": "William Green", "author_url": "", "post_date": "2018-10-26T02:08:17.123000", "content": "@Meshul great points. Slack is a great platform to have Q&A regarding competition rules and/or addressing issues.
\n\n@ taindow You actually have two options. Rather its practical, that's up in the air. You can skip the training and just test against the new test data. The second option is to retrain and test. I know that some of the participants in DSB 2018 just tested versus restraining their models. Some had some good results.
\n\nHopefully the 3rd time is a charm and Kaggle get the verbage right.
", "votes": 1, "replies": [] }, { "id": 410421, "author_name": "zzz333", "author_url": "", "post_date": "2018-10-26T03:11:32.893000", "content": "Agree。Confusion now。TT
", "votes": 2, "replies": [] } ] }, { "id": 410453, "author_name": "CPMP", "author_url": "", "post_date": "2018-10-26T05:00:58.937000", "content": "There were similar issues in DSB 2018 competition. Not exactly the same, but issues due to some rule changes during competition. I don't have the details, but I am with you on the principle: rules should NOT change during a competition. Or, if they change, then the change must happen soon enough before end of stage 1 to have teams adapt to the new rules.
", "votes": 10, "replies": [] }, { "id": 410442, "author_name": "Hadar", "author_url": "", "post_date": "2018-10-26T04:29:20.710000", "content": "I agree and i should add, that together with the strange fact the the training set and the test set were labeled in different methods and hence have different statistics, something that goes against the basics of statistical machine learning, the release of stage 1 labels and the possibility to retrain is crucial and completely changes the game. The respectful organizers were not responsive at all to our questions regarding this issue, that's a pity.
", "votes": 5, "replies": [ { "id": 410854, "author_name": "Fernando Camargo", "author_url": "", "post_date": "2018-10-26T19:03:03.333000", "content": "Was it stated somewhere that the datasets were labeled with different methods?
", "votes": 0, "replies": [] }, { "id": 410858, "author_name": "Fernando Camargo", "author_url": "", "post_date": "2018-10-26T19:13:30.073000", "content": "https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/64723#380608
", "votes": 0, "replies": [] }, { "id": 411448, "author_name": "Hadar", "author_url": "", "post_date": "2018-10-28T07:00:16.510000", "content": "Hi Fernando, yes it was stated: among ~25000 studies in stage 1 training set, only 1500 were triple labeled in the same way that stage 1 and stage 2 test sets were. The other 23500 were labeled by various radiologists, one per study. Moreover, see the comment of one of the organizers: https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/66323#391477
", "votes": 1, "replies": [] } ] }, { "id": 411607, "author_name": "", "author_url": "", "post_date": "2018-10-28T14:55:45.183000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "410322": "Following the recent announcements from the competition organizers I am really thinking whether I should continue and submit stage 2 predictions. After distinctly specifying that labels of stage 1 test set will not be released and our models cannot change once we upload them, now all of a sudden the organizers are pointing us to a set of rules that somehow went missing from the site. Even if those rules had been present, a distinct clarification from the organizers about the current challenge obviously supersedes any generic non-competition specific rules.\n\nAt the moment what our team uploaded doesn't retrain our model since that was what was required by the organizers. Had Julia said something else we would have submitted a completely different script which continues to train our models for a few more epochs on the released labels.\n\nAfter investing almost two months of very hard and intensive work. The current way the organizers are treating the competition is very frustrating.\n\nAs it stands our team placed fairly low in the ranking so I don't know if we had a realistic chance even if we could resubmit our script. I can't imagine what the top places of the LB are thinking after following the rules and making a submission that can now drop them much further than where they should be just because they were following the rules.\n\nFor myself, I'm consulting with my team members, but my current state of mind is not to submit our predictions at all.\n\nI would really appreciate the organizers response and hopefully a solution that will allow this competition to close with a more positive note.", "411301": "It would be so much cleaner not to release the stage1 test labels and not to allow re-training models, otherwise it only encourages cheating.\n\nAt least to keep the rules clearly stated ahead of competition, not clarified a few days before deadline.", "410348": "I agree, this is a very frustrating experience and it is not being handled well at all. I, like you and many others, spent months working on this competition. \n\nTo have no clarity until now - **after the submission stage!** - and even then through a **link buried in a post that doesn't automatically notify competitors** is absolutely insane.\n\nMy understanding at submisson was that my model and its weights were to be frozen and used to predict the stage 2 test set. I did not set aside time to retrain using stage 1 labels this week (which can be huge when we're talking ensembles and object detection!), so feel like I will now be at a major disadvantage despite all my hard work. ", "410453": "There were similar issues in DSB 2018 competition. Not exactly the same, but issues due to some rule changes during competition. I don't have the details, but I am with you on the principle: rules should NOT change during a competition. Or, if they change, then the change must happen soon enough before end of stage 1 to have teams adapt to the new rules.", "410442": "I agree and i should add, that together with the strange fact the the training set and the test set were labeled in different methods and hence have different statistics, something that goes against the basics of statistical machine learning, the release of stage 1 labels and the possibility to retrain is crucial and completely changes the game. The respectful organizers were not responsive at all to our questions regarding this issue, that's a pity.", "411607": "" } }