Hi all ,
\nAs this competition is coming to an end , I had few concerns regarding the distribution-shift between train and test . Just like everyone else , we have also done adverserial validation and have gotten an auc score of over 0.98 . So clearly there is a distribution shift when we go from train to test , we have tried to remove train examples which seem to be further apart from test in the hopes of closing cv-lb gap but to no success .
Now as the host have mentioned here that they have hidden completely new signals in the test set , now how many of them are in private and how good our models are on them will be deciding factor in the mild shakeup that is coming .
\nWe have tried some major things in order to flag the new distribution in the test but have been unsuccessful , I wonder what the other teams are doing to tackle the distribution shift , also it will be fun to read psi's team solution after all this is over π
", "messages": [ { "id": 1477122, "postDate": "2021-08-17T11:05:56.770Z", "content": "Hi all ,
\nAs this competition is coming to an end , I had few concerns regarding the distribution-shift between train and test . Just like everyone else , we have also done adverserial validation and have gotten an auc score of over 0.98 . So clearly there is a distribution shift when we go from train to test , we have tried to remove train examples which seem to be further apart from test in the hopes of closing cv-lb gap but to no success .
Now as the host have mentioned here that they have hidden completely new signals in the test set , now how many of them are in private and how good our models are on them will be deciding factor in the mild shakeup that is coming .
\nWe have tried some major things in order to flag the new distribution in the test but have been unsuccessful , I wonder what the other teams are doing to tackle the distribution shift , also it will be fun to read psi's team solution after all this is over π
", "rawMarkdown": "Hi all , \nAs this competition is coming to an end , I had few concerns regarding the distribution-shift between train and test . Just like everyone else , we have also done adverserial validation and have gotten an auc score of over 0.98 . So clearly there is a distribution shift when we go from train to test , we have tried to remove train examples which seem to be further apart from test in the hopes of closing cv-lb gap but to no success . \n\nNow as the host have mentioned [here](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/256043) that they have hidden completely new signals in the test set , now how many of them are in private and how good our models are on them will be deciding factor in the mild shakeup that is coming .\n\nWe have tried some major things in order to flag the new distribution in the test but have been unsuccessful , I wonder what the other teams are doing to tackle the distribution shift , also it will be fun to read psi's team solution after all this is over π", "votes": 16 }, { "id": 1479722, "postDate": "2021-08-18T15:50:14.830Z", "content": "I've lied down, looking forward 0.853 solutions.
", "rawMarkdown": "I've lied down, looking forward 0.853 solutions.", "votes": 1, "replies": [ { "id": 1479843, "postDate": "2021-08-18T17:04:22.690Z", "content": "more to the 0.965 now π―
", "rawMarkdown": "more to the 0.965 now π―", "votes": 1 }, { "id": 1480153, "postDate": "2021-08-18T20:28:52.087Z", "content": "That's crazyβ¦
", "rawMarkdown": "That's crazy..." } ] }, { "id": 1477815, "postDate": "2021-08-17T16:47:44.903Z", "content": "Agreed @tanulsingh077 I am excited to read the top solutions when the comp ends
", "rawMarkdown": "Agreed @tanulsingh077 I am excited to read the top solutions when the comp ends", "votes": 1 }, { "id": 1477235, "postDate": "2021-08-17T12:05:23.027Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1479722, "author_name": "Last Scene", "author_url": "", "post_date": "2021-08-18T15:50:14.830000", "content": "I've lied down, looking forward 0.853 solutions.
", "votes": 1, "replies": [ { "id": 1479843, "author_name": "yukiya", "author_url": "", "post_date": "2021-08-18T17:04:22.690000", "content": "more to the 0.965 now π―
", "votes": 1, "replies": [] }, { "id": 1480153, "author_name": "Last Scene", "author_url": "", "post_date": "2021-08-18T20:28:52.087000", "content": "That's crazyβ¦
", "votes": 0, "replies": [] } ] }, { "id": 1477815, "author_name": "Daniel Furman", "author_url": "", "post_date": "2021-08-17T16:47:44.903000", "content": "Agreed @tanulsingh077 I am excited to read the top solutions when the comp ends
", "votes": 1, "replies": [] }, { "id": 1477235, "author_name": "", "author_url": "", "post_date": "2021-08-17T12:05:23.027000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1477122": "Hi all , \nAs this competition is coming to an end , I had few concerns regarding the distribution-shift between train and test . Just like everyone else , we have also done adverserial validation and have gotten an auc score of over 0.98 . So clearly there is a distribution shift when we go from train to test , we have tried to remove train examples which seem to be further apart from test in the hopes of closing cv-lb gap but to no success . \n\nNow as the host have mentioned [here](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/256043) that they have hidden completely new signals in the test set , now how many of them are in private and how good our models are on them will be deciding factor in the mild shakeup that is coming .\n\nWe have tried some major things in order to flag the new distribution in the test but have been unsuccessful , I wonder what the other teams are doing to tackle the distribution shift , also it will be fun to read psi's team solution after all this is over π", "1479722": "I've lied down, looking forward 0.853 solutions.", "1477815": "Agreed @tanulsingh077 I am excited to read the top solutions when the comp ends", "1477235": "" } }