![\"t-l-chargement\"](\"https://i.ibb.co/2Nzdj5Y/t-l-chargement.png\")
Provided the fact that there are ~310 samples, it's fairly easy to compare how the LB looks compared to random submissions. Spoiler, they look the same.
\n
In this run, randomness is better than the private LB, but it's not generally the case.
\nFor instance, a 0.621 LB is top 0.02% over 100k runs - and scores higher than 0.65 can be reached …
The code looks like this, in particular one can increase the number of iterations for a more reliable distribution :
\nN = 310\n# N = 582\n# N = 87\naucs = []\n\nprop = 0.525 # same as train ?\nn_pos = int(N * prop)\ny = np.concatenate((np.zeros(N - n_pos), np.ones(n_pos)), 0)\n\nfor _ in range(1555):\n pred = np.random.random(N)\n auc = roc_auc_score(y, pred)\n pred = np.random.random(N)\n auc2 = roc_auc_score(y, pred)\n\n aucs.append(max(auc, auc2))\n\nref = 0.621\nsns.displot(aucs)\nplt.axvline(ref, c='salmon')\nplt.title(f'AUC {ref} : Top {np.mean(np.array(aucs) > ref) * 100 :.2f}%')\nplt.show()\nMy advice to the hosts would be the following :
\nTrain a baseline to see if there is signal behind the data ! Noticing that the models couldn't learn anything would've help better frame the problem.
\nPS : if someone plotted the actual private LB distribution plase share it :)
", "messages": [ { "id": 1549746, "postDate": "2021-10-19T07:24:59.647Z", "content": "Provided the fact that there are ~310 samples, it's fairly easy to compare how the LB looks compared to random submissions. Spoiler, they look the same.
\n
In this run, randomness is better than the private LB, but it's not generally the case.
\nFor instance, a 0.621 LB is top 0.02% over 100k runs - and scores higher than 0.65 can be reached …
The code looks like this, in particular one can increase the number of iterations for a more reliable distribution :
\nN = 310\n# N = 582\n# N = 87\naucs = []\n\nprop = 0.525 # same as train ?\nn_pos = int(N * prop)\ny = np.concatenate((np.zeros(N - n_pos), np.ones(n_pos)), 0)\n\nfor _ in range(1555):\n pred = np.random.random(N)\n auc = roc_auc_score(y, pred)\n pred = np.random.random(N)\n auc2 = roc_auc_score(y, pred)\n\n aucs.append(max(auc, auc2))\n\nref = 0.621\nsns.displot(aucs)\nplt.axvline(ref, c='salmon')\nplt.title(f'AUC {ref} : Top {np.mean(np.array(aucs) > ref) * 100 :.2f}%')\nplt.show()\nMy advice to the hosts would be the following :
\nTrain a baseline to see if there is signal behind the data ! Noticing that the models couldn't learn anything would've help better frame the problem.
\nPS : if someone plotted the actual private LB distribution plase share it :)
", "rawMarkdown": "Provided the fact that there are ~310 samples, it's fairly easy to compare how the LB looks compared to random submissions. Spoiler, they look the same.\n\n\n\nIn this run, randomness is better than the private LB, but it's not generally the case.\nFor instance, a 0.621 LB is top 0.02% over 100k runs - and scores higher than 0.65 can be reached ...\n\nThe code looks like this, in particular one can increase the number of iterations for a more reliable distribution :\n```\nN = 310\n# N = 582\n# N = 87\naucs = []\n\nprop = 0.525 # same as train ?\nn_pos = int(N * prop)\ny = np.concatenate((np.zeros(N - n_pos), np.ones(n_pos)), 0)\n\nfor _ in range(1555):\n pred = np.random.random(N)\n auc = roc_auc_score(y, pred)\n pred = np.random.random(N)\n auc2 = roc_auc_score(y, pred)\n \n aucs.append(max(auc, auc2))\n\nref = 0.621\nsns.displot(aucs)\nplt.axvline(ref, c='salmon')\nplt.title(f'AUC {ref} : Top {np.mean(np.array(aucs) > ref) * 100 :.2f}%')\nplt.show()\n``` \n\nMy advice to the hosts would be the following :\n\nTrain a baseline to see if there is signal behind the data ! Noticing that the models couldn't learn anything would've help better frame the problem.\n\n\nPS : if someone plotted the actual private LB distribution plase share it :)",
"votes": 84
},
{
"id": 1549815,
"postDate": "2021-10-19T08:06:06.680Z",
"content": "What I learned is that it's better to avoid competitions like this one
", "rawMarkdown": "What I learned is that it's better to avoid competitions like this one", "votes": 19, "replies": [ { "id": 1550672, "postDate": "2021-10-19T22:33:34.663Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 1549837, "postDate": "2021-10-19T08:19:27.467Z", "content": "It looks like I've made the right choice to focus on the Ventilator Pressure Prediction competition instead of this one.
\nYet, it feels pretty inspiring and self-explanatory that the 1st place team was named \"I hate this competition\" before the private LB was revealed 😊
Lol they even changed their name to \"I love this competition\" once private LB was revealed!
", "rawMarkdown": "Lol they even changed their name to \"I love this competition\" once private LB was revealed!", "votes": 9 } ] }, { "id": 1551624, "postDate": "2021-10-20T18:51:54.733Z", "content": "I have rarely seen such a lottery competition:
\nA gold medal and a prize with 1 single submission 2 months ago : that is the best time / money investment I have ever seen !
", "rawMarkdown": "I have rarely seen such a lottery competition:\n\nA gold medal and a prize with 1 single submission 2 months ago : that is the best time / money investment I have ever seen !\n\n", "votes": 9 }, { "id": 1550361, "postDate": "2021-10-19T16:37:22.283Z", "content": "Actual Private LB Distribution @theoviel
\n![\"\"](\"https://i.imgur.com/FkYp8SD.png\")
The mode is very interesting here, basically most submissions turned out to be as good as random guessing like a coin toss.
\nCurious to know what the mean and median are from this chart, I would think mean is very close to 0.5 as well since the even though it is left skewed, the mode of 0.5 would pull the mean to the left.
", "rawMarkdown": "The mode is very interesting here, basically most submissions turned out to be as good as random guessing like a coin toss.\n\nCurious to know what the mean and median are from this chart, I would think mean is very close to 0.5 as well since the even though it is left skewed, the mode of 0.5 would pull the mean to the left.", "votes": 2 }, { "id": 1550426, "postDate": "2021-10-19T17:47:59.530Z", "content": "Thanks @shivamb !
\nI think the distribution looks slightly better than random - I guess most models still found some cases.
The mode at of 0.5 probably comes from people submitting the same value everywhere and shouldn't really be considered
", "rawMarkdown": "Thanks @shivamb !\nI think the distribution looks slightly better than random - I guess most models still found some cases. \n\nThe mode at of 0.5 probably comes from people submitting the same value everywhere and shouldn't really be considered ", "votes": 2 }, { "id": 1550671, "postDate": "2021-10-19T22:32:22.597Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1550679, "postDate": "2021-10-19T23:06:03.337Z", "content": "@yuzheni where do you see multimodality here? If you just exclude the mode it looks normal. I think the multimodality you are seeing is possibly by random sampling a normal distribution of this size.
", "rawMarkdown": "@yuzheni where do you see multimodality here? If you just exclude the mode it looks normal. I think the multimodality you are seeing is possibly by random sampling a normal distribution of this size." } ] }, { "id": 1549920, "postDate": "2021-10-19T09:35:53.913Z", "content": "From my perspective, it wasn't a failure because I learned a lot from this competition. Some people in the comments should be in this mindset because otherwise you will be disappointed a lot. I shake down and didn't select gold medal solutions multiple times in my previous competitions. I think all of them were great learning experiences and medals weren't my priority in any of them.
", "rawMarkdown": "From my perspective, it wasn't a failure because I learned a lot from this competition. Some people in the comments should be in this mindset because otherwise you will be disappointed a lot. I shake down and didn't select gold medal solutions multiple times in my previous competitions. I think all of them were great learning experiences and medals weren't my priority in any of them. ", "votes": 7, "replies": [ { "id": 1549945, "postDate": "2021-10-19T10:08:50.240Z", "content": "I wholeheartedly agree with @gunesevitan: The only way one can totally loose a competition is to treat the leaderboards as the sole metric of success!
\nAll the best,
\ncarl
What I meant about \"failure\" was more an overall conclusion. Sure some people learnt from it, some even got a medal.
\nBut none of the models produced are worth the 50k+ $ the hosts invested on the making of the competition.
\nNone of the models will help detect the mgmt gene, and thus help for the cure of brain tumours.
Which is unfortunate
", "rawMarkdown": "What I meant about \"failure\" was more an overall conclusion. Sure some people learnt from it, some even got a medal.\nBut none of the models produced are worth the 50k+ $ the hosts invested on the making of the competition.\nNone of the models will help detect the mgmt gene, and thus help for the cure of brain tumours.\n\nWhich is unfortunate", "votes": 20 }, { "id": 1549974, "postDate": "2021-10-19T10:29:28.903Z", "content": "Dear @theoviel
\nI see your point of view. However, I spent 20 years as an academic researcher, and the number of unsuccessful projects/ideas/experiments etc often out-way the successes, but at the same time unsuccessful results are the paving stones on the way to excellent results, and an insightful postmortem of an experiment (or in this case; a competition) can be almost as valuable as any other type of work.
\nThat said, it would obviously be wonderful if science had a shiny new way of helping to cure brain tumours…
\nAll the best,
\ncarl
Perhaps there is much more insight in the MRI scans than we think now. The problem of this competition was a very small dataset. When you look at any tips regarding the minimum dataset size for a successful Deep Learning model, you can find numbers like 100k, 10k, 1k, depending on the problem. We had less then 600 samples, from which some of them are used as validation data during training. I had never faced a problem like this before (to train a model with not sufficient amount of data) and I agree that this could be a valuable lesson. What I learned is that with not enough data it is extremely difficult to succeed. Now I have this feeling that trying to tackle such a problem with so little amount of data might indeed be an inevitable failure.
\nAnother thing is that seeing this crazy public LB filled with completely unrealistic scores was really daunting and if the rules say that LB probing is forbidden, it should be forbidden. I definitely consider this as a failure. I bet that many people gave up because of this.
What would happen if we had more data? Maybe this competition would become a success.
", "rawMarkdown": "Perhaps there is much more insight in the MRI scans than we think now. The problem of this competition was a very small dataset. When you look at any tips regarding the minimum dataset size for a successful Deep Learning model, you can find numbers like 100k, 10k, 1k, depending on the problem. We had less then 600 samples, from which some of them are used as validation data during training. I had never faced a problem like this before (to train a model with not sufficient amount of data) and I agree that this could be a valuable lesson. What I learned is that with not enough data it is extremely difficult to succeed. Now I have this feeling that trying to tackle such a problem with so little amount of data might indeed be an inevitable failure. \nAnother thing is that seeing this crazy public LB filled with completely unrealistic scores was really daunting and if the rules say that LB probing is forbidden, it should be forbidden. I definitely consider this as a failure. I bet that many people gave up because of this.\n\nWhat would happen if we had more data? Maybe this competition would become a success.", "votes": 6 }, { "id": 1550129, "postDate": "2021-10-19T13:12:27.883Z", "content": "Great comment.Can you elaborate The unrealistic value thing again as I haven't understood how public LB has the unrealistic values
", "rawMarkdown": "Great comment.Can you elaborate The unrealistic value thing again as I haven't understood how public LB has the unrealistic values" }, { "id": 1550150, "postDate": "2021-10-19T13:31:35.790Z", "content": "@adityasharma01 In the public LB during the time of the competition you can normally have some approximation of your performance, and performance of the others. This would be an ideal scenario and it is often far from that.
\nHowever, in this competition a lot of people had scores 1.0 or 75+ which deviates so much from the final standings that this public LB was more confusing than helping anyone. And there were 2 reasons behind it:
\n\nAnother downside of the small public dataset - it was easier to submit a completely random result and be lucky enough to have a high score on the LB. There were many submissions of models that didn't learn anything but by chance they reached auc over 0.73 (many public notebooks had this score). You can see now that these models did not learn anything. It was just a roulette.
\n
That's why I never read the notebooks section. I only read through discussions and write my own code from scratch.
", "rawMarkdown": "> Another downside of the small public dataset - it was easier to submit a completely random result and be lucky enough to have a high score on the LB. There were many submissions of models that didn't learn anything but by chance they reached auc over 0.73 (many public notebooks had this score). You can see now that these models did not learn anything. It was just a roulette.\n\nThat's why I never read the notebooks section. I only read through discussions and write my own code from scratch.", "votes": 4 }, { "id": 1550199, "postDate": "2021-10-19T14:07:36.323Z", "content": "Thank you for the explanation @mikecho Now I got your point and you are completely correct. My submission of 0.91 validation just got nothing value in private leader board. And of less validation got high ranks. Means if I forget the public leader board and just go on the performance of validation, then also the private leader board is not giving that good response.
", "rawMarkdown": "Thank you for the explanation @mikecho Now I got your point and you are completely correct. My submission of 0.91 validation just got nothing value in private leader board. And of less validation got high ranks. Means if I forget the public leader board and just go on the performance of validation, then also the private leader board is not giving that good response.", "votes": 1 }, { "id": 1550207, "postDate": "2021-10-19T14:15:51.447Z", "content": "Agree. And for people who claim to have great experience are because they got the medal😂
", "rawMarkdown": "Agree. And for people who claim to have great experience are because they got the medal😂", "votes": 1 }, { "id": 1550247, "postDate": "2021-10-19T14:53:55.247Z", "content": "Haha a bit cynical @Swikwislkdj but understandable 😄
", "rawMarkdown": "Haha a bit cynical @Swikwislkdj but understandable 😄 ", "votes": 2 } ] }, { "id": 1549817, "postDate": "2021-10-19T08:08:15.207Z", "content": "In public score, I get > 0.76, but in private, it get <0.5 when my low score model<0.6 get nearly 0.6 in private… So confused at first saw.
", "rawMarkdown": "In public score, I get > 0.76, but in private, it get <0.5 when my low score model<0.6 get nearly 0.6 in private... So confused at first saw.", "votes": 3, "replies": [ { "id": 1550146, "postDate": "2021-10-19T13:30:04.047Z", "content": "Same… do you know why?
", "rawMarkdown": "Same… do you know why?" }, { "id": 1550497, "postDate": "2021-10-19T19:22:13.533Z", "content": "Going back to the medical background, the meaning of this competition tells us using brain tumors to predict MGMT value is impossible. These two things are not related.
", "rawMarkdown": "Going back to the medical background, the meaning of this competition tells us using brain tumors to predict MGMT value is impossible. These two things are not related. ", "votes": 2 }, { "id": 1550515, "postDate": "2021-10-19T19:33:33.497Z", "content": "@together i disagree, this claim is simply not true.
\nThe right formulation would be: with the given number of inputs noone could find any image based biomarker (or feature) within this competition, that shows a higher correlation with the methylation status of the tumors of this training set.
\nThe glioma have much more genotypic parameters that determine their mr-morphological appearance and methylation is only one - probably - weak influencer of this phenotyp.
\nI firmly believe that if we were given other genotypic descriptors and probably some additional base sequences (like diffusion or maybe perfusion imaging), we would be able to find a higher correlating fature.
Agree with your explanation.
", "rawMarkdown": "Agree with your explanation. ", "votes": 1 }, { "id": 1551947, "postDate": "2021-10-21T03:11:27.607Z", "content": "I agree with your explanation, in the medical background, there are some problems. But in the competitor perspective, I think the public test make some competitors distracted to not use the complicated model because can easily be overfitting, while some people just tried only backbone and they quit, they also get the high score, you can see on the leaderboard, someone just joins and last submit was 1-2 months with a low score on public and they quit, however, they get the high score, I think the dataset change make some team fall 800-1k rank on the private board although they try hard for this competition.
", "rawMarkdown": "I agree with your explanation, in the medical background, there are some problems. But in the competitor perspective, I think the public test make some competitors distracted to not use the complicated model because can easily be overfitting, while some people just tried only backbone and they quit, they also get the high score, you can see on the leaderboard, someone just joins and last submit was 1-2 months with a low score on public and they quit, however, they get the high score, I think the dataset change make some team fall 800-1k rank on the private board although they try hard for this competition. ", "votes": 1 } ] }, { "id": 1550731, "postDate": "2021-10-20T01:11:45.690Z", "content": "I think we can not say the models of top rankers are chance level because the distribution of LB score is similar to that of random sampling. The distribution of IQ is completely Gaussian, but I do not think Albert Einstein is as foolish as me. In addition, from the plot by @shivamb, the AUC scores of top rankers seem stable around 0.6 for both public and private datasets, while I admit that AUC 0.6 is too low for diagnoses.
", "rawMarkdown": "I think we can not say the models of top rankers are chance level because the distribution of LB score is similar to that of random sampling. The distribution of [IQ](https://en.wikipedia.org/wiki/Intelligence_quotient) is completely Gaussian, but I do not think Albert Einstein is as foolish as me. In addition, [from the plot](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279777) by @shivamb, the AUC scores of top rankers seem stable around 0.6 for both public and private datasets, while I admit that AUC 0.6 is too low for diagnoses. ", "votes": 1, "replies": [ { "id": 1550779, "postDate": "2021-10-20T02:41:13.777Z", "content": "![\"\"](\"https://i.ibb.co/qFT17Ds/einsum.jpg\")
The difference is that given a different test dataset, the rank of the top models can be switched, while given a different IQ test, Albert Einstein would still score higher than most people. Of course not all top models are random, but the randomness of the top is a bit too high in this competition.
", "rawMarkdown": "The difference is that given a different test dataset, the rank of the top models can be switched, while given a different IQ test, Albert Einstein would still score higher than most people. Of course not all top models are random, but the randomness of the top is a bit too high in this competition.", "votes": 2 }, { "id": 1551891, "postDate": "2021-10-21T00:23:33.613Z", "content": "Yes, we should see the (in)stability of CV and public AUC scores and not the distribution of them. We knew that the public score and ranking are far from reliable for many reasons.
", "rawMarkdown": "Yes, we should see the (in)stability of CV and public AUC scores and not the distribution of them. We knew that the public score and ranking are far from reliable for many reasons." } ] }, { "id": 1549782, "postDate": "2021-10-19T07:44:41.190Z", "content": "Thank you for the analysis!
\nI felt the same as i saw the <0.5 portion… was actually expecting much more results in the >0.6 area to be convinced that there is usable info in the images for this particular prediction!
Good analysis. Better to avoid these types of competitions.
", "rawMarkdown": "Good analysis. Better to avoid these types of competitions.", "votes": 2 }, { "id": 1550063, "postDate": "2021-10-19T12:09:35.160Z", "content": "Well sorry, I didn't understand what you meant, can you guys please explain in an easier way?
", "rawMarkdown": "Well sorry, I didn't understand what you meant, can you guys please explain in an easier way?" }, { "id": 1553374, "postDate": "2021-10-22T06:07:08.067Z", "content": "can't agree more.
", "rawMarkdown": "can't agree more." }, { "id": 1553079, "postDate": "2021-10-21T21:55:58.970Z", "content": "Thank you for analysis. Better to avoid these types of competitions.
", "rawMarkdown": "Thank you for analysis. Better to avoid these types of competitions." }, { "id": 1552192, "postDate": "2021-10-21T08:29:09.210Z", "content": "Does this happen frequently over Kaggle?
", "rawMarkdown": "Does this happen frequently over Kaggle?", "replies": [ { "id": 1553054, "postDate": "2021-10-21T21:05:02.997Z", "content": "Yes, about 1 out of 10 or 15 competitions.
\nUsually the big concerns are leakage problems (an information in the data that is hidden and not supposed to be there and is discovered by the competitors: too high correlation between the positive samples and the actual length of the audio file / pictures metadata with the real labels). Leaks can make the competition unfair.
\nHere clearly this is the worse case: the test set is different from the training set (training on zebra pictures and testing on what is the current temperature of the room). The last ones I remembered with such a check up were the Mercedes and VSB power line competitions: they have been others recently.
\nFor Kaggle defense, they need to manage the competition sponsors regarding the data and the sponsors are not always listening…
", "rawMarkdown": "Yes, about 1 out of 10 or 15 competitions.\n\nUsually the big concerns are leakage problems (an information in the data that is hidden and not supposed to be there and is discovered by the competitors: too high correlation between the positive samples and the actual length of the audio file / pictures metadata with the real labels). Leaks can make the competition unfair.\n\nHere clearly this is the worse case: the test set is different from the training set (training on zebra pictures and testing on what is the current temperature of the room). The last ones I remembered with such a check up were the Mercedes and VSB power line competitions: they have been others recently.\n\nFor Kaggle defense, they need to manage the competition sponsors regarding the data and the sponsors are not always listening...", "votes": 4 } ] }, { "id": 1551023, "postDate": "2021-10-20T08:11:48.847Z", "content": "That is a main challenge here . in this datatset
", "rawMarkdown": "That is a main challenge here . in this datatset" }, { "id": 1550092, "postDate": "2021-10-19T12:48:24.167Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 1549815, "author_name": "Michał Choiński", "author_url": "", "post_date": "2021-10-19T08:06:06.680000", "content": "What I learned is that it's better to avoid competitions like this one
", "votes": 19, "replies": [ { "id": 1550672, "author_name": "", "author_url": "", "post_date": "2021-10-19T22:33:34.663000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1549837, "author_name": "Ruslan Vdovychenko", "author_url": "", "post_date": "2021-10-19T08:19:27.467000", "content": "It looks like I've made the right choice to focus on the Ventilator Pressure Prediction competition instead of this one.
\nYet, it feels pretty inspiring and self-explanatory that the 1st place team was named \"I hate this competition\" before the private LB was revealed 😊
Lol they even changed their name to \"I love this competition\" once private LB was revealed!
", "votes": 9, "replies": [] } ] }, { "id": 1551624, "author_name": "eagle4", "author_url": "", "post_date": "2021-10-20T18:51:54.733000", "content": "I have rarely seen such a lottery competition:
\nA gold medal and a prize with 1 single submission 2 months ago : that is the best time / money investment I have ever seen !
", "votes": 9, "replies": [] }, { "id": 1550361, "author_name": "Shivam Bansal", "author_url": "", "post_date": "2021-10-19T16:37:22.283000", "content": "Actual Private LB Distribution @theoviel
\n
The mode is very interesting here, basically most submissions turned out to be as good as random guessing like a coin toss.
\nCurious to know what the mean and median are from this chart, I would think mean is very close to 0.5 as well since the even though it is left skewed, the mode of 0.5 would pull the mean to the left.
", "votes": 2, "replies": [] }, { "id": 1550426, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-10-19T17:47:59.530000", "content": "Thanks @shivamb !
\nI think the distribution looks slightly better than random - I guess most models still found some cases.
The mode at of 0.5 probably comes from people submitting the same value everywhere and shouldn't really be considered
", "votes": 2, "replies": [] }, { "id": 1550671, "author_name": "", "author_url": "", "post_date": "2021-10-19T22:32:22.597000", "content": "", "votes": 0, "replies": [] }, { "id": 1550679, "author_name": "Daniel Chen", "author_url": "", "post_date": "2021-10-19T23:06:03.337000", "content": "@yuzheni where do you see multimodality here? If you just exclude the mode it looks normal. I think the multimodality you are seeing is possibly by random sampling a normal distribution of this size.
", "votes": 0, "replies": [] } ] }, { "id": 1549920, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2021-10-19T09:35:53.913000", "content": "From my perspective, it wasn't a failure because I learned a lot from this competition. Some people in the comments should be in this mindset because otherwise you will be disappointed a lot. I shake down and didn't select gold medal solutions multiple times in my previous competitions. I think all of them were great learning experiences and medals weren't my priority in any of them.
", "votes": 7, "replies": [ { "id": 1549945, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2021-10-19T10:08:50.240000", "content": "I wholeheartedly agree with @gunesevitan: The only way one can totally loose a competition is to treat the leaderboards as the sole metric of success!
\nAll the best,
\ncarl
What I meant about \"failure\" was more an overall conclusion. Sure some people learnt from it, some even got a medal.
\nBut none of the models produced are worth the 50k+ $ the hosts invested on the making of the competition.
\nNone of the models will help detect the mgmt gene, and thus help for the cure of brain tumours.
Which is unfortunate
", "votes": 20, "replies": [] }, { "id": 1549974, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2021-10-19T10:29:28.903000", "content": "Dear @theoviel
\nI see your point of view. However, I spent 20 years as an academic researcher, and the number of unsuccessful projects/ideas/experiments etc often out-way the successes, but at the same time unsuccessful results are the paving stones on the way to excellent results, and an insightful postmortem of an experiment (or in this case; a competition) can be almost as valuable as any other type of work.
\nThat said, it would obviously be wonderful if science had a shiny new way of helping to cure brain tumours…
\nAll the best,
\ncarl
Perhaps there is much more insight in the MRI scans than we think now. The problem of this competition was a very small dataset. When you look at any tips regarding the minimum dataset size for a successful Deep Learning model, you can find numbers like 100k, 10k, 1k, depending on the problem. We had less then 600 samples, from which some of them are used as validation data during training. I had never faced a problem like this before (to train a model with not sufficient amount of data) and I agree that this could be a valuable lesson. What I learned is that with not enough data it is extremely difficult to succeed. Now I have this feeling that trying to tackle such a problem with so little amount of data might indeed be an inevitable failure.
\nAnother thing is that seeing this crazy public LB filled with completely unrealistic scores was really daunting and if the rules say that LB probing is forbidden, it should be forbidden. I definitely consider this as a failure. I bet that many people gave up because of this.
What would happen if we had more data? Maybe this competition would become a success.
", "votes": 6, "replies": [] }, { "id": 1550129, "author_name": "Aditya Sharma", "author_url": "", "post_date": "2021-10-19T13:12:27.883000", "content": "Great comment.Can you elaborate The unrealistic value thing again as I haven't understood how public LB has the unrealistic values
", "votes": 0, "replies": [] }, { "id": 1550150, "author_name": "Michał Choiński", "author_url": "", "post_date": "2021-10-19T13:31:35.790000", "content": "@adityasharma01 In the public LB during the time of the competition you can normally have some approximation of your performance, and performance of the others. This would be an ideal scenario and it is often far from that.
\nHowever, in this competition a lot of people had scores 1.0 or 75+ which deviates so much from the final standings that this public LB was more confusing than helping anyone. And there were 2 reasons behind it:
\n\nAnother downside of the small public dataset - it was easier to submit a completely random result and be lucky enough to have a high score on the LB. There were many submissions of models that didn't learn anything but by chance they reached auc over 0.73 (many public notebooks had this score). You can see now that these models did not learn anything. It was just a roulette.
\n
That's why I never read the notebooks section. I only read through discussions and write my own code from scratch.
", "votes": 4, "replies": [] }, { "id": 1550199, "author_name": "Aditya Sharma", "author_url": "", "post_date": "2021-10-19T14:07:36.323000", "content": "Thank you for the explanation @mikecho Now I got your point and you are completely correct. My submission of 0.91 validation just got nothing value in private leader board. And of less validation got high ranks. Means if I forget the public leader board and just go on the performance of validation, then also the private leader board is not giving that good response.
", "votes": 1, "replies": [] }, { "id": 1550207, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-10-19T14:15:51.447000", "content": "Agree. And for people who claim to have great experience are because they got the medal😂
", "votes": 1, "replies": [] }, { "id": 1550247, "author_name": "noidea", "author_url": "", "post_date": "2021-10-19T14:53:55.247000", "content": "Haha a bit cynical @Swikwislkdj but understandable 😄
", "votes": 2, "replies": [] } ] }, { "id": 1549817, "author_name": "BubuChacha", "author_url": "", "post_date": "2021-10-19T08:08:15.207000", "content": "In public score, I get > 0.76, but in private, it get <0.5 when my low score model<0.6 get nearly 0.6 in private… So confused at first saw.
", "votes": 3, "replies": [ { "id": 1550146, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-10-19T13:30:04.047000", "content": "Same… do you know why?
", "votes": 0, "replies": [] }, { "id": 1550497, "author_name": "Gaofeng Huang", "author_url": "", "post_date": "2021-10-19T19:22:13.533000", "content": "Going back to the medical background, the meaning of this competition tells us using brain tumors to predict MGMT value is impossible. These two things are not related.
", "votes": 2, "replies": [] }, { "id": 1550515, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-19T19:33:33.497000", "content": "@together i disagree, this claim is simply not true.
\nThe right formulation would be: with the given number of inputs noone could find any image based biomarker (or feature) within this competition, that shows a higher correlation with the methylation status of the tumors of this training set.
\nThe glioma have much more genotypic parameters that determine their mr-morphological appearance and methylation is only one - probably - weak influencer of this phenotyp.
\nI firmly believe that if we were given other genotypic descriptors and probably some additional base sequences (like diffusion or maybe perfusion imaging), we would be able to find a higher correlating fature.
Agree with your explanation.
", "votes": 1, "replies": [] }, { "id": 1551947, "author_name": "BubuChacha", "author_url": "", "post_date": "2021-10-21T03:11:27.607000", "content": "I agree with your explanation, in the medical background, there are some problems. But in the competitor perspective, I think the public test make some competitors distracted to not use the complicated model because can easily be overfitting, while some people just tried only backbone and they quit, they also get the high score, you can see on the leaderboard, someone just joins and last submit was 1-2 months with a low score on public and they quit, however, they get the high score, I think the dataset change make some team fall 800-1k rank on the private board although they try hard for this competition.
", "votes": 1, "replies": [] } ] }, { "id": 1550731, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-10-20T01:11:45.690000", "content": "I think we can not say the models of top rankers are chance level because the distribution of LB score is similar to that of random sampling. The distribution of IQ is completely Gaussian, but I do not think Albert Einstein is as foolish as me. In addition, from the plot by @shivamb, the AUC scores of top rankers seem stable around 0.6 for both public and private datasets, while I admit that AUC 0.6 is too low for diagnoses.
", "votes": 1, "replies": [ { "id": 1550779, "author_name": "عثمان", "author_url": "", "post_date": "2021-10-20T02:41:13.777000", "content": "
The difference is that given a different test dataset, the rank of the top models can be switched, while given a different IQ test, Albert Einstein would still score higher than most people. Of course not all top models are random, but the randomness of the top is a bit too high in this competition.
", "votes": 2, "replies": [] }, { "id": 1551891, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-10-21T00:23:33.613000", "content": "Yes, we should see the (in)stability of CV and public AUC scores and not the distribution of them. We knew that the public score and ranking are far from reliable for many reasons.
", "votes": 0, "replies": [] } ] }, { "id": 1549782, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-19T07:44:41.190000", "content": "Thank you for the analysis!
\nI felt the same as i saw the <0.5 portion… was actually expecting much more results in the >0.6 area to be convinced that there is usable info in the images for this particular prediction!
Good analysis. Better to avoid these types of competitions.
", "votes": 2, "replies": [] }, { "id": 1550063, "author_name": "Krish Yadav", "author_url": "", "post_date": "2021-10-19T12:09:35.160000", "content": "Well sorry, I didn't understand what you meant, can you guys please explain in an easier way?
", "votes": 0, "replies": [] }, { "id": 1553374, "author_name": "Tian", "author_url": "", "post_date": "2021-10-22T06:07:08.067000", "content": "can't agree more.
", "votes": 0, "replies": [] }, { "id": 1553079, "author_name": "Uroš Poček", "author_url": "", "post_date": "2021-10-21T21:55:58.970000", "content": "Thank you for analysis. Better to avoid these types of competitions.
", "votes": 0, "replies": [] }, { "id": 1552192, "author_name": "Yasser Shkeir", "author_url": "", "post_date": "2021-10-21T08:29:09.210000", "content": "Does this happen frequently over Kaggle?
", "votes": 0, "replies": [ { "id": 1553054, "author_name": "eagle4", "author_url": "", "post_date": "2021-10-21T21:05:02.997000", "content": "Yes, about 1 out of 10 or 15 competitions.
\nUsually the big concerns are leakage problems (an information in the data that is hidden and not supposed to be there and is discovered by the competitors: too high correlation between the positive samples and the actual length of the audio file / pictures metadata with the real labels). Leaks can make the competition unfair.
\nHere clearly this is the worse case: the test set is different from the training set (training on zebra pictures and testing on what is the current temperature of the room). The last ones I remembered with such a check up were the Mercedes and VSB power line competitions: they have been others recently.
\nFor Kaggle defense, they need to manage the competition sponsors regarding the data and the sponsors are not always listening…
", "votes": 4, "replies": [] } ] }, { "id": 1551023, "author_name": "Aniket Donode", "author_url": "", "post_date": "2021-10-20T08:11:48.847000", "content": "That is a main challenge here . in this datatset
", "votes": 0, "replies": [] }, { "id": 1550092, "author_name": "", "author_url": "", "post_date": "2021-10-19T12:48:24.167000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "1549746": "Provided the fact that there are ~310 samples, it's fairly easy to compare how the LB looks compared to random submissions. Spoiler, they look the same.\n\n\n\nIn this run, randomness is better than the private LB, but it's not generally the case.\nFor instance, a 0.621 LB is top 0.02% over 100k runs - and scores higher than 0.65 can be reached ...\n\nThe code looks like this, in particular one can increase the number of iterations for a more reliable distribution :\n```\nN = 310\n# N = 582\n# N = 87\naucs = []\n\nprop = 0.525 # same as train ?\nn_pos = int(N * prop)\ny = np.concatenate((np.zeros(N - n_pos), np.ones(n_pos)), 0)\n\nfor _ in range(1555):\n pred = np.random.random(N)\n auc = roc_auc_score(y, pred)\n pred = np.random.random(N)\n auc2 = roc_auc_score(y, pred)\n \n aucs.append(max(auc, auc2))\n\nref = 0.621\nsns.displot(aucs)\nplt.axvline(ref, c='salmon')\nplt.title(f'AUC {ref} : Top {np.mean(np.array(aucs) > ref) * 100 :.2f}%')\nplt.show()\n``` \n\nMy advice to the hosts would be the following :\n\nTrain a baseline to see if there is signal behind the data ! Noticing that the models couldn't learn anything would've help better frame the problem.\n\n\nPS : if someone plotted the actual private LB distribution plase share it :)",
"1549815": "What I learned is that it's better to avoid competitions like this one",
"1549837": "It looks like I've made the right choice to focus on the Ventilator Pressure Prediction competition instead of this one.\nYet, it feels pretty inspiring and self-explanatory that the 1st place team was named \"I hate this competition\" before the private LB was revealed 😊",
"1551624": "I have rarely seen such a lottery competition:\n\nA gold medal and a prize with 1 single submission 2 months ago : that is the best time / money investment I have ever seen !\n\n",
"1550361": "Actual Private LB Distribution @theoviel \n\n",
"1549920": "From my perspective, it wasn't a failure because I learned a lot from this competition. Some people in the comments should be in this mindset because otherwise you will be disappointed a lot. I shake down and didn't select gold medal solutions multiple times in my previous competitions. I think all of them were great learning experiences and medals weren't my priority in any of them. ",
"1549817": "In public score, I get > 0.76, but in private, it get <0.5 when my low score model<0.6 get nearly 0.6 in private... So confused at first saw.",
"1550731": "I think we can not say the models of top rankers are chance level because the distribution of LB score is similar to that of random sampling. The distribution of [IQ](https://en.wikipedia.org/wiki/Intelligence_quotient) is completely Gaussian, but I do not think Albert Einstein is as foolish as me. In addition, [from the plot](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279777) by @shivamb, the AUC scores of top rankers seem stable around 0.6 for both public and private datasets, while I admit that AUC 0.6 is too low for diagnoses. ",
"1549782": "Thank you for the analysis!\nI felt the same as i saw the <0.5 portion... was actually expecting much more results in the >0.6 area to be convinced that there is usable info in the images for this particular prediction!",
"1550909": "Good analysis. Better to avoid these types of competitions.",
"1550063": "Well sorry, I didn't understand what you meant, can you guys please explain in an easier way?",
"1553374": "can't agree more.",
"1553079": "Thank you for analysis. Better to avoid these types of competitions.",
"1552192": "Does this happen frequently over Kaggle?",
"1551023": "That is a main challenge here . in this datatset",
"1550092": ""
}
}