![\"https://i.ibb.co/PjTj84K/Selection-916.png\"](\"https://i.ibb.co/PjTj84K/Selection-916.png\")
In this competition, CV vs LB is very nice.
\nIt seems that no big shake will be happen as mentioned in previous discussion.
\n[CV vs LB]
\nShake up expectations ?
I was just curious to try shake estimation, so I published the notebook.
\nhttps://www.kaggle.com/cocoinit23/shake-estimation
I would be happy to get your feedbacks.
\nThanks!
In this competition, CV vs LB is very nice.
\nIt seems that no big shake will be happen as mentioned in previous discussion.
\n[CV vs LB]
\nShake up expectations ?
I was just curious to try shake estimation, so I published the notebook.
\nhttps://www.kaggle.com/cocoinit23/shake-estimation
I would be happy to get your feedbacks.
\nThanks!
our public board uses 16% of 226000 = 36160 samples.
\nthis is 33% (36160/112000) in my graph above.
\nthis is roughly sufficient.
\n(you should analyze for 5 folds, over different random subset)
private/public shakeup should be within 0.001~0.002.
\nIf the current top team does not overfit, he should be a clear winner.
\nit is interesting to make similar graphs if pretraining/normalisation/self-supervised learning is applied to both validation and training sets. or one may think of a way to make the curves (3.75% 10% 20% 50% …) closer
", "rawMarkdown": "\n\nour public board uses 16% of 226000 = 36160 samples.\nthis is 33% (36160/112000) in my graph above.\nthis is roughly sufficient.\n(you should analyze for 5 folds, over different random subset)\n\nprivate/public shakeup should be within 0.001~0.002.\n\nIf the current top team does not overfit, he should be a clear winner.\n\n---\n\nit is interesting to make similar graphs if pretraining/normalisation/self-supervised learning is applied to both validation and training sets. or one may think of a way to make the curves (3.75% 10% 20% 50% ...) closer", "votes": null }, { "id": "1526941", "postDate": "09/28/2021 12:38:20", "content": "I don't think you can call a constant shift from public to private a shakeup, as everyone will or will not have this shift. A shakeup is a severe shuffle of ranks.
", "rawMarkdown": "I don't think you can call a constant shift from public to private a shakeup, as everyone will or will not have this shift. A shakeup is a severe shuffle of ranks.", "votes": null }, { "id": "1527577", "postDate": "09/28/2021 22:11:24", "content": "I think that your analysis is too pessimistic because your sampling doesn't account for SNR in samples.
\n(basically, some of your sample sets may contain only low-SNR samples which are hard or impossible to predict, and some sample sets may contain only high-SNR samples which are relatively easy to predict)
I have an assumption that organizers will not only keep class balance the same, but also set of intrinsic parameters for generating GWs across public and private test (this will result in roughly the same PDFs of SNR of public-test/private-test and hence we won't experience the shakeup)
\nI think if there is a way to measure SNR of samples, it would be better to compare different sample sets with close PDFs of SNR, with this approach we'll get more reasonable estimation of shake-up.
\nPDF is a probability density function, if we can call it like that in this case.
\nIs will be great to discuss what would happen if distribution shift of SNR occurs.
", "rawMarkdown": "I think that your analysis is too pessimistic because your sampling doesn't account for SNR in samples.\n(basically, some of your sample sets may contain only low-SNR samples which are hard or impossible to predict, and some sample sets may contain only high-SNR samples which are relatively easy to predict)\n \nI have an assumption that organizers will not only keep class balance the same, but also set of intrinsic parameters for generating GWs across public and private test (this will result in roughly the same PDFs of SNR of public-test/private-test and hence we won't experience the shakeup)\n\nI think if there is a way to measure SNR of samples, it would be better to compare different sample sets with close PDFs of SNR, with this approach we'll get more reasonable estimation of shake-up.\n\nPDF is a probability density function, if we can call it like that in this case.\n\nIs will be great to discuss what would happen if distribution shift of SNR occurs.", "votes": null }, { "id": "1527650", "postDate": "09/29/2021 01:30:36", "content": "You are right, SNR is an important factor.
\nIf we could calculate SNR of samples, the d prime based on Signal Detection Theory might be better metric to analyze difference of probability density distributions.
That’s a sharp opinion, AUC is affected by the order of predicted scores.
\nBut I re-run my estimation notebook with rank transformation below, the results were almost the same.
\ndf[“y_pred”] = df[\"y_pred\"].rank(pct=True)\n
\nMaybe I have overlooked something important…
I think the point here is that the variance of score when you bootstrap dataset contains both fixed effect (constant shift as Psi mentioned) and random effect (this is what we call \"shakeup\").
", "rawMarkdown": "I think the point here is that the variance of score when you bootstrap dataset contains both fixed effect (constant shift as Psi mentioned) and random effect (this is what we call \"shakeup\").", "votes": null }, { "id": "1527787", "postDate": "09/29/2021 05:08:31", "content": "That make sense, I misunderstood.
\nThank you for elaboration!
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
our public board uses 16% of 226000 = 36160 samples.
\nthis is 33% (36160/112000) in my graph above.
\nthis is roughly sufficient.
\n(you should analyze for 5 folds, over different random subset)
private/public shakeup should be within 0.001~0.002.
\nIf the current top team does not overfit, he should be a clear winner.
\nit is interesting to make similar graphs if pretraining/normalisation/self-supervised learning is applied to both validation and training sets. or one may think of a way to make the curves (3.75% 10% 20% 50% …) closer
", "votes": null, "replies": [ { "id": 1526941, "author_name": "philippsinger", "author_url": "", "post_date": "09/28/2021 12:38:20", "content": "I don't think you can call a constant shift from public to private a shakeup, as everyone will or will not have this shift. A shakeup is a severe shuffle of ranks.
", "votes": null, "replies": [] }, { "id": 1527660, "author_name": "cocoinit23", "author_url": "", "post_date": "09/29/2021 01:43:32", "content": "That’s a sharp opinion, AUC is affected by the order of predicted scores.
\nBut I re-run my estimation notebook with rank transformation below, the results were almost the same.
\ndf[“y_pred”] = df[\"y_pred\"].rank(pct=True)\n
\nMaybe I have overlooked something important…
I think the point here is that the variance of score when you bootstrap dataset contains both fixed effect (constant shift as Psi mentioned) and random effect (this is what we call \"shakeup\").
", "votes": null, "replies": [] }, { "id": 1527787, "author_name": "cocoinit23", "author_url": "", "post_date": "09/29/2021 05:08:31", "content": "That make sense, I misunderstood.
\nThank you for elaboration!
I think that your analysis is too pessimistic because your sampling doesn't account for SNR in samples.
\n(basically, some of your sample sets may contain only low-SNR samples which are hard or impossible to predict, and some sample sets may contain only high-SNR samples which are relatively easy to predict)
I have an assumption that organizers will not only keep class balance the same, but also set of intrinsic parameters for generating GWs across public and private test (this will result in roughly the same PDFs of SNR of public-test/private-test and hence we won't experience the shakeup)
\nI think if there is a way to measure SNR of samples, it would be better to compare different sample sets with close PDFs of SNR, with this approach we'll get more reasonable estimation of shake-up.
\nPDF is a probability density function, if we can call it like that in this case.
\nIs will be great to discuss what would happen if distribution shift of SNR occurs.
", "votes": null, "replies": [ { "id": 1527650, "author_name": "cocoinit23", "author_url": "", "post_date": "09/29/2021 01:30:36", "content": "You are right, SNR is an important factor.
\nIf we could calculate SNR of samples, the d prime based on Signal Detection Theory might be better metric to analyze difference of probability density distributions.
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris