Should the organizers add a couple more decimals to the LB?
\nThis looks like the kind of competition that's gonna be settled with scores in the high 0.99s so a few more decimals would help w/ tracking your LB better.
", "messages": [ { "id": 1302798, "postDate": "2021-05-11T18:03:42.477Z", "content": "Should the organizers add a couple more decimals to the LB?
\nThis looks like the kind of competition that's gonna be settled with scores in the high 0.99s so a few more decimals would help w/ tracking your LB better.
", "rawMarkdown": "Should the organizers add a couple more decimals to the LB? \n\nThis looks like the kind of competition that's gonna be settled with scores in the high 0.99s so a few more decimals would help w/ tracking your LB better.", "votes": 21 }, { "id": 1302953, "postDate": "2021-05-11T19:56:32.637Z", "content": "For now, we'll hold off to discourage leaderboard probing.
", "rawMarkdown": "For now, we'll hold off to discourage leaderboard probing.", "votes": 5, "replies": [ { "id": 1304700, "postDate": "2021-05-12T19:26:54.830Z", "content": "Hi @inversion
\ngiven the imbalanced dataset, may we also consider F1 or Precision-Recall AUC vs current \"area under the ROC curve\"?
\nWe understand your concerns on LB probing… However, seeing scores of 0.97 on 2nd day makes it awkward…
Its early days and I am sure participants should be OK with a better metrics
", "rawMarkdown": "Hi @inversion \ngiven the imbalanced dataset, may we also consider F1 or Precision-Recall AUC vs current \"area under the ROC curve\"? \nWe understand your concerns on LB probing... However, seeing scores of 0.97 on 2nd day makes it awkward...\n \nIts early days and I am sure participants should be OK with a better metrics", "votes": 7 }, { "id": 1309559, "postDate": "2021-05-16T05:15:24.277Z", "content": "I think so
", "rawMarkdown": "I think so", "votes": 1 }, { "id": 1309810, "postDate": "2021-05-16T10:14:13.257Z", "content": "@kmldas I guess AUC is perfect in this case. It's not easy to cross 0.97.
\nThe thing is what we want from this competition. Criteria AUC relatively make sense than F1-Score in this competition.
Hi @micheomaano
\nOver 10% of the participants have reached 0.97 in less than a week of the competition. Perhaps it's not easy to cross 0.97, but not to difficult to reach as well.
\nAnd with 2 months to go, as @jbomitchell highlighted in his post, we may soon see:
\n\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"
\n
In an imbalanced dataset precision/recall are also useful. However, the competition organizers are best to decide the metrics
", "rawMarkdown": "Hi @micheomaano \nOver 10% of the participants have reached 0.97 in less than a week of the competition. Perhaps it's not easy to cross 0.97, but not to difficult to reach as well.\nAnd with 2 months to go, as @jbomitchell [highlighted in his post](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/238584), we may soon see:\n> \"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\" \n\nIn an imbalanced dataset precision/recall are also useful. However, the competition organizers are best to decide the metrics", "votes": 1 }, { "id": 1309849, "postDate": "2021-05-16T10:49:26.080Z", "content": "I'm not too fussed about the number of digits. Time and time again I've lamented my own overfitting to the public LB. Maybe this time I'll be forced to rely on CV for my own good.
", "rawMarkdown": "I'm not too fussed about the number of digits. Time and time again I've lamented my own overfitting to the public LB. Maybe this time I'll be forced to rely on CV for my own good.", "votes": 4 }, { "id": 1309873, "postDate": "2021-05-16T11:17:04.823Z", "content": "Agree with you and inversion on both (1) overfitting and (2) leaderboard probing…
\nbest to rely on CV.. and that has been the biggest learning of Kaggle and the private LB
while not fussing on number of decimal points, the choice of metrics and how the participants are clustered is not inconsequential to the process of the competition… there is a certain joy in slowing edging and improving one's LB score and LB rank.. and see it change…
\nthat said, fully appreciate the focus of CV-which should be the case..
", "rawMarkdown": "Agree with you and inversion on both (1) overfitting and (2) leaderboard probing...\nbest to rely on CV.. and that has been the biggest learning of Kaggle and the private LB\n\nwhile not fussing on number of decimal points, the choice of metrics and how the participants are clustered is not inconsequential to the process of the competition... there is a certain joy in slowing edging and improving one's LB score and LB rank.. and see it change...\n\nthat said, fully appreciate the focus of CV-which should be the case..", "votes": 1 } ] } ], "comments": [ { "id": 1302953, "author_name": "inversion", "author_url": "", "post_date": "2021-05-11T19:56:32.637000", "content": "For now, we'll hold off to discourage leaderboard probing.
", "votes": 5, "replies": [ { "id": 1304700, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-05-12T19:26:54.830000", "content": "Hi @inversion
\ngiven the imbalanced dataset, may we also consider F1 or Precision-Recall AUC vs current \"area under the ROC curve\"?
\nWe understand your concerns on LB probing… However, seeing scores of 0.97 on 2nd day makes it awkward…
Its early days and I am sure participants should be OK with a better metrics
", "votes": 7, "replies": [] }, { "id": 1309559, "author_name": "YTEP (Jiazhi Yang)", "author_url": "", "post_date": "2021-05-16T05:15:24.277000", "content": "I think so
", "votes": 1, "replies": [] }, { "id": 1309810, "author_name": "Salman", "author_url": "", "post_date": "2021-05-16T10:14:13.257000", "content": "@kmldas I guess AUC is perfect in this case. It's not easy to cross 0.97.
\nThe thing is what we want from this competition. Criteria AUC relatively make sense than F1-Score in this competition.
Hi @micheomaano
\nOver 10% of the participants have reached 0.97 in less than a week of the competition. Perhaps it's not easy to cross 0.97, but not to difficult to reach as well.
\nAnd with 2 months to go, as @jbomitchell highlighted in his post, we may soon see:
\n\n\"Your submission scored 1.00, which is not an improvement of your best score. Keep trying!\"
\n
In an imbalanced dataset precision/recall are also useful. However, the competition organizers are best to decide the metrics
", "votes": 1, "replies": [] }, { "id": 1309849, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-16T10:49:26.080000", "content": "I'm not too fussed about the number of digits. Time and time again I've lamented my own overfitting to the public LB. Maybe this time I'll be forced to rely on CV for my own good.
", "votes": 4, "replies": [] }, { "id": 1309873, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-05-16T11:17:04.823000", "content": "Agree with you and inversion on both (1) overfitting and (2) leaderboard probing…
\nbest to rely on CV.. and that has been the biggest learning of Kaggle and the private LB
while not fussing on number of decimal points, the choice of metrics and how the participants are clustered is not inconsequential to the process of the competition… there is a certain joy in slowing edging and improving one's LB score and LB rank.. and see it change…
\nthat said, fully appreciate the focus of CV-which should be the case..
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1302798": "Should the organizers add a couple more decimals to the LB? \n\nThis looks like the kind of competition that's gonna be settled with scores in the high 0.99s so a few more decimals would help w/ tracking your LB better.", "1302953": "For now, we'll hold off to discourage leaderboard probing." } }