I found many implementation just train the model with extremely imbalanced dataset. It will make model tend to predict very low probability that a team will score in next 10 seconds.
\nAlthough this method will give us a low log loss (see leaderboard), recall of model is really bad !
\nIf private testing dataset consists of many samples that either team A or team B will score in next 10 seconds, the performance of model will be bad.
\nWhy do not many implementations consider the recall of model ?
", "messages": [ { "id": "2005495", "postDate": "10/27/2022 03:57:00", "content": "I found many implementation just train the model with extremely imbalanced dataset. It will make model tend to predict very low probability that a team will score in next 10 seconds.
\nAlthough this method will give us a low log loss (see leaderboard), recall of model is really bad !
\nIf private testing dataset consists of many samples that either team A or team B will score in next 10 seconds, the performance of model will be bad.
\nWhy do not many implementations consider the recall of model ?
", "rawMarkdown": "I found many implementation just train the model with extremely imbalanced dataset. It will make model tend to predict very low probability that a team will score in next 10 seconds. \n\nAlthough this method will give us a low log loss (see leaderboard), recall of model is really bad !\n\nIf private testing dataset consists of many samples that either team A or team B will score in next 10 seconds, the performance of model will be bad. \n\nWhy do not many implementations consider the recall of model ?", "votes": null }, { "id": "2006118", "postDate": "10/27/2022 12:43:43", "content": "As shown in @alexryzhkov's informative notebook How to kill all your efforts? the scoring probabilities averaged over the training data are around 5-6%. I do something similar here. The Public LB values are in a broadly similar range.
\nWhile we don't know the balance of the Private LB sample, it would be reasonable to expect it to be somewhat similar. In the (unlikely) event that it isn't, the competition would turn into an extreme test of models' robustness.
\nAlso, the scoring metric for this competition is log loss, which is (almost always) optimised by predicting one's genuine best estimate of each probability.
", "rawMarkdown": "As shown in @alexryzhkov's informative notebook [How to kill all your efforts?](https://www.kaggle.com/code/alexryzhkov/how-to-kill-all-your-efforts) the scoring probabilities averaged over the training data are around 5-6%. I do something similar [here](https://www.kaggle.com/jbomitchell/mean-probabilities-from-training-data). The Public LB values are in a broadly similar range.\n\nWhile we don't know the balance of the Private LB sample, it would be reasonable to expect it to be somewhat similar. In the (unlikely) event that it isn't, the competition would turn into an extreme test of models' robustness.\n\nAlso, the scoring metric for this competition is log loss, which is (almost always) optimised by predicting one's genuine best estimate of each probability.", "votes": null }, { "id": "2007085", "postDate": "10/28/2022 02:48:11", "content": "@jbomitchell thank you for reply and recommended resource.
", "rawMarkdown": "jbomitchell thank you for reply and recommended resource.", "votes": null } ], "comments": [ { "id": 2006118, "author_name": "jbomitchell", "author_url": "", "post_date": "10/27/2022 12:43:43", "content": "As shown in @alexryzhkov's informative notebook How to kill all your efforts? the scoring probabilities averaged over the training data are around 5-6%. I do something similar here. The Public LB values are in a broadly similar range.
\nWhile we don't know the balance of the Private LB sample, it would be reasonable to expect it to be somewhat similar. In the (unlikely) event that it isn't, the competition would turn into an extreme test of models' robustness.
\nAlso, the scoring metric for this competition is log loss, which is (almost always) optimised by predicting one's genuine best estimate of each probability.
", "votes": null, "replies": [ { "id": 2007085, "author_name": "l066858998", "author_url": "", "post_date": "10/28/2022 02:48:11", "content": "@jbomitchell thank you for reply and recommended resource.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2005495": "I found many implementation just train the model with extremely imbalanced dataset. It will make model tend to predict very low probability that a team will score in next 10 seconds. \n\nAlthough this method will give us a low log loss (see leaderboard), recall of model is really bad !\n\nIf private testing dataset consists of many samples that either team A or team B will score in next 10 seconds, the performance of model will be bad. \n\nWhy do not many implementations consider the recall of model ?", "2006118": "As shown in @alexryzhkov's informative notebook [How to kill all your efforts?](https://www.kaggle.com/code/alexryzhkov/how-to-kill-all-your-efforts) the scoring probabilities averaged over the training data are around 5-6%. I do something similar [here](https://www.kaggle.com/jbomitchell/mean-probabilities-from-training-data). The Public LB values are in a broadly similar range.\n\nWhile we don't know the balance of the Private LB sample, it would be reasonable to expect it to be somewhat similar. In the (unlikely) event that it isn't, the competition would turn into an extreme test of models' robustness.\n\nAlso, the scoring metric for this competition is log loss, which is (almost always) optimised by predicting one's genuine best estimate of each probability.", "2007085": "jbomitchell thank you for reply and recommended resource." }, "source": "meta" }