\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F19099%2Fc322431ccac008b1209641fcdb02f78d%2F2022-10-24%20%2001.13.03.png?generation=1666563246475132&alt=media\")
Hi colleagues,
\nI have created the new kernel \"How to kill all your efforts?\" to show why you should be careful with target mean of your datasets if you are working with LogLoss metric.
To be clear - the target mean for the team A is around 6% and if you work carefully with the data splitting, you will receive the 0.20583 and 0.20236 LogLoss metrics for validation and test sets respectively using LightGBM model. The predictions histograms will look like this:
\n
But if you sample the train and validation data to have 33% target rate (less negative with all the positive objects), you will receive the dramatically worse scores - 0.38444 and 0.38042 LogLoss metrics for validation and test sets respectively using the same LightGBM model. The predictions histograms will now look like:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F19099%2F2e781446f4d82ac35475dbcb03959b86%2F2022-10-24%20%2001.13.38.png?generation=1666563435040349&alt=media\")
As you can see they move to the right and it kills all of your work - it doesn't mind that great features you have in your dataset. So please be careful with the data preparation if you want to get a good score.
\nHope this helps you!
\nAlex
", "messages": [ { "id": "2001286", "postDate": "10/23/2022 22:19:43", "content": "Hi colleagues,
\nI have created the new kernel \"How to kill all your efforts?\" to show why you should be careful with target mean of your datasets if you are working with LogLoss metric.
To be clear - the target mean for the team A is around 6% and if you work carefully with the data splitting, you will receive the 0.20583 and 0.20236 LogLoss metrics for validation and test sets respectively using LightGBM model. The predictions histograms will look like this:
\n
But if you sample the train and validation data to have 33% target rate (less negative with all the positive objects), you will receive the dramatically worse scores - 0.38444 and 0.38042 LogLoss metrics for validation and test sets respectively using the same LightGBM model. The predictions histograms will now look like:
\n
As you can see they move to the right and it kills all of your work - it doesn't mind that great features you have in your dataset. So please be careful with the data preparation if you want to get a good score.
\nHope this helps you!
\nAlex
", "rawMarkdown": "Hi colleagues,\n\nI have created the new kernel [\"How to kill all your efforts?\"](https://www.kaggle.com/code/alexryzhkov/how-to-kill-all-your-efforts) to show why you should be careful with target mean of your datasets if you are working with `LogLoss` metric.\n\nTo be clear - the target mean for the team A is around 6% and if you work carefully with the data splitting, you will receive the **0.20583** and **0.20236** LogLoss metrics for validation and test sets respectively using LightGBM model. The predictions histograms will look like this:\n\n\nBut if you sample the train and validation data to have 33% target rate (less negative with all the positive objects), you will receive the dramatically worse scores - **0.38444** and **0.38042** LogLoss metrics for validation and test sets respectively using the same LightGBM model. The predictions histograms will now look like:\n\n\nAs you can see they move to the right and it kills all of your work - it doesn't mind that great features you have in your dataset. So please **be careful with the data preparation if you want to get a good score**.\n\nHope this helps you!\n\nAlex", "votes": null }, { "id": "2001428", "postDate": "10/24/2022 03:29:27", "content": "Great explanation, so we should use stratified sample or always test the behavior of your split? I just read today about adversarial validation, but I am not sure if that could work in this case. Thanks.
", "rawMarkdown": "Great explanation, so we should use stratified sample or always test the behavior of your split? I just read today about adversarial validation, but I am not sure if that could work in this case. Thanks.", "votes": null }, { "id": "2013429", "postDate": "11/01/2022 22:34:19", "content": "Thank you for the explanation. It was very helpful to read.
", "rawMarkdown": "Thank you for the explanation. It was very helpful to read.", "votes": null } ], "comments": [ { "id": 2001428, "author_name": "pastorsoto", "author_url": "", "post_date": "10/24/2022 03:29:27", "content": "Great explanation, so we should use stratified sample or always test the behavior of your split? I just read today about adversarial validation, but I am not sure if that could work in this case. Thanks.
", "votes": null, "replies": [] }, { "id": 2013429, "author_name": "rafkhat", "author_url": "", "post_date": "11/01/2022 22:34:19", "content": "Thank you for the explanation. It was very helpful to read.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2001286": "Hi colleagues,\n\nI have created the new kernel [\"How to kill all your efforts?\"](https://www.kaggle.com/code/alexryzhkov/how-to-kill-all-your-efforts) to show why you should be careful with target mean of your datasets if you are working with `LogLoss` metric.\n\nTo be clear - the target mean for the team A is around 6% and if you work carefully with the data splitting, you will receive the **0.20583** and **0.20236** LogLoss metrics for validation and test sets respectively using LightGBM model. The predictions histograms will look like this:\n\n\nBut if you sample the train and validation data to have 33% target rate (less negative with all the positive objects), you will receive the dramatically worse scores - **0.38444** and **0.38042** LogLoss metrics for validation and test sets respectively using the same LightGBM model. The predictions histograms will now look like:\n\n\nAs you can see they move to the right and it kills all of your work - it doesn't mind that great features you have in your dataset. So please **be careful with the data preparation if you want to get a good score**.\n\nHope this helps you!\n\nAlex", "2001428": "Great explanation, so we should use stratified sample or always test the behavior of your split? I just read today about adversarial validation, but I am not sure if that could work in this case. Thanks.", "2013429": "Thank you for the explanation. It was very helpful to read." }, "source": "meta" }