![\"Real](\"https://storage.googleapis.com/kaggle-forum-message-attachments/536955/13296/real_vs_pred.png\")
\nHere is a simple Kernel with the idea.I tried Leave One Out Cross Validation, the MAE for the cross validation is 1.22, but the public leaderboard is 1.66, worse than some public kernels. This is what I call overfitting:\n\nHere is a simple Kernel with the idea.
I tried Leave One Out Cross Validation, the MAE for the cross validation is 1.22, but the public leaderboard is 1.66, worse than some public kernels. This is what I call overfitting:\n\nHere is a simple Kernel with the idea.
Are you only calculating MAE on predictions made out of fold? Or averaging the predictions on entire training set every fold?
", "rawMarkdown": "Are you only calculating MAE on predictions made out of fold? Or averaging the predictions on entire training set every fold?", "votes": null }, { "id": "537129", "postDate": "05/26/2019 09:00:53", "content": "Important warning: Leave one out and, in general, random sampling for cross validation will not work when observations are strongly correlated. In time series there is usually strong correlation among observations near in time what implies information leak to out of folds, what causes this overfitting.
", "rawMarkdown": "Important warning: Leave one out and, in general, random sampling for cross validation will not work when observations are strongly correlated. In time series there is usually strong correlation among observations near in time what implies information leak to out of folds, what causes this overfitting.", "votes": null }, { "id": "537217", "postDate": "05/26/2019 13:29:15", "content": "Isn't it disappointing when you get a great CV, send your submission with great anticipation, and then find it scores terribly? This has happened to me so many times with this competition, CV strategy will surely be key to the winning solution.
", "rawMarkdown": "Isn't it disappointing when you get a great CV, send your submission with great anticipation, and then find it scores terribly? This has happened to me so many times with this competition, CV strategy will surely be key to the winning solution.", "votes": null }, { "id": "537660", "postDate": "05/27/2019 12:10:14", "content": "Yes, it is disappointing, but in this specific case I was aware of the overfitting so I didn't expect a good score.
", "rawMarkdown": "Yes, it is disappointing, but in this specific case I was aware of the overfitting so I didn't expect a good score.", "votes": null }, { "id": "537661", "postDate": "05/27/2019 12:12:22", "content": "I couldn't agree more.
", "rawMarkdown": "I couldn't agree more.", "votes": null }, { "id": "537662", "postDate": "05/27/2019 12:15:16", "content": "Out of fold. Please see here.
", "rawMarkdown": "Out of fold. Please see [here](https://www.kaggle.com/felipefonte99/leave-one-out).", "votes": null } ], "comments": [ { "id": 537127, "author_name": "teeyee314", "author_url": "", "post_date": "05/26/2019 08:51:00", "content": "Are you only calculating MAE on predictions made out of fold? Or averaging the predictions on entire training set every fold?
", "votes": null, "replies": [ { "id": 537662, "author_name": "felipefonte99", "author_url": "", "post_date": "05/27/2019 12:15:16", "content": "Out of fold. Please see here.
", "votes": null, "replies": [] } ] }, { "id": 537129, "author_name": "miguelpm", "author_url": "", "post_date": "05/26/2019 09:00:53", "content": "Important warning: Leave one out and, in general, random sampling for cross validation will not work when observations are strongly correlated. In time series there is usually strong correlation among observations near in time what implies information leak to out of folds, what causes this overfitting.
", "votes": null, "replies": [ { "id": 537661, "author_name": "felipefonte99", "author_url": "", "post_date": "05/27/2019 12:12:22", "content": "I couldn't agree more.
", "votes": null, "replies": [] } ] }, { "id": 537217, "author_name": "bigironsphere", "author_url": "", "post_date": "05/26/2019 13:29:15", "content": "Isn't it disappointing when you get a great CV, send your submission with great anticipation, and then find it scores terribly? This has happened to me so many times with this competition, CV strategy will surely be key to the winning solution.
", "votes": null, "replies": [ { "id": 537660, "author_name": "felipefonte99", "author_url": "", "post_date": "05/27/2019 12:10:14", "content": "Yes, it is disappointing, but in this specific case I was aware of the overfitting so I didn't expect a good score.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "536955": "I tried Leave One Out Cross Validation, the MAE for the cross validation is 1.22, but the public leaderboard is 1.66, worse than some public kernels. This is what I call overfitting:\n\nHere is a simple [Kernel](https://www.kaggle.com/felipefonte99/leave-one-out) with the idea.", "537127": "Are you only calculating MAE on predictions made out of fold? Or averaging the predictions on entire training set every fold?", "537129": "Important warning: Leave one out and, in general, random sampling for cross validation will not work when observations are strongly correlated. In time series there is usually strong correlation among observations near in time what implies information leak to out of folds, what causes this overfitting.", "537217": "Isn't it disappointing when you get a great CV, send your submission with great anticipation, and then find it scores terribly? This has happened to me so many times with this competition, CV strategy will surely be key to the winning solution.", "537660": "Yes, it is disappointing, but in this specific case I was aware of the overfitting so I didn't expect a good score.", "537661": "I couldn't agree more.", "537662": "Out of fold. Please see [here](https://www.kaggle.com/felipefonte99/leave-one-out)." }, "source": "meta" }