When I compute locally my CV score using Kfold cross validation with 5 folds I get a score of 0.6937. However when I use 10 folds, I have CV of 0.6958. This increase is by quite a bit, so I wonder how it could be explained.
\nHas anybody experienced similar thing and what does that mean?
", "messages": [ { "id": "2240608", "postDate": "04/30/2023 17:21:33", "content": "When I compute locally my CV score using Kfold cross validation with 5 folds I get a score of 0.6937. However when I use 10 folds, I have CV of 0.6958. This increase is by quite a bit, so I wonder how it could be explained.
\nHas anybody experienced similar thing and what does that mean?
", "rawMarkdown": "When I compute locally my CV score using Kfold cross validation with 5 folds I get a score of 0.6937. However when I use 10 folds, I have CV of 0.6958. This increase is by quite a bit, so I wonder how it could be explained.\n\nHas anybody experienced similar thing and what does that mean?", "votes": null }, { "id": "2240892", "postDate": "05/01/2023 01:04:51", "content": "I'd say that this either mean the extra 12.5% data means alot to the model, or the noise have got to you. In my case, the CV varies +-0.002 with the slightest touch, such as changing the seed or the order of the columns etc.
", "rawMarkdown": "I'd say that this either mean the extra 12.5% data means alot to the model, or the noise have got to you. In my case, the CV varies +-0.002 with the slightest touch, such as changing the seed or the order of the columns etc.", "votes": null }, { "id": "2241113", "postDate": "05/01/2023 07:55:09", "content": "oh, that is so relieving to hear xax. I also noticed that changing order of columns or by adding just 1 new column as feature changes my CV by +-0.002. I am a bit afraid that means that the model is not generalizing well and might be screwed when the private LB is released. What do you think about this risk?
\nAlso what do you mean by the noise has got to me?
", "rawMarkdown": "oh, that is so relieving to hear xax. I also noticed that changing order of columns or by adding just 1 new column as feature changes my CV by +-0.002. I am a bit afraid that means that the model is not generalizing well and might be screwed when the private LB is released. What do you think about this risk?\n\nAlso what do you mean by the noise has got to me?", "votes": null }, { "id": "2241220", "postDate": "05/01/2023 10:02:00", "content": "Hi @ngocuong , I noticed the same improvement from 5 Folds to 15 Folds in my CV. My interpretation was that we get the ensemble effect, putting together more trained models.
", "rawMarkdown": "Hi @ngocuong , I noticed the same improvement from 5 Folds to 15 Folds in my CV. My interpretation was that we get the ensemble effect, putting together more trained models.", "votes": null }, { "id": "2241266", "postDate": "05/01/2023 10:48:56", "content": "I'm also afraid of overfitting, but since we have 3 submissions, I hope at least one of the model can perform OKish given the CV is high enough.
\n\n\nAlso what do you mean by the noise has got to me?
\n
By that, I just meant the random fluctuation of CV score, no hidden messages. :P
", "rawMarkdown": "I'm also afraid of overfitting, but since we have 3 submissions, I hope at least one of the model can perform OKish given the CV is high enough.\n\n> Also what do you mean by the noise has got to me?\n\nBy that, I just meant the random fluctuation of CV score, no hidden messages. :P", "votes": null }, { "id": "2241272", "postDate": "05/01/2023 10:55:22", "content": "cool, thanks for the response!
", "rawMarkdown": "cool, thanks for the response!", "votes": null }, { "id": "2241531", "postDate": "05/01/2023 15:26:41", "content": "This is quite normal actrually.
", "rawMarkdown": "This is quite normal actrually.", "votes": null }, { "id": "2241688", "postDate": "05/01/2023 17:56:14", "content": "@ngocuong, on a general note, I have seen a slight improvement in the CV score as I increase the number of folds. Usually I get my best CV score between 10-15 splits. As mentioned by @gehallak, this could occur due to the incremental benefit of ensembles among other contributing factors too.
", "rawMarkdown": "ngocuong, on a general note, I have seen a slight improvement in the CV score as I increase the number of folds. Usually I get my best CV score between 10-15 splits. As mentioned by @gehallak, this could occur due to the incremental benefit of ensembles among other contributing factors too.", "votes": null }, { "id": "2241737", "postDate": "05/01/2023 19:00:24", "content": "Sorry, I meant +-0.0002 of CV fluctuation, 0.002 sure sounds a lot to me. 😑
\nThe LB score on the other hand varies by +-0.002 depending model from different folds.
\nI tried to compare 10-folds and 5-folds CVs just now, but I only observed very slight improvements ~0.00015 going from 5- to 10- folds.
", "rawMarkdown": "Sorry, I meant +-0.0002 of CV fluctuation, 0.002 sure sounds a lot to me. 😑\n\nThe LB score on the other hand varies by +-0.002 depending model from different folds.\n\nI tried to compare 10-folds and 5-folds CVs just now, but I only observed very slight improvements ~0.00015 going from 5- to 10- folds.", "votes": null } ], "comments": [ { "id": 2240892, "author_name": "woprime", "author_url": "", "post_date": "05/01/2023 01:04:51", "content": "I'd say that this either mean the extra 12.5% data means alot to the model, or the noise have got to you. In my case, the CV varies +-0.002 with the slightest touch, such as changing the seed or the order of the columns etc.
", "votes": null, "replies": [ { "id": 2241113, "author_name": "ngocuong", "author_url": "", "post_date": "05/01/2023 07:55:09", "content": "oh, that is so relieving to hear xax. I also noticed that changing order of columns or by adding just 1 new column as feature changes my CV by +-0.002. I am a bit afraid that means that the model is not generalizing well and might be screwed when the private LB is released. What do you think about this risk?
\nAlso what do you mean by the noise has got to me?
", "votes": null, "replies": [ { "id": 2241266, "author_name": "woprime", "author_url": "", "post_date": "05/01/2023 10:48:56", "content": "I'm also afraid of overfitting, but since we have 3 submissions, I hope at least one of the model can perform OKish given the CV is high enough.
\n\n\nAlso what do you mean by the noise has got to me?
\n
By that, I just meant the random fluctuation of CV score, no hidden messages. :P
", "votes": null, "replies": [ { "id": 2241272, "author_name": "ngocuong", "author_url": "", "post_date": "05/01/2023 10:55:22", "content": "cool, thanks for the response!
", "votes": null, "replies": [ { "id": 2241737, "author_name": "woprime", "author_url": "", "post_date": "05/01/2023 19:00:24", "content": "Sorry, I meant +-0.0002 of CV fluctuation, 0.002 sure sounds a lot to me. 😑
\nThe LB score on the other hand varies by +-0.002 depending model from different folds.
\nI tried to compare 10-folds and 5-folds CVs just now, but I only observed very slight improvements ~0.00015 going from 5- to 10- folds.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2241220, "author_name": "gehallak", "author_url": "", "post_date": "05/01/2023 10:02:00", "content": "Hi @ngocuong , I noticed the same improvement from 5 Folds to 15 Folds in my CV. My interpretation was that we get the ensemble effect, putting together more trained models.
", "votes": null, "replies": [] }, { "id": 2241531, "author_name": "littlstar123", "author_url": "", "post_date": "05/01/2023 15:26:41", "content": "This is quite normal actrually.
", "votes": null, "replies": [] }, { "id": 2241688, "author_name": "ravi20076", "author_url": "", "post_date": "05/01/2023 17:56:14", "content": "@ngocuong, on a general note, I have seen a slight improvement in the CV score as I increase the number of folds. Usually I get my best CV score between 10-15 splits. As mentioned by @gehallak, this could occur due to the incremental benefit of ensembles among other contributing factors too.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2240608": "When I compute locally my CV score using Kfold cross validation with 5 folds I get a score of 0.6937. However when I use 10 folds, I have CV of 0.6958. This increase is by quite a bit, so I wonder how it could be explained.\n\nHas anybody experienced similar thing and what does that mean?", "2240892": "I'd say that this either mean the extra 12.5% data means alot to the model, or the noise have got to you. In my case, the CV varies +-0.002 with the slightest touch, such as changing the seed or the order of the columns etc.", "2241113": "oh, that is so relieving to hear xax. I also noticed that changing order of columns or by adding just 1 new column as feature changes my CV by +-0.002. I am a bit afraid that means that the model is not generalizing well and might be screwed when the private LB is released. What do you think about this risk?\n\nAlso what do you mean by the noise has got to me?", "2241220": "Hi @ngocuong , I noticed the same improvement from 5 Folds to 15 Folds in my CV. My interpretation was that we get the ensemble effect, putting together more trained models.", "2241266": "I'm also afraid of overfitting, but since we have 3 submissions, I hope at least one of the model can perform OKish given the CV is high enough.\n\n> Also what do you mean by the noise has got to me?\n\nBy that, I just meant the random fluctuation of CV score, no hidden messages. :P", "2241272": "cool, thanks for the response!", "2241531": "This is quite normal actrually.", "2241688": "ngocuong, on a general note, I have seen a slight improvement in the CV score as I increase the number of folds. Usually I get my best CV score between 10-15 splits. As mentioned by @gehallak, this could occur due to the incremental benefit of ensembles among other contributing factors too.", "2241737": "Sorry, I meant +-0.0002 of CV fluctuation, 0.002 sure sounds a lot to me. 😑\n\nThe LB score on the other hand varies by +-0.002 depending model from different folds.\n\nI tried to compare 10-folds and 5-folds CVs just now, but I only observed very slight improvements ~0.00015 going from 5- to 10- folds." }, "source": "meta" }