Hi all, I am ashamed to ask such a naive question, but I learn things slowly, layer by layer, and now I reached the stage of OOF score of a single fold.
From GM Nakama's notebook, it seems that each OOF score for his individual fold, is none other than the best validation accuracy of that said fold. In that case, is the word OOF for one single fold just a sugar coat for the best val loss/val acc (whichever metric you decide) in that fold?
I do know that, if one trains 5 folds, then the OOF score of the 5 folds means something different. It should be an aggregate of all 5 folds results. But then again, I got easily confused when one uses the word CV score vs OOF score. Hope someone can clarify my doubts (I did do my due diligence and plan to write a notebook to clear these concepts once and for al.
Hi all, I am ashamed to ask such a naive question, but I learn things slowly, layer by layer, and now I reached the stage of OOF score of a single fold.
From GM Nakama's notebook, it seems that each OOF score for his individual fold, is none other than the best validation accuracy of that said fold. In that case, is the word OOF for one single fold just a sugar coat for the best val loss/val acc (whichever metric you decide) in that fold?
I do know that, if one trains 5 folds, then the OOF score of the 5 folds means something different. It should be an aggregate of all 5 folds results. But then again, I got easily confused when one uses the word CV score vs OOF score. Hope someone can clarify my doubts (I did do my due diligence and plan to write a notebook to clear these concepts once and for al.
I guess the OOF score is actually the score you obtain when you concatenate the validation set predictions and their corresponding labels and then run the specific metric on this whole set. The CV score is the average of the validation fold metric.
", "rawMarkdown": "I guess the OOF score is actually the score you obtain when you concatenate the validation set predictions and their corresponding labels and then run the specific metric on this whole set. The CV score is the average of the validation fold metric.", "votes": null }, { "id": "1099147", "postDate": "12/02/2020 05:50:42", "content": "Hey In each fold, we generate validation predictions for a part of the dataset and add it to the OOF. At the end of KFold, we have a completely filled OOF. You have the best validation accuracy in each Fold, but the Accuracy of OOF gives you the best estimate because it is generated from all the models (from all the folds). This topic by Chris Deotte really helped me in understanding this, check it out. Also, I am a beginner too, so feel free to point out any corrections.
", "rawMarkdown": "Hey In each fold, we generate validation predictions for a part of the dataset and add it to the OOF. At the end of KFold, we have a completely filled OOF. You have the best validation accuracy in each Fold, but the Accuracy of OOF gives you the best estimate because it is generated from all the models (from all the folds). [This topic by Chris Deotte](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614) really helped me in understanding this, check it out. Also, I am a beginner too, so feel free to point out any corrections.", "votes": null }, { "id": "1099221", "postDate": "12/02/2020 07:30:34", "content": "To understand this, you just need to check the definition of the accuracy. Namely, how accuracy is calculated. Then you may look at BCE loss and how BCE is calculated. Question solved.
", "rawMarkdown": "To understand this, you just need to check the definition of the accuracy. Namely, how accuracy is calculated. Then you may look at BCE loss and how BCE is calculated. Question solved.", "votes": null }, { "id": "1099314", "postDate": "12/02/2020 08:55:31", "content": "@underwearfitting Seems like it is what it seems, I would take the definition of \"oof\" score for each fold as the best acc score (or whichever metric one chooses). I would also take the oof score for the whole 5 folds to be as follows: evaluating the model on each holdout fold, predictions are made and stored in a list. Then, at the end of the run, the predictions are compared to the expected values for each holdout test set and a single accuracy score is reported.
", "rawMarkdown": "underwearfitting Seems like it is what it seems, I would take the definition of \"oof\" score for each fold as the best acc score (or whichever metric one chooses). I would also take the oof score for the whole 5 folds to be as follows: evaluating the model on each holdout fold, predictions are made and stored in a list. Then, at the end of the run, the predictions are compared to the expected values for each holdout test set and a single accuracy score is reported.", "votes": null }, { "id": "1099691", "postDate": "12/02/2020 14:36:19", "content": "Thank you for sharing, love it
", "rawMarkdown": "Thank you for sharing, love it", "votes": null }, { "id": "1100252", "postDate": "12/03/2020 00:06:35", "content": "Out-of-fold (OOF) score is the score (logloss, auc, whatever the metric is) obtained for the validation set in a cross validation process. There is also the \"in-fold score\" term, which is the score obtained for the training set. The cross validation can be any kind of cross validation strategy (single train-test split, K-Fold, GroupKFold, etc).
\nIn the case of K-Fold, it is possible to obtain the OOF score for the entire training data as discussed in Chris' post.
\nOOF score = CV score. It's just the validation score.
\nSo, OOF (=CV score) can be calculated for each fold or for the entire training data depending on the cross validation strategy.
", "rawMarkdown": "Out-of-fold (OOF) score is the score (logloss, auc, whatever the metric is) obtained for the validation set in a cross validation process. There is also the \"in-fold score\" term, which is the score obtained for the training set. The cross validation can be any kind of cross validation strategy (single train-test split, K-Fold, GroupKFold, etc).\n\nIn the case of K-Fold, it is possible to obtain the OOF score for the entire training data as discussed in Chris' [post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614).\n\nOOF score = CV score. It's just the validation score.\n\nSo, OOF (=CV score) can be calculated for each fold or for the entire training data depending on the cross validation strategy.", "votes": null }, { "id": "1101212", "postDate": "12/03/2020 18:00:33", "content": "@reighns This question bothered me alot when I started out in Kaggle Refer this answer on stackexchangehope it will clarify your doubt.
", "rawMarkdown": "reighns This question bothered me alot when I started out in Kaggle Refer this answer on [stackexchange]( https://stackoverflow.com/questions/52396191/what-is-oof-approach-in-machine-learning )hope it will clarify your doubt.", "votes": null } ], "comments": [ { "id": 1099115, "author_name": "dragonpg2000", "author_url": "", "post_date": "12/02/2020 05:13:41", "content": "I guess the OOF score is actually the score you obtain when you concatenate the validation set predictions and their corresponding labels and then run the specific metric on this whole set. The CV score is the average of the validation fold metric.
", "votes": null, "replies": [] }, { "id": 1099147, "author_name": "yerramvarun", "author_url": "", "post_date": "12/02/2020 05:50:42", "content": "Hey In each fold, we generate validation predictions for a part of the dataset and add it to the OOF. At the end of KFold, we have a completely filled OOF. You have the best validation accuracy in each Fold, but the Accuracy of OOF gives you the best estimate because it is generated from all the models (from all the folds). This topic by Chris Deotte really helped me in understanding this, check it out. Also, I am a beginner too, so feel free to point out any corrections.
", "votes": null, "replies": [ { "id": 1099691, "author_name": "medali1992", "author_url": "", "post_date": "12/02/2020 14:36:19", "content": "Thank you for sharing, love it
", "votes": null, "replies": [] } ] }, { "id": 1099221, "author_name": "underwearfitting", "author_url": "", "post_date": "12/02/2020 07:30:34", "content": "To understand this, you just need to check the definition of the accuracy. Namely, how accuracy is calculated. Then you may look at BCE loss and how BCE is calculated. Question solved.
", "votes": null, "replies": [ { "id": 1099314, "author_name": "reighns", "author_url": "", "post_date": "12/02/2020 08:55:31", "content": "@underwearfitting Seems like it is what it seems, I would take the definition of \"oof\" score for each fold as the best acc score (or whichever metric one chooses). I would also take the oof score for the whole 5 folds to be as follows: evaluating the model on each holdout fold, predictions are made and stored in a list. Then, at the end of the run, the predictions are compared to the expected values for each holdout test set and a single accuracy score is reported.
", "votes": null, "replies": [] } ] }, { "id": 1100252, "author_name": "tolgadincer", "author_url": "", "post_date": "12/03/2020 00:06:35", "content": "Out-of-fold (OOF) score is the score (logloss, auc, whatever the metric is) obtained for the validation set in a cross validation process. There is also the \"in-fold score\" term, which is the score obtained for the training set. The cross validation can be any kind of cross validation strategy (single train-test split, K-Fold, GroupKFold, etc).
\nIn the case of K-Fold, it is possible to obtain the OOF score for the entire training data as discussed in Chris' post.
\nOOF score = CV score. It's just the validation score.
\nSo, OOF (=CV score) can be calculated for each fold or for the entire training data depending on the cross validation strategy.
", "votes": null, "replies": [] }, { "id": 1101212, "author_name": "sayedathar11", "author_url": "", "post_date": "12/03/2020 18:00:33", "content": "@reighns This question bothered me alot when I started out in Kaggle Refer this answer on stackexchangehope it will clarify your doubt.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1099030": "Hi all, I am ashamed to ask such a naive question, but I learn things slowly, layer by layer, and now I reached the stage of `OOF` score of a single fold.\n\nFrom [GM Nakama's notebook](https://www.kaggle.com/yasufuminakama/cassava-resnext50-32x4d-starter-training), it seems that each `OOF` score for his individual fold, is none other than the best `validation accuracy` of that said fold. In that case, is the word `OOF` for one single fold just a sugar coat for the `best val loss/val acc (whichever metric you decide)` in that fold?\n\nI do know that, if one trains 5 folds, then the `OOF` score of the 5 folds means something different. It should be an aggregate of all 5 folds results. But then again, I got easily confused when one uses the word `CV` score vs `OOF` score. Hope someone can clarify my doubts (I did do my due diligence and plan to write a notebook to clear these concepts once and for al.", "1099115": "I guess the OOF score is actually the score you obtain when you concatenate the validation set predictions and their corresponding labels and then run the specific metric on this whole set. The CV score is the average of the validation fold metric.", "1099147": "Hey In each fold, we generate validation predictions for a part of the dataset and add it to the OOF. At the end of KFold, we have a completely filled OOF. You have the best validation accuracy in each Fold, but the Accuracy of OOF gives you the best estimate because it is generated from all the models (from all the folds). [This topic by Chris Deotte](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614) really helped me in understanding this, check it out. Also, I am a beginner too, so feel free to point out any corrections.", "1099221": "To understand this, you just need to check the definition of the accuracy. Namely, how accuracy is calculated. Then you may look at BCE loss and how BCE is calculated. Question solved.", "1099314": "underwearfitting Seems like it is what it seems, I would take the definition of \"oof\" score for each fold as the best acc score (or whichever metric one chooses). I would also take the oof score for the whole 5 folds to be as follows: evaluating the model on each holdout fold, predictions are made and stored in a list. Then, at the end of the run, the predictions are compared to the expected values for each holdout test set and a single accuracy score is reported.", "1099691": "Thank you for sharing, love it", "1100252": "Out-of-fold (OOF) score is the score (logloss, auc, whatever the metric is) obtained for the validation set in a cross validation process. There is also the \"in-fold score\" term, which is the score obtained for the training set. The cross validation can be any kind of cross validation strategy (single train-test split, K-Fold, GroupKFold, etc).\n\nIn the case of K-Fold, it is possible to obtain the OOF score for the entire training data as discussed in Chris' [post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614).\n\nOOF score = CV score. It's just the validation score.\n\nSo, OOF (=CV score) can be calculated for each fold or for the entire training data depending on the cross validation strategy.", "1101212": "reighns This question bothered me alot when I started out in Kaggle Refer this answer on [stackexchange]( https://stackoverflow.com/questions/52396191/what-is-oof-approach-in-machine-learning )hope it will clarify your doubt." }, "source": "meta" }