Hi all,
\nThis is probably obvious to most of you but I'm struggling to understand some basic things about validation and the domain gap between the training data and the test data.
\nGiven the domain gap, what should I be using as the validation set while training? A part of my (augmented, transformed…) training set, or the provided soundscapes ?
\nIf the former: doesn't this always lead to overfitting?
\nIf the latter, what is the use of k-fold cross validation, since I will be using all training material that I can generate anyway? (Maybe I misunderstand CV…)
\nThank you all.
", "messages": [ { "id": "1303547", "postDate": "05/12/2021 06:08:07", "content": "Hi all,
\nThis is probably obvious to most of you but I'm struggling to understand some basic things about validation and the domain gap between the training data and the test data.
\nGiven the domain gap, what should I be using as the validation set while training? A part of my (augmented, transformed…) training set, or the provided soundscapes ?
\nIf the former: doesn't this always lead to overfitting?
\nIf the latter, what is the use of k-fold cross validation, since I will be using all training material that I can generate anyway? (Maybe I misunderstand CV…)
\nThank you all.
", "rawMarkdown": "Hi all,\n\nThis is probably obvious to most of you but I'm struggling to understand some basic things about validation and the domain gap between the training data and the test data.\n\nGiven the domain gap, what should I be using as the validation set while training? A part of my (augmented, transformed...) training set, or the provided soundscapes ?\n\nIf the former: doesn't this always lead to overfitting? \n\nIf the latter, what is the use of k-fold cross validation, since I will be using all training material that I can generate anyway? (Maybe I misunderstand CV...) \n\nThank you all.", "votes": null }, { "id": "1304029", "postDate": "05/12/2021 11:42:20", "content": "You can use any part of dataset as your validation set until you do not train it.
\nAs in this competition we are provided with train_short_audios and train_soundscapes many of the competitors are using the train_short_audios to train and then test the performance on train_soundscapes to decide if the model is good enough of or not. The performance on train_soundscapes is being considered as CV…
@nitindatta Thank you for your answer.
\nCould you advise if my approach makes sense.
\nMy dataloaders generate spectrograms on the fly from randomly selected and augmented signals.
\nAfter each epoch I 'validate' against the train_soundscapes.
\nSo, not against some fixed hold out set generated in the same way as (but not used by) the training set.
\nTherefore no folding or anything, which a lot of people seem to be using.
Does that make sense?
\nThank you.
train_short_audio -> split into n-folds -> keep one fold for validation and train on rest
\ntrained_model -> check F1 score on train_soundscapes…..
\nI hope this gives you a clear idea
", "rawMarkdown": "train_short_audio -> split into n-folds -> keep one fold for validation and train on rest\n\ntrained_model -> check F1 score on train_soundscapes..... \n\nI hope this gives you a clear idea", "votes": null }, { "id": "1304491", "postDate": "05/12/2021 16:45:27", "content": "Thank you once more.
\nIt's still not clear to me however, why you need a hold out/validation set based on the training data, given the domain shift between the training data and the test data.
You don't need a validation set. Depends on your approach. Some approaches may use the validation sets for stacking/blending. Some may use validation set to select hyper-parameters. Alternatively, you can use LB as feedback to select hyper-parameters, but you risk overfitting to LB.
", "rawMarkdown": "You don't need a validation set. Depends on your approach. Some approaches may use the validation sets for stacking/blending. Some may use validation set to select hyper-parameters. Alternatively, you can use LB as feedback to select hyper-parameters, but you risk overfitting to LB.", "votes": null }, { "id": "1304578", "postDate": "05/12/2021 17:52:11", "content": "@jy2tong Thank you. That's what I thought. But given my dismal score, I thought I'd ask.
", "rawMarkdown": "jy2tong Thank you. That's what I thought. But given my dismal score, I thought I'd ask.", "votes": null } ], "comments": [ { "id": 1304029, "author_name": "nitindatta", "author_url": "", "post_date": "05/12/2021 11:42:20", "content": "You can use any part of dataset as your validation set until you do not train it.
\nAs in this competition we are provided with train_short_audios and train_soundscapes many of the competitors are using the train_short_audios to train and then test the performance on train_soundscapes to decide if the model is good enough of or not. The performance on train_soundscapes is being considered as CV…
@nitindatta Thank you for your answer.
\nCould you advise if my approach makes sense.
\nMy dataloaders generate spectrograms on the fly from randomly selected and augmented signals.
\nAfter each epoch I 'validate' against the train_soundscapes.
\nSo, not against some fixed hold out set generated in the same way as (but not used by) the training set.
\nTherefore no folding or anything, which a lot of people seem to be using.
Does that make sense?
\nThank you.
train_short_audio -> split into n-folds -> keep one fold for validation and train on rest
\ntrained_model -> check F1 score on train_soundscapes…..
\nI hope this gives you a clear idea
", "votes": null, "replies": [] } ] }, { "id": 1304491, "author_name": "botkop", "author_url": "", "post_date": "05/12/2021 16:45:27", "content": "Thank you once more.
\nIt's still not clear to me however, why you need a hold out/validation set based on the training data, given the domain shift between the training data and the test data.
You don't need a validation set. Depends on your approach. Some approaches may use the validation sets for stacking/blending. Some may use validation set to select hyper-parameters. Alternatively, you can use LB as feedback to select hyper-parameters, but you risk overfitting to LB.
", "votes": null, "replies": [] } ] }, { "id": 1304578, "author_name": "botkop", "author_url": "", "post_date": "05/12/2021 17:52:11", "content": "@jy2tong Thank you. That's what I thought. But given my dismal score, I thought I'd ask.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1303547": "Hi all,\n\nThis is probably obvious to most of you but I'm struggling to understand some basic things about validation and the domain gap between the training data and the test data.\n\nGiven the domain gap, what should I be using as the validation set while training? A part of my (augmented, transformed...) training set, or the provided soundscapes ?\n\nIf the former: doesn't this always lead to overfitting? \n\nIf the latter, what is the use of k-fold cross validation, since I will be using all training material that I can generate anyway? (Maybe I misunderstand CV...) \n\nThank you all.", "1304029": "You can use any part of dataset as your validation set until you do not train it. \nAs in this competition we are provided with train_short_audios and train_soundscapes many of the competitors are using the train_short_audios to train and then test the performance on train_soundscapes to decide if the model is good enough of or not. The performance on train_soundscapes is being considered as CV...", "1304169": "nitindatta Thank you for your answer. \n \nCould you advise if my approach makes sense. \n \nMy dataloaders generate spectrograms on the fly from randomly selected and augmented signals. \nAfter each epoch I 'validate' against the train_soundscapes. \nSo, not against some fixed hold out set generated in the same way as (but not used by) the training set. \nTherefore no folding or anything, which a lot of people seem to be using. \n \nDoes that make sense?\nThank you.", "1304439": "train_short_audio -> split into n-folds -> keep one fold for validation and train on rest\n\ntrained_model -> check F1 score on train_soundscapes..... \n\nI hope this gives you a clear idea", "1304491": "Thank you once more.\nIt's still not clear to me however, why you need a hold out/validation set based on the training data, given the domain shift between the training data and the test data.", "1304558": "You don't need a validation set. Depends on your approach. Some approaches may use the validation sets for stacking/blending. Some may use validation set to select hyper-parameters. Alternatively, you can use LB as feedback to select hyper-parameters, but you risk overfitting to LB.", "1304578": "jy2tong Thank you. That's what I thought. But given my dismal score, I thought I'd ask." }, "source": "meta" }