Hey!
\n\nThanks for looking into this.\nSo I have been training with a CNN network. My train set is based on 40 folders and val set is based on the remaining 10 folders. The image resolution being fed into the network is 224.\nI'm pretty sure the network is overfitting really quickly. What I see happening is that, for the first 2-4 epochs, both the val loss and the training loss starts dropping, usually around 0.1 for train loss and 0.37 for val loss by the end of 3/4th epoch. And after this, val loss jumps super high to around 0.8-1 and training loss just keeps going lower. Seems like major overfitting going on. My next attempt would obviously be to lower the damned learning rate by the end of 3rd epoch itself. But I just wanted to know if other LB toppers are facing similar situations of overfitting this quick during training?
\n\nAnother really interesting thing going on is that the val loss just sits around the range of 0.8 with accuracy of 50% whenever I try image augmentation. The training loss keeps falling rapidly though. Extremely weird.
\n\nPS: I wish I had a bunch of GPUs. =/
\n\nThanks.\nAkash
", "messages": [ { "id": "742617", "postDate": "02/11/2020 12:21:28", "content": "Hey!
\n\nThanks for looking into this.\nSo I have been training with a CNN network. My train set is based on 40 folders and val set is based on the remaining 10 folders. The image resolution being fed into the network is 224.\nI'm pretty sure the network is overfitting really quickly. What I see happening is that, for the first 2-4 epochs, both the val loss and the training loss starts dropping, usually around 0.1 for train loss and 0.37 for val loss by the end of 3/4th epoch. And after this, val loss jumps super high to around 0.8-1 and training loss just keeps going lower. Seems like major overfitting going on. My next attempt would obviously be to lower the damned learning rate by the end of 3rd epoch itself. But I just wanted to know if other LB toppers are facing similar situations of overfitting this quick during training?
\n\nAnother really interesting thing going on is that the val loss just sits around the range of 0.8 with accuracy of 50% whenever I try image augmentation. The training loss keeps falling rapidly though. Extremely weird.
\n\nPS: I wish I had a bunch of GPUs. =/
\n\nThanks.\nAkash
", "rawMarkdown": "Hey!\n\nThanks for looking into this.\nSo I have been training with a CNN network. My train set is based on 40 folders and val set is based on the remaining 10 folders. The image resolution being fed into the network is 224.\nI'm pretty sure the network is overfitting really quickly. What I see happening is that, for the first 2-4 epochs, both the val loss and the training loss starts dropping, usually around 0.1 for train loss and 0.37 for val loss by the end of 3/4th epoch. And after this, val loss jumps super high to around 0.8-1 and training loss just keeps going lower. Seems like major overfitting going on. My next attempt would obviously be to lower the damned learning rate by the end of 3rd epoch itself. But I just wanted to know if other LB toppers are facing similar situations of overfitting this quick during training?\n\nAnother really interesting thing going on is that the val loss just sits around the range of 0.8 with accuracy of 50% whenever I try image augmentation. The training loss keeps falling rapidly though. Extremely weird. \n\nPS: I wish I had a bunch of GPUs. =/\n\nThanks.\nAkash", "votes": null }, { "id": "742653", "postDate": "02/11/2020 12:50:19", "content": "More augmentation really improved my score (but there's a limit to extreme you can make the augmentation).
", "rawMarkdown": "More augmentation really improved my score (but there's a limit to extreme you can make the augmentation).", "votes": null }, { "id": "743236", "postDate": "02/11/2020 23:01:13", "content": "I got my current score are just training 2 epochs...
", "rawMarkdown": "I got my current score are just training 2 epochs...", "votes": null }, { "id": "743382", "postDate": "02/12/2020 02:42:01", "content": "Can i ask if extracting more frames from videos achieve similar effect as augmentation? Thanks.
", "rawMarkdown": "Can i ask if extracting more frames from videos achieve similar effect as augmentation? Thanks.", "votes": null }, { "id": "743480", "postDate": "02/12/2020 05:19:16", "content": "I face a similar situation. After 6-7 epochs the model starts overfitting. Also my CV/LB scores do not match. There is too much of a difference .
\n\nMy train/validation splits was based on real videos. This was to ensure that there is no overlap of faces in the train & valid set. But then I did a face clustering on the entire dataset (https://www.kaggle.com/hmendonca/proper-clustering-with-facenet-embeddings-eda) and realised that there was a huge overlap of faces in the train & valid!
I guess I'll just make a submission to check whats going on. Fingers crossed. Will report back if things change.
", "rawMarkdown": "I guess I'll just make a submission to check whats going on. Fingers crossed. Will report back if things change.", "votes": null }, { "id": "744435", "postDate": "02/12/2020 20:37:41", "content": "Well, actually even I'm considering writing some code to cluster similar faces together and use that information for train/val split. But thats actually a significant amount of work if it has to be done perfectly and unfortunately, I don't have a GPU to spare at the moment. =|
\n\nDid you manually sample check your clustering to see whether everything looked good?
", "rawMarkdown": "Well, actually even I'm considering writing some code to cluster similar faces together and use that information for train/val split. But thats actually a significant amount of work if it has to be done perfectly and unfortunately, I don't have a GPU to spare at the moment. =|\n\nDid you manually sample check your clustering to see whether everything looked good?", "votes": null }, { "id": "744572", "postDate": "02/12/2020 23:53:20", "content": "Modern day deep neural networks need the decision boundary (threshold) to be calibrated. As you train and tune the network it becomes overly confident in its decisions particularly if there is little to no regularisation. This overconfidence can manifest itself in fluctuating log loss on the validation set. There are different methods to calibrate the threshold. Suggest do some research.
", "rawMarkdown": "Modern day deep neural networks need the decision boundary (threshold) to be calibrated. As you train and tune the network it becomes overly confident in its decisions particularly if there is little to no regularisation. This overconfidence can manifest itself in fluctuating log loss on the validation set. There are different methods to calibrate the threshold. Suggest do some research.", "votes": null }, { "id": "744590", "postDate": "02/13/2020 00:36:15", "content": "One of the reasons this is a delightful competition is a lot of the traditional approaches to regularise and prevent overfitting are unhelpful here. If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.
\n\nI think the better approaches will probably be those that use models which have a more limited capability of overfitting and inherently will struggle to learn individual actors' faces, and instead have to rely on lower-level differences between videos.
", "rawMarkdown": "One of the reasons this is a delightful competition is a lot of the traditional approaches to regularise and prevent overfitting are unhelpful here. If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.\n\nI think the better approaches will probably be those that use models which have a more limited capability of overfitting and inherently will struggle to learn individual actors' faces, and instead have to rely on lower-level differences between videos.", "votes": null }, { "id": "744684", "postDate": "02/13/2020 03:12:32", "content": "\"If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.\"
\n\nBang on.
", "rawMarkdown": "\"If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.\" \n\nBang on.", "votes": null }, { "id": "744878", "postDate": "02/13/2020 08:40:19", "content": "There's a face clustering notebook by Herique Link
\n\nI have modified it by using the RAPIDs package from NVidia Link \nyou can use it to cluster the faces.
", "rawMarkdown": "There's a face clustering notebook by Herique [Link](https://www.kaggle.com/hmendonca/proper-clustering-with-facenet-embeddings-eda)\n\nI have modified it by using the RAPIDs package from NVidia [Link](https://www.kaggle.com/skylord/deepfake-face-clusters-using-rapids) \nyou can use it to cluster the faces.", "votes": null } ], "comments": [ { "id": 742653, "author_name": "humananalog", "author_url": "", "post_date": "02/11/2020 12:50:19", "content": "More augmentation really improved my score (but there's a limit to extreme you can make the augmentation).
", "votes": null, "replies": [ { "id": 743382, "author_name": "mieemiee", "author_url": "", "post_date": "02/12/2020 02:42:01", "content": "Can i ask if extracting more frames from videos achieve similar effect as augmentation? Thanks.
", "votes": null, "replies": [] } ] }, { "id": 743236, "author_name": "unkownhihi", "author_url": "", "post_date": "02/11/2020 23:01:13", "content": "I got my current score are just training 2 epochs...
", "votes": null, "replies": [ { "id": 744422, "author_name": "akashnandi", "author_url": "", "post_date": "02/12/2020 20:28:59", "content": "I guess I'll just make a submission to check whats going on. Fingers crossed. Will report back if things change.
", "votes": null, "replies": [] } ] }, { "id": 743480, "author_name": "skylord", "author_url": "", "post_date": "02/12/2020 05:19:16", "content": "I face a similar situation. After 6-7 epochs the model starts overfitting. Also my CV/LB scores do not match. There is too much of a difference .
\n\nMy train/validation splits was based on real videos. This was to ensure that there is no overlap of faces in the train & valid set. But then I did a face clustering on the entire dataset (https://www.kaggle.com/hmendonca/proper-clustering-with-facenet-embeddings-eda) and realised that there was a huge overlap of faces in the train & valid!
Well, actually even I'm considering writing some code to cluster similar faces together and use that information for train/val split. But thats actually a significant amount of work if it has to be done perfectly and unfortunately, I don't have a GPU to spare at the moment. =|
\n\nDid you manually sample check your clustering to see whether everything looked good?
", "votes": null, "replies": [] }, { "id": 744878, "author_name": "skylord", "author_url": "", "post_date": "02/13/2020 08:40:19", "content": "There's a face clustering notebook by Herique Link
\n\nI have modified it by using the RAPIDs package from NVidia Link \nyou can use it to cluster the faces.
", "votes": null, "replies": [] } ] }, { "id": 744572, "author_name": "maralski", "author_url": "", "post_date": "02/12/2020 23:53:20", "content": "Modern day deep neural networks need the decision boundary (threshold) to be calibrated. As you train and tune the network it becomes overly confident in its decisions particularly if there is little to no regularisation. This overconfidence can manifest itself in fluctuating log loss on the validation set. There are different methods to calibrate the threshold. Suggest do some research.
", "votes": null, "replies": [] }, { "id": 744590, "author_name": "jamesphoward", "author_url": "", "post_date": "02/13/2020 00:36:15", "content": "One of the reasons this is a delightful competition is a lot of the traditional approaches to regularise and prevent overfitting are unhelpful here. If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.
\n\nI think the better approaches will probably be those that use models which have a more limited capability of overfitting and inherently will struggle to learn individual actors' faces, and instead have to rely on lower-level differences between videos.
", "votes": null, "replies": [ { "id": 744684, "author_name": "akashnandi", "author_url": "", "post_date": "02/13/2020 03:12:32", "content": "\"If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.\"
\n\nBang on.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "742617": "Hey!\n\nThanks for looking into this.\nSo I have been training with a CNN network. My train set is based on 40 folders and val set is based on the remaining 10 folders. The image resolution being fed into the network is 224.\nI'm pretty sure the network is overfitting really quickly. What I see happening is that, for the first 2-4 epochs, both the val loss and the training loss starts dropping, usually around 0.1 for train loss and 0.37 for val loss by the end of 3/4th epoch. And after this, val loss jumps super high to around 0.8-1 and training loss just keeps going lower. Seems like major overfitting going on. My next attempt would obviously be to lower the damned learning rate by the end of 3rd epoch itself. But I just wanted to know if other LB toppers are facing similar situations of overfitting this quick during training?\n\nAnother really interesting thing going on is that the val loss just sits around the range of 0.8 with accuracy of 50% whenever I try image augmentation. The training loss keeps falling rapidly though. Extremely weird. \n\nPS: I wish I had a bunch of GPUs. =/\n\nThanks.\nAkash", "742653": "More augmentation really improved my score (but there's a limit to extreme you can make the augmentation).", "743236": "I got my current score are just training 2 epochs...", "743382": "Can i ask if extracting more frames from videos achieve similar effect as augmentation? Thanks.", "743480": "I face a similar situation. After 6-7 epochs the model starts overfitting. Also my CV/LB scores do not match. There is too much of a difference . \n\nMy train/validation splits was based on real videos. This was to ensure that there is no overlap of faces in the train & valid set. But then I did a face clustering on the entire dataset (`https://www.kaggle.com/hmendonca/proper-clustering-with-facenet-embeddings-eda`) and realised that there was a huge overlap of faces in the train & valid!", "744422": "I guess I'll just make a submission to check whats going on. Fingers crossed. Will report back if things change.", "744435": "Well, actually even I'm considering writing some code to cluster similar faces together and use that information for train/val split. But thats actually a significant amount of work if it has to be done perfectly and unfortunately, I don't have a GPU to spare at the moment. =|\n\nDid you manually sample check your clustering to see whether everything looked good?", "744572": "Modern day deep neural networks need the decision boundary (threshold) to be calibrated. As you train and tune the network it becomes overly confident in its decisions particularly if there is little to no regularisation. This overconfidence can manifest itself in fluctuating log loss on the validation set. There are different methods to calibrate the threshold. Suggest do some research.", "744590": "One of the reasons this is a delightful competition is a lot of the traditional approaches to regularise and prevent overfitting are unhelpful here. If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.\n\nI think the better approaches will probably be those that use models which have a more limited capability of overfitting and inherently will struggle to learn individual actors' faces, and instead have to rely on lower-level differences between videos.", "744684": "\"If we add too much distortion to the images, 'real' videos become synthetic, and the 'real' label is not longer appropriate.\" \n\nBang on.", "744878": "There's a face clustering notebook by Herique [Link](https://www.kaggle.com/hmendonca/proper-clustering-with-facenet-embeddings-eda)\n\nI have modified it by using the RAPIDs package from NVidia [Link](https://www.kaggle.com/skylord/deepfake-face-clusters-using-rapids) \nyou can use it to cluster the faces." }, "source": "meta" }