I noticed that my pretrained resnets quickly overfit the data. After processing about 65k images (half real half fake), my validation loss starts to go up.
\n\nI'm using horizontal flip and shiftscalerotate from albumentations.
\n\nDo you have any tips for improving data augmentation or other parts of the pipeline to reduce overfitting?
", "messages": [ { "id": 737753, "postDate": "2020-02-05T18:13:18.320Z", "content": "I noticed that my pretrained resnets quickly overfit the data. After processing about 65k images (half real half fake), my validation loss starts to go up.
\n\nI'm using horizontal flip and shiftscalerotate from albumentations.
\n\nDo you have any tips for improving data augmentation or other parts of the pipeline to reduce overfitting?
", "rawMarkdown": "I noticed that my pretrained resnets quickly overfit the data. After processing about 65k images (half real half fake), my validation loss starts to go up. \n\nI'm using horizontal flip and shiftscalerotate from albumentations. \n\nDo you have any tips for improving data augmentation or other parts of the pipeline to reduce overfitting? ", "votes": 2 }, { "id": 744549, "postDate": "2020-02-12T23:29:50.397Z", "content": "For me, I just trained two epochs for my current best score so overfitting is not a problem. BTW, there's so many training data so the possibility of overfitting is pretty low.
", "rawMarkdown": "For me, I just trained two epochs for my current best score so overfitting is not a problem. BTW, there's so many training data so the possibility of overfitting is pretty low.", "replies": [ { "id": 744567, "postDate": "2020-02-12T23:49:29.573Z", "content": "@unkownhihi For me its 3-4 epochs. But sometimes I wonder if I am missing the full potential of the network by not being able to train further. I don't know. :(
", "rawMarkdown": "@unkownhihi For me its 3-4 epochs. But sometimes I wonder if I am missing the full potential of the network by not being able to train further. I don't know. :(" }, { "id": 765530, "postDate": "2020-03-06T18:38:35.927Z", "content": "How big is your training and validation set? Are you using all the data for training? Thanks!
", "rawMarkdown": "How big is your training and validation set? Are you using all the data for training? Thanks!" } ] }, { "id": 741301, "postDate": "2020-02-10T13:04:00.420Z", "content": "Typically , more trainning data will be helpful.
", "rawMarkdown": "Typically , more trainning data will be helpful." }, { "id": 741138, "postDate": "2020-02-10T08:30:00.633Z", "content": "Maybe give up some Neurons can deal with.\nAdd \"Dropout\" in your model.\n-> model.add(Dropout(N)) #give up N% Neurons\n((Sorry, I just a beginner ~~
", "rawMarkdown": "Maybe give up some Neurons can deal with.\nAdd \"Dropout\" in your model.\n-> model.add(Dropout(N)) #give up N% Neurons\n((Sorry, I just a beginner ~~ " }, { "id": 737906, "postDate": "2020-02-05T22:28:36.907Z", "content": "Probably you should try some regularization on the data and also use some Earlystopper as some also suggested. that helps so avoid overtting an and stop the training when appropriate.
", "rawMarkdown": "Probably you should try some regularization on the data and also use some Earlystopper as some also suggested. that helps so avoid overtting an and stop the training when appropriate." }, { "id": 737819, "postDate": "2020-02-05T20:04:35.607Z", "content": "It is a problem. My CV decreases almost in lockstep with the training loss - usually indicative of train/cv leakage although I have taken steps top avoid this. Currently, I am importing images from Face Forensics to create a truly independent CV but so far the losses are very high - 0.8 or so. Anybody else tried this?
", "rawMarkdown": "It is a problem. My CV decreases almost in lockstep with the training loss - usually indicative of train/cv leakage although I have taken steps top avoid this. Currently, I am importing images from Face Forensics to create a truly independent CV but so far the losses are very high - 0.8 or so. Anybody else tried this?", "replies": [ { "id": 737824, "postDate": "2020-02-05T20:18:13.463Z", "content": "My recommendation for cv is to use chunk 0 as your valid set.
\n\nChunk 0 only has one actor and that actor doesn't appear in other chunks.
\n\nIf you cv across all chunks, you will find that some actors appear in different chunks of data. This will cause your network to memorize that actor and your validation data includes that actor.
", "rawMarkdown": "My recommendation for cv is to use chunk 0 as your valid set. \n\nChunk 0 only has one actor and that actor doesn't appear in other chunks. \n\nIf you cv across all chunks, you will find that some actors appear in different chunks of data. This will cause your network to memorize that actor and your validation data includes that actor. ", "votes": 1 }, { "id": 737899, "postDate": "2020-02-05T22:04:17.413Z", "content": "Does your LB agree with your CV?
", "rawMarkdown": "Does your LB agree with your CV?", "votes": 1 } ] }, { "id": 737807, "postDate": "2020-02-05T19:42:19.337Z", "content": "From my experience, overfitting occurs pretty quickly, even for relatively simple models. The thing that helps the most: more data. I already noticed a big shift between half the data and all the data which becomes smoother. One frame from each video is also better than two frames from half the videos. Of course, complex models might still overfit on irrelevant details.
", "rawMarkdown": "From my experience, overfitting occurs pretty quickly, even for relatively simple models. The thing that helps the most: more data. I already noticed a big shift between half the data and all the data which becomes smoother. One frame from each video is also better than two frames from half the videos. Of course, complex models might still overfit on irrelevant details." }, { "id": 737758, "postDate": "2020-02-05T18:31:53.110Z", "content": "Usually, cross-validation should do the job. There are some other ways to recognize overfitting too: stopping your algorithm early, removing features, or training with different subsets of your data.
\n\nHere's an article that gives some good examples: https://elitedatascience.com/overfitting-in-machine-learning
", "rawMarkdown": "Usually, cross-validation should do the job. There are some other ways to recognize overfitting too: stopping your algorithm early, removing features, or training with different subsets of your data.\n\nHere's an article that gives some good examples: https://elitedatascience.com/overfitting-in-machine-learning", "replies": [ { "id": 737789, "postDate": "2020-02-05T19:15:09.360Z", "content": "Thank you for your response Gus. I have recognized that overfitting is occurring and my main concern is to prevent overfitting and continue to train the model for longer to get a better loss.
\n\nLet me know if you have any tips on managing an model and dataset that tends to overfit after identifying when it overfits. Perhaps augmentation?
\n\nMy guess is that my model is starting to memorize features specific to the actors in the training set and those features don't help in the validation set. This would explain my overfitting dilemma, where train loss improves but val loss gets worse.
", "rawMarkdown": "Thank you for your response Gus. I have recognized that overfitting is occurring and my main concern is to prevent overfitting and continue to train the model for longer to get a better loss.\n\nLet me know if you have any tips on managing an model and dataset that tends to overfit after identifying when it overfits. Perhaps augmentation? \n\nMy guess is that my model is starting to memorize features specific to the actors in the training set and those features don't help in the validation set. This would explain my overfitting dilemma, where train loss improves but val loss gets worse. \n" } ] }, { "id": 741184, "postDate": "2020-02-10T09:56:27.307Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 744549, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2020-02-12T23:29:50.397000", "content": "For me, I just trained two epochs for my current best score so overfitting is not a problem. BTW, there's so many training data so the possibility of overfitting is pretty low.
", "votes": 0, "replies": [ { "id": 744567, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-02-12T23:49:29.573000", "content": "@unkownhihi For me its 3-4 epochs. But sometimes I wonder if I am missing the full potential of the network by not being able to train further. I don't know. :(
", "votes": 0, "replies": [] }, { "id": 765530, "author_name": "ipekuyguner", "author_url": "", "post_date": "2020-03-06T18:38:35.927000", "content": "How big is your training and validation set? Are you using all the data for training? Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 741301, "author_name": "哈尔的移动城堡", "author_url": "", "post_date": "2020-02-10T13:04:00.420000", "content": "Typically , more trainning data will be helpful.
", "votes": 0, "replies": [] }, { "id": 741138, "author_name": "ZEOxOx", "author_url": "", "post_date": "2020-02-10T08:30:00.633000", "content": "Maybe give up some Neurons can deal with.\nAdd \"Dropout\" in your model.\n-> model.add(Dropout(N)) #give up N% Neurons\n((Sorry, I just a beginner ~~
", "votes": 0, "replies": [] }, { "id": 737906, "author_name": "Harphies", "author_url": "", "post_date": "2020-02-05T22:28:36.907000", "content": "Probably you should try some regularization on the data and also use some Earlystopper as some also suggested. that helps so avoid overtting an and stop the training when appropriate.
", "votes": 0, "replies": [] }, { "id": 737819, "author_name": "pete", "author_url": "", "post_date": "2020-02-05T20:04:35.607000", "content": "It is a problem. My CV decreases almost in lockstep with the training loss - usually indicative of train/cv leakage although I have taken steps top avoid this. Currently, I am importing images from Face Forensics to create a truly independent CV but so far the losses are very high - 0.8 or so. Anybody else tried this?
", "votes": 0, "replies": [ { "id": 737824, "author_name": "hongy", "author_url": "", "post_date": "2020-02-05T20:18:13.463000", "content": "My recommendation for cv is to use chunk 0 as your valid set.
\n\nChunk 0 only has one actor and that actor doesn't appear in other chunks.
\n\nIf you cv across all chunks, you will find that some actors appear in different chunks of data. This will cause your network to memorize that actor and your validation data includes that actor.
", "votes": 1, "replies": [] }, { "id": 737899, "author_name": "pete", "author_url": "", "post_date": "2020-02-05T22:04:17.413000", "content": "Does your LB agree with your CV?
", "votes": 1, "replies": [] } ] }, { "id": 737807, "author_name": "dagnelies", "author_url": "", "post_date": "2020-02-05T19:42:19.337000", "content": "From my experience, overfitting occurs pretty quickly, even for relatively simple models. The thing that helps the most: more data. I already noticed a big shift between half the data and all the data which becomes smoother. One frame from each video is also better than two frames from half the videos. Of course, complex models might still overfit on irrelevant details.
", "votes": 0, "replies": [] }, { "id": 737758, "author_name": "prose", "author_url": "", "post_date": "2020-02-05T18:31:53.110000", "content": "Usually, cross-validation should do the job. There are some other ways to recognize overfitting too: stopping your algorithm early, removing features, or training with different subsets of your data.
\n\nHere's an article that gives some good examples: https://elitedatascience.com/overfitting-in-machine-learning
", "votes": 0, "replies": [ { "id": 737789, "author_name": "hongy", "author_url": "", "post_date": "2020-02-05T19:15:09.360000", "content": "Thank you for your response Gus. I have recognized that overfitting is occurring and my main concern is to prevent overfitting and continue to train the model for longer to get a better loss.
\n\nLet me know if you have any tips on managing an model and dataset that tends to overfit after identifying when it overfits. Perhaps augmentation?
\n\nMy guess is that my model is starting to memorize features specific to the actors in the training set and those features don't help in the validation set. This would explain my overfitting dilemma, where train loss improves but val loss gets worse.
", "votes": 0, "replies": [] } ] }, { "id": 741184, "author_name": "", "author_url": "", "post_date": "2020-02-10T09:56:27.307000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "737753": "I noticed that my pretrained resnets quickly overfit the data. After processing about 65k images (half real half fake), my validation loss starts to go up. \n\nI'm using horizontal flip and shiftscalerotate from albumentations. \n\nDo you have any tips for improving data augmentation or other parts of the pipeline to reduce overfitting? ", "744549": "For me, I just trained two epochs for my current best score so overfitting is not a problem. BTW, there's so many training data so the possibility of overfitting is pretty low.", "741301": "Typically , more trainning data will be helpful.", "741138": "Maybe give up some Neurons can deal with.\nAdd \"Dropout\" in your model.\n-> model.add(Dropout(N)) #give up N% Neurons\n((Sorry, I just a beginner ~~ ", "737906": "Probably you should try some regularization on the data and also use some Earlystopper as some also suggested. that helps so avoid overtting an and stop the training when appropriate.", "737819": "It is a problem. My CV decreases almost in lockstep with the training loss - usually indicative of train/cv leakage although I have taken steps top avoid this. Currently, I am importing images from Face Forensics to create a truly independent CV but so far the losses are very high - 0.8 or so. Anybody else tried this?", "737807": "From my experience, overfitting occurs pretty quickly, even for relatively simple models. The thing that helps the most: more data. I already noticed a big shift between half the data and all the data which becomes smoother. One frame from each video is also better than two frames from half the videos. Of course, complex models might still overfit on irrelevant details.", "737758": "Usually, cross-validation should do the job. There are some other ways to recognize overfitting too: stopping your algorithm early, removing features, or training with different subsets of your data.\n\nHere's an article that gives some good examples: https://elitedatascience.com/overfitting-in-machine-learning", "741184": "" } }