Despite using the 75% of the available data, dropout and batch normalisation layers, and a low learning rate (as many people has suggested) my model isn't training. When using imagenet weights (withoud freezing the encoder) it converges too fast giving poor results. On the other hand if I don't use those weights and train the model end-to-end the loss keeps allways the same. Moreover, I have invested a sheer number of hours developing a custom Data Generator in order to balance the real/fake samples.
\n\nWhat could I have been doing wrong? any idea? Thank you in advance.
", "messages": [ { "id": "761229", "postDate": "03/02/2020 09:41:56", "content": "Despite using the 75% of the available data, dropout and batch normalisation layers, and a low learning rate (as many people has suggested) my model isn't training. When using imagenet weights (withoud freezing the encoder) it converges too fast giving poor results. On the other hand if I don't use those weights and train the model end-to-end the loss keeps allways the same. Moreover, I have invested a sheer number of hours developing a custom Data Generator in order to balance the real/fake samples.
\n\nWhat could I have been doing wrong? any idea? Thank you in advance.
", "rawMarkdown": "Despite using the 75% of the available data, dropout and batch normalisation layers, and a low learning rate (as many people has suggested) my model isn't training. When using imagenet weights (withoud freezing the encoder) it converges too fast giving poor results. On the other hand if I don't use those weights and train the model end-to-end the loss keeps allways the same. Moreover, I have invested a sheer number of hours developing a custom Data Generator in order to balance the real/fake samples.\n\nWhat could I have been doing wrong? any idea? Thank you in advance.", "votes": null }, { "id": "761262", "postDate": "03/02/2020 10:25:25", "content": "If the model \"converges too fast\", it means it's certainly learning something -- but not the thing you're expecting it to learn.
", "rawMarkdown": "If the model \"converges too fast\", it means it's certainly learning something -- but not the thing you're expecting it to learn.", "votes": null }, { "id": "761329", "postDate": "03/02/2020 11:50:00", "content": "Maybie \"converge\" is not the proper word to the current situation. It behaves as if I were using a random predictor (loss = 0.693).
", "rawMarkdown": "Maybie \"converge\" is not the proper word to the current situation. It behaves as if I were using a random predictor (loss = 0.693).", "votes": null }, { "id": "761357", "postDate": "03/02/2020 12:29:10", "content": "I've seen this too in some of my experiments, usually when I tried to do something clever with the loss function.
\n\nI'd suggest simplifying your code. Try to make the model overfit on a very small dataset (i.e. just a single batch trained for several epochs).
\n\nThere are a number of neural network troubleshooting guides on the web that have good tips for diagnosing such issues.
", "rawMarkdown": "I've seen this too in some of my experiments, usually when I tried to do something clever with the loss function. \n\nI'd suggest simplifying your code. Try to make the model overfit on a very small dataset (i.e. just a single batch trained for several epochs).\n\nThere are a number of neural network troubleshooting guides on the web that have good tips for diagnosing such issues.", "votes": null }, { "id": "761367", "postDate": "03/02/2020 12:43:56", "content": "I will follow you advice and I will try to make the model overfit on a small dataset, thank you!
", "rawMarkdown": "I will follow you advice and I will try to make the model overfit on a small dataset, thank you!", "votes": null }, { "id": "761505", "postDate": "03/02/2020 15:51:25", "content": "Sounds like a mislabeling issue, remember not all frames in a fake video are fake. If you train on every frame from a fake video and have them all labeled as fake then your model will not be able to learn the difference between real and fake frames. So random guess ~0.69 is the best loss you will get.
", "rawMarkdown": "Sounds like a mislabeling issue, remember not all frames in a fake video are fake. If you train on every frame from a fake video and have them all labeled as fake then your model will not be able to learn the difference between real and fake frames. So random guess ~0.69 is the best loss you will get.", "votes": null }, { "id": "761524", "postDate": "03/02/2020 16:12:55", "content": "Counter point: After I cleaned up my dataset and removed all the fake frames that were not true fakes (around 10% of the total), the performance of my model got worse... Noisy data isn't always a bad thing.
", "rawMarkdown": "Counter point: After I cleaned up my dataset and removed all the fake frames that were not true fakes (around 10% of the total), the performance of my model got worse... Noisy data isn't always a bad thing.", "votes": null }, { "id": "761571", "postDate": "03/02/2020 17:23:34", "content": "In my opinion misslabeled frames won't be a problem. I'm considering multiple frames per video, the model should be capable of learning the difference between real and fakes by its own, making it more robust to noisy data.
", "rawMarkdown": "In my opinion misslabeled frames won't be a problem. I'm considering multiple frames per video, the model should be capable of learning the difference between real and fakes by its own, making it more robust to noisy data.", "votes": null }, { "id": "761581", "postDate": "03/02/2020 17:33:58", "content": "@humananalog Interesting, and I will have to re-evaluate what I am considering a fake frame because I was averaging about ~25 fake frames per fake video (some had much more ~200, some had much less ~5).
\n\nWhat I done was compare the hashes of each frame in a fake video to it's corresponding original frame, if the hashes were different I consider it a fake. Going by your comment my approach sounds flawed.
", "rawMarkdown": "humananalog Interesting, and I will have to re-evaluate what I am considering a fake frame because I was averaging about ~25 fake frames per fake video (some had much more ~200, some had much less ~5). \n\nWhat I done was compare the hashes of each frame in a fake video to it's corresponding original frame, if the hashes were different I consider it a fake. Going by your comment my approach sounds flawed.", "votes": null } ], "comments": [ { "id": 761262, "author_name": "humananalog", "author_url": "", "post_date": "03/02/2020 10:25:25", "content": "If the model \"converges too fast\", it means it's certainly learning something -- but not the thing you're expecting it to learn.
", "votes": null, "replies": [ { "id": 761329, "author_name": "cayala", "author_url": "", "post_date": "03/02/2020 11:50:00", "content": "Maybie \"converge\" is not the proper word to the current situation. It behaves as if I were using a random predictor (loss = 0.693).
", "votes": null, "replies": [] }, { "id": 761505, "author_name": "jackvial", "author_url": "", "post_date": "03/02/2020 15:51:25", "content": "Sounds like a mislabeling issue, remember not all frames in a fake video are fake. If you train on every frame from a fake video and have them all labeled as fake then your model will not be able to learn the difference between real and fake frames. So random guess ~0.69 is the best loss you will get.
", "votes": null, "replies": [] }, { "id": 761524, "author_name": "humananalog", "author_url": "", "post_date": "03/02/2020 16:12:55", "content": "Counter point: After I cleaned up my dataset and removed all the fake frames that were not true fakes (around 10% of the total), the performance of my model got worse... Noisy data isn't always a bad thing.
", "votes": null, "replies": [] }, { "id": 761581, "author_name": "jackvial", "author_url": "", "post_date": "03/02/2020 17:33:58", "content": "@humananalog Interesting, and I will have to re-evaluate what I am considering a fake frame because I was averaging about ~25 fake frames per fake video (some had much more ~200, some had much less ~5).
\n\nWhat I done was compare the hashes of each frame in a fake video to it's corresponding original frame, if the hashes were different I consider it a fake. Going by your comment my approach sounds flawed.
", "votes": null, "replies": [] } ] }, { "id": 761357, "author_name": "humananalog", "author_url": "", "post_date": "03/02/2020 12:29:10", "content": "I've seen this too in some of my experiments, usually when I tried to do something clever with the loss function.
\n\nI'd suggest simplifying your code. Try to make the model overfit on a very small dataset (i.e. just a single batch trained for several epochs).
\n\nThere are a number of neural network troubleshooting guides on the web that have good tips for diagnosing such issues.
", "votes": null, "replies": [ { "id": 761367, "author_name": "cayala", "author_url": "", "post_date": "03/02/2020 12:43:56", "content": "I will follow you advice and I will try to make the model overfit on a small dataset, thank you!
", "votes": null, "replies": [] } ] }, { "id": 761571, "author_name": "cayala", "author_url": "", "post_date": "03/02/2020 17:23:34", "content": "In my opinion misslabeled frames won't be a problem. I'm considering multiple frames per video, the model should be capable of learning the difference between real and fakes by its own, making it more robust to noisy data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "761229": "Despite using the 75% of the available data, dropout and batch normalisation layers, and a low learning rate (as many people has suggested) my model isn't training. When using imagenet weights (withoud freezing the encoder) it converges too fast giving poor results. On the other hand if I don't use those weights and train the model end-to-end the loss keeps allways the same. Moreover, I have invested a sheer number of hours developing a custom Data Generator in order to balance the real/fake samples.\n\nWhat could I have been doing wrong? any idea? Thank you in advance.", "761262": "If the model \"converges too fast\", it means it's certainly learning something -- but not the thing you're expecting it to learn.", "761329": "Maybie \"converge\" is not the proper word to the current situation. It behaves as if I were using a random predictor (loss = 0.693).", "761357": "I've seen this too in some of my experiments, usually when I tried to do something clever with the loss function. \n\nI'd suggest simplifying your code. Try to make the model overfit on a very small dataset (i.e. just a single batch trained for several epochs).\n\nThere are a number of neural network troubleshooting guides on the web that have good tips for diagnosing such issues.", "761367": "I will follow you advice and I will try to make the model overfit on a small dataset, thank you!", "761505": "Sounds like a mislabeling issue, remember not all frames in a fake video are fake. If you train on every frame from a fake video and have them all labeled as fake then your model will not be able to learn the difference between real and fake frames. So random guess ~0.69 is the best loss you will get.", "761524": "Counter point: After I cleaned up my dataset and removed all the fake frames that were not true fakes (around 10% of the total), the performance of my model got worse... Noisy data isn't always a bad thing.", "761571": "In my opinion misslabeled frames won't be a problem. I'm considering multiple frames per video, the model should be capable of learning the difference between real and fakes by its own, making it more robust to noisy data.", "761581": "humananalog Interesting, and I will have to re-evaluate what I am considering a fake frame because I was averaging about ~25 fake frames per fake video (some had much more ~200, some had much less ~5). \n\nWhat I done was compare the hashes of each frame in a fake video to it's corresponding original frame, if the hashes were different I consider it a fake. Going by your comment my approach sounds flawed." }, "source": "meta" }