I trained a CNN based model on the training data which is 90% split from the main dataset and used other 10% split as the validation model. My model achieved to a very good performance on the validation set but the submission log score is very high. I have taken care of the skew and my model is not biased for one class, at least as seen from train and validation splits. I want to know if the problem is:\n(1) Overfitting as the validation and train split set are very identical OR \n(2) The test set used by Kaggle is very different from the dataset provided for training(in terms of structure as in generated videos using different methods or augmentations)? \nI am new to field, can someone help me figure out how can I improve my model from here?
", "messages": [ { "id": "698715", "postDate": "12/19/2019 16:10:04", "content": "I trained a CNN based model on the training data which is 90% split from the main dataset and used other 10% split as the validation model. My model achieved to a very good performance on the validation set but the submission log score is very high. I have taken care of the skew and my model is not biased for one class, at least as seen from train and validation splits. I want to know if the problem is:\n(1) Overfitting as the validation and train split set are very identical OR \n(2) The test set used by Kaggle is very different from the dataset provided for training(in terms of structure as in generated videos using different methods or augmentations)? \nI am new to field, can someone help me figure out how can I improve my model from here?
", "rawMarkdown": "I trained a CNN based model on the training data which is 90% split from the main dataset and used other 10% split as the validation model. My model achieved to a very good performance on the validation set but the submission log score is very high. I have taken care of the skew and my model is not biased for one class, at least as seen from train and validation splits. I want to know if the problem is:\n(1) Overfitting as the validation and train split set are very identical OR \n(2) The test set used by Kaggle is very different from the dataset provided for training(in terms of structure as in generated videos using different methods or augmentations)? \nI am new to field, can someone help me figure out how can I improve my model from here?", "votes": null }, { "id": "698855", "postDate": "12/19/2019 19:24:39", "content": "Make sure that the videos in your validation set are not also in the training set. For example, if real video A was used to create fake videos B, C, and D, then you shouldn't have B and C in the validation set and A and D in the training set. Either all of them should go in the training set, or all in the validation set.
\n\nI think each of the 50 subdirectories with training videos is self-contained in this way, so you might try using subdir 0 as the validation set and 1-49 to train on. (Although I'm not sure how good that particular split is.)
", "rawMarkdown": "Make sure that the videos in your validation set are not also in the training set. For example, if real video A was used to create fake videos B, C, and D, then you shouldn't have B and C in the validation set and A and D in the training set. Either all of them should go in the training set, or all in the validation set.\n\nI think each of the 50 subdirectories with training videos is self-contained in this way, so you might try using subdir 0 as the validation set and 1-49 to train on. (Although I'm not sure how good that particular split is.)", "votes": null }, { "id": "698858", "postDate": "12/19/2019 19:29:51", "content": "Train -> validation leak if you didn''t split by original video or even actor?
", "rawMarkdown": "Train -> validation leak if you didn''t split by original video or even actor?", "votes": null }, { "id": "698883", "postDate": "12/19/2019 20:21:34", "content": "\n\n\nThe test set used by Kaggle is very different from the dataset provided for training
\n
\"Public Test Set\" has these augmentations:
\n\n(1) reduce the FPS of the video to 15\n(2) reduce the resolution of the video to 1/4 of its original size\n(3) reduce the overall encoding quality
\n\nnumber 3 is a big problem when a video is real and your model think it's fake because it has a lower quality
", "rawMarkdown": "> The test set used by Kaggle is very different from the dataset provided for training\n\n\"Public Test Set\" has these augmentations:\n\n(1) reduce the FPS of the video to 15\n(2) reduce the resolution of the video to 1/4 of its original size\n(3) reduce the overall encoding quality\n\nnumber 3 is a big problem when a video is real and your model think it's fake because it has a lower quality", "votes": null }, { "id": "698926", "postDate": "12/19/2019 22:34:27", "content": "Where did you get the info from?\nAlso beware of using the first dirs (big faces) or the last dirs (only audio modification) when validating.
", "rawMarkdown": "Where did you get the info from?\nAlso beware of using the first dirs (big faces) or the last dirs (only audio modification) when validating.", "votes": null }, { "id": "699091", "postDate": "12/20/2019 03:30:30", "content": "Look at this paper https://arxiv.org/abs/1910.08854
", "rawMarkdown": "Look at this paper https://arxiv.org/abs/1910.08854", "votes": null }, { "id": "699368", "postDate": "12/20/2019 11:06:34", "content": "Thank you. This was helpful.
", "rawMarkdown": "Thank you. This was helpful.", "votes": null } ], "comments": [ { "id": 698855, "author_name": "humananalog", "author_url": "", "post_date": "12/19/2019 19:24:39", "content": "Make sure that the videos in your validation set are not also in the training set. For example, if real video A was used to create fake videos B, C, and D, then you shouldn't have B and C in the validation set and A and D in the training set. Either all of them should go in the training set, or all in the validation set.
\n\nI think each of the 50 subdirectories with training videos is self-contained in this way, so you might try using subdir 0 as the validation set and 1-49 to train on. (Although I'm not sure how good that particular split is.)
", "votes": null, "replies": [] }, { "id": 698858, "author_name": "bacterio", "author_url": "", "post_date": "12/19/2019 19:29:51", "content": "Train -> validation leak if you didn''t split by original video or even actor?
", "votes": null, "replies": [] }, { "id": 698883, "author_name": "behrouzp", "author_url": "", "post_date": "12/19/2019 20:21:34", "content": "\n\n\nThe test set used by Kaggle is very different from the dataset provided for training
\n
\"Public Test Set\" has these augmentations:
\n\n(1) reduce the FPS of the video to 15\n(2) reduce the resolution of the video to 1/4 of its original size\n(3) reduce the overall encoding quality
\n\nnumber 3 is a big problem when a video is real and your model think it's fake because it has a lower quality
", "votes": null, "replies": [ { "id": 698926, "author_name": "simoninparis", "author_url": "", "post_date": "12/19/2019 22:34:27", "content": "Where did you get the info from?\nAlso beware of using the first dirs (big faces) or the last dirs (only audio modification) when validating.
", "votes": null, "replies": [] }, { "id": 699091, "author_name": "behrouzp", "author_url": "", "post_date": "12/20/2019 03:30:30", "content": "Look at this paper https://arxiv.org/abs/1910.08854
", "votes": null, "replies": [] }, { "id": 699368, "author_name": "ameypatil", "author_url": "", "post_date": "12/20/2019 11:06:34", "content": "Thank you. This was helpful.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "698715": "I trained a CNN based model on the training data which is 90% split from the main dataset and used other 10% split as the validation model. My model achieved to a very good performance on the validation set but the submission log score is very high. I have taken care of the skew and my model is not biased for one class, at least as seen from train and validation splits. I want to know if the problem is:\n(1) Overfitting as the validation and train split set are very identical OR \n(2) The test set used by Kaggle is very different from the dataset provided for training(in terms of structure as in generated videos using different methods or augmentations)? \nI am new to field, can someone help me figure out how can I improve my model from here?", "698855": "Make sure that the videos in your validation set are not also in the training set. For example, if real video A was used to create fake videos B, C, and D, then you shouldn't have B and C in the validation set and A and D in the training set. Either all of them should go in the training set, or all in the validation set.\n\nI think each of the 50 subdirectories with training videos is self-contained in this way, so you might try using subdir 0 as the validation set and 1-49 to train on. (Although I'm not sure how good that particular split is.)", "698858": "Train -> validation leak if you didn''t split by original video or even actor?", "698883": "> The test set used by Kaggle is very different from the dataset provided for training\n\n\"Public Test Set\" has these augmentations:\n\n(1) reduce the FPS of the video to 15\n(2) reduce the resolution of the video to 1/4 of its original size\n(3) reduce the overall encoding quality\n\nnumber 3 is a big problem when a video is real and your model think it's fake because it has a lower quality", "698926": "Where did you get the info from?\nAlso beware of using the first dirs (big faces) or the last dirs (only audio modification) when validating.", "699091": "Look at this paper https://arxiv.org/abs/1910.08854", "699368": "Thank you. This was helpful." }, "source": "meta" }