I trained the models using Xception Net and can reach around 0.38 loss for my local validation set. The validation set is splitted by the index of folders. According to some discussions, this way of splitting the train/validation set should track the LB in some degree. But whenever I upload model to the kernel, I always get a LB around 0.6.
\n\nCan anyone share the local validation loss you get when getting a good score on LB or give some insights about this in improving the score? Thanks!
", "messages": [ { "id": "758467", "postDate": "02/27/2020 19:35:14", "content": "I trained the models using Xception Net and can reach around 0.38 loss for my local validation set. The validation set is splitted by the index of folders. According to some discussions, this way of splitting the train/validation set should track the LB in some degree. But whenever I upload model to the kernel, I always get a LB around 0.6.
\n\nCan anyone share the local validation loss you get when getting a good score on LB or give some insights about this in improving the score? Thanks!
", "rawMarkdown": "I trained the models using Xception Net and can reach around 0.38 loss for my local validation set. The validation set is splitted by the index of folders. According to some discussions, this way of splitting the train/validation set should track the LB in some degree. But whenever I upload model to the kernel, I always get a LB around 0.6. \n\nCan anyone share the local validation loss you get when getting a good score on LB or give some insights about this in improving the score? Thanks!", "votes": null }, { "id": "758498", "postDate": "02/27/2020 20:40:01", "content": "Your model is overfitting the training data moreover the number of fake videos are 100,000 and real are 19,000 (something). So the model can show low loss just by outputting 1 (FAKE).
", "rawMarkdown": "Your model is overfitting the training data moreover the number of fake videos are 100,000 and real are 19,000 (something). So the model can show low loss just by outputting 1 (FAKE).", "votes": null }, { "id": "758503", "postDate": "02/27/2020 20:46:28", "content": "Sorry, I should have mentioned that I balanced the dataset by selecting one real video and one corresponding fake video.
", "rawMarkdown": "Sorry, I should have mentioned that I balanced the dataset by selecting one real video and one corresponding fake video.", "votes": null }, { "id": "758505", "postDate": "02/27/2020 20:50:02", "content": "I did it too, then you underfitted the model. You must balance the data by using REAL faces from other datasets.
", "rawMarkdown": "I did it too, then you underfitted the model. You must balance the data by using REAL faces from other datasets.", "votes": null }, { "id": "758506", "postDate": "02/27/2020 20:51:55", "content": "Do you mean external dataset?
", "rawMarkdown": "Do you mean external dataset?", "votes": null }, { "id": "758507", "postDate": "02/27/2020 20:53:52", "content": "Yeah, I used http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/
", "rawMarkdown": "Yeah, I used http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/", "votes": null }, { "id": "758508", "postDate": "02/27/2020 20:55:34", "content": "Same here, however my model is not overfitting. My score is not always exact 0.2, its just around in 0.15-0.2, but I am using completely different faces then are in my training set, I am also training all the models on data of which method I can not reveal but it is no data loss and not even overbalancing, its custom.
\n\nI think there is just no reason for this, you can still know how much your leaderboard can be and get a lot of insights of ensembling.
", "rawMarkdown": "Same here, however my model is not overfitting. My score is not always exact 0.2, its just around in 0.15-0.2, but I am using completely different faces then are in my training set, I am also training all the models on data of which method I can not reveal but it is no data loss and not even overbalancing, its custom.\n\nI think there is just no reason for this, you can still know how much your leaderboard can be and get a lot of insights of ensembling.", "votes": null }, { "id": "758514", "postDate": "02/27/2020 21:08:38", "content": "BTW, I hope you have read this from RULES.\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.
", "rawMarkdown": "BTW, I hope you have read this from RULES.\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.", "votes": null }, { "id": "758522", "postDate": "02/27/2020 21:32:04", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "758526", "postDate": "02/27/2020 21:51:00", "content": "Bro, I am going all out but no externel data.
", "rawMarkdown": "Bro, I am going all out but no externel data.", "votes": null }, { "id": "760135", "postDate": "02/29/2020 22:22:38", "content": "Observing the same pattern.
", "rawMarkdown": "Observing the same pattern.", "votes": null }, { "id": "760151", "postDate": "02/29/2020 22:49:44", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "762946", "postDate": "03/04/2020 00:53:17", "content": "I had an issue like that I found that my training set and inference images where different (bug) and my libraries versions didn't match, go over things like this just to make sure your bug free
", "rawMarkdown": "I had an issue like that I found that my training set and inference images where different (bug) and my libraries versions didn't match, go over things like this just to make sure your bug free", "votes": null }, { "id": "763057", "postDate": "03/04/2020 04:54:18", "content": "What do you mean by inference images?
", "rawMarkdown": "What do you mean by inference images?", "votes": null }, { "id": "763262", "postDate": "03/04/2020 09:59:15", "content": "the data set I used used dlib cropped faces and the bounding box was only showing the face area, while in my inference I cropped the face with bigger bounding box (got hair and bit of background) which i assume did affect my inference
", "rawMarkdown": "the data set I used used dlib cropped faces and the bounding box was only showing the face area, while in my inference I cropped the face with bigger bounding box (got hair and bit of background) which i assume did affect my inference", "votes": null }, { "id": "789477", "postDate": "03/28/2020 17:23:30", "content": "me too.\nIn my case, validation scores always reach below 0.3 for any local validation set created from several folders.\nHowever, public scores always around 0.5. To me, folder based validation is doubtful.
", "rawMarkdown": "me too.\nIn my case, validation scores always reach below 0.3 for any local validation set created from several folders.\nHowever, public scores always around 0.5. To me, folder based validation is doubtful.", "votes": null } ], "comments": [ { "id": 758498, "author_name": "", "author_url": "", "post_date": "02/27/2020 20:40:01", "content": "Your model is overfitting the training data moreover the number of fake videos are 100,000 and real are 19,000 (something). So the model can show low loss just by outputting 1 (FAKE).
", "votes": null, "replies": [ { "id": 758503, "author_name": "jeffexu", "author_url": "", "post_date": "02/27/2020 20:46:28", "content": "Sorry, I should have mentioned that I balanced the dataset by selecting one real video and one corresponding fake video.
", "votes": null, "replies": [] }, { "id": 758505, "author_name": "", "author_url": "", "post_date": "02/27/2020 20:50:02", "content": "I did it too, then you underfitted the model. You must balance the data by using REAL faces from other datasets.
", "votes": null, "replies": [] }, { "id": 758506, "author_name": "jeffexu", "author_url": "", "post_date": "02/27/2020 20:51:55", "content": "Do you mean external dataset?
", "votes": null, "replies": [] }, { "id": 758507, "author_name": "", "author_url": "", "post_date": "02/27/2020 20:53:52", "content": "Yeah, I used http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/
", "votes": null, "replies": [] }, { "id": 758522, "author_name": "jeffexu", "author_url": "", "post_date": "02/27/2020 21:32:04", "content": "Thanks for sharing!
", "votes": null, "replies": [] } ] }, { "id": 758508, "author_name": "harshitsheoran", "author_url": "", "post_date": "02/27/2020 20:55:34", "content": "Same here, however my model is not overfitting. My score is not always exact 0.2, its just around in 0.15-0.2, but I am using completely different faces then are in my training set, I am also training all the models on data of which method I can not reveal but it is no data loss and not even overbalancing, its custom.
\n\nI think there is just no reason for this, you can still know how much your leaderboard can be and get a lot of insights of ensembling.
", "votes": null, "replies": [ { "id": 758514, "author_name": "", "author_url": "", "post_date": "02/27/2020 21:08:38", "content": "BTW, I hope you have read this from RULES.\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.
", "votes": null, "replies": [] }, { "id": 758526, "author_name": "harshitsheoran", "author_url": "", "post_date": "02/27/2020 21:51:00", "content": "Bro, I am going all out but no externel data.
", "votes": null, "replies": [] } ] }, { "id": 760135, "author_name": "unkownhihi", "author_url": "", "post_date": "02/29/2020 22:22:38", "content": "Observing the same pattern.
", "votes": null, "replies": [ { "id": 760151, "author_name": "jeffexu", "author_url": "", "post_date": "02/29/2020 22:49:44", "content": "Thanks for sharing!
", "votes": null, "replies": [] } ] }, { "id": 762946, "author_name": "basharallabadi", "author_url": "", "post_date": "03/04/2020 00:53:17", "content": "I had an issue like that I found that my training set and inference images where different (bug) and my libraries versions didn't match, go over things like this just to make sure your bug free
", "votes": null, "replies": [ { "id": 763057, "author_name": "jeffexu", "author_url": "", "post_date": "03/04/2020 04:54:18", "content": "What do you mean by inference images?
", "votes": null, "replies": [] }, { "id": 763262, "author_name": "basharallabadi", "author_url": "", "post_date": "03/04/2020 09:59:15", "content": "the data set I used used dlib cropped faces and the bounding box was only showing the face area, while in my inference I cropped the face with bigger bounding box (got hair and bit of background) which i assume did affect my inference
", "votes": null, "replies": [] } ] }, { "id": 789477, "author_name": "shimizumasaki", "author_url": "", "post_date": "03/28/2020 17:23:30", "content": "me too.\nIn my case, validation scores always reach below 0.3 for any local validation set created from several folders.\nHowever, public scores always around 0.5. To me, folder based validation is doubtful.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "758467": "I trained the models using Xception Net and can reach around 0.38 loss for my local validation set. The validation set is splitted by the index of folders. According to some discussions, this way of splitting the train/validation set should track the LB in some degree. But whenever I upload model to the kernel, I always get a LB around 0.6. \n\nCan anyone share the local validation loss you get when getting a good score on LB or give some insights about this in improving the score? Thanks!", "758498": "Your model is overfitting the training data moreover the number of fake videos are 100,000 and real are 19,000 (something). So the model can show low loss just by outputting 1 (FAKE).", "758503": "Sorry, I should have mentioned that I balanced the dataset by selecting one real video and one corresponding fake video.", "758505": "I did it too, then you underfitted the model. You must balance the data by using REAL faces from other datasets.", "758506": "Do you mean external dataset?", "758507": "Yeah, I used http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/", "758508": "Same here, however my model is not overfitting. My score is not always exact 0.2, its just around in 0.15-0.2, but I am using completely different faces then are in my training set, I am also training all the models on data of which method I can not reveal but it is no data loss and not even overbalancing, its custom.\n\nI think there is just no reason for this, you can still know how much your leaderboard can be and get a lot of insights of ensembling.", "758514": "BTW, I hope you have read this from RULES.\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.", "758522": "Thanks for sharing!", "758526": "Bro, I am going all out but no externel data.", "760135": "Observing the same pattern.", "760151": "Thanks for sharing!", "762946": "I had an issue like that I found that my training set and inference images where different (bug) and my libraries versions didn't match, go over things like this just to make sure your bug free", "763057": "What do you mean by inference images?", "763262": "the data set I used used dlib cropped faces and the bounding box was only showing the face area, while in my inference I cropped the face with bigger bounding box (got hair and bit of background) which i assume did affect my inference", "789477": "me too.\nIn my case, validation scores always reach below 0.3 for any local validation set created from several folders.\nHowever, public scores always around 0.5. To me, folder based validation is doubtful." }, "source": "meta" }