Looking at ahjnxtiamx.mp4 test video it's clearly fake.
\n\nWhen submitting a notebook with that label fake you would expect the LB score to be better than the 0.5 benchmark. However it's the same as the 0.5 benchmark (0.69314)! \nSee in this notebook: Are test labels wrong?
\n\nDoes anyone have an explanation for this?
\n\nIs the scoring performed on a random subset? \nIs the test set supposed to be reproducible when submitting the same result multiple times? If yes, are the labels wrong?
", "messages": [ { "id": "750270", "postDate": "02/19/2020 08:43:32", "content": "Looking at ahjnxtiamx.mp4 test video it's clearly fake.
\n\nWhen submitting a notebook with that label fake you would expect the LB score to be better than the 0.5 benchmark. However it's the same as the 0.5 benchmark (0.69314)! \nSee in this notebook: Are test labels wrong?
\n\nDoes anyone have an explanation for this?
\n\nIs the scoring performed on a random subset? \nIs the test set supposed to be reproducible when submitting the same result multiple times? If yes, are the labels wrong?
", "rawMarkdown": "Looking at ahjnxtiamx.mp4 test video it's clearly fake.\n\nWhen submitting a notebook with that label fake you would expect the LB score to be better than the 0.5 benchmark. However it's the same as the 0.5 benchmark (0.69314)! \n[See in this notebook: Are test labels wrong?](https://www.kaggle.com/dthrone/are-test-labels-wrong)\n\nDoes anyone have an explanation for this?\n\nIs the scoring performed on a random subset? \nIs the test set supposed to be reproducible when submitting the same result multiple times? If yes, are the labels wrong?", "votes": null }, { "id": "750279", "postDate": "02/19/2020 08:52:47", "content": "Test videos are a subsample from the full dataset. I would encourage you to use a wider test set (~20% == 10 parts) in order to cope with overfiting. Moreover, @harshitsheoran propose to use parts 40-50 to test here
", "rawMarkdown": "Test videos are a subsample from the full dataset. I would encourage you to use a wider test set (~20% == 10 parts) in order to cope with overfiting. Moreover, @harshitsheoran propose to use parts 40-50 to test [here](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/130915)", "votes": null }, { "id": "750289", "postDate": "02/19/2020 09:06:58", "content": "In getting started.\"3.Public Test Set: This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against. \"\nI think the public validation set is not included in the public test set.
", "rawMarkdown": "In getting started.\"3.Public Test Set: This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against. \"\nI think the public validation set is not included in the public test set.", "votes": null }, { "id": "750309", "postDate": "02/19/2020 09:20:53", "content": "Thank you, this is not what I have expected. Reading your comments I have the same conclusion. If the test_videos is a subset of the dataset the LB is calculated on the score should be different. The only explanation I have is that the test_videos files are not included in the set that the LB is calculated on.
", "rawMarkdown": "Thank you, this is not what I have expected. Reading your comments I have the same conclusion. If the test_videos is a subset of the dataset the LB is calculated on the score should be different. The only explanation I have is that the test_videos files are not included in the set that the LB is calculated on.", "votes": null }, { "id": "750322", "postDate": "02/19/2020 09:40:14", "content": "As @kueno55 has said in Getting Started you have all the information regarding to the different datasets.
", "rawMarkdown": "As @kueno55 has said in [Getting Started](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started) you have all the information regarding to the different datasets.", "votes": null }, { "id": "750424", "postDate": "02/19/2020 11:04:02", "content": "All of the test_videos are in the training data, so it would be very strange if they were also included in the LB test set. ;-)
Test videos are a subsample from the full dataset. I would encourage you to use a wider test set (~20% == 10 parts) in order to cope with overfiting. Moreover, @harshitsheoran propose to use parts 40-50 to test here
", "votes": null, "replies": [] }, { "id": 750289, "author_name": "kueno55", "author_url": "", "post_date": "02/19/2020 09:06:58", "content": "In getting started.\"3.Public Test Set: This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against. \"\nI think the public validation set is not included in the public test set.
", "votes": null, "replies": [ { "id": 750309, "author_name": "dthrone", "author_url": "", "post_date": "02/19/2020 09:20:53", "content": "Thank you, this is not what I have expected. Reading your comments I have the same conclusion. If the test_videos is a subset of the dataset the LB is calculated on the score should be different. The only explanation I have is that the test_videos files are not included in the set that the LB is calculated on.
", "votes": null, "replies": [] }, { "id": 750322, "author_name": "cayala", "author_url": "", "post_date": "02/19/2020 09:40:14", "content": "As @kueno55 has said in Getting Started you have all the information regarding to the different datasets.
", "votes": null, "replies": [] } ] }, { "id": 750424, "author_name": "humananalog", "author_url": "", "post_date": "02/19/2020 11:04:02", "content": "All of the test_videos are in the training data, so it would be very strange if they were also included in the LB test set. ;-)