In this kernel, I tried to figure out how many videos in the public test are failing for me, and what is the reason. So there are 27 videos that fail on conversion to PIL image. I tried the first and the last frame, and for the first frame - all videos are OK, so the problem appears closer to the end of those 27 videos.
\n\nLast time @juliaelliott commented on the submission errors she didn't mention corrupted videos, so just in case it is new to the organizers, can be nice to have a remark on that. As have already been proposed, can be good to have at least an example of such a video so we can incorporate it into the testing pipeline.
", "messages": [ { "id": "733272", "postDate": "01/30/2020 23:03:56", "content": "In this kernel, I tried to figure out how many videos in the public test are failing for me, and what is the reason. So there are 27 videos that fail on conversion to PIL image. I tried the first and the last frame, and for the first frame - all videos are OK, so the problem appears closer to the end of those 27 videos.
\n\nLast time @juliaelliott commented on the submission errors she didn't mention corrupted videos, so just in case it is new to the organizers, can be nice to have a remark on that. As have already been proposed, can be good to have at least an example of such a video so we can incorporate it into the testing pipeline.
", "rawMarkdown": "In [this kernel](https://www.kaggle.com/zaharch/public-test-errors), I tried to figure out how many videos in the public test are failing for me, and what is the reason. So there are 27 videos that fail on conversion to PIL image. I tried the first and the last frame, and for the first frame - all videos are OK, so the problem appears closer to the end of those 27 videos. \n\nLast time @juliaelliott commented [on the submission errors](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/126349#726575) she didn't mention corrupted videos, so just in case it is new to the organizers, can be nice to have a remark on that. As have already been proposed, can be good to have at least an example of such a video so we can incorporate it into the testing pipeline.", "votes": null }, { "id": "733584", "postDate": "01/31/2020 10:22:15", "content": "Awesome work. If they don't want to rejig the public testing set, then just reassurance it wont happen on the private test set would be helpful.
", "rawMarkdown": "Awesome work. If they don't want to rejig the public testing set, then just reassurance it wont happen on the private test set would be helpful.", "votes": null }, { "id": "733963", "postDate": "01/31/2020 18:38:10", "content": "The final submission (as I read the over view) will be a complexly different set of videos than the ones currently being used for the leader board scoring.
\n\nThose videos sound like they will have real and fake from the wild - the videos also are not under the control of Kaggle. So I do not believe they can offer any assurance regarding quality control on that set of videos.
\n\nMy belief is that lots of new things may happen with that final set of data. On ALL of the competitions that I have been involved in over the last two years there was always stuff wrong with data from the wild. Most often many links were no longer valid.
\n\nSo I would plan on trapping all the possible failure modes you can think of and that you should have 100% confidence that corrupt videos and other high crimes and misdemeanors will be present in that very last scoring.
", "rawMarkdown": "The final submission (as I read the over view) will be a complexly different set of videos than the ones currently being used for the leader board scoring.\n\nThose videos sound like they will have real and fake from the wild - the videos also are not under the control of Kaggle. So I do not believe they can offer any assurance regarding quality control on that set of videos.\n\nMy belief is that lots of new things may happen with that final set of data. On ALL of the competitions that I have been involved in over the last two years there was always stuff wrong with data from the wild. Most often many links were no longer valid.\n\nSo I would plan on trapping all the possible failure modes you can think of and that you should have 100% confidence that corrupt videos and other high crimes and misdemeanors will be present in that very last scoring.", "votes": null }, { "id": "734160", "postDate": "02/01/2020 03:10:23", "content": "@zaharch I forked your kernel to confirm if all of the corrupted videos are fake. But when I submit it, I always got \"Submission Scoring Error\". Does anyone knows the reason? Here is the link.
", "rawMarkdown": "zaharch I forked your kernel to confirm if all of the corrupted videos are fake. But when I submit it, I always got \"Submission Scoring Error\". Does anyone knows the reason? Here is the [link](https://www.kaggle.com/wuliaokaola/public-test-set-probe).", "votes": null }, { "id": "734381", "postDate": "02/01/2020 11:36:15", "content": "Cool idea! The problem in the kernel is that filename is a full path. Also, it is not an index, so you are basically can not do submission.loc[filename, 'label'].
Got it. Thank you. I'll try it again tomorrow.
", "rawMarkdown": "Got it. Thank you. I'll try it again tomorrow.", "votes": null } ], "comments": [ { "id": 733584, "author_name": "jamesphoward", "author_url": "", "post_date": "01/31/2020 10:22:15", "content": "Awesome work. If they don't want to rejig the public testing set, then just reassurance it wont happen on the private test set would be helpful.
", "votes": null, "replies": [ { "id": 733963, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/31/2020 18:38:10", "content": "The final submission (as I read the over view) will be a complexly different set of videos than the ones currently being used for the leader board scoring.
\n\nThose videos sound like they will have real and fake from the wild - the videos also are not under the control of Kaggle. So I do not believe they can offer any assurance regarding quality control on that set of videos.
\n\nMy belief is that lots of new things may happen with that final set of data. On ALL of the competitions that I have been involved in over the last two years there was always stuff wrong with data from the wild. Most often many links were no longer valid.
\n\nSo I would plan on trapping all the possible failure modes you can think of and that you should have 100% confidence that corrupt videos and other high crimes and misdemeanors will be present in that very last scoring.
", "votes": null, "replies": [] } ] }, { "id": 734160, "author_name": "wuliaokaola", "author_url": "", "post_date": "02/01/2020 03:10:23", "content": "@zaharch I forked your kernel to confirm if all of the corrupted videos are fake. But when I submit it, I always got \"Submission Scoring Error\". Does anyone knows the reason? Here is the link.
", "votes": null, "replies": [ { "id": 734381, "author_name": "zaharch", "author_url": "", "post_date": "02/01/2020 11:36:15", "content": "Cool idea! The problem in the kernel is that filename is a full path. Also, it is not an index, so you are basically can not do submission.loc[filename, 'label'].
Got it. Thank you. I'll try it again tomorrow.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "733272": "In [this kernel](https://www.kaggle.com/zaharch/public-test-errors), I tried to figure out how many videos in the public test are failing for me, and what is the reason. So there are 27 videos that fail on conversion to PIL image. I tried the first and the last frame, and for the first frame - all videos are OK, so the problem appears closer to the end of those 27 videos. \n\nLast time @juliaelliott commented [on the submission errors](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/126349#726575) she didn't mention corrupted videos, so just in case it is new to the organizers, can be nice to have a remark on that. As have already been proposed, can be good to have at least an example of such a video so we can incorporate it into the testing pipeline.", "733584": "Awesome work. If they don't want to rejig the public testing set, then just reassurance it wont happen on the private test set would be helpful.", "733963": "The final submission (as I read the over view) will be a complexly different set of videos than the ones currently being used for the leader board scoring.\n\nThose videos sound like they will have real and fake from the wild - the videos also are not under the control of Kaggle. So I do not believe they can offer any assurance regarding quality control on that set of videos.\n\nMy belief is that lots of new things may happen with that final set of data. On ALL of the competitions that I have been involved in over the last two years there was always stuff wrong with data from the wild. Most often many links were no longer valid.\n\nSo I would plan on trapping all the possible failure modes you can think of and that you should have 100% confidence that corrupt videos and other high crimes and misdemeanors will be present in that very last scoring.", "734160": "zaharch I forked your kernel to confirm if all of the corrupted videos are fake. But when I submit it, I always got \"Submission Scoring Error\". Does anyone knows the reason? Here is the [link](https://www.kaggle.com/wuliaokaola/public-test-set-probe).", "734381": "Cool idea! The problem in the kernel is that `filename` is a full path. Also, it is not an index, so you are basically can not do `submission.loc[filename, 'label']`.", "734390": "Got it. Thank you. I'll try it again tomorrow." }, "source": "meta" }