@zaharch found that there are 27 corrupted videos in the public test set. And this kernel can get LB 0.68846 without any model. That means all of the corrupted videos are fake. \n0.69314 * (4000 - 27) / 4000 = 0.68846\nI think when they generated deepfake videos, something went wrong that made video corrupted. @philculliton I don't know how about the private test set, but I think this is a leakage of public test set.
@zaharch found that there are 27 corrupted videos in the public test set. And this kernel can get LB 0.68846 without any model. That means all of the corrupted videos are fake. \n0.69314 * (4000 - 27) / 4000 = 0.68846\nI think when they generated deepfake videos, something went wrong that made video corrupted. @philculliton I don't know how about the private test set, but I think this is a leakage of public test set.
In many of the shared kernels when an error occurs most folks are using 0.5 as the default probability to use.
\n\nThis suggests that a value close to 1 would make more sense when a file error occurs. Many might have already decided that 0.5 was not the best to use for error default - I have been using 0.75 myself.
\n\nNot sure that 27/4000 is a significant leak however.
", "rawMarkdown": "In many of the shared kernels when an error occurs most folks are using 0.5 as the default probability to use.\n\nThis suggests that a value close to 1 would make more sense when a file error occurs. Many might have already decided that 0.5 was not the best to use for error default - I have been using 0.75 myself.\n\nNot sure that 27/4000 is a significant leak however.", "votes": null }, { "id": "736122", "postDate": "02/03/2020 21:01:57", "content": "This is one of those interesting situations where there's a real risk of overfitting to the public leaderboard. I currently predict 0.5 on any video that fails. I suspect my LB score could be improved by picking a higher score for failures, but with the logarithmic loss function there's a risk from it, too.\nI suspect failures are more likely to be deepfakes even in the final private test set, but it'd be better if we could get reassurance that such errors would be cleaned up prior?
", "rawMarkdown": "This is one of those interesting situations where there's a real risk of overfitting to the public leaderboard. I currently predict 0.5 on any video that fails. I suspect my LB score could be improved by picking a higher score for failures, but with the logarithmic loss function there's a risk from it, too.\nI suspect failures are more likely to be deepfakes even in the final private test set, but it'd be better if we could get reassurance that such errors would be cleaned up prior?", "votes": null }, { "id": "736137", "postDate": "02/03/2020 21:35:09", "content": "The final private test set would seem to me to have no failures or will a whole bunch more than 27. It's pretty big shot in the dark. But data from the wild tends to have more errors than a nicely developed data set.
\n\nIf I was you - sitting in 4th place than I would absolutely use the safe position of 0.5. I am not likely to be sitting so close to the top - as slow as my progress has been will almost certainly be out of the metals range. So I am going to probably choose to gamble and go even higher than 0.75.
", "rawMarkdown": "The final private test set would seem to me to have no failures or will a whole bunch more than 27. It's pretty big shot in the dark. But data from the wild tends to have more errors than a nicely developed data set. \n\nIf I was you - sitting in 4th place than I would absolutely use the safe position of 0.5. I am not likely to be sitting so close to the top - as slow as my progress has been will almost certainly be out of the metals range. So I am going to probably choose to gamble and go even higher than 0.75.", "votes": null }, { "id": "737130", "postDate": "02/05/2020 00:12:30", "content": "Thanks for raising this. For the private test set, we have been noting all of this feedback so that it goes smooth. My recommendation would be to try not to overfit or leverage these glitches.
", "rawMarkdown": "Thanks for raising this. For the private test set, we have been noting all of this feedback so that it goes smooth. My recommendation would be to try not to overfit or leverage these glitches.", "votes": null }, { "id": "737142", "postDate": "02/05/2020 00:42:13", "content": "Thank you for your reply and advice.
", "rawMarkdown": "Thank you for your reply and advice.", "votes": null } ], "comments": [ { "id": 734904, "author_name": "pcjimmmy", "author_url": "", "post_date": "02/02/2020 07:14:49", "content": "In many of the shared kernels when an error occurs most folks are using 0.5 as the default probability to use.
\n\nThis suggests that a value close to 1 would make more sense when a file error occurs. Many might have already decided that 0.5 was not the best to use for error default - I have been using 0.75 myself.
\n\nNot sure that 27/4000 is a significant leak however.
", "votes": null, "replies": [] }, { "id": 736122, "author_name": "jamesphoward", "author_url": "", "post_date": "02/03/2020 21:01:57", "content": "This is one of those interesting situations where there's a real risk of overfitting to the public leaderboard. I currently predict 0.5 on any video that fails. I suspect my LB score could be improved by picking a higher score for failures, but with the logarithmic loss function there's a risk from it, too.\nI suspect failures are more likely to be deepfakes even in the final private test set, but it'd be better if we could get reassurance that such errors would be cleaned up prior?
", "votes": null, "replies": [ { "id": 736137, "author_name": "pcjimmmy", "author_url": "", "post_date": "02/03/2020 21:35:09", "content": "The final private test set would seem to me to have no failures or will a whole bunch more than 27. It's pretty big shot in the dark. But data from the wild tends to have more errors than a nicely developed data set.
\n\nIf I was you - sitting in 4th place than I would absolutely use the safe position of 0.5. I am not likely to be sitting so close to the top - as slow as my progress has been will almost certainly be out of the metals range. So I am going to probably choose to gamble and go even higher than 0.75.
", "votes": null, "replies": [] } ] }, { "id": 737130, "author_name": "cristiancanton", "author_url": "", "post_date": "02/05/2020 00:12:30", "content": "Thanks for raising this. For the private test set, we have been noting all of this feedback so that it goes smooth. My recommendation would be to try not to overfit or leverage these glitches.
", "votes": null, "replies": [ { "id": 737142, "author_name": "wuliaokaola", "author_url": "", "post_date": "02/05/2020 00:42:13", "content": "Thank you for your reply and advice.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "734873": "zaharch found that there are 27 corrupted videos in the public test set. And [this kernel](https://www.kaggle.com/wuliaokaola/public-test-set-probe) can get LB 0.68846 without any model. That means all of the corrupted videos are fake. \n`0.69314 * (4000 - 27) / 4000 = 0.68846`\nI think when they generated deepfake videos, something went wrong that made video corrupted. @philculliton I don't know how about the private test set, but I think this is a leakage of public test set.", "734904": "In many of the shared kernels when an error occurs most folks are using 0.5 as the default probability to use.\n\nThis suggests that a value close to 1 would make more sense when a file error occurs. Many might have already decided that 0.5 was not the best to use for error default - I have been using 0.75 myself.\n\nNot sure that 27/4000 is a significant leak however.", "736122": "This is one of those interesting situations where there's a real risk of overfitting to the public leaderboard. I currently predict 0.5 on any video that fails. I suspect my LB score could be improved by picking a higher score for failures, but with the logarithmic loss function there's a risk from it, too.\nI suspect failures are more likely to be deepfakes even in the final private test set, but it'd be better if we could get reassurance that such errors would be cleaned up prior?", "736137": "The final private test set would seem to me to have no failures or will a whole bunch more than 27. It's pretty big shot in the dark. But data from the wild tends to have more errors than a nicely developed data set. \n\nIf I was you - sitting in 4th place than I would absolutely use the safe position of 0.5. I am not likely to be sitting so close to the top - as slow as my progress has been will almost certainly be out of the metals range. So I am going to probably choose to gamble and go even higher than 0.75.", "737130": "Thanks for raising this. For the private test set, we have been noting all of this feedback so that it goes smooth. My recommendation would be to try not to overfit or leverage these glitches.", "737142": "Thank you for your reply and advice." }, "source": "meta" }