![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2687432%2Fc71d3cd7153a830417ec7188d96c6d94%2F2020-04-26%2019-52-10.png?generation=1587902053447587&alt=media\")
\n1 - 3570 / (2265 * 2) = 21.2%, about 21.2% submissions went error.\n(21.2 - 6.5) * 2 + 6.5 = 35.9%, about 35.9% teams has at least one error submission.The private leaderboard has been revealed. The private scores were based on a re-run of participants’ selected submissions on the host’s private test set.
\n\nCongratulations to our winners! Please note that the leaderboard is still subject to verification of the winners' full code by the host team.
\n\nThere was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples). We were all frankly very impressed by that number - thanks to everyone for your hard work and attention to detail.
\n\nIt was a real pleasure to work on this competition with the host team and all of you. Thank you!
", "messages": [ { "id": 818448, "postDate": "2020-04-23T22:28:26.410Z", "content": "The private leaderboard has been revealed. The private scores were based on a re-run of participants’ selected submissions on the host’s private test set.
\n\nCongratulations to our winners! Please note that the leaderboard is still subject to verification of the winners' full code by the host team.
\n\nThere was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples). We were all frankly very impressed by that number - thanks to everyone for your hard work and attention to detail.
\n\nIt was a real pleasure to work on this competition with the host team and all of you. Thank you!
", "rawMarkdown": "The private leaderboard has been revealed. The private scores were based on a re-run of participants’ selected submissions on the host’s private test set.\n\nCongratulations to our winners! Please note that the leaderboard is still subject to verification of the winners' full code by the host team.\n\nThere was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples). We were all frankly very impressed by that number - thanks to everyone for your hard work and attention to detail.\n\nIt was a real pleasure to work on this competition with the host team and all of you. Thank you!", "votes": 32 }, { "id": 823327, "postDate": "2020-04-27T15:15:38.517Z", "content": "I did not take an active part in this competition but I regret the way Kaggle is turning a blind eye and a deaf ear to the issues raised here.
\n\nDo you guys realise people spend a huge amount of time, effort and computational resources in order to provide good and cutting edge solutions in such competitions ?
\n\nI think you should give them a full and reasonable feedback when their submissions are discarded in this way. Otherwise, you may discourage some people from participating in such highly demanding competitions in the future .
", "rawMarkdown": "I did not take an active part in this competition but I regret the way Kaggle is turning a blind eye and a deaf ear to the issues raised here.\n\nDo you guys realise people spend a huge amount of time, effort and computational resources in order to provide good and cutting edge solutions in such competitions ? \n\nI think you should give them a full and reasonable feedback when their submissions are discarded in this way. Otherwise, you may discourage some people from participating in such highly demanding competitions in the future .", "votes": 24 }, { "id": 822509, "postDate": "2020-04-27T00:52:14.487Z", "content": "WOW - MY GOODNESS! What a complete and utter breakdown! Kaggle and Facebook need to own what a poor contest this was, and offer a sincere apology to the engaged community here. My team saw the vast difference between the provided data (with killer models we created over months of work and testing) and the Leader Board and we correctly assumed if the data was so poorly put together with relation to the LB, there was certainly going to be more problems down the road. How prescient. I feel for the participants who worked to the very end to get in top 30, only to be bounced out by a very poorly designed challenge.
", "rawMarkdown": "WOW - MY GOODNESS! What a complete and utter breakdown! Kaggle and Facebook need to own what a poor contest this was, and offer a sincere apology to the engaged community here. My team saw the vast difference between the provided data (with killer models we created over months of work and testing) and the Leader Board and we correctly assumed if the data was so poorly put together with relation to the LB, there was certainly going to be more problems down the road. How prescient. I feel for the participants who worked to the very end to get in top 30, only to be bounced out by a very poorly designed challenge.", "votes": 21 }, { "id": 818596, "postDate": "2020-04-24T01:48:52.657Z", "content": "Can't believe it, our team ranked 3rd Public-LB. but get score 0.69314 in private board(2 submission with same score 0.69314). Which is a score of all prediction is 0.5. \nThis may be cause by (try.. exception). When the code throw exception the prediction will be 0.5. private dataset is held outside of Kaggle's platform, is the running environment is different from Kaggle's environment ?( i.e. system os version, cuda version, cuda driver version ....). \nIs it possible to provided detail log information of the re-run? Or just copy the private dataset to the kaggle's platform and re-run the notebook? @philculliton
", "rawMarkdown": "Can't believe it, our team ranked 3rd Public-LB. but get score 0.69314 in private board(2 submission with same score 0.69314). Which is a score of all prediction is 0.5. \nThis may be cause by (try.. exception). When the code throw exception the prediction will be 0.5. private dataset is held outside of Kaggle's platform, is the running environment is different from Kaggle's environment ?( i.e. system os version, cuda version, cuda driver version ....). \nIs it possible to provided detail log information of the re-run? Or just copy the private dataset to the kaggle's platform and re-run the notebook? @philculliton ", "votes": 21, "replies": [ { "id": 818600, "postDate": "2020-04-24T01:55:58.470Z", "content": "Surely if the OS/system/GPU were different that should have been communicated. What a shambles.
", "rawMarkdown": "Surely if the OS/system/GPU were different that should have been communicated. What a shambles.", "votes": 2 }, { "id": 818601, "postDate": "2020-04-24T01:57:54.660Z", "content": "It wasn't supposed to be different: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/123737
", "rawMarkdown": "It wasn't supposed to be different: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/123737", "votes": 2 }, { "id": 818604, "postDate": "2020-04-24T02:01:41.737Z", "content": "Yeah it's not a 6.5% failure rate. It's over 50% counting anything above the 0.5 benchmark really. Again, shambles.
", "rawMarkdown": "Yeah it's not a 6.5% failure rate. It's over 50% counting anything above the 0.5 benchmark really. Again, shambles.", "votes": 1 }, { "id": 818607, "postDate": "2020-04-24T02:10:10.960Z", "content": "so it is very necessary re-run the private board based on the kaggle's platform
", "rawMarkdown": "so it is very necessary re-run the private board based on the kaggle's platform ", "votes": 12 }, { "id": 818637, "postDate": "2020-04-24T02:42:31.587Z", "content": "I also have a similar issue (Private: 0.69314, Public: 0.37183). It's definitely not model overfitting but rather predictions all coming out as 0.5 from try-except block. I thought the whole purpose of Code Competitions were to make sure that code will re-run successfully on a held-out dataset. Just before the competition ended I specifically asked about potential issues and/or gotchas about the external running environment (including Kaggle dataset versions being used in the kernels) before we submit but there were no reply. This is very demotivating after putting lot of effort.
\n\nI don't know what to do in this situation.
\n\nhttps://www.kaggle.com/keremt/deepfake-single-frame-inference?scriptVersionId=31205131
\n\nI dropped 1415 places, got the score exactly same as 0.5 benchmark kernel.
\n\nI don't really know what is the common problem with other teams that dropped from medal zone to 0.5 benchmark level. @user155897, @greendolphin I believe you guys have the same issue, could you make your submission kernels public for further investigation? Thanks.
", "rawMarkdown": "I also have a similar issue (Private: 0.69314, Public: 0.37183). It's definitely not model overfitting but rather predictions all coming out as 0.5 from try-except block. I thought the whole purpose of Code Competitions were to make sure that code will re-run successfully on a held-out dataset. Just before the competition ended I specifically asked about potential issues and/or gotchas about the external running environment (including Kaggle dataset versions being used in the kernels) before we submit but there were no reply. This is very demotivating after putting lot of effort.\n\nI don't know what to do in this situation.\n\nhttps://www.kaggle.com/keremt/deepfake-single-frame-inference?scriptVersionId=31205131\n\nI dropped 1415 places, got the score exactly same as 0.5 benchmark kernel.\n\n- I don't think it's CUDA memory error - otherwise, submission would fail.\n- I used https://www.kaggle.com/leighplt/decord for video reading. If this caused problems reading private videos - then how did it work in Public?\n\nI don't really know what is the common problem with other teams that dropped from medal zone to 0.5 benchmark level. @user155897, @greendolphin I believe you guys have the same issue, could you make your submission kernels public for further investigation? Thanks.", "votes": 12 }, { "id": 818717, "postDate": "2020-04-24T04:34:35.583Z", "content": "We also use the video codec from Nvidia which compiled for CUDA 10.0. AFAIK kaggle kernel is tied to a specific CUDA environment. Probably, a private re-run was made with a different CUDA version.
", "rawMarkdown": "We also use the video codec from Nvidia which compiled for CUDA 10.0. AFAIK kaggle kernel is tied to a specific CUDA environment. Probably, a private re-run was made with a different CUDA version.", "votes": 5 }, { "id": 819106, "postDate": "2020-04-24T11:02:38.090Z", "content": "Welcome to the club @keremt, @user155897, @ekydna! Same here. I spent four months of work on my solution and now I don't even get a score. This is almost like being disqualified!
\n\nI see your kernel also relies on the existence of the competition dataset in the kaggle path:\npd.read_json('../input/deepfake-detection-challenge/train_sample_videos/metadata.json')
I also assumed the path would not change, but it is possible that only the test_videos.zip file in ./ was provided. Could anyone please confirm this?
In my kernel I have \nsrc_pth = Path('../input/deepfake-detection-challenge/test_videos')\nvideos = [f.name for f in os.scandir(src_pth) if f.is_file() and f.name.lower().endswith('.mp4')]
For the public validation set, Getting Started says:\n> Public Validation Set: When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as test_videos.zip
\n\nUnder \"Tips for successful submissions\" follows:
\n\n\n", "rawMarkdown": "Welcome to the club @keremt, @user155897, @ekydna! Same here. I spent four months of work on my solution and now I don't even get a score. This is almost like being disqualified! \n\nI see your kernel also relies on the existence of the competition dataset in the kaggle path:\n`pd.read_json('../input/deepfake-detection-challenge/train_sample_videos/metadata.json')`\n\nI also assumed the path would not change, but it is possible that only the `test_videos.zip` file in ./ was provided. Could anyone please confirm this?\n\nIn my kernel I have \n`src_pth = Path('../input/deepfake-detection-challenge/test_videos')`\n`videos = [f.name for f in os.scandir(src_pth) if f.is_file() and f.name.lower().endswith('.mp4')]`\n\nFor the public validation set, [Getting Started](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started) says:\n> Public Validation Set: When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as test_videos.zip\n\nUnder \"Tips for successful submissions\" follows:\n> Your code must output a submission.csv that predicts on any set of test_videos.zip that it is fed as an input. ", "votes": 3 }, { "id": 819190, "postDate": "2020-04-24T12:16:01.683Z", "content": "Your code must output a submission.csv that predicts on any set of test_videos.zip that it is fed as an input.
\n
This public and successful kernel seems to also use ../input/ relative path in some parts, so I doubt it's the problem. Our case seems to be unique in the sense that there are just a few people who dropped down from medals zone directly to all 0.5s benchmark. I guess there is no way of knowing the actual root cause without waiting for some clarification. @philculliton
@lytic It looks like Kaggle's docker file specifies the CUDA version very explicitly, but it also got upgraded on Feb 11: https://github.com/Kaggle/docker-python/blob/master/gpu.Dockerfile
", "rawMarkdown": "@lytic It looks like Kaggle's docker file specifies the CUDA version very explicitly, but it also got upgraded on Feb 11: https://github.com/Kaggle/docker-python/blob/master/gpu.Dockerfile" }, { "id": 821746, "postDate": "2020-04-26T11:46:26.153Z", "content": "Congrats
", "rawMarkdown": "Congrats\n" }, { "id": 822965, "postDate": "2020-04-27T09:35:02.727Z", "content": "I am really sorry for you guys. One thing though is that Decord and DALI have been reported as unstable (often crashed) in this public kernel, which advised to avoid them :
\n\nhttps://www.kaggle.com/mpware/videoreaders-benchmark
\n\nSo Decord can potentially produce memory leak and we can have crash and cannot catch errors with DALI .
", "rawMarkdown": "I am really sorry for you guys. One thing though is that Decord and DALI have been reported as unstable (often crashed) in this public kernel, which advised to avoid them : \n\nhttps://www.kaggle.com/mpware/videoreaders-benchmark\n\nSo Decord can potentially produce memory leak and we can have crash and cannot catch errors with DALI ." }, { "id": 823009, "postDate": "2020-04-27T10:24:54.377Z", "content": "@olegtrott I think a docker file is tied to a specific competition, but I'm not sure which docker file was provided for the host.
\n\n@ratthachat BTW, we did not use Decord or DALI, we used Nvidia Video Codec SDK directly.
", "rawMarkdown": "@olegtrott I think a docker file is tied to a specific competition, but I'm not sure which docker file was provided for the host.\n\n@ratthachat BTW, we did not use Decord or DALI, we used [Nvidia Video Codec SDK](https://developer.nvidia.com/nvidia-video-codec-sdk/download) directly.", "votes": 1 }, { "id": 823169, "postDate": "2020-04-27T13:19:41.343Z", "content": "@ratthachat Indeed, Decord GPU had the memory leak but I used the cpu version for that reason.
", "rawMarkdown": "@ratthachat Indeed, Decord GPU had the memory leak but I used the cpu version for that reason.", "votes": 1 }, { "id": 823233, "postDate": "2020-04-27T14:00:55.477Z", "content": "@ratthachat I do not use DALI or Decord. But I use nvidia's tensorRT to accelerate inference. There's a common issue that any nvida(cuda , cuda driver) related packages will cause invalid predictions.
", "rawMarkdown": "@ratthachat I do not use DALI or Decord. But I use nvidia's tensorRT to accelerate inference. There's a common issue that any nvida(cuda , cuda driver) related packages will cause invalid predictions. ", "votes": 2 }, { "id": 827800, "postDate": "2020-04-30T14:49:42.880Z", "content": "I agree if test data is not feasible we can at least ask for dockerfiles which was used for held out inference. Which shouldn't be a much ask IMHO. I am not pushing for medals at this point but the curiosity about what made or submissions invalid or all 0.5s is killing me :( if we could only learn something about what would our results be in private LB if we didn't have these technical difficult would be just helpful.
", "rawMarkdown": "I agree if test data is not feasible we can at least ask for dockerfiles which was used for held out inference. Which shouldn't be a much ask IMHO. I am not pushing for medals at this point but the curiosity about what made or submissions invalid or all 0.5s is killing me :( if we could only learn something about what would our results be in private LB if we didn't have these technical difficult would be just helpful.", "votes": 1 } ] }, { "id": 818594, "postDate": "2020-04-24T01:46:19.377Z", "content": "I would strongly recommend the organizers put their private board data on kaggle and re-run the submitted notebooks. All the submitted notebooks got error and I got no results. This is unbelievable and I'm sure the notebook I submitted is runable.
", "rawMarkdown": "I would strongly recommend the organizers put their private board data on kaggle and re-run the submitted notebooks. All the submitted notebooks got error and I got no results. This is unbelievable and I'm sure the notebook I submitted is runable.", "votes": 13 }, { "id": 818487, "postDate": "2020-04-23T23:28:13.377Z", "content": "🤔 Can't believe it! From Public 37th (was ~48th I remember) to Private 4th.
", "rawMarkdown": "🤔 Can't believe it! From Public 37th (was ~48th I remember) to Private 4th.", "votes": 11, "replies": [ { "id": 818493, "postDate": "2020-04-23T23:39:51.383Z", "content": "Hopefully this is final & real. I love money 😂
\n\nBut it is really strange that 4 of the public-LB top 5 dropped to nowhere in the private-LB. Very curious about what happened...
", "rawMarkdown": "Hopefully this is final & real. I love money 😂 \n\nBut it is really strange that 4 of the public-LB top 5 dropped to nowhere in the private-LB. Very curious about what happened...", "votes": 3 }, { "id": 818524, "postDate": "2020-04-24T00:30:40.210Z", "content": "Hi @cuihaoleo can we merge in this competition? :D
", "rawMarkdown": "Hi @cuihaoleo can we merge in this competition? :D" }, { "id": 818525, "postDate": "2020-04-24T00:32:16.657Z", "content": "Congratulations， have you clipped the prediction values?
", "rawMarkdown": "Congratulations， have you clipped the prediction values?" }, { "id": 818538, "postDate": "2020-04-24T00:49:43.147Z", "content": "I'm merging for the other 1 empty slot. 💯
", "rawMarkdown": "I'm merging for the other 1 empty slot. 💯 ", "votes": 1 } ] }, { "id": 818937, "postDate": "2020-04-24T08:14:44.040Z", "content": "I went from public 31st place to nowhere as many others here. All notebooks failed at stage 2; whatever that means.\nMy notebook has taken precautions to scale down images for face detection as I ran into errors (probably out of CUDA memory) during submissions in the past. That kind of rules out cuda memory issues as a reason for the failures.
\n\nMy video processing and face detection code is enclosed in a try/catch statement, which would result in 0.5 predictions for any video that causes an exception.
\n\nTo state the obvious: this is very disappointing. I expected more from a high profile competition such as this. Imagine how the top 10 teams must feel now that most are gone.
", "rawMarkdown": "I went from public 31st place to nowhere as many others here. All notebooks failed at stage 2; whatever that means.\nMy notebook has taken precautions to scale down images for face detection as I ran into errors (probably out of CUDA memory) during submissions in the past. That kind of rules out cuda memory issues as a reason for the failures.\n\nMy video processing and face detection code is enclosed in a try/catch statement, which would result in 0.5 predictions for any video that causes an exception.\n\nTo state the obvious: this is very disappointing. I expected more from a high profile competition such as this. Imagine how the top 10 teams must feel now that most are gone.", "votes": 10 }, { "id": 819816, "postDate": "2020-04-24T22:30:33.613Z", "content": "A few clarifications:
\n\nSorry, we are not able to provide team-specific feedback or debug your submissions.
", "rawMarkdown": "A few clarifications:\n\n- There were ~10k videos in the test set.\n- If you don't see any submissions for the competition, then your team was removed from the competition due to holding multiple accounts or file-sharing across teams. These removals are made with very conclusive evidence, but if you feel you were improperly removed, you are welcome to make a [compliance review submission](https://www.kaggle.com/compliance).\n- All submissions which were *not selected* will be shown as errored in your submissions, because we did not re-run or score those.\n- Here are the reasons you may have selected submissions which were invalidated, in rough order of how common the issue was:\n - Submission produced no or null predictions on some portion of the test set\n - Submission produced predictions but with the wrong number of rows\n - Your notebooks or linked notebooks/datasets for selected submissions were deleted, and therefore we could not re-run them\n - Insufficient error handling & Misc code failures (including out of memory & timeout errors)\n- As Phil mentioned, the private re-run success rate was ~94% teams \n\n**Sorry, we are not able to provide team-specific feedback or debug your submissions.**", "votes": 7, "replies": [ { "id": 819826, "postDate": "2020-04-24T22:57:17.170Z", "content": "@juliaelliott is there any chance that the re-run was performed in the different environment (docker image)? In particular, with a different CUDA version?
", "rawMarkdown": "@juliaelliott is there any chance that the re-run was performed in the different environment (docker image)? In particular, with a different CUDA version?", "votes": 1 }, { "id": 819858, "postDate": "2020-04-25T00:01:26.527Z", "content": "Is it possible to find out the reason for disqualification? As far as my team's aware, we did not break any rules. It might save everyone the trouble of going through the compliance review, if Kaggle tells us the reason behind the decision.
", "rawMarkdown": "Is it possible to find out the reason for disqualification? As far as my team's aware, we did not break any rules. It might save everyone the trouble of going through the compliance review, if Kaggle tells us the reason behind the decision." }, { "id": 819866, "postDate": "2020-04-25T00:08:50.840Z", "content": "@lytic, the docker images were replicated precisely (Kaggle's engineers handed this over to the host team's engineers), and errors were investigated to confirm they were not attributable to subtle image or environment differences.
\n\n@liovch, sorry, no. I've provided the categories of errors, but we're not able to support team-by-team debugging/error investigation.
", "rawMarkdown": "@lytic, the docker images were replicated precisely (Kaggle's engineers handed this over to the host team's engineers), and errors were investigated to confirm they were not attributable to subtle image or environment differences.\n\n@liovch, sorry, no. I've provided the categories of errors, but we're not able to support team-by-team debugging/error investigation.", "votes": 2 }, { "id": 819871, "postDate": "2020-04-25T00:12:58.963Z", "content": "Could you be able to share the 10k private test set? There are few teams which got all 0.5 predictions apparently. We can re-run our notebooks with the test set to verify that the fault is on our code.
", "rawMarkdown": "Could you be able to share the 10k private test set? There are few teams which got all 0.5 predictions apparently. We can re-run our notebooks with the test set to verify that the fault is on our code. ", "votes": 5 }, { "id": 819873, "postDate": "2020-04-25T00:23:29.080Z", "content": "Thanks for reply @juliaelliott, if Kaggle is unable to provide a reason for eliminating our team from the competition, we will apply for compliance review submission.
", "rawMarkdown": "Thanks for reply @juliaelliott, if Kaggle is unable to provide a reason for eliminating our team from the competition, we will apply for compliance review submission.", "votes": 3 }, { "id": 819874, "postDate": "2020-04-25T00:27:38.043Z", "content": "@juliaelliott could you release part of private test data（5%）and move to kaggle's platform? then we can check why we got all 0.5 predictions.
\n\nmy notebook will get 0.5 prediction when run on same docker image with different cuda driver version
", "rawMarkdown": "@juliaelliott could you release part of private test data（5%）and move to kaggle's platform? then we can check why we got all 0.5 predictions. \n\n\nmy notebook will get 0.5 prediction when run on same docker image with different cuda driver version ", "votes": 16 }, { "id": 820032, "postDate": "2020-04-25T05:22:31.677Z", "content": "@juliaelliott The results of online table statistics show that at least 9 of the top 120 teams have encountered similar situations and most of them are disqualified. \nThis ratio(9/120) is higher than 6.5%. \nEven at 6.5%, this is still a very high percentage, isn't it?
\n\nI want to know whether you have any plans to deal with the problem. \nIf (kagglers and facebookers)'s answers are no, I have no choice, because the initiative is in your hands.
", "rawMarkdown": "@juliaelliott The results of online table statistics show that at least 9 of the top 120 teams have encountered similar situations and most of them are disqualified. \nThis ratio(9/120) is higher than 6.5%. \nEven at 6.5%, this is still a very high percentage, isn't it?\n\nI want to know whether you have any plans to deal with the problem. \nIf (kagglers and facebookers)'s answers are no, I have no choice, because the initiative is in your hands.", "votes": 6 }, { "id": 821720, "postDate": "2020-04-26T11:16:11.347Z", "content": "@juliaelliott I think Kaggle should try to do something for all teams in the 6.5% complaining. It's unfair for all who worked hard in this competition (even maybe some teams in the 6.5% are in the money). It's a matter of image/reputation of Kaggle. The initial deal was if your submission works then final evaluation should work. 6.5% of failure means that this format of competition should be stopped as too random (no way to troubleshot). I'm not saying you will be able to reach 100% of successful re-run but you should consider some actions.
", "rawMarkdown": "@juliaelliott I think Kaggle should try to do something for all teams in the 6.5% complaining. It's unfair for all who worked hard in this competition (even maybe some teams in the 6.5% are in the money). It's a matter of image/reputation of Kaggle. The initial deal was if your submission works then final evaluation should work. 6.5% of failure means that this format of competition should be stopped as too random (no way to troubleshot). I'm not saying you will be able to reach 100% of successful re-run but you should consider some actions.", "votes": 14 }, { "id": 821770, "postDate": "2020-04-26T12:10:49.983Z", "content": "\n1 - 3570 / (2265 * 2) = 21.2%, about 21.2% submissions went error.\n(21.2 - 6.5) * 2 + 6.5 = 35.9%, about 35.9% teams has at least one error submission.
As an experienced organizer, Kaggle or Facebook should give a solution or decision to the 6.5% errors or incorrect scores.
", "rawMarkdown": "As an experienced organizer, Kaggle or Facebook should give a solution or decision to the 6.5% errors or incorrect scores. \n", "votes": 9 }, { "id": 821964, "postDate": "2020-04-26T14:41:36.147Z", "content": "Is there difference between bullet points 2 and 3?
\n\nI see my all my submissions in \"Error\" state. When I hover over the error message for selected entries, I see \"Invalidated for stage 2\".
\n\nNot clear if this means I broke some competition rule and was disqualified, Or was there a problem in running the selected notebooks. Hope it is the latter. Wish error messages could have been more specific. A lot of people have put a lot of hard work into this competition. Clear error message/ notebook log will at least help us avoid these mistakes in future competitions.
", "rawMarkdown": "Is there difference between bullet points 2 and 3?\n\nI see my all my submissions in \"Error\" state. When I hover over the error message for selected entries, I see \"Invalidated for stage 2\". \n\nNot clear if this means I broke some competition rule and was disqualified, Or was there a problem in running the selected notebooks. Hope it is the latter. Wish error messages could have been more specific. A lot of people have put a lot of hard work into this competition. Clear error message/ notebook log will at least help us avoid these mistakes in future competitions.", "votes": 3 }, { "id": 822910, "postDate": "2020-04-27T08:49:16.077Z", "content": "Rethinking why you held this competition. For giving out money? No, you want a better solution for deep fake detection. I don't think this result can help you in realistic applications, it can't help the development of deep fake detection either. This competition, which costs a lot of money, I don't think it should be ended this way.
", "rawMarkdown": "Rethinking why you held this competition. For giving out money? No, you want a better solution for deep fake detection. I don't think this result can help you in realistic applications, it can't help the development of deep fake detection either. This competition, which costs a lot of money, I don't think it should be ended this way.", "votes": 2 } ] }, { "id": 818534, "postDate": "2020-04-24T00:46:24.677Z", "content": "May I kindly ask to re-run the final solutions of the four teams that where \"in the medals\" on public LB, but aparently failed to read any video from the private test set (score==0.69314)?
\n\nI can speak only for myself, but the only way I see my script could have failed is the test videos no longer provided in \"../input/deepfake-detection-challenge/test_videos/\" but only in \"./test_videos.zip\". Or can anyone disconfirm this?
\n\nAnyway, congrats to all winners and thanks for the amazing contributions published in the kernels and discussions!
", "rawMarkdown": "May I kindly ask to re-run the final solutions of the four teams that where \"in the medals\" on public LB, but aparently failed to read any video from the private test set (score==0.69314)?\n\nI can speak only for myself, but the only way I see my script could have failed is the test videos no longer provided in \"../input/deepfake-detection-challenge/test_videos/\" but only in \"./test_videos.zip\". Or can anyone disconfirm this?\n\nAnyway, congrats to all winners and thanks for the amazing contributions published in the kernels and discussions!", "votes": 5 }, { "id": 818928, "postDate": "2020-04-24T08:05:55.533Z", "content": "Real sad to see lots of teams dropping ranks due to failed run. Though my team sort of benefitted from the shake-ups , but I do hope that the host will figure out the cause and provide us some solid feedbacks.
", "rawMarkdown": "Real sad to see lots of teams dropping ranks due to failed run. Though my team sort of benefitted from the shake-ups , but I do hope that the host will figure out the cause and provide us some solid feedbacks.", "votes": 6 }, { "id": 818676, "postDate": "2020-04-24T03:32:52.607Z", "content": "Congrats to the winners! 👍 What a surge from the all-star team, I feel blessed to have met many of you during the KaggleDays!\nAnd what an impressive top-position consistency from Selim Seferbekov @selimsef ! Віншаванні!!💪 👋 😃
", "rawMarkdown": "Congrats to the winners! 👍 What a surge from the all-star team, I feel blessed to have met many of you during the KaggleDays!\nAnd what an impressive top-position consistency from Selim Seferbekov @selimsef ! Віншаванні!!💪 👋 😃 ", "votes": 6 }, { "id": 879140, "postDate": "2020-06-09T08:22:19.590Z", "content": "@philculliton @juliaelliott \ndo you know by when we will be awarded with medals ?
", "rawMarkdown": "@philculliton @juliaelliott \ndo you know by when we will be awarded with medals ?", "votes": 4, "replies": [ { "id": 882264, "postDate": "2020-06-11T17:13:05.840Z", "content": "Never😭
", "rawMarkdown": "Never😭", "votes": 1 }, { "id": 882374, "postDate": "2020-06-11T18:48:32.140Z", "content": "waiting to hear back from @juliaelliott @philculliton
", "rawMarkdown": "waiting to hear back from @juliaelliott @philculliton ", "votes": 1 } ] }, { "id": 821833, "postDate": "2020-04-26T13:00:25.593Z", "content": "I think appealing on these forums is just resulting in the same standard clause. Only thing that is going to make a difference is to make it public.
\n\nAnthony Goldbloom - CEO Kaggle\nhttps://twitter.com/antgoldbloom\nhttps://www.linkedin.com/in/anthonygoldbloom\nhttps://www.kaggle.com/about/team
", "rawMarkdown": "I think appealing on these forums is just resulting in the same standard clause. Only thing that is going to make a difference is to make it public.\n\nAnthony Goldbloom - CEO Kaggle\nhttps://twitter.com/antgoldbloom\nhttps://www.linkedin.com/in/anthonygoldbloom\nhttps://www.kaggle.com/about/team", "votes": 4, "replies": [ { "id": 822895, "postDate": "2020-04-27T08:27:06.077Z", "content": "https://www.kaggle.com/antgoldbloom/contact . I think we should email to him and make him realize how poor their workers are 👎 I have also commented under his twitter. Hope he will see it.
", "rawMarkdown": "https://www.kaggle.com/antgoldbloom/contact . I think we should email to him and make him realize how poor their workers are 👎 I have also commented under his twitter. Hope he will see it." } ] }, { "id": 818650, "postDate": "2020-04-24T03:04:12.663Z", "content": "Hi folks, it's really sorry to see lots of people got reshuffled. Anyway, my dream to hunt gold from this competition break now. :| (can get your feelings). I'm sure many of you have considered the problem, here are my general 2 cents here: (definitely not for metal range players in public LB)
\n\nThere are several things may break the pipeline up:\n- OpenCV when reading videos.\n- Face decorator GPU memory overflow.\n- CNN models compete with face decorators on GPU memory.
\n\nFor the last 2 issues, we will need to clear the GPU memory, shrink batch process size (16 -> 8 -> 4 -> 2 -> 1), or even move some of the computation to CPU.
\n\nAlso, the resolution of the video may vary, so probably we'll need to normalize the video (say 1920 * 1080 / 1080 * 1920).
", "rawMarkdown": "Hi folks, it's really sorry to see lots of people got reshuffled. Anyway, my dream to hunt gold from this competition break now. :| (can get your feelings). I'm sure many of you have considered the problem, here are my general 2 cents here: (definitely not for metal range players in public LB)\n\nThere are several things may break the pipeline up:\n- OpenCV when reading videos.\n- Face decorator GPU memory overflow.\n- CNN models compete with face decorators on GPU memory.\n\nFor the last 2 issues, we will need to clear the GPU memory, shrink batch process size (16 -> 8 -> 4 -> 2 -> 1), or even move some of the computation to CPU. \n\nAlso, the resolution of the video may vary, so probably we'll need to normalize the video (say 1920 * 1080 / 1080 * 1920).\n", "votes": 4 }, { "id": 818459, "postDate": "2020-04-23T22:43:15.970Z", "content": "Grats, All Faces Are Real. :)
", "rawMarkdown": "Grats, All Faces Are Real. :) ", "votes": 5 }, { "id": 818803, "postDate": "2020-04-24T06:12:26.633Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F450289%2Fdf7b0fabaf59ee8fe7f9c789b239c2f6%2F1.png?generation=1587708373272288&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F450289%2Fdfc3de37fd736d29daa22a6f8cf3a100%2F2.png?generation=1587708374393936&alt=media\")
Can believe it, We can get a score of 0.34 in public-lb, but we don't get a score in private-lb. Why is that?
", "rawMarkdown": "\n\n\n@philculliton\n\nCan believe it, We can get a score of 0.34 in public-lb, but we don't get a score in private-lb. Why is that?", "votes": 3, "replies": [ { "id": 818812, "postDate": "2020-04-24T06:15:57.840Z", "content": "@philculliton The name of my team is relax, I don't know what happened, can you give me more detailed error information?
\n\nNow that both the public test set and the private test set have lost my name, am I breaking any rules?
", "rawMarkdown": "@philculliton **The name of my team is relax, I don't know what happened, can you give me more detailed error information?** \n\n**Now that both the public test set and the private test set have lost my name, am I breaking any rules?**\n\n\n ", "votes": 1 }, { "id": 818833, "postDate": "2020-04-24T06:31:15.693Z", "content": "I suppose they haven't run any of non-selected entires on their private lb, thus the errors. I've also got errors everywhere but on those 2 final entries.
", "rawMarkdown": "I suppose they haven't run any of non-selected entires on their private lb, thus the errors. I've also got errors everywhere but on those 2 final entries.", "votes": 3 }, { "id": 818839, "postDate": "2020-04-24T06:37:41.443Z", "content": "@petya The problem is that the entires I chose did not private lb either so I am now in a state of no ranking
", "rawMarkdown": "@petya The problem is that the entires I chose did not private lb either so I am now in a state of no ranking" }, { "id": 818869, "postDate": "2020-04-24T07:08:48.600Z", "content": "@ruyiwei I guess the competition holders will have to address the issue with most notebooks scoring around 0.69, which is the baseline. Something clearly has triggerred an exception for those guys, and those who didn't have try\\except at all got submission errors.
", "rawMarkdown": "@ruyiwei I guess the competition holders will have to address the issue with most notebooks scoring around 0.69, which is the baseline. Something clearly has triggerred an exception for those guys, and those who didn't have try\\except at all got submission errors. " }, { "id": 818873, "postDate": "2020-04-24T07:15:29.843Z", "content": "@petya It's hard for me to understand why the top 10 in public_lb is now 900 +
", "rawMarkdown": "@petya It's hard for me to understand why the top 10 in public_lb is now 900 +" } ] }, { "id": 818506, "postDate": "2020-04-23T23:59:37.077Z", "content": "Our team is gone from both public and private leaderboard. Team name is \"levsha\". Could you please explain what happened?
", "rawMarkdown": "Our team is gone from both public and private leaderboard. Team name is \"levsha\". Could you please explain what happened?", "votes": 3, "replies": [ { "id": 818509, "postDate": "2020-04-24T00:08:10.550Z", "content": "Same. Didn't use any external data, have only one account. Very disappointed, please explain what I'm guilty of
", "rawMarkdown": "Same. Didn't use any external data, have only one account. Very disappointed, please explain what I'm guilty of", "votes": 1 }, { "id": 818510, "postDate": "2020-04-24T00:08:40.437Z", "content": "It also happened to us. What kind of error does it show? We got \"invalidated for stage 2 error\".
", "rawMarkdown": "It also happened to us. What kind of error does it show? We got \"invalidated for stage 2 error\".", "votes": 1 }, { "id": 818512, "postDate": "2020-04-24T00:13:37.697Z", "content": "All our 155 submissions (including the 2 selected for private leaderboard) show \"Invalidated for stage 2 error\"
", "rawMarkdown": "All our 155 submissions (including the 2 selected for private leaderboard) show \"Invalidated for stage 2 error\"", "votes": 1 }, { "id": 818513, "postDate": "2020-04-24T00:17:27.170Z", "content": "@liovch, looks like we face the same situation. Any idea on what \"Invalidated for stage 2 error\" mean?
", "rawMarkdown": "@liovch, looks like we face the same situation. Any idea on what \"Invalidated for stage 2 error\" mean?", "votes": 1 }, { "id": 818515, "postDate": "2020-04-24T00:19:09.610Z", "content": "I have the same message, it seems we were disqualified
", "rawMarkdown": "I have the same message, it seems we were disqualified", "votes": 1 }, { "id": 818522, "postDate": "2020-04-24T00:29:54.397Z", "content": "It's possible 2 of your submissions were failed. In my case, I have only 1/2 submission rescored successful. I don't know what happened to the failed one.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1829450%2F3426f2272479b81d1db15fb9cd4e3e6c%2FUntitled.png?generation=1587688666663282&alt=media\")
what error message did you get on failed one?
", "rawMarkdown": "what error message did you get on failed one?" }, { "id": 818528, "postDate": "2020-04-24T00:34:45.767Z", "content": "Just Error as all other un-scored submissions.
", "rawMarkdown": "Just Error as all other un-scored submissions.", "votes": 1 }, { "id": 818532, "postDate": "2020-04-24T00:42:44.847Z", "content": "then I probably have a mistake in the code, better than being disqualified )
", "rawMarkdown": "then I probably have a mistake in the code, better than being disqualified )" }, { "id": 818557, "postDate": "2020-04-24T01:07:43.920Z", "content": "@khahuras , what error message did you get when you move the mouse onto the i icon?
", "rawMarkdown": "@khahuras , what error message did you get when you move the mouse onto the i icon?" }, { "id": 818586, "postDate": "2020-04-24T01:34:23.480Z", "content": "@khahuras If the submissions selected for private leaderboard failed due to some error, it doesn't explain why the score for all the other submissions on the public leaderboard changed to \"Error\".
", "rawMarkdown": "@khahuras If the submissions selected for private leaderboard failed due to some error, it doesn't explain why the score for all the other submissions on the public leaderboard changed to \"Error\"." }, { "id": 818615, "postDate": "2020-04-24T02:22:26.197Z", "content": "The pop-up error when hovering the mouse into \"Error\" is also the same as all other errors: \"Invalidated for stage 2\"
", "rawMarkdown": "The pop-up error when hovering the mouse into \"Error\" is also the same as all other errors: \"Invalidated for stage 2\"" }, { "id": 818638, "postDate": "2020-04-24T02:42:59.747Z", "content": "@khahuras what was the different between your 1st failed notebook and 2nd successful notebook?
", "rawMarkdown": "@khahuras what was the different between your 1st failed notebook and 2nd successful notebook?" }, { "id": 818645, "postDate": "2020-04-24T02:53:40.513Z", "content": "@maralski The selected but failed notebook uses more models, especially some LSTM. In my successful submission, there is no LSTM. So basically they are very different. I'm sure the LSTM/GRU sort of things caused the error. Maybe during loading, or during inference. We never know. Personally I hope this failed submission score better than the other, but it failed :(
", "rawMarkdown": "@maralski The selected but failed notebook uses more models, especially some LSTM. In my successful submission, there is no LSTM. So basically they are very different. I'm sure the LSTM/GRU sort of things caused the error. Maybe during loading, or during inference. We never know. Personally I hope this failed submission score better than the other, but it failed :(" }, { "id": 818881, "postDate": "2020-04-24T07:20:16.063Z", "content": "Same here but only for one of the submission. The differences between the 2 submissions:\n1 - Submission succeeded: 1 worker - 3 models ensembled - 73 frames.\n2 - Submission failed: 4 workers - 4 models ensembled - 83 frames.\nAs submission#2 did not get score, even with the try/catch, I suspect it could be a memory issue. And such issue could be explained if a lot images in final set are larger than the ones for public LB. There was no way to anticipate/test it.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2Feccceb954ff91328024811a35cd96999%2Fsubmissions.png?generation=1587712228858318&alt=media\")
hi, @philculliton ,my notebook will output two files(pred_full.csv, submission.csv), is it posible the failure caused by using pred_full.csv when calculation the score. \nIs it possible to provide log information of the re-run failure.
", "rawMarkdown": "hi, @philculliton ,my notebook will output two files(pred\\_full.csv, submission.csv), is it posible the failure caused by using pred\\_full.csv when calculation the score. \nIs it possible to provide log information of the re-run failure.", "votes": 1 }, { "id": 819114, "postDate": "2020-04-24T11:09:36.397Z", "content": "💔 👎 😫
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1920073%2F1e2e07d5a2097c993aa8acc006e239d1%2F2020-04-24%2014-07-37.png?generation=1587726520213001&alt=media\")
No sharing, I used only competition data. Kernels worked! Why I get error?
", "rawMarkdown": "No sharing, I used only competition data. Kernels worked! Why I get error? ", "votes": 2 } ] }, { "id": 818794, "postDate": "2020-04-24T06:08:46.313Z", "content": "Congratulations to final medallists. Looking at the private leaderboard for the first 50 teams, there are at least 35 of 50 teams whose ranking changed in double digits. AND the winner's scores in private leaderboard and public leaderboard differ by 0.23 on log loss of a binary classification task. It arouses my great curiosity on what's the data distribution in private test set.\nTherefore, I suggest the host to consider releasing some data samples of private test set. Furthermore, as the data domain varies so greatly in private set and public set, would it be more fair to integrat both the public set ranking and the private set ranking?
", "rawMarkdown": "Congratulations to final medallists. Looking at the private leaderboard for the first 50 teams, there are at least 35 of 50 teams whose ranking changed in double digits. AND the winner's scores in private leaderboard and public leaderboard differ by 0.23 on log loss of a binary classification task. It arouses my great curiosity on what's the data distribution in private test set.\nTherefore, I suggest the host to consider releasing some data samples of private test set. Furthermore, as the data domain varies so greatly in private set and public set, would it be more fair to integrat both the public set ranking and the private set ranking?", "votes": 2 }, { "id": 825577, "postDate": "2020-04-29T05:39:45.430Z", "content": "Wonderful . I am going to translate it to Chinese
", "rawMarkdown": "Wonderful . I am going to translate it to Chinese" }, { "id": 818671, "postDate": "2020-04-24T03:25:29.073Z", "content": "Reviewed my code and only thing I can explain that would have resulted in failure was I mapped my models to 'cuda:0'. Is there a situation where GPU ID was not 0 when run on private set?
", "rawMarkdown": "Reviewed my code and only thing I can explain that would have resulted in failure was I mapped my models to 'cuda:0'. Is there a situation where GPU ID was not 0 when run on private set?\n", "replies": [ { "id": 819010, "postDate": "2020-04-24T09:22:36.323Z", "content": "I did that too and my code run succesfully, so I do not think that's the issue. Hopefully, Kaggle will figure this out.
", "rawMarkdown": "I did that too and my code run succesfully, so I do not think that's the issue. Hopefully, Kaggle will figure this out." }, { "id": 819117, "postDate": "2020-04-24T11:11:44.797Z", "content": "Good to know. Thanks
", "rawMarkdown": "Good to know. Thanks" } ] }, { "id": 818644, "postDate": "2020-04-24T02:52:56.103Z", "content": "First of all congrats to winners, and final medalist. I know You all put a-lot of hard work and countless hours to achieve this, so kudos to all of you.
\n\nNow, Once again, most of us, including me, overfitted Public LB.... This is becoming common story of many Kaggle Kernel Competition, where high scoring overfitted public LB kernels will release earlier in competition by few and then most of contributors will end up following only these few directions discussed in these kernel and then producing more and more overfitted models on public LB and discuss more on more those direction.\nMany of less confident contributors like me, sometimes end up abandoning many possible less overfitted, generalized model direction in lieu of these public kernels score, discussion and approaches share by them.\nand then only In the end private LB will reveal how overfitted these directions were....
\n\nI too end up submitting overfitted model by abandoning my earlier generalized architecture approaches.
\n\nSomehow Only People who kept faith on their generalized original model till the end without getting distracted by those top scoring public kernels are the one who came out on top. \nI am seeing people from public LB in 1000's reaching bronze in private and vice versa. My own overfitted model drop from bronze in public to 900 in private.
\n\nPlease share if you happen to have any similar story.
", "rawMarkdown": "First of all congrats to winners, and final medalist. I know You all put a-lot of hard work and countless hours to achieve this, so kudos to all of you.\n\nNow, Once again, most of us, including me, overfitted Public LB.... This is becoming common story of many Kaggle Kernel Competition, where high scoring overfitted public LB kernels will release earlier in competition by few and then most of contributors will end up following only these few directions discussed in these kernel and then producing more and more overfitted models on public LB and discuss more on more those direction.\nMany of less confident contributors like me, sometimes end up abandoning many possible less overfitted, generalized model direction in lieu of these public kernels score, discussion and approaches share by them.\nand then only In the end private LB will reveal how overfitted these directions were.... \n\nI too end up submitting overfitted model by abandoning my earlier generalized architecture approaches.\n\nSomehow Only People who kept faith on their generalized original model till the end without getting distracted by those top scoring public kernels are the one who came out on top. \nI am seeing people from public LB in 1000's reaching bronze in private and vice versa. My own overfitted model drop from bronze in public to 900 in private. \n\nPlease share if you happen to have any similar story. ", "votes": 1 }, { "id": 826645, "postDate": "2020-04-29T18:56:56.243Z", "content": "So appealing and exciting.
", "rawMarkdown": "So appealing and exciting." }, { "id": 823249, "postDate": "2020-04-27T14:14:03.403Z", "content": "Eat melon
", "rawMarkdown": "Eat melon" }, { "id": 820390, "postDate": "2020-04-25T12:07:18.633Z", "content": "@juliaelliott can you clarify what was timeout for private test set?
", "rawMarkdown": "@juliaelliott can you clarify what was timeout for private test set?" }, { "id": 821202, "postDate": "2020-04-26T02:20:00.363Z", "rawMarkdown": "", "isDeleted": true }, { "id": 820117, "postDate": "2020-04-25T06:36:57.410Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 823327, "author_name": "Serigne ", "author_url": "", "post_date": "2020-04-27T15:15:38.517000", "content": "I did not take an active part in this competition but I regret the way Kaggle is turning a blind eye and a deaf ear to the issues raised here.
\n\nDo you guys realise people spend a huge amount of time, effort and computational resources in order to provide good and cutting edge solutions in such competitions ?
\n\nI think you should give them a full and reasonable feedback when their submissions are discarded in this way. Otherwise, you may discourage some people from participating in such highly demanding competitions in the future .
", "votes": 24, "replies": [] }, { "id": 822509, "author_name": "Mark Eckdahl", "author_url": "", "post_date": "2020-04-27T00:52:14.487000", "content": "WOW - MY GOODNESS! What a complete and utter breakdown! Kaggle and Facebook need to own what a poor contest this was, and offer a sincere apology to the engaged community here. My team saw the vast difference between the provided data (with killer models we created over months of work and testing) and the Leader Board and we correctly assumed if the data was so poorly put together with relation to the LB, there was certainly going to be more problems down the road. How prescient. I feel for the participants who worked to the very end to get in top 30, only to be bounced out by a very poorly designed challenge.
", "votes": 21, "replies": [] }, { "id": 818596, "author_name": "WestLake", "author_url": "", "post_date": "2020-04-24T01:48:52.657000", "content": "Can't believe it, our team ranked 3rd Public-LB. but get score 0.69314 in private board(2 submission with same score 0.69314). Which is a score of all prediction is 0.5. \nThis may be cause by (try.. exception). When the code throw exception the prediction will be 0.5. private dataset is held outside of Kaggle's platform, is the running environment is different from Kaggle's environment ?( i.e. system os version, cuda version, cuda driver version ....). \nIs it possible to provided detail log information of the re-run? Or just copy the private dataset to the kaggle's platform and re-run the notebook? @philculliton
", "votes": 21, "replies": [ { "id": 818600, "author_name": "maralski", "author_url": "", "post_date": "2020-04-24T01:55:58.470000", "content": "Surely if the OS/system/GPU were different that should have been communicated. What a shambles.
", "votes": 2, "replies": [] }, { "id": 818601, "author_name": "Oleg Trott", "author_url": "", "post_date": "2020-04-24T01:57:54.660000", "content": "It wasn't supposed to be different: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/123737
", "votes": 2, "replies": [] }, { "id": 818604, "author_name": "maralski", "author_url": "", "post_date": "2020-04-24T02:01:41.737000", "content": "Yeah it's not a 6.5% failure rate. It's over 50% counting anything above the 0.5 benchmark really. Again, shambles.
", "votes": 1, "replies": [] }, { "id": 818607, "author_name": "WestLake", "author_url": "", "post_date": "2020-04-24T02:10:10.960000", "content": "so it is very necessary re-run the private board based on the kaggle's platform
", "votes": 12, "replies": [] }, { "id": 818637, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-04-24T02:42:31.587000", "content": "I also have a similar issue (Private: 0.69314, Public: 0.37183). It's definitely not model overfitting but rather predictions all coming out as 0.5 from try-except block. I thought the whole purpose of Code Competitions were to make sure that code will re-run successfully on a held-out dataset. Just before the competition ended I specifically asked about potential issues and/or gotchas about the external running environment (including Kaggle dataset versions being used in the kernels) before we submit but there were no reply. This is very demotivating after putting lot of effort.
\n\nI don't know what to do in this situation.
\n\nhttps://www.kaggle.com/keremt/deepfake-single-frame-inference?scriptVersionId=31205131
\n\nI dropped 1415 places, got the score exactly same as 0.5 benchmark kernel.
\n\nI don't really know what is the common problem with other teams that dropped from medal zone to 0.5 benchmark level. @user155897, @greendolphin I believe you guys have the same issue, could you make your submission kernels public for further investigation? Thanks.
", "votes": 12, "replies": [] }, { "id": 818717, "author_name": "ivan", "author_url": "", "post_date": "2020-04-24T04:34:35.583000", "content": "We also use the video codec from Nvidia which compiled for CUDA 10.0. AFAIK kaggle kernel is tied to a specific CUDA environment. Probably, a private re-run was made with a different CUDA version.
", "votes": 5, "replies": [] }, { "id": 819106, "author_name": "Marius Wanko", "author_url": "", "post_date": "2020-04-24T11:02:38.090000", "content": "Welcome to the club @keremt, @user155897, @ekydna! Same here. I spent four months of work on my solution and now I don't even get a score. This is almost like being disqualified!
\n\nI see your kernel also relies on the existence of the competition dataset in the kaggle path:\npd.read_json('../input/deepfake-detection-challenge/train_sample_videos/metadata.json')
I also assumed the path would not change, but it is possible that only the test_videos.zip file in ./ was provided. Could anyone please confirm this?
In my kernel I have \nsrc_pth = Path('../input/deepfake-detection-challenge/test_videos')\nvideos = [f.name for f in os.scandir(src_pth) if f.is_file() and f.name.lower().endswith('.mp4')]
For the public validation set, Getting Started says:\n> Public Validation Set: When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as test_videos.zip
\n\nUnder \"Tips for successful submissions\" follows:
\n\n\n", "votes": 3, "replies": [] }, { "id": 819190, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-04-24T12:16:01.683000", "content": "Your code must output a submission.csv that predicts on any set of test_videos.zip that it is fed as an input.
\n
This public and successful kernel seems to also use ../input/ relative path in some parts, so I doubt it's the problem. Our case seems to be unique in the sense that there are just a few people who dropped down from medals zone directly to all 0.5s benchmark. I guess there is no way of knowing the actual root cause without waiting for some clarification. @philculliton
@lytic It looks like Kaggle's docker file specifies the CUDA version very explicitly, but it also got upgraded on Feb 11: https://github.com/Kaggle/docker-python/blob/master/gpu.Dockerfile
", "votes": 0, "replies": [] }, { "id": 821746, "author_name": "rs_works", "author_url": "", "post_date": "2020-04-26T11:46:26.153000", "content": "Congrats
", "votes": 0, "replies": [] }, { "id": 822965, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2020-04-27T09:35:02.727000", "content": "I am really sorry for you guys. One thing though is that Decord and DALI have been reported as unstable (often crashed) in this public kernel, which advised to avoid them :
\n\nhttps://www.kaggle.com/mpware/videoreaders-benchmark
\n\nSo Decord can potentially produce memory leak and we can have crash and cannot catch errors with DALI .
", "votes": 0, "replies": [] }, { "id": 823009, "author_name": "ivan", "author_url": "", "post_date": "2020-04-27T10:24:54.377000", "content": "@olegtrott I think a docker file is tied to a specific competition, but I'm not sure which docker file was provided for the host.
\n\n@ratthachat BTW, we did not use Decord or DALI, we used Nvidia Video Codec SDK directly.
", "votes": 1, "replies": [] }, { "id": 823169, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-04-27T13:19:41.343000", "content": "@ratthachat Indeed, Decord GPU had the memory leak but I used the cpu version for that reason.
", "votes": 1, "replies": [] }, { "id": 823233, "author_name": "WestLake", "author_url": "", "post_date": "2020-04-27T14:00:55.477000", "content": "@ratthachat I do not use DALI or Decord. But I use nvidia's tensorRT to accelerate inference. There's a common issue that any nvida(cuda , cuda driver) related packages will cause invalid predictions.
", "votes": 2, "replies": [] }, { "id": 827800, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-04-30T14:49:42.880000", "content": "I agree if test data is not feasible we can at least ask for dockerfiles which was used for held out inference. Which shouldn't be a much ask IMHO. I am not pushing for medals at this point but the curiosity about what made or submissions invalid or all 0.5s is killing me :( if we could only learn something about what would our results be in private LB if we didn't have these technical difficult would be just helpful.
", "votes": 1, "replies": [] } ] }, { "id": 818594, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-04-24T01:46:19.377000", "content": "I would strongly recommend the organizers put their private board data on kaggle and re-run the submitted notebooks. All the submitted notebooks got error and I got no results. This is unbelievable and I'm sure the notebook I submitted is runable.
", "votes": 13, "replies": [] }, { "id": 818487, "author_name": "CUI Hao", "author_url": "", "post_date": "2020-04-23T23:28:13.377000", "content": "🤔 Can't believe it! From Public 37th (was ~48th I remember) to Private 4th.
", "votes": 11, "replies": [ { "id": 818493, "author_name": "CUI Hao", "author_url": "", "post_date": "2020-04-23T23:39:51.383000", "content": "Hopefully this is final & real. I love money 😂
\n\nBut it is really strange that 4 of the public-LB top 5 dropped to nowhere in the private-LB. Very curious about what happened...
", "votes": 3, "replies": [] }, { "id": 818524, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-24T00:30:40.210000", "content": "Hi @cuihaoleo can we merge in this competition? :D
", "votes": 0, "replies": [] }, { "id": 818525, "author_name": "BokingChen", "author_url": "", "post_date": "2020-04-24T00:32:16.657000", "content": "Congratulations， have you clipped the prediction values?
", "votes": 0, "replies": [] }, { "id": 818538, "author_name": "Shane", "author_url": "", "post_date": "2020-04-24T00:49:43.147000", "content": "I'm merging for the other 1 empty slot. 💯
", "votes": 1, "replies": [] } ] }, { "id": 818937, "author_name": "DinoJuice", "author_url": "", "post_date": "2020-04-24T08:14:44.040000", "content": "I went from public 31st place to nowhere as many others here. All notebooks failed at stage 2; whatever that means.\nMy notebook has taken precautions to scale down images for face detection as I ran into errors (probably out of CUDA memory) during submissions in the past. That kind of rules out cuda memory issues as a reason for the failures.
\n\nMy video processing and face detection code is enclosed in a try/catch statement, which would result in 0.5 predictions for any video that causes an exception.
\n\nTo state the obvious: this is very disappointing. I expected more from a high profile competition such as this. Imagine how the top 10 teams must feel now that most are gone.
", "votes": 10, "replies": [] }, { "id": 819816, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-04-24T22:30:33.613000", "content": "A few clarifications:
\n\nSorry, we are not able to provide team-specific feedback or debug your submissions.
", "votes": 7, "replies": [ { "id": 819826, "author_name": "ivan", "author_url": "", "post_date": "2020-04-24T22:57:17.170000", "content": "@juliaelliott is there any chance that the re-run was performed in the different environment (docker image)? In particular, with a different CUDA version?
", "votes": 1, "replies": [] }, { "id": 819858, "author_name": "lev", "author_url": "", "post_date": "2020-04-25T00:01:26.527000", "content": "Is it possible to find out the reason for disqualification? As far as my team's aware, we did not break any rules. It might save everyone the trouble of going through the compliance review, if Kaggle tells us the reason behind the decision.
", "votes": 0, "replies": [] }, { "id": 819866, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-04-25T00:08:50.840000", "content": "@lytic, the docker images were replicated precisely (Kaggle's engineers handed this over to the host team's engineers), and errors were investigated to confirm they were not attributable to subtle image or environment differences.
\n\n@liovch, sorry, no. I've provided the categories of errors, but we're not able to support team-by-team debugging/error investigation.
", "votes": 2, "replies": [] }, { "id": 819871, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-04-25T00:12:58.963000", "content": "Could you be able to share the 10k private test set? There are few teams which got all 0.5 predictions apparently. We can re-run our notebooks with the test set to verify that the fault is on our code.
", "votes": 5, "replies": [] }, { "id": 819873, "author_name": "lev", "author_url": "", "post_date": "2020-04-25T00:23:29.080000", "content": "Thanks for reply @juliaelliott, if Kaggle is unable to provide a reason for eliminating our team from the competition, we will apply for compliance review submission.
", "votes": 3, "replies": [] }, { "id": 819874, "author_name": "WestLake", "author_url": "", "post_date": "2020-04-25T00:27:38.043000", "content": "@juliaelliott could you release part of private test data（5%）and move to kaggle's platform? then we can check why we got all 0.5 predictions.
\n\nmy notebook will get 0.5 prediction when run on same docker image with different cuda driver version
", "votes": 16, "replies": [] }, { "id": 820032, "author_name": "wengzejia", "author_url": "", "post_date": "2020-04-25T05:22:31.677000", "content": "@juliaelliott The results of online table statistics show that at least 9 of the top 120 teams have encountered similar situations and most of them are disqualified. \nThis ratio(9/120) is higher than 6.5%. \nEven at 6.5%, this is still a very high percentage, isn't it?
\n\nI want to know whether you have any plans to deal with the problem. \nIf (kagglers and facebookers)'s answers are no, I have no choice, because the initiative is in your hands.
", "votes": 6, "replies": [] }, { "id": 821720, "author_name": "MPWARE", "author_url": "", "post_date": "2020-04-26T11:16:11.347000", "content": "@juliaelliott I think Kaggle should try to do something for all teams in the 6.5% complaining. It's unfair for all who worked hard in this competition (even maybe some teams in the 6.5% are in the money). It's a matter of image/reputation of Kaggle. The initial deal was if your submission works then final evaluation should work. 6.5% of failure means that this format of competition should be stopped as too random (no way to troubleshot). I'm not saying you will be able to reach 100% of successful re-run but you should consider some actions.
", "votes": 14, "replies": [] }, { "id": 821770, "author_name": "Dayin", "author_url": "", "post_date": "2020-04-26T12:10:49.983000", "content": "\n1 - 3570 / (2265 * 2) = 21.2%, about 21.2% submissions went error.\n(21.2 - 6.5) * 2 + 6.5 = 35.9%, about 35.9% teams has at least one error submission.
As an experienced organizer, Kaggle or Facebook should give a solution or decision to the 6.5% errors or incorrect scores.
", "votes": 9, "replies": [] }, { "id": 821964, "author_name": "Rajaram G", "author_url": "", "post_date": "2020-04-26T14:41:36.147000", "content": "Is there difference between bullet points 2 and 3?
\n\nI see my all my submissions in \"Error\" state. When I hover over the error message for selected entries, I see \"Invalidated for stage 2\".
\n\nNot clear if this means I broke some competition rule and was disqualified, Or was there a problem in running the selected notebooks. Hope it is the latter. Wish error messages could have been more specific. A lot of people have put a lot of hard work into this competition. Clear error message/ notebook log will at least help us avoid these mistakes in future competitions.
", "votes": 3, "replies": [] }, { "id": 822910, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-04-27T08:49:16.077000", "content": "Rethinking why you held this competition. For giving out money? No, you want a better solution for deep fake detection. I don't think this result can help you in realistic applications, it can't help the development of deep fake detection either. This competition, which costs a lot of money, I don't think it should be ended this way.
", "votes": 2, "replies": [] } ] }, { "id": 818534, "author_name": "Marius Wanko", "author_url": "", "post_date": "2020-04-24T00:46:24.677000", "content": "May I kindly ask to re-run the final solutions of the four teams that where \"in the medals\" on public LB, but aparently failed to read any video from the private test set (score==0.69314)?
\n\nI can speak only for myself, but the only way I see my script could have failed is the test videos no longer provided in \"../input/deepfake-detection-challenge/test_videos/\" but only in \"./test_videos.zip\". Or can anyone disconfirm this?
\n\nAnyway, congrats to all winners and thanks for the amazing contributions published in the kernels and discussions!
", "votes": 5, "replies": [] }, { "id": 818928, "author_name": "Syndicate", "author_url": "", "post_date": "2020-04-24T08:05:55.533000", "content": "Real sad to see lots of teams dropping ranks due to failed run. Though my team sort of benefitted from the shake-ups , but I do hope that the host will figure out the cause and provide us some solid feedbacks.
", "votes": 6, "replies": [] }, { "id": 818676, "author_name": "Raman", "author_url": "", "post_date": "2020-04-24T03:32:52.607000", "content": "Congrats to the winners! 👍 What a surge from the all-star team, I feel blessed to have met many of you during the KaggleDays!\nAnd what an impressive top-position consistency from Selim Seferbekov @selimsef ! Віншаванні!!💪 👋 😃
", "votes": 6, "replies": [] }, { "id": 879140, "author_name": "Jaideep", "author_url": "", "post_date": "2020-06-09T08:22:19.590000", "content": "@philculliton @juliaelliott \ndo you know by when we will be awarded with medals ?
", "votes": 4, "replies": [ { "id": 882264, "author_name": "Kirill Balakhonov", "author_url": "", "post_date": "2020-06-11T17:13:05.840000", "content": "Never😭
", "votes": 1, "replies": [] }, { "id": 882374, "author_name": "Jaideep", "author_url": "", "post_date": "2020-06-11T18:48:32.140000", "content": "waiting to hear back from @juliaelliott @philculliton
", "votes": 1, "replies": [] } ] }, { "id": 821833, "author_name": "maralski", "author_url": "", "post_date": "2020-04-26T13:00:25.593000", "content": "I think appealing on these forums is just resulting in the same standard clause. Only thing that is going to make a difference is to make it public.
\n\nAnthony Goldbloom - CEO Kaggle\nhttps://twitter.com/antgoldbloom\nhttps://www.linkedin.com/in/anthonygoldbloom\nhttps://www.kaggle.com/about/team
", "votes": 4, "replies": [ { "id": 822895, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-04-27T08:27:06.077000", "content": "https://www.kaggle.com/antgoldbloom/contact . I think we should email to him and make him realize how poor their workers are 👎 I have also commented under his twitter. Hope he will see it.
", "votes": 0, "replies": [] } ] }, { "id": 818650, "author_name": "Shane", "author_url": "", "post_date": "2020-04-24T03:04:12.663000", "content": "Hi folks, it's really sorry to see lots of people got reshuffled. Anyway, my dream to hunt gold from this competition break now. :| (can get your feelings). I'm sure many of you have considered the problem, here are my general 2 cents here: (definitely not for metal range players in public LB)
\n\nThere are several things may break the pipeline up:\n- OpenCV when reading videos.\n- Face decorator GPU memory overflow.\n- CNN models compete with face decorators on GPU memory.
\n\nFor the last 2 issues, we will need to clear the GPU memory, shrink batch process size (16 -> 8 -> 4 -> 2 -> 1), or even move some of the computation to CPU.
\n\nAlso, the resolution of the video may vary, so probably we'll need to normalize the video (say 1920 * 1080 / 1080 * 1920).
", "votes": 4, "replies": [] }, { "id": 818459, "author_name": "Shane", "author_url": "", "post_date": "2020-04-23T22:43:15.970000", "content": "Grats, All Faces Are Real. :)
", "votes": 5, "replies": [] }, { "id": 818803, "author_name": "Rueven", "author_url": "", "post_date": "2020-04-24T06:12:26.633000", "content": "\n
Can believe it, We can get a score of 0.34 in public-lb, but we don't get a score in private-lb. Why is that?
", "votes": 3, "replies": [ { "id": 818812, "author_name": "Rueven", "author_url": "", "post_date": "2020-04-24T06:15:57.840000", "content": "@philculliton The name of my team is relax, I don't know what happened, can you give me more detailed error information?
\n\nNow that both the public test set and the private test set have lost my name, am I breaking any rules?
", "votes": 1, "replies": [] }, { "id": 818833, "author_name": "petya", "author_url": "", "post_date": "2020-04-24T06:31:15.693000", "content": "I suppose they haven't run any of non-selected entires on their private lb, thus the errors. I've also got errors everywhere but on those 2 final entries.
", "votes": 3, "replies": [] }, { "id": 818839, "author_name": "Rueven", "author_url": "", "post_date": "2020-04-24T06:37:41.443000", "content": "@petya The problem is that the entires I chose did not private lb either so I am now in a state of no ranking
", "votes": 0, "replies": [] }, { "id": 818869, "author_name": "petya", "author_url": "", "post_date": "2020-04-24T07:08:48.600000", "content": "@ruyiwei I guess the competition holders will have to address the issue with most notebooks scoring around 0.69, which is the baseline. Something clearly has triggerred an exception for those guys, and those who didn't have try\\except at all got submission errors.
", "votes": 0, "replies": [] }, { "id": 818873, "author_name": "Rueven", "author_url": "", "post_date": "2020-04-24T07:15:29.843000", "content": "@petya It's hard for me to understand why the top 10 in public_lb is now 900 +
", "votes": 0, "replies": [] } ] }, { "id": 818506, "author_name": "lev", "author_url": "", "post_date": "2020-04-23T23:59:37.077000", "content": "Our team is gone from both public and private leaderboard. Team name is \"levsha\". Could you please explain what happened?
", "votes": 3, "replies": [ { "id": 818509, "author_name": "apppa", "author_url": "", "post_date": "2020-04-24T00:08:10.550000", "content": "Same. Didn't use any external data, have only one account. Very disappointed, please explain what I'm guilty of
", "votes": 1, "replies": [] }, { "id": 818510, "author_name": "Daniel", "author_url": "", "post_date": "2020-04-24T00:08:40.437000", "content": "It also happened to us. What kind of error does it show? We got \"invalidated for stage 2 error\".
", "votes": 1, "replies": [] }, { "id": 818512, "author_name": "lev", "author_url": "", "post_date": "2020-04-24T00:13:37.697000", "content": "All our 155 submissions (including the 2 selected for private leaderboard) show \"Invalidated for stage 2 error\"
", "votes": 1, "replies": [] }, { "id": 818513, "author_name": "Daniel", "author_url": "", "post_date": "2020-04-24T00:17:27.170000", "content": "@liovch, looks like we face the same situation. Any idea on what \"Invalidated for stage 2 error\" mean?
", "votes": 1, "replies": [] }, { "id": 818515, "author_name": "apppa", "author_url": "", "post_date": "2020-04-24T00:19:09.610000", "content": "I have the same message, it seems we were disqualified
", "votes": 1, "replies": [] }, { "id": 818522, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-24T00:29:54.397000", "content": "It's possible 2 of your submissions were failed. In my case, I have only 1/2 submission rescored successful. I don't know what happened to the failed one.\n
what error message did you get on failed one?
", "votes": 0, "replies": [] }, { "id": 818528, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-24T00:34:45.767000", "content": "Just Error as all other un-scored submissions.
", "votes": 1, "replies": [] }, { "id": 818532, "author_name": "apppa", "author_url": "", "post_date": "2020-04-24T00:42:44.847000", "content": "then I probably have a mistake in the code, better than being disqualified )
", "votes": 0, "replies": [] }, { "id": 818557, "author_name": "Daniel", "author_url": "", "post_date": "2020-04-24T01:07:43.920000", "content": "@khahuras , what error message did you get when you move the mouse onto the i icon?
", "votes": 0, "replies": [] }, { "id": 818586, "author_name": "lev", "author_url": "", "post_date": "2020-04-24T01:34:23.480000", "content": "@khahuras If the submissions selected for private leaderboard failed due to some error, it doesn't explain why the score for all the other submissions on the public leaderboard changed to \"Error\".
", "votes": 0, "replies": [] }, { "id": 818615, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-24T02:22:26.197000", "content": "The pop-up error when hovering the mouse into \"Error\" is also the same as all other errors: \"Invalidated for stage 2\"
", "votes": 0, "replies": [] }, { "id": 818638, "author_name": "maralski", "author_url": "", "post_date": "2020-04-24T02:42:59.747000", "content": "@khahuras what was the different between your 1st failed notebook and 2nd successful notebook?
", "votes": 0, "replies": [] }, { "id": 818645, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-24T02:53:40.513000", "content": "@maralski The selected but failed notebook uses more models, especially some LSTM. In my successful submission, there is no LSTM. So basically they are very different. I'm sure the LSTM/GRU sort of things caused the error. Maybe during loading, or during inference. We never know. Personally I hope this failed submission score better than the other, but it failed :(
", "votes": 0, "replies": [] }, { "id": 818881, "author_name": "MPWARE", "author_url": "", "post_date": "2020-04-24T07:20:16.063000", "content": "Same here but only for one of the submission. The differences between the 2 submissions:\n1 - Submission succeeded: 1 worker - 3 models ensembled - 73 frames.\n2 - Submission failed: 4 workers - 4 models ensembled - 83 frames.\nAs submission#2 did not get score, even with the try/catch, I suspect it could be a memory issue. And such issue could be explained if a lot images in final set are larger than the ones for public LB. There was no way to anticipate/test it.
\n\n
hi, @philculliton ,my notebook will output two files(pred_full.csv, submission.csv), is it posible the failure caused by using pred_full.csv when calculation the score. \nIs it possible to provide log information of the re-run failure.
", "votes": 1, "replies": [] }, { "id": 819114, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-04-24T11:09:36.397000", "content": "💔 👎 😫
\n\n
No sharing, I used only competition data. Kernels worked! Why I get error?
", "votes": 2, "replies": [] } ] }, { "id": 818794, "author_name": "yzf233", "author_url": "", "post_date": "2020-04-24T06:08:46.313000", "content": "Congratulations to final medallists. Looking at the private leaderboard for the first 50 teams, there are at least 35 of 50 teams whose ranking changed in double digits. AND the winner's scores in private leaderboard and public leaderboard differ by 0.23 on log loss of a binary classification task. It arouses my great curiosity on what's the data distribution in private test set.\nTherefore, I suggest the host to consider releasing some data samples of private test set. Furthermore, as the data domain varies so greatly in private set and public set, would it be more fair to integrat both the public set ranking and the private set ranking?
", "votes": 2, "replies": [] }, { "id": 825577, "author_name": "Qinyue Chen", "author_url": "", "post_date": "2020-04-29T05:39:45.430000", "content": "Wonderful . I am going to translate it to Chinese
", "votes": 0, "replies": [] }, { "id": 818671, "author_name": "maralski", "author_url": "", "post_date": "2020-04-24T03:25:29.073000", "content": "Reviewed my code and only thing I can explain that would have resulted in failure was I mapped my models to 'cuda:0'. Is there a situation where GPU ID was not 0 when run on private set?
", "votes": 0, "replies": [ { "id": 819010, "author_name": "Polychronis Charitidis", "author_url": "", "post_date": "2020-04-24T09:22:36.323000", "content": "I did that too and my code run succesfully, so I do not think that's the issue. Hopefully, Kaggle will figure this out.
", "votes": 0, "replies": [] }, { "id": 819117, "author_name": "maralski", "author_url": "", "post_date": "2020-04-24T11:11:44.797000", "content": "Good to know. Thanks
", "votes": 0, "replies": [] } ] }, { "id": 818644, "author_name": "Pankaj Mathur", "author_url": "", "post_date": "2020-04-24T02:52:56.103000", "content": "First of all congrats to winners, and final medalist. I know You all put a-lot of hard work and countless hours to achieve this, so kudos to all of you.
\n\nNow, Once again, most of us, including me, overfitted Public LB.... This is becoming common story of many Kaggle Kernel Competition, where high scoring overfitted public LB kernels will release earlier in competition by few and then most of contributors will end up following only these few directions discussed in these kernel and then producing more and more overfitted models on public LB and discuss more on more those direction.\nMany of less confident contributors like me, sometimes end up abandoning many possible less overfitted, generalized model direction in lieu of these public kernels score, discussion and approaches share by them.\nand then only In the end private LB will reveal how overfitted these directions were....
\n\nI too end up submitting overfitted model by abandoning my earlier generalized architecture approaches.
\n\nSomehow Only People who kept faith on their generalized original model till the end without getting distracted by those top scoring public kernels are the one who came out on top. \nI am seeing people from public LB in 1000's reaching bronze in private and vice versa. My own overfitted model drop from bronze in public to 900 in private.
\n\nPlease share if you happen to have any similar story.
", "votes": 1, "replies": [] }, { "id": 826645, "author_name": "Shubham Sharma", "author_url": "", "post_date": "2020-04-29T18:56:56.243000", "content": "So appealing and exciting.
", "votes": 0, "replies": [] }, { "id": 823249, "author_name": "Jone", "author_url": "", "post_date": "2020-04-27T14:14:03.403000", "content": "Eat melon
", "votes": 0, "replies": [] }, { "id": 820390, "author_name": "maralski", "author_url": "", "post_date": "2020-04-25T12:07:18.633000", "content": "@juliaelliott can you clarify what was timeout for private test set?
", "votes": 0, "replies": [] }, { "id": 821202, "author_name": "", "author_url": "", "post_date": "2020-04-26T02:20:00.363000", "content": "", "votes": 0, "replies": [] }, { "id": 820117, "author_name": "", "author_url": "", "post_date": "2020-04-25T06:36:57.410000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "818448": "The private leaderboard has been revealed. The private scores were based on a re-run of participants’ selected submissions on the host’s private test set.\n\nCongratulations to our winners! Please note that the leaderboard is still subject to verification of the winners' full code by the host team.\n\nThere was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples). We were all frankly very impressed by that number - thanks to everyone for your hard work and attention to detail.\n\nIt was a real pleasure to work on this competition with the host team and all of you. Thank you!", "823327": "I did not take an active part in this competition but I regret the way Kaggle is turning a blind eye and a deaf ear to the issues raised here.\n\nDo you guys realise people spend a huge amount of time, effort and computational resources in order to provide good and cutting edge solutions in such competitions ? \n\nI think you should give them a full and reasonable feedback when their submissions are discarded in this way. Otherwise, you may discourage some people from participating in such highly demanding competitions in the future .", "822509": "WOW - MY GOODNESS! What a complete and utter breakdown! Kaggle and Facebook need to own what a poor contest this was, and offer a sincere apology to the engaged community here. My team saw the vast difference between the provided data (with killer models we created over months of work and testing) and the Leader Board and we correctly assumed if the data was so poorly put together with relation to the LB, there was certainly going to be more problems down the road. How prescient. I feel for the participants who worked to the very end to get in top 30, only to be bounced out by a very poorly designed challenge.", "818596": "Can't believe it, our team ranked 3rd Public-LB. but get score 0.69314 in private board(2 submission with same score 0.69314). Which is a score of all prediction is 0.5. \nThis may be cause by (try.. exception). When the code throw exception the prediction will be 0.5. private dataset is held outside of Kaggle's platform, is the running environment is different from Kaggle's environment ?( i.e. system os version, cuda version, cuda driver version ....). \nIs it possible to provided detail log information of the re-run? Or just copy the private dataset to the kaggle's platform and re-run the notebook? @philculliton ", "818594": "I would strongly recommend the organizers put their private board data on kaggle and re-run the submitted notebooks. All the submitted notebooks got error and I got no results. This is unbelievable and I'm sure the notebook I submitted is runable.", "818487": "🤔 Can't believe it! From Public 37th (was ~48th I remember) to Private 4th.", "818937": "I went from public 31st place to nowhere as many others here. All notebooks failed at stage 2; whatever that means.\nMy notebook has taken precautions to scale down images for face detection as I ran into errors (probably out of CUDA memory) during submissions in the past. That kind of rules out cuda memory issues as a reason for the failures.\n\nMy video processing and face detection code is enclosed in a try/catch statement, which would result in 0.5 predictions for any video that causes an exception.\n\nTo state the obvious: this is very disappointing. I expected more from a high profile competition such as this. Imagine how the top 10 teams must feel now that most are gone.", "819816": "A few clarifications:\n\n- There were ~10k videos in the test set.\n- If you don't see any submissions for the competition, then your team was removed from the competition due to holding multiple accounts or file-sharing across teams. These removals are made with very conclusive evidence, but if you feel you were improperly removed, you are welcome to make a [compliance review submission](https://www.kaggle.com/compliance).\n- All submissions which were *not selected* will be shown as errored in your submissions, because we did not re-run or score those.\n- Here are the reasons you may have selected submissions which were invalidated, in rough order of how common the issue was:\n - Submission produced no or null predictions on some portion of the test set\n - Submission produced predictions but with the wrong number of rows\n - Your notebooks or linked notebooks/datasets for selected submissions were deleted, and therefore we could not re-run them\n - Insufficient error handling & Misc code failures (including out of memory & timeout errors)\n- As Phil mentioned, the private re-run success rate was ~94% teams \n\n**Sorry, we are not able to provide team-specific feedback or debug your submissions.**", "818534": "May I kindly ask to re-run the final solutions of the four teams that where \"in the medals\" on public LB, but aparently failed to read any video from the private test set (score==0.69314)?\n\nI can speak only for myself, but the only way I see my script could have failed is the test videos no longer provided in \"../input/deepfake-detection-challenge/test_videos/\" but only in \"./test_videos.zip\". Or can anyone disconfirm this?\n\nAnyway, congrats to all winners and thanks for the amazing contributions published in the kernels and discussions!", "818928": "Real sad to see lots of teams dropping ranks due to failed run. Though my team sort of benefitted from the shake-ups , but I do hope that the host will figure out the cause and provide us some solid feedbacks.", "818676": "Congrats to the winners! 👍 What a surge from the all-star team, I feel blessed to have met many of you during the KaggleDays!\nAnd what an impressive top-position consistency from Selim Seferbekov @selimsef ! Віншаванні!!💪 👋 😃 ", "879140": "@philculliton @juliaelliott \ndo you know by when we will be awarded with medals ?", "821833": "I think appealing on these forums is just resulting in the same standard clause. Only thing that is going to make a difference is to make it public.\n\nAnthony Goldbloom - CEO Kaggle\nhttps://twitter.com/antgoldbloom\nhttps://www.linkedin.com/in/anthonygoldbloom\nhttps://www.kaggle.com/about/team", "818650": "Hi folks, it's really sorry to see lots of people got reshuffled. Anyway, my dream to hunt gold from this competition break now. :| (can get your feelings). I'm sure many of you have considered the problem, here are my general 2 cents here: (definitely not for metal range players in public LB)\n\nThere are several things may break the pipeline up:\n- OpenCV when reading videos.\n- Face decorator GPU memory overflow.\n- CNN models compete with face decorators on GPU memory.\n\nFor the last 2 issues, we will need to clear the GPU memory, shrink batch process size (16 -> 8 -> 4 -> 2 -> 1), or even move some of the computation to CPU. \n\nAlso, the resolution of the video may vary, so probably we'll need to normalize the video (say 1920 * 1080 / 1080 * 1920).\n", "818459": "Grats, All Faces Are Real. :) ", "818803": "\n\n\n@philculliton\n\nCan believe it, We can get a score of 0.34 in public-lb, but we don't get a score in private-lb. Why is that?", "818506": "Our team is gone from both public and private leaderboard. Team name is \"levsha\". Could you please explain what happened?", "818476": "hi, @philculliton ,my notebook will output two files(pred\\_full.csv, submission.csv), is it posible the failure caused by using pred\\_full.csv when calculation the score. \nIs it possible to provide log information of the re-run failure.", "819114": "💔 👎 😫 \n\n\n", "818794": "Congratulations to final medallists. Looking at the private leaderboard for the first 50 teams, there are at least 35 of 50 teams whose ranking changed in double digits. AND the winner's scores in private leaderboard and public leaderboard differ by 0.23 on log loss of a binary classification task. It arouses my great curiosity on what's the data distribution in private test set.\nTherefore, I suggest the host to consider releasing some data samples of private test set. Furthermore, as the data domain varies so greatly in private set and public set, would it be more fair to integrat both the public set ranking and the private set ranking?", "825577": "Wonderful . I am going to translate it to Chinese", "818671": "Reviewed my code and only thing I can explain that would have resulted in failure was I mapped my models to 'cuda:0'. Is there a situation where GPU ID was not 0 when run on private set?\n", "818644": "First of all congrats to winners, and final medalist. I know You all put a-lot of hard work and countless hours to achieve this, so kudos to all of you.\n\nNow, Once again, most of us, including me, overfitted Public LB.... This is becoming common story of many Kaggle Kernel Competition, where high scoring overfitted public LB kernels will release earlier in competition by few and then most of contributors will end up following only these few directions discussed in these kernel and then producing more and more overfitted models on public LB and discuss more on more those direction.\nMany of less confident contributors like me, sometimes end up abandoning many possible less overfitted, generalized model direction in lieu of these public kernels score, discussion and approaches share by them.\nand then only In the end private LB will reveal how overfitted these directions were.... \n\nI too end up submitting overfitted model by abandoning my earlier generalized architecture approaches.\n\nSomehow Only People who kept faith on their generalized original model till the end without getting distracted by those top scoring public kernels are the one who came out on top. \nI am seeing people from public LB in 1000's reaching bronze in private and vice versa. My own overfitted model drop from bronze in public to 900 in private. \n\nPlease share if you happen to have any similar story. ", "826645": "So appealing and exciting.", "823249": "Eat melon", "820390": "@juliaelliott can you clarify what was timeout for private test set?", "821202": "", "820117": "" } }