![\"trump\"](\"https://pbs.twimg.com/media/Eale1TWXsA07AV4?format=png&name=small\")
Kaggle, Facebook Host Team and Fellow Competitors:
\n\nFirst of all, we want to put on record our gratitude for Kaggle and the Facebook host team for putting the effort into creating the dataset and hosting this competition, and we give our congratulations to all eventual prize winners.
\n\nWe'd like to use this statement to further explain the circumstances which led to our winning solution being voided, and our position on the LB being moved with accordance to our second solution.
\n\nIn anticipation of shake-up of the competition on the private LB, we prepared our two solutions which finished with private LB scores 0.42320 and 0.44531 respectively. For the 0.44531 solution, which scored better on the public LB, we used competition data only and an unweighted mean of 12 models: this is the solution that enabled us to retain our 7th position on the LB. For our original winning solution (0.42320) we mixed 6 models trained using competition data with 9 models trained with some additional external data (our more adventurous submission).
\n\nFor our original winning model, we used the following additional data:
\n\nThe flickrface dataset: we used a resized version of this dataset. A few of these images had licenses which didn't allow commercial use, so in line with clarifications from Kaggle in the external data thread, we used the license information available from the original github to select and train only on images with license types that are acceptable for this competition (CC-BY, Public Domain Mark 1.0, Public Domain CC0 1.0, or U.S. Government Works)
Youtube videos images: we manually created a face image dataset from a handful of youtube videos with CC-BY license, which explicitly allows for commercial use.
We chose these data sources with the belief that they met the rules on external data, specifically that external data must be \"available to use by all participants of the competition for purposes of the competition at no cost to the other participants\", and the additional statements in the external data thread that they must be available for commercial use and not restricted to academics etc.
\n\nHowever, in our discussions with Facebook and Kaggle, we were told that despite fulfilling this we were contravening the rules on Winning Submission Documentation:
\n\n\n\n\nWINNING SUBMISSION DOCUMENTATION (Section 4 of the Competition-Specific Rules)\n In addition to compliance with the Kaggle Documentation Guidelines at https://www.kaggle.com/WinningModelDocumentationGuidelines, the winning submission documentation must conform with the following guidelines:
\n \nA. If any part of the submission documentation depicts, identifies, or includes any person that is not an individual participant or Team member, you must have all permissions and rights from the individual depicted, identified, or included and you agree to provide Competition Sponsor and PAI with written confirmation of those permissions and rights upon request.
\n \nB. Submission documentation must not infringe, misappropriate, or violate any rights of any third party including, without limitation, copyright (including moral rights), trademark, trade secret, patent or rights of privacy or publicity.
\n
Specifically, we were asked to provide \"additional permissions or licenses from individuals appearing in [our] external dataset\". Unfortunately, since the data was from public datasets, we didn't have specific written permission from each individual appearing in them, nor did we have any way of identifying these individuals. We didn't realise while competing that external data in this competition falls under 'documentation' as well as the external data rules, so we did not secure these permissions from individuals depicted above and beyond the licensing requirements.
\n\nWe suspect that most competitors also did not realise these additional restrictions existed - we are unable to find any data posted in the External Data Thread which meets this threshold with a brief scan. During the competition, the rules on external data were repeatedly clarified, so this leaves us wondering why Kaggle never took the opportunity to clarify that external data must additionally follow the more restrictive rules for winning submission documentation.
\n\nAn additional concern brought to us was that Facebook felt some of our external data \"clearly appears to infringe third party rights\" despite being labelled as CC-BY (it's not clear what data they were referring to specifically). Even if this were the case, it seems unreasonable to us that a Kaggle team should have to trace and verify that someone who publishes a dataset themselves has the rights to do so, and that we should have to engage rights clearance services in order to make a competition submission - it was suggested to us that we could have run our external data past our lawyer before making our submissions.
\n\nWhile we feel that these extra rules could have been made clear during the competition, and we hope that Kaggle will begin to clarify these rules in future competitions, we understand that there is little we can do in this instance. We have had a constructive call with both Kaggle and Facebook which we thank them for. After this call, it was agreed that because we did not knowingly seek to undermine any rules, that our submission that did not use any external data should be allowed to remain and only the winning submission is to be disqualified.
\n\nThat being said, we are very disappointed by this outcome after spending so many months on the competition. Successful Kaggle competitions rely on a trust between competitors and Kaggle that the rules will be fairly explained and applied, and this trust has been damaged. We welcome any thoughts from the community on this matter.
\n\nGiba, Mikel, Yifan, Gary and Qishen
\nAll Faces are Real
Kaggle, Facebook Host Team and Fellow Competitors:
\n\nFirst of all, we want to put on record our gratitude for Kaggle and the Facebook host team for putting the effort into creating the dataset and hosting this competition, and we give our congratulations to all eventual prize winners.
\n\nWe'd like to use this statement to further explain the circumstances which led to our winning solution being voided, and our position on the LB being moved with accordance to our second solution.
\n\nIn anticipation of shake-up of the competition on the private LB, we prepared our two solutions which finished with private LB scores 0.42320 and 0.44531 respectively. For the 0.44531 solution, which scored better on the public LB, we used competition data only and an unweighted mean of 12 models: this is the solution that enabled us to retain our 7th position on the LB. For our original winning solution (0.42320) we mixed 6 models trained using competition data with 9 models trained with some additional external data (our more adventurous submission).
\n\nFor our original winning model, we used the following additional data:
\n\nThe flickrface dataset: we used a resized version of this dataset. A few of these images had licenses which didn't allow commercial use, so in line with clarifications from Kaggle in the external data thread, we used the license information available from the original github to select and train only on images with license types that are acceptable for this competition (CC-BY, Public Domain Mark 1.0, Public Domain CC0 1.0, or U.S. Government Works)
Youtube videos images: we manually created a face image dataset from a handful of youtube videos with CC-BY license, which explicitly allows for commercial use.
We chose these data sources with the belief that they met the rules on external data, specifically that external data must be \"available to use by all participants of the competition for purposes of the competition at no cost to the other participants\", and the additional statements in the external data thread that they must be available for commercial use and not restricted to academics etc.
\n\nHowever, in our discussions with Facebook and Kaggle, we were told that despite fulfilling this we were contravening the rules on Winning Submission Documentation:
\n\n\n\n\nWINNING SUBMISSION DOCUMENTATION (Section 4 of the Competition-Specific Rules)\n In addition to compliance with the Kaggle Documentation Guidelines at https://www.kaggle.com/WinningModelDocumentationGuidelines, the winning submission documentation must conform with the following guidelines:
\n \nA. If any part of the submission documentation depicts, identifies, or includes any person that is not an individual participant or Team member, you must have all permissions and rights from the individual depicted, identified, or included and you agree to provide Competition Sponsor and PAI with written confirmation of those permissions and rights upon request.
\n \nB. Submission documentation must not infringe, misappropriate, or violate any rights of any third party including, without limitation, copyright (including moral rights), trademark, trade secret, patent or rights of privacy or publicity.
\n
Specifically, we were asked to provide \"additional permissions or licenses from individuals appearing in [our] external dataset\". Unfortunately, since the data was from public datasets, we didn't have specific written permission from each individual appearing in them, nor did we have any way of identifying these individuals. We didn't realise while competing that external data in this competition falls under 'documentation' as well as the external data rules, so we did not secure these permissions from individuals depicted above and beyond the licensing requirements.
\n\nWe suspect that most competitors also did not realise these additional restrictions existed - we are unable to find any data posted in the External Data Thread which meets this threshold with a brief scan. During the competition, the rules on external data were repeatedly clarified, so this leaves us wondering why Kaggle never took the opportunity to clarify that external data must additionally follow the more restrictive rules for winning submission documentation.
\n\nAn additional concern brought to us was that Facebook felt some of our external data \"clearly appears to infringe third party rights\" despite being labelled as CC-BY (it's not clear what data they were referring to specifically). Even if this were the case, it seems unreasonable to us that a Kaggle team should have to trace and verify that someone who publishes a dataset themselves has the rights to do so, and that we should have to engage rights clearance services in order to make a competition submission - it was suggested to us that we could have run our external data past our lawyer before making our submissions.
\n\nWhile we feel that these extra rules could have been made clear during the competition, and we hope that Kaggle will begin to clarify these rules in future competitions, we understand that there is little we can do in this instance. We have had a constructive call with both Kaggle and Facebook which we thank them for. After this call, it was agreed that because we did not knowingly seek to undermine any rules, that our submission that did not use any external data should be allowed to remain and only the winning submission is to be disqualified.
\n\nThat being said, we are very disappointed by this outcome after spending so many months on the competition. Successful Kaggle competitions rely on a trust between competitors and Kaggle that the rules will be fairly explained and applied, and this trust has been damaged. We welcome any thoughts from the community on this matter.
\n\nGiba, Mikel, Yifan, Gary and Qishen
\nAll Faces are Real
In the last four years, I have never seen such absurdity on Kaggle. As a Computer Vision guy, I am furious right now. If we go by the logic provided by Facebook for removing @titericz and team, then I have a bunch of points to make:
\n\nImageNet is a public dataset but nowhere we credit or trace the source of the images present in ImageNet before using it. The same goes for COCO, CIFAR, etc. As pointed out by @rwightman the situation is more complex if we add face detection/recognition datasets to the list. So, every time we use ImageNet, should we trace and provide credits for an individual sample even if the dataset is public?
Why stop at datasets? BatchNormalizattion, Dropout, CNNs, etc are patented by Google. Any use of BN or dropout indirectly involves that patent, so why allow that in the first place?
What are the expectations of the host here? Are they expecting us to be a lawyer first to understand such a ridiculous clause buried deep down somewhere and not clarified until the end of the competition?
Let's say we are naive and the claims of Facebook stand correct. So my question for FAIR: Is every researcher who works at FAIR, aware of this clause? If yes, why is it okay for the FAIR team to use these datasets without the consent of the source?
The situation would be worse if it comes to pretrained weights. Any architecture/dataset can be public but that doesn't mean you can directly use the weights without the consent of the person who trained the network. So, where exactly is the borderline?
To the All Faces Are Real team: I am very sorry that it happened to you. I am pretty sure that every sensible Kaggler is standing with you on this issue.
Agree.
\n\nI am starting to think competitions organisers at Facebook couldn't digest \"YouTube videos dataset\" and did the whole knit picking to void the team.
\n\nI wonder what if the team would have used \"Facebook Videos dataset\", that would make some real title in papers for facebook;
\n\n> FacebookAI have tackled DeepFake problem with FB videos platform
\n\nDissapointed more with fact that, Kaggle was complicit. What more could be worse in 2020?
", "rawMarkdown": "Agree. \n\nI am starting to think competitions organisers at Facebook couldn't digest **\"YouTube videos dataset\"** and did the whole knit picking to void the team.\n\nI wonder what if the team would have used **\"Facebook Videos dataset\"**, that would make some real title in papers for facebook;\n\n> FacebookAI have tackled DeepFake problem with FB videos platform\n\nDissapointed more with fact that, Kaggle was complicit. What more could be worse in 2020?", "votes": 15 }, { "id": 884595, "postDate": "2020-06-13T13:27:29.210Z", "content": "I am very much sold on this fact this is more like \"Facebook + Amazon\" vs \"Google\" thing.
", "rawMarkdown": "I am very much sold on this fact this is more like \"Facebook + Amazon\" vs \"Google\" thing.", "votes": 6 } ] }, { "id": 885456, "postDate": "2020-06-14T08:21:11.200Z", "content": "I spent most of Saturday just catching up with friends and reading all the comments via social media and here. Thank you all for your support.
\n\nFor most of last two months, we had spent much time \"behind the scene\" to coordinate with teammates, liaised with kaggle and host team, working with our legal representative to ensure we stay legally informed. Personally speaking, it was a period of significant stress. In a sense, it has been really good to finally share our side of the story with the community, and I can't overstate how heartening it is to see all the supportive messages.
\n\nThank you, everyone
", "rawMarkdown": "I spent most of Saturday just catching up with friends and reading all the comments via social media and here. Thank you all for your support.\n\nFor most of last two months, we had spent much time \"behind the scene\" to coordinate with teammates, liaised with kaggle and host team, working with our legal representative to ensure we stay legally informed. Personally speaking, it was a period of significant stress. In a sense, it has been really good to finally share our side of the story with the community, and I can't overstate how heartening it is to see all the supportive messages. \n\nThank you, everyone ", "votes": 45, "replies": [ { "id": 885504, "postDate": "2020-06-14T09:17:06.493Z", "content": "We are with you on this and we are ready to fight till the end. The community have learned a lot from you guys. If we don't support you on this where a company wants to get away with its wrong decision, then we don't deserve to be a part of the community.
", "rawMarkdown": "We are with you on this and we are ready to fight till the end. The community have learned a lot from you guys. If we don't support you on this where a company wants to get away with its wrong decision, then we don't deserve to be a part of the community. ", "votes": 16 } ] }, { "id": 884242, "postDate": "2020-06-13T09:00:48.243Z", "content": "So many heartbreaking and disappointing discussions after competitions could be avoided if admins more actively reply to concerns of Kagglers during competitions.
\n\nCompetitors are usually really clever, and more frequently than not point out vague aspects of rules early on, specifically with respect to external data. Frequently, potential issues about external data (is it allowed, or not) are raised early on, but are either only answered very late in the competition, or stay completely unanswered. This appears to have been an issue also here in this competition.
\n\nIt seems to me that the strategy is to not reply early enough, see what happens, and then make rulings afterwards. But this harms the integrity and trust of the platform, and demotivates Kagglers heavily.
\n\nI have talked with several others lately, and everyone has the same feeling, that they just don't know what is allowed and what is not allowed anylonger. I am the first person who always wants to follow the rules as precisely as possible, but how can I do that if I don't know them?
", "rawMarkdown": "So many heartbreaking and disappointing discussions after competitions could be avoided if admins more actively reply to concerns of Kagglers during competitions. \n\nCompetitors are usually really clever, and more frequently than not point out vague aspects of rules early on, specifically with respect to external data. Frequently, potential issues about external data (is it allowed, or not) are raised early on, but are either only answered very late in the competition, or stay completely unanswered. This appears to have been an issue also here in this competition.\n\nIt seems to me that the strategy is to not reply early enough, see what happens, and then make rulings afterwards. But this harms the integrity and trust of the platform, and demotivates Kagglers heavily.\n\nI have talked with several others lately, and everyone has the same feeling, that they just don't know what is allowed and what is not allowed anylonger. I am the first person who always wants to follow the rules as precisely as possible, but how can I do that if I don't know them?", "votes": 47, "replies": [ { "id": 884430, "postDate": "2020-06-13T11:20:56.330Z", "rawMarkdown": "", "votes": 22, "isDeleted": true }, { "id": 884629, "postDate": "2020-06-13T14:00:55.257Z", "content": "Fully agree here! Kagglers will often recognize if an external dataset is tricky to use and will ask questions about it in the external data thread. Unfortunately, these questions often go unanswered even after tagging competition hosts multiple times.
\n\nIt is understandable that the external data thread contains many comments and not all can be answered. However, if Kaggle is going to allow external data in a competition, a simple statement like \"so *dataset_X* is not allowed in this competition\" would be more integrous, then repeating vague rules that require legal knowledge to understand.
\n\nA similar situation is playing out in the Jigsaw competition where it is currently not clear what people can and cannot do with the test set (translation, cleaning, etc.). I really hope that the Kaggle hosts can answer these questions so that we can at least mitigate misunderstandings about external data.
", "rawMarkdown": "Fully agree here! Kagglers will often recognize if an external dataset is tricky to use and will ask questions about it in the external data thread. Unfortunately, these questions often go unanswered even after tagging competition hosts multiple times.\n\nIt is understandable that the external data thread contains many comments and not all can be answered. However, if Kaggle is going to allow external data in a competition, a simple statement like \"so *dataset_X* is not allowed in this competition\" would be more integrous, then repeating vague rules that require legal knowledge to understand. \n\nA similar situation is playing out in the Jigsaw competition where it is currently not clear what people can and cannot do with the test set (translation, cleaning, etc.). I really hope that the Kaggle hosts can answer these questions so that we can at least mitigate misunderstandings about external data.", "votes": 16 }, { "id": 886720, "postDate": "2020-06-15T08:30:03.283Z", "content": "\n\n\nIt is understandable that the external data thread contains many comments and not all can be answered.
\n
I think ALL should be answered.
", "rawMarkdown": "> It is understandable that the external data thread contains many comments and not all can be answered.\n\nI think ALL should be answered. ", "votes": 8 } ] }, { "id": 887796, "postDate": "2020-06-15T22:12:06.607Z", "content": "You guys are never gonna believe what just happened!
\n\nthis has to be a deepfake? wow!!!!
", "rawMarkdown": "this has to be a deepfake? wow!!!!", "votes": 8 }, { "id": 887799, "postDate": "2020-06-15T22:18:02.987Z", "content": "I am sure he has given his consent 😄
", "rawMarkdown": "I am sure he has given his consent 😄 ", "votes": 7 }, { "id": 887800, "postDate": "2020-06-15T22:22:59.427Z", "content": "Make Kaggle Great Again!!!
", "rawMarkdown": "Make Kaggle Great Again!!!", "votes": 5 }, { "id": 887804, "postDate": "2020-06-15T22:29:08.723Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F150338%2F67c8f247365a1c4887b6b55e57818c21%2F458vx8.jpg?generation=1592260106649426&alt=media\")
😂
", "rawMarkdown": "😂 ", "votes": 2 }, { "id": 888182, "postDate": "2020-06-16T07:09:01.353Z", "content": "😂
", "rawMarkdown": "😂 ", "votes": 3 }, { "id": 888869, "postDate": "2020-06-16T15:57:33.587Z", "content": "RULES & ORDER!
", "rawMarkdown": "RULES & ORDER!", "votes": 6 }, { "id": 888879, "postDate": "2020-06-16T16:06:57.960Z", "content": "We'll build a wall of legal issues and make Kagglers pay for it (no political view here)
", "rawMarkdown": "We'll build a wall of legal issues and make Kagglers pay for it (no political view here)", "votes": 8 }, { "id": 906130, "postDate": "2020-06-29T04:45:34.170Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3077375%2F70040f936f52130e3c25ccfc5e3f0166%2FScreenshot%20from%202020-06-29%2010-46-11.png?generation=1593405994550905&alt=media\")
For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored, under the excuse that it doesn't meet certain high-level standards that Kaggle supposedly adheres to. And now a team of honest, hard working, scrupulously principled and exceptionally talented Kagglers is punished because of some post-hoc pedantic scrupules coming from a powerful tech giant??? This hypocrisy cries to high heaven!
\n\nAs many have remarked below, it seems that Kaggle has become the victim of its own success. Juggling dual roles - a prize-awarding competition site AND a community of Data Scientists - seems to be becoming increasingly difficult. And if there is a conflict between those two roles, it is now painfully obvious on which side Kaggle will default. As far as I see it, it will never be possible to smoothly reconcile those two roles. I believe that if Kaggle truly cared about its community, then the best, and perhaps only, solution going forward would be to abandon being a prize-giving entity. Most of us here are very loosely motivated by the monetary prizes, if at all. The recent COVID competitions proved that you can get very competitive competitions without any prizes and even without medals. (At my current job I am contractually prohibited from taking any prize money anyways, and I have been more competitive than ever.) By not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work. It will free Kaggle to more forcefully stand for its community, focus fully on the community-building and promotion and advancement of Data Science.
", "rawMarkdown": "For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored, under the excuse that it doesn't meet certain high-level standards that Kaggle supposedly adheres to. And now a team of honest, hard working, scrupulously principled and exceptionally talented Kagglers is punished because of some post-hoc pedantic scrupules coming from a powerful tech giant??? This hypocrisy cries to high heaven!\n\nAs many have remarked below, it seems that Kaggle has become the victim of its own success. Juggling dual roles - a prize-awarding competition site **AND** a community of Data Scientists - seems to be becoming increasingly difficult. And if there is a conflict between those two roles, it is now painfully obvious on which side Kaggle will default. As far as I see it, it will never be possible to smoothly reconcile those two roles. I believe that if Kaggle truly cared about its community, then the best, and perhaps only, solution going forward would be to **abandon being a prize-giving entity.** Most of us here are very loosely motivated by the monetary prizes, if at all. The recent COVID competitions proved that you can get very competitive competitions without any prizes and even without medals. (At my current job I am contractually prohibited from taking any prize money anyways, and I have been more competitive than ever.) By not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work. It will free Kaggle to more forcefully stand for its community, focus fully on the community-building and promotion and advancement of Data Science. ", "votes": 41, "replies": [ { "id": 884767, "postDate": "2020-06-13T15:52:48.520Z", "content": "I have a problem following you
\n\n\n\n\nMany other notorious cheaters are still active.
\n
can you please elaborate?
\n\n\n\n\nthe best, and perhaps only, solution going forward would be to abandon being a prize-giving entity.
\n
the arguments for this I was not able to grasp from your message. The fact that it is not the main motivator for people is not an argument against. And this argument is not clear to me:
\n\n\n\n\nBy not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work.
\n
for example in DFDC case, for me the problem was Kaggle not clarifying what data could and could not be used. But if clarified, to whatever side, I think it would have been OK. So there is no problem that there is a regulation, the rules, but the problem is that it was not clear.
", "rawMarkdown": "I have a problem following you\n> Many other notorious cheaters are still active.\n\ncan you please elaborate?\n> the best, and perhaps only, solution going forward would be to abandon being a prize-giving entity.\n\nthe arguments for this I was not able to grasp from your message. The fact that it is not the main motivator for people is not an argument against. And this argument is not clear to me:\n> By not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work.\n\nfor example in DFDC case, for me the problem was Kaggle not clarifying what data could and could not be used. But if clarified, to whatever side, I think it would have been OK. So there is no problem that there is a regulation, the rules, but the problem is that it was not clear." }, { "id": 884781, "postDate": "2020-06-13T16:00:17.007Z", "content": "\n\n\nMany other notorious cheaters are still active.
\n
Several whistleblowers have come out in the past and documented cheating behavior (web-scraping of solutions, private sharing, etc.) by a few top Kagglers. I don't want to elaborate beyond this.
\n\nAs far as DFDC is concerned, it was not the elaboration of the rules that was at stake, but their arbitrary interpretation after the competition was finished.
", "rawMarkdown": "> Many other notorious cheaters are still active.\n\nSeveral whistleblowers have come out in the past and documented cheating behavior (web-scraping of solutions, private sharing, etc.) by a few top Kagglers. I don't want to elaborate beyond this.\n\nAs far as DFDC is concerned, it was not the **elaboration** of the rules that was at stake, but their arbitrary interpretation after the competition was finished.", "votes": 7 }, { "id": 884785, "postDate": "2020-06-13T16:10:51.367Z", "content": "Regarding DFDC, in the external data disclosure thread dozens of questions went unanswered. People were genuinely in the dark what could and could not be used. The story of the OP is the perfect example, even using their second submission to mitigate that exact risk of unclear regulation.
\n\nAfter the fact they just selected one of the possible interpretations. But different interpretations should not have been possible to begin with. It is still a competition, with or without prizes, the rules must be clear.
", "rawMarkdown": "Regarding DFDC, in the external data disclosure thread dozens of questions went unanswered. People were genuinely in the dark what could and could not be used. The story of the OP is the perfect example, even using their second submission to mitigate that exact risk of unclear regulation. \n\nAfter the fact they just selected one of the possible interpretations. But different interpretations should not have been possible to begin with. It is still a competition, with or without prizes, the rules must be clear.", "votes": 6 }, { "id": 885463, "postDate": "2020-06-14T08:28:54.113Z", "content": "Although I am sitting a bit on the sideline with regards to competitions (my activity has mostly been notebooks so far), I first want to say that I feel very sorry for team All faces are real. I am very likely repeating others but this feels totally unfair; a lot of damage with regards to honor and also prize money for a reason that feels extremely vague and maybe even random.
\n\n> For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored
\n\nI was not aware that many notorious cheaters in competitions are still active (only knew about the Bestpetting case). Don't want to change the subject, but I can say that we have the same experience in notebooks. We collected evidence on a number of high ranked cheaters (I don't want to call them high-profile as they don't deserve that word). At best Kaggle replies that they are closely following discussions, but never follow-up....
", "rawMarkdown": "Although I am sitting a bit on the sideline with regards to competitions (my activity has mostly been notebooks so far), I first want to say that I feel very sorry for team All faces are real. I am very likely repeating others but this feels totally unfair; a lot of damage with regards to honor and also prize money for a reason that feels extremely vague and maybe even random.\n\n> For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored\n\nI was not aware that many notorious cheaters in competitions are still active (only knew about the Bestpetting case). Don't want to change the subject, but I can say that we have the same experience in notebooks. We collected evidence on a number of high ranked cheaters (I don't want to call them high-profile as they don't deserve that word). At best Kaggle replies that they are closely following discussions, but never follow-up....\n\n ", "votes": 12 } ] }, { "id": 883688, "postDate": "2020-06-12T20:35:56.443Z", "content": "OMG. That is even worse nitpicking reasoning than I expected... Very disappointed. Next time we need a lawyer before feature engineering...
", "rawMarkdown": "OMG. That is even worse nitpicking reasoning than I expected... Very disappointed. Next time we need a lawyer before feature engineering...", "votes": 36, "replies": [ { "id": 884047, "postDate": "2020-06-13T07:02:45.217Z", "content": "\"Next time we need a lawyer before feature engineering\".. hahaha! :D
", "rawMarkdown": "\"Next time we need a lawyer before feature engineering\".. hahaha! :D", "votes": 3 }, { "id": 884345, "postDate": "2020-06-13T09:56:18.570Z", "content": "I would like to share a few words from my heart here.
\n\nI haven't been in kaggle for a long time, but it is precisely because of kaggle's fairness and justice, sincerity with most kagglers, and the spirit of sharing with knowledge from your guys that deeply attracted me. I have always maintained a high enthusiasm here, which has enabled me to become GrandMaster within one year.
\n\nAnd I would like to express my heartfelt thanks to every kagglers here, thank you for your support. In the past two months, for me personally, I have fallen into a very frustrated and wronged mood☹️ , even dealing with the new competition, or my personal work, the previous passion is gone. \nNow all the stories of our team are shared here, I feel relieved😃 😃 . Thanks a lot, guys.
", "rawMarkdown": "I would like to share a few words from my heart here.\n\nI haven't been in kaggle for a long time, but it is precisely because of kaggle's fairness and justice, sincerity with most kagglers, and the spirit of sharing with knowledge from your guys that deeply attracted me. I have always maintained a high enthusiasm here, which has enabled me to become GrandMaster within one year.\n\nAnd I would like to express my heartfelt thanks to every kagglers here, thank you for your support. In the past two months, for me personally, I have fallen into a very frustrated and wronged mood☹️ , even dealing with the new competition, or my personal work, the previous passion is gone. \nNow all the stories of our team are shared here, I feel relieved😃 😃 . Thanks a lot, guys.", "votes": 33, "replies": [ { "id": 885661, "postDate": "2020-06-14T11:27:22.627Z", "content": "I feel you pain. There is always a point where you get strongly disappointed by Kaggle because of some decision you feel are extremely unfair. For you it happens here. For me is was when a private competition (CAESARS for those who entered it) where I was at the top was reset because Kaggle staff decided that their data prep hadn't been right. I wasn't a GM yet, and that gold medal would have made me GM way earlier. I was so upset I stopped kaggling for two months.
\n\nWhen I came back I did not attach emotionally as I used to. I am probably less motivated as well, but still enough to have won more gold medals and prizes. I take care of not invest emotionally in Kaggle anymore. This is probably why my written reactions here aren't as extreme as others. It is not that I find what happened to your team acceptable. It is that I know these things happen from time to time at Kaggle.
\n\nI don't know if my story can help you heal. I hope it can to some extent.
\n\nIt would be a pity to see a great competitor like you desert Kaggle. Same for all the team.
", "rawMarkdown": "I feel you pain. There is always a point where you get strongly disappointed by Kaggle because of some decision you feel are extremely unfair. For you it happens here. For me is was when a private competition (CAESARS for those who entered it) where I was at the top was reset because Kaggle staff decided that their data prep hadn't been right. I wasn't a GM yet, and that gold medal would have made me GM way earlier. I was so upset I stopped kaggling for two months. \n\nWhen I came back I did not attach emotionally as I used to. I am probably less motivated as well, but still enough to have won more gold medals and prizes. I take care of not invest emotionally in Kaggle anymore. This is probably why my written reactions here aren't as extreme as others. It is not that I find what happened to your team acceptable. It is that I know these things happen from time to time at Kaggle. \n\nI don't know if my story can help you heal. I hope it can to some extent.\n\nIt would be a pity to see a great competitor like you desert Kaggle. Same for all the team.\n\n", "votes": 21 }, { "id": 885703, "postDate": "2020-06-14T12:07:41.937Z", "content": "@cpmpml Thanks for your comfort. It helps me a lot😄
", "rawMarkdown": "@cpmpml Thanks for your comfort. It helps me a lot😄 ", "votes": 5 }, { "id": 885904, "postDate": "2020-06-14T14:54:43.830Z", "content": "Happy to help a bit. Take care.
", "rawMarkdown": "Happy to help a bit. Take care.", "votes": 1 }, { "id": 887157, "postDate": "2020-06-15T14:04:08.087Z", "content": "\n\n\nI take care of not invest emotionally in Kaggle anymore.
\n
It is easier to say than to do. For instance, I did get emotional in covid-19 forecasting because of what I felt was very unfair behavior. As a result I did not enter the last round. In hindsight, the stress of forecasting fatalities, as well as lockdown in France probably explains it. Anyway, getting emotional wasn't good.
", "rawMarkdown": "> I take care of not invest emotionally in Kaggle anymore. \n\nIt is easier to say than to do. For instance, I did get emotional in covid-19 forecasting because of what I felt was very unfair behavior. As a result I did not enter the last round. In hindsight, the stress of forecasting fatalities, as well as lockdown in France probably explains it. Anyway, getting emotional wasn't good. \n\n", "votes": 4 } ] }, { "id": 885306, "postDate": "2020-06-14T05:02:55.543Z", "content": "Right now, the private LB looks like the biggest deepfake of all.
", "rawMarkdown": "Right now, the private LB looks like the biggest deepfake of all.", "votes": 33, "replies": [ { "id": 887083, "postDate": "2020-06-15T13:21:26.420Z", "content": "I can't agree more 😂
", "rawMarkdown": "I can't agree more 😂 ", "votes": 6 } ] }, { "id": 883752, "postDate": "2020-06-12T22:05:02.267Z", "content": "\n\n\nFacebook felt some of our external data \"clearly appears to infringe third party rights\"
\n
This is news to me. FB cares about peoples rights. #TIL
", "rawMarkdown": "> Facebook felt some of our external data \"clearly appears to infringe third party rights\"\n\nThis is news to me. FB cares about peoples rights. #TIL", "votes": 33, "replies": [ { "id": 883759, "postDate": "2020-06-12T22:13:31.370Z", "content": "My thoughts exactly, Authman. Boo and double boo.
\n\nAnd congratulations to the team for having the best solution.
", "rawMarkdown": "My thoughts exactly, Authman. Boo and double boo.\n\nAnd congratulations to the team for having the best solution.", "votes": 19 }, { "id": 884342, "postDate": "2020-06-13T09:46:41.807Z", "content": "FB only cares when its the \"other\" party. When they do the violation themselves.
\n\n\n\n\nThey tell - \"We are trying, and will do better\"
\n
Also, this bring me lights on creativity and modelling approaches by @titericz 's team.
\n\nShouldn't approach and creativity be part of winning decision rather than some luckiest 3rd decimal spitted out by a computer. We can definetly do better!
", "rawMarkdown": "FB only cares when its the \"other\" party. When they do the violation themselves. \n> They tell - \"We are trying, and will do better\" \n\nAlso, this bring me lights on creativity and modelling approaches by @titericz 's team. \n\nShouldn't approach and creativity be part of winning decision rather than some luckiest 3rd decimal spitted out by a computer. We can definetly do better!", "votes": 5 } ] }, { "id": 887403, "postDate": "2020-06-15T16:53:20.943Z", "content": "Not sure how I should take this, 6:30 into @cristiancanton presentation in the competition overview during Workshop on Media Forensics, and the competition best score are shown.
\n\nHere the 0.423 log loss is not the top final LB score, and the only submission with such score, as far as I know, is our voided solution at 0.4232.
\n\nIf our solution is not acceptable, why parade our score in your presentation?
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F150338%2F41ae4f1e06c6faa9204754266485846e%2Fdeepfake_presentation_CVPR2020.png?generation=1592239585071572&alt=media\")
wondering where did they grab the \"real\" Deepfake videos to score the private test 🤔
", "rawMarkdown": "wondering where did they grab the \"real\" Deepfake videos to score the private test 🤔 ", "votes": 21 }, { "id": 887420, "postDate": "2020-06-15T17:01:27.780Z", "content": "This is just morally unfair at this point - pretty sure people will notice the discrepancies.
", "rawMarkdown": "This is just morally unfair at this point - pretty sure people will notice the discrepancies.", "votes": 6 }, { "id": 887423, "postDate": "2020-06-15T17:04:01.777Z", "content": "I don't want to take this out to social media, in case I have jumped to some incorrect assumption. so post here for now - but this is really adding insult to injury
", "rawMarkdown": "I don't want to take this out to social media, in case I have jumped to some incorrect assumption. so post here for now - but this is really adding insult to injury", "votes": 13 }, { "id": 887472, "postDate": "2020-06-15T17:27:44.910Z", "content": "\n\n\nI don't want to take this out to social media
\n
I do. 😄
", "rawMarkdown": "> I don't want to take this out to social media\n\nI do. 😄 ", "votes": 12 }, { "id": 887474, "postDate": "2020-06-15T17:28:53.490Z", "content": "Today's joke: \n\"At Facebook we value intellectual property! \"
", "rawMarkdown": "Today's joke: \n\"At Facebook we value intellectual property! \"", "votes": 16 }, { "id": 887482, "postDate": "2020-06-15T17:37:27.943Z", "content": "My apologies, a typo from my side on the exact number. I will rebuild the video with the fix. Thanks for letting me know.\n(Update: slides/video have been amended.)
", "rawMarkdown": "My apologies, a typo from my side on the exact number. I will rebuild the video with the fix. Thanks for letting me know.\n(Update: [slides/video](https://www.facebook.com/mediaforensics2020/posts/128437558881803) have been amended.)", "votes": -34 }, { "id": 887488, "postDate": "2020-06-15T17:38:51.290Z", "content": "Sure, that will fix everything!
", "rawMarkdown": "Sure, that will fix everything!", "votes": 28 }, { "id": 887501, "postDate": "2020-06-15T17:43:45.987Z", "content": "Thaks @gaborfodor . Just about to comment the same.\nsorry I am sarcastic too. can't help
", "rawMarkdown": "Thaks @gaborfodor . Just about to comment the same.\nsorry I am sarcastic too. can't help", "votes": 6 }, { "id": 887982, "postDate": "2020-06-16T03:41:23.997Z", "content": "I just watched the video. They hope our algorithms to be robust and generalised to unseen cases. But using little external data to enhance generalization get people disqualified. I didn't see they mention anything about audio. Is fake audio a smoke bomb? 😭
", "rawMarkdown": "I just watched the video. They hope our algorithms to be robust and generalised to unseen cases. But using little external data to enhance generalization get people disqualified. I didn't see they mention anything about audio. Is fake audio a smoke bomb? 😭 ", "votes": 8 }, { "id": 889287, "postDate": "2020-06-16T22:10:45.713Z", "content": "This is bordering comic... If it wasn't so sad. I guess winning team should sue them for publishing their score without license? Oh wait, the whole presentation should be removed! We can't have him fix it!
", "rawMarkdown": "This is bordering comic... If it wasn't so sad. I guess winning team should sue them for publishing their score without license? Oh wait, the whole presentation should be removed! We can't have him fix it!", "votes": 2 } ] }, { "id": 886135, "postDate": "2020-06-14T18:19:30.430Z", "content": "I love kaggle community, but I think Kaggle Team doesn't appreciate contribution from anyone of us (Novice --> GM). Money award doesn't matter, we are solving any competition even if size of award is tiny, despite on almost anyone competitor (Novice --> GM) can earn this money during one month without hard-working as require competition.
\n\nWe expect only that Kaggle Team will fight for justice for everyone. But Kaggle Team doesn't care:
\n\nKaggle Team forgot that \"Business Power\" are professionals from Kaggle Community and their free hard work. Lose us = Lose quality of solutions = Lose reputation = Lose business
", "rawMarkdown": "I love kaggle community, but I think Kaggle Team doesn't appreciate contribution from anyone of us (Novice --> GM). Money award doesn't matter, we are solving any competition even if size of award is tiny, despite on almost anyone competitor (Novice --> GM) can earn this money during one month without hard-working as require competition. \n\nWe expect only that Kaggle Team will fight for justice for everyone. But Kaggle Team doesn't care:\n\n- remove teams without good reason (because organizers from facebook can't run working kernels, HAHAHA)\n- don't provide even a piece of data from private stage (for checking why many team got 0.5)\n- use double standards for interpretation rules\n- don't explain specificity of your rules during competition (ignore)\n\nKaggle Team forgot that \"Business Power\" are professionals from Kaggle Community and their free hard work. Lose us = Lose quality of solutions = Lose reputation = Lose business", "votes": 30, "replies": [ { "id": 886147, "postDate": "2020-06-14T18:28:41.697Z", "content": "I was expecting you here. Well said, brother. :)
", "rawMarkdown": "I was expecting you here. Well said, brother. :)", "votes": 2 }, { "id": 896536, "postDate": "2020-06-22T08:41:06.240Z", "content": "Lose us = Lose quality of solutions = Lose reputation = Lose business
Umm.... this phrase needs to be emphasized MORE. Simply because it's TRUE.
", "rawMarkdown": "*`Lose us = Lose quality of solutions = Lose reputation = Lose business`*\n\nUmm.... this phrase needs to be emphasized MORE. Simply because it's TRUE.", "votes": 4 }, { "id": 896545, "postDate": "2020-06-22T08:54:49.573Z", "content": "Let's not forget \"lose money\" which I am not sure any corporation would want.
", "rawMarkdown": "Let's not forget \"lose money\" which I am not sure any corporation would want.", "votes": -1 } ] }, { "id": 884841, "postDate": "2020-06-13T17:03:05.347Z", "content": "My impression, for what it's worth, is that most of the problematic cases that we have been seeing in the last years have been routed in a certain disconnect between the Kaggle team and the community. In a way, this is perfectly understandable, since the Kaggle team is still relatively small, and it's impossible, say, for 1 or 2 people in charge of a competition to keep up with the ingenuities and (crazy) ideas of thousands of competitors. And running (and building) a platform is a different business than using it; with different priorities and philosophies. In many cases, the disconnect is not a problem, but a normal consequence of our different roles in the community.
\n\nHowever, in cases like this (or the PetFinder scandal, or the Passenger Screening competition for those who remember it, or also the high-profile Kernel plagiarism cases); in these situations the disconnect becomes a serious problem, because it leads to a mismatch of expectations between the team (& the sponsor) vs the community. Conflicting expectations, especially when combined with a lack of communication, then lead to extremely frustrating situations like the current one; which likely could have been avoided if everyone had been on the same page from the beginning. In those cases, it becomes a weakness that Kaggle does not tap into the community hivemind when planning a competition strategy.
\n\nSo here's a suggestion: we could institute a Kaggle community advisory board, consisting of a large number (~100) of well-respected Masters and Grandmasters. (I could easily name a few dozen without even thinking; I'm sure we got the numbers.) For any given competition, the team and board would select, say, 6-10 board members who would then act as a liaison between the organisers and the competitors. They would also help to foresee problems and controversies (e.g. to avoid changes in metric or flag potential leaks). Of course, those 6-10 Kagglers wouldn't be allowed to compete in that specific competition, but with a large enough pool to draw from, this shouldn't be an issue either. Nobody joins all competitions, these days.
\n\nIn this way, there is a good chance that conversations like the one we're having now could happen before a competition; and before lots of time and effort have been spent working under \"wrong\" assumptions (which were very reasonable assumptions in this case). This scenario would create more work for the Kaggle board members, but I have the feeling that many Masters and Grandmasters would accept the responsibility (and could also learn a new thing or two by looking at a competition from a different perspective). Of course, certain Kaggle \"trade secrets\" might have to remain opaque so that the board members could continue to compete without having an unfair advantage in future competitions.
\n\nAnyway, that's my thoughts. I'd be curious to read feedback.
", "rawMarkdown": "My impression, for what it's worth, is that most of the problematic cases that we have been seeing in the last years have been routed in a certain disconnect between the Kaggle team and the community. In a way, this is perfectly understandable, since the Kaggle team is still relatively small, and it's impossible, say, for 1 or 2 people in charge of a competition to keep up with the ingenuities and (crazy) ideas of thousands of competitors. And running (and building) a platform is a different business than using it; with different priorities and philosophies. In many cases, the disconnect is not a problem, but a normal consequence of our different roles in the community.\n\nHowever, in cases like this (or the PetFinder scandal, or the Passenger Screening competition for those who remember it, or also the high-profile Kernel plagiarism cases); in these situations the disconnect becomes a serious problem, because it leads to a mismatch of expectations between the team (& the sponsor) vs the community. Conflicting expectations, especially when combined with a lack of communication, then lead to extremely frustrating situations like the current one; which likely could have been avoided if everyone had been on the same page from the beginning. In those cases, it becomes a weakness that Kaggle does not tap into the community hivemind when planning a competition strategy.\n\nSo here's a suggestion: we could institute a **Kaggle community advisory board**, consisting of a large number (~100) of well-respected Masters and Grandmasters. (I could easily name a few dozen without even thinking; I'm sure we got the numbers.) For any given competition, the team and board would select, say, 6-10 board members who would then act as a liaison between the organisers and the competitors. They would also help to foresee problems and controversies (e.g. to avoid changes in metric or flag potential leaks). Of course, those 6-10 Kagglers wouldn't be allowed to compete in that specific competition, but with a large enough pool to draw from, this shouldn't be an issue either. Nobody joins all competitions, these days.\n\nIn this way, there is a good chance that conversations like the one we're having now could happen before a competition; and before lots of time and effort have been spent working under \"wrong\" assumptions (which were very reasonable assumptions in this case). This scenario would create more work for the Kaggle board members, but I have the feeling that many Masters and Grandmasters would accept the responsibility (and could also learn a new thing or two by looking at a competition from a different perspective). Of course, certain Kaggle \"trade secrets\" might have to remain opaque so that the board members could continue to compete without having an unfair advantage in future competitions.\n\nAnyway, that's my thoughts. I'd be curious to read feedback.", "votes": 30, "replies": [ { "id": 884867, "postDate": "2020-06-13T17:31:05.983Z", "content": "This is a very reasonable suggestion if Kaggle is interested in exploring such kind of solutions.
\n\nI can even vouch that it works if planned and structured correctly. I have been part of an international community of organizing, authoring and monitoring puzzle competitions (which are structured pretty much the same as Kaggle competitions) for over a decade, and for every competition there is a pool of community members who help in the structure, testing and evaluation of the competition before it launches (with the obvious condition that they are not allowed to compete).
\n\nIt has solved most of the issues we used to face and almost removed the gap between organizers and competitors completely. And it just works!
", "rawMarkdown": "This is a very reasonable suggestion if Kaggle is interested in exploring such kind of solutions.\n\nI can even vouch that it works if planned and structured correctly. I have been part of an international community of organizing, authoring and monitoring puzzle competitions (which are structured pretty much the same as Kaggle competitions) for over a decade, and for every competition there is a pool of community members who help in the structure, testing and evaluation of the competition before it launches (with the obvious condition that they are not allowed to compete).\n\nIt has solved most of the issues we used to face and almost removed the gap between organizers and competitors completely. And it just works!", "votes": 8 }, { "id": 884897, "postDate": "2020-06-13T17:48:56.197Z", "content": "Thanks @rohanrao! This is very valuable feedback. Can you say a bit more about the size of these pools and how the members are selected?
", "rawMarkdown": "Thanks @rohanrao! This is very valuable feedback. Can you say a bit more about the size of these pools and how the members are selected?", "votes": 3 }, { "id": 884921, "postDate": "2020-06-13T18:21:04.713Z", "content": "We have a pool of ~ 40 members. These are all actual puzzle competitors who have been ranked in the Top-100 at the World Championships (something like a group of 40 Kaggle Grandmasters). This is the only criteria and it is voluntary. There are times when some folks are active and some are not, its a bit lenient there but it still meets the current requirements.
\n\nFor every international puzzle championship (ranges from 50-100 events in a year), with 70% of them online, there will be 5-6 of these members involved (we call them testers) in each of them who work closely with the organizers of the event. Their main role is in validating the correctness of puzzles, timing of the contest, points distribution, proof-reading the instructions and so on. There is almost always changes and improvements that come from testers that end up making it a better contest (not all, since at the end it is the organizer who has the power to decide). And it doesn't really take long. It's great feedback for organizers at no cost. Sometimes the testers themselves have differing opinions but that itself is valuable because you realize something is not natural and coming to a consensus leads to a thought-through decision.
\n\nThe testers are chosen on a first-come-first-serve basis. It's pretty simple, any organizer just sends out an email and the first 6 members to respond positively are chosen. The testers are not allowed to participate in the contest. And in over 10 years, we've always found few members willing to help who do not wish to participate in the contest.
\n\nSome contests still fail or end up with a problem (even testers are human after all), but it is significantly lesser than without having testers. I'm sure the Kaggle team do their best as testers for every competition to bridge the gap between the organizer (host) and competitors, but over time I think the gap between Kaggle and competitors has increased.
", "rawMarkdown": "We have a pool of ~ 40 members. These are all actual puzzle competitors who have been ranked in the Top-100 at the World Championships (something like a group of 40 Kaggle Grandmasters). This is the only criteria and it is voluntary. There are times when some folks are active and some are not, its a bit lenient there but it still meets the current requirements.\n\nFor every international puzzle championship (ranges from 50-100 events in a year), with 70% of them online, there will be 5-6 of these members involved (we call them **testers**) in each of them who work closely with the organizers of the event. Their main role is in validating the correctness of puzzles, timing of the contest, points distribution, proof-reading the instructions and so on. There is almost always changes and improvements that come from testers that end up making it a better contest (not all, since at the end it is the organizer who has the power to decide). And it doesn't really take long. It's great feedback for organizers at no cost. Sometimes the testers themselves have differing opinions but that itself is valuable because you realize something is not natural and coming to a consensus leads to a thought-through decision.\n\nThe testers are chosen on a first-come-first-serve basis. It's pretty simple, any organizer just sends out an email and the first 6 members to respond positively are chosen. The testers are not allowed to participate in the contest. And in over 10 years, we've always found few members willing to help who do not wish to participate in the contest.\n\nSome contests still fail or end up with a problem (even testers are human after all), but it is significantly lesser than without having testers. I'm sure the Kaggle team do their best as testers for every competition to bridge the gap between the organizer (host) and competitors, but over time I think the gap between Kaggle and competitors has increased.", "votes": 11 }, { "id": 888079, "postDate": "2020-06-16T05:39:01.887Z", "content": "I think that this won't solve many problems, unless this board will hava more direct channel to the team and the team will be willing to give PRECISE answers. I understand that the team is a bit intimidated by their accountability and responsibility when giving direct, precise and clear answer. Its much easier to not answer or just say \"carefully read the rules\". However, this is THEIR JOB. Mitigating their job to a \"board\" will not help as the problem is with the team being afraid to interpret the rules or give clear yes no answer.\nOn a slightly different tangent, it is really annoying that these questions goes in threads. An issue tracking system will be much better.
", "rawMarkdown": "I think that this won't solve many problems, unless this board will hava more direct channel to the team and the team will be willing to give PRECISE answers. I understand that the team is a bit intimidated by their accountability and responsibility when giving direct, precise and clear answer. Its much easier to not answer or just say \"carefully read the rules\". However, this is THEIR JOB. Mitigating their job to a \"board\" will not help as the problem is with the team being afraid to interpret the rules or give clear yes no answer.\nOn a slightly different tangent, it is really annoying that these questions goes in threads. An issue tracking system will be much better. ", "votes": 2 } ] }, { "id": 883728, "postDate": "2020-06-12T21:30:21.670Z", "content": "I'm really sorry that this happened. It looks like you did your due diligence and went out of your way to even remove data that had an inappropriate license in an explicit effort to avoid breaking the rules - yet were punished anyways.
\n\nThere were several posts in the external data thread that went unanswered regarding the use of various external data sources. As was mentioned before, if the stipulations regarding external data were so strict (as to essentially prevent any practical use of other data sources), external data (apart from pretrained models) should not have been permitted in the first place so that teams wouldn't go through all this effort just to have a winning solution disqualified.
", "rawMarkdown": "I'm really sorry that this happened. It looks like you did your due diligence and went out of your way to even remove data that had an inappropriate license in an explicit effort to avoid breaking the rules - yet were punished anyways. \n\nThere were several posts in the external data thread that went unanswered regarding the use of various external data sources. As was mentioned before, if the stipulations regarding external data were so strict (as to essentially prevent any practical use of other data sources), external data (apart from pretrained models) should not have been permitted in the first place so that teams wouldn't go through all this effort just to have a winning solution disqualified. ", "votes": 30 }, { "id": 884310, "postDate": "2020-06-13T09:25:18.270Z", "content": "To be simplfied, Kaggle and FB didn't make the rules clear.\nThen all the losses caused by this mistake were passed on to us.
\n\nWe worked very hard on understanding the rules and finally beated by \"The organizer own the final interpretation right to the rules\"
", "rawMarkdown": "To be simplfied, Kaggle and FB didn't make the rules clear.\nThen all the losses caused by this mistake were passed on to us.\n\nWe worked very hard on understanding the rules and finally beated by \"**The organizer own the final interpretation right to the rules**\"", "votes": 27 }, { "id": 883887, "postDate": "2020-06-13T03:28:11.890Z", "content": "I do hope that Kaggle takes this opportunity to rethink the use of external datasets.
\n\nThere's been a weird dynamic in competitions where Kagglers scramble to look for useful external datasets or pretrained models, post them in the \"External data thread\", never get a clear answer about whether they're allowed, and often end up using them anyway.
\n\nIt's bad because a) it creates an arms race (especially in multimedia competitions) for scraped data which takes away from the spirit of these competitions and b) it disadvantages teams that scrupulously follow the rules over teams that cross the line in terms of the data they use.
\n\nPerhaps consider switching to a fixed whitelist of external data/models that Kaggle administrators define?
", "rawMarkdown": "I do hope that Kaggle takes this opportunity to rethink the use of external datasets.\n\nThere's been a weird dynamic in competitions where Kagglers scramble to look for useful external datasets or pretrained models, post them in the \"External data thread\", never get a clear answer about whether they're allowed, and often end up using them anyway. \n\nIt's bad because a) it creates an arms race (especially in multimedia competitions) for scraped data which takes away from the spirit of these competitions and b) it disadvantages teams that scrupulously follow the rules over teams that cross the line in terms of the data they use.\n\nPerhaps consider switching to a fixed whitelist of external data/models that Kaggle administrators define? ", "votes": 27 }, { "id": 883865, "postDate": "2020-06-13T02:32:51.170Z", "content": "If this hasn't been pointed out already, the decision to remove this solution but keep (most) others in the competition is extremely arbitrary. Scanning many of the solutions, all face detectors/recognition models that I'm aware of were trained with datasets that are not compliant with the rules, actually much less so than the extra datasets these competitors used.
\n\ndlib face_detection, uses VGG face (Non commercial license, no consent), scrubface (no consent), manual scrubbing (duh). mtcnn, facenet, etc all use datasets based on faces without consent (the ones mentioned already, CASIA-WebFace, LFW, etc etc). Then there is ImageNet (non-commercial) pretrained weights.
\n\nSo, this is a long standing Kaggle question. Why does a pretrained model not apply to the rules but a self trained one does? Someone deciding to license their code with a given license (say dlib is Boosts) while the dataset they used (VGG face non-commercial) falls under a different one clearly does not make those weights fall under their code license. There should really be no difference.
", "rawMarkdown": "If this hasn't been pointed out already, the decision to remove this solution but keep (most) others in the competition is extremely arbitrary. Scanning many of the solutions, all face detectors/recognition models that I'm aware of were trained with datasets that are not compliant with the rules, actually much less so than the extra datasets these competitors used.\n\ndlib face_detection, uses VGG face (Non commercial license, no consent), scrubface (no consent), manual scrubbing (duh). mtcnn, facenet, etc all use datasets based on faces without consent (the ones mentioned already, CASIA-WebFace, LFW, etc etc). Then there is ImageNet (non-commercial) pretrained weights. \n\nSo, this is a long standing Kaggle question. Why does a pretrained model not apply to the rules but a self trained one does? Someone deciding to license their code with a given license (say dlib is Boosts) while the dataset they used (VGG face non-commercial) falls under a different one clearly does not make those weights fall under their code license. There should really be no difference. ", "votes": 27, "replies": [ { "id": 884833, "postDate": "2020-06-13T16:59:22.053Z", "content": "My take on this is : \"Authors Guild v. Google\" , when you have enough lawyers like us ... you can scrape, otherwise people will sue and we will DSQ you 😃
", "rawMarkdown": "My take on this is : \"Authors Guild v. Google\" , when you have enough lawyers like us ... you can scrape, otherwise people will sue and we will DSQ you 😃 ", "votes": 1 } ] }, { "id": 883712, "postDate": "2020-06-12T21:10:07.037Z", "content": "So do we now have to get a written and notarized permission from the ImageNet creators every time we use pretrained models in Kaggle solutions?
", "rawMarkdown": "So do we now have to get a written and notarized permission from the ImageNet creators every time we use pretrained models in Kaggle solutions? ", "votes": 28, "replies": [ { "id": 883718, "postDate": "2020-06-12T21:20:14.057Z", "content": "The analogy would be that you have to get notarized permission from anyone appearing in any image that appears in ImageNet.
", "rawMarkdown": "The analogy would be that you have to get notarized permission from anyone appearing in any image that appears in ImageNet.", "votes": 25 }, { "id": 883722, "postDate": "2020-06-12T21:27:42.483Z", "content": "That's would be insane. But not really surprising, knowing Facebook and their MO. What is really disappointing is that Kaggle allowed them to get away with it.
", "rawMarkdown": "That's would be insane. But not really surprising, knowing Facebook and their MO. What is really disappointing is that Kaggle allowed them to get away with it. ", "votes": 16 }, { "id": 883844, "postDate": "2020-06-13T01:43:38.223Z", "content": "For reference, pretrained models were specifically approved by the organisers: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121203#694466
\n\nSo in this case, there is no requirement to have consent from individuals depicted in ImageNet. I can see how this might be seen as a logical inconsistency.
", "rawMarkdown": "For reference, pretrained models were specifically approved by the organisers: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121203#694466\n\nSo in this case, there is no requirement to have consent from individuals depicted in ImageNet. I can see how this might be seen as a logical inconsistency.", "votes": 12 }, { "id": 884682, "postDate": "2020-06-13T14:53:03.760Z", "content": "Not only ImageNet - I feel that we have to also gain copyright permission(s) for the basic linear algebra too.
\n\nDear Gottfried Leibnitz, if you read this, please give us all your approval to use calculus wherever it is necessary.
", "rawMarkdown": "Not only ImageNet - I feel that we have to also gain copyright permission(s) for the basic linear algebra too.\n\nDear Gottfried Leibnitz, if you read this, please give us all your approval to use calculus wherever it is necessary.", "votes": 7 } ] }, { "id": 885153, "postDate": "2020-06-14T00:40:32.053Z", "content": "Giba, Mikel, Yifan, Gary and Qishen , you deserve the first place. We all see it.
\n\nI frankly don't know what else to say at this point. Many people made great comments about what to do next and I agree with many of them. But for you it won't matter much I guess. I hope you'll get over it.
\n\nAn afterthought: I would make clear to Facebook that they have no rights to reuse any part of your solution. AFAIK, only prize winners have an obligation to provide a licence for their code to the sponsor.
", "rawMarkdown": "Giba, Mikel, Yifan, Gary and Qishen , you deserve the first place. We all see it. \n\nI frankly don't know what else to say at this point. Many people made great comments about what to do next and I agree with many of them. But for you it won't matter much I guess. I hope you'll get over it. \n\nAn afterthought: I would make clear to Facebook that they have no rights to reuse any part of your solution. AFAIK, only prize winners have an obligation to provide a licence for their code to the sponsor.", "votes": 29, "replies": [ { "id": 886284, "postDate": "2020-06-14T21:36:14.347Z", "content": "Thank you @cpmpml. We really appreciate all the support from the community.
", "rawMarkdown": "Thank you @cpmpml. We really appreciate all the support from the community. ", "votes": 10 } ] }, { "id": 883685, "postDate": "2020-06-12T20:34:07.763Z", "content": "Thanks for giving this background and sorry it ended this way. I thought it was to do with the commercial licensing of the data. I was not even aware of the additional stipulations. In my opinion that should have been something much more prominently displayed, because that additional rule would have quickly eliminated all external data usage basically.
", "rawMarkdown": "Thanks for giving this background and sorry it ended this way. I thought it was to do with the commercial licensing of the data. I was not even aware of the additional stipulations. In my opinion that should have been something much more prominently displayed, because that additional rule would have quickly eliminated all external data usage basically. ", "votes": 26 }, { "id": 887946, "postDate": "2020-06-16T02:53:53.263Z", "content": "Up front, I should clearly state that Kaggle employees are employees of Google. In the rare event that a dispute is raised on Kaggle that requires review of competition rules, our actions, statements, and conduct require us to refrain from opinions, speculation, and casual commentary. It is for this reason that you can expect a slower response in matters like this.
\n\nWe are working on a more comprehensive response and will post when that’s ready.
", "rawMarkdown": "Up front, I should clearly state that Kaggle employees are employees of Google. In the rare event that a dispute is raised on Kaggle that requires review of competition rules, our actions, statements, and conduct require us to refrain from opinions, speculation, and casual commentary. It is for this reason that you can expect a slower response in matters like this.\n\nWe are working on a more comprehensive response and will post when that’s ready.\n", "votes": 26, "replies": [ { "id": 888191, "postDate": "2020-06-16T07:16:02.370Z", "content": "Thanks for this update. Discussions with Google legal and Facebook legal at the same time must be quite interesting. We all hope Kaggle will find a way out that is as fair as possible.
", "rawMarkdown": "Thanks for this update. Discussions with Google legal and Facebook legal at the same time must be quite interesting. We all hope Kaggle will find a way out that is as fair as possible.", "votes": 7 }, { "id": 888241, "postDate": "2020-06-16T08:08:55.267Z", "content": "Thanks Julia. I hope you will find the best possible solution soon. At the Zillow Prize Disqualification Issue kaggle found a solution after five days. I know this case is probably trickier.
\n\nBtw here is a chart that compares the heated forum threads.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F18102%2F633c1fa0dc2c33245b583b020217554a%2FScreenshot%202020-06-16%20at%2010.04.25.png?generation=1592294804666595&alt=media\")
\nSource
Discussions with Google legal and Facebook legal at the same time must be quite interesting
Same thought.
", "rawMarkdown": "`Discussions with Google legal and Facebook legal at the same time must be quite interesting`\n\nSame thought.", "votes": 3 }, { "id": 893487, "postDate": "2020-06-19T16:42:56.343Z", "content": "Sorry, Julia, but your attempt to clarify the mistake just make the mistake you did worse and worse.
", "rawMarkdown": "Sorry, Julia, but your attempt to clarify the mistake just make the mistake you did worse and worse. " } ] }, { "id": 884861, "postDate": "2020-06-13T17:21:21.913Z", "content": "No matter what way I look at this and no matter how much I try to understand from a host's perspective, I cannot come to convince myself that this was correct or fair in any way. There have been questionable decisions in competitions in the past but this is just preposterous and beyond acceptable.
\n\nCongratulations to Giba, Mikel, Yifan, Gary and Qishen for their work, effort, solution and for winning this competition!
", "rawMarkdown": "No matter what way I look at this and no matter how much I try to understand from a host's perspective, I cannot come to convince myself that this was correct or fair in any way. There have been questionable decisions in competitions in the past but this is just preposterous and beyond acceptable.\n\nCongratulations to Giba, Mikel, Yifan, Gary and Qishen for their work, effort, solution and for winning this competition!", "votes": 27 }, { "id": 883717, "postDate": "2020-06-12T21:18:20.530Z", "content": "Is my cat part of the documentation? It may well be if you hire the right lawyer.
\n\nThis is truly shocking – are we really entering times when to succeed in a Kaggle competition your team will need more lawyers than scientists? Perhaps just lawyers?
\n\nLet me get this straight – the world’s top technology to detect deepfakes has just been “voided” by Facebook because of their claim that the training data is the “documentation”. Really? I have looked very carefully through the Kaggle definition of documentation and could not find a single sentence stating that training data is the documentation.
\n\nSad times for science.
", "rawMarkdown": "Is my cat part of the documentation? It may well be if you hire the right lawyer. \n\nThis is truly shocking – are we really entering times when to succeed in a Kaggle competition your team will need more lawyers than scientists? Perhaps just lawyers? \n\nLet me get this straight – the world’s top technology to detect deepfakes has just been “voided” by Facebook because of their claim that the training data is the “documentation”. Really? I have looked very carefully through the Kaggle definition of documentation and could not find a single sentence stating that training data is the documentation. \n\nSad times for science. \n", "votes": 27 }, { "id": 892113, "postDate": "2020-06-18T17:05:46.877Z", "content": "I thought Kaggle and Google had developed a super duper spam detector algorithm recently ..
\n\nSo why this topic is still filled with bot users having not very subtle spams like \"Wow\", \"Great Job\", \"Nice\" etc. ?
", "rawMarkdown": "I thought Kaggle and Google had developed a [super duper spam detector](https://cloud.google.com/blog/products/ai-machine-learning/how-kaggle-solved-a-spam-problem-using-automl) algorithm recently ..\n\nSo why this topic is still filled with bot users having not very subtle spams like \"Wow\", \"Great Job\", \"Nice\" etc. ? ", "votes": 24 }, { "id": 886040, "postDate": "2020-06-14T17:02:08.433Z", "content": "This raises a bigger issue about all competitions having personal data involved... for example ongoing melanoma challenge - my reasoning here:
\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886037
", "rawMarkdown": "This raises a bigger issue about all competitions having personal data involved... for example ongoing melanoma challenge - my reasoning here:\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886037", "votes": 23, "replies": [ { "id": 886537, "postDate": "2020-06-15T05:40:46.697Z", "content": "There is a really bizarre line in the melanoma challenge stating that if you don't use external data, you are ineligible for a prize 🤷♂️: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886521
\n\nEdit: I think the rule is specifically around public vs private data. I guess this will need clear instructions on what is and is not public
", "rawMarkdown": "There is a really bizarre line in the melanoma challenge stating that if you **don't** use external data, you are ineligible for a prize 🤷♂️: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886521\n\nEdit: I think the rule is specifically around public vs private data. I guess this will need clear instructions on what is and is not public", "votes": 14 }, { "id": 886750, "postDate": "2020-06-15T08:51:57.397Z", "content": "Hahaha wow... Just wow...
", "rawMarkdown": "Hahaha wow... Just wow...", "votes": 2 }, { "id": 887563, "postDate": "2020-06-15T18:30:16.213Z", "content": "\n\n\nThere is a really bizarre line in the melanoma challenge stating that if you don't use external data, you are ineligible for a prize
\n
Although Gilles may have seen this already and many may already know this too, this is the comment by the competition host regarding the use of external data with eligibility to win prizes.
", "rawMarkdown": "> There is a really bizarre line in the melanoma challenge stating that if you don't use external data, you are ineligible for a prize\n\nAlthough Gilles may have seen this already and many may already know this too, [this](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158747#887350) is the comment by the competition host regarding the use of external data with eligibility to win prizes. \n", "votes": 2 } ] }, { "id": 884662, "postDate": "2020-06-13T14:28:11.973Z", "content": "This sucks. Sounds like a case of \"Let's keep the rules vague so we can disqualify submissions that we don't like for whatever reason.\"
", "rawMarkdown": "This sucks. Sounds like a case of \"Let's keep the rules vague so we can disqualify submissions that we don't like for whatever reason.\"", "votes": 24, "replies": [ { "id": 885009, "postDate": "2020-06-13T19:45:28.673Z", "content": "ah! I'm sure they followed it.
", "rawMarkdown": "ah! I'm sure they followed it. ", "votes": 2 } ] }, { "id": 883696, "postDate": "2020-06-12T20:48:00.103Z", "content": "This is awful. Truly reprehensible bullying behavior. I can't believe that Facebook will again get away with bending all standards of ethics and common decency.
", "rawMarkdown": "This is awful. Truly reprehensible bullying behavior. I can't believe that Facebook will again get away with bending all standards of ethics and common decency.", "votes": 28 }, { "id": 885964, "postDate": "2020-06-14T15:46:54.323Z", "content": "It is frustrating that Kaggle has only contributed to this dialogue insofar as making a single post containing a vague non-response that basically amounts to \"read the fine print\" without so much as a simple apology.
\n\nThis falls squarely on their shoulders, and it is unfortunate that part of the blame is shifted onto the competitors. It is disrespectful, especially when the team consists of 3 GMs and 2 Masters (who will probably be GMs soon) that have contributed so much to the community.
", "rawMarkdown": "It is frustrating that Kaggle has only contributed to this dialogue insofar as making a single post containing a vague non-response that basically amounts to \"read the fine print\" without so much as a simple apology. \n\nThis falls squarely on their shoulders, and it is unfortunate that part of the blame is shifted onto the competitors. It is disrespectful, especially when the team consists of 3 GMs and 2 Masters (who will probably be GMs soon) that have contributed so much to the community. ", "votes": 21, "replies": [ { "id": 885976, "postDate": "2020-06-14T16:00:50.553Z", "content": "I still hope they will respond to the issue properly. In my more emphatic moments I could understand why they haven't answered yet (weekend, tough topic, still exploring solutions etc.). Altough they must know this unfortunate (frustrating/outrageous/unacceptable... pick any) incident for weeks now.
", "rawMarkdown": "I still hope they will respond to the issue properly. In my more emphatic moments I could understand why they haven't answered yet (weekend, tough topic, still exploring solutions etc.). Altough they must know this unfortunate (frustrating/outrageous/unacceptable... pick any) incident for weeks now.", "votes": 15 }, { "id": 886114, "postDate": "2020-06-14T17:58:18.983Z", "content": "I think the main reason is that it's the weekend. I'm expecting more extensive statements by the Kaggle team on Monday/Tuesday (Pacific time) when everyone is back online. I think this is understandable; and I'm certain that no disrespect is intended. It was a little unfortunate timing to make the announcement on a Friday, though.
", "rawMarkdown": "I think the main reason is that it's the weekend. I'm expecting more extensive statements by the Kaggle team on Monday/Tuesday (Pacific time) when everyone is back online. I think this is understandable; and I'm certain that no disrespect is intended. It was a little unfortunate timing to make the announcement on a Friday, though.", "votes": 5 }, { "id": 886978, "postDate": "2020-06-15T12:12:29.787Z", "content": "It will depend on how fast Kaggle legal people and Facebook vet whatever they want to answer.
", "rawMarkdown": "It will depend on how fast Kaggle legal people and Facebook vet whatever they want to answer.", "votes": 3 } ] }, { "id": 883873, "postDate": "2020-06-13T02:56:13.997Z", "content": "Facebook is not a company it used to be; it does what it wants and this is just another case. Sadly one of most appreciated team of Kagglers are on the receiving end. I'm so sorry for your team ;(\nI have one suggestion to Kaggle: Please remove this from future competitions; there is no meaning to it and only makes the forums crowded \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1528571%2F052d3e16168fb81242a6ec3d8ffa2921%2Fexternal_data.bmp?generation=1592016734208101&alt=media\")
So not only were around 5% of the teams screwed out of the private leaderboard because their solution mysteriously didn't work and the top team on public and others lost due to misleading info on sound manipulation, BUT the top teams scores were then voided over ridiculous fine print rules and FB being picky just not liking their solution??!
\n\nTeams worked on this problem for months, incurring costs just to use the giant dataset, just to be screwed over because Kaggle didn't inform competitors. And for your team specially you were shut down even though you meticulously made sure to follow all the rules!
\n\nLike others have said Kaggle is a place to casually compete and move science forward but it seems we need lawyers to make sure we just follow the rules correctly!
\n\nAnd that is after we apparently just had to cross our fingers and hope we can even make it to the private leaderboard. The Kaggle team needs to rethink how to do these rerun solutions types of competitions for the future.
\n\nI definitely no longer hold any pride in doing decent in this comp when there are many who worked way harder and deserved something for it.
", "rawMarkdown": "So not only were around 5% of the teams screwed out of the private leaderboard because their solution mysteriously didn't work and the top team on public and others lost due to misleading info on sound manipulation, BUT the top teams scores were then voided over ridiculous fine print rules and FB being picky just not liking their solution??!\n\nTeams worked on this problem for months, incurring costs just to use the giant dataset, just to be screwed over because Kaggle didn't inform competitors. And for your team specially you were shut down even though you meticulously made sure to follow all the rules!\n\nLike others have said Kaggle is a place to casually compete and move science forward but it seems we need lawyers to make sure we just follow the rules correctly!\n\nAnd that is after we apparently just had to cross our fingers and hope we can even make it to the private leaderboard. The Kaggle team needs to rethink how to do these rerun solutions types of competitions for the future.\n\nI definitely no longer hold any pride in doing decent in this comp when there are many who worked way harder and deserved something for it.", "votes": 22 }, { "id": 883800, "postDate": "2020-06-13T00:23:05.290Z", "content": "While you are all blaming Facebook it is important to remember who was responsible for moderating this competition. That falls squarely on Kaggle. The platform owner (Kaggle) failed to moderate the competition by eliminating the ambiguity raised with the rules over at the official forum. Many of the questions were greeted with incomplete and ambiguous responses even after repeated attempts to seek clarity. That is where it all started to go downhill.
\n\nWithout users Kaggle doesn't exist. They should keep that in mind.
\n\n#KaggleDFDCFail
", "rawMarkdown": "While you are all blaming Facebook it is important to remember **who** was **responsible** for **moderating** this **competition**. That falls squarely on **Kaggle**. The **platform** owner (Kaggle) **failed** to **moderate** the competition by **eliminating** the **ambiguity** raised with the **rules** over at the **official forum**. Many of the **questions** were greeted with **incomplete** and **ambiguous responses** even after **repeated attempts** to seek **clarity**. That is where it all started to go **downhill**.\n\n### **At the very least there should be a public apology from Kaggle to all contestants.**\n\nWithout users Kaggle doesn't exist. They should keep that in mind.\n\n\\#KaggleDFDCFail\n", "votes": 20, "replies": [ { "id": 883811, "postDate": "2020-06-13T00:47:39.967Z", "content": "Indeed. And this is not just anyone that Kaggle allowed to be disqualified - this is Giba we are talking about, the most successful Kaggler of all time. If anyone understands the rules of Kaggle competitions, it would be him. The fact that Kaggle was so willing and ready to throw him under the bus is really infuriating. If they are unwilling to stand up for him, what can the rest of us expect?
", "rawMarkdown": "Indeed. And this is not just anyone that Kaggle allowed to be disqualified - this is Giba we are talking about, the most successful Kaggler of all time. If anyone understands the rules of Kaggle competitions, it would be him. The fact that Kaggle was so willing and ready to throw him under the bus is really infuriating. If they are unwilling to stand up for him, what can the rest of us expect?", "votes": 22 }, { "id": 883813, "postDate": "2020-06-13T00:51:15.237Z", "content": "Easy way to solve this. Add another $1 million to the pot and own up to their mistakes.
", "rawMarkdown": "Easy way to solve this. Add another $1 million to the pot and own up to their mistakes.", "votes": 5 }, { "id": 883869, "postDate": "2020-06-13T02:51:08.603Z", "content": "To be quite blunt, it's not really surprising that Kaggle would throw a competitor under the bus in favor of sponsors (especially given that it's a consortium including AWS, FB, and Microsoft).
\n\nBut it's a sign that the platform's moved away from its closeknit community roots to becoming a corporate platform.
", "rawMarkdown": "To be quite blunt, it's not really surprising that Kaggle would throw a competitor under the bus in favor of sponsors (especially given that it's a consortium including AWS, FB, and Microsoft).\n\nBut it's a sign that the platform's moved away from its closeknit community roots to becoming a corporate platform. ", "votes": 11 }, { "id": 883875, "postDate": "2020-06-13T03:00:02.793Z", "content": "Platforms come and go due to events like this. There are enough smart people here to start a new platform.
", "rawMarkdown": "Platforms come and go due to events like this. There are enough smart people here to start a new platform.", "votes": 9 } ] }, { "id": 889309, "postDate": "2020-06-16T22:40:24.507Z", "content": "@juliaelliott first of all, thank you for taking to time to engage with us.
\n\n\n\n\nI’d like to attempt to clarify the underlying issue with All Faces are Real’s disqualified submission. Some of the videos/images used in the disqualified submission were mis-licensed, in that they contained content belonging to other third parties (such as CNN), but were inappropriately offered under open source licenses. This content also clearly depicted third parties and used third party data whose permissions had not been obtained, in violation of the competition rules...
\n
Here is my thought - I struggle to pin down what \"mis-licensed\" mean here. If you search within Kaggle, you would find this is the first time this word (\"mis-licensed\" or \"mislicense\") is used in any discussion or notebook. Also it is definitely not shown in the competition rule.
\n\nFurthermore, we were asked to provide \"copies of any additional permissions or licenses from individuals\" appearing in the CC-BY youtube videos with deepfake manipulation that we used. and I am not sure by saying \"mis-license\": do you mean the youtube video providers can't/shouldn't give CC-BY license after applying deepfake on those videos? or do you mean they can't/shouldn't give CC-BY license on a videos that is CNN's property? or was it \"mis-licensed\" because the video providers didn't have individual consent from people appearing in the videos?
\n\nEither way, isn't it reasonable to infer that it should be Youtube's responsibility to decide if a video hosted on youtube is \"mis-licensed\", and not individual user like us?
\n\nI do apologies if my above statement is poorly constructed in a legal sense, after all I am not a legal professional. it would be great if people in our community, kaggle or host team can give legal definition of \"mis-licensed\" , and also advise if individual users should be penalised for using CC-BY license videos that are \"mis-licensed\"
\n\nThis further illustrates our frustration: it seems without hiring a legal representative before taking part in this competition, there is no way to determine if a piece of extra information can live up to our dear host team's morally superior standard.
\n\nAnd please allow me to again emphasis there is NO MENTION from you or host team whatsoever to look out for individual consent, or \"mis-licensed\" information during the competition - NONE. and yet, in @cristiancanton talk (see 5:03) - it was very clearly that the host team, at their brilliant and well-resourced effort to create the DFDC dataset, they made a point of securing individual consent in DF videos in 2nd half of 2019. you can tell they are rightly proud of it in this slide here. ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F150338%2Fde027bea12200c2ee089c985166ec2bd%2FDFDC_permission.png?generation=1592346426996673&alt=media\")
and yet, given they are so RIGHTLY PROUD of their dataset having individual consents, somehow this very important evaluation criterion for appropriateness of external dataset was NEVER mentioned, not even once in any of your clarification in the forum
\n\ndo you not see why our team, other participants - including prize winners who voiced their opinion here, and other fellow kagglers are frustrated and upset?
", "rawMarkdown": "@juliaelliott first of all, thank you for taking to time to engage with us.\n\n> I’d like to attempt to clarify the underlying issue with All Faces are Real’s disqualified submission. Some of the videos/images used in the disqualified submission were mis-licensed, in that they contained content belonging to other third parties (such as CNN), but were inappropriately offered under open source licenses. This content also clearly depicted third parties and used third party data whose permissions had not been obtained, in violation of the competition rules...\n\nHere is my thought - I struggle to pin down what \"mis-licensed\" mean here. If you search within Kaggle, you would find this is the first time this word (\"mis-licensed\" or \"mislicense\") is used in any discussion or notebook. Also it is definitely not shown in the competition rule.\n\nFurthermore, we were asked to provide \"copies of any additional permissions or licenses from individuals\" appearing in the CC-BY youtube videos with deepfake manipulation that we used. and I am not sure by saying \"mis-license\": do you mean the youtube video providers can't/shouldn't give CC-BY license after applying deepfake on those videos? or do you mean they can't/shouldn't give CC-BY license on a videos that is CNN's property? or was it \"mis-licensed\" because the video providers didn't have individual consent from people appearing in the videos?\n\nEither way, isn't it reasonable to infer that it should be Youtube's responsibility to decide if a video hosted on youtube is \"mis-licensed\", and not individual user like us?\n\nI do apologies if my above statement is poorly constructed in a legal sense, after all I am not a legal professional. it would be great if people in our community, kaggle or host team can give legal definition of \"mis-licensed\" , and also advise if individual users should be penalised for using CC-BY license videos that are \"mis-licensed\" \n\nThis further illustrates our frustration: it seems without hiring a legal representative before taking part in this competition, there is no way to determine if a piece of extra information can live up to our dear host team's morally superior standard. \n\nAnd please allow me to again emphasis there is NO MENTION from you or host team whatsoever to look out for individual consent, or \"mis-licensed\" information during the competition - NONE. and yet, in @cristiancanton [talk](https://www.facebook.com/watch/?v=1640779116079742) (see 5:03) - it was very clearly that the host team, at their brilliant and well-resourced effort to create the DFDC dataset, they made a point of securing individual consent in DF videos in **2nd half of 2019**. you can tell they are rightly proud of it in this slide here.  \n\nand yet, given they are so RIGHTLY PROUD of their dataset having individual consents, somehow this very important evaluation criterion for appropriateness of external dataset was NEVER mentioned, not even once in any of your clarification in the forum \n\ndo you not see why our team, other participants - including prize winners who voiced their opinion here, and other fellow kagglers are frustrated and upset?\n", "votes": 22, "replies": [ { "id": 889397, "postDate": "2020-06-17T00:20:56.700Z", "content": "I wonder if Facebook also got individual consent from participants in the organic fakes in the private dataset... Perhaps they should be disqualified as well....\nI bet there was probably at least one trump deep fake there :)
", "rawMarkdown": "I wonder if Facebook also got individual consent from participants in the organic fakes in the private dataset... Perhaps they should be disqualified as well....\nI bet there was probably at least one trump deep fake there :) ", "votes": 5 }, { "id": 890179, "postDate": "2020-06-17T11:15:14.030Z", "content": "Under the given explanation, you used YouTube videos that you thought were licensed properly but that actually were not.
\n\nI have to agree with the organizers that vetting the licensing of the data you're using is your responsibility. It is widely known that there is a lot of copyright-infringing material on YouTube, so assuming that the videos were properly licensed was a bit naive.
\n\nThat said, it sounds extremely hard to make sure any training data you collect from YouTube doesn't infringe someone else's rights. This is probably why the lawyers went in CYA mode when they saw you used videos from YouTube. I think we can conclude from this that YouTube is not a \"safe\" source of external training data.
\n\n\n\n\ndo you mean the youtube video providers can't/shouldn't give CC-BY license after applying deepfake on those videos? or do you mean they can't/shouldn't give CC-BY license on a videos that is CNN's property? or was it \"mis-licensed\" because the video providers didn't have individual consent from people appearing in the videos?
\n
I do think those are important questions. Also, if the people in the videos are public figures (which I assume was the case in the CNN videos), would consent really be required? I suppose the only way to find this out is in a court of law and the competition hosts may not feel that's worth it (or even want to avoid that at all costs, because as long is something is legally dubious it is not illegal).
", "rawMarkdown": "Under the given explanation, you used YouTube videos that you thought were licensed properly but that actually were not. \n\nI have to agree with the organizers that vetting the licensing of the data you're using is *your* responsibility. It is widely known that there is a lot of copyright-infringing material on YouTube, so assuming that the videos were properly licensed was a bit naive.\n\nThat said, it sounds extremely hard to make sure any training data you collect from YouTube doesn't infringe someone else's rights. This is probably why the lawyers went in CYA mode when they saw you used videos from YouTube. I think we can conclude from this that YouTube is not a \"safe\" source of external training data.\n\n> do you mean the youtube video providers can't/shouldn't give CC-BY license after applying deepfake on those videos? or do you mean they can't/shouldn't give CC-BY license on a videos that is CNN's property? or was it \"mis-licensed\" because the video providers didn't have individual consent from people appearing in the videos?\n\nI do think those are important questions. Also, if the people in the videos are public figures (which I assume was the case in the CNN videos), would consent really be required? I suppose the only way to find this out is in a court of law and the competition hosts may not feel that's worth it (or even want to avoid that at all costs, because as long is something is legally dubious it is not illegal).", "votes": 5 }, { "id": 890328, "postDate": "2020-06-17T12:58:55.037Z", "content": "@humananalog what about using non-licensed images for pretrained models and face detectors? Do they infringe someone else's right?
\n\nThat is the problem, or everything is bad or everything is good, there is no difference there.
", "rawMarkdown": "@humananalog what about using non-licensed images for pretrained models and face detectors? Do they infringe someone else's right?\n\n- images from Youtube = BAD\n- images from pretrained model scraped from internet = GOOD\n\nThat is the problem, or everything is bad or everything is good, there is no difference there.", "votes": 8 }, { "id": 890376, "postDate": "2020-06-17T13:31:42.260Z", "content": "I am not a lawyer, and I went into more detail about this on the other thread, but using a model that is pretrained on some dataset is indeed different from using a dataset that you don't have the rights for. Saying that what you did should be allowed because everyone else uses pretrained models too, is not the same thing.
\n\nWith the pretrained model you are not directly using that dataset. As far as I understand it, the pretrained model is not a derivative work from the dataset, so the license terms of the dataset do not apply to the pretrained model (unless you trained that model, since you had to accept the terms of the dataset in order to do so -- but anyone else using your trained model has nothing to do with this dataset or its license terms).
", "rawMarkdown": "I am not a lawyer, and I went into more detail about this on the other thread, but using a model that is pretrained on some dataset is indeed different from using a dataset that you don't have the rights for. Saying that what you did should be allowed because everyone else uses pretrained models too, is not the same thing.\n\nWith the pretrained model you are not directly using that dataset. As far as I understand it, the pretrained model is not a derivative work from the dataset, so the license terms of the dataset do not apply to the pretrained model (unless *you* trained that model, since you had to accept the terms of the dataset in order to do so -- but anyone else using your trained model has nothing to do with this dataset or its license terms).", "votes": 1 }, { "id": 890423, "postDate": "2020-06-17T14:00:43.297Z", "content": "@humananalog fully respect your points, and very good to have a chance to exchange on this topic. I was planning to write something along this line, but let's bounce idea here.
\n\nRegarding the copyright and third-party right issue of deepfake, I found this article to be quite relevant\nKim Kardashian vs. Deepfakes because the nature of the issue is fairly close to what we are talking about here.
\n\nI quote the following:
\n\n\n\n\ncopyright law isn’t the solution to the spread of deepfakes. The high-profile deepfake examples we’ve seen so far mostly appear to fall under the “fair use” exception to copyright infringement....
\n \nFair use is a doctrine in U.S. law that allows for some unlicensed use of material that would otherwise be copyright-protected. To determine whether a specific case qualifies as fair use, we look to four factors: (1) purpose and character of the use, (2) nature of the copyrighted work, (3) amount and substantiality of the portion taken, and (4) effect of the use upon the potential market.
\n \nLet’s use the Kardashian deepfake as an example. The doctored video used Vogue interview video and audio to make it seem like Kardashian was saying something she did not actually say—a confusing message about the truth behind being a social media influencer and manipulating an audience.
\n \nThe “purpose and character” factor seems to weigh in favor of the video being fair use. It does not appear that this video was made for a commercial purpose. It’s arguable that the video was a parody, a form of content often deemed to be “transformative use” for fair use analysis. Basically, this means that the new content added or changed the original content so much that the new content has a new purpose or character.
\n
As @titericz said we have actually never been told which of the videos we used are considered to be violating the rule, and that if @juliaelliott or @cristiancanton can be so kind to share, we can actually have an educated debate in the light of above related information, and perhaps other relevant literature - we may actually all emerged more enlightened.
\n\nFurthermore, this does not change the fact that the \"individual consent\" of video information was never mentioned in interpretation of external data rules by kaggle admin nor the host. As competitors who willingly spend months on this competition, we did our part to make sure the data we used were ok, but we could only act on the information we were given.
", "rawMarkdown": "@humananalog fully respect your points, and very good to have a chance to exchange on this topic. I was planning to write something along this line, but let's bounce idea here.\n\nRegarding the copyright and third-party right issue of deepfake, I found this article to be quite relevant\n**[Kim Kardashian vs. Deepfakes](https://slate.com/technology/2019/06/deepfake-kim-kardashian-copyright-law-fair-use.html)** because the nature of the issue is fairly close to what we are talking about here.\n\nI quote the following:\n\n> copyright law isn’t the solution to the spread of deepfakes. The high-profile deepfake examples we’ve seen so far mostly appear to fall under the “fair use” exception to copyright infringement....\n\n> Fair use is a doctrine in U.S. law that allows for some unlicensed use of material that would otherwise be copyright-protected. To determine whether a specific case qualifies as fair use, we look to four factors: (1) purpose and character of the use, (2) nature of the copyrighted work, (3) amount and substantiality of the portion taken, and (4) effect of the use upon the potential market.\n\n> Let’s use the Kardashian deepfake as an example. The doctored video used Vogue interview video and audio to make it seem like Kardashian was saying something she did not actually say—a confusing message about the truth behind being a social media influencer and manipulating an audience.\n\n> The “purpose and character” factor seems to weigh in favor of the video being fair use. It does not appear that this video was made for a commercial purpose. It’s arguable that the video was a parody, a form of content often deemed to be “transformative use” for fair use analysis. Basically, this means that the new content added or changed the original content so much that the new content has a new purpose or character.\n\nAs @titericz said we have actually never been told which of the videos we used are considered to be violating the rule, and that if @juliaelliott or @cristiancanton can be so kind to share, we can actually have an educated debate in the light of above related information, and perhaps other relevant literature - we may actually all emerged more enlightened.\n\nFurthermore, this does not change the fact that the \"individual consent\" of video information was never mentioned in interpretation of external data rules by kaggle admin nor the host. As competitors who willingly spend months on this competition, we did our part to make sure the data we used were ok, but we could only act on the information we were given. ", "votes": 8 }, { "id": 890531, "postDate": "2020-06-17T14:50:40.557Z", "content": "@yifanxie I agree that the way this was handled doesn't feel quite right. It doesn't sound like you were given a fair chance to defend yourselves.
\n\nAs for \"fair use\", this isn't something I'm qualified to discuss. I just know that a lot of people interpret it wrongly. ;-)
\n\nBut it does seem like the lawyers were very (overly?) cautious in deciding what they would allow and what not, and it sucks to find out about this afterwards.
", "rawMarkdown": "@yifanxie I agree that the way this was handled doesn't feel quite right. It doesn't sound like you were given a fair chance to defend yourselves.\n\nAs for \"fair use\", this isn't something I'm qualified to discuss. I just know that a lot of people interpret it wrongly. ;-) \n\nBut it does seem like the lawyers were very (overly?) cautious in deciding what they would allow and what not, and it sucks to find out about this afterwards.", "votes": 4 }, { "id": 891157, "postDate": "2020-06-18T00:50:47.873Z", "content": "Under similar assumptions, you are committing a serious offense just watching YouTube videos. There is a certain protection when you use \"mislicensed\" material. Enough to give them a chance to fix it. This is unacceptable requirement for a one strike disqualification.
", "rawMarkdown": "Under similar assumptions, you are committing a serious offense just watching YouTube videos. There is a certain protection when you use \"mislicensed\" material. Enough to give them a chance to fix it. This is unacceptable requirement for a one strike disqualification.", "votes": 6 } ] }, { "id": 899449, "postDate": "2020-06-24T08:50:18.980Z", "content": "A great dataset for bot detection is being made thanks to this topic!
", "rawMarkdown": "A great dataset for bot detection is being made thanks to this topic!", "votes": 17, "replies": [ { "id": 899889, "postDate": "2020-06-24T14:00:31.057Z", "content": "Haha, nice one.
\n\nIts truly insane to see at least 6 weird bot-like messages per day on this discussion post. Very confusing. 😐
", "rawMarkdown": "Haha, nice one. \n\nIts truly insane to see at least 6 weird bot-like messages per day on this discussion post. Very confusing. 😐 ", "votes": 3 }, { "id": 899917, "postDate": "2020-06-24T14:15:30.403Z", "content": "Nice! Awesome! Great!Thanks!
", "rawMarkdown": "Nice! Awesome! Great!Thanks!", "votes": 6 }, { "id": 899925, "postDate": "2020-06-24T14:19:07.143Z", "content": "Maybe we should take consent of bots?
", "rawMarkdown": "Maybe we should take consent of bots?", "votes": 5 }, { "id": 909736, "postDate": "2020-06-30T19:31:37.927Z", "content": "@khahuras is adding noise to the data 🤔
", "rawMarkdown": "@khahuras is adding noise to the data 🤔 ", "votes": 4 }, { "id": 936056, "postDate": "2020-07-19T23:52:29.877Z", "content": "Today I noticed something similar is going on here too 🤔
", "rawMarkdown": "Today I noticed something similar is going on [here](https://www.kaggle.com/general/24616) too 🤔 ", "votes": 1 } ] }, { "id": 886363, "postDate": "2020-06-15T01:36:17.343Z", "content": "It's in Kaggle's long-term interests to be more proactive about protecting Kagglers from the legal vagaries of competition hosts - it's terrible for morale and engagement if people are disqualified on a technicality. Kaggle has the ability to insist on being the final arbiter of who is the winner and who is not - Kaggle essentially has a monopoly on running ML competitions, so if a prospective competition host doesn't like Kaggle exerting that amount of control, then the host has limited options for running their competition elsewhere.
", "rawMarkdown": "It's in Kaggle's long-term interests to be more proactive about protecting Kagglers from the legal vagaries of competition hosts - it's terrible for morale and engagement if people are disqualified on a technicality. Kaggle has the ability to insist on being the final arbiter of who is the winner and who is not - Kaggle essentially has a monopoly on running ML competitions, so if a prospective competition host doesn't like Kaggle exerting that amount of control, then the host has limited options for running their competition elsewhere.", "votes": 18, "replies": [ { "id": 886924, "postDate": "2020-06-15T11:14:46.730Z", "content": "Facebook is now testing an alternative to Kaggle for its hate meme competition. I doubt they will get many kagglers...
", "rawMarkdown": "Facebook is now testing an alternative to Kaggle for its hate meme competition. I doubt they will get many kagglers...", "votes": 5 }, { "id": 888625, "postDate": "2020-06-16T13:24:30.910Z", "content": "Speaking from experience, it's definitely not handling things better than Kaggle post-competition. Moreover, there is no community (i.e. active forums, sharing of notebooks, ...) which does not make it appealing.
", "rawMarkdown": "Speaking from experience, it's definitely not handling things better than Kaggle post-competition. Moreover, there is no community (i.e. active forums, sharing of notebooks, ...) which does not make it appealing.", "votes": 6 }, { "id": 900525, "postDate": "2020-06-24T21:43:25.997Z", "content": "Interesting. What is the name of the platform? (I guess I could Google it as well :D)
", "rawMarkdown": "Interesting. What is the name of the platform? (I guess I could Google it as well :D)" } ] }, { "id": 883742, "postDate": "2020-06-12T21:45:47.007Z", "content": "There is a doubt here. Similar to MTCNN/RetinaFace/BlazeFace or more, these face detection models require a large number of face images dataset for training. I would like to ask, have all the face images in them been copyrighted? Or, if we just use the pretrained model, so we do not need to worry about the copyrights, and are also commercially available?
", "rawMarkdown": "There is a doubt here. Similar to MTCNN/RetinaFace/BlazeFace or more, these face detection models require a large number of face images dataset for training. I would like to ask, have all the face images in them been copyrighted? Or, if we just use the pretrained model, so we do not need to worry about the copyrights, and are also commercially available?", "votes": 17, "replies": [ { "id": 884838, "postDate": "2020-06-13T17:02:21.420Z", "content": "All academic databases ... no commercial even available ...
", "rawMarkdown": "All academic databases ... no commercial even available ... ", "votes": 5 } ] }, { "id": 884810, "postDate": "2020-06-13T16:32:04.087Z", "content": "This capricious enforcement of the rules makes no sense. Applying part A of the documentation section uniformly to all external data in this way would invalidate nearly all of the competition entries.
\n\nHas there been a misunderstanding on the part of the sponsors' legal team? They may believe that the competition solution would be directly plugged into their existing production systems and used as-is, thereby incurring potential liabilities.
\n\nI encourage the sponsors' engineering and research talent to reach out to the competition organizers and help them understand how the winning solutions will be used. The most likely scenario, in my mind, is that these solutions would form a set of promising templates for production systems. When viewed in that light, I hope the organizers might have a change of heart about this unfortunate decision.
", "rawMarkdown": "This capricious enforcement of the rules makes no sense. Applying part A of the documentation section uniformly to all external data in this way would invalidate nearly all of the competition entries.\n\nHas there been a misunderstanding on the part of the sponsors' legal team? They may believe that the competition solution would be directly plugged into their existing production systems and used as-is, thereby incurring potential liabilities.\n\nI encourage the sponsors' engineering and research talent to reach out to the competition organizers and help them understand how the winning solutions will be used. The most likely scenario, in my mind, is that these solutions would form a set of promising templates for production systems. When viewed in that light, I hope the organizers might have a change of heart about this unfortunate decision.", "votes": 18, "replies": [ { "id": 885873, "postDate": "2020-06-14T14:30:11.037Z", "content": "Interesting thought! It would be unreasonable to expect that winning Kaggle submissions could be directly plugged into existing production systems, but these expectations are probably unclear at times. Especially for non-technical parties like legal teams and management.
", "rawMarkdown": "Interesting thought! It would be unreasonable to expect that winning Kaggle submissions could be directly plugged into existing production systems, but these expectations are probably unclear at times. Especially for non-technical parties like legal teams and management.", "votes": 6 } ] }, { "id": 884152, "postDate": "2020-06-13T08:03:15.073Z", "content": "I feel very sorry how this ended for your team and thanks for sharing what happened.
\n\nKaggle team should take this seriously and come up with ideas to improve how they run competitions otherwise this tragedy happens again in the future.
\n\nHere is my suggestion.
\n\n\nKaggle team should clearly answer all questions regarding rules. \n
Look at External data thread in recent competitions for example and how often a question(clarification) is left unanswered. This leaves participants in very unconfortable position. We can not be sure what external data is allowed or not allowed, what method is allowed or not allowed beforehand. This ambiguity leads us to unfair competitions too because the ambiguity are solved after competitions by human not by written rules.
\n\nI have observed kaggle's attitude regarding rules. They say read the rules carefully and not answering questions. But what happens if the rules are not clear or interpretation of the rules are different among people.
\n\nAnswering all questions might be redundant and boring but I believe this is the most important part of the competition to be fair and avoid this kind of thing happening again.
", "rawMarkdown": "I feel very sorry how this ended for your team and thanks for sharing what happened. \n\nKaggle team should take this seriously and come up with ideas to improve how they run competitions otherwise this tragedy happens again in the future.\n\nHere is my suggestion.\n\n```\nKaggle team should clearly answer all questions regarding rules. \n```\n\nLook at External data thread in recent competitions for example and how often a question(clarification) is left unanswered. This leaves participants in very unconfortable position. We can not be sure what external data is allowed or not allowed, what method is allowed or not allowed beforehand. This ambiguity leads us to unfair competitions too because the ambiguity are solved after competitions by human not by written rules.\n\nI have observed kaggle's attitude regarding rules. They say read the rules carefully and not answering questions. But what happens if the rules are not clear or interpretation of the rules are different among people.\n\nAnswering all questions might be redundant and boring but I believe this is the most important part of the competition to be fair and avoid this kind of thing happening again.\n", "votes": 18, "replies": [ { "id": 884589, "postDate": "2020-06-13T13:21:11.683Z", "content": "Looks like @goldbloom needs a dashboard on # of unanswered questions relating to rules and not how many courses have been taken on his platform.
\n\nhttps://twitter.com/antgoldbloom/status/1268285195146235905
", "rawMarkdown": "Looks like @goldbloom needs a dashboard on # of unanswered questions relating to rules and not how many courses have been taken on his platform.\n\nhttps://twitter.com/antgoldbloom/status/1268285195146235905", "votes": 3 }, { "id": 884644, "postDate": "2020-06-13T14:17:02.027Z", "content": "Attention from person of position is needed I guess.
", "rawMarkdown": "Attention from person of position is needed I guess.", "votes": 3 } ] }, { "id": 883817, "postDate": "2020-06-13T00:58:08.050Z", "content": "From the perspective of someone who wasn't involved in this competition, the situation doesn't look good. The detailed write-up illustrates that the winning 'All Faces are Real' team put a lot of careful thought into selecting their external data to be compliant with the competition rules. It is also clear that the additional rules that the sponsor was requesting were much more restrictive than usual. Given the substantial effort contributed by all participants, and also the amount of prize money, it would have been very reasonable to expect that those special requirements are clearly communicated. I can empathize with the reactions here and with the winning team; this outcome must be extremely frustrating for them. I hope that we will see more transparent communications of new or unusual rules and conditions in the future.
", "rawMarkdown": "From the perspective of someone who wasn't involved in this competition, the situation doesn't look good. The detailed write-up illustrates that the winning 'All Faces are Real' team put a lot of careful thought into selecting their external data to be compliant with the competition rules. It is also clear that the additional rules that the sponsor was requesting were much more restrictive than usual. Given the substantial effort contributed by all participants, and also the amount of prize money, it would have been very reasonable to expect that those special requirements are clearly communicated. I can empathize with the reactions here and with the winning team; this outcome must be extremely frustrating for them. I hope that we will see more transparent communications of new or unusual rules and conditions in the future.", "votes": 18 }, { "id": 883750, "postDate": "2020-06-12T22:01:06.183Z", "content": "\n\n\nDISPUTE RESOLUTION\n Except where prohibited by law, any and all disputes, claims, and causes of action between you and any Competition Entity arising out of or connected with this Competition, the determination of any winner, or any prize awarded must be resolved individually, without resort to any form of class action. Further, in any such dispute, under no circumstances will any Competition participant be permitted or entitled to obtain awards for, and hereby waives all rights to claim punitive, incidental or consequential damages, or any other damages, including attorneys’ fees, other than the individual participant’s actual out-of-pocket expenses (if any), not to exceed ten dollars ($10 USD), and each individual participant further waives all rights to have damages multiplied or increased.
\n
Another \"funny\" section from the rules. Maybe you could sue for ten dollars...
", "rawMarkdown": "> **DISPUTE RESOLUTION**\nExcept where prohibited by law, any and all disputes, claims, and causes of action between you and any Competition Entity arising out of or connected with this Competition, the determination of any winner, or any prize awarded must be resolved individually, without resort to any form of class action. Further, in any such dispute, under no circumstances will any Competition participant be permitted or entitled to obtain awards for, and hereby waives all rights to claim punitive, incidental or consequential damages, or any other damages, including attorneys’ fees, other than the individual participant’s actual out-of-pocket expenses (if any), not to exceed ten dollars ($10 USD), and each individual participant further waives all rights to have damages multiplied or increased.\n\nAnother \"funny\" section from the rules. Maybe you could sue for ten dollars...", "votes": 18, "replies": [ { "id": 883753, "postDate": "2020-06-12T22:05:21.257Z", "content": "\n\n\nAnother \"funny\" section from the rules. Maybe you could sue for ten dollars...
\n
@gaborfodor yes indeed, they are extremely well protected
", "rawMarkdown": "> Another \"funny\" section from the rules. Maybe you could sue for ten dollars...\n\n@gaborfodor yes indeed, they are extremely well protected", "votes": 8 }, { "id": 884633, "postDate": "2020-06-13T14:05:51.633Z", "content": "Wow, thanks for pointing that out! That specification of ten dollars is pretty wicked. I'm not sure if that clause is even enforceable in the USA. But then again, litigating with Facebook is probably not worth it anyway in terms of costs.
", "rawMarkdown": "Wow, thanks for pointing that out! That specification of ten dollars is pretty wicked. I'm not sure if that clause is even enforceable in the USA. But then again, litigating with Facebook is probably not worth it anyway in terms of costs. ", "votes": 6 }, { "id": 884660, "postDate": "2020-06-13T14:25:18.403Z", "content": "\n\n\nCarlo Lepelaars wrote:
\n \nWow, thanks for pointing that out! That specification of ten dollars is pretty wicked. I'm not sure if that clause is even enforceable in the USA. But then again, litigating with Facebook is probably not worth it anyway in terms of costs.
\n
The suggestion given by our legal representative was that it is valid in California law, and it would be unrealistically expensive to challenge them in the US court for this situation for $10 return
", "rawMarkdown": "> **Carlo Lepelaars wrote:**\n> \n> Wow, thanks for pointing that out! That specification of ten dollars is pretty wicked. I'm not sure if that clause is even enforceable in the USA. But then again, litigating with Facebook is probably not worth it anyway in terms of costs. \n\nThe suggestion given by our legal representative was that it is valid in California law, and it would be unrealistically expensive to challenge them in the US court for this situation for $10 return \n", "votes": 8 }, { "id": 884945, "postDate": "2020-06-13T18:31:27.600Z", "content": "Ok, interesting to hear that something like this holds in California law. I really hope Facebook will still offer some compensation / settlement just to reduce the negative press caused by this decision. 🙂
", "rawMarkdown": "Ok, interesting to hear that something like this holds in California law. I really hope Facebook will still offer some compensation / settlement just to reduce the negative press caused by this decision. 🙂 ", "votes": 2 } ] }, { "id": 883721, "postDate": "2020-06-12T21:26:32.920Z", "content": "very sad moment for kaggle community.
", "rawMarkdown": "very sad moment for kaggle community.", "votes": 18 }, { "id": 883720, "postDate": "2020-06-12T21:26:31.233Z", "content": "That's really awful. Do we need to predict \"hidden rules\" too to win?\nI definitely believe they must be qualified, but kaggle team, if you said it's violated to the rules, please examine and update all of on-going and future competition's rules so that this tragedy would never happen.
", "rawMarkdown": "That's really awful. Do we need to predict \"hidden rules\" too to win?\nI definitely believe they must be qualified, but kaggle team, if you said it's violated to the rules, please examine and update all of on-going and future competition's rules so that this tragedy would never happen. ", "votes": 18 }, { "id": 883703, "postDate": "2020-06-12T20:55:07.773Z", "content": "Basically, you would need to hire a lawyer to diligently check all the data applicability, before you can use it. But even then the host and competition admin can still have their interpretation that would cancel yours
", "rawMarkdown": "Basically, you would need to hire a lawyer to diligently check all the data applicability, before you can use it. But even then the host and competition admin can still have their interpretation that would cancel yours", "votes": 18 }, { "id": 913915, "postDate": "2020-07-03T14:15:57.923Z", "content": "Maybe we need to leave a message once in a while to let new comers know that all the informatic comments are pushed to the bottom by those bot's comments ;)
", "rawMarkdown": "Maybe we need to leave a message once in a while to let new comers know that all the informatic comments are pushed to the bottom by those bot's comments ;)", "votes": 16 }, { "id": 887176, "postDate": "2020-06-15T14:20:19.023Z", "content": "One additional thought. Even if facebook is not willing to accept the top submissions due to some unique and different interpretation of the WINNING SUBMISSION DOCUMENTATION rules it is strange to remove that submission from LB. I remember a few previous competitions when the winner declined prizes and still stayed on LB. It would help at least in terms of points/fame. The loss of 500K$/200K would still hurt though...
", "rawMarkdown": "One additional thought. Even if facebook is not willing to accept the top submissions due to some unique and different interpretation of the WINNING SUBMISSION DOCUMENTATION rules it is strange to remove that submission from LB. I remember a few previous competitions when the winner declined prizes and still stayed on LB. It would help at least in terms of points/fame. The loss of 500K$/200K would still hurt though...", "votes": 15, "replies": [ { "id": 887333, "postDate": "2020-06-15T16:01:50.273Z", "content": "Yes, indeed, we proposed this.
\n\nYou will notice on the overview page it states:\n> Participants will have the option to make their submission open or closed when accepting the prize. Open proposals will be eligible for challenge prizes as long as they abide by the open source licensing terms. Closed proposals will be proprietary and not be eligible to accept the prizes.
\n\nWe suggested that if Facebook does not want to be associated with our solution, we could go for a closed submission which would remove issues about IP and liability for them. But I guess they decided for whatever reason they disagreed.
", "rawMarkdown": "Yes, indeed, we proposed this.\n\nYou will notice on the [overview page](https://www.kaggle.com/c/deepfake-detection-challenge/overview) it states:\n> Participants will have the option to make their submission open or closed when accepting the prize. Open proposals will be eligible for challenge prizes as long as they abide by the open source licensing terms. Closed proposals will be proprietary and not be eligible to accept the prizes.\n\nWe suggested that if Facebook does not want to be associated with our solution, we could go for a closed submission which would remove issues about IP and liability for them. But I guess they decided for whatever reason they disagreed.", "votes": 17 }, { "id": 887372, "postDate": "2020-06-15T16:25:35.170Z", "content": "I think this is completely unreasonable. They should have let you keep the 1st place without the prize
", "rawMarkdown": "I think this is completely unreasonable. They should have let you keep the 1st place without the prize", "votes": 12 }, { "id": 887391, "postDate": "2020-06-15T16:40:18.450Z", "content": "Still be unfair. The prize should be given as well.
", "rawMarkdown": "Still be unfair. The prize should be given as well.", "votes": 9 }, { "id": 887432, "postDate": "2020-06-15T17:08:01.557Z", "content": "Agree.
", "rawMarkdown": "Agree.", "votes": 3 }, { "id": 889801, "postDate": "2020-06-17T06:45:10.127Z", "content": "@juliaelliott Could you please answer why the teams can not be reinstated on LB?
", "rawMarkdown": "@juliaelliott Could you please answer why the teams can not be reinstated on LB?", "votes": 5 } ] }, { "id": 885800, "postDate": "2020-06-14T13:45:52.200Z", "content": "Sorry to see this happen to your team, this must be an emotional roller coaster for your team. There was so much ambiguity in the rules from the start and I had a strong feeling that there was going to be major drama validating results at the end. Seeing the haphazard nature of the rules in this competition probably affected motivation of many competitors and prevented others from participating altogether. Many valid points have been made about what should be done and what is fair. Obviously the legal frameworks around AI and data rights in training deep-learning models e.t.c are not mature, but that said, there are some glaring missteps by Kaggle and Facebook (1) taking the Kaggle community for granted by not thinking about the use of external data comprehensively enough (many points made) do we seek consent from the cats and dogs in imagenet as well? (2) taking the Kaggle community for granted by believing they can haphazardly enforce arbitrary contrived rational for whatever judgment they arrive at. Facebook has nothing to lose here but Kaggle should have the interest of the Kaggle community at heart. (3) taking the Kaggle community for granted by not responding to legitimate questions and concerns for clarification throughout the duration of the competition. To cut them some slack this challenge is not an easy one to setup because of all the data rights issues e.t.c and the problem itself is a Google or Facebook scale problem. My problem is the nitpicky nature of what we have witnessed here. In their internal efforts to detect deep fakes I am %100 sure they are violating people’s data rights if they apply the same arbitrary rules they offered as the reason for the disqualification.
", "rawMarkdown": "Sorry to see this happen to your team, this must be an emotional roller coaster for your team. There was so much ambiguity in the rules from the start and I had a strong feeling that there was going to be major drama validating results at the end. Seeing the haphazard nature of the rules in this competition probably affected motivation of many competitors and prevented others from participating altogether. Many valid points have been made about what should be done and what is fair. Obviously the legal frameworks around AI and data rights in training deep-learning models e.t.c are not mature, but that said, there are some glaring missteps by Kaggle and Facebook (1) taking the Kaggle community for granted by not thinking about the use of external data comprehensively enough (many points made) do we seek consent from the cats and dogs in imagenet as well? (2) taking the Kaggle community for granted by believing they can haphazardly enforce arbitrary contrived rational for whatever judgment they arrive at. Facebook has nothing to lose here but Kaggle should have the interest of the Kaggle community at heart. (3) taking the Kaggle community for granted by not responding to legitimate questions and concerns for clarification throughout the duration of the competition. To cut them some slack this challenge is not an easy one to setup because of all the data rights issues e.t.c and the problem itself is a Google or Facebook scale problem. My problem is the nitpicky nature of what we have witnessed here. In their internal efforts to detect deep fakes I am %100 sure they are violating people’s data rights if they apply the same arbitrary rules they offered as the reason for the disqualification.", "votes": 16 }, { "id": 890000, "postDate": "2020-06-17T08:47:18.033Z", "content": "\n\n\n\"“All of the final five winning teams were held to this same standard.”
\n
Yes, I am sure they were
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F150338%2F00546725ea41e44f41f02d93f5231975%2Ffair_selection.jpeg?generation=1592383589161372&alt=media\")
Last-minute decided standard are nothing but filters
", "rawMarkdown": "Last-minute decided standard are nothing but filters", "votes": 8 }, { "id": 890618, "postDate": "2020-06-17T15:53:19.940Z", "content": "\n\n\nTwist : Fish somehow climbs the tree
\n
Organiser : Oh, You should have taken consent from tree before climbling. You are disqualified 💔
", "rawMarkdown": "> Twist : Fish somehow climbs the tree \n\nOrganiser : Oh, You should have taken consent from **tree** before climbling. You are disqualified 💔 ", "votes": 6 } ] }, { "id": 891696, "postDate": "2020-06-18T11:48:40.613Z", "content": "I just read https://www.theregister.com/2020/06/18/facebook_deepfake_kaggle_contest/
\n\nIt reads:
\n\n\n\n\nFacebook and Kaggle allowed them to resubmit their machine-learning model without any training from external datasets. This bumped them down to seventh place, and they narrowly missed out collecting anything from the prize pot.
\n
I thought the 7th rank solution was another submission, not a retraining of the top solution with less data.
", "rawMarkdown": "I just read https://www.theregister.com/2020/06/18/facebook_deepfake_kaggle_contest/\n\nIt reads:\n\n> Facebook and Kaggle allowed them to resubmit their machine-learning model without any training from external datasets. This bumped them down to seventh place, and they narrowly missed out collecting anything from the prize pot.\n\nI thought the 7th rank solution was another submission, not a retraining of the top solution with less data. ", "votes": 13, "replies": [ { "id": 891713, "postDate": "2020-06-18T12:02:42.053Z", "content": "I guess it is just bad understanding/phrasing from the journalist.
", "rawMarkdown": "I guess it is just bad understanding/phrasing from the journalist.", "votes": 8 }, { "id": 891737, "postDate": "2020-06-18T12:28:07.260Z", "content": "This situation has so many twists, turns and nuances of how Kaggle competitions work. It is hard enough for those of us who are in the know and have been following the story closely to get everything straight. Can only imagine what it would be like for an external person who is not nearly as versed in these subtleties.
", "rawMarkdown": "This situation has so many twists, turns and nuances of how Kaggle competitions work. It is hard enough for those of us who are in the know and have been following the story closely to get everything straight. Can only imagine what it would be like for an external person who is not nearly as versed in these subtleties.", "votes": 10 }, { "id": 891750, "postDate": "2020-06-18T12:40:35.493Z", "content": "This is a misunderstanding from the journalist. Facebook didn't allowed us to resubmit our best solution without Youtube data. Certainly this would scored very the same as our winning solution 0.423.
", "rawMarkdown": "This is a misunderstanding from the journalist. Facebook didn't allowed us to resubmit our best solution without Youtube data. Certainly this would scored very the same as our winning solution 0.423. ", "votes": 19 }, { "id": 891785, "postDate": "2020-06-18T12:59:22.873Z", "content": "\n\n\nour best solution without Youtube data [...] would scored very the same as our winning solution 0.423.
\n
This is what makes this whole story really bad for your team. I feel for you guys.
", "rawMarkdown": "> our best solution without Youtube data [...] would scored very the same as our winning solution 0.423. \n\nThis is what makes this whole story really bad for your team. I feel for you guys.", "votes": 12 }, { "id": 891931, "postDate": "2020-06-18T14:41:57.103Z", "content": "Thank you, uncle. You got the one of the important points😏
", "rawMarkdown": "Thank you, uncle. You got the one of the important points😏 \n", "votes": 14 } ] }, { "id": 893495, "postDate": "2020-06-19T16:50:58.923Z", "content": "Next time, I'd like to suggest to @titericz, @yifanxie, @haqishen, @anokas and @garybios to take the team under the \"Cambridge Analytica Faces Matter\" name. This way Facebook will not bother you about license, privacy or any topic regarding the content to any purpose.
", "rawMarkdown": "Next time, I'd like to suggest to @titericz, @yifanxie, @haqishen, @anokas and @garybios to take the team under the \"Cambridge Analytica Faces Matter\" name. This way Facebook will not bother you about license, privacy or any topic regarding the content to any purpose.", "votes": 14 }, { "id": 887003, "postDate": "2020-06-15T12:28:24.970Z", "content": "I finally read the rules. From @juliaelliott comment, this seems to be the reason why the team was disqualified.
\n\n> A. If any part of the submission documentation depicts, identifies, or includes any person that is not an individual participant or Team member, you must have all permissions and rights from the individual depicted, identified, or included and you agree to provide Competition Sponsor and PAI with written confirmation of those permissions and rights upon request.
\n\nIt means that \"submission documentation\" includes all data used to train models. I frankly doubt this would stand in court. But I am not a lawyer.
", "rawMarkdown": "I finally read the rules. From @juliaelliott comment, this seems to be the reason why the team was disqualified.\n\n> A. If any part of the submission documentation depicts, identifies, or includes any person that is not an individual participant or Team member, you must have all permissions and rights from the individual depicted, identified, or included and you agree to provide Competition Sponsor and PAI with written confirmation of those permissions and rights upon request.\n\nIt means that \"submission documentation\" includes all data used to train models. I frankly doubt this would stand in court. But I am not a lawyer.", "votes": 13, "replies": [ { "id": 887361, "postDate": "2020-06-15T16:20:57.030Z", "content": "I for one would love to see a All Faces Are Real Team vs Facebook, Microsoft, Kaggle case (or really just FB) and would be the very first to donate to a GoFundMe page.
", "rawMarkdown": "I for one would love to see a All Faces Are Real Team vs Facebook, Microsoft, Kaggle case (or really just FB) and would be the very first to donate to a GoFundMe page.", "votes": 7 }, { "id": 887389, "postDate": "2020-06-15T16:39:23.670Z", "content": "Yes, I don't think this interpretation holds much water.
\n\nThe main barrier is that we don't want to spend several years and $100Ks fighting this in californian court, also challenging the \"$10 max dispute\" clause if we don't have to. We would rather be doing more productive things!
", "rawMarkdown": "Yes, I don't think this interpretation holds much water.\n\nThe main barrier is that we don't want to spend several years and $100Ks fighting this in californian court, also challenging the \"$10 max dispute\" clause if we don't have to. We would rather be doing more productive things!", "votes": 14 }, { "id": 887404, "postDate": "2020-06-15T16:54:43.077Z", "content": "So basically, Facebook has just wound themselves into a corner.
\n\nI am sure that this is an ultra-cautious move on their side - they NEVER want to be accused of using \"third-party\" data again, even it means doing something completely out of the blue like this,
", "rawMarkdown": "So basically, Facebook has just wound themselves into a corner.\n\nI am sure that this is an ultra-cautious move on their side - they NEVER want to be accused of using \"third-party\" data again, even it means doing something completely out of the blue like this,", "votes": 5 }, { "id": 887883, "postDate": "2020-06-16T01:09:51.913Z", "content": "Like building a new platform? :)
", "rawMarkdown": "Like building a new platform? :)", "votes": 3 }, { "id": 889080, "postDate": "2020-06-16T18:45:52.983Z", "content": "Yup, but as pointed out, that being the reason is hugely inconsistent. Most teams appear to have at least used some sort of face detection or recognition model. Many (most/all?) of those models (MTCNN, FaceNet, dlib face recognition, etc) are trained on datasets with either non-commercial licenses (like VGG Face, FaceSrub, CASIA-WebFace), no apparent license (WIDER), and none of them have sign-off from the individuals depicted. Also, they likely have less legal protections than Flickr or Youtube based content where the content creator agreed to a CC license. Most of the above datasets were scrubbed from the internet, contain celebs, etc and weren't sourced under a creative commons agreement that the youtube dataset in question adhered to. So, to be consistent, you'd likely have to disqualify a majority of the contestants.
", "rawMarkdown": "Yup, but as pointed out, that being the reason is hugely inconsistent. Most teams appear to have at least used some sort of face detection or recognition model. Many (most/all?) of those models (MTCNN, FaceNet, dlib face recognition, etc) are trained on datasets with either non-commercial licenses (like VGG Face, FaceSrub, CASIA-WebFace), no apparent license (WIDER), and none of them have sign-off from the individuals depicted. Also, they likely have less legal protections than Flickr or Youtube based content where the content creator agreed to a CC license. Most of the above datasets were scrubbed from the internet, contain celebs, etc and weren't sourced under a creative commons agreement that the youtube dataset in question adhered to. So, to be consistent, you'd likely have to disqualify a majority of the contestants.", "votes": 6 } ] }, { "id": 884483, "postDate": "2020-06-13T12:10:36.720Z", "content": "Surprised that no one has cited Elon Musk yet. \"Facebook sucks\"(c).
", "rawMarkdown": "Surprised that no one has cited Elon Musk yet. \"Facebook sucks\"(c).", "votes": 13 }, { "id": 884138, "postDate": "2020-06-13T07:54:09.463Z", "content": "To be honest, I'm not surprised that Kaggle bogged down in front of the giant that is Facebook. \nThe sad part is that they've broken the trust of the Kaggle community, and the don't even seem to be apologetic about it !!
\n\nI agree with @kazanova here. We are NOT lawyers. We are here to learn data science, not law ! It's the responsibility of the Kaggle team to ensure we are made aware of the Do's & Dont's in a clear fashion.
", "rawMarkdown": "To be honest, I'm not surprised that Kaggle bogged down in front of the giant that is Facebook. \nThe sad part is that they've broken the trust of the Kaggle community, and the don't even seem to be apologetic about it !!\n\nI agree with @kazanova here. We are NOT lawyers. We are here to learn data science, not law ! It's the responsibility of the Kaggle team to ensure we are made aware of the Do's & Dont's in a clear fashion.", "votes": 13, "replies": [ { "id": 885149, "postDate": "2020-06-14T00:31:34.657Z", "content": "\n\n\nKaggle bogged down in front of the giant that is Facebook.
\n
Google is a larger giant than Facebook. That's not the issue.
", "rawMarkdown": "> Kaggle bogged down in front of the giant that is Facebook. \n\nGoogle is a larger giant than Facebook. That's not the issue.", "votes": 4 } ] }, { "id": 883697, "postDate": "2020-06-12T20:48:43.880Z", "content": "How Winning Submission Documentation is related to the data that you used? Isn't it just a document describing your approach?
Kaggle not providing any clarification on the external data applicability besides \"read the rules\" was ugly.
", "rawMarkdown": "How `Winning Submission Documentation` is related to the data that you used? Isn't it just a document describing your approach?\n\nKaggle not providing any clarification on the external data applicability besides \"read the rules\" was ugly.", "votes": 13, "replies": [ { "id": 883701, "postDate": "2020-06-12T20:51:54.947Z", "content": "Indeed, this was always my interpretation of the rules too.
\nBut apparently, the entire solution is included under the \"winning submission documentation\", which therefore includes the external data that we link to in our solution. It's flaky at best
Thank you for the transparency, sharing the statement.\nIt looks like there was a twisting of rules/the restrictions weren't made 100% clear, even though the team very meticulously engineered the solution and it appears followed the rules;\nWe all can only imagine the frustrations and empathise with the real winning team. 😞
", "rawMarkdown": "Thank you for the transparency, sharing the statement.\nIt looks like there was a twisting of rules/the restrictions weren't made 100% clear, even though the team very meticulously engineered the solution and it appears followed the rules;\nWe all can only imagine the frustrations and empathise with the real winning team. 😞 ", "votes": 12 }, { "id": 883707, "postDate": "2020-06-12T21:01:17.410Z", "content": "This is totally unfair.
\n\nOne needs a degree in Law to interpret the meaning of all the rules exactly or kaggle should start providing a lawyer assistant to each competitor.
\n\nI guess some faces had Masked On them
", "rawMarkdown": " This is totally unfair.\n\nOne needs a degree in Law to interpret the meaning of all the rules exactly or kaggle should start providing a lawyer assistant to each competitor. \n\nI guess some faces had Masked On them \n", "votes": 11 }, { "id": 883881, "postDate": "2020-06-13T03:19:21.930Z", "content": "Feeling very sorry for the \"All Faces Are Real\" team. I can't even imagine what they must be going through. If Facebook is being too strict due to lawyers, Kaggle should pool in the prize money from their side and declare 2 winners as \"All Faces Are Real\" team is the real winner here.
", "rawMarkdown": "Feeling very sorry for the \"All Faces Are Real\" team. I can't even imagine what they must be going through. If Facebook is being too strict due to lawyers, Kaggle should pool in the prize money from their side and declare 2 winners as \"All Faces Are Real\" team is the real winner here. ", "votes": 12 }, { "id": 883727, "postDate": "2020-06-12T21:30:17.783Z", "content": "This sounds similar to a standard IRB (institutional review board) human subjects research issue to me. Even if there's a public dataset, you have to get permission from the people in that dataset to use it. However, I had asked much earlier in the challenge if Facebook had gone through any sort of IRB process for this dataset, and they said no (https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122113). If they are taking this stance from an IRB perspective, then it's out of the blue.
", "rawMarkdown": "This sounds similar to a standard IRB (institutional review board) human subjects research issue to me. Even if there's a public dataset, you have to get permission from the people in that dataset to use it. However, I had asked much earlier in the challenge if Facebook had gone through any sort of IRB process for this dataset, and they said no (https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122113). If they are taking this stance from an IRB perspective, then it's out of the blue.", "votes": 12, "replies": [ { "id": 883735, "postDate": "2020-06-12T21:36:13.967Z", "content": "I agree, such restrictions would be reasonable if and only if they were actually mentioned by the competition hosts, which they weren't. We just made extra sure we were following the external data rules, including all clarifications posted in the external data thread.
\n\nI don't think it should be up to competitors to research the host's legal/ethical/PR needs and self-impose them too.
", "rawMarkdown": "I agree, such restrictions would be reasonable **if and only if they were actually mentioned by the competition hosts**, which they weren't. We just made extra sure we were following the external data rules, including all clarifications posted in the external data thread.\n\nI don't think it should be up to competitors to research the host's legal/ethical/PR needs and self-impose them too.", "votes": 25 }, { "id": 883747, "postDate": "2020-06-12T21:53:12.450Z", "content": "One of the biggest selling points of Kaggle and Kaggle competitions is that the vast majority of us here are NOT professional researchers. It is extremely unreasonable to expect us to follow all the minutiae of the copyright laws and IRB research standards. Even professional researchers often have hard time discerning what is ethically permissible and what is out of bounds. I know, because I just asked my wife who has been chairing the IRB committee at her university for years, and she had hard time deciding if the human subject consent rules applied in this instance.
\n\nIf Facebook wanted to have that level of rigor, they should have gone to some university department instead of crowdsourcing this project. This is a truly dastardly behavior on their part. 😠
", "rawMarkdown": "One of the biggest selling points of Kaggle and Kaggle competitions is that the vast majority of us here are **NOT** professional researchers. It is extremely unreasonable to expect us to follow all the minutiae of the copyright laws and IRB research standards. Even professional researchers often have hard time discerning what is ethically permissible and what is out of bounds. I know, because I just asked my wife who has been chairing the IRB committee at her university for years, and she had hard time deciding if the human subject consent rules applied in this instance. \n\nIf Facebook wanted to have that level of rigor, they should have gone to some university department instead of crowdsourcing this project. This is a truly dastardly behavior on their part. 😠 ", "votes": 26 } ] }, { "id": 883715, "postDate": "2020-06-12T21:12:55.937Z", "content": "\n\n\nSpecifically, we were asked to provide \"additional permissions or licenses from individuals appearing in [our] external dataset\".
\n
This reminds me of all those non-sense restrictions written in smallest font size on the medical insurance policy. Maybe a J.D degree will be prerequisite to win a competition in the near future.
", "rawMarkdown": "> Specifically, we were asked to provide \"additional permissions or licenses from individuals appearing in [our] external dataset\".\n\nThis reminds me of all those non-sense restrictions written in smallest font size on the medical insurance policy. Maybe a J.D degree will be prerequisite to win a competition in the near future. ", "votes": 12, "replies": [ { "id": 885022, "postDate": "2020-06-13T19:57:13.097Z", "content": "\n\n\nThis reminds me of all those non-sense restrictions written in smallest font size on the medical insurance policy.
\n
huh! you nailed it. :D
", "rawMarkdown": "> This reminds me of all those non-sense restrictions written in smallest font size on the medical insurance policy.\n\nhuh! you nailed it. :D ", "votes": 2 } ] }, { "id": 883710, "postDate": "2020-06-12T21:01:56.063Z", "content": "btw I have deja vu\nGiba was disqualified from the zillow competition because he worked at airbnb...\nhttps://www.kaggle.com/c/zillow-prize-1/discussion/45770
", "rawMarkdown": "btw I have deja vu\nGiba was disqualified from the zillow competition because he worked at airbnb...\nhttps://www.kaggle.com/c/zillow-prize-1/discussion/45770", "votes": 12, "replies": [ { "id": 883757, "postDate": "2020-06-12T22:12:54.880Z", "content": "Yes, its not my first time... and my team mate won the USD1M Zillow competition. Maybe I'm cursed... people won't teamup with me anymore 😭
", "rawMarkdown": "Yes, its not my first time... and my team mate won the USD1M Zillow competition. Maybe I'm cursed... people won't teamup with me anymore 😭 ", "votes": 29 }, { "id": 884264, "postDate": "2020-06-13T09:08:26.410Z", "content": "The rule is that we can team up with you only for competitions below USD1M :-)\nOk, joking aside, it's really unfair because a lot of questions had been posted about rules clarifications and most got no answer.
", "rawMarkdown": "The rule is that we can team up with you only for competitions below USD1M :-)\nOk, joking aside, it's really unfair because a lot of questions had been posted about rules clarifications and most got no answer.", "votes": 8 }, { "id": 884298, "postDate": "2020-06-13T09:20:43.133Z", "content": "Maybe it's Kaggle $1M Competition curse but not yours 😤
", "rawMarkdown": "Maybe it's Kaggle $1M Competition curse but not yours 😤 ", "votes": 11 }, { "id": 884839, "postDate": "2020-06-13T17:02:42.723Z", "content": "@haqishen You might be on to something:\n+ Passenger screening - No non-US residents could claim prize\n+ Zillow - real estate industry employees were removed\n+ Deepfake - this whole issue
", "rawMarkdown": "@haqishen You might be on to something:\n+ Passenger screening - No non-US residents could claim prize\n+ Zillow - real estate industry employees were removed\n+ Deepfake - this whole issue", "votes": 7 }, { "id": 884842, "postDate": "2020-06-13T17:04:38.790Z", "content": "lol ... that is some dark comedy right there ... Maybe stop aiming for that cursed number then?
", "rawMarkdown": "lol ... that is some dark comedy right there ... Maybe stop aiming for that cursed number then?", "votes": 1 }, { "id": 887790, "postDate": "2020-06-15T21:59:05.703Z", "content": "Don't be sad Giba... I'll always be happy to team up with you. This is very frustrating. I recently started doing competitions on other sires like drivendata because of kaggle becoming painful. Very late, ambiguous responses to questions are the standard. It's a bit like they are doing this as a second job...
", "rawMarkdown": "Don't be sad Giba... I'll always be happy to team up with you. This is very frustrating. I recently started doing competitions on other sires like drivendata because of kaggle becoming painful. Very late, ambiguous responses to questions are the standard. It's a bit like they are doing this as a second job... ", "votes": 1 } ] }, { "id": 892714, "postDate": "2020-06-19T05:24:22.700Z", "content": "I know it is a lot to ask after this - but please - don't lose faith and stay on Kaggle! Kaggle community needs you to keep the bar high.
\n\nI was shocked when I saw this story, and I feel for you guys. For me, a bare minimum would be to allow you to retrain your sub without doubtful data (I am putting asterisks because it does not seem to me that the data you used violated competition rules)
", "rawMarkdown": "I know it is a lot to ask after this - but please - don't lose faith and stay on Kaggle! Kaggle community needs you to keep the bar high.\n\nI was shocked when I saw this story, and I feel for you guys. For me, a bare minimum would be to allow you to retrain your sub without ***doubtful data*** (I am putting asterisks because it does not seem to me that the data you used violated competition rules)", "votes": 11, "replies": [ { "id": 893100, "postDate": "2020-06-19T11:44:42.987Z", "content": "Why? How stay comfortable with this disrespectful decision? Who knows it will not happen again? All the work they have, almost in their free time I think, to the competition to be disqualified by an unclear and unilateral decision? I think even other competitors will think twice from now before start any other competition if they can’t rely on the rules or the rulers. It’s a sad event to Kaggle and all the community.
", "rawMarkdown": "Why? How stay comfortable with this disrespectful decision? Who knows it will not happen again? All the work they have, almost in their free time I think, to the competition to be disqualified by an unclear and unilateral decision? I think even other competitors will think twice from now before start any other competition if they can’t rely on the rules or the rulers. It’s a sad event to Kaggle and all the community.", "votes": 14 } ] }, { "id": 883694, "postDate": "2020-06-12T20:44:26.220Z", "content": "I have a feeling that the Kaggle team sometimes not take enough efforts to think about the need of competition partitioner. For example, I remember there is a UI change for the kernel at the last few days of an competition. I think there is a lot of thing Kaggle can do to avoid such frustrating result
", "rawMarkdown": "I have a feeling that the Kaggle team sometimes not take enough efforts to think about the need of competition partitioner. For example, I remember there is a UI change for the kernel at the last few days of an competition. I think there is a lot of thing Kaggle can do to avoid such frustrating result", "votes": 10 }, { "id": 888733, "postDate": "2020-06-16T14:33:52.110Z", "content": "I think we need a new competition: \"Predict when a Deep Fake Challenge is Fake\"
", "rawMarkdown": "I think we need a new competition: \"Predict when a Deep Fake Challenge is Fake\"", "votes": 10 }, { "id": 885606, "postDate": "2020-06-14T10:45:17.997Z", "content": "Sorry to see this, Giba! Organizers should have made rules clear before the competition started.
", "rawMarkdown": "Sorry to see this, Giba! Organizers should have made rules clear before the competition started.", "votes": 10 }, { "id": 885301, "postDate": "2020-06-14T04:52:46.703Z", "content": "After some thoughts, here's my perspective (disregarding my earlier, snarkier comments):
\n\nThe principal entity behind this whole competition was Facebook - a company notorious for not giving a damn (sorry for language) about privacy of the individuals who use its platform, as evidenced by what happened with Cambridge Analytica a few years prior.
\n\nFacebook probably wants to avoid using the data of any third-party individual as it would most likely be attacked a lot, not so much by the affected entity/individual, but more so by the people who dislike these big Silicon Valley companies and want to break them down (it would make a good election premise).
\n\nAs such, this is probably an attempt by Facebook to prevent any sort of damage to its image after all that it has been through. It may seem a bit too much (it most definitely is a bit too much) because Facebook must be trying to oppress anything which could risk even the slightest amount of damage to the company's image post-Cambridge Analytica, or, as @aakashnain said, it could be contempt between Google (which owns Kaggle) and Facebook, or it could be some combination of both.
\n\nEither way this is highly unjust and unfair to the two top teams who worked so hard in this competition - I truly believe Giba, Mikel, Gary, Qishen and Yifan should have won 1st place (no contempt towards Selim Serfebekov - he too deserves GM tier). So regardless of what happened, I think the entire community will regard All Faces are Real as the true winners.
", "rawMarkdown": "After some thoughts, here's my perspective (disregarding my earlier, snarkier comments):\n\nThe principal entity behind this whole competition was Facebook - a company notorious for not giving a damn (sorry for language) about privacy of the individuals who use its platform, as evidenced by what happened with Cambridge Analytica a few years prior.\n\nFacebook probably wants to avoid using the data of any third-party individual as it would most likely be attacked a lot, not so much by the affected entity/individual, but more so by the people who dislike these big Silicon Valley companies and want to break them down (it would make a good election premise).\n\nAs such, this is probably an attempt by Facebook to prevent any sort of damage to its image after all that it has been through. It may seem a bit too much (it most definitely is a bit too much) because Facebook must be trying to oppress anything which could risk even the slightest amount of damage to the company's image post-Cambridge Analytica, or, as @aakashnain said, it could be contempt between Google (which owns Kaggle) and Facebook, or it could be some combination of both.\n\nEither way this is highly unjust and unfair to the two top teams who worked so hard in this competition - I truly believe Giba, Mikel, Gary, Qishen and Yifan should have won 1st place (no contempt towards Selim Serfebekov - he too deserves GM tier). So regardless of what happened, I think the entire community will regard All Faces are Real as the true winners.", "votes": 10 }, { "id": 884406, "postDate": "2020-06-13T11:02:22.020Z", "content": "At this rate, we're going to have lawsuits with headings, \"Kaggle Competitors vs. Facebook\"!! After the Cambridge Analytics scandal, they lost the public's trust, and now they've lost ours.
", "rawMarkdown": "At this rate, we're going to have lawsuits with headings, \"Kaggle Competitors vs. Facebook\"!! After the Cambridge Analytics scandal, they lost the public's trust, and now they've lost ours. ", "votes": 10 }, { "id": 884465, "postDate": "2020-06-13T11:53:39.660Z", "content": "As a matter of fact, there have already been several news articles on this.
\n\nThe irony is that NONE of them mention these unfortunate incidents - your team has used a lot of time and effort and yet this gets swept under the rug.
", "rawMarkdown": "As a matter of fact, there have already been several news articles on this.\n\nThe irony is that NONE of them mention these unfortunate incidents - your team has used a lot of time and effort and yet this gets swept under the rug.", "votes": 8 }, { "id": 888060, "postDate": "2020-06-16T05:12:45.137Z", "content": "I can imagine how the team All Faces are Realfelt about it. It must have been a harrowing month!
I have organized few hackathons and (its a personal opinion) that external datasets are a big legal black hole. But it also reflects the reality of working in data science.... where the maximum benefit/ ROI comes only from mixing & matching data from a number of different sources.
\n\nSo I had a suggestion for a new feature or a widget (for a lack of a better word)
\n\nIt should be an interface which will allow participants to enter a link & name of the external dataset. Once given they can be analyzed for suitability for the competition. This will create a green-list of dataset & highlight ones which cannot be used. So whenever a new dataset is submitted, it automatically gets checked with the green-list & there are no gray areas in the competition.
\n\nIt will remove the current in-efficient means of submitting the information through a message on the forum thread. I have seen Julia and other organizers trying to respond to the same queries again & again. Also for the participant it gives an easy interface to check for green-listed datasets.
\n\nSo that this suggestion gets highlighted and the Kaggle team can evaluate it, I am marking them -
\n\n@antgoldbloom & @mrisdal @juliaelliott
", "rawMarkdown": "\nI can imagine how the team `All Faces are Real `felt about it. It must have been a harrowing month! \n\nI have organized few hackathons and (*its a personal opinion*) that *external datasets* are a big legal black hole. But it also reflects the reality of working in data science.... where the maximum benefit/ ROI comes only from mixing & matching data from a number of different sources. \n\nSo I had a suggestion for a new feature or a widget (for a lack of a better word)\n\nIt should be an interface which will allow participants to enter a link & name of the external dataset. Once given they can be analyzed for suitability for the competition. This will create a green-list of dataset & highlight ones which cannot be used. So whenever a new dataset is submitted, it automatically gets checked with the green-list & there are no gray areas in the competition. \n\nIt will remove the current in-efficient means of submitting the information through a message on the forum thread. I have seen Julia and other organizers trying to respond to the same queries again & again. Also for the participant it gives an easy interface to check for green-listed datasets. \n\n\nSo that this suggestion gets highlighted and the Kaggle team can evaluate it, I am marking them - \n\n@antgoldbloom & @mrisdal @juliaelliott \n", "votes": 7 }, { "id": 885005, "postDate": "2020-06-13T19:43:25.253Z", "content": "This is truly unacceptable. 🙁
", "rawMarkdown": "This is truly unacceptable. 🙁 ", "votes": 7 }, { "id": 884718, "postDate": "2020-06-13T15:20:32.787Z", "content": "My suggestion: what if changing future competitions rules so that competitors have to submit external datasets for approval of Kaggle team during competition. This way there will be no risk of disqualification after competition ends.
", "rawMarkdown": "My suggestion: what if changing future competitions rules so that competitors have to submit external datasets for approval of Kaggle team during competition. This way there will be no risk of disqualification after competition ends.\n", "votes": 7, "replies": [ { "id": 888029, "postDate": "2020-06-16T04:32:18.093Z", "content": "Nah too much work for them. Tbh, i don't even understand what they are doing, these kaggle representatives. The response time is so long it seems like they work only one day a week.
", "rawMarkdown": "Nah too much work for them. Tbh, i don't even understand what they are doing, these kaggle representatives. The response time is so long it seems like they work only one day a week. ", "votes": 3 } ] }, { "id": 884371, "postDate": "2020-06-13T10:19:16.880Z", "content": "I'm sad about what happened to you. It's absolutely absurd. No sane person can figure out such vague rules. As Selim mentioned in one of his replies, he did ask about CC-BY videos multiple times, but he didn't receive any reply from the hosts or the competition organizers. Kaggle has set a precedent that is going to create further confusion in upcoming competitions.
\n\nAlso, I'd like to ask you, are you planning to share your solution code with us?
", "rawMarkdown": "I'm sad about what happened to you. It's absolutely absurd. No sane person can figure out such vague rules. As Selim mentioned in one of his replies, he did ask about CC-BY videos multiple times, but he didn't receive any reply from the hosts or the competition organizers. Kaggle has set a precedent that is going to create further confusion in upcoming competitions. \n\nAlso, I'd like to ask you, are you planning to share your solution code with us? ", "votes": 7 }, { "id": 891631, "postDate": "2020-06-18T10:43:58.360Z", "content": "\n\nJust saw this, may help to get accountability of the host team. Way to go!!
", "rawMarkdown": "[Facebook's $500k deepfake-detector AI contest drama: Winning team disqualified on buried consent technicality](https://www.theregister.com/2020/06/18/facebook_deepfake_kaggle_contest/) \n\nJust saw this, may help to get accountability of the host team. Way to go!!", "votes": 8 }, { "id": 885049, "postDate": "2020-06-13T20:32:53.747Z", "content": "This doesn't sound very fair
", "rawMarkdown": "This doesn't sound very fair", "votes": 8 }, { "id": 884835, "postDate": "2020-06-13T17:01:10.830Z", "content": "This is sad !
\n\nEven with in-depth legal training, you may not get by with Kaggle's unclear rules.
\n\nThe most unfortunate thing is they do not really answer when asked to clarify this or that part of the rules.
\n\nIt reminds me of when they arbitrarily removed competitors from LB after 4 months of hardwork during the Zillow competition because they put on their profiles they were working at Real State Companies (and regardless the job they were doing there)
", "rawMarkdown": "This is sad !\n\nEven with in-depth legal training, you may not get by with Kaggle's unclear rules.\n\nThe most unfortunate thing is they do not really answer when asked to clarify this or that part of the rules.\n\nIt reminds me of when they arbitrarily removed competitors from LB after 4 months of hardwork during the Zillow competition because they put on their profiles they were working at Real State Companies (and regardless the job they were doing there) ", "votes": 8 }, { "id": 884107, "postDate": "2020-06-13T07:35:50.337Z", "content": "This competition will go down, as Franklin D. Roosevelt said, \"a day which will live in infamy.\" (dear President, please do not copyright this)
\n\nNext time we use backprop, remind me to consult Geoff Hinton beforehand.....
", "rawMarkdown": "This competition will go down, as Franklin D. Roosevelt said, \"a day which will live in infamy.\" (dear President, please do not copyright this)\n\nNext time we use backprop, remind me to consult Geoff Hinton beforehand.....", "votes": 8 }, { "id": 883960, "postDate": "2020-06-13T06:32:25.730Z", "content": "Extra licenses?? Are there actually competitions where participants produce extra licenses?
\n\nSorry for what has happened. I think the rule-makers are to blame for this...
", "rawMarkdown": "Extra licenses?? Are there actually competitions where participants produce extra licenses?\n\nSorry for what has happened. I think the rule-makers are to blame for this...", "votes": 8 }, { "id": 883906, "postDate": "2020-06-13T04:14:21.370Z", "content": "My condolences. This sudden requirement was unexpected and the rules were very vague. I'm sad that this happened.
", "rawMarkdown": "My condolences. This sudden requirement was unexpected and the rules were very vague. I'm sad that this happened.", "votes": 6 }, { "id": 884343, "postDate": "2020-06-13T09:48:39.210Z", "content": "I lost 1st place in Liverpool due to a simple leak, and Kaggle understated it in the Recap topic there. It was a mess there. Another story: I just requested inversion (Kaggle staff) to see which tasks were solved by each team in Abstract Reasoning, but he didn’t respond although he disclosed how many teams solved each task (I asked because it can be the evidence for private sharing among top teams). And now this. Too many disappointments.
", "rawMarkdown": "I lost 1st place in Liverpool due to a simple leak, and Kaggle understated it in the Recap topic there. It was a mess there. Another story: I just requested inversion (Kaggle staff) to see which tasks were solved by each team in Abstract Reasoning, but he didn’t respond although he disclosed how many teams solved each task (I asked because it can be the evidence for private sharing among top teams). And now this. Too many disappointments. ", "votes": 7 }, { "id": 915968, "postDate": "2020-07-05T08:50:10.657Z", "content": "That is so unfair. Kaggle usually makes their rules and guidelines pretty straight forward. But this is odd. It is so disappointing when you don't get what you deserve after putting so much efforts. I suggest you inform this to the team at kaggle so that it never happens again.
", "rawMarkdown": "That is so unfair. Kaggle usually makes their rules and guidelines pretty straight forward. But this is odd. It is so disappointing when you don't get what you deserve after putting so much efforts. I suggest you inform this to the team at kaggle so that it never happens again.", "votes": 6 }, { "id": 885165, "postDate": "2020-06-14T01:08:38.647Z", "content": "There is still another team disqualified from top 5 and they have been quiet. What is their problem and their perspective on this?
", "rawMarkdown": "There is still another team disqualified from top 5 and they have been quiet. What is their problem and their perspective on this?", "votes": 5, "replies": [ { "id": 886012, "postDate": "2020-06-14T16:39:55.877Z", "content": "We just published our solution (https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158506). The rules are confusing and we only realised that FF++ is not allowed by the DQ e-mail😭 . Unfortunately, both results are removed from us.
", "rawMarkdown": "We just published our solution (https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158506). The rules are confusing and we only realised that FF++ is not allowed by the DQ e-mail😭 . Unfortunately, both results are removed from us. ", "votes": 8 }, { "id": 888460, "postDate": "2020-06-16T11:22:55Z", "content": "I must say that selective enforcement of the rules is not something I agree with. In the last open imaged competition several teams used the open360 database. This database grants access only to people who have selective email addresses. You can't even request access to it without this email address. The license of use is permissive. @juliaelliott responded that they decided it was ok. In my opinion it was exactly the same as yours situation, perhaps even worse.
", "rawMarkdown": "I must say that selective enforcement of the rules is not something I agree with. In the last open imaged competition several teams used the open360 database. This database grants access only to people who have selective email addresses. You can't even request access to it without this email address. The license of use is permissive. @juliaelliott responded that they decided it was ok. In my opinion it was exactly the same as yours situation, perhaps even worse. ", "votes": 2 } ] }, { "id": 895434, "postDate": "2020-06-21T10:45:14.143Z", "content": "I don't know what these bots are here to achieve, but they are definitely keeping the topic and the questions raised hot! :)
", "rawMarkdown": "I don't know what these bots are here to achieve, but they are definitely keeping the topic and the questions raised hot! :)", "votes": 6 }, { "id": 891911, "postDate": "2020-06-18T14:26:21.723Z", "content": "That’s another misleading decision by Kaggle. I think we may group and start to create another platform and stay away from here. That was very unfair with the team who worked hard and now is disqualified with a pretty unclear statement. Sorry for you guys and let’s move on.
", "rawMarkdown": "That’s another misleading decision by Kaggle. I think we may group and start to create another platform and stay away from here. That was very unfair with the team who worked hard and now is disqualified with a pretty unclear statement. Sorry for you guys and let’s move on.", "votes": 6 }, { "id": 888979, "postDate": "2020-06-16T17:36:48.677Z", "content": "That's really bad and unfortunate =/\nThis competition was really confusing.. If they didn't want any external data usage just explicitly say it in the rules. But instead we got a game of words and our questions were answered vaguely or not answered at all.
", "rawMarkdown": "That's really bad and unfortunate =/\nThis competition was really confusing.. If they didn't want any external data usage just explicitly say it in the rules. But instead we got a game of words and our questions were answered vaguely or not answered at all. ", "votes": 6 }, { "id": 883942, "postDate": "2020-06-13T06:21:16.160Z", "content": "Let's support Trump to defund Facebook, Twitter, etc!
", "rawMarkdown": "Let's support Trump to defund Facebook, Twitter, etc!", "votes": 6 }, { "id": 917355, "postDate": "2020-07-06T12:38:41.770Z", "content": "Deepfake can be used in lots of ways. Hope this can be promoted and to be used in biotech.
", "rawMarkdown": "Deepfake can be used in lots of ways. Hope this can be promoted and to be used in biotech.", "votes": 4 }, { "id": 884516, "postDate": "2020-06-13T12:30:53.717Z", "content": "I felt bad for you for losing 1st position in this very unfortunate way, but congratulations for Gold and being first Kaggler to get 50+ competition golds.
", "rawMarkdown": "I felt bad for you for losing 1st position in this very unfortunate way, but congratulations for Gold and being first Kaggler to get 50+ competition golds.", "votes": 5 }, { "id": 898361, "postDate": "2020-06-23T13:37:04.430Z", "content": "Hmmmm so bots from fb, twitter and ig also want to become data-scientist ,no doubt this field is very competitive .
", "rawMarkdown": "Hmmmm so bots from fb, twitter and ig also want to become data-scientist ,no doubt this field is very competitive .", "votes": 3 }, { "id": 936127, "postDate": "2020-07-20T02:30:22.800Z", "content": "Finally, all the bot comments are deleted.
", "rawMarkdown": "Finally, all the bot comments are deleted.", "votes": 4 }, { "id": 891703, "postDate": "2020-06-18T11:55:02.100Z", "content": "Have there been any other competitions where an in-the-money team was disqualified for non-cheating reasons?
", "rawMarkdown": "Have there been any other competitions where an in-the-money team was disqualified for non-cheating reasons?", "votes": 3, "replies": [ { "id": 891710, "postDate": "2020-06-18T11:59:28.483Z", "content": "Giba was removed from Zillow because he was working at Airbnb at the time. Zillow said Airbnb was in same industry as them, which was forbidden in rules. But Airbnb is not in real estate.
\n\nIt was another $1M prize competition...
", "rawMarkdown": "Giba was removed from Zillow because he was working at Airbnb at the time. Zillow said Airbnb was in same industry as them, which was forbidden in rules. But Airbnb is not in real estate.\n\nIt was another $1M prize competition...", "votes": 9 }, { "id": 891715, "postDate": "2020-06-18T12:06:19.137Z", "content": "Moral of the story seems to be that you should get a lawyer on your team if you're expecting to win a high-dollar Kaggle competition.
", "rawMarkdown": "Moral of the story seems to be that you should get a lawyer on your team if you're expecting to win a high-dollar Kaggle competition. ", "votes": 14 } ] }, { "id": 886338, "postDate": "2020-06-15T00:11:30.220Z", "content": "I would like to add that kaggle is selective in the enforcement of the rules. They might be following the host guidelines in that, but I think this is not fair to us. In the open images competition the winning solution disclosed its external sources after the deadline. It might have been that Google are more relaxed about the rules (and it was a small prize) but for us, the competitors, this should be enforced by kaggle to ensure level ground.
", "rawMarkdown": "I would like to add that kaggle is selective in the enforcement of the rules. They might be following the host guidelines in that, but I think this is not fair to us. In the open images competition the winning solution disclosed its external sources after the deadline. It might have been that Google are more relaxed about the rules (and it was a small prize) but for us, the competitors, this should be enforced by kaggle to ensure level ground. \n", "votes": 3 }, { "id": 909731, "postDate": "2020-06-30T19:24:42.660Z", "content": "Amazing FAIR I guess. However, how can we optimize fronters between law policies and AI human well-being?
", "rawMarkdown": "Amazing FAIR I guess. However, how can we optimize fronters between law policies and AI human well-being?\n", "votes": 2 }, { "id": 884809, "postDate": "2020-06-13T16:30:56.647Z", "content": "Dear all,
\n\nI have published a separate thread here trying to address the problems - https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158272 .
", "rawMarkdown": "Dear all, \n\nI have published a separate thread here trying to address the problems - https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158272 . ", "votes": 3 }, { "id": 961432, "postDate": "2020-08-07T07:23:37.470Z", "content": "Gray area of rules attacks again!
", "rawMarkdown": "Gray area of rules attacks again!", "votes": 1 }, { "id": 904170, "postDate": "2020-06-27T12:08:13.713Z", "content": "As we learn, we will better understand what can be done with these bots
", "rawMarkdown": "As we learn, we will better understand what can be done with these bots", "votes": 1, "replies": [ { "id": 904188, "postDate": "2020-06-27T12:28:44.363Z", "content": "Or not... They are driving us crazy for over a year....
", "rawMarkdown": "Or not... They are driving us crazy for over a year.... ", "votes": 1 } ] }, { "id": 889389, "postDate": "2020-06-17T00:16:44.113Z", "content": "Not wanting to hijack this thread in any way, but I contemplated using YouTube videos as well but dropped the idea as it seems YouTube eula (terms of service) \"discourage\" downloads. It is a very difficult and convoluted agreement to read and understand so i dropped the idea.
", "rawMarkdown": "Not wanting to hijack this thread in any way, but I contemplated using YouTube videos as well but dropped the idea as it seems YouTube eula (terms of service) \"discourage\" downloads. It is a very difficult and convoluted agreement to read and understand so i dropped the idea. ", "votes": 1 }, { "id": 889085, "postDate": "2020-06-16T18:49:41.313Z", "content": "I’d like to attempt to clarify the underlying issue with All Faces are Real’s disqualified submission. Some of the videos/images used in the disqualified submission were mis-licensed, in that they contained content belonging to other third parties (such as CNN), but were inappropriately offered under open source licenses. This content also clearly depicted third parties and used third party data whose permissions had not been obtained, in violation of the competition rules.
\n\nWe cannot overlook holding users responsible to the competition-specific rules for the data brought into this competition. All of the final five winning teams were held to this same standard. Likewise, Facebook created the dataset for this competition with consenting actors specifically to ensure fair data use practices and avoid the risk of using external sources for which rights could not be guaranteed.
\n\nRetrospectively, we recognize that the \"submission documentation\" rule’s application to include external data could have been reinforced. It was not our expectation that this was unclear, given the inclusion of external data as part of that documentation. Unfortunately, the specific videos' contents and mislicensing were not anticipated in order to know this would arise as an issue. However, we now acknowledge this was a source of misunderstanding.
\n\nWe absolutely could have done better.
\n\nWe will be taking steps to increase the clarity and host responsiveness around external data use and their competition-specific rules interpretation generally, especially when there is ambiguity. In fact, we’ve begun proactively raising the need for heightened forum engagement with current and prospective hosts.
\n\nKnow that every host will have varied levels of risk tolerance and code scrutiny. While it’s realistically not possible for all questions to receive a response, we commit to better alignment between the forum response level and the level of scrutiny and enforcement expected by a host. We appreciate your constructive recommendations, including the possibility of a community advisory board, and will explore which make sense to implement.
\n\nWithout our community, Kaggle would cease to exist. Our hosts will tell you that we consistently default to standing and siding with our users. We will continue to advocate for our community and commit to not allowing this to happen again.
", "rawMarkdown": "I’d like to attempt to clarify the underlying issue with All Faces are Real’s disqualified submission. Some of the videos/images used in the disqualified submission were mis-licensed, in that they contained content belonging to other third parties (such as CNN), but were inappropriately offered under open source licenses. This content also clearly depicted third parties and used third party data whose permissions had not been obtained, in violation of the competition rules.\n\nWe cannot overlook holding users responsible to the competition-specific rules for the data brought into this competition. All of the final five winning teams were held to this same standard. Likewise, Facebook created the dataset for this competition with consenting actors specifically to ensure fair data use practices and avoid the risk of using external sources for which rights could not be guaranteed.\n\nRetrospectively, we recognize that the \"submission documentation\" rule’s application to include external data could have been reinforced. It was not our expectation that this was unclear, given the inclusion of external data as part of that documentation. Unfortunately, the specific videos' contents and mislicensing were not anticipated in order to know this would arise as an issue. However, we now acknowledge this was a source of misunderstanding. \n\nWe absolutely could have done better.\n\nWe will be taking steps to increase the clarity and host responsiveness around external data use and their competition-specific rules interpretation generally, especially when there is ambiguity. In fact, we’ve begun proactively raising the need for heightened forum engagement with current and prospective hosts. \n\nKnow that every host will have varied levels of risk tolerance and code scrutiny. While it’s realistically not possible for all questions to receive a response, we commit to better alignment between the forum response level and the level of scrutiny and enforcement expected by a host. We appreciate your constructive recommendations, including the possibility of a community advisory board, and will explore which make sense to implement.\n\nWithout our community, Kaggle would cease to exist. Our hosts will tell you that we consistently default to standing and siding with our users. We will continue to advocate for our community and commit to not allowing this to happen again.\n", "votes": -10, "replies": [ { "id": 889120, "postDate": "2020-06-16T19:21:07.860Z", "content": "The pretrained models used in this competition did not even have commercial friendly licenses and were basically web scrubbed datasets containing celebrities and inviduals with zero consent, not even a user violating TOS of a platform by uploading content they didn't own. Those few violating clips can be removed and you'd likely see minimal/if any change in the score. Now try elminating the pretrained face detection/recognition models from the challenge and see how it impacts the scores...
", "rawMarkdown": "The pretrained models used in this competition did not even have commercial friendly licenses and were basically web scrubbed datasets containing celebrities and inviduals with zero consent, not even a user violating TOS of a platform by uploading content they didn't own. Those few violating clips can be removed and you'd likely see minimal/if any change in the score. Now try elminating the pretrained face detection/recognition models from the challenge and see how it impacts the scores...", "votes": 36 }, { "id": 889145, "postDate": "2020-06-16T19:48:27.540Z", "content": "@juliaelliott Also, I'd like to point out, the Kaggle team / Google has not been decisive on the ImageNet pretrained weights for challenges requiring commercial friendly code/data licenses in a competition. I don't believe a court has set a precedent for this, but by never taking a stand on this one way or the other for years, you keep leaving that door open for another company in a future competition to do something similar to this and say 'no, our legal team doesn't like this solution because ImageNet is licensed under a non-commercial license'. You can go back in numerous competitions and find this question unaswered. Same for the face detect/recognition models here, except many weren't even used as pretrained starting points, but used directly.
", "rawMarkdown": "@juliaelliott Also, I'd like to point out, the Kaggle team / Google has not been decisive on the ImageNet pretrained weights for challenges requiring commercial friendly code/data licenses in a competition. I don't believe a court has set a precedent for this, but by never taking a stand on this one way or the other for years, you keep leaving that door open for another company in a future competition to do something similar to this and say 'no, our legal team doesn't like this solution because ImageNet is licensed under a non-commercial license'. You can go back in numerous competitions and find this question unaswered. Same for the face detect/recognition models here, except many weren't even used as pretrained starting points, but used directly.", "votes": 20 }, { "id": 889151, "postDate": "2020-06-16T19:52:51.957Z", "content": "To make things more fair and transparent in the future, Kaggle should have an agreement with the Hosts, that when a challenge is underway and Kaggle does not have a definitive answer on an external dataset/pretrained model question, the Host should be under some sort of obligation to respond in a timely manner so everyone is on the same page. And if mistakes such as a large dataset that's SUPPOSED to be CC has some mistakes due to users of a platform violating a TOS, then there should be an opportunity to retrain the model without the infringing data in a timely manner when a prize or top spot is on the line.
", "rawMarkdown": "To make things more fair and transparent in the future, Kaggle should have an agreement with the Hosts, that when a challenge is underway and Kaggle does not have a definitive answer on an external dataset/pretrained model question, the Host should be under some sort of obligation to respond in a timely manner so everyone is on the same page. And if mistakes such as a large dataset that's SUPPOSED to be CC has some mistakes due to users of a platform violating a TOS, then there should be an opportunity to retrain the model without the infringing data in a timely manner when a prize or top spot is on the line.", "votes": 22 }, { "id": 889163, "postDate": "2020-06-16T20:06:09.903Z", "content": "After reading your explanation it still completely unfair to disqualify our top solution.
\n\nTaking into account what you said about \"mis-licensed\" videos, I must ask you about \"mis-licensed\" pre-trained models, specially the ones that contains images of people scraped from internet.
\n\nOne example: MTCNN face detection algorithm was used by many top teams. This model was trained with faces scraped from internet. It uses two datasets:
\n\nCASIA-WebFace: The images in this dataset are crawled from internet and are for non-commercial use only.
VGGface2: This dataset is scraped from internet and realeased under CC license, but it doesn't hold individual licenses for the people in the dataset.
Are these datasets really allowed and do not infringe any license? Or after training an open-source model with \"mis-licensed\" datasets it becomes licensed?
\n\nIt would have been so easy if external data had not been allowed from the beginning...
", "rawMarkdown": "After reading your explanation it still completely unfair to disqualify our top solution.\n\nTaking into account what you said about \"mis-licensed\" videos, I must ask you about \"mis-licensed\" pre-trained models, specially the ones that contains images of people scraped from internet.\n\n\nOne example: MTCNN face detection algorithm was used by many top teams. This model was trained with faces scraped from internet. It uses two datasets: \n\n- [CASIA-WebFace](http://www.cbsr.ia.ac.cn/english/CASIA-WebFace/CASIA-WebFace_Agreements.pdf): The images in this dataset are crawled from internet and are for non-commercial use only.\n\n- [VGGface2]( https://www.robots.ox.ac.uk/~vgg/data/vgg_face2/): This dataset is scraped from internet and realeased under CC license, but it doesn't hold individual licenses for the people in the dataset.\n\n\nAre these datasets really allowed and do not infringe any license? Or after training an open-source model with \"mis-licensed\" datasets it becomes licensed?\n\n\nIt would have been so easy if external data had not been allowed from the beginning...", "votes": 30 }, { "id": 889175, "postDate": "2020-06-16T20:11:12.283Z", "content": "People ask me why I got disqualified and I still don't know why. Every time we asked hosts we got a different answer.
\n\nPlease, tell me: what is the \"real\" reason for disqualifying us?
", "rawMarkdown": "People ask me why I got disqualified and I still don't know why. Every time we asked hosts we got a different answer. \n\nPlease, tell me: what is the \"real\" reason for disqualifying us?", "votes": 22 }, { "id": 889217, "postDate": "2020-06-16T20:42:11.230Z", "content": "FYI import face_recognition also used quite a bit, is VGG Face (version before 2, non-commercial), FaceScrub, and manually scrubbed images by the dlib authors.
The rules as far as most kaggler understand them are that we use things by their license. You can't expect us to go and verify the individual license for every image/video in every dataset we use!\nMoreover, in such case, where the team really went out of their way to be honest, they should be given a chance to correct their submission. This is basic moral behavior. To do otherwise just shows lack of respect for the people. \nAlso ALL QUESTIONS MUST BE ANSWERED. I honestly don't get where this lofty attitude is coming from. The amount of questions in the external thread would not amount to more than an hour work a day.
", "rawMarkdown": "The rules as far as most kaggler understand them are that we use things by their license. You can't expect us to go and verify the individual license for every image/video in every dataset we use!\nMoreover, in such case, where the team really went out of their way to be honest, they should be given a chance to correct their submission. This is basic moral behavior. To do otherwise just shows lack of respect for the people. \nAlso ALL QUESTIONS MUST BE ANSWERED. I honestly don't get where this lofty attitude is coming from. The amount of questions in the external thread would not amount to more than an hour work a day. ", "votes": 13 }, { "id": 889228, "postDate": "2020-06-16T20:54:48.493Z", "content": "IMHO, this mis-licensed stuff makes everything to a infinite loop. Yes, top team used mis-licensed data to train a model, then what about other pretrained models used mis-licensed data? Did Kaggle/Facebook check the licenses of each individual image used to obtain all those pretrained models? What about other models/packages using non-commercial licenses?
\n\nIt is hard to convince people there is only one standard applied here.
", "rawMarkdown": "IMHO, this mis-licensed stuff makes everything to a infinite loop. Yes, top team used mis-licensed data to train a model, then what about other pretrained models used mis-licensed data? Did Kaggle/Facebook check the licenses of each **individual image** used to obtain all those pretrained models? What about other models/packages using non-commercial licenses?\n\nIt is hard to convince people there is only one standard applied here. ", "votes": 15 }, { "id": 889235, "postDate": "2020-06-16T21:00:15.390Z", "content": "Even if it were true.
\n\n\n\n\nIn the event that the Submission demonstrates non-compliance with these Competition Rules, Competition Sponsor may at its discretion take either of the following actions: (i) disqualify the Submission(s); or (ii) require the potential winner to remediate within one week after notice all issues identified in the Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).
\n
Would not be ii) better solution? If the host has issues with a few mislicensed images/videos why not allow the team to retrain the models without them? They had the last two months...
", "rawMarkdown": "Even if it were true.\n> In the event that the Submission demonstrates non-compliance with these Competition Rules, Competition Sponsor may at its discretion take either of the following actions: (i) disqualify the Submission(s); or (ii) require the potential winner to remediate within one week after notice all issues identified in the Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).\n\nWould not be ii) better solution? If the host has issues with a few mislicensed images/videos why not allow the team to retrain the models without them? They had the last two months...", "votes": 10 }, { "id": 889239, "postDate": "2020-06-16T21:03:18.677Z", "content": "Again. Even if it were true and the host decides to not pay for the best submission why remove it from the LB (ranking/points/fame)?
", "rawMarkdown": "Again. Even if it were true and the host decides to not pay for the best submission why remove it from the LB (ranking/points/fame)?", "votes": 9 }, { "id": 889260, "postDate": "2020-06-16T21:26:09.557Z", "content": "Allowing the hosts to disqualify teams on the slightest license issues without providing exact and detailed evidence to the teams and not allowing the teams to challenge it in court is also unfair. I would like that section of the rules to be removed in future competitions. That would encourage the hosts to be more constructive during the winner documentation discussions.
", "rawMarkdown": "Allowing the hosts to disqualify teams on the slightest license issues without providing exact and detailed evidence to the teams and not allowing the teams to challenge it in court is also unfair. I would like that section of the rules to be removed in future competitions. That would encourage the hosts to be more constructive during the winner documentation discussions.", "votes": 13 }, { "id": 889271, "postDate": "2020-06-16T21:47:41.887Z", "content": "My response regarding the pre-trained model questions is on this thread.
", "rawMarkdown": "My response regarding the pre-trained model questions is [on this thread](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158244#889086). ", "votes": -4 }, { "id": 889280, "postDate": "2020-06-16T21:58:10.357Z", "content": "BTW, just came up with an idea: use whatever data (well, there might be some mis-licensed stuff or forbidden stuff, who knows) to train a model myself, open source it as a pretrained model and give it BSD 2-Clause \"Simplified\" License (yes, I may mis-license it). Don't I just by-pass the external data/license limit?
", "rawMarkdown": "BTW, just came up with an idea: use whatever data (well, there might be some mis-licensed stuff or forbidden stuff, who knows) to train a model myself, open source it as a pretrained model and give it BSD 2-Clause \"Simplified\" License (yes, I may mis-license it). Don't I just by-pass the external data/license limit? ", "votes": 12 }, { "id": 889560, "postDate": "2020-06-17T02:27:17.893Z", "content": "I’m neutral. The feeling of losing 1st prize is bitter. You are a victim of rule vagueness. However, I suppose the only thing that makes sense here is:\n“All of the final five winning teams were held to this same standard.”\nI think this reason would alleviate your pain because at least you still have a submission of no external data that ranks very high (7th).
", "rawMarkdown": "I’m neutral. The feeling of losing 1st prize is bitter. You are a victim of rule vagueness. However, I suppose the only thing that makes sense here is:\n“All of the final five winning teams were held to this same standard.”\nI think this reason would alleviate your pain because at least you still have a submission of no external data that ranks very high (7th). ", "votes": 1 }, { "id": 889876, "postDate": "2020-06-17T07:38:39.987Z", "content": "Unfortunately this explanation just raises more questions. So if ImageNet pre-trained weights are acceptable then all we need to do to comply to these rules is train a model on any data - commercial or otherwise - post the model to github under an open-source license and we are in compliance with all rules.
\n\nThat is the precedence Kaggle has set here for this ridiculous interpretation of the rules.
\n\nIf Microsoft and Google (the largest companies in the world) cannot determine whether an image is subject to copyright and openly copy and distribute them infringing on licenses then how do you expect contestants to validate whether a dataset with a permitted license published on the Internet is not in compliance with the stated license?
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F389345%2F162fdbe3629321eb5656734c458e090b%2FScreen%20Shot%202020-06-17%20at%205.40.26%20pm%201.png?generation=1592379663135365&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F389345%2Fe0cd0ad96016a921b17232a0bd2b6c9e%2FScreen%20Shot%202020-06-17%20at%205.48.00%20pm.png?generation=1592380101338726&alt=media\")
Most of the points I had in my mind have been laid out correctly by @rwightman . Thank you @rwightman . I have responded to the ImageNet pretrained models query in this thread.
Since the community cannot obtain datasets by themselves and that too with written consent for each sample in the datasets, here is what I suggest:
\n\nApart from these, I have one question that is still unanswered. Google and Facebook have done exceptional ML research. But for any research, the team needs data. Are the teams aided by legal teams? Is there any validation for any dataset being used prior to a research paper written by the team and the paper being public?
", "rawMarkdown": "Most of the points I had in my mind have been laid out correctly by @rwightman . Thank you @rwightman . I have responded to the `ImageNet` pretrained models query in this [thread](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158244#889968). \n\nSince the community cannot obtain datasets by themselves and that too with written consent for each sample in the datasets, here is what I suggest:\n\n1. Ban the usage of IamgeNet and ImageNet pretrained models. If we don't have a clear cut answer regarding licensing, copyright, consent, etc for individual samples, then this can always be misused against the competitors.\n2. Ban any external data. Neither Kaggle nor any individual can validate those. So, why allow it in the first place? Ask the host to provide links to any external data they are okay to use with.\n\nApart from these, I have one question that is still unanswered. Google and Facebook have done exceptional ML research. But for any research, the team needs data. Are the teams aided by legal teams? Is there any validation for any dataset being used prior to a research paper written by the team and the paper being public?", "votes": 11 }, { "id": 891537, "postDate": "2020-06-18T08:43:14.363Z", "content": "@juliaelliott: aside from those static Welcome, External Data Disclosure, Looking for a Team, and New to Machine Learning or Kaggle thread in every competition, I would suggest to kindly add this new thread \"Challenge Competition Rules\". This is for clarification of competition rules given that every seasoned and/or newbie competitors sometimes have difficulty in the interpretation of any such competition rules.
", "rawMarkdown": "@juliaelliott: aside from those static Welcome, External Data Disclosure, Looking for a Team, and New to Machine Learning or Kaggle thread in every competition, I would suggest to kindly add this new thread \"Challenge Competition Rules\". This is for clarification of competition rules given that every seasoned and/or newbie competitors sometimes have difficulty in the interpretation of any such competition rules.", "votes": 2 } ] }, { "id": 917439, "postDate": "2020-07-06T14:09:24.777Z", "content": "Sometimes I think why they make such guidelines when they can make it pretty straight forward great work guys \nall the best for upcoming competitions.
", "rawMarkdown": "Sometimes I think why they make such guidelines when they can make it pretty straight forward great work guys \nall the best for upcoming competitions.", "votes": 1 }, { "id": 915972, "postDate": "2020-07-05T08:53:55.687Z", "content": "Using external datasets for modeling this was a cunning move. I believe such an attitude and techniques are expected from a real-life data scientist. Great work!
", "rawMarkdown": "Using external datasets for modeling this was a cunning move. I believe such an attitude and techniques are expected from a real-life data scientist. Great work!", "votes": 1 }, { "id": 914348, "postDate": "2020-07-03T19:27:03.753Z", "content": "Great job, congratulation for the efforts you have made, you deserve a lot and the best
", "rawMarkdown": "Great job, congratulation for the efforts you have made, you deserve a lot and the best", "votes": 1 }, { "id": 891502, "postDate": "2020-06-18T08:07:00.370Z", "content": "silly world
", "rawMarkdown": "silly world" }, { "id": 893614, "postDate": "2020-06-19T18:47:47.713Z", "content": "Nevertheless, Keep competing on kaggle. Kaggle community runs because of many people but such grand masters who post their great solutions that helps many beginners to start with. Your time will come.
\n\nRead the solution and didn't agreed with reasons provided for the disqualification. Keep up the hope!
", "rawMarkdown": "Nevertheless, Keep competing on kaggle. Kaggle community runs because of many people but such grand masters who post their great solutions that helps many beginners to start with. Your time will come.\n\nRead the solution and didn't agreed with reasons provided for the disqualification. Keep up the hope! " }, { "id": 883799, "postDate": "2020-06-13T00:15:58.413Z", "content": "Thank you for sharing your perspective and inviting this dialogue. This competition was unique in many ways, from the topic it tackled to its design and the host-defined rules. At Kaggle, we spend a great deal of time attempting to foresee possible issues and guide our hosts and users alike towards positive outcomes. However, we recognize this situation was not as clear as it needed to be.
\n\nI am obliged to remind participants of the importance of reviewing the rules for each and every competition you wish to enter in its entirety, as the rules form a binding legal agreement between you and the competition sponsor. In particular, hosts call out their “Competition-Specific Rules” in Section A. That said, this experience reminds us that we can better advocate for improving participants’ understanding of these rules.
\n\nFor the time and effort every user invests in contributions on Kaggle, we owe you the commitment to minimizing barriers to achieving what we’re actually here for: doing machine learning work and fostering a community for sharing that work. We have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.
", "rawMarkdown": "Thank you for sharing your perspective and inviting this dialogue. This competition was unique in many ways, from the topic it tackled to its design and the host-defined rules. At Kaggle, we spend a great deal of time attempting to foresee possible issues and guide our hosts and users alike towards positive outcomes. However, we recognize this situation was not as clear as it needed to be.\n\nI am obliged to remind participants of the importance of reviewing the rules for each and every competition you wish to enter in its entirety, as the rules form a binding legal agreement between you and the competition sponsor. In particular, hosts call out their “Competition-Specific Rules” in Section A. That said, this experience reminds us that we can better advocate for improving participants’ understanding of these rules.\n\nFor the time and effort every user invests in contributions on Kaggle, we owe you the commitment to minimizing barriers to achieving what we’re actually here for: doing machine learning work and fostering a community for sharing that work. We have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.\t", "votes": -32, "replies": [ { "id": 883802, "postDate": "2020-06-13T00:30:15.407Z", "content": "@juliaelliott let me ask you a very simple question: do pre-trained models that use human images require a written consent from those humans before they are qualified for prizes in this competition?
", "rawMarkdown": "@juliaelliott let me ask you a very simple question: do pre-trained models that use human images require a written consent from those humans before they are qualified for prizes in this competition?", "votes": 40 }, { "id": 883840, "postDate": "2020-06-13T01:32:07.110Z", "content": "\n\nFirst of all, I understand the importance for the organiser to be able to get a solution that can be used given the amount of money spent here . I also understand you've spent a lot of time trying to come up with this solution and probably was not easy.
\n\nIt goes without saying that all participants need to carefully read the rules. However it seems to me that some rules are open to interpretation- at least for the typical kaggle member (who is an analyst/data scientist) . For example it is not clear whether you are allowed to use a certain video with a CC licence . We are not lawyers. The grand majority of the kaggle community (I would presume) are not lawyers nor can afford one to participate in the competitions.
\n\nKaggle needs to put more effort to address all queries regarding rules and external data (as the external data causing the issue seems to be referenced there) throughout the course of the competition. Some questions regarding the external data remain unanswered (or get an answer really late). Same goes for questions regarding the rules. Then there are also changes in the rules midway (or late) in a competition that people might miss.
\n\nI think for bigger prize competitions , the organiser can provide a lawyer (or another qualified admin) to address these copyright/rules-related queries . For the rest, more effort is needed on addressing these queries as soon as they are raised.
", "rawMarkdown": "@juliaelliott . \n\nFirst of all, I understand the importance for the organiser to be able to get a solution that can be used given the amount of money spent here . I also understand you've spent a lot of time trying to come up with this solution and probably was not easy.\n\nIt goes without saying that all participants need to carefully read the rules. However it seems to me that some rules are open to interpretation- at least for the typical kaggle member (who is an analyst/data scientist) . For example it is not clear whether you are allowed to use a certain video with a CC licence . We are not lawyers. The grand majority of the kaggle community (I would presume) are not lawyers nor can afford one to participate in the competitions.\n\n**Kaggle needs to put more effort** to address all queries regarding rules and external data (as the external data causing the issue seems to be referenced there) throughout the course of the competition. Some questions regarding the external data remain unanswered (or get an answer really late). Same goes for questions regarding the rules. Then there are also changes in the rules midway (or late) in a competition that people might miss.\n\nI think for bigger prize competitions , **the organiser can provide a lawyer (or another qualified admin)** to address these copyright/rules-related queries . For the rest, more effort is needed on addressing these queries as soon as they are raised. \n\n", "votes": 27 }, { "id": 883841, "postDate": "2020-06-13T01:35:51.553Z", "rawMarkdown": "", "votes": 23, "isDeleted": true }, { "id": 883842, "postDate": "2020-06-13T01:38:01.887Z", "content": "Julia - thanks for your input regarding the need to follow the rules. I think all teams appreciate this point and actually did their best to adhere to the rules.
\n\nIn their post, \"All Faces are Real\" team has made it crystal clear how they took great care to follow all competition rules. To me, their explanation is 100% convincing - I can't see how any rule was broken. I am happy to be proven wrong, but I would like to understand. It seems many Kaggles are sharing my view and would also like to understand what went wrong.
\n\nFor the sake of transparecy and community trust, I request that a clear explanation is given regarding which particular rule was broken.
", "rawMarkdown": "Julia - thanks for your input regarding the need to follow the rules. I think all teams appreciate this point and actually did their best to adhere to the rules. \n\nIn their post, \"All Faces are Real\" team has made it crystal clear how they took great care to follow all competition rules. To me, their explanation is 100% convincing - I can't see how any rule was broken. I am happy to be proven wrong, but I would like to understand. It seems many Kaggles are sharing my view and would also like to understand what went wrong. \n\nFor the sake of transparecy and community trust, I request that a clear explanation is given regarding which particular rule was broken. \n\n", "votes": 11 }, { "id": 883849, "postDate": "2020-06-13T01:49:24.853Z", "content": "Hi Julia - in my view if a reasonable person read the rules about \"Winning Submission Documentation\", they would not immediately think that this referred to external data. In fact, the rules make a point of differentiating \"documentation\" and \"code\", so it is not a stretch that external data is also not documentation. So I respectfully reject the notion that reading the rules more carefully would have prevented such a misunderstanding.
\n\nIt would genuinely surprise me if a single competitor interpreted it this way when they read the clause (maybe I am wrong!)
", "rawMarkdown": "Hi Julia - in my view if a reasonable person read the rules about \"Winning Submission Documentation\", they would not immediately think that this referred to external data. In fact, the rules make a point of differentiating \"documentation\" and \"code\", so it is not a stretch that external data is also not documentation. So I respectfully reject the notion that reading the rules more carefully would have prevented such a misunderstanding.\n\nIt would genuinely surprise me if a single competitor interpreted it this way when they read the clause (maybe I am wrong!)", "votes": 28 }, { "id": 884127, "postDate": "2020-06-13T07:47:35.127Z", "content": "This is really bad. Why bury it in \"Winning Submission Documentation\"? The context makes no sense. It should have been in section B7.
\n\nThe reputational damage to Kaggle here is clear. I think there is an opportunity here for Kaggle to clear up the documentation on external datasets. It seems that the same (unanswered) questions appear in most competitions, yet there is no one place to clarify the differences between things like licences etc.
", "rawMarkdown": "This is really bad. Why bury it in \"Winning Submission Documentation\"? The context makes no sense. It should have been in section B7.\n\nThe reputational damage to Kaggle here is clear. I think there is an opportunity here for Kaggle to clear up the documentation on external datasets. It seems that the same (unanswered) questions appear in most competitions, yet there is no one place to clarify the differences between things like licences etc. ", "votes": 12 }, { "id": 885710, "postDate": "2020-06-14T12:13:10.517Z", "content": "\n\n\nWe have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.
\n
It's been two days since I've asked my one simple question about the rules, and still have not gotten the answer. I doubt that I ever will. Which is why I and many others still have very strong doubts that anything will change on Kaggle when it comes to \"prioritizing communication and [explaining] rules clearly.\"
", "rawMarkdown": "> We have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.\n\nIt's been two days since I've asked my one simple question about the rules, and still have not gotten the answer. I doubt that I ever will. Which is why I and many others still have very strong doubts that anything will change on Kaggle when it comes to \"prioritizing communication and [explaining] rules clearly.\"", "votes": 9 }, { "id": 886606, "postDate": "2020-06-15T06:44:44.900Z", "content": "I believe this important thread is buried deep down when it deserves the highest attention.
\n\nWe as a community deserve answers to questions;
\n\n@tunguz \n> do pre-trained models that use human images require a written consent from those humans before they are qualified for prizes in this competition?
\n\n@aakashnain \n> So, every time we use ImageNet, should we trace and provide credits for an individual sample even if the dataset is public?
\n\nWe understand @kaggle in recent times is not able to cope up with leaks, rules, and external data set clarifications. Every competition now a days has a new story, a spoiled LB or infamous leak.
\n\nBut, if you messed it up. You messed it up. Make for it.
\n\nI understand talking on above may question whole point of the competitions and lot of other shuffles in leaderboard. Well, if no answers then @titericz team deserves an official apology. This platform is all about community and competitive spirit. Please don't make me feel other wise.
", "rawMarkdown": "I believe this **important** thread is buried deep down when it deserves the highest attention. \n\nWe as a community deserve answers to questions;\n\n @tunguz \n> do pre-trained models that use human images require a written consent from those humans before they are qualified for prizes in this competition?\n\n@aakashnain \n> So, every time we use ImageNet, should we trace and provide credits for an individual sample even if the dataset is public?\n\nWe understand @kaggle in recent times is not able to cope up with leaks, rules, and external data set clarifications. Every competition now a days has a new story, a spoiled LB or infamous leak. \n\nBut, if you messed it up. You messed it up. Make for it. \n\nI understand talking on above may question whole point of the competitions and lot of other shuffles in leaderboard. Well, if no answers then @titericz team deserves an official apology. This platform is all about community and competitive spirit. Please don't make me feel other wise.", "votes": 5 }, { "id": 887090, "postDate": "2020-06-15T13:25:00.667Z", "content": "Thanks and I really appreciate all the support here. Also I would like to hear all the answers.
", "rawMarkdown": "Thanks and I really appreciate all the support here. Also I would like to hear all the answers.", "votes": 7 }, { "id": 952261, "postDate": "2020-07-30T18:58:25.453Z", "content": "\n\n\n@juliaelliott - For the time and effort every user invests in contributions on Kaggle, we owe you the commitment to minimizing barriers to achieving what we’re actually here for: doing machine learning work and fostering a community for sharing that work. We have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.
\n
Our team spent a considerable amount of time on this competition and realized that we were wasting our time as each LB score made no sense. Kaggle failed the community with this competition on many levels and I for one am much more reluctant to trust Kaggle as a company to honor your above statement, because you have not honored the time and effort from your users here.
", "rawMarkdown": "> @juliaelliott - For the time and effort every user invests in contributions on Kaggle, we owe you the commitment to minimizing barriers to achieving what we’re actually here for: doing machine learning work and fostering a community for sharing that work. We have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.\n\nOur team spent a considerable amount of time on this competition and realized that we were wasting our time as each LB score made no sense. Kaggle failed the community with this competition on many levels and I for one am much more reluctant to trust Kaggle as a company to honor your above statement, because you have not honored the time and effort from your users here.", "votes": 1 } ] }, { "id": 886350, "postDate": "2020-06-15T00:56:33.580Z", "content": "I have a very genuine concern. Hope to be clarified.
\n\nAs shown in 1 comment below, the 2nd removed team used FaceForensics (FF) dataset. This dataset consists of individual youtube videos with given links. FF is the best deepfake dataset we can find online. Its quality and quantity of videos are much better than any deepfake dataset we can find. It alone can significantly boost score.
\n\nDuring the course of competition, squeezing improvement was extremely hard, except using external data. Maybe some teams will find ways to use external data, and some teams determined not to do that. It seems first 5 teams (after removal) don't use any external data at all?
\n\nSo, basically maybe 1st removed team also did this (use FF) by mentioning “youtube videos”? And FF is disallowed clearly from the beginning. I’m curious what “youtube videos” are? Please correct me if I’m wrong.
", "rawMarkdown": "I have a very genuine concern. Hope to be clarified.\n\nAs shown in 1 comment below, the 2nd removed team used FaceForensics (FF) dataset. This dataset consists of individual youtube videos with given links. FF is the best deepfake dataset we can find online. Its quality and quantity of videos are much better than any deepfake dataset we can find. It alone can significantly boost score.\n\nDuring the course of competition, squeezing improvement was extremely hard, except using external data. Maybe some teams will find ways to use external data, and some teams determined not to do that. It seems first 5 teams (after removal) don't use any external data at all?\n\nSo, basically maybe 1st removed team also did this (use FF) by mentioning “youtube videos”? And FF is disallowed clearly from the beginning. I’m curious what “youtube videos” are? Please correct me if I’m wrong. ", "votes": -6, "replies": [ { "id": 886974, "postDate": "2020-06-15T12:05:09.207Z", "content": "\n\n\nbasically maybe 1st removed team also did this (use FF) by mentioning “youtube videos”?
\n
No, we did not use FF or FF++
", "rawMarkdown": "> basically maybe 1st removed team also did this (use FF) by mentioning “youtube videos”?\n\nNo, we did not use FF or FF++", "votes": 8 }, { "id": 886987, "postDate": "2020-06-15T12:17:28.557Z", "content": "What are \"youtube videos\", and how many videos you collect so you can make 0.02 difference? I'm really curious and how do you collect them and make fake versions of them, can you elaborate? I just want to learn the method to do that. We can basically do the same thing with all links provided by FF or FF++ that way (and using the tools that FF used to make fake versions for each video, or in other words, we just can simply state that \"use FF = use youtube + relevant tools\".
\n\nPlease be clear about \"youtube videos\". That's really unclear what they are.
\n\nOne strange thing is that if you collect videos on youtube, make fake versions of each, that would be a really painful process, why didn't you mention in the solution about this effort?
\n\nFor those not in this competition, original Facebook/Kaggle data consists of about hundred of thousands of videos, each 10 seconds.
", "rawMarkdown": "What are \"youtube videos\", and how many videos you collect so you can make 0.02 difference? I'm really curious and how do you collect them and make fake versions of them, can you elaborate? I just want to learn the method to do that. We can basically do the same thing with all links provided by FF or FF++ that way (and using the tools that FF used to make fake versions for each video, or in other words, we just can simply state that \"use FF = use youtube + relevant tools\".\n\nPlease be clear about \"youtube videos\". That's really unclear what they are.\n\nOne strange thing is that if you collect videos on youtube, make fake versions of each, that would be a really painful process, why didn't you mention in the solution about this effort? \n\nFor those not in this competition, original Facebook/Kaggle data consists of about hundred of thousands of videos, each 10 seconds.", "votes": 4 }, { "id": 887015, "postDate": "2020-06-15T12:33:30.220Z", "content": "We extracted only 16 Deepfake videos from Youtube with Creative Commons License and I'm sure it made no difference in our scores because the videos from Youtube are made using a different deepfake algorithm and also the competition dataset have 100k+ videos.
\n\nUnfortunately late submission are not enabled to test some hypothesis, but Its more probable that the 0.02 difference comes from better generalization in our models due Flickrface dataset or by the fact that we blended 3 additional models in out best solution.
", "rawMarkdown": "We extracted only 16 Deepfake videos from Youtube with Creative Commons License and I'm sure it made no difference in our scores because the videos from Youtube are made using a different deepfake algorithm and also the competition dataset have 100k+ videos. \n\nUnfortunately late submission are not enabled to test some hypothesis, but Its more probable that the 0.02 difference comes from better generalization in our models due Flickrface dataset or by the fact that we blended 3 additional models in out best solution.", "votes": 12 }, { "id": 887036, "postDate": "2020-06-15T12:46:24.343Z", "content": "I’m glad if I’m wrong. I hope so. And I believe you. \nA strange thing is that using flickrface gaining 0.02 is something large. But I did not use this dataset so I can’t comment more.... it’s an image dataset though (not video)
", "rawMarkdown": "I’m glad if I’m wrong. I hope so. And I believe you. \nA strange thing is that using flickrface gaining 0.02 is something large. But I did not use this dataset so I can’t comment more.... it’s an image dataset though (not video)", "votes": 1 }, { "id": 887066, "postDate": "2020-06-15T13:13:51.743Z", "content": "Well, we made most of our effort on the 1st plan.\nThe 2nd plan is something like illegitimate child... We have to say that we didn't do our best on it.
", "rawMarkdown": "Well, we made most of our effort on the 1st plan.\nThe 2nd plan is something like illegitimate child... We have to say that we didn't do our best on it.\n", "votes": 6 }, { "id": 887076, "postDate": "2020-06-15T13:16:45.497Z", "content": "0.02 is not too much, but Flickrface have faces different than the ones that appears in the competition dataset, so it may help to generalizes to new faces. Also I told that are 3 more models in the blend of the best solution, the difference could be coming from that models diversity.
", "rawMarkdown": "0.02 is not too much, but Flickrface have faces different than the ones that appears in the competition dataset, so it may help to generalizes to new faces. Also I told that are 3 more models in the blend of the best solution, the difference could be coming from that models diversity.", "votes": 7 }, { "id": 887077, "postDate": "2020-06-15T13:17:48.817Z", "content": "Unfortunately we can't experiment on LB to prove any point.
", "rawMarkdown": "Unfortunately we can't experiment on LB to prove any point. ", "votes": 7 }, { "id": 887134, "postDate": "2020-06-15T13:52:52.773Z", "content": "That sucks. What if prize winners don’t use any external dataset but still cannot reproduce results because they used 20 models that were trained differently (they forget how to train 20 those models) and they cannot experiment on LB to check? I wonder how can the host verify and how top teams reproduce, I think that’s not simple.
", "rawMarkdown": "That sucks. What if prize winners don’t use any external dataset but still cannot reproduce results because they used 20 models that were trained differently (they forget how to train 20 those models) and they cannot experiment on LB to check? I wonder how can the host verify and how top teams reproduce, I think that’s not simple. ", "votes": 2 } ] }, { "id": 910325, "postDate": "2020-07-01T05:25:54.920Z", "content": "That's a cool topic!
", "rawMarkdown": "That's a cool topic!", "votes": -9 }, { "id": 887444, "postDate": "2020-06-15T17:15:27.487Z", "content": "That's too bad. Don't worry, I'm sure there'll be another chance to showcase your skills. Please keep the torch in your heart alight.
", "rawMarkdown": "That's too bad. Don't worry, I'm sure there'll be another chance to showcase your skills. Please keep the torch in your heart alight.", "votes": -9, "replies": [ { "id": 888012, "postDate": "2020-06-16T04:18:17.987Z", "content": "haha you're really funny
", "rawMarkdown": "haha you're really funny", "votes": 3 } ] }, { "id": 885863, "postDate": "2020-06-14T14:26:30.130Z", "content": "great!
", "rawMarkdown": "great!", "votes": -9 }, { "id": 889471, "postDate": "2020-06-17T01:11:29.340Z", "content": "great job!
", "rawMarkdown": "great job!", "votes": -23, "replies": [ { "id": 890001, "postDate": "2020-06-17T08:48:24.527Z", "content": "\"Great job\"
\n\nFor god's sake, read the article.
", "rawMarkdown": "\"Great job\"\n\nFor god's sake, read the article.", "votes": 2 }, { "id": 890433, "postDate": "2020-06-17T14:05:53.677Z", "content": "even though this is not really funny, i exhaled some air out of my nose quickly
", "rawMarkdown": "even though this is not really funny, i exhaled some air out of my nose quickly", "votes": 2 } ] }, { "id": 894041, "postDate": "2020-06-20T06:03:16.507Z", "content": "will love to connect with more people here
", "rawMarkdown": "will love to connect with more people here ", "votes": -25, "replies": [ { "id": 894459, "postDate": "2020-06-20T12:55:18.823Z", "content": "More advanced bot. Later we will have a smart bot that can earn upvotes ;)
", "rawMarkdown": "More advanced bot. Later we will have a smart bot that can earn upvotes ;)", "votes": 8 }, { "id": 894468, "postDate": "2020-06-20T13:01:29.020Z", "content": "Much advanced case: The bots will compete in such competitions and if removed by the \"mistake\" of a unreliable host team, they won't get hurt!
", "rawMarkdown": "Much advanced case: The bots will compete in such competitions and if removed by the \"mistake\" of a unreliable host team, they won't get hurt!", "votes": 4 } ] }, { "id": 905258, "postDate": "2020-06-28T11:44:28.363Z", "content": "Yes Exactly
", "rawMarkdown": "Yes Exactly", "votes": -13 }, { "id": 902553, "postDate": "2020-06-26T08:11:03.640Z", "content": "Good transparency
", "rawMarkdown": "Good transparency", "votes": -16 }, { "id": 901794, "postDate": "2020-06-25T17:36:31.013Z", "content": "dropped comment just to complete kaggle task.
\n\nbut this one is really cool though. i'm gonna give it a try.
", "rawMarkdown": "dropped comment just to complete kaggle task.\n\nbut this one is really cool though. i'm gonna give it a try.", "votes": -14 }, { "id": 892093, "postDate": "2020-06-18T16:44:39.263Z", "content": "wow! This is great 👍
", "rawMarkdown": "wow! This is great 👍 ", "votes": -23 }, { "id": 886247, "postDate": "2020-06-14T20:35:35.130Z", "content": "Nice to be work on. I will give it a try :)
", "rawMarkdown": "Nice to be work on. I will give it a try :)", "votes": -11 }, { "id": 884016, "postDate": "2020-06-13T06:52:36.070Z", "content": "great
", "rawMarkdown": "great", "votes": -18, "replies": [ { "id": 885014, "postDate": "2020-06-13T19:51:19.660Z", "content": "@kaggleteam \nWhy the honorable kaggle team doesn't take steps on such suspicious profiles?
", "rawMarkdown": "@kaggleteam \nWhy the honorable kaggle team doesn't take steps on such suspicious profiles? ", "votes": 3 } ] }, { "id": 2150658, "postDate": "2023-02-19T13:38:29.863Z", "content": "pandas or feature engine method,which is generally more prefered for endTial Imputer,For both numerical and categirical data,i fyou can explain,
\n if NA dominant then i)feature engine for numerical data. .aand ii) if,miissing data for arbitrary imputation for categorical data,but wether to choose pandas method or feature engine method or base don largeness of dataset and features KnnImputer is directly applied.?,kindly help me with these..!!!
ohh,Deepfake
", "rawMarkdown": "ohh,Deepfake" }, { "id": 918705, "postDate": "2020-07-07T12:16:06.720Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 918577, "postDate": "2020-07-07T10:32:36.420Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 918354, "postDate": "2020-07-07T07:48:12.443Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 918212, "postDate": "2020-07-07T04:48:25.133Z", "rawMarkdown": "", "votes": -3, "isDeleted": true }, { "id": 918090, "postDate": "2020-07-07T01:25:09.060Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 917734, "postDate": "2020-07-06T18:13:37.507Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 917534, "postDate": "2020-07-06T15:41:28.657Z", "rawMarkdown": "", "votes": -3, "isDeleted": true }, { "id": 917349, "postDate": "2020-07-06T12:33:18.777Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 917328, "postDate": "2020-07-06T12:18:16.277Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 917240, "postDate": "2020-07-06T11:05:54.720Z", "rawMarkdown": "", "votes": -6, "isDeleted": true }, { "id": 916648, "postDate": "2020-07-05T21:57:30.003Z", "rawMarkdown": "", "isDeleted": true }, { "id": 916154, "postDate": "2020-07-05T12:24:15.603Z", "rawMarkdown": "", "votes": -8, "isDeleted": true }, { "id": 916143, "postDate": "2020-07-05T12:09:59.647Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 916097, "postDate": "2020-07-05T11:02:58.887Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 915591, "postDate": "2020-07-04T22:55:11.147Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 915440, "postDate": "2020-07-04T18:24:10.547Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 915030, "postDate": "2020-07-04T12:32:22.167Z", "rawMarkdown": "", "votes": -8, "isDeleted": true }, { "id": 914939, "postDate": "2020-07-04T11:20:07.767Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 914349, "postDate": "2020-07-03T19:28:54.027Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 914055, "postDate": "2020-07-03T15:43:54.153Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 913991, "postDate": "2020-07-03T15:06:10.267Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 913569, "postDate": "2020-07-03T09:31:56.690Z", "rawMarkdown": "", "votes": -10, "isDeleted": true }, { "id": 913565, "postDate": "2020-07-03T09:29:23.193Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 913524, "postDate": "2020-07-03T09:03:10.600Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 913265, "postDate": "2020-07-03T05:02:55.847Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 913201, "postDate": "2020-07-03T04:00:27.270Z", "rawMarkdown": "", "votes": -10, "isDeleted": true }, { "id": 912813, "postDate": "2020-07-02T18:40:54.720Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 912557, "postDate": "2020-07-02T15:07:58.127Z", "rawMarkdown": "", "votes": -12, "isDeleted": true }, { "id": 912509, "postDate": "2020-07-02T14:29:02.183Z", "rawMarkdown": "", "votes": -12, "isDeleted": true }, { "id": 911397, "postDate": "2020-07-01T17:46:54.100Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 911060, "postDate": "2020-07-01T14:40:46.477Z", "rawMarkdown": "", "isDeleted": true }, { "id": 910915, "postDate": "2020-07-01T12:57:42.327Z", "rawMarkdown": "", "votes": -10, "isDeleted": true }, { "id": 910768, "postDate": "2020-07-01T11:06:17.140Z", "rawMarkdown": "", "votes": -10, "isDeleted": true }, { "id": 909937, "postDate": "2020-07-01T00:11:35.090Z", "rawMarkdown": "", "isDeleted": true }, { "id": 909880, "postDate": "2020-06-30T22:17:44.213Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 909634, "postDate": "2020-06-30T17:49:38.087Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 908189, "postDate": "2020-06-30T12:40:46.613Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 907943, "postDate": "2020-06-30T09:14:57.913Z", "rawMarkdown": "", "votes": -13, "isDeleted": true }, { "id": 907821, "postDate": "2020-06-30T07:19:39.560Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 907758, "postDate": "2020-06-30T06:38:34.280Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 907751, "postDate": "2020-06-30T06:30:59.610Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 907601, "postDate": "2020-06-30T03:51:59.917Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 907089, "postDate": "2020-06-29T17:19:37.397Z", "rawMarkdown": "", "votes": -10, "isDeleted": true }, { "id": 906874, "postDate": "2020-06-29T15:24:01.873Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 906707, "postDate": "2020-06-29T13:58:57.830Z", "rawMarkdown": "", "votes": -10, "isDeleted": true }, { "id": 906598, "postDate": "2020-06-29T12:43:40.340Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 906236, "postDate": "2020-06-29T06:31:34.380Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 905793, "postDate": "2020-06-28T19:48:48.807Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 905139, "postDate": "2020-06-28T09:35:05.183Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 904338, "postDate": "2020-06-27T14:50:04.563Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 904057, "postDate": "2020-06-27T10:20:32.390Z", "rawMarkdown": "", "votes": -9, "isDeleted": true }, { "id": 903851, "postDate": "2020-06-27T06:50:40.793Z", "rawMarkdown": "", "isDeleted": true }, { "id": 903849, "postDate": "2020-06-27T06:49:16.167Z", "rawMarkdown": "", "isDeleted": true }, { "id": 903482, "postDate": "2020-06-26T21:35:27.487Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 903436, "postDate": "2020-06-26T20:29:28.133Z", "rawMarkdown": "", "votes": -11, "isDeleted": true }, { "id": 903294, "postDate": "2020-06-26T17:56:40.757Z", "rawMarkdown": "", "votes": -13, "isDeleted": true }, { "id": 902953, "postDate": "2020-06-26T13:37:32.013Z", "rawMarkdown": "", "votes": -16, "isDeleted": true }, { "id": 902802, "postDate": "2020-06-26T11:21:08.837Z", "rawMarkdown": "", "votes": -15, "isDeleted": true }, { "id": 902799, "postDate": "2020-06-26T11:18:41.527Z", "rawMarkdown": "", "votes": -14, "isDeleted": true }, { "id": 902783, "postDate": "2020-06-26T11:03:48.820Z", "rawMarkdown": "", "votes": -3, "isDeleted": true }, { "id": 902777, "postDate": "2020-06-26T11:01:00.080Z", "rawMarkdown": "", "votes": -14, "isDeleted": true }, { "id": 899431, "postDate": "2020-06-24T08:34:16.630Z", "rawMarkdown": "", "votes": -12, "isDeleted": true }, { "id": 898842, "postDate": "2020-06-23T19:36:51.787Z", "rawMarkdown": "", "votes": -12, "isDeleted": true }, { "id": 898416, "postDate": "2020-06-23T14:04:31.537Z", "rawMarkdown": "", "votes": -13, "isDeleted": true }, { "id": 898182, "postDate": "2020-06-23T11:09:31.333Z", "rawMarkdown": "", "votes": -15, "isDeleted": true }, { "id": 898088, "postDate": "2020-06-23T09:36:55.867Z", "rawMarkdown": "", "isDeleted": true }, { "id": 897770, "postDate": "2020-06-23T04:53:07.567Z", "rawMarkdown": "", "votes": -15, "isDeleted": true }, { "id": 897157, "postDate": "2020-06-22T16:53:45.393Z", "rawMarkdown": "", "votes": -17, "isDeleted": true }, { "id": 897031, "postDate": "2020-06-22T15:27:10.493Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 897024, "postDate": "2020-06-22T15:22:21.697Z", "rawMarkdown": "", "votes": -16, "isDeleted": true }, { "id": 896868, "postDate": "2020-06-22T13:42:33.403Z", "rawMarkdown": "", "votes": -15, "isDeleted": true }, { "id": 896851, "postDate": "2020-06-22T13:28:53.937Z", "rawMarkdown": "", "votes": -17, "isDeleted": true }, { "id": 896445, "postDate": "2020-06-22T07:22:35.663Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 896305, "postDate": "2020-06-22T04:17:05.177Z", "rawMarkdown": "", "votes": -19, "isDeleted": true }, { "id": 896005, "postDate": "2020-06-21T18:40:04.633Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 895699, "postDate": "2020-06-21T14:48:47.540Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 895455, "postDate": "2020-06-21T11:03:07.603Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 895411, "postDate": "2020-06-21T10:22:18.497Z", "rawMarkdown": "", "votes": -20, "isDeleted": true }, { "id": 895367, "postDate": "2020-06-21T09:37:29.297Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 895014, "postDate": "2020-06-21T03:19:55.453Z", "rawMarkdown": "", "votes": -20, "isDeleted": true }, { "id": 894634, "postDate": "2020-06-20T15:55:27.123Z", "rawMarkdown": "", "votes": -21, "isDeleted": true }, { "id": 894509, "postDate": "2020-06-20T13:43:30.093Z", "rawMarkdown": "", "votes": -20, "isDeleted": true }, { "id": 894447, "postDate": "2020-06-20T12:43:29.660Z", "rawMarkdown": "", "votes": -22, "isDeleted": true }, { "id": 892558, "postDate": "2020-06-19T02:15:49.593Z", "rawMarkdown": "", "isDeleted": true }, { "id": 892284, "postDate": "2020-06-18T19:14:25.050Z", "rawMarkdown": "", "votes": -21, "isDeleted": true }, { "id": 892209, "postDate": "2020-06-18T18:00:07.420Z", "rawMarkdown": "", "votes": -22, "isDeleted": true }, { "id": 891936, "postDate": "2020-06-18T14:45:56.033Z", "rawMarkdown": "", "votes": -21, "isDeleted": true }, { "id": 891827, "postDate": "2020-06-18T13:22:30.227Z", "rawMarkdown": "", "votes": -21, "isDeleted": true }, { "id": 891589, "postDate": "2020-06-18T09:30:27.917Z", "rawMarkdown": "", "votes": -19, "isDeleted": true }, { "id": 891523, "postDate": "2020-06-18T08:28:12.207Z", "rawMarkdown": "", "isDeleted": true }, { "id": 891518, "postDate": "2020-06-18T08:23:18.387Z", "rawMarkdown": "", "votes": -13, "isDeleted": true }, { "id": 889877, "postDate": "2020-06-17T07:40:09.060Z", "rawMarkdown": "", "votes": -23, "isDeleted": true }, { "id": 889850, "postDate": "2020-06-17T07:26:50.440Z", "rawMarkdown": "", "votes": -24, "isDeleted": true }, { "id": 889516, "postDate": "2020-06-17T01:46:31.980Z", "rawMarkdown": "", "votes": -28, "isDeleted": true, "replies": [ { "id": 889775, "postDate": "2020-06-17T06:13:36.220Z", "rawMarkdown": "", "votes": -21, "isDeleted": true }, { "id": 889780, "postDate": "2020-06-17T06:23:11.550Z", "content": "nice!
", "rawMarkdown": "nice!", "votes": -1 }, { "id": 889791, "postDate": "2020-06-17T06:35:14.077Z", "content": "What's up with the bots Facebook?
", "rawMarkdown": "What's up with the bots Facebook?", "votes": 5 }, { "id": 890025, "postDate": "2020-06-17T09:06:51.823Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 890032, "postDate": "2020-06-17T09:12:20.480Z", "rawMarkdown": "", "votes": -18, "isDeleted": true }, { "id": 891673, "postDate": "2020-06-18T11:23:38.690Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 914357, "postDate": "2020-07-03T19:36:53.313Z", "content": "I guess someone is building a \"fake\" comments dataset for the next competition. 😬
", "rawMarkdown": "I guess someone is building a \"fake\" comments dataset for the next competition. 😬 " } ] }, { "id": 888984, "postDate": "2020-06-16T17:38:54.200Z", "rawMarkdown": "", "votes": -23, "isDeleted": true }, { "id": 888855, "postDate": "2020-06-16T15:44:47.440Z", "rawMarkdown": "", "votes": -20, "isDeleted": true }, { "id": 888441, "postDate": "2020-06-16T11:05:25.853Z", "rawMarkdown": "", "votes": -25, "isDeleted": true, "replies": [ { "id": 888525, "postDate": "2020-06-16T12:22:01.070Z", "content": "After these comments facebook will prepare statistics about \"opinions were separated\"? :D
\n\nKaggle Team, please, run simple competition about creating models with finding these fraud users that abuse commentary and notebooks for getting medals.
\n\nIt is the case of the shoemaker's kids having no shoes.
", "rawMarkdown": "After these comments facebook will prepare statistics about \"opinions were separated\"? :D\n\nKaggle Team, please, run simple competition about creating models with finding these fraud users that abuse commentary and notebooks for getting medals.\n\nIt is the case of the shoemaker's kids having no shoes.", "votes": 16 } ] }, { "id": 888411, "postDate": "2020-06-16T10:29:01.050Z", "rawMarkdown": "", "votes": -26, "isDeleted": true }, { "id": 888223, "postDate": "2020-06-16T07:48:02.853Z", "rawMarkdown": "", "votes": -22, "isDeleted": true }, { "id": 885038, "postDate": "2020-06-13T20:15:51.857Z", "rawMarkdown": "", "isDeleted": true }, { "id": 884812, "postDate": "2020-06-13T16:34:34.737Z", "rawMarkdown": "", "votes": 4, "isDeleted": true }, { "id": 884496, "postDate": "2020-06-13T12:20:42.467Z", "rawMarkdown": "", "votes": -32, "isDeleted": true }, { "id": 884440, "postDate": "2020-06-13T11:30:54.637Z", "content": "It's a pity, it's not easy to be the first, must consuming lots of energy. But still marvellous! I respect you!!👍
", "rawMarkdown": "It's a pity, it's not easy to be the first, must consuming lots of energy. But still marvellous! I respect you!!👍 ", "votes": 6, "isDeleted": true }, { "id": 884421, "postDate": "2020-06-13T11:11:18.790Z", "content": "thank you for share!
", "rawMarkdown": "thank you for share!", "votes": 1 }, { "id": 916268, "postDate": "2020-07-05T14:20:32.973Z", "content": "Great ! Thanks for sharing 👌
", "rawMarkdown": "Great ! Thanks for sharing 👌 ", "votes": -10 }, { "id": 900763, "postDate": "2020-06-25T03:44:33.783Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": -15 }, { "id": 884659, "postDate": "2020-06-13T14:24:54.950Z", "content": "Thanks for sharing !! Nice work!!!
", "rawMarkdown": "Thanks for sharing !! Nice work!!!", "votes": -4 } ], "comments": [ { "id": 884238, "author_name": "NAIN", "author_url": "", "post_date": "2020-06-13T08:56:14.047000", "content": "In the last four years, I have never seen such absurdity on Kaggle. As a Computer Vision guy, I am furious right now. If we go by the logic provided by Facebook for removing @titericz and team, then I have a bunch of points to make:
\n\nImageNet is a public dataset but nowhere we credit or trace the source of the images present in ImageNet before using it. The same goes for COCO, CIFAR, etc. As pointed out by @rwightman the situation is more complex if we add face detection/recognition datasets to the list. So, every time we use ImageNet, should we trace and provide credits for an individual sample even if the dataset is public?
Why stop at datasets? BatchNormalizattion, Dropout, CNNs, etc are patented by Google. Any use of BN or dropout indirectly involves that patent, so why allow that in the first place?
What are the expectations of the host here? Are they expecting us to be a lawyer first to understand such a ridiculous clause buried deep down somewhere and not clarified until the end of the competition?
Let's say we are naive and the claims of Facebook stand correct. So my question for FAIR: Is every researcher who works at FAIR, aware of this clause? If yes, why is it okay for the FAIR team to use these datasets without the consent of the source?
The situation would be worse if it comes to pretrained weights. Any architecture/dataset can be public but that doesn't mean you can directly use the weights without the consent of the person who trained the network. So, where exactly is the borderline?
To the All Faces Are Real team: I am very sorry that it happened to you. I am pretty sure that every sensible Kaggler is standing with you on this issue.
Agree.
\n\nI am starting to think competitions organisers at Facebook couldn't digest \"YouTube videos dataset\" and did the whole knit picking to void the team.
\n\nI wonder what if the team would have used \"Facebook Videos dataset\", that would make some real title in papers for facebook;
\n\n> FacebookAI have tackled DeepFake problem with FB videos platform
\n\nDissapointed more with fact that, Kaggle was complicit. What more could be worse in 2020?
", "votes": 15, "replies": [] }, { "id": 884595, "author_name": "NAIN", "author_url": "", "post_date": "2020-06-13T13:27:29.210000", "content": "I am very much sold on this fact this is more like \"Facebook + Amazon\" vs \"Google\" thing.
", "votes": 6, "replies": [] } ] }, { "id": 885456, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-06-14T08:21:11.200000", "content": "I spent most of Saturday just catching up with friends and reading all the comments via social media and here. Thank you all for your support.
\n\nFor most of last two months, we had spent much time \"behind the scene\" to coordinate with teammates, liaised with kaggle and host team, working with our legal representative to ensure we stay legally informed. Personally speaking, it was a period of significant stress. In a sense, it has been really good to finally share our side of the story with the community, and I can't overstate how heartening it is to see all the supportive messages.
\n\nThank you, everyone
", "votes": 45, "replies": [ { "id": 885504, "author_name": "NAIN", "author_url": "", "post_date": "2020-06-14T09:17:06.493000", "content": "We are with you on this and we are ready to fight till the end. The community have learned a lot from you guys. If we don't support you on this where a company wants to get away with its wrong decision, then we don't deserve to be a part of the community.
", "votes": 16, "replies": [] } ] }, { "id": 884242, "author_name": "Psi", "author_url": "", "post_date": "2020-06-13T09:00:48.243000", "content": "So many heartbreaking and disappointing discussions after competitions could be avoided if admins more actively reply to concerns of Kagglers during competitions.
\n\nCompetitors are usually really clever, and more frequently than not point out vague aspects of rules early on, specifically with respect to external data. Frequently, potential issues about external data (is it allowed, or not) are raised early on, but are either only answered very late in the competition, or stay completely unanswered. This appears to have been an issue also here in this competition.
\n\nIt seems to me that the strategy is to not reply early enough, see what happens, and then make rulings afterwards. But this harms the integrity and trust of the platform, and demotivates Kagglers heavily.
\n\nI have talked with several others lately, and everyone has the same feeling, that they just don't know what is allowed and what is not allowed anylonger. I am the first person who always wants to follow the rules as precisely as possible, but how can I do that if I don't know them?
", "votes": 47, "replies": [ { "id": 884430, "author_name": "", "author_url": "", "post_date": "2020-06-13T11:20:56.330000", "content": "", "votes": 22, "replies": [] }, { "id": 884629, "author_name": "Carlo", "author_url": "", "post_date": "2020-06-13T14:00:55.257000", "content": "Fully agree here! Kagglers will often recognize if an external dataset is tricky to use and will ask questions about it in the external data thread. Unfortunately, these questions often go unanswered even after tagging competition hosts multiple times.
\n\nIt is understandable that the external data thread contains many comments and not all can be answered. However, if Kaggle is going to allow external data in a competition, a simple statement like \"so *dataset_X* is not allowed in this competition\" would be more integrous, then repeating vague rules that require legal knowledge to understand.
\n\nA similar situation is playing out in the Jigsaw competition where it is currently not clear what people can and cannot do with the test set (translation, cleaning, etc.). I really hope that the Kaggle hosts can answer these questions so that we can at least mitigate misunderstandings about external data.
", "votes": 16, "replies": [] }, { "id": 886720, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2020-06-15T08:30:03.283000", "content": "\n\n\nIt is understandable that the external data thread contains many comments and not all can be answered.
\n
I think ALL should be answered.
", "votes": 8, "replies": [] } ] }, { "id": 887796, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-15T22:12:06.607000", "content": "You guys are never gonna believe what just happened!
\n\nthis has to be a deepfake? wow!!!!
", "votes": 8, "replies": [] }, { "id": 887799, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-06-15T22:18:02.987000", "content": "I am sure he has given his consent 😄
", "votes": 7, "replies": [] }, { "id": 887800, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-15T22:22:59.427000", "content": "Make Kaggle Great Again!!!
", "votes": 5, "replies": [] }, { "id": 887804, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-06-15T22:29:08.723000", "content": "
😂
", "votes": 2, "replies": [] }, { "id": 888182, "author_name": "Jainam Shah", "author_url": "", "post_date": "2020-06-16T07:09:01.353000", "content": "😂
", "votes": 3, "replies": [] }, { "id": 888869, "author_name": "Serigne ", "author_url": "", "post_date": "2020-06-16T15:57:33.587000", "content": "RULES & ORDER!
", "votes": 6, "replies": [] }, { "id": 888879, "author_name": "Trigram", "author_url": "", "post_date": "2020-06-16T16:06:57.960000", "content": "We'll build a wall of legal issues and make Kagglers pay for it (no political view here)
", "votes": 8, "replies": [] }, { "id": 906130, "author_name": "Abu Noman Md. Sakib", "author_url": "", "post_date": "2020-06-29T04:45:34.170000", "content": "
For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored, under the excuse that it doesn't meet certain high-level standards that Kaggle supposedly adheres to. And now a team of honest, hard working, scrupulously principled and exceptionally talented Kagglers is punished because of some post-hoc pedantic scrupules coming from a powerful tech giant??? This hypocrisy cries to high heaven!
\n\nAs many have remarked below, it seems that Kaggle has become the victim of its own success. Juggling dual roles - a prize-awarding competition site AND a community of Data Scientists - seems to be becoming increasingly difficult. And if there is a conflict between those two roles, it is now painfully obvious on which side Kaggle will default. As far as I see it, it will never be possible to smoothly reconcile those two roles. I believe that if Kaggle truly cared about its community, then the best, and perhaps only, solution going forward would be to abandon being a prize-giving entity. Most of us here are very loosely motivated by the monetary prizes, if at all. The recent COVID competitions proved that you can get very competitive competitions without any prizes and even without medals. (At my current job I am contractually prohibited from taking any prize money anyways, and I have been more competitive than ever.) By not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work. It will free Kaggle to more forcefully stand for its community, focus fully on the community-building and promotion and advancement of Data Science.
", "votes": 41, "replies": [ { "id": 884767, "author_name": "nosound", "author_url": "", "post_date": "2020-06-13T15:52:48.520000", "content": "I have a problem following you
\n\n\n\n\nMany other notorious cheaters are still active.
\n
can you please elaborate?
\n\n\n\n\nthe best, and perhaps only, solution going forward would be to abandon being a prize-giving entity.
\n
the arguments for this I was not able to grasp from your message. The fact that it is not the main motivator for people is not an argument against. And this argument is not clear to me:
\n\n\n\n\nBy not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work.
\n
for example in DFDC case, for me the problem was Kaggle not clarifying what data could and could not be used. But if clarified, to whatever side, I think it would have been OK. So there is no problem that there is a regulation, the rules, but the problem is that it was not clear.
", "votes": 0, "replies": [] }, { "id": 884781, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-13T16:00:17.007000", "content": "\n\n\nMany other notorious cheaters are still active.
\n
Several whistleblowers have come out in the past and documented cheating behavior (web-scraping of solutions, private sharing, etc.) by a few top Kagglers. I don't want to elaborate beyond this.
\n\nAs far as DFDC is concerned, it was not the elaboration of the rules that was at stake, but their arbitrary interpretation after the competition was finished.
", "votes": 7, "replies": [] }, { "id": 884785, "author_name": "nosound", "author_url": "", "post_date": "2020-06-13T16:10:51.367000", "content": "Regarding DFDC, in the external data disclosure thread dozens of questions went unanswered. People were genuinely in the dark what could and could not be used. The story of the OP is the perfect example, even using their second submission to mitigate that exact risk of unclear regulation.
\n\nAfter the fact they just selected one of the possible interpretations. But different interpretations should not have been possible to begin with. It is still a competition, with or without prizes, the rules must be clear.
", "votes": 6, "replies": [] }, { "id": 885463, "author_name": "Erik Bruin", "author_url": "", "post_date": "2020-06-14T08:28:54.113000", "content": "Although I am sitting a bit on the sideline with regards to competitions (my activity has mostly been notebooks so far), I first want to say that I feel very sorry for team All faces are real. I am very likely repeating others but this feels totally unfair; a lot of damage with regards to honor and also prize money for a reason that feels extremely vague and maybe even random.
\n\n> For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored
\n\nI was not aware that many notorious cheaters in competitions are still active (only knew about the Bestpetting case). Don't want to change the subject, but I can say that we have the same experience in notebooks. We collected evidence on a number of high ranked cheaters (I don't want to call them high-profile as they don't deserve that word). At best Kaggle replies that they are closely following discussions, but never follow-up....
", "votes": 12, "replies": [] } ] }, { "id": 883688, "author_name": "beluga", "author_url": "", "post_date": "2020-06-12T20:35:56.443000", "content": "OMG. That is even worse nitpicking reasoning than I expected... Very disappointed. Next time we need a lawyer before feature engineering...
", "votes": 36, "replies": [ { "id": 884047, "author_name": "FGPC", "author_url": "", "post_date": "2020-06-13T07:02:45.217000", "content": "\"Next time we need a lawyer before feature engineering\".. hahaha! :D
", "votes": 3, "replies": [] }, { "id": 884345, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2020-06-13T09:56:18.570000", "content": "I would like to share a few words from my heart here.
\n\nI haven't been in kaggle for a long time, but it is precisely because of kaggle's fairness and justice, sincerity with most kagglers, and the spirit of sharing with knowledge from your guys that deeply attracted me. I have always maintained a high enthusiasm here, which has enabled me to become GrandMaster within one year.
\n\nAnd I would like to express my heartfelt thanks to every kagglers here, thank you for your support. In the past two months, for me personally, I have fallen into a very frustrated and wronged mood☹️ , even dealing with the new competition, or my personal work, the previous passion is gone. \nNow all the stories of our team are shared here, I feel relieved😃 😃 . Thanks a lot, guys.
", "votes": 33, "replies": [ { "id": 885661, "author_name": "CPMP", "author_url": "", "post_date": "2020-06-14T11:27:22.627000", "content": "I feel you pain. There is always a point where you get strongly disappointed by Kaggle because of some decision you feel are extremely unfair. For you it happens here. For me is was when a private competition (CAESARS for those who entered it) where I was at the top was reset because Kaggle staff decided that their data prep hadn't been right. I wasn't a GM yet, and that gold medal would have made me GM way earlier. I was so upset I stopped kaggling for two months.
\n\nWhen I came back I did not attach emotionally as I used to. I am probably less motivated as well, but still enough to have won more gold medals and prizes. I take care of not invest emotionally in Kaggle anymore. This is probably why my written reactions here aren't as extreme as others. It is not that I find what happened to your team acceptable. It is that I know these things happen from time to time at Kaggle.
\n\nI don't know if my story can help you heal. I hope it can to some extent.
\n\nIt would be a pity to see a great competitor like you desert Kaggle. Same for all the team.
", "votes": 21, "replies": [] }, { "id": 885703, "author_name": "Gary", "author_url": "", "post_date": "2020-06-14T12:07:41.937000", "content": "@cpmpml Thanks for your comfort. It helps me a lot😄
", "votes": 5, "replies": [] }, { "id": 885904, "author_name": "CPMP", "author_url": "", "post_date": "2020-06-14T14:54:43.830000", "content": "Happy to help a bit. Take care.
", "votes": 1, "replies": [] }, { "id": 887157, "author_name": "CPMP", "author_url": "", "post_date": "2020-06-15T14:04:08.087000", "content": "\n\n\nI take care of not invest emotionally in Kaggle anymore.
\n
It is easier to say than to do. For instance, I did get emotional in covid-19 forecasting because of what I felt was very unfair behavior. As a result I did not enter the last round. In hindsight, the stress of forecasting fatalities, as well as lockdown in France probably explains it. Anyway, getting emotional wasn't good.
", "votes": 4, "replies": [] } ] }, { "id": 885306, "author_name": "Vopani", "author_url": "", "post_date": "2020-06-14T05:02:55.543000", "content": "Right now, the private LB looks like the biggest deepfake of all.
", "votes": 33, "replies": [ { "id": 887083, "author_name": "Giba", "author_url": "", "post_date": "2020-06-15T13:21:26.420000", "content": "I can't agree more 😂
", "votes": 6, "replies": [] } ] }, { "id": 883752, "author_name": "عثمان", "author_url": "", "post_date": "2020-06-12T22:05:02.267000", "content": "\n\n\nFacebook felt some of our external data \"clearly appears to infringe third party rights\"
\n
This is news to me. FB cares about peoples rights. #TIL
", "votes": 33, "replies": [ { "id": 883759, "author_name": "JohnM", "author_url": "", "post_date": "2020-06-12T22:13:31.370000", "content": "My thoughts exactly, Authman. Boo and double boo.
\n\nAnd congratulations to the team for having the best solution.
", "votes": 19, "replies": [] }, { "id": 884342, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2020-06-13T09:46:41.807000", "content": "FB only cares when its the \"other\" party. When they do the violation themselves.
\n\n\n\n\nThey tell - \"We are trying, and will do better\"
\n
Also, this bring me lights on creativity and modelling approaches by @titericz 's team.
\n\nShouldn't approach and creativity be part of winning decision rather than some luckiest 3rd decimal spitted out by a computer. We can definetly do better!
", "votes": 5, "replies": [] } ] }, { "id": 887403, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-06-15T16:53:20.943000", "content": "Not sure how I should take this, 6:30 into @cristiancanton presentation in the competition overview during Workshop on Media Forensics, and the competition best score are shown.
\n\nHere the 0.423 log loss is not the top final LB score, and the only submission with such score, as far as I know, is our voided solution at 0.4232.
\n\nIf our solution is not acceptable, why parade our score in your presentation?
\n\n
wondering where did they grab the \"real\" Deepfake videos to score the private test 🤔
", "votes": 21, "replies": [] }, { "id": 887420, "author_name": "Trigram", "author_url": "", "post_date": "2020-06-15T17:01:27.780000", "content": "This is just morally unfair at this point - pretty sure people will notice the discrepancies.
", "votes": 6, "replies": [] }, { "id": 887423, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-06-15T17:04:01.777000", "content": "I don't want to take this out to social media, in case I have jumped to some incorrect assumption. so post here for now - but this is really adding insult to injury
", "votes": 13, "replies": [] }, { "id": 887472, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-15T17:27:44.910000", "content": "\n\n\nI don't want to take this out to social media
\n
I do. 😄
", "votes": 12, "replies": [] }, { "id": 887474, "author_name": "Xuan Cao", "author_url": "", "post_date": "2020-06-15T17:28:53.490000", "content": "Today's joke: \n\"At Facebook we value intellectual property! \"
", "votes": 16, "replies": [] }, { "id": 887482, "author_name": "Mozaic", "author_url": "", "post_date": "2020-06-15T17:37:27.943000", "content": "My apologies, a typo from my side on the exact number. I will rebuild the video with the fix. Thanks for letting me know.\n(Update: slides/video have been amended.)
", "votes": -34, "replies": [] }, { "id": 887488, "author_name": "beluga", "author_url": "", "post_date": "2020-06-15T17:38:51.290000", "content": "Sure, that will fix everything!
", "votes": 28, "replies": [] }, { "id": 887501, "author_name": "Manoj", "author_url": "", "post_date": "2020-06-15T17:43:45.987000", "content": "Thaks @gaborfodor . Just about to comment the same.\nsorry I am sarcastic too. can't help
", "votes": 6, "replies": [] }, { "id": 887982, "author_name": "Shengtao Xiao", "author_url": "", "post_date": "2020-06-16T03:41:23.997000", "content": "I just watched the video. They hope our algorithms to be robust and generalised to unseen cases. But using little external data to enhance generalization get people disqualified. I didn't see they mention anything about audio. Is fake audio a smoke bomb? 😭
", "votes": 8, "replies": [] }, { "id": 889287, "author_name": "Moshel", "author_url": "", "post_date": "2020-06-16T22:10:45.713000", "content": "This is bordering comic... If it wasn't so sad. I guess winning team should sue them for publishing their score without license? Oh wait, the whole presentation should be removed! We can't have him fix it!
", "votes": 2, "replies": [] } ] }, { "id": 886135, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-06-14T18:19:30.430000", "content": "I love kaggle community, but I think Kaggle Team doesn't appreciate contribution from anyone of us (Novice --> GM). Money award doesn't matter, we are solving any competition even if size of award is tiny, despite on almost anyone competitor (Novice --> GM) can earn this money during one month without hard-working as require competition.
\n\nWe expect only that Kaggle Team will fight for justice for everyone. But Kaggle Team doesn't care:
\n\nKaggle Team forgot that \"Business Power\" are professionals from Kaggle Community and their free hard work. Lose us = Lose quality of solutions = Lose reputation = Lose business
", "votes": 30, "replies": [ { "id": 886147, "author_name": "Innat", "author_url": "", "post_date": "2020-06-14T18:28:41.697000", "content": "I was expecting you here. Well said, brother. :)
", "votes": 2, "replies": [] }, { "id": 896536, "author_name": "Andrada", "author_url": "", "post_date": "2020-06-22T08:41:06.240000", "content": "Lose us = Lose quality of solutions = Lose reputation = Lose business
Umm.... this phrase needs to be emphasized MORE. Simply because it's TRUE.
", "votes": 4, "replies": [] }, { "id": 896545, "author_name": "Trigram", "author_url": "", "post_date": "2020-06-22T08:54:49.573000", "content": "Let's not forget \"lose money\" which I am not sure any corporation would want.
", "votes": -1, "replies": [] } ] }, { "id": 884841, "author_name": "Heads or Tails", "author_url": "", "post_date": "2020-06-13T17:03:05.347000", "content": "My impression, for what it's worth, is that most of the problematic cases that we have been seeing in the last years have been routed in a certain disconnect between the Kaggle team and the community. In a way, this is perfectly understandable, since the Kaggle team is still relatively small, and it's impossible, say, for 1 or 2 people in charge of a competition to keep up with the ingenuities and (crazy) ideas of thousands of competitors. And running (and building) a platform is a different business than using it; with different priorities and philosophies. In many cases, the disconnect is not a problem, but a normal consequence of our different roles in the community.
\n\nHowever, in cases like this (or the PetFinder scandal, or the Passenger Screening competition for those who remember it, or also the high-profile Kernel plagiarism cases); in these situations the disconnect becomes a serious problem, because it leads to a mismatch of expectations between the team (& the sponsor) vs the community. Conflicting expectations, especially when combined with a lack of communication, then lead to extremely frustrating situations like the current one; which likely could have been avoided if everyone had been on the same page from the beginning. In those cases, it becomes a weakness that Kaggle does not tap into the community hivemind when planning a competition strategy.
\n\nSo here's a suggestion: we could institute a Kaggle community advisory board, consisting of a large number (~100) of well-respected Masters and Grandmasters. (I could easily name a few dozen without even thinking; I'm sure we got the numbers.) For any given competition, the team and board would select, say, 6-10 board members who would then act as a liaison between the organisers and the competitors. They would also help to foresee problems and controversies (e.g. to avoid changes in metric or flag potential leaks). Of course, those 6-10 Kagglers wouldn't be allowed to compete in that specific competition, but with a large enough pool to draw from, this shouldn't be an issue either. Nobody joins all competitions, these days.
\n\nIn this way, there is a good chance that conversations like the one we're having now could happen before a competition; and before lots of time and effort have been spent working under \"wrong\" assumptions (which were very reasonable assumptions in this case). This scenario would create more work for the Kaggle board members, but I have the feeling that many Masters and Grandmasters would accept the responsibility (and could also learn a new thing or two by looking at a competition from a different perspective). Of course, certain Kaggle \"trade secrets\" might have to remain opaque so that the board members could continue to compete without having an unfair advantage in future competitions.
\n\nAnyway, that's my thoughts. I'd be curious to read feedback.
", "votes": 30, "replies": [ { "id": 884867, "author_name": "Vopani", "author_url": "", "post_date": "2020-06-13T17:31:05.983000", "content": "This is a very reasonable suggestion if Kaggle is interested in exploring such kind of solutions.
\n\nI can even vouch that it works if planned and structured correctly. I have been part of an international community of organizing, authoring and monitoring puzzle competitions (which are structured pretty much the same as Kaggle competitions) for over a decade, and for every competition there is a pool of community members who help in the structure, testing and evaluation of the competition before it launches (with the obvious condition that they are not allowed to compete).
\n\nIt has solved most of the issues we used to face and almost removed the gap between organizers and competitors completely. And it just works!
", "votes": 8, "replies": [] }, { "id": 884897, "author_name": "Heads or Tails", "author_url": "", "post_date": "2020-06-13T17:48:56.197000", "content": "Thanks @rohanrao! This is very valuable feedback. Can you say a bit more about the size of these pools and how the members are selected?
", "votes": 3, "replies": [] }, { "id": 884921, "author_name": "Vopani", "author_url": "", "post_date": "2020-06-13T18:21:04.713000", "content": "We have a pool of ~ 40 members. These are all actual puzzle competitors who have been ranked in the Top-100 at the World Championships (something like a group of 40 Kaggle Grandmasters). This is the only criteria and it is voluntary. There are times when some folks are active and some are not, its a bit lenient there but it still meets the current requirements.
\n\nFor every international puzzle championship (ranges from 50-100 events in a year), with 70% of them online, there will be 5-6 of these members involved (we call them testers) in each of them who work closely with the organizers of the event. Their main role is in validating the correctness of puzzles, timing of the contest, points distribution, proof-reading the instructions and so on. There is almost always changes and improvements that come from testers that end up making it a better contest (not all, since at the end it is the organizer who has the power to decide). And it doesn't really take long. It's great feedback for organizers at no cost. Sometimes the testers themselves have differing opinions but that itself is valuable because you realize something is not natural and coming to a consensus leads to a thought-through decision.
\n\nThe testers are chosen on a first-come-first-serve basis. It's pretty simple, any organizer just sends out an email and the first 6 members to respond positively are chosen. The testers are not allowed to participate in the contest. And in over 10 years, we've always found few members willing to help who do not wish to participate in the contest.
\n\nSome contests still fail or end up with a problem (even testers are human after all), but it is significantly lesser than without having testers. I'm sure the Kaggle team do their best as testers for every competition to bridge the gap between the organizer (host) and competitors, but over time I think the gap between Kaggle and competitors has increased.
", "votes": 11, "replies": [] }, { "id": 888079, "author_name": "Moshel", "author_url": "", "post_date": "2020-06-16T05:39:01.887000", "content": "I think that this won't solve many problems, unless this board will hava more direct channel to the team and the team will be willing to give PRECISE answers. I understand that the team is a bit intimidated by their accountability and responsibility when giving direct, precise and clear answer. Its much easier to not answer or just say \"carefully read the rules\". However, this is THEIR JOB. Mitigating their job to a \"board\" will not help as the problem is with the team being afraid to interpret the rules or give clear yes no answer.\nOn a slightly different tangent, it is really annoying that these questions goes in threads. An issue tracking system will be much better.
", "votes": 2, "replies": [] } ] }, { "id": 883728, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-06-12T21:30:21.670000", "content": "I'm really sorry that this happened. It looks like you did your due diligence and went out of your way to even remove data that had an inappropriate license in an explicit effort to avoid breaking the rules - yet were punished anyways.
\n\nThere were several posts in the external data thread that went unanswered regarding the use of various external data sources. As was mentioned before, if the stipulations regarding external data were so strict (as to essentially prevent any practical use of other data sources), external data (apart from pretrained models) should not have been permitted in the first place so that teams wouldn't go through all this effort just to have a winning solution disqualified.
", "votes": 30, "replies": [] }, { "id": 884310, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-06-13T09:25:18.270000", "content": "To be simplfied, Kaggle and FB didn't make the rules clear.\nThen all the losses caused by this mistake were passed on to us.
\n\nWe worked very hard on understanding the rules and finally beated by \"The organizer own the final interpretation right to the rules\"
", "votes": 27, "replies": [] }, { "id": 883887, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2020-06-13T03:28:11.890000", "content": "I do hope that Kaggle takes this opportunity to rethink the use of external datasets.
\n\nThere's been a weird dynamic in competitions where Kagglers scramble to look for useful external datasets or pretrained models, post them in the \"External data thread\", never get a clear answer about whether they're allowed, and often end up using them anyway.
\n\nIt's bad because a) it creates an arms race (especially in multimedia competitions) for scraped data which takes away from the spirit of these competitions and b) it disadvantages teams that scrupulously follow the rules over teams that cross the line in terms of the data they use.
\n\nPerhaps consider switching to a fixed whitelist of external data/models that Kaggle administrators define?
", "votes": 27, "replies": [] }, { "id": 883865, "author_name": "RossWightman", "author_url": "", "post_date": "2020-06-13T02:32:51.170000", "content": "If this hasn't been pointed out already, the decision to remove this solution but keep (most) others in the competition is extremely arbitrary. Scanning many of the solutions, all face detectors/recognition models that I'm aware of were trained with datasets that are not compliant with the rules, actually much less so than the extra datasets these competitors used.
\n\ndlib face_detection, uses VGG face (Non commercial license, no consent), scrubface (no consent), manual scrubbing (duh). mtcnn, facenet, etc all use datasets based on faces without consent (the ones mentioned already, CASIA-WebFace, LFW, etc etc). Then there is ImageNet (non-commercial) pretrained weights.
\n\nSo, this is a long standing Kaggle question. Why does a pretrained model not apply to the rules but a self trained one does? Someone deciding to license their code with a given license (say dlib is Boosts) while the dataset they used (VGG face non-commercial) falls under a different one clearly does not make those weights fall under their code license. There should really be no difference.
", "votes": 27, "replies": [ { "id": 884833, "author_name": "mjsML", "author_url": "", "post_date": "2020-06-13T16:59:22.053000", "content": "My take on this is : \"Authors Guild v. Google\" , when you have enough lawyers like us ... you can scrape, otherwise people will sue and we will DSQ you 😃
", "votes": 1, "replies": [] } ] }, { "id": 883712, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-12T21:10:07.037000", "content": "So do we now have to get a written and notarized permission from the ImageNet creators every time we use pretrained models in Kaggle solutions?
", "votes": 28, "replies": [ { "id": 883718, "author_name": "anokas", "author_url": "", "post_date": "2020-06-12T21:20:14.057000", "content": "The analogy would be that you have to get notarized permission from anyone appearing in any image that appears in ImageNet.
", "votes": 25, "replies": [] }, { "id": 883722, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-12T21:27:42.483000", "content": "That's would be insane. But not really surprising, knowing Facebook and their MO. What is really disappointing is that Kaggle allowed them to get away with it.
", "votes": 16, "replies": [] }, { "id": 883844, "author_name": "anokas", "author_url": "", "post_date": "2020-06-13T01:43:38.223000", "content": "For reference, pretrained models were specifically approved by the organisers: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121203#694466
\n\nSo in this case, there is no requirement to have consent from individuals depicted in ImageNet. I can see how this might be seen as a logical inconsistency.
", "votes": 12, "replies": [] }, { "id": 884682, "author_name": "Trigram", "author_url": "", "post_date": "2020-06-13T14:53:03.760000", "content": "Not only ImageNet - I feel that we have to also gain copyright permission(s) for the basic linear algebra too.
\n\nDear Gottfried Leibnitz, if you read this, please give us all your approval to use calculus wherever it is necessary.
", "votes": 7, "replies": [] } ] }, { "id": 885153, "author_name": "CPMP", "author_url": "", "post_date": "2020-06-14T00:40:32.053000", "content": "Giba, Mikel, Yifan, Gary and Qishen , you deserve the first place. We all see it.
\n\nI frankly don't know what else to say at this point. Many people made great comments about what to do next and I agree with many of them. But for you it won't matter much I guess. I hope you'll get over it.
\n\nAn afterthought: I would make clear to Facebook that they have no rights to reuse any part of your solution. AFAIK, only prize winners have an obligation to provide a licence for their code to the sponsor.
", "votes": 29, "replies": [ { "id": 886284, "author_name": "Giba", "author_url": "", "post_date": "2020-06-14T21:36:14.347000", "content": "Thank you @cpmpml. We really appreciate all the support from the community.
", "votes": 10, "replies": [] } ] }, { "id": 883685, "author_name": "ryches", "author_url": "", "post_date": "2020-06-12T20:34:07.763000", "content": "Thanks for giving this background and sorry it ended this way. I thought it was to do with the commercial licensing of the data. I was not even aware of the additional stipulations. In my opinion that should have been something much more prominently displayed, because that additional rule would have quickly eliminated all external data usage basically.
", "votes": 26, "replies": [] }, { "id": 887946, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-06-16T02:53:53.263000", "content": "Up front, I should clearly state that Kaggle employees are employees of Google. In the rare event that a dispute is raised on Kaggle that requires review of competition rules, our actions, statements, and conduct require us to refrain from opinions, speculation, and casual commentary. It is for this reason that you can expect a slower response in matters like this.
\n\nWe are working on a more comprehensive response and will post when that’s ready.
", "votes": 26, "replies": [ { "id": 888191, "author_name": "CPMP", "author_url": "", "post_date": "2020-06-16T07:16:02.370000", "content": "Thanks for this update. Discussions with Google legal and Facebook legal at the same time must be quite interesting. We all hope Kaggle will find a way out that is as fair as possible.
", "votes": 7, "replies": [] }, { "id": 888241, "author_name": "beluga", "author_url": "", "post_date": "2020-06-16T08:08:55.267000", "content": "Thanks Julia. I hope you will find the best possible solution soon. At the Zillow Prize Disqualification Issue kaggle found a solution after five days. I know this case is probably trickier.
\n\nBtw here is a chart that compares the heated forum threads.\n\nSource
Discussions with Google legal and Facebook legal at the same time must be quite interesting
Same thought.
", "votes": 3, "replies": [] }, { "id": 893487, "author_name": "Hector", "author_url": "", "post_date": "2020-06-19T16:42:56.343000", "content": "Sorry, Julia, but your attempt to clarify the mistake just make the mistake you did worse and worse.
", "votes": 0, "replies": [] } ] }, { "id": 884861, "author_name": "Vopani", "author_url": "", "post_date": "2020-06-13T17:21:21.913000", "content": "No matter what way I look at this and no matter how much I try to understand from a host's perspective, I cannot come to convince myself that this was correct or fair in any way. There have been questionable decisions in competitions in the past but this is just preposterous and beyond acceptable.
\n\nCongratulations to Giba, Mikel, Yifan, Gary and Qishen for their work, effort, solution and for winning this competition!
", "votes": 27, "replies": [] }, { "id": 883717, "author_name": "Mirek Bober", "author_url": "", "post_date": "2020-06-12T21:18:20.530000", "content": "Is my cat part of the documentation? It may well be if you hire the right lawyer.
\n\nThis is truly shocking – are we really entering times when to succeed in a Kaggle competition your team will need more lawyers than scientists? Perhaps just lawyers?
\n\nLet me get this straight – the world’s top technology to detect deepfakes has just been “voided” by Facebook because of their claim that the training data is the “documentation”. Really? I have looked very carefully through the Kaggle definition of documentation and could not find a single sentence stating that training data is the documentation.
\n\nSad times for science.
", "votes": 27, "replies": [] }, { "id": 892113, "author_name": "Serigne ", "author_url": "", "post_date": "2020-06-18T17:05:46.877000", "content": "I thought Kaggle and Google had developed a super duper spam detector algorithm recently ..
\n\nSo why this topic is still filled with bot users having not very subtle spams like \"Wow\", \"Great Job\", \"Nice\" etc. ?
", "votes": 24, "replies": [] }, { "id": 886040, "author_name": "raddar", "author_url": "", "post_date": "2020-06-14T17:02:08.433000", "content": "This raises a bigger issue about all competitions having personal data involved... for example ongoing melanoma challenge - my reasoning here:
\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886037
", "votes": 23, "replies": [ { "id": 886537, "author_name": "datasaurus", "author_url": "", "post_date": "2020-06-15T05:40:46.697000", "content": "There is a really bizarre line in the melanoma challenge stating that if you don't use external data, you are ineligible for a prize 🤷♂️: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886521
\n\nEdit: I think the rule is specifically around public vs private data. I guess this will need clear instructions on what is and is not public
", "votes": 14, "replies": [] }, { "id": 886750, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-06-15T08:51:57.397000", "content": "Hahaha wow... Just wow...
", "votes": 2, "replies": [] }, { "id": 887563, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-15T18:30:16.213000", "content": "\n\n\nThere is a really bizarre line in the melanoma challenge stating that if you don't use external data, you are ineligible for a prize
\n
Although Gilles may have seen this already and many may already know this too, this is the comment by the competition host regarding the use of external data with eligibility to win prizes.
", "votes": 2, "replies": [] } ] }, { "id": 884662, "author_name": "Human Analog", "author_url": "", "post_date": "2020-06-13T14:28:11.973000", "content": "This sucks. Sounds like a case of \"Let's keep the rules vague so we can disqualify submissions that we don't like for whatever reason.\"
", "votes": 24, "replies": [ { "id": 885009, "author_name": "Innat", "author_url": "", "post_date": "2020-06-13T19:45:28.673000", "content": "ah! I'm sure they followed it.
", "votes": 2, "replies": [] } ] }, { "id": 883696, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2020-06-12T20:48:00.103000", "content": "This is awful. Truly reprehensible bullying behavior. I can't believe that Facebook will again get away with bending all standards of ethics and common decency.
", "votes": 28, "replies": [] }, { "id": 885964, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-06-14T15:46:54.323000", "content": "It is frustrating that Kaggle has only contributed to this dialogue insofar as making a single post containing a vague non-response that basically amounts to \"read the fine print\" without so much as a simple apology.
\n\nThis falls squarely on their shoulders, and it is unfortunate that part of the blame is shifted onto the competitors. It is disrespectful, especially when the team consists of 3 GMs and 2 Masters (who will probably be GMs soon) that have contributed so much to the community.
", "votes": 21, "replies": [ { "id": 885976, "author_name": "beluga", "author_url": "", "post_date": "2020-06-14T16:00:50.553000", "content": "I still hope they will respond to the issue properly. In my more emphatic moments I could understand why they haven't answered yet (weekend, tough topic, still exploring solutions etc.). Altough they must know this unfortunate (frustrating/outrageous/unacceptable... pick any) incident for weeks now.
", "votes": 15, "replies": [] }, { "id": 886114, "author_name": "Heads or Tails", "author_url": "", "post_date": "2020-06-14T17:58:18.983000", "content": "I think the main reason is that it's the weekend. I'm expecting more extensive statements by the Kaggle team on Monday/Tuesday (Pacific time) when everyone is back online. I think this is understandable; and I'm certain that no disrespect is intended. It was a little unfortunate timing to make the announcement on a Friday, though.
", "votes": 5, "replies": [] }, { "id": 886978, "author_name": "CPMP", "author_url": "", "post_date": "2020-06-15T12:12:29.787000", "content": "It will depend on how fast Kaggle legal people and Facebook vet whatever they want to answer.
", "votes": 3, "replies": [] } ] }, { "id": 883873, "author_name": "Bibek", "author_url": "", "post_date": "2020-06-13T02:56:13.997000", "content": "Facebook is not a company it used to be; it does what it wants and this is just another case. Sadly one of most appreciated team of Kagglers are on the receiving end. I'm so sorry for your team ;(\nI have one suggestion to Kaggle: Please remove this from future competitions; there is no meaning to it and only makes the forums crowded \n
So not only were around 5% of the teams screwed out of the private leaderboard because their solution mysteriously didn't work and the top team on public and others lost due to misleading info on sound manipulation, BUT the top teams scores were then voided over ridiculous fine print rules and FB being picky just not liking their solution??!
\n\nTeams worked on this problem for months, incurring costs just to use the giant dataset, just to be screwed over because Kaggle didn't inform competitors. And for your team specially you were shut down even though you meticulously made sure to follow all the rules!
\n\nLike others have said Kaggle is a place to casually compete and move science forward but it seems we need lawyers to make sure we just follow the rules correctly!
\n\nAnd that is after we apparently just had to cross our fingers and hope we can even make it to the private leaderboard. The Kaggle team needs to rethink how to do these rerun solutions types of competitions for the future.
\n\nI definitely no longer hold any pride in doing decent in this comp when there are many who worked way harder and deserved something for it.
", "votes": 22, "replies": [] }, { "id": 883800, "author_name": "", "author_url": "", "post_date": "2020-06-13T00:23:05.290000", "content": "", "votes": 20, "replies": [ { "id": 883811, "author_name": "", "author_url": "", "post_date": "2020-06-13T00:47:39.967000", "content": "", "votes": 22, "replies": [] }, { "id": 883813, "author_name": "", "author_url": "", "post_date": "2020-06-13T00:51:15.237000", "content": "", "votes": 5, "replies": [] }, { "id": 883869, "author_name": "", "author_url": "", "post_date": "2020-06-13T02:51:08.603000", "content": "", "votes": 11, "replies": [] }, { "id": 883875, "author_name": "", "author_url": "", "post_date": "2020-06-13T03:00:02.793000", "content": "", "votes": 9, "replies": [] } ] }, { "id": 889309, "author_name": "", "author_url": "", "post_date": "2020-06-16T22:40:24.507000", "content": "", "votes": 22, "replies": [ { "id": 889397, "author_name": "", "author_url": "", "post_date": "2020-06-17T00:20:56.700000", "content": "", "votes": 5, "replies": [] }, { "id": 890179, "author_name": "", "author_url": "", "post_date": "2020-06-17T11:15:14.030000", "content": "", "votes": 5, "replies": [] }, { "id": 890328, "author_name": "", "author_url": "", "post_date": "2020-06-17T12:58:55.037000", "content": "", "votes": 8, "replies": [] }, { "id": 890376, "author_name": "", "author_url": "", "post_date": "2020-06-17T13:31:42.260000", "content": "", "votes": 1, "replies": [] }, { "id": 890423, "author_name": "", "author_url": "", "post_date": "2020-06-17T14:00:43.297000", "content": "", "votes": 8, "replies": [] }, { "id": 890531, "author_name": "", "author_url": "", "post_date": "2020-06-17T14:50:40.557000", "content": "", "votes": 4, "replies": [] }, { "id": 891157, "author_name": "", "author_url": "", "post_date": "2020-06-18T00:50:47.873000", "content": "", "votes": 6, "replies": [] } ] }, { "id": 899449, "author_name": "", "author_url": "", "post_date": "2020-06-24T08:50:18.980000", "content": "", "votes": 17, "replies": [ { "id": 899889, "author_name": "", "author_url": "", "post_date": "2020-06-24T14:00:31.057000", "content": "", "votes": 3, "replies": [] }, { "id": 899917, "author_name": "", "author_url": "", "post_date": "2020-06-24T14:15:30.403000", "content": "", "votes": 6, "replies": [] }, { "id": 899925, "author_name": "", "author_url": "", "post_date": "2020-06-24T14:19:07.143000", "content": "", "votes": 5, "replies": [] }, { "id": 909736, "author_name": "", "author_url": "", "post_date": "2020-06-30T19:31:37.927000", "content": "", "votes": 4, "replies": [] }, { "id": 936056, "author_name": "", "author_url": "", "post_date": "2020-07-19T23:52:29.877000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 886363, "author_name": "", "author_url": "", "post_date": "2020-06-15T01:36:17.343000", "content": "", "votes": 18, "replies": [ { "id": 886924, "author_name": "", "author_url": "", "post_date": "2020-06-15T11:14:46.730000", "content": "", "votes": 5, "replies": [] }, { "id": 888625, "author_name": "", "author_url": "", "post_date": "2020-06-16T13:24:30.910000", "content": "", "votes": 6, "replies": [] }, { "id": 900525, "author_name": "", "author_url": "", "post_date": "2020-06-24T21:43:25.997000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 883742, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:45:47.007000", "content": "", "votes": 17, "replies": [ { "id": 884838, "author_name": "", "author_url": "", "post_date": "2020-06-13T17:02:21.420000", "content": "", "votes": 5, "replies": [] } ] }, { "id": 884810, "author_name": "", "author_url": "", "post_date": "2020-06-13T16:32:04.087000", "content": "", "votes": 18, "replies": [ { "id": 885873, "author_name": "", "author_url": "", "post_date": "2020-06-14T14:30:11.037000", "content": "", "votes": 6, "replies": [] } ] }, { "id": 884152, "author_name": "", "author_url": "", "post_date": "2020-06-13T08:03:15.073000", "content": "", "votes": 18, "replies": [ { "id": 884589, "author_name": "", "author_url": "", "post_date": "2020-06-13T13:21:11.683000", "content": "", "votes": 3, "replies": [] }, { "id": 884644, "author_name": "", "author_url": "", "post_date": "2020-06-13T14:17:02.027000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 883817, "author_name": "", "author_url": "", "post_date": "2020-06-13T00:58:08.050000", "content": "", "votes": 18, "replies": [] }, { "id": 883750, "author_name": "", "author_url": "", "post_date": "2020-06-12T22:01:06.183000", "content": "", "votes": 18, "replies": [ { "id": 883753, "author_name": "", "author_url": "", "post_date": "2020-06-12T22:05:21.257000", "content": "", "votes": 8, "replies": [] }, { "id": 884633, "author_name": "", "author_url": "", "post_date": "2020-06-13T14:05:51.633000", "content": "", "votes": 6, "replies": [] }, { "id": 884660, "author_name": "", "author_url": "", "post_date": "2020-06-13T14:25:18.403000", "content": "", "votes": 8, "replies": [] }, { "id": 884945, "author_name": "", "author_url": "", "post_date": "2020-06-13T18:31:27.600000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 883721, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:26:32.920000", "content": "", "votes": 18, "replies": [] }, { "id": 883720, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:26:31.233000", "content": "", "votes": 18, "replies": [] }, { "id": 883703, "author_name": "", "author_url": "", "post_date": "2020-06-12T20:55:07.773000", "content": "", "votes": 18, "replies": [] }, { "id": 913915, "author_name": "", "author_url": "", "post_date": "2020-07-03T14:15:57.923000", "content": "", "votes": 16, "replies": [] }, { "id": 887176, "author_name": "", "author_url": "", "post_date": "2020-06-15T14:20:19.023000", "content": "", "votes": 15, "replies": [ { "id": 887333, "author_name": "", "author_url": "", "post_date": "2020-06-15T16:01:50.273000", "content": "", "votes": 17, "replies": [] }, { "id": 887372, "author_name": "", "author_url": "", "post_date": "2020-06-15T16:25:35.170000", "content": "", "votes": 12, "replies": [] }, { "id": 887391, "author_name": "", "author_url": "", "post_date": "2020-06-15T16:40:18.450000", "content": "", "votes": 9, "replies": [] }, { "id": 887432, "author_name": "", "author_url": "", "post_date": "2020-06-15T17:08:01.557000", "content": "", "votes": 3, "replies": [] }, { "id": 889801, "author_name": "", "author_url": "", "post_date": "2020-06-17T06:45:10.127000", "content": "", "votes": 5, "replies": [] } ] }, { "id": 885800, "author_name": "", "author_url": "", "post_date": "2020-06-14T13:45:52.200000", "content": "", "votes": 16, "replies": [] }, { "id": 890000, "author_name": "", "author_url": "", "post_date": "2020-06-17T08:47:18.033000", "content": "", "votes": 17, "replies": [ { "id": 890207, "author_name": "", "author_url": "", "post_date": "2020-06-17T11:38:09.800000", "content": "", "votes": 8, "replies": [] }, { "id": 890618, "author_name": "", "author_url": "", "post_date": "2020-06-17T15:53:19.940000", "content": "", "votes": 6, "replies": [] } ] }, { "id": 891696, "author_name": "", "author_url": "", "post_date": "2020-06-18T11:48:40.613000", "content": "", "votes": 13, "replies": [ { "id": 891713, "author_name": "", "author_url": "", "post_date": "2020-06-18T12:02:42.053000", "content": "", "votes": 8, "replies": [] }, { "id": 891737, "author_name": "", "author_url": "", "post_date": "2020-06-18T12:28:07.260000", "content": "", "votes": 10, "replies": [] }, { "id": 891750, "author_name": "", "author_url": "", "post_date": "2020-06-18T12:40:35.493000", "content": "", "votes": 19, "replies": [] }, { "id": 891785, "author_name": "", "author_url": "", "post_date": "2020-06-18T12:59:22.873000", "content": "", "votes": 12, "replies": [] }, { "id": 891931, "author_name": "", "author_url": "", "post_date": "2020-06-18T14:41:57.103000", "content": "", "votes": 14, "replies": [] } ] }, { "id": 893495, "author_name": "", "author_url": "", "post_date": "2020-06-19T16:50:58.923000", "content": "", "votes": 14, "replies": [] }, { "id": 887003, "author_name": "", "author_url": "", "post_date": "2020-06-15T12:28:24.970000", "content": "", "votes": 13, "replies": [ { "id": 887361, "author_name": "", "author_url": "", "post_date": "2020-06-15T16:20:57.030000", "content": "", "votes": 7, "replies": [] }, { "id": 887389, "author_name": "", "author_url": "", "post_date": "2020-06-15T16:39:23.670000", "content": "", "votes": 14, "replies": [] }, { "id": 887404, "author_name": "", "author_url": "", "post_date": "2020-06-15T16:54:43.077000", "content": "", "votes": 5, "replies": [] }, { "id": 887883, "author_name": "", "author_url": "", "post_date": "2020-06-16T01:09:51.913000", "content": "", "votes": 3, "replies": [] }, { "id": 889080, "author_name": "", "author_url": "", "post_date": "2020-06-16T18:45:52.983000", "content": "", "votes": 6, "replies": [] } ] }, { "id": 884483, "author_name": "", "author_url": "", "post_date": "2020-06-13T12:10:36.720000", "content": "", "votes": 13, "replies": [] }, { "id": 884138, "author_name": "", "author_url": "", "post_date": "2020-06-13T07:54:09.463000", "content": "", "votes": 13, "replies": [ { "id": 885149, "author_name": "", "author_url": "", "post_date": "2020-06-14T00:31:34.657000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 883697, "author_name": "", "author_url": "", "post_date": "2020-06-12T20:48:43.880000", "content": "", "votes": 13, "replies": [ { "id": 883701, "author_name": "", "author_url": "", "post_date": "2020-06-12T20:51:54.947000", "content": "", "votes": 18, "replies": [] } ] }, { "id": 883866, "author_name": "", "author_url": "", "post_date": "2020-06-13T02:35:18.077000", "content": "", "votes": 12, "replies": [] }, { "id": 883707, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:01:17.410000", "content": "", "votes": 11, "replies": [] }, { "id": 883881, "author_name": "", "author_url": "", "post_date": "2020-06-13T03:19:21.930000", "content": "", "votes": 12, "replies": [] }, { "id": 883727, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:30:17.783000", "content": "", "votes": 12, "replies": [ { "id": 883735, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:36:13.967000", "content": "", "votes": 25, "replies": [] }, { "id": 883747, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:53:12.450000", "content": "", "votes": 26, "replies": [] } ] }, { "id": 883715, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:12:55.937000", "content": "", "votes": 12, "replies": [ { "id": 885022, "author_name": "", "author_url": "", "post_date": "2020-06-13T19:57:13.097000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 883710, "author_name": "", "author_url": "", "post_date": "2020-06-12T21:01:56.063000", "content": "", "votes": 12, "replies": [ { "id": 883757, "author_name": "", "author_url": "", "post_date": "2020-06-12T22:12:54.880000", "content": "", "votes": 29, "replies": [] }, { "id": 884264, "author_name": "", "author_url": "", "post_date": "2020-06-13T09:08:26.410000", "content": "", "votes": 8, "replies": [] }, { "id": 884298, "author_name": "", "author_url": "", "post_date": "2020-06-13T09:20:43.133000", "content": "", "votes": 11, "replies": [] }, { "id": 884839, "author_name": "", "author_url": "", "post_date": "2020-06-13T17:02:42.723000", "content": "", "votes": 7, "replies": [] }, { "id": 884842, "author_name": "", "author_url": "", "post_date": "2020-06-13T17:04:38.790000", "content": "", "votes": 1, "replies": [] }, { "id": 887790, "author_name": "", "author_url": "", "post_date": "2020-06-15T21:59:05.703000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 892714, "author_name": "", "author_url": "", "post_date": "2020-06-19T05:24:22.700000", "content": "", "votes": 11, "replies": [ { "id": 893100, "author_name": "", "author_url": "", "post_date": "2020-06-19T11:44:42.987000", "content": "", "votes": 14, "replies": [] } ] }, { "id": 883694, "author_name": "", "author_url": "", "post_date": "2020-06-12T20:44:26.220000", "content": "", "votes": 10, "replies": [] }, { "id": 888733, "author_name": "", "author_url": "", "post_date": "2020-06-16T14:33:52.110000", "content": "", "votes": 10, "replies": [] }, { "id": 885606, "author_name": "", "author_url": "", "post_date": "2020-06-14T10:45:17.997000", "content": "", "votes": 10, "replies": [] }, { "id": 885301, "author_name": "", "author_url": "", "post_date": "2020-06-14T04:52:46.703000", "content": "", "votes": 10, "replies": [] }, { "id": 884406, "author_name": "", "author_url": "", "post_date": "2020-06-13T11:02:22.020000", "content": "", "votes": 10, "replies": [] }, { "id": 884465, "author_name": "", "author_url": "", "post_date": "2020-06-13T11:53:39.660000", "content": "", "votes": 8, "replies": [] }, { "id": 888060, "author_name": "", "author_url": "", "post_date": "2020-06-16T05:12:45.137000", "content": "", "votes": 7, "replies": [] }, { "id": 885005, "author_name": "", "author_url": "", "post_date": "2020-06-13T19:43:25.253000", "content": "", "votes": 7, "replies": [] }, { "id": 884718, "author_name": "", "author_url": "", "post_date": "2020-06-13T15:20:32.787000", "content": "", "votes": 7, "replies": [ { "id": 888029, "author_name": "", "author_url": "", "post_date": "2020-06-16T04:32:18.093000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 884371, "author_name": "", "author_url": "", "post_date": "2020-06-13T10:19:16.880000", "content": "", "votes": 7, "replies": [] }, { "id": 891631, "author_name": "", "author_url": "", "post_date": "2020-06-18T10:43:58.360000", "content": "", "votes": 8, "replies": [] }, { "id": 885049, "author_name": "", "author_url": "", "post_date": "2020-06-13T20:32:53.747000", "content": "", "votes": 8, "replies": [] }, { "id": 884835, "author_name": "", "author_url": "", "post_date": "2020-06-13T17:01:10.830000", "content": "", "votes": 8, "replies": [] }, { "id": 884107, "author_name": "", "author_url": "", "post_date": "2020-06-13T07:35:50.337000", "content": "", "votes": 8, "replies": [] }, { "id": 883960, "author_name": "", "author_url": "", "post_date": "2020-06-13T06:32:25.730000", "content": "", "votes": 8, "replies": [] }, { "id": 883906, "author_name": "", "author_url": "", "post_date": "2020-06-13T04:14:21.370000", "content": "", "votes": 6, "replies": [] }, { "id": 884343, "author_name": "", "author_url": "", "post_date": "2020-06-13T09:48:39.210000", "content": "", "votes": 7, "replies": [] }, { "id": 915968, "author_name": "", "author_url": "", "post_date": "2020-07-05T08:50:10.657000", "content": "", "votes": 6, "replies": [] }, { "id": 885165, "author_name": "", "author_url": "", "post_date": "2020-06-14T01:08:38.647000", "content": "", "votes": 5, "replies": [ { "id": 886012, "author_name": "", "author_url": "", "post_date": "2020-06-14T16:39:55.877000", "content": "", "votes": 8, "replies": [] }, { "id": 888460, "author_name": "", "author_url": "", "post_date": "2020-06-16T11:22:55", "content": "", "votes": 2, "replies": [] } ] }, { "id": 895434, "author_name": "", "author_url": "", "post_date": "2020-06-21T10:45:14.143000", "content": "", "votes": 6, "replies": [] }, { "id": 891911, "author_name": "", "author_url": "", "post_date": "2020-06-18T14:26:21.723000", "content": "", "votes": 6, "replies": [] }, { "id": 888979, "author_name": "", "author_url": "", "post_date": "2020-06-16T17:36:48.677000", "content": "", "votes": 6, "replies": [] }, { "id": 883942, "author_name": "", "author_url": "", "post_date": "2020-06-13T06:21:16.160000", "content": "", "votes": 6, "replies": [] }, { "id": 917355, "author_name": "", "author_url": "", "post_date": "2020-07-06T12:38:41.770000", "content": "", "votes": 4, "replies": [] }, { "id": 884516, "author_name": "", "author_url": "", "post_date": "2020-06-13T12:30:53.717000", "content": "", "votes": 5, "replies": [] }, { "id": 898361, "author_name": "", "author_url": "", "post_date": "2020-06-23T13:37:04.430000", "content": "", "votes": 3, "replies": [] }, { "id": 936127, "author_name": "", "author_url": "", "post_date": "2020-07-20T02:30:22.800000", "content": "", "votes": 4, "replies": [] }, { "id": 891703, "author_name": "", "author_url": "", "post_date": "2020-06-18T11:55:02.100000", "content": "", "votes": 3, "replies": [ { "id": 891710, "author_name": "", "author_url": "", "post_date": "2020-06-18T11:59:28.483000", "content": "", "votes": 9, "replies": [] }, { "id": 891715, "author_name": "", "author_url": "", "post_date": "2020-06-18T12:06:19.137000", "content": "", "votes": 14, "replies": [] } ] }, { "id": 886338, "author_name": "", "author_url": "", "post_date": "2020-06-15T00:11:30.220000", "content": "", "votes": 3, "replies": [] }, { "id": 909731, "author_name": "", "author_url": "", "post_date": "2020-06-30T19:24:42.660000", "content": "", "votes": 2, "replies": [] }, { "id": 884809, "author_name": "", "author_url": "", "post_date": "2020-06-13T16:30:56.647000", "content": "", "votes": 3, "replies": [] }, { "id": 961432, "author_name": "", "author_url": "", "post_date": "2020-08-07T07:23:37.470000", "content": "", "votes": 1, "replies": [] }, { "id": 904170, "author_name": "", "author_url": "", "post_date": "2020-06-27T12:08:13.713000", "content": "", "votes": 1, "replies": [ { "id": 904188, "author_name": "", "author_url": "", "post_date": "2020-06-27T12:28:44.363000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 889389, "author_name": "", "author_url": "", "post_date": "2020-06-17T00:16:44.113000", "content": "", "votes": 1, "replies": [] }, { "id": 889085, "author_name": "", "author_url": "", "post_date": "2020-06-16T18:49:41.313000", "content": "", "votes": -10, "replies": [ { "id": 889120, "author_name": "", "author_url": "", "post_date": "2020-06-16T19:21:07.860000", "content": "", "votes": 36, "replies": [] }, { "id": 889145, "author_name": "", "author_url": "", "post_date": "2020-06-16T19:48:27.540000", "content": "", "votes": 20, "replies": [] }, { "id": 889151, "author_name": "", "author_url": "", "post_date": "2020-06-16T19:52:51.957000", "content": "", "votes": 22, "replies": [] }, { "id": 889163, "author_name": "", "author_url": "", "post_date": "2020-06-16T20:06:09.903000", "content": "", "votes": 30, "replies": [] }, { "id": 889175, "author_name": "", "author_url": "", "post_date": "2020-06-16T20:11:12.283000", "content": "", "votes": 22, "replies": [] }, { "id": 889217, "author_name": "", "author_url": "", "post_date": "2020-06-16T20:42:11.230000", "content": "", "votes": 12, "replies": [] }, { "id": 889226, "author_name": "", "author_url": "", "post_date": "2020-06-16T20:51:04.510000", "content": "", "votes": 13, "replies": [] }, { "id": 889228, "author_name": "", "author_url": "", "post_date": "2020-06-16T20:54:48.493000", "content": "", "votes": 15, "replies": [] }, { "id": 889235, "author_name": "", "author_url": "", "post_date": "2020-06-16T21:00:15.390000", "content": "", "votes": 10, "replies": [] }, { "id": 889239, "author_name": "", "author_url": "", "post_date": "2020-06-16T21:03:18.677000", "content": "", "votes": 9, "replies": [] }, { "id": 889260, "author_name": "", "author_url": "", "post_date": "2020-06-16T21:26:09.557000", "content": "", "votes": 13, "replies": [] }, { "id": 889271, "author_name": "", "author_url": "", "post_date": "2020-06-16T21:47:41.887000", "content": "", "votes": -4, "replies": [] }, { "id": 889280, "author_name": "", "author_url": "", "post_date": "2020-06-16T21:58:10.357000", "content": "", "votes": 12, "replies": [] }, { "id": 889560, "author_name": "", "author_url": "", "post_date": "2020-06-17T02:27:17.893000", "content": "", "votes": 1, "replies": [] }, { "id": 889876, "author_name": "", "author_url": "", "post_date": "2020-06-17T07:38:39.987000", "content": "", "votes": 10, "replies": [] }, { "id": 889988, "author_name": "", "author_url": "", "post_date": "2020-06-17T08:36:36.333000", "content": "", "votes": 11, "replies": [] }, { "id": 891537, "author_name": "", "author_url": "", "post_date": "2020-06-18T08:43:14.363000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 917439, "author_name": "", "author_url": "", "post_date": "2020-07-06T14:09:24.777000", "content": "", "votes": 1, "replies": [] }, { "id": 915972, "author_name": "", "author_url": "", "post_date": "2020-07-05T08:53:55.687000", "content": "", "votes": 1, "replies": [] }, { "id": 914348, "author_name": "", "author_url": "", "post_date": "2020-07-03T19:27:03.753000", "content": "", "votes": 1, "replies": [] }, { "id": 891502, "author_name": "", "author_url": "", "post_date": "2020-06-18T08:07:00.370000", "content": "", "votes": 0, "replies": [] }, { "id": 893614, "author_name": "", "author_url": "", "post_date": "2020-06-19T18:47:47.713000", "content": "", "votes": 0, "replies": [] }, { "id": 883799, "author_name": "", "author_url": "", "post_date": "2020-06-13T00:15:58.413000", "content": "", "votes": -32, "replies": [ { "id": 883802, "author_name": "", "author_url": "", "post_date": "2020-06-13T00:30:15.407000", "content": "", "votes": 40, "replies": [] }, { "id": 883840, "author_name": "", "author_url": "", "post_date": "2020-06-13T01:32:07.110000", "content": "", "votes": 27, "replies": [] }, { "id": 883841, "author_name": "", "author_url": "", "post_date": "2020-06-13T01:35:51.553000", "content": "", "votes": 23, "replies": [] }, { "id": 883842, "author_name": "", "author_url": "", "post_date": "2020-06-13T01:38:01.887000", "content": "", "votes": 11, "replies": [] }, { "id": 883849, "author_name": "", "author_url": "", "post_date": "2020-06-13T01:49:24.853000", "content": "", "votes": 28, "replies": [] }, { "id": 884127, "author_name": "", "author_url": "", "post_date": "2020-06-13T07:47:35.127000", "content": "", "votes": 12, "replies": [] }, { "id": 885710, "author_name": "", "author_url": "", "post_date": "2020-06-14T12:13:10.517000", "content": "", "votes": 9, "replies": [] }, { "id": 886606, "author_name": "", "author_url": "", "post_date": "2020-06-15T06:44:44.900000", "content": "", "votes": 5, "replies": [] }, { "id": 887090, "author_name": "", "author_url": "", "post_date": "2020-06-15T13:25:00.667000", "content": "", "votes": 7, "replies": [] }, { "id": 952261, "author_name": "", "author_url": "", "post_date": "2020-07-30T18:58:25.453000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 886350, "author_name": "", "author_url": "", "post_date": "2020-06-15T00:56:33.580000", "content": "", "votes": -6, "replies": [ { "id": 886974, "author_name": "", "author_url": "", "post_date": "2020-06-15T12:05:09.207000", "content": "", "votes": 8, "replies": [] }, { "id": 886987, "author_name": "", "author_url": "", "post_date": "2020-06-15T12:17:28.557000", "content": "", "votes": 4, "replies": [] }, { "id": 887015, "author_name": "", "author_url": "", "post_date": "2020-06-15T12:33:30.220000", "content": "", "votes": 12, "replies": [] }, { "id": 887036, "author_name": "", "author_url": "", "post_date": "2020-06-15T12:46:24.343000", "content": "", "votes": 1, "replies": [] }, { "id": 887066, "author_name": "", "author_url": "", "post_date": "2020-06-15T13:13:51.743000", "content": "", "votes": 6, "replies": [] }, { "id": 887076, "author_name": "", "author_url": "", "post_date": "2020-06-15T13:16:45.497000", "content": "", "votes": 7, "replies": [] }, { "id": 887077, "author_name": "", "author_url": "", "post_date": "2020-06-15T13:17:48.817000", "content": "", "votes": 7, "replies": [] }, { "id": 887134, "author_name": "", "author_url": "", "post_date": "2020-06-15T13:52:52.773000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 910325, "author_name": "", "author_url": "", "post_date": "2020-07-01T05:25:54.920000", "content": "", "votes": -9, "replies": [] }, { "id": 887444, "author_name": "", "author_url": "", "post_date": "2020-06-15T17:15:27.487000", "content": "", "votes": -9, "replies": [ { "id": 888012, "author_name": "", "author_url": "", "post_date": "2020-06-16T04:18:17.987000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 885863, "author_name": "", "author_url": "", "post_date": "2020-06-14T14:26:30.130000", "content": "", "votes": -9, "replies": [] }, { "id": 889471, "author_name": "", "author_url": "", "post_date": "2020-06-17T01:11:29.340000", "content": "", "votes": -23, "replies": [ { "id": 890001, "author_name": "", "author_url": "", "post_date": "2020-06-17T08:48:24.527000", "content": "", "votes": 2, "replies": [] }, { "id": 890433, "author_name": "", "author_url": "", "post_date": "2020-06-17T14:05:53.677000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 894041, "author_name": "", "author_url": "", "post_date": "2020-06-20T06:03:16.507000", "content": "", "votes": -25, "replies": [ { "id": 894459, "author_name": "", "author_url": "", "post_date": "2020-06-20T12:55:18.823000", "content": "", "votes": 8, "replies": [] }, { "id": 894468, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:01:29.020000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 905258, "author_name": "", "author_url": "", "post_date": "2020-06-28T11:44:28.363000", "content": "", "votes": -13, "replies": [] }, { "id": 902553, "author_name": "", "author_url": "", "post_date": "2020-06-26T08:11:03.640000", "content": "", "votes": -16, "replies": [] }, { "id": 901794, "author_name": "", "author_url": "", "post_date": "2020-06-25T17:36:31.013000", "content": "", "votes": -14, "replies": [] }, { "id": 892093, "author_name": "", "author_url": "", "post_date": "2020-06-18T16:44:39.263000", "content": "", "votes": -23, "replies": [] }, { "id": 886247, "author_name": "", "author_url": "", "post_date": "2020-06-14T20:35:35.130000", "content": "", "votes": -11, "replies": [] }, { "id": 884016, "author_name": "", "author_url": "", "post_date": "2020-06-13T06:52:36.070000", "content": "", "votes": -18, "replies": [ { "id": 885014, "author_name": "", "author_url": "", "post_date": "2020-06-13T19:51:19.660000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 2150658, "author_name": "", "author_url": "", "post_date": "2023-02-19T13:38:29.863000", "content": "", "votes": 0, "replies": [] }, { "id": 918080, "author_name": "", "author_url": "", "post_date": "2020-07-07T01:03:34.950000", "content": "", "votes": 0, "replies": [] }, { "id": 918705, "author_name": "", "author_url": "", "post_date": "2020-07-07T12:16:06.720000", "content": "", "votes": -1, "replies": [] }, { "id": 918577, "author_name": "", "author_url": "", "post_date": "2020-07-07T10:32:36.420000", "content": "", "votes": -1, "replies": [] }, { "id": 918354, "author_name": "", "author_url": "", "post_date": "2020-07-07T07:48:12.443000", "content": "", "votes": -1, "replies": [] }, { "id": 918212, "author_name": "", "author_url": "", "post_date": "2020-07-07T04:48:25.133000", "content": "", "votes": -3, "replies": [] }, { "id": 918090, "author_name": "", "author_url": "", "post_date": "2020-07-07T01:25:09.060000", "content": "", "votes": -4, "replies": [] }, { "id": 917734, "author_name": "", "author_url": "", "post_date": "2020-07-06T18:13:37.507000", "content": "", "votes": -4, "replies": [] }, { "id": 917534, "author_name": "", "author_url": "", "post_date": "2020-07-06T15:41:28.657000", "content": "", "votes": -3, "replies": [] }, { "id": 917349, "author_name": "", "author_url": "", "post_date": "2020-07-06T12:33:18.777000", "content": "", "votes": -2, "replies": [] }, { "id": 917328, "author_name": "", "author_url": "", "post_date": "2020-07-06T12:18:16.277000", "content": "", "votes": -4, "replies": [] }, { "id": 917240, "author_name": "", "author_url": "", "post_date": "2020-07-06T11:05:54.720000", "content": "", "votes": -6, "replies": [] }, { "id": 916648, "author_name": "", "author_url": "", "post_date": "2020-07-05T21:57:30.003000", "content": "", "votes": 0, "replies": [] }, { "id": 916154, "author_name": "", "author_url": "", "post_date": "2020-07-05T12:24:15.603000", "content": "", "votes": -8, "replies": [] }, { "id": 916143, "author_name": "", "author_url": "", "post_date": "2020-07-05T12:09:59.647000", "content": "", "votes": -7, "replies": [] }, { "id": 916097, "author_name": "", "author_url": "", "post_date": "2020-07-05T11:02:58.887000", "content": "", "votes": -9, "replies": [] }, { "id": 915591, "author_name": "", "author_url": "", "post_date": "2020-07-04T22:55:11.147000", "content": "", "votes": -7, "replies": [] }, { "id": 915440, "author_name": "", "author_url": "", "post_date": "2020-07-04T18:24:10.547000", "content": "", "votes": -7, "replies": [] }, { "id": 915030, "author_name": "", "author_url": "", "post_date": "2020-07-04T12:32:22.167000", "content": "", "votes": -8, "replies": [] }, { "id": 914939, "author_name": "", "author_url": "", "post_date": "2020-07-04T11:20:07.767000", "content": "", "votes": -7, "replies": [] }, { "id": 914349, "author_name": "", "author_url": "", "post_date": "2020-07-03T19:28:54.027000", "content": "", "votes": -9, "replies": [] }, { "id": 914055, "author_name": "", "author_url": "", "post_date": "2020-07-03T15:43:54.153000", "content": "", "votes": -1, "replies": [] }, { "id": 913991, "author_name": "", "author_url": "", "post_date": "2020-07-03T15:06:10.267000", "content": "", "votes": -9, "replies": [] }, { "id": 913569, "author_name": "", "author_url": "", "post_date": "2020-07-03T09:31:56.690000", "content": "", "votes": -10, "replies": [] }, { "id": 913565, "author_name": "", "author_url": "", "post_date": "2020-07-03T09:29:23.193000", "content": "", "votes": -11, "replies": [] }, { "id": 913524, "author_name": "", "author_url": "", "post_date": "2020-07-03T09:03:10.600000", "content": "", "votes": -9, "replies": [] }, { "id": 913265, "author_name": "", "author_url": "", "post_date": "2020-07-03T05:02:55.847000", "content": "", "votes": -11, "replies": [] }, { "id": 913201, "author_name": "", "author_url": "", "post_date": "2020-07-03T04:00:27.270000", "content": "", "votes": -10, "replies": [] }, { "id": 912813, "author_name": "", "author_url": "", "post_date": "2020-07-02T18:40:54.720000", "content": "", "votes": -9, "replies": [] }, { "id": 912557, "author_name": "", "author_url": "", "post_date": "2020-07-02T15:07:58.127000", "content": "", "votes": -12, "replies": [] }, { "id": 912509, "author_name": "", "author_url": "", "post_date": "2020-07-02T14:29:02.183000", "content": "", "votes": -12, "replies": [] }, { "id": 911397, "author_name": "", "author_url": "", "post_date": "2020-07-01T17:46:54.100000", "content": "", "votes": -11, "replies": [] }, { "id": 911060, "author_name": "", "author_url": "", "post_date": "2020-07-01T14:40:46.477000", "content": "", "votes": 0, "replies": [] }, { "id": 910915, "author_name": "", "author_url": "", "post_date": "2020-07-01T12:57:42.327000", "content": "", "votes": -10, "replies": [] }, { "id": 910768, "author_name": "", "author_url": "", "post_date": "2020-07-01T11:06:17.140000", "content": "", "votes": -10, "replies": [] }, { "id": 909937, "author_name": "", "author_url": "", "post_date": "2020-07-01T00:11:35.090000", "content": "", "votes": 0, "replies": [] }, { "id": 909880, "author_name": "", "author_url": "", "post_date": "2020-06-30T22:17:44.213000", "content": "", "votes": -1, "replies": [] }, { "id": 909634, "author_name": "", "author_url": "", "post_date": "2020-06-30T17:49:38.087000", "content": "", "votes": -9, "replies": [] }, { "id": 908189, "author_name": "", "author_url": "", "post_date": "2020-06-30T12:40:46.613000", "content": "", "votes": -4, "replies": [] }, { "id": 907943, "author_name": "", "author_url": "", "post_date": "2020-06-30T09:14:57.913000", "content": "", "votes": -13, "replies": [] }, { "id": 907821, "author_name": "", "author_url": "", "post_date": "2020-06-30T07:19:39.560000", "content": "", "votes": -11, "replies": [] }, { "id": 907758, "author_name": "", "author_url": "", "post_date": "2020-06-30T06:38:34.280000", "content": "", "votes": -9, "replies": [] }, { "id": 907751, "author_name": "", "author_url": "", "post_date": "2020-06-30T06:30:59.610000", "content": "", "votes": -9, "replies": [] }, { "id": 907601, "author_name": "", "author_url": "", "post_date": "2020-06-30T03:51:59.917000", "content": "", "votes": -11, "replies": [] }, { "id": 907089, "author_name": "", "author_url": "", "post_date": "2020-06-29T17:19:37.397000", "content": "", "votes": -10, "replies": [] }, { "id": 906874, "author_name": "", "author_url": "", "post_date": "2020-06-29T15:24:01.873000", "content": "", "votes": -9, "replies": [] }, { "id": 906707, "author_name": "", "author_url": "", "post_date": "2020-06-29T13:58:57.830000", "content": "", "votes": -10, "replies": [] }, { "id": 906598, "author_name": "", "author_url": "", "post_date": "2020-06-29T12:43:40.340000", "content": "", "votes": -11, "replies": [] }, { "id": 906236, "author_name": "", "author_url": "", "post_date": "2020-06-29T06:31:34.380000", "content": "", "votes": -11, "replies": [] }, { "id": 905793, "author_name": "", "author_url": "", "post_date": "2020-06-28T19:48:48.807000", "content": "", "votes": -11, "replies": [] }, { "id": 905139, "author_name": "", "author_url": "", "post_date": "2020-06-28T09:35:05.183000", "content": "", "votes": -2, "replies": [] }, { "id": 904338, "author_name": "", "author_url": "", "post_date": "2020-06-27T14:50:04.563000", "content": "", "votes": -11, "replies": [] }, { "id": 904057, "author_name": "", "author_url": "", "post_date": "2020-06-27T10:20:32.390000", "content": "", "votes": -9, "replies": [] }, { "id": 903851, "author_name": "", "author_url": "", "post_date": "2020-06-27T06:50:40.793000", "content": "", "votes": 0, "replies": [] }, { "id": 903849, "author_name": "", "author_url": "", "post_date": "2020-06-27T06:49:16.167000", "content": "", "votes": 0, "replies": [] }, { "id": 903482, "author_name": "", "author_url": "", "post_date": "2020-06-26T21:35:27.487000", "content": "", "votes": -4, "replies": [] }, { "id": 903436, "author_name": "", "author_url": "", "post_date": "2020-06-26T20:29:28.133000", "content": "", "votes": -11, "replies": [] }, { "id": 903294, "author_name": "", "author_url": "", "post_date": "2020-06-26T17:56:40.757000", "content": "", "votes": -13, "replies": [] }, { "id": 902953, "author_name": "", "author_url": "", "post_date": "2020-06-26T13:37:32.013000", "content": "", "votes": -16, "replies": [] }, { "id": 902802, "author_name": "", "author_url": "", "post_date": "2020-06-26T11:21:08.837000", "content": "", "votes": -15, "replies": [] }, { "id": 902799, "author_name": "", "author_url": "", "post_date": "2020-06-26T11:18:41.527000", "content": "", "votes": -14, "replies": [] }, { "id": 902783, "author_name": "", "author_url": "", "post_date": "2020-06-26T11:03:48.820000", "content": "", "votes": -3, "replies": [] }, { "id": 902777, "author_name": "", "author_url": "", "post_date": "2020-06-26T11:01:00.080000", "content": "", "votes": -14, "replies": [] }, { "id": 899431, "author_name": "", "author_url": "", "post_date": "2020-06-24T08:34:16.630000", "content": "", "votes": -12, "replies": [] }, { "id": 898842, "author_name": "", "author_url": "", "post_date": "2020-06-23T19:36:51.787000", "content": "", "votes": -12, "replies": [] }, { "id": 898416, "author_name": "", "author_url": "", "post_date": "2020-06-23T14:04:31.537000", "content": "", "votes": -13, "replies": [] }, { "id": 898182, "author_name": "", "author_url": "", "post_date": "2020-06-23T11:09:31.333000", "content": "", "votes": -15, "replies": [] }, { "id": 898088, "author_name": "", "author_url": "", "post_date": "2020-06-23T09:36:55.867000", "content": "", "votes": 0, "replies": [] }, { "id": 897770, "author_name": "", "author_url": "", "post_date": "2020-06-23T04:53:07.567000", "content": "", "votes": -15, "replies": [] }, { "id": 897157, "author_name": "", "author_url": "", "post_date": "2020-06-22T16:53:45.393000", "content": "", "votes": -17, "replies": [] }, { "id": 897031, "author_name": "", "author_url": "", "post_date": "2020-06-22T15:27:10.493000", "content": "", "votes": -18, "replies": [] }, { "id": 897024, "author_name": "", "author_url": "", "post_date": "2020-06-22T15:22:21.697000", "content": "", "votes": -16, "replies": [] }, { "id": 896868, "author_name": "", "author_url": "", "post_date": "2020-06-22T13:42:33.403000", "content": "", "votes": -15, "replies": [] }, { "id": 896851, "author_name": "", "author_url": "", "post_date": "2020-06-22T13:28:53.937000", "content": "", "votes": -17, "replies": [] }, { "id": 896445, "author_name": "", "author_url": "", "post_date": "2020-06-22T07:22:35.663000", "content": "", "votes": -4, "replies": [] }, { "id": 896305, "author_name": "", "author_url": "", "post_date": "2020-06-22T04:17:05.177000", "content": "", "votes": -19, "replies": [] }, { "id": 896005, "author_name": "", "author_url": "", "post_date": "2020-06-21T18:40:04.633000", "content": "", "votes": -18, "replies": [] }, { "id": 895699, "author_name": "", "author_url": "", "post_date": "2020-06-21T14:48:47.540000", "content": "", "votes": -18, "replies": [] }, { "id": 895455, "author_name": "", "author_url": "", "post_date": "2020-06-21T11:03:07.603000", "content": "", "votes": -18, "replies": [] }, { "id": 895411, "author_name": "", "author_url": "", "post_date": "2020-06-21T10:22:18.497000", "content": "", "votes": -20, "replies": [] }, { "id": 895367, "author_name": "", "author_url": "", "post_date": "2020-06-21T09:37:29.297000", "content": "", "votes": -18, "replies": [] }, { "id": 895014, "author_name": "", "author_url": "", "post_date": "2020-06-21T03:19:55.453000", "content": "", "votes": -20, "replies": [] }, { "id": 894634, "author_name": "", "author_url": "", "post_date": "2020-06-20T15:55:27.123000", "content": "", "votes": -21, "replies": [] }, { "id": 894509, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:43:30.093000", "content": "", "votes": -20, "replies": [] }, { "id": 894447, "author_name": "", "author_url": "", "post_date": "2020-06-20T12:43:29.660000", "content": "", "votes": -22, "replies": [] }, { "id": 892558, "author_name": "", "author_url": "", "post_date": "2020-06-19T02:15:49.593000", "content": "", "votes": 0, "replies": [] }, { "id": 892284, "author_name": "", "author_url": "", "post_date": "2020-06-18T19:14:25.050000", "content": "", "votes": -21, "replies": [] }, { "id": 892209, "author_name": "", "author_url": "", "post_date": "2020-06-18T18:00:07.420000", "content": "", "votes": -22, "replies": [] }, { "id": 891936, "author_name": "", "author_url": "", "post_date": "2020-06-18T14:45:56.033000", "content": "", "votes": -21, "replies": [] }, { "id": 891827, "author_name": "", "author_url": "", "post_date": "2020-06-18T13:22:30.227000", "content": "", "votes": -21, "replies": [] }, { "id": 891589, "author_name": "", "author_url": "", "post_date": "2020-06-18T09:30:27.917000", "content": "", "votes": -19, "replies": [] }, { "id": 891523, "author_name": "", "author_url": "", "post_date": "2020-06-18T08:28:12.207000", "content": "", "votes": 0, "replies": [] }, { "id": 891518, "author_name": "", "author_url": "", "post_date": "2020-06-18T08:23:18.387000", "content": "", "votes": -13, "replies": [] }, { "id": 889877, "author_name": "", "author_url": "", "post_date": "2020-06-17T07:40:09.060000", "content": "", "votes": -23, "replies": [] }, { "id": 889850, "author_name": "", "author_url": "", "post_date": "2020-06-17T07:26:50.440000", "content": "", "votes": -24, "replies": [] }, { "id": 889516, "author_name": "", "author_url": "", "post_date": "2020-06-17T01:46:31.980000", "content": "", "votes": -28, "replies": [ { "id": 889775, "author_name": "", "author_url": "", "post_date": "2020-06-17T06:13:36.220000", "content": "", "votes": -21, "replies": [] }, { "id": 889780, "author_name": "", "author_url": "", "post_date": "2020-06-17T06:23:11.550000", "content": "", "votes": -1, "replies": [] }, { "id": 889791, "author_name": "", "author_url": "", "post_date": "2020-06-17T06:35:14.077000", "content": "", "votes": 5, "replies": [] }, { "id": 890025, "author_name": "", "author_url": "", "post_date": "2020-06-17T09:06:51.823000", "content": "", "votes": -18, "replies": [] }, { "id": 890032, "author_name": "", "author_url": "", "post_date": "2020-06-17T09:12:20.480000", "content": "", "votes": -18, "replies": [] }, { "id": 891673, "author_name": "", "author_url": "", "post_date": "2020-06-18T11:23:38.690000", "content": "", "votes": -1, "replies": [] }, { "id": 914357, "author_name": "", "author_url": "", "post_date": "2020-07-03T19:36:53.313000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 888984, "author_name": "", "author_url": "", "post_date": "2020-06-16T17:38:54.200000", "content": "", "votes": -23, "replies": [] }, { "id": 888855, "author_name": "", "author_url": "", "post_date": "2020-06-16T15:44:47.440000", "content": "", "votes": -20, "replies": [] }, { "id": 888441, "author_name": "", "author_url": "", "post_date": "2020-06-16T11:05:25.853000", "content": "", "votes": -25, "replies": [ { "id": 888525, "author_name": "", "author_url": "", "post_date": "2020-06-16T12:22:01.070000", "content": "", "votes": 16, "replies": [] } ] }, { "id": 888411, "author_name": "", "author_url": "", "post_date": "2020-06-16T10:29:01.050000", "content": "", "votes": -26, "replies": [] }, { "id": 888223, "author_name": "", "author_url": "", "post_date": "2020-06-16T07:48:02.853000", "content": "", "votes": -22, "replies": [] }, { "id": 885038, "author_name": "", "author_url": "", "post_date": "2020-06-13T20:15:51.857000", "content": "", "votes": 0, "replies": [] }, { "id": 884812, "author_name": "", "author_url": "", "post_date": "2020-06-13T16:34:34.737000", "content": "", "votes": 4, "replies": [] }, { "id": 884496, "author_name": "", "author_url": "", "post_date": "2020-06-13T12:20:42.467000", "content": "", "votes": -32, "replies": [] }, { "id": 884440, "author_name": "", "author_url": "", "post_date": "2020-06-13T11:30:54.637000", "content": "", "votes": 6, "replies": [] }, { "id": 884421, "author_name": "", "author_url": "", "post_date": "2020-06-13T11:11:18.790000", "content": "", "votes": 1, "replies": [] }, { "id": 916268, "author_name": "", "author_url": "", "post_date": "2020-07-05T14:20:32.973000", "content": "", "votes": -10, "replies": [] }, { "id": 900763, "author_name": "", "author_url": "", "post_date": "2020-06-25T03:44:33.783000", "content": "", "votes": -15, "replies": [] }, { "id": 884659, "author_name": "", "author_url": "", "post_date": "2020-06-13T14:24:54.950000", "content": "", "votes": -4, "replies": [] } ], "raw_markdown_by_id": { "883677": "**Kaggle, Facebook Host Team and Fellow Competitors:**\n\nFirst of all, we want to put on record our gratitude for Kaggle and the Facebook host team for putting the effort into creating the dataset and hosting this competition, and we give our congratulations to all eventual prize winners.\n\nWe'd like to use this statement to further explain the circumstances which led to our winning solution being voided, and our position on the LB being moved with accordance to our second solution. \n\nIn anticipation of shake-up of the competition on the private LB, we prepared our two solutions which finished with private LB scores 0.42320 and 0.44531 respectively. For the 0.44531 solution, which scored better on the public LB, we used competition data only and an unweighted mean of 12 models: this is the solution that enabled us to retain our 7th position on the LB. For our original winning solution (0.42320) we mixed 6 models trained using competition data with 9 models trained with some additional external data (our more adventurous submission).\n\nFor our original winning model, we used the following additional data:\n\n- **The flickrface dataset**: we used a [resized version](https://www.kaggle.com/xhlulu/flickrfaceshq-dataset-nvidia-resized-256px) of this dataset. A few of these images had licenses which didn't allow commercial use, so in line with clarifications from Kaggle in the external data thread, we used the license information available from the original github to select and train **only** on images with license types that are acceptable for this competition ([CC-BY](https://creativecommons.org/licenses/by/2.0/), [Public Domain Mark 1.0](https://creativecommons.org/publicdomain/mark/1.0/), [Public Domain CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/), or [U.S. Government Works](http://www.usa.gov/copyright.shtml))\n\n- **Youtube videos images**: we manually created a face image dataset from a handful of youtube videos with [CC-BY](https://support.google.com/youtube/answer/2797468?hl=en-GB) license, which explicitly [allows for commercial use](https://creativecommons.org/licenses/by/3.0/).\n\nWe chose these data sources with the belief that they met the rules on external data, specifically that external data must be *\"available to use by all participants of the competition for purposes of the competition at no cost to the other participants\"*, and the additional statements in the external data thread that they must be available for commercial use and not restricted to academics etc.\n\nHowever, in our discussions with Facebook and Kaggle, we were told that despite fulfilling this we were contravening the rules on Winning Submission Documentation:\n\n\n> WINNING SUBMISSION DOCUMENTATION (Section 4 of the Competition-Specific Rules)\nIn addition to compliance with the Kaggle Documentation Guidelines at [https://www.kaggle.com/WinningModelDocumentationGuidelines](https://www.kaggle.com/WinningModelDocumentationGuidelines), the winning submission documentation must conform with the following guidelines:\n\n> A. If any part of the submission documentation depicts, identifies, or includes any person that is not an individual participant or Team member, you must have all permissions and rights from the individual depicted, identified, or included and you agree to provide Competition Sponsor and PAI with written confirmation of those permissions and rights upon request.\n\n> B. Submission documentation must not infringe, misappropriate, or violate any rights of any third party including, without limitation, copyright (including moral rights), trademark, trade secret, patent or rights of privacy or publicity.\n\n\n**Specifically, we were asked to provide \"additional permissions or licenses from individuals appearing in [our] external dataset\"**. Unfortunately, since the data was from public datasets, we didn't have specific written permission from each individual appearing in them, nor did we have any way of identifying these individuals. We didn't realise while competing that external data in this competition falls under 'documentation' as well as the external data rules, so we did not secure these permissions from individuals depicted above and beyond the licensing requirements. \n\nWe suspect that most competitors also did not realise these additional restrictions existed - we are unable to find any data posted in the External Data Thread which meets this threshold with a brief scan. During the competition, the rules on external data were repeatedly clarified, so this leaves us wondering why Kaggle never took the opportunity to clarify that external data must additionally follow the more restrictive rules for winning submission documentation.\n\nAn additional concern brought to us was that **Facebook felt some of our external data \"clearly appears to infringe third party rights\" despite being labelled as CC-BY** (it's not clear what data they were referring to specifically). Even if this were the case, it seems unreasonable to us that a Kaggle team should have to trace and verify that someone who publishes a dataset themselves has the rights to do so, and that we should have to engage rights clearance services in order to make a competition submission - it was suggested to us that we could have run our external data past our lawyer before making our submissions.\n\n**While we feel that these extra rules could have been made clear during the competition, and we hope that Kaggle will begin to clarify these rules in future competitions, we understand that there is little we can do in this instance.** We have had a constructive call with both Kaggle and Facebook which we thank them for. After this call, it was agreed that because we did not knowingly seek to undermine any rules, that our submission that did not use any external data should be allowed to remain and only the winning submission is to be disqualified.\n\nThat being said, we are very disappointed by this outcome after spending so many months on the competition. **Successful Kaggle competitions rely on a trust between competitors and Kaggle that the rules will be fairly explained and applied, and this trust has been damaged.** We welcome any thoughts from the community on this matter.\n\n\nGiba, Mikel, Yifan, Gary and Qishen \nAll Faces are Real", "884238": "In the last four years, I have never seen such absurdity on Kaggle. As a Computer Vision guy, I am furious right now. If we go by the logic provided by Facebook for removing @titericz and team, then I have a bunch of points to make:\n\n1. ImageNet is a public dataset but nowhere we credit or trace the source of the images present in ImageNet before using it. The same goes for COCO, CIFAR, etc. As pointed out by @rwightman the situation is more complex if we add face detection/recognition datasets to the list. So, every time we use ImageNet, should we trace and provide credits for an individual sample even if the dataset is public?\n\n2. Why stop at datasets? BatchNormalizattion, Dropout, CNNs, etc are patented by Google. Any use of BN or dropout indirectly involves that patent, so why allow that in the first place?\n\n3. What are the expectations of the host here? Are they expecting us to be a lawyer first to understand such a ridiculous clause buried deep down somewhere and not clarified until the end of the competition?\n\n4. Let's say we are naive and the claims of Facebook stand correct. So my question for FAIR: Is every researcher who works at FAIR, aware of this clause? If yes, why is it okay for the FAIR team to use these datasets without the consent of the source? \n\n5. The situation would be worse if it comes to pretrained weights. Any architecture/dataset can be public but that doesn't mean you can directly use the weights without the consent of the person who trained the network. So, where exactly is the borderline?\n\nTo the `All Faces Are Real` team: I am very sorry that it happened to you. I am pretty sure that every sensible Kaggler is standing with you on this issue. ", "885456": "I spent most of Saturday just catching up with friends and reading all the comments via social media and here. Thank you all for your support.\n\nFor most of last two months, we had spent much time \"behind the scene\" to coordinate with teammates, liaised with kaggle and host team, working with our legal representative to ensure we stay legally informed. Personally speaking, it was a period of significant stress. In a sense, it has been really good to finally share our side of the story with the community, and I can't overstate how heartening it is to see all the supportive messages. \n\nThank you, everyone ", "884242": "So many heartbreaking and disappointing discussions after competitions could be avoided if admins more actively reply to concerns of Kagglers during competitions. \n\nCompetitors are usually really clever, and more frequently than not point out vague aspects of rules early on, specifically with respect to external data. Frequently, potential issues about external data (is it allowed, or not) are raised early on, but are either only answered very late in the competition, or stay completely unanswered. This appears to have been an issue also here in this competition.\n\nIt seems to me that the strategy is to not reply early enough, see what happens, and then make rulings afterwards. But this harms the integrity and trust of the platform, and demotivates Kagglers heavily.\n\nI have talked with several others lately, and everyone has the same feeling, that they just don't know what is allowed and what is not allowed anylonger. I am the first person who always wants to follow the rules as precisely as possible, but how can I do that if I don't know them?", "887796": "You guys are never gonna believe what just happened!\n\n", "884745": "For years we've witnessed cases of outrageous cheating behavior on Kaggle that went unpunished. Finally the most notorious cheater was removed earlier this year, but only when a teenager and a Malaysian pet agency documented his cheating behavior. Many other notorious cheaters are still active. All the evidence that is regularly brought against them is ignored, under the excuse that it doesn't meet certain high-level standards that Kaggle supposedly adheres to. And now a team of honest, hard working, scrupulously principled and exceptionally talented Kagglers is punished because of some post-hoc pedantic scrupules coming from a powerful tech giant??? This hypocrisy cries to high heaven!\n\nAs many have remarked below, it seems that Kaggle has become the victim of its own success. Juggling dual roles - a prize-awarding competition site **AND** a community of Data Scientists - seems to be becoming increasingly difficult. And if there is a conflict between those two roles, it is now painfully obvious on which side Kaggle will default. As far as I see it, it will never be possible to smoothly reconcile those two roles. I believe that if Kaggle truly cared about its community, then the best, and perhaps only, solution going forward would be to **abandon being a prize-giving entity.** Most of us here are very loosely motivated by the monetary prizes, if at all. The recent COVID competitions proved that you can get very competitive competitions without any prizes and even without medals. (At my current job I am contractually prohibited from taking any prize money anyways, and I have been more competitive than ever.) By not awarding prizes, Kaggle will be freed from all legal constraints under which it now operates that pertain to that form of work. It will free Kaggle to more forcefully stand for its community, focus fully on the community-building and promotion and advancement of Data Science. ", "883688": "OMG. That is even worse nitpicking reasoning than I expected... Very disappointed. Next time we need a lawyer before feature engineering...", "885598": "I would like to share a few words from my heart here.\n\nI haven't been in kaggle for a long time, but it is precisely because of kaggle's fairness and justice, sincerity with most kagglers, and the spirit of sharing with knowledge from your guys that deeply attracted me. I have always maintained a high enthusiasm here, which has enabled me to become GrandMaster within one year.\n\nAnd I would like to express my heartfelt thanks to every kagglers here, thank you for your support. In the past two months, for me personally, I have fallen into a very frustrated and wronged mood☹️ , even dealing with the new competition, or my personal work, the previous passion is gone. \nNow all the stories of our team are shared here, I feel relieved😃 😃 . Thanks a lot, guys.", "885306": "Right now, the private LB looks like the biggest deepfake of all.", "883752": "> Facebook felt some of our external data \"clearly appears to infringe third party rights\"\n\nThis is news to me. FB cares about peoples rights. #TIL", "887403": "Not sure how I should take this, 6:30 into @cristiancanton [presentation](https://www.facebook.com/mediaforensics2020/videos/1640779116079742/?v=1640779116079742) in the competition overview during [Workshop on Media Forensics](https://sites.google.com/view/wmediaforensics2020/program?authuser=0), and the competition best score are shown. \n\nHere the 0.423 log loss is not the top final LB score, and the only submission with such score, as far as I know, is our voided solution at 0.4232.\n\nIf our solution is not acceptable, why parade our score in your presentation? \n\n\n", "886135": "I love kaggle community, but I think Kaggle Team doesn't appreciate contribution from anyone of us (Novice --> GM). Money award doesn't matter, we are solving any competition even if size of award is tiny, despite on almost anyone competitor (Novice --> GM) can earn this money during one month without hard-working as require competition. \n\nWe expect only that Kaggle Team will fight for justice for everyone. But Kaggle Team doesn't care:\n\n- remove teams without good reason (because organizers from facebook can't run working kernels, HAHAHA)\n- don't provide even a piece of data from private stage (for checking why many team got 0.5)\n- use double standards for interpretation rules\n- don't explain specificity of your rules during competition (ignore)\n\nKaggle Team forgot that \"Business Power\" are professionals from Kaggle Community and their free hard work. Lose us = Lose quality of solutions = Lose reputation = Lose business", "884841": "My impression, for what it's worth, is that most of the problematic cases that we have been seeing in the last years have been routed in a certain disconnect between the Kaggle team and the community. In a way, this is perfectly understandable, since the Kaggle team is still relatively small, and it's impossible, say, for 1 or 2 people in charge of a competition to keep up with the ingenuities and (crazy) ideas of thousands of competitors. And running (and building) a platform is a different business than using it; with different priorities and philosophies. In many cases, the disconnect is not a problem, but a normal consequence of our different roles in the community.\n\nHowever, in cases like this (or the PetFinder scandal, or the Passenger Screening competition for those who remember it, or also the high-profile Kernel plagiarism cases); in these situations the disconnect becomes a serious problem, because it leads to a mismatch of expectations between the team (& the sponsor) vs the community. Conflicting expectations, especially when combined with a lack of communication, then lead to extremely frustrating situations like the current one; which likely could have been avoided if everyone had been on the same page from the beginning. In those cases, it becomes a weakness that Kaggle does not tap into the community hivemind when planning a competition strategy.\n\nSo here's a suggestion: we could institute a **Kaggle community advisory board**, consisting of a large number (~100) of well-respected Masters and Grandmasters. (I could easily name a few dozen without even thinking; I'm sure we got the numbers.) For any given competition, the team and board would select, say, 6-10 board members who would then act as a liaison between the organisers and the competitors. They would also help to foresee problems and controversies (e.g. to avoid changes in metric or flag potential leaks). Of course, those 6-10 Kagglers wouldn't be allowed to compete in that specific competition, but with a large enough pool to draw from, this shouldn't be an issue either. Nobody joins all competitions, these days.\n\nIn this way, there is a good chance that conversations like the one we're having now could happen before a competition; and before lots of time and effort have been spent working under \"wrong\" assumptions (which were very reasonable assumptions in this case). This scenario would create more work for the Kaggle board members, but I have the feeling that many Masters and Grandmasters would accept the responsibility (and could also learn a new thing or two by looking at a competition from a different perspective). Of course, certain Kaggle \"trade secrets\" might have to remain opaque so that the board members could continue to compete without having an unfair advantage in future competitions.\n\nAnyway, that's my thoughts. I'd be curious to read feedback.", "883728": "I'm really sorry that this happened. It looks like you did your due diligence and went out of your way to even remove data that had an inappropriate license in an explicit effort to avoid breaking the rules - yet were punished anyways. \n\nThere were several posts in the external data thread that went unanswered regarding the use of various external data sources. As was mentioned before, if the stipulations regarding external data were so strict (as to essentially prevent any practical use of other data sources), external data (apart from pretrained models) should not have been permitted in the first place so that teams wouldn't go through all this effort just to have a winning solution disqualified. ", "884310": "To be simplfied, Kaggle and FB didn't make the rules clear.\nThen all the losses caused by this mistake were passed on to us.\n\nWe worked very hard on understanding the rules and finally beated by \"**The organizer own the final interpretation right to the rules**\"", "883887": "I do hope that Kaggle takes this opportunity to rethink the use of external datasets.\n\nThere's been a weird dynamic in competitions where Kagglers scramble to look for useful external datasets or pretrained models, post them in the \"External data thread\", never get a clear answer about whether they're allowed, and often end up using them anyway. \n\nIt's bad because a) it creates an arms race (especially in multimedia competitions) for scraped data which takes away from the spirit of these competitions and b) it disadvantages teams that scrupulously follow the rules over teams that cross the line in terms of the data they use.\n\nPerhaps consider switching to a fixed whitelist of external data/models that Kaggle administrators define? ", "883865": "If this hasn't been pointed out already, the decision to remove this solution but keep (most) others in the competition is extremely arbitrary. Scanning many of the solutions, all face detectors/recognition models that I'm aware of were trained with datasets that are not compliant with the rules, actually much less so than the extra datasets these competitors used.\n\ndlib face_detection, uses VGG face (Non commercial license, no consent), scrubface (no consent), manual scrubbing (duh). mtcnn, facenet, etc all use datasets based on faces without consent (the ones mentioned already, CASIA-WebFace, LFW, etc etc). Then there is ImageNet (non-commercial) pretrained weights. \n\nSo, this is a long standing Kaggle question. Why does a pretrained model not apply to the rules but a self trained one does? Someone deciding to license their code with a given license (say dlib is Boosts) while the dataset they used (VGG face non-commercial) falls under a different one clearly does not make those weights fall under their code license. There should really be no difference. ", "883712": "So do we now have to get a written and notarized permission from the ImageNet creators every time we use pretrained models in Kaggle solutions? ", "885153": "Giba, Mikel, Yifan, Gary and Qishen , you deserve the first place. We all see it. \n\nI frankly don't know what else to say at this point. Many people made great comments about what to do next and I agree with many of them. But for you it won't matter much I guess. I hope you'll get over it. \n\nAn afterthought: I would make clear to Facebook that they have no rights to reuse any part of your solution. AFAIK, only prize winners have an obligation to provide a licence for their code to the sponsor.", "883685": "Thanks for giving this background and sorry it ended this way. I thought it was to do with the commercial licensing of the data. I was not even aware of the additional stipulations. In my opinion that should have been something much more prominently displayed, because that additional rule would have quickly eliminated all external data usage basically. ", "887946": "Up front, I should clearly state that Kaggle employees are employees of Google. In the rare event that a dispute is raised on Kaggle that requires review of competition rules, our actions, statements, and conduct require us to refrain from opinions, speculation, and casual commentary. It is for this reason that you can expect a slower response in matters like this.\n\nWe are working on a more comprehensive response and will post when that’s ready.\n", "884861": "No matter what way I look at this and no matter how much I try to understand from a host's perspective, I cannot come to convince myself that this was correct or fair in any way. There have been questionable decisions in competitions in the past but this is just preposterous and beyond acceptable.\n\nCongratulations to Giba, Mikel, Yifan, Gary and Qishen for their work, effort, solution and for winning this competition!", "883717": "Is my cat part of the documentation? It may well be if you hire the right lawyer. \n\nThis is truly shocking – are we really entering times when to succeed in a Kaggle competition your team will need more lawyers than scientists? Perhaps just lawyers? \n\nLet me get this straight – the world’s top technology to detect deepfakes has just been “voided” by Facebook because of their claim that the training data is the “documentation”. Really? I have looked very carefully through the Kaggle definition of documentation and could not find a single sentence stating that training data is the documentation. \n\nSad times for science. \n", "892113": "I thought Kaggle and Google had developed a [super duper spam detector](https://cloud.google.com/blog/products/ai-machine-learning/how-kaggle-solved-a-spam-problem-using-automl) algorithm recently ..\n\nSo why this topic is still filled with bot users having not very subtle spams like \"Wow\", \"Great Job\", \"Nice\" etc. ? ", "886040": "This raises a bigger issue about all competitions having personal data involved... for example ongoing melanoma challenge - my reasoning here:\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#886037", "884662": "This sucks. Sounds like a case of \"Let's keep the rules vague so we can disqualify submissions that we don't like for whatever reason.\"", "883696": "This is awful. Truly reprehensible bullying behavior. I can't believe that Facebook will again get away with bending all standards of ethics and common decency.", "885964": "It is frustrating that Kaggle has only contributed to this dialogue insofar as making a single post containing a vague non-response that basically amounts to \"read the fine print\" without so much as a simple apology. \n\nThis falls squarely on their shoulders, and it is unfortunate that part of the blame is shifted onto the competitors. It is disrespectful, especially when the team consists of 3 GMs and 2 Masters (who will probably be GMs soon) that have contributed so much to the community. ", "883873": "Facebook is not a company it used to be; it does what it wants and this is just another case. Sadly one of most appreciated team of Kagglers are on the receiving end. I'm so sorry for your team ;(\nI have one suggestion to Kaggle: Please remove this from future competitions; there is no meaning to it and only makes the forums crowded \n\n", "883766": "So not only were around 5% of the teams screwed out of the private leaderboard because their solution mysteriously didn't work and the top team on public and others lost due to misleading info on sound manipulation, BUT the top teams scores were then voided over ridiculous fine print rules and FB being picky just not liking their solution??!\n\nTeams worked on this problem for months, incurring costs just to use the giant dataset, just to be screwed over because Kaggle didn't inform competitors. And for your team specially you were shut down even though you meticulously made sure to follow all the rules!\n\nLike others have said Kaggle is a place to casually compete and move science forward but it seems we need lawyers to make sure we just follow the rules correctly!\n\nAnd that is after we apparently just had to cross our fingers and hope we can even make it to the private leaderboard. The Kaggle team needs to rethink how to do these rerun solutions types of competitions for the future.\n\nI definitely no longer hold any pride in doing decent in this comp when there are many who worked way harder and deserved something for it.", "883800": "While you are all blaming Facebook it is important to remember **who** was **responsible** for **moderating** this **competition**. That falls squarely on **Kaggle**. The **platform** owner (Kaggle) **failed** to **moderate** the competition by **eliminating** the **ambiguity** raised with the **rules** over at the **official forum**. Many of the **questions** were greeted with **incomplete** and **ambiguous responses** even after **repeated attempts** to seek **clarity**. That is where it all started to go **downhill**.\n\n### **At the very least there should be a public apology from Kaggle to all contestants.**\n\nWithout users Kaggle doesn't exist. They should keep that in mind.\n\n\\#KaggleDFDCFail\n", "889309": "@juliaelliott first of all, thank you for taking to time to engage with us.\n\n> I’d like to attempt to clarify the underlying issue with All Faces are Real’s disqualified submission. Some of the videos/images used in the disqualified submission were mis-licensed, in that they contained content belonging to other third parties (such as CNN), but were inappropriately offered under open source licenses. This content also clearly depicted third parties and used third party data whose permissions had not been obtained, in violation of the competition rules...\n\nHere is my thought - I struggle to pin down what \"mis-licensed\" mean here. If you search within Kaggle, you would find this is the first time this word (\"mis-licensed\" or \"mislicense\") is used in any discussion or notebook. Also it is definitely not shown in the competition rule.\n\nFurthermore, we were asked to provide \"copies of any additional permissions or licenses from individuals\" appearing in the CC-BY youtube videos with deepfake manipulation that we used. and I am not sure by saying \"mis-license\": do you mean the youtube video providers can't/shouldn't give CC-BY license after applying deepfake on those videos? or do you mean they can't/shouldn't give CC-BY license on a videos that is CNN's property? or was it \"mis-licensed\" because the video providers didn't have individual consent from people appearing in the videos?\n\nEither way, isn't it reasonable to infer that it should be Youtube's responsibility to decide if a video hosted on youtube is \"mis-licensed\", and not individual user like us?\n\nI do apologies if my above statement is poorly constructed in a legal sense, after all I am not a legal professional. it would be great if people in our community, kaggle or host team can give legal definition of \"mis-licensed\" , and also advise if individual users should be penalised for using CC-BY license videos that are \"mis-licensed\" \n\nThis further illustrates our frustration: it seems without hiring a legal representative before taking part in this competition, there is no way to determine if a piece of extra information can live up to our dear host team's morally superior standard. \n\nAnd please allow me to again emphasis there is NO MENTION from you or host team whatsoever to look out for individual consent, or \"mis-licensed\" information during the competition - NONE. and yet, in @cristiancanton [talk](https://www.facebook.com/watch/?v=1640779116079742) (see 5:03) - it was very clearly that the host team, at their brilliant and well-resourced effort to create the DFDC dataset, they made a point of securing individual consent in DF videos in **2nd half of 2019**. you can tell they are rightly proud of it in this slide here.  \n\nand yet, given they are so RIGHTLY PROUD of their dataset having individual consents, somehow this very important evaluation criterion for appropriateness of external dataset was NEVER mentioned, not even once in any of your clarification in the forum \n\ndo you not see why our team, other participants - including prize winners who voiced their opinion here, and other fellow kagglers are frustrated and upset?\n", "899449": "A great dataset for bot detection is being made thanks to this topic!", "886363": "It's in Kaggle's long-term interests to be more proactive about protecting Kagglers from the legal vagaries of competition hosts - it's terrible for morale and engagement if people are disqualified on a technicality. Kaggle has the ability to insist on being the final arbiter of who is the winner and who is not - Kaggle essentially has a monopoly on running ML competitions, so if a prospective competition host doesn't like Kaggle exerting that amount of control, then the host has limited options for running their competition elsewhere.", "883742": "There is a doubt here. Similar to MTCNN/RetinaFace/BlazeFace or more, these face detection models require a large number of face images dataset for training. I would like to ask, have all the face images in them been copyrighted? Or, if we just use the pretrained model, so we do not need to worry about the copyrights, and are also commercially available?", "884810": "This capricious enforcement of the rules makes no sense. Applying part A of the documentation section uniformly to all external data in this way would invalidate nearly all of the competition entries.\n\nHas there been a misunderstanding on the part of the sponsors' legal team? They may believe that the competition solution would be directly plugged into their existing production systems and used as-is, thereby incurring potential liabilities.\n\nI encourage the sponsors' engineering and research talent to reach out to the competition organizers and help them understand how the winning solutions will be used. The most likely scenario, in my mind, is that these solutions would form a set of promising templates for production systems. When viewed in that light, I hope the organizers might have a change of heart about this unfortunate decision.", "884152": "I feel very sorry how this ended for your team and thanks for sharing what happened. \n\nKaggle team should take this seriously and come up with ideas to improve how they run competitions otherwise this tragedy happens again in the future.\n\nHere is my suggestion.\n\n```\nKaggle team should clearly answer all questions regarding rules. \n```\n\nLook at External data thread in recent competitions for example and how often a question(clarification) is left unanswered. This leaves participants in very unconfortable position. We can not be sure what external data is allowed or not allowed, what method is allowed or not allowed beforehand. This ambiguity leads us to unfair competitions too because the ambiguity are solved after competitions by human not by written rules.\n\nI have observed kaggle's attitude regarding rules. They say read the rules carefully and not answering questions. But what happens if the rules are not clear or interpretation of the rules are different among people.\n\nAnswering all questions might be redundant and boring but I believe this is the most important part of the competition to be fair and avoid this kind of thing happening again.\n", "883817": "From the perspective of someone who wasn't involved in this competition, the situation doesn't look good. The detailed write-up illustrates that the winning 'All Faces are Real' team put a lot of careful thought into selecting their external data to be compliant with the competition rules. It is also clear that the additional rules that the sponsor was requesting were much more restrictive than usual. Given the substantial effort contributed by all participants, and also the amount of prize money, it would have been very reasonable to expect that those special requirements are clearly communicated. I can empathize with the reactions here and with the winning team; this outcome must be extremely frustrating for them. I hope that we will see more transparent communications of new or unusual rules and conditions in the future.", "883750": "> **DISPUTE RESOLUTION**\nExcept where prohibited by law, any and all disputes, claims, and causes of action between you and any Competition Entity arising out of or connected with this Competition, the determination of any winner, or any prize awarded must be resolved individually, without resort to any form of class action. Further, in any such dispute, under no circumstances will any Competition participant be permitted or entitled to obtain awards for, and hereby waives all rights to claim punitive, incidental or consequential damages, or any other damages, including attorneys’ fees, other than the individual participant’s actual out-of-pocket expenses (if any), not to exceed ten dollars ($10 USD), and each individual participant further waives all rights to have damages multiplied or increased.\n\nAnother \"funny\" section from the rules. Maybe you could sue for ten dollars...", "883721": "very sad moment for kaggle community.", "883720": "That's really awful. Do we need to predict \"hidden rules\" too to win?\nI definitely believe they must be qualified, but kaggle team, if you said it's violated to the rules, please examine and update all of on-going and future competition's rules so that this tragedy would never happen. ", "883703": "Basically, you would need to hire a lawyer to diligently check all the data applicability, before you can use it. But even then the host and competition admin can still have their interpretation that would cancel yours", "913915": "Maybe we need to leave a message once in a while to let new comers know that all the informatic comments are pushed to the bottom by those bot's comments ;)", "887176": "One additional thought. Even if facebook is not willing to accept the top submissions due to some unique and different interpretation of the WINNING SUBMISSION DOCUMENTATION rules it is strange to remove that submission from LB. I remember a few previous competitions when the winner declined prizes and still stayed on LB. It would help at least in terms of points/fame. The loss of 500K$/200K would still hurt though...", "885800": "Sorry to see this happen to your team, this must be an emotional roller coaster for your team. There was so much ambiguity in the rules from the start and I had a strong feeling that there was going to be major drama validating results at the end. Seeing the haphazard nature of the rules in this competition probably affected motivation of many competitors and prevented others from participating altogether. Many valid points have been made about what should be done and what is fair. Obviously the legal frameworks around AI and data rights in training deep-learning models e.t.c are not mature, but that said, there are some glaring missteps by Kaggle and Facebook (1) taking the Kaggle community for granted by not thinking about the use of external data comprehensively enough (many points made) do we seek consent from the cats and dogs in imagenet as well? (2) taking the Kaggle community for granted by believing they can haphazardly enforce arbitrary contrived rational for whatever judgment they arrive at. Facebook has nothing to lose here but Kaggle should have the interest of the Kaggle community at heart. (3) taking the Kaggle community for granted by not responding to legitimate questions and concerns for clarification throughout the duration of the competition. To cut them some slack this challenge is not an easy one to setup because of all the data rights issues e.t.c and the problem itself is a Google or Facebook scale problem. My problem is the nitpicky nature of what we have witnessed here. In their internal efforts to detect deep fakes I am %100 sure they are violating people’s data rights if they apply the same arbitrary rules they offered as the reason for the disqualification.", "890000": "> \"“All of the final five winning teams were held to this same standard.”\n\nYes, I am sure they were \n\n\n", "891696": "I just read https://www.theregister.com/2020/06/18/facebook_deepfake_kaggle_contest/\n\nIt reads:\n\n> Facebook and Kaggle allowed them to resubmit their machine-learning model without any training from external datasets. This bumped them down to seventh place, and they narrowly missed out collecting anything from the prize pot.\n\nI thought the 7th rank solution was another submission, not a retraining of the top solution with less data. ", "893495": "Next time, I'd like to suggest to @titericz, @yifanxie, @haqishen, @anokas and @garybios to take the team under the \"Cambridge Analytica Faces Matter\" name. This way Facebook will not bother you about license, privacy or any topic regarding the content to any purpose.", "887003": "I finally read the rules. From @juliaelliott comment, this seems to be the reason why the team was disqualified.\n\n> A. If any part of the submission documentation depicts, identifies, or includes any person that is not an individual participant or Team member, you must have all permissions and rights from the individual depicted, identified, or included and you agree to provide Competition Sponsor and PAI with written confirmation of those permissions and rights upon request.\n\nIt means that \"submission documentation\" includes all data used to train models. I frankly doubt this would stand in court. But I am not a lawyer.", "884483": "Surprised that no one has cited Elon Musk yet. \"Facebook sucks\"(c).", "884138": "To be honest, I'm not surprised that Kaggle bogged down in front of the giant that is Facebook. \nThe sad part is that they've broken the trust of the Kaggle community, and the don't even seem to be apologetic about it !!\n\nI agree with @kazanova here. We are NOT lawyers. We are here to learn data science, not law ! It's the responsibility of the Kaggle team to ensure we are made aware of the Do's & Dont's in a clear fashion.", "883697": "How `Winning Submission Documentation` is related to the data that you used? Isn't it just a document describing your approach?\n\nKaggle not providing any clarification on the external data applicability besides \"read the rules\" was ugly.", "883866": "Thank you for the transparency, sharing the statement.\nIt looks like there was a twisting of rules/the restrictions weren't made 100% clear, even though the team very meticulously engineered the solution and it appears followed the rules;\nWe all can only imagine the frustrations and empathise with the real winning team. 😞 ", "883707": " This is totally unfair.\n\nOne needs a degree in Law to interpret the meaning of all the rules exactly or kaggle should start providing a lawyer assistant to each competitor. \n\nI guess some faces had Masked On them \n", "883881": "Feeling very sorry for the \"All Faces Are Real\" team. I can't even imagine what they must be going through. If Facebook is being too strict due to lawyers, Kaggle should pool in the prize money from their side and declare 2 winners as \"All Faces Are Real\" team is the real winner here. ", "883727": "This sounds similar to a standard IRB (institutional review board) human subjects research issue to me. Even if there's a public dataset, you have to get permission from the people in that dataset to use it. However, I had asked much earlier in the challenge if Facebook had gone through any sort of IRB process for this dataset, and they said no (https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122113). If they are taking this stance from an IRB perspective, then it's out of the blue.", "883715": "> Specifically, we were asked to provide \"additional permissions or licenses from individuals appearing in [our] external dataset\".\n\nThis reminds me of all those non-sense restrictions written in smallest font size on the medical insurance policy. Maybe a J.D degree will be prerequisite to win a competition in the near future. ", "883710": "btw I have deja vu\nGiba was disqualified from the zillow competition because he worked at airbnb...\nhttps://www.kaggle.com/c/zillow-prize-1/discussion/45770", "892714": "I know it is a lot to ask after this - but please - don't lose faith and stay on Kaggle! Kaggle community needs you to keep the bar high.\n\nI was shocked when I saw this story, and I feel for you guys. For me, a bare minimum would be to allow you to retrain your sub without ***doubtful data*** (I am putting asterisks because it does not seem to me that the data you used violated competition rules)", "883694": "I have a feeling that the Kaggle team sometimes not take enough efforts to think about the need of competition partitioner. For example, I remember there is a UI change for the kernel at the last few days of an competition. I think there is a lot of thing Kaggle can do to avoid such frustrating result", "888733": "I think we need a new competition: \"Predict when a Deep Fake Challenge is Fake\"", "885606": "Sorry to see this, Giba! Organizers should have made rules clear before the competition started.", "885301": "After some thoughts, here's my perspective (disregarding my earlier, snarkier comments):\n\nThe principal entity behind this whole competition was Facebook - a company notorious for not giving a damn (sorry for language) about privacy of the individuals who use its platform, as evidenced by what happened with Cambridge Analytica a few years prior.\n\nFacebook probably wants to avoid using the data of any third-party individual as it would most likely be attacked a lot, not so much by the affected entity/individual, but more so by the people who dislike these big Silicon Valley companies and want to break them down (it would make a good election premise).\n\nAs such, this is probably an attempt by Facebook to prevent any sort of damage to its image after all that it has been through. It may seem a bit too much (it most definitely is a bit too much) because Facebook must be trying to oppress anything which could risk even the slightest amount of damage to the company's image post-Cambridge Analytica, or, as @aakashnain said, it could be contempt between Google (which owns Kaggle) and Facebook, or it could be some combination of both.\n\nEither way this is highly unjust and unfair to the two top teams who worked so hard in this competition - I truly believe Giba, Mikel, Gary, Qishen and Yifan should have won 1st place (no contempt towards Selim Serfebekov - he too deserves GM tier). So regardless of what happened, I think the entire community will regard All Faces are Real as the true winners.", "884406": "At this rate, we're going to have lawsuits with headings, \"Kaggle Competitors vs. Facebook\"!! After the Cambridge Analytics scandal, they lost the public's trust, and now they've lost ours. ", "884465": "As a matter of fact, there have already been several news articles on this.\n\nThe irony is that NONE of them mention these unfortunate incidents - your team has used a lot of time and effort and yet this gets swept under the rug.", "888060": "\nI can imagine how the team `All Faces are Real `felt about it. It must have been a harrowing month! \n\nI have organized few hackathons and (*its a personal opinion*) that *external datasets* are a big legal black hole. But it also reflects the reality of working in data science.... where the maximum benefit/ ROI comes only from mixing & matching data from a number of different sources. \n\nSo I had a suggestion for a new feature or a widget (for a lack of a better word)\n\nIt should be an interface which will allow participants to enter a link & name of the external dataset. Once given they can be analyzed for suitability for the competition. This will create a green-list of dataset & highlight ones which cannot be used. So whenever a new dataset is submitted, it automatically gets checked with the green-list & there are no gray areas in the competition. \n\nIt will remove the current in-efficient means of submitting the information through a message on the forum thread. I have seen Julia and other organizers trying to respond to the same queries again & again. Also for the participant it gives an easy interface to check for green-listed datasets. \n\n\nSo that this suggestion gets highlighted and the Kaggle team can evaluate it, I am marking them - \n\n@antgoldbloom & @mrisdal @juliaelliott \n", "885005": "This is truly unacceptable. 🙁 ", "884718": "My suggestion: what if changing future competitions rules so that competitors have to submit external datasets for approval of Kaggle team during competition. This way there will be no risk of disqualification after competition ends.\n", "884371": "I'm sad about what happened to you. It's absolutely absurd. No sane person can figure out such vague rules. As Selim mentioned in one of his replies, he did ask about CC-BY videos multiple times, but he didn't receive any reply from the hosts or the competition organizers. Kaggle has set a precedent that is going to create further confusion in upcoming competitions. \n\nAlso, I'd like to ask you, are you planning to share your solution code with us? ", "891631": "[Facebook's $500k deepfake-detector AI contest drama: Winning team disqualified on buried consent technicality](https://www.theregister.com/2020/06/18/facebook_deepfake_kaggle_contest/) \n\nJust saw this, may help to get accountability of the host team. Way to go!!", "885049": "This doesn't sound very fair", "884835": "This is sad !\n\nEven with in-depth legal training, you may not get by with Kaggle's unclear rules.\n\nThe most unfortunate thing is they do not really answer when asked to clarify this or that part of the rules.\n\nIt reminds me of when they arbitrarily removed competitors from LB after 4 months of hardwork during the Zillow competition because they put on their profiles they were working at Real State Companies (and regardless the job they were doing there) ", "884107": "This competition will go down, as Franklin D. Roosevelt said, \"a day which will live in infamy.\" (dear President, please do not copyright this)\n\nNext time we use backprop, remind me to consult Geoff Hinton beforehand.....", "883960": "Extra licenses?? Are there actually competitions where participants produce extra licenses?\n\nSorry for what has happened. I think the rule-makers are to blame for this...", "883906": "My condolences. This sudden requirement was unexpected and the rules were very vague. I'm sad that this happened.", "884343": "I lost 1st place in Liverpool due to a simple leak, and Kaggle understated it in the Recap topic there. It was a mess there. Another story: I just requested inversion (Kaggle staff) to see which tasks were solved by each team in Abstract Reasoning, but he didn’t respond although he disclosed how many teams solved each task (I asked because it can be the evidence for private sharing among top teams). And now this. Too many disappointments. ", "915968": "That is so unfair. Kaggle usually makes their rules and guidelines pretty straight forward. But this is odd. It is so disappointing when you don't get what you deserve after putting so much efforts. I suggest you inform this to the team at kaggle so that it never happens again.", "885165": "There is still another team disqualified from top 5 and they have been quiet. What is their problem and their perspective on this?", "895434": "I don't know what these bots are here to achieve, but they are definitely keeping the topic and the questions raised hot! :)", "891911": "That’s another misleading decision by Kaggle. I think we may group and start to create another platform and stay away from here. That was very unfair with the team who worked hard and now is disqualified with a pretty unclear statement. Sorry for you guys and let’s move on.", "888979": "That's really bad and unfortunate =/\nThis competition was really confusing.. If they didn't want any external data usage just explicitly say it in the rules. But instead we got a game of words and our questions were answered vaguely or not answered at all. ", "883942": "Let's support Trump to defund Facebook, Twitter, etc!", "917355": "Deepfake can be used in lots of ways. Hope this can be promoted and to be used in biotech.", "884516": "I felt bad for you for losing 1st position in this very unfortunate way, but congratulations for Gold and being first Kaggler to get 50+ competition golds.", "898361": "Hmmmm so bots from fb, twitter and ig also want to become data-scientist ,no doubt this field is very competitive .", "936127": "Finally, all the bot comments are deleted.", "891703": "Have there been any other competitions where an in-the-money team was disqualified for non-cheating reasons?", "886338": "I would like to add that kaggle is selective in the enforcement of the rules. They might be following the host guidelines in that, but I think this is not fair to us. In the open images competition the winning solution disclosed its external sources after the deadline. It might have been that Google are more relaxed about the rules (and it was a small prize) but for us, the competitors, this should be enforced by kaggle to ensure level ground. \n", "909731": "Amazing FAIR I guess. However, how can we optimize fronters between law policies and AI human well-being?\n", "884809": "Dear all, \n\nI have published a separate thread here trying to address the problems - https://www.kaggle.com/c/deepfake-detection-challenge/discussion/158272 . ", "961432": "Gray area of rules attacks again!", "904170": "As we learn, we will better understand what can be done with these bots", "889389": "Not wanting to hijack this thread in any way, but I contemplated using YouTube videos as well but dropped the idea as it seems YouTube eula (terms of service) \"discourage\" downloads. It is a very difficult and convoluted agreement to read and understand so i dropped the idea. ", "889085": "I’d like to attempt to clarify the underlying issue with All Faces are Real’s disqualified submission. Some of the videos/images used in the disqualified submission were mis-licensed, in that they contained content belonging to other third parties (such as CNN), but were inappropriately offered under open source licenses. This content also clearly depicted third parties and used third party data whose permissions had not been obtained, in violation of the competition rules.\n\nWe cannot overlook holding users responsible to the competition-specific rules for the data brought into this competition. All of the final five winning teams were held to this same standard. Likewise, Facebook created the dataset for this competition with consenting actors specifically to ensure fair data use practices and avoid the risk of using external sources for which rights could not be guaranteed.\n\nRetrospectively, we recognize that the \"submission documentation\" rule’s application to include external data could have been reinforced. It was not our expectation that this was unclear, given the inclusion of external data as part of that documentation. Unfortunately, the specific videos' contents and mislicensing were not anticipated in order to know this would arise as an issue. However, we now acknowledge this was a source of misunderstanding. \n\nWe absolutely could have done better.\n\nWe will be taking steps to increase the clarity and host responsiveness around external data use and their competition-specific rules interpretation generally, especially when there is ambiguity. In fact, we’ve begun proactively raising the need for heightened forum engagement with current and prospective hosts. \n\nKnow that every host will have varied levels of risk tolerance and code scrutiny. While it’s realistically not possible for all questions to receive a response, we commit to better alignment between the forum response level and the level of scrutiny and enforcement expected by a host. We appreciate your constructive recommendations, including the possibility of a community advisory board, and will explore which make sense to implement.\n\nWithout our community, Kaggle would cease to exist. Our hosts will tell you that we consistently default to standing and siding with our users. We will continue to advocate for our community and commit to not allowing this to happen again.\n", "917439": "Sometimes I think why they make such guidelines when they can make it pretty straight forward great work guys \nall the best for upcoming competitions.", "915972": "Using external datasets for modeling this was a cunning move. I believe such an attitude and techniques are expected from a real-life data scientist. Great work!", "914348": "Great job, congratulation for the efforts you have made, you deserve a lot and the best", "891502": "silly world", "893614": "Nevertheless, Keep competing on kaggle. Kaggle community runs because of many people but such grand masters who post their great solutions that helps many beginners to start with. Your time will come.\n\nRead the solution and didn't agreed with reasons provided for the disqualification. Keep up the hope! ", "883799": "Thank you for sharing your perspective and inviting this dialogue. This competition was unique in many ways, from the topic it tackled to its design and the host-defined rules. At Kaggle, we spend a great deal of time attempting to foresee possible issues and guide our hosts and users alike towards positive outcomes. However, we recognize this situation was not as clear as it needed to be.\n\nI am obliged to remind participants of the importance of reviewing the rules for each and every competition you wish to enter in its entirety, as the rules form a binding legal agreement between you and the competition sponsor. In particular, hosts call out their “Competition-Specific Rules” in Section A. That said, this experience reminds us that we can better advocate for improving participants’ understanding of these rules.\n\nFor the time and effort every user invests in contributions on Kaggle, we owe you the commitment to minimizing barriers to achieving what we’re actually here for: doing machine learning work and fostering a community for sharing that work. We have learned a great deal from this and will work with our hosts to prioritize communication and rules clarity.\t", "886350": "I have a very genuine concern. Hope to be clarified.\n\nAs shown in 1 comment below, the 2nd removed team used FaceForensics (FF) dataset. This dataset consists of individual youtube videos with given links. FF is the best deepfake dataset we can find online. Its quality and quantity of videos are much better than any deepfake dataset we can find. It alone can significantly boost score.\n\nDuring the course of competition, squeezing improvement was extremely hard, except using external data. Maybe some teams will find ways to use external data, and some teams determined not to do that. It seems first 5 teams (after removal) don't use any external data at all?\n\nSo, basically maybe 1st removed team also did this (use FF) by mentioning “youtube videos”? And FF is disallowed clearly from the beginning. I’m curious what “youtube videos” are? Please correct me if I’m wrong. ", "910325": "That's a cool topic!", "887444": "That's too bad. Don't worry, I'm sure there'll be another chance to showcase your skills. Please keep the torch in your heart alight.", "885863": "great!", "889471": "great job!", "894041": "will love to connect with more people here ", "905258": "Yes Exactly", "902553": "Good transparency", "901794": "dropped comment just to complete kaggle task.\n\nbut this one is really cool though. i'm gonna give it a try.", "892093": "wow! This is great 👍 ", "886247": "Nice to be work on. I will give it a try :)", "884016": "great", "2150658": "pandas or feature engine method,which is generally more prefered for endTial Imputer,For both numerical and categirical data,i fyou can explain,\n if NA dominant then i)feature engine for numerical data. .aand ii) if,miissing data for arbitrary imputation for categorical data,but wether to choose pandas method or feature engine method or base don largeness of dataset and features KnnImputer is directly applied.?,kindly help me with these..!!!", "918080": "ohh,Deepfake", "918705": "", "918577": "", "918354": "", "918212": "", "918090": "", "917734": "", "917534": "", "917349": "", "917328": "", "917240": "", "916648": "", "916154": "", "916143": "", "916097": "", "915591": "", "915440": "", "915030": "", "914939": "", "914349": "", "914055": "", "913991": "", "913569": "", "913565": "", "913524": "", "913265": "", "913201": "", "912813": "", "912557": "", "912509": "", "911397": "", "911060": "", "910915": "", "910768": "", "909937": "", "909880": "", "909634": "", "908189": "", "907943": "", "907821": "", "907758": "", "907751": "", "907601": "", "907089": "", "906874": "", "906707": "", "906598": "", "906236": "", "905793": "", "905139": "", "904338": "", "904057": "", "903851": "", "903849": "", "903482": "", "903436": "", "903294": "", "902953": "", "902802": "", "902799": "", "902783": "", "902777": "", "899431": "", "898842": "", "898416": "", "898182": "", "898088": "", "897770": "", "897157": "", "897031": "", "897024": "", "896868": "", "896851": "", "896445": "", "896305": "", "896005": "", "895699": "", "895455": "", "895411": "", "895367": "", "895014": "", "894634": "", "894509": "", "894447": "", "892558": "", "892284": "", "892209": "", "891936": "", "891827": "", "891589": "", "891523": "", "891518": "", "889877": "", "889850": "", "889516": "", "888984": "", "888855": "", "888441": "", "888411": "", "888223": "", "885038": "", "884812": "", "884496": "", "884440": "It's a pity, it's not easy to be the first, must consuming lots of energy. But still marvellous! I respect you!!👍 ", "884421": "thank you for share!", "916268": "Great ! Thanks for sharing 👌 ", "900763": "Thanks for sharing", "884659": "Thanks for sharing !! Nice work!!!" } }