![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4218673%2Fd59c60112a9f0dc14a48d346f9c0fc89%2Fdlpaibcwsy.mp4.png?generation=1586212515405190&alt=media\")
Submissions for the Deepfake Detection Challenge closed on March 31st at 11:59pm UTC. First of all, thank you for your participation in building innovative new technologies to help detect deepfakes. As detailed on the Getting Started page here's what you can expect for the next few weeks:\n- Facebook is working with Kaggle to export the code for your 2 selected submissions to our local servers, where we'll re-run your code submissions against a privately held test set. This private set has been created by a reduced closed set of engineers from Facebook who did not participate in the competition.\n- The private test set includes:\n - Synthetic videos with a similar format and nature as the training and public validation/test sets (generated with the same methods used to create the training dataset)\n - Organic videos with and without deepfakes\n- With the predictions generated by this private re-run, Kaggle will calculate your scores on the private leaderboard.\n- We'll reveal those scores on the private leaderboard in ~3 weeks time.\n- Those in the winners circle will have their code inspected by Facebook to validate that no violations have been committed. As a reminder, to be eligible for the challenge prizes, winners must agree to open-source their work so others in the research community can benefit. Entrants will retain rights to their models trained on the training dataset.\n- We'll officially announce the final winners in May!
\n\nKeep tuned!
", "messages": [ { "id": 794298, "postDate": "2020-04-01T18:10:06.827Z", "content": "Submissions for the Deepfake Detection Challenge closed on March 31st at 11:59pm UTC. First of all, thank you for your participation in building innovative new technologies to help detect deepfakes. As detailed on the Getting Started page here's what you can expect for the next few weeks:\n- Facebook is working with Kaggle to export the code for your 2 selected submissions to our local servers, where we'll re-run your code submissions against a privately held test set. This private set has been created by a reduced closed set of engineers from Facebook who did not participate in the competition.\n- The private test set includes:\n - Synthetic videos with a similar format and nature as the training and public validation/test sets (generated with the same methods used to create the training dataset)\n - Organic videos with and without deepfakes\n- With the predictions generated by this private re-run, Kaggle will calculate your scores on the private leaderboard.\n- We'll reveal those scores on the private leaderboard in ~3 weeks time.\n- Those in the winners circle will have their code inspected by Facebook to validate that no violations have been committed. As a reminder, to be eligible for the challenge prizes, winners must agree to open-source their work so others in the research community can benefit. Entrants will retain rights to their models trained on the training dataset.\n- We'll officially announce the final winners in May!
\n\nKeep tuned!
", "rawMarkdown": "Submissions for the Deepfake Detection Challenge closed on March 31st at 11:59pm UTC. First of all, thank you for your participation in building innovative new technologies to help detect deepfakes. As detailed on the [Getting Started page](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started) here's what you can expect for the next few weeks:\n- Facebook is working with Kaggle to export the code for your 2 selected submissions to our local servers, where we'll re-run your code submissions against a privately held test set. This private set has been created by a reduced closed set of engineers from Facebook who did not participate in the competition.\n- The private test set includes:\n - Synthetic videos with a similar format and nature as the training and public validation/test sets (generated with the same methods used to create the training dataset)\n - Organic videos with and without deepfakes\n- With the predictions generated by this private re-run, Kaggle will calculate your scores on the private leaderboard.\n- We'll reveal those scores on the private leaderboard in ~3 weeks time.\n- Those in the winners circle will have their code inspected by Facebook to validate that no violations have been committed. As a reminder, to be eligible for the challenge prizes, winners must agree to open-source their work so others in the research community can benefit. Entrants will retain rights to their models trained on the training dataset.\n- We'll officially announce the final winners in May!\n\nKeep tuned!\n", "votes": 27 }, { "id": 794385, "postDate": "2020-04-01T19:11:50.983Z", "content": "Exciting! Out of interest, can you tell us the relative proportion of synthetic vs organic videos?
", "rawMarkdown": "Exciting! Out of interest, can you tell us the relative proportion of synthetic vs organic videos?", "votes": 8 }, { "id": 815834, "postDate": "2020-04-21T21:47:33.507Z", "content": "Update: We've pushed the expected private code re-run completion date to April 24th, as we're needing a few extra days to complete the re-runs. We'll post an update when the tentative private leaderboard (pending full code & documentation validation by the host) is revealed.
", "rawMarkdown": "**Update**: We've pushed the expected private code re-run completion date to April 24th, as we're needing a few extra days to complete the re-runs. We'll post an update when the tentative private leaderboard (pending full code & documentation validation by the host) is revealed.", "votes": 6, "replies": [ { "id": 816406, "postDate": "2020-04-22T10:52:38.697Z", "content": "Hi, have you adjusted the number of teams recently, since yesterday? Saw that I droped in the medalzone but thats maybe because of less teams and than a tighter medalzone. Is the PL medelzone calculated based on the original total of teams? Now my result will certainly differ against LB but good to know.
", "rawMarkdown": "Hi, have you adjusted the number of teams recently, since yesterday? Saw that I droped in the medalzone but thats maybe because of less teams and than a tighter medalzone. Is the PL medelzone calculated based on the original total of teams? Now my result will certainly differ against LB but good to know." }, { "id": 816893, "postDate": "2020-04-22T17:26:47.440Z", "content": "Private scoring is actively happening. You may expect to see some shuffling/changes in the leaderboard appearance during this process. It's not to be considered final until we announce that it is. There may be some continuous changes in total numbers of teams since the competition close, as we work through rules violators' eliminations.
", "rawMarkdown": "Private scoring is actively happening. You may expect to see some shuffling/changes in the leaderboard appearance during this process. It's not to be considered final until we announce that it is. There may be some continuous changes in total numbers of teams since the competition close, as we work through rules violators' eliminations.", "votes": 1 }, { "id": 816900, "postDate": "2020-04-22T17:35:42.193Z", "content": "So Public LB shows both the public and private or a combo during the update process? Or is the shuffling/changes in the Public leaderboard happening due to rules violators' eliminations only?
", "rawMarkdown": "So Public LB shows both the public and private or a combo during the update process? Or is the shuffling/changes in the Public leaderboard happening due to rules violators' eliminations only?", "votes": 1 }, { "id": 818908, "postDate": "2020-04-24T07:45:33.003Z", "content": "I would recommend organizers put their data on kaggle and re-run the submitted notebooks again. The results are really unbelievable.
", "rawMarkdown": "I would recommend organizers put their data on kaggle and re-run the submitted notebooks again. The results are really unbelievable.", "votes": 2 } ] }, { "id": 852654, "postDate": "2020-05-18T15:31:54.093Z", "content": "Nearly a month already since deadline but this comp still cannot be concluded.
", "rawMarkdown": "Nearly a month already since deadline but this comp still cannot be concluded.", "votes": 3, "replies": [ { "id": 852841, "postDate": "2020-05-18T18:01:03.777Z", "content": "I guess they are checking the code thoroughly? 🤔
", "rawMarkdown": "I guess they are checking the code thoroughly? 🤔 ", "votes": 1 } ] }, { "id": 823325, "postDate": "2020-04-27T15:14:56.507Z", "content": "@cristiancanton I would hope that you will be able keep even the private testset on a benchmark server or something or release it as part of the full dataset for the broader community to work with ... I'm have a feeling this would be the imagenet of forensics ;)
", "rawMarkdown": "@cristiancanton I would hope that you will be able keep even the private testset on a benchmark server or something or release it as part of the full dataset for the broader community to work with ... I'm have a feeling this would be the imagenet of forensics ;)", "votes": 1 }, { "id": 797153, "postDate": "2020-04-04T08:44:42.587Z", "content": "I’m asking a basic question, please somebody explain what is “organic real/fake video”?
", "rawMarkdown": "I’m asking a basic question, please somebody explain what is “organic real/fake video”?", "votes": 1, "replies": [ { "id": 797749, "postDate": "2020-04-04T21:31:59.920Z", "content": "The videos in the competition training set were synthetic, meaning they were composed and recorded for this competition. Organic means \"real\" videos from YouTube /Facebook (probably ones that were removed) made by many different people using different methods. Organic is the opposite of synthetic.
", "rawMarkdown": "The videos in the competition training set were synthetic, meaning they were composed and recorded for this competition. Organic means \"real\" videos from YouTube /Facebook (probably ones that were removed) made by many different people using different methods. Organic is the opposite of synthetic. ", "votes": 2 } ] }, { "id": 794516, "postDate": "2020-04-01T21:54:19.200Z", "content": "Great work đź‘Ť
", "rawMarkdown": "Great work đź‘Ť ", "votes": 1 }, { "id": 800278, "postDate": "2020-04-07T08:35:59.330Z", "content": "\n\nI have a question regarding the inclusion of synthetic videos in the private test set:
\n\nWill the recently announced synthetic videos in the private test set be inspected/chosen in a way to avoid inclusion of synthetic videos whose properties have virtually no resemblance to “real” organic deepfakes?
\n\nBackground
\n\nTo my understanding the information that the private test set will include synthetic videos (and not exclusively real “organic” videos) was disclosed entirely after the submission deadline.
\n\nDuring the competition the information given was that the evaluation would be made entirely on \"real, organic videos with and without deepfakes\".
\n\nIn the instructions on the Getting Started page, it says:
\n\n\n\n\n“Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.”
\n
Also, towards the end of the competition the exact above information was quoted in your post Reminder from the organizers.
\n\nAside the apparent possible difference between described evaluation in instructions during the competition and actual evaluation afterwards, I see another problem with an evaluation in part based on synthetic videos (given that the selection of synthetic videos would directly resemble the overall DFDC training set):
\n\nThe DFDC training set includes a fair proportion of videos labeled as fake but whose synthetic manipulation obviously has little, if any, resemblance to what could be considered the properties of a real “organic” audio- and/or video-deepfake. In relation to the competition objective to identify real organic deepfakes this could be viewed as problematic.
\n\nSuch examples in the DFDC training set include videos labeled as “fake” where the manipulation has been done, not on the actors themselves but on “thin air” beside the actors' bodies (see picture example below). Other examples include \"fake\" videos in the training set with close to zero difference (both audio and video) to their corresponding original videos. A third example is videos so dark that virtually nothing can be seen without ridiculously heavy brightness adjustment.
\n\nFor example, if included as basis for evaluation, should a video similiar to \"dlpaibcwsy.mp4\" be considered a deepfake? The tilted “fake” square to the left of the actor never gets even near the actors face region throughout the video. A strange special effect, sure. But how good does it serve as basis for deepfake detection evaluation?
\n\nFake (first frame of dlpaibcwsy.mp4 from training part 0)\n
Original (first frame of fopjiyxiqd.mp4 from training part 0)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4218673%2F0e273c093bb8d6b8e9ac47f9032d26be%2Ffopjiyxiqd.mp4.png?generation=1586212516017440&alt=media\")
I am of course in no position to object to the organizers decision to include synthetic videos in the private test set. In fact I can see good motives for doing so. However, my opinion is that studying the winning submissions will be a much more interesting subject if the evaluation, as far as possible – with or without inclusion of synthetic videos, is done in a way relevant for the overall objective, i.e. to identify real world “organic” deepfakes.
\n\nA broad inclusion of synthetic videos, without any kind of selection, could obviously have the undesirable effect of benefitting submissions over-adjusted towards non representative properties of the training set.
\n\nThus, if possible, I believe synthetic videos in the private test set should be inspected/chosen in a way that avoids inclusion of synthetic videos without any resemblance to “real” organic deepfakes.
\n\nBest regards
", "rawMarkdown": "@cristiancanton \n\nI have a question regarding the inclusion of synthetic videos in the private test set:\n\n**Will the recently announced synthetic videos in the private test set be inspected/chosen in a way to avoid inclusion of synthetic videos whose properties have virtually no resemblance to “real” organic deepfakes?**\n\n______\n**Background**\n\nTo my understanding the information that the private test set will include synthetic videos (and not exclusively real “organic” videos) was disclosed entirely after the submission deadline. \n\nDuring the competition the information given was that the evaluation would be made entirely on \"real, organic videos with and without deepfakes\".\n\nIn the instructions on the [Getting Started page](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started), it says:\n> “**Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard.** It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.”\n\nAlso, towards the end of the competition the exact above information was quoted in your post [Reminder from the organizers](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/133421).\n\nAside the apparent possible difference between described evaluation in instructions during the competition and actual evaluation afterwards, I see another problem with an evaluation in part based on synthetic videos (given that the selection of synthetic videos would directly resemble the overall DFDC training set):\n\nThe DFDC training set includes a fair proportion of videos labeled as fake but whose synthetic manipulation obviously has little, if any, resemblance to what could be considered the properties of a real “organic” audio- and/or video-deepfake. In relation to the competition objective to identify real organic deepfakes this could be viewed as problematic. \n\nSuch examples in the DFDC training set include videos labeled as “fake” where the manipulation has been done, not on the actors themselves but on “thin air” beside the actors' bodies (see picture example below). Other examples include \"fake\" videos in the training set with close to zero difference (both audio and video) to their corresponding original videos. A third example is videos so dark that virtually nothing can be seen without ridiculously heavy brightness adjustment.\n\nFor example, if included as basis for evaluation, should a video similiar to \"dlpaibcwsy.mp4\" be considered a deepfake? The tilted “fake” square to the left of the actor never gets even near the actors face region throughout the video. A strange special effect, sure. But how good does it serve as basis for deepfake detection evaluation? \n\nFake (first frame of dlpaibcwsy.mp4 from training part 0)\n\n\nOriginal (first frame of fopjiyxiqd.mp4 from training part 0)\n\n\n\n\n\nI am of course in no position to object to the organizers decision to include synthetic videos in the private test set. In fact I can see good motives for doing so. However, my opinion is that studying the winning submissions will be a much more interesting subject if the evaluation, as far as possible – with or without inclusion of synthetic videos, is done in a way relevant for the overall objective, i.e. to identify real world “organic” deepfakes.\n\nA broad inclusion of synthetic videos, without any kind of selection, could obviously have the undesirable effect of benefitting submissions over-adjusted towards non representative properties of the training set.\n\nThus, if possible, I believe synthetic videos in the private test set should be inspected/chosen in a way that avoids inclusion of synthetic videos without any resemblance to “real” organic deepfakes.\n\nBest regards\n\n", "votes": 2, "replies": [ { "id": 800402, "postDate": "2020-04-07T11:32:58.810Z", "content": "Couldn't agree more.. No Organic video out there in the real deepfake world has data like the ones above with things moving around
", "rawMarkdown": "Couldn't agree more.. No Organic video out there in the real deepfake world has data like the ones above with things moving around", "votes": 1 } ] }, { "id": 794504, "postDate": "2020-04-01T21:37:29.527Z", "content": "For those asking: Sorry, late submissions will not be possible until the winners have been locked in and the private leaderboard is finalized. Check back in late April/May!
\n\nThe \"end date\" for the competition will continue to be a date into the future (currently April 22nd is our projected date), until late submissions (which don't count towards the leaderboard) open back up.
", "rawMarkdown": "For those asking: Sorry, late submissions will not be possible until the winners have been locked in and the private leaderboard is finalized. Check back in late April/May!\n\nThe \"end date\" for the competition will continue to be a date into the future (currently April 22nd is our projected date), until late submissions (which don't count towards the leaderboard) open back up.", "replies": [ { "id": 794691, "postDate": "2020-04-02T02:36:43.657Z", "content": "Could you open the test videos of public leaderboard, so we can evaluate models locally
", "rawMarkdown": "Could you open the test videos of public leaderboard, so we can evaluate models locally", "votes": 1 }, { "id": 795588, "postDate": "2020-04-02T21:05:20.597Z", "content": "@gaohaibin The host will be releasing the dataset and notify participants when that's available for continued exploration/work.
", "rawMarkdown": "@gaohaibin The host will be releasing the dataset and notify participants when that's available for continued exploration/work.", "votes": 1 }, { "id": 796974, "postDate": "2020-04-04T06:23:29.100Z", "content": "@juliaelliott post declaration of winners.. if some one is able to produce new state of Art results is any thing there for such solutions ?
", "rawMarkdown": "@juliaelliott post declaration of winners.. if some one is able to produce new state of Art results is any thing there for such solutions ?" }, { "id": 809837, "postDate": "2020-04-16T14:09:44.240Z", "content": "Is the private test set going to be available at some point later?
", "rawMarkdown": "Is the private test set going to be available at some point later?" }, { "id": 830790, "postDate": "2020-05-02T22:42:43.507Z", "content": "Is there any update as to when late submissions will open up? Thanks.
", "rawMarkdown": "Is there any update as to when late submissions will open up? Thanks. " } ] }, { "id": 814208, "postDate": "2020-04-20T13:00:01.163Z", "content": "Hi, hope the re-runs went well. Regarding the approximal PL releaseday, is Wednesday still in the loop you think?
", "rawMarkdown": "Hi, hope the re-runs went well. Regarding the approximal PL releaseday, is Wednesday still in the loop you think?" }, { "id": 794430, "postDate": "2020-04-01T20:09:26.880Z", "content": "Great.\nJust wondering if the late submissions will open earlier ..
", "rawMarkdown": "Great.\nJust wondering if the late submissions will open earlier .." }, { "id": 794410, "postDate": "2020-04-01T19:40:34.843Z", "content": "Awesome. Will there be a paper about the testing dataset generation?
", "rawMarkdown": "Awesome. Will there be a paper about the testing dataset generation? ", "replies": [ { "id": 794769, "postDate": "2020-04-02T04:34:26.050Z", "content": "Everything at due time but yes, there will be a paper.
", "rawMarkdown": "Everything at due time but yes, there will be a paper.", "votes": 3 } ] }, { "id": 853092, "postDate": "2020-05-19T00:45:20.173Z", "content": "Just wonder, If there is no late submission, what are we gonna to do with the benchmark?
", "rawMarkdown": "Just wonder, If there is no late submission, what are we gonna to do with the benchmark?", "votes": 2, "isDeleted": true, "replies": [ { "id": 853149, "postDate": "2020-05-19T02:11:23.310Z", "content": "Nothing. :D
", "rawMarkdown": "Nothing. :D " }, { "id": 853164, "postDate": "2020-05-19T02:18:57.030Z", "content": "I don't think the results on private board could be regarded as the final result since nearly 6% competitors got no result caused by env differences between public and private LB. It's said that kaggles' employees are dealing with this. But nearly one month has gone....We got no updates.
", "rawMarkdown": "I don't think the results on private board could be regarded as the final result since nearly 6% competitors got no result caused by env differences between public and private LB. It's said that kaggles' employees are dealing with this. But nearly one month has gone....We got no updates.", "votes": 3 } ] }, { "id": 799304, "postDate": "2020-04-06T11:00:45.590Z", "content": "Hi, can the learderboard keep open when the competion is over?
", "rawMarkdown": "Hi, can the learderboard keep open when the competion is over?", "isDeleted": true }, { "id": 795456, "postDate": "2020-04-02T18:17:23.137Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 794385, "author_name": "James Howard", "author_url": "", "post_date": "2020-04-01T19:11:50.983000", "content": "Exciting! Out of interest, can you tell us the relative proportion of synthetic vs organic videos?
", "votes": 8, "replies": [] }, { "id": 815834, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-04-21T21:47:33.507000", "content": "Update: We've pushed the expected private code re-run completion date to April 24th, as we're needing a few extra days to complete the re-runs. We'll post an update when the tentative private leaderboard (pending full code & documentation validation by the host) is revealed.
", "votes": 6, "replies": [ { "id": 816406, "author_name": "Kirderf", "author_url": "", "post_date": "2020-04-22T10:52:38.697000", "content": "Hi, have you adjusted the number of teams recently, since yesterday? Saw that I droped in the medalzone but thats maybe because of less teams and than a tighter medalzone. Is the PL medelzone calculated based on the original total of teams? Now my result will certainly differ against LB but good to know.
", "votes": 0, "replies": [] }, { "id": 816893, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-04-22T17:26:47.440000", "content": "Private scoring is actively happening. You may expect to see some shuffling/changes in the leaderboard appearance during this process. It's not to be considered final until we announce that it is. There may be some continuous changes in total numbers of teams since the competition close, as we work through rules violators' eliminations.
", "votes": 1, "replies": [] }, { "id": 816900, "author_name": "Kirderf", "author_url": "", "post_date": "2020-04-22T17:35:42.193000", "content": "So Public LB shows both the public and private or a combo during the update process? Or is the shuffling/changes in the Public leaderboard happening due to rules violators' eliminations only?
", "votes": 1, "replies": [] }, { "id": 818908, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-04-24T07:45:33.003000", "content": "I would recommend organizers put their data on kaggle and re-run the submitted notebooks again. The results are really unbelievable.
", "votes": 2, "replies": [] } ] }, { "id": 852654, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-05-18T15:31:54.093000", "content": "Nearly a month already since deadline but this comp still cannot be concluded.
", "votes": 3, "replies": [ { "id": 852841, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-05-18T18:01:03.777000", "content": "I guess they are checking the code thoroughly? 🤔
", "votes": 1, "replies": [] } ] }, { "id": 823325, "author_name": "mjsML", "author_url": "", "post_date": "2020-04-27T15:14:56.507000", "content": "@cristiancanton I would hope that you will be able keep even the private testset on a benchmark server or something or release it as part of the full dataset for the broader community to work with ... I'm have a feeling this would be the imagenet of forensics ;)
", "votes": 1, "replies": [] }, { "id": 797153, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-04T08:44:42.587000", "content": "I’m asking a basic question, please somebody explain what is “organic real/fake video”?
", "votes": 1, "replies": [ { "id": 797749, "author_name": "Moshel", "author_url": "", "post_date": "2020-04-04T21:31:59.920000", "content": "The videos in the competition training set were synthetic, meaning they were composed and recorded for this competition. Organic means \"real\" videos from YouTube /Facebook (probably ones that were removed) made by many different people using different methods. Organic is the opposite of synthetic.
", "votes": 2, "replies": [] } ] }, { "id": 794516, "author_name": "podsyp", "author_url": "", "post_date": "2020-04-01T21:54:19.200000", "content": "Great work đź‘Ť
", "votes": 1, "replies": [] }, { "id": 800278, "author_name": "Richard_U", "author_url": "", "post_date": "2020-04-07T08:35:59.330000", "content": "\n\nI have a question regarding the inclusion of synthetic videos in the private test set:
\n\nWill the recently announced synthetic videos in the private test set be inspected/chosen in a way to avoid inclusion of synthetic videos whose properties have virtually no resemblance to “real” organic deepfakes?
\n\nBackground
\n\nTo my understanding the information that the private test set will include synthetic videos (and not exclusively real “organic” videos) was disclosed entirely after the submission deadline.
\n\nDuring the competition the information given was that the evaluation would be made entirely on \"real, organic videos with and without deepfakes\".
\n\nIn the instructions on the Getting Started page, it says:
\n\n\n\n\n“Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.”
\n
Also, towards the end of the competition the exact above information was quoted in your post Reminder from the organizers.
\n\nAside the apparent possible difference between described evaluation in instructions during the competition and actual evaluation afterwards, I see another problem with an evaluation in part based on synthetic videos (given that the selection of synthetic videos would directly resemble the overall DFDC training set):
\n\nThe DFDC training set includes a fair proportion of videos labeled as fake but whose synthetic manipulation obviously has little, if any, resemblance to what could be considered the properties of a real “organic” audio- and/or video-deepfake. In relation to the competition objective to identify real organic deepfakes this could be viewed as problematic.
\n\nSuch examples in the DFDC training set include videos labeled as “fake” where the manipulation has been done, not on the actors themselves but on “thin air” beside the actors' bodies (see picture example below). Other examples include \"fake\" videos in the training set with close to zero difference (both audio and video) to their corresponding original videos. A third example is videos so dark that virtually nothing can be seen without ridiculously heavy brightness adjustment.
\n\nFor example, if included as basis for evaluation, should a video similiar to \"dlpaibcwsy.mp4\" be considered a deepfake? The tilted “fake” square to the left of the actor never gets even near the actors face region throughout the video. A strange special effect, sure. But how good does it serve as basis for deepfake detection evaluation?
\n\nFake (first frame of dlpaibcwsy.mp4 from training part 0)\n
Original (first frame of fopjiyxiqd.mp4 from training part 0)\n
I am of course in no position to object to the organizers decision to include synthetic videos in the private test set. In fact I can see good motives for doing so. However, my opinion is that studying the winning submissions will be a much more interesting subject if the evaluation, as far as possible – with or without inclusion of synthetic videos, is done in a way relevant for the overall objective, i.e. to identify real world “organic” deepfakes.
\n\nA broad inclusion of synthetic videos, without any kind of selection, could obviously have the undesirable effect of benefitting submissions over-adjusted towards non representative properties of the training set.
\n\nThus, if possible, I believe synthetic videos in the private test set should be inspected/chosen in a way that avoids inclusion of synthetic videos without any resemblance to “real” organic deepfakes.
\n\nBest regards
", "votes": 2, "replies": [ { "id": 800402, "author_name": "WiseLearner", "author_url": "", "post_date": "2020-04-07T11:32:58.810000", "content": "Couldn't agree more.. No Organic video out there in the real deepfake world has data like the ones above with things moving around
", "votes": 1, "replies": [] } ] }, { "id": 794504, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-04-01T21:37:29.527000", "content": "For those asking: Sorry, late submissions will not be possible until the winners have been locked in and the private leaderboard is finalized. Check back in late April/May!
\n\nThe \"end date\" for the competition will continue to be a date into the future (currently April 22nd is our projected date), until late submissions (which don't count towards the leaderboard) open back up.
", "votes": 0, "replies": [ { "id": 794691, "author_name": "Gao Haibin", "author_url": "", "post_date": "2020-04-02T02:36:43.657000", "content": "Could you open the test videos of public leaderboard, so we can evaluate models locally
", "votes": 1, "replies": [] }, { "id": 795588, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-04-02T21:05:20.597000", "content": "@gaohaibin The host will be releasing the dataset and notify participants when that's available for continued exploration/work.
", "votes": 1, "replies": [] }, { "id": 796974, "author_name": "Jaideep", "author_url": "", "post_date": "2020-04-04T06:23:29.100000", "content": "@juliaelliott post declaration of winners.. if some one is able to produce new state of Art results is any thing there for such solutions ?
", "votes": 0, "replies": [] }, { "id": 809837, "author_name": "MohamedMaher", "author_url": "", "post_date": "2020-04-16T14:09:44.240000", "content": "Is the private test set going to be available at some point later?
", "votes": 0, "replies": [] }, { "id": 830790, "author_name": "qwerty", "author_url": "", "post_date": "2020-05-02T22:42:43.507000", "content": "Is there any update as to when late submissions will open up? Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 814208, "author_name": "Kirderf", "author_url": "", "post_date": "2020-04-20T13:00:01.163000", "content": "Hi, hope the re-runs went well. Regarding the approximal PL releaseday, is Wednesday still in the loop you think?
", "votes": 0, "replies": [] }, { "id": 794430, "author_name": "Saurabh7", "author_url": "", "post_date": "2020-04-01T20:09:26.880000", "content": "Great.\nJust wondering if the late submissions will open earlier ..
", "votes": 0, "replies": [] }, { "id": 794410, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-04-01T19:40:34.843000", "content": "Awesome. Will there be a paper about the testing dataset generation?
", "votes": 0, "replies": [ { "id": 794769, "author_name": "Mozaic", "author_url": "", "post_date": "2020-04-02T04:34:26.050000", "content": "Everything at due time but yes, there will be a paper.
", "votes": 3, "replies": [] } ] }, { "id": 853092, "author_name": "", "author_url": "", "post_date": "2020-05-19T00:45:20.173000", "content": "Just wonder, If there is no late submission, what are we gonna to do with the benchmark?
", "votes": 2, "replies": [ { "id": 853149, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-05-19T02:11:23.310000", "content": "Nothing. :D
", "votes": 0, "replies": [] }, { "id": 853164, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-05-19T02:18:57.030000", "content": "I don't think the results on private board could be regarded as the final result since nearly 6% competitors got no result caused by env differences between public and private LB. It's said that kaggles' employees are dealing with this. But nearly one month has gone....We got no updates.
", "votes": 3, "replies": [] } ] }, { "id": 799304, "author_name": "", "author_url": "", "post_date": "2020-04-06T11:00:45.590000", "content": "Hi, can the learderboard keep open when the competion is over?
", "votes": 0, "replies": [] }, { "id": 795456, "author_name": "", "author_url": "", "post_date": "2020-04-02T18:17:23.137000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "794298": "Submissions for the Deepfake Detection Challenge closed on March 31st at 11:59pm UTC. First of all, thank you for your participation in building innovative new technologies to help detect deepfakes. As detailed on the [Getting Started page](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started) here's what you can expect for the next few weeks:\n- Facebook is working with Kaggle to export the code for your 2 selected submissions to our local servers, where we'll re-run your code submissions against a privately held test set. This private set has been created by a reduced closed set of engineers from Facebook who did not participate in the competition.\n- The private test set includes:\n - Synthetic videos with a similar format and nature as the training and public validation/test sets (generated with the same methods used to create the training dataset)\n - Organic videos with and without deepfakes\n- With the predictions generated by this private re-run, Kaggle will calculate your scores on the private leaderboard.\n- We'll reveal those scores on the private leaderboard in ~3 weeks time.\n- Those in the winners circle will have their code inspected by Facebook to validate that no violations have been committed. As a reminder, to be eligible for the challenge prizes, winners must agree to open-source their work so others in the research community can benefit. Entrants will retain rights to their models trained on the training dataset.\n- We'll officially announce the final winners in May!\n\nKeep tuned!\n", "794385": "Exciting! Out of interest, can you tell us the relative proportion of synthetic vs organic videos?", "815834": "**Update**: We've pushed the expected private code re-run completion date to April 24th, as we're needing a few extra days to complete the re-runs. We'll post an update when the tentative private leaderboard (pending full code & documentation validation by the host) is revealed.", "852654": "Nearly a month already since deadline but this comp still cannot be concluded.", "823325": "@cristiancanton I would hope that you will be able keep even the private testset on a benchmark server or something or release it as part of the full dataset for the broader community to work with ... I'm have a feeling this would be the imagenet of forensics ;)", "797153": "I’m asking a basic question, please somebody explain what is “organic real/fake video”?", "794516": "Great work 👍 ", "800278": "@cristiancanton \n\nI have a question regarding the inclusion of synthetic videos in the private test set:\n\n**Will the recently announced synthetic videos in the private test set be inspected/chosen in a way to avoid inclusion of synthetic videos whose properties have virtually no resemblance to “real” organic deepfakes?**\n\n______\n**Background**\n\nTo my understanding the information that the private test set will include synthetic videos (and not exclusively real “organic” videos) was disclosed entirely after the submission deadline. \n\nDuring the competition the information given was that the evaluation would be made entirely on \"real, organic videos with and without deepfakes\".\n\nIn the instructions on the [Getting Started page](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started), it says:\n> “**Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard.** It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.”\n\nAlso, towards the end of the competition the exact above information was quoted in your post [Reminder from the organizers](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/133421).\n\nAside the apparent possible difference between described evaluation in instructions during the competition and actual evaluation afterwards, I see another problem with an evaluation in part based on synthetic videos (given that the selection of synthetic videos would directly resemble the overall DFDC training set):\n\nThe DFDC training set includes a fair proportion of videos labeled as fake but whose synthetic manipulation obviously has little, if any, resemblance to what could be considered the properties of a real “organic” audio- and/or video-deepfake. In relation to the competition objective to identify real organic deepfakes this could be viewed as problematic. \n\nSuch examples in the DFDC training set include videos labeled as “fake” where the manipulation has been done, not on the actors themselves but on “thin air” beside the actors' bodies (see picture example below). Other examples include \"fake\" videos in the training set with close to zero difference (both audio and video) to their corresponding original videos. A third example is videos so dark that virtually nothing can be seen without ridiculously heavy brightness adjustment.\n\nFor example, if included as basis for evaluation, should a video similiar to \"dlpaibcwsy.mp4\" be considered a deepfake? The tilted “fake” square to the left of the actor never gets even near the actors face region throughout the video. A strange special effect, sure. But how good does it serve as basis for deepfake detection evaluation? \n\nFake (first frame of dlpaibcwsy.mp4 from training part 0)\n\n\nOriginal (first frame of fopjiyxiqd.mp4 from training part 0)\n\n\n\n\n\nI am of course in no position to object to the organizers decision to include synthetic videos in the private test set. In fact I can see good motives for doing so. However, my opinion is that studying the winning submissions will be a much more interesting subject if the evaluation, as far as possible – with or without inclusion of synthetic videos, is done in a way relevant for the overall objective, i.e. to identify real world “organic” deepfakes.\n\nA broad inclusion of synthetic videos, without any kind of selection, could obviously have the undesirable effect of benefitting submissions over-adjusted towards non representative properties of the training set.\n\nThus, if possible, I believe synthetic videos in the private test set should be inspected/chosen in a way that avoids inclusion of synthetic videos without any resemblance to “real” organic deepfakes.\n\nBest regards\n\n", "794504": "For those asking: Sorry, late submissions will not be possible until the winners have been locked in and the private leaderboard is finalized. Check back in late April/May!\n\nThe \"end date\" for the competition will continue to be a date into the future (currently April 22nd is our projected date), until late submissions (which don't count towards the leaderboard) open back up.", "814208": "Hi, hope the re-runs went well. Regarding the approximal PL releaseday, is Wednesday still in the loop you think?", "794430": "Great.\nJust wondering if the late submissions will open earlier ..", "794410": "Awesome. Will there be a paper about the testing dataset generation? ", "853092": "Just wonder, If there is no late submission, what are we gonna to do with the benchmark?", "799304": "Hi, can the learderboard keep open when the competion is over?", "795456": "" } }