{ "id": 140878, "title": "Shake up or not shake up ?", "url": "/competitions/deepfake-detection-challenge/discussion/140878", "author_name": "", "post_date": "2020-04-03T15:05:47.011596100Z", "votes": 23, "comment_count": 20, "views": 0, "content": "

First of all, I would like to thank my teammates, especially @harangdev, for their amazing work. In this thread I would like to know your opinion about the possibility of shake up. Here is my thought.

\n\n

There might be a significant shake up in this competition due to

\n\n\n\n

In term of the 2 final submissions: We selected the best public LB solution and a backup one.

\n\n

PS: Our backup solution is mainly selected based on the CV score. We split the train set by folders. We grouped folders based on the similarity of their background (the video scene).

", "messages": [ { "id": "796410", "postDate": "04/03/2020 15:05:47", "content": "

First of all, I would like to thank my teammates, especially @harangdev, for their amazing work. In this thread I would like to know your opinion about the possibility of shake up. Here is my thought.

\n\n

There might be a significant shake up in this competition due to

\n\n\n\n

In term of the 2 final submissions: We selected the best public LB solution and a backup one.

\n\n

PS: Our backup solution is mainly selected based on the CV score. We split the train set by folders. We grouped folders based on the similarity of their background (the video scene).

", "rawMarkdown": "First of all, I would like to thank my teammates, especially @harangdev, for their amazing work. In this thread I would like to know your opinion about the possibility of shake up. Here is my thought.\n\nThere might be a significant shake up in this competition due to\n\n-\tVideo size: we probed the LB to estimate video size of the public test set. The average video size is 12345 KB, which makes us confused because the average video size of the train set is around 6 MB. We needed to probed LB twice using different calculation to confirm this number.\n\n-\t Audio quality: Some of videos of the public test set have no audio and even if we were able to extract audios we were unable to generate the spectrogram for all audios. If I remember well, there are around 25% of videos with no good audios. And the loss for these videos is quite high, let’s say 0.4 compared to 0.2 for the good-audio videos.\n\n-\tFace quality: The face detection confidence score of the train set is around 0.99 and the one of the public test set is around 0.98. \n\nIn term of the 2 final submissions: We selected the best public LB solution and a backup one.\n\nPS: Our backup solution is mainly selected based on the CV score. We split the train set by folders. We grouped folders based on the similarity of their background (the video scene).", "votes": null }, { "id": "796521", "postDate": "04/03/2020 16:54:46", "content": "

I suspect the introduction of organic videos in the privately held test set, some with faces and some without faces, some with audio and others without audio or weird audio will be a source of significant shake up. Majority of the training data had at least 1 face. Models that make the assumption that there will always be a face in the video and did not handle this situation gracefully might experience significant shake down. The impact of audio is questionable because I suspect most teams probably could not get much from using audio. I had to go back and check that I did not just set the probability to 0.5 if there was no face in the video.

", "rawMarkdown": "I suspect the introduction of organic videos in the privately held test set, some with faces and some without faces, some with audio and others without audio or weird audio will be a source of significant shake up. Majority of the training data had at least 1 face. Models that make the assumption that there will always be a face in the video and did not handle this situation gracefully might experience significant shake down. The impact of audio is questionable because I suspect most teams probably could not get much from using audio. I had to go back and check that I did not just set the probability to 0.5 if there was no face in the video.", "votes": null }, { "id": "796574", "postDate": "04/03/2020 17:48:08", "content": "

My team also tried to build an audio model. It worked but we were unable to properly stack it together with the image models. So we did not use the audio model at the end.

", "rawMarkdown": "My team also tried to build an audio model. It worked but we were unable to properly stack it together with the image models. So we did not use the audio model at the end.", "votes": null }, { "id": "796728", "postDate": "04/03/2020 21:21:41", "content": "

I am sure there will be a huge shakeup and here are my reasonings...

\n\n
    \n
  1. Because stable cv was very difficult to achieve, everyone honed their models on the lb. This is usually a sure recipe for shakeup.
    2. \n
  2. The limited number of algorithms employed and the lack of wild organic fakes in the training means models did not generalize well. The setups were similar. I know some teams used deepfakes (or perhaps only real?) from YouTube to generalize (deducting from the external data thread). I tried to do that myself but the process is of questionable legality even though it is under the right license because of the term and conditions of YouTube (and i also ran out of time. I am living on an island connected by a shoestring to the world).
    3. \n
  3. This is a bit of a continuation to 2. Most of the methods used were direct learning by cnn. As such the cnn could learn either to detect real or to detect fake. Detecting real would probably result in very good scires after shakeup, while detecting fake would fail on unseen very different video. I have invested some time in thinking how can I make the model detect real but couldn't come up with anything. Teams that used non cnn methods might have a better chance at that.
    4. \n
\n\n

Anyways, time will tell. My model is a single model, so i think it is more fragile than ensamble so not holding much hope.

\n\n

Good luck to everyone!

", "rawMarkdown": "I am sure there will be a huge shakeup and here are my reasonings...\n\n1. Because stable cv was very difficult to achieve, everyone honed their models on the lb. This is usually a sure recipe for shakeup. \n2. The limited number of algorithms employed and the lack of wild organic fakes in the training means models did not generalize well. The setups were similar. I know some teams used deepfakes (or perhaps only real?) from YouTube to generalize (deducting from the external data thread). I tried to do that myself but the process is of questionable legality even though it is under the right license because of the term and conditions of YouTube (and i also ran out of time. I am living on an island connected by a shoestring to the world).\n3. This is a bit of a continuation to 2. Most of the methods used were direct learning by cnn. As such the cnn could learn either to detect real or to detect fake. Detecting real would probably result in very good scires after shakeup, while detecting fake would fail on unseen very different video. I have invested some time in thinking how can I make the model detect real but couldn't come up with anything. Teams that used non cnn methods might have a better chance at that.\n\nAnyways, time will tell. My model is a single model, so i think it is more fragile than ensamble so not holding much hope. \n\nGood luck to everyone!", "votes": null }, { "id": "796811", "postDate": "04/04/2020 00:13:33", "content": "

How did you get logs?

", "rawMarkdown": "How did you get logs?", "votes": null }, { "id": "796841", "postDate": "04/04/2020 01:25:05", "content": "

They probed... Probing is done by sending a signal through the score. For example, 0.5 for everything if one thing, 1 for everything if another. Very difficult in this competition as number of submissions per day was low.

", "rawMarkdown": "They probed... Probing is done by sending a signal through the score. For example, 0.5 for everything if one thing, 1 for everything if another. Very difficult in this competition as number of submissions per day was low.", "votes": null }, { "id": "796881", "postDate": "04/04/2020 03:10:44", "content": "

Wow, thanks

", "rawMarkdown": "Wow, thanks", "votes": null }, { "id": "796940", "postDate": "04/04/2020 05:24:53", "content": "

There will be, so good luck to your team.

", "rawMarkdown": "There will be, so good luck to your team.", "votes": null }, { "id": "798402", "postDate": "04/05/2020 13:17:10", "content": "

Definitely shakeup should happen since the organizers told they plan to include both organic and synthetic videos into private test. Participants whose solutions were overfitted on public LB (and who didnt use \"organic\" videos in training) may shift down.

", "rawMarkdown": "Definitely shakeup should happen since the organizers told they plan to include both organic and synthetic videos into private test. Participants whose solutions were overfitted on public LB (and who didnt use \"organic\" videos in training) may shift down.", "votes": null }, { "id": "798434", "postDate": "04/05/2020 13:48:27", "content": "

🤝

", "rawMarkdown": "🤝", "votes": null }, { "id": "798491", "postDate": "04/05/2020 14:46:35", "content": "

The background information introduced by large margin size makes the model more robust than models with small margin size (only face). IMO, the only reason for a shake up is a complete different kind of background information.

", "rawMarkdown": "The background information introduced by large margin size makes the model more robust than models with small margin size (only face). IMO, the only reason for a shake up is a complete different kind of background information.", "votes": null }, { "id": "798810", "postDate": "04/05/2020 21:55:01", "content": "

DFDC dataset has a lot of low quality deepfakes. There are good ones but it is a small percentage of train set.

\n\n

Hence the reasons for shake up could be:\n- better visual quality deepfakes (autoencoder trained on 256x256 is much better than 128x128 for example)\n- better face blending (poisson blending, deepblending) see https://arxiv.org/abs/1912.13458\n- better color correction

", "rawMarkdown": "DFDC dataset has a lot of low quality deepfakes. There are good ones but it is a small percentage of train set.\n\nHence the reasons for shake up could be:\n- better visual quality deepfakes (autoencoder trained on 256x256 is much better than 128x128 for example)\n- better face blending (poisson blending, deepblending) see https://arxiv.org/abs/1912.13458\n- better color correction", "votes": null }, { "id": "798814", "postDate": "04/05/2020 21:57:48", "content": "

and those who did will be disqualified😃

", "rawMarkdown": "and those who did will be disqualified😃", "votes": null }, { "id": "798851", "postDate": "04/05/2020 23:10:17", "content": "

It's tricky, as instructions were confusing. I decided not to use any external training data just to be on the safe side. However, imho, downloading videos from YouTube that has the right license should be ok by the competition rules. Not sure it is ok by YouTube terms and conditions. Too complicated!

", "rawMarkdown": "It's tricky, as instructions were confusing. I decided not to use any external training data just to be on the safe side. However, imho, downloading videos from YouTube that has the right license should be ok by the competition rules. Not sure it is ok by YouTube terms and conditions. Too complicated!", "votes": null }, { "id": "799002", "postDate": "04/06/2020 04:21:43", "content": "

You are right. I hope the private test set is really 'similar' to public test set...

", "rawMarkdown": "You are right. I hope the private test set is really 'similar' to public test set...", "votes": null }, { "id": "799398", "postDate": "04/06/2020 12:11:37", "content": "

Public test set is the same as train set but was augmented (resize, compression, subsampling). Has nothing to do with organic videos.

", "rawMarkdown": "Public test set is the same as train set but was augmented (resize, compression, subsampling). Has nothing to do with organic videos.", "votes": null }, { "id": "799672", "postDate": "04/06/2020 16:18:59", "content": "

Agree. Same for us. We did not add any organic videos in our training set because it's quite impossible to know if it was allowed (i.e. from YouTube) or not. I think main contributor for the shake up will come from organic videos and great deep fakes.

", "rawMarkdown": "Agree. Same for us. We did not add any organic videos in our training set because it's quite impossible to know if it was allowed (i.e. from YouTube) or not. I think main contributor for the shake up will come from organic videos and great deep fakes.", "votes": null }, { "id": "804642", "postDate": "04/11/2020 18:59:39", "content": "

@tarobxl How did you managed to get something out of the audio? I tried several models, on the audio, and it didn't help :( Which approach did you follow?

", "rawMarkdown": "tarobxl How did you managed to get something out of the audio? I tried several models, on the audio, and it didn't help :( Which approach did you follow?", "votes": null }, { "id": "804663", "postDate": "04/11/2020 19:24:09", "content": "

We just run the deepfake audio detection benchmark. The main trick is to create the dataset. Only 10% of fake videos have fake audios. And they are duplicated as well.

", "rawMarkdown": "We just run the deepfake audio detection benchmark. The main trick is to create the dataset. Only 10% of fake videos have fake audios. And they are duplicated as well.", "votes": null }, { "id": "804712", "postDate": "04/11/2020 20:46:48", "content": "

@tarobxl Thanks for your reply. How did you identify which videos had fake audio to create the dataset?

", "rawMarkdown": "tarobxl Thanks for your reply. How did you identify which videos had fake audio to create the dataset?", "votes": null }, { "id": "818504", "postDate": "04/23/2020 23:59:15", "content": "

Looks like the correct answer was yes, shake up

", "rawMarkdown": "Looks like the correct answer was yes, shake up", "votes": null } ], "comments": [ { "id": 796521, "author_name": "godaibo", "author_url": "", "post_date": "04/03/2020 16:54:46", "content": "

I suspect the introduction of organic videos in the privately held test set, some with faces and some without faces, some with audio and others without audio or weird audio will be a source of significant shake up. Majority of the training data had at least 1 face. Models that make the assumption that there will always be a face in the video and did not handle this situation gracefully might experience significant shake down. The impact of audio is questionable because I suspect most teams probably could not get much from using audio. I had to go back and check that I did not just set the probability to 0.5 if there was no face in the video.

", "votes": null, "replies": [ { "id": 796574, "author_name": "tarobxl", "author_url": "", "post_date": "04/03/2020 17:48:08", "content": "

My team also tried to build an audio model. It worked but we were unable to properly stack it together with the image models. So we did not use the audio model at the end.

", "votes": null, "replies": [] }, { "id": 804642, "author_name": "ngcferreira", "author_url": "", "post_date": "04/11/2020 18:59:39", "content": "

@tarobxl How did you managed to get something out of the audio? I tried several models, on the audio, and it didn't help :( Which approach did you follow?

", "votes": null, "replies": [] }, { "id": 804663, "author_name": "tarobxl", "author_url": "", "post_date": "04/11/2020 19:24:09", "content": "

We just run the deepfake audio detection benchmark. The main trick is to create the dataset. Only 10% of fake videos have fake audios. And they are duplicated as well.

", "votes": null, "replies": [] }, { "id": 804712, "author_name": "ngcferreira", "author_url": "", "post_date": "04/11/2020 20:46:48", "content": "

@tarobxl Thanks for your reply. How did you identify which videos had fake audio to create the dataset?

", "votes": null, "replies": [] } ] }, { "id": 796728, "author_name": "moshel", "author_url": "", "post_date": "04/03/2020 21:21:41", "content": "

I am sure there will be a huge shakeup and here are my reasonings...

\n\n
    \n
  1. Because stable cv was very difficult to achieve, everyone honed their models on the lb. This is usually a sure recipe for shakeup.
    2. \n
  2. The limited number of algorithms employed and the lack of wild organic fakes in the training means models did not generalize well. The setups were similar. I know some teams used deepfakes (or perhaps only real?) from YouTube to generalize (deducting from the external data thread). I tried to do that myself but the process is of questionable legality even though it is under the right license because of the term and conditions of YouTube (and i also ran out of time. I am living on an island connected by a shoestring to the world).
    3. \n
  3. This is a bit of a continuation to 2. Most of the methods used were direct learning by cnn. As such the cnn could learn either to detect real or to detect fake. Detecting real would probably result in very good scires after shakeup, while detecting fake would fail on unseen very different video. I have invested some time in thinking how can I make the model detect real but couldn't come up with anything. Teams that used non cnn methods might have a better chance at that.
    4. \n
\n\n

Anyways, time will tell. My model is a single model, so i think it is more fragile than ensamble so not holding much hope.

\n\n

Good luck to everyone!

", "votes": null, "replies": [] }, { "id": 796811, "author_name": "azamatk", "author_url": "", "post_date": "04/04/2020 00:13:33", "content": "

How did you get logs?

", "votes": null, "replies": [ { "id": 796841, "author_name": "moshel", "author_url": "", "post_date": "04/04/2020 01:25:05", "content": "

They probed... Probing is done by sending a signal through the score. For example, 0.5 for everything if one thing, 1 for everything if another. Very difficult in this competition as number of submissions per day was low.

", "votes": null, "replies": [] }, { "id": 796881, "author_name": "azamatk", "author_url": "", "post_date": "04/04/2020 03:10:44", "content": "

Wow, thanks

", "votes": null, "replies": [] } ] }, { "id": 796940, "author_name": "khahuras", "author_url": "", "post_date": "04/04/2020 05:24:53", "content": "

There will be, so good luck to your team.

", "votes": null, "replies": [] }, { "id": 798402, "author_name": "konstantinsimonchik", "author_url": "", "post_date": "04/05/2020 13:17:10", "content": "

Definitely shakeup should happen since the organizers told they plan to include both organic and synthetic videos into private test. Participants whose solutions were overfitted on public LB (and who didnt use \"organic\" videos in training) may shift down.

", "votes": null, "replies": [ { "id": 798814, "author_name": "selimsef", "author_url": "", "post_date": "04/05/2020 21:57:48", "content": "

and those who did will be disqualified😃

", "votes": null, "replies": [] }, { "id": 798851, "author_name": "moshel", "author_url": "", "post_date": "04/05/2020 23:10:17", "content": "

It's tricky, as instructions were confusing. I decided not to use any external training data just to be on the safe side. However, imho, downloading videos from YouTube that has the right license should be ok by the competition rules. Not sure it is ok by YouTube terms and conditions. Too complicated!

", "votes": null, "replies": [] }, { "id": 799672, "author_name": "mpware", "author_url": "", "post_date": "04/06/2020 16:18:59", "content": "

Agree. Same for us. We did not add any organic videos in our training set because it's quite impossible to know if it was allowed (i.e. from YouTube) or not. I think main contributor for the shake up will come from organic videos and great deep fakes.

", "votes": null, "replies": [] } ] }, { "id": 798434, "author_name": "jeffzhenyuqiu", "author_url": "", "post_date": "04/05/2020 13:48:27", "content": "

🤝

", "votes": null, "replies": [] }, { "id": 798491, "author_name": "yuanzhezhou", "author_url": "", "post_date": "04/05/2020 14:46:35", "content": "

The background information introduced by large margin size makes the model more robust than models with small margin size (only face). IMO, the only reason for a shake up is a complete different kind of background information.

", "votes": null, "replies": [ { "id": 798810, "author_name": "selimsef", "author_url": "", "post_date": "04/05/2020 21:55:01", "content": "

DFDC dataset has a lot of low quality deepfakes. There are good ones but it is a small percentage of train set.

\n\n

Hence the reasons for shake up could be:\n- better visual quality deepfakes (autoencoder trained on 256x256 is much better than 128x128 for example)\n- better face blending (poisson blending, deepblending) see https://arxiv.org/abs/1912.13458\n- better color correction

", "votes": null, "replies": [] }, { "id": 799002, "author_name": "yuanzhezhou", "author_url": "", "post_date": "04/06/2020 04:21:43", "content": "

You are right. I hope the private test set is really 'similar' to public test set...

", "votes": null, "replies": [] }, { "id": 799398, "author_name": "selimsef", "author_url": "", "post_date": "04/06/2020 12:11:37", "content": "

Public test set is the same as train set but was augmented (resize, compression, subsampling). Has nothing to do with organic videos.

", "votes": null, "replies": [] } ] }, { "id": 818504, "author_name": "ryches", "author_url": "", "post_date": "04/23/2020 23:59:15", "content": "

Looks like the correct answer was yes, shake up

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "796410": "First of all, I would like to thank my teammates, especially @harangdev, for their amazing work. In this thread I would like to know your opinion about the possibility of shake up. Here is my thought.\n\nThere might be a significant shake up in this competition due to\n\n-\tVideo size: we probed the LB to estimate video size of the public test set. The average video size is 12345 KB, which makes us confused because the average video size of the train set is around 6 MB. We needed to probed LB twice using different calculation to confirm this number.\n\n-\t Audio quality: Some of videos of the public test set have no audio and even if we were able to extract audios we were unable to generate the spectrogram for all audios. If I remember well, there are around 25% of videos with no good audios. And the loss for these videos is quite high, let’s say 0.4 compared to 0.2 for the good-audio videos.\n\n-\tFace quality: The face detection confidence score of the train set is around 0.99 and the one of the public test set is around 0.98. \n\nIn term of the 2 final submissions: We selected the best public LB solution and a backup one.\n\nPS: Our backup solution is mainly selected based on the CV score. We split the train set by folders. We grouped folders based on the similarity of their background (the video scene).", "796521": "I suspect the introduction of organic videos in the privately held test set, some with faces and some without faces, some with audio and others without audio or weird audio will be a source of significant shake up. Majority of the training data had at least 1 face. Models that make the assumption that there will always be a face in the video and did not handle this situation gracefully might experience significant shake down. The impact of audio is questionable because I suspect most teams probably could not get much from using audio. I had to go back and check that I did not just set the probability to 0.5 if there was no face in the video.", "796574": "My team also tried to build an audio model. It worked but we were unable to properly stack it together with the image models. So we did not use the audio model at the end.", "796728": "I am sure there will be a huge shakeup and here are my reasonings...\n\n1. Because stable cv was very difficult to achieve, everyone honed their models on the lb. This is usually a sure recipe for shakeup. \n2. The limited number of algorithms employed and the lack of wild organic fakes in the training means models did not generalize well. The setups were similar. I know some teams used deepfakes (or perhaps only real?) from YouTube to generalize (deducting from the external data thread). I tried to do that myself but the process is of questionable legality even though it is under the right license because of the term and conditions of YouTube (and i also ran out of time. I am living on an island connected by a shoestring to the world).\n3. This is a bit of a continuation to 2. Most of the methods used were direct learning by cnn. As such the cnn could learn either to detect real or to detect fake. Detecting real would probably result in very good scires after shakeup, while detecting fake would fail on unseen very different video. I have invested some time in thinking how can I make the model detect real but couldn't come up with anything. Teams that used non cnn methods might have a better chance at that.\n\nAnyways, time will tell. My model is a single model, so i think it is more fragile than ensamble so not holding much hope. \n\nGood luck to everyone!", "796811": "How did you get logs?", "796841": "They probed... Probing is done by sending a signal through the score. For example, 0.5 for everything if one thing, 1 for everything if another. Very difficult in this competition as number of submissions per day was low.", "796881": "Wow, thanks", "796940": "There will be, so good luck to your team.", "798402": "Definitely shakeup should happen since the organizers told they plan to include both organic and synthetic videos into private test. Participants whose solutions were overfitted on public LB (and who didnt use \"organic\" videos in training) may shift down.", "798434": "🤝", "798491": "The background information introduced by large margin size makes the model more robust than models with small margin size (only face). IMO, the only reason for a shake up is a complete different kind of background information.", "798810": "DFDC dataset has a lot of low quality deepfakes. There are good ones but it is a small percentage of train set.\n\nHence the reasons for shake up could be:\n- better visual quality deepfakes (autoencoder trained on 256x256 is much better than 128x128 for example)\n- better face blending (poisson blending, deepblending) see https://arxiv.org/abs/1912.13458\n- better color correction", "798814": "and those who did will be disqualified😃", "798851": "It's tricky, as instructions were confusing. I decided not to use any external training data just to be on the safe side. However, imho, downloading videos from YouTube that has the right license should be ok by the competition rules. Not sure it is ok by YouTube terms and conditions. Too complicated!", "799002": "You are right. I hope the private test set is really 'similar' to public test set...", "799398": "Public test set is the same as train set but was augmented (resize, compression, subsampling). Has nothing to do with organic videos.", "799672": "Agree. Same for us. We did not add any organic videos in our training set because it's quite impossible to know if it was allowed (i.e. from YouTube) or not. I think main contributor for the shake up will come from organic videos and great deep fakes.", "804642": "tarobxl How did you managed to get something out of the audio? I tried several models, on the audio, and it didn't help :( Which approach did you follow?", "804663": "We just run the deepfake audio detection benchmark. The main trick is to create the dataset. Only 10% of fake videos have fake audios. And they are duplicated as well.", "804712": "tarobxl Thanks for your reply. How did you identify which videos had fake audio to create the dataset?", "818504": "Looks like the correct answer was yes, shake up" }, "source": "meta" }