![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F351627%2F127f6d6809a28d6f31db4f6f794794d8%2Fretinaface_license.png?generation=1585709695747954&alt=media\")
Long 4 months have passed. Applaud to participants who have been working hard and thanks to hosts who arranged this dataset and competition.
\nIn social aspect, automatic deepfake detection algorithm will be a must in near future. In personal aspect, this competition was held in a good timing for me. So dfdc took priority in my head for the last 4 months.
\nLuckily, I somehow managed to attain 2nd place in public leaderboard. While waiting for the private leaderboard to be revealed, I'll share my solution.
\nNote that all of the following are what I have done for this competition. (This post does not include what my teammates have done)
\nFor those who are curious, I'll first share the major methods that I came up with, which improved public lb score.
\nExtensive augmentations significantly improved public lb score and reduced cv-lb gap. I guess it makes model robust on varying data.
\nI set margin when cropping face, where new_face_width=face_width*(1+margin) (same with height). Margin of 0.5~0.7 worked significantly better than 0 and further reduced cv-lb gap. I guess model learns to detect some inconsistency between manipulated region and surrounding region.
Tuning augmentations and face margin were the two main ingredients that boosted public lb significantly.
\nEfficientnet-b4 did quite better than efficientnet-b0. With extensive augmentations, appropriate model size improved the score.
\nI used Unet with classification branch for model architecture. I calculated pixel-wise difference between fake and corresponding original video, generated mask with pixels that have outstanding differences, and used it as segmentation part target. By using mask information, the model is able to know which part of the image is modified and I guess it helped boost performance.
\nTrain data and test data have quite different distribution, as can be seen in cv-lb discrepancy. Multiplying constant (<1) to the logit then taking sigmoid helped improve logloss in this situation.
\nEnsemble always helps.
\nI used simple average of frames to get probability, so the more frames extracted from the video, the better, until it hits 9 hours restriction.
\nNow, I'll go into details of my journey. I'll separate it into 10 phases, each of where I concentrated on certain subject. (number inside parenthesis in the title is approximate public lb score at that phase)
\nThis was my first encounter to video type vision task and deepfake detection. So at the very beginning of the competition, I started searching google for articles and papers, and also there were nice posts in kaggle discussion that introduced related papers.
\nAt first, I tried to go with network architectures that take care of temporal information, but I found out that they lack pretrained weights and are very heavy to train, so I decided to make baseline based on XceptionNet which was introduced in faceforensics paper( https://arxiv.org/abs/1901.08971 ).
\nAt my initial experiments, I tried to integrate audio part in the image model by using spectrogram as second input, but it didn't help that much and complicated the process, so I abandoned audio part. Anyway my teammate wanted audio information, so I set up the pipeline which uses ffmpeg with subprocess to extract audio stably in both train and public test videos. (I used PyAv first, but it wasn't stable with public test videos.)
\nReading video was the main bottleneck for the runtime, so I struggled with optimizing the process. At last, I selected method kindly provided at https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122328 which was the fastest among those I tried. I extracted 20 frames evenly. Duration is 10 seconds so I extracted frame every 0.5s.
\nNext step was to select fast but accurate face detector. I searched and found out that retinaface is the best available face detector evaluated on widerface dataset( http://shuoyang1213.me/WIDERFACE/WiderFace_Results.html ). Thereafter, I used its pytorch implementation https://github.com/biubug6/Pytorch_Retinaface .
To save data loading time when training the network, I first processed each original videos and saved them as compressed joblib files at disk. I extract 20 frames from each video, detect face, crop them, and with additional meta information extracted, save them. The saved video will be a numpy array with shape (n_people, n_detected_frames, height, width, 3). There were a lot of trivial issues and bugs when processing videos. I'll introduce some.
After processing all original videos, I used the bounding box information of original videos to process corresponding fake videos. I could use multiprocessing to speed up this process, since it didn't use cuda. After I processed all videos, I updated given metadata with the new extracted information.
\nIt took about 12 hours on my computer to process all of the train dataset. Then I trained the model with processed data.
\nInferencing in Kaggle notebook was another obstacle. Initially my notebook failed many times with varying error messages. I concluded there are some corrupt videos in public test set, so I introduced try except block and the submission went well thereafter.
In the Understanding Cloud Organization competition( https://www.kaggle.com/c/understanding_cloud_organization/discussion ), I learned that multi-task learning with classification and segmentation helps improve model's classification score.
\nI could generate masks by selecting pixels from the fake video that have large differences from corresponding real video. All of the training videos are compressed, so the masks were not perfect, but the strategy worked in my validation set and public test set (about 0.01 improvement in lb).
\nBy using mask information, the model was able to know which part of the image is modified and I guess it helped boost performance.
\nI used UNet with classification branch. Also, since efficientnets are the state of the art architectures in imagenet, I used them as the encoder of Unet.
\nAt this moment, I was using first frame from each videos.
\nThen this huge problem came. My validation score and public lb score didn't correlate and had large gap (0.2 vs 0.6). It was the main problem in this competition from the start to the end. I managed to decrease the gap to ~0.05 at the end, but still don't fully understand how public test data differ from train data.
\nI was only using horizontal flip since I thought augmentation will distort features that were introduced by manipulation. But it was wrong after all. Augmentation made the model robust on public test set.
\nI added some basic augmentation such as shift, scale, rotate, rgbshift, brightness, contrast, hue, saturation, value. Also, referencing dfdc preview paper( https://arxiv.org/abs/1910.08854 ), I added JpegCompression and Downscale augmentation too.
\nLocal validation score improved a bit but the public score jumped massively to 0.4.
\nI was using efficientnet-b0 and I switched to efficientnet-b3. It scored 0.36, then I switched to efficientnet-b7 and it scored 0.33.
By that time I thought the improvement was solely due to model capacity, but some of the improvement was actually due to learning rate - batch size relationship.
\nI thought I had some correlation with cv and lb by this time, so I tried a lot of experiment with local validation. I'll introduce some.
\nconservative fix: multiply constant(<1) to the logits than take sigmoid. It helps improve logloss when the training and testing distributions differ.After testing the tuned model on public lb, I once again found out cv and lb didn't correlate at most experiments I did. I needed to select cv vs lb at this time, and decided to value lb more, since the hosts implied that the private test set will be different than train set. So I started to validate experiments on public lb. Also, I decided to use b5 for my experiment, since I judged b0 had too low capacity when augmentations are applied.
\nThen I remembered I got significant boost when I did more augmentations, so I added extensive augmentations including noise and blur, and increased the degree of augmentations. Score jumped to 0.269. I did harder augmentations but it didn't improve, so I stopped tuning augmentations.
\nI did more experiments, but sadly nothing worked.
\nI had an experience of increasing the face margin from 0.05->0.1, and it seemed to have increased overall public lb score. So I decided to increase face margin more.
\nI experimented face margin of 0.5, and single b5 scored astonishing 0.222. So I increased face margin to 1.0 and started to tune how much I should crop the margins.
\nWith margin crop of 0.15 (margin of 0.7), I could get 0.218, and could get 0.214 with b4.
\nI trained 7 b4s and 3 b5s with different seeds and did simple average. Also I increased nframes to 30 when inferencing. It resulted in my final public lb score of 0.199.
", "messages": [ { "id": 793392, "postDate": "2020-04-01T02:03:10.507Z", "content": "Long 4 months have passed. Applaud to participants who have been working hard and thanks to hosts who arranged this dataset and competition.
\nIn social aspect, automatic deepfake detection algorithm will be a must in near future. In personal aspect, this competition was held in a good timing for me. So dfdc took priority in my head for the last 4 months.
\nLuckily, I somehow managed to attain 2nd place in public leaderboard. While waiting for the private leaderboard to be revealed, I'll share my solution.
\nNote that all of the following are what I have done for this competition. (This post does not include what my teammates have done)
\nFor those who are curious, I'll first share the major methods that I came up with, which improved public lb score.
\nExtensive augmentations significantly improved public lb score and reduced cv-lb gap. I guess it makes model robust on varying data.
\nI set margin when cropping face, where new_face_width=face_width*(1+margin) (same with height). Margin of 0.5~0.7 worked significantly better than 0 and further reduced cv-lb gap. I guess model learns to detect some inconsistency between manipulated region and surrounding region.
Tuning augmentations and face margin were the two main ingredients that boosted public lb significantly.
\nEfficientnet-b4 did quite better than efficientnet-b0. With extensive augmentations, appropriate model size improved the score.
\nI used Unet with classification branch for model architecture. I calculated pixel-wise difference between fake and corresponding original video, generated mask with pixels that have outstanding differences, and used it as segmentation part target. By using mask information, the model is able to know which part of the image is modified and I guess it helped boost performance.
\nTrain data and test data have quite different distribution, as can be seen in cv-lb discrepancy. Multiplying constant (<1) to the logit then taking sigmoid helped improve logloss in this situation.
\nEnsemble always helps.
\nI used simple average of frames to get probability, so the more frames extracted from the video, the better, until it hits 9 hours restriction.
\nNow, I'll go into details of my journey. I'll separate it into 10 phases, each of where I concentrated on certain subject. (number inside parenthesis in the title is approximate public lb score at that phase)
\nThis was my first encounter to video type vision task and deepfake detection. So at the very beginning of the competition, I started searching google for articles and papers, and also there were nice posts in kaggle discussion that introduced related papers.
\nAt first, I tried to go with network architectures that take care of temporal information, but I found out that they lack pretrained weights and are very heavy to train, so I decided to make baseline based on XceptionNet which was introduced in faceforensics paper( https://arxiv.org/abs/1901.08971 ).
\nAt my initial experiments, I tried to integrate audio part in the image model by using spectrogram as second input, but it didn't help that much and complicated the process, so I abandoned audio part. Anyway my teammate wanted audio information, so I set up the pipeline which uses ffmpeg with subprocess to extract audio stably in both train and public test videos. (I used PyAv first, but it wasn't stable with public test videos.)
\nReading video was the main bottleneck for the runtime, so I struggled with optimizing the process. At last, I selected method kindly provided at https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122328 which was the fastest among those I tried. I extracted 20 frames evenly. Duration is 10 seconds so I extracted frame every 0.5s.
\nNext step was to select fast but accurate face detector. I searched and found out that retinaface is the best available face detector evaluated on widerface dataset( http://shuoyang1213.me/WIDERFACE/WiderFace_Results.html ). Thereafter, I used its pytorch implementation https://github.com/biubug6/Pytorch_Retinaface .
To save data loading time when training the network, I first processed each original videos and saved them as compressed joblib files at disk. I extract 20 frames from each video, detect face, crop them, and with additional meta information extracted, save them. The saved video will be a numpy array with shape (n_people, n_detected_frames, height, width, 3). There were a lot of trivial issues and bugs when processing videos. I'll introduce some.
After processing all original videos, I used the bounding box information of original videos to process corresponding fake videos. I could use multiprocessing to speed up this process, since it didn't use cuda. After I processed all videos, I updated given metadata with the new extracted information.
\nIt took about 12 hours on my computer to process all of the train dataset. Then I trained the model with processed data.
\nInferencing in Kaggle notebook was another obstacle. Initially my notebook failed many times with varying error messages. I concluded there are some corrupt videos in public test set, so I introduced try except block and the submission went well thereafter.
In the Understanding Cloud Organization competition( https://www.kaggle.com/c/understanding_cloud_organization/discussion ), I learned that multi-task learning with classification and segmentation helps improve model's classification score.
\nI could generate masks by selecting pixels from the fake video that have large differences from corresponding real video. All of the training videos are compressed, so the masks were not perfect, but the strategy worked in my validation set and public test set (about 0.01 improvement in lb).
\nBy using mask information, the model was able to know which part of the image is modified and I guess it helped boost performance.
\nI used UNet with classification branch. Also, since efficientnets are the state of the art architectures in imagenet, I used them as the encoder of Unet.
\nAt this moment, I was using first frame from each videos.
\nThen this huge problem came. My validation score and public lb score didn't correlate and had large gap (0.2 vs 0.6). It was the main problem in this competition from the start to the end. I managed to decrease the gap to ~0.05 at the end, but still don't fully understand how public test data differ from train data.
\nI was only using horizontal flip since I thought augmentation will distort features that were introduced by manipulation. But it was wrong after all. Augmentation made the model robust on public test set.
\nI added some basic augmentation such as shift, scale, rotate, rgbshift, brightness, contrast, hue, saturation, value. Also, referencing dfdc preview paper( https://arxiv.org/abs/1910.08854 ), I added JpegCompression and Downscale augmentation too.
\nLocal validation score improved a bit but the public score jumped massively to 0.4.
\nI was using efficientnet-b0 and I switched to efficientnet-b3. It scored 0.36, then I switched to efficientnet-b7 and it scored 0.33.
By that time I thought the improvement was solely due to model capacity, but some of the improvement was actually due to learning rate - batch size relationship.
\nI thought I had some correlation with cv and lb by this time, so I tried a lot of experiment with local validation. I'll introduce some.
\nconservative fix: multiply constant(<1) to the logits than take sigmoid. It helps improve logloss when the training and testing distributions differ.After testing the tuned model on public lb, I once again found out cv and lb didn't correlate at most experiments I did. I needed to select cv vs lb at this time, and decided to value lb more, since the hosts implied that the private test set will be different than train set. So I started to validate experiments on public lb. Also, I decided to use b5 for my experiment, since I judged b0 had too low capacity when augmentations are applied.
\nThen I remembered I got significant boost when I did more augmentations, so I added extensive augmentations including noise and blur, and increased the degree of augmentations. Score jumped to 0.269. I did harder augmentations but it didn't improve, so I stopped tuning augmentations.
\nI did more experiments, but sadly nothing worked.
\nI had an experience of increasing the face margin from 0.05->0.1, and it seemed to have increased overall public lb score. So I decided to increase face margin more.
\nI experimented face margin of 0.5, and single b5 scored astonishing 0.222. So I increased face margin to 1.0 and started to tune how much I should crop the margins.
\nWith margin crop of 0.15 (margin of 0.7), I could get 0.218, and could get 0.214 with b4.
\nI trained 7 b4s and 3 b5s with different seeds and did simple average. Also I increased nframes to 30 when inferencing. It resulted in my final public lb score of 0.199.
", "rawMarkdown": "Long 4 months have passed. Applaud to participants who have been working hard and thanks to hosts who arranged this dataset and competition.\n\nIn social aspect, automatic deepfake detection algorithm will be a must in near future. In personal aspect, this competition was held in a good timing for me. So dfdc took priority in my head for the last 4 months.\n\nLuckily, I somehow managed to attain 2nd place in public leaderboard. While waiting for the private leaderboard to be revealed, I'll share my solution. \n\nNote that all of the following are what ***I*** have done for this competition. (This post does not include what my teammates have done)\n\nFor those who are curious, I'll first share the major methods that I came up with, which improved public lb score.\n\n# Ingredients for Public LB\n\n### 1. Augmentations\n\nExtensive augmentations significantly improved public lb score and reduced cv-lb gap. I guess it makes model robust on varying data.\n\n### 2. Face Margin\n\nI set margin when cropping face, where `new_face_width=face_width*(1+margin)` (same with height). Margin of 0.5~0.7 worked significantly better than 0 and further reduced cv-lb gap. I guess model learns to detect some inconsistency between manipulated region and surrounding region.\n\nTuning augmentations and face margin were the two main ingredients that boosted public lb significantly.\n\n### 3. Model Capacity\n\nEfficientnet-b4 did quite better than efficientnet-b0. With extensive augmentations, appropriate model size improved the score.\n\n### 4. Multi-task Learning\n\nI used Unet with classification branch for model architecture. I calculated pixel-wise difference between fake and corresponding original video, generated mask with pixels that have outstanding differences, and used it as segmentation part target. By using mask information, the model is able to know which part of the image is modified and I guess it helped boost performance.\n\n### 5. Conservative Fix\n\nTrain data and test data have quite different distribution, as can be seen in cv-lb discrepancy. Multiplying constant (<1) to the logit then taking sigmoid helped improve logloss in this situation.\n\n### 6. Ensemble\n\nEnsemble always helps.\n\n### 7. N Frames when Inferencing\n\nI used simple average of frames to get probability, so the more frames extracted from the video, the better, until it hits 9 hours restriction.\n\nThank you for sharing your solution. It has many similarities to mine. Did you try taking the median prediction from each model? Or appending all the predictions from every model into a single list and then taking the median value from that list? I found that taking the median outperforms simple averaging. And to go one step further, one can consider the confidence of every prediction in the list and make a more well informed prediction that outperforms taking the median (I'll go into more detail when I post my solution when I get some free time).
", "rawMarkdown": "Thank you for sharing your solution. It has many similarities to mine. Did you try taking the median prediction from each model? Or appending all the predictions from every model into a single list and then taking the median value from that list? I found that taking the median outperforms simple averaging. And to go one step further, one can consider the confidence of every prediction in the list and make a more well informed prediction that outperforms taking the median (I'll go into more detail when I post my solution when I get some free time).", "votes": 12, "replies": [ { "id": 794622, "postDate": "2020-04-02T00:16:15.003Z", "content": "Taking median didn't work for my validation set, so I abandoned it, but it seems it worked on public lb! Achieving your score within only 1 month or so is very impressive. Looking forward for your solution.
", "rawMarkdown": "Taking median didn't work for my validation set, so I abandoned it, but it seems it worked on public lb! Achieving your score within only 1 month or so is very impressive. Looking forward for your solution. ", "votes": 4 } ] }, { "id": 793804, "postDate": "2020-04-01T09:47:02.023Z", "content": "Nice work! In hindsight it does seem logical to use segmentation models to improve predictions, but it's another thing to actually think of it and implement. Awesome!
\n\nDo you think label smoothing still has its place to improve generalization for this competition? For some videos only the audio was swapped, so there is noise in the labels if you only use the frames to train on.
\n\nWhy did you settle on EfficientNetB4 and B5 models for the ensemble while B7 gave an even better result (even though some of the improvement was due to the learning rate)?
", "rawMarkdown": "Nice work! In hindsight it does seem logical to use segmentation models to improve predictions, but it's another thing to actually think of it and implement. Awesome!\n\nDo you think label smoothing still has its place to improve generalization for this competition? For some videos only the audio was swapped, so there is noise in the labels if you only use the frames to train on.\n\nWhy did you settle on EfficientNetB4 and B5 models for the ensemble while B7 gave an even better result (even though some of the improvement was due to the learning rate)?", "votes": 5, "replies": [ { "id": 793979, "postDate": "2020-04-01T13:03:03.020Z", "content": "Thanks @carlolepelaars . I did label smoothing, but it strangely gave me significantly bad cv. I thought mixup or label smoothing will make model more conservative and generalize better, but it didn't work for my case.\nI found B4 did better than B5 in my last version and settled on b4+b5.
", "rawMarkdown": "Thanks @carlolepelaars . I did label smoothing, but it strangely gave me significantly bad cv. I thought mixup or label smoothing will make model more conservative and generalize better, but it didn't work for my case.\nI found B4 did better than B5 in my last version and settled on b4+b5.", "votes": 3 }, { "id": 794039, "postDate": "2020-04-01T14:03:17.967Z", "content": "Thank you for the explanation, @harangdev ! Our team also had good performance with EfficientNetB6 so we settled on 2x B6 (200x200 and 224x224 resolution), 2x B5 (224x224) and 1x B4 (224x224 with more data augmentation and label smoothing). Very curious to see the end results!
", "rawMarkdown": "Thank you for the explanation, @harangdev ! Our team also had good performance with EfficientNetB6 so we settled on 2x B6 (200x200 and 224x224 resolution), 2x B5 (224x224) and 1x B4 (224x224 with more data augmentation and label smoothing). Very curious to see the end results!", "votes": 1 } ] }, { "id": 797752, "postDate": "2020-04-04T21:33:58.740Z", "content": "Great work! Its appreciable. Thanks for sharing.
", "rawMarkdown": "Great work! Its appreciable. Thanks for sharing.", "votes": 4 }, { "id": 796154, "postDate": "2020-04-03T11:00:50.130Z", "content": "*Seeing the approaches made by one of the smartest minds of the world gimme a lot of pleasure. Thank you for sharing. *
", "rawMarkdown": "**Seeing the approaches made by one of the smartest minds of the world gimme a lot of pleasure. Thank you for sharing. **", "votes": 3, "replies": [ { "id": 796159, "postDate": "2020-04-03T11:04:41.203Z", "content": "Well said.
", "rawMarkdown": "Well said.", "votes": 1 } ] }, { "id": 793993, "postDate": "2020-04-01T13:17:58.043Z", "content": "Nice work
", "rawMarkdown": "Nice work", "votes": 3 }, { "id": 793793, "postDate": "2020-04-01T09:32:21.127Z", "content": "Amazing work :)
\n\nVery well explained discussions. Thanks for sharing. Hope you get money !
", "rawMarkdown": "Amazing work :) \n\nVery well explained discussions. Thanks for sharing. Hope you get money !", "votes": 3, "replies": [ { "id": 793973, "postDate": "2020-04-01T12:58:43.677Z", "content": "Thanks @youhanlee Hoping for your last step to GM!
", "rawMarkdown": "Thanks @youhanlee Hoping for your last step to GM!", "votes": 1 } ] }, { "id": 793665, "postDate": "2020-04-01T07:39:41.610Z", "content": "Nice work! I found it useful that you included what didn't work in your write up.
", "rawMarkdown": "Nice work! I found it useful that you included what didn't work in your write up.", "votes": 4 }, { "id": 816093, "postDate": "2020-04-22T05:35:12.030Z", "content": "Nice write up! We had some promising code, including deep analysis of the mp4 structural file content. But once we found after 2 months of work that the LB and data provided were very loosely coupled, if at all, our team lost steam. It really felt like Kaggle failed (let down the community) to provide quality training data, and certainly wasted a lot of our teams time. If they could not provide better training data, that should have been clearly stated up front. I see in Phase 6 you just started testing for the LB instead of using the training data, nice call.
", "rawMarkdown": "Nice write up! We had some promising code, including deep analysis of the mp4 structural file content. But once we found after 2 months of work that the LB and data provided were very loosely coupled, if at all, our team lost steam. It really felt like Kaggle failed (let down the community) to provide quality training data, and certainly wasted a lot of our teams time. If they could not provide better training data, that should have been clearly stated up front. I see in Phase 6 you just started testing for the LB instead of using the training data, nice call. ", "votes": 1, "replies": [ { "id": 816197, "postDate": "2020-04-22T07:01:14.793Z", "content": "Working with bad quality data, generate a good model to perform well on future data, is a job in data science. Not just this competition but with other competitions. This is real-life problem, so everything is difficult, and we will need to tackle it. Furthermore, data was prepared by Facebook, not Kaggle, and Facebook want us to solve problem for them.
", "rawMarkdown": "Working with bad quality data, generate a good model to perform well on future data, is a job in data science. Not just this competition but with other competitions. This is real-life problem, so everything is difficult, and we will need to tackle it. Furthermore, data was prepared by Facebook, not Kaggle, and Facebook want us to solve problem for them.", "votes": 1 }, { "id": 816921, "postDate": "2020-04-22T18:00:35.097Z", "content": "I agree this was the worse data that I have seen on Kaggle, I lost steam too because most of if was not really fake face videos, the moved a blur cursor around the faces back and forward, so when trying to see how many frames and how many to skip, you may end up with the blur not being on the face most to the times.... so you where mixing good faces with a fake face on the fake dataset.... with no time to clean the data, family, work, and other things, I got frustrated and asked my self what was the real purpose of making the fake videos like this, because if you see the other fake videos external sources, and train with external data, I will get a very low public score... what mess they made in Kaggle.
", "rawMarkdown": "I agree this was the worse data that I have seen on Kaggle, I lost steam too because most of if was not really fake face videos, the moved a blur cursor around the faces back and forward, so when trying to see how many frames and how many to skip, you may end up with the blur not being on the face most to the times.... so you where mixing good faces with a fake face on the fake dataset.... with no time to clean the data, family, work, and other things, I got frustrated and asked my self what was the real purpose of making the fake videos like this, because if you see the other fake videos external sources, and train with external data, I will get a very low public score... what mess they made in Kaggle. ", "votes": 1 }, { "id": 817217, "postDate": "2020-04-23T01:06:25.947Z", "content": "Yes, it was to help Facebook, but for a $1M prize, they should have better spent half that on getting quality data, instead of the prize money to get spot on results. Real world is one thing, mailing it in for creating data when you have deep pockets is very unfortunate.
", "rawMarkdown": "Yes, it was to help Facebook, but for a $1M prize, they should have better spent half that on getting quality data, instead of the prize money to get spot on results. Real world is one thing, mailing it in for creating data when you have deep pockets is very unfortunate." }, { "id": 817221, "postDate": "2020-04-23T01:13:45.383Z", "content": "When they removed the header leaked data, the entire LB test set was uniformly re-produced to all be fake data. So the provided test data MP4 deep file structure had errors in some of the fakes and not in others, which would be natural if fakes were created by a bunch of different creators. After the leaked data fix, none of the individual file differences were present anymore, because they had uniformly made them all fakes by re-producing them all.
\n\nIn the end, it was just an unfortunate waste of a lot of people's time, when you expect the dataset to be what it is presented as.
", "rawMarkdown": "When they removed the header leaked data, the entire LB test set was uniformly re-produced to all be fake data. So the provided test data MP4 deep file structure had errors in some of the fakes and not in others, which would be natural if fakes were created by a bunch of different creators. After the leaked data fix, none of the individual file differences were present anymore, because they had uniformly made them all fakes by re-producing them all.\n\nIn the end, it was just an unfortunate waste of a lot of people's time, when you expect the dataset to be what it is presented as." } ] }, { "id": 812407, "postDate": "2020-04-18T18:11:45.990Z", "content": "Great work mate! Your Solution is really helpful.
", "rawMarkdown": "Great work mate! Your Solution is really helpful.", "votes": 1 }, { "id": 811510, "postDate": "2020-04-18T02:08:59.057Z", "content": "Nice job!
", "rawMarkdown": "Nice job!", "votes": 1 }, { "id": 794750, "postDate": "2020-04-02T04:08:47.497Z", "content": "Great work ! Thank you for you sharing. Increase the face margin give a significant improvement in LB( there are many fake videos have mask not in the face region, but around the faces). We also using the Increase the face margin trick, but do not have such big improvement as yours.
\nHave you considered more than one faces in a frame and not all faces in video are fakes， so the average strategy will smoothing the predict score. Have you tried some attention strategy to model the feature sequence ？
@user155897 you've done great too! I've done nothing special with two faces. Just used averaging. I didn't use attention.
", "rawMarkdown": "@user155897 you've done great too! I've done nothing special with two faces. Just used averaging. I didn't use attention.", "votes": 1 }, { "id": 796047, "postDate": "2020-04-03T08:36:44.063Z", "content": "@user155897 \nAny insights about attention strategy for more than one faces in a frame ?
", "rawMarkdown": "@user155897 \nAny insights about attention strategy for more than one faces in a frame ? " } ] }, { "id": 794573, "postDate": "2020-04-01T22:46:55.177Z", "content": "First of all, great job and amazing work.
\n\nI see that you have made many different iteration to reach that goal. I had questions related to the hardware setup that you used.
\n\nThanks again for the amazing results.
", "rawMarkdown": "First of all, great job and amazing work. \n\nI see that you have made many different iteration to reach that goal. I had questions related to the hardware setup that you used. \n\n- Did you have your own machine or you used cloud services like AWS or GC?\n- If you used cloud services, may I ask a rough cost estimate for the setup that you used?\n- What was the hard drive space requirements you needed for your files management?\n- Roughly, how much time did you give this project of your time?\n\nThanks again for the amazing results.", "votes": 1, "replies": [ { "id": 794623, "postDate": "2020-04-02T00:18:20.820Z", "content": "Great write up! Best of luck on the private LB.
", "rawMarkdown": "Great write up! Best of luck on the private LB. ", "votes": 1 }, { "id": 794005, "postDate": "2020-04-01T13:29:12.960Z", "content": "Thanks for sharing your work. I respect your effort on this competition.. hope shake-up works for you :)
", "rawMarkdown": "Thanks for sharing your work. I respect your effort on this competition.. hope shake-up works for you :)", "votes": 1 }, { "id": 793854, "postDate": "2020-04-01T10:59:49.290Z", "content": "Congrats on the 2nd place solution! How were you able to fit so many models+weights in a limit of 1024 MB? We were only able to squeeze in one B5 (350MB) and B6 (500MB) using keras.
", "rawMarkdown": "Congrats on the 2nd place solution! How were you able to fit so many models+weights in a limit of 1024 MB? We were only able to squeeze in one B5 (350MB) and B6 (500MB) using keras.", "votes": 1, "replies": [ { "id": 793981, "postDate": "2020-04-01T13:04:56.583Z", "content": "My single pytorch B5 only takes 54.51MB. I did model.half() to save space.
Congratulation for your team's result and thank you for this detailed solution presentation.
", "rawMarkdown": "Congratulation for your team's result and thank you for this detailed solution presentation. ", "votes": 1 }, { "id": 793773, "postDate": "2020-04-01T09:06:12.050Z", "content": "Congrats and thanks for the great write up! Love the Unet/mask idea.
\n\nHow did you manage to fit 10 model weights in the inference notebook when it was limited to 1.024GB of external data? Or am I missing something?
\n\nAlso, do you think you’ll be releasing your extraction/preprocessing code? Video was new to me so I spent a lot of time just making that work, and I feel my extraction code could have been a lot cleaner (also, corrupt public test set videos are the worst 😂)
", "rawMarkdown": "Congrats and thanks for the great write up! Love the Unet/mask idea. \n\nHow did you manage to fit 10 model weights in the inference notebook when it was limited to 1.024GB of external data? Or am I missing something?\n\nAlso, do you think you’ll be releasing your extraction/preprocessing code? Video was new to me so I spent a lot of time just making that work, and I feel my extraction code could have been a lot cleaner (also, corrupt public test set videos are the worst 😂)", "votes": 1, "replies": [ { "id": 793967, "postDate": "2020-04-01T12:55:46.543Z", "content": "I'll update code after private leaderboard reveals.\nSince I didn't need decoder part when inferencing, I excluded them. Also I did model.half(). but since my dataset is less than 500MB, halving wasn't needed after all.
ah gotcha, thanks and congrats again!
", "rawMarkdown": "ah gotcha, thanks and congrats again!" } ] }, { "id": 793720, "postDate": "2020-04-01T08:23:36.123Z", "content": "Congratulations on getting to the second position, and thank you for sharing your solution! All of your hard work paid off!\nI actually joined the competition 3 weeks ago, and this my first time using computer vision. My model idea was very similar to yours, I created masks of the fake features by using the difference between fake and real, and I trained a semantic segmentation model using resnet50 and unet. However, my model wasn't good at classifying the real videos, but I was able to detect many of the fake features. But this could also be due to my poor training methods.\nSo, I just wanted to ask you about your mask classes, was it only one class that corresponds to fakeness. So, if it finds fake features then it generates a mask, and if it is real then no mask is generated?
", "rawMarkdown": "Congratulations on getting to the second position, and thank you for sharing your solution! All of your hard work paid off!\nI actually joined the competition 3 weeks ago, and this my first time using computer vision. My model idea was very similar to yours, I created masks of the fake features by using the difference between fake and real, and I trained a semantic segmentation model using resnet50 and unet. However, my model wasn't good at classifying the real videos, but I was able to detect many of the fake features. But this could also be due to my poor training methods.\nSo, I just wanted to ask you about your mask classes, was it only one class that corresponds to fakeness. So, if it finds fake features then it generates a mask, and if it is real then no mask is generated?", "votes": 1, "replies": [ { "id": 793966, "postDate": "2020-04-01T12:53:45.587Z", "content": "Yeah if it is real mask contains only 0. I generated mask (segmentation target) by following pseudo code.\n\n(abs(frame - original).mean(axis=-1) > adaptive_threshold).astype(int)\n
I used a DeeplabV3-Resnet101 segmentation model to predict a generated binary mask from the difference of the real and corresponding fake face. I struggled to design a method to generate the binary mask that could handle the compression artifacts, the image noise of some videos, differences in brightness, textures, etc. I ended up applying gaussian blurring on both real and fake image prior to doing the pixel difference, and using the mean and std of the difference to set a threshold. \nThe masks seemed ok, and the predicted segmentation masks looked good. As I was running out of time, I didn't try to train a combined classifier and segmentation model, instead I tried training a classifier using the predicted segmentation mask as input. It provided ok results, but not better than my other model. I didn't give myself enough time to tweak it as I had limited time to choose what to work on. I wonder how much of it it had to do with the face margin I used. I used a margin of 0.2\nHow did you determine the adaptive threshold?
", "rawMarkdown": "I used a DeeplabV3-Resnet101 segmentation model to predict a generated binary mask from the difference of the real and corresponding fake face. I struggled to design a method to generate the binary mask that could handle the compression artifacts, the image noise of some videos, differences in brightness, textures, etc. I ended up applying gaussian blurring on both real and fake image prior to doing the pixel difference, and using the mean and std of the difference to set a threshold. \nThe masks seemed ok, and the predicted segmentation masks looked good. As I was running out of time, I didn't try to train a combined classifier and segmentation model, instead I tried training a classifier using the predicted segmentation mask as input. It provided ok results, but not better than my other model. I didn't give myself enough time to tweak it as I had limited time to choose what to work on. I wonder how much of it it had to do with the face margin I used. I used a margin of 0.2\nHow did you determine the adaptive threshold? " }, { "id": 794894, "postDate": "2020-04-02T07:23:40.187Z", "content": "@cesb45 I set adaptive threshold constant*mean_of_abs_pixel_differences for each frame.
Thanks for sharing, It was my first serious competition and definitely I learned a lot. Kaggle and you guys are amazing. And also I see lot of people using efficientnet, is it really superior for all vision tasks ? It would be really helpful if someone can please give some pointers.
", "rawMarkdown": "Thanks for sharing, It was my first serious competition and definitely I learned a lot. Kaggle and you guys are amazing. And also I see lot of people using efficientnet, is it really superior for all vision tasks ? It would be really helpful if someone can please give some pointers.", "votes": 1, "replies": [ { "id": 793716, "postDate": "2020-04-01T08:19:34.973Z", "content": "You can explore https://paperswithcode.com/sota which compares models on diverse datasets.
", "rawMarkdown": "You can explore https://paperswithcode.com/sota which compares models on diverse datasets.", "votes": 6 } ] }, { "id": 793616, "postDate": "2020-04-01T06:40:51.753Z", "content": "\"... I calculated pixel-wise difference between fake and corresponding original video, generated mask with pixels that have outstanding differences, and used it as segmentation part target. By using mask information, the model is able to know which part of the image is modified and I guess it helped boost performance.\"
\n\n\"Use conservative fix: multiply constant(<1) to the logits than take sigmoid. It helps improve logloss when the training and testing distributions differ.\"\n- Towards the later submissions I started using a custom (gentler) sigmoid function in inference and that seemed to boost my score from extremely low to very low
\n\n\ndef gsigmoid(logit):\n return (logit/ (1 + abs(logit)) + 1.) / 2.\n
@harangdev , thank you for sharing your amazing solution in such detail! What a fantastic work you've done!
\n\nCould you please share what image resolution you've been using till phase 6 and how long the training took in this case for efficeintnet-b0/-b3/-b7? I'd be grateful to learn more about your best practices for faster experiment iterations. I stuck with effnet-b0 and 160/224 pixels per face for faster iterations (I started working on this amazing comp rather late, making 5 out of total 7 submissions in the past 3 days😅 )
Thank you and congrats again!
", "rawMarkdown": "@harangdev , thank you for sharing your amazing solution in such detail! What a fantastic work you've done!\n\nCould you please share what image resolution you've been using till phase 6 and how long the training took in this case for `efficeintnet-b0`/`-b3`/`-b7`? I'd be grateful to learn more about your best practices for faster experiment iterations. I stuck with `effnet-b0` and `160/224` pixels per face for faster iterations (I started working on this amazing comp rather late, making 5 out of total 7 submissions in the past 3 days😅 ) \n\nThank you and congrats again!", "votes": 1, "replies": [ { "id": 793614, "postDate": "2020-04-01T06:36:40.227Z", "content": "Thanks, @samusram . I used 256 resolution for my experiments. I don't recall precisely, but b0 took about 3 hours and b7 took about 12 hours. My final model b4 with resolution 352 took about 9 hours to train.
", "rawMarkdown": "Thanks, @samusram . I used 256 resolution for my experiments. I don't recall precisely, but b0 took about 3 hours and b7 took about 12 hours. My final model b4 with resolution 352 took about 9 hours to train.", "votes": 1 }, { "id": 793615, "postDate": "2020-04-01T06:40:50.663Z", "content": "Thanks for the reply!👍
", "rawMarkdown": "Thanks for the reply!👍 " }, { "id": 793674, "postDate": "2020-04-01T07:45:30.677Z", "content": "@harangdev Amazing work! Thanks for the insights.\nDid you resize the already cropped faces from 256 to 352 or did you do another crop from the original frame with size 352?
", "rawMarkdown": "@harangdev Amazing work! Thanks for the insights.\nDid you resize the already cropped faces from 256 to 352 or did you do another crop from the original frame with size 352?", "votes": 1 }, { "id": 793678, "postDate": "2020-04-01T07:47:43.993Z", "content": "@catochris Thanks. The saved faces are resized to match the maximum face size within each video so that I can resize flexibly in the dataloader.
", "rawMarkdown": "@catochris Thanks. The saved faces are resized to match the maximum face size within each video so that I can resize flexibly in the dataloader.", "votes": 3 }, { "id": 794099, "postDate": "2020-04-01T15:00:16.620Z", "content": "@harangdev How much disk space did the extracted faces take up? Did you save them as JPEG, PNG or compressed Numpy arrays?
", "rawMarkdown": "@harangdev How much disk space did the extracted faces take up? Did you save them as JPEG, PNG or compressed Numpy arrays?" }, { "id": 794898, "postDate": "2020-04-02T07:28:55.543Z", "content": "@olegtrott about 700GB. I saved them as compressed numpy arrays, using joblib.
", "rawMarkdown": "@olegtrott about 700GB. I saved them as compressed numpy arrays, using joblib.", "votes": 3 } ] }, { "id": 793573, "postDate": "2020-04-01T05:54:12.990Z", "content": "Nice work! Could you tell us what method do you use to decrease the LB/CV gap to ~0.05. In my case, CV is all 0.23-0.20, but LB is range between 0.36 - 0.30. And do you have the problem that many real faces are classified to fake in your CV?
", "rawMarkdown": "Nice work! Could you tell us what method do you use to decrease the LB/CV gap to ~0.05. In my case, CV is all 0.23-0.20, but LB is range between 0.36 - 0.30. And do you have the problem that many real faces are classified to fake in your CV?", "votes": 1, "replies": [ { "id": 793604, "postDate": "2020-04-01T06:26:31.967Z", "content": "Thanks! Like said in the post, augmentations and face margin decreased the cv-lb gap. I think these two methods made the model learn robust features. \nAnd no I didn't have such problem that many real faces are classified to fake in my cv.
", "rawMarkdown": "Thanks! Like said in the post, augmentations and face margin decreased the cv-lb gap. I think these two methods made the model learn robust features. \nAnd no I didn't have such problem that many real faces are classified to fake in my cv." }, { "id": 793761, "postDate": "2020-04-01T08:50:53.653Z", "content": "Thank you for your reply! Do you wash the dirty data?
", "rawMarkdown": "Thank you for your reply! Do you wash the dirty data?" }, { "id": 794030, "postDate": "2020-04-01T13:53:41.453Z", "content": "@xavierlin No I didn't
", "rawMarkdown": "@xavierlin No I didn't", "votes": 2 } ] }, { "id": 793535, "postDate": "2020-04-01T05:16:58.390Z", "content": "Congrats! Interestingly all the phases are exactly same as mine:) I used resnext instead but my scores are very similar until phase 9. (As an ML noob very happy to see that.).\nExcept increasing the face margin; I used a fixed face margin(5%). Instead of experimenting bigger face margins I have focused on two different experiments: 1- whole frame, 2- zooming into inner faces.\nCongrats again!
", "rawMarkdown": "Congrats! Interestingly all the phases are exactly same as mine:) I used resnext instead but my scores are very similar until phase 9. (As an ML noob very happy to see that.).\nExcept increasing the face margin; I used a fixed face margin(5%). Instead of experimenting bigger face margins I have focused on two different experiments: 1- whole frame, 2- zooming into inner faces.\nCongrats again!", "votes": 1 }, { "id": 793500, "postDate": "2020-04-01T04:23:38.567Z", "content": "Good job! I also tried segmentaion-based method simiar with your work, which is called face x-ray,https://arxiv.org/abs/1912.13458. But I can't get a better score than classification-based method. I also choosed Unet as model structure but I did'nt realliezed I can generate mask by fake video! It'a brilliant idea, congratulations!
", "rawMarkdown": "Good job! I also tried segmentaion-based method simiar with your work, which is called face x-ray,https://arxiv.org/abs/1912.13458. But I can't get a better score than classification-based method. I also choosed Unet as model structure but I did'nt realliezed I can generate mask by fake video! It'a brilliant idea, congratulations!", "votes": 1 }, { "id": 793477, "postDate": "2020-04-01T03:59:38.387Z", "content": "Good work and congratulations! I just wonder you mentioned that CNN+LSTM does not work for you and average predictions of each frame also does not work, then how did you make the final decision based on multiple frames?
", "rawMarkdown": "Good work and congratulations! I just wonder you mentioned that CNN+LSTM does not work for you and average predictions of each frame also does not work, then how did you make the final decision based on multiple frames?", "votes": 1, "replies": [ { "id": 793485, "postDate": "2020-04-01T04:10:24.047Z", "content": "Averaging predictions of each frame didn't work when training. That is, I tried architecture that uses n frames and puts each of them into efficientnet. The output will be (n_frames, n_channels, h, w). Then the model calculates statistical features such as mean, std over the n_frames axis. Then it concatenates them, and connects to dense layer to output single logit. But it didn't work out good.\nI did averaging when inferencing.\nI'll edit the post to avoid confusion.
I really appreciate your sharing of valuable experience. Thank you sincerely!
", "rawMarkdown": " I really appreciate your sharing of valuable experience. Thank you sincerely!", "votes": 1, "replies": [ { "id": 793526, "postDate": "2020-04-01T05:01:24.423Z", "content": "Face margin seemed to really crucial to improve performance.\nSo you cropped faces with margin 1.0(face width x2) from original images \nand randomly cropped again with margin 0.7(face width x1.7) finally?
\n\nI got really many lessons from your article. Thank you again!
", "rawMarkdown": "Face margin seemed to really crucial to improve performance.\nSo you cropped faces with margin 1.0(face width x2) from original images \nand randomly cropped again with margin 0.7(face width x1.7) finally?\n\nI got really many lessons from your article. Thank you again!" }, { "id": 793527, "postDate": "2020-04-01T05:04:43.637Z", "content": "It's good to hear that thanks! Yeah you are right, but not random. I center cropped so that the final face width is x1.7.
", "rawMarkdown": "It's good to hear that thanks! Yeah you are right, but not random. I center cropped so that the final face width is x1.7.", "votes": 2 } ] }, { "id": 793436, "postDate": "2020-04-01T03:00:49.223Z", "content": "Thanks for sharing.
\n\nI don't know if it's the power of the face margin or the capacity of efficientnet. I was trying from the beginning to exploit the blurring feature manually. But it turned out that this feature can be learned by efficientnet by brute-force... Maybe I should all in efficientnet if I will try more cv competitions.
", "rawMarkdown": "Thanks for sharing.\n\n I don't know if it's the power of the face margin or the capacity of efficientnet. I was trying from the beginning to exploit the blurring feature manually. But it turned out that this feature can be learned by efficientnet by brute-force... Maybe I should all in efficientnet if I will try more cv competitions.", "votes": 1 }, { "id": 793433, "postDate": "2020-04-01T02:56:54.697Z", "content": "Thank you for releasing your solution.\nI will read it carefully !
", "rawMarkdown": "Thank you for releasing your solution.\nI will read it carefully !", "votes": 1 }, { "id": 793429, "postDate": "2020-04-01T02:56:14.703Z", "content": "Really helpful, and most conclusions are the same.\nWhat multiply constant you choose for conservative fix. Is it tuned based on public score?
", "rawMarkdown": "Really helpful, and most conclusions are the same.\nWhat multiply constant you choose for conservative fix. Is it tuned based on public score?\n", "votes": 1, "replies": [ { "id": 793434, "postDate": "2020-04-01T02:58:22.103Z", "content": "0.9 and yes it is tuned on public score
", "rawMarkdown": "0.9 and yes it is tuned on public score" } ] }, { "id": 793423, "postDate": "2020-04-01T02:49:56.960Z", "content": "Many thanks for this awesome solution. will try to recreate this one.
", "rawMarkdown": "Many thanks for this awesome solution. will try to recreate this one.", "votes": 1 }, { "id": 794279, "postDate": "2020-04-01T17:52:41.880Z", "content": "Lots of great tips and things to try for my next CV competition. Overfitting was an issue indeed and you found a lot of creative ways to deal with it. Thanks again for sharing and good luck for the private results!
", "rawMarkdown": "Lots of great tips and things to try for my next CV competition. Overfitting was an issue indeed and you found a lot of creative ways to deal with it. Thanks again for sharing and good luck for the private results!", "votes": 2 }, { "id": 793995, "postDate": "2020-04-01T13:20:41.483Z", "content": "Great job! Did you use the landmark information of retinaface to apply face alignment transform to the face image?
", "rawMarkdown": "Great job! Did you use the landmark information of retinaface to apply face alignment transform to the face image? ", "votes": 2, "replies": [ { "id": 794028, "postDate": "2020-04-01T13:52:52.907Z", "content": "No I didn't
", "rawMarkdown": "No I didn't" } ] }, { "id": 793660, "postDate": "2020-04-01T07:33:52.817Z", "content": "Thanks for sharing!\nBtw, I wonder whether have you considered the situation of multiple faces appearing in one video?
", "rawMarkdown": "Thanks for sharing!\nBtw, I wonder whether have you considered the situation of multiple faces appearing in one video?", "votes": 2, "replies": [ { "id": 793671, "postDate": "2020-04-01T07:44:14.263Z", "content": "If 2 faces are spotted and n_frames=20, I get 40 outputs, and I simply average them. I tried max between prob of each face and mean of 20 largest predictions but they didn't work.
Here by \"mean of 20 largest prediction\" you mean for the case you have n_frames=20 and 2 faces, right?
", "rawMarkdown": "Here by \"mean of 20 largest prediction\" you mean for the case you have n_frames=20 and 2 faces, right? " }, { "id": 793984, "postDate": "2020-04-01T13:08:39.550Z", "content": "Yep
", "rawMarkdown": "Yep", "votes": 1 } ] }, { "id": 793622, "postDate": "2020-04-01T06:47:02.820Z", "content": "Thanks for sharing ! I was not able to notice that Face Margin is so important ...\nBtw, what is your validation strategy? (folder wise, holdout, k-fold ...)
", "rawMarkdown": "Thanks for sharing ! I was not able to notice that Face Margin is so important ...\nBtw, what is your validation strategy? (folder wise, holdout, k-fold ...)", "votes": 2, "replies": [ { "id": 793628, "postDate": "2020-04-01T06:55:32.320Z", "content": "I used kfold or sometimes just fold0 with folder-wise split for the experiments
", "rawMarkdown": "I used kfold or sometimes just fold0 with folder-wise split for the experiments" }, { "id": 794686, "postDate": "2020-04-02T02:29:04.180Z", "content": "Did kfold mean (folder0 - folder39 for train, folder40 - folder49 for validataion), (folder10-folder49 for train and folder 0-9 for validation) ... ?\nIs k = 5? \nAnd did just fold0 can trace LB well?
", "rawMarkdown": "Did kfold mean (folder0 - folder39 for train, folder40 - folder49 for validataion), (folder10-folder49 for train and folder 0-9 for validation) ... ?\nIs k = 5? \nAnd did just fold0 can trace LB well?" }, { "id": 794895, "postDate": "2020-04-02T07:27:30.577Z", "content": "@xavierlin \n1. Yeah but shuffled\n2. k=5\n3. No
", "rawMarkdown": "@xavierlin \n1. Yeah but shuffled\n2. k=5\n3. No", "votes": 1 } ] }, { "id": 793586, "postDate": "2020-04-01T06:04:36.227Z", "content": "Thank you for sharing your solution. It reflects a big work congrats!\nI didn't join this competition because I am always thinking that video data needs a lot of computational power. Can you please tell me:\nWhat and how many GPUs did you use?\nHow many hours (average) do you spend on the competition daily?\nHow much time did it take to train your final model?
", "rawMarkdown": "Thank you for sharing your solution. It reflects a big work congrats!\nI didn't join this competition because I am always thinking that video data needs a lot of computational power. Can you please tell me:\nWhat and how many GPUs did you use?\nHow many hours (average) do you spend on the competition daily?\nHow much time did it take to train your final model? ", "votes": 2, "replies": [ { "id": 793608, "postDate": "2020-04-01T06:32:35.773Z", "content": "1 2080ti\nWith slight exaggeration, I can say that local computer ran 24 hours daily for last 1~2 months.\nApproximately 9 hours per model
", "rawMarkdown": "1 2080ti\nWith slight exaggeration, I can say that local computer ran 24 hours daily for last 1~2 months.\nApproximately 9 hours per model", "votes": 2 }, { "id": 793625, "postDate": "2020-04-01T06:52:42.497Z", "content": "Thanks for your sharing, and I want to know how many images did you use when training. Because 24 batch size and 9 hours can' t allow 20 frames per video(about total 2.4m.). @harangdev
", "rawMarkdown": "Thanks for your sharing, and I want to know how many images did you use when training. Because 24 batch size and 9 hours can' t allow 20 frames per video(about total 2.4m.). @harangdev " }, { "id": 793640, "postDate": "2020-04-01T07:13:34.757Z", "content": "use only 1 2080ti GPU to train efficientnet b4-b5, res 252 or higher, batchsize 24 ?
", "rawMarkdown": "use only 1 2080ti GPU to train efficientnet b4-b5, res 252 or higher, batchsize 24 ? " }, { "id": 793650, "postDate": "2020-04-01T07:17:59.007Z", "content": "@gaohaibin I use apex fp16\n@lookcc I sample one frame per epoch, like I said in the post \n\nUse different frames of each video every epoch (#0->#7->#14->#1->#8->…)\n
Thanks, did you use apex with opt_level = 'O1' or another?
", "rawMarkdown": "Thanks, did you use apex with opt_level = 'O1' or another? " }, { "id": 793656, "postDate": "2020-04-01T07:25:14.140Z", "content": "I used default config
", "rawMarkdown": "I used default config" }, { "id": 813733, "postDate": "2020-04-19T23:55:32.420Z", "content": "This is very impressive
", "rawMarkdown": "This is very impressive" } ] }, { "id": 793476, "postDate": "2020-04-01T03:57:37.717Z", "content": "Thanks for sharing. Now that you mentioned face margin I noticed the same. When I reduce margin < 1.0 my score went backwards. When I increased to 1.1 it improved. I should have kept going 😑
", "rawMarkdown": "Thanks for sharing. Now that you mentioned face margin I noticed the same. When I reduce margin < 1.0 my score went backwards. When I increased to 1.1 it improved. I should have kept going 😑 ", "votes": 2, "replies": [ { "id": 793504, "postDate": "2020-04-01T04:29:56.653Z", "content": "damn it, same here, why did we never tried beyond 40 margin, which was 0.2 in your ratio but making it 0 made things worse, looks like we all missed with the minor details.
", "rawMarkdown": "damn it, same here, why did we never tried beyond 40 margin, which was 0.2 in your ratio but making it 0 made things worse, looks like we all missed with the minor details.", "votes": 1 } ] }, { "id": 793459, "postDate": "2020-04-01T03:35:44.830Z", "content": "Thanks for sharing! I've reached stage 5, and from this perspective I guess it's an awesome result. I've had 0.33 with b3 :p. I don't think I would've come up with loss constant multiplication though.
", "rawMarkdown": "Thanks for sharing! I've reached stage 5, and from this perspective I guess it's an awesome result. I've had 0.33 with b3 :p. I don't think I would've come up with loss constant multiplication though. ", "votes": 2 }, { "id": 793422, "postDate": "2020-04-01T02:49:10.873Z", "content": "Wow! your efforts are truly commendable. \nI just reached your phase-4 two days ago. 😂
", "rawMarkdown": "Wow! your efforts are truly commendable. \nI just reached your phase-4 two days ago. 😂 ", "votes": 2, "replies": [ { "id": 793499, "postDate": "2020-04-01T04:23:16.723Z", "content": "@limerobot Thanks! You should have achieved great result if you started this comp a bit earlier 😂
", "rawMarkdown": "@limerobot Thanks! You should have achieved great result if you started this comp a bit earlier 😂 ", "votes": 2 }, { "id": 793510, "postDate": "2020-04-01T04:34:10.967Z", "content": "Have you joined at the very beginning? :)
", "rawMarkdown": "Have you joined at the very beginning? :) " }, { "id": 793515, "postDate": "2020-04-01T04:37:11.260Z", "content": "I remember about one week after
", "rawMarkdown": "I remember about one week after" } ] }, { "id": 793413, "postDate": "2020-04-01T02:26:12.100Z", "content": "Thank you! Can I check the inference code?
", "rawMarkdown": "Thank you! Can I check the inference code?", "votes": 2, "replies": [ { "id": 793415, "postDate": "2020-04-01T02:28:08.687Z", "content": "I'll update code after release of private leaderboard
", "rawMarkdown": "I'll update code after release of private leaderboard", "votes": 1 }, { "id": 793428, "postDate": "2020-04-01T02:54:28.627Z", "content": "Thanks in detailed insight hope you stay around at samoe plc .\nDid you align batches that is to process original and their fakes together while taking loss ?
", "rawMarkdown": "Thanks in detailed insight hope you stay around at samoe plc .\nDid you align batches that is to process original and their fakes together while taking loss ?", "votes": 1 }, { "id": 793484, "postDate": "2020-04-01T04:08:21.457Z", "content": "I did it, but I don't think it helps a lot. It made the training a bit smoother and prevented overfitting (imo). It stopped real faces of fake videos from making the model overfit (with 0 margin).
", "rawMarkdown": "I did it, but I don't think it helps a lot. It made the training a bit smoother and prevented overfitting (imo). It stopped real faces of fake videos from making the model overfit (with 0 margin). " }, { "id": 793490, "postDate": "2020-04-01T04:16:12.720Z", "content": "@jaideepvalani , no. Like I said in the post, I tried the dataloading function that guarantees fake and corresponding real to appear together in one batch, but it didn't help. I just did weighted loss with standard random dataloader.
", "rawMarkdown": "@jaideepvalani , no. Like I said in the post, I tried the dataloading function that guarantees fake and corresponding real to appear together in one batch, but it didn't help. I just did weighted loss with standard random dataloader.", "votes": 1 }, { "id": 793589, "postDate": "2020-04-01T06:12:10.843Z", "content": "1) which efficient net did u use.. i tried the one that is used by many \nhttps://github.com/lukemelas/EfficientNet-PyTorch\nbut it was not converging .. \n2) I tried training in both way one Real-Fake together and one just treating each image as stand alone ..i dont know my results were poor with second approach. \nWhat i could be missing ? \nHow much ratio you kept in your training set for real and fake . \nWas it like Taking all reals and may be randomly taking one or two fakes for each real to maintain ratio of 1:2 or 1:1 for classes
", "rawMarkdown": "1) which efficient net did u use.. i tried the one that is used by many \nhttps://github.com/lukemelas/EfficientNet-PyTorch\nbut it was not converging .. \n2) I tried training in both way one Real-Fake together and one just treating each image as stand alone ..i dont know my results were poor with second approach. \nWhat i could be missing ? \nHow much ratio you kept in your training set for real and fake . \nWas it like Taking all reals and may be randomly taking one or two fakes for each real to maintain ratio of 1:2 or 1:1 for classes\n" }, { "id": 793626, "postDate": "2020-04-01T06:53:41.537Z", "content": "1) Yeah I used that one for experiments.\n2) I used all training data and used pos_weight parameter in nn.BCEWithLogitsLoss parameter to balance the loss. I didn't gain anything from changing the data loading method, and didn't dig further, so I cannot consult your specific situation.
Did you use fixed pos_weight or did you calculate per batch since random batch does not guarantee same ratio?
", "rawMarkdown": "Did you use fixed pos_weight or did you calculate per batch since random batch does not guarantee same ratio?" }, { "id": 794032, "postDate": "2020-04-01T13:54:19.300Z", "content": "@maralski I used fixed pos_weight
", "rawMarkdown": "@maralski I used fixed pos_weight", "votes": 2 }, { "id": 794797, "postDate": "2020-04-02T05:23:01.143Z", "content": "Thanks..\nRequiring more clarification on mask gen process\n(abs(video - original).mean(axis=-1) > adaptive_threshold).astype(int)
1) What is Video is it fake frame ?\n2) Appropriate Threshold value ?\n3) Each original had its fake face in fake videos ,did u take difference between Real frame and its corresponding fake frame of fake video ?\n4) your code would do the pixel wise difference ,what could be the size of your diff here diff(3,224,224)-dff2(3,224,224) , if we do mean it would be column vector is suppose.
\n\n5) How can we do these augmentation shift, scale, rgbshift & JpegCompression and Downscale augmentation
", "rawMarkdown": "Thanks..\nRequiring more clarification on mask gen process\n`(abs(video - original).mean(axis=-1) > adaptive_threshold).astype(int)`\n\n1) What is Video is it fake frame ?\n2) Appropriate Threshold value ?\n3) Each original had its fake face in fake videos ,did u take difference between Real frame and its corresponding fake frame of fake video ?\n4) your code would do the pixel wise difference ,what could be the size of your diff here diff(3,224,224)-dff2(3,224,224) , if we do mean it would be column vector is suppose.\n\n5) How can we do these augmentation shift, scale, rgbshift & JpegCompression and Downscale augmentation \n" }, { "id": 794902, "postDate": "2020-04-02T07:37:43.283Z", "content": "@jaideepvalani \n1) every frame. I'll edit it to frame.\n2) constant*mean_of_abs_pixel_differences for each frame.\n3) as it is. frame - original will yield 0 mask if frame is original\n4) like I mentioned in the post, I worked in channel_last format except in training.\n5) albumentations library\nBTW, I cannot possibly explain every details of my methods. You should consult the code later(I'm planning to share it when I retain similar position in private lb)
\"Face Margin\" is a great idea.
", "rawMarkdown": "\"Face Margin\" is a great idea.", "votes": 2 }, { "id": 793404, "postDate": "2020-04-01T02:18:10.930Z", "content": "Your solution is pure hard work. I did not expect something like tuning augmentations and other things that would yield much better LB.
", "rawMarkdown": "Your solution is pure hard work. I did not expect something like tuning augmentations and other things that would yield much better LB. ", "votes": 2, "replies": [ { "id": 793424, "postDate": "2020-04-01T02:51:12.220Z", "content": "It is nature to use the difference between the environment and the blurred face. I was wondering all time how to use this feature. But I didn't expect the margin can help to exploit this feature... I think it is working due to the high capacity of efficientnet.
", "rawMarkdown": "It is nature to use the difference between the environment and the blurred face. I was wondering all time how to use this feature. But I didn't expect the margin can help to exploit this feature... I think it is working due to the high capacity of efficientnet. ", "votes": 1 } ] }, { "id": 810791, "postDate": "2020-04-17T09:20:43.323Z", "content": "Really nice work!
", "rawMarkdown": "Really nice work!" }, { "id": 1222378, "postDate": "2021-03-01T18:25:54.873Z", "content": "That's Really an insightful solution!!
", "rawMarkdown": "That's Really an insightful solution!!" }, { "id": 1077914, "postDate": "2020-11-14T04:50:56.100Z", "content": "Hello did you publish code? I saw place #67 shared code. Not sure if I missed something.
", "rawMarkdown": "Hello did you publish code? I saw place #67 shared code. Not sure if I missed something. " }, { "id": 818243, "postDate": "2020-04-23T18:08:17.043Z", "content": "Супер
", "rawMarkdown": "Супер" }, { "id": 816823, "postDate": "2020-04-22T16:15:52.830Z", "content": "Excellent.
", "rawMarkdown": "Excellent." }, { "id": 815543, "postDate": "2020-04-21T16:22:58.910Z", "content": "A masterpiece, well done!
", "rawMarkdown": "A masterpiece, well done!" }, { "id": 814619, "postDate": "2020-04-20T20:17:25.983Z", "content": "Thanks for your sharing!!!!!\nGreat work mate!!!!Your Solution is really helpful.....
", "rawMarkdown": "Thanks for your sharing!!!!!\nGreat work mate!!!!Your Solution is really helpful....." }, { "id": 814591, "postDate": "2020-04-20T19:47:00.443Z", "content": "Awesome work! the methodologies and with the reasoning of each step is really helpful. Thanks for sharing.
", "rawMarkdown": "Awesome work! the methodologies and with the reasoning of each step is really helpful. Thanks for sharing. " }, { "id": 814583, "postDate": "2020-04-20T19:34:01.697Z", "content": "Nice Job!
", "rawMarkdown": "Nice Job!" }, { "id": 814474, "postDate": "2020-04-20T17:32:15.393Z", "content": "Thanks for sharing, your work is really inspiring, hope you win the competition!!!
", "rawMarkdown": "Thanks for sharing, your work is really inspiring, hope you win the competition!!!" }, { "id": 813806, "postDate": "2020-04-20T03:09:23.360Z", "content": "\n\n\nDifferent learning rate between encoder - decoder
\n
is there some Learning materials？ Thx!!
", "rawMarkdown": "> Different learning rate between encoder - decoder \n\n\nis there some Learning materials？ Thx!!" }, { "id": 813521, "postDate": "2020-04-19T18:15:00.133Z", "content": "Fantastic work and very useful!
", "rawMarkdown": "Fantastic work and very useful!" }, { "id": 812462, "postDate": "2020-04-18T18:57:55.263Z", "content": "Great work! Congrats!
", "rawMarkdown": "Great work! Congrats!" }, { "id": 811834, "postDate": "2020-04-18T09:09:33.293Z", "content": "Great work
", "rawMarkdown": "Great work" }, { "id": 809701, "postDate": "2020-04-16T12:12:16.230Z", "content": "nice
", "rawMarkdown": "nice" }, { "id": 808270, "postDate": "2020-04-15T09:12:28.750Z", "content": "it's brilliant to see approach taken so well described, thanks for sharing
", "rawMarkdown": "it's brilliant to see approach taken so well described, thanks for sharing" }, { "id": 808030, "postDate": "2020-04-15T05:24:37.007Z", "content": "Thanks sooo much for sharing! Great resources to learn from
", "rawMarkdown": "Thanks sooo much for sharing! Great resources to learn from" }, { "id": 807854, "postDate": "2020-04-15T01:18:33.987Z", "content": "Thank you for sharing your amazing solution! :)
", "rawMarkdown": "Thank you for sharing your amazing solution! :)" }, { "id": 807423, "postDate": "2020-04-14T16:23:37.790Z", "content": "👍
", "rawMarkdown": "👍 " }, { "id": 806855, "postDate": "2020-04-14T06:22:26.597Z", "content": "cool
", "rawMarkdown": "cool" }, { "id": 806627, "postDate": "2020-04-13T22:34:23.900Z", "content": "@harangdev thank you for sharing your solution. I would like to know about how you implement inferencing. Since this model is essentially an image classifier, how does it translate to video classification? Thanks in advance!
", "rawMarkdown": "@harangdev thank you for sharing your solution. I would like to know about how you implement inferencing. Since this model is essentially an image classifier, how does it translate to video classification? Thanks in advance!" }, { "id": 805986, "postDate": "2020-04-13T10:23:43.777Z", "content": "Congratulation and thank you for this detailed solution presentation.
", "rawMarkdown": "Congratulation and thank you for this detailed solution presentation." }, { "id": 805975, "postDate": "2020-04-13T10:05:58.900Z", "content": "Nice work! Very useful.
", "rawMarkdown": "Nice work! Very useful." }, { "id": 805885, "postDate": "2020-04-13T07:36:31.840Z", "content": "Useful💙
", "rawMarkdown": "Useful💙 " }, { "id": 805840, "postDate": "2020-04-13T06:41:09.930Z", "content": "?
", "rawMarkdown": "?\n\n" }, { "id": 805732, "postDate": "2020-04-13T02:27:37.130Z", "content": "Great work! + kind explanation
", "rawMarkdown": "Great work! + kind explanation" }, { "id": 805486, "postDate": "2020-04-12T18:26:59.937Z", "content": "Wow, a lot of work you've done here. Thanks for sharing and congrats the 2nd place you deserve more.
", "rawMarkdown": "Wow, a lot of work you've done here. Thanks for sharing and congrats the 2nd place you deserve more." }, { "id": 805189, "postDate": "2020-04-12T12:53:32.347Z", "content": "Thank you so much for sharing this. It was very good that you included what was tried but didn't improve the score.
", "rawMarkdown": "Thank you so much for sharing this. It was very good that you included what was tried but didn't improve the score.\n" }, { "id": 804761, "postDate": "2020-04-11T22:34:15.577Z", "content": "Nice work!
", "rawMarkdown": "Nice work!" }, { "id": 804444, "postDate": "2020-04-11T15:30:11.727Z", "content": "Awesome, great work!!
", "rawMarkdown": "Awesome, great work!!" }, { "id": 804179, "postDate": "2020-04-11T09:59:29.950Z", "content": "well said
", "rawMarkdown": "well said" }, { "id": 804155, "postDate": "2020-04-11T09:24:20.157Z", "content": "This was very helpful, thank you so much
", "rawMarkdown": "This was very helpful, thank you so much" }, { "id": 804149, "postDate": "2020-04-11T09:14:17.690Z", "content": "Congratulations. It was very insightful. Thanks for sharing.
", "rawMarkdown": "Congratulations. It was very insightful. Thanks for sharing." }, { "id": 802482, "postDate": "2020-04-09T14:52:26.390Z", "content": "Congrats! One question, did you feed a 4 channel image (1mask + 3ch) to your efficientNet? So I'm guessing you didn't use transfer learning.
", "rawMarkdown": "Congrats! One question, did you feed a 4 channel image (1mask + 3ch) to your efficientNet? So I'm guessing you didn't use transfer learning." }, { "id": 802127, "postDate": "2020-04-09T06:50:37.360Z", "content": "Did you crop the part from the video based on the difference from the original one?
", "rawMarkdown": "Did you crop the part from the video based on the difference from the original one?" }, { "id": 801980, "postDate": "2020-04-09T02:18:51.893Z", "content": "good
", "rawMarkdown": "good" }, { "id": 801865, "postDate": "2020-04-08T21:42:29.430Z", "content": "Nice work!
", "rawMarkdown": "Nice work!" }, { "id": 801538, "postDate": "2020-04-08T15:14:07.440Z", "content": "Nice work
", "rawMarkdown": "Nice work" }, { "id": 801181, "postDate": "2020-04-08T07:01:20.593Z", "content": "well, i'm new this platform so how can join the challenge
", "rawMarkdown": "well, i'm new this platform so how can join the challenge" }, { "id": 800615, "postDate": "2020-04-07T15:21:50.107Z", "content": "good
", "rawMarkdown": "good" }, { "id": 799244, "postDate": "2020-04-06T09:42:49.427Z", "content": "Great post with detailed explaination. Thanks for sharing.!
", "rawMarkdown": "Great post with detailed explaination. Thanks for sharing.!" }, { "id": 799141, "postDate": "2020-04-06T08:16:32.777Z", "content": "upgrade
", "rawMarkdown": "upgrade\n" }, { "id": 798411, "postDate": "2020-04-05T13:22:16.520Z", "content": "well told
", "rawMarkdown": "well told" }, { "id": 798410, "postDate": "2020-04-05T13:21:44.640Z", "content": "well said
", "rawMarkdown": "well said" }, { "id": 796118, "postDate": "2020-04-03T10:22:18.620Z", "content": "Nice one, thanks for sharing. Wish you luck in the private leaderboard
", "rawMarkdown": "Nice one, thanks for sharing. Wish you luck in the private leaderboard" }, { "id": 795971, "postDate": "2020-04-03T06:55:53.293Z", "content": "good
", "rawMarkdown": "good" }, { "id": 795887, "postDate": "2020-04-03T05:54:39.823Z", "content": "The real vs fake frame difference is 3-channel, how do you calc distance? Euc distance?
", "rawMarkdown": "The real vs fake frame difference is 3-channel, how do you calc distance? Euc distance?" }, { "id": 795484, "postDate": "2020-04-02T18:48:08.717Z", "content": "Thank you for sharing: but did you use multi-task in the final solution? Or was it just EfficientNet?
", "rawMarkdown": "Thank you for sharing: but did you use multi-task in the final solution? Or was it just EfficientNet?", "replies": [ { "id": 795692, "postDate": "2020-04-03T00:38:48.053Z", "content": "I used multi-task in the final solution.
", "rawMarkdown": "I used multi-task in the final solution.", "votes": 3 } ] }, { "id": 795344, "postDate": "2020-04-02T16:32:11.660Z", "content": "\bWhat stopping strategy did you use during training? I evaluated the performance of the validation set in each epoch, and then selected the checkpoint with the lowest loss in the validation set. However, I find that the checkpoint performance of different epochs is quite large, which wastes a lot of time.
", "rawMarkdown": "\bWhat stopping strategy did you use during training? I evaluated the performance of the validation set in each epoch, and then selected the checkpoint with the lowest loss in the validation set. However, I find that the checkpoint performance of different epochs is quite large, which wastes a lot of time.", "replies": [ { "id": 795696, "postDate": "2020-04-03T00:42:58.400Z", "content": "I followed basic approach - select best epoch with val set logloss.
", "rawMarkdown": "I followed basic approach - select best epoch with val set logloss.", "votes": 2 }, { "id": 795750, "postDate": "2020-04-03T02:28:45.750Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!" }, { "id": 796068, "postDate": "2020-04-03T09:04:32.843Z", "content": "@harangdev \nHow do u split the train and val dataset ?
", "rawMarkdown": "@harangdev \nHow do u split the train and val dataset ?" } ] }, { "id": 794998, "postDate": "2020-04-02T09:28:40.770Z", "content": "Great competition phases and tricks sharing. Thank you very much. @harangdev I found offline Training / Validation set could affect LB results largely. How do you split them?
", "rawMarkdown": "Great competition phases and tricks sharing. Thank you very much. @harangdev I found offline Training / Validation set could affect LB results largely. How do you split them?", "replies": [ { "id": 795016, "postDate": "2020-04-02T09:52:34.893Z", "content": "Like I said in another reply, I split folder-wise, but in the end pretty much ignored local validation and rather validated on public lb.
", "rawMarkdown": "Like I said in another reply, I split folder-wise, but in the end pretty much ignored local validation and rather validated on public lb.", "votes": 1 } ] }, { "id": 794917, "postDate": "2020-04-02T08:00:23.480Z", "content": "Amazing work.
", "rawMarkdown": "Amazing work." }, { "id": 794780, "postDate": "2020-04-02T04:56:42.263Z", "content": "Congratulations to you and your teammates. Thanks for the impressive and detailed solution @harangdev!
", "rawMarkdown": "Congratulations to you and your teammates. Thanks for the impressive and detailed solution @harangdev!" }, { "id": 794724, "postDate": "2020-04-02T03:25:59.407Z", "content": "Nice work!I have learned so many tricks in your Discussion.In this contest I try to use CNN+LSTM structure,but it is only about 60% acurracy.i wonder know if you try to design a multi-fullyConnected net for classification?
", "rawMarkdown": "Nice work!I have learned so many tricks in your Discussion.In this contest I try to use CNN+LSTM structure,but it is only about 60% acurracy.i wonder know if you try to design a multi-fullyConnected net for classification?", "replies": [ { "id": 794893, "postDate": "2020-04-02T07:21:36.937Z", "content": "What do you mean by multi-fullyConnected net?
just a network contains some fullyconnected layer.Because I have read a paper named fakespotter.they use different activation layer's output to set an activate threshold, and they design a shallow network with five full-connected layers as their classifier.
", "rawMarkdown": "just a network contains some fullyconnected layer.Because I have read a paper named fakespotter.they use different activation layer's output to set an activate threshold, and they design a shallow network with five full-connected layers as their classifier.\n\n" }, { "id": 794905, "postDate": "2020-04-02T07:41:13.803Z", "content": "hmm I didn't know the paper and also didn't try that method.
", "rawMarkdown": "hmm I didn't know the paper and also didn't try that method.", "votes": 1 } ] }, { "id": 794339, "postDate": "2020-04-01T18:37:52.127Z", "content": "Thanks for sharing your work and congratulations on your team's result.
", "rawMarkdown": "Thanks for sharing your work and congratulations on your team's result." }, { "id": 794318, "postDate": "2020-04-01T18:26:59.893Z", "content": "Good Job
", "rawMarkdown": "Good Job\n" }, { "id": 794281, "postDate": "2020-04-01T17:55:03.333Z", "content": "@harangdev Amazing work, thank you for sharing! How did you implement parameter tuning? Did you use a particular library? I have used Keras LR finder and GridSearch but I'm having trouble using them.
", "rawMarkdown": "@harangdev Amazing work, thank you for sharing! How did you implement parameter tuning? Did you use a particular library? I have used Keras LR finder and GridSearch but I'm having trouble using them.", "replies": [ { "id": 794618, "postDate": "2020-04-02T00:10:38.090Z", "content": "I tuned manually
", "rawMarkdown": "I tuned manually", "votes": 2 } ] }, { "id": 794174, "postDate": "2020-04-01T16:19:56.023Z", "content": "Thank you for sharing your work, it is well explained and congratulation for your team on getting to the second position
", "rawMarkdown": "Thank you for sharing your work, it is well explained and congratulation for your team on getting to the second position" }, { "id": 794117, "postDate": "2020-04-01T15:18:39.903Z", "content": "Did you use a u-net architecture in all the ensembled models? I tried something similar to your u-net for multitask learning but instead of the u-net I utilized mask-rcnn on the whole frame but did not get good results. Also, unlike using diff btw original image and fake image to get face ground truth, my ground truth face bounding-box was predicted in the data-generator with a small face detection model. Mask-rcnn's task then was to learn to detect faces and classify them. Some other approach would have been used to aggregate results across multiple frames. Since this initial phase did not perform well, I abandoned this approach. Understandably the classification branch in mask-rcnn is a tiny cnn that probably could not deal with the difficult deepfake faces. Most approaches used in this competition are at the mercy of the face detectors which are pretty weak. Your approach really solves many issues with false positive and negatives because the Eff-7 u-net is a pretty strong face detector and the ground truth strategy will also catch off face modifications that other approaches might miss.
", "rawMarkdown": "Did you use a u-net architecture in all the ensembled models? I tried something similar to your u-net for multitask learning but instead of the u-net I utilized mask-rcnn on the whole frame but did not get good results. Also, unlike using diff btw original image and fake image to get face ground truth, my ground truth face bounding-box was predicted in the data-generator with a small face detection model. Mask-rcnn's task then was to learn to detect faces and classify them. Some other approach would have been used to aggregate results across multiple frames. Since this initial phase did not perform well, I abandoned this approach. Understandably the classification branch in mask-rcnn is a tiny cnn that probably could not deal with the difficult deepfake faces. Most approaches used in this competition are at the mercy of the face detectors which are pretty weak. Your approach really solves many issues with false positive and negatives because the Eff-7 u-net is a pretty strong face detector and the ground truth strategy will also catch off face modifications that other approaches might miss. ", "replies": [ { "id": 794617, "postDate": "2020-04-02T00:10:01.883Z", "content": "@godaibo Yeah I used unet in all of my models. If the hosts want to utilize mask informations, they have uncompressed videos, so they know ground true modified regions, and the process will be much cleaner.
", "rawMarkdown": "@godaibo Yeah I used unet in all of my models. If the hosts want to utilize mask informations, they have uncompressed videos, so they know ground true modified regions, and the process will be much cleaner.", "votes": 1 } ] }, { "id": 793832, "postDate": "2020-04-01T10:20:28.173Z", "content": "Nice work, and very useful explanation!
", "rawMarkdown": "Nice work, and very useful explanation!" }, { "id": 793798, "postDate": "2020-04-01T09:37:30.620Z", "content": "nice work
", "rawMarkdown": "nice work\n" }, { "id": 793609, "postDate": "2020-04-01T06:35:12.683Z", "content": "I've googled multi-task learning with classification and segmentation and got this paper - https://arxiv.org/pdf/1906.06876.pdf from mid-2019.
\n\nI wonder where implemeting it as is would've gotten us 😏
", "rawMarkdown": "I've googled multi-task learning with classification and segmentation and got this paper - https://arxiv.org/pdf/1906.06876.pdf from mid-2019.\n\nI wonder where implemeting it as is would've gotten us 😏 " }, { "id": 793458, "postDate": "2020-04-01T03:33:18.623Z", "content": "\n\n\nPhase 9. Increase Face Margin (0.214)
\n
How come you didn't take this toward its logical conclusion (just ignore faces) like @Moshel did?
", "rawMarkdown": "> Phase 9. Increase Face Margin (0.214)\n\nHow come you didn't take this toward its logical conclusion (just ignore faces) like @Moshel did?\n\n", "replies": [ { "id": 793463, "postDate": "2020-04-01T03:42:01.303Z", "content": "@olegtrott I didn't understand your question. I increased face margin so that it contains more outer part of face and some backgrounds.
", "rawMarkdown": "@olegtrott I didn't understand your question. I increased face margin so that it contains more outer part of face and some backgrounds." }, { "id": 793468, "postDate": "2020-04-01T03:45:17.297Z", "content": "@Moshel \"increased the margin\" so much that it included the whole frame! 🤔
", "rawMarkdown": "@Moshel \"increased the margin\" so much that it included the whole frame! 🤔 " }, { "id": 793469, "postDate": "2020-04-01T03:48:47.627Z", "content": "I found increasing too much margin worsen the results.
", "rawMarkdown": "I found increasing too much margin worsen the results.", "votes": 1 } ] }, { "id": 793439, "postDate": "2020-04-01T03:07:15.327Z", "content": "I used Retinaface too, at first, because they had the best WIDERFACE accuracy, but then I noticed that it says it's for non-commercial use only:
\n\n
This forced me to try to look for other face-detection methods and to post this (which mentions RetinaFace specifically)
\n\nhttps://www.kaggle.com/c/deepfake-detection-challenge/discussion/133316
\n\nThe Pytorch re-implementation also used the proprietary Insightface face annotations. The whole uncertainty over licensing was annoying and a huge time-sink.
", "rawMarkdown": "I used Retinaface too, at first, because they had the best WIDERFACE accuracy, but then I noticed that it says it's for non-commercial use only:\n\n\n\nThis forced me to try to look for other face-detection methods and to post this (which mentions RetinaFace specifically)\n\nhttps://www.kaggle.com/c/deepfake-detection-challenge/discussion/133316\n\nThe Pytorch re-implementation also used the proprietary Insightface face annotations. The whole uncertainty over licensing was annoying and a huge time-sink. \n", "replies": [ { "id": 793448, "postDate": "2020-04-01T03:17:04.220Z", "content": "Yeah I also read them. Uncertainty over licensing was an obstacle to us too. The model is publicly available so I decided to use it.
", "rawMarkdown": "Yeah I also read them. Uncertainty over licensing was an obstacle to us too. The model is publicly available so I decided to use it." } ] }, { "id": 813085, "postDate": "2020-04-19T11:15:41.040Z", "rawMarkdown": "", "isDeleted": true }, { "id": 803848, "postDate": "2020-04-10T22:41:50.363Z", "rawMarkdown": "", "isDeleted": true }, { "id": 796796, "postDate": "2020-04-03T23:24:37.053Z", "rawMarkdown": "", "isDeleted": true }, { "id": 795851, "postDate": "2020-04-03T04:52:20.443Z", "rawMarkdown": "", "isDeleted": true }, { "id": 795548, "postDate": "2020-04-02T20:10:12.843Z", "rawMarkdown": "", "isDeleted": true }, { "id": 794673, "postDate": "2020-04-02T02:01:24.113Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 794682, "postDate": "2020-04-02T02:21:23.033Z", "content": "You dont, you dont need to even do that in the inference to start with, that is the role of model to output mask for the input image in the inference. That step is only needed while training, in inference if the mask (output of the model) exists or larger than a particular size, we can say that the image is fake else the image is real.
", "rawMarkdown": "You dont, you dont need to even do that in the inference to start with, that is the role of model to output mask for the input image in the inference. That step is only needed while training, in inference if the mask (output of the model) exists or larger than a particular size, we can say that the image is fake else the image is real." }, { "id": 794709, "postDate": "2020-04-02T03:06:07.183Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 793408, "postDate": "2020-04-01T02:25:14.530Z", "rawMarkdown": "", "isDeleted": true }, { "id": 799343, "postDate": "2020-04-06T11:32:31.440Z", "content": "thank you
", "rawMarkdown": "thank you\n", "votes": 3 }, { "id": 806544, "postDate": "2020-04-13T20:05:09.953Z", "content": "Thanks for sharing your work!
", "rawMarkdown": "Thanks for sharing your work!", "votes": 2 }, { "id": 811579, "postDate": "2020-04-18T04:58:56.457Z", "content": "Thanks for your sharing!
", "rawMarkdown": "Thanks for your sharing!", "votes": 1 }, { "id": 794594, "postDate": "2020-04-01T23:31:31.163Z", "content": "thanks for sharing!
", "rawMarkdown": "thanks for sharing!", "votes": 1 }, { "id": 794482, "postDate": "2020-04-01T21:02:47.923Z", "content": "Great post! Thanks for sharing!
", "rawMarkdown": "Great post! Thanks for sharing! ", "votes": 1 }, { "id": 793465, "postDate": "2020-04-01T03:44:01.870Z", "content": "Thanks for sharing your approach
", "rawMarkdown": "Thanks for sharing your approach", "votes": 1 }, { "id": 817406, "postDate": "2020-04-23T05:59:47.907Z", "content": "Thanks
", "rawMarkdown": "Thanks" }, { "id": 816991, "postDate": "2020-04-22T18:51:35.317Z", "content": "Nice work. Thanks for your share
", "rawMarkdown": "Nice work. Thanks for your share" }, { "id": 816130, "postDate": "2020-04-22T06:12:33.397Z", "content": "Thanks for your sharing...
", "rawMarkdown": "Thanks for your sharing..." }, { "id": 816050, "postDate": "2020-04-22T04:32:38.047Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 815523, "postDate": "2020-04-21T15:53:46.247Z", "content": "Thank you for sharing
", "rawMarkdown": "Thank you for sharing" }, { "id": 815079, "postDate": "2020-04-21T08:20:43.817Z", "content": "thanks for sharing!
", "rawMarkdown": "thanks for sharing!" }, { "id": 811435, "postDate": "2020-04-17T23:04:46.250Z", "content": "Thanks for sharing. Congrats!
", "rawMarkdown": "Thanks for sharing. Congrats!" }, { "id": 810651, "postDate": "2020-04-17T06:43:38.717Z", "content": "Thanks for sharing. Great content.
", "rawMarkdown": "Thanks for sharing. Great content." }, { "id": 810637, "postDate": "2020-04-17T06:31:44.463Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 810385, "postDate": "2020-04-16T22:54:17.427Z", "content": "Thanks you!
", "rawMarkdown": "Thanks you!" }, { "id": 810000, "postDate": "2020-04-16T16:44:34.753Z", "content": "Thank you for this!
", "rawMarkdown": "Thank you for this!" }, { "id": 804751, "postDate": "2020-04-11T22:12:28.450Z", "content": "Great job!!\nThanks for sharing
", "rawMarkdown": "Great job!!\nThanks for sharing\n" }, { "id": 803598, "postDate": "2020-04-10T17:36:32.817Z", "content": "Thanks for sharing! Well Explained!!
", "rawMarkdown": "Thanks for sharing! Well Explained!!" }, { "id": 803367, "postDate": "2020-04-10T12:41:48.243Z", "content": "Thanks for sharing :)
", "rawMarkdown": "Thanks for sharing :)" }, { "id": 801554, "postDate": "2020-04-08T15:30:26.940Z", "content": "Really cool! Thanks for sharing.
", "rawMarkdown": "Really cool! Thanks for sharing." }, { "id": 800918, "postDate": "2020-04-07T21:39:19.640Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 800128, "postDate": "2020-04-07T05:59:10.700Z", "content": "amazing work. thank you for sharing
", "rawMarkdown": "amazing work. thank you for sharing" }, { "id": 800074, "postDate": "2020-04-07T03:59:59.297Z", "content": "Great! Thanks for sharing!
", "rawMarkdown": "Great! Thanks for sharing!" }, { "id": 799919, "postDate": "2020-04-06T23:11:22.647Z", "content": "thanks for sharing!
", "rawMarkdown": "thanks for sharing!" }, { "id": 798545, "postDate": "2020-04-05T16:03:05.080Z", "content": "Awesome! Thanks for sharing.
", "rawMarkdown": "Awesome! Thanks for sharing." }, { "id": 797565, "postDate": "2020-04-04T17:03:15.200Z", "content": "Thanks for your share👍
", "rawMarkdown": "Thanks for your share👍 " }, { "id": 796276, "postDate": "2020-04-03T13:09:54.610Z", "content": "Great post! Thanks for sharing!
", "rawMarkdown": "Great post! Thanks for sharing!" }, { "id": 795575, "postDate": "2020-04-02T20:50:15.433Z", "content": "Thanks for sharing!!!
", "rawMarkdown": "Thanks for sharing!!!" }, { "id": 795522, "postDate": "2020-04-02T19:33:08.633Z", "content": "nice work thanks for sharing.
", "rawMarkdown": "nice work thanks for sharing." }, { "id": 794695, "postDate": "2020-04-02T02:41:41.307Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing" }, { "id": 794195, "postDate": "2020-04-01T16:31:28.997Z", "content": "nice work thanks for sharing.
", "rawMarkdown": "nice work thanks for sharing." } ], "comments": [ { "id": 794262, "author_name": "David", "author_url": "", "post_date": "2020-04-01T17:33:03.957000", "content": "Thank you for sharing your solution. It has many similarities to mine. Did you try taking the median prediction from each model? Or appending all the predictions from every model into a single list and then taking the median value from that list? I found that taking the median outperforms simple averaging. And to go one step further, one can consider the confidence of every prediction in the list and make a more well informed prediction that outperforms taking the median (I'll go into more detail when I post my solution when I get some free time).
", "votes": 12, "replies": [ { "id": 794622, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-02T00:16:15.003000", "content": "Taking median didn't work for my validation set, so I abandoned it, but it seems it worked on public lb! Achieving your score within only 1 month or so is very impressive. Looking forward for your solution.
", "votes": 4, "replies": [] } ] }, { "id": 793804, "author_name": "Carlo", "author_url": "", "post_date": "2020-04-01T09:47:02.023000", "content": "Nice work! In hindsight it does seem logical to use segmentation models to improve predictions, but it's another thing to actually think of it and implement. Awesome!
\n\nDo you think label smoothing still has its place to improve generalization for this competition? For some videos only the audio was swapped, so there is noise in the labels if you only use the frames to train on.
\n\nWhy did you settle on EfficientNetB4 and B5 models for the ensemble while B7 gave an even better result (even though some of the improvement was due to the learning rate)?
", "votes": 5, "replies": [ { "id": 793979, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T13:03:03.020000", "content": "Thanks @carlolepelaars . I did label smoothing, but it strangely gave me significantly bad cv. I thought mixup or label smoothing will make model more conservative and generalize better, but it didn't work for my case.\nI found B4 did better than B5 in my last version and settled on b4+b5.
", "votes": 3, "replies": [] }, { "id": 794039, "author_name": "Carlo", "author_url": "", "post_date": "2020-04-01T14:03:17.967000", "content": "Thank you for the explanation, @harangdev ! Our team also had good performance with EfficientNetB6 so we settled on 2x B6 (200x200 and 224x224 resolution), 2x B5 (224x224) and 1x B4 (224x224 with more data augmentation and label smoothing). Very curious to see the end results!
", "votes": 1, "replies": [] } ] }, { "id": 797752, "author_name": "Aman Kumar", "author_url": "", "post_date": "2020-04-04T21:33:58.740000", "content": "Great work! Its appreciable. Thanks for sharing.
", "votes": 4, "replies": [] }, { "id": 796154, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2020-04-03T11:00:50.130000", "content": "*Seeing the approaches made by one of the smartest minds of the world gimme a lot of pleasure. Thank you for sharing. *
", "votes": 3, "replies": [ { "id": 796159, "author_name": "", "author_url": "", "post_date": "2020-04-03T11:04:41.203000", "content": "Well said.
", "votes": 1, "replies": [] } ] }, { "id": 793993, "author_name": "Ragampudy Sandeep", "author_url": "", "post_date": "2020-04-01T13:17:58.043000", "content": "Nice work
", "votes": 3, "replies": [] }, { "id": 793793, "author_name": "Youhan Lee", "author_url": "", "post_date": "2020-04-01T09:32:21.127000", "content": "Amazing work :)
\n\nVery well explained discussions. Thanks for sharing. Hope you get money !
", "votes": 3, "replies": [ { "id": 793973, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T12:58:43.677000", "content": "Thanks @youhanlee Hoping for your last step to GM!
", "votes": 1, "replies": [] } ] }, { "id": 793665, "author_name": "Alex Kaechele", "author_url": "", "post_date": "2020-04-01T07:39:41.610000", "content": "Nice work! I found it useful that you included what didn't work in your write up.
", "votes": 4, "replies": [] }, { "id": 816093, "author_name": "Mark Eckdahl", "author_url": "", "post_date": "2020-04-22T05:35:12.030000", "content": "Nice write up! We had some promising code, including deep analysis of the mp4 structural file content. But once we found after 2 months of work that the LB and data provided were very loosely coupled, if at all, our team lost steam. It really felt like Kaggle failed (let down the community) to provide quality training data, and certainly wasted a lot of our teams time. If they could not provide better training data, that should have been clearly stated up front. I see in Phase 6 you just started testing for the LB instead of using the training data, nice call.
", "votes": 1, "replies": [ { "id": 816197, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-04-22T07:01:14.793000", "content": "Working with bad quality data, generate a good model to perform well on future data, is a job in data science. Not just this competition but with other competitions. This is real-life problem, so everything is difficult, and we will need to tackle it. Furthermore, data was prepared by Facebook, not Kaggle, and Facebook want us to solve problem for them.
", "votes": 1, "replies": [] }, { "id": 816921, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-04-22T18:00:35.097000", "content": "I agree this was the worse data that I have seen on Kaggle, I lost steam too because most of if was not really fake face videos, the moved a blur cursor around the faces back and forward, so when trying to see how many frames and how many to skip, you may end up with the blur not being on the face most to the times.... so you where mixing good faces with a fake face on the fake dataset.... with no time to clean the data, family, work, and other things, I got frustrated and asked my self what was the real purpose of making the fake videos like this, because if you see the other fake videos external sources, and train with external data, I will get a very low public score... what mess they made in Kaggle.
", "votes": 1, "replies": [] }, { "id": 817217, "author_name": "Mark Eckdahl", "author_url": "", "post_date": "2020-04-23T01:06:25.947000", "content": "Yes, it was to help Facebook, but for a $1M prize, they should have better spent half that on getting quality data, instead of the prize money to get spot on results. Real world is one thing, mailing it in for creating data when you have deep pockets is very unfortunate.
", "votes": 0, "replies": [] }, { "id": 817221, "author_name": "Mark Eckdahl", "author_url": "", "post_date": "2020-04-23T01:13:45.383000", "content": "When they removed the header leaked data, the entire LB test set was uniformly re-produced to all be fake data. So the provided test data MP4 deep file structure had errors in some of the fakes and not in others, which would be natural if fakes were created by a bunch of different creators. After the leaked data fix, none of the individual file differences were present anymore, because they had uniformly made them all fakes by re-producing them all.
\n\nIn the end, it was just an unfortunate waste of a lot of people's time, when you expect the dataset to be what it is presented as.
", "votes": 0, "replies": [] } ] }, { "id": 812407, "author_name": "siddhanth chopra", "author_url": "", "post_date": "2020-04-18T18:11:45.990000", "content": "Great work mate! Your Solution is really helpful.
", "votes": 1, "replies": [] }, { "id": 811510, "author_name": "Nick Fleece", "author_url": "", "post_date": "2020-04-18T02:08:59.057000", "content": "Nice job!
", "votes": 1, "replies": [] }, { "id": 794750, "author_name": "WestLake", "author_url": "", "post_date": "2020-04-02T04:08:47.497000", "content": "Great work ! Thank you for you sharing. Increase the face margin give a significant improvement in LB( there are many fake videos have mask not in the face region, but around the faces). We also using the Increase the face margin trick, but do not have such big improvement as yours.
\nHave you considered more than one faces in a frame and not all faces in video are fakes， so the average strategy will smoothing the predict score. Have you tried some attention strategy to model the feature sequence ？
@user155897 you've done great too! I've done nothing special with two faces. Just used averaging. I didn't use attention.
", "votes": 1, "replies": [] }, { "id": 796047, "author_name": "xmaocai", "author_url": "", "post_date": "2020-04-03T08:36:44.063000", "content": "@user155897 \nAny insights about attention strategy for more than one faces in a frame ?
", "votes": 0, "replies": [] } ] }, { "id": 794573, "author_name": "Amro", "author_url": "", "post_date": "2020-04-01T22:46:55.177000", "content": "First of all, great job and amazing work.
\n\nI see that you have made many different iteration to reach that goal. I had questions related to the hardware setup that you used.
\n\nThanks again for the amazing results.
", "votes": 1, "replies": [ { "id": 794623, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-02T00:18:20.820000", "content": "Great write up! Best of luck on the private LB.
", "votes": 1, "replies": [] }, { "id": 794005, "author_name": "saewonYang", "author_url": "", "post_date": "2020-04-01T13:29:12.960000", "content": "Thanks for sharing your work. I respect your effort on this competition.. hope shake-up works for you :)
", "votes": 1, "replies": [] }, { "id": 793854, "author_name": "shinig4mi", "author_url": "", "post_date": "2020-04-01T10:59:49.290000", "content": "Congrats on the 2nd place solution! How were you able to fit so many models+weights in a limit of 1024 MB? We were only able to squeeze in one B5 (350MB) and B6 (500MB) using keras.
", "votes": 1, "replies": [ { "id": 793981, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T13:04:56.583000", "content": "My single pytorch B5 only takes 54.51MB. I did model.half() to save space.
Congratulation for your team's result and thank you for this detailed solution presentation.
", "votes": 1, "replies": [] }, { "id": 793773, "author_name": "morg", "author_url": "", "post_date": "2020-04-01T09:06:12.050000", "content": "Congrats and thanks for the great write up! Love the Unet/mask idea.
\n\nHow did you manage to fit 10 model weights in the inference notebook when it was limited to 1.024GB of external data? Or am I missing something?
\n\nAlso, do you think you’ll be releasing your extraction/preprocessing code? Video was new to me so I spent a lot of time just making that work, and I feel my extraction code could have been a lot cleaner (also, corrupt public test set videos are the worst 😂)
", "votes": 1, "replies": [ { "id": 793967, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T12:55:46.543000", "content": "I'll update code after private leaderboard reveals.\nSince I didn't need decoder part when inferencing, I excluded them. Also I did model.half(). but since my dataset is less than 500MB, halving wasn't needed after all.
ah gotcha, thanks and congrats again!
", "votes": 0, "replies": [] } ] }, { "id": 793720, "author_name": "Dr Octopus", "author_url": "", "post_date": "2020-04-01T08:23:36.123000", "content": "Congratulations on getting to the second position, and thank you for sharing your solution! All of your hard work paid off!\nI actually joined the competition 3 weeks ago, and this my first time using computer vision. My model idea was very similar to yours, I created masks of the fake features by using the difference between fake and real, and I trained a semantic segmentation model using resnet50 and unet. However, my model wasn't good at classifying the real videos, but I was able to detect many of the fake features. But this could also be due to my poor training methods.\nSo, I just wanted to ask you about your mask classes, was it only one class that corresponds to fakeness. So, if it finds fake features then it generates a mask, and if it is real then no mask is generated?
", "votes": 1, "replies": [ { "id": 793966, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T12:53:45.587000", "content": "Yeah if it is real mask contains only 0. I generated mask (segmentation target) by following pseudo code.\n\n(abs(frame - original).mean(axis=-1) > adaptive_threshold).astype(int)\n
I used a DeeplabV3-Resnet101 segmentation model to predict a generated binary mask from the difference of the real and corresponding fake face. I struggled to design a method to generate the binary mask that could handle the compression artifacts, the image noise of some videos, differences in brightness, textures, etc. I ended up applying gaussian blurring on both real and fake image prior to doing the pixel difference, and using the mean and std of the difference to set a threshold. \nThe masks seemed ok, and the predicted segmentation masks looked good. As I was running out of time, I didn't try to train a combined classifier and segmentation model, instead I tried training a classifier using the predicted segmentation mask as input. It provided ok results, but not better than my other model. I didn't give myself enough time to tweak it as I had limited time to choose what to work on. I wonder how much of it it had to do with the face margin I used. I used a margin of 0.2\nHow did you determine the adaptive threshold?
", "votes": 0, "replies": [] }, { "id": 794894, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-02T07:23:40.187000", "content": "@cesb45 I set adaptive threshold constant*mean_of_abs_pixel_differences for each frame.
Thanks for sharing, It was my first serious competition and definitely I learned a lot. Kaggle and you guys are amazing. And also I see lot of people using efficientnet, is it really superior for all vision tasks ? It would be really helpful if someone can please give some pointers.
", "votes": 1, "replies": [ { "id": 793716, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T08:19:34.973000", "content": "You can explore https://paperswithcode.com/sota which compares models on diverse datasets.
", "votes": 6, "replies": [] } ] }, { "id": 793616, "author_name": "Rajaram G", "author_url": "", "post_date": "2020-04-01T06:40:51.753000", "content": "\"... I calculated pixel-wise difference between fake and corresponding original video, generated mask with pixels that have outstanding differences, and used it as segmentation part target. By using mask information, the model is able to know which part of the image is modified and I guess it helped boost performance.\"
\n\n\"Use conservative fix: multiply constant(<1) to the logits than take sigmoid. It helps improve logloss when the training and testing distributions differ.\"\n- Towards the later submissions I started using a custom (gentler) sigmoid function in inference and that seemed to boost my score from extremely low to very low
\n\n\ndef gsigmoid(logit):\n return (logit/ (1 + abs(logit)) + 1.) / 2.\n
@harangdev , thank you for sharing your amazing solution in such detail! What a fantastic work you've done!
\n\nCould you please share what image resolution you've been using till phase 6 and how long the training took in this case for efficeintnet-b0/-b3/-b7? I'd be grateful to learn more about your best practices for faster experiment iterations. I stuck with effnet-b0 and 160/224 pixels per face for faster iterations (I started working on this amazing comp rather late, making 5 out of total 7 submissions in the past 3 days😅 )
Thank you and congrats again!
", "votes": 1, "replies": [ { "id": 793614, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T06:36:40.227000", "content": "Thanks, @samusram . I used 256 resolution for my experiments. I don't recall precisely, but b0 took about 3 hours and b7 took about 12 hours. My final model b4 with resolution 352 took about 9 hours to train.
", "votes": 1, "replies": [] }, { "id": 793615, "author_name": "Raman", "author_url": "", "post_date": "2020-04-01T06:40:50.663000", "content": "Thanks for the reply!👍
", "votes": 0, "replies": [] }, { "id": 793674, "author_name": "Cato", "author_url": "", "post_date": "2020-04-01T07:45:30.677000", "content": "@harangdev Amazing work! Thanks for the insights.\nDid you resize the already cropped faces from 256 to 352 or did you do another crop from the original frame with size 352?
", "votes": 1, "replies": [] }, { "id": 793678, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T07:47:43.993000", "content": "@catochris Thanks. The saved faces are resized to match the maximum face size within each video so that I can resize flexibly in the dataloader.
", "votes": 3, "replies": [] }, { "id": 794099, "author_name": "Oleg Trott", "author_url": "", "post_date": "2020-04-01T15:00:16.620000", "content": "@harangdev How much disk space did the extracted faces take up? Did you save them as JPEG, PNG or compressed Numpy arrays?
", "votes": 0, "replies": [] }, { "id": 794898, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-02T07:28:55.543000", "content": "@olegtrott about 700GB. I saved them as compressed numpy arrays, using joblib.
", "votes": 3, "replies": [] } ] }, { "id": 793573, "author_name": "Xavier_lin", "author_url": "", "post_date": "2020-04-01T05:54:12.990000", "content": "Nice work! Could you tell us what method do you use to decrease the LB/CV gap to ~0.05. In my case, CV is all 0.23-0.20, but LB is range between 0.36 - 0.30. And do you have the problem that many real faces are classified to fake in your CV?
", "votes": 1, "replies": [ { "id": 793604, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T06:26:31.967000", "content": "Thanks! Like said in the post, augmentations and face margin decreased the cv-lb gap. I think these two methods made the model learn robust features. \nAnd no I didn't have such problem that many real faces are classified to fake in my cv.
", "votes": 0, "replies": [] }, { "id": 793761, "author_name": "Xavier_lin", "author_url": "", "post_date": "2020-04-01T08:50:53.653000", "content": "Thank you for your reply! Do you wash the dirty data?
", "votes": 0, "replies": [] }, { "id": 794030, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T13:53:41.453000", "content": "@xavierlin No I didn't
", "votes": 2, "replies": [] } ] }, { "id": 793535, "author_name": "Emre Bayram", "author_url": "", "post_date": "2020-04-01T05:16:58.390000", "content": "Congrats! Interestingly all the phases are exactly same as mine:) I used resnext instead but my scores are very similar until phase 9. (As an ML noob very happy to see that.).\nExcept increasing the face margin; I used a fixed face margin(5%). Instead of experimenting bigger face margins I have focused on two different experiments: 1- whole frame, 2- zooming into inner faces.\nCongrats again!
", "votes": 1, "replies": [] }, { "id": 793500, "author_name": "ant1", "author_url": "", "post_date": "2020-04-01T04:23:38.567000", "content": "Good job! I also tried segmentaion-based method simiar with your work, which is called face x-ray,https://arxiv.org/abs/1912.13458. But I can't get a better score than classification-based method. I also choosed Unet as model structure but I did'nt realliezed I can generate mask by fake video! It'a brilliant idea, congratulations!
", "votes": 1, "replies": [] }, { "id": 793477, "author_name": "CoooolerData", "author_url": "", "post_date": "2020-04-01T03:59:38.387000", "content": "Good work and congratulations! I just wonder you mentioned that CNN+LSTM does not work for you and average predictions of each frame also does not work, then how did you make the final decision based on multiple frames?
", "votes": 1, "replies": [ { "id": 793485, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T04:10:24.047000", "content": "Averaging predictions of each frame didn't work when training. That is, I tried architecture that uses n frames and puts each of them into efficientnet. The output will be (n_frames, n_channels, h, w). Then the model calculates statistical features such as mean, std over the n_frames axis. Then it concatenates them, and connects to dense layer to output single logit. But it didn't work out good.\nI did averaging when inferencing.\nI'll edit the post to avoid confusion.
I really appreciate your sharing of valuable experience. Thank you sincerely!
", "votes": 1, "replies": [ { "id": 793526, "author_name": "gw song", "author_url": "", "post_date": "2020-04-01T05:01:24.423000", "content": "Face margin seemed to really crucial to improve performance.\nSo you cropped faces with margin 1.0(face width x2) from original images \nand randomly cropped again with margin 0.7(face width x1.7) finally?
\n\nI got really many lessons from your article. Thank you again!
", "votes": 0, "replies": [] }, { "id": 793527, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T05:04:43.637000", "content": "It's good to hear that thanks! Yeah you are right, but not random. I center cropped so that the final face width is x1.7.
", "votes": 2, "replies": [] } ] }, { "id": 793436, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2020-04-01T03:00:49.223000", "content": "Thanks for sharing.
\n\nI don't know if it's the power of the face margin or the capacity of efficientnet. I was trying from the beginning to exploit the blurring feature manually. But it turned out that this feature can be learned by efficientnet by brute-force... Maybe I should all in efficientnet if I will try more cv competitions.
", "votes": 1, "replies": [] }, { "id": 793433, "author_name": "islet", "author_url": "", "post_date": "2020-04-01T02:56:54.697000", "content": "Thank you for releasing your solution.\nI will read it carefully !
", "votes": 1, "replies": [] }, { "id": 793429, "author_name": "vtddggg", "author_url": "", "post_date": "2020-04-01T02:56:14.703000", "content": "Really helpful, and most conclusions are the same.\nWhat multiply constant you choose for conservative fix. Is it tuned based on public score?
", "votes": 1, "replies": [ { "id": 793434, "author_name": "YoonSoo", "author_url": "", "post_date": "2020-04-01T02:58:22.103000", "content": "0.9 and yes it is tuned on public score
", "votes": 0, "replies": [] } ] }, { "id": 793423, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:49:56.960000", "content": "", "votes": 1, "replies": [] }, { "id": 794279, "author_name": "", "author_url": "", "post_date": "2020-04-01T17:52:41.880000", "content": "", "votes": 2, "replies": [] }, { "id": 793995, "author_name": "", "author_url": "", "post_date": "2020-04-01T13:20:41.483000", "content": "", "votes": 2, "replies": [ { "id": 794028, "author_name": "", "author_url": "", "post_date": "2020-04-01T13:52:52.907000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 793660, "author_name": "", "author_url": "", "post_date": "2020-04-01T07:33:52.817000", "content": "", "votes": 2, "replies": [ { "id": 793671, "author_name": "", "author_url": "", "post_date": "2020-04-01T07:44:14.263000", "content": "", "votes": 3, "replies": [] }, { "id": 793796, "author_name": "", "author_url": "", "post_date": "2020-04-01T09:35:13.360000", "content": "", "votes": 0, "replies": [] }, { "id": 793984, "author_name": "", "author_url": "", "post_date": "2020-04-01T13:08:39.550000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 793622, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:47:02.820000", "content": "", "votes": 2, "replies": [ { "id": 793628, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:55:32.320000", "content": "", "votes": 0, "replies": [] }, { "id": 794686, "author_name": "", "author_url": "", "post_date": "2020-04-02T02:29:04.180000", "content": "", "votes": 0, "replies": [] }, { "id": 794895, "author_name": "", "author_url": "", "post_date": "2020-04-02T07:27:30.577000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 793586, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:04:36.227000", "content": "", "votes": 2, "replies": [ { "id": 793608, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:32:35.773000", "content": "", "votes": 2, "replies": [] }, { "id": 793625, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:52:42.497000", "content": "", "votes": 0, "replies": [] }, { "id": 793640, "author_name": "", "author_url": "", "post_date": "2020-04-01T07:13:34.757000", "content": "", "votes": 0, "replies": [] }, { "id": 793650, "author_name": "", "author_url": "", "post_date": "2020-04-01T07:17:59.007000", "content": "", "votes": 5, "replies": [] }, { "id": 793654, "author_name": "", "author_url": "", "post_date": "2020-04-01T07:24:11.290000", "content": "", "votes": 0, "replies": [] }, { "id": 793656, "author_name": "", "author_url": "", "post_date": "2020-04-01T07:25:14.140000", "content": "", "votes": 0, "replies": [] }, { "id": 813733, "author_name": "", "author_url": "", "post_date": "2020-04-19T23:55:32.420000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 793476, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:57:37.717000", "content": "", "votes": 2, "replies": [ { "id": 793504, "author_name": "", "author_url": "", "post_date": "2020-04-01T04:29:56.653000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 793459, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:35:44.830000", "content": "", "votes": 2, "replies": [] }, { "id": 793422, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:49:10.873000", "content": "", "votes": 2, "replies": [ { "id": 793499, "author_name": "", "author_url": "", "post_date": "2020-04-01T04:23:16.723000", "content": "", "votes": 2, "replies": [] }, { "id": 793510, "author_name": "", "author_url": "", "post_date": "2020-04-01T04:34:10.967000", "content": "", "votes": 0, "replies": [] }, { "id": 793515, "author_name": "", "author_url": "", "post_date": "2020-04-01T04:37:11.260000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 793413, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:26:12.100000", "content": "", "votes": 2, "replies": [ { "id": 793415, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:28:08.687000", "content": "", "votes": 1, "replies": [] }, { "id": 793428, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:54:28.627000", "content": "", "votes": 1, "replies": [] }, { "id": 793484, "author_name": "", "author_url": "", "post_date": "2020-04-01T04:08:21.457000", "content": "", "votes": 0, "replies": [] }, { "id": 793490, "author_name": "", "author_url": "", "post_date": "2020-04-01T04:16:12.720000", "content": "", "votes": 1, "replies": [] }, { "id": 793589, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:12:10.843000", "content": "", "votes": 0, "replies": [] }, { "id": 793626, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:53:41.537000", "content": "", "votes": 1, "replies": [] }, { "id": 793988, "author_name": "", "author_url": "", "post_date": "2020-04-01T13:12:09.207000", "content": "", "votes": 0, "replies": [] }, { "id": 794032, "author_name": "", "author_url": "", "post_date": "2020-04-01T13:54:19.300000", "content": "", "votes": 2, "replies": [] }, { "id": 794797, "author_name": "", "author_url": "", "post_date": "2020-04-02T05:23:01.143000", "content": "", "votes": 0, "replies": [] }, { "id": 794902, "author_name": "", "author_url": "", "post_date": "2020-04-02T07:37:43.283000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 793407, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:23:21.960000", "content": "", "votes": 2, "replies": [] }, { "id": 793404, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:18:10.930000", "content": "", "votes": 2, "replies": [ { "id": 793424, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:51:12.220000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 810791, "author_name": "", "author_url": "", "post_date": "2020-04-17T09:20:43.323000", "content": "", "votes": 0, "replies": [] }, { "id": 1222378, "author_name": "", "author_url": "", "post_date": "2021-03-01T18:25:54.873000", "content": "", "votes": 0, "replies": [] }, { "id": 1077914, "author_name": "", "author_url": "", "post_date": "2020-11-14T04:50:56.100000", "content": "", "votes": 0, "replies": [] }, { "id": 818243, "author_name": "", "author_url": "", "post_date": "2020-04-23T18:08:17.043000", "content": "", "votes": 0, "replies": [] }, { "id": 816823, "author_name": "", "author_url": "", "post_date": "2020-04-22T16:15:52.830000", "content": "", "votes": 0, "replies": [] }, { "id": 815543, "author_name": "", "author_url": "", "post_date": "2020-04-21T16:22:58.910000", "content": "", "votes": 0, "replies": [] }, { "id": 814619, "author_name": "", "author_url": "", "post_date": "2020-04-20T20:17:25.983000", "content": "", "votes": 0, "replies": [] }, { "id": 814591, "author_name": "", "author_url": "", "post_date": "2020-04-20T19:47:00.443000", "content": "", "votes": 0, "replies": [] }, { "id": 814583, "author_name": "", "author_url": "", "post_date": "2020-04-20T19:34:01.697000", "content": "", "votes": 0, "replies": [] }, { "id": 814474, "author_name": "", "author_url": "", "post_date": "2020-04-20T17:32:15.393000", "content": "", "votes": 0, "replies": [] }, { "id": 813806, "author_name": "", "author_url": "", "post_date": "2020-04-20T03:09:23.360000", "content": "", "votes": 0, "replies": [] }, { "id": 813521, "author_name": "", "author_url": "", "post_date": "2020-04-19T18:15:00.133000", "content": "", "votes": 0, "replies": [] }, { "id": 812462, "author_name": "", "author_url": "", "post_date": "2020-04-18T18:57:55.263000", "content": "", "votes": 0, "replies": [] }, { "id": 811834, "author_name": "", "author_url": "", "post_date": "2020-04-18T09:09:33.293000", "content": "", "votes": 0, "replies": [] }, { "id": 809701, "author_name": "", "author_url": "", "post_date": "2020-04-16T12:12:16.230000", "content": "", "votes": 0, "replies": [] }, { "id": 808270, "author_name": "", "author_url": "", "post_date": "2020-04-15T09:12:28.750000", "content": "", "votes": 0, "replies": [] }, { "id": 808030, "author_name": "", "author_url": "", "post_date": "2020-04-15T05:24:37.007000", "content": "", "votes": 0, "replies": [] }, { "id": 807854, "author_name": "", "author_url": "", "post_date": "2020-04-15T01:18:33.987000", "content": "", "votes": 0, "replies": [] }, { "id": 807423, "author_name": "", "author_url": "", "post_date": "2020-04-14T16:23:37.790000", "content": "", "votes": 0, "replies": [] }, { "id": 806855, "author_name": "", "author_url": "", "post_date": "2020-04-14T06:22:26.597000", "content": "", "votes": 0, "replies": [] }, { "id": 806627, "author_name": "", "author_url": "", "post_date": "2020-04-13T22:34:23.900000", "content": "", "votes": 0, "replies": [] }, { "id": 805986, "author_name": "", "author_url": "", "post_date": "2020-04-13T10:23:43.777000", "content": "", "votes": 0, "replies": [] }, { "id": 805975, "author_name": "", "author_url": "", "post_date": "2020-04-13T10:05:58.900000", "content": "", "votes": 0, "replies": [] }, { "id": 805885, "author_name": "", "author_url": "", "post_date": "2020-04-13T07:36:31.840000", "content": "", "votes": 0, "replies": [] }, { "id": 805840, "author_name": "", "author_url": "", "post_date": "2020-04-13T06:41:09.930000", "content": "", "votes": 0, "replies": [] }, { "id": 805732, "author_name": "", "author_url": "", "post_date": "2020-04-13T02:27:37.130000", "content": "", "votes": 0, "replies": [] }, { "id": 805486, "author_name": "", "author_url": "", "post_date": "2020-04-12T18:26:59.937000", "content": "", "votes": 0, "replies": [] }, { "id": 805189, "author_name": "", "author_url": "", "post_date": "2020-04-12T12:53:32.347000", "content": "", "votes": 0, "replies": [] }, { "id": 804761, "author_name": "", "author_url": "", "post_date": "2020-04-11T22:34:15.577000", "content": "", "votes": 0, "replies": [] }, { "id": 804444, "author_name": "", "author_url": "", "post_date": "2020-04-11T15:30:11.727000", "content": "", "votes": 0, "replies": [] }, { "id": 804179, "author_name": "", "author_url": "", "post_date": "2020-04-11T09:59:29.950000", "content": "", "votes": 0, "replies": [] }, { "id": 804155, "author_name": "", "author_url": "", "post_date": "2020-04-11T09:24:20.157000", "content": "", "votes": 0, "replies": [] }, { "id": 804149, "author_name": "", "author_url": "", "post_date": "2020-04-11T09:14:17.690000", "content": "", "votes": 0, "replies": [] }, { "id": 802482, "author_name": "", "author_url": "", "post_date": "2020-04-09T14:52:26.390000", "content": "", "votes": 0, "replies": [] }, { "id": 802127, "author_name": "", "author_url": "", "post_date": "2020-04-09T06:50:37.360000", "content": "", "votes": 0, "replies": [] }, { "id": 801980, "author_name": "", "author_url": "", "post_date": "2020-04-09T02:18:51.893000", "content": "", "votes": 0, "replies": [] }, { "id": 801865, "author_name": "", "author_url": "", "post_date": "2020-04-08T21:42:29.430000", "content": "", "votes": 0, "replies": [] }, { "id": 801538, "author_name": "", "author_url": "", "post_date": "2020-04-08T15:14:07.440000", "content": "", "votes": 0, "replies": [] }, { "id": 801181, "author_name": "", "author_url": "", "post_date": "2020-04-08T07:01:20.593000", "content": "", "votes": 0, "replies": [] }, { "id": 800615, "author_name": "", "author_url": "", "post_date": "2020-04-07T15:21:50.107000", "content": "", "votes": 0, "replies": [] }, { "id": 799244, "author_name": "", "author_url": "", "post_date": "2020-04-06T09:42:49.427000", "content": "", "votes": 0, "replies": [] }, { "id": 799141, "author_name": "", "author_url": "", "post_date": "2020-04-06T08:16:32.777000", "content": "", "votes": 0, "replies": [] }, { "id": 798411, "author_name": "", "author_url": "", "post_date": "2020-04-05T13:22:16.520000", "content": "", "votes": 0, "replies": [] }, { "id": 798410, "author_name": "", "author_url": "", "post_date": "2020-04-05T13:21:44.640000", "content": "", "votes": 0, "replies": [] }, { "id": 796118, "author_name": "", "author_url": "", "post_date": "2020-04-03T10:22:18.620000", "content": "", "votes": 0, "replies": [] }, { "id": 795971, "author_name": "", "author_url": "", "post_date": "2020-04-03T06:55:53.293000", "content": "", "votes": 0, "replies": [] }, { "id": 795887, "author_name": "", "author_url": "", "post_date": "2020-04-03T05:54:39.823000", "content": "", "votes": 0, "replies": [] }, { "id": 795484, "author_name": "", "author_url": "", "post_date": "2020-04-02T18:48:08.717000", "content": "", "votes": 0, "replies": [ { "id": 795692, "author_name": "", "author_url": "", "post_date": "2020-04-03T00:38:48.053000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 795344, "author_name": "", "author_url": "", "post_date": "2020-04-02T16:32:11.660000", "content": "", "votes": 0, "replies": [ { "id": 795696, "author_name": "", "author_url": "", "post_date": "2020-04-03T00:42:58.400000", "content": "", "votes": 2, "replies": [] }, { "id": 795750, "author_name": "", "author_url": "", "post_date": "2020-04-03T02:28:45.750000", "content": "", "votes": 0, "replies": [] }, { "id": 796068, "author_name": "", "author_url": "", "post_date": "2020-04-03T09:04:32.843000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 794998, "author_name": "", "author_url": "", "post_date": "2020-04-02T09:28:40.770000", "content": "", "votes": 0, "replies": [ { "id": 795016, "author_name": "", "author_url": "", "post_date": "2020-04-02T09:52:34.893000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 794917, "author_name": "", "author_url": "", "post_date": "2020-04-02T08:00:23.480000", "content": "", "votes": 0, "replies": [] }, { "id": 794780, "author_name": "", "author_url": "", "post_date": "2020-04-02T04:56:42.263000", "content": "", "votes": 0, "replies": [] }, { "id": 794724, "author_name": "", "author_url": "", "post_date": "2020-04-02T03:25:59.407000", "content": "", "votes": 0, "replies": [ { "id": 794893, "author_name": "", "author_url": "", "post_date": "2020-04-02T07:21:36.937000", "content": "", "votes": 1, "replies": [] }, { "id": 794900, "author_name": "", "author_url": "", "post_date": "2020-04-02T07:32:52.973000", "content": "", "votes": 0, "replies": [] }, { "id": 794905, "author_name": "", "author_url": "", "post_date": "2020-04-02T07:41:13.803000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 794339, "author_name": "", "author_url": "", "post_date": "2020-04-01T18:37:52.127000", "content": "", "votes": 0, "replies": [] }, { "id": 794318, "author_name": "", "author_url": "", "post_date": "2020-04-01T18:26:59.893000", "content": "", "votes": 0, "replies": [] }, { "id": 794281, "author_name": "", "author_url": "", "post_date": "2020-04-01T17:55:03.333000", "content": "", "votes": 0, "replies": [ { "id": 794618, "author_name": "", "author_url": "", "post_date": "2020-04-02T00:10:38.090000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 794174, "author_name": "", "author_url": "", "post_date": "2020-04-01T16:19:56.023000", "content": "", "votes": 0, "replies": [] }, { "id": 794117, "author_name": "", "author_url": "", "post_date": "2020-04-01T15:18:39.903000", "content": "", "votes": 0, "replies": [ { "id": 794617, "author_name": "", "author_url": "", "post_date": "2020-04-02T00:10:01.883000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 793832, "author_name": "", "author_url": "", "post_date": "2020-04-01T10:20:28.173000", "content": "", "votes": 0, "replies": [] }, { "id": 793798, "author_name": "", "author_url": "", "post_date": "2020-04-01T09:37:30.620000", "content": "", "votes": 0, "replies": [] }, { "id": 793609, "author_name": "", "author_url": "", "post_date": "2020-04-01T06:35:12.683000", "content": "", "votes": 0, "replies": [] }, { "id": 793458, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:33:18.623000", "content": "", "votes": 0, "replies": [ { "id": 793463, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:42:01.303000", "content": "", "votes": 0, "replies": [] }, { "id": 793468, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:45:17.297000", "content": "", "votes": 0, "replies": [] }, { "id": 793469, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:48:47.627000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 793439, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:07:15.327000", "content": "", "votes": 0, "replies": [ { "id": 793448, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:17:04.220000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 813085, "author_name": "", "author_url": "", "post_date": "2020-04-19T11:15:41.040000", "content": "", "votes": 0, "replies": [] }, { "id": 803848, "author_name": "", "author_url": "", "post_date": "2020-04-10T22:41:50.363000", "content": "", "votes": 0, "replies": [] }, { "id": 796796, "author_name": "", "author_url": "", "post_date": "2020-04-03T23:24:37.053000", "content": "", "votes": 0, "replies": [] }, { "id": 795851, "author_name": "", "author_url": "", "post_date": "2020-04-03T04:52:20.443000", "content": "", "votes": 0, "replies": [] }, { "id": 795548, "author_name": "", "author_url": "", "post_date": "2020-04-02T20:10:12.843000", "content": "", "votes": 0, "replies": [] }, { "id": 794673, "author_name": "", "author_url": "", "post_date": "2020-04-02T02:01:24.113000", "content": "", "votes": 0, "replies": [ { "id": 794682, "author_name": "", "author_url": "", "post_date": "2020-04-02T02:21:23.033000", "content": "", "votes": 0, "replies": [] }, { "id": 794709, "author_name": "", "author_url": "", "post_date": "2020-04-02T03:06:07.183000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 793408, "author_name": "", "author_url": "", "post_date": "2020-04-01T02:25:14.530000", "content": "", "votes": 0, "replies": [] }, { "id": 799343, "author_name": "", "author_url": "", "post_date": "2020-04-06T11:32:31.440000", "content": "", "votes": 3, "replies": [] }, { "id": 806544, "author_name": "", "author_url": "", "post_date": "2020-04-13T20:05:09.953000", "content": "", "votes": 2, "replies": [] }, { "id": 811579, "author_name": "", "author_url": "", "post_date": "2020-04-18T04:58:56.457000", "content": "", "votes": 1, "replies": [] }, { "id": 794594, "author_name": "", "author_url": "", "post_date": "2020-04-01T23:31:31.163000", "content": "", "votes": 1, "replies": [] }, { "id": 794482, "author_name": "", "author_url": "", "post_date": "2020-04-01T21:02:47.923000", "content": "", "votes": 1, "replies": [] }, { "id": 793465, "author_name": "", "author_url": "", "post_date": "2020-04-01T03:44:01.870000", "content": "", "votes": 1, "replies": [] }, { "id": 817406, "author_name": "", "author_url": "", "post_date": "2020-04-23T05:59:47.907000", "content": "", "votes": 0, "replies": [] }, { "id": 816991, "author_name": "", "author_url": "", "post_date": "2020-04-22T18:51:35.317000", "content": "", "votes": 0, "replies": [] }, { "id": 816130, "author_name": "", "author_url": "", "post_date": "2020-04-22T06:12:33.397000", "content": "", "votes": 0, "replies": [] }, { "id": 816050, "author_name": "", "author_url": "", "post_date": "2020-04-22T04:32:38.047000", "content": "", "votes": 0, "replies": [] }, { "id": 815523, "author_name": "", "author_url": "", "post_date": "2020-04-21T15:53:46.247000", "content": "", "votes": 0, "replies": [] }, { "id": 815079, "author_name": "", "author_url": "", "post_date": "2020-04-21T08:20:43.817000", "content": "", "votes": 0, "replies": [] }, { "id": 811435, "author_name": "", "author_url": "", "post_date": "2020-04-17T23:04:46.250000", "content": "", "votes": 0, "replies": [] }, { "id": 810651, "author_name": "", "author_url": "", "post_date": "2020-04-17T06:43:38.717000", "content": "", "votes": 0, "replies": [] }, { "id": 810637, "author_name": "", "author_url": "", "post_date": "2020-04-17T06:31:44.463000", "content": "", "votes": 0, "replies": [] }, { "id": 810385, "author_name": "", "author_url": "", "post_date": "2020-04-16T22:54:17.427000", "content": "", "votes": 0, "replies": [] }, { "id": 810000, "author_name": "", "author_url": "", "post_date": "2020-04-16T16:44:34.753000", "content": "", "votes": 0, "replies": [] }, { "id": 804751, "author_name": "", "author_url": "", "post_date": "2020-04-11T22:12:28.450000", "content": "", "votes": 0, "replies": [] }, { "id": 803598, "author_name": "", "author_url": "", "post_date": "2020-04-10T17:36:32.817000", "content": "", "votes": 0, "replies": [] }, { "id": 803367, "author_name": "", "author_url": "", "post_date": "2020-04-10T12:41:48.243000", "content": "", "votes": 0, "replies": [] }, { "id": 801554, "author_name": "", "author_url": "", "post_date": "2020-04-08T15:30:26.940000", "content": "", "votes": 0, "replies": [] }, { "id": 800918, "author_name": "", "author_url": "", "post_date": "2020-04-07T21:39:19.640000", "content": "", "votes": 0, "replies": [] }, { "id": 800128, "author_name": "", "author_url": "", "post_date": "2020-04-07T05:59:10.700000", "content": "", "votes": 0, "replies": [] }, { "id": 800074, "author_name": "", "author_url": "", "post_date": "2020-04-07T03:59:59.297000", "content": "", "votes": 0, "replies": [] }, { "id": 799919, "author_name": "", "author_url": "", "post_date": "2020-04-06T23:11:22.647000", "content": "", "votes": 0, "replies": [] }, { "id": 798545, "author_name": "", "author_url": "", "post_date": "2020-04-05T16:03:05.080000", "content": "", "votes": 0, "replies": [] }, { "id": 797565, "author_name": "", "author_url": "", "post_date": "2020-04-04T17:03:15.200000", "content": "", "votes": 0, "replies": [] }, { "id": 796276, "author_name": "", "author_url": "", "post_date": "2020-04-03T13:09:54.610000", "content": "", "votes": 0, "replies": [] }, { "id": 795575, "author_name": "", "author_url": "", "post_date": "2020-04-02T20:50:15.433000", "content": "", "votes": 0, "replies": [] }, { "id": 795522, "author_name": "", "author_url": "", "post_date": "2020-04-02T19:33:08.633000", "content": "", "votes": 0, "replies": [] }, { "id": 794695, "author_name": "", "author_url": "", "post_date": "2020-04-02T02:41:41.307000", "content": "", "votes": 0, "replies": [] }, { "id": 794195, "author_name": "", "author_url": "", "post_date": "2020-04-01T16:31:28.997000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "793392": "Long 4 months have passed. Applaud to participants who have been working hard and thanks to hosts who arranged this dataset and competition.\n\nIn social aspect, automatic deepfake detection algorithm will be a must in near future. In personal aspect, this competition was held in a good timing for me. So dfdc took priority in my head for the last 4 months.\n\nLuckily, I somehow managed to attain 2nd place in public leaderboard. While waiting for the private leaderboard to be revealed, I'll share my solution. \n\nNote that all of the following are what ***I*** have done for this competition. (This post does not include what my teammates have done)\n\nFor those who are curious, I'll first share the major methods that I came up with, which improved public lb score.\n\n# Ingredients for Public LB\n\n### 1. Augmentations\n\nExtensive augmentations significantly improved public lb score and reduced cv-lb gap. I guess it makes model robust on varying data.\n\n### 2. Face Margin\n\nI set margin when cropping face, where `new_face_width=face_width*(1+margin)` (same with height). Margin of 0.5~0.7 worked significantly better than 0 and further reduced cv-lb gap. I guess model learns to detect some inconsistency between manipulated region and surrounding region.\n\nTuning augmentations and face margin were the two main ingredients that boosted public lb significantly.\n\n### 3. Model Capacity\n\nEfficientnet-b4 did quite better than efficientnet-b0. With extensive augmentations, appropriate model size improved the score.\n\n### 4. Multi-task Learning\n\nI used Unet with classification branch for model architecture. I calculated pixel-wise difference between fake and corresponding original video, generated mask with pixels that have outstanding differences, and used it as segmentation part target. By using mask information, the model is able to know which part of the image is modified and I guess it helped boost performance.\n\n### 5. Conservative Fix\n\nTrain data and test data have quite different distribution, as can be seen in cv-lb discrepancy. Multiplying constant (<1) to the logit then taking sigmoid helped improve logloss in this situation.\n\n### 6. Ensemble\n\nEnsemble always helps.\n\n### 7. N Frames when Inferencing\n\nI used simple average of frames to get probability, so the more frames extracted from the video, the better, until it hits 9 hours restriction.\n\n