First of all, we congratulate the winners of the Deepfake Detection competition and also would like to thank every competitor who worked hard to solve the deepfakes problem. We learned a lot throughout this amazing journey with you all, and we hope to see you again in the upcoming Deepfake challenges.
\n\nBefore intensely experimenting with deep neural network models, we decided to inspect the deepfake data with established computer vision techniques, to see if there's a simple cue which can reveal a deepfake, having these features in our mind for later feature-engineering purposes. Initially, we could observe indicators of deepfakes in frame inspections; such small artifacts were too hard to generalize for a novel solution. Although these experiments haven't resulted with great success, they provided a better understanding of deepfakes for us. Later on, we started experimenting with deep learnig based models, which then showed significant performance increase in detecting fake faces.
\n\nFor face extraction and processing purposes, we used FaceNet-Pytorch,
\n\nAlso, we used InceptionResNET features to differentiate individuals in the same frame from each other.
\n\nMargin: After square cropping faces with respect to bounding boxes, we’ve scaled each face with “1.2”, which is determined by experimentation.
During our experiments, we have used different levels of augmentation techniques to increase robustness towards uncertainty in the wild.
\n\ndef chain_aug(img):\n return A.Compose([A.HorizontalFlip(p=1),\n A.JpegCompression(quality_lower=70, quality_upper=100, p=1.),\n A.Blur(blur_limit=7, p=1.),\n A.IAASharpen(alpha=(0.2, 0.5), lightness=(0.5, 1.0), p=1.)\n ], p=1)(image=img)['image']\ndef light_aug(img):\n return A.Compose([A.HorizontalFlip(p=1),\n A.RandomBrightnessContrast(p=1),\n A.RandomGamma(p=1),\n A.CLAHE(p=1), ], p=1)(image=img)['image']\ndef draw_ellipse(img):\n img = Image.fromarray(img)\n draw = ImageDraw.Draw(img)\n w, h = img.size\n cx, cy = w//2, h//2\n x0 = np.random.randint(0, cx)\n y0 = np.random.randint(0, cy)\n x1 = np.random.randint(cx, w)\n y1 = np.random.randint(cy, h)\n draw.ellipse((x0, y0, x1, y1), fill='black')\n return np.asarray(img)\nEarly studies preferred backbones such as ResNet, Xception, VGG whereas we have updated our backbone choice as EfficientNet. The main idea with the backbone network was to train a simple binary classifier and to discriminate between real and fake faces. Experiments for binary-classification models started from EfficientNet-b0 and scaled up to EfficientNet-b5. Here’s a couple of things we’ve tried.\n- Low res effnet-b3 + Effnet-b5 (224x224) 0.30656\n- Effnet-b4 0.38793 (224x224)\n- Effnet-b2 0.43068 (224x224)\n- Triplet loss models (224x224)\n- An ensemble of XGB, LGB, CATBoost, KNN models
\n\nFor final submissions:\n- An ensemble of EfficientNets: 0.31499 [FAILED] (Although scripts worked Kaggle submissions, as well as tests, sadly, our best performing model failed on private board scoring.)\n- Ensembles of EfficientNets + Other models mentioned above: 0.33552
\n\nIn our development environment, we used Nvidia-docker, with a 4x 2080TI Lambda workstation.
\n\nAt first, like many competitors, we measured our performance with 400 public videos, after some time, we switched to folders 0-1-2. Later on, our team developed a validation split strategy based on similarity scores.
\n\nDuring the competition, we were tracking;\n- Log loss score for each class\n- Accuracy, confusion matrix scores\n- Error, gradient map analysis with custom debug scripts
\n\nThrough the DFDC journey, we have had many challenges of generalizing our models to deepfakes in the wild. Training with only the DFDC dataset, our models resulted in high performance with \"face swap\" auto-encoder based generation techniques. Still, they failed to generalize other variations of deepfakes and generation techniques. We are very excited for future contributions, and we are happy to announce a deployed version of our solution at https://deepware.ai. Once again, we would like to thank the amazing Kaggle community and especially the Facebook team and everyone who worked hard on the Deepfakes problem to provide novel solutions.
", "messages": [ { "id": 825006, "postDate": "2020-04-28T18:14:35.333Z", "content": "First of all, we congratulate the winners of the Deepfake Detection competition and also would like to thank every competitor who worked hard to solve the deepfakes problem. We learned a lot throughout this amazing journey with you all, and we hope to see you again in the upcoming Deepfake challenges.
\n\nBefore intensely experimenting with deep neural network models, we decided to inspect the deepfake data with established computer vision techniques, to see if there's a simple cue which can reveal a deepfake, having these features in our mind for later feature-engineering purposes. Initially, we could observe indicators of deepfakes in frame inspections; such small artifacts were too hard to generalize for a novel solution. Although these experiments haven't resulted with great success, they provided a better understanding of deepfakes for us. Later on, we started experimenting with deep learnig based models, which then showed significant performance increase in detecting fake faces.
\n\nFor face extraction and processing purposes, we used FaceNet-Pytorch,
\n\nAlso, we used InceptionResNET features to differentiate individuals in the same frame from each other.
\n\nMargin: After square cropping faces with respect to bounding boxes, we’ve scaled each face with “1.2”, which is determined by experimentation.
During our experiments, we have used different levels of augmentation techniques to increase robustness towards uncertainty in the wild.
\n\ndef chain_aug(img):\n return A.Compose([A.HorizontalFlip(p=1),\n A.JpegCompression(quality_lower=70, quality_upper=100, p=1.),\n A.Blur(blur_limit=7, p=1.),\n A.IAASharpen(alpha=(0.2, 0.5), lightness=(0.5, 1.0), p=1.)\n ], p=1)(image=img)['image']\ndef light_aug(img):\n return A.Compose([A.HorizontalFlip(p=1),\n A.RandomBrightnessContrast(p=1),\n A.RandomGamma(p=1),\n A.CLAHE(p=1), ], p=1)(image=img)['image']\ndef draw_ellipse(img):\n img = Image.fromarray(img)\n draw = ImageDraw.Draw(img)\n w, h = img.size\n cx, cy = w//2, h//2\n x0 = np.random.randint(0, cx)\n y0 = np.random.randint(0, cy)\n x1 = np.random.randint(cx, w)\n y1 = np.random.randint(cy, h)\n draw.ellipse((x0, y0, x1, y1), fill='black')\n return np.asarray(img)\nEarly studies preferred backbones such as ResNet, Xception, VGG whereas we have updated our backbone choice as EfficientNet. The main idea with the backbone network was to train a simple binary classifier and to discriminate between real and fake faces. Experiments for binary-classification models started from EfficientNet-b0 and scaled up to EfficientNet-b5. Here’s a couple of things we’ve tried.\n- Low res effnet-b3 + Effnet-b5 (224x224) 0.30656\n- Effnet-b4 0.38793 (224x224)\n- Effnet-b2 0.43068 (224x224)\n- Triplet loss models (224x224)\n- An ensemble of XGB, LGB, CATBoost, KNN models
\n\nFor final submissions:\n- An ensemble of EfficientNets: 0.31499 [FAILED] (Although scripts worked Kaggle submissions, as well as tests, sadly, our best performing model failed on private board scoring.)\n- Ensembles of EfficientNets + Other models mentioned above: 0.33552
\n\nIn our development environment, we used Nvidia-docker, with a 4x 2080TI Lambda workstation.
\n\nAt first, like many competitors, we measured our performance with 400 public videos, after some time, we switched to folders 0-1-2. Later on, our team developed a validation split strategy based on similarity scores.
\n\nDuring the competition, we were tracking;\n- Log loss score for each class\n- Accuracy, confusion matrix scores\n- Error, gradient map analysis with custom debug scripts
\n\nThrough the DFDC journey, we have had many challenges of generalizing our models to deepfakes in the wild. Training with only the DFDC dataset, our models resulted in high performance with \"face swap\" auto-encoder based generation techniques. Still, they failed to generalize other variations of deepfakes and generation techniques. We are very excited for future contributions, and we are happy to announce a deployed version of our solution at https://deepware.ai. Once again, we would like to thank the amazing Kaggle community and especially the Facebook team and everyone who worked hard on the Deepfakes problem to provide novel solutions.
", "rawMarkdown": "First of all, we congratulate the winners of the Deepfake Detection competition and also would like to thank every competitor who worked hard to solve the deepfakes problem. We learned a lot throughout this amazing journey with you all, and we hope to see you again in the upcoming Deepfake challenges.\n\nBefore intensely experimenting with deep neural network models, we decided to inspect the deepfake data with established computer vision techniques, to see if there's a simple cue which can reveal a deepfake, having these features in our mind for later feature-engineering purposes. Initially, we could observe indicators of deepfakes in frame inspections; such small artifacts were too hard to generalize for a novel solution. Although these experiments haven't resulted with great success, they provided a better understanding of deepfakes for us. Later on, we started experimenting with deep learnig based models, which then showed significant performance increase in detecting fake faces.\n## Face Extraction\nFor face extraction and processing purposes, we used FaceNet-Pytorch,\n\n- Extracted landmarks and bounding boxes via MTCNN, then faces were cropped and saved in PNG format for high-quality image representation.\n- Also, we used InceptionResNET features to differentiate individuals in the same frame from each other.\n## Data Preparation\n- **Margin**: After square cropping faces with respect to bounding boxes, we’ve scaled each face with “1.2”, which is determined by experimentation.\n- **Resize**: Scaled faces resized to 224x224, with BICUBIC sampling.\n- **Augmentations**: We’ve experimented with “kornia” and “albumentations”.\n- Used techniques such as Adaptive Histogram Equalization, Object Detection algorithms to increase detection robustness.\n- Fine-tuned MTCNN for robust and rapid face detection.\n- Used specific methods such as Compression, Edge-detection, Landmark Extraction, and Low-Quality to generate several datasets.\n- Additionally extracted features such as Quality Metrics, Optical Flow, FFT, DoG, Glcm, Lbp\n\n## Augmentations and generalization\nDuring our experiments, we have used different levels of augmentation techniques to increase robustness towards uncertainty in the wild. \n\n def chain_aug(img):\n return A.Compose([A.HorizontalFlip(p=1),\n A.JpegCompression(quality_lower=70, quality_upper=100, p=1.),\n A.Blur(blur_limit=7, p=1.),\n A.IAASharpen(alpha=(0.2, 0.5), lightness=(0.5, 1.0), p=1.)\n ], p=1)(image=img)['image']\n@deepware Thanks for sharing! Many things to learn from it.
", "rawMarkdown": "@deepware Thanks for sharing! Many things to learn from it.", "votes": 1 }, { "id": 827222, "postDate": "2020-04-30T06:33:44.210Z", "content": "Thanks for sharing! Gonna test your agumentations.
", "rawMarkdown": "Thanks for sharing! Gonna test your agumentations. ", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 826771, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-04-29T21:11:59.350000", "content": "@deepware Thanks for sharing! Many things to learn from it.
", "votes": 1, "replies": [] }, { "id": 827222, "author_name": "", "author_url": "", "post_date": "2020-04-30T06:33:44.210000", "content": "Thanks for sharing! Gonna test your agumentations.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "825006": "First of all, we congratulate the winners of the Deepfake Detection competition and also would like to thank every competitor who worked hard to solve the deepfakes problem. We learned a lot throughout this amazing journey with you all, and we hope to see you again in the upcoming Deepfake challenges.\n\nBefore intensely experimenting with deep neural network models, we decided to inspect the deepfake data with established computer vision techniques, to see if there's a simple cue which can reveal a deepfake, having these features in our mind for later feature-engineering purposes. Initially, we could observe indicators of deepfakes in frame inspections; such small artifacts were too hard to generalize for a novel solution. Although these experiments haven't resulted with great success, they provided a better understanding of deepfakes for us. Later on, we started experimenting with deep learnig based models, which then showed significant performance increase in detecting fake faces.\n## Face Extraction\nFor face extraction and processing purposes, we used FaceNet-Pytorch,\n\n- Extracted landmarks and bounding boxes via MTCNN, then faces were cropped and saved in PNG format for high-quality image representation.\n- Also, we used InceptionResNET features to differentiate individuals in the same frame from each other.\n## Data Preparation\n- **Margin**: After square cropping faces with respect to bounding boxes, we’ve scaled each face with “1.2”, which is determined by experimentation.\n- **Resize**: Scaled faces resized to 224x224, with BICUBIC sampling.\n- **Augmentations**: We’ve experimented with “kornia” and “albumentations”.\n- Used techniques such as Adaptive Histogram Equalization, Object Detection algorithms to increase detection robustness.\n- Fine-tuned MTCNN for robust and rapid face detection.\n- Used specific methods such as Compression, Edge-detection, Landmark Extraction, and Low-Quality to generate several datasets.\n- Additionally extracted features such as Quality Metrics, Optical Flow, FFT, DoG, Glcm, Lbp\n\n## Augmentations and generalization\nDuring our experiments, we have used different levels of augmentation techniques to increase robustness towards uncertainty in the wild. \n\n def chain_aug(img):\n return A.Compose([A.HorizontalFlip(p=1),\n A.JpegCompression(quality_lower=70, quality_upper=100, p=1.),\n A.Blur(blur_limit=7, p=1.),\n A.IAASharpen(alpha=(0.2, 0.5), lightness=(0.5, 1.0), p=1.)\n ], p=1)(image=img)['image']\n