{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n# Table of Contents\n1. [Introduction](#introduction)\n1. [Install libraries and packages](#install_libraries_and_packages)\n1. [Import libraries](#import_libraries)\n1. [Configure hyper-parameters](#configure_hyper_parameters)\n1. [Define useful classes](#define_useful_classes)\n1. [Start the inference process](#start_the_inference_process)\n1. [Save the submission](#save_the_submission)\n1. [Conclusion](#conclusion)"},{"metadata":{},"cell_type":"markdown","source":"\n# Introduction\n\nSo, I have successfully trained a classifier by using a bunch of datasets listed in the [*Other useful datasets*](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/128954) discussion and upload it into Kaggle as an external [*dataset*](https://www.kaggle.com/phunghieu/dfdcmultifacef5-resnet18). Now, let's use this model to infer all videos in the test set then complete this end-to-end solution by submitting the final result to the host :) .\n\nIf you do not know how to train the classifier, please follow this [*link*](https://www.kaggle.com/phunghieu/dfdc-multiface-training).\n\n---\n## Multiface's general diagram\n\n\n---\n## Implementation\nI will loop through all test videos and try to get face images by using the same strategy as I have applied to the validation process in the [*DFDC-Multiface-Training*](https://www.kaggle.com/phunghieu/dfdc-multiface-training) kernel. The only difference is instead of having well-prepared data, I must run a face-detector, the same as I used to prepare the training dataset in the [*Data Preparation*](https://www.kaggle.com/phunghieu/deepfake-detection-face-extractor) kernel, to directly extract faces from each frame of one input video.\n\nIf I fail to get enough faces from a video, I will mark it as `invalid` and assign a `default predicted value` (probability) to this video near the end of the notebook.\n\n---\n## Pipeline\nThis end-to-end solution includes 3 steps:\n1. [*Data Preparation*](https://www.kaggle.com/phunghieu/deepfake-detection-face-extractor)\n1. [*Training*](https://www.kaggle.com/phunghieu/dfdc-multiface-training)\n1. *Inference* <- **you're here**\n\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"\n# Install libraries and packages\n[Back to Table of Contents](#toc)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Install facenet-pytorch\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.0.0-py3-none-any.whl\n\nfrom facenet_pytorch.models.inception_resnet_v1 import get_torch_home\ntorch_home = get_torch_home()\n\n# Copy model checkpoints to torch cache so they are loaded automatically by the package\n!mkdir -p $torch_home/checkpoints/\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-logits.pth $torch_home/checkpoints/vggface2_DG3kwML46X.pt\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-features.pth $torch_home/checkpoints/vggface2_G5aNV2VSMn.pt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Import libraries\n[Back to Table of Contents](#toc)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import torch\nfrom torch import nn\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.models import resnet18\nfrom facenet_pytorch import MTCNN\nfrom albumentations import Normalize, Compose\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nimport os\nimport glob\nimport multiprocessing as mp\n\nif torch.cuda.is_available():\n device = 'cuda:0'\n torch.set_default_tensor_type('torch.cuda.FloatTensor')\nelse:\n device = 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Configure hyper-parameters\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_DIR = '/kaggle/input/deepfake-detection-challenge/test_videos/'\nMODEL_PATH = '/kaggle/input/dfdcmultifacef5-resnet18/f5_resnet18.pth'\n\nN_FACES = 5\nBATCH_SIZE = 64\nNUM_WORKERS = mp.cpu_count()\n\nFRAME_SCALE = 0.25\nFACE_BATCH_SHAPE = (N_FACES*3, 160, 160)\n\nDEFAULT_PROB = 0.5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Define useful classes\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DeepfakeClassifier(nn.Module):\n def __init__(self, encoder, in_channels=3, num_classes=1):\n super(DeepfakeClassifier, self).__init__()\n self.encoder = encoder\n \n # Modify input layer.\n self.encoder.conv1 = nn.Conv2d(\n in_channels,\n 64,\n kernel_size=7,\n stride=2,\n padding=3,\n bias=False\n )\n \n # Modify output layer.\n self.encoder.fc = nn.Linear(512 * 1, num_classes)\n\n def forward(self, x):\n return torch.sigmoid(self.encoder(x))\n \n def freeze_all_layers(self):\n for param in self.encoder.parameters():\n param.requires_grad = False\n\n def freeze_middle_layers(self):\n self.freeze_all_layers()\n \n for param in self.encoder.conv1.parameters():\n param.requires_grad = True\n \n for param in self.encoder.fc.parameters():\n param.requires_grad = True\n\n def unfreeze_all_layers(self):\n for param in self.encoder.parameters():\n param.requires_grad = True\n \n \nclass TestVideoDataset(Dataset):\n def __init__(self, test_dir, frame_resize=None, face_detector=None, n_faces=1, preprocess=None):\n self.test_dir = test_dir\n self.test_video_paths = glob.glob(os.path.join(self.test_dir, '*.mp4'))\n self.face_detector = face_detector\n self.n_faces = n_faces\n self.frame_resize = frame_resize\n self.preprocess = preprocess\n\n def __len__(self):\n return len(self.test_video_paths)\n \n def __getitem__(self, idx):\n if torch.is_tensor(idx):\n idx = idx.tolist()\n\n test_video_path = self.test_video_paths[idx]\n test_video = test_video_path.split('/')[-1]\n \n # Get faces until enough (try limit: n_faces)\n faces = []\n \n for i in range(self.n_faces):\n # Create video reader and find length\n v_cap = cv2.VideoCapture(test_video_path)\n v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n \n stride = int(v_len/(self.n_faces**2))\n sample = np.linspace(i*stride, (v_len - 1) + i*stride, self.n_faces).astype(int)\n frames = []\n\n # Get frames\n for j in range(v_len):\n success = v_cap.grab()\n \n if j in sample:\n success, frame = v_cap.retrieve()\n\n if not success:\n continue\n\n frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n frame = Image.fromarray(frame)\n\n # Resize frame to desired size\n if self.frame_resize is not None:\n frame = frame.resize([int(d * self.frame_resize) for d in frame.size])\n frames.append(frame)\n \n if len(frames) > 0:\n all_faces_in_frames = [\n detected_face\n for detected_faces in self.face_detector(frames)\n if detected_faces is not None\n for detected_face in detected_faces\n ]\n\n faces.extend(all_faces_in_frames)\n \n if len(faces) >= self.n_faces: # Get enough faces\n break\n\n v_cap.release()\n\n if len(faces) >= self.n_faces: # Get enough faces\n faces = faces[:self.n_faces] # Get top\n \n if self.preprocess is not None:\n for j in range(len(faces)):\n augmented = self.preprocess(image=faces[j].cpu().detach().numpy().transpose(1, 2, 0))\n faces[j] = augmented['image']\n \n faces = np.concatenate(faces, axis=-1).transpose(2, 0, 1)\n \n return {\n 'video_name': test_video,\n 'faces': faces,\n 'is_valid': True\n }\n else:\n return {\n 'video_name': test_video,\n 'faces': np.zeros(FACE_BATCH_SHAPE, dtype=np.float32),\n 'is_valid': False # Those invalid videos will get DEFAULT_PROB\n }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Start the inference process\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Load face detector.\nface_detector = MTCNN(margin=14, keep_all=True, factor=0.5, post_process=False, device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"encoder = resnet18(pretrained=False)\n\nclassifier = DeepfakeClassifier(encoder=encoder, in_channels=3*N_FACES, num_classes=1)\nclassifier.to(device)\nstate = torch.load(MODEL_PATH, map_location=lambda storage, loc: storage)\nclassifier.load_state_dict(state['state_dict'])\nclassifier.eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preprocess = Compose([\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], p=1)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = TestVideoDataset(\n TEST_DIR,\n frame_resize=FRAME_SCALE,\n face_detector=face_detector,\n n_faces=N_FACES,\n preprocess=preprocess\n)\n\ntest_dataloader = DataLoader(\n test_dataset,\n batch_size=BATCH_SIZE,\n shuffle=False,\n num_workers=NUM_WORKERS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = []\n\nwith torch.no_grad():\n try:\n for videos in tqdm(test_dataloader):\n y_pred = classifier(videos['faces']).squeeze(dim=-1).cpu().detach().numpy()\n submission.extend(list(zip(videos['video_name'], y_pred, videos['is_valid'].cpu().detach().numpy())))\n except Exception as e:\n print(e)\n \nsubmission = pd.DataFrame(submission, columns=['filename', 'label', 'is_valid'])\nsubmission.sort_values('filename', inplace=True)\nsubmission.loc[submission.is_valid == False, 'label'] = DEFAULT_PROB","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Save the submission\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission[['filename', 'label']].to_csv('submission.csv', index=False)\n\nplt.hist(submission.label, 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Conclusion\nFinally, we made it! Let's submit the result to see whether we can get a better position in the Public Leaderboard =]]\n\nIf you have any questions or suggestions, feel free to move to the `comments` section below.\n\n---\nThe content in this notebook is too complicated to understand and you want a simpler solution to get started, I have already prepared one for you based on `@timesler`'s solution in this series:\n* [Deepfake Detection - Data Preparation (baseline)](https://www.kaggle.com/phunghieu/deepfake-detection-data-preparation-baseline)\n* [Deepfake Detection - Training (baseline)](https://www.kaggle.com/phunghieu/deepfake-detection-training-baseline)\n* [Deepfake Detection - Inference (baseline)](https://www.kaggle.com/phunghieu/deepfake-detection-inference-baseline)\n\n---\nPlease upvote this kernel if you think it is worth reading.\n\nThank you so much!\n\n[Back to Table of Contents](#toc)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}