{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Baseline submission using Facenet\n\nThis notebook demonstrates how to use the `facenet-pytorch` package to build a rudimentary deepfake detector without training any models.\nThe following steps are performed:\n\n1. Create pretrained facial detection (MTCNN) and recognition (Inception Resnet) models.\n1. For each test video, calculate face feature vectors for N faces evenly spaced through each video.\n1. Calculate the distance from each face to the centroid for its video.\n1. Use these distances as your means of discrimination.\n\nFor (much) better results, finetune the resnet to the fake/real binary classification task instead - this is just a baseline. Alternatively, I'm sure there is much more interesting things that can be done with the feature vectors."},{"metadata":{},"cell_type":"markdown","source":"## Install dependencies"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%capture\n# Install facenet-pytorch\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-1.0.1-py3-none-any.whl\n\n# Copy model checkpoints to torch cache so they are loaded automatically by the package\n!mkdir -p /tmp/.cache/torch/checkpoints/\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-logits.pth /tmp/.cache/torch/checkpoints/vggface2_DG3kwML46X.pt\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-features.pth /tmp/.cache/torch/checkpoints/vggface2_G5aNV2VSMn.pt\n\n# Install ffmpeg\n! tar xvf ../input/ffmpeg-static-build/ffmpeg-git-amd64-static.tar.xz","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport torch\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom tqdm import tqdm_notebook\n\n# See github.com/timesler/facenet-pytorch:\nfrom facenet_pytorch import MTCNN, InceptionResnetV1\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create MTCNN and Inception Resnet models\n\nBoth models are pretrained. The Inception Resnet weights will be downloaded the first time it is instantiated; after that, they will be loaded from the torch cache."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load face detector\nmtcnn = MTCNN(device='cuda').eval()\n\n# Load facial recognition model\nresnet = InceptionResnetV1(pretrained='vggface2', num_classes=2, device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Process test videos\n\nLoop through all videos and pass N frames from each through the face detector followed by facenet. Calculate the distance from the centroid to the extracted feature for each face."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get all test videos\nfilenames = glob.glob('/kaggle/input/deepfake-detection-challenge/test_videos/*.mp4')\n\n# Number of frames to sample (evenly spaced) from each video\nn_frames = 10\n\nX = []\nwith torch.no_grad():\n    for i, filename in enumerate(filenames):\n        print(f'Processing {i+1:5n} of {len(filenames):5n} videos\\r', end='')\n        \n        try:\n            # Create video reader and find length\n            v_cap = cv2.VideoCapture(filename)\n            v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n            \n            # Pick 'n_frames' evenly spaced frames to sample\n            sample = np.linspace(0, v_len - 1, n_frames).round().astype(int)\n            imgs = []\n            for j in range(v_len):\n                success = v_cap.grab()\n                if j in sample:\n                    success, vframe = v_cap.retrieve()\n                    vframe = cv2.cvtColor(vframe, cv2.COLOR_BGR2RGB)\n                    imgs.append(Image.fromarray(vframe))\n            v_cap.release()\n            \n            # Pass image batch to MTCNN as a list of PIL images\n            faces = mtcnn(imgs)\n            \n            # Filter out frames without faces\n            faces = [f for f in faces if f is not None]\n            faces = torch.stack(faces).cuda()\n            \n            # Generate facial feature vectors using a pretrained model\n            embeddings = resnet(faces).cuda()\n            \n            # Calculate centroid for video and distance of each face's feature vector from centroid\n            centroid = embeddings.mean(dim=0)\n            X.append((embeddings - centroid).norm(dim=1).cpu().numpy())\n        except KeyboardInterrupt:\n            raise Exception(\"Stopped.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.save('X.npy',X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict classes\n\nThe below weights were selected by following the same process as above for the train sample videos and then using a logistic regression model to fit to the labels. Note that, intuitively, this is not a very good approach as it does nothing to take into account the progression of feature vectors throughout a video, just combines them together using the weights below. This step is provided as a placeholder only; it should be replaced with a more thoughtful mapping from a sequence of feature vectors to a single prediction."},{"metadata":{"trusted":true},"cell_type":"code","source":"bias = -0.2942\nweight = 0.068235746\n\nsubmission = []\nfor filename, x_i in zip(filenames, X):\n    if x_i is not None and len(x_i) == 10:\n        prob = 1 / (1 + np.exp(-(bias + (weight * x_i).sum())))\n    else:\n        prob = 0.5\n    submission.append([os.path.basename(filename), prob])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(submission, columns=['filename', 'label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('../input/deepfake-detection-challenge/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result_map=dict(zip(submission.filename,submission.label))\nsub['label']=sub['filename'].map(result_map)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.label = np.where(sub['label']>0.5,1,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}