{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Baseline submission using Facenet\n\nThis notebook demonstrates how to use the `facenet-pytorch` package to build a rudimentary deepfake detector without training any models.\nThe following steps are performed:\n\n1. Create pretrained facial detection (MTCNN) and recognition (Inception Resnet) models.\n1. For each test video, calculate face feature vectors for N faces evenly spaced through each video.\n1. Calculate the distance from each face to the centroid for its video.\n1. Use these distances as your means of discrimination.\n\nFor (much) better results, finetune the resnet to the fake/real binary classification task instead - this is just a baseline. Alternatively, I'm sure there is much more interesting things that can be done with the feature vectors."},{"metadata":{},"cell_type":"markdown","source":"#### Install dependencies"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%capture\n# Install facenet-pytorch\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-1.0.1-py3-none-any.whl\n\n# Copy model checkpoints to torch cache so they are loaded automatically by the package\n!mkdir -p /tmp/.cache/torch/checkpoints/\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-logits.pth /tmp/.cache/torch/checkpoints/vggface2_DG3kwML46X.pt\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-features.pth /tmp/.cache/torch/checkpoints/vggface2_G5aNV2VSMn.pt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport torch\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\n# See github.com/timesler/facenet-pytorch:\nfrom facenet_pytorch import MTCNN, InceptionResnetV1\n\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Create MTCNN and Inception Resnet models\n\nBoth models are pretrained. The Inception Resnet weights will be downloaded the first time it is instantiated; after that, they will be loaded from the torch cache."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load face detector\nmtcnn = MTCNN(device=device).eval()\n\n# Load facial recognition model\nresnet = InceptionResnetV1(pretrained='vggface2', num_classes=2, device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Process test videos\n\nLoop through all videos and pass N frames from each through the face detector followed by facenet. Calculate the distance from the centroid to the extracted feature for each face."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get all test videos\nfilenames = glob.glob('/kaggle/input/deepfake-detection-challenge/test_videos/*.mp4')\n\nn_frames = 10\nX = []\nwith torch.no_grad():\n    for i, filename in enumerate(filenames):\n        print(f'Processing {i+1:5n} of {len(filenames):5n} videos\\r', end='')\n        try:\n            v_cap = cv2.VideoCapture(filename)\n            v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n            sample = np.linspace(0, v_len - 1, n_frames).round().astype(int)\n            imgs = []\n            for j in range(v_len):\n                success, vframe = v_cap.read()\n                vframe = cv2.cvtColor(vframe, cv2.COLOR_BGR2RGB)\n                if j in sample:\n                    imgs.append(Image.fromarray(vframe))\n            v_cap.release()\n            faces = mtcnn(imgs)\n            faces = [f for f in faces if f is not None]\n            faces = torch.stack(faces).to(device)\n            embeddings = resnet(faces)\n            centroid = embeddings.mean(dim=0)\n            X.append((embeddings - centroid).norm(dim=1).cpu().numpy())\n        except:\n            X.append(None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Predict classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"bias = -0.31777231\nweights = np.array([\n    0.13553764,\n    0.3220863,\n    0.03451057,\n    0.20375968,\n    0.13409478,\n    -0.16768392,\n    0.30386854,\n    0.02896564,\n    0.3549986,\n    -0.66778037\n])\n\nsubmission = []\nfor filename, x_i in zip(filenames, X):\n    if x_i is not None and len(x_i) == 10:\n        prob = 1 / (1 + np.exp(-(bias + np.matmul(weights, x_i))))\n    else:\n        prob = 0.5\n    submission.append([os.path.basename(filename), prob])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Build submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(submission, columns=['filename', 'label'])\nsubmission.sort_values('filename').to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}