{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Baseline submission using Facenet\n\nThis notebook demonstrates how to use the `facenet-pytorch` package to build a rudimentary deepfake detector without training any models. It also demonstrates a method for (1) loading all video frames, (2) finding all faces, and (3) calculating face embeddings at over 30 frames per second (or greater than 1 video per 10 seconds).\n\nThe following steps are performed:\n\n1. Create pretrained facial detection (MTCNN) and recognition (Inception Resnet) models.\n  * See the following kernel for a strided implementation of MTCNN that is able to process all frames in each video: https://www.kaggle.com/timesler/facenet-pytorch-mtcnn-process-every-frame\n  * See the following kernel for a performance comparison for different face detection implementations: https://www.kaggle.com/timesler/comparison-of-face-detection-packages\n1. For each test video, calculate face feature vectors for **ALL** faces in each video.\n1. Calculate the distance from each face to the centroid for its video.\n1. Use these distances as your means of discrimination.\n\nFor (much) better results, finetune the resnet to the fake/real binary classification task instead - this is just a baseline. Alternatively, I'm sure there is much more interesting things that can be done with the feature vectors."},{"metadata":{},"cell_type":"markdown","source":"## Install dependencies"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Install facenet-pytorch\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.7-py3-none-any.whl\n\nfrom facenet_pytorch.models.inception_resnet_v1 import get_torch_home\ntorch_home = get_torch_home()\n\n# Copy model checkpoints to torch cache so they are loaded automatically by the package\n!mkdir -p $torch_home/checkpoints/\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-logits.pth $torch_home/checkpoints/vggface2_DG3kwML46X.pt\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-features.pth $torch_home/checkpoints/vggface2_G5aNV2VSMn.pt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport time\nimport torch\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom tqdm.notebook import tqdm\n\n# See github.com/timesler/facenet-pytorch:\nfrom facenet_pytorch import MTCNN, InceptionResnetV1, extract_face\n\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create MTCNN and Inception Resnet models\n\nBoth models are pretrained. The Inception Resnet weights will be downloaded the first time it is instantiated; after that, they will be loaded from the torch cache."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load face detector\nmtcnn = MTCNN(margin=14, keep_all=True, factor=0.5, device=device).eval()\n\n# Load facial recognition model\nresnet = InceptionResnetV1(pretrained='vggface2', device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Process test videos\n\nAfter defining a few helper functions, this code loops through all videos and passes **_all_** frames from each through the face detector followed by facenet. Finally, we calculate the distance from the centroid to the extracted feature for each face."},{"metadata":{"trusted":true},"cell_type":"code","source":"class DetectionPipeline:\n    \"\"\"Pipeline class for detecting faces in the frames of a video file.\"\"\"\n    \n    def __init__(self, detector, n_frames=None, batch_size=60, resize=None):\n        \"\"\"Constructor for DetectionPipeline class.\n        \n        Keyword Arguments:\n            n_frames {int} -- Total number of frames to load. These will be evenly spaced\n                throughout the video. If not specified (i.e., None), all frames will be loaded.\n                (default: {None})\n            batch_size {int} -- Batch size to use with MTCNN face detector. (default: {32})\n            resize {float} -- Fraction by which to resize frames from original prior to face\n                detection. A value less than 1 results in downsampling and a value greater than\n                1 result in upsampling. (default: {None})\n        \"\"\"\n        self.detector = detector\n        self.n_frames = n_frames\n        self.batch_size = batch_size\n        self.resize = resize\n    \n    def __call__(self, filename):\n        \"\"\"Load frames from an MP4 video and detect faces.\n\n        Arguments:\n            filename {str} -- Path to video.\n        \"\"\"\n        # Create video reader and find length\n        v_cap = cv2.VideoCapture(filename)\n        v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n        # Pick 'n_frames' evenly spaced frames to sample\n        if self.n_frames is None:\n            sample = np.arange(0, v_len)\n        else:\n            sample = np.linspace(0, v_len - 1, self.n_frames).astype(int)\n\n        # Loop through frames\n        faces = []\n        frames = []\n        for j in range(v_len):\n            success = v_cap.grab()\n            if j in sample:\n                # Load frame\n                success, frame = v_cap.retrieve()\n                if not success:\n                    continue\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n                frame = Image.fromarray(frame)\n                \n                # Resize frame to desired size\n                if self.resize is not None:\n                    frame = frame.resize([int(d * self.resize) for d in frame.size])\n                frames.append(frame)\n\n                # When batch is full, detect faces and reset frame list\n                if len(frames) % self.batch_size == 0 or j == sample[-1]:\n                    faces.extend(self.detector(frames))\n                    frames = []\n\n        v_cap.release()\n\n        return faces    \n\n\ndef process_faces(faces, resnet):\n    # Filter out frames without faces\n    faces = [f for f in faces if f is not None]\n    faces = torch.cat(faces).to(device)\n\n    # Generate facial feature vectors using a pretrained model\n    embeddings = resnet(faces)\n\n    # Calculate centroid for video and distance of each face's feature vector from centroid\n    centroid = embeddings.mean(dim=0)\n    x = (embeddings - centroid).norm(dim=1).cpu().numpy()\n    \n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define face detection pipeline\ndetection_pipeline = DetectionPipeline(detector=mtcnn, batch_size=60, resize=0.25)\n\n# Get all test videos\nfilenames = glob.glob('/kaggle/input/deepfake-detection-challenge/test_videos/*.mp4')\n\nX = []\nstart = time.time()\nn_processed = 0\nwith torch.no_grad():\n    for i, filename in tqdm(enumerate(filenames), total=len(filenames)):\n        try:\n            # Load frames and find faces\n            faces = detection_pipeline(filename)\n            \n            # Calculate embeddings\n            X.append(process_faces(faces, resnet))\n\n        except KeyboardInterrupt:\n            print('\\nStopped.')\n            break\n\n        except Exception as e:\n            print(e)\n            X.append(None)\n        \n        n_processed += len(faces)\n        print(f'Frames per second (load+detect+embed): {n_processed / (time.time() - start):6.3}\\r', end='')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict classes\n\nThe below weights were selected by following the same process as above for the train sample videos and then using a logistic regression model to fit to the labels. Note that, intuitively, this is not a very good approach as it does nothing to take into account the progression of feature vectors throughout a video, just combines them together using the weights below. This step is provided as a placeholder only; it should be replaced with a more thoughtful mapping from a sequence of feature vectors to a single prediction."},{"metadata":{"trusted":true},"cell_type":"code","source":"bias = -0.2942\nweight = 0.68235746\n\nsubmission = []\nfor filename, x_i in zip(filenames, X):\n    if x_i is not None:\n        prob = 1 / (1 + np.exp(-(bias + (weight * x_i).mean())))\n    else:\n        prob = 0.5\n    submission.append([os.path.basename(filename), prob])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(submission, columns=['filename', 'label'])\nsubmission.sort_values('filename').to_csv('submission.csv', index=False)\n\nplt.hist(submission.label, 20)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}