{"cells":[{"metadata":{},"cell_type":"markdown","source":"# GRU model using Facenet embeddings\n\nThis notebook is based on the following notebook:\nhttps://www.kaggle.com/timesler/facial-recognition-model-in-pytorch\n\nThe difference is that this notebook uses GRU, a sequence based model to classify the videos to fake/real. The input of the model is a sequence of embeddings of faces from the video frames. It uses Facenet MTCNN for face detection and Inception Resnet for creating embeddings from faces. In the training of the GRU, I have used only the first 50 faces from the videos and only one face per frame. It could be an improvement to use all of the frames and possibly multiple faces from one frame.\n\nThe GRU model is trained with Keras Tensorflow."},{"metadata":{},"cell_type":"markdown","source":"## Install dependencies"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Install facenet-pytorch\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.0.0-py3-none-any.whl\n\nfrom facenet_pytorch.models.inception_resnet_v1 import get_torch_home\ntorch_home = get_torch_home()\n\n# Copy model checkpoints to torch cache so they are loaded automatically by the package\n!mkdir -p $torch_home/checkpoints/\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-logits.pth $torch_home/checkpoints/vggface2_DG3kwML46X.pt\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-features.pth $torch_home/checkpoints/vggface2_G5aNV2VSMn.pt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport time\nimport torch\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom tqdm.notebook import tqdm\n\n# See github.com/timesler/facenet-pytorch:\nfrom facenet_pytorch import MTCNN, InceptionResnetV1, extract_face\n\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create MTCNN and Inception Resnet models\n\nBoth models are pretrained. The Inception Resnet weights will be downloaded the first time it is instantiated; after that, they will be loaded from the torch cache."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load face detector\nmtcnn = MTCNN(margin=14, keep_all=False,  select_largest=True, factor=0.5, device=device).eval()\n\n# Load facial recognition model\nresnet = InceptionResnetV1(pretrained='vggface2', device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Process test videos\n\nAfter defining a few helper functions, this code loops through all videos and passes **_all_** frames from each through the face detector followed by facenet."},{"metadata":{"trusted":true},"cell_type":"code","source":"class DetectionPipeline:\n    \"\"\"Pipeline class for detecting faces in the frames of a video file.\"\"\"\n    \n    def __init__(self, detector, batch_size=50, resize=None):\n        \"\"\"Constructor for DetectionPipeline class.\n        \n        Keyword Arguments:\n            batch_size {int} -- Batch size to use with MTCNN face detector. (default: {32})\n            resize {float} -- Fraction by which to resize frames from original prior to face\n                detection. A value less than 1 results in downsampling and a value greater than\n                1 result in upsampling. (default: {None})\n        \"\"\"\n        self.detector = detector\n        self.batch_size = batch_size\n        self.resize = resize\n    \n    def __call__(self, filename):\n        \"\"\"Load frames from an MP4 video and detect faces.\n\n        Arguments:\n            filename {str} -- Path to video.\n        \"\"\"\n        # Create video reader and find length\n        v_cap = cv2.VideoCapture(filename)\n        v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n        \n        sample = np.arange(0, v_len)\n       \n        # Loop through frames\n        faces = []\n        frames = []\n        for j in range(50):\n            success = v_cap.grab()\n            if j in sample:\n                # Load frame\n                success, frame = v_cap.retrieve()\n                if not success:\n                    continue\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n                frame = Image.fromarray(frame)\n                \n                # Resize frame to desired size\n                if self.resize is not None:\n                    frame = frame.resize([int(d * self.resize) for d in frame.size])\n                frames.append(frame)\n\n                # When batch is full, detect faces and reset frame list\n                #if len(frames) % self.batch_size == 0 or j == sample[-1]:\n                if j == 49:\n                    faces.extend(self.detector(frames))\n                    frames = []\n                    v_cap.release()\n                    return faces\n\ndef process_faces(faces, resnet):\n    # Filter out frames without faces\n    faces = [f for f in faces if f is not None]\n    faces = torch.stack(faces).to(device)\n    # use torch.cat to concatenate along the same dimension - if keepAll\n\n    # Generate facial feature vectors using a pretrained model\n    embeddings = resnet(faces)\n\n    return embeddings.cpu().numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define face detection pipeline\ndetection_pipeline = DetectionPipeline(detector=mtcnn, batch_size=60, resize=0.25)\n\n# Get all test videos\nfilenames = glob.glob('/kaggle/input/deepfake-detection-challenge/test_videos/*.mp4')\n\nX = []\nstart = time.time()\nn_processed = 0\nwith torch.no_grad():\n    for i, filename in tqdm(enumerate(filenames), total=len(filenames)):\n        try:\n            # Load frames and find faces\n            faces = detection_pipeline(filename)\n            embeddings = process_faces(faces, resnet)\n\n            # Calculate embeddings\n            X.append(embeddings)\n\n        except KeyboardInterrupt:\n            print('\\nStopped.')\n            break\n\n        except Exception as e:\n            print(e)\n            X.append(None)\n        \n        n_processed += len(faces)\n        print(f'Frames per second (load+detect+embed): {n_processed / (time.time() - start):6.3}\\r', end='')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load GRU mdel"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.models import load_model\n\ndeepfake_detector = load_model('/kaggle/input/alldetector/model_all.hdf5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def pad_sample(sample):\n    result = np.zeros((50, 512))\n    result[:sample.shape[0],:sample.shape[1]] = sample\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make predictions with the GRU model"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = []\nfor filename, x_i in zip(filenames, X):\n    if x_i is not None:\n        x_i = pad_sample(x_i)\n        x_i = np.expand_dims(x_i, axis=0)\n        prob = deepfake_detector.predict(x_i)[0][0]\n    else:\n        prob = 0.5\n    submission.append([os.path.basename(filename), prob])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(submission, columns=['filename', 'label'])\nsubmission.sort_values('filename').to_csv('submission.csv', index=False)\n\nplt.hist(submission.label, 20)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}