{"cells":[{"metadata":{},"cell_type":"markdown","source":"<a id=\"toc\"></a>\n# Table of Contents\n1. [Introduction](#introduction)\n1. [Configure hyper-parameters](#configure_hyper_parameters)\n1. [Install required libraries and tools](#install_required_libraries_and_tools)\n1. [Import libraries](#import_libraries)\n1. [Define FaceExtractor](#define_face_extractor)\n1. [Get metadata](#get_metadata)\n1. [Start the detection process](#start_the_detection_process)\n1. [Save metadata.csv](#save_metadata_csv)\n1. [Compress results](#compress_results)\n1. [Conclusion](#conclusion)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"introduction\"></a>\n# Introduction\nThis notebook dedicated to extracting all faces appears in each video of the training dataset. The name of each face image corresponds to the index of the frame that this face appears, plus a suffix _2, or _3, etc. if the number of faces in a frame is greater than 1.\n\n---\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"configure_hyper_parameters\"></a>\n# Configure hyper-parameters\n[Back to Table of Contents](#toc)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"TRAIN_DIR = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/'\nTMP_DIR = '/kaggle/tmp/'\nZIP_NAME = 'dfdc_train_faces_sample.zip'\nMETADATA_PATH = TRAIN_DIR + 'metadata.json'\n\nSCALE = 0.25\nN_FRAMES = None","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"install_required_libraries_and_tools\"></a>\n# Install required libraries and tools\n[Back to Table of Contents](#toc)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install facenet-pytorch > /dev/null 2>&1\n!apt install zip > /dev/null 2>&1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"import_libraries\"></a>\n# Import libraries\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport json\nimport torch\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom tqdm.notebook import tqdm\nfrom facenet_pytorch import MTCNN\n\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"define_face_extractor\"></a>\n# Define FaceExtractor\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class FaceExtractor:\n    def __init__(self, detector, n_frames=None, resize=None):\n        \"\"\"\n        Parameters:\n            n_frames {int} -- Total number of frames to load. These will be evenly spaced\n                throughout the video. If not specified (i.e., None), all frames will be loaded.\n                (default: {None})\n            resize {float} -- Fraction by which to resize frames from original prior to face\n                detection. A value less than 1 results in downsampling and a value greater than\n                1 result in upsampling. (default: {None})\n        \"\"\"\n\n        self.detector = detector\n        self.n_frames = n_frames\n        self.resize = resize\n    \n    def __call__(self, filename, save_dir):\n        \"\"\"Load frames from an MP4 video, detect faces and save the results.\n\n        Parameters:\n            filename {str} -- Path to video.\n            save_dir {str} -- The directory where results are saved.\n        \"\"\"\n\n        # Create video reader and find length\n        v_cap = cv2.VideoCapture(filename)\n        v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n        # Pick 'n_frames' evenly spaced frames to sample\n        if self.n_frames is None:\n            sample = np.arange(0, v_len)\n        else:\n            sample = np.linspace(0, v_len - 1, self.n_frames).astype(int)\n\n        # Loop through frames\n        for j in range(v_len):\n            success = v_cap.grab()\n            if j in sample:\n                # Load frame\n                success, frame = v_cap.retrieve()\n                if not success:\n                    continue\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n                frame = Image.fromarray(frame)\n                \n                # Resize frame to desired size\n                if self.resize is not None:\n                    frame = frame.resize([int(d * self.resize) for d in frame.size])\n\n                save_path = os.path.join(save_dir, f'{j}.png')\n\n                self.detector([frame], save_path=save_path)\n\n        v_cap.release()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"get_metadata\"></a>\n# Get metadata\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(METADATA_PATH, 'r') as f:\n    metadata = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.DataFrame(\n    [\n        (video_file, metadata[video_file]['label'], metadata[video_file]['split'], metadata[video_file]['original'] if 'original' in metadata[video_file].keys() else '')\n        for video_file in metadata.keys()\n    ],\n    columns=['filename', 'label', 'split', 'original']\n)\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"start_the_detection_process\"></a>\n# Start the detection process\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load face detector\nface_detector = MTCNN(margin=14, keep_all=True, factor=0.5, device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define face extractor\nface_extractor = FaceExtractor(detector=face_detector, n_frames=N_FRAMES, resize=SCALE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get the paths of all train videos\nall_train_videos = glob.glob(os.path.join(TRAIN_DIR, '*.mp4'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir -p $TMP_DIR","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with torch.no_grad():\n    for path in tqdm(all_train_videos):\n        file_name = path.split('/')[-1]\n\n        save_dir = os.path.join(TMP_DIR, file_name.split(\".\")[0])\n\n        if not os.path.exists(save_dir):\n            os.makedirs(save_dir)\n\n        # Detect all faces appear in the video and save them.\n        face_extractor(path, save_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"save_metadata_csv\"></a>\n# Save metadata.csv\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"cd $TMP_DIR","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.to_csv('metadata.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"compress_results\"></a>\n# Compress results\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!zip -r -m -q /kaggle/working/$ZIP_NAME *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"conclusion\"></a>\n# Conclusion\nData preprocessing is an integral part of machine learning and usually costs data scientists lots of time and effort. Sometimes, this process can even take weeks to be done before we can move on to the next step, and the huge dataset from the Deepfake Detection Challenge gives me the feeling that I must spend even months to digest all of it with my \"Napoleon\" PC. So, to worth the time and effort I've spent when struggling with this nearly-half-terabyte dataset, I think it's better to share all of my work, hope that they will be useful for many others.\n\nBesides the result of this demo on the sample training dataset, I have finished some small parts of the whole dataset which download links are shared in my [*Other useful datasets*](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/128954) discussion. I'll gradually update the other parts in the next few days and public a kernel to demonstrate how to load and prepare these data to train a simple classifier as soon as possible.\n\n---\nFor some who want a demo of how to load and merge multiple datasets at once before training a classifier, let check this [*kernel*](https://www.kaggle.com/phunghieu/loading-merging-multiple-kaggle-datasets-demo).\n\nFor training and inference examples, let check [*DFDC-Multiface-Training*](https://www.kaggle.com/phunghieu/dfdc-multiface-training) and [*\nDFDC-Multiface-Inference*](https://www.kaggle.com/phunghieu/dfdc-multiface-inference).\n\n---\nIf you think this kernel is worth your time of reading, consider to upvote it :3\n\nMany thanks! ^^\n\n---\n[Back to Table of Contents](#toc)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}