{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Comparison of face detection packages"},{"metadata":{},"cell_type":"markdown","source":"This notebook demonstrates the use of three face detection packages:\n\n1. facenet-pytorch: https://pypi.org/project/facenet-pytorch/\n1. mtcnn: https://pypi.org/project/mtcnn/\n1. dlib: https://pypi.org/project/dlib/\n\nThanks to Carlos Souza and \"Need to think of a good name\" for creating the datasets used to install the packages offline.\n\nEach package is tested for its speed in detecting the faces in a set of 300 images (all frames from one video), with GPU support enabled. Detection is performed at 3 different resolutions. Any one-off initialization steps, such as model instantiation, are performed prior to performance testing.\n\nPlease let me know if you know of better/faster ways to use these packages.\n\n*****\n**UPDATE (2020-01-05)**: facenet-pytorch version 2.0.0 has been released, offering some additional performance gains, particularly for batched processing.\n*****\n\n## Summary of results\n\n|Package|FPS (1080x1920)|FPS (720x1280)|FPS (540x960)|\n|-|-|-|\n|***facenet-pytorch***|12.97|20.32|25.50|\n|***facenet-pytorch (non-batched)***|9.75|14.81|19.68|\n|***dlib***|3.80|8.39|14.53|\n|***mtcnn***|3.04|5.70|8.23|\n\n## Other resources\n\nSee the following kernel for a guide to using the MTCNN functionality of facenet-pytorch: https://www.kaggle.com/timesler/guide-to-mtcnn-in-facenet-pytorch"},{"metadata":{},"cell_type":"markdown","source":"## Install packages\n\nNormally, each package can be installed with `pip install <package>`, but this notebook is offline to demonstrate their use in this competition."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%capture\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.7-py3-none-any.whl\n!pip install /kaggle/input/dlibpkg/dlib-19.19.0\n!pip install /kaggle/input/mtcnn-package/mtcnn-0.1.0-py3-none-any.whl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import cv2\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom PIL import Image\nimport torch\nfrom tqdm.notebook import tqdm\nimport time\n\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Read in the frames of a video using cv2's `VideoCapture`."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/aagfhgtpmv.mp4'\n\nreader = cv2.VideoCapture(sample)\nimages_1080_1920 = []\nimages_720_1280 = []\nimages_540_960 = []\nfor i in tqdm(range(int(reader.get(cv2.CAP_PROP_FRAME_COUNT)))):\n    _, image = reader.read()\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    images_1080_1920.append(image)\n    images_720_1280.append(cv2.resize(image, (1280, 720)))\n    images_540_960.append(cv2.resize(image, (960, 540)))\nreader.release()\n\nimages_1080_1920 = np.stack(images_1080_1920)\nimages_720_1280 = np.stack(images_720_1280)\nimages_540_960 = np.stack(images_540_960)\n\nprint('Shapes:')\nprint(images_1080_1920.shape)\nprint(images_720_1280.shape)\nprint(images_540_960.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_faces(images, figsize=(10.8/2, 19.2/2)):\n    shape = images[0].shape\n    images = images[np.linspace(0, len(images)-1, 16).astype(int)]\n    im_plot = []\n    for i in range(0, 16, 4):\n        im_plot.append(np.concatenate(images[i:i+4], axis=0))\n    im_plot = np.concatenate(im_plot, axis=1)\n    \n    fig, ax = plt.subplots(1, 1, figsize=figsize)\n    ax.imshow(im_plot)\n    ax.xaxis.set_visible(False)\n    ax.yaxis.set_visible(False)\n\n    ax.grid(False)\n    fig.tight_layout()\n\ndef timer(detector, detect_fn, images, *args):\n    start = time.time()\n    faces = detect_fn(detector, images, *args)\n    elapsed = time.time() - start\n    print(f', {elapsed:.3f} seconds')\n    return faces, elapsed","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_faces(images_540_960, figsize=(10.8, 19.2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The facenet-pytorch package"},{"metadata":{"trusted":true},"cell_type":"code","source":"from facenet_pytorch import MTCNN\ndetector = MTCNN(device=device, post_process=False)\n\ndef detect_facenet_pytorch(detector, images, batch_size):\n    faces = []\n    for lb in np.arange(0, len(images), batch_size):\n        imgs = [img for img in images[lb:lb+batch_size]]\n        faces.extend(detector(imgs))\n    return faces\n\ntimes_facenet_pytorch = []    # batched\ntimes_facenet_pytorch_nb = [] # non-batched","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Detecting faces in 540x960 frames', end='')\n_, elapsed = timer(detector, detect_facenet_pytorch, images_540_960, 60)\ntimes_facenet_pytorch.append(elapsed)\n\nprint('Detecting faces in 720x1280 frames', end='')\n_, elapsed = timer(detector, detect_facenet_pytorch, images_720_1280, 40)\ntimes_facenet_pytorch.append(elapsed)\n\nprint('Detecting faces in 1080x1920 frames', end='')\nfaces, elapsed = timer(detector, detect_facenet_pytorch, images_1080_1920, 20)\ntimes_facenet_pytorch.append(elapsed)\n\nplot_faces(torch.stack(faces).permute(0, 2, 3, 1).int().numpy())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The facenet-pytorch package (non-batched)"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Detecting faces in 540x960 frames', end='')\n_, elapsed = timer(detector, detect_facenet_pytorch, images_540_960, 1)\ntimes_facenet_pytorch_nb.append(elapsed)\n\nprint('Detecting faces in 720x1280 frames', end='')\n_, elapsed = timer(detector, detect_facenet_pytorch, images_720_1280, 1)\ntimes_facenet_pytorch_nb.append(elapsed)\n\nprint('Detecting faces in 1080x1920 frames', end='')\nfaces, elapsed = timer(detector, detect_facenet_pytorch, images_1080_1920, 1)\ntimes_facenet_pytorch_nb.append(elapsed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del detector\ntorch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The dlib package"},{"metadata":{"trusted":true},"cell_type":"code","source":"from dlib import get_frontal_face_detector\ndetector = get_frontal_face_detector()\n\ndef detect_dlib(detector, images):\n    faces = []\n    for image in images:\n        image_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n        boxes = detector(image_gray)\n        box = boxes[0]\n        face = image[box.top():box.bottom(), box.left():box.right()]\n        faces.append(face)\n    return faces\n\ntimes_dlib = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Detecting faces in 540x960 frames', end='')\n_, elapsed = timer(detector, detect_dlib, images_540_960)\ntimes_dlib.append(elapsed)\n\nprint('Detecting faces in 720x1280 frames', end='')\n_, elapsed = timer(detector, detect_dlib, images_720_1280)\ntimes_dlib.append(elapsed)\n\nprint('Detecting faces in 1080x1920 frames', end='')\nfaces, elapsed = timer(detector, detect_dlib, images_1080_1920)\ntimes_dlib.append(elapsed)\n\nplot_faces(np.stack([cv2.resize(f, (160, 160)) for f in faces]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del detector\ntorch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The mtcnn package"},{"metadata":{"trusted":true},"cell_type":"code","source":"from mtcnn import MTCNN\ndetector = MTCNN()\n\ndef detect_mtcnn(detector, images):\n    faces = []\n    for image in images:\n        boxes = detector.detect_faces(image)\n        box = boxes[0]['box']\n        face = image[box[1]:box[3]+box[1], box[0]:box[2]+box[0]]\n        faces.append(face)\n    return faces\n\ntimes_mtcnn = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Detecting faces in 540x960 frames', end='')\n_, elapsed = timer(detector, detect_mtcnn, images_540_960)\ntimes_mtcnn.append(elapsed)\n\nprint('Detecting faces in 720x1280 frames', end='')\n_, elapsed = timer(detector, detect_mtcnn, images_720_1280)\ntimes_mtcnn.append(elapsed)\n\nprint('Detecting faces in 1080x1920 frames', end='')\nfaces, elapsed = timer(detector, detect_mtcnn, images_1080_1920)\ntimes_mtcnn.append(elapsed)\n\nplot_faces(np.stack([cv2.resize(face, (160, 160)) for face in faces]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del detector\ntorch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Performance comparison"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10,6))\n\npos = np.arange(3)\nplt.bar(pos, times_facenet_pytorch, 0.2, label='facenet-pytorch')\nplt.bar(pos + 0.2, times_facenet_pytorch_nb, 0.2, label='facenet-pytorch (non-batched)')\nplt.bar(pos + 0.4, times_dlib, 0.2, label='dlib')\nplt.bar(pos + 0.6, times_mtcnn, 0.2, label='mtcnn')\n\nax.set_ylabel('Elapsed time (seconds)')\nax.set_xticks(pos + 0.25)\nax.set_xticklabels(['540x960', '720x1280', '1080x1920'])\nplt.legend();","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}