{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":16880,"databundleVersionId":858837,"sourceType":"competition"},{"sourceId":1003630,"sourceType":"datasetVersion","datasetId":442595}],"dockerImageVersionId":29845,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook aims to demonstrate the different ways to use the MTCNN face detection module of `facenet-pytorch`. Originally reported in [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://arxiv.org/abs/1604.02878), the MTCNN network is able to simultaneously propose bounding boxes, five-point facial landmarks, and detection probabilities. Taken from the original paper:\n\n> Face detection and alignment in unconstrained environments are challenging due to various poses, illuminations and occlusions. Recent studies show that deep learning approaches can achieve impressive performance on these two tasks. In this paper, we propose a deep cascaded multi-task framework which exploits the inherent correlation between them to boost up their performance. In particular, our framework adopts a cascaded structure with three stages of carefully designed deep convolutional networks that predict face and landmark location in a coarse-to-fine manner. In addition, in the learning process, we propose a new online hard sample mining strategy that can improve the performance automatically without manual sample selection. Our method achieves superior accuracy over the state-of-the-art techniques on the challenging FDDB and WIDER FACE benchmark for face detection, and AFLW benchmark for face alignment, while keeps real time performance.\n\n`facenet-pytorch` includes an efficient, cuda-ready implementation of MTCNN that will be demonstrated in this notebook. The following topics will be covered:\n\n1. <a href='#1'>Documentation</a>\n1. <a href='#2'>Basic usage</a>\n1. <a href='#3'>Preventing image normalization</a>\n1. <a href='#4'>Margin adjustment</a>\n1. <a href='#5'>Multiple faces in a single image</a>\n1. <a href='#6'>Batched detection</a>\n1. <a href='#7'>Bounding boxes and facial landmarks</a>\n1. <a href='#8'>Saving face datasets</a>\n\nOther resources:\n\n1. The facenet-pytorch [github repo](https://github.com/timesler/facenet-pytorch)\n1. [Notebook demonstrating combined use of face detection and recognition](https://www.kaggle.com/timesler/facial-recognition-model-in-pytorch)\n1. [The FastMTCNN algorithm](https://www.kaggle.com/timesler/fast-mtcnn-detector-45-fps-at-full-resolution) ","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.7-py3-none-any.whl","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:33:18.939521Z","iopub.execute_input":"2024-10-23T11:33:18.939814Z","iopub.status.idle":"2024-10-23T11:33:46.781613Z","shell.execute_reply.started":"2024-10-23T11:33:18.939755Z","shell.execute_reply":"2024-10-23T11:33:46.780673Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from facenet_pytorch import MTCNN\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom tqdm.notebook import tqdm","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:33:53.747461Z","iopub.execute_input":"2024-10-23T11:33:53.747829Z","iopub.status.idle":"2024-10-23T11:33:55.466561Z","shell.execute_reply.started":"2024-10-23T11:33:53.747757Z","shell.execute_reply":"2024-10-23T11:33:55.465870Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='1'>Documentation</a>\n\nDetailed usage information is contained in the MTCNN docstring:\n\n```\nhelp(MTCNN)\n```","metadata":{}},{"cell_type":"markdown","source":"# <a id='2'>Basic usage</a>\n\nUnlike other implementations, calling a `facenet-pytorch` MTCNN object directly with an image (i.e., using the forward method for those familiar with pytorch) will return torch tensors containing the detected face(s), rather than just the bounding boxes. This is to enable using the module easily as the first stage of a facial recognition pipeline, in which the faces are passed directly to an additional network or algorithm.\n\nIn order to return the detected boxes instead (and optionally, the facial landmarks), see the `MTCNN.detect()` method. Its use will be described below also.\n\nTo create an MTCNN detector that runs on the GPU, instantiate the model with `device='cuda:0'` or equivalent.\n\nFor this competition, it will be best to set `select_largest=False` to ensure detected faces are ordered according to detection probability rather than size.","metadata":{}},{"cell_type":"code","source":"# Create face detector\nmtcnn = MTCNN(select_largest=False, device='cuda')\n\n# Load a single image and display\nv_cap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/train_sample_videos/agqphdxmwt.mp4')\nsuccess, frame = v_cap.read()\nframe = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\nframe = Image.fromarray(frame)\n\nplt.figure(figsize=(12, 8))\nplt.imshow(frame)\nplt.axis('off')\n\n# Detect face\nface = mtcnn(frame)\nface.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:34:28.852178Z","iopub.execute_input":"2024-10-23T11:34:28.852575Z","iopub.status.idle":"2024-10-23T11:34:34.670616Z","shell.execute_reply.started":"2024-10-23T11:34:28.852517Z","shell.execute_reply":"2024-10-23T11:34:34.669971Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='3'>Preventing image normalization</a>\n\nBy default, the MTCNN module of `facenet-pytorch` applies fixed image standardization to faces before returning so they are well suited for the package's face recognition model.\n\nIf you want to get out images that look more normal to the human eye, this normalization can be prevented by creating the detector with `post_process=False`.","metadata":{}},{"cell_type":"code","source":"# Create face detector\nmtcnn = MTCNN(select_largest=False, post_process=False, device='cuda:0')\n\n# Detect face\nface = mtcnn(frame)\n\n# Visualize\nplt.imshow(face.permute(1, 2, 0).int().numpy())\nplt.axis('off');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:35:40.193293Z","iopub.execute_input":"2024-10-23T11:35:40.193620Z","iopub.status.idle":"2024-10-23T11:35:40.369603Z","shell.execute_reply.started":"2024-10-23T11:35:40.193575Z","shell.execute_reply":"2024-10-23T11:35:40.368884Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='4'>Margin adjustment</a>\n\nDepending on your downstream processing and how fakes can be identified, you may want to add more (or less) of a margin around the detected faces. This is controlled using the `margin` argument.","metadata":{}},{"cell_type":"code","source":"# Create face detector\nmtcnn = MTCNN(margin=40, select_largest=False, post_process=False, device='cuda:0')\n\n# Detect face\nface = mtcnn(frame)\n\n# Visualize\nplt.imshow(face.permute(1, 2, 0).int().numpy())\nplt.axis('off');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:36:02.212512Z","iopub.execute_input":"2024-10-23T11:36:02.212934Z","iopub.status.idle":"2024-10-23T11:36:02.476358Z","shell.execute_reply.started":"2024-10-23T11:36:02.212873Z","shell.execute_reply":"2024-10-23T11:36:02.475312Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='5'>Multiple faces in a single image</a>\n\nUsing MTCNN as above will only return a single face from each frame (or None if none are detected). Since some of the videos in the dataset contain more than one face, you will likely want to return all detected faces as any/all of them may have been modified. This is acheived by setting `keep_all=True`","metadata":{}},{"cell_type":"code","source":"# Create face detector\nmtcnn = MTCNN(margin=20, keep_all=True, post_process=False, device='cuda:0')\n\n# Load a single image and display\nv_cap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/train_sample_videos/avibnnhwhp.mp4')\nsuccess, frame = v_cap.read()\nframe = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\nframe = Image.fromarray(frame)\n\nplt.figure(figsize=(12, 8))\nplt.imshow(frame)\nplt.axis('off')\nplt.show()\n\n# Detect face\nfaces = mtcnn(frame)\n\n# Visualize\nfig, axes = plt.subplots(1, len(faces))\nfor face, ax in zip(faces, axes):\n    ax.imshow(face.permute(1, 2, 0).int().numpy())\n    ax.axis('off')\nfig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:37:19.041549Z","iopub.execute_input":"2024-10-23T11:37:19.041917Z","iopub.status.idle":"2024-10-23T11:37:19.758977Z","shell.execute_reply.started":"2024-10-23T11:37:19.041870Z","shell.execute_reply":"2024-10-23T11:37:19.757939Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='6'>Batched detection</a>\n\n`facenet-pytorch` is also capable of performing face detection on batches of images, typically providing considerable speed-up. A batch should be structured as list of PIL images of equal dimension. The returned object will have an additional first dimension corresponding to the batch. Each image in the batch may have one or more faces detected.\n\nIn the following example, we use MTCNN to detect multiple faces in:\n1. A single batch of frames, and\n1. Every frame of a video","metadata":{}},{"cell_type":"code","source":"# Create face detector\nmtcnn = MTCNN(margin=20, keep_all=True, post_process=False, device='cuda:0')\n\n# Load a video\nv_cap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/train_sample_videos/avibnnhwhp.mp4')\nv_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n# Loop through video, taking a handful of frames to form a batch\nframes = []\nfor i in tqdm(range(v_len)):\n    \n    # Load frame\n    success = v_cap.grab()\n    if i % 50 == 0:\n        success, frame = v_cap.retrieve()\n    else:\n        continue\n    if not success:\n        continue\n        \n    # Add to batch\n    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    frames.append(Image.fromarray(frame))\n\n# Detect faces in batch\nfaces = mtcnn(frames)\n\nfig, axes = plt.subplots(len(faces), 2, figsize=(6, 15))\nfor i, frame_faces in enumerate(faces):\n    for j, face in enumerate(frame_faces):\n        axes[i, j].imshow(face.permute(1, 2, 0).int().numpy())\n        axes[i, j].axis('off')\nfig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:37:53.632350Z","iopub.execute_input":"2024-10-23T11:37:53.632650Z","iopub.status.idle":"2024-10-23T11:37:56.068322Z","shell.execute_reply.started":"2024-10-23T11:37:53.632606Z","shell.execute_reply":"2024-10-23T11:37:56.067612Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The following example uses a similar approach to detect all faces in all frames in a video.","metadata":{}},{"cell_type":"code","source":"# Load a video\nv_cap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/train_sample_videos/avibnnhwhp.mp4')\nv_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n# Loop through video\nbatch_size = 16\nframes = []\nfaces = []\nfor _ in tqdm(range(v_len)):\n    \n    # Load frame\n    success, frame = v_cap.read()\n    if not success:\n        continue\n        \n    # Add to batch\n    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    frames.append(Image.fromarray(frame))\n    \n    # When batch is full, detect faces and reset batch list\n    if len(frames) >= batch_size:\n        faces.extend(mtcnn(frames))\n        frames = []\n\nplt.figure(figsize=(12, 4))\nplt.plot([len(f) for f in faces])\nplt.title('Detected faces per frame');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:38:15.411369Z","iopub.execute_input":"2024-10-23T11:38:15.411859Z","iopub.status.idle":"2024-10-23T11:38:32.392791Z","shell.execute_reply.started":"2024-10-23T11:38:15.411630Z","shell.execute_reply":"2024-10-23T11:38:32.390883Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='7'>Bounding boxes and facial landmarks</a>\n\nTo return bounding boxes and facial landmarks from MTCNN, instead of calling the `mtcnn` object directly, call `mtcnn.detect()` instead.\n\nUnlike the forward method (shown in each of examples above), the `.detect()` method will always return all detected bounding boxes (and optional landmarks) in an image.\n\nThe following example demonstrates the use of the `.detect()` method on a single image.\n\nNote that the `margin` argument, if used when creating the MTCNN detector, is not used in the `detect()` method. `detect()` returns the true bounding boxes, so the margin can be applied subsequently by the user if desired.","metadata":{}},{"cell_type":"code","source":"# Create face detector\nmtcnn = MTCNN(keep_all=True, device='cuda:0')\n\n# Load a single image and display\nv_cap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/train_sample_videos/agqphdxmwt.mp4')\nsuccess, frame = v_cap.read()\nframe = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\nframe = Image.fromarray(frame)\n\n# Detect face\nboxes, probs, landmarks = mtcnn.detect(frame, landmarks=True)\n\n# Visualize\nfig, ax = plt.subplots(figsize=(16, 12))\nax.imshow(frame)\nax.axis('off')\n\nfor box, landmark in zip(boxes, landmarks):\n    ax.scatter(*np.meshgrid(box[[0, 2]], box[[1, 3]]))\n    ax.scatter(landmark[:, 0], landmark[:, 1], s=8)\nfig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:40:20.376006Z","iopub.execute_input":"2024-10-23T11:40:20.376346Z","iopub.status.idle":"2024-10-23T11:40:20.938716Z","shell.execute_reply.started":"2024-10-23T11:40:20.376297Z","shell.execute_reply":"2024-10-23T11:40:20.937746Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The following example demonstrates how to show bounding boxes and facial landmarks in every frame in a video.","metadata":{}},{"cell_type":"code","source":"# Load a video\nv_cap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/train_sample_videos/avibnnhwhp.mp4')\n\n# Loop through video\nbatch_size = 32\nframes = []\nboxes = []\nlandmarks = []\nview_frames = []\nview_boxes = []\nview_landmarks = []\nfor _ in tqdm(range(v_len)):\n    \n    # Load frame\n    success, frame = v_cap.read()\n    if not success:\n        continue\n        \n    # Add to batch, resizing for speed\n    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    frame = Image.fromarray(frame)\n    frame = frame.resize([int(f * 0.25) for f in frame.size])\n    frames.append(frame)\n    \n    # When batch is full, detect faces and reset batch list\n    if len(frames) >= batch_size:\n        batch_boxes, _, batch_landmarks = mtcnn.detect(frames, landmarks=True)\n        boxes.extend(batch_boxes)\n        landmarks.extend(batch_landmarks)\n        \n        view_frames.append(frames[-1])\n        view_boxes.append(boxes[-1])\n        view_landmarks.append(landmarks[-1])\n        \n        frames = []\n\n# Visualize\nfig, ax = plt.subplots(3, 3, figsize=(18, 12))\nfor i in range(9):\n    ax[int(i / 3), i % 3].imshow(view_frames[i])\n    ax[int(i / 3), i % 3].axis('off')\n    for box, landmark in zip(view_boxes[i], view_landmarks[i]):\n        ax[int(i / 3), i % 3].scatter(*np.meshgrid(box[[0, 2]], box[[1, 3]]), s=8)\n        ax[int(i / 3), i % 3].scatter(landmark[:, 0], landmark[:, 1], s=6)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:40:33.274363Z","iopub.execute_input":"2024-10-23T11:40:33.274677Z","iopub.status.idle":"2024-10-23T11:40:40.782927Z","shell.execute_reply.started":"2024-10-23T11:40:33.274632Z","shell.execute_reply":"2024-10-23T11:40:40.781438Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <a id='8'>Saving face datasets</a>\n\nIn order to save detected faces directly to file, use MTCNN's `save_path` argument in the forward function. This is compatible with both single images and batch processing.\n\n- For single images, pass a single path string (e.g., '{videoname}\\_{frame}.jpg')\n- For batches of images, pass a list of path strings (one for each frame)\n\nWhen multiple faces are detected in a single image, additional faces are each saved with an incremental integer appended to the end of the save path (e.g., '{videoname}\\_{frame}.jpg' and '{videoname}\\_{frame}\\_1.jpg')\n\nSee example below.","metadata":{}},{"cell_type":"code","source":"# Single image\nmtcnn(frame, save_path='single_image.jpg');\n\n# Batch\nsave_paths = [f'image_{i}.jpg' for i in range(len(frames))]\nmtcnn(frames, save_path=save_paths);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-10-23T11:41:34.080550Z","iopub.execute_input":"2024-10-23T11:41:34.080902Z","iopub.status.idle":"2024-10-23T11:41:34.347878Z","shell.execute_reply.started":"2024-10-23T11:41:34.080843Z","shell.execute_reply":"2024-10-23T11:41:34.347071Z"}},"outputs":[],"execution_count":null}]}