{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Fast MTCNN detector\n\nThis notebook demonstrates how to achieve 45 frames per second speeds for loading frames and detecting faces on full resolution videos.\n\n## Algorithm\n\n**Striding**: The algorithm used is a strided modification of MTCNN in which face detection is performed on only every _N_ frames, and applied to all frames. For example, with a batch of 9 frames, we could pass frames 0, 3, and 6 to MTCNN. Then, the bounding boxes (and potentially landmarks) returned for frame 0 would be naively applied to frames 1 and 2. Similarly, the detections for frame 3 are applied to frames 4 and 5, and the detections for frames 6 are applied to frames 7 and 8.\n\nAlthough this assume that faces do not move between frames significantly, this is generally a good approximation for low stride numbers. If the stride is 3, we are assuming that the face does not significantly alter position for an additional 2 frames, or ~0.07 seconds. If faces are moving faster than this, they are likely to be extremely blurry anyway. Furthermore, ensuring that faces are cropped with a small margin mitigates the impact of face drift.\n\n**Scale pyramid**: The algorithm uses a slightly smaller scaling factor (0.6 vs 0.709) than the original MTCNN algorithm to construct the scaling pyramid applied to input images. For details of the scaling pyramid, see the [original paper](https://arxiv.org/abs/1604.02878) for details of the scaling pyramid approach.\n\n**Multi-threading**: A modest performance gain comes from loading video frames (with `cv2.VideoCapture`) using threading. This functionality is provided by the `FileVideoStream` class of the imutils package.\n\n## Other resources\n\nSee the following kernel for a guide to using the MTCNN functionality of facenet-pytorch: https://www.kaggle.com/timesler/guide-to-mtcnn-in-facenet-pytorch","metadata":{}},{"cell_type":"code","source":"%%capture\n# Install facenet-pytorch (with internet use \"pip install facenet-pytorch\")\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.7-py3-none-any.whl\n#!pip install /kaggle/input/imutils/imutils-0.5.3\n!pip install /kaggle/input/d/datasets/meckdahl/imutils/imutils-0.5.3","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-06-12T02:59:04.786574Z","iopub.execute_input":"2022-06-12T02:59:04.786882Z","iopub.status.idle":"2022-06-12T02:59:57.291950Z","shell.execute_reply.started":"2022-06-12T02:59:04.786826Z","shell.execute_reply":"2022-06-12T02:59:57.290942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"from facenet_pytorch import MTCNN\nfrom PIL import Image\nimport torch\nfrom imutils.video import FileVideoStream\nimport cv2\nimport time\nimport glob\nfrom tqdm.notebook import tqdm\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\n\nfilenames = glob.glob('/kaggle/input/deepfake-detection-challenge/test_videos/*.mp4')[:100]","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2022-06-12T03:00:45.782573Z","iopub.execute_input":"2022-06-12T03:00:45.782910Z","iopub.status.idle":"2022-06-12T03:00:46.036335Z","shell.execute_reply.started":"2022-06-12T03:00:45.782854Z","shell.execute_reply":"2022-06-12T03:00:46.035535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The FastMTCNN class\n\nThe class below is a thin wrapper for the MTCNN implementation in the `facenet-pytorch` package that implements the algorithm described above.","metadata":{}},{"cell_type":"code","source":"class FastMTCNN(object):\n    \"\"\"Fast MTCNN implementation.\"\"\"\n    \n    def __init__(self, stride, resize=1, *args, **kwargs):\n        \"\"\"Constructor for FastMTCNN class.\n        \n        Arguments:\n            stride (int): The detection stride. Faces will be detected every `stride` frames\n                and remembered for `stride-1` frames.\n        \n        Keyword arguments:\n            resize (float): Fractional frame scaling. [default: {1}]\n            *args: Arguments to pass to the MTCNN constructor. See help(MTCNN).\n            **kwargs: Keyword arguments to pass to the MTCNN constructor. See help(MTCNN).\n        \"\"\"\n        self.stride = stride\n        self.resize = resize\n        self.mtcnn = MTCNN(*args, **kwargs)\n        \n    def __call__(self, frames):\n        \"\"\"Detect faces in frames using strided MTCNN.\"\"\"\n        if self.resize != 1:\n            frames = [\n                cv2.resize(f, (int(f.shape[1] * self.resize), int(f.shape[0] * self.resize)))\n                    for f in frames\n            ]\n                      \n        boxes, probs = self.mtcnn.detect(frames[::self.stride])\n\n        faces = []\n        for i, frame in enumerate(frames):\n            box_ind = int(i / self.stride)\n            if boxes[box_ind] is None:\n                continue\n            for box in boxes[box_ind]:\n                box = [int(b) for b in box]\n                faces.append(frame[box[1]:box[3], box[0]:box[2]])\n        \n        return faces","metadata":{"execution":{"iopub.status.busy":"2022-06-12T03:00:54.241683Z","iopub.execute_input":"2022-06-12T03:00:54.242029Z","iopub.status.idle":"2022-06-12T03:00:54.254326Z","shell.execute_reply.started":"2022-06-12T03:00:54.241971Z","shell.execute_reply":"2022-06-12T03:00:54.253311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Full resolution detection\n\nIn this example, we demonstrate how to detect faces using full resolution frames (i.e., `resize=1`).","metadata":{}},{"cell_type":"code","source":"fast_mtcnn = FastMTCNN(\n    stride=4,\n    resize=1,\n    margin=14,\n    factor=0.6,\n    keep_all=True,\n    device=device\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-12T03:01:18.217749Z","iopub.execute_input":"2022-06-12T03:01:18.218094Z","iopub.status.idle":"2022-06-12T03:01:18.243040Z","shell.execute_reply.started":"2022-06-12T03:01:18.218039Z","shell.execute_reply":"2022-06-12T03:01:18.242376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def run_detection(fast_mtcnn, filenames):\n    frames = []\n    frames_processed = 0\n    faces_detected = 0\n    batch_size = 60\n    start = time.time()\n\n    for filename in tqdm(filenames):\n\n        v_cap = FileVideoStream(filename).start()\n        v_len = int(v_cap.stream.get(cv2.CAP_PROP_FRAME_COUNT))\n\n        for j in range(v_len):\n\n            frame = v_cap.read()\n            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n            frames.append(frame)\n\n            if len(frames) >= batch_size or j == v_len - 1:\n\n                faces = fast_mtcnn(frames)\n\n                frames_processed += len(frames)\n                faces_detected += len(faces)\n                frames = []\n\n                print(\n                    f'Frames per second: {frames_processed / (time.time() - start):.3f},',\n                    f'faces detected: {faces_detected}\\r',\n                    end=''\n                )\n\n        v_cap.stop()\n\nrun_detection(fast_mtcnn, filenames)","metadata":{"execution":{"iopub.status.busy":"2022-06-12T03:01:22.354612Z","iopub.execute_input":"2022-06-12T03:01:22.354951Z","iopub.status.idle":"2022-06-12T03:10:13.761844Z","shell.execute_reply.started":"2022-06-12T03:01:22.354889Z","shell.execute_reply":"2022-06-12T03:10:13.761057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Half resolution detection\n\nIn this example, we demonstrate how to detect faces using half resolution frames (i.e., `resize=0.5`).","metadata":{}},{"cell_type":"code","source":"fast_mtcnn = FastMTCNN(\n    stride=4,\n    resize=0.5,\n    margin=14,\n    factor=0.5,\n    keep_all=True,\n    device=device\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-12T03:17:11.265625Z","iopub.execute_input":"2022-06-12T03:17:11.265956Z","iopub.status.idle":"2022-06-12T03:17:11.288761Z","shell.execute_reply.started":"2022-06-12T03:17:11.265898Z","shell.execute_reply":"2022-06-12T03:17:11.288098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run_detection(fast_mtcnn, filenames)","metadata":{"execution":{"iopub.status.busy":"2022-06-12T03:17:15.964421Z","iopub.execute_input":"2022-06-12T03:17:15.964737Z","iopub.status.idle":"2022-06-12T03:24:40.588079Z","shell.execute_reply.started":"2022-06-12T03:17:15.964682Z","shell.execute_reply":"2022-06-12T03:24:40.587325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}