{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Working with facenet-pytorch and decord\n\nAs of version 2.2, the MTCNN module of facenet-pytorch can work directly with images represented as numpy arrays. This change achieves higher performance when reading video frames with either `cv2.VideoCapture` or `decord.VideoReader` as it avoids conversion to PIL format. A number of additional enhancements have been added to improve detection efficiency.\n\n**This notebook demonstrates how to detect every face in every frame in every video of the dataset at full resolution in approximately 3 hours.**\n\n---\n\n**UPDATE (2020-03-04):** Video reading has been switched from cv2 to decord for improved performance.\n\n---"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%capture\n# Install facenet-pytorch (with internet use \"pip install facenet-pytorch\")\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.9-py3-none-any.whl\n!cp /kaggle/input/decord/install.sh . && chmod  +x install.sh && ./install.sh","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import sys, os\nsys.path.insert(0,'/kaggle/working/reader/python')\n\nfrom facenet_pytorch import MTCNN\nimport torch\nimport cupy\nfrom decord import VideoReader, gpu\nimport glob\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The FastMTCNN Class\n\nThe following class implements a strided version of MTCNN. See [here](https://www.kaggle.com/timesler/fast-mtcnn-detector-55-fps-at-full-resolution) for the original implementation."},{"metadata":{"trusted":true},"cell_type":"code","source":"class FastMTCNN(object):\n    \"\"\"Fast MTCNN implementation.\"\"\"\n    \n    def __init__(self, stride, *args, **kwargs):\n        \"\"\"Constructor for FastMTCNN class.\n        \n        Arguments:\n            stride (int): The detection stride. Faces will be detected every `stride` frames\n                and remembered for `stride-1` frames.\n        \n        Keyword arguments:\n            resize (float): Fractional frame scaling. [default: {1}]\n            *args: Arguments to pass to the MTCNN constructor. See help(MTCNN).\n            **kwargs: Keyword arguments to pass to the MTCNN constructor. See help(MTCNN).\n        \"\"\"\n        self.stride = stride\n        self.mtcnn = MTCNN(*args, **kwargs)\n        \n    def __call__(self, frames):\n        \"\"\"Detect faces in frames using strided MTCNN.\"\"\"\n                      \n        boxes, probs = self.mtcnn.detect(frames[::self.stride])\n\n        faces = []\n        probs_out = []\n        frame_index = []\n        for i, frame in enumerate(frames):\n            box_ind = int(i / self.stride)\n            if boxes[box_ind] is None:\n                continue\n            for box, prob in zip(boxes[box_ind], probs[box_ind]):\n                box = [int(b) for b in box]\n                faces.append(frame[box[1]:box[3], box[0]:box[2]].copy())\n                probs_out.append(prob)\n                frame_index.append(i)\n                \n        \n        return faces, probs, frame_index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define face detector\n\nThe following face detector can detect all faces in a video in approximately 2.8 seconds, allowing all videos in the public test set to be processed in 2.8 * 4000 = 11200 seconds = 3.1 hours."},{"metadata":{"trusted":true},"cell_type":"code","source":"fast_mtcnn = FastMTCNN(\n    stride=10,\n    margin=20,\n    factor=0.6,\n    keep_all=True,\n    device=device,\n    thresholds=[0.6, 0.7, 0.98]\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Process all videos"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndef mean_detection_prob(prob):\n    cnt_p = 0\n    sum_p = 0\n    for p in prob:\n        for pp in p:\n            if pp is not None:\n                cnt_p += 1\n                sum_p += pp\n    return sum_p / cnt_p\n\n\ndef get_frames(filename, batch_size=30):\n    v_cap = VideoReader(filename, ctx=gpu())\n    v_len = len(v_cap)\n\n    frames = []\n    for i in range(0, v_len, batch_size):\n        batch = v_cap.get_batch(range(i, min(i + batch_size, v_len - 1))).asnumpy()\n        frames.extend(batch.copy())\n    \n    frames = np.array(frames)\n    \n    del v_cap, v_len, batch\n    \n    return frames\n\n\nfilenames = glob.glob('/kaggle/input/deepfake-detection-challenge/test_videos/*.mp4')\n\nnum_faces = 0\nprobs = []\nindexes = []\npbar = tqdm(filenames)\nfor filename in pbar:\n    frames = get_frames(filename)\n\n    faces, prob, index = fast_mtcnn(frames)        \n    probs.append(mean_detection_prob(prob))\n\n    num_faces += len(faces)\n    pbar.set_description(f'Faces found: {num_faces}')\n\n    del frames","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probs = np.asarray(probs)\nprobs = np.clip((1 - probs) ** (1 / 6) * 1.7, 0.0, 1.0)\nplt.hist(probs, 40);\n\nfilenames = [os.path.basename(f) for f in filenames]\n\nsubmission = pd.DataFrame({'filename': filenames, 'label': probs})\nsubmission.to_csv('submission.csv', index=False)\nsubmission","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}