{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Face Sequence Ensemble - Inference\n\nThis notebook combines a series of pretrained video classifiers by averaging the output logits of each model.\n\nThe input to each model are the faces from every frame in a video. The dataset class used is able to identify distinct face sequences (i.e., from different people in a video), and these are classified separately. The output from each submodel is then the logit from the face sequence with the highest probability of being fake.\n\n## Contents\n\n1. <a href=\"#1\">Install dependencies</a>\n1. <a href=\"#2\">Imports</a>\n1. <a href=\"#3\">Define ensemble module</a>\n1. <a href=\"#4\">Load pretrained face models</a>\n1. <a href=\"#5\">Create dataset object</a>\n1. <a href=\"#6\">Load pretrained video models</a>\n1. <a href=\"#7\">Sense checking</a>\n1. <a href=\"#8\">Inference on test set</a>\n1. <a href=\"#9\">Save predictions</a>"},{"metadata":{},"cell_type":"markdown","source":"## <a id='1'>Install dependencies</a>"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%capture\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.9-py3-none-any.whl\n!pip install /kaggle/input/xt-training/pynvml-8.0.4-py3-none-any.whl\n!pip install /kaggle/input/xt-training/xt_training-1.4.0-py3-none-any.whl\n!pip install /kaggle/input/imageio-ffmpeg/imageio_ffmpeg-0.3.0-py3-none-manylinux2010_x86_64.whl\n!pip install /kaggle/input/imutils/imutils-0.5.3/\n!cp -R /kaggle/input/xtract-ai-dfdc/dfdc ./","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='2'>Imports</a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport glob\n\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch.utils.data import DataLoader, Subset\nfrom torch import optim\nfrom tqdm.notebook import tqdm\nfrom matplotlib import pyplot as plt\nimport albumentations as A\nfrom tqdm.notebook import tqdm\nfrom xt_training import metrics, Runner\nfrom xt_training.runner import Logger\n\nfrom dfdc.datasets.video_dataset import VideoDataset\nfrom dfdc.models.video_models import FaceSequenceClassifier, FaceClassifier","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\nprint('Running on device: {}'.format(device))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='3'>Ensemble module</a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Ensemble(torch.nn.Module):\n    \n    def __init__(self, unpack=None, permute=False, mapping=None, **kwargs):\n        super().__init__()\n        self.permute = permute\n        self.mapping = mapping\n        if unpack is None:\n            self.unpack = lambda x, i: x\n        else:\n            self.unpack = unpack\n        for name, model in kwargs.items():\n            self.add_module(name, model)\n    \n    def forward(self, x):\n        out = []\n\n        for i, layer in enumerate(self._modules.values()):\n            if self.mapping is None:\n                in_i = i\n            else:\n                in_i = self.mapping[i]\n            x_i = self.unpack(x, in_i)\n            out.append(layer(x_i))\n        \n        out = torch.stack(out)\n        \n        if self.permute:\n            out = out.permute((1, 0, 2))\n\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='4'>Load pretrained face models</a>\n\nThe primary component of each video classifier is a resnet-based model that returns logits for each face from each frame."},{"metadata":{"trusted":true},"cell_type":"code","source":"face_model1 = FaceClassifier(pretrained=False, base_model='resnext')\nface_model1.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/face_model_best_alltrain_moreaug_20200317.pt'))\nface_model1.classifier.fc = torch.nn.Sequential()\n\nface_model2 = FaceClassifier(pretrained=False, base_model='resnet')\nface_model2.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/face_model_best_lessaug_resnet101_20200321.pt'))\nface_model2.classifier.fc = torch.nn.Sequential()\n\nface_model3 = FaceClassifier(pretrained=False, base_model='resnext')\nface_model3.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/face_model_best_20200314.pt'))\nface_model3.classifier.fc = torch.nn.Sequential()\n\nface_model4 = FaceClassifier(pretrained=False, base_model='resnext')\nface_model4.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/face_model_best_20200223.pt'))\nface_model4.classifier.fc = torch.nn.Sequential()\n\n\nface_model = Ensemble(\n    unpack=None,\n    m1=face_model1,\n    m2=face_model2,\n    m3=face_model3,\n    m4=face_model4,\n)\nface_model.to(device)\nface_model.eval()\n\n@torch.no_grad()\ndef face_model_transform(x):\n    return face_model(x.to(device))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='5'>Create dataset object</a>\n\nThis dataset object is responsible for:\n\n1. Loading images\n1. Applying transformations\n1. Detecting all faces\n1. Constructing sequences of individual faces"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_trans = A.ReplayCompose([\n    A.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),\n    A.Resize(160, 160, always_apply=True)\n])\n\nvideo_root = '/kaggle/input/deepfake-detection-challenge/'\n\ntest_dataset = VideoDataset(\n    video_root,\n    transform=test_trans,\n    out_transform=face_model_transform,\n    is_test=True,\n    sample_frames=-1,\n    shuffle=False,\n    stride=10,\n    n_frames=-1,\n    device=device,\n    reader='imutils',\n    path_include='test_videos/',\n)\n\nbatch_size = 1\nnum_workers = 0\ntest_loader = DataLoader(\n    test_dataset,\n    batch_size=batch_size,\n    num_workers=num_workers\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='6'>Load pretrained video models</a>\n\nEach of these models takes as input the feature vectors returned from the above face models for each face in each frame of a video."},{"metadata":{"trusted":true},"cell_type":"code","source":"video_model1 = FaceSequenceClassifier(mode='linear')\nvideo_model1.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/video_model_best_alltrain_lessaug_20200319.pt'))\n\nvideo_model2 = FaceSequenceClassifier(mode='linear')\nvideo_model2.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/video_model_best_lessaug_resnet101_20200321.pt'))\n\nvideo_model3 = FaceSequenceClassifier(mode='linear')\nvideo_model3.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/video_model_best_ep8_20200314.pt'))\n\nvideo_model4 = FaceSequenceClassifier(mode='linear')\nvideo_model4.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/video_model_best_20200223.pt'))\n\nvideo_model5 = FaceSequenceClassifier(mode='conv')\nvideo_model5.load_state_dict(torch.load('/kaggle/input/face-sequence-classifier/video_model_best_alltrain_lessaug_conv_20200319.pt'))\n\nscales = [1, 1, 1, 1, 0.5]\n\ndef unpack(x, i):\n    return [x[0][:, i], x[1]]\n\nvideo_model = Ensemble(\n    unpack=unpack,\n    permute=True,\n    m1=video_model1,\n    m2=video_model2,\n    m3=video_model3,\n    m4=video_model4,\n    m5=video_model5,\n    mapping=[0, 1, 2, 3, 0]\n)\nvideo_model.to(device)\nvideo_model.eval()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='7'>Sense checking</a>\n\nProduce a prediction to ensure the value matches what I've seen locally."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_dataset = VideoDataset(\n    video_root,\n    transform=test_trans,\n    out_transform=face_model_transform,\n    is_test=True,\n    sample_frames=-1,\n    shuffle=False,\n    stride=10,\n    n_frames=-1,\n    device=device,\n    reader='imutils',\n    path_include='train_sample_videos/',\n)\n\nsample_dataset.samples[0] = ('/kaggle/input/deepfake-detection-challenge/train_sample_videos/aagfhgtpmv.mp4', 1)\n\nsample_loader = DataLoader(\n    sample_dataset,\n    batch_size=batch_size,\n    num_workers=num_workers\n)\n\nfor x, y in sample_loader:\n    break\nprint(x[0].abs().mean())\nx = [x_i.to(device) for x_i in x]\ny.to(device)\n\nwith torch.no_grad():\n    y_pred = video_model(x)\n\nfor i, scale in enumerate(scales):\n    y_pred[:, i] = y_pred[:, i] * scale\n\ny_pred","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='8'>Inference on test set</a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"runner = Runner(model=video_model, device=device)\n\ny_pred, _ = runner(test_loader, 'test', return_preds=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a id='9'>Save predictions</a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, scale in enumerate(scales):\n    y_pred[:, i] = y_pred[:, i] * scale\n\nlabels = torch.nn.functional.softmax(y_pred.mean(dim=1), dim=1)[:, 1].numpy()\nfilenames = [os.path.basename(f) for f in test_dataset.video_files]\n\nsubmission = pd.DataFrame({'filename': filenames, 'label': labels})\n# submission.label = submission.label.clip(0.005, 0.995)\nsubmission.to_csv('submission.csv', index=False)\nplt.hist(submission.label, 30)\nsubmission","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}