{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# resnext and audio\n+ This kernel is modified from Inference Kernel Demo.\n+ I‘ve added a voice processing framework to the previous video processing framework.\n+ When the voice-video model exceeds the single video model, it means that the voice model is useful.","metadata":{}},{"cell_type":"markdown","source":"# Inference Kernel Demo\n\nThis is the kernel I’ve used for my recent submissions. It takes about 5-6 hours on the test set, using only CPU. \n\nI’ve provided this kernel because a lot of people have problems making submissions. This method works and has never errored out for me. (Although I haven't tried making a submission using the GPU yet -- so no guarantees there.)\n\nIt uses BlazeFace for face extraction (see also [my BlazeFace kernel](https://www.kaggle.com/humananalog/starter-blazeface-pytorch)) and ResNeXt50 as the classifier model.\n\nWe take the average prediction over 17 frames from each video. (Why 17? Using more frames makes the kernel slower, but doesn't appear to improve the score much. I used an odd number so we don't always land on even frames.)\n\n**Please use this kernel only to learn from...** Included is the checkpoint for a ResNeXt50 model that hasn't really been trained very well yet. I'm sure you can improve on it by training your own model!\n\nYou could use the included trained weights to get yourself an easy top-50 score on the leaderboard (as of 24 Jan 2020) but it’s nicer to use it as a starting point for your own work. :-)","metadata":{}},{"cell_type":"code","source":"! tar xvf ../input/ffmpeg/ffmpeg-git-amd64-static.tar.xz","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport librosa\nimport subprocess\nimport shutil\nfrom pathlib import Path\noutput_format = 'wav'  # can also use aac, wav, etc\n\noutput_dir = Path(f\"{output_format}s\")\nPath(output_dir).mkdir(exist_ok=True, parents=True)\nWAV_PATH = './wavs/'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_wav(file):\n    command = f\"../working/ffmpeg-git-20191209-amd64-static/ffmpeg -i {file} -ab 192000 -ac 2 -ar 44100 -vn {output_dir/file[-14:-4]}.{output_format}\"\n    subprocess.call(command, shell=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test_dir = \"/kaggle/input/deepfake-detection-challenge/test_videos/\"\n# tmp_dir = \"./kaggle/working/\"\n# test_videos = sorted([x for x in os.listdir(test_dir) if x[-4:] == \".mp4\"])\n# for test_video in test_videos:\n#     video_path = os.path.join(test_dir, test_video)\n#     create_wav(video_path)\n#     name = test_video[:-4] + \".wav\"\n#     wave, sr = librosa.load(WAV_PATH+f'{name}', mono=True)\n#     os.unlink(WAV_PATH+f'{name}')\n#     print(sr)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os, sys, time\nimport cv2\nimport numpy as np\nimport pandas as pd\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n%matplotlib inline\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get the test videos","metadata":{}},{"cell_type":"code","source":"test_dir = \"/kaggle/input/deepfake-detection-challenge/test_videos/\"\n\ntest_videos = sorted([x for x in os.listdir(test_dir) if x[-4:] == \".mp4\"])\nlen(test_videos)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create helpers","metadata":{}},{"cell_type":"code","source":"print(\"PyTorch version:\", torch.__version__)\nprint(\"CUDA version:\", torch.version.cuda)\nprint(\"cuDNN version:\", torch.backends.cudnn.version())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gpu = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ngpu","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.insert(0, \"/kaggle/input/blazeface-pytorch\")\nsys.path.insert(0, \"/kaggle/input/deepfakes-inference-demo\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from blazeface import BlazeFace\nfacedet = BlazeFace().to(gpu)\nfacedet.load_weights(\"/kaggle/input/blazeface-pytorch/blazeface.pth\")\nfacedet.load_anchors(\"/kaggle/input/blazeface-pytorch/anchors.npy\")\n_ = facedet.train(False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from helpers.read_video_1 import VideoReader\nfrom helpers.face_extract_1 import FaceExtractor\n\nframes_per_video = 21\n\nvideo_reader = VideoReader()\nvideo_read_fn = lambda x: video_reader.read_frames(x, num_frames=frames_per_video)\nface_extractor = FaceExtractor(video_read_fn, facedet)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"input_size = 224","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torchvision.transforms import Normalize\n\nmean = [0.485, 0.456, 0.406]\nstd = [0.229, 0.224, 0.225]\nnormalize_transform = Normalize(mean, std)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def isotropically_resize_image(img, size, resample=cv2.INTER_AREA):\n    h, w = img.shape[:2]\n    if w > h:\n        h = h * size // w\n        w = size\n    else:\n        w = w * size // h\n        h = size\n\n    resized = cv2.resize(img, (w, h), interpolation=resample)\n    return resized\n\n\ndef make_square_image(img):\n    h, w = img.shape[:2]\n    size = max(h, w)\n    t = 0\n    b = size - h\n    l = 0\n    r = size - w\n    return cv2.copyMakeBorder(img, t, b, l, r, cv2.BORDER_CONSTANT, value=0)\n\ndef load_audio_mfcc(audio_path):\n    wave, sr = librosa.load(audio_path, mono=True)\n    mfcc = librosa.feature.mfcc(wave, sr)\n    if mfcc.shape[1] > 400:\n        mfcc = mfcc[:, :400]\n    else:\n        mfcc = np.pad(mfcc, ((0, 0), (0, 400 - len(mfcc[0]))), mode='constant', constant_values=0)\n    return mfcc","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch.nn as nn\nimport torchvision.models as models\n\nclass MyResNeXt(models.resnet.ResNet):\n    def __init__(self, training=True):\n        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,\n                                        layers=[3, 4, 6, 3], \n                                        groups=32, \n                                        width_per_group=4)\n        self.fc = nn.Linear(2048, 1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# audio_model","metadata":{}},{"cell_type":"code","source":"class MASRCNN_activate(nn.Module):\n    def __init__(self, max_sent_len, embedding_dim, num_conv_blocks, init_neurons, num_classes=2):\n        super(MASRCNN_activate, self).__init__()\n        num_dense_neurons = 50\n        convnet_3 = []\n        convnet_5 = []\n        convnet_7 = []\n        for ly in range(0, num_conv_blocks):\n            if ly == 0:\n                convnet_3.append(nn.Conv1d(in_channels=embedding_dim, out_channels=init_neurons, kernel_size=3))\n                convnet_3.append(nn.LeakyReLU(0.2))\n                convnet_5.append(nn.Conv1d(in_channels=embedding_dim, out_channels=init_neurons, kernel_size=5))\n                convnet_5.append(nn.LeakyReLU(0.2))\n                convnet_7.append(nn.Conv1d(in_channels=embedding_dim, out_channels=init_neurons, kernel_size=7))\n                convnet_7.append(nn.LeakyReLU(0.2))\n            elif ly == 1:\n                convnet_3.append(nn.Conv1d(in_channels=init_neurons, out_channels=init_neurons*(ly*2), kernel_size=3))\n                convnet_3.append(nn.LeakyReLU(0.2))\n                convnet_5.append(nn.Conv1d(in_channels=init_neurons, out_channels=init_neurons*(ly*2), kernel_size=5))\n                convnet_5.append(nn.LeakyReLU(0.2))\n                convnet_7.append(nn.Conv1d(in_channels=init_neurons, out_channels=init_neurons*(ly*2), kernel_size=7))\n                convnet_7.append(nn.LeakyReLU(0.2))\n            else:\n                convnet_3.append(nn.Conv1d(in_channels=init_neurons*((ly - 1)*2), out_channels=init_neurons*(ly*2), kernel_size=3))\n                convnet_3.append(nn.LeakyReLU(0.2))\n                convnet_5.append(nn.Conv1d(in_channels=init_neurons*((ly - 1)*2), out_channels=init_neurons*(ly*2), kernel_size=5))\n                convnet_5.append(nn.LeakyReLU(0.2))\n                convnet_7.append(nn.Conv1d(in_channels=init_neurons*((ly - 1)*2), out_channels=init_neurons*(ly*2), kernel_size=7))\n                convnet_7.append(nn.LeakyReLU(0.2))\n        self.conv_blocks_3 = nn.Sequential(*convnet_3)\n        self.conv_blocks_5 = nn.Sequential(*convnet_5)\n        self.conv_blocks_7 = nn.Sequential(*convnet_7)\n        self.leakyrelu = nn.LeakyReLU(0.2)\n        self.maxpool = nn.AdaptiveMaxPool1d(1)\n        self.dense = nn.Sequential(nn.Linear(448*3, num_dense_neurons),\n                                    nn.BatchNorm1d(num_dense_neurons),\n                                    nn.LeakyReLU(0.2)\n                                    )\n        self.fc = nn.Linear(50, num_classes)\n    \n    def forward(self, x):\n        x_3 = self.conv_blocks_3(x)\n        x_5 = self.conv_blocks_5(x)\n        x_7 = self.conv_blocks_7(x)\n        x_3 = self.maxpool(x_3)\n        x_5 = self.maxpool(x_5)\n        x_7 = self.maxpool(x_7)\n        x = torch.cat([x_3, x_5, x_7], 2)\n        x = x.view(x.size(0), -1)\n        x = self.dense(x)\n        x = F.dropout(x, p=0.5, training=self.training)\n        x = self.fc(x)\n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# load weights","metadata":{}},{"cell_type":"code","source":"def load_weights(model, checkpoint_path, multi_gpu=False):\n    if torch.cuda.is_available():\n        checkpoint = torch.load(checkpoint_path)\n    else:\n        checkpoint = torch.load(checkpoint_path, map_location=torch.device('cpu'))\n    if multi_gpu:\n        model.module.load_state_dict(checkpoint['model'], strict=False)\n    else:\n        model.load_state_dict(checkpoint['model'], strict=False)\n    del checkpoint\n    torch.cuda.empty_cache()\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_path = \"/kaggle/input/audio-model/ASRCNN_27000.pth\"\naudio_model = MASRCNN_activate(max_sent_len=400, embedding_dim=20, num_conv_blocks=8, init_neurons=32)\naudio_model = load_weights(audio_model, checkpoint_path)\naudio_model = audio_model.to(gpu)\naudio_model.eval()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_on_audio(audio_path):\n    try:\n        mfcc = load_audio_mfcc(audio_path)\n        mfcc = torch.tensor(mfcc, device=gpu).float()\n        mfcc = torch.unsqueeze(mfcc, dim=0)\n        output = audio_model(mfcc)\n        output = torch.softmax(output, dim=1)\n        pred = output.detach().cpu().numpy()[0][1]\n        return pred\n    except Exception as e:\n        print(\"Prediction error on audio %s: %s\" % (audio_path, str(e)))\n    return 0.5","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint = torch.load(\"/kaggle/input/deepfakes-inference-demo/resnext.pth\", map_location=gpu)\n\nmodel = MyResNeXt().to(gpu)\nmodel.load_state_dict(checkpoint)\n_ = model.eval()\n\ndel checkpoint","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction loop","metadata":{}},{"cell_type":"code","source":"def predict_on_video(video_path, batch_size):\n    try:\n        # Find the faces for N frames in the video.\n        faces = face_extractor.process_video(video_path)\n\n        # Only look at one face per frame.\n        face_extractor.keep_only_best_face(faces)\n        \n        if len(faces) > 0:\n            # NOTE: When running on the CPU, the batch size must be fixed\n            # or else memory usage will blow up. (Bug in PyTorch?)\n            x = np.zeros((batch_size, input_size, input_size, 3), dtype=np.uint8)\n\n            # If we found any faces, prepare them for the model.\n            n = 0\n            for frame_data in faces:\n                for face in frame_data[\"faces\"]:\n                    # Resize to the model's required input size.\n                    # We keep the aspect ratio intact and add zero\n                    # padding if necessary.                    \n                    resized_face = isotropically_resize_image(face, input_size)\n                    resized_face = make_square_image(resized_face)\n\n                    if n < batch_size:\n                        x[n] = resized_face\n                        n += 1\n                    else:\n                        print(\"WARNING: have %d faces but batch size is %d\" % (n, batch_size))\n                    \n                    # Test time augmentation: horizontal flips.\n                    # TODO: not sure yet if this helps or not\n                    #x[n] = cv2.flip(resized_face, 1)\n                    #n += 1\n\n            if n > 0:\n                x = torch.tensor(x, device=gpu).float()\n\n                # Preprocess the images.\n                x = x.permute((0, 3, 1, 2))\n\n                for i in range(len(x)):\n                    x[i] = normalize_transform(x[i] / 255.)\n\n                # Make a prediction, then take the average.\n                with torch.no_grad():\n                    y_pred = model(x)\n                    y_pred = torch.sigmoid(y_pred.squeeze())\n                    return y_pred[:n].mean().item()\n\n    except Exception as e:\n        print(\"Prediction error on video %s: %s\" % (video_path, str(e)))\n\n    return 0.5","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 多线程(快十倍？)","metadata":{}},{"cell_type":"code","source":"from concurrent.futures import ThreadPoolExecutor\n\ndef predict_on_video_set(videos, num_workers):\n    def process_file(i):\n        filename = videos[i]\n        video_path = os.path.join(test_dir, filename)\n        # predict video\n        # y_pred_frame = predict_on_video(video_path, batch_size=frames_per_video)\n        # predict audio\n        create_wav(video_path)\n        audio_path = WAV_PATH + filename[:-4] + \".wav\"\n        y_pred_audio = predict_on_audio(audio_path)\n        os.unlink(audio_path)\n        y_pred = y_pred_audio\n        # y_pred = max(y_pred_frame, y_pred_audio)\n        return y_pred\n\n    with ThreadPoolExecutor(max_workers=num_workers) as ex:\n        predictions = ex.map(process_file, range(len(videos)))\n\n    return list(predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 单线程","metadata":{}},{"cell_type":"code","source":"def predict_on_video_set_single(videos):\n    predictions = []\n    for filename in videos:\n        video_path = os.path.join(test_dir, filename)\n        # predict video\n        y_pred_frame = predict_on_video(video_path, batch_size=frames_per_video)\n        # predict audio\n        try:\n            create_wav(video_path)\n        except Exception as e:\n            print(\"create wav error\", e)\n            predictions.append(y_pred_frame)\n            continue\n        audio_path = WAV_PATH + filename[:-4] + \".wav\"\n        if not os.path.exists(audio_path):\n            predictions.append(y_pred_frame)\n            continue\n        y_pred_audio = predict_on_audio(audio_path)\n        os.unlink(audio_path)\n        if y_pred_frame > 0.5:\n            y_pred = y_pred_frame\n        elif y_pred_audio >= 0.5:\n            y_pred = y_pred_audio\n        else:\n            y_pred = min(y_pred_frame, y_pred_audio)\n        predictions.append(y_pred)\n    return predictions","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Speed test\n\nThe leaderboard submission must finish within 9 hours. With 4000 test videos, that is `9*60*60/4000 = 8.1` seconds per video. So if the average time per video is greater than ~8 seconds, the kernel will be too slow!","metadata":{}},{"cell_type":"code","source":"speed_test = True  # you have to enable this manually","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if speed_test:\n    start_time = time.time()\n    speedtest_videos = test_videos[:5]\n    # predictions = predict_on_video_set(speedtest_videos, num_workers=4)\n    predictions = predict_on_video_set_single(speedtest_videos)\n    print(predictions)\n    elapsed = time.time() - start_time\n    print(\"Elapsed %f sec. Average per video: %f sec.\" % (elapsed, elapsed / len(speedtest_videos)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make the submission","metadata":{}},{"cell_type":"code","source":"if not speed_test:\n    # predictions = predict_on_video_set(test_videos, num_workers=4)\n    predictions = predict_on_video_set_single(test_videos)\n    submission_df = pd.DataFrame({\"filename\": test_videos, \"label\": predictions})\n    submission_df.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shutil.rmtree(\"../working/ffmpeg-git-20191209-amd64-static\")\nshutil.rmtree(\"../working/wavs\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#submission_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}