{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Audio Model Notes"},{"metadata":{},"cell_type":"markdown","source":"**Notebook**\n\nWe created a fake-audio detection model to detect videos with 'fake audio' and 'real faces'. This model should stop other notebooks from predicting 'real' when they should be predicting 'fake'. This notebook implements this 'fake audio' model with a resnext model and if the fake audio model predicts > .7, the final prediction is .95 fake. If it is lower than .7, it feeds the video to the resnext model and uses that prediction.\n\nWe are releasing this model because of covid-19 (time constraints and lack of access to previous resources). We can get this notebook to commit, but we cannot get it to 'submit'. If any of you can get it to commit, please share to the public.\n\n**Dataset**\n\nWe created a fake-audio detection model from fake and real audio that we isolated from the dataset. Each 'fake' video has a corresponding 'real' video - we labeled any audio from a 'fake' video that deviated from the corresponding audio from the 'real' video as 'fake audio'. Audio from 'real' video was labeled 'real audio'. The fake audio files can be found here: https://www.kaggle.com/sherkt1/fakeaudio\n\n**Model**\n\nWe used librosa to extract features from the audio and trained a neural network with three hidden layers. Test accuracy of just fake-audio and real-audio was ~ 93%. Fake-Audio detection accuracy with a threshold above .7 for fake-audio, real, and fake-video was ~ 92%.\n\nNotebook to train model can be found here: https://www.kaggle.com/sherkt1/train-audio-model?scriptVersionId=30962766\n\nAudoModel can be found here: https://www.kaggle.com/sherkt1/audiomodel2\n\n\n\n\n\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport librosa\nimport os, sys, time\nimport cv2\nimport numpy as np\nimport pandas as pd\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport sys\nsys.path.insert(0, \"/kaggle/input/helpers/\")\n\nfrom helpers.read_video_1 import VideoReader\nfrom helpers.face_extract_1 import FaceExtractor\nfrom torchvision.transforms import Normalize\nimport torch.nn as nn\nimport torchvision.models as models\nfrom concurrent.futures import ThreadPoolExecutor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gpu = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ngpu","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/deepfake-xception-trained-model\n!ls ../input/deepfake-kernel-data\n!pip install ../input/deepfake-xception-trained-model/pytorchcv-0.0.55-py2.py3-none-any.whl --quiet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install /kaggle/input/video-tools/video-tools/wheelhouse/*.whl\n!dpkg -i --force-confdef /kaggle/input/video-tools/video-tools/dpkgs/*.deb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import moviepy.editor as mp\nimport keras\nimport pickle\nimport os\nimport csv\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\n\nscalerfile = 'scaler.sav'\nscaler = pickle.load(open('/kaggle/input/audiomodel2/' + scalerfile, 'rb'))\naudiomodel = keras.models.load_model('/kaggle/input/audiomodel2/audiomodel2.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Def"},{"metadata":{"trusted":true},"cell_type":"code","source":"def isotropically_resize_image(img, size, resample=cv2.INTER_AREA):\n    h, w = img.shape[:2]\n    if w > h:\n        h = h * size // w\n        w = size\n    else:\n        w = w * size // h\n        h = size\n\n    resized = cv2.resize(img, (w, h), interpolation=resample)\n    return resized\n\n\ndef make_square_image(img):\n    h, w = img.shape[:2]\n    size = max(h, w)\n    t = 0\n    b = size - h\n    l = 0\n    r = size - w\n    return cv2.copyMakeBorder(img, t, b, l, r, cv2.BORDER_CONSTANT, value=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initializing Face Extractor"},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\nsys.path.insert(0, \"/kaggle/input/blazeface-pytorch\")\nsys.path.insert(0, \"/kaggle/input/deepfakes-inference-demo\")\n\nfrom blazeface import BlazeFace\nfacedet = BlazeFace().to(gpu)\nfacedet.load_weights(\"/kaggle/input/blazeface-pytorch/blazeface.pth\")\nfacedet.load_anchors(\"/kaggle/input/blazeface-pytorch/anchors.npy\")\n_ = facedet.train(False)\n\nframes_per_video = 10\nvideo_reader = VideoReader()\nvideo_read_fn = lambda x: video_reader.read_frames(x, num_frames=frames_per_video)\nface_extractor = FaceExtractor(video_read_fn, facedet)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction Def"},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = [0.485, 0.456, 0.406]\nstd = [0.229, 0.224, 0.225]\nnormalize_transform = Normalize(mean, std)\n    \ndef videoprediction(video_path, batch_size):\n    try:\n        faces = face_extractor.process_video(video_path)\n        face_extractor.keep_only_best_face(faces)\n        \n        if len(faces) > 0:\n            x = np.zeros((batch_size, 224, 224, 3), dtype=np.uint8)\n            n = 0\n            for frame_data in faces:\n                for face in frame_data[\"faces\"]:     \n                    resized_face = isotropically_resize_image(face, 224)\n                    resized_face = make_square_image(resized_face)\n\n                    if n < batch_size:\n                        x[n] = resized_face\n                        n += 1\n\n            if n > 0:\n                x = torch.tensor(x, device=gpu).float()\n                x = x.permute((0, 3, 1, 2))\n\n                for i in range(len(x)):\n                    x[i] = normalize_transform(x[i] / 255.)\n                    \n                with torch.no_grad():\n                    #Need to return both resnext and xception predictions\n                    resnext_pred = resnext_model(x)\n                    resnext_pred = torch.sigmoid(resnext_pred.squeeze())\n                    torch.cuda.empty_cache()\n                    \n                    return resnext_pred[:n].mean().item()\n\n    except Exception as e:\n        print(\"Prediction error on video %s: %s\" % (video_path, str(e)))\n\n    return 0.5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initialize Models"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MyResNeXt(models.resnet.ResNet):\n    def __init__(self, training=True):\n        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,\n                                        layers=[3, 4, 6, 3], \n                                        groups=32, \n                                        width_per_group=4)\n        self.fc = nn.Linear(2048, 1)\n        \nclass Pooling(nn.Module):\n    def __init__(self):\n        super(Pooling, self).__init__()\n\n        self.p1 = nn.AdaptiveAvgPool2d((1,1))\n        self.p2 = nn.AdaptiveMaxPool2d((1,1))\n\n    def forward(self, x):\n        x1 = self.p1(x)\n        x2 = self.p2(x)\n        return (x1+x2) * 0.5\n\n\nclass Head(torch.nn.Module):\n    def __init__(self, in_f, out_f):\n        super(Head, self).__init__()\n\n        self.f = nn.Flatten()\n        self.l = nn.Linear(in_f, 512)\n        self.d = nn.Dropout(0.5)\n        self.o = nn.Linear(512, out_f)\n        self.b1 = nn.BatchNorm1d(in_f)\n        self.b2 = nn.BatchNorm1d(512)\n        self.r = nn.ReLU()\n\n    def forward(self, x):\n        x = self.f(x)\n        x = self.b1(x)\n        x = self.d(x)\n\n        x = self.l(x)\n        x = self.r(x)\n        x = self.b2(x)\n        x = self.d(x)\n\n        out = self.o(x)\n        return out\n\nclass FCN(torch.nn.Module):\n    def __init__(self, base, in_f):\n        super(FCN, self).__init__()\n        self.base = base\n        self.h1 = Head(in_f, 1)\n  \n    def forward(self, x):\n        x = self.base(x)\n        return self.h1(x)\n    \ncheckpoint = torch.load(\"/kaggle/input/deepfakes-inference-demo/resnext.pth\", map_location=gpu)\nmodel = MyResNeXt().to(gpu)\nmodel.load_state_dict(checkpoint)\n_ = model.eval()\ndel checkpoint\n\nresnext_model = model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Print to log"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport gc\n__print__ = print\ndef print(string):\n    os.system(f'echo \\\"{string}\\\"')\n    __print__(string)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction Loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir temp\n\npath = '/kaggle/input/deepfake-detection-challenge/test_videos/'\nmovies = os.listdir(path)\n\nfinalpred = []\nfinalmovies = []\ncount = 0\nfor movie in movies:\n    try:\n        try:\n            gc.collect()\n            torch.cuda.empty_cache()\n            if movie[-4:] != \".mp4\":\n                continue\n\n            print('count: ' + str(count))\n            count += 1\n            #Load audio, save as temporary .mp3, load with librosa\n            audioclip = mp.VideoFileClip(path + movie)\n            audioclip.audio.write_audiofile('/kaggle/working/temp/temp.wav', verbose=False, logger=None)\n            y, sr = librosa.load('/kaggle/working/temp/temp.wav', mono=True, duration=30)\n\n            #Extract Features\n            chroma_stft = librosa.feature.chroma_stft(y=y, sr=sr)\n            rmse = librosa.feature.rms(y=y)\n            spec_cent = librosa.feature.spectral_centroid(y=y, sr=sr)\n            spec_bw = librosa.feature.spectral_bandwidth(y=y, sr=sr)\n            rolloff = librosa.feature.spectral_rolloff(y=y, sr=sr)\n            zcr = librosa.feature.zero_crossing_rate(y)\n            mfcc = librosa.feature.mfcc(y=y, sr=sr)\n\n            #Concatenate Features, Scale\n            audiodata = np.concatenate((np.expand_dims(np.mean(chroma_stft), axis=0),np.expand_dims(np.mean(rmse), axis=0),np.expand_dims(np.mean(spec_cent), axis=0),np.expand_dims(np.mean(spec_bw), axis=0),np.expand_dims(np.mean(rolloff), axis=0),np.expand_dims(np.mean(zcr), axis=0)),axis=0)\n            for e in mfcc:\n                 audiodata = np.concatenate((audiodata,np.expand_dims(np.mean(e), axis=0)),axis=0)\n            audiodata = scaler.transform(np.expand_dims(audiodata,axis=0))\n\n            #Predict, choose tocsv or to cnn\n            prediction = audiomodel.predict(audiodata)\n\n            if prediction[0][0] > .7:\n                finalpred.append(.95)\n                finalmovies.append(movie)\n            else:\n                prediction1 = videoprediction(path+movie,30)\n                if prediction1 >= 1:\n                    prediction1 = .99\n                if prediction1 <= 0:\n                    prediction1 = .01\n                finalpred.append(prediction1)\n                finalmovies.append(movie)\n        except:\n            prediction1 = videoprediction(path+movie,30)\n            if prediction1 >= 1:\n                prediction1 = .99\n            if prediction1 <= 0:\n                prediction1 = .01\n            finalpred.append(prediction1)\n            finalmovies.append(movie)\n    except:\n        finalmovies.append(movie)\n        finalpred.append(.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({\"filename\": finalmovies, \"label\": finalpred})\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}