{"cells":[{"metadata":{},"cell_type":"markdown","source":"Yuming Yao 2032754\n\nDataset：https://www.kaggle.com/c/deepfake-detection-challenge\n"},{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction"},{"metadata":{},"cell_type":"markdown","source":"The continuous development of science and technology enables deep learning (DL) models to be applied to a variety of applications. For example, Generative Adversarial Networks (GAN) models can produce super-realistic images, languages, and even videos. For example, the so-called deep forgeries made by GANs (Generative Adversarial Networks) manipulating audio or video clips are too close to real content to be distinguished from real images in human perception. \n\nOne of its most influential applications, DeepFake, is very advanced. In principle, it can turn the protagonist of the video into anyone. In principle, DeepFake uses one of the most well-known deep learning algorithms, GANs. Its first appearance at the end of 2017 caused a sensation. At that time, DeepFake made its first appearance in the adult exchange community of the American social news website Reddit, and it caused a lot of shock. This is a user named Deepfakes in a community who grafted the face of \"Wonder Woman\" Gal Gadot to the heroine of an adult movie and uploaded the video to the website. After uploading this fake video to this website, it caused a lot of discomfort. Reddit officially banned the face-changing video produced and uploaded by Deepfakes for inappropriate content and infringed on the privacy of others. Of course, this is only one of many bad influences, and there are many evil or illegal ways to use these fake content in propaganda, political campaigns, cyber crime, blackmail, etc. If it is allowed to continue, the consequences will be disastrous. Therefore, it is very necessary to do some work to identify these fake content generated by algorithms through technical means, which is a crucial part of solving the problems caused by deepfake described above. First identify the false content, and then we can better prevent, warn, and punish. This work is to protect people's personal rights, safeguard the people's legitimate interests, and maintain the stable development of society, which is extremely beneficial.\n\nIn this case, the researchers decided to find a method of deepfake detection to protect people from this huge danger. It was at that time that offensive deepfakes and defensive detection methods began to compete. This is why the first data sets containing generated fake videos appeared. What I want to do this time is to detect the fake faces generated by deepfake. The input of the predictor is the data set provided by the DFDC challenge, and the Resnext and Xception neural network algorithms are used to detect real and fake faces."},{"metadata":{},"cell_type":"markdown","source":"# 2.\tRelated Work"},{"metadata":{},"cell_type":"markdown","source":"Since the development of deepfake, even the detection of deepfake has long become a hot research topic. In order to find a solution to this problem, relevant researchers have actively explored relevant research strategies and made many attempts. Usually, the solution approach uses visible artifacts, which is common in most deepfakes. The most successful method is based on blinking eyes, mismatched color profiles and facial distortion artifacts. Such artifacts provide good accuracy for deepfakes to a large extent, especially for relatively older fakes. On the other hand, this problem is considered to be more complex and requires other connections beyond vision [1]. Therefore, it is necessary to combine other technologies to study and solve the deepfake problem.\n\nThe most used research on this type of problem is directly based on machine learning and deep learning for recognition and detection, but recently integrated learning has been proposed to further improve the recognition accuracy. Next, let’s look at the work that has made some research progress in this area.\n\nFirst, let's look at some related work directly based on machine learning and deep learning. As early as 2018, D. Afchar et al. proposed an automatic and efficient method for detecting face tampering in videos based on DeepFake and Face2Face, which can generate fake videos that cannot be distinguished by the human eyes. Because traditional image forensics technology is usually not suitable for video, and compression will seriously reduce the quality of the data. Therefore, this paper follows the method of deep learning and proposes two networks, both with fewer layers, to focus on the mesoscopic properties of the image [2]. This is one of the earlier work done in deepfake detection. In [3], they focused on the problem of detecting fake faces by matching falsely replaced face components, specifically by matching the eyebrow region to detect fake phenomena. Further, other researchers can add other parts to this work to enhance the recognition effect. With the continuous improvement of the deepfake algorithm, one can expect that the altered image artifacts will disappear. Therefore, the biometrics of the exchange components may be more preferable than marking forged images. B. Malolan et al. are analyzing the following methods, two convolutional neural network (CNN) architectures, namely Meso-4 and MesoInception-4; Convolutional-LSTM network, the former extracts frame-level features, the latter performs sequence processing and sends the dense layer, and finally determines whether the video is real or fake. Changing faces is never perfect. Obvious distortions and blurring of the surrounding facial areas usually expose them. The use of exchanged face resolution mismatches and affine transformations leads to the existence of fake images, a CNN is trained to capture these features and match. Then a framework for using deep learning methods to detect these deepfake videos is proposed: a convolutional neural network architecture is trained on the face database extracted from the FaceForensics and DeepFakeDetection datasets. In addition, the model has been tested on various interpretable artificial intelligence technologies (such as LRP and LIME) to provide a clear visualization of the salient areas of the image focused by the model [4]. The above work has achieved quite good results, but deepfake is also making progress. Obviously we need further research to combat deepfake.\n\nDeepfakeStack [5] is a deep ensemble learning technology that combines a series of the latest classification models based on deep learning and creates an improved combined classifier. Based on training meta-learners on top of pre-trained basic learners, and provide an interface to adapt meta-learners according to the predictions of basic learners, and show how the integrated technology performs classification tasks. The architecture of DeepfakeStack includes two or more basic learners, called level 0 models, and a meta-learner, called level 1 model, which combines the predictions of these level 0 models. The level 1 model is trained based on the predictions made by the basic model of out-of-sample data. That is to say, the data not used for training the basic model is provided to the basic model to make predictions, and these predictions and the expected output provide input and output pairs of the training data set used to fit the meta-model. Compared with a single machine learning and deep learning model, deep ensemble learning shows better results in deepfake detection. The accuracy of DeepfakeStack can reach 99.65%, and the AUC can reach 1.0.\n\nThe current work is based on machine learning, deep learning and deep integrated learning to identify and detect deepfakes. Our work is also based on deep learning. The difference is that what we do has the following characteristics. We do it on the original video data set. A comparison of several ways of extracting faces, such as MTCNN, BlazeFace, YOLO, etc., used the Resnet model for training and prediction, and at the same time, using Resnetx and Xception pre-trained models to perform predictions. In the end we found that BlazeFace has the fastest detection speed, and the Resnetx model AUC can reach 0.98, which is a good data.\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport matplotlib.pyplot as plt\nfrom matplotlib.animation import FuncAnimation, ArtistAnimation \n%matplotlib inline\nimport glob\nimport cv2\nfrom albumentations import *\nfrom tqdm import tqdm_notebook as tqdm\nimport gc\nimport seaborn as sns\nimport warnings\nimport torch\nimport tensorflow as tf\nimport math\nimport sys\nimport time\nsys.path.insert(0, \"/kaggle/input/blazefacepytorch\")\nfrom blazeface import BlazeFace\nimport os, sys, time\n\nimport torch.nn as nn\nimport torch.nn.functional as F\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport random\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nfrom tqdm.notebook import tqdm\n\n\nfrom tensorflow.keras.layers import Conv2D, Input, ZeroPadding2D, Dense, Lambda\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.applications.mobilenet_v2 import MobileNetV2\n\n\nimport keras\nfrom keras import Model,Sequential\nfrom keras.layers import *\nfrom keras.optimizers import *\nfrom sklearn.metrics import log_loss\nfrom sklearn.model_selection import train_test_split\n\nPATH = '../input/deepfake-detection-challenge/'\nprint(os.listdir(PATH))\nsorted(glob.glob('../input/deepfakes/meta*'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, we need to declare the path of training and test samples and metadata files (this is just a small part of it, more training data we will show later):"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_PATH = '../input/deepfake-detection-challenge/test_videos/'\nTRAIN_PATH = '../input/deepfake-detection-challenge/train_sample_videos/'\nmetadata = '../input/deepfake-detection-challenge/train_sample_videos/metadata.json'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"View the number of samples in the test and training set:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#  加载训练视频的文件名\ntrain_fns = sorted(glob.glob(TRAIN_PATH + '*.mp4'))\n\n# 加载测试视频的文件名\ntest_fns = sorted(glob.glob(TEST_PATH + '*.mp4'))\n\nprint('The training set has {} sample videos'.format(len(train_fns)))\nprint('The test set has {} sample videos '.format(len(test_fns)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"meta = pd.read_json(metadata).transpose()\nlabel_df = meta.reset_index()\nidx = label_df['index']\nimage_meta = []\nfirst_trn_images = []","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Analyze the number of true and false videos in the training set and describe them with pie charts and histograms respectively"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = 'FAKE', 'REAL'\nsizes = [meta[meta.label == 'FAKE'].label.count(), meta[meta.label == 'REAL'].label.count()]\n\nfig1, ax1 = plt.subplots(figsize=(10,7))\nax1.pie(sizes, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90, colors=['#f4d53f', '#02a1d8'])\nax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\nplt.title('Labels', fontsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this histogram, we find that only 19% of the samples are real videos"},{"metadata":{"trusted":true},"cell_type":"code","source":"def gather_info(train_img):\n    train_img = os.path.join(PATH, f\"train_sample_videos/{train_img}\")\n\n    cap = cv2.VideoCapture(train_img)\n\n    success, image = cap.read()\n    count = 0\n    first_trn_image = None\n\n    while success:\n        try:\n            success, image = cap.read()\n            if count == 0:\n                first_trn_image = image\n            x, y, z = image.shape\n\n        except:\n            break\n        count += 1\n        \n    cap.release()\n    cv2.destroyAllWindows()\n    \n    return [x,y,z,count], first_trn_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Collect training image size\nfor i in tqdm(range(len(idx))):\n    test, first = gather_info(idx.values[i])\n    image_meta.append(test)\n    first_trn_images.append(first)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#View the first frame of the first 10 videos\nfor img, lbl in zip(first_trn_images[:10], label_df['label'].values[:10]):\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    plt.title(f'{lbl}')\n    plt.imshow(img)\n    plt.show()\n    \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sample size distribution\nplt.title('Count of each video type')\nsns.countplot(y=label_df['label'].values)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training samples have two resolutions: 1080x1920 and 1920x1080. When training and testing, the number of frames of each video is 297/299 and 298/299, of which 299 is the most frequent. It is not clear whether the private test will be the same release. Obviously, from the perspective of frame resolution, this challenge will require a lot of resources."},{"metadata":{},"cell_type":"markdown","source":"Video data analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_frame(filename):\n    # 读取视频\n    cap = cv2.VideoCapture(filename)\n    ret, frame = cap.read()\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    cap.release()\n    cv2.destroyAllWindows()\n    \n    return image\n\ndef get_label(filename, meta):\n    # 获取标签\n    video_id = filename.split('/')[-1]\n    return meta.loc[video_id].label\n\ndef get_original_filename(filename, meta):\n    \n    video_id = filename.split('/')[-1]\n    original_id = meta.loc[video_id].original\n    \n    return original_id\n\ndef visualize_frame(filename, meta, train = True):\n\n    # 获取第一帧\n    image = get_frame(filename)\n\n    #　展示第一帧视频\n    fig, axs = plt.subplots(1,3, figsize=(20,7))\n    axs[0].imshow(image) \n    axs[0].axis('off')\n    axs[0].set_title('Original frame')\n    \n    # 提取脸部图片\n    face_cascade = cv2.CascadeClassifier('../input/haarcascades/haarcascade_frontalface_default.xml')\n\n\n    faces = face_cascade.detectMultiScale(image, 1.2, 3)\n\n    image_with_detections = image.copy()\n\n    for (x,y,w,h) in faces:\n\n        cv2.rectangle(image_with_detections,(x,y),(x+w,y+h),(255,0,0),3)\n\n    axs[1].imshow(image_with_detections)\n    axs[1].axis('off')\n    axs[1].set_title('Highlight faces')\n    \n    crop_img = image.copy()\n    for (x,y,w,h) in faces:\n        crop_img = image[y:y+h, x:x+w]\n        break;\n        \n    axs[2].imshow(crop_img)\n    axs[2].axis('off')\n    axs[2].set_title('Zoom-in face')\n    \n    if train:\n        plt.suptitle('Image {image} label: {label}'.format(image = filename.split('/')[-1], label=get_label(filename, meta)))\n    else:\n        plt.suptitle('Image {image}'.format(image = filename.split('/')[-1]))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta = pd.read_json(metadata).transpose()\nmeta.head()\nmeta","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.\tProblem Formulation"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nIn this work, the data set we selected is a competition on Kaggle: Deepfake Detection Challenge (DFDC), which is the data set provided by the Deepfake synthetic face detection competition. There are 4 data sets related to this competition.\n\n1.Training set: This data set contains target labels, which can be downloaded outside of Kaggle for competitors to build their models. It is divided into 50 files for easy access and download. Due to its large size, it must be accessed through the GCS bucket.\n\n2.Public verification set: The submitted file output generation will be based on a small set of 400 videos/id included in this public verification set. This is test_videos.zip on the Kaggle data page.\n\n3.Public test set: This data set is completely rejected and the Kaggle platform calculates the public ranking. When you submit the competition from the output file of the submitted notebook containing the competition data set, your code will rerun in the background, and when the retest is completed, the score will be posted on the public leaderboard.\n\n4.Private test set: This data set is private outside the Kaggle platform and is used to calculate private rankings. It contains videos of similar format and nature to the training and public verification/test sets, but real, organic videos, with or without deepfakes.\n\nHere is another introduction to the training set. The complete training set exceeds 470GB. The official has provided 50 small data sets after segmentation. In the training data, whether the video is forged by deepfake is determined by the string \"REAL\" or \"FAKE\" in the label column. Here are the main columns: filename: the file name of the video; label: whether the video is real or fake; original: if the video of the training set is fake, the original video is listed here; split: this is always equal to \"training\". We intercept a certain frame or a few frames in each video as input for training and detection, as shown in the following figure\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train0 = pd.read_json('../input/deepfakes/metadata0.json')\ndf_train1 = pd.read_json('../input/deepfakes/metadata1.json')\ndf_train2 = pd.read_json('../input/deepfakes/metadata2.json')\ndf_train3 = pd.read_json('../input/deepfakes/metadata3.json')\ndf_train4 = pd.read_json('../input/deepfakes/metadata4.json')\ndf_train5 = pd.read_json('../input/deepfakes/metadata5.json')\ndf_train6 = pd.read_json('../input/deepfakes/metadata6.json')\ndf_train7 = pd.read_json('../input/deepfakes/metadata7.json')\ndf_train8 = pd.read_json('../input/deepfakes/metadata8.json')\ndf_train9 = pd.read_json('../input/deepfakes/metadata9.json')\ndf_train10 = pd.read_json('../input/deepfakes/metadata10.json')\ndf_train11 = pd.read_json('../input/deepfakes/metadata11.json')\ndf_train12 = pd.read_json('../input/deepfakes/metadata12.json')\ndf_train13 = pd.read_json('../input/deepfakes/metadata13.json')\ndf_train14 = pd.read_json('../input/deepfakes/metadata14.json')\ndf_train15 = pd.read_json('../input/deepfakes/metadata15.json')\ndf_train16 = pd.read_json('../input/deepfakes/metadata16.json')\ndf_train17 = pd.read_json('../input/deepfakes/metadata17.json')\ndf_train18 = pd.read_json('../input/deepfakes/metadata18.json')\ndf_train19 = pd.read_json('../input/deepfakes/metadata19.json')\ndf_train20 = pd.read_json('../input/deepfakes/metadata20.json')\ndf_train21 = pd.read_json('../input/deepfakes/metadata21.json')\ndf_train22 = pd.read_json('../input/deepfakes/metadata22.json')\ndf_train23 = pd.read_json('../input/deepfakes/metadata23.json')\ndf_train24 = pd.read_json('../input/deepfakes/metadata24.json')\ndf_train25 = pd.read_json('../input/deepfakes/metadata25.json')\ndf_train26 = pd.read_json('../input/deepfakes/metadata26.json')\ndf_train27 = pd.read_json('../input/deepfakes/metadata27.json')\ndf_train28 = pd.read_json('../input/deepfakes/metadata28.json')\ndf_train29 = pd.read_json('../input/deepfakes/metadata29.json')\ndf_train30 = pd.read_json('../input/deepfakes/metadata30.json')\ndf_train31 = pd.read_json('../input/deepfakes/metadata31.json')\ndf_train32 = pd.read_json('../input/deepfakes/metadata32.json')\ndf_train33 = pd.read_json('../input/deepfakes/metadata33.json')\ndf_train34 = pd.read_json('../input/deepfakes/metadata34.json')\ndf_train35 = pd.read_json('../input/deepfakes/metadata35.json')\ndf_train36 = pd.read_json('../input/deepfakes/metadata36.json')\ndf_train37 = pd.read_json('../input/deepfakes/metadata37.json')\ndf_train38 = pd.read_json('../input/deepfakes/metadata38.json')\ndf_train39 = pd.read_json('../input/deepfakes/metadata39.json')\ndf_train40 = pd.read_json('../input/deepfakes/metadata40.json')\ndf_train41 = pd.read_json('../input/deepfakes/metadata41.json')\ndf_train42 = pd.read_json('../input/deepfakes/metadata42.json')\ndf_train43 = pd.read_json('../input/deepfakes/metadata43.json')\ndf_train44 = pd.read_json('../input/deepfakes/metadata44.json')\ndf_train45 = pd.read_json('../input/deepfakes/metadata45.json')\ndf_train46 = pd.read_json('../input/deepfakes/metadata46.json')\ndf_val1 = pd.read_json('../input/deepfakes/metadata47.json')\ndf_val2 = pd.read_json('../input/deepfakes/metadata48.json')\ndf_val3 = pd.read_json('../input/deepfakes/metadata49.json')\ndf_trains = [df_train0 ,df_train1, df_train2, df_train3, df_train4,\n             df_train5, df_train6, df_train7, df_train8, df_train9,df_train10,\n            df_train11, df_train12, df_train13, df_train14, df_train15,df_train16, \n            df_train17, df_train18, df_train19, df_train20, df_train21, df_train22, \n            df_train23, df_train24, df_train25, df_train26, df_train27, df_train28, \n            df_train29, df_train30, df_train31, df_train32, df_train33, df_train34,\n            df_train34, df_train35, df_train36, df_train37, df_train38, df_train39,\n            df_train40, df_train41, df_train42, df_train43, df_train44, df_train45,\n            df_train46]\ndf_vals=[df_val1, df_val2, df_val3]\nnums = list(range(len(df_trains)+1))\nLABELS = ['REAL','FAKE']\nval_nums=[47, 48, 49]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we show the photos in the training set"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_path(num,x):\n    num=str(num)\n    if len(num)==2:\n        path='../input/deepfakes/DeepFake'+num+'/DeepFake'+num+'/' + x.replace('.mp4', '') + '.jpg'\n    else:\n        path='../input/deepfakes/DeepFake0'+num+'/DeepFake0'+num+'/' + x.replace('.mp4', '') + '.jpg'\n    if not os.path.exists(path):\n       raise Exception\n    return path\npaths=[]\ny=[]\nfor df_train,num in tqdm(zip(df_trains,nums),total=len(df_trains)):\n    images = list(df_train.columns.values)\n    for x in images:\n        try:\n            paths.append(get_path(num,x))\n            y.append(LABELS.index(df_train[x]['label']))\n        except Exception as err:\n            #print(err)\n            pass\n\nval_paths=[]\nval_y=[]\nfor df_val,num in tqdm(zip(df_vals,val_nums),total=len(df_vals)):\n    images = list(df_val.columns.values)\n    for x in images:\n        try:\n            val_paths.append(get_path(num,x))\n            val_y.append(LABELS.index(df_val[x]['label']))\n        except Exception as err:\n            #print(err)\n            pass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('There are '+str(y.count(1))+' fake train samples')\nprint('There are '+str(y.count(0))+' real train samples')\nprint('There are '+str(val_y.count(1))+' fake val samples')\nprint('There are '+str(val_y.count(0))+' real val samples')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[0], meta)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[4], meta)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"In the selection of extraction methods, we have chosen four face extraction methods: BlazeFace, MTCNN, MobileNet, and YOLO."},{"metadata":{},"cell_type":"markdown","source":"# Initialize BlazeFace"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nFirst look at BlazeFace. This is a very efficient and lightweight face detector. It performs very well in face detection tasks in close-up frontal scenes. It can not only ensure the accuracy of detection, but also the fast reasoning speed can reach the sub-millisecond level in the general mobile phone CPU. Why it can have such high performance? One is the design of the backbone network, which uses the depthwise convolution of the large receptive field and 1x1 convolution kernel to speed up the features, and uses the nested single blaze method to increase the expression of the model ability; the second is because of the design of the detection regression. Since the main task of the algorithm is the detection of positive faces, in most cases, the anchor with a ratio of 1.0 can get good results. BlazeFace only cascades 2 scale features for face detection, and each point uses only two anchors under 16x16 features, and each point uses 6 anchors under 8x8 features. In the case of uncomplicated data distribution, both it can solve problems and increase the speed of network reasoning, which makes BlazeFace efficient and lightweight\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"blazeface = BlazeFace()\nblazeface.load_weights(\"/kaggle/input/blazefacepytorch/blazeface.pth\")\nblazeface.load_anchors(\"/kaggle/input/blazefacepytorch/anchors.npy\")\n\nblazeface.min_score_thresh = 0.75\nblazeface.min_suppression_threshold = 0.3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_blaze_boxes(detections, with_keypoints=False):\n    result = []\n    if isinstance(detections, torch.Tensor):\n        detections = detections.cpu().numpy()\n\n    if detections.ndim == 1:\n        detections = np.expand_dims(detections, axis=0)\n\n    img_shape = (128, 128)\n    for i in range(detections.shape[0]):\n        ymin = detections[i, 0] * img_shape[0]\n        xmin = detections[i, 1] * img_shape[1]\n        ymax = detections[i, 2] * img_shape[0]\n        xmax = detections[i, 3] * img_shape[1]\n        result.append((xmin, ymin, xmax, ymax))\n    return result\ndef scale_boxes(boxes, scale_w, scale_h):\n    sb = []\n    for b in boxes:\n        sb.append((b[0] * scale_w, b[1] * scale_h, b[2] * scale_w, b[3] * scale_h))\n    return sb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initialize Yolo"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nYOLO, its full name is You Only Look Once, which basically summarizes the characteristics of YOLO completely, that is, only one CNN operation is required. YOLO solves object detection as a regression problem. Based on a single end-to-end network, complete the input from the original image to the output of the object position and category. Both YOLO training and detection are performed in a separate network. YOLO did not show the process of obtaining the region proposal. The YOLO network draws on the GoogLeNet classification network structure. The difference is that YOLO does not use the inception module, but uses 1x1 convolutional layer (here 1x1 convolutional layer exists for cross-channel information integration) + 3x3 convolutional layer simple replacement. YOLO model training is divided into two steps, one is pre-training, using data to train the first 20 convolutional layers of the YOLO network + 1 average pooling layer + 1 fully connected layer. Resize the training image resolution to 224x224; then use the first 20 convolutional layer network parameters obtained in step 1 to initialize the network parameters of the first 20 convolutional layers of the YOLO model, and then use the VOC 20 type of annotation data for YOLO model training. In order to improve the image accuracy, when training the detection model, the input image resolution is resized to 448x448. In general, YOLO has the characteristics of fast speed, low background false detection rate and strong versatility.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_mobilenetv2_224_075_detector(path):\n    input_tensor = Input(shape=(224, 224, 3))\n    output_tensor = MobileNetV2(weights=None, include_top=False, input_tensor=input_tensor, alpha=0.75).output\n    output_tensor = ZeroPadding2D()(output_tensor)\n    output_tensor = Conv2D(kernel_size=(3, 3), filters=5)(output_tensor)\n\n    model = Model(inputs=input_tensor, outputs=output_tensor)\n    model.load_weights(path)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.layers import Conv2D, Input, ZeroPadding2D, Dense, Lambda\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.applications.mobilenet_v2 import MobileNetV2\nmobilenetv2 = load_mobilenetv2_224_075_detector(\"../input/facedetectionmobilenetv2/facedetection-mobilenetv2-size224-alpha0.75.h5\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transpose_shots(shots):\n    return [(shot[1], shot[0], shot[3], shot[2], shot[4]) for shot in shots]\n\n#That constant describe pieces for 16:9 images\nSHOTS = {\n    # fast less accurate\n    '2-16/9' : {\n        'aspect_ratio' : 16/9,\n        'shots' : [\n             (0, 0, 9/16, 1, 1),\n             (7/16, 0, 9/16, 1, 1)\n        ]\n    },\n    # slower more accurate\n    '10-16/9' : {\n        'aspect_ratio' : 16/9,\n        'shots' : [\n             (0, 0, 9/16, 1, 1),\n             (7/16, 0, 9/16, 1, 1),\n             (0, 0, 5/16, 5/9, 0.5),\n             (0, 4/9, 5/16, 5/9, 0.5),\n             (11/48, 0, 5/16, 5/9, 0.5),\n             (11/48, 4/9, 5/16, 5/9, 0.5),\n             (22/48, 0, 5/16, 5/9, 0.5),\n             (22/48, 4/9, 5/16, 5/9, 0.5),\n             (11/16, 0, 5/16, 5/9, 0.5),\n             (11/16, 4/9, 5/16, 5/9, 0.5),\n        ]\n    }\n}\n\n# 9:16 respectively\nSHOTS_T = {\n    '2-9/16' : {\n        'aspect_ratio' : 9/16,\n        'shots' : transpose_shots(SHOTS['2-16/9']['shots'])\n    },\n    '10-9/16' : {\n        'aspect_ratio' : 9/16,\n        'shots' : transpose_shots(SHOTS['10-16/9']['shots'])\n    }\n}\n\ndef r(x):\n    return int(round(x))\n\ndef sigmoid(x):\n    return 1 / (np.exp(-x) + 1)\n\ndef non_max_suppression(boxes, p, iou_threshold):\n    if len(boxes) == 0:\n        return np.array([])\n\n    x1 = boxes[:, 0]\n    y1 = boxes[:, 1]\n    x2 = boxes[:, 2]\n    y2 = boxes[:, 3]\n\n    indexes = np.argsort(p)\n    true_boxes_indexes = []\n\n    while len(indexes) > 0:\n        true_boxes_indexes.append(indexes[-1])\n\n        intersection = np.maximum(np.minimum(x2[indexes[:-1]], x2[indexes[-1]]) - np.maximum(x1[indexes[:-1]], x1[indexes[-1]]), 0) * np.maximum(np.minimum(y2[indexes[:-1]], y2[indexes[-1]]) - np.maximum(y1[indexes[:-1]], y1[indexes[-1]]), 0)\n        iou = intersection / ((x2[indexes[:-1]] - x1[indexes[:-1]]) * (y2[indexes[:-1]] - y1[indexes[:-1]]) + (x2[indexes[-1]] - x1[indexes[-1]]) * (y2[indexes[-1]] - y1[indexes[-1]]) - intersection)\n\n        indexes = np.delete(indexes, -1)\n        indexes = np.delete(indexes, np.where(iou >= iou_threshold)[0])\n\n    return boxes[true_boxes_indexes]\n\ndef union_suppression(boxes, threshold):\n    if len(boxes) == 0:\n        return np.array([])\n\n    x1 = boxes[:, 0]\n    y1 = boxes[:, 1]\n    x2 = boxes[:, 2]\n    y2 = boxes[:, 3]\n\n    indexes = np.argsort((x2 - x1) * (y2 - y1))\n    result_boxes = []\n\n    while len(indexes) > 0:\n        intersection = np.maximum(np.minimum(x2[indexes[:-1]], x2[indexes[-1]]) - np.maximum(x1[indexes[:-1]], x1[indexes[-1]]), 0) * np.maximum(np.minimum(y2[indexes[:-1]], y2[indexes[-1]]) - np.maximum(y1[indexes[:-1]], y1[indexes[-1]]), 0)\n        min_s = np.minimum((x2[indexes[:-1]] - x1[indexes[:-1]]) * (y2[indexes[:-1]] - y1[indexes[:-1]]), (x2[indexes[-1]] - x1[indexes[-1]]) * (y2[indexes[-1]] - y1[indexes[-1]]))\n        ioms = intersection / (min_s + 1e-9)\n        neighbours = np.where(ioms >= threshold)[0]\n        if len(neighbours) > 0:\n            result_boxes.append([min(np.min(x1[indexes[neighbours]]), x1[indexes[-1]]), min(np.min(y1[indexes[neighbours]]), y1[indexes[-1]]), max(np.max(x2[indexes[neighbours]]), x2[indexes[-1]]), max(np.max(y2[indexes[neighbours]]), y2[indexes[-1]])])\n        else:\n            result_boxes.append([x1[indexes[-1]], y1[indexes[-1]], x2[indexes[-1]], y2[indexes[-1]]])\n\n        indexes = np.delete(indexes, -1)\n        indexes = np.delete(indexes, neighbours)\n\n    return result_boxes\n\nclass FaceDetector():\n# Face Detection\n    def __init__(self, model=mobilenetv2, shots=[SHOTS['10-16/9'], SHOTS_T['10-9/16']], image_size=224, grids=7, iou_threshold=0.1, union_threshold=0.1):\n        self.model = model\n        self.shots = shots\n        self.image_size = image_size\n        self.grids = grids\n        self.iou_threshold = iou_threshold\n        self.union_threshold = union_threshold\n        self.prob_threshold = 0.7\n        \n    \n    def detect(self, frame, threshold = 0.7):\n        original_frame_shape = frame.shape\n        self.prob_threshold = threshold\n        aspect_ratio = None\n        for shot in self.shots:\n            if abs(frame.shape[1] / frame.shape[0] - shot[\"aspect_ratio\"]) < 1e-9:\n                aspect_ratio = shot[\"aspect_ratio\"]\n                shots = shot\n        \n        assert aspect_ratio is not None\n        \n        c = min(frame.shape[0], frame.shape[1] / aspect_ratio)\n        slice_h_shift = r((frame.shape[0] - c) / 2)\n        slice_w_shift = r((frame.shape[1] - c * aspect_ratio) / 2)\n        if slice_w_shift != 0 and slice_h_shift == 0:\n            frame = frame[:, slice_w_shift:-slice_w_shift]\n        elif slice_w_shift == 0 and slice_h_shift != 0:\n            frame = frame[slice_h_shift:-slice_h_shift, :]\n\n        frames = []\n        for s in shots[\"shots\"]:\n            frames.append(cv2.resize(frame[r(s[1] * frame.shape[0]):r((s[1] + s[3]) * frame.shape[0]), r(s[0] * frame.shape[1]):r((s[0] + s[2]) * frame.shape[1])], (self.image_size, self.image_size), interpolation=cv2.INTER_NEAREST))\n        frames = np.array(frames)\n\n        predictions = self.model.predict(frames, batch_size=len(frames), verbose=0)\n\n        boxes = []\n        prob = []\n        shots = shots['shots']\n        for i in range(len(shots)):\n            slice_boxes = []\n            slice_prob = []\n            for j in range(predictions.shape[1]):\n                for k in range(predictions.shape[2]):\n                    p = sigmoid(predictions[i][j][k][4])\n                    if not(p is None) and p > self.prob_threshold:\n                        px = sigmoid(predictions[i][j][k][0])\n                        py = sigmoid(predictions[i][j][k][1])\n                        pw = min(math.exp(predictions[i][j][k][2] / self.grids), self.grids)\n                        ph = min(math.exp(predictions[i][j][k][3] / self.grids), self.grids)\n                        if not(px is None) and not(py is None) and not(pw is None) and not(ph is None) and pw > 1e-9 and ph > 1e-9:\n                            cx = (px + j) / self.grids\n                            cy = (py + k) / self.grids\n                            wx = pw / self.grids\n                            wy = ph / self.grids\n                            if wx <= shots[i][4] and wy <= shots[i][4]:\n                                lx = min(max(cx - wx / 2, 0), 1)\n                                ly = min(max(cy - wy / 2, 0), 1)\n                                rx = min(max(cx + wx / 2, 0), 1)\n                                ry = min(max(cy + wy / 2, 0), 1)\n\n                                lx *= shots[i][2]\n                                ly *= shots[i][3]\n                                rx *= shots[i][2]\n                                ry *= shots[i][3]\n\n                                lx += shots[i][0]\n                                ly += shots[i][1]\n                                rx += shots[i][0]\n                                ry += shots[i][1]\n\n                                slice_boxes.append([lx, ly, rx, ry])\n                                slice_prob.append(p)\n\n            slice_boxes = np.array(slice_boxes)\n            slice_prob = np.array(slice_prob)\n\n            slice_boxes = non_max_suppression(slice_boxes, slice_prob, self.iou_threshold)\n\n            for sb in slice_boxes:\n                boxes.append(sb)\n\n\n        boxes = np.array(boxes)\n        boxes = union_suppression(boxes, self.union_threshold)\n\n        for i in range(len(boxes)):\n            boxes[i][0] /= original_frame_shape[1] / frame.shape[1]\n            boxes[i][1] /= original_frame_shape[0] / frame.shape[0]\n            boxes[i][2] /= original_frame_shape[1] / frame.shape[1]\n            boxes[i][3] /= original_frame_shape[0] / frame.shape[0]\n\n            boxes[i][0] += slice_w_shift / original_frame_shape[1]\n            boxes[i][1] += slice_h_shift / original_frame_shape[0]\n            boxes[i][2] += slice_w_shift / original_frame_shape[1]\n            boxes[i][3] += slice_h_shift / original_frame_shape[0]\n\n        return list(boxes)\ndef get_boxes_points(boxes, frame_shape):\n    result = []\n    for box in boxes:\n        lx = int(round(box[0] * frame_shape[1]))\n        ly = int(round(box[1] * frame_shape[0]))\n        rx = int(round(box[2] * frame_shape[1]))\n        ry = int(round(box[3] * frame_shape[0]))\n        result.append((lx,rx, ly, ry))\n    return result ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"yolo_model = FaceDetector()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initialize MTCNN"},{"metadata":{},"cell_type":"markdown","source":"MTCNN, Multi-task convolutional neural network, is a multi-task neural network model for face detection tasks proposed by Shenzhen Research Institute of Chinese Academy of Sciences in 2016. This model mainly uses three cascaded networks, and uses the idea of candidate box plus the classifier is to perform fast and efficient face detection. The three cascaded networks are P-Net for quickly generating candidate windows, R-Net for filtering and selecting high-precision candidate windows, and O-Net for generating final bounding boxes and face key points. The full name of P-Net is Proposal Network, and its basic structure is a fully convolutional network. For the image pyramid constructed in the previous step, an FCN is used to perform preliminary feature extraction and frame calibration, and perform the Bounding-Box Regression adjustment window and NMS to filter most of the windows. P-Net is a region suggestion network for the face region. After the features are input to the three convolutional layers of the result, the network uses a face classifier to determine whether the region is a face, and uses border regression and a face key point. The locator is used to make the initial proposal of the face area. This part will eventually output a lot of face areas that may have faces, and input these areas into R-Net for further processing. The full name of R-Net is Refine Network. Its basic structure is a convolutional neural network. Compared with the first layer of P-Net, a fully connected layer is added, so the input data will be more stringent. After the picture passes through the P-Net, many prediction windows will be left. We send all the prediction windows to R-Net. This network will filter out a large number of candidate frames with poor results, and finally perform Bounding-Box Regression and NMS on the selected candidate boxes to further optimize the prediction results. Because the output of P-Net is only a possible face area with a certain degree of credibility, in this network, the input will be refined and selected, and most of the wrong input will be discarded, and again use the border regression and facial key point locator to carry out the border regression and key point positioning of the face area, and finally outputs a more credible face area for O-Net. Compared with the 1x1x32 feature of P-Net using full convolution output, R-Net uses a 128 fully connected layer after the last convolution layer, which retains more image features, and the accuracy performance is also better than P-Net. O-Net is called Output Network. The basic structure is a more complicated convolutional neural network. Compared with R-Net, there is one more convolutional layer. The difference between the effect of O-Net and R-Net is that this layer of structure will recognize the area of the face through more supervision, and will regress the facial feature points of the person, and finally output five facial feature points. It is a more complex convolutional network with more input features. At the end of the network structure, there is also a larger 256 fully connected layer, which retains more image features, and at the same time performs face discrimination and human face area border regression and face feature positioning, and finally output the coordinates of the upper left corner and the lower right corner of the face area and the five feature points of the face area. O-Net has more characteristic input and more complex network structure, and also has better performance. The output of this layer is used as the final network model output. In order to balance performance and accuracy, MTCNN avoids the huge performance consumption caused by traditional ideas such as sliding window plus classifier, first use a small model to generate a certain possibility of target area candidate frame, then use more complex models for fine classification and higher precision area box regression, and let this step be executed recursively, this idea forms a three-layer network, namely P-Net, R-Net, and O-Net, to achieve fast and efficient face detection."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install mtcnn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from mtcnn import MTCNN\nmtcnn = MTCNN()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initialize MobilenetFace"},{"metadata":{},"cell_type":"markdown","source":"MobileNet, which is a small and efficient CNN model proposed by Google, has a compromise between accuracy and latency. The basic unit of MobileNet is depthwise separable convolution. In fact, this structure has been used in the Inception model before. Depth-level separable convolution is actually a factorized convolutions, which can be decomposed into two smaller operations: depthwise convolution and pointwise convolution. Depthwise convolution is different from standard convolution. For standard convolution, the convolution kernel is used on all input channels, while depthwise convolution uses different convolution kernels for each input channel, that is, a convolution kernel. Corresponds to an input channel, so depthwise convolution is a depth-level operation. The pointwise convolution is actually an ordinary convolution, but it uses a 1x1 convolution kernel. For depthwise separable convolution, it first uses depthwise convolution to convolve different input channels separately, and then uses pointwise convolution to combine the above outputs, so that the overall effect is similar to a standard convolution, but it will greatly reduce the calculation quantity and model parameter quantity. The above described depthwise separable convolution, which is the basic component of MobileNet, but batchnorm will be added to the real application and the ReLU activation function will be used. In general, its core is the use of decomposable depthwise separable convolution, which can not only reduce the computational complexity of the model, but also greatly reduce the model size."},{"metadata":{"trusted":true},"cell_type":"code","source":"detection_graph = tf.Graph()\nwith detection_graph.as_default():\n    od_graph_def = tf.compat.v1.GraphDef()\n    with tf.io.gfile.GFile('../input/mobilenetface/frozen_inference_graph_face.pb', 'rb') as fid:\n        serialized_graph = fid.read()\n        od_graph_def.ParseFromString(serialized_graph)\n        tf.import_graph_def(od_graph_def, name='')\n        config = tf.compat.v1.ConfigProto()\n    config.gpu_options.allow_growth = True\n    sess=tf.compat.v1.Session(graph=detection_graph, config=config)\n    image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')\n    boxes_tensor = detection_graph.get_tensor_by_name('detection_boxes:0')    \n    scores_tensor = detection_graph.get_tensor_by_name('detection_scores:0')\n    num_detections = detection_graph.get_tensor_by_name('num_detections:0')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"video='../input/deepfake-detection-challenge/train_sample_videos/bdnaqemxmr.mp4'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"video_cap = cv2.VideoCapture(video)\n\nframe_count = 0\nall_frames = []\nwhile(True):\n    ret, frame = video_cap.read()\n    if ret is False:\n        break\n    all_frames.append(frame)\n    frame_count = frame_count + 1\n\n# The value below are both the number of frames\nprint (frame_count)\nprint (len(all_frames))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cap = cv2.VideoCapture(video)\n# Frame rate\nfps = int(round(cap.get(cv2.CAP_PROP_FPS)))\n# Resolution-Width\nwidth = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\n# Resolution-Height\nheight = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n# Total number of frames\nframe_counter = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\ncap.release()\ncv2.destroyAllWindows()\n# Duration, unit s\nduration = frame_counter / fps","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cap=cv2.VideoCapture(video)\nret,frame=cap.read()\nframe=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\nplt.imshow(frame)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" ****"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mtcnn_face(img):\n    start=time.time()\n    bboxes=mtcnn.detect_faces(frame)[0]['box']\n    x,y,w,h=bboxes\n    bboxes=x,x+w,y,y+h\n    return time.time()-start, bboxes\ndef get_blazeface_face(img):\n    start=time.time()\n    scale_w = img.shape[1] / 128.0 \n    scale_h = img.shape[0] / 128.0\n    blaze_output=blazeface.predict_on_image(cv2.resize(frame, (128,128)))\n    blaze_bboxes=scale_boxes(get_blaze_boxes(blaze_output), scale_w, scale_h)\n    if blaze_bboxes==[]:\n        return time.time()-start,[]\n    lx, ly, rx, ry = blaze_bboxes[0]\n    bboxes=int(lx), int(rx), int(ly), int(ry)\n    return time.time()-start, bboxes\ndef get_mobilenet_face(image):\n    start=time.time()\n    global boxes,scores,num_detections\n    (im_height,im_width)=image.shape[:-1]\n    imgs=np.array([image])\n    (boxes, scores) = sess.run(\n        [boxes_tensor, scores_tensor],\n        feed_dict={image_tensor: imgs})\n    max_=np.where(scores==scores.max())[0][0]\n    box=boxes[0][max_]\n    ymin, xmin, ymax, xmax = box\n    (left, right, top, bottom) = (xmin * im_width, xmax * im_width,\n                                ymin * im_height, ymax * im_height)\n    left, right, top, bottom = int(left), int(right), int(top), int(bottom)\n    return time.time()-start,(left, right, top, bottom)\ndef get_yolo_face(image):\n    start=time.time()\n    bbox=yolo_model.detect(frame, 0.7)\n    bbox=get_boxes_points(bbox,frame.shape)[0]\n    return time.time()-start,bbox","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def annotate_image(frame,bbox,color):\n    if bbox==[]:\n        return frame\n    frame=frame.copy()\n    return cv2.rectangle(frame,(bbox[0],bbox[2]),(bbox[1],bbox[3]),color,10)\ndef crop_image(frame,bbox):\n    left, right, top, bottom=bbox\n    return frame[top:bottom,left:right]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_=get_blazeface_face(frame)\n_=get_mtcnn_face(frame)\n_=get_mobilenet_face(frame)\n_=get_yolo_face(frame)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"blaze_time,blaze_bboxes=get_blazeface_face(frame)\nmtcnn_time,mtcnn_bboxes=get_mtcnn_face(frame)\nmobilenet_time,mobilenet_bboxes=get_mobilenet_face(frame)\nyolo_time,yolo_bboxes=get_yolo_face(frame)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"BlazeFace detection time:\"+str(blaze_time))\nprint(\"MTCNN detection time:\"+str(mtcnn_time))\nprint(\"Yolo detection time:\"+str(yolo_time))\nprint(\"Mobilenet detection time:\"+str(mobilenet_time))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ability to Detect Face"},{"metadata":{"trusted":true},"cell_type":"code","source":"if blaze_bboxes==[]:\n    print('⚠️BlazeFace cannot detect faces in this picture.。')\nif mtcnn_bboxes==[]:\n    print('⚠️MTCNN cannot detect faces in this picture.。')\nif mobilenet_bboxes==[]:\n    print('⚠️mobilenet cannot detect faces in this picture.。')\nif yolo_bboxes==[]:\n    print('⚠️mobilenet cannot detect faces in this picture.。')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Accuracy Comparison\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"annotated=annotate_image(frame,mobilenet_bboxes,(255,0,0))\nannotated=annotate_image(annotated,mtcnn_bboxes,(0,255,0))\nannotated=annotate_image(annotated,blaze_bboxes,(0,0,255))\nannotated=annotate_image(annotated,yolo_bboxes,(255,0,255))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Blue: BlazeFace\nRed: Mobilenet\nGreen: MTCNN\nPurple: YOLO"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(annotated)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cropped Images"},{"metadata":{},"cell_type":"markdown","source":"# BlazeFace\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(crop_image(frame,blaze_bboxes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MTCNN\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(crop_image(frame,mtcnn_bboxes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mobilenet"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(crop_image(frame,mobilenet_bboxes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# YOLO"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(crop_image(frame,yolo_bboxes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extra Comparisons\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"video='../input/deepfake-detection-challenge/train_sample_videos/eqnoqyfquo.mp4'\ncap=cv2.VideoCapture(video)    \nret,frame=cap.read()\nframe=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\nblaze_time,blaze_bboxes=get_blazeface_face(frame)\nmtcnn_time,mtcnn_bboxes=get_mtcnn_face(frame)\nmobilenet_time,mobilenet_bboxes=get_mobilenet_face(frame)\nyolo_time,yolo_bboxes=get_yolo_face(frame)  \nprint(\"MTCNN detection time:\"+str(mtcnn_time))\nprint(\"Yolo detection time:\"+str(yolo_time))\nprint(\"Mobilenet detection time:\"+str(mobilenet_time))\nprint(\"BlazeFace detection time:\"+str(blaze_time))\nannotated=annotate_image(frame,mobilenet_bboxes,(255,0,0))\nannotated=annotate_image(annotated,mtcnn_bboxes,(0,255,0))\nannotated=annotate_image(annotated,blaze_bboxes,(0,0,255))\nannotated=annotate_image(annotated,yolo_bboxes,(255,0,255))\nplt.imshow(annotated)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"video='../input/deepfake-detection-challenge/train_sample_videos/eqjscdagiv.mp4'\ncap=cv2.VideoCapture(video)    \nret,frame=cap.read()\nframe=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\nblaze_time,blaze_bboxes=get_blazeface_face(frame)\nmtcnn_time,mtcnn_bboxes=get_mtcnn_face(frame)\nmobilenet_time,mobilenet_bboxes=get_mobilenet_face(frame)\nyolo_time,yolo_bboxes=get_yolo_face(frame)  \nprint(\"MTCNN detection time:\"+str(mtcnn_time))\nprint(\"Yolo detection time:\"+str(yolo_time))\nprint(\"Mobilenet detection time:\"+str(mobilenet_time))\nprint(\"BlazeFace detection time:\"+str(blaze_time))\nannotated=annotate_image(frame,mobilenet_bboxes,(255,0,0))\nannotated=annotate_image(annotated,mtcnn_bboxes,(0,255,0))\nannotated=annotate_image(annotated,blaze_bboxes,(0,0,255))\nannotated=annotate_image(annotated,yolo_bboxes,(255,0,255))\nplt.imshow(annotated)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"video='../input/deepfake-detection-challenge/train_sample_videos/emgjphonqb.mp4'\ncap=cv2.VideoCapture(video)    \nret,frame=cap.read()\nframe=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\nblaze_time,blaze_bboxes=get_blazeface_face(frame)\nmtcnn_time,mtcnn_bboxes=get_mtcnn_face(frame)\nmobilenet_time,mobilenet_bboxes=get_mobilenet_face(frame)\nyolo_time,yolo_bboxes=get_yolo_face(frame)  \nprint(\"MTCNN detection time :\"+str(mtcnn_time))\nprint(\"Yolo detection time:\"+str(yolo_time))\nprint(\"Mobilenet detection time:\"+str(mobilenet_time))\nprint(\"BlazeFace detection time:\"+str(blaze_time))\nannotated=annotate_image(frame,mobilenet_bboxes,(255,0,0))\nannotated=annotate_image(annotated,mtcnn_bboxes,(0,255,0))\nannotated=annotate_image(annotated,blaze_bboxes,(0,0,255))\nannotated=annotate_image(annotated,yolo_bboxes,(255,0,255))\nplt.imshow(annotated)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4.\tMethods"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nIn this part, we will introduce the related algorithms used in this work, mainly Resnet, Xception and ResNext. First look at Resnet. The proposal of the Deep Residual Network (ResNet) is a milestone event in the history of CNN images. It achieved 5 first results in ILSVRC and COCO 2015. In fact, ResNet solved the deep CNN model. The problem of difficulty in training, 14 years of VGG has only 19 layers, and 15 years of ResNet has as many as 152 layers, which is not an order of magnitude in network depth. Although ResNet wins by depth, there are tricks in its architecture, which makes the depth of the network play a role. This trick is Residual Learning. Use residual learning to solve the degradation problem. For a stacked layer structure (several layers stacked) when the input is x, the learned feature is recorded as H(x), and now I hope it can learn the residual F(x)=H(x)-x, so the original learning feature is F(x)+x. The reason for this is that residual learning is easier than direct learning of original features. When the residual is 0, the stacked layer only does the identity mapping at this time, at least the network performance will not decrease, in fact the residual will not be 0, which will also make the stacked layer learn new features based on the input features for better performance.\n"},{"metadata":{},"cell_type":"markdown","source":"The ResNet network is based on the VGG19 network, modified on the basis of it, and added a residual unit through a short-circuit mechanism. The change is mainly reflected in ResNet directly using stride=2 convolution for downsampling, and replacing the fully connected layer with the global average pool layer. An important design principle of ResNet is: when the size of the feature map is reduced by half, the number of feature maps doubles, which maintains the complexity of the network layer. Compared with ordinary networks, ResNet adds a short-circuit mechanism between every two layers, which forms residual learning. Of course, you can also build a deeper network. For 18-layer and 34-layer ResNet, it performs residual learning between two layers. When the network is deeper, it performs residual learning between three layers, three-layer volume The product cores are 1x1, 3x3, and 1x1. It is worth noting that the number of feature maps in the hidden layer is relatively small, and it is 1/4 of the number of output feature maps. Then analyze the residual unit, ResNet uses two residual units, shallow network and deep network. For short-circuit connection, when the input and output dimensions are the same, the input can be directly added to the output. But when the dimensions are inconsistent (corresponding to double the dimensions), this cannot be added directly. There are two strategies: (1) Use zero-padding to increase the dimension. At this time, a downsamp is generally required. You can use the pooling with strde=2, so that no parameters will be added; (2) Use a new mapping (projection shortcut), Generally, a 1x1 convolution is used, which will increase the parameters and increase the amount of calculation. In addition to directly using identity mapping for short-circuit connections, of course, projection shortcuts can be used.\n\nThe second is Xception, which is another improvement to Inception-v3 proposed by Google after Inception. The structure of Xception is based on ResNet, but the convolutional layer is replaced with Separable Convolution. The Xception architecture has 36 convolutional layers, which form the basis of the feature extraction of the network. In studying the problem of image classification, the basis of convolution will follow the logistic regression layer. Optionally, a fully connected layer can be inserted before the logistic regression layer. The 36 convolutional layers are constructed into 14 modules, all of which have linear residual connections around them except for the first and last modules. In short, the Xception architecture is a linear stack of deeply separable convolutional layers with residual connections. This makes the architecture very easy to define and modify. Using high-level libraries (such as Keras or TensorFlow-Slim) only requires 30 to 40 lines of code, which is different from architectures such as VGG-16, but is much more complicated to define than architectures such as Inception V2 or V3. Compared with Inception V3, in terms of classification performance, Xception has a smaller lead on ImageNet, but is much ahead of JFT; in terms of parameters and speed, Xception has fewer parameters than Inception, but is faster.\n\nAnother is ResNext, which is a combination of ResNet and Inception. Unlike Inception v4, ResNext does not require manual design of complex Inception structural details, but each branch uses the same topology. The essence of ResNeXt is Group Convolution, which controls the number of groups by variable cardinality. The convolution machine is a compromise between ordinary convolution and depth separable convolution, that is, the number of channels of the Feature Map generated by each branch is n (n>1). Another difference from Inception v4 is that ResNeXt first performs 1×1 convolution and then performs unit addition. Inception V4 performs splicing and then performs 1×1 convolution. There is also grouped convolution. Grouped convolution is a compromise between the depth of the ordinary convolution kernel and the separable convolution. It does not completely assign each channel to an independent convolution kernel or the entire Feature Map. Use the same convolution kernel. In general, ResNeXt proposes a strategy of separable convolution between the depth of the ordinary convolution kernel: grouped convolution, which achieves a balance between the two strategies by controlling the number of groups (base). The idea of grouped convolution is derived from Inception. Unlike Inception, which requires manual design of each branch, the topology of each branch of ResNeXt is the same. Finally, combined with the residual network, the final ResNext is obtained. ResNeXt does have fewer hyperparameters than Inception V4, but it does not seem to be very reasonable to directly abolish Inception's characteristics of including different receptive fields. It is found that Inception V4 is better than ResNeXt in more environments. The running speed of ResNeXt of similar structure should be better than Inception V4, because the design of the branch of the same topology of ResNeXt is more in line with GPU hardware design principles.\n\nThe above three algorithms are the algorithms used in this work. We use AUC-ROC (Area Under Curve) to measure the quality of machine learning models. AUC-ROC is a commonly used evaluation index for classifiers in machine learning. Its rules are summarized in one sentence: a positive example, a negative example, the probability that the prediction is positive is greater than the probability that the prediction is negative . We show the effect of our work by drawing the ROC curve. The area under the ROC curve is the value of AUC. The specific diagram will be given in detail in the next part. For the training of the model mentioned above, we use packages such as numpy, pandas, matplotlib.pyplot, glob, cv2, torch, and tensorflow in python.\n"},{"metadata":{},"cell_type":"markdown","source":"# 5.\tExperiments and Results"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"# MesoNet"},{"metadata":{},"cell_type":"markdown","source":"现在我们使用MesoNet网络进行训练，在这之前我们做了数据集的一些分析，发现这并不是一个均衡的数据集，为此我们采用了下采样的方式对数据集进行处理，通过K-5交叉验证的方式进行，并通过log_loss作为训练结果评估的标准，准确率在测试集上大约有60%以上的正确率。我们发现自己训练出的模型效果并不出色，我想这与我们数据集的使用不够科学有关，为此我们尝试了其他的神经网络模型进行预测，通过实验可以表明，相对比MesoNet、Xception和Resnext，Resnext的表现结果是最好的。"},{"metadata":{"trusted":true},"cell_type":"code","source":"real=[]\nfake=[]\nfor m,n in zip(paths,y):\n    if n==0:\n        real.append(m)\n    else:\n        fake.append(m)\nfake=random.sample(fake,len(real))\npaths,y=[],[]\nfor x in real:\n    paths.append(x)\n    y.append(0)\nfor x in fake:\n    paths.append(x)\n    y.append(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"real=[]\nfake=[]\nfor m,n in zip(val_paths,val_y):\n    if n==0:\n        real.append(m)\n    else:\n        fake.append(m)\nfake=random.sample(fake,len(real))\nval_paths,val_y=[],[]\nfor x in real:\n    val_paths.append(x)\n    val_y.append(0)\nfor x in fake:\n    val_paths.append(x)\n    val_y.append(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('There are '+str(y.count(1))+' fake train samples')\nprint('There are '+str(y.count(0))+' real train samples')\nprint('There are '+str(val_y.count(1))+' fake val samples')\nprint('There are '+str(val_y.count(0))+' real val samples')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_img(path):\n    return cv2.cvtColor(cv2.imread(path),cv2.COLOR_BGR2RGB)\nX=[]\nfor img in tqdm(paths):\n    X.append(read_img(img))\nval_X=[]\nfor img in tqdm(val_paths):\n    val_X.append(read_img(img))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_img(path):\n    return cv2.cvtColor(cv2.imread(path),cv2.COLOR_BGR2RGB)\nX=[]\nfor img in tqdm(paths):\n    X.append(read_img(img))\nval_X=[]\nfor img in tqdm(val_paths):\n    val_X.append(read_img(img))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\ndef shuffle(X,y):\n    new_train=[]\n    for m,n in zip(X,y):\n        new_train.append([m,n])\n    random.shuffle(new_train)\n    X,y=[],[]\n    for x in new_train:\n        X.append(x[0])\n        y.append(x[1])\n    return X,y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport keras\nimport os\nimport numpy as np\nfrom sklearn.metrics import log_loss\nfrom keras import Model,Sequential\nfrom keras.layers import *\nfrom keras.optimizers import *\nfrom sklearn.model_selection import train_test_split\nimport cv2\nfrom tqdm.notebook import tqdm\nimport glob\nfrom mtcnn import MTCNN","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def InceptionLayer(a, b, c, d):\n    def func(x):\n        x1 = Conv2D(a, (1, 1), padding='same', activation='elu')(x)\n        \n        x2 = Conv2D(b, (1, 1), padding='same', activation='elu')(x)\n        x2 = Conv2D(b, (3, 3), padding='same', activation='elu')(x2)\n            \n        x3 = Conv2D(c, (1, 1), padding='same', activation='elu')(x)\n        x3 = Conv2D(c, (3, 3), dilation_rate = 2, strides = 1, padding='same', activation='elu')(x3)\n        \n        x4 = Conv2D(d, (1, 1), padding='same', activation='elu')(x)\n        x4 = Conv2D(d, (3, 3), dilation_rate = 3, strides = 1, padding='same', activation='elu')(x4)\n        y = Concatenate(axis = -1)([x1, x2, x3, x4])\n            \n        return y\n    return func\n    \ndef define_model(shape=(256,256,3)):\n    x = Input(shape = shape)\n    \n    x1 = InceptionLayer(1, 4, 4, 2)(x)\n    x1 = BatchNormalization()(x1)\n    x1 = MaxPooling2D(pool_size=(2, 2), padding='same')(x1)\n    \n    x2 = InceptionLayer(2, 4, 4, 2)(x1)\n    x2 = BatchNormalization()(x2)        \n    x2 = MaxPooling2D(pool_size=(2, 2), padding='same')(x2)        \n        \n    x3 = Conv2D(16, (5, 5), padding='same', activation = 'elu')(x2)\n    x3 = BatchNormalization()(x3)\n    x3 = MaxPooling2D(pool_size=(2, 2), padding='same')(x3)\n        \n    x4 = Conv2D(16, (5, 5), padding='same', activation = 'elu')(x3)\n    x4 = BatchNormalization()(x4)\n    if shape==(256,256,3):\n        x4 = MaxPooling2D(pool_size=(4, 4), padding='same')(x4)\n    else:\n        x4 = MaxPooling2D(pool_size=(2, 2), padding='same')(x4)\n    y = Flatten()(x4)\n    y = Dropout(0.5)(y)\n    y = Dense(16)(y)\n    y = LeakyReLU(alpha=0.1)(y)\n    y = Dropout(0.5)(y)\n    y = Dense(1, activation = 'sigmoid')(y)\n    model=Model(inputs = x, outputs = y)\n    model.compile(loss='binary_crossentropy',optimizer=Adam(lr=1e-4))\n    #model.summary()\n    return model\ndf_model=define_model()\ndf_model.load_weights('../input/mesopretrain/MesoInception_DF')\nf2f_model=define_model()\nf2f_model.load_weights('../input/mesopretrain/MesoInception_F2F')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import LearningRateScheduler\nlrs=[1e-3,5e-4,1e-4]\ndef schedule(epoch):\n    return lrs[epoch]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"LOAD_PRETRAIN=True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nkfolds=5\nlosses=[]\nif LOAD_PRETRAIN:\n    # import keras.backend as K\n    df_models=[]\n    f2f_models=[]\n    i=0\n    while len(df_models)<kfolds:\n        model=define_model((150,150,3))\n        if i==0:\n            model.summary()\n        #model.load_weights('../input/meso-pretrain/MesoInception_DF')\n        for new_layer, layer in zip(model.layers[1:-8], df_model.layers[1:-8]):\n            new_layer.set_weights(layer.get_weights())\n        model.fit([X],[y],epochs=2,callbacks=[LearningRateScheduler(schedule)])\n        pred=model.predict([val_X])\n        loss=log_loss(val_y,pred)\n        losses.append(loss)\n        print('fold '+str(i)+' model loss: '+str(loss))\n        df_models.append(model)\n        K.clear_session()\n        del model\n        gc.collect()\n        i+=1\n    i=0\n    while len(f2f_models)<kfolds:\n        model=define_model((150,150,3))\n        #model.load_weights('../input/meso-pretrain/MesoInception_DF')\n        for new_layer, layer in zip(model.layers[1:-8], f2f_model.layers[1:-8]):\n            new_layer.set_weights(layer.get_weights())\n        model.fit([X],[y],epochs=2,callbacks=[LearningRateScheduler(schedule)])\n        pred=model.predict([val_X])\n        loss=log_loss(val_y,pred)\n        losses.append(loss)\n        print('fold '+str(i)+' model loss: '+str(loss))\n        f2f_models.append(model)\n        K.clear_session()\n        del model\n        gc.collect()\n        i+=1\n        models=f2f_models+df_models\nelse:\n    models=[]\n    i=0\n    while len(models)<kfolds:\n        model=define_model((150,150,3))\n        if i==0:\n            model.summary()\n        model.fit([X],[y],epochs=2,callbacks=[LearningRateScheduler(schedule)])\n        pred=model.predict([val_X])\n        loss=log_loss(val_y,pred)\n        losses.append(loss)\n        print('fold '+str(i)+' model loss: '+str(loss))\n        if loss<0.68:\n            models.append(model)\n        else:\n            print('loss too bad, retrain!')\n        K.clear_session()\n        del model\n        gc.collect()\n        i+=1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prediction_pipline(X,two_times=False):\n    preds=[]\n    for model in tqdm(models):\n        pred=model.predict([X])\n        preds.append(pred)\n    preds=sum(preds)/len(preds)\n    if two_times:\n        return larger_range(preds,2)\n    else:\n        return preds\ndef larger_range(model_pred,time):\n    return (((model_pred-0.5)*time)+0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_pred=prediction_pipline(val_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"random_pred=np.random.random(len(val_X))\nprint('random loss: ' + str(log_loss(val_y,random_pred.clip(0.35,0.65))))\nallone_pred=np.array([1 for _ in range(len(val_X))])\nprint('1 loss: ' + str(log_loss(val_y,allone_pred)))\nallzero_pred=np.array([0 for _ in range(len(val_X))])\nprint('0 loss: ' + str(log_loss(val_y,allzero_pred)))\nallpoint5_pred=np.array([0.5 for _ in range(len(val_X))])\nprint('0.5 loss: ' + str(log_loss(val_y,allpoint5_pred)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Simple Averaging Loss: '+str(log_loss(val_y,model_pred.clip(0.35,0.65))))\nprint('Two Times Larger Range(Averaging) Loss: '+str(log_loss(val_y,larger_range(model_pred,2).clip(0.35,0.65))))\nif log_loss(val_y,model_pred.clip(0.35,0.65))<log_loss(val_y,larger_range(model_pred,2).clip(0.35,0.65)):\n    two_times=False\n    print('simple averaging is better')\nelse:\n    two_times=True\n    print('two times larger range is better')\ntwo_times=False \n#This is not a bug. I did this intentionally because the model can't get most of the private validation set right(based on LB)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import scipy\nprint(model_pred.clip(0.35,0.65).mean())\nprint(scipy.stats.median_absolute_deviation(model_pred.clip(0.35,0.65))[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_answers(pred,real,num):\n    for i,(x,y) in enumerate(zip(pred,real)):\n        correct_incorrect='correct ✅ ' if round(float(x),0)==round(float(y),0) else 'incorrect❌'\n        print(correct_incorrect+' prediction: '+str(x[0])+', answer: '+str(y))\n        if i>num:\n            return\ndef correct_precentile(pred,real):\n    correct=0\n    incorrect=0\n    for x,y in zip(pred,real):\n        if round(float(x),0)==round(float(y),0):\n            correct+=1\n        else:\n            incorrect+=1\n    print('number correct: '+str(correct)+', number incorrect: '+str(incorrect))\n    print(str(round(correct/len(real)*100,1))+'% correct'+', '+str(round(incorrect/len(real)*100,1))+'% incorrect')\ncheck_answers(model_pred,val_y,15)\ncorrect_precentile(model_pred,val_y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"在这之前我们也已经介绍了Resnext和Xception的网络，下面我们会通过对这两个预训练的网络模型进行预测，并通过AUC进行评估"},{"metadata":{},"cell_type":"markdown","source":"# Resnext Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_dir = \"/kaggle/input/deepfake-detection-challenge/train_sample_videos/\"\n\ntest_videos = sorted([x for x in os.listdir(test_dir) if x[-4:] == \".mp4\"])\nlen(test_videos)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"PyTorch version:\", torch.__version__)\nprint(\"CUDA version:\", torch.version.cuda)\nprint(\"cuDNN version:\", torch.backends.cudnn.version())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gpu = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sys.path.insert(0, \"../input/blazefaceepytorch\")\nsys.path.insert(0, \"../input/deepfakesinferencedemo\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from blazeface import BlazeFace\nfacedet = BlazeFace()\nfacedet.load_weights(\"../input/blazefaceepytorch/blazeface.pth\")\nfacedet.load_anchors(\"../input/blazefaceepytorch/anchors.npy\")\n_ = facedet.train(False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from read_video_1 import VideoReader\nfrom face_extract_1 import FaceExtractor\nimport cv2\n\nframes_per_video = 10 #frame_h * frame_l\nvideo_reader = VideoReader()\nvideo_read_fn = lambda x: video_reader.read_frames(x, num_frames=frames_per_video)\nface_extractor = FaceExtractor(video_read_fn, facedet)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_size = 224","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision.transforms import Normalize\n\nmean = [0.485, 0.456, 0.406]\nstd = [0.229, 0.224, 0.225]\nnormalize_transform = Normalize(mean, std)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def isotropically_resize_image(img, size, resample=cv2.INTER_AREA):\n    h, w = img.shape[:2]\n    if w > h:\n        h = h * size // w\n        w = size\n    else:\n        w = w * size // h\n        h = size\n\n    resized = cv2.resize(img, (w, h), interpolation=resample)\n    return resized\n\n\ndef make_square_image(img):\n    h, w = img.shape[:2]\n    size = max(h, w)\n    t = 0\n    b = size - h\n    l = 0\n    r = size - w\n    return cv2.copyMakeBorder(img, t, b, l, r, cv2.BORDER_CONSTANT, value=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torchvision.models as models\n\nclass MyResNeXt(models.resnet.ResNet):\n    def __init__(self, training=True):\n        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,\n                                        layers=[3, 4, 6, 3], \n                                        groups=32, \n                                        width_per_group=4)\n        self.fc = nn.Linear(2048, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"checkpoint = torch.load(\"../input/deepfakesinferencedemo/resnext.pth\")\n\nmodel = MyResNeXt().to(gpu)\nmodel.load_state_dict(checkpoint)\n_ = model.eval()\n\ndel checkpoint","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_on_video(video_path, batch_size):\n    try:\n        # Find the faces for N frames in the video.\n        faces = face_extractor.process_video(video_path)\n\n        # Only look at one face per frame.\n        face_extractor.keep_only_best_face(faces)\n                \n        if len(faces) > 0:\n            # NOTE: When running on the CPU, the batch size must be fixed\n            # or else memory usage will blow up. (Bug in PyTorch?)\n            x = np.zeros((batch_size, input_size, input_size, 3), dtype=np.uint8)\n\n            # If we found any faces, prepare them for the model.\n            n = 0\n            for frame_data in faces:\n                for face in frame_data[\"faces\"]:\n                    # Resize to the model's required input size.\n                    # We keep the aspect ratio intact and add zero\n                    # padding if necessary.                    \n                    resized_face = isotropically_resize_image(face, input_size)\n                    resized_face = make_square_image(resized_face)\n\n                    if n < batch_size:\n                        x[n] = resized_face\n                        n += 1\n                    else:\n                        print(\"WARNING: have %d faces but batch size is %d\" % (n, batch_size))\n                    \n                    # Test time augmentation: horizontal flips.\n                    # TODO: not sure yet if this helps or not\n                    #x[n] = cv2.flip(resized_face, 1)\n                    #n += 1\n\n            if n > 0:\n                x = torch.tensor(x, device=gpu).float()\n\n                # Preprocess the images.\n                x = x.permute((0, 3, 1, 2))\n\n                for i in range(len(x)):\n                    x[i] = normalize_transform(x[i] / 255.)\n\n                # Make a prediction, then take the average.\n                with torch.no_grad():\n                    y_pred = model(x)\n                    y_pred = torch.sigmoid(y_pred.squeeze())\n                    return y_pred[:n].mean().item()\n\n    except Exception as e:\n        print(\"Prediction error on video %s: %s\" % (video_path, str(e)))\n\n    return 0.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from concurrent.futures import ThreadPoolExecutor\n\ndef predict_on_video_set(videos, num_workers):\n    def process_file(i):\n        filename = videos[i]\n        y_pred = predict_on_video(os.path.join(test_dir, filename), batch_size=frames_per_video)\n        return y_pred\n\n    with ThreadPoolExecutor(max_workers=num_workers) as ex:\n        predictions = ex.map(process_file, range(len(videos)))\n\n    return list(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"speed_test = True  # you have to enable this manually","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if speed_test:\n    start_time = time.time()\n    speedtest_videos = test_videos[:5]\n    predictions = predict_on_video_set(speedtest_videos, num_workers=4)\n    elapsed = time.time() - start_time\n    print(\"Elapsed %f sec. Average per video: %f sec.\" % (elapsed, elapsed / len(speedtest_videos)))\n    print(speedtest_videos)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = predict_on_video_set(test_videos, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df_resnext = pd.DataFrame({\"filename\": test_videos, \"label\": predictions})#\nsubmission_df_resnext.to_csv(\"submission_Resnext_10.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When using regression models and machine learning models, all the survey data are numerical values to make it easier to get good results. Because regression and machine learning are both based on mathematical function methods, when categorical data appears in the data set we want to analyze, the data at this time is not ideal because we cannot process them mathematically. For example, when processing Fake or Real data, we replace them with 0 and 1, and then analyze. Because of this situation, we need a way to digitize text to preprocess the training data. In this work, we use LabelEncoder to implement the above conversion process. LabelEncoder is a function in the scikit-learn package. First, we need to create a variable encoder_x for encoding, and then when the program is executed, these Fake and Real category data are converted into values 0 and 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder \nle = LabelEncoder()\ny = le.fit_transform(meta['label'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pr=[]\nfor  item in predictions:\n    item = 1-item\n    pr.append(item)\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#path_xcep = \"./submission_Resnext.csv\"\n\n#obj=pd.read_csv(path_xcep)\ny=y\ny_pre_xcep=pr\n\nimport sklearn.metrics as metrics\n# calculate the fpr and tpr for all thresholds of the classification\nfpr, tpr, threshold = metrics.roc_curve(y, y_pre_xcep)\nroc_auc = metrics.auc(fpr, tpr)\n\n# method I: plt\nimport matplotlib.pyplot as plt\nplt.title('Receiver Operating Characteristic')\nplt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)\nplt.legend(loc = 'lower right')\nplt.plot([0, 1], [0, 1],'r--')\nplt.xlim([0, 1])\nplt.ylim([0, 1])\nplt.ylabel('True Positive Rate')\nplt.xlabel('False Positive Rate')\nplt.savefig('./xception.png')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Xception"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/deepfakexceptiontrainmodel/pytorchcv-0.0.55-py2.py3-none-any.whl --quiet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_size = 224","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def isotropically_resize_image(img, size, resample=cv2.INTER_AREA):\n    h, w = img.shape[:2]\n    if w > h:\n        h = h * size // w\n        w = size\n    else:\n        w = w * size // h\n        h = size\n\n    resized = cv2.resize(img, (w, h), interpolation=resample)\n    return resized\n\n\ndef make_square_image(img):\n    h, w = img.shape[:2]\n    size = max(h, w)\n    t = 0\n    b = size - h\n    l = 0\n    r = size - w\n    return cv2.copyMakeBorder(img, t, b, l, r, cv2.BORDER_CONSTANT, value=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/deepfakexceptiontrainmodel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pytorchcv.model_provider import get_model\nmodel = get_model(\"xception\", pretrained=False)\nmodel = nn.Sequential(*list(model.children())[:-1]) # Remove original output layer\n\nclass Pooling(nn.Module):\n  def __init__(self):\n    super(Pooling, self).__init__()\n    \n    self.p1 = nn.AdaptiveAvgPool2d((1,1))\n    self.p2 = nn.AdaptiveMaxPool2d((1,1))\n\n  def forward(self, x):\n    x1 = self.p1(x)\n    x2 = self.p2(x)\n    return (x1+x2) * 0.5\n\nmodel[0].final_block.pool = nn.Sequential(nn.AdaptiveAvgPool2d((1,1)))\n\nclass Head(torch.nn.Module):\n  def __init__(self, in_f, out_f):\n    super(Head, self).__init__()\n    \n    self.f = nn.Flatten()\n    self.l = nn.Linear(in_f, 512)\n    self.d = nn.Dropout(0.5)\n    self.o = nn.Linear(512, out_f)\n    self.b1 = nn.BatchNorm1d(in_f)\n    self.b2 = nn.BatchNorm1d(512)\n    self.r = nn.ReLU()\n\n  def forward(self, x):\n    x = self.f(x)\n    x = self.b1(x)\n    x = self.d(x)\n\n    x = self.l(x)\n    x = self.r(x)\n    x = self.b2(x)\n    x = self.d(x)\n\n    out = self.o(x)\n    return out\n\nclass FCN(torch.nn.Module):\n  def __init__(self, base, in_f):\n    super(FCN, self).__init__()\n    self.base = base\n    self.h1 = Head(in_f, 1)\n  \n  def forward(self, x):\n    x = self.base(x)\n    return self.h1(x)\n\n\nnet = []\nmodel = FCN(model, 2048)\nmodel = model.cuda()\nmodel.load_state_dict(torch.load('../input/deepfakexceptiontrainmodel/model.pth')) # new, updated\nnet.append(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_on_video(video_path, batch_size):\n    try:\n        # Find the faces for N frames in the video.\n        faces = face_extractor.process_video(video_path)\n\n        # Only look at one face per frame.\n        face_extractor.keep_only_best_face(faces)\n        \n        if len(faces) > 0:\n            # NOTE: When running on the CPU, the batch size must be fixed\n            # or else memory usage will blow up. (Bug in PyTorch?)\n            x = np.zeros((batch_size, input_size, input_size, 3), dtype=np.uint8)\n\n            # If we found any faces, prepare them for the model.\n            n = 0\n            for frame_data in faces:\n                for face in frame_data[\"faces\"]:\n                    # Resize to the model's required input size.\n                    # We keep the aspect ratio intact and add zero\n                    # padding if necessary.                    \n                    resized_face = isotropically_resize_image(face, input_size)\n                    resized_face = make_square_image(resized_face)\n\n                    if n < batch_size:\n                        x[n] = resized_face\n                        n += 1\n                    else:\n                        print(\"WARNING: have %d faces but batch size is %d\" % (n, batch_size))\n                    \n                    # Test time augmentation: horizontal flips.\n                    # TODO: not sure yet if this helps or not\n                    #x[n] = cv2.flip(resized_face, 1)\n                    #n += 1\n\n            if n > 0:\n                x = torch.tensor(x, device=gpu).float()\n\n                # Preprocess the images.\n                x = x.permute((0, 3, 1, 2))\n\n                for i in range(len(x)):\n                    x[i] = normalize_transform(x[i] / 255.)\n#                     x[i] = x[i] / 255.\n\n                # Make a prediction, then take the average.\n                with torch.no_grad():\n                    y_pred = model(x)\n                    y_pred = torch.sigmoid(y_pred.squeeze())\n                    return y_pred[:n].mean().item()\n\n    except Exception as e:\n        print(\"Prediction error on video %s: %s\" % (video_path, str(e)))\n\n    return 0.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from concurrent.futures import ThreadPoolExecutor\n\ndef predict_on_video_set(videos, num_workers):\n    def process_file(i):\n        filename = videos[i]\n        y_pred = predict_on_video(os.path.join(test_dir, filename), batch_size=frames_per_video)\n        return y_pred\n\n    with ThreadPoolExecutor(max_workers=num_workers) as ex:\n        predictions = ex.map(process_file, range(len(videos)))\n\n    return list(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if True:\n    start_time = time.time()\n    speedtest_videos = test_videos[:1]\n    predictions = predict_on_video_set(speedtest_videos, num_workers=4)\n    elapsed = time.time() - start_time\n    print(\"Elapsed %f sec. Average per video: %f sec.\" % (elapsed, elapsed / len(speedtest_videos)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = predict_on_video_set(test_videos, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pr1=[]\nfor  item in predictions:\n    item = 1-item\n    pr1.append(item)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y=y\ny_pre_xcep=pr1\n\n\n\nimport sklearn.metrics as metrics\n# calculate the fpr and tpr for all thresholds of the classification\nfpr, tpr, threshold = metrics.roc_curve(y, y_pre_xcep)\nroc_auc = metrics.auc(fpr, tpr)\n\n# method I: plt\nimport matplotlib.pyplot as plt\nplt.title('Receiver Operating Characteristic')\nplt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)\nplt.legend(loc = 'lower right')\nplt.plot([0, 1], [0, 1],'r--')\nplt.xlim([0, 1])\nplt.ylim([0, 1])\nplt.ylabel('True Positive Rate')\nplt.xlabel('False Positive Rate')\nplt.savefig('./xception1.png')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"以上三种算法就是本次工作中用到的算法，我们使用AUC-ROC（Area Under Curve）来衡量机器学习模型质量的指标。\nAUC-ROC是机器学习中常用的一个分类器评价指标，其规则用一句话概括就是：一个正例，一个负例，预测为正的概率值比预测为负的概率值还要大的可能性。我们通过绘制ROC曲线来体现我们的工作效果，ROC曲线下面的面积就是AUC的值，具体图示会在下一部分详细给出。\n对于上面提到的模型的训练，我们使用了python中的numpy、pandas、matplotlib.pyplot、glob、cv2、torch、tensorflow等package。\n"},{"metadata":{},"cell_type":"markdown","source":"# 6.\tConclusion and Future Works"},{"metadata":{},"cell_type":"markdown","source":"In the whole work, we first tried to segment the video data set and divide the data set into pictures. Then we used 4 face detection methods and compared its running time. Finally, we chose the shortest use time FaceBlaze was used to extract faces, and then we used a ResNet model similar to CNN for training and verification, but our method did not perform well. We changed the model, chose the pre-trained Rest and Xception, and we got the best results on Rest. In the next work, we will try to do some work similar to the DeepfakeStack deep integration learning algorithm to achieve better recognition results and find the optimal integration strategy."},{"metadata":{},"cell_type":"markdown","source":"# 7.\tBibliography"},{"metadata":{},"cell_type":"markdown","source":"[1] N. S. Ivanov, A. V. Arzhskov and V. G. Ivanenko, \"Combining Deep Learning and Super-Resolution Algorithms for Deep Fake Detection,\" 2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), St. Petersburg and Moscow, Russia, 2020, pp. 326-328, doi: 10.1109/EIConRus49466.2020.9039498.\n[2] D. Afchar, V. Nozick, J. Yamagishi and I. Echizen, \"MesoNet: a Compact Facial Video Forgery Detection Network,\" 2018 IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, Hong Kong, 2018, pp. 1-7, doi: 10.1109/WIFS.2018.8630761.\n[3] H. M. Nguyen and R. Derakhshani, \"Eyebrow Recognition for Identifying Deepfake Videos,\" 2020 International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, 2020, pp. 1-5.\n[4] B. Malolan, A. Parekh and F. Kazi, \"Explainable Deep-Fake Detection Using Visual Interpretability Methods,\" 2020 3rd International Conference on Information and Computer Technologies (ICICT), San Jose, CA, USA, 2020, pp. 289-293, doi: 10.1109/ICICT50521.2020.00051.\n[5] M. S. Rana and A. H. Sung, \"DeepfakeStack: A Deep Ensemble-based Learning Technique for Deepfake Detection,\" 2020 7th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2020 6th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom), New York, NY, USA, 2020, pp. 70-75, doi: 10.1109/CSCloud-EdgeCom49738.2020.00021.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"好　、　我们的工作就到这里","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}