{"cells":[{"metadata":{},"cell_type":"markdown","source":"# DeepFake Introductory EDA"},{"metadata":{},"cell_type":"markdown","source":"## Introduction\nToday's technology allows us to do the incredible things such as creation of fake videos or images of real people, [deepfakes](https://en.wikipedia.org/wiki/Deepfake). Deepfakes [are going viral](https://www.creativebloq.com/features/deepfake-examples) and creating a lot of credibility and security concerns. That is why deepfake detection is a fast growing area of research (I put some of the papers related to deepfakes in the end of the notebook).\n\nIn this analysis I will try to look close at the videos from the sample dataset on Kaggle and find traits which can help us distinguish the fakes from the real videos."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import file utilities\nimport os\nimport glob\n\n# import charting\nimport matplotlib.pyplot as plt\nfrom matplotlib.animation import FuncAnimation, ArtistAnimation \n%matplotlib inline\n\nfrom IPython.display import HTML\n\n# import computer vision\nimport cv2\nfrom skimage.measure import compare_ssim","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data"},{"metadata":{},"cell_type":"markdown","source":"First of all, we need to declare the paths to train and test samples and metadata file:"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_PATH = '../input/deepfake-detection-challenge/test_videos/'\nTRAIN_PATH = '../input/deepfake-detection-challenge/train_sample_videos/'\n\nmetadata = '../input/deepfake-detection-challenge/train_sample_videos/metadata.json'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Look at the number of samples in test and train sets:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the filenames for train videos\ntrain_fns = sorted(glob.glob(TRAIN_PATH + '*.mp4'))\n\n# load the filenames for test videos\ntest_fns = sorted(glob.glob(TEST_PATH + '*.mp4'))\n\nprint('There are {} samples in the train set.'.format(len(train_fns)))\nprint('There are {} samples in the test set.'.format(len(test_fns)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And load the metadata:"},{"metadata":{"trusted":true},"cell_type":"code","source":"meta = pd.read_json(metadata).transpose()\nmeta.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the metadata we have a reference to the original video, but those videos can't be found among the samples on Kaggle.\n\nYou can find the original videos if you download the whole dataset."},{"metadata":{},"cell_type":"markdown","source":"Analyze the number or fake and real samples:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Pie chart, where the slices will be ordered and plotted counter-clockwise:\nlabels = 'FAKE', 'REAL'\nsizes = [meta[meta.label == 'FAKE'].label.count(), meta[meta.label == 'REAL'].label.count()]\n\nfig1, ax1 = plt.subplots(figsize=(10,7))\nax1.pie(sizes, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90, colors=['G', 'B'])\nax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\nplt.title('Labels', fontsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__Only 19% of samples are real videos.__ I don't know if this is the same for the whole dataset."},{"metadata":{},"cell_type":"markdown","source":"## Preview Videos and Zoom into Faces"},{"metadata":{},"cell_type":"markdown","source":"Let's start with looking at some frames of the videos and trying to look closer at the faces.\nI used [HAAR cascades](https://docs.opencv.org/trunk/db/d28/tutorial_cascade_classifier.html) to detect the areas containing the faces on the image."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frame(filename):\n    '''\n    Helper function to return the 1st frame of the video by filename\n    INPUT: \n        filename - the filename of the video\n    OUTPUT:\n        image - 1st frame of the video (RGB)\n    '''\n    # Playing video from file\n    cap = cv2.VideoCapture(filename)\n    ret, frame = cap.read()\n\n    # convert into RGB\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    \n    # When everything done, release the capture\n    cap.release()\n    cv2.destroyAllWindows()\n    \n    return image\n\ndef get_label(filename, meta):\n    '''\n    Helper function to get a label from the filepath.\n    INPUT:\n        filename - filename of the video\n        meta - dataframe containing metadata.json\n    OUTPUT:\n        label - label of the video 'FAKE' or 'REAL'\n    '''\n    video_id = filename.split('/')[-1]\n    return meta.loc[video_id].label\n\ndef get_original_filename(filename, meta):\n    '''\n    Helper function to get the filename of the original image\n    INPUT:\n        filename - filename of the video\n        meta - dataframe containing metadata.json\n    OUTPUT:\n        original_filename - name of the original video\n    '''\n    video_id = filename.split('/')[-2]\n    original_id = meta.loc[video_id].original\n    \n    return original_id\n\ndef visualize_frame(filename, meta, train = True):\n    '''\n    Helper function to visualize the 1st frame of the video by filename and metadata\n    INPUT:\n        filename - video filename\n        meta - dataframe containing metadata.json\n        train - indicates that the video is among train samples and the label can be retrived from metadata\n    '''\n    # get the 1st frame of the video\n    image = get_frame(filename)\n\n    # Display the 1st frame of the video\n    fig, axs = plt.subplots(1,3, figsize=(20,7))\n    axs[0].imshow(image) \n    axs[0].axis('off')\n    axs[0].set_title('Original frame')\n    \n    # Extract the face with haar cascades\n    face_cascade = cv2.CascadeClassifier('../input/haarcascades/haarcascade_frontalface_default.xml')\n\n    # run the detector\n    # the output here is an array of detections; the corners of each detection box\n    # if necessary, modify these parameters until you successfully identify every face in a given image\n    faces = face_cascade.detectMultiScale(image, 1.2, 3)\n\n    # make a copy of the original image to plot detections on\n    image_with_detections = image.copy()\n\n    # loop over the detected faces, mark the image where each face is found\n    for (x,y,w,h) in faces:\n        # draw a rectangle around each detected face\n        # you may also need to change the width of the rectangle drawn depending on image resolution\n        cv2.rectangle(image_with_detections,(x,y),(x+w,y+h),(250,0,0),3)\n\n    axs[1].imshow(image_with_detections)\n    axs[1].axis('off')\n    axs[1].set_title('Highlight faces')\n    \n    # crop out the 1st face\n    crop_img = image.copy()\n    for (x,y,w,h) in faces:\n        crop_img = image[y:y+h, x:x+w]\n        break;\n        \n    # plot the 1st face\n    axs[2].imshow(crop_img)\n    axs[2].axis('off')\n    axs[2].set_title('Zoom-in face')\n    \n    if train:\n        plt.suptitle('Image {image} label: {label}'.format(image = filename.split('/')[-1], label=get_label(filename, meta)))\n    else:\n        plt.suptitle('Image {image}'.format(image = filename.split('/')[-1]))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[0], meta)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[9], meta)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On this video the nose of the person is strange."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[4], meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The glasses of this don't look very realistic. There is also a strange rounded shape around the right eye of the lady. Strange white spot to the right of the mouth."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[8], meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The face of this person is so blurry."},{"metadata":{},"cell_type":"markdown","source":"Let's also look at a couple of real images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame('../input/deepfake-detection-challenge/train_sample_videos/afoovlsmtx.mp4', meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that real faces have such details as:\n* actual teeth (not just one white blob);\n* glasses with reflections."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame('../input/deepfake-detection-challenge/train_sample_videos/agrmhtjdlk.mp4', meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This one will be really hard to predict! To my mind, the videos like these are garbage and should be removed from the training set."},{"metadata":{},"cell_type":"markdown","source":"These fakes are really nice! Only small details tell that those are not real."},{"metadata":{},"cell_type":"markdown","source":"## Preview Multiple Frames"},{"metadata":{},"cell_type":"markdown","source":"Let's look at multiple frames:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frames(filename):\n    '''\n    Get all frames from the video\n    INPUT:\n        filename - video filename\n    OUTPUT:\n        frames - the array of video frames\n    '''\n    frames = []\n    cap = cv2.VideoCapture(filename)\n\n    while(cap.isOpened()):\n        ret, frame = cap.read()\n                \n        if not ret:\n            break;\n            \n        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n        frames.append(image)\n\n    cap.release()\n    cv2.destroyAllWindows()\n    return frames\n\ndef create_animation(filename):\n    '''\n    Function to plot the animation with matplotlib\n    INPUT:\n        filename - filename of the video\n    '''\n    fig = plt.figure(figsize=(10,7))\n    frames = get_frames(filename)\n\n    ims = []\n    for frame in frames:\n        im = plt.imshow(frame, animated=True)\n        ims.append([im])\n\n    animation = ArtistAnimation(fig, ims, interval=30, repeat_delay=1000)\n    plt.show()\n    return animation\n\ndef visualize_several_frames(frames, step=100, cols = 3, title=''):\n    '''\n    Function to visualize the frames from the video\n    INPUT:\n        filename - filename of the video\n        step - the step between the video frames to visualize\n        cols - number of columns of frame grid\n    '''\n    n_frames = len(range(0, len(frames), step))\n    rows = n_frames // cols\n    if n_frames % cols > 0:\n        rows = rows + 1\n    \n    fig, axs = plt.subplots(rows, cols, figsize=(20,20))\n    for i in range(0, n_frames):\n        frame = frames[i]\n        \n        r = i // cols\n        c = i % cols\n        \n        axs[r,c].imshow(frame)\n        axs[r,c].axis('off')\n        axs[r,c].set_title(str(i))\n        \n    plt.suptitle(title)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"frames = get_frames(train_fns[0])\nvisualize_several_frames(frames, step=50, cols = 2, title=train_fns[0].split('/')[-1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Static images don't look so bad, but if we look at the video (use the code for animation: `create_animation` function above) we see a lot of artifacts, which tell us that the video is fake."},{"metadata":{},"cell_type":"markdown","source":"Now let's look closer at the person's face in motion:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frames_zoomed(filename):\n    '''\n    Get all frames from the video zoomed into the face\n    INPUT:\n        filename - video filename\n    OUTPUT:\n        frames - the array of video frames\n    '''\n    frames = []\n    cap = cv2.VideoCapture(filename)\n    \n    face_cascade = cv2.CascadeClassifier('../input/haarcascades/haarcascade_frontalface_default.xml')\n\n    while(cap.isOpened()):\n        ret, frame = cap.read()\n                \n        if not ret:\n            break;\n            \n        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n        \n        faces = face_cascade.detectMultiScale(image, 1.2, 3)\n        image_with_detections = image.copy()\n\n        crop_img = image.copy()\n        for (x,y,w,h) in faces:\n            crop_img = image[y:y+h, x:x+w]\n            break;\n        \n        frames.append(crop_img)\n\n    cap.release()\n    cv2.destroyAllWindows()\n    return frames\n\ndef create_animation_zoomed(filename):\n    '''\n    Function to create the animated cropped faces out of the video\n    INPUT:\n        filename - filename of the video\n    '''\n    fig, ax = plt.subplots(1,1, figsize=(10,7))\n    frames = get_frames_zoomed(filename)\n\n    def update(frame_number):\n        plt.axis('off')\n        plt.imshow(frames[frame_number])\n\n    animation = FuncAnimation(fig, update, interval=30, repeat=True)\n    return animation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"animation = create_animation_zoomed(train_fns[0])\nHTML(animation.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We clearly see that the video is fake looking closer at the face! Some frames are really creepy. And there is flickering."},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize the zoomed in frames\nframes_face = get_frames_zoomed(train_fns[0])\nvisualize_several_frames(frames_face, step=55, cols = 2, title=train_fns[0].split('/')[-1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Individual frames don't look too bad. This means that we have to build models using maximum frames, we can't just sample some frames. But we can use only frames containing faces to train the model."},{"metadata":{},"cell_type":"markdown","source":"## Explore the Similarity between Frames"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_similarity_scores(frames):\n    '''\n    Get the list of similarity scores between the frames.\n    '''\n    scores = []\n    for i in range(1, len(frames)):\n        frame = frames[i]\n        prev_frame = frames[i-1]\n        \n        if frame.shape[0] != prev_frame.shape[0]:\n            if  frame.shape[0] > prev_frame.shape[0]:\n                frame = frame[:prev_frame.shape[0], :prev_frame.shape[0], :]\n            else:\n                prev_frame = prev_frame[:frame.shape[0], :frame.shape[0], :]\n        \n        (score, diff) = compare_ssim(frame, prev_frame, full=True, multichannel=True)\n        scores.append(score)\n    return scores\n\ndef plot_scores(scores):\n    '''\n    Plot the similarity scores\n    '''\n    plt.figure(figsize=(12,7))\n    plt.plot(scores)\n    plt.title('Similarity Scores')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = get_similarity_scores(frames)\nplot_scores(scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are some similarity drops, let's try to look at the frames in this area:"},{"metadata":{"trusted":true},"cell_type":"code","source":"max_dist = np.argmax(scores[1:50])\nmax_dist\nplt.imshow(frames_face[max_dist])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(frames_face[max_dist+5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's compare similarity score with the original video (it is not among samples, I uploaded it in separate dataset):"},{"metadata":{},"cell_type":"markdown","source":"Open video and look at the first frame:"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame('../input/deepfake-utils/vudstovrck.mp4', meta, train = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The difference between real and fake is quite clear. Just look at the nose."},{"metadata":{},"cell_type":"markdown","source":"Let's get the frames and plot the similarity scores:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get frames from the original video\norig_frames = get_frames('../input/deepfake-utils/vudstovrck.mp4')\n# plot similarity scores\norig_scores = get_similarity_scores(orig_frames)\nplot_scores(orig_scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot similarity scores together:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nplt.plot(scores, label = 'fake image', color='g')\nplt.plot(orig_scores, label = 'real image', color='orange')\nplt.title('Similarity Scores (Real and Fake)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The similarity scores of the original image are almost identical __if we take the whole frames__. Image similarity could still work if taking only frames containing faces. But I should fix the face detection first."},{"metadata":{},"cell_type":"markdown","source":"## Improvement\n\n1. Haar cascades don't seem to be very convenient tool. Setting up the parameters to match various images seems to be nearly manual. Consider using deep learning techniques, such as [MTCNN](https://github.com/ipazc/mtcnn) to detect faces."},{"metadata":{},"cell_type":"markdown","source":"## Deepfake Research Papers"},{"metadata":{},"cell_type":"markdown","source":"1. [Unmasking DeepFakes with simple Features](https://arxiv.org/pdf/1911.00686v2.pdf): The method is based on a classical frequency domain analysis\nfollowed by a basic classifier. Compared to previous systems, which need to be fed with large amounts of labeled data, this\napproach showed very good results using only a few annotated training samples and even achieved good accuracies in fully\nunsupervised scenarios. [Github repo](https://github.com/cc-hpc-itwm/DeepFakeDetection)\n\n2. [FaceForensics++: Learning to Detect Manipulated Facial Images](https://arxiv.org/pdf/1901.08971v3.pdf): This paper\nexamines the realism of state-of- the-art image manipulations, and how difficult it is to detect them, either automatically\nor by humans. [Github repo](https://github.com/ondyari/FaceForensics)\n\n3. [In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking](https://arxiv.org/pdf/1806.02877v2.pdf): Method is based on detection of eye blinking in the videos,\nwhich is a physiological signal that is not well presented in the synthesized fake videos. Method is tested over\nbenchmarks of eye-blinking detection datasets and also show promising performance on detecting videos generated with DeepFake.\n[Github repo](https://github.com/danmohaha/WIFS2018_In_Ictu_Oculi)\n\n4. [USE OF A CAPSULE NETWORK TO DETECT FAKE IMAGES AND VIDEOS](https://arxiv.org/pdf/1910.12467v2.pdf): \"Capsule-Forensics\"\nmethod to detect fake images and videos. [Github repo](https://github.com/nii-yamagishilab/Capsule-Forensics-v2)\n\n5. [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://arxiv.org/pdf/1811.00656v3.pdf): Deep learning based\nmethod that can effectively distinguish AI-generated fake videos (referred to as DeepFake videos hereafter) from real videos.\nMethod is based on the observations that current DeepFake algorithm can only generate images of limited resolutions, which\nneed to be further warped to match the original faces in the source video. Such transforms leave distinctive artifacts in\nthe resulting DeepFake videos, and we show that they can be effectively captured by convolutional neural networks (CNNs).\n[Github repo](https://github.com/danmohaha/CVPRW2019_Face_Artifacts)\n\n6. [Limits of Deepfake Detection: A Robust Estimation Viewpoint](https://arxiv.org/pdf/1905.03493v1.pdf): This work gives a\ngeneralizable statistical framework with guarantees on its reliability. In particular, we build on the information-theoretic\nstudy of authentication to cast deepfake detection as a hypothesis testing problem specifically for outputs of GANs,\nthemselves viewed through a generalized robust statistics framework."},{"metadata":{},"cell_type":"markdown","source":"## Conclusion"},{"metadata":{},"cell_type":"markdown","source":"In this notebook:\n* I loaded saparate frames of the videos and sequences of video frames.\n* I created some animation of fake videos.\n* I used Haar cascades for face detection and zoomed into real and fake faces.\n* I looked at the similarity between frames.\n\nFake videos can be detected by:\n* Small missing details (nose, glasses, teeth),\n* Blurry contours of the face,\n* Flickering.\n\n__This notebook is refered by other kernals __"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}