{"cells":[{"metadata":{},"cell_type":"markdown","source":"# DeepFake Introductory EDA"},{"metadata":{},"cell_type":"markdown","source":"## Disclaimer\nThis code / eda is **originally** not mine, this is from the user \"aleksandradeis\", you can visit her in https://www.kaggle.com/aleksandradeis . I only modified a part of the code."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import file utilities\nimport os\nimport glob\n\n# import charting\nimport matplotlib.pyplot as plt\nfrom matplotlib.animation import FuncAnimation, ArtistAnimation \n%matplotlib inline\n\nfrom IPython.display import HTML\n\n# import computer vision\nimport cv2\nfrom skimage.measure import compare_ssim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_PATH = '../input/deepfake-detection-challenge/test_videos/'\nTRAIN_PATH = '../input/deepfake-detection-challenge/train_sample_videos/'\n\nmetadata = '../input/deepfake-detection-challenge/train_sample_videos/metadata.json'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the filenames for train videos\ntrain_fns = sorted(glob.glob(TRAIN_PATH + '*.mp4'))\n\n# load the filenames for test videos\ntest_fns = sorted(glob.glob(TEST_PATH + '*.mp4'))\n\nprint('There are {} samples in the train set.'.format(len(train_fns)))\nprint('There are {} samples in the test set.'.format(len(test_fns)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta = pd.read_json(metadata).transpose()\nmeta.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Analyze the number or fake and real samples:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Pie chart, where the slices will be ordered and plotted counter-clockwise:\nlabels = 'FAKE', 'REAL'\nsizes = [meta[meta.label == 'FAKE'].label.count(), meta[meta.label == 'REAL'].label.count()]\n\nfig1, ax1 = plt.subplots(figsize=(10,7))\nax1.pie(sizes, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90, colors=['#f4d53f', '#02a1d8'])\nax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\nplt.title('Labels', fontsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frame(filename):\n    '''\n    Helper function to return the 1st frame of the video by filename\n    INPUT: \n        filename - the filename of the video\n    OUTPUT:\n        image - 1st frame of the video (RGB)\n    '''\n    # Aquí se retienen frames y se reproduce el video del archivo\n    cap = cv2.VideoCapture(filename)\n    ret, frame = cap.read()\n\n    # Our operations on the frame come here\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    \n    # When everything done, release the capture\n    cap.release()\n    cv2.destroyAllWindows()\n    \n    return image\n\ndef get_label(filename, meta):\n    '''\n    Helper function to get a label from the filepath.\n    INPUT:\n        filename - filename of the video\n        meta - dataframe containing metadata.json\n    OUTPUT:\n        label - label of the video 'FAKE' or 'REAL'\n    '''\n    video_id = filename.split('/')[-1]\n    return meta.loc[video_id].label\n\ndef get_original_filename(filename, meta):\n    '''\n    Helper function to get the filename of the original image\n    INPUT:\n        filename - filename of the video\n        meta - dataframe containing metadata.json\n    OUTPUT:\n        original_filename - name of the original video\n    '''\n    video_id = filename.split('/')[-1]\n    original_id = meta.loc[video_id].original\n    \n    return original_id\n\ndef visualize_frame(filename, meta, train = True):\n    '''\n    Helper function to visualize the 1st frame of the video by filename and metadata\n    INPUT:\n        filename - video filename\n        meta - dataframe containing metadata.json\n        train - indicates that the video is among train samples and the label can be retrived from metadata\n    '''\n    # get the 1st frame of the video\n    image = get_frame(filename)\n\n    # Display the 1st frame of the video\n    fig, axs = plt.subplots(1,3, figsize=(20,7))\n    axs[0].imshow(image) \n    axs[0].axis('off')\n    axs[0].set_title('Original frame')\n    \n    # Extract the face with haar cascades\n    face_cascade = cv2.CascadeClassifier('../input/haarcascades/haarcascade_frontalface_default.xml')\n\n    # run the detector\n    # the output here is an array of detections; the corners of each detection box\n    # if necessary, modify these parameters until you successfully identify every face in a given image\n    faces = face_cascade.detectMultiScale(image, 1.2, 3)\n\n    # make a copy of the original image to plot detections on\n    image_with_detections = image.copy()\n\n    # loop over the detected faces, mark the image where each face is found\n    for (x,y,w,h) in faces:\n        # draw a rectangle around each detected face\n        # you may also need to change the width of the rectangle drawn depending on image resolution\n        cv2.rectangle(image_with_detections,(x,y),(x+w,y+h),(255,0,0),3)\n\n    axs[1].imshow(image_with_detections)\n    axs[1].axis('off')\n    axs[1].set_title('Highlight faces')\n    \n    # crop out the 1st face\n    crop_img = image.copy()\n    for (x,y,w,h) in faces:\n        crop_img = image[y:y+h, x:x+w]\n        break;\n        \n    # plot the 1st face\n    axs[2].imshow(crop_img)\n    axs[2].axis('off')\n    axs[2].set_title('Zoom-in face')\n    \n    if train:\n        plt.suptitle('Image {image} label: {label}'.format(image = filename.split('/')[-1], label=get_label(filename, meta)))\n    else:\n        plt.suptitle('Image {image}'.format(image = filename.split('/')[-1]))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[0], meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On this video the nose of the person is strange."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame(train_fns[4], meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The face of this person is so blurry."},{"metadata":{},"cell_type":"markdown","source":"Let's also look at a couple of real images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame('../input/deepfake-detection-challenge/train_sample_videos/afoovlsmtx.mp4', meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that real faces have such details as:\n* actual teeth (not just one white blob);\n* glasses with reflections."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame('../input/deepfake-detection-challenge/train_sample_videos/agrmhtjdlk.mp4', meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This one will be really hard to predict! To my mind, the videos like these are garbage and should be removed from the training set."},{"metadata":{},"cell_type":"markdown","source":"These fakes are really nice! Only small details tell that those are not real."},{"metadata":{},"cell_type":"markdown","source":"## Preview Multiple Frames"},{"metadata":{},"cell_type":"markdown","source":"Let's look at multiple frames:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frames(filename):\n    '''\n    Get all frames from the video\n    INPUT:\n        filename - video filename\n    OUTPUT:\n        frames - the array of video frames\n    '''\n    frames = []\n    cap = cv2.VideoCapture(filename)\n\n    while(cap.isOpened()):\n        ret, frame = cap.read()\n                \n        if not ret:\n            break;\n            \n        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n        frames.append(image)\n\n    cap.release()\n    cv2.destroyAllWindows()\n    return frames\n\ndef create_animation(filename):\n    '''\n    Function to plot the animation with matplotlib\n    INPUT:\n        filename - filename of the video\n    '''\n    fig = plt.figure(figsize=(10,7))\n    frames = get_frames(filename)\n\n    ims = []\n    for frame in frames:\n        im = plt.imshow(frame, animated=True)\n        ims.append([im])\n\n    animation = ArtistAnimation(fig, ims, interval=30, repeat_delay=1000)\n    plt.show()\n    return animation\n\ndef visualize_several_frames(frames, step=100, cols = 3, title=''):\n    '''\n    Function to visualize the frames from the video\n    INPUT:\n        filename - filename of the video\n        step - the step between the video frames to visualize\n        cols - number of columns of frame grid\n    '''\n    n_frames = len(range(0, len(frames), step))\n    rows = n_frames // cols\n    if n_frames % cols > 0:\n        rows = rows + 1\n    \n    fig, axs = plt.subplots(rows, cols, figsize=(20,20))\n    for i in range(0, n_frames):\n        frame = frames[i]\n        \n        r = i // cols\n        c = i % cols\n        \n        axs[r,c].imshow(frame)\n        axs[r,c].axis('off')\n        axs[r,c].set_title(str(i))\n        \n    plt.suptitle(title)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"frames = get_frames(train_fns[0])\nvisualize_several_frames(frames, step=50, cols = 2, title=train_fns[0].split('/')[-1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Static images don't look so bad, but if we look at the video (use the code for animation: `create_animation` function above) we see a lot of artifacts, which tell us that the video is fake."},{"metadata":{},"cell_type":"markdown","source":"Now let's look closer at the person's face in motion:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frames_zoomed(filename):\n    '''\n    Get all frames from the video zoomed into the face\n    INPUT:\n        filename - video filename\n    OUTPUT:\n        frames - the array of video frames\n    '''\n    frames = []\n    cap = cv2.VideoCapture(filename)\n    \n    face_cascade = cv2.CascadeClassifier('../input/haarcascades/haarcascade_frontalface_default.xml')\n\n    while(cap.isOpened()):\n        ret, frame = cap.read()\n                \n        if not ret:\n            break;\n            \n        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n        \n        faces = face_cascade.detectMultiScale(image, 1.2, 3)\n        image_with_detections = image.copy()\n\n        crop_img = image.copy()\n        for (x,y,w,h) in faces:\n            crop_img = image[y:y+h, x:x+w]\n            break;\n        \n        frames.append(crop_img)\n\n    cap.release()\n    cv2.destroyAllWindows()\n    return frames\n\ndef create_animation_zoomed(filename):\n    '''\n    Function to create the animated cropped faces out of the video\n    INPUT:\n        filename - filename of the video\n    '''\n    fig, ax = plt.subplots(1,1, figsize=(10,7))\n    frames = get_frames_zoomed(filename)\n\n    def update(frame_number):\n        plt.axis('off')\n        plt.imshow(frames[frame_number])\n\n    animation = FuncAnimation(fig, update, interval=30, repeat=True)\n    return animation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"animation = create_animation_zoomed(train_fns[0])\nHTML(animation.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We clearly see that the video is fake looking closer at the face! Some frames are really creepy. And there is flickering."},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize the zoomed in frames\nframes_face = get_frames_zoomed(train_fns[0])\nvisualize_several_frames(frames_face, step=55, cols = 2, title=train_fns[0].split('/')[-1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Individual frames don't look too bad. This means that we have to build models using maximum frames, we can't just sample some frames. But we can use only frames containing faces to train the model."},{"metadata":{},"cell_type":"markdown","source":"## Explore the Similarity between Frames"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_similarity_scores(frames):\n    '''\n    Get the list of similarity scores between the frames.\n    '''\n    scores = []\n    for i in range(1, len(frames)):\n        frame = frames[i]\n        prev_frame = frames[i-1]\n        \n        if frame.shape[0] != prev_frame.shape[0]:\n            if  frame.shape[0] > prev_frame.shape[0]:\n                frame = frame[:prev_frame.shape[0], :prev_frame.shape[0], :]\n            else:\n                prev_frame = prev_frame[:frame.shape[0], :frame.shape[0], :]\n        \n        (score, diff) = compare_ssim(frame, prev_frame, full=True, multichannel=True)\n        scores.append(score)\n    return scores\n\ndef plot_scores(scores):\n    '''\n    Plot the similarity scores\n    '''\n    plt.figure(figsize=(12,7))\n    plt.plot(scores)\n    plt.title('Similarity Scores')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = get_similarity_scores(frames)\nplot_scores(scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are some similarity drops, let's try to look at the frames in this area:"},{"metadata":{"trusted":true},"cell_type":"code","source":"max_dist = np.argmax(scores[1:50])\nmax_dist\nplt.imshow(frames_face[max_dist])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(frames_face[max_dist+5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's compare similarity score with the original video (it is not among samples, I uploaded it in separate dataset):"},{"metadata":{},"cell_type":"markdown","source":"Open video and look at the first frame:"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frame('../input/deepfake-utils/vudstovrck.mp4', meta, train = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The difference between real and fake is quite clear. Just look at the nose."},{"metadata":{},"cell_type":"markdown","source":"Let's get the frames and plot the similarity scores:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get frames from the original video\norig_frames = get_frames('../input/deepfake-utils/vudstovrck.mp4')\n# plot similarity scores\norig_scores = get_similarity_scores(orig_frames)\nplot_scores(orig_scores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot similarity scores together:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nplt.plot(scores, label = 'fake image', color='g')\nplt.plot(orig_scores, label = 'real image', color='orange')\nplt.title('Similarity Scores (Real and Fake)')\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}