{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\npd.set_option('display.max_colwidth', -1)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport json\nfrom collections import defaultdict\nimport cv2\nimport matplotlib.pyplot as plt\nfrom matplotlib.animation import FuncAnimation\nfrom IPython.display import HTML\n%matplotlib inline\n!pip install /kaggle/input/mtcnn-package/mtcnn-0.1.0-py3-none-any.whl;\nfrom mtcnn import MTCNN\nfrom skimage.metrics import structural_similarity","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train/Test Split Ratio in Public Validation Set"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_files, test_files = [], []\ntrain_exts, test_exts = defaultdict(int), defaultdict(int)\n\nfor dirname, _, filenames in os.walk('/kaggle/input/deepfake-detection-challenge/'):\n    for filename in filenames:\n        if ('train' in dirname):\n            train_exts[filename.split('.')[1]] += 1\n            train_files.append(os.path.join(dirname, filename))\n        elif ('test' in dirname):\n            test_exts[filename.split('.')[1]] += 1\n            test_files.append(os.path.join(dirname, filename))\n\nprint(f'we have {train_exts[\"mp4\"]} training samples: {train_exts.items()}')\nprint(f'we have {test_exts[\"mp4\"]} testing samples: {test_exts.items()}')\n\ntrain_files = sorted(train_files)\ntest_files = sorted(test_files)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"metadata_df = pd.read_json('/kaggle/input/deepfake-detection-challenge/train_sample_videos/metadata.json').T\ndisplay(metadata_df.head())\n# Pie chart, where the slices will be ordered and plotted counter-clockwise:\nlabels = [f'{k} ({metadata_df[\"label\"].value_counts()[k]} samples)' for k in metadata_df['label'].value_counts().keys()]\nsizes = dict(metadata_df['label'].value_counts()).values()\nexplode = (0.1, 0)\n\nfig1, ax1 = plt.subplots()\nax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90)\nax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'metadata.json has the ground truth for all the {metadata_df.shape[0]} training video samples.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check if we have corresponding original videos of fakes"},{"metadata":{},"cell_type":"markdown","source":"for all fake videos,```metadata.json``` has a reference to its corresponding original video.  \nLets check if we have those original videos with us(included in the public validation set)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"reference_to_original_list=[item for item in metadata_df['original'].tolist() if item]\nreference_to_original_set=set(reference_to_original_list)\ntrain_videos_list=set(metadata_df.index)\nintersect_list=list(reference_to_original_set.intersection(train_videos_list))\nprint(f'train_videos_list:{len(train_videos_list)}, reference_to_original_list:{len(reference_to_original_list)}, reference_to_original_set:{len(reference_to_original_set)}, intersect_list:{len(intersect_list)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, ```train_videos_list``` is the list of all the videos in public validation set,  \n```reference_to_original_set``` is a list of all the unique original videos used to create the fake ones,  \nand ```intersect_list``` is a list of videos found in ```reference_to_original_set``` which exists in ```train_videos_list```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"orig_to_fake = metadata_df[metadata_df[\"original\"].isin(intersect_list)].reset_index().set_index('original').rename(columns={'index':'fakes'})[\"fakes\"]\ndisplay(orig_to_fake.groupby('original').apply(list).to_frame().head())\ndisplay(orig_to_fake.groupby('original').apply(list).apply(lambda x : len(x)).to_frame().rename(columns={'fakes':'length(fakes)'}).head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"lets try to see the original video and their fake counterparts side by side maybe also zooming in on the faces as done in [this notebook](https://www.kaggle.com/aleksandradeis/deepfake-challenge-eda/notebook) by [aleksandradeis](https://www.kaggle.com/aleksandradeis)"},{"metadata":{},"cell_type":"markdown","source":"## Explore the Differences/Similaries b/w fake and real videos"},{"metadata":{},"cell_type":"markdown","source":"### Understanding Haar CascadeClassifier (Viola Jones Face detection Algorithm)  \n\nIts better to know firsthand how the face detection works in general since it will be a major part of our overall workflow leading to the detection of deepfakes.  \nI recommend reading/watching below contents mainly because its absolute beauty IMHO.  \n[Face Detection using Haar Cascades](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html)  \n[Detecting Faces (Viola Jones Algorithm) - Computerphile](https://www.youtube.com/watch?v=uEJ71VlUmMQ)   \n[Viola Jones face detection and tracking explained](https://www.youtube.com/watch?v=WfdYYNamHZ8)  "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_label(filename):\n    return metadata_df.loc[filename.split('/')[-1]].label\n\ndef get_videoid(filename):\n    return filename.split('/')[-1]\n\ndef get_first_frame(filename):\n    fig,ax = plt.subplots(1,3,figsize=(20,7))\n    \n    cap = cv2.VideoCapture(filename)\n    ret,frame = cap.read()\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    cap.release()\n    cv2.destroyAllWindows()\n    ax[0].axis('off')\n    ax[0].set_title(f'{get_videoid(filename)} - {get_label(filename)}')\n    ax[0].imshow(image)\n    \n    hcascade = cv2.CascadeClassifier('/kaggle/input/haarcascades/haarcascade_frontalface_default.xml')\n    face = hcascade.detectMultiScale(image,1.2,3)\n    \n    img_copy = image.copy()\n    for (x,y,w,h) in face:\n        cv2.rectangle(img_copy,(x,y),(x+w,y+h),(0,255,0),3)\n        break;\n    \n    ax[1].axis('off')\n    ax[1].set_title(f'highlight face')\n    ax[1].imshow(img_copy)\n    \n    face_crop = image.copy()\n    for (x,y,w,h) in face:\n        face_crop = image[y:y+h,x:x+w]\n        break; #to get just the first detection\n    \n    ax[2].axis('off')\n    ax[2].set_title(f'face cropped')\n    ax[2].imshow(face_crop)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Viewing first frame of a few videos (do note the image label(real/fake) in the title)"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"get_first_frame(train_files[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame(train_files[4])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame(train_files[11])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame(train_files[14])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"till now we seen samples of fake videos, below one is of a real one"},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame(train_files[13])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_filename(video_id,train=True):\n    if train:\n        return f'/kaggle/input/deepfake-detection-challenge/train_sample_videos/{video_id}'\n    else:\n        return f'/kaggle/input/deepfake-detection-challenge/test_videos/{video_id}'\n\ndef get_face_crop(filename):\n    cap = cv2.VideoCapture(filename)\n    ret,frame = cap.read()\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    cap.release()\n    cv2.destroyAllWindows()\n    hcascade = cv2.CascadeClassifier('/kaggle/input/haarcascades/haarcascade_frontalface_default.xml')\n    face = hcascade.detectMultiScale(image,1.2,3)\n    face_crop = image.copy()\n    for (x,y,w,h) in face:\n        face_crop = image[y:y+h,x:x+w]\n        break; #to get just the first detection\n    return face_crop\n\ndef compare_orig_to_fakes(orig,fake_files):\n    if(len(fake_files)<=2):\n        fig,ax = plt.subplots(1,len(fake_files)+1,figsize=(20,7))\n        face_crop = get_face_crop(get_filename(orig))\n        ax[0].axis('off')\n        ax[0].set_title(f'face cropped(real) - {orig}')\n        ax[0].imshow(face_crop)\n    \n        for i in range(len(fake_files)):\n            face_crop = get_face_crop(get_filename(fake_files[i]))\n            ax[i+1].axis('off')\n            ax[i+1].set_title(f'face cropped (fake) - {fake_files[i]}')\n            ax[i+1].imshow(face_crop)\n    else:\n        fig,ax = plt.subplots(int(len(fake_files)/2)+1,2,figsize=(20,7*int(len(fake_files)/2)))\n        face_crop = get_face_crop(get_filename(orig))\n        ax[0,0].axis('off')\n        ax[0,0].set_title(f'face cropped(real) - {orig}')\n        ax[0,0].imshow(face_crop)\n    \n        for i in range(1,len(fake_files)+1):\n            face_crop = get_face_crop(get_filename(fake_files[i-1]))\n            ax[int(i-(i/2)),i%2].axis('off')\n            ax[int(i-(i/2)),i%2].set_title(f'face cropped (fake) - {fake_files[i-1]}')\n            ax[int(i-(i/2)),i%2].imshow(face_crop)\n        \n        if(len(fake_files)%2==0):\n            ax[int(len(fake_files)/2),1].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Viewing Original and their corresponsing Fake videos side by side"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"orig_to_fake.groupby('original').apply(list).to_frame().iloc[:10].reset_index().apply(lambda row : compare_orig_to_fakes(row['original'],row['fakes']),axis=1);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"as we can see above some faces are not being recognized by haar cascadeclassifier(since we didnt get a zoomed version - see 1st and last set above) and some are misidentified as faces(see 5th set).  \nneed to find some other method to detect faces.  lets try mtcnn method as suggested [here](https://www.kaggle.com/aleksandradeis/deepfake-challenge-eda/notebook).  "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_first_frame_mtcnn(filename):\n    fig,ax = plt.subplots(1,3,figsize=(20,7))\n    \n    cap = cv2.VideoCapture(filename)\n    ret,frame = cap.read()\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    cap.release()\n    cv2.destroyAllWindows()\n    ax[0].axis('off')\n    ax[0].set_title(f'{get_videoid(filename)} - {get_label(filename)}')\n    ax[0].imshow(image)\n    \n    detector = MTCNN()\n    mtcnn_op = detector.detect_faces(image)\n    \n    img_copy = image.copy()\n    for boxes in mtcnn_op:\n        x,y,w,h = boxes['box']\n        cv2.rectangle(img_copy,(x,y),(x+w,y+h),(0,255,0),3)\n    \n    ax[1].axis('off')\n    ax[1].set_title(f'highlight face')\n    ax[1].imshow(img_copy)\n    \n    face_crop = image.copy()\n    for boxes in mtcnn_op:\n        x,y,w,h = boxes['box']\n        face_crop = image[y:y+h,x:x+w]\n        break; #to get just the first detection\n    \n    ax[2].axis('off')\n    ax[2].set_title(f'face cropped')\n    ax[2].imshow(face_crop)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame_mtcnn(train_files[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame_mtcnn(train_files[4])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame_mtcnn(train_files[11])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame_mtcnn(train_files[14])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_frame_mtcnn(train_files[13])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_face_crop_mtcnn(filename):\n    cap = cv2.VideoCapture(filename)\n    ret,frame = cap.read()\n    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    cap.release()\n    cv2.destroyAllWindows()\n    \n    detector = MTCNN()\n    mtcnn_op = detector.detect_faces(image)    \n        \n    face_crop = image.copy()\n    for boxes in mtcnn_op:\n        x,y,w,h = boxes['box']\n        face_crop = image[y:y+h,x:x+w]\n        break; #to get just the first detection\n    return face_crop\n\ndef compare_orig_to_fakes_mtcnn(orig,fake_files):\n    if(len(fake_files)<=2):\n        fig,ax = plt.subplots(1,len(fake_files)+1,figsize=(20,7))\n        face_crop = get_face_crop_mtcnn(get_filename(orig))\n        ax[0].axis('off')\n        ax[0].set_title(f'face cropped(real) - {orig}')\n        ax[0].imshow(face_crop)\n    \n        for i in range(len(fake_files)):\n            face_crop = get_face_crop_mtcnn(get_filename(fake_files[i]))\n            ax[i+1].axis('off')\n            ax[i+1].set_title(f'face cropped (fake) - {fake_files[i]}')\n            ax[i+1].imshow(face_crop)\n    else:\n        fig,ax = plt.subplots(int(len(fake_files)/2)+1,2,figsize=(20,7*int(len(fake_files)/2)))\n        face_crop = get_face_crop_mtcnn(get_filename(orig))\n        ax[0,0].axis('off')\n        ax[0,0].set_title(f'face cropped(real) - {orig}')\n        ax[0,0].imshow(face_crop)\n    \n        for i in range(1,len(fake_files)+1):\n            face_crop = get_face_crop_mtcnn(get_filename(fake_files[i-1]))\n            ax[int(i-(i/2)),i%2].axis('off')\n            ax[int(i-(i/2)),i%2].set_title(f'face cropped (fake) - {fake_files[i-1]}')\n            ax[int(i-(i/2)),i%2].imshow(face_crop)\n        \n        if(len(fake_files)%2==0):\n            ax[int(len(fake_files)/2),1].axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"orig_to_fake.groupby('original').apply(list).to_frame().iloc[:10].reset_index().apply(lambda row : compare_orig_to_fakes_mtcnn(row['original'],row['fakes']),axis=1);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"as you can see mtcnn is able to fill the gaps in haarcascade classifier by detecting the faces in 1st and the last set.  \nThe face in the 5th set is somehow identifiable as a face(as a human) which was missed by haarcascade classifier.  \nAlthough it fails for the 4th set as before.  \nMaybe we can try even more face detection methods, possibly dlib.  \nMore on that [here](https://www.kaggle.com/timesler/comparison-of-face-detection-packages)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frames(filename,zoomed=False,interval=100):\n    frames = []\n    cap = cv2.VideoCapture(filename)\n    detector = MTCNN()\n    frame_n = 0\n    while(cap.isOpened()):\n        ret,frame = cap.read() #ret is a boolean variable that returns true if the frame is available\n        \n        if not frame_n%interval:\n            if not ret:\n                break\n\n            image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n            if zoomed:\n                mtcnn_op = detector.detect_faces(image)\n                face_crop = image.copy()\n                for boxes in mtcnn_op:\n                    x,y,w,h = boxes['box']\n                    face_crop = image[y:y+h,x:x+w]\n                    frames.append(face_crop)\n                    break; #to get just the first detection\n            else:\n                frames.append(image)\n\n        frame_n += 1\n        \n    cap.release()\n    cv2.destroyAllWindows()\n    return frames\n\ndef visualize_frames(filename, zoomed=False, interval=100, cols=2, title=''):\n    frames = get_frames(filename, zoomed=zoomed, interval=interval)\n    n_frames = len(frames)\n    rows = n_frames // cols\n    if n_frames % cols:\n        rows = rows + 1\n    fig,axis = plt.subplots(rows,cols,figsize=(20,7*rows))\n    for i in range(n_frames):\n        r = i // cols\n        c = i % cols\n        if n_frames <= cols:\n            axis[c].imshow(frames[i])\n            axis[c].axis('off')\n            axis[c].set_title(str(i))\n        else:\n            axis[r,c].imshow(frames[i])\n            axis[r,c].axis('off')\n            axis[r,c].set_title(str(i))\n    \n    if(n_frames%cols):\n        for i in range(n_frames%cols,cols):\n            for i in range(n_frames%cols,cols):\n                if n_frames <= cols:\n                    axis[i].axis('off')\n                else:\n                    axis[n_frames//cols,i].axis('off')\n    \n    plt.suptitle(f'{get_videoid(filename)} {(\"(zoomed)\") if zoomed else \"\"}' if title=='' else title)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing sequences of frames of a few videos  \nhaving explored the first frame of each video, lets explore the sequential frames"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frames(train_files[0],interval=50)\nvisualize_frames(train_files[0],interval=50,zoomed=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_frames_alongwith_faces(filename, interval=100):\n    frames = []\n    frames_no = []\n    cap = cv2.VideoCapture(filename)\n    detector = MTCNN()\n    frame_n = 0\n    while(cap.isOpened()):\n        ret,frame = cap.read() #ret is a boolean variable that returns true if the frame is available\n        \n        if not frame_n%interval:\n            if not ret:\n                break\n\n            image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n            frames.append(image)\n            frames_no.append(frame_n)\n            mtcnn_op = detector.detect_faces(image)\n            face_crop = image.copy()\n            for boxes in mtcnn_op:\n                x,y,w,h = boxes['box']\n                face_crop = image[y:y+h,x:x+w]\n                frames.append(face_crop)\n                frames_no.append(frame_n)\n                break; #to get just the first detection\n        frame_n += 1\n        \n    cap.release()\n    cv2.destroyAllWindows()\n    return frames,frames_no\n\ndef visualize_frames_alongside_faces(filename, interval=100, cols=2, title=''):\n    frames,frames_no = get_frames_alongwith_faces(filename, interval=interval)\n    n_frames = len(frames)\n    rows = n_frames // cols\n    if n_frames % cols:\n        rows = rows + 1\n    fig,axis = plt.subplots(rows,cols,figsize=(20,6*rows))\n    for i in range(n_frames):\n        r = i // cols\n        c = i % cols\n        axis[r,c].imshow(frames[i])\n        axis[r,c].axis('off')\n        axis[r,c].set_title(f'frame #{frames_no[i]}' if c==0 else f'frame #{frames_no[i]} (zoomed)')\n    \n    if(n_frames%cols):\n        for i in range(n_frames%cols,cols):\n            axis[n_frames//cols,i].axis('off')\n    \n    plt.suptitle(f'{get_videoid(filename)}' if title=='' else title)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_frames_alongside_faces(train_files[0],interval=50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_faces(filename):\n    frames = []\n    frames_no = []\n    cap = cv2.VideoCapture(filename)\n    detector = MTCNN()\n    frame_n = 0\n    while(cap.isOpened()):\n        ret,frame = cap.read() #ret is a boolean variable that returns true if the frame is available\n        \n        if not ret:\n            break\n\n        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n        mtcnn_op = detector.detect_faces(image)\n        face_crop = image.copy()\n        for boxes in mtcnn_op:\n            x,y,w,h = boxes['box']\n            face_crop = image[y:y+h,x:x+w]\n            frames.append(face_crop)\n            frames_no.append(frame_n)\n            break; #to get just the first detection\n        frame_n += 1\n        \n    cap.release()\n    cv2.destroyAllWindows()\n    return frames,frames_no\n\ndef animate_frames(filename):\n    frames,_ = get_faces(filename)\n    fig = plt.figure(figsize=(16,9))\n    \n    def update(frame_number):\n        plt.axis('off')\n        plt.imshow(frames[frame_number])\n\n    return FuncAnimation(fig, update, interval=30, repeat=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Animating the sequences of frames (with just faces extracted)"},{"metadata":{"trusted":true},"cell_type":"code","source":"animation = animate_frames(train_files[0])\nHTML(animation.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_ssim_score(frame_a,frame_b,get_str_sim_image=False):\n    \"\"\"\n    Compute the mean structural similarity index between two images.\n    \"\"\"\n    if frame_a.shape[0] != frame_b.shape[0]: #resizing height on mismatch\n        if frame_a.shape[0] < frame_b.shape[0]:\n            frame_b = frame_b[:frame_a.shape[0],:,:]\n        else:\n            frame_a = frame_a[:frame_b.shape[0],:,:]\n\n    if frame_a.shape[1] != frame_b.shape[1]: #resizing width on mismatch\n        if frame_a.shape[1] < frame_b.shape[1]:\n            frame_b = frame_b[:,:frame_a.shape[1],:]\n        else:\n            frame_a = frame_a[:,:frame_b.shape[1],:]\n    \n    if get_str_sim_image:\n        (score,image)=structural_similarity(frame_a,frame_b,multichannel=True, full=True) #image has full structural similarity image\n        return score,image\n    else:\n        return structural_similarity(frame_a,frame_b,multichannel=True)\n\ndef get_frames_similarity_scores(filename, zoomed=False, interval=10):\n    frames = get_frames(filename, zoomed=zoomed, interval=interval)\n    print(len(frames),\"frames\")\n    scores = []\n    for i in range(1,len(frames)):\n        frame = frames[i]\n        prev_frame = frames[i-1]\n        score = get_ssim_score(frame,prev_frame)\n        # score,image=get_ssim_score(frame,prev_frame,get_str_sim_image=True) #image has full structural similarity image\n        scores.append(score)\n    return scores,frames\n\ndef plot_scores(scores,title=\"\"):\n    plt.figure(figsize=(12,7))\n    plt.plot(scores)\n    plt.title(f\"Similarity Scores ({title})\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting the structural similarity scores of 2 consecutive frames for all the frames of the video  \nlets plot the scores of just the face part zoomed in first. "},{"metadata":{"trusted":true},"cell_type":"code","source":"(scores_fake_face,frames_fake_face) = get_frames_similarity_scores(get_filename(orig_to_fake.reset_index().iloc[5]['fakes']), zoomed=True, interval=2)\nplot_scores(scores_fake_face,\"fake face\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"since it doesn't give a clear picture of the similarities b/w frame as there is noise in it (a possible explanation of this noise is because if we play the face extracted parts as a video it's not stabilized meaning the face parts extracted by mtcnn from each frame has some irregularities as far as the location from where it extracted is concerned and is not consistent - eg. tip of the left eye may be a few pixels off in the next consecutive frame), lets try plotting for the images as is(without face extraction)."},{"metadata":{"trusted":true},"cell_type":"code","source":"(scores_fake,frames_fake) = get_frames_similarity_scores(get_filename(orig_to_fake.reset_index().iloc[5]['fakes']), interval=1)\nplot_scores(scores_fake,\"fake\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(scores_real,frames_real) = get_frames_similarity_scores(get_filename(orig_to_fake.reset_index().iloc[5]['original']), interval=1)\nplot_scores(scores_real,\"real\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nplt.plot(scores_fake, label = 'fake', color='red')\nplt.plot(scores_real, label = 'real', color='g')\nplt.title(\"Similarity Scores (fake vs real)\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Comparing them there's hardly any major difference probably because \"fake\" part is just on the face and face constitutes a very small area in the entire image. But remember if we just take the face part, it adds to the noise due to unstabilized aspect of extracted frames?"},{"metadata":{},"cell_type":"markdown","source":"## Visualizing the frames where there is a drop in the similarity score"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def visualize_faces(frames, cols=3):\n    n_frames = len(frames)\n    rows = n_frames // cols\n    if n_frames % cols:\n        rows = rows + 1\n    fig,axis = plt.subplots(rows,cols,figsize=(20,7*rows))\n    for i in range(n_frames):\n        r = i // cols\n        c = i % cols\n        if n_frames <= cols:\n            axis[c].imshow(frames[i])\n            axis[c].axis('off')\n        else:\n            axis[r,c].imshow(frames[i])\n            axis[r,c].axis('off')\n    \n    if(n_frames%cols):\n        for i in range(n_frames%cols,cols):\n            if n_frames <= cols:\n                axis[i].axis('off')\n            else:\n                axis[n_frames//cols,i].axis('off')\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Credits where its due:\n[Aleksandra Deis](https://www.kaggle.com/aleksandradeis) for [this notebook](https://www.kaggle.com/aleksandradeis/deepfake-challenge-eda/notebook) : took most of the idea and references from here, added upon it things like face detection using mtcnn where it lacked."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}