{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Can you detect the deepfake?\n![](https://i.blogs.es/841847/jennifer_buscemi/450_1000.jpg)\n\nCome here for a summary? Here is a tl;dr:\n- I strongly encourage you start first with the [official Getting Started guide here](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started).\n- What is the goal of the Deepfake Detection Challenge? According to the FAQ \"The AI technologies that power deepfakes and other tampered media are rapidly evolving, making deepfakes so hard to detect that, at times, even human evaluators can’t reliably tell the difference. The Deepfake Detection Challenge is designed to incentivize rapid progress in this area by inviting participants to compete to create new ways of detecting and preventing manipulated media.\"\n- This is a Code Competition:\n    - CPU Notebook <= 9 hours run-time, GPU Notebook <= 9 hours run-time on Kaggle's P100 GPUs, No internet access enabled\n    - External data is **allowed up to 1 GB in size**. External data must be freely & publicly available, including pre-trained models\n- This code competition's **training set is not available directly on Kaggle**, as its size is prohibitively large to train in Kaggle. Instead, it's **strongly recommended that you train offline** and load the externally trained model as an external dataset into Kaggle Notebooks to perform inference on the Test Set. Review Getting Started for more detailed information."},{"metadata":{},"cell_type":"markdown","source":"# Scoring\nSubmissions are scored on log loss:\n$$ LogLoss = - \\frac{1}{n} \\sum\\limits_{i=1}^n [y_i \\cdot log_e(\\hat{y_i}) + (1-y_i) \\cdot log_e(1-\\hat{y_i})]  $$\nwhere:\n- $n$ is the number of videos being predicted\n- $y^i$ is the predicted probability of the video being FAKE\n- $y_i$ is 1 if the video is FAKE, 0 if REAL\n- $log()$ is the natural (base e) logarithm\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# SKLearn Implemention\nfrom sklearn.metrics import log_loss\nlog_loss([\"REAL\", \"FAKE\", \"FAKE\", \"REAL\"],\n         [[.1, .9], [.9, .1], [.8, .2], [.35, .65]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data\n- The data is comprised of .mp4 files, split into compressed sets of ~10GB apiece. A metadata.json accompanies each set of .mp4 files, and contains filename, label (REAL/FAKE), original and split columns, listed below under Columns.\n- The full training set is just over 470 GB.\n\n*References: https://deepfakedetectionchallenge.ai/faqs*"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pylab as plt\nimport cv2\nplt.style.use('ggplot')\nfrom IPython.display import Video\nfrom IPython.display import HTML","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Take a look at the input folder files"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!ls -GFlash ../input/deepfake-detection-challenge","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can take a look at the total disk use of the data accessable from within the kernel. It is only 4.2G while the entire dataset is over 100x larger (470GB)."},{"metadata":{"trusted":true},"cell_type":"code","source":"!du -sh ../input/deepfake-detection-challenge/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Description of Datsets\nPer the competition description [here](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started):\n\n*There are 4 groups of datasets associated with this competition.*\n\n**Training Set:** *This dataset, containing labels for the target, is available for download outside of Kaggle for competitors to build their models. It is broken up into 50 files, for ease of access and download. Due to its large size, it must be accessed through a GCS bucket which is only made available to participants after accepting the competition’s rules. Please read the rules fully before accessing the dataset, as they contain important details about the dataset’s permitted use. It is expected and encouraged that you train your models outside of Kaggle’s notebooks environment and submit to Kaggle by uploading the trained model as an external data source.*\n\n**Public Validation Set:** *When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as test_videos.zip*\n\n**Public Test Set:** *This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against. When you “Submit to Competition” from the “Output” file of a committed notebook that contains the competition’s dataset, your code will be re-run in the background against this Public Test Set. When the re-run is complete, the score will be posted to the public leaderboard. If the re-run fails, you will see an error reflected in your “My Submissions” page. Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. You are limited to 2 submissions per day, including submissions which error.*\n\n**Private Test Set:** *This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes. After the competition deadline, Kaggle transfers your 2 final selected submissions’ code to the host. They will re-run your code against this private dataset and return prediction submissions back to Kaggle for computing your final private leaderboard scores.*"},{"metadata":{},"cell_type":"markdown","source":"# Review of Data Files Accessable within kernel\n\n### Files\n- **train_sample_videos.zip** - a ZIP file containing a sample set of training videos and a metadata.json with labels. the full set of training videos is available through the links provided above.\n- **sample_submission.csv** - a sample submission file in the correct format.\n- **test_videos.zip** - a zip file containing a small set of videos to be used as a public validation set.\nTo understand the datasets available for this competition, review the Getting Started information.\n\n### Metadata Columns\n- **filename** - the filename of the video\n- **label** - whether the video is REAL or FAKE\n- **original** - in the case that a train set video is FAKE, the original video is listed here\n- **split** - this is always equal to \"train\"."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_sample_metadata = pd.read_json('../input/deepfake-detection-challenge/train_sample_videos/metadata.json').T\ntrain_sample_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_sample_metadata.groupby('label')['label'].count().plot(figsize=(15, 5), kind='bar', title='Distribution of Labels in the Training Set')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Example of fake: `aagfhgtpmv.mp4`\n![](https://i.imgur.com/ToWjusQ.gif)"},{"metadata":{},"cell_type":"markdown","source":"# Lets use the face_recognition package to detect faces in the video*\nCheck out this great kernel here https://www.kaggle.com/brassmonkey381/a-quick-look-at-the-first-frame-of-each-video for how I learned to capture a frame from the video file.\n\n* Note that in this kernel I use `pip` to install the `face_recognition` package. This is for demonstration purposes. In the final evaluation kernel you will not be able to have internet access. We can request that kaggle add this package to the official kernel docker image."},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2 as cv\nimport os\nimport matplotlib.pylab as plt\ntrain_dir = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/'\nfig, ax = plt.subplots(1,1, figsize=(15, 15))\ntrain_video_files = [train_dir + x for x in os.listdir(train_dir)]\n# video_file = train_video_files[30]\nvideo_file = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/akxoopqjqz.mp4'\ncap = cv.VideoCapture(video_file)\nsuccess, image = cap.read()\nimage = cv.cvtColor(image, cv.COLOR_BGR2RGB)\ncap.release()   \nax.imshow(image)\nax.xaxis.set_visible(False)\nax.yaxis.set_visible(False)\nax.title.set_text(f\"FRAME 0: {video_file.split('/')[-1]}\")\nplt.grid(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now lets use opencv to detect the faces using the `face_recognition` package! First we need to pip install it. Make sure you have internet turned on in your kernel.\n\nReference: https://github.com/ageitgey/face_recognition"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install face_recognition","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Locating a face within an image"},{"metadata":{"trusted":true},"cell_type":"code","source":"import face_recognition\nface_locations = face_recognition.face_locations(image)\n\n# https://github.com/ageitgey/face_recognition/blob/master/examples/find_faces_in_picture.py\nfrom PIL import Image\n\nprint(\"I found {} face(s) in this photograph.\".format(len(face_locations)))\n\nfor face_location in face_locations:\n\n    # Print the location of each face in this image\n    top, right, bottom, left = face_location\n    print(\"A face is located at pixel location Top: {}, Left: {}, Bottom: {}, Right: {}\".format(top, left, bottom, right))\n\n    # You can access the actual face itself like this:\n    face_image = image[top:bottom, left:right]\n    fig, ax = plt.subplots(1,1, figsize=(5, 5))\n    plt.grid(False)\n    ax.xaxis.set_visible(False)\n    ax.yaxis.set_visible(False)\n    ax.imshow(face_image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Locating a face landmarks within an image"},{"metadata":{"trusted":true},"cell_type":"code","source":"face_landmarks_list = face_recognition.face_landmarks(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://github.com/ageitgey/face_recognition/blob/master/examples/find_facial_features_in_picture.py\n# face_landmarks_list\nfrom PIL import Image, ImageDraw\npil_image = Image.fromarray(image)\nd = ImageDraw.Draw(pil_image)\n\nfor face_landmarks in face_landmarks_list:\n\n    # Print the location of each facial feature in this image\n    for facial_feature in face_landmarks.keys():\n        print(\"The {} in this face has the following points: {}\".format(facial_feature, face_landmarks[facial_feature]))\n\n    # Let's trace out each facial feature in the image with a line!\n    for facial_feature in face_landmarks.keys():\n        d.line(face_landmarks[facial_feature], width=3)\n\n# Show the picture\ndisplay(pil_image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Displaying many test examples and labels\nCan you tell which are fakes?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(19, 2, figsize=(15, 80))\naxs = np.array(axs)\naxs = axs.reshape(-1)\ni = 0\nfor fn in train_sample_metadata.index[:23]:\n    label = train_sample_metadata.loc[fn]['label']\n    orig = train_sample_metadata.loc[fn]['label']\n    video_file = f'/kaggle/input/deepfake-detection-challenge/train_sample_videos/{fn}'\n    ax = axs[i]\n    cap = cv.VideoCapture(video_file)\n    success, image = cap.read()\n    image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n    face_locations = face_recognition.face_locations(image)\n    if len(face_locations) > 0:\n        # Print first face\n        face_location = face_locations[0]\n        top, right, bottom, left = face_location\n        face_image = image[top:bottom, left:right]\n        ax.imshow(face_image)\n        ax.grid(False)\n        ax.title.set_text(f'{fn} - {label}')\n        ax.xaxis.set_visible(False)\n        ax.yaxis.set_visible(False)\n        # Find landmarks\n        face_landmarks_list = face_recognition.face_landmarks(face_image)\n        face_landmarks = face_landmarks_list[0]\n        pil_image = Image.fromarray(face_image)\n        d = ImageDraw.Draw(pil_image)\n        for facial_feature in face_landmarks.keys():\n            d.line(face_landmarks[facial_feature], width=2)\n        landmark_face_array = np.array(pil_image)\n        ax2 = axs[i+1]\n        ax2.imshow(landmark_face_array)\n        ax2.grid(False)\n        ax2.title.set_text(f'{fn} - {label}')\n        ax2.xaxis.set_visible(False)\n        ax2.yaxis.set_visible(False)\n        i += 2\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Add padding to zoom out of face"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(19, 2, figsize=(10, 80))\naxs = np.array(axs)\naxs = axs.reshape(-1)\ni = 0\npad = 60\nfor fn in train_sample_metadata.index[23:44]:\n    label = train_sample_metadata.loc[fn]['label']\n    orig = train_sample_metadata.loc[fn]['label']\n    video_file = f'/kaggle/input/deepfake-detection-challenge/train_sample_videos/{fn}'\n    ax = axs[i]\n    cap = cv.VideoCapture(video_file)\n    success, image = cap.read()\n    image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n    face_locations = face_recognition.face_locations(image)\n    if len(face_locations) > 0:\n        # Print first face\n        face_location = face_locations[0]\n        top, right, bottom, left = face_location\n        face_image = image[top-pad:bottom+pad, left-pad:right+pad]\n        ax.imshow(face_image)\n        ax.grid(False)\n        ax.title.set_text(f'{fn} - {label}')\n        ax.xaxis.set_visible(False)\n        ax.yaxis.set_visible(False)\n        # Find landmarks\n        face_landmarks_list = face_recognition.face_landmarks(face_image)\n        try:\n            face_landmarks = face_landmarks_list[0]\n            pil_image = Image.fromarray(face_image)\n            d = ImageDraw.Draw(pil_image)\n            for facial_feature in face_landmarks.keys():\n                d.line(face_landmarks[facial_feature], width=2, fill='white')\n            landmark_face_array = np.array(pil_image)\n            ax2 = axs[i+1]\n            ax2.imshow(landmark_face_array)\n            ax2.grid(False)\n            ax2.title.set_text(f'{fn} - {label}')\n            ax2.xaxis.set_visible(False)\n            ax2.yaxis.set_visible(False)\n            i += 2\n        except:\n            pass\nplt.grid(False)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Frame by Frame Face Detection\n- The real power may come from looking at how the \"face\" changes or doesn't change as the video progresses\n- We will take the FAKE example video `akxoopqjqz.mp4`\n- First we will loop through the frames of the video file and append them to a list called `frames`"},{"metadata":{"trusted":true},"cell_type":"code","source":"video_file = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/akxoopqjqz.mp4'\n\ncap = cv2.VideoCapture(video_file)\n\nframes = []\nwhile(cap.isOpened()):\n    ret, frame = cap.read()\n    if ret==True:\n        frames.append(frame)\n        if cv2.waitKey(1) & 0xFF == ord('q'):\n            break\n    else:\n        break\ncap.release()\n\nprint('The number of frames saved: ', len(frames))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can display some of the frames of this video"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3, 3, figsize=(15, 10))\naxes = np.array(axes)\naxes = axes.reshape(-1)\n\nax_ix = 0\nfor i in [0, 25, 50, 75, 100, 125, 150, 175, 250]:\n    frame = frames[i]\n    #fig, ax = plt.subplots(1,1, figsize=(5, 5))\n    image = cv.cvtColor(frame, cv.COLOR_BGR2RGB)\n    axes[ax_ix].imshow(image)\n    axes[ax_ix].xaxis.set_visible(False)\n    axes[ax_ix].yaxis.set_visible(False)\n    axes[ax_ix].set_title(f'Frame {i}')\n    ax_ix += 1\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can use the face detection to pull the faces from each frame in the video. Notice that the face coun't be detected for one of the frames."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3, 3, figsize=(15, 15))\naxes = np.array(axes)\naxes = axes.reshape(-1)\nax_ix = 0\npadding = 40\nfor i in [0, 25, 50, 75, 100, 125, 150, 175, 250, 275]:\n    frame = frames[i]\n    #fig, ax = plt.subplots(1,1, figsize=(5, 5))\n    face_locations = face_recognition.face_locations(frame)\n    if len(face_locations) == 0:\n        print(f'Could not find face in frame {i}')\n        continue\n    top, right, bottom, left = face_locations[0]\n    frame_face = frame[top-padding:bottom+padding, left-padding:right+padding]\n    image = cv.cvtColor(frame_face, cv.COLOR_BGR2RGB)\n    axes[ax_ix].imshow(image)\n    axes[ax_ix].xaxis.set_visible(False)\n    axes[ax_ix].yaxis.set_visible(False)\n    axes[ax_ix].set_title(f'Frame {i}')\n    ax_ix += 1\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting facial landmarks for each frame"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3, 3, figsize=(15, 15))\naxes = np.array(axes)\naxes = axes.reshape(-1)\nax_ix = 0\npadding = 40\nfor i in [0, 25, 50, 75, 100, 125, 150, 175, 250, 275]:\n    frame = frames[i]\n    #fig, ax = plt.subplots(1,1, figsize=(5, 5))\n    face_locations = face_recognition.face_locations(frame)\n    if len(face_locations) == 0:\n        print(f'Count find face in frame {i}')\n        continue\n    top, right, bottom, left = face_locations[0]\n    frame_face = frame[top-padding:bottom+padding, left-padding:right+padding]\n    face_landmarks_list = face_recognition.face_landmarks(frame_face)\n    if len(face_landmarks_list) == 0:\n        print(f'Could not identify face landmarks for frame {i}')\n        continue\n    face_landmarks = face_landmarks_list[0]\n    pil_image = Image.fromarray(frame_face)\n    d = ImageDraw.Draw(pil_image)\n    for facial_feature in face_landmarks.keys():\n        d.line(face_landmarks[facial_feature], width=3, fill='white')\n    landmark_face_array = np.array(pil_image)\n    image = cv.cvtColor(landmark_face_array, cv.COLOR_BGR2RGB)\n    axes[ax_ix].imshow(image)\n    axes[ax_ix].grid(False)\n    axes[ax_ix].set_title(f'FAKE example - Frame {i}')\n    axes[ax_ix].xaxis.set_visible(False)\n    axes[ax_ix].yaxis.set_visible(False)\n    ax_ix += 1\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Frame by frame of REAL example"},{"metadata":{"trusted":true},"cell_type":"code","source":"fn = 'ahqqqilsxt.mp4'\nvideo_file = f'/kaggle/input/deepfake-detection-challenge/train_sample_videos/{fn}'\n\ncap = cv2.VideoCapture(video_file)\n\nframes = []\nwhile(cap.isOpened()):\n    ret, frame = cap.read()\n    if ret==True:\n        frames.append(frame)\n        if cv2.waitKey(1) & 0xFF == ord('q'):\n            break\n    else:\n        break\ncap.release()\n\nprint('The number of frames saved: ', len(frames))\n\nfig, axes = plt.subplots(3, 3, figsize=(15, 15))\naxes = np.array(axes)\naxes = axes.reshape(-1)\nax_ix = 0\npadding = 40\nfor i in [0, 25, 50, 75, 100, 125, 150, 175, 250, 275]:\n    frame = frames[i]\n    #fig, ax = plt.subplots(1,1, figsize=(5, 5))\n    face_locations = face_recognition.face_locations(frame)\n    if len(face_locations) == 0:\n        print(f'Count find face in frame {i}')\n        continue\n    top, right, bottom, left = face_locations[0]\n    frame_face = frame[top-padding:bottom+padding, left-padding:right+padding]\n    face_landmarks_list = face_recognition.face_landmarks(frame_face)\n    if len(face_landmarks_list) == 0:\n        print(f'Could not identify face landmarks for frame {i}')\n        continue\n    face_landmarks = face_landmarks_list[0]\n    pil_image = Image.fromarray(frame_face)\n    d = ImageDraw.Draw(pil_image)\n    for facial_feature in face_landmarks.keys():\n        d.line(face_landmarks[facial_feature], width=2, fill='white')\n    landmark_face_array = np.array(pil_image)\n    image = cv.cvtColor(landmark_face_array, cv.COLOR_BGR2RGB)\n    axes[ax_ix].imshow(image)\n    axes[ax_ix].grid(False)\n    axes[ax_ix].set_title(f'REAL example - Frame {i}')\n    axes[ax_ix].xaxis.set_visible(False)\n    axes[ax_ix].yaxis.set_visible(False)\n    ax_ix += 1\n    if ax_ix >= len(axes):\n        break\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sample Submission\n- Lets use the distribution in the training make a guess on the test set.\n- We are predicting the probability that the video is a **fake**.\n- 80.75% of the training videos are fake. Turns out the test set does not share the same distribution, as predicting 0.80 scores worse than simply guessing 0.5"},{"metadata":{"trusted":true},"cell_type":"code","source":"ss = pd.read_csv(\"/kaggle/input/deepfake-detection-challenge/sample_submission.csv\")\nss['label'] = 0.5\nss.loc[ss['filename'] == 'aassnaulhq.mp4', 'label'] = 0 # Guess the true value\nss.loc[ss['filename'] == 'aayfryxljh.mp4', 'label'] = 0\nss.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ss.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}