{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Deepfakes make me feel [cagey](https://imgur.com/zzsW0Zf).\n![Deepfakes make me feel cagey](https://i.imgur.com/zzsW0Zf.gif)\n\nThis kernel is supposed to give an introduction to deepfakes and detectors. check out some of the data and point out some general challenge things.\n\nWho would be better at introducing this than Károly from [Two Minute Papers](https://www.youtube.com/watch?v=RoGHVI-w9bE), so hold on to your papers. \n\n(It is a video about a SOTA video of FaceForensics++ after all)"},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.lib.display import YouTubeVideo\nkarol = YouTubeVideo('RoGHVI-w9bE', width=560, height=315)\ndisplay(karol)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Deepfakes are fakes generated by deep learning. So far so easy. \n\n# How are Deep Fakes Generated?\n\nThis usually means someone used a generative model like an Auto - Encoder or most likely a Generative Adversarial Network, short GAN. GANs are technically two networks that work against each other, illustrated below. The artist (generator) draws its inspiration from a noise sample and creates a rendering of the data you are trying to generate with said GAN. The private investigator (discriminator) randomly gets assigned real and fake data to investigate. \n\nThe learning process is collaborative. The generator gets better at fooling the discriminator and the discriminator gets better at figuring out which data is real and which isn't. In mathematical terms they are learning until a [Nash equilibrium](https://en.wikipedia.org/wiki/Nash_equilibrium) is reached, which means neither can learn new tricks and get better. They're a really cool concept and even used in scientific simulation at [CERN](https://indico.cern.ch/event/595059/contributions/2497383/attachments/1431666/2199445/gan_presentation_IML.pdf).\n\nYou can probably guess that they can be tricky to train, due to so many moving parts. This has become a very popular area of research, warranting a [GAN Zoo](https://github.com/hindupuravinash/the-gan-zoo) of all named GANs. Some important stuff you may want to check out if your interested are keywords like Wasserstein GANs, Gradient Penalization, Attention, and in this context Style Transfer (namely face2face). Maybe you'll even find some in the comments.\n\n\n![GAN from Dramsch PhD thesis.](https://dramsch.net/assets/images/GAN.PNG)\nFigure describing Generative Adversarial Networks from my [PhD thesis](https://orbit.dtu.dk/en/publications/machine-learning-in-4d-seismic-data-analysis-deep-neural-networks)."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport cv2\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom pathlib import Path\n\nfrom sklearn_pandas import DataFrameMapper\nimport sklearn, sklearn.preprocessing\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom sklearn.model_selection import train_test_split, cross_val_score\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Handling Video\nSince we're dealing with video here, let's have a quick look at the data.\n\nJust keep in mind that the data is so large that you'll want to use a generator to concurrently load data for training. In PyTorch they're called Dataset and Dataloader and the only one I found is [here](https://github.com/MohsenFayyaz89/PyTorch_Video_Dataset). In keras it's the custom ImageGenerator, where you'll have to subclass [Sequence](https://keras.io/utils/), I found a good starter [here](https://github.com/jphdotam/keras_generator_example), which unfortunately expects numpy files. However, you may be able to change it to mp4 using OpenCV.\n\nSo let's have a quick look at OpenCV, also check out their tutorial on [loading video data](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_gui/py_video_display/py_video_display.html). The documentation for [VideoCapture.get()](https://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#videocapture-get) is a bit dense, but let's try some stuff."},{"metadata":{},"cell_type":"markdown","source":"0.     CV_CAP_PROP_POS_MSEC Current position of the video file in milliseconds or video capture timestamp.\n1.     CV_CAP_PROP_POS_FRAMES 0-based index of the frame to be decoded/captured next.\n2.     CV_CAP_PROP_POS_AVI_RATIO Relative position of the video file: 0 - start of the film, 1 - end of the film.\n3.     CV_CAP_PROP_FRAME_WIDTH Width of the frames in the video stream.\n4.     CV_CAP_PROP_FRAME_HEIGHT Height of the frames in the video stream.\n5.     CV_CAP_PROP_FPS Frame rate.\n6.     CV_CAP_PROP_FOURCC 4-character code of codec.\n7.     CV_CAP_PROP_FRAME_COUNT Number of frames in the video file.\n8.     CV_CAP_PROP_FORMAT Format of the Mat objects returned by retrieve() .\n9.     CV_CAP_PROP_MODE Backend-specific value indicating the current capture mode.\n1.     CV_CAP_PROP_BRIGHTNESS Brightness of the image (only for cameras).\n1.     CV_CAP_PROP_CONTRAST Contrast of the image (only for cameras).\n1.     CV_CAP_PROP_SATURATION Saturation of the image (only for cameras).\n1.     CV_CAP_PROP_HUE Hue of the image (only for cameras).\n1.     CV_CAP_PROP_GAIN Gain of the image (only for cameras).\n1.     CV_CAP_PROP_EXPOSURE Exposure (only for cameras).\n16.     CV_CAP_PROP_CONVERT_RGB Boolean flags indicating whether images should be converted to RGB.\n17.     CV_CAP_PROP_WHITE_BALANCE_U The U value of the whitebalance setting (note: only supported by DC1394 v 2.x backend currently)\n18.     CV_CAP_PROP_WHITE_BALANCE_V The V value of the whitebalance setting (note: only supported by DC1394 v 2.x backend currently)\n19.     CV_CAP_PROP_RECTIFICATION Rectification flag for stereo cameras (note: only supported by DC1394 v 2.x backend currently)\n20.     CV_CAP_PROP_ISO_SPEED The ISO speed of the camera (note: only supported by DC1394 v 2.x backend currently)\n21.     CV_CAP_PROP_BUFFERSIZE Amount of frames stored in internal buffer memory (note: only supported by DC1394 v 2.x backend currently)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def vid_stat(vid):\n    cap = cv2.VideoCapture(vid)\n    width = cap.get(3)\n    height = cap.get(4)\n    frame_rate = cap.get(5)\n    frame_num = cap.get(7)\n    #_, frame = cap.read()\n    cap.release()\n\n    return vid.rsplit(\"/\")[-1], width, height, frame_rate, frame_num","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis\nMeta-data like a good wine. Are there variations in framerate? Is the length slightly off?\n\nLots of funny things happen when you try to fake videos (did I mention that GANs are a nuisance to train?)"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain = Path(\"/kaggle/input/deepfake-detection-challenge/train_sample_videos/\")\ntest = Path(\"/kaggle/input/deepfake-detection-challenge/test_videos/\")\n\n\ntrain_stats = [vid_stat(str(vid)) for vid in train.glob('*.mp4')]\ntest_stats = [vid_stat(str(vid)) for vid in test.glob('*.mp4')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at a single fake video. Could you tell?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,axs = plt.subplots(ncols=4, figsize=(40,10))\nvid = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/aagfhgtpmv.mp4'\ncap = cv2.VideoCapture(str(vid))\nfor i in range(4):\n    [cap.read() for _ in range(50)]\n    _, frame = cap.read()\n    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    axs[i].imshow(frame)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I picked this video for the visual artifacts. Unfortunately, this competition may fake images or voices. This one looks like it may be a voice swap."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,axs = plt.subplots(ncols=4, figsize=(40,10))\nvid = '/kaggle/input/deepfake-detection-challenge/train_sample_videos/etohcvnzbj.mp4'\ncap = cv2.VideoCapture(str(vid))\nfor i in range(4):\n    [cap.read() for _ in range(50)]\n    _, frame = cap.read()\n    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    axs[i].imshow(frame)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the deep learniing models, temporal signal will probably be quite important. GANs often cause flickering artifacts, however, recent generative models have become much better at this."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame(train_stats, columns = ['Id', 'Width', 'Height', 'FPS', 'Number Frames'])\ndf[\"Length\"] = df.FPS * df[\"Number Frames\"]\ndf[\"Area\"] = df.Width * df.Height\ndf.set_index('Id', inplace=True)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_ = pd.DataFrame(test_stats, columns =['Id', 'Width', 'Height', 'FPS', 'Number Frames']) \ndf_[\"Length\"] = df_.FPS * df_[\"Number Frames\"]\ndf_[\"Area\"] = df_.Width * df_.Height\ndf_.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_ = df_.drop([ \"Area\"], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All data is in HD and 30ish FPS.\n\nLooks like the length is either 299 or 300 frames and the fps varying slightly.\nThe variation in width and height seems to stem from flipped axes. That is definitely something you'll want to *look out for in your submission notebook*."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_meta = pd.read_json('/kaggle/input/deepfake-detection-challenge/train_sample_videos/metadata.json').T\ntrain = pd.concat([train_meta, df], axis=1, sort=False).drop([\"split\", \"original\", \"Area\"],axis=1)\ntrain","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(\"label\")[\"label\"].count().plot(kind=\"pie\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(\"label\")[\"Length\"].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(\"label\")[\"Length\"].std()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Try something stupid!\nStupid in the sense that we'll just try to predict on the metadata. Maybe it works, it probably doesn't. However, let's consider that the mean real video is a bit shorter and the real video length varies a bit more. Nothing significant, but if we don't try we aren't the wiser."},{"metadata":{"trusted":true},"cell_type":"code","source":"mapper = DataFrameMapper([\n     ('label', sklearn.preprocessing.LabelBinarizer()),\n     (['Length', 'FPS'], sklearn.preprocessing.StandardScaler()),\n     (['Width', 'Height', 'Number Frames'], sklearn.preprocessing.OneHotEncoder())\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_[\"label\"] = 0\n\ny, y_val, X, X_val = train_test_split(mapper.fit_transform(train)[:, 0], mapper.transform(train)[:, 1:], test_size=.5, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = GradientBoostingClassifier(random_state=42)\nclf.fit(X, y)\n\ncross_val_score(clf, X_val, y_val, cv=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf_dummy = DummyClassifier(\"prior\", random_state=42)\nclf_dummy.fit(X, y)\n\ncross_val_score(clf_dummy, X_val, y_val, cv=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Dangit. At least you don't have to try it now.\n\nYou made it to the end, so remember that your submission notebooks will be run on the kaggle servers and will count towards your *maximum allowance of 9h*."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.7.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}