\n\n\n\n# Content\n\n- Introduction \n- Preliminary data exploration \n * Load the packages \n * Load the data \n * Check files type \n- Meta data exploration \n * Missing data \n * Unique values \n * Most frequent originals \n- Video data exploration \n * Missing video (or meta) data \n * Few fake videos \n * Few real videos \n * Videos with same original \n * Test video files \n * Play video files\n- Face detection \n- Resources \n- References \n\n"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n\nDeepFake is composed from Deep Learning and Fake and means taking one person from an image or video and replacing with someone else\nlikeness using technology such as Deep Artificial Neural Networks [1]. Large companies like Google invest very much in fighting the DeepFake, this including release of large datasets to help training models to counter this threat [2].The phenomen invades rapidly the film industry and threatens to compromise news agencies. Large digital companies, including content providers and social platforms are in the frontrun of fighting Deep Fakes. GANs that generate DeepFakes becomes better every day and, of course, if you include in a new GAN model all the information we collected until now how to combat various existent models, we create a model that cannot be beatten by the existing ones. \n\nIn the **Data Exploration** section we perform a (partial) Exploratory Data Analysis (EDA) on the training and testing data. After we are checking the files types, we are focusing first on the **metadata** files, which we are exploring in details, after we are importing in dataframes. Then, we move to explore video files, by looking first to a sample of fake videos, then to real videos. After that, we are also exploring few of the videos with the same origin. We are visualizing one frame extracted from the video, for both real and fake videos. Then we are also playing few videos. \nThen, we move to perform face (and other `objects` from the persons in the videos) extraction. More precisely, we are using OpenCV Haar Cascade resources to identify frontal face, eyes, smile and profile face from still images in the videos.\n\n**Important note**: The data we analyze here is just a very small sample of data. The competition specifies that the train data is provided as archived chunks. Training of models should pe performed offline using the data provided by Kaggle as archives, models should be loaded (max 1GB memory) in a Kernel, where inference should be performed (submission sample file provided) and prediction should be prepared as an output file from the Kernel.\n\n\nIn the **Resources** section I provide a short list of various resources for GAN and DeepFake, with blog posts, Kaggle Kernels and Github repos. \n\n"},{"metadata":{},"cell_type":"markdown","source":"\n"},{"metadata":{},"cell_type":"markdown","source":"# Preliminary data exploration"},{"metadata":{},"cell_type":"markdown","source":"## Load packages"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm_notebook\n%matplotlib inline \nimport cv2 as cv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"DATA_FOLDER = '../input/deepfake-detection-challenge'\nTRAIN_SAMPLE_FOLDER = 'train_sample_videos'\nTEST_FOLDER = 'test_videos'\n\nprint(f\"Train samples: {len(os.listdir(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER)))}\")\nprint(f\"Test samples: {len(os.listdir(os.path.join(DATA_FOLDER, TEST_FOLDER)))}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also added a face detection resource."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"FACE_DETECTION_FOLDER = '../input/haar-cascades-for-face-detection'\nprint(f\"Face detection resources: {os.listdir(FACE_DETECTION_FOLDER)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check files type\n\nHere we check the train data files extensions. Most of the files looks to have `mp4` extension, let's check if there is other extension as well."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_list = list(os.listdir(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER)))\next_dict = []\nfor file in train_list:\n file_ext = file.split('.')[1]\n if (file_ext not in ext_dict):\n ext_dict.append(file_ext)\nprint(f\"Extensions: {ext_dict}\") ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's count how many files with each extensions there are."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for file_ext in ext_dict:\n print(f\"Files with extension `{file_ext}`: {len([file for file in train_list if file.endswith(file_ext)])}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's repeat the same process for test videos folder."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test_list = list(os.listdir(os.path.join(DATA_FOLDER, TEST_FOLDER)))\next_dict = []\nfor file in test_list:\n file_ext = file.split('.')[1]\n if (file_ext not in ext_dict):\n ext_dict.append(file_ext)\nprint(f\"Extensions: {ext_dict}\")\nfor file_ext in ext_dict:\n print(f\"Files with extension `{file_ext}`: {len([file for file in train_list if file.endswith(file_ext)])}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the `json` file first."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"json_file = [file for file in train_list if file.endswith('json')][0]\nprint(f\"JSON file: {json_file}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Aparently here is a metadata file. Let's explore this JSON file."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_meta_from_json(path):\n df = pd.read_json(os.path.join(DATA_FOLDER, path, json_file))\n df = df.T\n return df\n\nmeta_train_df = get_meta_from_json(TRAIN_SAMPLE_FOLDER)\nmeta_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Meta data exploration\n\nLet's explore now the meta data in train sample. \n\n## Missing data\n\nWe start by checking for any missing values. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def missing_data(data):\n total = data.isnull().sum()\n percent = (data.isnull().sum()/data.isnull().count()*100)\n tt = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n types = []\n for col in data.columns:\n dtype = str(data[col].dtype)\n types.append(dtype)\n tt['Types'] = types\n return(np.transpose(tt))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"missing_data(meta_train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are missing data 19.25% of the samples (or 77). We suspect that actually the real data has missing original (if we generalize from the data we glimpsed). Let's check this hypothesis."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"missing_data(meta_train_df.loc[meta_train_df.label=='REAL'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed, all missing `original` data are the one associated with `REAL` label. \n\n## Unique values\n\nLet's check into more details the unique values."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def unique_values(data):\n total = data.count()\n tt = pd.DataFrame(total)\n tt.columns = ['Total']\n uniques = []\n for col in data.columns:\n unique = data[col].nunique()\n uniques.append(unique)\n tt['Uniques'] = uniques\n return(np.transpose(tt))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"unique_values(meta_train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We observe that `original` label has the same pattern for uniques values. We know that we have 77 missing data (that's why total is only 323) and we observe that we do have 209 unique examples. \n\n## Most frequent originals\n\nLet's look now to the most frequent originals uniques in train sample data. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def most_frequent_values(data):\n total = data.count()\n tt = pd.DataFrame(total)\n tt.columns = ['Total']\n items = []\n vals = []\n for col in data.columns:\n itm = data[col].value_counts().index[0]\n val = data[col].value_counts().values[0]\n items.append(itm)\n vals.append(val)\n tt['Most frequent item'] = items\n tt['Frequence'] = vals\n tt['Percent from total'] = np.round(vals / total * 100, 3)\n return(np.transpose(tt))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"most_frequent_values(meta_train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that most frequent **label** is `FAKE` (80.75%), `meawmsgiti.mp4` is the most frequent **original** (6 samples)."},{"metadata":{},"cell_type":"markdown","source":"Let's do now some data distribution visualizations."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_count(feature, title, df, size=1):\n '''\n Plot count of classes / feature\n param: feature - the feature to analyze\n param: title - title to add to the graph\n param: df - dataframe from which we plot feature's classes distribution \n param: size - default 1.\n '''\n f, ax = plt.subplots(1,1, figsize=(4*size,4))\n total = float(len(df))\n g = sns.countplot(df[feature], order = df[feature].value_counts().index[:20], palette='Set3')\n g.set_title(\"Number and percentage of {}\".format(title))\n if(size > 2):\n plt.xticks(rotation=90, size=8)\n for p in ax.patches:\n height = p.get_height()\n ax.text(p.get_x()+p.get_width()/2.,\n height + 3,\n '{:1.2f}%'.format(100*height/total),\n ha=\"center\") \n plt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_count('split', 'split (train)', meta_train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_count('label', 'label (train)', meta_train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the `REAL` are only 19.25% in train sample videos, with the `FAKE`s acounting for 80.75% of the samples. \n\n\n# Video data exploration\n\n\nIn the following we will explore some of the video data. \n\n\n## Missing video (or meta) data\n\nWe check first if the list of files in the meta info and the list from the folder are the same.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"meta = np.array(list(meta_train_df.index))\nstorage = np.array([file for file in train_list if file.endswith('mp4')])\nprint(f\"Metadata: {meta.shape[0]}, Folder: {storage.shape[0]}\")\nprint(f\"Files in metadata and not in folder: {np.setdiff1d(meta,storage,assume_unique=False).shape[0]}\")\nprint(f\"Files in folder and not in metadata: {np.setdiff1d(storage,meta,assume_unique=False).shape[0]}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize now the data. \n\nWe select first a list of fake videos.\n\n## Few fake videos"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fake_train_sample_video = list(meta_train_df.loc[meta_train_df.label=='FAKE'].sample(3).index)\nfake_train_sample_video","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From [4] ([Basic EDA Face Detection, split video, ROI](https://www.kaggle.com/marcovasquez/basic-eda-face-detection-split-video-roi)) we modified a function for displaying a selected image from a video."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def display_image_from_video(video_path):\n '''\n input: video_path - path for video\n process:\n 1. perform a video capture from the video\n 2. read the image\n 3. display the image\n '''\n capture_image = cv.VideoCapture(video_path) \n ret, frame = capture_image.read()\n fig = plt.figure(figsize=(10,10))\n ax = fig.add_subplot(111)\n frame = cv.cvtColor(frame, cv.COLOR_BGR2RGB)\n ax.imshow(frame)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for video_file in fake_train_sample_video:\n display_image_from_video(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, video_file))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's try now the same for few of the images that are real. \n\n\n## Few real videos"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"real_train_sample_video = list(meta_train_df.loc[meta_train_df.label=='REAL'].sample(3).index)\nreal_train_sample_video","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for video_file in real_train_sample_video:\n display_image_from_video(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, video_file))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Videos with same original\n\nLet's look now to set of samples with the same original."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"meta_train_df['original'].value_counts()[0:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We pick one of the originals with largest number of samples. \n\nWe also modify our visualization function to work with multiple images."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def display_image_from_video_list(video_path_list, video_folder=TRAIN_SAMPLE_FOLDER):\n '''\n input: video_path_list - path for video\n process:\n 0. for each video in the video path list\n 1. perform a video capture from the video\n 2. read the image\n 3. display the image\n '''\n plt.figure()\n fig, ax = plt.subplots(2,3,figsize=(16,8))\n # we only show images extracted from the first 6 videos\n for i, video_file in enumerate(video_path_list[0:6]):\n video_path = os.path.join(DATA_FOLDER, video_folder,video_file)\n capture_image = cv.VideoCapture(video_path) \n ret, frame = capture_image.read()\n frame = cv.cvtColor(frame, cv.COLOR_BGR2RGB)\n ax[i//3, i%3].imshow(frame)\n ax[i//3, i%3].set_title(f\"Video: {video_file}\")\n ax[i//3, i%3].axis('on')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"same_original_fake_train_sample_video = list(meta_train_df.loc[meta_train_df.original=='meawmsgiti.mp4'].index)\ndisplay_image_from_video_list(same_original_fake_train_sample_video)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look now to a different selection of videos with the same original. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"same_original_fake_train_sample_video = list(meta_train_df.loc[meta_train_df.original=='atvmxvwyns.mp4'].index)\ndisplay_image_from_video_list(same_original_fake_train_sample_video)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"same_original_fake_train_sample_video = list(meta_train_df.loc[meta_train_df.original=='qeumxirsme.mp4'].index)\ndisplay_image_from_video_list(same_original_fake_train_sample_video)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"same_original_fake_train_sample_video = list(meta_train_df.loc[meta_train_df.original=='kgbkktcjxf.mp4'].index)\ndisplay_image_from_video_list(same_original_fake_train_sample_video)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test video files\n\nLet's also look to few of the test data files."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_videos = pd.DataFrame(list(os.listdir(os.path.join(DATA_FOLDER, TEST_FOLDER))), columns=['video'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_videos.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize now one of the videos."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_image_from_video(os.path.join(DATA_FOLDER, TEST_FOLDER, test_videos.iloc[0].video))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look to some more videos from test set."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_image_from_video_list(test_videos.sample(6).video, TEST_FOLDER)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Face detection \n\nFrom [5] ([Face Detection using OpenCV](https://www.kaggle.com/serkanpeldek/face-detection-with-opencv)) by [@serkanpeldek](https://www.kaggle.com/serkanpeldek) we got and slightly modified the functions to extract face, profile face, eyes and smile. \n\nThe class ObjectDetector initialize the cascade classifier (using the imported resource). The function **detect** uses a method of the CascadeClassifier to detect objects into images - in this case the face, eye, smile or profile face."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class ObjectDetector():\n '''\n Class for Object Detection\n '''\n def __init__(self,object_cascade_path):\n '''\n param: object_cascade_path - path for the *.xml defining the parameters for {face, eye, smile, profile}\n detection algorithm\n source of the haarcascade resource is: https://github.com/opencv/opencv/tree/master/data/haarcascades\n '''\n\n self.objectCascade=cv.CascadeClassifier(object_cascade_path)\n\n\n def detect(self, image, scale_factor=1.3,\n min_neighbors=5,\n min_size=(20,20)):\n '''\n Function return rectangle coordinates of object for given image\n param: image - image to process\n param: scale_factor - scale factor used for object detection\n param: min_neighbors - minimum number of parameters considered during object detection\n param: min_size - minimum size of bounding box for object detected\n '''\n rects=self.objectCascade.detectMultiScale(image,\n scaleFactor=scale_factor,\n minNeighbors=min_neighbors,\n minSize=min_size)\n return rects","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We load the resources for frontal face, eye, smile and profile face detection. \n\nThen we initialize the `ObjectDetector` objects defined above with the respective resources, to use CascadeClassfier for each specific task."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Frontal face, profile, eye and smile haar cascade loaded\nfrontal_cascade_path= os.path.join(FACE_DETECTION_FOLDER,'haarcascade_frontalface_default.xml')\neye_cascade_path= os.path.join(FACE_DETECTION_FOLDER,'haarcascade_eye.xml')\nprofile_cascade_path= os.path.join(FACE_DETECTION_FOLDER,'haarcascade_profileface.xml')\nsmile_cascade_path= os.path.join(FACE_DETECTION_FOLDER,'haarcascade_smile.xml')\n\n#Detector object created\n# frontal face\nfd=ObjectDetector(frontal_cascade_path)\n# eye\ned=ObjectDetector(eye_cascade_path)\n# profile face\npd=ObjectDetector(profile_cascade_path)\n# smile\nsd=ObjectDetector(smile_cascade_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also define a function for detection and display of all these specific objects. \n\nThe function call the **detect** method of the **ObjectDetector** object. For each object we are using a different shape and color, as following:\n* Frontal face: green rectangle; \n* Eye: red circle; \n* Smile: red rectangle; \n* Profile face: blue rectangle. \n\nNote: due to a huge amount of false positive, we deactivate for now the smile detector."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def detect_objects(image, scale_factor, min_neighbors, min_size):\n '''\n Objects detection function\n Identify frontal face, eyes, smile and profile face and display the detected objects over the image\n param: image - the image extracted from the video\n param: scale_factor - scale factor parameter for `detect` function of ObjectDetector object\n param: min_neighbors - min neighbors parameter for `detect` function of ObjectDetector object\n param: min_size - minimum size parameter for f`detect` function of ObjectDetector object\n '''\n \n image_gray=cv.cvtColor(image, cv.COLOR_BGR2GRAY)\n\n\n eyes=ed.detect(image_gray,\n scale_factor=scale_factor,\n min_neighbors=min_neighbors,\n min_size=(int(min_size[0]/2), int(min_size[1]/2)))\n\n for x, y, w, h in eyes:\n #detected eyes shown in color image\n cv.circle(image,(int(x+w/2),int(y+h/2)),(int((w + h)/4)),(0, 0,255),3)\n \n # deactivated due to many false positive\n #smiles=sd.detect(image_gray,\n # scale_factor=scale_factor,\n # min_neighbors=min_neighbors,\n # min_size=(int(min_size[0]/2), int(min_size[1]/2)))\n\n #for x, y, w, h in smiles:\n # #detected smiles shown in color image\n # cv.rectangle(image,(x,y),(x+w, y+h),(0, 0,255),3)\n\n\n profiles=pd.detect(image_gray,\n scale_factor=scale_factor,\n min_neighbors=min_neighbors,\n min_size=min_size)\n\n for x, y, w, h in profiles:\n #detected profiles shown in color image\n cv.rectangle(image,(x,y),(x+w, y+h),(255, 0,0),3)\n\n faces=fd.detect(image_gray,\n scale_factor=scale_factor,\n min_neighbors=min_neighbors,\n min_size=min_size)\n\n for x, y, w, h in faces:\n #detected faces shown in color image\n cv.rectangle(image,(x,y),(x+w, y+h),(0, 255,0),3)\n\n # image\n fig = plt.figure(figsize=(10,10))\n ax = fig.add_subplot(111)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n ax.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following function extracts an image from a video and then call the function that extracts the face rectangle from the image and display the rectangle above the image."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def extract_image_objects(video_file, video_set_folder=TRAIN_SAMPLE_FOLDER):\n '''\n Extract one image from the video and then perform face/eyes/smile/profile detection on the image\n param: video_file - the video from which to extract the image from which we extract the face\n '''\n video_path = os.path.join(DATA_FOLDER, video_set_folder,video_file)\n capture_image = cv.VideoCapture(video_path) \n ret, frame = capture_image.read()\n #frame = cv.cvtColor(frame, cv.COLOR_BGR2RGB)\n detect_objects(image=frame, \n scale_factor=1.3, \n min_neighbors=5, \n min_size=(50, 50)) \n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We apply the function for face detection for a selection of images from train sample videos."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"same_original_fake_train_sample_video = list(meta_train_df.loc[meta_train_df.original=='kgbkktcjxf.mp4'].index)\nfor video_file in same_original_fake_train_sample_video[1:4]:\n print(video_file)\n extract_image_objects(video_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_subsample_video = list(meta_train_df.sample(3).index)\nfor video_file in train_subsample_video:\n print(video_file)\n extract_image_objects(video_file)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look to a small collection of samples from test videos."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"subsample_test_videos = list(test_videos.sample(3).video)\nfor video_file in subsample_test_videos:\n print(video_file)\n extract_image_objects(video_file, TEST_FOLDER)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can observe that in some cases, when the subject is not looking frontaly or when the luminosity is low, the algorithm for face detection is not detecting the face or eyes correctly. Due to a large amount of false positive, we deactivated for now the smile detector."},{"metadata":{},"cell_type":"markdown","source":"## Play video files \n\nFrom [Play video and processing](https://www.kaggle.com/hamditarek/play-video-and-processing) Kernel by [@hamditarek](https://www.kaggle.com/hamditarek) we learned how to play video files in a Kaggle Kernel. \nLet's look to few fake videos."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fake_videos = list(meta_train_df.loc[meta_train_df.label=='FAKE'].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from IPython.display import HTML\nfrom base64 import b64encode\n\ndef play_video(video_file, subset=TRAIN_SAMPLE_FOLDER):\n '''\n Display video\n param: video_file - the name of the video file to display\n param: subset - the folder where the video file is located (can be TRAIN_SAMPLE_FOLDER or TEST_Folder)\n '''\n video_url = open(os.path.join(DATA_FOLDER, subset,video_file),'rb').read()\n data_url = \"data:video/mp4;base64,\" + b64encode(video_url).decode()\n return HTML(\"\"\"\"\"\" % data_url)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"play_video(fake_videos[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"play_video(fake_videos[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"play_video(fake_videos[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[4])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[12])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[15])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"play_video(fake_videos[18])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From visual inspection of these fakes videos, in some cases is very easy to spot the anomalies created when engineering the deep fake, in some cases is more difficult."},{"metadata":{},"cell_type":"markdown","source":"# Resources \n\nThis resources list is not exhaustive, it provides just a starting point for Kagglers that would like to join this competition in order to learn, like myself. \n\nI provide here technical articles links, small blog articles, links to Github projects and some inspirational Kaggle Kernels about DeepFake, DCGANs and GANs. \n\n**Blogs, Technical Articles**\n\n* Towards Data Science DeepFakes subject selection: [DeepFakes](https://towardsdatascience.com/tagged/deepfakes) \n\n* An introduction to DeepFakes from Towards Data Science: [Deepfakes: The Ugly, and The Good](https://towardsdatascience.com/deepfakes-the-ugly-and-the-good-49115643d8dd) \n\n* Introduction to GAN from Towards Data Science (with code): [GANs from Scratch 1: A deep introduction. With code in PyTorch and TensorFlow](https://medium.com/ai-society/gans-from-scratch-1-a-deep-introduction-with-code-in-pytorch-and-tensorflow-cb03cdcdba0f) \n\n* A cGAN introduction from Mastering Data Science (with code): [How to Develop a Conditional GAN (cGAN) From Scratch](https://machinelearningmastery.com/how-to-develop-a-conditional-generative-adversarial-network-from-scratch/) \n\n* An introduction to GANs on Analytics Vidhya, [GANs — A Brief Introduction to Generative Adversarial Networks](https://medium.com/analytics-vidhya/gans-a-brief-introduction-to-generative-adversarial-networks-f06216c7200e?)\n\n**Tutorials**\n\n* A nice tutorial for GAN using Pytorch: [DCGAN Faces Tutorial](https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) \n\n* A tutorial for GANs: [A Beginner's Guide to Generative Adversarial Networks (GANs)](https://pathmind.com/wiki/generative-adversarial-network-gan) \n\n* Deep Convolutional Generative Adversial Networks, Tensorflow, [DCGAN](https://www.tensorflow.org/tutorials/generative/dcgan) \n\n* A tutorial for video and image colorization and resolution improvement using fast.ai and PyTorch recommended by [@init27](https://www.kaggle.com/init27), [Decrappification, DeOldification, and Super Resolution](https://www.fast.ai/2019/05/03/decrappify/) \n\n* A tutorial from OpenCV for face detection using Cascade Classifiers, [Cascade Classifier](https://docs.opencv.org/3.4/db/d28/tutorial_cascade_classifier.html) \n\n**Kaggle Kernels**\n\n* A very good intro to GAN, by [@nanashi](http://kaggle/nanashi): [GAN Introduction](https://www.kaggle.com/jesucristo/gan-introduction) \n\n* A GAN developed for Dog face generation competition by [@cdeotte](http://kaggle/cdeotte): [Dog Memorizer GAN](https://www.kaggle.com/cdeotte/dog-memorizer-gan) \n\n* A Kernel for Face detection using OpenCV with Haarcascade, by [@serkanpeldek](https://www.kaggle.com/serkanpeldek), [Face Detection with OpenCV](https://www.kaggle.com/serkanpeldek/face-detection-with-opencv) \n\n* Play video and processing, by [@hamditarek](https://www.kaggle.com/hamditarek), https://www.kaggle.com/hamditarek/play-video-and-processing \n\n\n**Github repos**\n\n* Github topic for DeeFakes: [deepfakes](https://github.com/topics/) \n\n* A Github project using Pytorch: [Faceswap-Deepfake-Pytorch](https://github.com/Oldpan/Faceswap-Deepfake-Pytorch) \n\n* A Github project for GAN with PyTorch: [PyTorch-GAN](https://github.com/eriklindernoren/PyTorch-GAN) \n\n* A Github recommended by [@shwetagoyal4](https://www.kaggle.com/shwetagoyal4), [Generative-model-using-PyTorch](https://github.com/Shwetago/Generative-model-using-PyTorch)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# References\n\n[1] Deepfake, Wikipedia, https://en.wikipedia.org/wiki/Deepfake \n[2] Google DeepFake Database, Endgadget, https://www.engadget.com/2019/09/25/google-deepfake-database/ \n[3] A quick look at the first frame of each video, https://www.kaggle.com/brassmonkey381/a-quick-look-at-the-first-frame-of-each-video \n[4] Basic EDA Face Detection, split video, ROI, https://www.kaggle.com/marcovasquez/basic-eda-face-detection-split-video-roi \n[5] Face Detection with OpenCV, https://www.kaggle.com/serkanpeldek/face-detection-with-opencv \n[6] Play video and processing, https://www.kaggle.com/hamditarek/play-video-and-processing/\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}