{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

DeepFake Starter Kit

\n\n

Content

 \n\n* Introduction \n* Preliminary Data Exploration \n * Load the Packages \n * Import Utility Scripts \n * Load the Data \n * Check Files Type\n* Metadata Exploration \n * Missing Data \n * Unique Values \n * Most Frequent Originals \n* Video Data Exploration \n * Missing Video data or Metadata \n * Few Fake Videos \n * Few Real Videos \n * Videos with Same Original\n * Test Video Files \n* Face Detection \n * Haar Cascades \n * MTCNN \n* Play Video Files \n* References ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T13:24:35.412025Z","iopub.execute_input":"2023-08-30T13:24:35.412614Z","iopub.status.idle":"2023-08-30T13:24:35.427685Z","shell.execute_reply.started":"2023-08-30T13:24:35.41256Z","shell.execute_reply":"2023-08-30T13:24:35.425264Z"}}},{"cell_type":"markdown","source":"# Introduction \n\nDeepFake is composed from Deep Learning and Fake and means taking one person from an image or video and replacing with someone else likeness using technology such as Deep Artificial Neural Networks [1]. Large companies like Google invest very much in fighting the DeepFake, this including release of large datasets to help training models to counter this threat [2].The phenomen invades rapidly the film industry and threatens to compromise news agencies. Large digital companies, including content providers and social platforms are in the frontrun of fighting Deep Fakes. GANs that generate DeepFakes becomes better every day and, of course, if you include in a new GAN model all the information we collected until now how to combat various existent models, we create a model that cannot be beatten by the existing ones.\n\nIn the **Data Exploration** section we perform a (partial) Exploratory Data Analysis (EDA) on the training and testing data. After we are checking the files types, we are focusing first on the **metadata** files, which we are exploring in details, after we are importing in dataframes. Then, we move to explore video files, by looking first to a sample of fake videos, then to real videos. After that, we are also exploring few of the videos with the same origin. We are visualizing one frame extracted from the video, for both real and fake videos. Then we are also playing few videos.\nThen, we move to perform face (and other objects from the persons in the videos) extraction. More precisely, we are using OpenCV Haar Cascade resources to identify frontal face, eyes, smile and profile face from still images in the videos.\n\n**Important note**: The data we analyze here is just a very small sample of data. The competition specifies that the train data is provided as archived chunks. Training of models should pe performed offline using the data provided by Kaggle as archives, models should be loaded (max 1GB memory) in a Kernel, where inference should be performed (submission sample file provided) and prediction should be prepared as an output file from the Kernel.\n\nIn the Resources section I provide a short list of various resources for GAN and DeepFake, with blog posts, Kaggle Kernels and Github repos.","metadata":{}},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"# Preliminary Data Exploration ","metadata":{}},{"cell_type":"markdown","source":"## Load Packages","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfrom tqdm import tqdm_notebook\nimport cv2 as cv","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:40:55.729674Z","iopub.execute_input":"2023-09-16T07:40:55.730160Z","iopub.status.idle":"2023-09-16T07:40:56.556433Z","shell.execute_reply.started":"2023-09-16T07:40:55.730094Z","shell.execute_reply":"2023-09-16T07:40:56.555588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import Utility Scripts","metadata":{}},{"cell_type":"code","source":"from data_quality_stats import missing_data, unique_values, most_frequent_values\nfrom plot_style_utils import set_color_map, plot_count\nfrom video_utils import display_image_from_video, display_images_from_video_list, play_video\nfrom face_object_detection import CascadeObjectDetector, FaceObjectDetector\nfrom face_detection_mtcnn import MTCNNFaceDetector","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:15.534699Z","iopub.execute_input":"2023-09-16T07:41:15.535035Z","iopub.status.idle":"2023-09-16T07:41:17.582606Z","shell.execute_reply.started":"2023-09-16T07:41:15.534944Z","shell.execute_reply":"2023-09-16T07:41:17.581939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load Data","metadata":{}},{"cell_type":"code","source":"DATA_FOLDER = '../input/deepfake-detection-challenge'\nTRAIN_SAMPLE_FOLDER = 'train_sample_videos'\nTEST_FOLDER = 'test_videos'\n\nprint(f\"Train samples: {len(os.listdir(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER)))}\")\nprint(f\"Test samples: {len(os.listdir(os.path.join(DATA_FOLDER, TEST_FOLDER)))}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:19.261061Z","iopub.execute_input":"2023-09-16T07:41:19.261557Z","iopub.status.idle":"2023-09-16T07:41:19.465513Z","shell.execute_reply.started":"2023-09-16T07:41:19.261501Z","shell.execute_reply":"2023-09-16T07:41:19.464412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We also add face detection resources.","metadata":{}},{"cell_type":"code","source":"FACE_DETECTION_FOLDER = '../input/haar-cascades-for-face-detection'\nprint(f\"Face detection resources: {os.listdir(FACE_DETECTION_FOLDER)}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:22.126347Z","iopub.execute_input":"2023-09-16T07:41:22.126691Z","iopub.status.idle":"2023-09-16T07:41:22.135587Z","shell.execute_reply.started":"2023-09-16T07:41:22.126605Z","shell.execute_reply":"2023-09-16T07:41:22.134787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"## Check Files Type \n\nHere we check the train data files extensions. Most of the files looks to have mp4 extension, let's check if there is other extension as well.","metadata":{}},{"cell_type":"code","source":"train_list = list(os.listdir(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER)))\next_dict = []\nfor file in train_list:\n file_ext = file.split('.')[1]\n if (file_ext not in ext_dict):\n ext_dict.append(file_ext)\nprint(f\"Extensions: {ext_dict}\") ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:25.808309Z","iopub.execute_input":"2023-09-16T07:41:25.808782Z","iopub.status.idle":"2023-09-16T07:41:25.815996Z","shell.execute_reply.started":"2023-09-16T07:41:25.808725Z","shell.execute_reply":"2023-09-16T07:41:25.814612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's count how many files with each extensions there are.","metadata":{}},{"cell_type":"code","source":"for file_ext in ext_dict:\n print(f\"Files with extension `{file_ext}`: {len([file for file in train_list if file.endswith(file_ext)])}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:28.826382Z","iopub.execute_input":"2023-09-16T07:41:28.826710Z","iopub.status.idle":"2023-09-16T07:41:28.833256Z","shell.execute_reply.started":"2023-09-16T07:41:28.826653Z","shell.execute_reply":"2023-09-16T07:41:28.831791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's repeat the same process for test videos folder.","metadata":{}},{"cell_type":"code","source":"json_file = [file for file in train_list if file.endswith('json')][0]\nprint(f\"JSON file: {json_file}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:31.427479Z","iopub.execute_input":"2023-09-16T07:41:31.427823Z","iopub.status.idle":"2023-09-16T07:41:31.434498Z","shell.execute_reply.started":"2023-09-16T07:41:31.427771Z","shell.execute_reply":"2023-09-16T07:41:31.433073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aparently here is a metadata file. Let's explore this JSON file.","metadata":{}},{"cell_type":"code","source":"def get_meta_from_json(path):\n df = pd.read_json(os.path.join(DATA_FOLDER, path, json_file))\n df = df.T\n return df\n\nmeta_train_df = get_meta_from_json(TRAIN_SAMPLE_FOLDER)\nmeta_train_df.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:34.420284Z","iopub.execute_input":"2023-09-16T07:41:34.420716Z","iopub.status.idle":"2023-09-16T07:41:35.118510Z","shell.execute_reply.started":"2023-09-16T07:41:34.420658Z","shell.execute_reply":"2023-09-16T07:41:35.117589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"# Metadata Exploration ","metadata":{}},{"cell_type":"markdown","source":"Let's explore now the meta data in train sample.\n\n## Missing data ","metadata":{}},{"cell_type":"code","source":"missing_data(meta_train_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:38.182511Z","iopub.execute_input":"2023-09-16T07:41:38.182831Z","iopub.status.idle":"2023-09-16T07:41:38.308170Z","shell.execute_reply.started":"2023-09-16T07:41:38.182779Z","shell.execute_reply":"2023-09-16T07:41:38.307214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, all missing `original` data are the one associated with `REAL` label.","metadata":{}},{"cell_type":"markdown","source":"## Unique data ","metadata":{}},{"cell_type":"code","source":"unique_values(meta_train_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:41.008579Z","iopub.execute_input":"2023-09-16T07:41:41.008937Z","iopub.status.idle":"2023-09-16T07:41:41.025062Z","shell.execute_reply.started":"2023-09-16T07:41:41.008870Z","shell.execute_reply":"2023-09-16T07:41:41.023439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe that original label has the same pattern for uniques values. We know that we have 77 missing data (that's why total is only 323) and we observe that we do have 209 unique examples.","metadata":{}},{"cell_type":"markdown","source":"## Most frequent originals ","metadata":{}},{"cell_type":"code","source":"most_frequent_values(meta_train_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:44.066541Z","iopub.execute_input":"2023-09-16T07:41:44.066882Z","iopub.status.idle":"2023-09-16T07:41:44.089554Z","shell.execute_reply.started":"2023-09-16T07:41:44.066831Z","shell.execute_reply":"2023-09-16T07:41:44.088515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that most frequent label is `FAKE` (80.75%), `meawmsgiti.mp4` is the most frequent original (6 samples).\n\nLet's do now some data distribution visualizations.","metadata":{}},{"cell_type":"code","source":"color_list = ['#4166AA', '#06BDDD', '#83CEEC', '#EDE8E4', '#C2AFA8']\ncmap_custom = set_color_map(color_list)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:41:46.962957Z","iopub.execute_input":"2023-09-16T07:41:46.963295Z","iopub.status.idle":"2023-09-16T07:41:47.033718Z","shell.execute_reply.started":"2023-09-16T07:41:46.963236Z","shell.execute_reply":"2023-09-16T07:41:47.032744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_count(meta_train_df, 'split', 'split (train)', color_list, size=1)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:41:49.458404Z","iopub.execute_input":"2023-09-16T07:41:49.458893Z","iopub.status.idle":"2023-09-16T07:41:49.598186Z","shell.execute_reply.started":"2023-09-16T07:41:49.458841Z","shell.execute_reply":"2023-09-16T07:41:49.597238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_count(meta_train_df, 'label', 'label (train)', color_list, size=2)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:41:52.542815Z","iopub.execute_input":"2023-09-16T07:41:52.543103Z","iopub.status.idle":"2023-09-16T07:41:52.675670Z","shell.execute_reply.started":"2023-09-16T07:41:52.543055Z","shell.execute_reply":"2023-09-16T07:41:52.674668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the `REAL` are only 19.25% in train sample videos, with the `FAKE` acounting for 80.75% of the samples.","metadata":{}},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"# Video Data Exploration","metadata":{}},{"cell_type":"markdown","source":"In the following we will explore some of the video data.\n\n## Missing video (or meta) data \n\nWe check first if the list of files in the meta info and the list from the folder are the same. ","metadata":{}},{"cell_type":"code","source":"meta = np.array(list(meta_train_df.index))\nstorage = np.array([file for file in train_list if file.endswith('mp4')])\nprint(f\"Metadata: {meta.shape[0]}, Folder: {storage.shape[0]}\")\nprint(f\"Files in metadata and not in folder: {np.setdiff1d(meta,storage,assume_unique=False).shape[0]}\")\nprint(f\"Files in folder and not in metadata: {np.setdiff1d(storage,meta,assume_unique=False).shape[0]}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:56.137687Z","iopub.execute_input":"2023-09-16T07:41:56.137982Z","iopub.status.idle":"2023-09-16T07:41:56.146572Z","shell.execute_reply.started":"2023-09-16T07:41:56.137933Z","shell.execute_reply":"2023-09-16T07:41:56.145350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize now the data.\n\nWe select first a list of fake videos.","metadata":{}},{"cell_type":"code","source":"fake_train_sample_video = list(meta_train_df.loc[meta_train_df.label=='FAKE'].sample(3).index)\nfake_train_sample_video","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:41:58.972976Z","iopub.execute_input":"2023-09-16T07:41:58.973328Z","iopub.status.idle":"2023-09-16T07:41:58.982743Z","shell.execute_reply.started":"2023-09-16T07:41:58.973265Z","shell.execute_reply":"2023-09-16T07:41:58.982221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the utility script `video_utils` we are using a function for displaying a selected image from a video.","metadata":{}},{"cell_type":"code","source":"for video_file in fake_train_sample_video:\n display_image_from_video(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, video_file))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:01.763606Z","iopub.execute_input":"2023-09-16T07:42:01.764195Z","iopub.status.idle":"2023-09-16T07:42:03.149038Z","shell.execute_reply.started":"2023-09-16T07:42:01.764137Z","shell.execute_reply":"2023-09-16T07:42:03.148060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's try now the same for few of the images that are real.\n\n## Few real videos","metadata":{}},{"cell_type":"code","source":"real_train_sample_video = list(meta_train_df.loc[meta_train_df.label=='REAL'].sample(3).index)\nreal_train_sample_video","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:07.405089Z","iopub.execute_input":"2023-09-16T07:42:07.405380Z","iopub.status.idle":"2023-09-16T07:42:07.412644Z","shell.execute_reply.started":"2023-09-16T07:42:07.405332Z","shell.execute_reply":"2023-09-16T07:42:07.411859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for video_file in real_train_sample_video:\n display_image_from_video(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, video_file))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:10.215733Z","iopub.execute_input":"2023-09-16T07:42:10.216026Z","iopub.status.idle":"2023-09-16T07:42:11.434761Z","shell.execute_reply.started":"2023-09-16T07:42:10.215974Z","shell.execute_reply":"2023-09-16T07:42:11.433554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Videos with same original \n\nLet's look now to set of samples with the same original.","metadata":{}},{"cell_type":"code","source":"meta_train_df['original'].value_counts()[0:5]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:16.201007Z","iopub.execute_input":"2023-09-16T07:42:16.201543Z","iopub.status.idle":"2023-09-16T07:42:16.210604Z","shell.execute_reply.started":"2023-09-16T07:42:16.201485Z","shell.execute_reply":"2023-09-16T07:42:16.209606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We pick one of the originals with largest number of samples.\n\nWe also modify our visualization function to work with multiple images.","metadata":{}},{"cell_type":"code","source":"same_original_fake_train_sample_video = \\\n list(meta_train_df.loc[meta_train_df.original=='meawmsgiti.mp4'].index)\n\ndisplay_images_from_video_list(video_path_list=same_original_fake_train_sample_video,\n data_folder=DATA_FOLDER,\n video_folder=TRAIN_SAMPLE_FOLDER)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:19.093308Z","iopub.execute_input":"2023-09-16T07:42:19.093587Z","iopub.status.idle":"2023-09-16T07:42:20.876384Z","shell.execute_reply.started":"2023-09-16T07:42:19.093537Z","shell.execute_reply":"2023-09-16T07:42:20.875756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look now to a different selection of videos with the same original.","metadata":{}},{"cell_type":"code","source":"same_original_fake_train_sample_video = \\\n list(meta_train_df.loc[meta_train_df.original=='atvmxvwyns.mp4'].index)\n\ndisplay_images_from_video_list(video_path_list=same_original_fake_train_sample_video,\n data_folder=DATA_FOLDER,\n video_folder=TRAIN_SAMPLE_FOLDER)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:32.010608Z","iopub.execute_input":"2023-09-16T07:42:32.011103Z","iopub.status.idle":"2023-09-16T07:42:33.739939Z","shell.execute_reply.started":"2023-09-16T07:42:32.011017Z","shell.execute_reply":"2023-09-16T07:42:33.739055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"same_original_fake_train_sample_video = \\\n list(meta_train_df.loc[meta_train_df.original=='qeumxirsme.mp4'].index)\n\ndisplay_images_from_video_list(video_path_list=same_original_fake_train_sample_video,\n data_folder=DATA_FOLDER,\n video_folder=TRAIN_SAMPLE_FOLDER)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:38.563526Z","iopub.execute_input":"2023-09-16T07:42:38.563872Z","iopub.status.idle":"2023-09-16T07:42:40.093144Z","shell.execute_reply.started":"2023-09-16T07:42:38.563817Z","shell.execute_reply":"2023-09-16T07:42:40.091945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"same_original_fake_train_sample_video = \\\n list(meta_train_df.loc[meta_train_df.original=='kgbkktcjxf.mp4'].index) \n\ndisplay_images_from_video_list(video_path_list=same_original_fake_train_sample_video,\n data_folder=DATA_FOLDER,\n video_folder=TRAIN_SAMPLE_FOLDER)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:43.382267Z","iopub.execute_input":"2023-09-16T07:42:43.382775Z","iopub.status.idle":"2023-09-16T07:42:44.969153Z","shell.execute_reply.started":"2023-09-16T07:42:43.382713Z","shell.execute_reply":"2023-09-16T07:42:44.968071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test video files \n\nLet's also look to few of the test data files.","metadata":{}},{"cell_type":"code","source":"test_videos = pd.DataFrame(list(os.listdir(os.path.join(DATA_FOLDER, TEST_FOLDER))), columns=['video'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:47.799833Z","iopub.execute_input":"2023-09-16T07:42:47.800140Z","iopub.status.idle":"2023-09-16T07:42:47.813186Z","shell.execute_reply.started":"2023-09-16T07:42:47.800080Z","shell.execute_reply":"2023-09-16T07:42:47.811950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_videos.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:49.665690Z","iopub.execute_input":"2023-09-16T07:42:49.665997Z","iopub.status.idle":"2023-09-16T07:42:49.675689Z","shell.execute_reply.started":"2023-09-16T07:42:49.665935Z","shell.execute_reply":"2023-09-16T07:42:49.674736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize now one of the videos.","metadata":{}},{"cell_type":"code","source":"display_image_from_video(os.path.join(DATA_FOLDER, TEST_FOLDER, test_videos.iloc[0].video))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:53.177880Z","iopub.execute_input":"2023-09-16T07:42:53.178173Z","iopub.status.idle":"2023-09-16T07:42:53.612365Z","shell.execute_reply.started":"2023-09-16T07:42:53.178114Z","shell.execute_reply":"2023-09-16T07:42:53.611682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look to some more videos from test set.","metadata":{}},{"cell_type":"code","source":"display_images_from_video_list(test_videos.sample(6).video, DATA_FOLDER, TEST_FOLDER)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:42:56.820379Z","iopub.execute_input":"2023-09-16T07:42:56.820812Z","iopub.status.idle":"2023-09-16T07:42:58.679195Z","shell.execute_reply.started":"2023-09-16T07:42:56.820768Z","shell.execute_reply":"2023-09-16T07:42:58.677993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"# Face detection \n\nFor face detection we will use two different approaches. In the first one, we will use Haar cascades and in the second one we will use MTCNN models.\n\n## Haar Cascades\n\nIn the first approach for face detection we will use the FaceObjectDetector class from `face_object_detection` utility script. This was modified from [5] (Face Detection using OpenCV) by @serkanpeldek we got and slightly modified the functions to extract face, profile face, eyes and smile.\n\nThe class CascadeObjectDetector initialize the cascade classifier (using the imported resource). The function detect uses a method of the CascadeClassifier to detect objects into images - in this case the face, eye, smile or profile face.\n\nWe load the resources for frontal face, eye, smile and profile face in an object of type FaceObjectDetector which will initialize specialized CascadeObjectDetector objects. \n","metadata":{}},{"cell_type":"code","source":"face_object_detector = FaceObjectDetector(FACE_DETECTION_FOLDER)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:45:11.214715Z","iopub.execute_input":"2023-09-16T07:45:11.214974Z","iopub.status.idle":"2023-09-16T07:45:11.312705Z","shell.execute_reply.started":"2023-09-16T07:45:11.214940Z","shell.execute_reply":"2023-09-16T07:45:11.311940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We defined also the `detect` method of the `FaceObjectDetector` object. For each object to extract we are using a different shape and color, as following:\n\n* Frontal face: green rectangle; \n* Eye: red circle; \n* Smile: red rectangle; \n* Profile face: blue rectangle. \n\n**Note**: due to a huge amount of false positive, we deactivate for now the smile detector.","metadata":{}},{"cell_type":"markdown","source":"The function `extract_image_objects`, as well defined as a member function of `FaceObjectDetector` in `face_object_detection` utility script, extracts an image from a video and then call the function that extracts the face rectangle from the image and display the rectangle above the image.","metadata":{}},{"cell_type":"markdown","source":"We apply the function for face detection for a selection of images from train sample videos.","metadata":{}},{"cell_type":"code","source":"same_original_fake_train_sample_video = \\\n list(meta_train_df.loc[meta_train_df.original=='kgbkktcjxf.mp4'].index)\n\nfor video_file in same_original_fake_train_sample_video[1:4]:\n print(video_file)\n face_object_detector.extract_image_objects(video_file=video_file,\n data_folder=DATA_FOLDER,\n video_set_folder=TRAIN_SAMPLE_FOLDER,\n show_smile=False \n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:45:26.359660Z","iopub.execute_input":"2023-09-16T07:45:26.359916Z","iopub.status.idle":"2023-09-16T07:45:27.954226Z","shell.execute_reply.started":"2023-09-16T07:45:26.359872Z","shell.execute_reply":"2023-09-16T07:45:27.953463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's do the same by enabling smile detection as well.","metadata":{}},{"cell_type":"code","source":"for video_file in same_original_fake_train_sample_video[1:2]:\n print(video_file)\n face_object_detector.extract_image_objects(video_file=video_file,\n data_folder=DATA_FOLDER,\n video_set_folder=TRAIN_SAMPLE_FOLDER,\n show_smile=True \n )","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:45:31.011165Z","iopub.execute_input":"2023-09-16T07:45:31.011488Z","iopub.status.idle":"2023-09-16T07:45:31.633404Z","shell.execute_reply.started":"2023-09-16T07:45:31.011428Z","shell.execute_reply":"2023-09-16T07:45:31.632766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, the smile detection gives too many false positives.","metadata":{}},{"cell_type":"code","source":"train_subsample_video = list(meta_train_df.sample(3).index)\nfor video_file in train_subsample_video:\n print(video_file)\n face_object_detector.extract_image_objects(video_file=video_file,\n data_folder=DATA_FOLDER,\n video_set_folder=TRAIN_SAMPLE_FOLDER,\n show_smile=False \n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:45:35.546991Z","iopub.execute_input":"2023-09-16T07:45:35.547305Z","iopub.status.idle":"2023-09-16T07:45:37.160917Z","shell.execute_reply.started":"2023-09-16T07:45:35.547245Z","shell.execute_reply":"2023-09-16T07:45:37.159774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look to a small collection of samples from test videos.","metadata":{}},{"cell_type":"code","source":"subsample_test_videos = list(test_videos.sample(3).video)\nfor video_file in subsample_test_videos:\n print(video_file)\n face_object_detector.extract_image_objects(video_file=video_file,\n data_folder=DATA_FOLDER,\n video_set_folder=TEST_FOLDER,\n show_smile=False \n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:45:44.533873Z","iopub.execute_input":"2023-09-16T07:45:44.534145Z","iopub.status.idle":"2023-09-16T07:45:46.452191Z","shell.execute_reply.started":"2023-09-16T07:45:44.534098Z","shell.execute_reply":"2023-09-16T07:45:46.447767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe that in some cases, when the subject is not looking frontaly or when the luminosity is low, the algorithm for face detection is not detecting the face or eyes correctly. Due to a large amount of false positive, we deactivated for now the smile detector.\n\nLet's retry now with a different algorithm, MTCNN model.","metadata":{}},{"cell_type":"markdown","source":"## MTCNN Model","metadata":{}},{"cell_type":"markdown","source":"First we `pip install` mtcnn library.","metadata":{}},{"cell_type":"code","source":"!pip install mtcnn","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:48:20.816741Z","iopub.execute_input":"2023-09-16T07:48:20.817037Z","iopub.status.idle":"2023-09-16T07:48:27.319266Z","shell.execute_reply.started":"2023-09-16T07:48:20.816983Z","shell.execute_reply":"2023-09-16T07:48:27.317971Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then we import the model from the newly installed library.","metadata":{}},{"cell_type":"code","source":"from mtcnn.mtcnn import MTCNN\nmtcnn_model = MTCNN()","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:48:42.270620Z","iopub.execute_input":"2023-09-16T07:48:42.270977Z","iopub.status.idle":"2023-09-16T07:48:47.026685Z","shell.execute_reply.started":"2023-09-16T07:48:42.270922Z","shell.execute_reply":"2023-09-16T07:48:47.025283Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the instantiated object `mtcnn_model` we initialize a `MTCNNFaceDetector` type object from the Utility Script `face_detection_mtcnn`.","metadata":{}},{"cell_type":"code","source":"from face_detection_mtcnn import MTCNNFaceDetector\nmtcnn_face_detector = MTCNNFaceDetector(mtcnn_model)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:51:28.525412Z","iopub.execute_input":"2023-09-16T07:51:28.525704Z","iopub.status.idle":"2023-09-16T07:51:28.529771Z","shell.execute_reply.started":"2023-09-16T07:51:28.525665Z","shell.execute_reply":"2023-09-16T07:51:28.528910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We prepare a path for one video to perform face detection. ","metadata":{}},{"cell_type":"code","source":"video_path = os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, fake_train_sample_video[1])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-16T07:52:19.768922Z","iopub.execute_input":"2023-09-16T07:52:19.769163Z","iopub.status.idle":"2023-09-16T07:52:19.774580Z","shell.execute_reply.started":"2023-09-16T07:52:19.769133Z","shell.execute_reply":"2023-09-16T07:52:19.773395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We run the `detect` function of `MTCNNFaceDetector`","metadata":{}},{"cell_type":"code","source":"mtcnn_face_detector.detect(video_path)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:53:21.438456Z","iopub.execute_input":"2023-09-16T07:53:21.438810Z","iopub.status.idle":"2023-09-16T07:53:23.916106Z","shell.execute_reply.started":"2023-09-16T07:53:21.438746Z","shell.execute_reply":"2023-09-16T07:53:23.914873Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's repeat this face extraction for few more images.","metadata":{}},{"cell_type":"code","source":"video_path = os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, fake_train_sample_video[0])\nmtcnn_face_detector.detect(video_path)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:54:24.213160Z","iopub.execute_input":"2023-09-16T07:54:24.213482Z","iopub.status.idle":"2023-09-16T07:54:26.343278Z","shell.execute_reply.started":"2023-09-16T07:54:24.213413Z","shell.execute_reply":"2023-09-16T07:54:26.342566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"video_path = os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER, fake_train_sample_video[2])\nmtcnn_face_detector.detect(video_path)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:54:40.015961Z","iopub.execute_input":"2023-09-16T07:54:40.016404Z","iopub.status.idle":"2023-09-16T07:54:41.238878Z","shell.execute_reply.started":"2023-09-16T07:54:40.016358Z","shell.execute_reply":"2023-09-16T07:54:41.237893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe that this method is more robust. It detects correctly the face and the face features even in cases when the image is less illuminated and the subject is not looking frontally.\n\nWith the implementation we did for `MTCNNFaceDetector` we display the following elements in the image:\n* the bounding box for the face (with red)\n* the position of keypoints (with green points), as following: \n * left eye\n * right eye\n * nose\n * mouth left \n * mouth right\n* the confidence score (with magenta text, above the face bounding box). This score is shown as a rounded value of the first 4 decimals.\n\nBesides the elements shown in the image, we also print the entire detection JSON.","metadata":{}},{"cell_type":"markdown","source":"Let's also look to some of the test videos.","metadata":{}},{"cell_type":"code","source":"for i in range(0, 3):\n video_path = os.path.join(DATA_FOLDER, TEST_FOLDER, subsample_test_videos[i])\n mtcnn_face_detector.detect(video_path)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T08:05:47.257978Z","iopub.execute_input":"2023-09-16T08:05:47.258238Z","iopub.status.idle":"2023-09-16T08:05:51.400658Z","shell.execute_reply.started":"2023-09-16T08:05:47.258194Z","shell.execute_reply":"2023-09-16T08:05:51.399328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"# Play video files \n\nFrom [Play video and processing Kernel](https://www.kaggle.com/code/hamditarek/deepfake-detection-challenge-kaggle?scriptVersionId=28503498) by @hamditarek we learned how to play video files in a Kaggle Kernel. We included the function to play videos as well in the `video_utils` utilty script.\n","metadata":{}},{"cell_type":"markdown","source":"Let's look to few fake videos. ","metadata":{}},{"cell_type":"code","source":"fake_videos = list(meta_train_df.loc[meta_train_df.label=='FAKE'].index)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-02T12:48:08.134391Z","iopub.execute_input":"2023-09-02T12:48:08.134805Z","iopub.status.idle":"2023-09-02T12:48:08.142516Z","shell.execute_reply.started":"2023-09-02T12:48:08.134733Z","shell.execute_reply":"2023-09-02T12:48:08.141281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"play_video(fake_videos[0], DATA_FOLDER, TRAIN_SAMPLE_FOLDER) ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-02T12:48:10.493137Z","iopub.execute_input":"2023-09-02T12:48:10.493608Z","iopub.status.idle":"2023-09-02T12:48:11.30223Z","shell.execute_reply.started":"2023-09-02T12:48:10.493546Z","shell.execute_reply":"2023-09-02T12:48:11.300742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"play_video(fake_videos[1], DATA_FOLDER, TRAIN_SAMPLE_FOLDER) ","metadata":{"execution":{"iopub.status.busy":"2023-09-02T12:48:20.73367Z","iopub.execute_input":"2023-09-02T12:48:20.734037Z","iopub.status.idle":"2023-09-02T12:48:21.057167Z","shell.execute_reply.started":"2023-09-02T12:48:20.733992Z","shell.execute_reply":"2023-09-02T12:48:21.05505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"play_video(fake_videos[2], DATA_FOLDER, TRAIN_SAMPLE_FOLDER) ","metadata":{"execution":{"iopub.status.busy":"2023-09-02T12:48:27.581627Z","iopub.execute_input":"2023-09-02T12:48:27.582035Z","iopub.status.idle":"2023-09-02T12:48:27.746848Z","shell.execute_reply.started":"2023-09-02T12:48:27.581975Z","shell.execute_reply":"2023-09-02T12:48:27.745251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"play_video(fake_videos[3], DATA_FOLDER, TRAIN_SAMPLE_FOLDER) ","metadata":{"execution":{"iopub.status.busy":"2023-09-02T12:48:32.368382Z","iopub.execute_input":"2023-09-02T12:48:32.368762Z","iopub.status.idle":"2023-09-02T12:48:32.609719Z","shell.execute_reply.started":"2023-09-02T12:48:32.368705Z","shell.execute_reply":"2023-09-02T12:48:32.608443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"play_video(fake_videos[4], DATA_FOLDER, TRAIN_SAMPLE_FOLDER) ","metadata":{"execution":{"iopub.status.busy":"2023-09-02T12:46:11.312369Z","iopub.status.idle":"2023-09-02T12:46:11.313152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From visual inspection of these fakes videos, in some cases is very easy to spot the anomalies created when engineering the deep fake, in some cases is more difficult.","metadata":{}},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"# References\n\n[1] Deepfake, Wikipedia, https://en.wikipedia.org/wiki/Deepfake \n[2] Google DeepFake Database, Endgadget, https://www.engadget.com/2019/09/25/google-deepfake-database/ \n[3] A quick look at the first frame of each video, https://www.kaggle.com/brassmonkey381/a-quick-look-at-the-first-frame-of-each-video \n[4] Basic EDA Face Detection, split video, ROI, https://www.kaggle.com/marcovasquez/basic-eda-face-detection-split-video-roi \n[5] Face Detection with OpenCV, https://www.kaggle.com/serkanpeldek/face-detection-with-opencv \n[6] Face Detection using MTCNN — a guide for face extraction with a focus on speed, https://towardsdatascience.com/face-detection-using-mtcnn-a-guide-for-face-extraction-with-a-focus-on-speed-c6d59f82d49 \n[7] Play video and processing, https://www.kaggle.com/hamditarek/play-video-and-processing/ ","metadata":{}},{"cell_type":"markdown","source":"---\n
\n Go to Top\n
","metadata":{}},{"cell_type":"markdown","source":"## ","metadata":{}}]}