{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Problem","metadata":{"id":"IByDRoRyvzMY"}},{"cell_type":"markdown","source":"**Task [[kaggle](https://www.kaggle.com/c/reface-fake-detection)]:** recognize fake videos. You need to train the binary classifier to distinguish real videos from fake ones (the provided fake data is the result of the technologies developed in Reface).\n\n****\n\n### What I should get?\n\nIn order to complete this stage, you should meet one of 2 conditions below:\n+ either make a solution with a minimum target metric value of 0.92475\n+ or be in the top 30 of all competitors.\n\n****\n\n### Evaluation\n\nThe evaluation metric for this competition is F1-Score, average='micro'. The F1 score, commonly used in information retrieval, measures accuracy using the statistics precision p and recall r. Precision is the ratio of true positives (tp) to all predicted positives (tp + fp). Recall is the ratio of true positives to all actual positives (tp + fn).\n\nThe F1 metric weights recall and precision equally, and a good retrieval algorithm will maximize both precision and recall simultaneously. Thus, moderately good performance on both will be favored over extremely good performance on one and poor performance on the other.\n\nMore information you can find at sklearn docs:\nhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html\n\n****\n\n### Submission\n\nFor each filename in the test set, you must predict either this file is fake video (label 1) or this file is real video (label 0). The file should contain a header and have the following format:\n\n```\nfilename,label\n004582.mp4,1\n003603.mp4,0\n```","metadata":{"id":"cVcVuY0Uvm5m"}},{"cell_type":"markdown","source":"## Install external modules and load our data","metadata":{"id":"RneR2ZDQw-_F"}},{"cell_type":"code","source":"!pip install -qq av","metadata":{"id":"yKTZNbxGAZTz","outputId":"1eebd139-d64e-4d44-ecbf-09c43bde3d21"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install facenet-pytorch > /dev/null 2>&1\n!apt install zip > /dev/null 2>&1","metadata":{"id":"RP7ozXs_L5Q0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from google.colab import drive\ndrive.mount('/content/drive')","metadata":{"id":"0XVuCB3FxCya","outputId":"a2fd054b-25ad-4ab2-b179-f9cf865c331a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -rf reface-fake-detection","metadata":{"id":"VhlKzYn7P3d8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!unzip -qq /content/drive/MyDrive/dl-creator-school/reface-fake-detection.zip -d reface-fake-detection","metadata":{"id":"nvk5JdEBxlyx"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modules importing","metadata":{"id":"dCV4IWc34im0"}},{"cell_type":"code","source":"import os\nimport glob\nimport json\nimport cv2\n\nimport pandas as pd\nimport numpy as np\n\nimport torch\nimport torch.nn.functional as F\nimport torchvision\n\nfrom torch import nn, optim\nfrom torch.utils.data import sampler\nfrom torch.utils.data.sampler import SubsetRandomSampler\nfrom torch.utils import data\nfrom torchvision import transforms, datasets\nfrom torchsummary import summary\n\nfrom facenet_pytorch import MTCNN\n\nfrom sklearn.model_selection import train_test_split\n\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nfrom joblib import Parallel, delayed\n\nfrom typing import List, Dict, Tuple, Union, Optional\nfrom pathlib import Path","metadata":{"id":"J3uT46T04iT2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\nplt.rcParams['figure.dpi'] = 150","metadata":{"id":"aUIGxneS_3lz"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Settings","metadata":{"id":"c1bv67Wf65ck"}},{"cell_type":"code","source":"device = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","metadata":{"id":"KALKKU9OMLuF","outputId":"488519df-f985-451d-9c5f-c541e7c3066a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PATH2PROJECT = Path('')\nPATH2DRIVE = Path('/content/drive/MyDrive/dl-creator-school/')\n\nPATH2DATA = PATH2PROJECT / 'reface-fake-detection'\nPATH2TRAIN = PATH2DATA / 'train'\nPATH2TEST = PATH2DATA / 'test'\n# PATH2SUBMISSIONS = PATH2DRIVE / 'submissions'\n# PATH2CHECKOUTS = PATH2DRIVE / 'checkouts'\n\nPATH2RESULT = PATH2PROJECT / 'reface-fake-detection-result'\nPATH2RESULT_TRAIN = PATH2RESULT / 'train'\nPATH2RESULT_TEST = PATH2RESULT / 'test'","metadata":{"id":"Q3nOA0Y765OW"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# try: PATH2SUBMISSIONS.mkdir()\n# except: pass\n# try: PATH2CHECKOUTS.mkdir()\n# except: pass\ntry: PATH2RESULT.mkdir()\nexcept: pass\ntry: PATH2RESULT_TRAIN.mkdir()\nexcept: pass\ntry: PATH2RESULT_TEST.mkdir()\nexcept: pass","metadata":{"id":"ZnGPv8Si_8J1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SEED = 42\nN_FRAMES = 8","metadata":{"id":"AuwPZruG8VDD"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training metadata","metadata":{"id":"nyxg-uyf60_q"}},{"cell_type":"code","source":"meta_df = pd.read_csv(PATH2DATA / 'train.csv')\nmeta_df.shape","metadata":{"id":"o2uK3RTF60v_","outputId":"1e2396ac-ad77-4fae-f8e3-f0331b43e72a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.sample(n=5, random_state=SEED)","metadata":{"id":"KDSOHBAI8Q2d","outputId":"15222a67-1cbb-4436-bb41-749fd6e8cd8c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.label.value_counts(normalize=True)","metadata":{"id":"1YAaHThY8xCN","outputId":"0fe613e6-e07e-4fb6-bb83-d3f43ab92b42"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Show few frames","metadata":{"id":"HOk8EDeV_zuN"}},{"cell_type":"code","source":"t = os.listdir(PATH2TRAIN / 'train')\ntemp_filename = t[np.random.randint(0, high=len(t))]\ndata = torchvision.io.read_video(str(PATH2TRAIN / 'train' / temp_filename), pts_unit='sec')\ndel t","metadata":{"id":"eCww0kXI_ze8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vid = data[0]\naud = data[1]\n\nprint(\"Video Shape: {}\\nAudio Shape: {}\\n\".format(vid.shape, aud.shape))","metadata":{"id":"LgziLCJpAPkh","outputId":"1ec93873-93e3-4732-b6ed-7b1a2589749f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(vid[10])\nplt.title(f\"DeepFake Dataset: {temp_filename}; label={meta_df[meta_df.filename == temp_filename].label.values[0]}\", loc='left', fontsize=8, pad=5);\nplt.axis(\"off\");","metadata":{"id":"JT9iG_x5BEcO","outputId":"c31d5a26-df6e-4f4d-b12b-3f4fbe0ffdea"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define Face Extractor","metadata":{"id":"kS0ef7AT4mRY"}},{"cell_type":"code","source":"class FaceExtractor:\n    def __init__(self, detector, n_frames=None):\n        \"\"\"\n        Parameters:\n            n_frames {int} -- Total number of frames to load. These will be evenly spaced\n                throughout the video. If not specified (i.e., None), all frames will be loaded.\n                (default: {None})\n            resize {float} -- Fraction by which to resize frames from original prior to face\n                detection. A value less than 1 results in downsampling and a value greater than\n                1 result in upsampling. (default: {None})\n        \"\"\"\n\n        self.detector = detector\n        self.n_frames = n_frames\n    \n    def __call__(self, filename, save_dir):\n        \"\"\"Load frames from an MP4 video, detect faces and save the results.\n\n        Parameters:\n            filename {str} -- Path to video.\n            save_dir {str} -- The directory where results are saved.\n        \"\"\"\n\n        # Create video reader and find length\n        v_cap = cv2.VideoCapture(filename)\n        v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n        # Pick 'n_frames' evenly spaced frames to sample\n        if self.n_frames is None:\n            sample = np.arange(0, v_len)\n        else:\n            sample = np.linspace(0, v_len - 1, self.n_frames).astype(int)\n\n        # Loop through frames\n        for j in range(v_len):\n            success = v_cap.grab()\n            if j in sample:\n                # Load frame\n                success, frame = v_cap.retrieve()\n                if not success:\n                    continue\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n                frame = Image.fromarray(frame)\n                \n                save_path = os.path.join(save_dir, f'{j}.png')\n\n                self.detector([frame], save_path=save_path)\n\n        v_cap.release()","metadata":{"id":"SMnOO2_W9YDS"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Face extraction process","metadata":{"id":"O-hkaybEM2Hs"}},{"cell_type":"code","source":"# Load face detector\nface_detector = MTCNN(margin=14, keep_all=True, factor=0.5, device=device).eval()","metadata":{"id":"dlTgu7wQ4iHL"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define face extractor\nface_extractor = FaceExtractor(detector=face_detector, n_frames=N_FRAMES)","metadata":{"id":"BTIamlI_M_hx"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the paths of all train videos\nall_train_videos = glob.glob(os.path.join(PATH2TRAIN / 'train', '*.mp4'))\nall_res_train_videos = os.listdir(PATH2RESULT_TRAIN)\n\nall_train_videos = [path for path in all_train_videos if (path.split('/')[-1].split('.')[0] not in all_res_train_videos)]","metadata":{"id":"5G0k1VS3NC-k"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(glob.glob(os.path.join(PATH2TRAIN / 'train', '*.mp4'))), len(all_train_videos)","metadata":{"id":"PO-dQLHMuA3T","outputId":"09a1daaa-de3a-4d23-eb7e-d5e48e990e7a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the paths of all test videos\nall_test_videos = glob.glob(os.path.join(PATH2TEST / 'test', '*.mp4'))\nall_res_test_videos = os.listdir(PATH2RESULT_TEST)\n\nall_test_videos = [path for path in all_test_videos if (path.split('/')[-1].split('.')[0] not in all_res_test_videos)]","metadata":{"id":"ChsRWBG-OsZr"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compute_face_extractor(path, save_path):\n    file_name = path.split('/')[-1]\n\n    save_dir = os.path.join(save_path, file_name.split(\".\")[0])\n\n    if not os.path.exists(save_dir):\n        os.makedirs(save_dir)\n\n    # Detect all faces appear in the video and save them.\n    face_extractor(path, save_dir)","metadata":{"id":"3LjPuHTwtUdj"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with torch.no_grad():\n    Parallel(n_jobs=-1, verbose=5)(delayed(compute_face_extractor)(path, PATH2RESULT_TRAIN) for path in tqdm(all_train_videos))","metadata":{"id":"sctMYkLdt9Ml","outputId":"a390acff-b323-4e94-a3d5-658c157e1c48"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with torch.no_grad():\n    Parallel(n_jobs=-1, verbose=5)(delayed(compute_face_extractor)(path, PATH2RESULT_TEST) for path in tqdm(all_test_videos))","metadata":{"id":"APtvHTibSAm7","outputId":"4ccc1bc2-62f7-40a5-ee37-552a11dc9615"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.to_csv(PATH2RESULT / 'train.csv', index=False)","metadata":{"id":"-dN1vA4hdlQz"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!tar -zcvf reface-fake-detection-result.tar.gz reface-fake-detection-result","metadata":{"id":"RB6rt_x4VKtf","outputId":"8c57a208-f4c4-4db4-991d-0f5c21f2b8b6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm /content/drive/MyDrive/dl-creator-school/reface-fake-detection.zip","metadata":{"id":"YaMREReKVsrv"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!cp reface-fake-detection-result.tar.gz /content/drive/MyDrive/dl-creator-school/reface-fake-detection-result.tar.gz","metadata":{"id":"WpsT1WDVZCv3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /content/drive/MyDrive/dl-creator-school","metadata":{"id":"26LWEn1ZaOwK","outputId":"3ef65f03-76e0-4178-f241-59e68da57afc"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"-yxEe0SYcARX"},"execution_count":null,"outputs":[]}]}