{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n# Table of Contents\n1. [Introduction](#introduction)\n1. [Import libraries](#import_libraries)\n1. [Configure hyper-parameters](#configure_hyper_parameters)\n1. [Define helper-functions](#define_helper_functions)\n1. [Define useful classes](#define_useful_classes)\n1. [Get datasets](#get_datasets)\n 1. [Find all folders contain training data](#fine_all_folders_contain_training_data)\n 1. [Get and merge all metadata files](#get_and_merge_all_metadata_files)\n 1. [Clean data](#clean_data)\n 1. [Remove corrupt videos or ones in what cannot detect any faces](#remove_corrupt_videos_or_ones_in_what_cannot_detect_any_faces)\n 1. [Remove videos in which do not have enough faces](#remove_videos_in_which_do_not_have_enough_faces)\n 1. [Change label format](#change_label_format)\n 1. [Split dataset](#split_dataset)\n1. [Start the training process](#start_the_training_process)\n 1. [Create dataloaders, classifier, etc.](#create_dataloaders_classifier_etc)\n 1. [Train the classifier](#train_the_classifier)\n 1. [Start the warm-up rounds](#start_the_warm_up_rounds)\n 1. [Visualize the results of the warm-up phase](#visualize_the_results_of_the_warm_up_phase)\n 1. [Start the fine-tune rounds](#start_the_fine_tune_rounds)\n 1. [Visualize the results of the fine-tune phase](#visualize_the_results_of_the_fine_tune_phase)\n1. [Conclusion](#conclusion)"},{"metadata":{},"cell_type":"markdown","source":"\n# Introduction\n\nIn this notebook, I will implement a simple pipeline which includes my suggested solution named `multiface classifier` to determine if an input video is FAKE or NOT.\n\nNote:\n* This solution only used for learning and research purposes, and I do not guarantee that it will produce good results.\n* All the datasets I used to train the classifier are well prepared beforehand, and you can find the full list of them in the [*Other useful datasets*](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/128954) discussion. For simplicity, I only choose a few of these datasets to be added in this kernel; you can add more if you wish. I have tested by loading 32 datasets at the same time and it works just fine.\n\nYou are in the second step of the whole pipeline, to read about the data preparation process, let follow this [*link*](https://www.kaggle.com/phunghieu/deepfake-detection-face-extractor).\n\n---\n## Idea\nSometimes, it is hard to determine a video if it is `real` or `fake` by only using a single face appears in this video, or separately classify each face then combine the results in some ways, for example averaging, to predict the label of the input video. I think we can give the classifier more meaningful information by feeding it with multiple face images at once (the strategy to sample these images will be discussed later.) Then, based on a sequence of faces, the classifier can give a better judgment on the video it gets.\n\n---\n## Multiface's general diagram\n\n\n---\n## Implementation\nTo start with, I've tried ResNet18 as the classifier and chose to use 5 sampled faces in each video to classify it.\n* Each chosen face image will be preprocessed separately then stacked together depth-wise (along the third axis) into a single tensor before fed into the model.\n* In the training process, I will generate a uniform random sample of size 5 (the number of faces.) Note: I am not sure random sampling can help prevent overfitting so further experiments must be conducted.\n* In the validation process, I will not choose input faces randomly like before; instead, I use a different strategy to obtain these images. I will try to get enough faces throughout the video by evenly spaced sampling; if I cannot get enough in the first run, I will continue this strategy but with a little shift (or stride) in the interval [*start, stop*] to get different faces if possible, and continue this process until the fifth try (just a hyper-parameter to prevent infinite loop.) Note: I've preprocessed the whole dataset before the validation process, so I can ensure that the `selector` will always get enough face images from each video.\n* The model will be led by the Focal Loss and optimized by the Adam algorithm.\n\n---\n## Pipeline\nThis end-to-end solution includes 3 steps:\n1. [*Data Preparation*](https://www.kaggle.com/phunghieu/deepfake-detection-face-extractor)\n1. *Training* <- **you're here**\n1. [*Inference*](https://www.kaggle.com/phunghieu/dfdc-multiface-inference)\n\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"\n# Import libraries\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch import nn, optim\nfrom torch.nn import functional as F\nfrom torchvision.models import resnet18\nfrom albumentations import Normalize, Compose\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2\nfrom tqdm.notebook import tqdm\nfrom sklearn.model_selection import train_test_split\nimport os\nimport glob\nimport multiprocessing as mp\n\nif torch.cuda.is_available():\n device = 'cuda:0'\n torch.set_default_tensor_type('torch.cuda.FloatTensor')\nelse:\n device = 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Configure hyper-parameters\n[Back to Table of Contents](#toc)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"INPUT_DIR = '/kaggle/input/'\nSAVE_PATH = '/kaggle/working/f5_resnet18.pth' # The location where the model should be saved.\nPRETRAINED_MODEL_PATH = ''\n\nN_FACES = 5\nTEST_SIZE = 0.3\nRANDOM_STATE = 123\n\nBATCH_SIZE = 32\nNUM_WORKERS = mp.cpu_count()\n\nWARM_UP_EPOCHS = 10\nWARM_UP_LR = 1e-4\nFINE_TUNE_EPOCHS = 100\nFINE_TUNE_LR = 1e-6\n\nTHRESHOLD = 0.5\nEPSILON = 1e-7","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Define helper-functions\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_f1(preds, labels):\n '''\n Parameters:\n preds: The predictions.\n labels: The labels.\n\n Returns:\n f1 score\n '''\n\n labels = np.array(labels, dtype=np.uint8)\n preds = (np.array(preds) >= THRESHOLD).astype(np.uint8)\n tp = np.count_nonzero(np.logical_and(labels, preds))\n tn = np.count_nonzero(np.logical_not(np.logical_or(labels, preds)))\n fp = np.count_nonzero(np.logical_not(labels)) - tn\n fn = np.count_nonzero(labels) - tp\n precision = tp / (tp + fp + EPSILON)\n recall = tp / (tp + fn + EPSILON)\n f1 = (2 * precision * recall) / (precision + recall + EPSILON)\n \n return f1\n\n\ndef train_the_model(\n model,\n criterion,\n optimizer,\n epochs,\n train_dataloader,\n val_dataloader,\n best_val_loss=1e7,\n best_val_logloss=1e7,\n save_the_best_on='val_logloss'\n):\n '''\n Parameters:\n model: The model needs to be trained.\n criterion: Loss function.\n optimizer: The optimizer.\n epochs: The number of epochs\n train_dataloader: The dataloader used to generate training samples.\n val_dataloader: The dataloader used to generate validation samples.\n best_val_loss: The initial value of the best val loss (default: 1e7.)\n best_val_logloss: The initial value of the best val log loss (default: 1e7.)\n save_the_best_on: Whether to save the best model based on \"val_loss\" or \"val_logloss\" (default: val_logloss.)\n\n Returns:\n losses: All computed losses.\n val_losses: All computed val_losses.\n loglosses: All computed loglosses.\n val_loglosses: All computed val_loglosses.\n f1_scores: All computed f1_scores.\n val_f1_scores: All computed val_f1_scores.\n best_val_loss: New value of the best val loss.\n best_val_logloss: New value of the best val log loss.\n best_model_state_dict: The state_dict of the best model.\n best_optimizer_state_dict: The state_dict of the optimizer corresponds to the best model.\n '''\n\n losses = np.zeros(epochs)\n val_losses = np.zeros(epochs)\n loglosses = np.zeros(epochs)\n val_loglosses = np.zeros(epochs)\n f1_scores = np.zeros(epochs)\n val_f1_scores = np.zeros(epochs)\n best_model_state_dict = None\n best_optimizer_state_dict = None\n\n logloss = nn.BCELoss()\n\n for i in tqdm(range(epochs)):\n batch_losses = []\n train_pbar = tqdm(train_dataloader)\n train_pbar.desc = f'Epoch {i+1}'\n classifier.train()\n\n all_labels = []\n all_preds = []\n\n for i_batch, sample_batched in enumerate(train_pbar):\n # Make prediction.\n y_pred = classifier(sample_batched['faces'])\n\n all_labels.extend(sample_batched['label'].squeeze(dim=-1).tolist())\n all_preds.extend(y_pred.squeeze(dim=-1).tolist())\n\n # Compute loss.\n loss = criterion(y_pred, sample_batched['label'])\n batch_losses.append(loss.item())\n\n # Zero gradients, perform a backward pass, and update the weights.\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n # Display some information in progress-bar.\n train_pbar.set_postfix({\n 'loss': batch_losses[-1]\n })\n\n # Compute scores.\n loglosses[i] = logloss(torch.tensor(all_preds).to(device), torch.tensor(all_labels).to(device))\n f1_scores[i] = calculate_f1(all_preds, all_labels)\n\n # Compute batch loss (average).\n losses[i] = np.array(batch_losses).mean()\n\n\n # Compute val loss\n val_batch_losses = []\n val_pbar = tqdm(val_dataloader)\n val_pbar.desc = 'Validating'\n classifier.eval()\n\n all_labels = []\n all_preds = []\n\n for i_batch, sample_batched in enumerate(val_pbar):\n # Make prediction.\n y_pred = classifier(sample_batched['faces'])\n\n all_labels.extend(sample_batched['label'].squeeze(dim=-1).tolist())\n all_preds.extend(y_pred.squeeze(dim=-1).tolist())\n\n # Compute val loss.\n val_loss = criterion(y_pred, sample_batched['label'])\n val_batch_losses.append(val_loss.item())\n\n # Display some information in progress-bar.\n val_pbar.set_postfix({\n 'val_loss': val_batch_losses[-1]\n })\n\n # Compute val scores.\n val_loglosses[i] = logloss(torch.tensor(all_preds).to(device), torch.tensor(all_labels).to(device))\n val_f1_scores[i] = calculate_f1(all_preds, all_labels)\n\n val_losses[i] = np.array(val_batch_losses).mean()\n print(f'loss: {losses[i]} | val loss: {val_losses[i]} | f1: {f1_scores[i]} | val f1: {val_f1_scores[i]} | log loss: {loglosses[i]} | val log loss: {val_loglosses[i]}')\n \n # Update the best values\n if val_losses[i] < best_val_loss:\n best_val_loss = val_losses[i]\n if save_the_best_on == 'val_loss':\n print('Found a better checkpoint!')\n best_model_state_dict = classifier.state_dict()\n best_optimizer_state_dict = optimizer.state_dict()\n if val_loglosses[i] < best_val_logloss:\n best_val_logloss = val_loglosses[i]\n if save_the_best_on == 'val_logloss':\n print('Found a better checkpoint!')\n best_model_state_dict = classifier.state_dict()\n best_optimizer_state_dict = optimizer.state_dict()\n \n return losses, val_losses, loglosses, val_loglosses, f1_scores, val_f1_scores, best_val_loss, best_val_logloss, best_model_state_dict, best_optimizer_state_dict\n\n\ndef visualize_results(\n losses,\n val_losses,\n loglosses,\n val_loglosses,\n f1_scores,\n val_f1_scores\n):\n '''\n Parameters:\n losses: A list of losses.\n val_losses: A list of val losses.\n loglosses: A list of loglosses.\n val_loglosses: A list of val loglosses.\n f1_scores: A list of f1 scores.\n val_f1_scores: A list of val f1 scores.\n '''\n\n fig = plt.figure(figsize=(16, 8))\n ax = fig.add_axes([0, 0, 1, 1])\n\n ax.plot(np.arange(1, len(losses) + 1), losses)\n ax.plot(np.arange(1, len(val_losses) + 1), val_losses)\n ax.set_xlabel('epoch', fontsize='xx-large')\n ax.set_ylabel('focal loss', fontsize='xx-large')\n ax.legend(\n ['loss', 'val loss'],\n loc='upper right',\n fontsize='xx-large',\n shadow=True\n )\n plt.show()\n\n \n fig = plt.figure(figsize=(16, 8))\n ax = fig.add_axes([0, 0, 1, 1])\n\n ax.plot(np.arange(1, len(loglosses) + 1), loglosses)\n ax.plot(np.arange(1, len(val_loglosses) + 1), val_loglosses)\n ax.set_xlabel('epoch', fontsize='xx-large')\n ax.set_ylabel('log loss', fontsize='xx-large')\n ax.legend(\n ['log loss', 'val log loss'],\n loc='upper right',\n fontsize='xx-large',\n shadow=True\n )\n plt.show()\n\n\n fig = plt.figure(figsize=(16, 8))\n ax = fig.add_axes([0, 0, 1, 1])\n\n ax.plot(np.arange(1, len(f1_scores) + 1), f1_scores)\n ax.plot(np.arange(1, len(val_f1_scores) + 1), val_f1_scores)\n ax.set_xlabel('epoch', fontsize='xx-large')\n ax.set_ylabel('f1 score', fontsize='xx-large')\n ax.legend(\n ['f1', 'val f1'],\n loc='upper left',\n fontsize='xx-large',\n shadow=True\n )\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Define useful classes\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DeepfakeClassifier(nn.Module):\n def __init__(self, encoder, in_channels=3, num_classes=1):\n super(DeepfakeClassifier, self).__init__()\n self.encoder = encoder\n \n # Modify input layer.\n self.encoder.conv1 = nn.Conv2d(\n in_channels,\n 64,\n kernel_size=7,\n stride=2,\n padding=3,\n bias=False\n )\n \n # Modify output layer.\n self.encoder.fc = nn.Linear(512 * 1, num_classes)\n\n def forward(self, x):\n return torch.sigmoid(self.encoder(x))\n \n def freeze_all_layers(self):\n for param in self.encoder.parameters():\n param.requires_grad = False\n\n def freeze_middle_layers(self):\n self.freeze_all_layers()\n \n for param in self.encoder.conv1.parameters():\n param.requires_grad = True\n \n for param in self.encoder.fc.parameters():\n param.requires_grad = True\n\n def unfreeze_all_layers(self):\n for param in self.encoder.parameters():\n param.requires_grad = True\n\n\nclass FaceDataset(Dataset):\n def __init__(self, img_dirs, labels, n_faces=1, preprocess=None):\n self.img_dirs = img_dirs\n self.labels = labels\n self.n_faces = n_faces\n self.preprocess = preprocess\n\n def __len__(self):\n return len(self.img_dirs)\n \n def __getitem__(self, idx):\n if torch.is_tensor(idx):\n idx = idx.tolist()\n\n img_dir = self.img_dirs[idx]\n label = self.labels[idx]\n face_paths = glob.glob(f'{img_dir}/*.png')\n\n if len(face_paths) >= self.n_faces:\n sample = np.random.choice(face_paths, self.n_faces, replace=False)\n else:\n sample = np.random.choice(face_paths, self.n_faces, replace=True)\n \n faces = []\n \n for face_path in sample:\n face = cv2.imread(face_path, 1)\n face = cv2.cvtColor(face, cv2.COLOR_BGR2RGB)\n if self.preprocess is not None:\n augmented = self.preprocess(image=face)\n face = augmented['image']\n faces.append(face)\n\n return {'faces': np.concatenate(faces, axis=-1).transpose(2, 0, 1), 'label': np.array([label], dtype=float)}\n \n \nclass FaceValDataset(Dataset):\n def __init__(self, img_dirs, labels, n_faces=1, preprocess=None):\n self.img_dirs = img_dirs\n self.labels = labels\n self.n_faces = n_faces\n self.preprocess = preprocess\n\n def __len__(self):\n return len(self.img_dirs)\n \n def __getitem__(self, idx):\n if torch.is_tensor(idx):\n idx = idx.tolist()\n\n img_dir = self.img_dirs[idx]\n label = self.labels[idx]\n face_paths = glob.glob(f'{img_dir}/*.png')\n\n face_indices = [\n path.split('/')[-1].split('.')[0].split('_')[0]\n for path in face_paths\n ] \n max_idx = np.max(np.array(face_indices, dtype=np.uint32))\n\n selected_paths = []\n\n for i in range(self.n_faces):\n stride = int((max_idx + 1)/(self.n_faces**2))\n sample = np.linspace(i*stride, max_idx + i*stride, self.n_faces).astype(int)\n\n # Get faces\n for idx in sample:\n paths = glob.glob(f'{img_dir}/{idx}*.png')\n\n selected_paths.extend(paths)\n\n if len(selected_paths) >= self.n_faces:\n break\n \n if len(selected_paths) >= self.n_faces:\n break\n\n faces = []\n\n selected_paths = selected_paths[:self.n_faces] # Get top\n for selected_path in selected_paths:\n img = cv2.imread(selected_path, 1)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n faces.append(img)\n\n if self.preprocess is not None:\n for j in range(len(faces)):\n augmented = self.preprocess(image=faces[j])\n faces[j] = augmented['image']\n\n faces = np.concatenate(faces, axis=-1).transpose(2, 0, 1)\n\n return {\n 'faces': faces,\n 'label': np.array([label], dtype=float)\n }\n\n\nclass FocalLoss(nn.Module):\n def __init__(self, gamma=2, sample_weight=None):\n super().__init__()\n self.gamma = gamma\n self.sample_weight = sample_weight\n\n def forward(self, logit, target):\n target = target.float()\n max_val = (-logit).clamp(min=0)\n loss = logit - logit * target + max_val + \\\n ((-max_val).exp() + (-logit - max_val).exp()).log()\n\n invprobs = F.logsigmoid(-logit * (target * 2.0 - 1.0))\n loss = (invprobs * self.gamma).exp() * loss\n if len(loss.size())==2:\n loss = loss.sum(dim=1)\n if self.sample_weight is not None:\n loss = loss * self.sample_weight\n return loss.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Get datasets\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"\n## Find all folders contain training data\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_dirs = glob.glob(INPUT_DIR + 'deepfake-detection-faces-*')\nall_train_dirs = sorted(all_train_dirs, key=lambda x: x)\nfor i, train_dir in enumerate(all_train_dirs):\n print('[{:02}]'.format(i), train_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Get and merge all metadata files\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"all_dataframes = []\nfor train_dir in all_train_dirs:\n df = pd.read_csv(os.path.join(train_dir, 'metadata.csv'))\n df['path'] = df['filename'].apply(lambda x: os.path.join(train_dir, x.split('.')[0]))\n all_dataframes.append(df)\n\ntrain_df = pd.concat(all_dataframes, ignore_index=True, sort=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Clean data\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"\n### Remove corrupt videos or ones in what cannot detect any faces\n*These videos will result in empty folders after the data preparation process.*\n\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove empty folders\ntrain_df = train_df[train_df['path'].map(lambda x: os.path.exists(x))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Remove videos in which do not have enough faces\n*We must do this to prevent producing inappropriate samples during the training and validation processes.*\n\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_train_df = pd.DataFrame(columns=['filename', 'label', 'split', 'original', 'path'])\n\n# for row_idx, row in tqdm(train_df.iterrows()):\nfor row_idx in tqdm(train_df.index):\n row = train_df.loc[row_idx]\n img_dir = row['path']\n face_paths = glob.glob(f'{img_dir}/*.png')\n\n if len(face_paths) >= N_FACES: # Satisfy the minimum requirement for the number of faces\n face_indices = [\n path.split('/')[-1].split('.')[0].split('_')[0]\n for path in face_paths\n ]\n max_idx = np.max(np.array(face_indices, dtype=np.uint32))\n\n selected_paths = []\n\n for i in range(N_FACES):\n stride = int((max_idx + 1)/(N_FACES**2))\n sample = np.linspace(i*stride, max_idx + i*stride, N_FACES).astype(int)\n\n # Get faces\n for idx in sample:\n paths = glob.glob(f'{img_dir}/{idx}*.png')\n\n selected_paths.extend(paths)\n if len(selected_paths) >= N_FACES: # Get enough faces\n break\n\n if len(selected_paths) >= N_FACES: # Get enough faces\n valid_train_df = valid_train_df.append(row, ignore_index=True)\n break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Change label format\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_train_df['label'].replace({'FAKE': 1, 'REAL': 0}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Split dataset\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"label_count = valid_train_df.groupby('label').count()['filename']\nprint(label_count)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = valid_train_df['path'].to_numpy()\ny = valid_train_df['label'].to_numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=TEST_SIZE, random_state=RANDOM_STATE, stratify=y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Start the training process\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"\n## Create dataloaders, classifier, etc.\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"preprocess = Compose([\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], p=1)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = FaceDataset(\n img_dirs=X_train,\n labels=y_train,\n n_faces=N_FACES,\n preprocess=preprocess\n)\nval_dataset = FaceValDataset(\n img_dirs=X_val,\n labels=y_val,\n n_faces=N_FACES,\n preprocess=preprocess\n)\n\ntrain_dataloader = DataLoader(\n train_dataset,\n batch_size=BATCH_SIZE,\n shuffle=True,\n num_workers=NUM_WORKERS\n)\nval_dataloader = DataLoader(\n val_dataset,\n batch_size=BATCH_SIZE,\n shuffle=False,\n num_workers=NUM_WORKERS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"if os.path.exists(PRETRAINED_MODEL_PATH):\n encoder = resnet18(pretrained=False)\n classifier = DeepfakeClassifier(encoder=encoder, in_channels=3*N_FACES, num_classes=1)\n state = torch.load(PRETRAINED_MODEL_PATH, map_location=lambda storage, loc: storage)\n classifier.load_state_dict(state['state_dict'])\nelse:\n encoder = resnet18(pretrained=True)\n classifier = DeepfakeClassifier(encoder=encoder, in_channels=3*N_FACES, num_classes=1)\n\nclassifier.to(device)\nclassifier.train()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = FocalLoss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"losses = np.zeros(WARM_UP_EPOCHS + FINE_TUNE_EPOCHS)\nval_losses = np.zeros(WARM_UP_EPOCHS + FINE_TUNE_EPOCHS)\nloglosses = np.zeros(WARM_UP_EPOCHS + FINE_TUNE_EPOCHS)\nval_loglosses = np.zeros(WARM_UP_EPOCHS + FINE_TUNE_EPOCHS)\nf1_scores = np.zeros(WARM_UP_EPOCHS + FINE_TUNE_EPOCHS)\nval_f1_scores = np.zeros(WARM_UP_EPOCHS + FINE_TUNE_EPOCHS)\n\nif os.path.exists(PRETRAINED_MODEL_PATH):\n best_val_loss = state['best_val_loss']\nelse:\n best_val_loss = 1e7\n\nif os.path.exists(PRETRAINED_MODEL_PATH):\n best_val_logloss = state['best_val_logloss']\nelse:\n best_val_logloss = 1e7","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Train the classifier\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"\n### Start the warm-up rounds\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"classifier.freeze_middle_layers()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"warmup_optimizer = optim.Adam(filter(lambda p: p.requires_grad, classifier.parameters()), lr=WARM_UP_LR)\nif os.path.exists(PRETRAINED_MODEL_PATH) and 'warmup_optimizer' in state.keys():\n warmup_optimizer.load_state_dict(state['warmup_optimizer'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"losses[:WARM_UP_EPOCHS], val_losses[:WARM_UP_EPOCHS], \\\nloglosses[:WARM_UP_EPOCHS], val_loglosses[:WARM_UP_EPOCHS], \\\nf1_scores[:WARM_UP_EPOCHS], val_f1_scores[:WARM_UP_EPOCHS], \\\nbest_val_loss, best_val_logloss, \\\nbest_model_state_dict, best_optimizer_state_dict \\\n= train_the_model(\n model=classifier,\n criterion=criterion,\n optimizer=warmup_optimizer,\n epochs=WARM_UP_EPOCHS,\n train_dataloader=train_dataloader,\n val_dataloader=val_dataloader,\n best_val_loss=best_val_loss,\n best_val_logloss=best_val_logloss,\n save_the_best_on='val_logloss'\n)\n\n# Save the best checkpoint.\nif best_model_state_dict is not None:\n state = {\n 'state_dict': best_model_state_dict,\n 'warmup_optimizer': best_optimizer_state_dict,\n 'best_val_loss': best_val_loss,\n 'best_val_logloss': best_val_logloss\n }\n\n torch.save(state, SAVE_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Visualize the results of the warm-up phase\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_results(\n losses=losses[:WARM_UP_EPOCHS],\n val_losses=val_losses[:WARM_UP_EPOCHS],\n loglosses=loglosses[:WARM_UP_EPOCHS],\n val_loglosses=val_loglosses[:WARM_UP_EPOCHS],\n f1_scores=f1_scores[:WARM_UP_EPOCHS],\n val_f1_scores=val_f1_scores[:WARM_UP_EPOCHS]\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Start the fine-tune rounds\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"classifier.unfreeze_all_layers()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"finetune_optimizer = optim.Adam(filter(lambda p: p.requires_grad, classifier.parameters()), lr=FINE_TUNE_LR)\nif os.path.exists(PRETRAINED_MODEL_PATH) and 'finetune_optimizer' in state.keys() and WARM_UP_EPOCHS == 0:\n finetune_optimizer.load_state_dict(state['finetune_optimizer'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"losses[WARM_UP_EPOCHS:WARM_UP_EPOCHS+FINE_TUNE_EPOCHS], val_losses[WARM_UP_EPOCHS:WARM_UP_EPOCHS+FINE_TUNE_EPOCHS], \\\nloglosses[WARM_UP_EPOCHS:WARM_UP_EPOCHS+FINE_TUNE_EPOCHS], val_loglosses[WARM_UP_EPOCHS:WARM_UP_EPOCHS+FINE_TUNE_EPOCHS], \\\nf1_scores[WARM_UP_EPOCHS:WARM_UP_EPOCHS+FINE_TUNE_EPOCHS], val_f1_scores[WARM_UP_EPOCHS:WARM_UP_EPOCHS+FINE_TUNE_EPOCHS], \\\nbest_val_loss, best_val_logloss, \\\nbest_model_state_dict, best_optimizer_state_dict \\\n= train_the_model(\n model=classifier,\n criterion=criterion,\n optimizer=finetune_optimizer,\n epochs=FINE_TUNE_EPOCHS,\n train_dataloader=train_dataloader,\n val_dataloader=val_dataloader,\n best_val_loss=best_val_loss,\n best_val_logloss=best_val_logloss,\n save_the_best_on='val_logloss'\n)\n\n# Save the best checkpoint.\nif best_model_state_dict is not None:\n state = {\n 'state_dict': best_model_state_dict,\n 'finetune_optimizer': best_optimizer_state_dict,\n 'best_val_loss': best_val_loss,\n 'best_val_logloss': best_val_logloss\n }\n\n torch.save(state, SAVE_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Visualize the results of the fine-tune phase\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_results(\n losses=losses,\n val_losses=val_losses,\n loglosses=loglosses,\n val_loglosses=val_loglosses,\n f1_scores=f1_scores,\n val_f1_scores=val_f1_scores\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Conclusion\nSo, we have done the second step in the whole pipeline. Let's move on to the final step -> [*DFDC-Multiface-Inference*](https://www.kaggle.com/phunghieu/dfdc-multiface-inference).\n\nIf you have any questions or suggestions, feel free to move to the `comments` section below.\n\n---\nThe content in this notebook is too complicated to understand and you want a simpler solution to get started, I have already prepared one for you based on `@timesler`'s solution in this series:\n* [Deepfake Detection - Data Preparation (baseline)](https://www.kaggle.com/phunghieu/deepfake-detection-data-preparation-baseline)\n* [Deepfake Detection - Training (baseline)](https://www.kaggle.com/phunghieu/deepfake-detection-training-baseline)\n* [Deepfake Detection - Inference (baseline)](https://www.kaggle.com/phunghieu/deepfake-detection-inference-baseline)\n\n---\nPlease upvote this kernel if you think it is worth reading.\n\nThank you so much!\n\n[Back to Table of Contents](#toc)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}