{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Facenet Embeddings + Clustering + LSTM for Inference\nThis kernel shows how to use facenet embeddings to accomplish several things:\n\n1) Cluster similar faces throughout the training data and create a safe validation strategy for trainining and validation splits. You can see below how to use PCA, T-SNE and DBSCAN to efficiently cluster high-dimensional data. The found clusters are exported and can be used to improve your training and validation split.\n\n2) Provide inputs to an LSTM to classify REAL vs. FAKE videos\n\nSome of the code is borrowed from @carlossouza and @timesler kernels, so thanks heaps to both. However, the results with facenet seem considerably better and more consistent than what is showed on the original kernel."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Install facenet-pytorch\n!pip install /kaggle/input/facenet-pytorch-vggface2/facenet_pytorch-2.2.7-py3-none-any.whl\n\nfrom facenet_pytorch.models.inception_resnet_v1 import get_torch_home\ntorch_home = get_torch_home()\n\n# Copy model checkpoints to torch cache so they are loaded automatically by the package\n!mkdir -p $torch_home/checkpoints/\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-logits.pth $torch_home/checkpoints/vggface2_DG3kwML46X.pt\n!cp /kaggle/input/facenet-pytorch-vggface2/20180402-114759-vggface2-features.pth $torch_home/checkpoints/vggface2_G5aNV2VSMn.pt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom pathlib import Path\nfrom facenet_pytorch import MTCNN\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport cv2\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.axes_grid1 import ImageGrid\nimport PIL.Image\nfrom tqdm.notebook import tqdm\nfrom time import time\nimport shutil\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# https://www.kaggle.com/hmendonca/kaggle-pytorch-utility-script\nfrom kaggle_pytorch_utility_script import *\n\nseed_everything(42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Model, Sequential\nfrom keras.layers import Dense, Input, Dropout, LSTM, Activation\nfrom keras.preprocessing import sequence\nfrom keras.initializers import glorot_uniform","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# See github.com/timesler/facenet-pytorch:\nfrom facenet_pytorch import MTCNN, InceptionResnetV1, extract_face\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision.transforms import ToTensor\n\n# load image processor\ntf_img = lambda i: ToTensor()(i).unsqueeze(0)\n\n# Load facial recognition model\nresnet = InceptionResnetV1(pretrained='vggface2', device=device).eval()\n\n# set up embedding calculator for resnet\nembeddings = lambda input_data: resnet(input_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set Configs"},{"metadata":{"trusted":true},"cell_type":"code","source":"DEBUG = False\nDEBUG_COUNT = 1000 # Limit on original training and new videos \nREPROCESS = True # Reprocess source videos from scratch, or use embeddings already extracted\nSUBMIT = False # Test run or for submission\n\n# data locations\nTMP_DIR = '/kaggle/working/datasets' # for data created during the notebook run\nINPUT_FOLDER = '/kaggle/input'\nTEST_DIR = '/kaggle/input/deepfake-detection-challenge/test_videos/'\nTRAIN_DIR = '/kaggle/input/deepfake-detection-faces*.mp4'\nAUGMENT_DIR = '/kaggle/input/additional-deepfake-training-data/'\n\n# data processing\nAUGMENT_VIDEO_COUNT = 400\nN_FRAMES = None\nSCALE = 0.25\nNUM_SAMPLES = 1\nNUM_FRAMES = 4\n\n# train/val/test setup\nTRAIN_SPLIT_FRACTION = .80","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Face Extraction Setup\nUsing additional videos we can augment the face detection data with additional cases for training"},{"metadata":{"trusted":true},"cell_type":"code","source":"class FaceExtractor:\n    def __init__(self, detector, n_frames=None, resize=None):\n        \"\"\"\n        Parameters:\n            n_frames {int} -- Total number of frames to load. These will be evenly spaced\n                throughout the video. If not specified (i.e., None), all frames will be loaded.\n                (default: {None})\n            resize {float} -- Fraction by which to resize frames from original prior to face\n                detection. A value less than 1 results in downsampling and a value greater than\n                1 result in upsampling. (default: {None})\n        \"\"\"\n\n        self.detector = detector\n        self.n_frames = n_frames\n        self.resize = resize\n    \n    def __call__(self, filename, save_dir):\n        \"\"\"Load frames from an MP4 video, detect faces and save the results.\n\n        Parameters:\n            filename {str} -- Path to video.\n            save_dir {str} -- The directory where results are saved.\n        \"\"\"\n\n        # Create video reader and find length\n        v_cap = cv2.VideoCapture(filename)\n        frame_count = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n        v_len = 100 if frame_count > 100 else frame_count\n\n        # Pick 'n_frames' evenly spaced frames to sample\n        if self.n_frames is None:\n            sample = np.arange(0, v_len)\n        else:\n            sample = np.linspace(0, v_len - 1, self.n_frames).astype(int)\n\n        # Loop through frames\n        for j in range(v_len):\n            success = v_cap.grab()\n            if j in sample:\n                # Load frame\n                success, frame = v_cap.retrieve()\n                if not success:\n                    continue\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n                frame = PIL.Image.fromarray(frame)\n                \n                # Resize frame to desired size\n                if self.resize is not None:\n                    frame = frame.resize([int(d * self.resize) for d in frame.size])\n\n                save_path = os.path.join(save_dir, f'{j}.png')\n\n                self.detector([frame], save_path=save_path)\n\n        v_cap.release()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"face_detector = MTCNN(margin=14, keep_all=True, factor=0.5, device=device).eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define face extractor\nface_extractor = FaceExtractor(detector=face_detector, n_frames=N_FRAMES, resize=SCALE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_faces_from_videos(extract_video_paths, face_extractor, save_dir):\n    \"\"\"Run face extractor module and save results to TMP_DIR\"\"\"\n    with torch.no_grad():\n        for path in tqdm(extract_video_paths):\n            file_name = path.split('/')[-1]\n\n            save_dir = os.path.join(save_dir, file_name.split(\".\")[0])\n\n            if not os.path.exists(save_dir):\n                os.makedirs(save_dir)\n\n            # Detect all faces appear in the video and save them.\n            face_extractor(path, save_dir)\n            \n            # reset save_dir\n            save_dir = '/'.join(save_dir.split('/')[:-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir -p $TMP_DIR","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extract Faces from Augmentation Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"augment_videos = glob.glob(os.path.join(AUGMENT_DIR, '*.mp4'))\n\nif DEBUG:\n    augment_videos = augment_videos[0:AUGMENT_VIDEO_COUNT]\nelse:\n    augment_videos = augment_videos[0:AUGMENT_VIDEO_COUNT]\n\nprint(len(augment_videos))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"augment_extract_dir = TMP_DIR + '/augment-video-extracts'\nextract_faces_from_videos(augment_videos, face_extractor, augment_extract_dir)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Extract sample frames from all videos\nThanks @unkownhihi for his dataset with all training faces."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(augment_videos[0:5])\nlen(augment_videos)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_videos = []\nmetadata_dfs = []\n\naugment_videos = glob.glob(augment_extract_dir + '/*')\ntrain_data_folders = glob.glob('/kaggle/input/deepfake-detection-face*')\n\nfor tdf in train_data_folders:\n    videos = os.listdir(tdf)\n    metadata_dfs.append(pd.read_csv(tdf + '/metadata.csv'))\n    for v in videos:\n        if v == 'metadata.csv':\n            continue\n        path = os.path.join(tdf, v)\n        train_videos.append(path)\n\n        \ntrain_videos = train_videos + augment_videos\n\nif DEBUG:\n    train_videos = train_videos[0:DEBUG_COUNT] + augment_videos[0:DEBUG_COUNT]\n    \nprint(f\"Train samples: {len(train_videos)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata = pd.concat(metadata_dfs)\nmetadata['video_source'] = 'competition'\n\nadf_count = len(augment_videos)\n\naugment_df = pd.DataFrame.from_dict({'filename': [av.split('/')[-1] + '.mp4' for av in augment_videos],\n                                      'split': ['train' for i in range(adf_count)],\n                                      'original': [np.nan for i in range(adf_count)],\n                                      'label': ['REAL' for i in range(adf_count)],\n                                      'video_source': ['augment' for i in range(adf_count)]})\n\nmetadata = pd.concat([metadata, augment_df])\nfig, ax = plt.subplots(figsize=(15,7))\nmetadata.groupby('original').count().sort_values(by='filename', ascending=False).plot(ax=ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_indices(fname):\n    \"\"\"Adds leading zeros to input filename\"\"\"\n    fname_list = fname.split('.')\n    index = fname_list[0].split('_')[0] # don't care whether frames have > 1 face within\n    if not index:\n        print('issue')\n    return (index.zfill(3), fname)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sort_frames(train_videos):\n    result = []\n    for v in train_videos:\n        video_files = os.listdir(v)\n        video_indices = [get_indices(f) for f in video_files]\n        video_indices = sorted(video_indices, key = lambda x: x[0])\n        result.append({'name': v, 'indices': video_indices})\n            \n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"originals = metadata.groupby('filename')['original'].min().to_dict()\nadded_tracker = {}\nfiltered_train_videos = []\nfor tv in train_videos:\n    video_name = tv.split('/')[-1] + '.mp4'\n    #original = originals[video_name]\n    filtered_train_videos.append(tv)\n    \n#     if original is np.nan:\n#         filtered_train_videos.append(tv)\n#         continue\n        \n#     elif original not in added_tracker:\n#         added_tracker[original] = ''\n#         filtered_train_videos.append(tv)\n#     else:\n#         continue\n        \nprint(len(filtered_train_videos))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sorted_frames = sort_frames(filtered_train_videos)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_sample(video, samp_idx, sample_length):\n    samp_rng = list(range(samp_idx.astype(int), samp_idx.astype(int) + sample_length))\n    sampled = [video['indices'][i][1] for i in samp_rng]\n    return sampled","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sample_frames(sorted_frames, metadata, n_sample_sets=2, sample_length=4):\n    \"\"\"Randomly sample n_sample_sets of sample_length from the recorded face\n    frames\"\"\"\n    result = []\n    for video in sorted_frames:\n        num_frames = len(video['indices'])\n        sample_indices = np.random.choice(range(num_frames - (sample_length + 1)), n_sample_sets)\n        for samp_idx in sample_indices:\n            selected_frames = draw_sample(video, samp_idx, sample_length)\n            names = [video['name'] for sf in selected_frames]\n            idxs = [samp_idx for sf in selected_frames]\n            result.append({'name': names, 'samp_idx': idxs, 'selected_frames': selected_frames})  \n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(1)\nsampled_frames = sample_frames(sorted_frames, metadata, n_sample_sets=NUM_SAMPLES, sample_length=NUM_FRAMES)\nsampled_frames[0:2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_embedding(path_to_data):\n    \"\"\"Uses pretrained model to extract embeddings from a frame at path_to_data\"\"\"\n    t = tf_img(PIL.Image.open(path_to_data)).to(device)\n    return embeddings(t).squeeze().cpu().tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_sample(video):    \n    result = []\n    with torch.no_grad():\n        video_name, samp_img_list = video['name'][0], video['selected_frames']\n        for img in samp_img_list:\n            path_to_data = '/'.join([video_name, img])\n            emb = get_embedding(path_to_data)\n            result.append(emb)\n        video['embeddings'] = result\n    return video","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_to_dataframe(processed_frames, metadata):\n    \"\"\"Conver embeddings dataset to a pandas dataframe\"\"\"\n    dfs = []\n    for pf in processed_frames:\n        dfs.append(pd.DataFrame(pf))\n    df =  pd.concat(dfs, ignore_index=True) # make sure to ignore the index to recreate from 0 to n!\n    # explode embedding vectors to columns\n    num_embeddings = len(df['embeddings'].values[0])\n    emb_df = pd.DataFrame(df['embeddings'].values.tolist(), columns=['emb_{}'.format(i) for i in range(num_embeddings)])\n    df = df.join(emb_df)\n    df['file_path'] = df['name']\n    df['name'] = df['name'].apply(lambda x: x.split('/')[-1] + '.mp4')\n    return df.merge(metadata[['filename', 'split', 'label', 'original']], \n              left_on='name', \n              right_on='filename', \n              how='inner').drop(['filename', 'embeddings'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def output_dataframe(dataset, path, fname):\n    \"\"\"Check if path exists, create if it doesn't, then output file\"\"\"\n    file_path = '/'.join([path, fname])\n    if not os.path.exists(path):\n        os.mkdir(path)\n        dataset.to_csv(file_path)\n    \n    else:\n        dataset.to_csv(file_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"OUTPUT_PATH = '../working/processed'\nOUTPUT_FILE = 'embeddings_data.csv'\n\nif REPROCESS:\n    processed_frames = [result for result in map(process_sample, sampled_frames)]\n    df = convert_to_dataframe(processed_frames, metadata)\n    print('shape of dataframe created: ', str(df.shape))\n    output_dataframe(df, OUTPUT_PATH, OUTPUT_FILE)\n\nelse:\n    df = pd.read_csv('/'.join([OUTPUT_PATH, OUTPUT_FILE]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Convert embeddings dataframe to lists of vectors\n\nGoal is to convert each set of embeddings for a given video to a list of vectors for training the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['label_bin'] = df['label'].apply(lambda x: 1 if x == 'REAL' else 0)\ndf.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nfilenames = df.groupby('name')['name'].min().values\nfname_to_original = df.groupby('name')['original'].min().to_dict()\nn = len(filenames)\nidx = int(n * TRAIN_SPLIT_FRACTION)\n\nrandom.shuffle(filenames) #shuffle randomly to mix original and augment datasets\n\ntrain_fnames = {fn: '0' for fn in filenames[0:idx]}\ntest_fnames = {fn: '0' for fn in filenames[idx:]}\n\nprint(len(train_fnames) + len(test_fnames) == n)\nprint('{} training videos || {} testing videos'.format(str(len(train_fnames)), str(len(test_fnames))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_X, train_Y = [], [] \ntest_X, test_Y = [], []\ntest_names = []\n\nfor pf in processed_frames:\n    video_name = pf['name'][0]\n    lookup_name = video_name.split('/')[-1] + '.mp4'\n    y_value = df.loc[df['name'] == lookup_name, 'label_bin'].values[0] # extract 1 for REAL and 0 for FAKE \n    \n    # grab embeddings and normalize across faces\n    embs = pf['embeddings']\n    if lookup_name in train_fnames:\n        train_X.append(embs)\n        train_Y.append(y_value)\n        \n    elif fname_to_original[lookup_name] not in train_fnames: # make sure originals haven't been seen in testing data\n        test_X.append(embs)\n        test_Y.append(y_value)\n        test_names.append(video_name)\n        \n    else:\n        continue","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_X_len = len(train_X)\ntest_X_len = len(test_X)\ntrain_X_arr = np.reshape(train_X, (train_X_len, NUM_FRAMES, 512))\ntrain_X_mn = train_X_arr.mean(0)\ntrain_X_std = train_X_arr.std(0)\ntrain_X_arr = (train_X_arr - train_X_mn) / train_X_std\n\ntest_X_arr = np.reshape(test_X, (test_X_len, NUM_FRAMES, 512))\ntest_X_mn = test_X_arr.mean(0)\ntest_X_std = test_X_arr.std(0)\ntest_X_arr = (test_X_arr - test_X_mn) / test_X_std\ntrain_Y = np.array(train_Y)\n\nprint('{} training video || {} testing videos'.format(str(train_X_len), str(test_X_len)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#np.save('/kaggle/working/models/train_X_mn', train_X_mn)\n#np.save('/kaggle/working/models/train_X_std', train_X_std)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\n\n# summarize class distribution\ncounter = Counter(train_Y)\n\n# indexes of reals and fakes\nreal_idx = np.where(train_Y == 1)[0]\nfake_idx = np.where(train_Y == 0)[0]\n\n# upsample reals to match the fakes\n# add in Xception net kernel idea with readily available faces\nreal_samples = np.random.choice(real_idx, size=counter[0], replace=True)\n\ntrain_X_samp = np.concatenate([train_X_arr[real_samples], train_X_arr[fake_idx]], axis=0)\ntrain_Y_samp = np.concatenate([train_Y[real_samples], train_Y[fake_idx]], axis=0)\n\nprint(train_X_samp.shape)\nprint(train_Y_samp.shape)\nprint(np.unique(train_Y_samp, return_counts=True))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 64\nepochs = 25\nbatch_input_shape = (None, NUM_FRAMES, 512)\nmodel = Sequential()\nmodel.add(LSTM(256, activation='relu', return_sequences=True, batch_input_shape=batch_input_shape, stateful=False))\nmodel.add(Dropout(0.5))\nmodel.add(LSTM(128, activation='relu', return_sequences=False, batch_input_shape=batch_input_shape, stateful=False))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1, activation='sigmoid'))\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nmodel.fit(train_X_samp, train_Y_samp, epochs=epochs, batch_size=batch_size, verbose=2, shuffle=True)\n#model.reset_states()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.mkdir('/kaggle/working/models')\nmodel.save('/kaggle/working/models/all-augment-400.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluate Results"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import accuracy_score\nfrom sklearn.metrics import precision_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import cohen_kappa_score\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import precision_recall_curve\nfrom matplotlib import pyplot\n\n\n# predict probabilities for test set\nyhat_probs = model.predict(test_X_arr, verbose=0)\n# predict crisp classes for test set\nyhat_classes = model.predict_classes(test_X_arr, verbose=0)\n# reduce to 1d array\nyhat_probs = yhat_probs[:, 0]\nyhat_classes = yhat_classes[:, 0]\n \n# accuracy: (tp + tn) / (p + n)\naccuracy = accuracy_score(test_Y, yhat_classes)\nprint('Accuracy: %f' % accuracy)\n# precision tp / (tp + fp)\nprecision = precision_score(test_Y, yhat_classes)\nprint('Precision: %f' % precision)\n# recall: tp / (tp + fn)\nrecall = recall_score(test_Y, yhat_classes)\nprint('Recall: %f' % recall)\n# f1: 2 tp / (2 tp + fp + fn)\nf1 = f1_score(test_Y, yhat_classes)\nprint('F1 score: %f' % f1)\n \n# kappa\nkappa = cohen_kappa_score(test_Y, yhat_classes)\nprint('Cohens kappa: %f' % kappa)\n# ROC AUC\nauc = roc_auc_score(test_Y, yhat_probs)\nprint('ROC AUC: %f' % auc)\n\nlr_precision, lr_recall, _ = precision_recall_curve(test_Y, yhat_probs)\nno_skill = len(yhat_classes[yhat_classes==1]) / len(yhat_classes)\n\npyplot.plot([0, 1], [no_skill, no_skill], linestyle='--', label='No Skill')\npyplot.plot(lr_recall, lr_precision, marker='.', label='NN Classifier')\n# axis labels\npyplot.xlabel('Recall')\npyplot.ylabel('Precision')\n# show the legend\npyplot.legend()\n# show the plot\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\ncm = confusion_matrix(test_Y, yhat_classes)\nax= plt.subplot()\nsns.heatmap(cm, annot=True, ax = ax); #annot=True to annotate cells\n\n# labels, title and ticks\nax.set_xlabel('Predicted labels');ax.set_ylabel('True labels'); \nax.set_title('Confusion Matrix'); \nax.xaxis.set_ticklabels(['fake', 'real']); ax.yaxis.set_ticklabels(['fake', 'real']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(yhat_probs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Examine misclassified framesets"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_names[0:3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#from itertools import compress\n#error_idx = (test_Y != yhat_classes)\n#list(compress(test_names, error_idx))\n\neval_dict = {'tp': [], 'fp': [], 'tn': [], 'fn': []}\ndef make_eval_dict(eval_dict, test_Y, yhat_classes):\n    for i in range(len(test_Y)):\n        eval_array = [test_Y[i], yhat_classes[i]]\n        fname = test_names[i]\n        if eval_array == [1, 0]:\n            eval_dict['fn'].append(fname)\n        \n        elif eval_array == [0, 0]:\n            eval_dict['tn'].append(fname)\n        \n        elif eval_array == [0, 1]:\n            eval_dict['fp'].append(fname)\n        \n        else:\n            eval_dict['tp'].append(fname)\n    return eval_dict\n\neval_dict = make_eval_dict(eval_dict, test_Y, yhat_classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# quick check\nfor k, v in eval_dict.items():\n    print(k, len(v))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.core.display import display, Image\n\ndef show_errors(eval_dict, error_type, limit=20):\n    \"\"\"Get examples of videos where model makes mistakes\"\"\"\n    lookup_videos = eval_dict[error_type]\n    for lv in lookup_videos[:limit]:\n        print(lv)\n        frames = df.loc[df['file_path'] == lv, 'selected_frames'].values\n        real_fake = df.loc[df['file_path'] == lv, 'label'].values[0]\n        print(real_fake)\n        for frame in frames:\n            frame_path = lv + '/' + frame\n            display(Image(filename=frame_path))\n            \nshow_errors(eval_dict, 'fp')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}