{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Fast Fake Face Only Extractor\n## This notebook is an update of this [work](https://www.kaggle.com/phunghieu/deepfake-detection-face-extractor). \n## Thank you, Phung.\n## If it is useful for you, please consider your upvote."},{"metadata":{},"cell_type":"markdown","source":"- REAL only face extract with MTCNN FACTOR = 0.9\n- FAKE only cuts out images using coordinates acquired by REAL\n- Cluster faces in nearby coordinates. The top two clusters with the largest number of face images are set as person candidates\n- Take the pixel difference between REAL image and FAKE image and calculate the standard deviation. A face image having a large standard deviation is set as a final Fake image candidate.\n- This script has continue mode\n- Using GCP 64 CPUs(no GPU), you get all face images of dfdc_train_all in 28 hours. output: about 1.5TB, 1.6M faces (almost fake only :-p)"},{"metadata":{},"cell_type":"markdown","source":"# fffoe.py script part\n- [For example] \n- \\$ nohup python fffoe.py part_0 &\n- \\$ nohup python fffoe.py part_1 &\n- \\$ nohup python fffoe.py part_2 &\n- ...\n- \\$ nohup python fffoe.py part_49 &"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install facenet-pytorch > /dev/null 2>&1\n!apt install zip > /dev/null 2>&1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls ../input/dfdc-full-data-csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mkdir /kaggle/working/tmp","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"##################################\nTMP_DIR = '/kaggle/working/tmp'\nFULL_DATA_CSV = '../input/dfdc-full-data-csv/full_data.csv'\nDFDC_FULL_PATH ='../input/dfdc-train-part-04/'\nCANDIDATE_ALL = 'candidate_all'\n\n##################################\n# for MTCNN and CV2\nSCALE = 0.25\nN_FRAMES = None\n\n##################################\n# parameters\n##################################\n# for MTCNN\nMARGIN=0\nSIZE=160\nFACTOR = 0.9  # 0.5: 27s, 0.7:30s, 0.9:70s, 1.0:no end\n#MAX_FRAMES=27   # for DEBUG\nMAX_FRAMES=300\n\n##################################\n# for FACE cluster matching\n# matching range in M_FRAME frames\nM_FRAME = 50\n# video frame size is 1920x1080 or 1080x1920\nNEAR_X = 100 # about 5% of 1920 \nNEAR_Y = 50  # about 5% of 1080 \nFACE_RATE = 1.1  # for face size matching\n##################################\n# For CopyCandidate\n# number os train candidate faces\nCANDIDATE = 10\n# threshold of std\nSTD_TH = 4.5\n##################################\n\ndef auto_contrast(img):\n    # BGR->HSV\n    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)\n    # V histogram equalize\n    hsv[:,:,2] = cv2.equalizeHist(hsv[:,:,2])\n    # HSV->BGR\n    bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR) \n    \n    return bgr\n\n###########################################################\nimport PIL\nPIL.PILLOW_VERSION = PIL.__version__\n\nimport os\nimport sys\nimport shutil\nimport re\nimport glob\nimport json\nimport torch\nimport cv2\nfrom PIL import Image\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nimport matplotlib.patches as patches\nfrom tqdm.auto import tqdm\nfrom facenet_pytorch import MTCNN\n\nfrom functools import wraps\nimport time\n\ndevice = 'cuda:0' if torch.cuda.is_available() else 'cpu'\nprint(f'Running on device: {device}')\n\n###########################################################\nDEBUG = False\ndef isSameCenter(sxc, syc, xc, yc):\n    ret = True\n    \n    if (sxc-NEAR_X > xc) | (sxc+NEAR_X < xc):\n        if DEBUG :\n            print(f'False1:{sxc},{xc}')\n        return False\n    elif (syc-NEAR_Y > yc) | (syc+NEAR_Y < yc):\n        if DEBUG :\n            print(f'False2:{syc},{yc}')\n        return False\n    # currently not using FACE size, if it used then clusters could be more divided\n    \n    return ret\n\n###########################################################\ndef InitDrawImage(draw_flag, ax, frame):\n    if draw_flag is not True:\n        return\n\n    ax.imshow(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))\n    ax.axis('off')\n\n    return\n\ndef DrawLandmarks(draw_flag, ax, box, landmark):\n    if draw_flag is not True:\n        return\n\n    xl, yb, xr, yt = box[0], box[1], box[2], box[3]\n    width, height=xr-xl, yt-yb\n    \n    #draw corner\n    ax.scatter(*np.meshgrid(box[[0, 2]], box[[1, 3]]))\n    #draw landmark\n    ax.scatter(landmark[:, 0], landmark[:, 1], s=8)\n    #draw rectangle\n    r = patches.Rectangle(xy=(xl, yb), width=width, height=height, ec='green', fill=False)\n    ax.add_patch(r)\n\n    return\n\ndef SaveLandmarkPlot(draw_flag, save_dir, file_base, f_num):\n    if draw_flag is not True:\n        plt.close('all')\n        return\n    \n    #save plot\n    save_path = os.path.join(save_dir, file_base+f'{f_num:04}.png')\n    plt.savefig(save_path) \n    plt.close('all')\n    \n    return\n\ndef SaveCharts(df, save_dir, file_base):\n    plt.figure()\n    df.plot(kind='scatter', x='xc', y='yc', c='c_id', cmap='rainbow', xlim=(0,1920), ylim=(1080,0), title=f'{file_base}:xc-yc graph')\n    plt.savefig(os.path.join(save_dir, 'graph-xc-yc'), dpi=300)\n    plt.close('all')\n            \n    plt.figure()\n    df.plot(kind='scatter', x='frame', y='yc', c='c_id', cmap='rainbow', ylim=(1080,0), title=f'{file_base}:frame-yc graph')\n    plt.savefig(os.path.join(save_dir, 'graph-frame-yc'), dpi=300)\n    plt.close('all')\n    \n    plt.figure()            \n    ax = df.plot(kind='scatter', x='frame', y='xc', c='c_id', cmap='rainbow', ylim=(1920,0), title=f'{file_base}:frame-xc graph')\n    plt.savefig(os.path.join(save_dir, 'graph-frame-xc'), dpi=300)\n    plt.close('all')\n\n    return\n\ndef SaveStdCharts(df, save_dir, file_base):\n    plt.figure()\n    df.plot(kind='scatter', x='frame', y='std', c='c_id', cmap='rainbow', title=f'{file_base}:frame-std graph')\n    plt.savefig(os.path.join(save_dir, 'graph-frame-std'), dpi=300)\n    plt.close('all')\n    \n    return\n\n###########################################################\ndef MoveClusters(df, save_dir):\n    # get TOP 2 clusters\n    Clusters = df['cluster'].value_counts().index[:2]\n    #if 2nd cluster length is < 5, only use TOP 1 cluster\n    if len(Clusters) > 1:\n        if len(df[df['cluster']==Clusters[1]]) < 5:\n            # use only 1st cluster\n            Clusters = df['cluster'].value_counts().index[:1]\n    \n    # move to cluster folders\n    for c in Clusters:\n        path_c = os.path.join(save_dir, c)\n        if not os.path.exists(path_c):\n            os.makedirs(path_c)\n\n        #for i, row in df[df['cluster']==c].iterrows():\n        #    png_file = os.path.join(save_dir, row['png'])\n        #    if os.path.exists(png_file):\n        #        shutil.move(png_file, path_c)\n        for png in df[df['cluster']==c]['png']:\n            png_file = os.path.join(save_dir, png)\n            if os.path.exists(png_file):\n                shutil.move(png_file, path_c)\n    \n    # remaining png -> etc\n    path_etc = os.path.join(save_dir, 'etc')\n    if not os.path.exists(path_etc):\n        os.makedirs(path_etc)\n    remaining_png = glob.glob(os.path.join(save_dir, '*_????_????.png'))\n    for png_file in remaining_png:\n        if os.path.exists(png_file):\n            shutil.move(png_file, path_etc)\n    \n    return\n\n###########################################################\nclass FaceCExtractor:\n    def __init__(self, detector, n_frames=None, resize=None, max_frames=None, margin=0, size=160):\n        \"\"\"\n        Parameters:\n            n_frames {int} -- Total number of frames to load. These will be evenly spaced\n                throughout the video. If not specified (i.e., None), all frames will be loaded.\n                (default: {None})\n            resize {float} -- Fraction by which to resize frames from original prior to face\n                detection. A value less than 1 results in downsampling and a value greater than\n                1 result in upsampling. (default: {None})\n            max_frames [int] -- Maximum number of frames to process.\n        \"\"\"\n\n        self.detector = detector\n        self.n_frames = n_frames\n        self.resize = resize\n        self.max_frames = max_frames\n        self.margin = margin\n        self.size = size\n    \n    def __call__(self, filename, tmp_dir):\n        \"\"\"Load frames from an MP4 video, detect faces and save the results.\n\n        Parameters:\n            filename {str} -- Path to video.\n            tmp_dir  [str] -- The tmp directory for save\n        \"\"\"\n        \n        file_base = os.path.splitext(os.path.basename(filename))[0]\n        save_dir = os.path.join(tmp_dir, file_base)\n        if os.path.exists(save_dir):\n            #shutil.rmtree(save_dir)\n            # continue with using save_dir\n            return\n        else:\n            os.makedirs(save_dir)\n\n        face_df = pd.DataFrame()\n        idx = 0\n        \n        # high speed DataFrame generate : list -> pd \n        df = pd.DataFrame()\n        video_list = [] \n        png_list = [] \n        frame_list = [] \n        xc_list = [] \n        yc_list = [] \n        l_list = []\n        w_list = [] \n        h_list = []  \n        xl_list = []\n        xr_list = []\n        yb_list = []\n        yt_list = []\n        cluster_list = []\n\n        \n        save_dir = os.path.join(tmp_dir, file_base)\n\n        # Create video reader and find length\n        v_cap = cv2.VideoCapture(filename)\n        v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n        # for temporary\n        loop = v_len if self.max_frames is None else min(v_len, self.max_frames)\n        # Loop through frames\n        for j in tqdm(range(loop), desc=f'MTCNN f{FACTOR}:{file_base}'):\n        #for j in range(loop):\n            success = v_cap.grab()\n\n            # Load frame\n            success, frame_cv2 = v_cap.retrieve()\n            if not success:\n                continue\n                    \n            frame_cv2 = auto_contrast(frame_cv2)\n\n            # Resize frame to desired size\n            if self.resize is not None:\n                frame_cv2_re = cv2.resize(frame_cv2, dsize=None, fx=self.resize, fy=self.resize)\n            else:\n                frame_cv2_re = frame_cv2\n            \n            frame_h, frame_w, frame_ch = frame_cv2_re.shape[:3]\n                \n            # Detect face\n            boxes, probs, landmarks = self.detector.detect(frame_cv2_re, landmarks=True)\n            \n            # Visualize for DEBUG\n            draw_flag = True if j==0 else False\n            #draw_flag = True # all frame draw for DEBUG\n            \n            ### Draw\n            #plt.figure()\n            fig, ax = plt.subplots(figsize=(16/2, 12/2))\n            plt.subplots_adjust(left=0, right=1, bottom=0, top=1)\n            InitDrawImage(draw_flag, ax, frame_cv2_re)\n\n            if boxes is None:\n                ### just Image draw and closing and skip frame loop\n                SaveLandmarkPlot(draw_flag, save_dir, file_base, j)\n                continue\n            \n            for box, landmark in zip(boxes, landmarks):\n                xl, yb, xr, yt = box[0], box[1], box[2], box[3]\n                if xl<0 or xl>frame_w or xr<0 or xr>frame_w or yb<0 or yb>frame_h or yt<0 or yt>frame_h:\n                    # print('SKIP!!')\n                    # invalid float number: skip this face\n                    continue\n                width, height=xr-xl, yt-yb\n                ### Draw\n                DrawLandmarks(draw_flag, ax, box, landmark)\n                    \n                # cut & save face rectangle\n                length = width if width > height else height\n                #print(f'frame {j}:length={length}')\n                        \n                xc, yc = (xl+xr)/2, (yb+yt)/2\n                    \n                # Square BOX corners with MARGIN\n                # clip in resized small framesize for MTCNN\n                bxl = min(frame_w, max(0, xc - (length+self.margin)/2))\n                bxr = min(frame_w, max(0, xc + (length+self.margin)/2))\n                byb = min(frame_h, max(0, yc - (length+self.margin)/2))\n                byt = min(frame_h, max(0, yc + (length+self.margin)/2))\n                if (bxr-bxl)<10 or (byt-byb)<10:\n                    # print('SKIP!!')\n                    # too small or out of screen : skip this face\n                    continue\n\n                # SCALED corners\n                sxl = bxl/self.resize\n                sxr = bxr/self.resize\n                syb = byb/self.resize\n                syt = byt/self.resize\n                sxc = xc/self.resize\n                syc = yc/self.resize\n                sl  = length/self.resize\n                sw  = width/self.resize\n                sh  = height/self.resize\n                \n                face_cv2 = frame_cv2[int(syb):int(syt), int(sxl):int(sxr)]\n                #print(f'frame {j}: {syb}, {syt}, {sxl}, {sxr}')  # for DEBUG\n                face_cv2 = cv2.resize(face_cv2, dsize=(self.size, self.size))\n                png_file = f'{file_base}_r_{j:04}_{int(sxc):04}_{int(syc):04}.png'\n                save_face_path = os.path.join(save_dir, png_file)\n                cv2.imwrite(save_face_path, face_cv2)\n                \n                #idx = f'{file_base}_{j:04}_{int(sxc):04}_{int(syc):04}'\n                # high speed DataFrame generate : list -> pd \n                video_list.append(file_base) \n                png_list.append(png_file) \n                frame_list.append(j) \n                xc_list.append(sxc) \n                yc_list.append(syc) \n                l_list.append(sl)\n                w_list.append(sw) \n                h_list.append(sh) \n                xl_list.append(sxl) \n                xr_list.append(sxr) \n                yb_list.append(syb) \n                yt_list.append(syt) \n\n                \n                c = f'{int(sxc):04}_{int(syc):04}'\n                if j >= 1:\n                    #for index, row in face_df[(face_df['video']==file_base) & (face_df['frame']==j-1)].iterrows():\n                    # matching range in M_FRAME frames\n                    stop = len(xc_list)-1 # until previous frame\n                    start = max(0, stop-M_FRAME)  # matching range\n                    for i in range(start, len(xc_list)-1):\n                        if isSameCenter(sxc, syc, xc_list[i], yc_list[i]):\n                            c = cluster_list[i]\n                    \n                #face_df.loc[idx, 'cluster'] = c\n                cluster_list.append(c)\n\n            ### Draw\n            SaveLandmarkPlot(draw_flag, save_dir, file_base, j)\n                \n        v_cap.release()\n        \n        # high speed DataFrame generate : list -> pd \n        df = pd.DataFrame(\n            data={'video': video_list, 'png': png_list, 'frame': frame_list, \n                  'xc': xc_list, 'yc': yc_list, \n                  'length': l_list, 'width': w_list, 'height' : h_list, \n                  'xl': xl_list, 'xr' : xr_list, 'yb' : yb_list, 'yt' : yt_list, \n                  'cluster' : cluster_list},\n            columns=['video', 'png', 'frame', 'xc', 'yc', 'length', 'width', 'height', 'xl', 'xr', 'yb', 'yt', 'cluster']\n        )\n        \n        \n        # create cluster_id\n        df['c_id'] = pd.factorize(df['cluster'])[0]\n\n        # Draw & Save Charts\n        SaveCharts(df, save_dir, file_base)\n        \n        # Cluster: move to sub-folders\n        MoveClusters(df, save_dir)\n        \n        # save csv for face information\n        df.to_csv(os.path.join(save_dir, file_base+'.csv'))\n\n        return\n    \n###########################################################\ndef CopyCandidate(fake_df, fake_tmp_dir, fake_base, real_tmp_dir, real_base):\n    if not os.path.exists(fake_tmp_dir):\n        print('ERROR! not exists:', fake_tmp_dir)\n        return\n    if not os.path.exists(real_tmp_dir):\n        print('ERROR! not exists:', real_tmp_dir)\n        return\n    \n    # reduce fake_df\n    c_df = fake_df.loc[:, ['png', 'frame', 'cluster', 'std']]\n    #c_df = fake_df.copy()\n    # sort by 'std'\n    c_df = c_df.sort_values(by=['std'], ascending=False)\n    # get clusters form FAKE foder\n    Clusters = glob.glob(os.path.join(fake_tmp_dir, fake_base, '????_????'))\n\n    candidate_dir = os.path.join(fake_tmp_dir, fake_base, 'candidate')\n    os.makedirs(candidate_dir)\n    \n    candidate_all_dir = os.path.join(real_tmp_dir, '..', CANDIDATE_ALL)\n    if not os.path.exists(candidate_all_dir):\n        os.makedirs(candidate_all_dir)\n    candidate_all_dir = os.path.join(candidate_all_dir, real_base)\n    if not os.path.exists(candidate_all_dir):\n        os.makedirs(candidate_all_dir)\n    \n    for path_c in Clusters :\n        c = os.path.basename(path_c)\n        #print(c)\n        df = c_df[c_df['cluster']==c]\n        df = df.reset_index()\n        for i, row in df.iterrows():\n            #print(i, row['png'], row['cluster'], row['std'])\n            fake_png = row['png']\n            base, ext = os.path.splitext(fake_png)\n            std = row['std']\n            fake_new_png = real_base + '-' + base + f'_{std:05.1f}' + ext\n            \n            cluster = row['cluster']\n            real_png = fake_png.replace(fake_base, real_base).replace('_f_', '_r_')\n            if row['std'] > STD_TH:\n                shutil.copy2(os.path.join(fake_tmp_dir, fake_base, cluster, fake_png), os.path.join(candidate_dir, fake_new_png))\n                shutil.copy2(os.path.join(real_tmp_dir, real_base, cluster, real_png), os.path.join(candidate_dir, real_png))\n\n                shutil.copy2(os.path.join(fake_tmp_dir, fake_base, cluster, fake_png), os.path.join(candidate_all_dir, fake_new_png))\n                shutil.copy2(os.path.join(real_tmp_dir, real_base, cluster, real_png), os.path.join(candidate_all_dir, real_png))\n\n            # copy only CANDIDATE files\n            if i >= CANDIDATE-1:\n                break\n    \n    return\n\n###########################################################\ndef fake_diff_extractor(filename, fake_tmp_dir, real_tmp_dir, real_base):\n    file_base = os.path.splitext(os.path.basename(filename))[0]\n    fake_base = file_base\n    save_dir = os.path.join(fake_tmp_dir, file_base)\n    if os.path.exists(save_dir):\n        #shutil.rmtree(save_dir)\n        # continue with using save_dir\n        return\n    else:\n        os.makedirs(save_dir)\n\n    real_dir = os.path.join(real_tmp_dir, real_base)\n    real_csv = os.path.join(real_dir, real_base+'.csv')\n    real_df = pd.read_csv(real_csv, index_col=0)\n    fake_df = real_df\n    fake_df['video'] = fake_base\n    \n    save_dir = os.path.join(fake_tmp_dir, file_base)\n    save_csv = os.path.join(save_dir, real_base+'.csv')\n\n    # copy: csv, init_image, charts\n    #shutil.copy2(real_csv, save_csv)\n    shutil.copy2(os.path.join(real_dir, real_base+'0000.png'), os.path.join(save_dir, real+'0000.png'))\n    shutil.copy2(os.path.join(real_dir, 'graph-frame-xc.png'), os.path.join(save_dir, 'graph-frame-xc.png'))\n    shutil.copy2(os.path.join(real_dir, 'graph-frame-yc.png'), os.path.join(save_dir, 'graph-frame-yc.png'))\n    shutil.copy2(os.path.join(real_dir, 'graph-xc-yc.png'), os.path.join(save_dir, 'graph-xc-yc.png'))\n    \n    # Create video reader and find length\n    v_cap = cv2.VideoCapture(filename)\n    v_len = int(v_cap.get(cv2.CAP_PROP_FRAME_COUNT))\n\n    fake_df['std'] = fake_df['png']\n    loop = min(v_len, MAX_FRAMES)\n    # Loop through frames\n    for j in range(loop):\n        success = v_cap.grab()\n\n        # Load frame\n        success, frame_cv2 = v_cap.retrieve()\n        if not success:\n            continue\n                    \n        frame_cv2 = auto_contrast(frame_cv2)\n\n        # Detect face\n        #boxes, probs, landmarks = self.detector.detect(frame_cv2_re, landmarks=True)\n        frame_df = real_df[real_df['frame']==j]\n        if len(frame_df)==0:   # case : no face\n            continue\n        \n        for real_png, sxc, syc, sxl, sxr, syb, syt, c in zip(frame_df['png'], frame_df['xc'], frame_df['yc'], \n                                                             frame_df['xl'], frame_df['xr'], frame_df['yb'], frame_df['yt'],\n                                                            frame_df['cluster']):\n\n            fake_face_cv2 = frame_cv2[int(syb):int(syt), int(sxl):int(sxr)]\n            fake_face_cv2 = cv2.resize(fake_face_cv2, dsize=(SIZE, SIZE))\n            fake_png = f'{fake_base}_f_{j:04}_{int(sxc):04}_{int(syc):04}.png'\n            fake_face_path = os.path.join(save_dir, fake_png)\n            cv2.imwrite(fake_face_path, fake_face_cv2)\n            \n            fake_df['png'] = fake_df['png'].str.replace(real_png, fake_png)\n            \n            real_face_path = os.path.join(real_dir, c, real_png)\n            if os.path.exists(real_face_path):\n                real_face_cv2 = cv2.imread(real_face_path)\n                \n                img_diff=abs(fake_face_cv2.astype(\"int\") - real_face_cv2.astype(\"int\")).astype(\"uint8\")\n                #img_diff=abs(auto_contrast(fake_face_cv2).astype(\"int\") - auto_contrast(real_face_cv2).astype(\"int\")).astype(\"uint8\")\n                \n                #diff_mean = np.mean(img_diff)\n                diff_std  = np.std(img_diff)\n                fake_df['std'] = fake_df['std'].replace(real_png, diff_std)\n                \n\n    v_cap.release()\n    \n    # out of clusters\n    fake_df['std'] = fake_df['std'].replace(real_base, 0, regex=True)\n    \n    # Cluster: move to sub-folders\n    MoveClusters(fake_df, save_dir)\n\n    # save csv for face information\n    fake_df.to_csv(os.path.join(save_dir, fake_base+'.csv'))\n    \n    # save std charts\n    SaveStdCharts(fake_df, os.path.join(fake_tmp_dir, fake_base), fake_base)\n    \n    # Copy the face-png file of the high std score along with the corresponding real-png to the 'candidate' folder\n    CopyCandidate(fake_df, fake_tmp_dir, fake_base, real_tmp_dir, real_base)\n    \n    return\n\n###########################################################\n### log\nimport datetime\n\n#path_log = \"../tmp-c/tmp/log.txt\"\n\ndef log_file_base(label_str, file_base, file_part, tmp_dir):\n    m=re.match('^part_\\d+$', file_part)\n    if not m:\n        file_part = \"no-part\"\n\n    path_log = os.path.join(tmp_dir, 'log.txt')\n    with open(path_log, mode='a') as f:\n        dt_now = datetime.datetime.now()\n        if label_str is None:\n            head = ''\n        elif label_str == 'FAKE':\n            head = '  FAKE '\n        else:\n            head ='REAL '\n        \n        f.write(dt_now.strftime(head+'[%Y%m%d %H:%M:%S] '))\n        f.write(file_base + ':' + file_part +  \" ... \")  \n    return\n\ndef log_skip(label_str, file_base, file_part, tmp_dir):\n    path_log = os.path.join(tmp_dir, 'log.txt')\n    with open(path_log, mode='a') as f:\n        dt_now = datetime.datetime.now()\n        if label_str == 'REAL':\n            head = '[SKIP REAL]'\n        else:\n            head = '  [SKIP FAKE]'\n        f.write(dt_now.strftime(head+'[%Y%m%d %H:%M:%S] '))\n\n    return\n    \ndef log_done(tmp_dir):\n    path_log = os.path.join(tmp_dir, 'log.txt')\n    with open(path_log, mode='a') as f:\n        f.write(\"done\\n\")\n    return\n\ndef log_fixdone(tmp_dir):\n    path_log = os.path.join(tmp_dir, 'log.txt')\n    with open(path_log, mode='a') as f:\n        f.write(\"fix done\\n\")\n    return\n\ndef log_finish(tmp_dir):\n    path_log = os.path.join(tmp_dir, 'log.txt')\n    with open(path_log, mode='a') as f:\n        f.write(\"finish!\\n\")\n    return\n\n###########################################################\ndef load_full_real():\n    ########################\n    # load full_df\n    ########################\n    '''\n    [NOTICE] These videos are missing.\n    Part18：\n      pvohowzowy.mp4 : Fake\n      wipjitfmta.mp4 : Fake\n      wpuxmawbkj.mp4 : Fake\n    Part35\n      cfxiikrhep.mp4 : Fake\n      dzjjtfwiqc.mp4 : Fake\n      glleqxulnn.mp4 : Fake\n      innmztffzd.mp4 : Fake\n      zzfhqvpsyp.mp4 : Fake\n    \n    '''\n    full_df = pd.read_csv(FULL_DATA_CSV, index_col=0)\n    # delete missing video rows\n    #     not is in [list]\n    full_df = full_df[~full_df['filename'].isin([\n        'pvohowzowy.mp4', \n        'wipjitfmta.mp4', \n        'wpuxmawbkj.mp4',\n        'cfxiikrhep.mp4', \n        'dzjjtfwiqc.mp4', \n        'glleqxulnn.mp4', \n        'innmztffzd.mp4', \n        'zzfhqvpsyp.mp4'])]\n\n    ########################\n    # make real_df\n    ########################\n    #### create real_df\n    real_df = full_df[['orig_part', 'original', 'orig_label']]\n    # counts duplicated number\n    vc = real_df['original'].value_counts()\n    # drop duplicates\n    real_df = real_df.drop_duplicates()\n    # set index\n    real_df.set_index('original', inplace=True)\n    # add duplicated number\n    for real in vc.index:\n        real_df.loc[real, 'fake_num'] = vc[real]\n    # sort by reverse order, top is big number\n    real_df=real_df.sort_values(by=['fake_num'], ascending=False)\n    \n    return full_df, real_df\n\n###########################################################\n###########################################################\n### Load face detector\n#face_detector = MTCNN(image_size=SIZE, margin=14, keep_all=True, factor=0.5, device=device).eval()\nface_detector = MTCNN(image_size=SIZE, margin=MARGIN, keep_all=True, factor=FACTOR, device=device).eval()\n\n## Define face extractor\nface_extractor = FaceCExtractor(detector=face_detector, n_frames=N_FRAMES, resize=SCALE, \n                                max_frames=MAX_FRAMES, margin=MARGIN, size=SIZE)\n###########################################################\n# about 5 sec\nfull_df, real_df = load_full_real()\n\n###########################################################\n# Real->Fake loop\n######################\n# select input data\n######################\n\n# get command line args\nargs = sys.argv\nprint(args)\n# check invalid args and jupyter notebook \nif (len(args) == 2) and (not args[1]=='-f()') : # check args and no jupyter notebook\n    # set run mode by command line args\n    m = re.match(r'(part_\\d\\d*)', args[1])\n    if m :\n        SELECT_PART = m.group()\n        real_df=real_df[real_df['orig_part']==SELECT_PART]\n    else :\n        print('ERROR : invalid args')\n        exit(1)\nelse : \n    # set run mode by manualy here\n    SELECT_PART=''    # MUST need this line\n    ### select part\n    #SELECT_PART='part_4'\n\n    # remove comment out, if you want to use SELECT_PART\n    #real_df=real_df[real_df['orig_part']==SELECT_PART]\n\n    ##### SELECT_PART or SELECT_REAL #####\n\n    ### select real\n    #SELECT_REAL=['bxgkydnxzv.mp4', 'vudstovrck.mp4']\n    SELECT_REAL=['bgpoldvzrh.mp4']\n\n    # remove comment out, if you want to use SELECT_REAL\n    real_df=real_df.reset_index()\n    real_df=real_df[real_df['original'].isin(SELECT_REAL)]\n    real_df=real_df.set_index('original')\n\n\n###########################################################\n###########################\n### main Real-to-Fake loop\n###########################\nif SELECT_PART == '':\n    print('selecting real:', SELECT_REAL)\n    TMP_DIR_PART=TMP_DIR\nelse:\n    print('selecting part:', SELECT_PART)\n    TMP_DIR_PART=TMP_DIR+'_'+ SELECT_PART\n\n##################################################\n# Check TMP_DIR_PART with log.txt and try continue\n##################################################\nif os.path.exists(TMP_DIR_PART) and SELECT_PART != \"\":\n    print(f'Checking [{TMP_DIR_PART}] for continue')\n    \n    log_real = None\n    log_fake = None\n    continue_flag = False\n    if os.path.exists(os.path.join(TMP_DIR_PART, 'log.txt')):\n        with open(os.path.join(TMP_DIR_PART, 'log.txt'),'r') as f:\n            for line in f:\n                # for DEBUG\n                #print(line.strip())\n                log_label = None\n                log_base = None\n                m = re.search(r'^finish!$', line)\n                if m :\n                    print(\"previous log.txt is finishd!\")\n                    exit(0)\n                m = re.search(r'^\\s*(REAL|FAKE).* (..........):part.*', line)\n                if m :\n                    log_label = m.group(1)\n                    log_base = m.group(2)\n                    if log_label == 'REAL':\n                        log_real = log_base\n                    else:\n                        log_fake = log_base\n                    #print(f'label:{log_label}, base:{log_base}, log_real:{log_real}')\n                else :\n                    print('log.txt format error:', line)\n                    exit(1)\n\n                m = re.search(r'(done)$', line)\n                if not m :\n                    if log_label == \"REAL\":\n                        path_log_base = os.path.join(TMP_DIR_PART, log_real)\n                        if os.path.exists(path_log_base):\n                                shutil.rmtree(path_log_base)\n                    else:\n                        path_log_base = os.path.join(TMP_DIR_PART, log_real, 'FAKES', log_fake)\n                        if os.path.exists(path_log_base):\n                                shutil.rmtree(path_log_base)\n\n                    print('Continue OK:', line.strip())\n                    continue_flag = True\n\n    if continue_flag:\n        log_fixdone(TMP_DIR_PART)\n    else:\n        print(f'Fail continue')\n        print(f'Removing [{TMP_DIR_PART}] ... ', end='')\n        shutil.rmtree(TMP_DIR_PART)\n        print('done')\n        os.makedirs(TMP_DIR_PART)\nelse:\n    if os.path.exists(TMP_DIR_PART):\n        shutil.rmtree(TMP_DIR_PART)\n        os.makedirs(TMP_DIR_PART)\n\n##################\n#### real loop\n##################\nwith tqdm(real_df.index) as preal:\n    for i, real in enumerate(preal):\n        #print(i, real, real_df.loc[real, 'fake_num'])\n        real_base = os.path.splitext(os.path.basename(real))[0]\n        preal.set_description(f'[REAL]{real_base}')\n        \n        fake_df = full_df[full_df['original']==real]\n        fake_df.set_index('filename', inplace=True)\n        \n        real_part = real_df.loc[real]['orig_part']\n        path_movie = os.path.join(DFDC_FULL_PATH, 'dfdc_train_'+real_part, real)\n\n        if not os.path.exists(path_movie):\n            log_skip('REAL', real_base, real_part, TMP_DIR_PART)\n            continue\n        \n        #### extract real\n\n        #### for DEBUG\n        #print('REAL:', real, real_part)\n        \n        log_file_base('REAL', real_base, real_part, TMP_DIR_PART)\n        #start = time.time()\n        face_extractor(path_movie, TMP_DIR_PART)\n        #elapsed_time = time.time() - start\n        #preal.set_postfix(time=elapsed_time)\n        log_done(TMP_DIR_PART)\n        \n        ##################\n        #### fake loop\n        ##################\n        real_tmp_dir = os.path.join(TMP_DIR_PART, real_base)\n        fake_df.to_csv(os.path.join(real_tmp_dir, real_base+'-'+real_part+'-fakes.csv'))\n                       \n        fake_tmp_dir = os.path.join(real_tmp_dir, 'FAKES')\n        if not os.path.exists(fake_tmp_dir):\n            os.makedirs(fake_tmp_dir)\n\n        with tqdm(fake_df.index, desc='FAKE') as pfake:\n            for j, fake in enumerate(pfake):\n                fake_base = os.path.splitext(os.path.basename(fake))[0]\n                pfake.set_description(f' [FAKE]{fake_base}')\n\n                fake_part = fake_df.loc[fake, 'part']\n                real = fake_df.loc[fake, 'original']\n            \n                real_base = os.path.splitext(os.path.basename(real))[0]\n\n                #### for DEBUG\n                #print(\"FAKE:\", fake, fake_part, real, real_part)\n            \n                fake_part=fake_df.loc[fake, 'part']\n                path_movie = os.path.join(DFDC_FULL_PATH, 'dfdc_train_'+fake_part, fake)\n\n                if not os.path.exists(path_movie):\n                    log_skip('SKIP', fake_base, fake_part, TMP_DIR_PART)\n                    continue\n                \n                #### extract fake\n                log_file_base('FAKE', fake_base, fake_part, TMP_DIR_PART)\n                fake_diff_extractor(path_movie, fake_tmp_dir, TMP_DIR_PART, real_base)\n                log_done(TMP_DIR_PART)\n            \n                ### for DEBUG\n                ### break FAKE loop\n                #if j>=1:\n                #    break\n        \n        ### for DEBUG\n        ### break REAL loop\n        #if i=>0:\n        #    break\n\nlog_finish(TMP_DIR_PART)\n\n###########################################################","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls /kaggle/working/candidate_all/bgpoldvzrh","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### face file naming rule\n\nREAL: [ream movie]_r_[frame no]_[center x]_[center y].png\n\nFAKE: [base real movie]-[fake movie]_f_[frame no]_[center x]_[center y]_[diff-std].png"},{"metadata":{"trusted":true},"cell_type":"code","source":"ls /kaggle/working/tmp/bgpoldvzrh/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# [REAL]\n## Create a cluster and select two candidates"},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_png(png):\n    ffig, ax = plt.subplots(figsize=(16/2, 12/2))\n    plt.subplots_adjust(left=0, right=1, bottom=0, top=1)\n    f = cv2.imread(png)\n    ax.imshow(cv2.cvtColor(f, cv2.COLOR_BGR2RGB))\n    ax.axis('off')\n    plt.show()\n    return\n\ndef draw_png1_4(png1, png2, png3, png4):\n    f1 = cv2.imread(png1)\n    f2 = cv2.imread(png2)\n    f3 = cv2.imread(png3)\n    f4 = cv2.imread(png4)\n\n    fig = plt.figure(figsize=(16, 12))\n    \n    ax1 = fig.add_subplot(1, 4, 1)\n    ax2 = fig.add_subplot(1, 4, 2)\n    ax3 = fig.add_subplot(1, 4, 3)\n    ax4 = fig.add_subplot(1, 4, 4)\n\n    ax1.axis('off')\n    ax2.axis('off')\n    ax3.axis('off')\n    ax4.axis('off')\n\n    ax1.imshow(cv2.cvtColor(f1, cv2.COLOR_BGR2RGB))\n    ax2.imshow(cv2.cvtColor(f2, cv2.COLOR_BGR2RGB))\n    ax3.imshow(cv2.cvtColor(f3, cv2.COLOR_BGR2RGB))\n    ax4.imshow(cv2.cvtColor(f4, cv2.COLOR_BGR2RGB))\n\n    plt.show()\n    plt.close('all')\n\n    return\n\ndef draw_png2_2(png1, png2, png3, png4):\n    f1 = cv2.imread(png1)\n    f2 = cv2.imread(png2)\n    f3 = cv2.imread(png3)\n    f4 = cv2.imread(png4)\n\n    fig = plt.figure(figsize=(16, 12))\n    \n    ax1 = fig.add_subplot(2, 2, 1)\n    ax2 = fig.add_subplot(2, 2, 2)\n    ax3 = fig.add_subplot(2, 2, 3)\n    ax4 = fig.add_subplot(2, 2, 4)\n\n    ax1.axis('off')\n    ax2.axis('off')\n    ax3.axis('off')\n    ax4.axis('off')\n\n    ax1.imshow(cv2.cvtColor(f1, cv2.COLOR_BGR2RGB))\n    ax2.imshow(cv2.cvtColor(f2, cv2.COLOR_BGR2RGB))\n    ax3.imshow(cv2.cvtColor(f3, cv2.COLOR_BGR2RGB))\n    ax4.imshow(cv2.cvtColor(f4, cv2.COLOR_BGR2RGB))\n\n    plt.show()\n    plt.close('all')\n\n    return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"draw_png2_2('/kaggle/working/tmp/bgpoldvzrh/bgpoldvzrh0000.png',\n          '/kaggle/working/tmp/bgpoldvzrh/graph-xc-yc.png',\n          '/kaggle/working/tmp/bgpoldvzrh/graph-frame-xc.png',\n          '/kaggle/working/tmp/bgpoldvzrh/graph-frame-yc.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# [FAKE]\n\n## The one with the larger value of diff-std is the final candidate. Ignore diff-std = 4.5 or less."},{"metadata":{"trusted":true},"cell_type":"code","source":"ls /kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"draw_png('/kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa/graph-frame-std.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls /kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa/candidate","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"draw_png1_4(\n    '/kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa/candidate/bgpoldvzrh-txdcmspaaa_f_0092_1431_0419_006.7.png',\n    '/kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa/candidate/bgpoldvzrh-txdcmspaaa_f_0095_1427_0421_006.7.png',\n    '/kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa/candidate/bgpoldvzrh-txdcmspaaa_f_0103_0540_0508_007.7.png',\n    '/kaggle/working/tmp/bgpoldvzrh/FAKES/txdcmspaaa/candidate/bgpoldvzrh-txdcmspaaa_f_0113_0562_0515_006.0.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}