{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nInspired by the cool kernel [Animating and Smoothing 3D Facial Keypoints](https://www.kaggle.com/selfishgene/animating-and-smoothing-3d-facial-keypoints), I decided apply similar techniques for smooth face tracker."},{"metadata":{},"cell_type":"markdown","source":"Suppose you extracted keypoints in each frame of a video.\n\nIn particular, I use MTCNN face recognition framework, which produce a bounding box and five facial points.\n\n**NOTE:** Before performing smoothing, you should consider removing outliers and interpolating the missing values!"},{"metadata":{},"cell_type":"markdown","source":"## Inputs\n\nAll of the following methods will use one specific test video *aayfryxljh.mp4*"},{"metadata":{"trusted":true,"_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"data = \"\"\"913,383,1085,632,956,489,1037,483,993,537,973,583,1031,577\n906,388,1094,636,958,492,1038,481,996,533,971,576,1044,567\n904,380,1095,634,957,489,1039,481,994,531,966,576,1040,569\n914,386,1092,637,960,487,1037,483,992,531,974,582,1030,578\n914,384,1089,635,955,486,1036,481,992,535,973,583,1028,579\n914,384,1089,635,955,488,1036,482,993,536,974,584,1029,580\n911,386,1087,633,955,491,1037,485,995,540,973,584,1030,580\n915,387,1090,637,959,491,1040,487,996,542,975,587,1032,583\n919,393,1090,634,959,492,1039,488,996,542,974,584,1035,580\n916,389,1091,636,956,493,1039,487,992,543,972,587,1037,582\n916,387,1086,636,957,492,1039,487,996,543,974,586,1036,581\n910,379,1088,634,952,491,1038,487,996,543,968,590,1035,581\n922,393,1091,641,959,495,1038,490,994,543,972,586,1035,582\n912,387,1085,633,956,494,1036,489,996,547,972,586,1033,581\n920,399,1090,639,958,495,1037,488,993,542,974,586,1032,582\n911,384,1084,633,955,492,1035,486,995,545,972,586,1032,580\n911,386,1089,633,956,493,1037,487,994,544,973,586,1034,580\n911,385,1089,633,957,491,1040,484,994,540,972,585,1034,580\n908,387,1089,634,955,492,1037,485,992,538,972,583,1035,577\n906,386,1087,640,952,494,1037,489,993,544,971,589,1036,586\n911,386,1088,631,954,490,1036,485,991,537,970,581,1035,575\n907,384,1087,633,952,489,1035,484,991,538,970,583,1033,578\n905,385,1083,633,951,490,1034,485,992,541,970,584,1032,578\n904,389,1085,642,950,495,1034,492,989,547,968,590,1032,589\n910,396,1083,637,953,499,1034,493,990,546,968,588,1031,583\n905,389,1083,640,950,496,1033,494,988,547,965,588,1030,585\n905,391,1084,643,950,495,1033,492,989,547,967,588,1032,584\n908,394,1087,649,951,498,1034,495,987,549,962,590,1034,586\n909,392,1081,640,950,497,1031,493,989,549,966,589,1030,585\n911,401,1079,640,952,500,1030,495,990,549,966,589,1028,585\n903,396,1076,640,947,500,1027,494,986,552,964,593,1025,587\n895,389,1083,638,944,503,1027,496,980,552,957,593,1027,586\n893,386,1080,636,942,502,1026,498,978,553,954,597,1019,595\n903,395,1085,649,943,498,1022,491,974,549,959,593,1017,589\n894,385,1073,642,938,492,1021,487,976,546,956,591,1014,588\n894,381,1074,642,936,491,1021,488,975,543,952,590,1012,586\n892,388,1075,643,940,496,1024,492,980,549,961,594,1018,591\n885,391,1074,649,937,495,1026,492,978,544,946,588,1019,586\n886,392,1076,649,938,497,1027,492,981,544,947,587,1023,584\n886,392,1076,650,938,497,1027,492,980,545,946,587,1022,585\n881,389,1079,657,937,497,1026,492,979,545,944,590,1021,587\n885,390,1076,653,937,496,1026,492,979,545,947,591,1023,589\n899,385,1073,638,942,492,1025,486,982,544,959,586,1025,580\n898,385,1070,635,942,490,1023,484,985,542,959,583,1023,578\n898,386,1072,630,942,486,1024,479,984,534,961,577,1024,572\n897,384,1070,627,944,484,1023,476,983,528,960,573,1021,567\n891,373,1076,635,943,482,1025,472,986,526,961,579,1025,569\n894,375,1075,624,947,477,1026,470,989,518,965,570,1027,563\n903,369,1074,623,948,472,1030,464,991,515,967,567,1029,560\n903,371,1075,622,949,472,1031,462,993,512,969,565,1029,559\n902,367,1077,623,950,469,1032,462,994,508,969,565,1031,559\n907,370,1073,619,953,468,1033,461,994,504,971,560,1031,555\n906,379,1073,610,954,465,1032,456,998,501,974,557,1030,550\n899,372,1075,618,953,467,1035,461,999,501,964,557,1032,554\n905,376,1079,615,952,464,1031,453,994,499,974,561,1031,552\n898,371,1073,619,951,468,1032,460,995,501,962,556,1030,552\n904,372,1075,609,950,468,1030,459,992,500,963,554,1028,547\n901,375,1074,608,950,466,1028,453,994,498,973,559,1029,549\n901,368,1075,600,949,472,1031,463,993,503,963,555,1033,550\n897,367,1077,606,947,473,1030,464,990,504,959,557,1032,552\n896,367,1077,606,946,473,1031,463,990,503,959,557,1032,552\n882,364,1079,619,945,473,1028,464,989,504,955,555,1031,549\n888,363,1081,619,945,472,1029,466,986,503,953,557,1030,553\n884,363,1080,620,943,472,1029,466,987,502,952,557,1028,553\n892,367,1078,616,944,473,1029,464,987,502,959,561,1028,557\n886,364,1081,626,943,475,1030,469,985,507,953,564,1026,562\n896,368,1078,616,945,475,1028,467,985,506,959,564,1025,560\n893,367,1081,616,943,475,1029,466,983,507,954,563,1023,560\n891,366,1080,618,944,476,1029,469,983,507,953,563,1021,561\n882,360,1079,614,942,476,1026,469,981,507,949,556,1022,550\n881,361,1080,616,942,477,1026,469,981,509,948,558,1022,551\n880,362,1079,619,942,478,1025,470,981,511,948,559,1022,552\n881,360,1079,617,943,479,1027,470,986,512,950,562,1026,556\n883,360,1079,616,943,479,1029,472,985,513,950,563,1026,558\n885,364,1079,610,945,475,1024,467,981,509,948,550,1023,545\n886,367,1079,611,948,470,1028,462,988,501,956,540,1032,534\n902,387,1067,613,949,474,1025,468,985,506,964,563,1019,559\n884,363,1081,611,947,475,1028,463,988,506,958,548,1033,538\n884,363,1081,612,944,476,1027,468,982,508,950,554,1027,546\n884,363,1082,615,944,478,1027,469,981,509,951,558,1026,549\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\npoints = np.array([x.split(',') for x in data.split()], dtype=np.int)\nprint(points.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\ncap = cv2.VideoCapture('/kaggle/input/deepfake-detection-challenge/test_videos/aayfryxljh.mp4')\nframes = []\nwhile(True):\n    status, frame = cap.read()\n    frames.append(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))\n    if len(frames) == len(points):\n        break\nprint(len(frames))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Functions\n\nHere we define few functions for cropping and drawing faces."},{"metadata":{"trusted":true,"_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport matplotlib.animation\nplt.rcParams[\"animation.html\"] = \"jshtml\"\n\n\ndef get_boundingbox(x1, y1, x2, y2, height, width, scale=1.3):\n    \"\"\"\n    :param width: frame width\n    :param height: frame height\n    :param scale: bounding box size multiplier to get a bigger face region\n    :return: x, y, bounding_box_size in opencv form\n    \"\"\"\n    size_bb = int(max(x2 - x1, y2 - y1) * scale)\n    center_x, center_y = (x1 + x2) // 2, (y1 + y2) // 2\n    # Check for out of bounds, x-y top left corner\n    x1 = max(int(center_x - size_bb // 2), 0)\n    y1 = max(int(center_y - size_bb // 2), 0)\n    # Check for too big bb size for given x, y\n    size_bb = min(width - x1, size_bb)\n    size_bb = min(height - y1, size_bb)\n    return x1, y1, size_bb\n\n\ndef crop_faces(points, imsize=224):\n    \n    faces = []\n    \n    points = points.astype(np.int)\n    \n    for row, frame in zip(points, frames):\n        \n        box = row[:4]\n        landmarks = row[4:].reshape(5, 2)\n        height, width = frame.shape[:2]\n        \n        frame = frame.copy()\n        for point in landmarks:\n            cv2.rectangle(frame, tuple(point-5), tuple(point+5), (0, 255, 0), 3)\n        \n        x, y, size = get_boundingbox(*box, height, width)\n        \n        face = frame[y:y + size, x:x + size]\n        face = cv2.resize(face, (imsize, imsize))\n        faces.append(face)\n    \n    return faces\n\n\ndef build_animation(faces, figsize=5):\n    fig = plt.figure(figsize=(figsize, figsize))\n    fig.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=None, hspace=None)\n    plt.axis('off')\n    images = [[plt.imshow(face)] for face in faces]\n    return matplotlib.animation.ArtistAnimation(fig, images, interval=50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stage 0: initial"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\nfaces = crop_faces(points)\nanimation = build_animation(faces)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"animation","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Without any post-processing steps, it looks very shaky."},{"metadata":{},"cell_type":"markdown","source":"# Stage 1: PCA model\n\nThe PCA projection of points and reconstructions it back in each frame can be considered as spatial smoothing."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\n\n# normalize\nscaler = StandardScaler().fit(points)\npoints_norm = scaler.transform(points)\n# build model\nface_model = PCA(n_components=2, random_state=0).fit(points_norm)\nprint('Total explained percent by PCA model with %d components is %.1f%%' % (face_model.n_components_, 100 * face_model.explained_variance_ratio_.sum()))\n# project onto model and reconstruct\npoints_pca = face_model.inverse_transform(face_model.transform(points_norm))\n# transform back to image coords\npoints_pca = scaler.inverse_transform(points_pca)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\nfaces_pca = crop_faces(points_pca)\nanimation_pca = build_animation(faces_pca)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"animation_pca","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A little better, but still shaking a lot."},{"metadata":{},"cell_type":"markdown","source":"# Stage 2: temporal filter\n\nTemporal smoothing applied independently for each point."},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.signal import savgol_filter\n\npoints_filter = savgol_filter(points_pca, window_length=31, polyorder=2, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\nfaces_filter = crop_faces(points_filter)\nanimation_filter = build_animation(faces_filter)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"animation_filter","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Temporal smoothing gives much better result"},{"metadata":{},"cell_type":"markdown","source":"# Stage 3: torch conv2d\n\nFinally, we can align the frames with each other using the content of the images."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torch.nn.functional import conv2d\n\n\ndef align_frames(frames, points, size=56, padding=10):\n\n    frames = torch.tensor(frames).float()\n\n    h, w = frames.shape[1:3]\n\n    h = (h - size) // 2\n    w = (w - size) // 2\n    squares = frames[:, h:h+size, w:w+size, :].mean(dim=-1) / 255\n\n    h, w = squares.shape[1:3]\n\n    squares = squares.reshape(-1, 1, h, w)\n    kernel = squares.mean(dim=0).reshape(1, 1, h, w)\n    kernel = kernel[:, :, padding:-padding, padding:-padding]\n\n    kernel -= kernel.mean()\n    squares -= squares.mean()\n\n    corr_img = conv2d(squares, kernel)\n\n    h, w = corr_img.shape[-2:]\n\n    corr_img_max = corr_img.reshape(-1, h * w).argmax(dim=1)\n\n    roll_y = padding - corr_img_max / w\n    roll_x = padding - corr_img_max % w\n    \n    points = points.copy().reshape((-1, 7, 2))\n\n    for t in range(frames.shape[0]):\n        h = roll_y[t]\n        w = roll_x[t]\n        if h != 0 or w != 0:\n            print(t, h, w)\n            frames[t] = torch.roll(frames[t], (h, w), dims=[0, 1])\n            points[t, :, 0] += w.item()\n            points[t, :, 1] += h.item()\n\n    return frames.cpu().numpy().astype(np.int), points.reshape((-1, 14))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\nfaces_align, points_align = align_frames(faces_filter, points_filter)\nanimation_align = build_animation(faces_align)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"animation_align","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here you can see how the nose stays in one place, because we align the frames relative to a small region in the center of the images."},{"metadata":{},"cell_type":"markdown","source":"# Comparison"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 5))\nplt.title('Comparison of different methods of smoothing the Y coordinate of the nose')\nplt.xlabel('Frame')\nplt.ylabel('Coordinate')\nplt.plot(points[:, 9], label='inital')\nplt.plot(points_pca[:, 9], label='spatial')\nplt.plot(points_filter[:, 9], label='temporal')\nplt.plot(points_align[:, 9], label='content')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nAll demonstrated methods is quite fast and in combination gives a good result. But there are still some issues with rotation and scaling transformations. It is also worth noting that I did not experiment much with the order of these methods. Perhaps applying content smoothing before temporal will give a better result."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}