{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"A Comprehensive Introduction to SuperPoint by Magic Leap
\n\n\n* Magic Leap is a company that creates AR(Augmented Reality) Headsets.\n* The business was started in 2010 by Rony Abovitz, and it has raised over $2.06 billion in funding.\n* The Magic Leap One, the company's initial product, was released in 2018.\n* The Magic Leap One is a premium augmented reality headset intended for use by businesses and developers.\n* The headset has a 1080p display, a 6DoF head-tracking system, and numerous sensors.\n* A second-generation augmented reality headset has been unveiled by Magic Leap.\nVisit the [link](https://www.magicleap.com/en-us/) to learn more about it.\n* The company is also creating a range of augmented reality (AR) applications for business, gaming, and entertainment.\n\nThe following are some of the main aspects of Magic Leap's augmented reality technology.\n\n* **True integrative augmented reality:** Magic Leap's AR technology fuses digital content into users' views of their surroundings in the real world. In contrast, other AR devices usually superimpose digital content over the real world.\n* **Dynamic DimmingTM technology:** Magic Leap's Dynamic DimmingTM technology eliminates 99.7 percent of ambient light from either segments or the entire display, making content solid and vibrant in almost any lighting environment.\n* **High-resolution display:** Magic Leap's AR headsets come with a high-resolution display that offers a clear and lifelike AR experience.\n* **6DoF head-tracking:** Magic Leap's AR headsets track the user's head movements with the aid of 6DoF head-tracking, enabling the AR content to move in real time with the user's head.\n* **Variety of sensors:** Magic Leap's augmented reality headsets come equipped with a number of sensors, including a depth sensor, a microphone, and a camera. With the aid of these sensors, the headset can follow the user's movements, identify objects, and track their environment.\n\n\n\n* SuperPoint is a deep neural network architecture designed for feature detection and description in images.\n* It is based on a fully-convolutional neural network that takes an image as input and outputs a set of keypoints and descriptors.\n* The network is trained end-to-end on a large dataset of image pairs with ground-truth correspondences.\n* SuperPoint uses a novel loss function that encourages the network to learn repeatable and discriminative keypoints and descriptors.\n* The architecture is designed to be lightweight and efficient, making it suitable for real-time applications on mobile and embedded devices.\n* SuperPoint achieves state-of-the-art performance on several benchmark datasets for feature detection and description, including the HPatches and Aachen Day-Night datasets.\n* The architecture has been integrated into the MagicLeap AR platform, where it is used for real-time tracking and localization of objects in the environment.\n\n Architecture
\n\n* SuperPoint is a fully convolutional neural network that takes an image as input and outputs a set of keypoints and descriptors.\n* The network consists of several convolutional layers followed by a set of fully connected layers.\n* The output of the network is a set of heatmaps that represent the probability of a keypoint being present at each pixel in the image, as well as a set of descriptors that describe the local image patch around each keypoint.\n\n Working
\n\n* To detect keypoints, the network first generates a set of coarse-scale heatmaps that represent the probability of a keypoint being present at each pixel in the image.\n* The network then uses a non-maximum suppression algorithm to select the most salient keypoints from the heatmaps.\n* To compute descriptors, the network extracts a local image patch around each keypoint and passes it through a separate branch of the network that outputs a descriptor vector.\n* The descriptors are then normalized to have unit length and are used for matching and tracking.\n\n Use cases
\n\n* SuperPoint can be used for feature detection and description in a wide range of computer vision applications, including object recognition, image retrieval, and visual localization.\n* SuperPoint has been integrated into the Magic Leap AR platform, where it is used for real-time tracking and localization of objects in the environment.\n* SuperPoint has also been used in several research projects, including the SuperGlue algorithm for image matching and the SuperPoint SLAM system for visual odometry and mapping.\n\n Demo
\n","metadata":{}},{"cell_type":"markdown","source":" Importing modules
\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport cv2\nimport torch\nfrom torchvision.transforms import functional as F\nimport ipywidgets as widgets\nfrom IPython.display import display\nimport subprocess\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:06.052767Z","iopub.execute_input":"2023-06-13T00:07:06.053157Z","iopub.status.idle":"2023-06-13T00:07:09.189986Z","shell.execute_reply.started":"2023-06-13T00:07:06.053122Z","shell.execute_reply":"2023-06-13T00:07:09.188899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" SuperPointNet class definition
\nThe following code defines a PyTorch class - `SuperPointNet`, which represents the SuperPoint Network. Here's, how the code works: \n\n1. The `SuperPointNet` class inherits from the `torch.nn.Module` class, which is the base class for all neural network modules in PyTorch.\n2. The `__init__` method is the constructor for the `SuperPointNet` class. It initializes the various layers and modules of the network.\n3. The network architecture consists of a shared encoder, a detector head, and a descriptor head.\n4. The shared encoder comprises a series of convolutional layers (`Conv2d`) followed by ReLU activation functions and max pooling (`MaxPool2d`) operations.\n5. The detector head consists of a convolutional layer (`convPa`) followed by a convolutional layer with 65 output channels (`convPb`). It is responsible for predicting the point heatmaps.\n6. The descriptor head consists of a convolutional layer (`convDa`) followed by a convolutional layer with 256 output channels (`convDb`). It is responsible for predicting the descriptors for each point.\n7. The `forward` method defines the forward pass of the network. It takes an input tensor `x` representing the image and performs the necessary computations to generate the output tensors `semi` (point heatmaps) and `desc` (descriptors).\n8. The input tensor `x` goes through the shared encoder, which applies a series of convolutional layers with ReLU activations and max pooling operations.\n9. The output of the shared encoder is then fed into the detector head and the descriptor head separately.\n10. The output of the detector head (`cPa`) undergoes ReLU activation and is passed through the final convolutional layer (`convPb`) to generate the `semi` tensor, representing the point heatmaps.\n11. The output of the descriptor head (`cDa`) undergoes ReLU activation and is passed through the final convolutional layer (`convDb`) to generate the `desc` tensor, representing the descriptors for each point.\n12. The descriptors are then normalized by dividing each descriptor vector by its L2 norm (`dn`), ensuring that they have unit length.\n\nOverall, the `SuperPointNet` class defines the architecture and forward pass of the SuperPoint Network, which is designed for keypoint detection and description tasks. \n\n","metadata":{}},{"cell_type":"code","source":"class SuperPointNet(torch.nn.Module):\n \"\"\" Pytorch definition of SuperPoint Network. \"\"\"\n def __init__(self):\n super(SuperPointNet, self).__init__()\n self.relu = torch.nn.ReLU(inplace=True)\n self.pool = torch.nn.MaxPool2d(kernel_size=2, stride=2)\n c1, c2, c3, c4, c5, d1 = 64, 64, 128, 128, 256, 256\n # Shared Encoder.\n self.conv1a = torch.nn.Conv2d(1, c1, kernel_size=3, stride=1, padding=1)\n self.conv1b = torch.nn.Conv2d(c1, c1, kernel_size=3, stride=1, padding=1)\n self.conv2a = torch.nn.Conv2d(c1, c2, kernel_size=3, stride=1, padding=1)\n self.conv2b = torch.nn.Conv2d(c2, c2, kernel_size=3, stride=1, padding=1)\n self.conv3a = torch.nn.Conv2d(c2, c3, kernel_size=3, stride=1, padding=1)\n self.conv3b = torch.nn.Conv2d(c3, c3, kernel_size=3, stride=1, padding=1)\n self.conv4a = torch.nn.Conv2d(c3, c4, kernel_size=3, stride=1, padding=1)\n self.conv4b = torch.nn.Conv2d(c4, c4, kernel_size=3, stride=1, padding=1)\n # Detector Head.\n self.convPa = torch.nn.Conv2d(c4, c5, kernel_size=3, stride=1, padding=1)\n self.convPb = torch.nn.Conv2d(c5, 65, kernel_size=1, stride=1, padding=0)\n # Descriptor Head.\n self.convDa = torch.nn.Conv2d(c4, c5, kernel_size=3, stride=1, padding=1)\n self.convDb = torch.nn.Conv2d(c5, d1, kernel_size=1, stride=1, padding=0)\n\n def forward(self, x):\n \"\"\" Forward pass that jointly computes unprocessed point and descriptor\n tensors.\n Input\n x: Image pytorch tensor shaped N x 1 x H x W.\n Output\n semi: Output point pytorch tensor shaped N x 65 x H/8 x W/8.\n desc: Output descriptor pytorch tensor shaped N x 256 x H/8 x W/8.\n \"\"\"\n # Shared Encoder.\n x = self.relu(self.conv1a(x))\n x = self.relu(self.conv1b(x))\n x = self.pool(x)\n x = self.relu(self.conv2a(x))\n x = self.relu(self.conv2b(x))\n x = self.pool(x)\n x = self.relu(self.conv3a(x))\n x = self.relu(self.conv3b(x))\n x = self.pool(x)\n x = self.relu(self.conv4a(x))\n x = self.relu(self.conv4b(x))\n # Detector Head.\n cPa = self.relu(self.convPa(x))\n semi = self.convPb(cPa)\n # Descriptor Head.\n cDa = self.relu(self.convDa(x))\n desc = self.convDb(cDa)\n dn = torch.norm(desc, p=2, dim=1) # Compute the norm.\n desc = desc.div(torch.unsqueeze(dn, 1)) # Divide by norm to normalize.\n return semi, desc\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.194178Z","iopub.execute_input":"2023-06-13T00:07:09.195000Z","iopub.status.idle":"2023-06-13T00:07:09.211188Z","shell.execute_reply.started":"2023-06-13T00:07:09.194938Z","shell.execute_reply":"2023-06-13T00:07:09.210073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" SuperPointFrontend class definition
\nThe following code defines class called `SuperPointFrontend` that serves as a wrapper around a PyTorch network for performing SuperPoint feature extraction on images. Here's, how the code works:\n\n1. The `__init__` method initializes the SuperPoint frontend by loading the pre-trained network weights and setting the required parameters such as NMS distance, confidence threshold, nearest neighbor threshold, etc.\n\n2. The `nms_fast` method implements a faster approximate non-maximum suppression algorithm. It takes a numpy array of corners (each corner represented by [x, y, confidence]) and suppresses nearby corners to obtain a set of surviving corners with their corresponding indices.\n\n3. The `run` method processes a grayscale input image to extract SuperPoint features. It performs the following steps:\n - Checks that the image has the required dimensions and data type.\n - Reshapes and prepares the input image for feeding it into the network.\n - Performs a forward pass on the network to obtain the outputs.\n - Processes the output heatmap to obtain a set of surviving corners above the confidence threshold.\n - Applies non-maximum suppression to remove redundant corners.\n - Filters out corners near the image border.\n - Interpolates into the descriptor map using the 2D point locations to obtain descriptors corresponding to the surviving corners.\n - Normalizes the descriptors.\n - Returns the surviving corners, descriptors, and heatmap.\n\nAltogether, The code provides a convenient way for using the SuperPoint network to extract feature points and descriptors from images. It handles the pre-processing, post-processing, and integration with the network, making it easier to incorporate SuperPoint feature extraction into applications.","metadata":{}},{"cell_type":"code","source":"class SuperPointFrontend(object):\n \"\"\" Wrapper around pytorch net to help with pre and post image processing. \"\"\"\n def __init__(self, weights_path, nms_dist, conf_thresh, nn_thresh,\n cuda=False):\n self.name = 'SuperPoint'\n self.cuda = cuda\n self.nms_dist = nms_dist\n self.conf_thresh = conf_thresh\n self.nn_thresh = nn_thresh # L2 descriptor distance for good match.\n self.cell = 8 # Size of each output cell. Keep this fixed.\n self.border_remove = 4 # Remove points this close to the border.\n\n # Load the network in inference mode.\n self.net = SuperPointNet()\n if cuda:\n # Train on GPU, deploy on GPU.\n self.net.load_state_dict(torch.load(weights_path))\n self.net = self.net.cuda()\n else:\n # Train on GPU, deploy on CPU.\n self.net.load_state_dict(torch.load(weights_path,\n map_location=lambda storage, loc: storage))\n self.net.eval()\n\n def nms_fast(self, in_corners, H, W, dist_thresh):\n \"\"\"\n Run a faster approximate Non-Max-Suppression on numpy corners shaped:\n 3xN [x_i,y_i,conf_i]^T\n \n Algo summary: Create a grid sized HxW. Assign each corner location a 1, rest\n are zeros. Iterate through all the 1's and convert them either to -1 or 0.\n Suppress points by setting nearby values to 0.\n \n Grid Value Legend:\n -1 : Kept.\n 0 : Empty or suppressed.\n 1 : To be processed (converted to either kept or supressed).\n \n NOTE: The NMS first rounds points to integers, so NMS distance might not\n be exactly dist_thresh. It also assumes points are within image boundaries.\n \n Inputs\n in_corners - 3xN numpy array with corners [x_i, y_i, confidence_i]^T.\n H - Image height.\n W - Image width.\n dist_thresh - Distance to suppress, measured as an infinty norm distance.\n Returns\n nmsed_corners - 3xN numpy matrix with surviving corners.\n nmsed_inds - N length numpy vector with surviving corner indices.\n \"\"\"\n grid = np.zeros((H, W)).astype(int) # Track NMS data.\n inds = np.zeros((H, W)).astype(int) # Store indices of points.\n # Sort by confidence and round to nearest int.\n inds1 = np.argsort(-in_corners[2,:])\n corners = in_corners[:,inds1]\n rcorners = corners[:2,:].round().astype(int) # Rounded corners.\n # Check for edge case of 0 or 1 corners.\n if rcorners.shape[1] == 0:\n return np.zeros((3,0)).astype(int), np.zeros(0).astype(int)\n if rcorners.shape[1] == 1:\n out = np.vstack((rcorners, in_corners[2])).reshape(3,1)\n return out, np.zeros((1)).astype(int)\n # Initialize the grid.\n for i, rc in enumerate(rcorners.T):\n grid[rcorners[1,i], rcorners[0,i]] = 1\n inds[rcorners[1,i], rcorners[0,i]] = i\n # Pad the border of the grid, so that we can NMS points near the border.\n pad = dist_thresh\n grid = np.pad(grid, ((pad,pad), (pad,pad)), mode='constant')\n # Iterate through points, highest to lowest conf, suppress neighborhood.\n count = 0\n for i, rc in enumerate(rcorners.T):\n # Account for top and left padding.\n pt = (rc[0]+pad, rc[1]+pad)\n if grid[pt[1], pt[0]] == 1: # If not yet suppressed.\n grid[pt[1]-pad:pt[1]+pad+1, pt[0]-pad:pt[0]+pad+1] = 0\n grid[pt[1], pt[0]] = -1\n count += 1\n # Get all surviving -1's and return sorted array of remaining corners.\n keepy, keepx = np.where(grid==-1)\n keepy, keepx = keepy - pad, keepx - pad\n inds_keep = inds[keepy, keepx]\n out = corners[:, inds_keep]\n values = out[-1, :]\n inds2 = np.argsort(-values)\n out = out[:, inds2]\n out_inds = inds1[inds_keep[inds2]]\n return out, out_inds\n\n def run(self, img):\n \"\"\" Process a numpy image to extract points and descriptors.\n Input\n img - HxW numpy float32 input image in range [0,1].\n Output\n corners - 3xN numpy array with corners [x_i, y_i, confidence_i]^T.\n desc - 256xN numpy array of corresponding unit normalized descriptors.\n heatmap - HxW numpy heatmap in range [0,1] of point confidences.\n \"\"\"\n assert img.ndim == 2, 'Image must be grayscale.'\n assert img.dtype == np.float32, 'Image must be float32.'\n H, W = img.shape[0], img.shape[1]\n inp = img.copy()\n inp = (inp.reshape(1, H, W))\n inp = torch.from_numpy(inp)\n inp = torch.autograd.Variable(inp).view(1, 1, H, W)\n if self.cuda:\n inp = inp.cuda()\n # Forward pass of network.\n outs = self.net.forward(inp)\n semi, coarse_desc = outs[0], outs[1]\n # Convert pytorch -> numpy.\n semi = semi.data.cpu().numpy().squeeze()\n # --- Process points.\n dense = np.exp(semi) # Softmax.\n dense = dense / (np.sum(dense, axis=0)+.00001) # Should sum to 1.\n # Remove dustbin.\n nodust = dense[:-1, :, :]\n # Reshape to get full resolution heatmap.\n Hc = int(H / self.cell)\n Wc = int(W / self.cell)\n nodust = nodust.transpose(1, 2, 0)\n heatmap = np.reshape(nodust, [Hc, Wc, self.cell, self.cell])\n heatmap = np.transpose(heatmap, [0, 2, 1, 3])\n heatmap = np.reshape(heatmap, [Hc*self.cell, Wc*self.cell])\n xs, ys = np.where(heatmap >= self.conf_thresh) # Confidence threshold.\n if len(xs) == 0:\n return np.zeros((3, 0)), None, None\n pts = np.zeros((3, len(xs))) # Populate point data sized 3xN.\n pts[0, :] = ys\n pts[1, :] = xs\n pts[2, :] = heatmap[xs, ys]\n pts, _ = self.nms_fast(pts, H, W, dist_thresh=self.nms_dist) # Apply NMS.\n inds = np.argsort(pts[2,:])\n pts = pts[:,inds[::-1]] # Sort by confidence.\n # Remove points along border.\n bord = self.border_remove\n toremoveW = np.logical_or(pts[0, :] < bord, pts[0, :] >= (W-bord))\n toremoveH = np.logical_or(pts[1, :] < bord, pts[1, :] >= (H-bord))\n toremove = np.logical_or(toremoveW, toremoveH)\n pts = pts[:, ~toremove]\n # --- Process descriptor.\n D = coarse_desc.shape[1]\n if pts.shape[1] == 0:\n desc = np.zeros((D, 0))\n else:\n # Interpolate into descriptor map using 2D point locations.\n samp_pts = torch.from_numpy(pts[:2, :].copy())\n samp_pts[0, :] = (samp_pts[0, :] / (float(W)/2.)) - 1.\n samp_pts[1, :] = (samp_pts[1, :] / (float(H)/2.)) - 1.\n samp_pts = samp_pts.transpose(0, 1).contiguous()\n samp_pts = samp_pts.view(1, 1, -1, 2)\n samp_pts = samp_pts.float()\n if self.cuda:\n samp_pts = samp_pts.cuda()\n desc = torch.nn.functional.grid_sample(coarse_desc, samp_pts)\n desc = desc.data.cpu().numpy().reshape(D, -1)\n desc /= np.linalg.norm(desc, axis=0)[np.newaxis, :]\n return pts, desc, heatmap\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.213552Z","iopub.execute_input":"2023-06-13T00:07:09.214012Z","iopub.status.idle":"2023-06-13T00:07:09.257566Z","shell.execute_reply.started":"2023-06-13T00:07:09.213972Z","shell.execute_reply":"2023-06-13T00:07:09.255858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Setting the path of pre-trained weights
\n","metadata":{}},{"cell_type":"code","source":"weights_path = '/kaggle/input/superpoint-magicleap/superpoint_v1.pth'","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.261459Z","iopub.execute_input":"2023-06-13T00:07:09.262712Z","iopub.status.idle":"2023-06-13T00:07:09.282185Z","shell.execute_reply.started":"2023-06-13T00:07:09.262670Z","shell.execute_reply":"2023-06-13T00:07:09.280836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Instantiating frontend
\n","metadata":{}},{"cell_type":"code","source":"frontend = SuperPointFrontend(weights_path, nms_dist=4, conf_thresh=0.015, nn_thresh=0.7, cuda=False)\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.283417Z","iopub.execute_input":"2023-06-13T00:07:09.283849Z","iopub.status.idle":"2023-06-13T00:07:09.546096Z","shell.execute_reply.started":"2023-06-13T00:07:09.283806Z","shell.execute_reply":"2023-06-13T00:07:09.545048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Setting up the input video path
\n","metadata":{}},{"cell_type":"code","source":"video_path = '/kaggle/input/superpoint-magicleap/nyu_snippet.mp4'\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.550199Z","iopub.execute_input":"2023-06-13T00:07:09.554220Z","iopub.status.idle":"2023-06-13T00:07:09.558951Z","shell.execute_reply.started":"2023-06-13T00:07:09.554159Z","shell.execute_reply":"2023-06-13T00:07:09.557865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Creating a videocap object
\n","metadata":{}},{"cell_type":"code","source":"cap = cv2.VideoCapture(video_path)\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.560661Z","iopub.execute_input":"2023-06-13T00:07:09.561075Z","iopub.status.idle":"2023-06-13T00:07:09.666719Z","shell.execute_reply.started":"2023-06-13T00:07:09.561036Z","shell.execute_reply":"2023-06-13T00:07:09.665788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Defining the properties of video
\n","metadata":{}},{"cell_type":"code","source":"fps = cap.get(cv2.CAP_PROP_FPS)\nframe_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\nframe_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\nframe_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.671042Z","iopub.execute_input":"2023-06-13T00:07:09.671420Z","iopub.status.idle":"2023-06-13T00:07:09.680914Z","shell.execute_reply.started":"2023-06-13T00:07:09.671386Z","shell.execute_reply":"2023-06-13T00:07:09.679909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Creating output writer.
\n","metadata":{}},{"cell_type":"code","source":"output_path = '/kaggle/working/output_video.mp4'\nfourcc = cv2.VideoWriter_fourcc(*'mp4v')\nout = cv2.VideoWriter(output_path, fourcc, fps, (frame_width, frame_height))\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.683169Z","iopub.execute_input":"2023-06-13T00:07:09.683571Z","iopub.status.idle":"2023-06-13T00:07:09.698178Z","shell.execute_reply.started":"2023-06-13T00:07:09.683533Z","shell.execute_reply":"2023-06-13T00:07:09.697116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Processing the input video to generate output video
\n\n###### Working of the code : \n\n1. The code starts by reading each frame of the input video using the `cap.read()` function. If there are no more frames to read, the loop is terminated.\n\n2. The frame is then converted to grayscale using `cv2.cvtColor()` function with the `cv2.COLOR_BGR2GRAY` flag. This is done because SuperPoint expects grayscale images as input.\n\n3. The frame is optionally resized to the desired size using `cv2.resize()` function. This step is useful if you want to change the dimensions of the frame.\n\n4. The resized frame is converted to a PyTorch tensor using `F.to_tensor()` function from the `torchvision.transforms` module. The tensor is then squeezed to remove any singleton dimensions and converted to a NumPy array with the `numpy()` method. Finally, the data type of the array is changed to `np.float32` using `astype()`.\n\n5. The SuperPoint algorithm is applied to the resized frame by calling `frontend.run(resized_frame)`. This returns the detected corners, descriptors, and heatmap.\n\n6. The keypoints (corners) are extracted from the `corners` array and converted to integer coordinates. Each keypoint is represented as a circle drawn on the original frame using `cv2.circle()` function.\n\n7. The frame with keypoints is written to the output video using the `out.write()` function. This accumulates the frames with keypoints to create the final output video.\n\n8. The process continues with the next frame until there are no more frames to process.\n\n9. Finally, the video capture and writer are released using `cap.release()` and `out.release()` respectively, to free up system resources.\n","metadata":{}},{"cell_type":"code","source":"# Process each frame of the video\nwhile True:\n # Read the frame\n ret, frame = cap.read()\n if not ret:\n break\n \n # Convert the frame to grayscale\n gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n \n # Resize the frame to the desired size (optional)\n resized_frame = cv2.resize(gray, (frame_width, frame_height))\n \n resized_frame = F.to_tensor(resized_frame).squeeze(0).numpy().astype(np.float32)\n\n #print(resized_frame.type)\n # Run SuperPoint on the frame\n corners, descriptors, heatmap = frontend.run(resized_frame)\n \n # Draw the keypoints on the frame\n keypoints = corners[:2, :].T.astype(np.int32)\n for kp in keypoints:\n x, y = kp\n cv2.circle(frame, (x, y), 2, (0, 255, 0), -1)\n \n # Write the frame with keypoints to the output video\n out.write(frame)\n\n# Release the video capture and writer\ncap.release()\nout.release()\n","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:09.702076Z","iopub.execute_input":"2023-06-13T00:07:09.702434Z","iopub.status.idle":"2023-06-13T00:07:25.746919Z","shell.execute_reply.started":"2023-06-13T00:07:09.702404Z","shell.execute_reply":"2023-06-13T00:07:25.745601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Displaying input video
\n\n* The video can also be viewed at link ","metadata":{}},{"cell_type":"code","source":"video_path = '/kaggle/input/superpoint-magicleap/nyu_snippet.mp4'\n\n# Create a Video widget\nvideo_widget = widgets.Video.from_file(video_path, autoplay=False)\n\n# Display the Video widget\ndisplay(video_widget)","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:25.747957Z","iopub.execute_input":"2023-06-13T00:07:25.748252Z","iopub.status.idle":"2023-06-13T00:07:25.814837Z","shell.execute_reply.started":"2023-06-13T00:07:25.748224Z","shell.execute_reply":"2023-06-13T00:07:25.812291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Displaying output video after converting it to proper encoding.
\n\n* The video can also be viewed at link ","metadata":{}},{"cell_type":"code","source":"\n# Specify the paths\noutput_path = '/kaggle/working/output_video.mp4'\nconverted_path = '/kaggle/working/output_video_converted.mp4'\n\n# Convert the video using ffmpeg\nsubprocess.run(['ffmpeg', '-i', output_path, converted_path])\n\n# Create a Video widget\nvideo_widget = widgets.Video.from_file(converted_path, autoplay=False)\n\n# Display the Video widget\ndisplay(video_widget)","metadata":{"execution":{"iopub.status.busy":"2023-06-13T00:07:25.816203Z","iopub.execute_input":"2023-06-13T00:07:25.816719Z","iopub.status.idle":"2023-06-13T00:07:26.979321Z","shell.execute_reply.started":"2023-06-13T00:07:25.816686Z","shell.execute_reply":"2023-06-13T00:07:26.977576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" References
\n\n1. https://github.com/magicleap/SuperPointPretrainedNetwork\n2. [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629)\n3. [MagicLeap](https://www.magicleap.com/en-us/)\n4. [Presentation PDF: Talk at CVPR Deep Learning for Visual SLAM Workshop 2018](https://github.com/magicleap/SuperPointPretrainedNetwork/blob/master/assets/DL4VSLAM_talk.pdf)\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}