{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Level 5 Kaggle Reference Model\nAuthor: **Guido Zuidhof** - [gzuidhof@lyft.com](mailto:gzuidhof@lyft.com)\n\n---\n\nIn this Kernel we provide a (near) end-to-end example solution for the Lyft Level 5 Kaggle competition.\n\nWe train a [U-Net](https://arxiv.org/abs/1505.04597) fully convolutional neural network to predict whether a car or other object is present for every pixel in a birds eye view of the world centered on the car. We can then threshold this probability map and fit boxes around each of the detections.\n\nYou can expect to train the model in a couple of hours on a modern GPU, with inference times under 30ms per image.\n\n### Outline\n\n##### A. Creating an index and splitting into train and validation scenes\n1. Loading the dataset\n2. Creating a dataframe with one scene per row.\n3. Splitting all data into a train and validation set by car\n\n#### B. Creating input and targets\n1. We produce top-down images and targets\n2. Running this on all of the data in parallel\n\n#### C. Training a network to segment objects\n1. Defining datasets / dataloaders\n2. Defining the network architecture (U-net)\n3. Training the model\n\n#### D. Inference and postprocessing\n4. Predicting our validation set.\n5. Thresholding the probability map.\n6. Performing a morphological closing operation to filter out tiny objects (presuming they are false positives)\n7. Loading the ground truth\n8. backprojecting our predicted boxes into world space\n\n#### E. Visualizing the results (not included in this kernel)\nx. Creating top down visualizations of the ground truth and predictions using the nuScenes SDK.  \nx. (Optional) Creating a GIF of a scene.  \n\n#### F. Evaluation\nx. Computing mAP."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install shapely -U\n!pip install lyft-dataset-sdk","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Our code will generate data, visualization and model checkpoints, they will be persisted to disk in this folder\nARTIFACTS_FOLDER = \"./artifacts\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from datetime import datetime\nfrom functools import partial\nimport glob\nfrom multiprocessing import Pool\n\n# Disable multiprocesing for numpy/opencv. We already multiprocess ourselves, this would mean every subprocess produces\n# even more threads which would lead to a lot of context switching, slowing things down a lot.\nimport os\nos.environ[\"OMP_NUM_THREADS\"] = \"1\"\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport pandas as pd\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom tqdm import tqdm, tqdm_notebook\nimport scipy\nimport scipy.ndimage\nimport scipy.special\nfrom scipy.spatial.transform import Rotation as R\n\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset\nfrom lyft_dataset_sdk.utils.data_classes import LidarPointCloud, Box, Quaternion\nfrom lyft_dataset_sdk.utils.geometry_utils import view_points, transform_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data = LyftDataset(data_path='.', json_path='/kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data', verbose=True)\nos.makedirs(ARTIFACTS_FOLDER, exist_ok=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classes = [\"car\", \"motorcycle\", \"bus\", \"bicycle\", \"truck\", \"pedestrian\", \"other_vehicle\", \"animal\", \"emergency_vehicle\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"records = [(level5data.get('sample', record['first_sample_token'])['timestamp'], record) for record in level5data.scene]\n\nentries = []\n\nfor start_time, record in sorted(records):\n    start_time = level5data.get('sample', record['first_sample_token'])['timestamp'] / 1000000\n\n    token = record['token']\n    name = record['name']\n    date = datetime.utcfromtimestamp(start_time)\n    host = \"-\".join(record['name'].split(\"-\")[:2])\n    first_sample_token = record[\"first_sample_token\"]\n\n    entries.append((host, name, date, token, first_sample_token))\n            \ndf = pd.DataFrame(entries, columns=[\"host\", \"scene_name\", \"date\", \"scene_token\", \"first_sample_token\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"host_count_df = df.groupby(\"host\")['scene_token'].count()\nprint(host_count_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train/Validation split\nLet's split the data by car to get a validation set.\nAlternatively we could consider doing it by scenes, date, or completely randomly."},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_hosts = [\"host-a101\"]\n\nvalidation_df = df[df[\"host\"].isin(validation_hosts)]\nvi = validation_df.index\ntrain_df = df[~df.index.isin(vi)]\n\nprint(len(train_df), len(validation_df), \"train/validation split scene counts\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## B. Creating input and targets\n\nLet's load the first sample in the train set. We can use that to test the functions we'll define next that transform the data to the format we want to input into the model we are training."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_token = train_df.first_sample_token.values[0]\nsample = level5data.get(\"sample\", sample_token)\n\nsample_lidar_token = sample[\"data\"][\"LIDAR_TOP\"]\nlidar_data = level5data.get(\"sample_data\", sample_lidar_token)\nlidar_filepath = level5data.get_sample_data_path(sample_lidar_token)\n\nego_pose = level5data.get(\"ego_pose\", lidar_data[\"ego_pose_token\"])\ncalibrated_sensor = level5data.get(\"calibrated_sensor\", lidar_data[\"calibrated_sensor_token\"])\n\n# Homogeneous transformation matrix from car frame to world frame.\nglobal_from_car = transform_matrix(ego_pose['translation'],\n                                   Quaternion(ego_pose['rotation']), inverse=False)\n\n# Homogeneous transformation matrix from sensor coordinate frame to ego car frame.\ncar_from_sensor = transform_matrix(calibrated_sensor['translation'], Quaternion(calibrated_sensor['rotation']),\n                                    inverse=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lidar_pointcloud = LidarPointCloud.from_file(lidar_filepath)\n\n# The lidar pointcloud is defined in the sensor's reference frame.\n# We want it in the car's reference frame, so we transform each point\nlidar_pointcloud.transform(car_from_sensor)\n\n# A sanity check, the points should be centered around 0 in car space.\nplt.hist(lidar_pointcloud.points[0], alpha=0.5, bins=30, label=\"X\")\nplt.hist(lidar_pointcloud.points[1], alpha=0.5, bins=30, label=\"Y\")\nplt.legend()\nplt.xlabel(\"Distance from car along axis\")\nplt.ylabel(\"Amount of points\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As input for our network we voxelize the LIDAR points. That means that we go from a list of coordinates of points, to a X by Y by Z space."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_transformation_matrix_to_voxel_space(shape, voxel_size, offset):\n    \"\"\"\n    Constructs a transformation matrix given an output voxel shape such that (0,0,0) ends up in the center.\n    Voxel_size defines how large every voxel is in world coordinate, (1,1,1) would be the same as Minecraft voxels.\n    \n    An offset per axis in world coordinates (metric) can be provided, this is useful for Z (up-down) in lidar points.\n    \"\"\"\n    \n    shape, voxel_size, offset = np.array(shape), np.array(voxel_size), np.array(offset)\n    \n    tm = np.eye(4, dtype=np.float32)\n    translation = shape/2 + offset/voxel_size\n    \n    tm = tm * np.array(np.hstack((1/voxel_size, [1])))\n    tm[:3, 3] = np.transpose(translation)\n    return tm\n\ndef transform_points(points, transf_matrix):\n    \"\"\"\n    Transform (3,N) or (4,N) points using transformation matrix.\n    \"\"\"\n    if points.shape[0] not in [3,4]:\n        raise Exception(\"Points input should be (3,N) or (4,N) shape, received {}\".format(points.shape))\n    return transf_matrix.dot(np.vstack((points[:3, :], np.ones(points.shape[1]))))[:3, :]\n\n# Let's try it with some example values\ntm = create_transformation_matrix_to_voxel_space(shape=(100,100,4), voxel_size=(0.5,0.5,0.5), offset=(0,0,0.5))\np = transform_points(np.array([[10, 10, 0, 0, 0], [10, 5, 0, 0, 0],[0, 0, 0, 2, 0]], dtype=np.float32), tm)\nprint(p)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def car_to_voxel_coords(points, shape, voxel_size, z_offset=0):\n    if len(shape) != 3:\n        raise Exception(\"Voxel volume shape should be 3 dimensions (x,y,z)\")\n        \n    if len(points.shape) != 2 or points.shape[0] not in [3, 4]:\n        raise Exception(\"Input points should be (3,N) or (4,N) in shape, found {}\".format(points.shape))\n\n    tm = create_transformation_matrix_to_voxel_space(shape, voxel_size, (0, 0, z_offset))\n    p = transform_points(points, tm)\n    return p\n\ndef create_voxel_pointcloud(points, shape, voxel_size=(0.5,0.5,1), z_offset=0):\n\n    points_voxel_coords = car_to_voxel_coords(points.copy(), shape, voxel_size, z_offset)\n    points_voxel_coords = points_voxel_coords[:3].transpose(1,0)\n    points_voxel_coords = np.int0(points_voxel_coords)\n    \n    bev = np.zeros(shape, dtype=np.float32)\n    bev_shape = np.array(shape)\n\n    within_bounds = (np.all(points_voxel_coords >= 0, axis=1) * np.all(points_voxel_coords < bev_shape, axis=1))\n    \n    points_voxel_coords = points_voxel_coords[within_bounds]\n    coord, count = np.unique(points_voxel_coords, axis=0, return_counts=True)\n        \n    # Note X and Y are flipped:\n    bev[coord[:,1], coord[:,0], coord[:,2]] = count\n    \n    return bev\n\ndef normalize_voxel_intensities(bev, max_intensity=16):\n    return (bev/max_intensity).clip(0,1)\n\n\nvoxel_size = (0.4,0.4,1.5)\nz_offset = -2.0\nbev_shape = (336, 336, 3)\n\nbev = create_voxel_pointcloud(lidar_pointcloud.points, bev_shape, voxel_size=voxel_size, z_offset=z_offset)\n\n# So that the values in the voxels range from 0,1 we set a maximum intensity.\nbev = normalize_voxel_intensities(bev)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,8))\nplt.imshow(bev)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above is an example of what the input for our network will look like. It's a top-down projection of the world around the car (the car faces to the right in the image). The height of the lidar points are separated into three bins, which visualized like this these are the RGB channels of the image."},{"metadata":{"trusted":true},"cell_type":"code","source":"boxes = level5data.get_boxes(sample_lidar_token)\n\ntarget_im = np.zeros(bev.shape[:3], dtype=np.uint8)\n\ndef move_boxes_to_car_space(boxes, ego_pose):\n    \"\"\"\n    Move boxes from world space to car space.\n    Note: mutates input boxes.\n    \"\"\"\n    translation = -np.array(ego_pose['translation'])\n    rotation = Quaternion(ego_pose['rotation']).inverse\n    \n    for box in boxes:\n        # Bring box to car space\n        box.translate(translation)\n        box.rotate(rotation)\n        \ndef scale_boxes(boxes, factor):\n    \"\"\"\n    Note: mutates input boxes\n    \"\"\"\n    for box in boxes:\n        box.wlh = box.wlh * factor\n\ndef draw_boxes(im, voxel_size, boxes, classes, z_offset=0.0):\n    for box in boxes:\n        # We only care about the bottom corners\n        corners = box.bottom_corners()\n        corners_voxel = car_to_voxel_coords(corners, im.shape, voxel_size, z_offset).transpose(1,0)\n        corners_voxel = corners_voxel[:,:2] # Drop z coord\n\n        class_color = classes.index(box.name) + 1\n        \n        if class_color == 0:\n            raise Exception(\"Unknown class: {}\".format(box.name))\n\n        cv2.drawContours(im, np.int0([corners_voxel]), 0, (class_color, class_color, class_color), -1)\n\n\n\nmove_boxes_to_car_space(boxes, ego_pose)\nscale_boxes(boxes, 0.8)\ndraw_boxes(target_im, voxel_size, boxes, classes, z_offset=z_offset)\n\nplt.figure(figsize=(8,8))\nplt.imshow((target_im > 0).astype(np.float32), cmap='Set2')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_lidar_of_sample(sample_token, axes_limit=80):\n    sample = level5data.get(\"sample\", sample_token)\n    sample_lidar_token = sample[\"data\"][\"LIDAR_TOP\"]\n    level5data.render_sample_data(sample_lidar_token, axes_limit=axes_limit)\n    \n# Don't worry about it being mirrored.\nvisualize_lidar_of_sample(sample_token)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del bev, lidar_pointcloud, boxes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some hyperparameters we'll need to define for the system\nvoxel_size = (0.4, 0.4, 1.5)\nz_offset = -2.0\nbev_shape = (336, 336, 3)\n\n# We scale down each box so they are more separated when projected into our coarse voxel space.\nbox_scale = 0.8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# \"bev\" stands for birds eye view\ntrain_data_folder = os.path.join(ARTIFACTS_FOLDER, \"bev_train_data\")\nvalidation_data_folder = os.path.join(ARTIFACTS_FOLDER, \"./bev_validation_data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_WORKERS = os.cpu_count() * 3\n\ndef prepare_training_data_for_scene(first_sample_token, output_folder, bev_shape, voxel_size, z_offset, box_scale):\n    \"\"\"\n    Given a first sample token (in a scene), output rasterized input volumes and targets in birds-eye-view perspective.\n\n    \"\"\"\n    sample_token = first_sample_token\n    \n    while sample_token:\n        \n        sample = level5data.get(\"sample\", sample_token)\n\n        sample_lidar_token = sample[\"data\"][\"LIDAR_TOP\"]\n        lidar_data = level5data.get(\"sample_data\", sample_lidar_token)\n        lidar_filepath = level5data.get_sample_data_path(sample_lidar_token)\n\n        ego_pose = level5data.get(\"ego_pose\", lidar_data[\"ego_pose_token\"])\n        calibrated_sensor = level5data.get(\"calibrated_sensor\", lidar_data[\"calibrated_sensor_token\"])\n\n\n        global_from_car = transform_matrix(ego_pose['translation'],\n                                           Quaternion(ego_pose['rotation']), inverse=False)\n\n        car_from_sensor = transform_matrix(calibrated_sensor['translation'], Quaternion(calibrated_sensor['rotation']),\n                                            inverse=False)\n\n        try:\n            lidar_pointcloud = LidarPointCloud.from_file(lidar_filepath)\n            lidar_pointcloud.transform(car_from_sensor)\n        except Exception as e:\n            print (\"Failed to load Lidar Pointcloud for {}: {}:\".format(sample_token, e))\n            sample_token = sample[\"next\"]\n            continue\n        \n        bev = create_voxel_pointcloud(lidar_pointcloud.points, bev_shape, voxel_size=voxel_size, z_offset=z_offset)\n        bev = normalize_voxel_intensities(bev)\n\n        \n        boxes = level5data.get_boxes(sample_lidar_token)\n\n        target = np.zeros_like(bev)\n\n        move_boxes_to_car_space(boxes, ego_pose)\n        scale_boxes(boxes, box_scale)\n        draw_boxes(target, voxel_size, boxes=boxes, classes=classes, z_offset=z_offset)\n\n        bev_im = np.round(bev*255).astype(np.uint8)\n        target_im = target[:,:,0] # take one channel only\n\n        cv2.imwrite(os.path.join(output_folder, \"{}_input.png\".format(sample_token)), bev_im)\n        cv2.imwrite(os.path.join(output_folder, \"{}_target.png\".format(sample_token)), target_im)\n        \n        sample_token = sample[\"next\"]\n\nfor df, data_folder in [(train_df, train_data_folder), (validation_df, validation_data_folder)]:\n    print(\"Preparing data into {} using {} workers\".format(data_folder, NUM_WORKERS))\n    first_samples = df.first_sample_token.values\n\n    os.makedirs(data_folder, exist_ok=True)\n    \n    process_func = partial(prepare_training_data_for_scene,\n                           output_folder=data_folder, bev_shape=bev_shape, voxel_size=voxel_size, z_offset=z_offset, box_scale=box_scale)\n\n    pool = Pool(NUM_WORKERS)\n    for _ in tqdm_notebook(pool.imap_unordered(process_func, first_samples), total=len(first_samples)):\n        pass\n    pool.close()\n    del pool","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## C. Training a network to segment objects"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.utils.data\n\nclass BEVImageDataset(torch.utils.data.Dataset):\n    def __init__(self, input_filepaths, target_filepaths, map_filepaths=None):\n        self.input_filepaths = input_filepaths\n        self.target_filepaths = target_filepaths\n        self.map_filepaths = map_filepaths\n        \n        if map_filepaths is not None:\n            assert len(input_filepaths) == len(map_filepaths)\n        \n        assert len(input_filepaths) == len(target_filepaths)\n\n    def __len__(self):\n        return len(self.input_filepaths)\n\n    def __getitem__(self, idx):\n        input_filepath = self.input_filepaths[idx]\n        target_filepath = self.target_filepaths[idx]\n        \n        sample_token = input_filepath.split(\"/\")[-1].replace(\"_input.png\",\"\")\n        \n        im = cv2.imread(input_filepath, cv2.IMREAD_UNCHANGED)\n        \n        if self.map_filepaths:\n            map_filepath = self.map_filepaths[idx]\n            map_im = cv2.imread(map_filepath, cv2.IMREAD_UNCHANGED)\n            im = np.concatenate((im, map_im), axis=2)\n        \n        target = cv2.imread(target_filepath, cv2.IMREAD_UNCHANGED)\n        \n        im = im.astype(np.float32)/255\n        target = target.astype(np.int64)\n        \n        im = torch.from_numpy(im.transpose(2,0,1))\n        target = torch.from_numpy(target)\n        \n        return im, target, sample_token\n\ninput_filepaths = sorted(glob.glob(os.path.join(train_data_folder, \"*_input.png\")))\ntarget_filepaths = sorted(glob.glob(os.path.join(train_data_folder, \"*_target.png\")))\n\ntrain_dataset = BEVImageDataset(input_filepaths, target_filepaths)\n    \nim, target, sample_token = train_dataset[1]\nim = im.numpy()\ntarget = target.numpy()\n\nplt.figure(figsize=(16,8))\n\ntarget_as_rgb = np.repeat(target[...,None], 3, 2)\n# Transpose the input volume CXY to XYC order, which is what matplotlib requires.\nplt.imshow(np.hstack((im.transpose(1,2,0)[...,:3], target_as_rgb)))\nplt.title(sample_token)\nplt.show()\n\nvisualize_lidar_of_sample(sample_token)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This implementation was copied from https://github.com/jvanvugt/pytorch-unet, it is MIT licensed.\n\nclass UNet(nn.Module):\n    def __init__(\n        self,\n        in_channels=1,\n        n_classes=2,\n        depth=5,\n        wf=6,\n        padding=False,\n        batch_norm=False,\n        up_mode='upconv',\n    ):\n        \"\"\"\n        Implementation of\n        U-Net: Convolutional Networks for Biomedical Image Segmentation\n        (Ronneberger et al., 2015)\n        https://arxiv.org/abs/1505.04597\n        Using the default arguments will yield the exact version used\n        in the original paper\n        Args:\n            in_channels (int): number of input channels\n            n_classes (int): number of output channels\n            depth (int): depth of the network\n            wf (int): number of filters in the first layer is 2**wf\n            padding (bool): if True, apply padding such that the input shape\n                            is the same as the output.\n                            This may introduce artifacts\n            batch_norm (bool): Use BatchNorm after layers with an\n                               activation function\n            up_mode (str): one of 'upconv' or 'upsample'.\n                           'upconv' will use transposed convolutions for\n                           learned upsampling.\n                           'upsample' will use bilinear upsampling.\n        \"\"\"\n        super(UNet, self).__init__()\n        assert up_mode in ('upconv', 'upsample')\n        self.padding = padding\n        self.depth = depth\n        prev_channels = in_channels\n        self.down_path = nn.ModuleList()\n        for i in range(depth):\n            self.down_path.append(\n                UNetConvBlock(prev_channels, 2 ** (wf + i), padding, batch_norm)\n            )\n            prev_channels = 2 ** (wf + i)\n\n        self.up_path = nn.ModuleList()\n        for i in reversed(range(depth - 1)):\n            self.up_path.append(\n                UNetUpBlock(prev_channels, 2 ** (wf + i), up_mode, padding, batch_norm)\n            )\n            prev_channels = 2 ** (wf + i)\n\n        self.last = nn.Conv2d(prev_channels, n_classes, kernel_size=1)\n\n    def forward(self, x):\n        blocks = []\n        for i, down in enumerate(self.down_path):\n            x = down(x)\n            if i != len(self.down_path) - 1:\n                blocks.append(x)\n                x = F.max_pool2d(x, 2)\n\n        for i, up in enumerate(self.up_path):\n            x = up(x, blocks[-i - 1])\n\n        return self.last(x)\n\n\nclass UNetConvBlock(nn.Module):\n    def __init__(self, in_size, out_size, padding, batch_norm):\n        super(UNetConvBlock, self).__init__()\n        block = []\n\n        block.append(nn.Conv2d(in_size, out_size, kernel_size=3, padding=int(padding)))\n        block.append(nn.ReLU())\n        if batch_norm:\n            block.append(nn.BatchNorm2d(out_size))\n\n        block.append(nn.Conv2d(out_size, out_size, kernel_size=3, padding=int(padding)))\n        block.append(nn.ReLU())\n        if batch_norm:\n            block.append(nn.BatchNorm2d(out_size))\n\n        self.block = nn.Sequential(*block)\n\n    def forward(self, x):\n        out = self.block(x)\n        return out\n\n\nclass UNetUpBlock(nn.Module):\n    def __init__(self, in_size, out_size, up_mode, padding, batch_norm):\n        super(UNetUpBlock, self).__init__()\n        if up_mode == 'upconv':\n            self.up = nn.ConvTranspose2d(in_size, out_size, kernel_size=2, stride=2)\n        elif up_mode == 'upsample':\n            self.up = nn.Sequential(\n                nn.Upsample(mode='bilinear', scale_factor=2),\n                nn.Conv2d(in_size, out_size, kernel_size=1),\n            )\n\n        self.conv_block = UNetConvBlock(in_size, out_size, padding, batch_norm)\n\n    def center_crop(self, layer, target_size):\n        _, _, layer_height, layer_width = layer.size()\n        diff_y = (layer_height - target_size[0]) // 2\n        diff_x = (layer_width - target_size[1]) // 2\n        return layer[\n            :, :, diff_y : (diff_y + target_size[0]), diff_x : (diff_x + target_size[1])\n        ]\n\n    def forward(self, x, bridge):\n        up = self.up(x)\n        crop1 = self.center_crop(bridge, up.shape[2:])\n        out = torch.cat([up, crop1], 1)\n        out = self.conv_block(out)\n\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We train a U-net fully convolutional neural network, we create a network that is less deep and with only half the amount of filters compared to the original U-net paper implementation. We do this to keep training and inference time low."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_unet_model(in_channels=3, num_output_classes=2):\n    model = UNet(in_channels=in_channels, n_classes=num_output_classes, wf=5, depth=4, padding=True, up_mode='upsample')\n    \n    # Optional, for multi GPU training and inference\n    model = nn.DataParallel(model)\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_predictions(input_image, prediction, target, n_images=2, apply_softmax=True):\n    \"\"\"\n    Takes as input 3 PyTorch tensors, plots the input image, predictions and targets.\n    \"\"\"\n    # Only select the first n images\n    prediction = prediction[:n_images]\n    target = target[:n_images]\n    input_image = input_image[:n_images]\n\n    prediction = prediction.detach().cpu().numpy()\n    if apply_softmax:\n        prediction = scipy.special.softmax(prediction, axis=1)\n    class_one_preds = np.hstack(1-prediction[:,0])\n\n    target = np.hstack(target.detach().cpu().numpy())\n\n    class_rgb = np.repeat(class_one_preds[..., None], 3, axis=2)\n    class_rgb[...,2] = 0\n    class_rgb[...,1] = target\n\n    \n    input_im = np.hstack(input_image.cpu().numpy().transpose(0,2,3,1))\n    \n    if input_im.shape[2] == 3:\n        input_im_grayscale = np.repeat(input_im.mean(axis=2)[..., None], 3, axis=2)\n        overlayed_im = (input_im_grayscale*0.6 + class_rgb*0.7).clip(0,1)\n    else:\n        input_map = input_im[...,3:]\n        overlayed_im = (input_map*0.6 + class_rgb*0.7).clip(0,1)\n\n    thresholded_pred = np.repeat(class_one_preds[..., None] > 0.5, 3, axis=2)\n\n    fig = plt.figure(figsize=(12,26))\n    plot_im = np.vstack([class_rgb, input_im[...,:3], overlayed_im, thresholded_pred]).clip(0,1).astype(np.float32)\n    plt.imshow(plot_im)\n    plt.axis(\"off\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We weigh the loss for the 0 class lower to account for (some of) the big class imbalance.\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nclass_weights = torch.from_numpy(np.array([0.2] + [1.0]*len(classes), dtype=np.float32))\nclass_weights = class_weights.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 8\nepochs = 15 # Note: We may be able to train for longer and expect better results, the reason this number is low is to keep the runtime short.\n\nmodel = get_unet_model(num_output_classes=len(classes)+1)\nmodel = model.to(device)\n\noptim = torch.optim.Adam(model.parameters(), lr=1e-3)\ndataloader = torch.utils.data.DataLoader(train_dataset, batch_size, shuffle=True, num_workers=os.cpu_count()*2)\n\nall_losses = []\n\nfor epoch in range(1, epochs+1):\n    print(\"Epoch\", epoch)\n    \n    epoch_losses = []\n    progress_bar = tqdm_notebook(dataloader)\n    \n    for ii, (X, target, sample_ids) in enumerate(progress_bar):\n        X = X.to(device)  # [N, 3, H, W]\n        target = target.to(device)  # [N, H, W] with class indices (0, 1)\n        prediction = model(X)  # [N, 2, H, W]\n        loss = F.cross_entropy(prediction, target, weight=class_weights)\n\n        optim.zero_grad()\n        loss.backward()\n        optim.step()\n        \n        epoch_losses.append(loss.detach().cpu().numpy())\n\n        if ii == 0:\n            visualize_predictions(X, prediction, target)\n    \n    print(\"Loss:\", np.mean(epoch_losses))\n    all_losses.extend(epoch_losses)\n    \n    checkpoint_filename = \"unet_checkpoint_epoch_{}.pth\".format(epoch)\n    checkpoint_filepath = os.path.join(ARTIFACTS_FOLDER, checkpoint_filename)\n    torch.save(model.state_dict(), checkpoint_filepath)\n    \nplt.figure(figsize=(12,12))\nplt.plot(all_losses, alpha=0.75)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### You can interpret the above visualizations as follows:  \nThere are four different visualizations stacked on top of eachother:\n1. The top images have two color channels: red for predictions, green for targets. Note that red+green=yellow. In other words:  \n> **Black**: True Negative  \n**Green**: False Negative  \n**Yellow**: True Positive  \n**Red**: False Positive \n2. The input image\n3. The input image (or semantic input map, not in this kernel) blended together with targets+predictions\n4. The predictions thresholded at 0.5 probability."},{"metadata":{"trusted":true},"cell_type":"code","source":"input_filepaths = sorted(glob.glob(os.path.join(validation_data_folder, \"*_input.png\")))\ntarget_filepaths = sorted(glob.glob(os.path.join(validation_data_folder, \"*_target.png\")))\n\nbatch_size=16\nvalidation_dataset = BEVImageDataset(input_filepaths, target_filepaths)\nvalidation_dataloader = torch.utils.data.DataLoader(validation_dataset, batch_size, shuffle=False, num_workers=os.cpu_count())\n\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nmodel = get_unet_model(num_output_classes=1+len(classes))\nmodel = model.to(device)\n\nepoch_to_load=15\ncheckpoint_filename = \"unet_checkpoint_epoch_{}.pth\".format(epoch_to_load)\ncheckpoint_filepath = os.path.join(ARTIFACTS_FOLDER, checkpoint_filename)\nmodel.load_state_dict(torch.load(checkpoint_filepath))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"progress_bar = tqdm_notebook(validation_dataloader)\n\ntargets = np.zeros((len(target_filepaths), 336, 336), dtype=np.uint8)\n\n# We quantize to uint8 here to conserve memory. We're allocating >20GB of memory otherwise.\npredictions = np.zeros((len(target_filepaths), 1+len(classes), 336, 336), dtype=np.uint8)\n\nsample_tokens = []\nall_losses = []\n\nwith torch.no_grad():\n    model.eval()\n    for ii, (X, target, batch_sample_tokens) in enumerate(progress_bar):\n\n        offset = ii*batch_size\n        targets[offset:offset+batch_size] = target.numpy()\n        sample_tokens.extend(batch_sample_tokens)\n        \n        X = X.to(device)  # [N, 1, H, W]\n        target = target.to(device)  # [N, H, W] with class indices (0, 1)\n        prediction = model(X)  # [N, 2, H, W]\n        loss = F.cross_entropy(prediction, target, weight=class_weights)\n        all_losses.append(loss.detach().cpu().numpy())\n        \n        prediction = F.softmax(prediction, dim=1)\n        \n        prediction_cpu = prediction.cpu().numpy()\n        predictions[offset:offset+batch_size] = np.round(prediction_cpu*255).astype(np.uint8)\n        \n        # Visualize the first prediction\n        if ii == 0:\n            visualize_predictions(X, prediction, target, apply_softmax=False)\n            \nprint(\"Mean loss:\", np.mean(all_losses))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#del model, class_weights, validation_dataloader, validation_dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get probabilities for non-background\npredictions_non_class0 = 255 - predictions[:,0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Arbitrary threshold in our system to create a binary image to fit boxes around.\nbackground_threshold = 255//2\n\nfor i in range(3):\n    fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(16, 6))\n    axes[0].imshow(predictions_non_class0[i])\n    axes[0].set_title(\"predictions\")\n    axes[1].imshow(predictions_non_class0[i] > background_threshold)\n    axes[1].set_title(\"thresholded predictions\")\n    axes[2].imshow((targets[i] > 0).astype(np.uint8), interpolation=\"nearest\")\n    axes[2].set_title(\"targets\")\n    fig.tight_layout()\n    fig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We perform an opening morphological operation to filter tiny detections\n# Note that this may be problematic for classes that are inherently small (e.g. pedestrians)..\nkernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(3,3))\npredictions_opened = np.zeros((predictions_non_class0.shape), dtype=np.uint8)\n\nfor i, p in enumerate(tqdm(predictions_non_class0)):\n    thresholded_p = (p > background_threshold).astype(np.uint8)\n    predictions_opened[i] = cv2.morphologyEx(thresholded_p, cv2.MORPH_OPEN, kernel)\n\nplt.figure(figsize=(12,12))\nfig, axes = plt.subplots(nrows=1, ncols=2, figsize=(16, 6))\naxes[0].imshow(predictions_non_class0[0] > 255//2)\naxes[0].set_title(\"thresholded prediction\")\naxes[1].imshow(predictions_opened[0])\naxes[1].set_title(\"opened thresholded prediction\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sanity check: let's count the amount of connected components in an image\nlabels, n = scipy.ndimage.label(predictions_opened[0])\nplt.imshow(labels, cmap=\"tab20b\")\nplt.xlabel(\"N predictions: {}\".format(n))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above image looks pretty well separated, some boxes seem to be wrongly merged together and may be problematic. Let's continue.\nFor each scene we fit boxes to the segmentations. For each box and each class we write it's probability in the center pixel."},{"metadata":{"trusted":true},"cell_type":"code","source":"detection_boxes = []\ndetection_scores = []\ndetection_classes = []\n\nfor i in tqdm_notebook(range(len(predictions))):\n    prediction_opened = predictions_opened[i]\n    probability_non_class0 = predictions_non_class0[i]\n    class_probability = predictions[i]\n\n    sample_boxes = []\n    sample_detection_scores = []\n    sample_detection_classes = []\n    \n    contours, hierarchy = cv2.findContours(prediction_opened, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE) \n    \n    for cnt in contours:\n        rect = cv2.minAreaRect(cnt)\n        box = cv2.boxPoints(rect)\n        \n        # Let's take the center pixel value as the confidence value\n        box_center_index = np.int0(np.mean(box, axis=0))\n        \n        for class_index in range(len(classes)):\n            box_center_value = class_probability[class_index+1, box_center_index[1], box_center_index[0]]\n            \n            # Let's remove candidates with very low probability\n            if box_center_value < 0.01:\n                continue\n            \n            box_center_class = classes[class_index]\n\n            box_detection_score = box_center_value\n            sample_detection_classes.append(box_center_class)\n            sample_detection_scores.append(box_detection_score)\n            sample_boxes.append(box)\n        \n    \n    detection_boxes.append(np.array(sample_boxes))\n    detection_scores.append(sample_detection_scores)\n    detection_classes.append(sample_detection_classes)\n    \nprint(\"Total amount of boxes:\", np.sum([len(x) for x in detection_boxes]))\n    \n\n# Visualize the boxes in the first sample\nt = np.zeros_like(predictions_opened[0])\nfor sample_boxes in detection_boxes[0]:\n    box_pix = np.int0(sample_boxes)\n    cv2.drawContours(t,[box_pix],0,(255),2)\nplt.imshow(t)\nplt.show()\n\n# Visualize their probabilities\nplt.hist(detection_scores[0], bins=20)\nplt.xlabel(\"Detection Score\")\nplt.ylabel(\"Count\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load the ground truth for the validation set."},{"metadata":{"trusted":true},"cell_type":"code","source":"from lyft_dataset_sdk.eval.detection.mAP_evaluation import Box3D, recall_precision\n\ndef load_groundtruth_boxes(nuscenes, sample_tokens):\n    gt_box3ds = []\n\n    # Load annotations and filter predictions and annotations.\n    for sample_token in tqdm_notebook(sample_tokens):\n\n        sample = nuscenes.get('sample', sample_token)\n        sample_annotation_tokens = sample['anns']\n\n        sample_lidar_token = sample[\"data\"][\"LIDAR_TOP\"]\n        lidar_data = level5data.get(\"sample_data\", sample_lidar_token)\n        ego_pose = level5data.get(\"ego_pose\", lidar_data[\"ego_pose_token\"])\n        ego_translation = np.array(ego_pose['translation'])\n        \n        for sample_annotation_token in sample_annotation_tokens:\n            sample_annotation = nuscenes.get('sample_annotation', sample_annotation_token)\n            sample_annotation_translation = sample_annotation['translation']\n            \n            class_name = sample_annotation['category_name']\n            \n            box3d = Box3D(\n                sample_token=sample_token,\n                translation=sample_annotation_translation,\n                size=sample_annotation['size'],\n                rotation=sample_annotation['rotation'],\n                name=class_name\n            )\n            gt_box3ds.append(box3d)\n            \n    return gt_box3ds\n\ngt_box3ds = load_groundtruth_boxes(level5data, sample_tokens)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we take our predicted boxes, transform them back into world space and make them 3D."},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_box3ds = []\n\n# This could use some refactoring..\nfor (sample_token, sample_boxes, sample_detection_scores, sample_detection_class) in tqdm_notebook(zip(sample_tokens, detection_boxes, detection_scores, detection_classes), total=len(sample_tokens)):\n    sample_boxes = sample_boxes.reshape(-1, 2) # (N, 4, 2) -> (N*4, 2)\n    sample_boxes = sample_boxes.transpose(1,0) # (N*4, 2) -> (2, N*4)\n\n    # Add Z dimension\n    sample_boxes = np.vstack((sample_boxes, np.zeros(sample_boxes.shape[1]),)) # (2, N*4) -> (3, N*4)\n\n    sample = level5data.get(\"sample\", sample_token)\n    sample_lidar_token = sample[\"data\"][\"LIDAR_TOP\"]\n    lidar_data = level5data.get(\"sample_data\", sample_lidar_token)\n    lidar_filepath = level5data.get_sample_data_path(sample_lidar_token)\n    ego_pose = level5data.get(\"ego_pose\", lidar_data[\"ego_pose_token\"])\n    ego_translation = np.array(ego_pose['translation'])\n\n    global_from_car = transform_matrix(ego_pose['translation'],\n                                       Quaternion(ego_pose['rotation']), inverse=False)\n\n    car_from_voxel = np.linalg.inv(create_transformation_matrix_to_voxel_space(bev_shape, voxel_size, (0, 0, z_offset)))\n\n\n    global_from_voxel = np.dot(global_from_car, car_from_voxel)\n    sample_boxes = transform_points(sample_boxes, global_from_voxel)\n\n    # We don't know at where the boxes are in the scene on the z-axis (up-down), let's assume all of them are at\n    # the same height as the ego vehicle.\n    sample_boxes[2,:] = ego_pose[\"translation\"][2]\n\n\n    # (3, N*4) -> (N, 4, 3)\n    sample_boxes = sample_boxes.transpose(1,0).reshape(-1, 4, 3)\n\n\n    # We don't know the height of our boxes, let's assume every object is the same height.\n    box_height = 1.75\n\n    # Note: Each of these boxes describes the ground corners of a 3D box.\n    # To get the center of the box in 3D, we'll have to add half the height to it.\n    sample_boxes_centers = sample_boxes.mean(axis=1)\n    sample_boxes_centers[:,2] += box_height/2\n\n    # Width and height is arbitrary - we don't know what way the vehicles are pointing from our prediction segmentation\n    # It doesn't matter for evaluation, so no need to worry about that here.\n    # Note: We scaled our targets to be 0.8 the actual size, we need to adjust for that\n    sample_lengths = np.linalg.norm(sample_boxes[:,0,:] - sample_boxes[:,1,:], axis=1) * 1/box_scale\n    sample_widths = np.linalg.norm(sample_boxes[:,1,:] - sample_boxes[:,2,:], axis=1) * 1/box_scale\n    \n    sample_boxes_dimensions = np.zeros_like(sample_boxes_centers) \n    sample_boxes_dimensions[:,0] = sample_widths\n    sample_boxes_dimensions[:,1] = sample_lengths\n    sample_boxes_dimensions[:,2] = box_height\n\n    for i in range(len(sample_boxes)):\n        translation = sample_boxes_centers[i]\n        size = sample_boxes_dimensions[i]\n        class_name = sample_detection_class[i]\n        ego_distance = float(np.linalg.norm(ego_translation - translation))\n    \n        \n        # Determine the rotation of the box\n        v = (sample_boxes[i,0] - sample_boxes[i,1])\n        v /= np.linalg.norm(v)\n        r = R.from_dcm([\n            [v[0], -v[1], 0],\n            [v[1],  v[0], 0],\n            [   0,     0, 1],\n        ])\n        quat = r.as_quat()\n        # XYZW -> WXYZ order of elements\n        quat = quat[[3,0,1,2]]\n        \n        detection_score = float(sample_detection_scores[i])\n\n        \n        box3d = Box3D(\n            sample_token=sample_token,\n            translation=list(translation),\n            size=list(size),\n            rotation=list(quat),\n            name=class_name,\n            score=detection_score\n        )\n        pred_box3ds.append(box3d)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ⚠️ Note: This kernel is a work in progress\n\nI didn't want to hold off on releasing this kernel as I think it will help with getting started in this competition as it is :). \n\nAt this point we have `pred_box3ds` and `gt_box3ds`, they are the predictions and the targets on the validation set.\nNext steps: \n* Compute mAP on the validation set using the evaluation script provided in the SDK\n* Run inference on the test set.\n* Making a submission.\n\n"},{"metadata":{},"cell_type":"markdown","source":"### Model limitations\n- The model performs very poorly on uncommon classes.\n- The boxes are imprecise: the input has a very low resolution (one pixel is 40x40cm in the real world!), and we arbitrarily threshold the predictions and fit boxes around these boxes. As we evaluate with IoUs between 0.4 and 0.75, we can expect that to hurt the score.\n- The model is barely converged - we could train for longer.\n- We only use LIDAR data, and we only use one lidar sweep.\n- We compress the height dimension into only 3 channels. We assume every object is 1.75 meters tall and is at the same height of the ego vehicle, which is surely a wrong assumption."},{"metadata":{"trusted":true},"cell_type":"code","source":"# del model\nbatch_size = 8\nepochs = 15 # Note: We may be able to train for longer and expect better results, the reason this number is low is to keep the runtime short.\n\nmodel = get_unet_model(num_output_classes=len(classes)+1)\n\nepoch_to_load=15\ncheckpoint_filename = \"unet_checkpoint_epoch_{}.pth\".format(epoch_to_load)\ncheckpoint_filepath = os.path.join(ARTIFACTS_FOLDER, checkpoint_filename)\nmodel.load_state_dict(torch.load(checkpoint_filepath))\nprint(model.eval())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calc_detection_box(prediction_opened,class_probability):\n\n    sample_boxes = []\n    sample_detection_scores = []\n    sample_detection_classes = []\n    \n    contours, hierarchy = cv2.findContours(prediction_opened, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE) \n    \n    for cnt in contours:\n        rect = cv2.minAreaRect(cnt)\n        box = cv2.boxPoints(rect)\n        \n        # Let's take the center pixel value as the confidence value\n        box_center_index = np.int0(np.median(box, axis=0))\n        \n        for class_index in range(len(classes)):\n            box_center_value = class_probability[class_index+1, box_center_index[1], box_center_index[0]]\n            \n            # Let's remove candidates with very low probability\n            if box_center_value < 0.3:\n                continue\n            \n            box_center_class = classes[class_index]\n\n            box_detection_score = box_center_value\n            sample_detection_classes.append(box_center_class)\n            sample_detection_scores.append(box_detection_score)\n            sample_boxes.append(box)\n            \n    return np.array(sample_boxes),sample_detection_scores,sample_detection_classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We perform an opening morphological operation to filter tiny detections\n# Note that this may be problematic for classes that are inherently small (e.g. pedestrians)..\nkernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(3,3))\n    \ndef open_preds(predictions_non_class0):\n\n    predictions_opened = np.zeros((predictions_non_class0.shape), dtype=np.uint8)\n\n    for i, p in enumerate(tqdm(predictions_non_class0)):\n        thresholded_p = (p > background_threshold).astype(np.uint8)\n        predictions_opened[i] = cv2.morphologyEx(thresholded_p, cv2.MORPH_OPEN, kernel)\n        \n    return predictions_opened","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()\n\ntest_data_folder = './artifacts/'\ninput_filepaths = sorted(glob.glob(os.path.join(test_data_folder, \"*_input.png\")))\ntarget_filepaths = sorted(glob.glob(os.path.join(test_data_folder, \"*_target.png\")))\n\ntest_dataset = BEVImageDataset(input_filepaths, target_filepaths)\n\ntest_loader = torch.utils.data.DataLoader(test_dataset, batch_size, shuffle=True, num_workers=os.cpu_count()*2)\nprogress_bar = tqdm_notebook(test_loader)\n\n# We quantize to uint8 here to conserve memory. We're allocating >20GB of memory otherwise.\n# predictions = np.zeros((len(test_loader), 1+len(classes), 336, 336), dtype=np.uint8)\n\nsample_tokens = []\nall_losses = []\n\ndetection_boxes = []\ndetection_scores = []\ndetection_classes = []\n\n# Arbitrary threshold in our system to create a binary image to fit boxes around.\nbackground_threshold = 200\n\nwith torch.no_grad():\n    model.eval()\n    for ii, (X, batch_sample_tokens) in enumerate(progress_bar):\n\n        sample_tokens.extend(batch_sample_tokens)\n        \n        X = X.to(device)  # [N, 1, H, W]\n        prediction = model(X)  # [N, 2, H, W]\n        \n        prediction = F.softmax(prediction, dim=1)\n        \n        prediction_cpu = prediction.cpu().numpy()\n        predictions = np.round(prediction_cpu*255).astype(np.uint8)\n        \n        # Get probabilities for non-background\n        predictions_non_class0 = 255 - predictions[:,0]\n        \n        predictions_opened = np.zeros((predictions_non_class0.shape), dtype=np.uint8)\n\n        for i, p in enumerate(predictions_non_class0):\n            thresholded_p = (p > background_threshold).astype(np.uint8)\n            predictions_opened[i] = cv2.morphologyEx(thresholded_p, cv2.MORPH_OPEN, kernel)\n    \n            sample_boxes,sample_detection_scores,sample_detection_classes = calc_detection_box(predictions_opened[i],\n                                                                                              predictions[i])\n        \n            detection_boxes.append(np.array(sample_boxes))\n            detection_scores.append(sample_detection_scores)\n            detection_classes.append(sample_detection_classes)\n        \n#         # Visualize the first prediction\n#         if ii == 0:\n#             visualize_predictions(X, prediction, apply_softmax=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}