{"cells":[{"metadata":{},"cell_type":"markdown","source":"<a id=\"toc\"></a>\n# Table of Contents\n1. [Introduction](#introduction)\n1. [Download some useful packages](#download_some_useful_packages)\n1. [Import packages](#import_packages)\n1. [Load json files from dataset](#load_json_files_from_dataset)\n1. [Data example](#data_example)\n1. [Define generator](#define_generator)\n1. [Define metrics](#define_metrics)\n1. [Define loss function](#define_loss_function)\n1. [Configure parameters](#configure_parameters)\n1. [Create train and validation generators](#create_train_and_validation_generators)\n1. [Create callbacks](#create_callbacks)\n1. [Load pre-trained model](#load_pretrained_model)\n1. [Train 3D U-Net model](#train_3d_unet_model)\n1. [Visualize some results of validation set](#visualize_some_results_of_validation_set)\n1. [Conclusion](#conclusion)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"introduction\"></a>\n# Introduction\n- This kernel is forked from [3d segmentation approach](https://www.kaggle.com/fartuk1/3d-segmentation-approach) and modified to handle multi-label 3D semantic segmentation on Lyft's dataset.\n- The approach used in this kernel cannot directly solved 3D object-detection task of Lyft's competition, but it may be useful for who are unfamiliar with 3D data and/or using Neural-Network model to apply in 3D problems.\n\n---\n[Back to Table of Contents](#toc)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"download_some_useful_packages\"></a>\n# Download some useful packages\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pyquaternion","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"import_packages\"></a>\n# Import packages\n[Back to Table of Contents](#toc)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import json\nimport os\nimport os.path\nimport numpy as np\nimport pandas as pd\nimport random\nimport itertools\nimport copy\nimport math\nfrom matplotlib import pyplot as plt\nfrom pyquaternion import Quaternion\nfrom matplotlib import pyplot\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom skimage.morphology import convex_hull_image\n\nimport tensorflow as tf\n\nimport keras\nfrom keras import backend as K\nfrom keras.engine import Input, Model\nfrom keras.layers import Conv3D, MaxPooling3D, UpSampling3D, Activation, BatchNormalization, PReLU, Deconvolution3D\nfrom keras.optimizers import Adam\nfrom keras.models import Model, Sequential\nfrom keras.callbacks import ModelCheckpoint, TensorBoard, ReduceLROnPlateau\nfrom keras.losses import binary_crossentropy\n\nK.set_image_data_format(\"channels_first\")\n\ntry:\n    from keras.engine import merge\nexcept ImportError:\n    from keras.layers.merge import concatenate","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir('/kaggle/input/3d-object-detection-for-autonomous-vehicles'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"load_json_files_from_dataset\"></a>\n# Load json files from dataset\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Table:\n    def __init__(self, data):\n        self.data = data\n        self.index = {x['token']: x for x in data}\n\n\nDATA_ROOT = '/kaggle/input/3d-object-detection-for-autonomous-vehicles/'\n\n\ndef load_table(name, root=os.path.join(DATA_ROOT, 'train_data')):\n    with open(os.path.join(root, name), 'rb') as f:\n        return Table(json.load(f))\n\n    \nscene = load_table('scene.json')\nsample = load_table('sample.json')\nsample_data = load_table('sample_data.json')\nego_pose = load_table('ego_pose.json')\ncalibrated_sensor = load_table('calibrated_sensor.json')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(DATA_ROOT, 'train.csv')).set_index('Id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Translations of coordinates from https://www.kaggle.com/lopuhin/lyft-3d-join-all-lidars-annotations-from-scratch"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rotate_points(points, rotation, inverse=False):\n    assert points.shape[1] == 3\n    q = Quaternion(rotation)\n    if inverse:\n        q = q.inverse\n    return np.dot(q.rotation_matrix, points.T).T\n    \ndef apply_pose(points, cs, inverse=False):\n    \"\"\" Translate (lidar) points to vehicle coordinates, given a calibrated sensor.\n    \"\"\"\n    points = rotate_points(points, cs['rotation'])\n    points = points + np.array(cs['translation'])\n    return points\n\ndef inverse_apply_pose(points, cs):\n    \"\"\" Reverse of apply_pose (we'll need it later).\n    \"\"\"\n    points = points - np.array(cs['translation']) \n    points = rotate_points(points, np.array(cs['rotation']), inverse=True)\n    return points\n\ndef get_annotations(token):\n    annotations = np.array(train_df.loc[token].PredictionString.split()).reshape(-1, 8)\n    return {\n        'point': annotations[:, :3].astype(np.float32),\n        'wlh': annotations[:, 3:6].astype(np.float32),\n        'rotation': annotations[:, 6].astype(np.float32),\n        'cls': np.array(annotations[:, 7]),\n    }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Helpers to rotate bounding box points"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rotate(origin, point, angle):\n    ox, oy, _ = origin\n    px, py, pz = point\n\n    qx = ox + math.cos(angle) * (px - ox) - math.sin(angle) * (py - oy)\n    qy = oy + math.sin(angle) * (px - ox) + math.cos(angle) * (py - oy)\n    return [qx, qy, pz]\n\n\ndef make_box_coords(center, wlh, rotation, ep):\n\n    planar_wlh = copy.deepcopy(wlh)\n    planar_wlh = planar_wlh[[1,0,2]]\n\n    bottom_center = copy.deepcopy(center)\n    bottom_center[-1] = bottom_center[-1] - planar_wlh[-1] / 2\n\n    bottom_points = []\n    bottom_points.append(bottom_center + planar_wlh * [1, 1, 0] / 2)\n    bottom_points.append(bottom_center + planar_wlh * [-1, -1, 0] / 2)\n    bottom_points.append(bottom_center + planar_wlh * [1, -1, 0] / 2)\n    bottom_points.append(bottom_center + planar_wlh * [-1, 1, 0] / 2)\n    bottom_points = np.array(bottom_points)\n\n    rotated_bottom_points = []\n    for point in bottom_points:\n        rotated_bottom_points.append(rotate(bottom_center, point, rotation))\n\n    rotated_bottom_points = np.array(rotated_bottom_points)\n    rotated_top_points = rotated_bottom_points + planar_wlh * [0,0,1]\n\n    box_points = np.concatenate([rotated_bottom_points, rotated_top_points], axis=0)\n\n    box_points = inverse_apply_pose(box_points, ep)\n    \n    return box_points","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Helpers to get training data in raster format. To each sample we can create raster 1000x1000x100 image as 3-dimensional histogram of points. Mask is bounding boxes of cars. There is get_crop_positive function to create small crops from sample image to put it in neural network "},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_sample_data(sample_token):\n    lidars = []\n    for x in sample_data.data:\n        if x['sample_token'] == sample_token and 'lidar' in x['filename']:\n            lidars.append(x)\n\n    lidars_data = [\n        # here, sorry\n        np.fromfile(os.path.join(DATA_ROOT, x['filename'].replace('lidar/', 'train_lidar/')), dtype=np.float32)\n        .reshape(-1, 5)[:, :3] for x in lidars]\n\n\n    all_points = []\n    all_colors = []\n    for points, lidar in zip(lidars_data, lidars):\n        cs = calibrated_sensor.index[lidar['calibrated_sensor_token']]\n        points = apply_pose(points, cs)\n        all_points.append(points)\n    all_points = np.concatenate(all_points)\n\n\n    ego_pose_token, = {x['ego_pose_token'] for x in lidars}\n    ep = ego_pose.index[ego_pose_token]\n    annotations = get_annotations(sample_token)\n\n    car_centers = annotations['point'][annotations['cls'] == 'car']\n    car_wlhs = annotations['wlh'][annotations['cls'] == 'car']\n    car_rotations = annotations['rotation'][annotations['cls'] == 'car']\n\n    truck_centers = annotations['point'][annotations['cls'] == 'truck']\n    truck_wlhs = annotations['wlh'][annotations['cls'] == 'truck']\n    truck_rotations = annotations['rotation'][annotations['cls'] == 'truck']\n\n    other_vehicle_centers = annotations['point'][annotations['cls'] == 'other_vehicle']\n    other_vehicle_wlhs = annotations['wlh'][annotations['cls'] == 'other_vehicle']\n    other_vehicle_rotations = annotations['rotation'][annotations['cls'] == 'other_vehicle']\n    \n    bus_centers = annotations['point'][annotations['cls'] == 'bus']\n    bus_wlhs = annotations['wlh'][annotations['cls'] == 'bus']\n    bus_rotations = annotations['rotation'][annotations['cls'] == 'bus']\n\n    pedestrian_centers = annotations['point'][annotations['cls'] == 'pedestrian']\n    pedestrian_wlhs = annotations['wlh'][annotations['cls'] == 'pedestrian']\n    pedestrian_rotations = annotations['rotation'][annotations['cls'] == 'pedestrian']\n\n    bicycle_centers = annotations['point'][annotations['cls'] == 'bicycle']\n    bicycle_wlhs = annotations['wlh'][annotations['cls'] == 'bicycle']\n    bicycle_rotations = annotations['rotation'][annotations['cls'] == 'bicycle']\n\n\n    # Car\n    car_boxes = []\n    for k in range(len(car_centers)):\n        center = car_centers[k]\n        wlh = car_wlhs[k]\n        rotation = car_rotations[k]\n\n        box_coords = make_box_coords(center, wlh, rotation, ep)\n        car_boxes.append(box_coords)\n\n    car_boxes = np.array(car_boxes)    \n    car_centers = inverse_apply_pose(car_centers, ep)\n    \n    # Truck\n    truck_boxes = []\n    for k in range(len(truck_centers)):\n        center = truck_centers[k]\n        wlh = truck_wlhs[k]\n        rotation = truck_rotations[k]\n\n        box_coords = make_box_coords(center, wlh, rotation, ep)\n        truck_boxes.append(box_coords)\n\n    truck_boxes = np.array(truck_boxes)    \n    truck_centers = inverse_apply_pose(truck_centers, ep)\n    \n    # Other vehicle\n    other_vehicle_boxes = []\n    for k in range(len(other_vehicle_centers)):\n        center = other_vehicle_centers[k]\n        wlh = other_vehicle_wlhs[k]\n        rotation = other_vehicle_rotations[k]\n\n        box_coords = make_box_coords(center, wlh, rotation, ep)\n        other_vehicle_boxes.append(box_coords)\n\n    other_vehicle_boxes = np.array(other_vehicle_boxes)    \n    other_vehicle_centers = inverse_apply_pose(other_vehicle_centers, ep)\n    \n    # Bus\n    bus_boxes = []\n    for k in range(len(bus_centers)):\n        center = bus_centers[k]\n        wlh = bus_wlhs[k]\n        rotation = bus_rotations[k]\n\n        box_coords = make_box_coords(center, wlh, rotation, ep)\n        bus_boxes.append(box_coords)\n\n    bus_boxes = np.array(bus_boxes)    \n    bus_centers = inverse_apply_pose(bus_centers, ep)\n    \n    # Pedestrian\n    pedestrian_boxes = []\n    for k in range(len(pedestrian_centers)):\n        center = pedestrian_centers[k]\n        wlh = pedestrian_wlhs[k]\n        rotation = pedestrian_rotations[k]\n\n        box_coords = make_box_coords(center, wlh, rotation, ep)\n        pedestrian_boxes.append(box_coords)\n\n    pedestrian_boxes = np.array(pedestrian_boxes)    \n    pedestrian_centers = inverse_apply_pose(pedestrian_centers, ep)\n    \n    # Bicycle\n    bicycle_boxes = []\n    for k in range(len(bicycle_centers)):\n        center = bicycle_centers[k]\n        wlh = bicycle_wlhs[k]\n        rotation = bicycle_rotations[k]\n\n        box_coords = make_box_coords(center, wlh, rotation, ep)\n        bicycle_boxes.append(box_coords)\n\n    bicycle_boxes = np.array(bicycle_boxes)    \n    bicycle_centers = inverse_apply_pose(bicycle_centers, ep)\n    \n    return all_points, \\\n        car_boxes, car_centers, \\\n        truck_boxes, truck_centers, \\\n        other_vehicle_boxes, other_vehicle_centers, \\\n        bus_boxes, bus_centers, \\\n        pedestrian_boxes, pedestrian_centers, \\\n        bicycle_boxes, bicycle_centers\n\n\ndef get_sample_raster(all_points, all_boxes): \n    x_bounds = np.linspace(-100, 100, 1001)\n    y_bounds = np.linspace(-100, 100, 1001)\n    z_bounds = np.linspace(-10, 10, 101)\n\n    sample_hist = np.histogramdd(all_points[:], [x_bounds, y_bounds, z_bounds])[0]\n    sample_mask = np.zeros((len(x_bounds)-1, len(y_bounds)-1, len(z_bounds)-1))\n\n\n\n    for box in all_boxes:\n        x_min, y_min, z_min = box.min(axis=0)\n        x_max, y_max, z_max = box.max(axis=0)\n\n        x_box_bound_cnt = int(1001 / 200 * (x_max - x_min))\n        y_box_bound_cnt = int(1001 / 200 * (y_max - y_min))\n        z_box_bound_cnt = int(101 / 20 * (z_max - z_min))\n\n        box_hist = np.histogramdd(box, [np.linspace(x_min, x_max, x_box_bound_cnt),\n                                        np.linspace(y_min, y_max, y_box_bound_cnt),\n                                        np.linspace(z_min, z_max, z_box_bound_cnt)])[0]\n\n\n        try:\n            box_mask = convex_hull_image(box_hist)\n        except:\n            continue\n\n\n        x_start_idx = np.where(x_bounds > x_min)[0][0]\n        y_start_idx = np.where(y_bounds > y_min)[0][0]\n        z_start_idx = np.where(z_bounds > z_min)[0][0]\n\n\n        x_cnt = min(sample_mask.shape[0] - x_start_idx - 1, x_box_bound_cnt - 1)\n        y_cnt = min(sample_mask.shape[1] - y_start_idx - 1, y_box_bound_cnt - 1)\n        z_cnt = min(sample_mask.shape[2] - z_start_idx - 1, z_box_bound_cnt - 1)\n\n        sample_mask[x_start_idx:x_start_idx+x_cnt,\n                   y_start_idx:y_start_idx+y_cnt,\n                   z_start_idx:z_start_idx+z_cnt] = sample_mask[x_start_idx:x_start_idx+x_cnt,\n                                                                           y_start_idx:y_start_idx+y_cnt,\n                                                                           z_start_idx:z_start_idx+z_cnt] + box_mask[:x_cnt, :y_cnt, :z_cnt]\n\n    return sample_hist, sample_mask, (x_bounds, y_bounds, z_bounds)\n\n\ndef get_crop_positive(sample_hist, sample_mask, bounds, centers, crop_size=(64, 64, 32)):\n    \n    half_x_size = crop_size[0] // 2\n    half_y_size = crop_size[1] // 2\n    half_z_size = crop_size[2] // 2\n    \n    (x_bounds, y_bounds, z_bounds) = bounds\n    if len(centers) > 0:\n        idx = np.random.choice(range(len(centers)))\n        x_center, y_center, z_center = centers[idx]\n    else:\n        x_center, y_center = np.random.randint(-30, 30, 2)\n        z_center = np.random.randint(-10, 10)\n\n    x_center, y_center, z_center = [x_center, y_center, z_center] + np.random.randint(-3, 3, 3)\n\n    x_center = min(x_center, 100 - np.abs(x_bounds[-1] - x_bounds[-2]) * half_x_size - 1)\n    x_center = max(x_center, -100 + np.abs(x_bounds[-1] - x_bounds[-2]) * half_x_size + 1)\n\n    y_center = min(y_center, 100 - np.abs(y_bounds[-1] - y_bounds[-2]) * half_y_size - 1)\n    y_center = max(y_center, -100 + np.abs(y_bounds[-1] - y_bounds[-2]) * half_y_size + 1)\n\n    z_center = min(z_center, 10 - np.abs(z_bounds[-1] - z_bounds[-2]) * half_z_size - 1)\n    z_center = max(z_center, -10 + np.abs(z_bounds[-1] - z_bounds[-2]) * half_z_size + 1)\n\n\n\n\n    x_center_idx = np.where(x_bounds > x_center)[0][0]\n    y_center_idx = np.where(y_bounds > y_center)[0][0]\n    z_center_idx = np.where(z_bounds > z_center)[0][0]\n\n        \n    crop_hist = sample_hist[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    \n    crop_mask = sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n\n    return crop_hist, crop_mask > 0\n\n\n\ndef get_crop_over_all_classes(sample_hist, bounds, \\\n                              car_sample_mask, truck_sample_mask, \\\n                              other_vehicle_sample_mask, bus_sample_mask, \\\n                              pedestrian_sample_mask, bicycle_sample_mask, \\\n                              car_centers, truck_centers, \\\n                              other_vehicle_centers, bus_centers, \\\n                              pedestrian_centers, bicycle_centers, \\\n                              crop_size=(64, 64, 32)):\n    \n    half_x_size = crop_size[0] // 2\n    half_y_size = crop_size[1] // 2\n    half_z_size = crop_size[2] // 2\n    \n    (x_bounds, y_bounds, z_bounds) = bounds\n    \n    centers = np.concatenate((car_centers, truck_centers, other_vehicle_centers, bus_centers, pedestrian_centers, bicycle_centers), axis=0)\n    if len(centers) > 0:\n        idx = np.random.choice(range(len(centers)))\n        x_center, y_center, z_center = centers[idx]\n    else:\n        x_center, y_center = np.random.randint(-30, 30, 2)\n        z_center = np.random.randint(-10, 10)\n\n\n    x_center, y_center, z_center = [x_center, y_center, z_center] + np.random.randint(-3, 3, 3)\n\n    x_center = min(x_center, 100 - np.abs(x_bounds[-1] - x_bounds[-2]) * half_x_size - 1)\n    x_center = max(x_center, -100 + np.abs(x_bounds[-1] - x_bounds[-2]) * half_x_size + 1)\n\n    y_center = min(y_center, 100 - np.abs(y_bounds[-1] - y_bounds[-2]) * half_y_size - 1)\n    y_center = max(y_center, -100 + np.abs(y_bounds[-1] - y_bounds[-2]) * half_y_size + 1)\n\n    z_center = min(z_center, 10 - np.abs(z_bounds[-1] - z_bounds[-2]) * half_z_size - 1)\n    z_center = max(z_center, -10 + np.abs(z_bounds[-1] - z_bounds[-2]) * half_z_size + 1)\n\n\n\n\n    x_center_idx = np.where(x_bounds > x_center)[0][0]\n    y_center_idx = np.where(y_bounds > y_center)[0][0]\n    z_center_idx = np.where(z_bounds > z_center)[0][0]\n\n        \n    crop_hist = sample_hist[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    \n    car_crop_mask = car_sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    truck_crop_mask = truck_sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    other_vehicle_crop_mask = other_vehicle_sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    bus_crop_mask = bus_sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    pedestrian_crop_mask = pedestrian_sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n    bicycle_crop_mask = bicycle_sample_mask[x_center_idx-half_x_size:x_center_idx+half_x_size,\n                            y_center_idx-half_y_size:y_center_idx+half_y_size,\n                            z_center_idx-half_z_size:z_center_idx+half_z_size]\n\n    crop_mask = np.array((car_crop_mask, truck_crop_mask, other_vehicle_crop_mask, bus_crop_mask, pedestrian_crop_mask, bicycle_crop_mask))\n    \n\n    return crop_hist, crop_mask > 0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"data_example\"></a>\n# Data example\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_token = train_df.reset_index()['Id'].values[71]\n\nall_points, car_boxes, car_centers, \\\ntruck_boxes, truck_centers, \\\nother_vehicle_boxes, other_vehicle_centers, \\\nbus_boxes, bus_centers, \\\npedestrian_boxes, pedestrian_centers, \\\nbicycle_boxes, bicycle_centers = get_sample_data(sample_token)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Full raster image for one sample"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sample_hist, sample_mask, bounds = get_sample_raster(all_points, all_boxes)\n\ncar_sample_hist, car_sample_mask, car_bounds = get_sample_raster(all_points, car_boxes)\ntruck_sample_hist, truck_sample_mask, truck_bounds = get_sample_raster(all_points, truck_boxes)\nother_vehicle_sample_hist, other_vehicle_sample_mask, other_vehicle_bounds = get_sample_raster(all_points, other_vehicle_boxes)\nbus_sample_hist, bus_sample_mask, bus_bounds = get_sample_raster(all_points, bus_boxes)\npedestrian_sample_hist, pedestrian_sample_mask, pedestrian_bounds = get_sample_raster(all_points, pedestrian_boxes)\nbicycle_sample_hist, bicycle_sample_mask, bicycle_bounds = get_sample_raster(all_points, bicycle_boxes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Crop from this image"},{"metadata":{"trusted":true},"cell_type":"code","source":"crop_hist, crop_mask = get_crop_over_all_classes(\n    car_sample_hist, car_bounds, \\\n    car_sample_mask, truck_sample_mask, \\\n    other_vehicle_sample_mask, bus_sample_mask, \\\n    pedestrian_sample_mask, bicycle_sample_mask, \\\n    car_centers, truck_centers, \\\n    other_vehicle_centers, bus_centers, \\\n    pedestrian_centers, bicycle_centers, \\\n    crop_size=(128,128,64))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Source point cloud for full sample"},{"metadata":{"trusted":true},"cell_type":"code","source":"boxes_coords = np.concatenate(car_boxes, axis=0)\n\n\nplt.figure(figsize=(25,15))\nplt.scatter(all_points[:, 0], all_points[:, 1],s=[0.1]*len(all_points))\nplt.scatter(boxes_coords[:, 0], boxes_coords[:, 1],s=[15]*len(boxes_coords),color='r')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Source point cloud for small region with car"},{"metadata":{"trusted":true},"cell_type":"code","source":"ann_idx = 20\n\ncenter_point = car_centers[ann_idx]\nx_min = center_point[0] - 5\nx_max = center_point[0] + 5\ny_min = center_point[1] - 5\ny_max = center_point[1] + 5\nz_min= center_point[2] - 5\nz_max = center_point[2] + 5\n\n\narea_mask = (all_points[:, 0] > x_min) * (all_points[:, 0] < x_max) * (all_points[:, 1] > y_min) * (all_points[:, 1] < y_max) * (all_points[:, 2] > z_min) * (all_points[:, 2] < z_max)\narea_mask = np.where(area_mask)[0]\n\n\nfig = pyplot.figure(figsize=(25,15))\nax = Axes3D(fig)\nax.scatter(all_points[area_mask, 0], all_points[area_mask, 1], all_points[area_mask, 2])\n\n\nax.scatter(car_boxes[ann_idx][:, 0], car_boxes[ann_idx][:, 1], car_boxes[ann_idx][:, 2], color='r', s=[100])\n\n\n\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Rastered crop of a small region"},{"metadata":{},"cell_type":"markdown","source":"Show only top projection of 3d image"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(6, 2, figsize=(40, 40))\nfor i in range(6):\n    axes[i][0].imshow(crop_hist.sum(axis=-1))\n    axes[i][1].imshow(crop_mask[i].sum(axis=-1))\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"define_3d_unet_model\"></a>\n# Define 3D U-Net model\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def unet_model_3d(input_shape, pool_size=(2, 2, 2), n_labels=1, initial_learning_rate=0.00001, deconvolution=False,\n                  depth=4, n_base_filters=32,\n                  batch_normalization=False, activation_name=\"sigmoid\"):\n    \n    inputs = Input(input_shape)\n    current_layer = inputs\n    levels = list()\n\n    # add levels with max pooling\n    for layer_depth in range(depth):\n        layer1 = create_convolution_block(input_layer=current_layer, n_filters=n_base_filters*(2**layer_depth),\n                                          batch_normalization=batch_normalization)\n        layer2 = create_convolution_block(input_layer=layer1, n_filters=n_base_filters*(2**layer_depth)*2,\n                                          batch_normalization=batch_normalization)\n        if layer_depth < depth - 1:\n            current_layer = MaxPooling3D(pool_size=pool_size)(layer2)\n            levels.append([layer1, layer2, current_layer])\n        else:\n            current_layer = layer2\n            levels.append([layer1, layer2])\n\n    # add levels with up-convolution or up-sampling\n    for layer_depth in range(depth-2, -1, -1):\n        up_convolution = get_up_convolution(pool_size=pool_size, deconvolution=deconvolution,\n                                            n_filters=current_layer._keras_shape[1])(current_layer)\n        concat = concatenate([up_convolution, levels[layer_depth][1]], axis=1)\n        current_layer = create_convolution_block(n_filters=levels[layer_depth][1]._keras_shape[1],\n                                                 input_layer=concat, batch_normalization=batch_normalization)\n        current_layer = create_convolution_block(n_filters=levels[layer_depth][1]._keras_shape[1],\n                                                 input_layer=current_layer,\n                                                 batch_normalization=batch_normalization)\n\n    final_convolution = Conv3D(n_labels, (1, 1, 1))(current_layer)\n    act = Activation(activation_name)(final_convolution)\n    model = Model(inputs=inputs, outputs=act)\n\n    return model\n\n\ndef create_convolution_block(input_layer, n_filters, batch_normalization=False, kernel=(3, 3, 3), activation=None,\n                             padding='same', strides=(1, 1, 1), instance_normalization=False):\n\n    layer = Conv3D(n_filters, kernel, padding=padding, strides=strides)(input_layer)\n    if batch_normalization:\n        layer = BatchNormalization(axis=1)(layer)\n    elif instance_normalization:\n        from keras_contrib.layers.normalization import InstanceNormalization\n\n        layer = InstanceNormalization(axis=1)(layer)\n    if activation is None:\n        return Activation('relu')(layer)\n    else:\n        return activation()(layer)\n\n\ndef compute_level_output_shape(n_filters, depth, pool_size, image_shape):\n    output_image_shape = np.asarray(np.divide(image_shape, np.power(pool_size, depth)), dtype=np.int32).tolist()\n    return tuple([None, n_filters] + output_image_shape)\n\n\ndef get_up_convolution(n_filters, pool_size, kernel_size=(2, 2, 2), strides=(2, 2, 2),\n                       deconvolution=False):\n    if deconvolution:\n        return Deconvolution3D(filters=n_filters, kernel_size=kernel_size,\n                               strides=strides)\n    else:\n        return UpSampling3D(size=pool_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"define_generator\"></a>\n# Define generator\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"tokens = train_df.reset_index()['Id'].values\n\ndef generator(tokens, crop_size, batch_size):\n    while True:\n        sample_token = np.random.choice(tokens)\n        all_points, car_boxes, car_centers, truck_boxes, truck_centers, other_vehicle_boxes, other_vehicle_centers, bus_boxes, bus_centers, pedestrian_boxes, pedestrian_centers, bicycle_boxes, bicycle_centers = get_sample_data(sample_token)\n\n\n        # All generated sample_hists of all classes are the same, so we just pick one\n        sample_hist, car_sample_mask, bounds = get_sample_raster(all_points, car_boxes)\n        _, truck_sample_mask, _ = get_sample_raster(all_points, truck_boxes)\n        _, other_vehicle_sample_mask, _ = get_sample_raster(all_points, other_vehicle_boxes)\n        _, bus_sample_mask, _ = get_sample_raster(all_points, bus_boxes)\n        _, pedestrian_sample_mask, _ = get_sample_raster(all_points, pedestrian_boxes)\n        _, bicycle_sample_mask, _ = get_sample_raster(all_points, bicycle_boxes)\n        \n        x_batch = []\n        y_batch = []\n        for _ in range(batch_size):\n            crop_hist, crop_mask = get_crop_over_all_classes(\n                sample_hist, bounds, \\\n                car_sample_mask, truck_sample_mask, \\\n                other_vehicle_sample_mask, bus_sample_mask, \\\n                pedestrian_sample_mask, bicycle_sample_mask, \\\n                car_centers, truck_centers, \\\n                other_vehicle_centers, bus_centers, \\\n                pedestrian_centers, bicycle_centers, \\\n                crop_size=crop_size)\n\n            x_batch.append(crop_hist)\n            y_batch.append(crop_mask)\n        \n        x_batch = np.array(x_batch)\n        y_batch = np.array(y_batch)\n        \n        x_batch = np.expand_dims(x_batch, axis=1)\n        \n        yield x_batch , y_batch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"define_metrics\"></a>\n# Define metrics\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def jaccard_coef(y_true, y_pred):\n    smooth = 1e-12\n    intersection = K.sum(y_true * y_pred, axis=[-1, -2, -3])\n    sum_ = K.sum(y_true + y_pred, axis=[-1, -2, -3])\n    jac = (intersection + smooth) / (sum_ - intersection + smooth)\n    return K.mean(jac)\n\n\ndef jaccard_coef_int(y_true, y_pred):\n    smooth = 1e-12\n    y_pred_pos = K.round(K.clip(y_pred, 0, 1))\n    intersection = K.sum(y_true * y_pred_pos, axis=[-1, -2, -3])\n    sum_ = K.sum(y_true + y_pred_pos, axis=[-1, -2, -3])\n    jac = (intersection + smooth) / (sum_ - intersection + smooth)\n    return K.mean(jac)\n\n\ndef dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"define_loss_function\"></a>\n# Define loss function\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def weighted_binary_crossentropy(alpha=0.75):\n    def weighted_binary_crossentropy_fixed(y_true, y_pred):\n        binary_crossentropy = K.binary_crossentropy(y_true, y_pred)\n        weight_vec = alpha * y_true + (1 - alpha) * (1 - y_true)\n        \n        return K.mean(binary_crossentropy * weight_vec)\n        \n    return weighted_binary_crossentropy_fixed\n\n\ndef binary_focal_loss(gamma=2., alpha=.25):\n    \"\"\"\n    Binary form of focal loss.\n\n      FL(p_t) = -alpha * (1 - p_t)**gamma * log(p_t)\n\n      where p = sigmoid(x), p_t = p or 1 - p depending on if the label is 1 or 0, respectively.\n\n    References:\n        https://arxiv.org/pdf/1708.02002.pdf\n    Usage:\n     model.compile(loss=[binary_focal_loss(alpha=.25, gamma=2)], metrics=[\"accuracy\"], optimizer=adam)\n\n    \"\"\"\n    def binary_focal_loss_fixed(y_true, y_pred):\n        \"\"\"\n        :param y_true: A tensor of the same shape as `y_pred`\n        :param y_pred:  A tensor resulting from a sigmoid\n        :return: Output tensor.\n        \"\"\"\n        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n\n        epsilon = K.epsilon()\n        # clip to prevent NaN's and Inf's\n        pt_1 = K.clip(pt_1, epsilon, 1. - epsilon)\n        pt_0 = K.clip(pt_0, epsilon, 1. - epsilon)\n\n        return -K.mean(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) \\\n               -K.mean((1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n\n    return binary_focal_loss_fixed\n\n\ndef dice_loss(y_true, y_pred):\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = y_true_f * y_pred_f\n    score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n    return 1. - score\n\n\ndef bce_dice_loss(y_true, y_pred):\n    return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"configure_parameters\"></a>\n# Configure parameters\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 16\nlr = 1e-5\nepochs = 10","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"create_train_and_validation_generators\"></a>\n# Create train and validation generators\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader = generator(tokens[:15000], (64,64,32), batch_size)\nval_loader = generator(tokens[15000:], (64,64,32), batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for x_batch, y_batch in train_loader:\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(x_batch.shape)\nprint(y_batch.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"idx = 0\nfig, axes = plt.subplots(6, 2, figsize=(40, 40))\nfor i in range(6):\n    axes[i][0].imshow(x_batch[idx].sum(axis=(0, -1)))\n    axes[i][1].imshow(np.expand_dims(y_batch[idx][i], axis=0).sum(axis=(0, -1)))\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"create_callbacks\"></a>\n# Create callbacks\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"tensorboard_callback = TensorBoard(\n    log_dir='./',\n    histogram_freq=0,\n    batch_size=batch_size,\n    write_graph=False,\n    write_grads=False,\n    write_images=False,\n    embeddings_freq=0,\n    embeddings_layer_names=None,\n    embeddings_metadata=None\n)\n\nreduce_lr_on_plateau_callback = ReduceLROnPlateau(\n    monitor='val_loss',\n    factor=0.2,\n    patience=5,\n    verbose=1,\n    mode='auto',\n    min_delta=0.0001,\n    cooldown=0,\n    min_lr=0\n)\n\nmodel_checkpoint = ModelCheckpoint(\n    'weights.{epoch:02d}.h5',\n    monitor='val_loss',\n    verbose=0,\n    save_best_only=False,\n    save_weights_only=True,\n    mode='auto',\n    period=1\n)\n\ncallbacks=[\n    tensorboard_callback,\n    reduce_lr_on_plateau_callback,\n    model_checkpoint\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model = unet_model_3d((1, 64,64,32))\nmodel = unet_model_3d((1, 64,64,32), n_labels=6)\nadam = keras.optimizers.Adam(lr=lr, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n\nmodel.compile(loss=bce_dice_loss,\n        optimizer=adam,\n        metrics=[dice_coef, jaccard_coef, jaccard_coef_int])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"load_pretrained_model\"></a>\n# Load pre-trained model\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget \"https://drive.google.com/uc?id=11BXmZn3lGMKE-ukHnKiRvb6ZyVV9XZ1g&export=download\" -O pretrained_weights.h5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('pretrained_weights.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"train_3d_unet_model\"></a>\n# Train 3D U-Net model\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# NOTE: Uncomment below lines to train new model from scratch or pre-trained weights\n\n# model.fit_generator(generator=train_loader,\n#                     steps_per_epoch=100,\n#                     epochs=epochs,\n#                     verbose=1,\n#                     callbacks=callbacks,\n#                     validation_data=val_loader,\n#                     validation_steps=50,\n#                     class_weight=None,\n#                     max_queue_size=10,\n#                     use_multiprocessing=False,\n#                     shuffle=True,\n#                     initial_epoch=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"visualize_some_results_of_validation_set\"></a>\n# Visualize some results of validation set\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_loader = generator(tokens[15000:], (64,64,32), 16)\n\nfor x_batch, y_batch in val_loader:\n    break \n    \npred = model.predict(x_batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(x_batch.shape)\n# print(y_batch.shape)\n# print(pred.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"visualize_predictions_in_all_3_projections\"></a>\n# Vizualize predictions in all 3 projections\n[Back to Table of Contents](#toc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(16):\n    fig, axes = plt.subplots(6, 4, figsize=(40,40))\n    axes[0][0].imshow(x_batch[i].sum(axis=(0, -1)))\n    axes[0][1].imshow(np.expand_dims(y_batch[i][0], axis=0).sum(axis=(0, -1)))\n    axes[0][2].imshow((np.expand_dims(pred[i][0], axis=0)  ).sum(axis=(0, -1)))\n    axes[0][3].imshow((np.expand_dims(pred[i][0], axis=0) > 0.5 ).sum(axis=(0, -1)))\n\n    axes[1][0].imshow(x_batch[i].sum(axis=(0, -1)))\n    axes[1][1].imshow(np.expand_dims(y_batch[i][1], axis=0).sum(axis=(0, -1)))\n    axes[1][2].imshow((np.expand_dims(pred[i][1], axis=0)  ).sum(axis=(0, -1)))\n    axes[1][3].imshow((np.expand_dims(pred[i][1], axis=0) > 0.5 ).sum(axis=(0, -1)))\n\n    axes[2][0].imshow(x_batch[i].sum(axis=(0, -1)))\n    axes[2][1].imshow(np.expand_dims(y_batch[i][2], axis=0).sum(axis=(0, -1)))\n    axes[2][2].imshow((np.expand_dims(pred[i][2], axis=0)  ).sum(axis=(0, -1)))\n    axes[2][3].imshow((np.expand_dims(pred[i][2], axis=0) > 0.5 ).sum(axis=(0, -1)))\n\n    axes[3][0].imshow(x_batch[i].sum(axis=(0, -1)))\n    axes[3][1].imshow(np.expand_dims(y_batch[i][3], axis=0).sum(axis=(0, -1)))\n    axes[3][2].imshow((np.expand_dims(pred[i][3], axis=0)  ).sum(axis=(0, -1)))\n    axes[3][3].imshow((np.expand_dims(pred[i][3], axis=0) > 0.5 ).sum(axis=(0, -1)))\n\n    axes[4][0].imshow(x_batch[i].sum(axis=(0, -1)))\n    axes[4][1].imshow(np.expand_dims(y_batch[i][4], axis=0).sum(axis=(0, -1)))\n    axes[4][2].imshow((np.expand_dims(pred[i][4], axis=0)  ).sum(axis=(0, -1)))\n    axes[4][3].imshow((np.expand_dims(pred[i][4], axis=0) > 0.5 ).sum(axis=(0, -1)))\n\n    axes[5][0].imshow(x_batch[i].sum(axis=(0, -1)))\n    axes[5][1].imshow(np.expand_dims(y_batch[i][5], axis=0).sum(axis=(0, -1)))\n    axes[5][2].imshow((np.expand_dims(pred[i][5], axis=0)  ).sum(axis=(0, -1)))\n    axes[5][3].imshow((np.expand_dims(pred[i][5], axis=0) > 0.5 ).sum(axis=(0, -1)))\n    plt.show()\n\n\n    fig, axes = plt.subplots(6, 4, figsize=(40,40))\n    axes[0][0].imshow(x_batch[i].sum(axis=(0, 1)).T)\n    axes[0][1].imshow(np.expand_dims(y_batch[i][0], axis=0).sum(axis=(0, 1)).T)\n    axes[0][2].imshow((np.expand_dims(pred[i][0], axis=0)).sum(axis=(0, 1)).T)\n    axes[0][3].imshow((np.expand_dims(pred[i][0], axis=0) > 0.5 ).sum(axis=(0, 1)).T)\n\n    axes[1][0].imshow(x_batch[i].sum(axis=(0, 1)).T)\n    axes[1][1].imshow(np.expand_dims(y_batch[i][1], axis=0).sum(axis=(0, 1)).T)\n    axes[1][2].imshow((np.expand_dims(pred[i][1], axis=0)).sum(axis=(0, 1)).T)\n    axes[1][3].imshow((np.expand_dims(pred[i][1], axis=0) > 0.5 ).sum(axis=(0, 1)).T)\n\n    axes[2][0].imshow(x_batch[i].sum(axis=(0, 1)).T)\n    axes[2][1].imshow(np.expand_dims(y_batch[i][2], axis=0).sum(axis=(0, 1)).T)\n    axes[2][2].imshow((np.expand_dims(pred[i][2], axis=0)).sum(axis=(0, 1)).T)\n    axes[2][3].imshow((np.expand_dims(pred[i][2], axis=0) > 0.5 ).sum(axis=(0, 1)).T)\n\n    axes[3][0].imshow(x_batch[i].sum(axis=(0, 1)).T)\n    axes[3][1].imshow(np.expand_dims(y_batch[i][3], axis=0).sum(axis=(0, 1)).T)\n    axes[3][2].imshow((np.expand_dims(pred[i][3], axis=0)).sum(axis=(0, 1)).T)\n    axes[3][3].imshow((np.expand_dims(pred[i][3], axis=0) > 0.5 ).sum(axis=(0, 1)).T)\n\n    axes[4][0].imshow(x_batch[i].sum(axis=(0, 1)).T)\n    axes[4][1].imshow(np.expand_dims(y_batch[i][4], axis=0).sum(axis=(0, 1)).T)\n    axes[4][2].imshow((np.expand_dims(pred[i][4], axis=0)).sum(axis=(0, 1)).T)\n    axes[4][3].imshow((np.expand_dims(pred[i][4], axis=0) > 0.5 ).sum(axis=(0, 1)).T)\n\n    axes[5][0].imshow(x_batch[i].sum(axis=(0, 1)).T)\n    axes[5][1].imshow(np.expand_dims(y_batch[i][5], axis=0).sum(axis=(0, 1)).T)\n    axes[5][2].imshow((np.expand_dims(pred[i][5], axis=0)).sum(axis=(0, 1)).T)\n    axes[5][3].imshow((np.expand_dims(pred[i][5], axis=0) > 0.5 ).sum(axis=(0, 1)).T)\n    plt.show()\n\n\n    fig, axes = plt.subplots(6, 4, figsize=(40,40))\n    axes[0][0].imshow(x_batch[i].sum(axis=(0, 2)).T)\n    axes[0][1].imshow(np.expand_dims(y_batch[i][0], axis=0).sum(axis=(0, 2)).T)\n    axes[0][2].imshow((np.expand_dims(pred[i][0], axis=0)).sum(axis=(0, 2)).T)\n    axes[0][3].imshow((np.expand_dims(pred[i][0], axis=0) > 0.5 ).sum(axis=(0, 2)).T)\n\n    axes[1][0].imshow(x_batch[i].sum(axis=(0, 2)).T)\n    axes[1][1].imshow(np.expand_dims(y_batch[i][1], axis=0).sum(axis=(0, 2)).T)\n    axes[1][2].imshow((np.expand_dims(pred[i][1], axis=0)).sum(axis=(0, 2)).T)\n    axes[1][3].imshow((np.expand_dims(pred[i][1], axis=0) > 0.5 ).sum(axis=(0, 2)).T)\n\n    axes[2][0].imshow(x_batch[i].sum(axis=(0, 2)).T)\n    axes[2][1].imshow(np.expand_dims(y_batch[i][2], axis=0).sum(axis=(0, 2)).T)\n    axes[2][2].imshow((np.expand_dims(pred[i][2], axis=0)).sum(axis=(0, 2)).T)\n    axes[2][3].imshow((np.expand_dims(pred[i][2], axis=0) > 0.5 ).sum(axis=(0, 2)).T)\n\n    axes[3][0].imshow(x_batch[i].sum(axis=(0, 2)).T)\n    axes[3][1].imshow(np.expand_dims(y_batch[i][3], axis=0).sum(axis=(0, 2)).T)\n    axes[3][2].imshow((np.expand_dims(pred[i][3], axis=0)).sum(axis=(0, 2)).T)\n    axes[3][3].imshow((np.expand_dims(pred[i][3], axis=0) > 0.5 ).sum(axis=(0, 2)).T)\n\n    axes[4][0].imshow(x_batch[i].sum(axis=(0, 2)).T)\n    axes[4][1].imshow(np.expand_dims(y_batch[i][4], axis=0).sum(axis=(0, 2)).T)\n    axes[4][2].imshow((np.expand_dims(pred[i][4], axis=0)).sum(axis=(0, 2)).T)\n    axes[4][3].imshow((np.expand_dims(pred[i][4], axis=0) > 0.5 ).sum(axis=(0, 2)).T)\n\n    axes[5][0].imshow(x_batch[i].sum(axis=(0, 2)).T)\n    axes[5][1].imshow(np.expand_dims(y_batch[i][5], axis=0).sum(axis=(0, 2)).T)\n    axes[5][2].imshow((np.expand_dims(pred[i][5], axis=0)).sum(axis=(0, 2)).T)\n    axes[5][3].imshow((np.expand_dims(pred[i][5], axis=0) > 0.5 ).sum(axis=(0, 2)).T)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"conclusion\"></a>\n# Conclusion\n- This is the first time I use 3D model to solve a real-world problem so I can make some mistakes when modifying the original kernel; if you guys notice some, please let me know.\n- If you find this kernel useful to you, please drop an *upvote* :) , and do not forget to upvote the original [kernel](https://www.kaggle.com/fartuk1/3d-segmentation-approach), too.\n\n[Back to Table of Contents](#toc)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}