{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Clustering approach to generate proposals for LIDAR Object Detection\n\nThis notebook is an attempt to cluster the LIDAR pointclouds into object proposals (which can then be run through a classification model for identification).\n\n\n- Uses scikit-learn DBSCAN.\n- Homespun grid search and visual checking to identify epsilon and min samples hyper0parameters (optimum is 0.3 and 4 respectively)\n\n**Steps**\n1. Read LIDAR point clouds for sample.\n\n2. Project them into world space and merge them.\n\n3. Eliminate any points higher than Z meters from the local ground (as we are not interested in trees, buildings etc.).\n\n4. Eliminate any points lower than 1 meter from the local ground (ground LIDAR points really mess up the clustering).\n\n5. Cluster using DBSCAN (see note on hyperparameters).\n\n6. Eliminate noise points.\n\n7. Calculate Yaw: Find the XY regression line through the point cluster. Angle of regression line with y-axis is assumed to be the yaw.\n\n8. Rotate transform the point cluster using yaw angle to compute min/max values for x, y and z  (in the presumed object's coordinates). Use these to get the bounding box in object's coordindates. Reverse rotate to get bounding box in world coordinates.\n\n9. Eliminate all boxes that are wider than 5m, longer than 20m or taller than 5m. (Note: The tallest bounding box in the training data set is 9 m. It still did not make sense to consider anything taller than 5m for vehicles).\n\n**Why DBSCAN**\n- Seems to be the only unsupervised clustering approach that does not require pre-specifying the number of clusters.\n\n**Hyperparameters**\n- There seems to be a trade-off on the epsilon hyperparameter - high epsilon performs better on far away objects, whereas low epsilon performs better for nearby objects. No epsilon value seems to pickup more than 65% of the objects.\n\n- While a rudimentary grid search on the sample data suggests 1.75 as the best value for accuracy, 0.3 performs better on nearby objects.  \n\n- According to this [article][https://towardsdatascience.com/how-dbscan-works-and-why-should-i-use-it-443b4a191c80], min samples should be >= number of dimensions + 1. Based on grid search, 4 seems to provide the best IOU for any given epsilon value.\n\n**Limitations and improvements**\n- See note on epsilon above. On the whole, DBSCAN is not at all suitable for proposal generation. However a modified approach could work.\n\n- Could not get DBSCAN to work on GPU. Not sure why Tensorflow / Keras do not support DBSCAN. Had to use CPU for the clustering step and it was painfully slow.\n\n- Chopping off points >  Z meters from local ground - is based on a risky assumption of ground z (here we assume it is vehicle ego pose ground).  Better would be to somehow calculate ground Z local to each object. Even better would be to not use ground z at all and deal with the noise in some other way. \n\n- Bounding box calculation is crude. Better approach would have been to use [Point Cloud Library's moment of intertia](http://pointclouds.org/documentation/tutorials/moment_of_inertia.php) method. However could not install PCL and py-pcl due to compatibility issues with kaggle kernel.\n\n- Bounding box volume is restricted to the number of visible points. This limits IOU to low values (average across sample < 20%, peak < 75%). Need some other way to project visible BB to whole object BB (either by enhancing classification model or augmentating the proposal data).\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Operating system\nimport sys\nimport os\nfrom pathlib import Path\nos.environ[\"OMP_NUM_THREADS\"] = \"1\"\n\n# math\nimport numpy as np\nfrom numpy import arange\nimport math\nfrom numpy import linalg as LA\nfrom scipy import stats\n#progress bar\nfrom tqdm import tqdm, tqdm_notebook\ntqdm.pandas()\nfrom datetime import timezone, datetime, timedelta\n\n# data analysis\nimport pandas as pd\n\n#plotting\nimport matplotlib.pyplot as plt\nfrom matplotlib.axes import Axes\nfrom matplotlib import animation, rc\n\n#plotting 3d\nimport plotly.graph_objects as go\n\n\n#machine learning\nimport sklearn\nimport h5py\nimport sklearn.metrics\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.cluster import DBSCAN\nfrom sklearn import metrics\nfrom sklearn.datasets.samples_generator import make_blobs\nfrom sklearn.preprocessing import StandardScaler\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Lyft dataset SDK\n!pip install lyft-dataset-sdk\nfrom lyft_dataset_sdk.utils.map_mask import MapMask\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset\nfrom lyft_dataset_sdk.utils.geometry_utils import view_points, box_in_image, BoxVisibility\nfrom lyft_dataset_sdk.utils.geometry_utils import view_points, transform_matrix\nfrom lyft_dataset_sdk.utils.data_classes import LidarPointCloud, Box, Quaternion\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = './'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DEBUG = True\ndef log(message):\n    if(DEBUG == True):\n        time_string = datetime.now().strftime('%Y-%m-%d-%H-%M-%S.%f')\n        print(time_string + ' : ', message )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.system('rm -f data && ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data data')\nos.system('rm  -f images && ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images')\nos.system('rm  -f maps && ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps')\nos.system('rm  -f lidar && ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"LYFT = LyftDataset(data_path=DATA_PATH, json_path=DATA_PATH + 'data', verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Examine object sizes\nwlhs = np.array([ann['size'] for ann in  LYFT.sample_annotation])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# max, min  height\nnp.max(wlhs[:,2]), np.min(wlhs[:,2])\n# (8.862, 0.333)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# max, min  length\nnp.max(wlhs[:,1]), np.min(wlhs[:,1])\n#  (22.802, 0.261)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# max, min  width\nnp.max(wlhs[:,0]), np.min(wlhs[:,0])\n# (4.157, 0.223)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# To unzip lidar files when needed (not needed for Kaggle execution)\ndef unzip(row, mode='train'):\n    zip_command = \"unzip ../3d-object-detection-for-autonomous-vehicles.zip \" + mode + '_' + row['filename'].astype(str)           + \" -d \"            + CWD + \"/../data/\"        \n    os.system(zip_command)    \n\ndef removefile(row, mode='train'):\n    rm_command = \"rm -f  \" + CWD + \"/../data/\"+ mode + '_' + row['filename'].astype(str)                  \n    os.system(zip_command)    \n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sort_points_by_coord(points, coord):\n    indices = np.argsort(points[:,coord],axis=0).reshape(-1,1)\n    indices = np.repeat(indices, points.shape[-1],axis=-1)\n    # print(indices)\n    sorted = np.take_along_axis(points,indices,axis=0)\n    del indices\n    return sorted\n\ndef sort_points(points):\n    sorted_points = sort_points_by_coord(points, 2)\n    sorted_points = sort_points_by_coord(sorted_points, 1)\n    return sort_points_by_coord(sorted_points, 0)\n\n    \n# https://www.geeksforgeeks.org/linear-regression-python-implementation/\ndef estimate_regression_coef(points): \n\n    x = points[:,0]\n    y = points[:,1]\n    # number of observations/points \n    n = np.size(x) \n  \n    # mean of x and y vector \n    m_x, m_y = np.mean(x), np.mean(y) \n  \n    # calculating cross-deviation and deviation about x \n    SS_xy = np.sum(y*x) - n*m_y*m_x \n    SS_xx = np.sum(x*x) - n*m_x*m_x \n  \n    # calculating regression coefficients \n    b_1 = SS_xy / SS_xx \n    b_0 = m_y - b_1*m_x \n  \n    return b_1 \n\ndef length_of_xy_diagonal(points):\n    x = points[:,0]\n    y = points[:,1]\n    return LA.norm([np.max(x)- np.min(x), np.max(y)- np.min(y)])\n\ndef length_of_xz_diagonal(points):\n    x = points[:,0]\n    z = points[:,2]\n    return LA.norm([np.max(x)- np.min(x), np.max(z)- np.min(z)])\n\n\ndef slope(points):\n    x = points[:,0]\n    y = points[:,1]\n    m, _, _, _, _ = stats.linregress(x, y)\n    return m\n\ndef yaw(points):\n#     b1 = estimate_regression_coef (points)\n    b1 = slope(points)\n    if b1 == 0:\n        return math.pi / 2\n    else:\n        return np.arctan(1/b1)\n\ndef rotation_matrix_xy(theta):\n     return np.array([ \n        [math.cos(theta), -math.sin(theta), 0],\n        [math.sin(theta), math.cos(theta), 0],\n        [0, 0, 1],\n    ])\n\n# Rotate point cluster using yaw -> calclulate min, max and construct boxs -> inverse rotate to world coords\ndef get_min_bbox_corners(candidate, yw):\n    xyz = np.delete(candidate, np.s_[3], axis=1) \n    world_to_candidate_rotation_matrix = rotation_matrix_xy(-yw)\n    rotated = np.matmul(xyz, world_to_candidate_rotation_matrix)\n    minx = np.min(rotated[:,0])\n    maxx = np.max(rotated[:,0])\n    miny = np.min(rotated[:,1])\n    maxy = np.max(rotated[:,1])\n    minz = np.min(rotated[:,2])\n    maxz = np.max(rotated[:,2])\n    corners_rotated = np.array([\n        [minx, miny, minz],\n        [maxx, miny, minz],\n        [maxx, miny, maxz],\n        [minx, miny, maxz],               \n        [minx, maxy, minz],\n        [maxx, maxy, minz],\n        [maxx, maxy, maxz],\n        [minx, maxy, maxz],               \n    ])\n    del xyz\n    del rotated\n    corners = np.matmul(corners_rotated, world_to_candidate_rotation_matrix.T)\n    return (corners_rotated, corners)\n\ndef get_centroid(corners):\n    minx = np.min(corners[:,0])\n    maxx = np.max(corners[:,0])\n    miny = np.min(corners[:,1])\n    maxy = np.max(corners[:,1])\n    minz = np.min(corners[:,2])\n    maxz = np.max(corners[:,2])\n    x = (minx + maxx) / 2\n    y = (miny + maxy) / 2\n    z = (minz + maxz) / 2\n    return x, y, z\n\ndef get_dimensions(corners):\n    minx = np.min(corners[:,0])\n    maxx = np.max(corners[:,0])\n    miny = np.min(corners[:,1])\n    maxy = np.max(corners[:,1])\n    minz = np.min(corners[:,2])\n    maxz = np.max(corners[:,2])\n    w = (maxx - minx) \n    l = (maxy - miny) \n    h = (maxz - minz) \n    return w, l, h\n\n\n\n# https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/\n\ndef minmax(box):\n    minx = np.min(box[:,0])\n    maxx = np.max(box[:,0])\n    miny = np.min(box[:,1])\n    maxy = np.max(box[:,1])\n    minz = np.min(box[:,2])\n    maxz = np.max(box[:,2])    \n    return minx, maxx, miny , maxy, minz, maxz\n    \ndef intersection(boxA, boxB):\n    # determine the (x, y)-coordinates of the intersection rectangle\n    minxa, maxxa, minya, maxya, minza, maxza = minmax(boxA)\n    minxb, maxxb, minyb, maxyb, minzb, maxzb = minmax(boxB)\n\n    \n    xA = max(minxa, minxb)\n    yA = max(minya, minyb)\n    zA = max(minza, minzb)\n    xB = min(maxxa, maxxb)\n    yB = min(maxya, maxyb)\n    zB = min(maxza, maxzb)\n\n    # compute the volume of intersection cuboid\n    intersectionVolume = max(0, xB - xA ) * max(0, yB - yA )  * max(0, zB - zA) \n    return intersectionVolume\n\ndef union(boxA, boxB):\n    aw, al, ah = get_dimensions(boxA)\n    bw, bl, bh = get_dimensions(boxB)\n\n\n    totalVolume = aw * al * ah + bw * bl * bh - intersection(boxA, boxB)\n    return totalVolume\n\ndef iou(boxA, boxB):\n    return intersection(boxA,boxB) / union(boxA, boxB)\n\ndef make_a_box(minx, maxx, miny, maxy, minz,maxz):\n    return np.array([\n        [minx, miny, minz],\n        [maxx, miny, minz],\n        [maxx, miny, maxz],\n        [minx, miny, maxz],               \n        [minx, maxy, minz],\n        [maxx, maxy, minz],\n        [maxx, maxy, maxz],\n        [minx, maxy, maxz],               \n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Unit test IOU\n\nboxA = make_a_box(2,3,2,3,2,3)\nboxB = make_a_box(1,5,1,5,1,5)\n\n\nboxB = np.array([[2174.97305545, 988.72661905,  -17.47796403],\n [2173.87907608,  986.95519783 , -17.47796403],\n [2173.87907608,  986.95519783 , -19.45096403],\n [2174.97305545,  988.72661905 , -19.45096403],\n [2170.5096185 ,  991.48311077 , -17.47796403],\n [2169.41563913,  989.71168955 , -17.47796403],\n [2169.41563913 , 989.71168955 , -19.45096403],\n [2170.5096185 ,  991.48311077,  -19.45096403]] )\nboxA = np.array( [[2173.90940966 , 987.70527066,  -18.97512237],\n [2174.19097651 , 987.47654011,  -18.97512237],\n [2174.19097651 , 987.47654011,  -18.40049185],\n [2173.90940966 , 987.70527066,  -18.40049185],\n [2174.39607111 , 988.30434993,  -18.97512237],\n [2174.67763796 , 988.07561938,  -18.97512237],\n [2174.67763796,  988.07561938,  -18.40049185],\n [2174.39607111,  988.30434993,  -18.40049185]])\n\n#intersection(boxA, boxB)\n#minmax(boxA), minmax(boxB)\niou(boxA, boxB), union(boxA, boxB), intersection(boxA, boxB)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract composite dataframe for list of sample tokens\ndef extract_data_for_clustering(tokens):\n    sampledata_df = pd.DataFrame(LYFT.sample_data)\n    sampledata_df = sampledata_df[sampledata_df['sample_token'].isin(tokens)]\n    sampledata_df = sampledata_df[sampledata_df['fileformat'] == 'bin']\n    sampledata_df.rename(columns={'token':'sampledata_token'}, inplace=True)\n    sampledata_df = sampledata_df[[\n        'sample_token', \n        'sampledata_token',\n        'ego_pose_token', \n        'channel',\n        'calibrated_sensor_token',\n        'fileformat',\n        'filename']]\n\n    ep_df = pd.DataFrame(LYFT.ego_pose)\n    ep_df.rename(columns={'token':'ego_pose_token', 'rotation': 'ep_rotation', 'translation': 'ep_translation'}, inplace=True)\n    ep_df = ep_df[['ego_pose_token',\n                     'ep_rotation',\n                     'ep_translation']]\n    sampledata_df = pd.merge(sampledata_df, ep_df, left_on='ego_pose_token', right_on='ego_pose_token',how='inner')\n\n\n    cs_df = pd.DataFrame(LYFT.calibrated_sensor)\n    cs_df.rename(columns={'token':'calibrated_sensor_token', 'rotation': 'cs_rotation', 'translation': 'cs_translation'}, inplace=True)\n    cs_df = cs_df[['calibrated_sensor_token',\\\n                     'cs_rotation',\\\n                     'cs_translation',\\\n                     'camera_intrinsic'\n                  ]]\n    sampledata_df = pd.merge(sampledata_df, cs_df, left_on='calibrated_sensor_token', right_on='calibrated_sensor_token',how='inner')\n    # sampledata_df['filepath'] = sampledata_df.apply(lambda row: LYFT.get_sample_data_path(row['sampledata_token']), axis=1)\n    sampledata_df['pointcloud'] = sampledata_df.apply(lambda row: LidarPointCloud.from_file(LYFT.get_sample_data_path(row['sampledata_token'])).points, axis=1)\n    sampledata_df = sampledata_df[[\n        'sample_token', \n        'sampledata_token',\n        'ep_rotation',\n        'ep_translation',\n        'channel',\n        'cs_rotation',\n        'cs_translation',\n        # 'filepath',\n        'pointcloud',\n        'fileformat',\n        'filename']]\n    \n\n    return sampledata_df.copy(deep=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def car_to_world_points_notf (points, translation, rotation):\n    rotated = np.dot(Quaternion(rotation).rotation_matrix, points.T)\n    translated = np.add(rotated.T, translation)\n    return translated\n\ndef sensor_to_car_points_notf (points, translation, rotation):\n    rotated = np.dot(Quaternion(rotation).rotation_matrix, points.T)\n    translated = np.add(rotated.T, translation)\n    return translated\n\n# Take all LIDAR points for a sample and merge them in world coordinates\ndef get_lidar_points_for_clustering(sample_token, token_input, data_path):\n    pt_cloud = np.zeros((0,3))\n    ground_z = -20.0\n    for i in range(len(token_input)):\n\n        sampledatarow = token_input.iloc[i]\n        cs_t = sampledatarow['cs_translation']\n        cs_r = sampledatarow['cs_rotation']\n        ep_t = sampledatarow['ep_translation']\n        ep_r = sampledatarow['ep_rotation']\n        ground_z = min(ground_z, ep_t[2])\n        ego_x = ep_t[0]\n        ego_y = ep_t[1]\n\n\n        pointcloud = sampledatarow['pointcloud']\n        # pointcloud = get_lidar_pointcloud_for_clustering(sampledatarow['filepath'])\n        pc_points_t = pointcloud.T\n        pc_points_t = pc_points_t[:,:3]\n        pc_points_t = sensor_to_car_points_notf(pc_points_t, cs_t, cs_r)\n\n        pc_points_t = car_to_world_points_notf(pc_points_t, ep_t, ep_r)\n        print(pc_points_t.shape, ground_z, ep_t[2],np.min(pc_points_t[:,2]), np.max(pc_points_t[:,2]))\n        pc_points_t = pc_points_t[pc_points_t[:,0] < (ego_x+100)]\n        pc_points_t = pc_points_t[pc_points_t[:,0] > (ego_x-100)]\n\n        pc_points_t = pc_points_t[pc_points_t[:,1] < (ego_y+100)]\n        pc_points_t = pc_points_t[pc_points_t[:,1] > (ego_y-100)]\n\n\n        pc_points_t = pc_points_t[pc_points_t[:,2] < (ground_z+5.3)]\n        pc_points_t = pc_points_t[pc_points_t[:,2] > (ground_z+1)]\n        print(pc_points_t.shape,  ground_z,ep_t[2], np.min(pc_points_t[:,2]), np.max(pc_points_t[:,2]))\n\n\n        pt_cloud = np.concatenate([pt_cloud, pc_points_t], axis=0)\n        #del pointcloud\n        #del pc_points_t\n    return ground_z,  pt_cloud      ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Given a sample token, identify point clusters with bounding box dimensions for that sample\n# Uses DBSCAN clustering\ndef identify_clusters(sample_token, token_input, data_path, eps=0.3, min_samples=10):\n    yaw_list = []\n    corners_list = []\n    dimensions_list = []\n    xyzs = np.zeros((0,3))\n    wlhs = np.zeros((0,3))\n    xs = []\n    ys = []\n    zs = []\n    ws = []\n    ls = []\n    hs = []\n    equisized = []\n    has_rows = False\n\n    log('getting lidar points' + sample_token)\n    ground_z, all_points = get_lidar_points_for_clustering(sample_token, token_input, data_path)\n    log('got lidar points' + sample_token)\n\n    if(all_points.shape[0] > 0):\n        clusters = DBSCAN(eps=eps, min_samples=min_samples).fit(all_points)\n        log('clustered' + sample_token)\n        labels = clusters.labels_\n        unique_labels = set(labels)\n        points_with_labels = np.concatenate((all_points, labels.reshape((-1,1))), axis=1)\n        noise_points = labels == -1\n        points_with_labels = points_with_labels[~noise_points]\n        if(points_with_labels.shape[0] > 0):\n            ulabels = [x for x in unique_labels if x != -1]\n            # ulabels = unique_labels[unique_labels != -1]\n            if(len(ulabels) > 0):\n                pt_clouds = []\n                max_points = 0\n\n                for ulabel in list(ulabels):\n                    box_pt_cloud = points_with_labels[points_with_labels[:,3] == ulabel]\n\n                    if(box_pt_cloud.shape[0] == 0):\n                        continue\n#                     if(length_of_xy_diagonal(box_pt_cloud) > 26):\n#                         continue;\n                    if(length_of_xz_diagonal(box_pt_cloud) > 10):\n                        continue;\n\n                    yw = yaw(box_pt_cloud)\n                    corners_rotated, corners = get_min_bbox_corners(box_pt_cloud, yw)\n                    if (corners[0][2] > ground_z + 2):\n                        continue\n                    w, l, h = get_dimensions(corners_rotated)\n                    if (w > 5):\n                        continue\n                    if (l > 25):\n                        continue\n                    if (h > 5):\n                        continue\n                    has_rows = True\n                    yaw_list.append(yw)\n                    corners_list.append(corners)\n                    dimensions_list.append((w,l,h))\n                    box_pt_cloud = np.delete(box_pt_cloud, np.s_[3], axis=1)\n\n                    num_points = box_pt_cloud.shape[0]\n\n                    if num_points > max_points:\n                        max_points = num_points\n                    # box_pt_cloud = box_pt_cloud.astype('f')\n                    pt_clouds.append((box_pt_cloud))\n                if(len(pt_clouds) > 0):\n                    for cloud in pt_clouds:\n                        ones = np.ones((cloud.shape[0])).reshape(-1,1)\n                        cloud = np.concatenate((cloud, ones), axis=1)\n                        zeros = np.zeros((max_points - cloud.shape[0],4))\n                        fullsize = np.c_[cloud.T, zeros.T].T\n                        fullsize = fullsize.astype('f')\n                        equisized.append(fullsize)\n#                     del pt_clouds\n#                     del points_with_labels\n#                     del clusters\n                    xyzs = np.array([ get_centroid(c) for c in corners_list])\n                    wlhs = np.array(dimensions_list)\n    log('proposals ready' + sample_token)\n        \n    proposal_df = pd.DataFrame()\n    proposal_df['yaw'] = yaw_list\n    proposal_df['x'] = xyzs[:,0]\n    proposal_df['y'] = xyzs[:,1]\n    proposal_df['z'] = xyzs[:,2]\n    proposal_df['w'] = wlhs[:,0]\n    proposal_df['l'] = wlhs[:,1]\n    proposal_df['h'] = wlhs[:,2]\n    \n    proposal_df['corners'] = corners_list\n    proposal_df['dimensions'] = dimensions_list\n    proposal_df['candidate'] = equisized\n    if(len(equisized)> 0):\n        proposal_df['token'] = sample_token\n    return all_points, has_rows, proposal_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Render pointcloud, annotations and object proposals in 3D\ndef render_act_vs_pred_boxes_in_world(\n        points,\n        act_boxes,\n        pred_box_corners\n    ):\n\n        c = np.array([255, 158, 0]) / 255.0\n        \n        print(points.shape)        \n        df_tmp = pd.DataFrame(points, columns=[\"x\", \"y\", \"z\"])\n        df_tmp[\"norm\"] = np.sqrt(np.power(df_tmp[[\"x\", \"y\", \"z\"]].values, 2).sum(axis=1))\n        scatter = go.Scatter3d(\n            x=df_tmp[\"x\"],\n            y=df_tmp[\"y\"],\n            z=df_tmp[\"z\"],\n            mode=\"markers\",\n            marker=dict(size=1, color=df_tmp[\"norm\"], opacity=0.8),\n        )\n        \n        \n        x_lines = []\n        y_lines = []\n        z_lines = []\n\n        def f_lines_add_nones():\n            x_lines.append(None)\n            y_lines.append(None)\n            z_lines.append(None)\n\n        ixs_box_0 = [0, 1, 2, 3, 0]\n        ixs_box_1 = [4, 5, 6, 7, 4]\n\n        for box in act_boxes:\n            bpoints = view_points(box.corners(), view=np.eye(3), normalize=False)\n            x_lines.extend(bpoints[0, ixs_box_0])\n            y_lines.extend(bpoints[1, ixs_box_0])\n            z_lines.extend(bpoints[2, ixs_box_0])\n            f_lines_add_nones()\n            x_lines.extend(bpoints[0, ixs_box_1])\n            y_lines.extend(bpoints[1, ixs_box_1])\n            z_lines.extend(bpoints[2, ixs_box_1])\n            f_lines_add_nones()\n            for i in range(4):\n                x_lines.extend(bpoints[0, [ixs_box_0[i], ixs_box_1[i]]])\n                y_lines.extend(bpoints[1, [ixs_box_0[i], ixs_box_1[i]]])\n                z_lines.extend(bpoints[2, [ixs_box_0[i], ixs_box_1[i]]])\n                f_lines_add_nones()\n\n        lines = go.Scatter3d(x=x_lines, y=y_lines, z=z_lines, mode=\"lines\", name=\"lines\")\n        \n        \n        cx_lines = []\n        cy_lines = []\n        cz_lines = []\n\n        def cf_lines_add_nones():\n            cx_lines.append(None)\n            cy_lines.append(None)\n            cz_lines.append(None)\n\n        cixs_box_0 = [0, 1, 2, 3, 0]\n        cixs_box_1 = [4, 5, 6, 7, 4]\n\n\n        for corners in pred_box_corners:\n            cpoints = view_points(corners.T, view=np.eye(3), normalize=False)\n            cx_lines.extend(cpoints[0, cixs_box_0])\n            cy_lines.extend(cpoints[1, cixs_box_0])\n            cz_lines.extend(cpoints[2, cixs_box_0])\n            cf_lines_add_nones()\n            cx_lines.extend(cpoints[0, cixs_box_1])\n            cy_lines.extend(cpoints[1, cixs_box_1])\n            cz_lines.extend(cpoints[2, cixs_box_1])\n            cf_lines_add_nones()\n            for i in range(4):\n                cx_lines.extend(cpoints[0, [cixs_box_0[i], cixs_box_1[i]]])\n                cy_lines.extend(cpoints[1, [cixs_box_0[i], cixs_box_1[i]]])\n                cz_lines.extend(cpoints[2, [cixs_box_0[i], cixs_box_1[i]]])\n                cf_lines_add_nones()\n\n        clines = go.Scatter3d(x=cx_lines, y=cy_lines, z=cz_lines, mode=\"lines\", name=\"clines\")\n\n        \n        fig = go.Figure(data=[scatter,lines, clines])\n        # fig = go.Figure(data=[scatter])\n        fig.update_layout(scene_aspectmode=\"data\")\n        fig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_sample_tokens(log_token):\n    scene_df =  pd.DataFrame(LYFT.scene)\n    scene_df =  scene_df[scene_df['log_token']==log_token]\n    scene_df.rename(columns={'token':'scene_token'}, inplace=True)\n\n    sample_df = pd.DataFrame(LYFT.sample)\n\n    s_df = pd.merge(sample_df, scene_df, left_on='scene_token', right_on='scene_token',how='inner')\n    s_df = s_df.iloc[:10]\n    return s_df['token']\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compute rough accuracy and mean IOU for given hyperparameters\ndef score_clustering(eps, min_samples):\n    log_token= '71dfb15d2f88bf2aab2c5d4800c0d10a76c279b9fda98720781a406cbacc583b'\n    tokens = get_sample_tokens(log_token)\n    tokens = tokens[:2]\n    batch_input = extract_data_for_clustering(tokens)\n    \n    ious = []\n    numcorrect = 0\n    total = 0\n    pred = 0\n    for sample_token in tokens:\n        token_input = batch_input[batch_input['sample_token'] == sample_token]\n        pointcloud, has_rows, proposals_df = identify_clusters(sample_token, token_input, DATA_PATH, eps=eps, min_samples=min_samples)\n        sampledata_token = token_input.iloc[0]['sampledata_token']\n        pred = pred + len(proposals_df)\n        boxes = LYFT.get_boxes(sampledata_token)\n        \n        for i  in range(len(boxes)):\n            total = total+1\n    \n            matches = 0\n            box = boxes[i]\n            act_corners = box.corners().T\n            for j in range(len(proposals_df)):\n                pred_corners = np.array(proposals_df.iloc[j]['corners'])\n                ioveru = iou(act_corners, pred_corners)\n                if ioveru > 0:\n                    matches = matches + 1\n                    ious.append(ioveru)\n            if (matches == 1):\n                numcorrect = numcorrect + 1\n            if (matches == 0):\n                ious.append(0)\n    print(\"eps:%.2f min_samples:%d -> correct:%d actual:%d  pred %d acc:%.4f mean-iou:%.4f max-iou:%.4f\"% (eps, min_samples , numcorrect, total, pred, numcorrect / total, np.mean(ious), np.max(ious))  )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Render points , actual and proposed bounding boxes in 3d \ndef render_clustering(eps, min_samples):\n    log_token= '71dfb15d2f88bf2aab2c5d4800c0d10a76c279b9fda98720781a406cbacc583b'\n    tokens = get_sample_tokens(log_token)\n    tokens = tokens[:2]\n    batch_input = extract_data_for_clustering(tokens)\n    sample_token = tokens[0]\n    token_input = batch_input[batch_input['sample_token'] == sample_token]\n    sampledata_token = token_input.iloc[0]['sampledata_token']\n    pointcloud, has_rows, proposals_df = identify_clusters(sample_token, token_input, DATA_PATH, eps=eps, min_samples=min_samples)\n    render_act_vs_pred_boxes_in_world(pointcloud, LYFT.get_boxes(sampledata_token), proposals_df['corners'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#render_clustering(1,40)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# grid search\n# for eps in np.arange(1.7,1.8,0.5):\n#     for min_samples in np.arange(4, 6, 1):\n#          score_clustering(eps, min_samples)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#score_clustering(1, 40)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}