{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this kernel\n\nThis kernel takes a look at the dataset for the Lyft competition, with some visual exploration (display images, animations, etc.). I also convert some of the JSON files into CSV, so feel free to use this kernel output as supplementary data.\n\n## Updates\n\nV3: Added Animations! Please go to the \"Animating the images\" section at the end!\n\n## References\n\n* Starter Devkit Lyft3D: https://www.kaggle.com/jesucristo/starter-devkit-lyft3d\n* Devkit for the public 2019 Lyft Level 5 AV Dataset: https://github.com/lyft/nuscenes-devkit"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Thanks for [Nanashi's notebook](https://www.kaggle.com/jesucristo/starter-devkit-lyft3d) for going over the setup for the sdk!"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install lyft-dataset-sdk --quiet\n# Load the dataset\n# Adjust the dataroot parameter below to point to your local dataset path.\n# The correct dataset path contains at least the following four folders (or similar): images, lidar, maps, v1.0.1-train\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport json\nfrom pprint import pprint\n\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation, rc\nfrom IPython.display import HTML\n\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_PATH = '/kaggle/input/3d-object-detection-for-autonomous-vehicles/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to Nanashi!\nlyft_data = LyftDataset(\n    data_path='.',\n    json_path='/kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data', \n    verbose=True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preliminary Exploration\n\n## Load CSV files"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/3d-object-detection-for-autonomous-vehicles/train.csv')\nsub = pd.read_csv('/kaggle/input/3d-object-detection-for-autonomous-vehicles/sample_submission.csv')\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Taking a look at train_images"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(BASE_PATH + \"/train_images\")[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring JSON files"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(BASE_PATH + \"/train_data\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### sample_data.json\nLet's load `sample_data.json`, since it contains information about our training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(BASE_PATH + '/train_data/sample_data.json') as f:\n    data_json = json.load(f)\n\nprint(\"There are\", len(data_json), \"records in sample_data.json\")\n\nprint(\"\\nBelow is a record containing lidar data:\")\npprint(data_json[0])\n\nprint('\\n This one contains information about image data:')\npprint(data_json[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Exploring scene.json"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(BASE_PATH + '/train_data/scene.json') as f:\n    scene_json = json.load(f)\n\nprint(\"There are\", len(scene_json), \"records in sample_data.json\")\n\npprint(scene_json[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating Useful Files\n\n## Converting sample_data.json to dataframes"},{"metadata":{},"cell_type":"markdown","source":"We can safely store the two types into separate dataframes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"lidar_data = []\nimage_data = []\n\nfor record in data_json:\n    if record['fileformat'] == 'jpeg':\n        image_data.append(record)\n    else:\n        lidar_data.append(record)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lidar_df = pd.DataFrame(lidar_data)\nimage_df = pd.DataFrame(image_data)\n\nprint(lidar_df.shape)\nprint(image_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at what they look like:"},{"metadata":{"trusted":true},"cell_type":"code","source":"lidar_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augment image_df\nFinally, we will augment the image_df with some information about its host and camera:"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_df['host'] = image_df['filename'].apply(lambda st: st.strip('images/host-').split('_')[0])\nimage_df['cam'] = image_df['filename'].apply(lambda st: st.split('_')[1])\nimage_df['timestamp'] = image_df['filename'].apply(lambda st: st.split('_')[2].strip('.jpeg'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's store the dataframes we just created:"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"image_df.to_csv(\"sample_data_images.csv\")\nlidar_df.to_csv(\"lidar_data_images.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploring images"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_df['host'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_df['cam'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like there is only 12 hosts, of which there are up to 6 cameras. We also notice that each of the cameras are getting an equal amount of pictures (therefore there are little to no defects, or pictures being \"thrown away\"). We also notice that each host contains a different number of pictures, ranging from 882 to 44k. This could be caused by some of the cars being used on the road for longer than others."},{"metadata":{},"cell_type":"markdown","source":"## Displaying sample image by host\n\nIn the sample below, we display all the images in a given timeframe (10 pictures taken from the first timestamp), in the order they were taken. Each column are the images given by each camera. We do so for a few hosts.\n\nUnhide the cell below to see the definition of `display_host_sample(host, n_images, jumps=1)`."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def display_host_sample(host, n_images, jumps=1):\n    cams = list(sorted(image_df['cam'].unique()))\n    \n    fig, axs = plt.subplots(\n        n_images, len(cams), figsize=(3*len(cams), 3*n_images), \n        sharex=True, sharey=True, gridspec_kw = {'wspace':0.1, 'hspace':0.1}\n    )\n    \n    for i in range(n_images):\n        for c, cam in enumerate(cams):\n            if i == 0:\n                axs[i, c].set_title(cam)\n            \n            mask1 = image_df.cam == cam\n            mask2 = image_df.host == host\n            image_path = image_df[mask1 & mask2]\n            image_path = image_path.sort_values('timestamp')['filename'].iloc[i*jumps]\n            \n            img = cv2.imread(BASE_PATH + '/train_' + image_path)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n            img = cv2.resize(img, (200, 200))\n            \n            axs[i, c].imshow(img)\n            axs[i, c].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Host 017, no frame skip"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_host_sample('017', 5, jumps=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Host 009, skip 5 frames"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_host_sample('009', 5, jumps=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Host 012, skips 10 frames"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_host_sample('012', 5, jumps=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Animating the images\n\nNote: Please use the \"-\" button to slow down the animation speed.\n\nUnhide below for the definition of `animate_images(scene, frames, pointsensor_channel='LIDAR_TOP', interval=1)`"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def generate_next_token(scene):\n    scene = lyft_data.scene[scene]\n    sample_token = scene['first_sample_token']\n    sample_record = lyft_data.get(\"sample\", sample_token)\n    \n    while sample_record['next']:\n        sample_token = sample_record['next']\n        sample_record = lyft_data.get(\"sample\", sample_token)\n        \n        yield sample_token\n\ndef animate_images(scene, frames, pointsensor_channel='LIDAR_TOP', interval=1):\n    cams = [\n        'CAM_FRONT',\n        'CAM_FRONT_RIGHT',\n        'CAM_BACK_RIGHT',\n        'CAM_BACK',\n        'CAM_BACK_LEFT',\n        'CAM_FRONT_LEFT',\n    ]\n\n    generator = generate_next_token(scene)\n\n    fig, axs = plt.subplots(\n        2, len(cams), figsize=(3*len(cams), 6), \n        sharex=True, sharey=True, gridspec_kw = {'wspace': 0, 'hspace': 0.1}\n    )\n    \n    plt.close(fig)\n\n    def animate_fn(i):\n        for _ in range(interval):\n            sample_token = next(generator)\n            \n        for c, camera_channel in enumerate(cams):    \n            sample_record = lyft_data.get(\"sample\", sample_token)\n\n            pointsensor_token = sample_record[\"data\"][pointsensor_channel]\n            camera_token = sample_record[\"data\"][camera_channel]\n            \n            axs[0, c].clear()\n            axs[1, c].clear()\n            \n            lyft_data.render_sample_data(camera_token, with_anns=False, ax=axs[0, c])\n            lyft_data.render_sample_data(camera_token, with_anns=True, ax=axs[1, c])\n            \n            axs[0, c].set_title(\"\")\n            axs[1, c].set_title(\"\")\n\n    anim = animation.FuncAnimation(fig, animate_fn, frames=frames, interval=interval)\n    \n    return anim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nanim = animate_images(scene=0, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"anim = animate_images(scene=10, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"anim = animate_images(scene=50, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"anim = animate_images(scene=100, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Animating LIDAR"},{"metadata":{},"cell_type":"markdown","source":"Unhide below to see the definition of `animate_lidar(scene, frames, pointsensor_channel='LIDAR_TOP', with_anns=True, interval=1)`."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def animate_lidar(scene, frames, pointsensor_channel='LIDAR_TOP', with_anns=True, interval=1):\n    generator = generate_next_token(scene)\n\n    fig, axs = plt.subplots(1, 1, figsize=(8, 8))\n    plt.close(fig)\n\n    def animate_fn(i):\n        for _ in range(interval):\n            sample_token = next(generator)\n        \n        axs.clear()\n        sample_record = lyft_data.get(\"sample\", sample_token)\n        pointsensor_token = sample_record[\"data\"][pointsensor_channel]\n        lyft_data.render_sample_data(pointsensor_token, with_anns=with_anns, ax=axs)\n\n    anim = animation.FuncAnimation(fig, animate_fn, frames=frames, interval=interval)\n    \n    return anim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nanim = animate_lidar(scene=0, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nanim = animate_lidar(scene=10, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"anim = animate_lidar(scene=50, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nanim = animate_lidar(scene=100, frames=100, interval=1)\nHTML(anim.to_jshtml(fps=8))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}