{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft: Understanding the data and EDA"},{"metadata":{},"cell_type":"markdown","source":"### Credits:\n\n**https://www.kaggle.com/t3nyks/lyft-working-with-map-api** \n**https://www.kaggle.com/jpbremer/lyft-scene-visualisations** \n**https://www.kaggle.com/pestipeti/pytorch-baseline-train**"},{"metadata":{},"cell_type":"markdown","source":"This new Lyft competition is tasking us, the participants, to predict the motion of external cars, cyclists, pedestrians etc. to assist self-driving cars. This is a step ahead from last year's competition, where we were tasked with detecting three-dimensional objects, like stop signs, to teach AVs how to recognize these. "},{"metadata":{},"cell_type":"markdown","source":"**TIP: Use plt.imshow instead of IPython.display**"},{"metadata":{},"cell_type":"markdown","source":"This is apparently the **largest collection of traffic agent motion data.** The files are stored in the .zarr file format with Python, which we can easily load using the Level 5 Kit (l5kit for the pip package). Within our training ZARRs, we have the agents, the masks for agents, frames and scenes (which you might recollect from last year) and traffic light faces.\n\nThe test ZARR however is almost practically the same format, but the only exclusion is that of the data masks. for the agents. "},{"metadata":{},"cell_type":"markdown","source":"# Get started with the data"},{"metadata":{},"cell_type":"markdown","source":"Wait! Before we directly get into the fancy visualization and all it entails, why not watch a short YouTube video to enlighten us on the subject of operating an autonomous vehicle?"},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('
')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seems like this car casually handles all the normal challenges a driver faces, and that too with remarkable accuracy. Over here, we are tasked with somethign to faciliate this sort of thing - **predicting the motion of extraneous vehicles and based on that, predicting the motion path of an AV.** To predict the motion of these extraneous factors, there are many approaches which I shall discuss later, but for now let's dive in.\n\nHere's a brief FAQ section about the dataset and all it entails:\n\n**What is the structure of the dataset?** \nThe dataset is structured as follows:\n```\naerial_map\nscenes\nsemantic_map\n```\n\nwhere each scene contains roughly a minute or so of information about the motion of several extraneous vehicles and the corresponding movement of the AV.\n\nUnder scenes, we have:\n```\nsample.zarr\ntest.zarr\ntrain.zarr\nvalidate.zarr\n```\n\nNow this ZARR format is a little bit interesting, as I am willing to fathom a guess most of the participants have never worked with these. Fear not, for they are very much interoperable with NumPy and the Lyft Level 5 Kit also gives us easy ways to handle the processing of the data. Of course, there also might be a few ways to use a Pandas DataFrame in the process, which leads up a road for LightGBM.\n\nThe train.zarr contains the agents, the mask for the agents, the frames, the scenes and the traffic light faces, which I'll go into more depth later."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"Now, as we can see here, we have a sensor input (in the form of last year's sensory data) and they had to detect traffic agents in last year's competition. Here, we follow the next two steps of the process: predicting the agent motion and mapping out a path for the autonomous vehicle. The sensor is a LIDAR sensor, which basically gives us a rough perspective of the motion on the road. The sensor then feeds the data back to Lyft, who then collects the data from multiple sensors/cars all over Palo Alto, collates the data and gives it to us."},{"metadata":{},"cell_type":"markdown","source":"We may now import the lyft level 5 kit, and all that comes with it. We have quite a lot of imports and installations required..."},{"metadata":{},"cell_type":"markdown","source":"**UPDATE: Finally got GPU to work by manually installing everything, adding utility scripts did not help at all. Took a painfully long time to get myself to realize that everything needs to be done in the kernel or things will break.**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install neptune-client segmentation_models_pytorch alchemy -q\n!pip install --target=/kaggle/working pymap3d==2.1.0 -q\n!pip install --target=/kaggle/working protobuf==3.12.2 -q\n!pip install --target=/kaggle/working transforms3d -q\n!pip install --target=/kaggle/working zarr -q\n!pip install --target=/kaggle/working ptable -q\n!pip install --no-dependencies --target=/kaggle/working l5kit -q\n!cp ../input/lyft-config-files/agent_motion_config.yaml config.yaml\nimport l5kit, os\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom l5kit.evaluation.metrics import neg_multi_log_likelihood\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\nimport matplotlib.pyplot as plt\n# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"../input/lyft-motion-prediction-autonomous-vehicles\"\n# get config\nMONITORING = True # set this to false if you want to fork and train\nif MONITORING:\n import utilsforlyft as U\ncfg = load_config_data(\"../input/lyft-config-files/visualisation_config.yaml\")\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport torch\nimport torch.nn.functional as F\nimport torch.nn as nn\nfrom catalyst import dl, data\nfrom catalyst.utils import metrics\nfrom torch.utils.data import DataLoader\nfrom catalyst.dl import utils, BatchOverfitCallback\nfrom torch.optim.lr_scheduler import OneCycleLR\nfrom catalyst.contrib.nn.criterion.dice import DiceLoss\nfrom catalyst.dl.callbacks.metrics.accuracy import AccuracyCallback\nimport segmentation_models_pytorch as smp\nfrom catalyst.core.callbacks.early_stop import EarlyStoppingCallback\nfrom catalyst.contrib.dl.callbacks import WandbLogger\nif MONITORING:\n from catalyst.contrib.dl.callbacks.neptune_logger import NeptuneLogger\n neptune_logger = NeptuneLogger(\n api_token=U.TOKEN + '=', \n project_name=\"trigram19/\"+U.NAME_PROJ,\n offline_mode=False, \n name=U.NAME,\n params={'epoch_nr': 5}, \n properties={'data_source': 'lyft'}, \n tags=['resnet']\n )\n from torch.utils.tensorboard import SummaryWriter\n\n import multiprocessing\n pool = multiprocessing.Pool(processes = 10)\n\nfrom IPython.display import display, clear_output\nfrom IPython.display import HTML\nimport PIL\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation as ani, rc\nimport numpy as np\nfrom segmentation_models_pytorch.encoders import get_preprocessing_fn\n!rm -rf ./logs/ \n!mkdir ./logs/\n# Download Ngrok to tunnel the tensorboard port to an external port\n!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n!unzip ngrok-stable-linux-amd64.zip\npool = multiprocessing.Pool(processes = 10)\nresults_of_processes = [\n pool.apply_async(os.system, args=(cmd, ), callback=None) for cmd in [\n f\"tensorboard --logdir ./logs/ --host 0.0.0.0 --port 6006 &\",\n \"./ngrok http 6006 &\"\n ]\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we go using some helpful functions to visualize the data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_image(map_type):\n cfg[\"raster_params\"][\"map_type\"] = map_type\n rast = build_rasterizer(cfg, dm)\n dataset = EgoDataset(cfg, zarr_dataset, rast)\n scene_idx = 2\n indexes = dataset.get_scene_indices(scene_idx)\n images = []\n\n for idx in indexes:\n \n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n display(PIL.Image.fromarray(im[::-1]))\n \ndef plot_from_agent(map_type):\n cfg[\"raster_params\"][\"map_type\"] = map_type\n rast = build_rasterizer(cfg, dm)\n dataset = AgentDataset(cfg, zarr_dataset, rast)\n scene_idx = 2\n indexes = dataset.get_scene_indices(scene_idx)\n images = []\n\n for idx in indexes:\n \n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n display(PIL.Image.fromarray(im[::-1]))\n \ndef animate_solution(images):\n\n def animate(i):\n im.set_data(images[i])\n \n fig, ax = plt.subplots()\n im = ax.imshow(images[0])\n \n return ani.FuncAnimation(fig, animate, frames=len(images), interval=60)\n\ndef animation(type_):\n cfg[\"raster_params\"][\"map_type\"] = type_\n rast = build_rasterizer(cfg, dm)\n dataset = EgoDataset(cfg, zarr_dataset, rast)\n scene_idx = 34\n indexes = dataset.get_scene_indices(scene_idx)\n images = []\n\n for idx in indexes:\n \n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n images.append(PIL.Image.fromarray(im[::-1]))\n anim = animate_solution(images)\n HTML(anim.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's get a sense of the configuration data. This will include metadata pertaining to the agents, the total time, the frames-per-scene, the scene time and the frame frequency."},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\ndm = LocalDataManager()\ndataset_path = dm.require(cfg[\"val_data_loader\"][\"key\"]);rasterizer = build_rasterizer(cfg, dm)\nzarr_dataset = ChunkedDataset(dataset_path)\ntrain_dataset_a = AgentDataset(cfg, zarr_dataset, rasterizer)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, however it's time for us to look at the scenes and analyze them in depth. Theoretically, we could create a nifty little data-loader to do some heavy lifting for us."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_image(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, there's a lot of information in this one image. I'll try my best to point everything out, but do notify me if I make any errors. OK, let's get started with dissecting the image:\n+ We have an intersection of four roads over here.\n+ The green blob represents the AV's motion, and we would require to predict the movement of the AV in these traffic conditions as a sample."},{"metadata":{},"cell_type":"markdown","source":"I don't exactly know what other inferences we can make without more detail on this data, so let's try a satellite-format viewing of these images. "},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image(\"py_satellite\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes! This allows for far more detail than a simple plot without detail. I'd haphazard an educated guess, and make the following inferences:\n+ Green still represents the autonomous vehicle (AV), and blue is primarily all the other cars/vehicles/exogenous factors we need to predict for.\n+ My hypothesis is that the blue represents the path the vehicle needs to go through.\n+ If we are able to accurately predict the path the vehicles go through, it will make it easier for an AV to compute its trajectory on the fly."},{"metadata":{},"cell_type":"markdown","source":"We also want to see how the whole charade of vehicles "},{"metadata":{"trusted":true},"cell_type":"code","source":"animation(\"py_satellite\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So this is a demonstration of the movement of the other vehicles and (in relation to the movement and placement of the other vehicles) the movement of the AV. The AV is currently taking only a straight path in its motion, and a straight path seems logical with the movement and placement of other vehicles."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"animation(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We're also able to take a more low-level move by using the semantic option in the Lyft level 5 kit."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The semantic view is good for a less clustered view but if we want a more detailed, more high-level overview of the data we should perhaps try to use the satellite view voer semantic."},{"metadata":{},"cell_type":"markdown","source":"Now, how about from the agent perspective? This would be quite interesting to consider, as we're modeling from principally the agent perspective in most public notebooks so far."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_from_agent(\"py_satellite\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So yes, I probably should save these as a GIF to visualize the agent movements. Let's try a simpler form of this and use the semantic view for the agent dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_from_agent(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also take a general view of the street from a matplotlib-type perspective. I borrow this from [this wonderful notebook](https://www.kaggle.com/t3nyks/lyft-working-with-map-api)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from l5kit.data.map_api import MapAPI\nfrom l5kit.rasterization.rasterizer_builder import _load_metadata\n\nsemantic_map_filepath = dm.require(cfg[\"raster_params\"][\"semantic_map_key\"])\ndataset_meta = _load_metadata(cfg[\"raster_params\"][\"dataset_meta_key\"], dm)\nworld_to_ecef = np.array(dataset_meta[\"world_to_ecef\"], dtype=np.float64)\n\nmap_api = MapAPI(semantic_map_filepath, world_to_ecef)\nMAP_LAYERS = [\"junction\", \"node\", \"segment\", \"lane\"]\n\n\ndef element_of_type(elem, layer_name):\n return elem.element.HasField(layer_name)\n\n\ndef get_elements_from_layer(map_api, layer_name):\n return [elem for elem in map_api.elements if element_of_type(elem, layer_name)]\n\n\nclass MapRenderer:\n \n def __init__(self, map_api):\n self._color_map = dict(drivable_area='#a6cee3',\n road_segment='#1f78b4',\n road_block='#b2df8a',\n lane='#474747')\n self._map_api = map_api\n \n def render_layer(self, layer_name):\n fig = plt.figure(figsize=(10, 10))\n ax = fig.add_axes([0, 0, 1, 1])\n \n def render_lanes(self):\n all_lanes = get_elements_from_layer(self._map_api, \"lane\")\n fig = plt.figure(figsize=(10, 10))\n ax = fig.add_axes([0, 0, 1, 1])\n for lane in all_lanes:\n self.render_lane(ax, lane)\n return fig, ax\n \n def render_lane(self, ax, lane):\n coords = self._map_api.get_lane_coords(MapAPI.id_as_str(lane.id))\n self.render_boundary(ax, coords[\"xyz_left\"])\n self.render_boundary(ax, coords[\"xyz_right\"])\n \n def render_boundary(self, ax, boundary):\n xs = boundary[:, 0]\n ys = boundary[:, 1] \n ax.plot(xs, ys, color=self._color_map[\"lane\"], label=\"lane\")\n \n \nrenderer = MapRenderer(map_api)\nfig, ax = renderer.render_lanes()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Uh it seems the rasterizer renders rather well the satellite and semantic views, and both in conjunction help one to get a good sense of the positioning of each vehicle in relation to the road. You can easily understand the placement and motion of the vehicles and highway layout in satellite by taking a good look at the semantic view too."},{"metadata":{},"cell_type":"markdown","source":"Again, box and stub will also give a good representaton of the data albeit with less low-level detail than the semantic view, seeing as the highways are not into much consideration here. The box view helps to just take a low-level look at the vehicles and their projected path whereas the stub view functions similarly to semantic. We can now proceed to taking a good look at the metadata provided by kkiller and potentially train a good model."},{"metadata":{},"cell_type":"markdown","source":"# Metadata Exploration"},{"metadata":{},"cell_type":"markdown","source":"Now that we can explore the images, we can also get a little down and dirty when it comes to the ZARR files. It's rather simple to use with the Python library for exploring them, especially the fact that it's NumPy interoperable."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"scenes\", zarr_dataset.scenes)\nprint(\"scenes[0]\", zarr_dataset.scenes[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also, a gentle note that we can use the ChunkedDataset to generate CSV files of scenes."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nscenes = zarr_dataset.scenes\nscenes_df = pd.DataFrame(scenes)\nscenes_df.columns = [\"data\"]; features = ['frame_index_interval', 'host', 'start_time', 'end_time']\nfor i, feature in enumerate(features):\n scenes_df[feature] = scenes_df['data'].apply(lambda x: x[i])\nscenes_df.drop(columns=[\"data\"],inplace=True)\nprint(f\"scenes dataset: {scenes_df.shape}\")\nscenes_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But are we sure this is enough? Enough knowledge to satisfy us? No it's not. I will be using Kkiller's dataset for further tabular data exploration from henceforth."},{"metadata":{"trusted":true},"cell_type":"code","source":"agents = pd.read_csv('../input/lyft-motion-prediction-autonomous-vehicles-as-csv/agents_0_10019001_10019001.csv')\nagents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have a literal wealth of information that we can use here to our benefits, including familiar features like:\n1. x, y, and z coords\n2. yaw\n3. probabilites of other extraneous factors."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import seaborn as sns\ncolormap = plt.cm.magma\ncont_feats = [\"centroid_x\", \"centroid_y\", \"extent_x\", \"extent_y\", \"extent_z\", \"yaw\"]\nplt.figure(figsize=(16,12));\nplt.title('Pearson correlation of features', y=1.05, size=15);\nsns.heatmap(agents[cont_feats].corr(),linewidths=0.1,vmax=1.0, square=True, \n cmap=colormap, linecolor='white', annot=True);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we can extrapolate that the variables **centroid_x** and **centroid_y** have strongly negative correlations, and the strongest correlations are between **extent_z** and **extent_x** more than any other, coming in at 0.4. We can also try using an XGBoost/LightGBM model as kkiller has demonstrated in his brilliant kernel as an alternative approach to the problem."},{"metadata":{},"cell_type":"markdown","source":"### centroid_x and centroid_y"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplot = sns.jointplot(x=agents['centroid_x'][:1000], y=agents['centroid_y'][:1000], kind='hexbin', color='blueviolet')\nplot.set_axis_labels('center_x', 'center_y', fontsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like the two centroids have a somewhat strongly negative correlation and seemingly similar variable distributions. It seems that as such there is a negative correlation between both the variables."},{"metadata":{},"cell_type":"markdown","source":"### extent_x, extent_y and extent_z"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 15));\nsns.distplot(agents['extent_x'], color='steelblue');\nsns.distplot(agents['extent_y'], color='purple');\n\nplt.title(\"Distributions of Extents X and Y\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems both the distributions of extent X and extent Y are heavily right skewed, as is centroid X. However, I have left out extent Z is order for readability of the plot, let's look at it now.\n\nTry to smooth the data and get:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 15));\nsns.distplot(agents['extent_z'], color='steelblue');\n\nplt.title(\"Distributions of Extents z\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once again, we have a right-skewed distribution as is the same with all the `extent` variables. "},{"metadata":{},"cell_type":"markdown","source":"### yaw"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 15));\nsns.distplot(agents['yaw'], color='steelblue');\n\nplt.title(\"Distributions of Extents z\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So yes it seems like this distribution has several \"protrusions\" as I shall call them. We can now move on to exploring the frames data to check how feasible it is for our tabular purposes."},{"metadata":{"trusted":true},"cell_type":"code","source":"frms = pd.read_csv(\"../input/lyft-motion-prediction-autonomous-vehicles-as-csv/frames_0_124167_124167.csv\")\nfrms.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So here we have the ego rotations with regards to the centroids, which will be very interesting to consider. It seems like we will require to check multiple of these variables at once:"},{"metadata":{},"cell_type":"markdown","source":"### ego_rotatations"},{"metadata":{},"cell_type":"markdown","source":"First of all, we have nine ego rotation columns corresponding to each. So I would want to do a quick check of the correlation of these variables before moving on to some more high-level analyses."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import seaborn as sns\ncolormap = plt.cm.magma\ncont_feats = [\"ego_rotation_xx\", \"ego_rotation_xy\", \"ego_rotation_xz\", \"ego_rotation_yx\", \"ego_rotation_yy\", \"ego_rotation_yz\", \"ego_rotation_zx\", \"ego_rotation_zy\", \"ego_rotation_zz\"]\nplt.figure(figsize=(16,12));\nplt.title('Pearson correlation of features', y=1.05, size=15);\nsns.heatmap(frms[cont_feats].corr(),linewidths=0.1,vmax=1.0, square=True, \n cmap=colormap, linecolor='white', annot=True);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Things to note from this correlation analysis:\n1. The rotation coordinates with `y` and `z` seem to be uncorrelated most of the time\n2. The coordinates which have `x` are correlated strongly with the z-dimensional rotation (could this be indicative of something? I very much think so)"},{"metadata":{},"cell_type":"markdown","source":"# Baseline model (source: [here](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb) and [here](https://www.kaggle.com/pestipeti/pytorch-baseline-inference))"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"This is mainly me using the Lyft baseline model and training it, for the purpose of demonstrating how we can fit to a PyTorch model with the provided dataset format. Also, I am using the tool neptune.ai for monitoring the epoch progress.\n\nAlso using wonderful ML library catalyst, which helps your PyTorch training greatly. For now, let's get started on the modelling with some basic import setup:"},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg2 = load_config_data(\"../input/lyft-config-files/agent_motion_config.yaml\")\ntrain_cfg = cfg2[\"train_data_loader\"]\nvalidation_cfg = cfg2[\"val_data_loader\"]\n# Rasterizer\nrasterizer = build_rasterizer(cfg2, dm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is basically initializing the rasterization process for the training data which basically functions to pass the .zarr files to our model and make it a coherent data format easily modular with our PyTorch setup."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LyftModel(torch.nn.Module):\n \n def __init__(self, cfg):\n super().__init__()\n \n self.backbone = smp.FPN(encoder_name=\"resnext50_32x4d\", classes=1)\n \n num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n num_in_channels = 3 + num_history_channels\n\n self.backbone.encoder.conv1 = nn.Conv2d(\n num_in_channels,\n self.backbone.encoder.conv1.out_channels,\n kernel_size= self.backbone.encoder.conv1.kernel_size,\n stride= self.backbone.encoder.conv1.stride,\n padding= self.backbone.encoder.conv1.padding,\n bias=False,\n )\n backbone_out_features = 14\n\n # X, Y coords for the future positions (output shape: Bx50x2)\n num_targets = 2 * cfg[\"model_params\"][\"future_num_frames\"]\n\n self.head = nn.Sequential(\n nn.Dropout(0.2),\n nn.Linear(in_features=14, out_features=4096),\n )\n self.backbone.segmentation_head = nn.Sequential(nn.Conv1d(56, 1, kernel_size=3, stride=2), nn.Dropout(0.2), nn.ReLU())\n self.logit = nn.Linear(4096, out_features=num_targets)\n self.logit_final = nn.Linear(128, 12)\n self.num_preds = num_targets * 3\n self.num_modes = 3\n \n def forward(self, x):\n x = self.backbone.encoder.conv1(x)\n x = self.backbone.encoder.bn1(x)\n x = self.backbone.encoder.relu(x)\n x = self.backbone.encoder.maxpool(x)\n\n x = self.backbone.encoder.layer1(x)\n x = self.backbone.encoder.layer2(x)\n x = self.backbone.encoder.layer3(x)\n x = self.backbone.encoder.layer4(x)\n\n x = self.backbone.decoder.p5(x)\n x = self.backbone.decoder.seg_blocks[0](x)\n x = self.backbone.decoder.merge(x)\n x = self.backbone.segmentation_head(x)\n x = self.backbone.encoder.maxpool(x)\n \n x = torch.flatten(x, 1)\n x = self.head(x)\n x = self.logit(x)\n x = x.permute(1, 0)\n x = self.logit_final(x)\n\n\n return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"~~Unhide the above cell if you want to see the model, it's basically Peter's original work. All I am doing here is modifying the training pipeline to be Catalyst-compatible.~~\n\nUpdate 17-09-2020: The model is roughly based on what kkiller has accomplished with PointNet and is basically me using a modified FPN network - to first segment (encode and decode) - and then use a simple MLP to classify. The model is roughly simple but it gives me pretty good results on loss, maybe something worth looking into?"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = LyftModel(cfg2)\nmodel.to(device)\n\n# Train dataset/dataloader\ntrain_zarr = ChunkedDataset(dm.require(train_cfg[\"key\"])).open()\ntrain_dataset = AgentDataset(cfg2, train_zarr, rasterizer)\nsubset = torch.utils.data.Subset(train_dataset, range(0, 400))\ntrain_dataloader = DataLoader(subset,\n shuffle=train_cfg[\"shuffle\"],\n batch_size=train_cfg[\"batch_size\"],\n num_workers=train_cfg[\"num_workers\"])\n\nval_zarr = ChunkedDataset(dm.require(validation_cfg[\"key\"])).open()\nval_dataset = AgentDataset(cfg2, val_zarr, rasterizer)\nsubset = torch.utils.data.Subset(val_dataset, range(0, 50))\nval_dataloader = DataLoader(subset,\n shuffle=validation_cfg[\"shuffle\"],\n batch_size=validation_cfg[\"batch_size\"],\n num_workers=validation_cfg[\"num_workers\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This takes a very small subset of the data that we have here (mainly because this training pipeline is purely for demonstration purposes with regards to Catalyst)."},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = torch.optim.SGD(model.parameters(), lr=0.02, momentum=0.9)\nsched = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.15, patience=2)\n\nloaders = {\n \"train\": train_dataloader,\n \"valid\": val_dataloader\n}\n\nclass LyftRunner(dl.Runner):\n\n def predict_batch(self, batch):\n return self.model(batch[0].to(self.device).view(batch[0].size(0), -1))\n\n def _handle_batch(self, batch):\n x, y = batch['image'], batch['target_positions']\n self.model.logit_final = nn.Linear(128, y.shape[0]).cuda()\n y_hat = self.model(x).view(y.shape)\n target_availabilities = batch[\"target_availabilities\"].unsqueeze(-1)\n criterion = torch.nn.MSELoss(reduction=\"none\")\n \n loss = criterion(y_hat, y)\n loss = loss * target_availabilities\n loss = loss.mean()\n self.batch_metrics.update(\n {\"loss\": loss}\n )\n\n if self.is_train_loader:\n loss.backward()\n self.optimizer.step()\n self.optimizer.zero_grad()\n \n if MONITORING:\n writer.add_scalar('Loss', loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Few things:\n+ Configuration of the data loaders\n+ Beginning the Catalyst process\n+ Neptune, wandb initialization"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%time\ndevice = utils.get_device()\nrunner = LyftRunner(device=device)\nif MONITORING:\n writer = SummaryWriter('./logs')\n\n runner.train(\n model=model,\n optimizer=optimizer,\n loaders=loaders,\n logdir=\"./logs\",\n num_epochs=6,\n verbose=True,\n load_best_on_end=True,\n scheduler=sched,\n callbacks=[neptune_logger, BatchOverfitCallback(train=10, valid=0.5), \n EarlyStoppingCallback(\n patience=5,\n metric=\"loss\",\n minimize=True,\n ), WandbLogger(project=\"dertaismus\",name= 'Example')\n ]\n )\nelse:\n print(\"Woops! Looks like you disabled monitoring. To be able to run the model without monitoring, try running the above catalyst model without the Neptune logger and the Wandb logger.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And this is how the Neptune metrics look like:\n"},{"metadata":{},"cell_type":"markdown","source":"To properly work with Neptune callbacks, replace the `U.` fields in the callback class with your API token, project name etc. Then, add it as a callback to Catalyst and watch the magic happen."},{"metadata":{},"cell_type":"markdown","source":"Source: https://github.com/lyft/l5kit/blob/master/examples/visualisation/visualise_data.ipynb\n\n**WORK IN PROGRESS - MORE TO COME.**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}
![](\"https://self-driving.lyft.com/wp-content/uploads/2020/06/dataset-steps-longimg.png\")
"},{"metadata":{},"cell_type":"markdown","source":"Now, as we can see here, we have a sensor input (in the form of last year's sensory data) and they had to detect traffic agents in last year's competition. Here, we follow the next two steps of the process: predicting the agent motion and mapping out a path for the autonomous vehicle. The sensor is a LIDAR sensor, which basically gives us a rough perspective of the motion on the road. The sensor then feeds the data back to Lyft, who then collects the data from multiple sensors/cars all over Palo Alto, collates the data and gives it to us."},{"metadata":{},"cell_type":"markdown","source":"We may now import the lyft level 5 kit, and all that comes with it. We have quite a lot of imports and installations required..."},{"metadata":{},"cell_type":"markdown","source":"**UPDATE: Finally got GPU to work by manually installing everything, adding utility scripts did not help at all. Took a painfully long time to get myself to realize that everything needs to be done in the kernel or things will break.**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install neptune-client segmentation_models_pytorch alchemy -q\n!pip install --target=/kaggle/working pymap3d==2.1.0 -q\n!pip install --target=/kaggle/working protobuf==3.12.2 -q\n!pip install --target=/kaggle/working transforms3d -q\n!pip install --target=/kaggle/working zarr -q\n!pip install --target=/kaggle/working ptable -q\n!pip install --no-dependencies --target=/kaggle/working l5kit -q\n!cp ../input/lyft-config-files/agent_motion_config.yaml config.yaml\nimport l5kit, os\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom l5kit.evaluation.metrics import neg_multi_log_likelihood\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\nimport matplotlib.pyplot as plt\n# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"../input/lyft-motion-prediction-autonomous-vehicles\"\n# get config\nMONITORING = True # set this to false if you want to fork and train\nif MONITORING:\n import utilsforlyft as U\ncfg = load_config_data(\"../input/lyft-config-files/visualisation_config.yaml\")\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport torch\nimport torch.nn.functional as F\nimport torch.nn as nn\nfrom catalyst import dl, data\nfrom catalyst.utils import metrics\nfrom torch.utils.data import DataLoader\nfrom catalyst.dl import utils, BatchOverfitCallback\nfrom torch.optim.lr_scheduler import OneCycleLR\nfrom catalyst.contrib.nn.criterion.dice import DiceLoss\nfrom catalyst.dl.callbacks.metrics.accuracy import AccuracyCallback\nimport segmentation_models_pytorch as smp\nfrom catalyst.core.callbacks.early_stop import EarlyStoppingCallback\nfrom catalyst.contrib.dl.callbacks import WandbLogger\nif MONITORING:\n from catalyst.contrib.dl.callbacks.neptune_logger import NeptuneLogger\n neptune_logger = NeptuneLogger(\n api_token=U.TOKEN + '=', \n project_name=\"trigram19/\"+U.NAME_PROJ,\n offline_mode=False, \n name=U.NAME,\n params={'epoch_nr': 5}, \n properties={'data_source': 'lyft'}, \n tags=['resnet']\n )\n from torch.utils.tensorboard import SummaryWriter\n\n import multiprocessing\n pool = multiprocessing.Pool(processes = 10)\n\nfrom IPython.display import display, clear_output\nfrom IPython.display import HTML\nimport PIL\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation as ani, rc\nimport numpy as np\nfrom segmentation_models_pytorch.encoders import get_preprocessing_fn\n!rm -rf ./logs/ \n!mkdir ./logs/\n# Download Ngrok to tunnel the tensorboard port to an external port\n!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n!unzip ngrok-stable-linux-amd64.zip\npool = multiprocessing.Pool(processes = 10)\nresults_of_processes = [\n pool.apply_async(os.system, args=(cmd, ), callback=None) for cmd in [\n f\"tensorboard --logdir ./logs/ --host 0.0.0.0 --port 6006 &\",\n \"./ngrok http 6006 &\"\n ]\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we go using some helpful functions to visualize the data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_image(map_type):\n cfg[\"raster_params\"][\"map_type\"] = map_type\n rast = build_rasterizer(cfg, dm)\n dataset = EgoDataset(cfg, zarr_dataset, rast)\n scene_idx = 2\n indexes = dataset.get_scene_indices(scene_idx)\n images = []\n\n for idx in indexes:\n \n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n display(PIL.Image.fromarray(im[::-1]))\n \ndef plot_from_agent(map_type):\n cfg[\"raster_params\"][\"map_type\"] = map_type\n rast = build_rasterizer(cfg, dm)\n dataset = AgentDataset(cfg, zarr_dataset, rast)\n scene_idx = 2\n indexes = dataset.get_scene_indices(scene_idx)\n images = []\n\n for idx in indexes:\n \n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n display(PIL.Image.fromarray(im[::-1]))\n \ndef animate_solution(images):\n\n def animate(i):\n im.set_data(images[i])\n \n fig, ax = plt.subplots()\n im = ax.imshow(images[0])\n \n return ani.FuncAnimation(fig, animate, frames=len(images), interval=60)\n\ndef animation(type_):\n cfg[\"raster_params\"][\"map_type\"] = type_\n rast = build_rasterizer(cfg, dm)\n dataset = EgoDataset(cfg, zarr_dataset, rast)\n scene_idx = 34\n indexes = dataset.get_scene_indices(scene_idx)\n images = []\n\n for idx in indexes:\n \n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n images.append(PIL.Image.fromarray(im[::-1]))\n anim = animate_solution(images)\n HTML(anim.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's get a sense of the configuration data. This will include metadata pertaining to the agents, the total time, the frames-per-scene, the scene time and the frame frequency."},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\ndm = LocalDataManager()\ndataset_path = dm.require(cfg[\"val_data_loader\"][\"key\"]);rasterizer = build_rasterizer(cfg, dm)\nzarr_dataset = ChunkedDataset(dataset_path)\ntrain_dataset_a = AgentDataset(cfg, zarr_dataset, rasterizer)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, however it's time for us to look at the scenes and analyze them in depth. Theoretically, we could create a nifty little data-loader to do some heavy lifting for us."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_image(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, there's a lot of information in this one image. I'll try my best to point everything out, but do notify me if I make any errors. OK, let's get started with dissecting the image:\n+ We have an intersection of four roads over here.\n+ The green blob represents the AV's motion, and we would require to predict the movement of the AV in these traffic conditions as a sample."},{"metadata":{},"cell_type":"markdown","source":"I don't exactly know what other inferences we can make without more detail on this data, so let's try a satellite-format viewing of these images. "},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image(\"py_satellite\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes! This allows for far more detail than a simple plot without detail. I'd haphazard an educated guess, and make the following inferences:\n+ Green still represents the autonomous vehicle (AV), and blue is primarily all the other cars/vehicles/exogenous factors we need to predict for.\n+ My hypothesis is that the blue represents the path the vehicle needs to go through.\n+ If we are able to accurately predict the path the vehicles go through, it will make it easier for an AV to compute its trajectory on the fly."},{"metadata":{},"cell_type":"markdown","source":"We also want to see how the whole charade of vehicles "},{"metadata":{"trusted":true},"cell_type":"code","source":"animation(\"py_satellite\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So this is a demonstration of the movement of the other vehicles and (in relation to the movement and placement of the other vehicles) the movement of the AV. The AV is currently taking only a straight path in its motion, and a straight path seems logical with the movement and placement of other vehicles."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"animation(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We're also able to take a more low-level move by using the semantic option in the Lyft level 5 kit."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The semantic view is good for a less clustered view but if we want a more detailed, more high-level overview of the data we should perhaps try to use the satellite view voer semantic."},{"metadata":{},"cell_type":"markdown","source":"Now, how about from the agent perspective? This would be quite interesting to consider, as we're modeling from principally the agent perspective in most public notebooks so far."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_from_agent(\"py_satellite\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So yes, I probably should save these as a GIF to visualize the agent movements. Let's try a simpler form of this and use the semantic view for the agent dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_from_agent(\"py_semantic\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also take a general view of the street from a matplotlib-type perspective. I borrow this from [this wonderful notebook](https://www.kaggle.com/t3nyks/lyft-working-with-map-api)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from l5kit.data.map_api import MapAPI\nfrom l5kit.rasterization.rasterizer_builder import _load_metadata\n\nsemantic_map_filepath = dm.require(cfg[\"raster_params\"][\"semantic_map_key\"])\ndataset_meta = _load_metadata(cfg[\"raster_params\"][\"dataset_meta_key\"], dm)\nworld_to_ecef = np.array(dataset_meta[\"world_to_ecef\"], dtype=np.float64)\n\nmap_api = MapAPI(semantic_map_filepath, world_to_ecef)\nMAP_LAYERS = [\"junction\", \"node\", \"segment\", \"lane\"]\n\n\ndef element_of_type(elem, layer_name):\n return elem.element.HasField(layer_name)\n\n\ndef get_elements_from_layer(map_api, layer_name):\n return [elem for elem in map_api.elements if element_of_type(elem, layer_name)]\n\n\nclass MapRenderer:\n \n def __init__(self, map_api):\n self._color_map = dict(drivable_area='#a6cee3',\n road_segment='#1f78b4',\n road_block='#b2df8a',\n lane='#474747')\n self._map_api = map_api\n \n def render_layer(self, layer_name):\n fig = plt.figure(figsize=(10, 10))\n ax = fig.add_axes([0, 0, 1, 1])\n \n def render_lanes(self):\n all_lanes = get_elements_from_layer(self._map_api, \"lane\")\n fig = plt.figure(figsize=(10, 10))\n ax = fig.add_axes([0, 0, 1, 1])\n for lane in all_lanes:\n self.render_lane(ax, lane)\n return fig, ax\n \n def render_lane(self, ax, lane):\n coords = self._map_api.get_lane_coords(MapAPI.id_as_str(lane.id))\n self.render_boundary(ax, coords[\"xyz_left\"])\n self.render_boundary(ax, coords[\"xyz_right\"])\n \n def render_boundary(self, ax, boundary):\n xs = boundary[:, 0]\n ys = boundary[:, 1] \n ax.plot(xs, ys, color=self._color_map[\"lane\"], label=\"lane\")\n \n \nrenderer = MapRenderer(map_api)\nfig, ax = renderer.render_lanes()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Uh it seems the rasterizer renders rather well the satellite and semantic views, and both in conjunction help one to get a good sense of the positioning of each vehicle in relation to the road. You can easily understand the placement and motion of the vehicles and highway layout in satellite by taking a good look at the semantic view too."},{"metadata":{},"cell_type":"markdown","source":"Again, box and stub will also give a good representaton of the data albeit with less low-level detail than the semantic view, seeing as the highways are not into much consideration here. The box view helps to just take a low-level look at the vehicles and their projected path whereas the stub view functions similarly to semantic. We can now proceed to taking a good look at the metadata provided by kkiller and potentially train a good model."},{"metadata":{},"cell_type":"markdown","source":"# Metadata Exploration"},{"metadata":{},"cell_type":"markdown","source":"Now that we can explore the images, we can also get a little down and dirty when it comes to the ZARR files. It's rather simple to use with the Python library for exploring them, especially the fact that it's NumPy interoperable."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"scenes\", zarr_dataset.scenes)\nprint(\"scenes[0]\", zarr_dataset.scenes[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also, a gentle note that we can use the ChunkedDataset to generate CSV files of scenes."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nscenes = zarr_dataset.scenes\nscenes_df = pd.DataFrame(scenes)\nscenes_df.columns = [\"data\"]; features = ['frame_index_interval', 'host', 'start_time', 'end_time']\nfor i, feature in enumerate(features):\n scenes_df[feature] = scenes_df['data'].apply(lambda x: x[i])\nscenes_df.drop(columns=[\"data\"],inplace=True)\nprint(f\"scenes dataset: {scenes_df.shape}\")\nscenes_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But are we sure this is enough? Enough knowledge to satisfy us? No it's not. I will be using Kkiller's dataset for further tabular data exploration from henceforth."},{"metadata":{"trusted":true},"cell_type":"code","source":"agents = pd.read_csv('../input/lyft-motion-prediction-autonomous-vehicles-as-csv/agents_0_10019001_10019001.csv')\nagents","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have a literal wealth of information that we can use here to our benefits, including familiar features like:\n1. x, y, and z coords\n2. yaw\n3. probabilites of other extraneous factors."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import seaborn as sns\ncolormap = plt.cm.magma\ncont_feats = [\"centroid_x\", \"centroid_y\", \"extent_x\", \"extent_y\", \"extent_z\", \"yaw\"]\nplt.figure(figsize=(16,12));\nplt.title('Pearson correlation of features', y=1.05, size=15);\nsns.heatmap(agents[cont_feats].corr(),linewidths=0.1,vmax=1.0, square=True, \n cmap=colormap, linecolor='white', annot=True);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we can extrapolate that the variables **centroid_x** and **centroid_y** have strongly negative correlations, and the strongest correlations are between **extent_z** and **extent_x** more than any other, coming in at 0.4. We can also try using an XGBoost/LightGBM model as kkiller has demonstrated in his brilliant kernel as an alternative approach to the problem."},{"metadata":{},"cell_type":"markdown","source":"### centroid_x and centroid_y"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplot = sns.jointplot(x=agents['centroid_x'][:1000], y=agents['centroid_y'][:1000], kind='hexbin', color='blueviolet')\nplot.set_axis_labels('center_x', 'center_y', fontsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like the two centroids have a somewhat strongly negative correlation and seemingly similar variable distributions. It seems that as such there is a negative correlation between both the variables."},{"metadata":{},"cell_type":"markdown","source":"### extent_x, extent_y and extent_z"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 15));\nsns.distplot(agents['extent_x'], color='steelblue');\nsns.distplot(agents['extent_y'], color='purple');\n\nplt.title(\"Distributions of Extents X and Y\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems both the distributions of extent X and extent Y are heavily right skewed, as is centroid X. However, I have left out extent Z is order for readability of the plot, let's look at it now.\n\nTry to smooth the data and get:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 15));\nsns.distplot(agents['extent_z'], color='steelblue');\n\nplt.title(\"Distributions of Extents z\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once again, we have a right-skewed distribution as is the same with all the `extent` variables. "},{"metadata":{},"cell_type":"markdown","source":"### yaw"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15, 15));\nsns.distplot(agents['yaw'], color='steelblue');\n\nplt.title(\"Distributions of Extents z\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So yes it seems like this distribution has several \"protrusions\" as I shall call them. We can now move on to exploring the frames data to check how feasible it is for our tabular purposes."},{"metadata":{"trusted":true},"cell_type":"code","source":"frms = pd.read_csv(\"../input/lyft-motion-prediction-autonomous-vehicles-as-csv/frames_0_124167_124167.csv\")\nfrms.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So here we have the ego rotations with regards to the centroids, which will be very interesting to consider. It seems like we will require to check multiple of these variables at once:"},{"metadata":{},"cell_type":"markdown","source":"### ego_rotatations"},{"metadata":{},"cell_type":"markdown","source":"First of all, we have nine ego rotation columns corresponding to each. So I would want to do a quick check of the correlation of these variables before moving on to some more high-level analyses."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import seaborn as sns\ncolormap = plt.cm.magma\ncont_feats = [\"ego_rotation_xx\", \"ego_rotation_xy\", \"ego_rotation_xz\", \"ego_rotation_yx\", \"ego_rotation_yy\", \"ego_rotation_yz\", \"ego_rotation_zx\", \"ego_rotation_zy\", \"ego_rotation_zz\"]\nplt.figure(figsize=(16,12));\nplt.title('Pearson correlation of features', y=1.05, size=15);\nsns.heatmap(frms[cont_feats].corr(),linewidths=0.1,vmax=1.0, square=True, \n cmap=colormap, linecolor='white', annot=True);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Things to note from this correlation analysis:\n1. The rotation coordinates with `y` and `z` seem to be uncorrelated most of the time\n2. The coordinates which have `x` are correlated strongly with the z-dimensional rotation (could this be indicative of something? I very much think so)"},{"metadata":{},"cell_type":"markdown","source":"# Baseline model (source: [here](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb) and [here](https://www.kaggle.com/pestipeti/pytorch-baseline-inference))"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"This is mainly me using the Lyft baseline model and training it, for the purpose of demonstrating how we can fit to a PyTorch model with the provided dataset format. Also, I am using the tool neptune.ai for monitoring the epoch progress.\n\nAlso using wonderful ML library catalyst, which helps your PyTorch training greatly. For now, let's get started on the modelling with some basic import setup:"},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg2 = load_config_data(\"../input/lyft-config-files/agent_motion_config.yaml\")\ntrain_cfg = cfg2[\"train_data_loader\"]\nvalidation_cfg = cfg2[\"val_data_loader\"]\n# Rasterizer\nrasterizer = build_rasterizer(cfg2, dm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is basically initializing the rasterization process for the training data which basically functions to pass the .zarr files to our model and make it a coherent data format easily modular with our PyTorch setup."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LyftModel(torch.nn.Module):\n \n def __init__(self, cfg):\n super().__init__()\n \n self.backbone = smp.FPN(encoder_name=\"resnext50_32x4d\", classes=1)\n \n num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n num_in_channels = 3 + num_history_channels\n\n self.backbone.encoder.conv1 = nn.Conv2d(\n num_in_channels,\n self.backbone.encoder.conv1.out_channels,\n kernel_size= self.backbone.encoder.conv1.kernel_size,\n stride= self.backbone.encoder.conv1.stride,\n padding= self.backbone.encoder.conv1.padding,\n bias=False,\n )\n backbone_out_features = 14\n\n # X, Y coords for the future positions (output shape: Bx50x2)\n num_targets = 2 * cfg[\"model_params\"][\"future_num_frames\"]\n\n self.head = nn.Sequential(\n nn.Dropout(0.2),\n nn.Linear(in_features=14, out_features=4096),\n )\n self.backbone.segmentation_head = nn.Sequential(nn.Conv1d(56, 1, kernel_size=3, stride=2), nn.Dropout(0.2), nn.ReLU())\n self.logit = nn.Linear(4096, out_features=num_targets)\n self.logit_final = nn.Linear(128, 12)\n self.num_preds = num_targets * 3\n self.num_modes = 3\n \n def forward(self, x):\n x = self.backbone.encoder.conv1(x)\n x = self.backbone.encoder.bn1(x)\n x = self.backbone.encoder.relu(x)\n x = self.backbone.encoder.maxpool(x)\n\n x = self.backbone.encoder.layer1(x)\n x = self.backbone.encoder.layer2(x)\n x = self.backbone.encoder.layer3(x)\n x = self.backbone.encoder.layer4(x)\n\n x = self.backbone.decoder.p5(x)\n x = self.backbone.decoder.seg_blocks[0](x)\n x = self.backbone.decoder.merge(x)\n x = self.backbone.segmentation_head(x)\n x = self.backbone.encoder.maxpool(x)\n \n x = torch.flatten(x, 1)\n x = self.head(x)\n x = self.logit(x)\n x = x.permute(1, 0)\n x = self.logit_final(x)\n\n\n return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"~~Unhide the above cell if you want to see the model, it's basically Peter's original work. All I am doing here is modifying the training pipeline to be Catalyst-compatible.~~\n\nUpdate 17-09-2020: The model is roughly based on what kkiller has accomplished with PointNet and is basically me using a modified FPN network - to first segment (encode and decode) - and then use a simple MLP to classify. The model is roughly simple but it gives me pretty good results on loss, maybe something worth looking into?"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = LyftModel(cfg2)\nmodel.to(device)\n\n# Train dataset/dataloader\ntrain_zarr = ChunkedDataset(dm.require(train_cfg[\"key\"])).open()\ntrain_dataset = AgentDataset(cfg2, train_zarr, rasterizer)\nsubset = torch.utils.data.Subset(train_dataset, range(0, 400))\ntrain_dataloader = DataLoader(subset,\n shuffle=train_cfg[\"shuffle\"],\n batch_size=train_cfg[\"batch_size\"],\n num_workers=train_cfg[\"num_workers\"])\n\nval_zarr = ChunkedDataset(dm.require(validation_cfg[\"key\"])).open()\nval_dataset = AgentDataset(cfg2, val_zarr, rasterizer)\nsubset = torch.utils.data.Subset(val_dataset, range(0, 50))\nval_dataloader = DataLoader(subset,\n shuffle=validation_cfg[\"shuffle\"],\n batch_size=validation_cfg[\"batch_size\"],\n num_workers=validation_cfg[\"num_workers\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This takes a very small subset of the data that we have here (mainly because this training pipeline is purely for demonstration purposes with regards to Catalyst)."},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = torch.optim.SGD(model.parameters(), lr=0.02, momentum=0.9)\nsched = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.15, patience=2)\n\nloaders = {\n \"train\": train_dataloader,\n \"valid\": val_dataloader\n}\n\nclass LyftRunner(dl.Runner):\n\n def predict_batch(self, batch):\n return self.model(batch[0].to(self.device).view(batch[0].size(0), -1))\n\n def _handle_batch(self, batch):\n x, y = batch['image'], batch['target_positions']\n self.model.logit_final = nn.Linear(128, y.shape[0]).cuda()\n y_hat = self.model(x).view(y.shape)\n target_availabilities = batch[\"target_availabilities\"].unsqueeze(-1)\n criterion = torch.nn.MSELoss(reduction=\"none\")\n \n loss = criterion(y_hat, y)\n loss = loss * target_availabilities\n loss = loss.mean()\n self.batch_metrics.update(\n {\"loss\": loss}\n )\n\n if self.is_train_loader:\n loss.backward()\n self.optimizer.step()\n self.optimizer.zero_grad()\n \n if MONITORING:\n writer.add_scalar('Loss', loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Few things:\n+ Configuration of the data loaders\n+ Beginning the Catalyst process\n+ Neptune, wandb initialization"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%time\ndevice = utils.get_device()\nrunner = LyftRunner(device=device)\nif MONITORING:\n writer = SummaryWriter('./logs')\n\n runner.train(\n model=model,\n optimizer=optimizer,\n loaders=loaders,\n logdir=\"./logs\",\n num_epochs=6,\n verbose=True,\n load_best_on_end=True,\n scheduler=sched,\n callbacks=[neptune_logger, BatchOverfitCallback(train=10, valid=0.5), \n EarlyStoppingCallback(\n patience=5,\n metric=\"loss\",\n minimize=True,\n ), WandbLogger(project=\"dertaismus\",name= 'Example')\n ]\n )\nelse:\n print(\"Woops! Looks like you disabled monitoring. To be able to run the model without monitoring, try running the above catalyst model without the Neptune logger and the Wandb logger.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And this is how the Neptune metrics look like:\n"},{"metadata":{},"cell_type":"markdown","source":"To properly work with Neptune callbacks, replace the `U.` fields in the callback class with your API token, project name etc. Then, add it as a callback to Catalyst and watch the magic happen."},{"metadata":{},"cell_type":"markdown","source":"Source: https://github.com/lyft/l5kit/blob/master/examples/visualisation/visualise_data.ipynb\n\n**WORK IN PROGRESS - MORE TO COME.**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}