{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Preface\n**In this notebook we would be looking at fine tuning an [EfficientNet model](https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html). EfficientNets are the class of models that work on computer vision tasks and have been implemented by Google. These models achieve state-of-art performance on computer vision tasks. Hence I decided to use it as the pretrained model for this task.**\n\n**This notebook contains only the training part of the model. The inference and ensemble tasks should be carried out in seperate notebooks to iterate faster.**\n\n**You could also look over my [ensemble notebook](https://www.kaggle.com/forwet/lyft-ensemble-and-submission) for a structured way to perform ensembling.**\n "},{"metadata":{},"cell_type":"markdown","source":"**Version 2 Updates:**\n1. raster_size - 300\n2. max_num_steps - 500\n3. train_batch_size - 12\n4. train_num_workers - 4\n5. model_save - On minimum training loss + At final iteration\n6. WEIGHT_FILE - None\n\n\n**Version 3 Updates**\n1. pixel_size = 0.45\n2. raster_size = 222 (calculated using pixel size)\n3. epochs = 5\n4. used [catalyst module](https://catalyst-team.github.io/catalyst/) for training of the model\n5. trained the model on a subset of `12000` samples of the original dataset.\n6. Validated the model on a subset of `500` samples.\n7. Used function to get image size based on pixel size."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Declaring the path to load efficientNet models.\nimport sys\nsys.path.append('../input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"#IMPORTS\n\n# PyTorch\nimport torch\nfrom torch import nn, optim\nfrom torch.utils.data import DataLoader\nfrom torchvision.models.resnet import resnet50\n\n# L5kit\nfrom l5kit.configs import load_config_data\nfrom l5kit.geometry import transform_points\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.dataset import AgentDataset, EgoDataset\nfrom l5kit.data import LocalDataManager, ChunkedDataset\nfrom l5kit.evaluation.metrics import neg_multi_log_likelihood, time_displace\nfrom l5kit.visualization import PREDICTED_POINTS_COLOR, TARGET_POINTS_COLOR, draw_trajectory\nfrom l5kit.evaluation import write_pred_csv, compute_metrics_csv, read_gt_csv, create_chopped_dataset\n\n# EfficientNet \nfrom efficientnet_pytorch import model as enet\n\n# Catalyst module\nfrom catalyst import dl\nfrom catalyst.utils import metrics\nfrom catalyst.dl import utils\n\n# Miscellaneous\nimport os\nimport gc\nimport sys\nimport numpy as np\nimport pandas as pd\nfrom tqdm.auto import tqdm\nfrom typing import Dict\nimport matplotlib.pyplot as plt\nfrom tempfile import gettempdir\nfrom prettytable import PrettyTable","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configrations"},{"metadata":{"trusted":true},"cell_type":"code","source":"# L5KIT'S CONFIGRATIONS\n\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"../input/lyft-motion-prediction-autonomous-vehicles\"\ndm = LocalDataManager()\ncfg = {\n        'model_params': {'model_architecture': 'efficientnet-b6',\n          'history_num_frames': 0,\n          'history_step_size': 1,\n          'history_delta_time': 0.1,\n          'future_num_frames': 50,\n          'future_step_size': 1,\n          'future_delta_time': 0.1},\n\n        'raster_params': {'raster_size': [300, 300],\n          'pixel_size': [0.33, 0.33],\n          'ego_center': [0.25, 0.5],\n          'map_type': 'py_semantic',\n          'satellite_map_key': 'aerial_map/aerial_map.png',\n          'semantic_map_key': 'semantic_map/semantic_map.pb',\n          'dataset_meta_key': 'meta.json',\n          'filter_agents_threshold': 0.5},\n\n        'train_data_loader': {'key': 'scenes/train.zarr',\n          'batch_size': 12,\n          'shuffle': True,\n          'num_workers': 4},\n\n        \"valid_data_loader\":{\"key\": \"scenes/validation.zarr\",\n                            \"batch_size\": 8,\n                            \"shuffle\": False,\n                            \"num_workers\": 4},\n    \n        }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Using Peter's great explanation of `raster size` and `pixel size`, I built up a function to\ncalculate the `raster size` parameter from the given `pixel size`. Using the same values of velocity and time as Peter used in his explanation.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def calc_img_size(px_size):\n    return int(100/px_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CONFIGRATIONS\n\nWEIGHT_FILE = None # Model state_dict path of previously trained model\nMODEL_NAME = \"efficientnet-b0\"\nIMG_SIZE = calc_img_size(cfg[\"raster_params\"][\"pixel_size\"][0])\nVALIDATION = True # A hyperparameter you could use to toggle for validating the model\n\ncfg[\"raster_params\"][\"raster_size\"] = [IMG_SIZE, IMG_SIZE]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"MODEL_NAME: It could be one of-\n- \"efficientnet-b0\"\n- \"efficientnet-b1\"\n- \"efficientnet-b2\" \n- \"efficientnet-b3\"\n- \"efficientnet-b4\" \n- \"efficientnet-b5\" \n- \"efficientnet-b6\"\n- \"efficientnet-b7\" (Would likely to cause OOM due to larger architecture.)\n\nIMG_SIZE: You can take either one of the following settings or try some other combinations-\n- 300 (pixel size = 0.33)\n- 224 (pixel size = 0.45)\n- 267 (pixel size = 0.38)\n- 245 (pixel size = 0.41)\n\n**More info over `raster_size` and `pixel_size` can be found out [here](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/178323).** "},{"metadata":{},"cell_type":"markdown","source":"# Utility Scripts"},{"metadata":{"trusted":true},"cell_type":"code","source":"%time\ndef build_model(cfg) -> torch.nn.Module:\n    \"\"\"Creates an instance of the pretrained model with custom input and output\"\"\"\n    model = enet.EfficientNet.from_name(MODEL_NAME)\n    \n    num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n    num_in_channels = 3 + num_history_channels\n    num_targets = 2*cfg[\"model_params\"][\"future_num_frames\"]\n    \n    model._conv_stem = nn.Conv2d(\n        num_in_channels,\n        model._conv_stem.out_channels,\n        kernel_size=model._conv_stem.kernel_size,\n        stride=model._conv_stem.stride,\n        padding=model._conv_stem.padding,\n        bias=False\n    )\n    \n    model._fc = nn.Linear(in_features=model._fc.in_features, out_features=num_targets)\n    return model\n\ndef forward(data, model, device, criterion):\n    \"\"\"Forward Propogation function\"\"\"\n    inputs = data[\"image\"].to(device)\n    target_availabilities = data[\"target_availabilities\"].unsqueeze(-1).to(device)\n    targets = data[\"target_positions\"].to(device)\n    \n    outputs = model(inputs)\n    outputs = outputs.reshape(targets.shape)\n    loss = criterion(outputs, targets)\n    \n    loss = loss * target_availabilities\n    loss = loss.mean()\n\n    # Disabling rmse loss pertaining the mse loss.\n    # loss = torch.sqrt(loss) # Using RMSE loss\n    return loss, outputs\n\n\ndef get_dataloader(config, zarr_data, subset_len, map_type=\"py_semantic\"):\n    \"\"\"Creates DataLoader instance for the given dataset.\"\"\"\n    cfg[\"raster_params\"][\"map_type\"] = map_type\n    rasterizer = build_rasterizer(cfg, dm)\n    chunk_data = ChunkedDataset(zarr_data).open()\n    agent_data = AgentDataset(cfg, chunk_data, rasterizer)\n    \n    # Sample the dataset\n    subset_data = torch.utils.data.Subset(agent_data, range(0, subset_len))\n    \n    dataloader = DataLoader(subset_data, \n                            batch_size=config[\"batch_size\"],\n                            num_workers=config[\"num_workers\"],\n                            shuffle=config[\"shuffle\"]\n                           )\n    return dataloader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time\ndef train(opt=None, criterion=None, lrate=1e-2):\n        \"\"\"Function for training the model\"\"\"\n        print(\"Building Model...\")\n        device=torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n        model = build_model(cfg).to(device)\n        optimizer = optim.Adam(model.parameters(), lr=lrate) if opt is None else opt\n        criterion = nn.MSELoss(reduction=\"none\")\n        \n        if WEIGHT_FILE is not None:\n            state_dict = torch.load(WEIGHT_FILE, map_location=device)\n            model.load_state_dict(state_dict)\n        \n        print(\"Prepairing Dataloader...\")\n        train_dataloader = get_dataloader(cfg[\"train_data_loader\"], dm.require(\"scenes/train.zarr\"), 12000)\n        \n        if VALIDATION:\n            valid_dataloader = get_dataloader(cfg[\"valid_data_loader\"], dm.require(\"scenes/validate.zarr\"), 500)\n            \n        print(\"Training...\")\n        loaders = {\n                    \"train\": train_dataloader,\n                    \"valid\": valid_dataloader\n                }\n\n        device = utils.get_device()\n        runner = LyftRunner(device=device)\n        \n        runner.train(\n                model=model,\n                optimizer=optimizer,\n                loaders=loaders,\n                logdir=\"./logs\",\n                num_epochs=5,\n                verbose=True,\n                load_best_on_end=True\n            )\n        return model\n\nclass LyftRunner(dl.Runner):\n\n    def predict_batch(self, batch):\n        return self.model(batch[0].to(self.device).view(batch[0].size(0), -1))\n\n    def _handle_batch(self, batch):\n        x, y = batch['image'], batch['target_positions']\n        y_hat = self.model(x).reshape(y.shape)\n        target_availabilities = batch[\"target_availabilities\"].unsqueeze(-1)\n        criterion = torch.nn.MSELoss(reduction=\"none\")\n        loss = criterion(y_hat, y)\n        loss = loss * target_availabilities\n        loss = loss.mean()\n        self.batch_metrics.update(\n            {\"loss\": loss}\n        )\n\n        if self.is_train_loader:\n            loss.backward()\n            self.optimizer.step()\n            self.optimizer.zero_grad()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing images\n\n**Let's just have a look at how the image would be formed using the current settings for `raster_size` and `pixel_size`.**\n> The more you decrease the `pixel_size`, the more the image will be zoomed-in. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing pixel_size and raster_size START ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Preparing the EgoDataset from sample zarr file.\nsample_zarr = dm.require(\"scenes/sample.zarr\")\nsample_chunk = ChunkedDataset(sample_zarr).open()\nrasterizer = build_rasterizer(cfg, dm)\nsample_ego = EgoDataset(cfg, sample_chunk, rasterizer)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 3, figsize=(15,15))\nax = ax.flatten()\nfor i in range(9):\n    idx = np.random.randint(500)\n    data = sample_ego[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = sample_ego.rasterizer.to_rgb(im)\n    data_positions = transform_points(data[\"target_positions\"]+data[\"centroid\"], data[\"world_to_image\"])\n    draw_trajectory(im, data_positions, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    ax[i].imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing END","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use the above defined utility script to train the model.\nmodel = train()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Saving model on final iteration\ntorch.save(model.state_dict(), f\"{MODEL_NAME}.pth\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<body>\n    <p style=\"color:#bf190d;font-size:18px;\">Hope you liked my work.</p>\n    <p style=\"color:#bf19dd;font-size:18px;\">Please upvote if you did.</p>\n</body>"},{"metadata":{},"cell_type":"markdown","source":"<body>\n    <p style=\"color:#bf190d;font-size:15px;\">I would be updating my <a href=\"https://www.kaggle.com/forwet/lyft-ensemble-and-submission\">ensemble notebook</a> for evaluation of this model. Stay tuned!</p>\n    <p style=\"color:#6a018a;font-size:15px;\">If you've any queries please comment down below. Thanks!</p>"},{"metadata":{},"cell_type":"markdown","source":"# References\n\n- https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html\n- https://arxiv.org/pdf/1905.11946.pdf\n- https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/178323\n- https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/180468\n- https://www.kaggle.com/nxrprime/lyft-understanding-the-data-catalyst-training"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}