{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Keras Starter.\n**\nThank you to @pestipeti who did the heavy lifting at https://www.kaggle.com/pestipeti/pytorch-baseline-train \n\nI am newer to deep learning models and have been trying to learn Keras.  Please let me know of any blatent issues!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"v3: Learned How to Attach the proper input to the ResNet model.  So now I can preload imagenet weights\n\nv2: Fixed bug in creating submission where I was using the same timestamp and agent_id for all test shots\n\n\nv1: add predictions.  Try 10000 * 16 training images with a 1/8 validation.  SGD and mse loss.  Turning GPU on.\n\nv0: ResNet50 with random weights, train on 20,000 samples.  Timed out on gpu.  Got ~65% through training.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Start with some imports and the utility script from the comments","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%time\nimport os\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nimport numpy as np\nimport gc\nimport tensorflow as tf\nimport time\nt0 = time.time()\n\n## this script transports l5kit and dependencies\nos.system('pip uninstall typing -y')\nos.system('pip install --target=/kaggle/working pymap3d==2.1.0')\nos.system('pip install --target=/kaggle/working protobuf==3.12.2')\nos.system('pip install --target=/kaggle/working transforms3d')\nos.system('pip install --target=/kaggle/working zarr')\nos.system('pip install --target=/kaggle/working ptable')\n\nos.system('pip install --no-dependencies --target=/kaggle/working l5kit')\n#!pip install --upgrade pip\n#!pip install pymap3d==2.1.0\n#!pip install -U l5kit","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Importing some stuff from the l5kit and setting the directories.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"DIR_INPUT = \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\"\nimport os\nos.environ[\"L5KIT_DATA_FOLDER\"] = DIR_INPUT\nSINGLE_MODE_SUBMISSION = f\"{DIR_INPUT}/single_mode_sample_submission.csv\"\nMULTI_MODE_SUBMISSION = f\"{DIR_INPUT}/multi_mode_sample_submission.csv\"\nfrom l5kit.data import LocalDataManager, ChunkedDataset\nfrom l5kit.dataset import AgentDataset, EgoDataset\nfrom l5kit.evaluation import write_pred_csv\nfrom l5kit.rasterization import build_rasterizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Config File (dictionary)  from the linked notebook.  This contains parameters for the training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DEBUG = False  # True just trains for 10 steps instead of the full dataset\ncfg = {\n    'format_version': 4,\n    'model_params': {\n        'model_architecture': 'resnet50',\n        'history_num_frames': 20,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1\n    },\n    \n    'raster_params': {\n        'raster_size': [224, 224],\n        'pixel_size': [0.5, 0.5],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n    \n    'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 12,\n        'shuffle': True,\n        'num_workers': 4\n    },\n    \n    'train_params': {\n        'max_num_steps': 10*16 if DEBUG else 3200,\n        'checkpoint_every_n_steps': 5000,\n        'train_batch' : 32,\n        'num_batch' : 10\n        \n        # 'eval_every_n_steps': -1\n    },\n    \n    'test_data_loader': {\n        'key': 'scenes/test.zarr',\n        'batch_size': 8,\n        'shuffle': False,\n        'num_workers': 4\n    },\n    \n    \n    \n    'valid_data_loader': {\n        'key': 'scenes/validate.zarr',\n        'batch_size': 8,\n        'shuffle': False,\n        'num_workers': 4\n    },\n    \n    \n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load In the train dataset.  I notice the pytorch folks can just import this with DataLoader, but I am not familiar with anything similar in keras.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_cfg = cfg[\"train_data_loader\"]\n\n# Rasterizer\ndm = LocalDataManager(None)\nrasterizer = build_rasterizer(cfg, dm)\n\n# Train dataset/dataloader\n\ntrain_zarr = ChunkedDataset(dm.require(train_cfg[\"key\"])).open()\ntrain_dataset = AgentDataset(cfg, train_zarr, rasterizer)\nhist_shape = train_dataset[0]['history_positions'].shape\nnum_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\nnum_in_channels = 3 + num_history_channels\nnum_targets = 2 * cfg[\"model_params\"][\"future_num_frames\"]\n\nprint(train_dataset)\n\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_path1 = dm.require(cfg[\"valid_data_loader\"][\"key\"])\nvalid_zarr = ChunkedDataset(dataset_path1).open()\nvalid_dataset = AgentDataset(cfg, valid_zarr, rasterizer)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_itr = iter(valid_dataset)\nn_valid = 100\n\nval_inputs = np.zeros(shape=(n_valid,224,224, num_in_channels) )\nval_targets = np.zeros(shape=(n_valid,num_targets))\nfor itr in tqdm(range(n_valid)):\n    data = next(valid_itr)\n\n    val_inputs[itr] = data['image'].transpose(1,2,0)    \n    val_targets[itr] = data['target_positions'].reshape(-1,num_targets)\n    gc.collect()\ndel valid_dataset\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"idx = 100\nplt.scatter(train_dataset[idx]['history_positions'][:,0],train_dataset[idx]['history_positions'][:,1])\nplt.scatter(train_dataset[idx]['target_positions'][:,0],train_dataset[idx]['target_positions'][:,1],c='r')\nplt.show()\nprint(train_dataset[0]['target_positions'].shape) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define the model architecture.\n\nGiven what the examples use, lets start with ResNet50.  I tried to initialize with imagenet weights, but I get dimension mismatches, so that is something to look into.\n\nThe avg pooling layer after ResNet50 will be 2048, so I add in some Dense+ BN + dropout layers, with the last one being 2xnum_targets units.\n\nFirst run will just use sgd optimizer with mse loss.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications.resnet50 import ResNet50\nfrom keras.utils.conv_utils import convert_kernel\nfrom keras.layers import (Input, Conv2D, Flatten,Dense,AveragePooling2D,Dropout,MaxPooling2D,BatchNormalization)\nfrom keras.models import Model, Sequential\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\nfrom keras import optimizers\n\nbase_in = Input(shape=(224,224,num_in_channels))\nbase_model=Conv2D(20,kernel_size=1,use_bias=False,padding=\"same\")(base_in)\nbase_model=Conv2D(3,kernel_size=3,use_bias=False,padding=\"same\")(base_model)\n\n\nbase_model = ResNet50(include_top=False,\n                      weights= 'imagenet',\n                      input_tensor= Input(shape = (224,224,3)),\n                      pooling='max'\n                ) (base_model)\n\n#dense_model = base_model.output\ndense_model = Dense(1000, activation=\"linear\")(base_model)\ndense_model = Dropout(.25)(dense_model)\ndense_model = Dense(500, activation=\"linear\")(dense_model)\ndense_model = Dropout(.25)(dense_model)\ndense_model = Dense(num_targets, activation=\"linear\")(dense_model)\n\nmodel = Model(inputs=base_in, outputs=dense_model)\nopt = optimizers.Adam(lr=0.002)\nmodel.compile(optimizer=opt, loss='mse')\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to loop through the train_dataset and use a batch_size variable to train the model in batches.  ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nMC = ModelCheckpoint('./model.h5', verbose=True,monitor='val_loss',mode='min',\n        save_weights_only=True,save_best_only=True)\n\nstop = EarlyStopping(monitor = 'val_loss', restore_best_weights=True , patience = 5)\n\ntr_it = iter(train_dataset)\nbatch_size = cfg['train_params']['train_batch']\n#progress_bar = tqdm(range(0,cfg[\"train_params\"][\"max_num_steps\"],batch_size))\nprogress_bar = tqdm(range(cfg[\"train_params\"][\"max_num_steps\"]))\ntwo_hours = 60 * 60 * 4\nlosses = []\nhist = []\nfor itr in progress_bar:#range(0,cfg[\"train_params\"][\"max_num_steps\"],batch_size):\n    inputs = np.zeros(shape=(batch_size,224,224,num_in_channels))\n    targets = np.zeros(shape=(batch_size, num_targets))\n    \n    for i in range(batch_size):\n        \n        try:\n            data = next(tr_it)\n        except StopIteration:\n            tr_it = iter(train_dataset)\n            data = next(tr_it)\n            \n        inputs[i] = data['image'].transpose(1,2,0)\n        targets[i] = data['target_positions'].reshape(-1,num_targets)\n   \n    h = model.fit(inputs, targets,\n                  batch_size = batch_size / 2 ,\n                  validation_data = (val_inputs, val_targets),\n                  verbose = 0,\n                 callbacks = [MC, stop])\n                  \n    hist.append(h.history)\n    gc.collect()\n    # For training + submission, break if training exceeds 6 hours\n    if (time.time()-t0) > four_hours:\n        break\n    \n    \nvl = [hi['val_loss'] for hi in hist]\nl = [hi['loss'] for hi in hist]\nplt.plot(np.log(vl), label = 'val_loss')\nplt.plot(np.log(l), label = 'loss')\nplt.legend(loc=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('modelv0.h5')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Example Prediction:\nimport matplotlib.pyplot as plt\na1 = next(tr_it)\ninp = a1['image'].transpose(1,2,0)\nact = a1['target_positions']\npred = model.predict(inp.reshape(-1,224,224,num_in_channels)).reshape(50,2)\nplt.scatter(act[:,0], act[:,1])\nplt.scatter(pred[:,0],pred[:,1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can use this model to predict from the test set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_cfg = cfg[\"test_data_loader\"]\n\n# Rasterizer\nrasterizer = build_rasterizer(cfg, dm)\n\n# Test dataset/dataloader\ntest_zarr = ChunkedDataset(dm.require(test_cfg[\"key\"])).open()\ntest_mask = np.load(f\"{DIR_INPUT}/scenes/mask.npz\")[\"arr_0\"]\ntest_dataset = AgentDataset(cfg, test_zarr, rasterizer, agents_mask=test_mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"t_shape = test_dataset[0][\"target_positions\"].shape\ntimestamps = []\nagent_ids = []\ncoords = []\nfor it in tqdm(test_dataset):\n    \n    dat = it['image'].transpose(1,2,0)\n    coords.append(np.array(model.predict(dat.reshape(1,224,224,num_in_channels)).reshape(t_shape)))\n    timestamps.append(it[\"timestamp\"])\n    agent_ids.append(it[\"track_id\"])\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.evaluation import write_pred_csv\n\n\nwrite_pred_csv('submission.csv',\n                timestamps = np.array(timestamps),\n                track_ids = np.array(agent_ids),\n                coords = np.array(coords) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I will play with some parameters later.  I am finally glad to just have something that functions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}