{"cells":[{"metadata":{},"cell_type":"markdown","source":"# A Simple Feed Forward Network on Lyft Tabular Data With the Competition Loss","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Recalls\n\n* The training step uses a [custom parquet version](https://www.kaggle.com/kneroma/lyft-train-as-parquet) of the official training datatset.\n* The submission step uses a [custom parquet version](https://www.kaggle.com/kneroma/lyft-train-as-parquet) of the official test datatset\n* To reduce execution time, I use [my own checkpoint](https://www.kaggle.com/kneroma/neural-net-on-lyft-tabular-data), even if everything was trained with this same script.\n* Here are [details on how I build those datasets](https://www.kaggle.com/kneroma/zarr-files-and-l5kit-data-for-dummies)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nimport seaborn as sns\nfrom pathlib import Path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom tqdm.notebook import tqdm\nimport re,json,time,pickle\nfrom sklearn.preprocessing import MinMaxScaler","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.options.display.max_columns=305","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_ROOT = Path(\"../input/lyft-train-as-parquet/train\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scaler = MinMaxScaler()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading the train data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_scene_path(scene):\n meta = \"meta_{}_{}.json\".format(*re.search( r\"scenes_(\\d+)_(\\d+)\", scene.stem).groups())\n with open(DATA_ROOT/meta) as f:\n meta = json.load(f)\n frame = DATA_ROOT/meta[\"frames\"][\"results\"][\"filename\"]\n agent = DATA_ROOT/meta[\"agents\"][\"results\"][\"filename\"]\n return (scene, frame, agent)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"SCENES = np.array(list(DATA_ROOT.glob(\"scenes_*.parquet.snappy\")))\nSCENES = SCENES[np.random.permutation(len(SCENES))]\nprint(\"NB SCENES:\", len(SCENES))\nscene = SCENES[0]\nscene","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_scene_path(scene)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reader = pd.read_parquet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_COLS = [\n# 'ego_translation_x', \n# 'ego_translation_y', \n# 'ego_translation_z', \n# 'ego_rotation_xx', \n# 'ego_rotation_xy', \n# 'ego_rotation_xz', \n# 'ego_rotation_yx', \n# 'ego_rotation_yy', \n# 'ego_rotation_yz', \n# 'ego_rotation_zx', \n# 'ego_rotation_zy', \n# 'ego_rotation_zz', \n 'extent_x_shift_50', \n 'extent_y_shift_50', \n 'extent_z_shift_50', \n 'velocity_x_shift_50', \n 'velocity_y_shift_50', \n 'label_probabilities_PERCEPTION_LABEL_UNKNOWN_shift_50', \n 'label_probabilities_PERCEPTION_LABEL_CAR_shift_50', \n 'label_probabilities_PERCEPTION_LABEL_CYCLIST_shift_50', \n 'label_probabilities_PERCEPTION_LABEL_PEDESTRIAN_shift_50', \n 'yaw_shift_50', \n 'nagents_shift_50', \n 'nlights_shift_50', \n 'centroid_x_shift_50', \n 'centroid_y_shift_50',\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_list(df_names):\n df = None\n for df_name in df_names:\n temp = reader(DATA_ROOT/df_name)\n df = df.append(temp) if df is not None else temp\n \n return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_shifted_col_names(cols, shift):\n new_cols = []\n for col in cols:\n col = re.sub(r\"_shift_(\\d\\d$)\", \"\", col)\n new_col = f\"{col}_shift_{shift:02d}\"\n new_cols.append(new_col)\n return new_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def df_shifter(group_cols, shift_cols, max_shift=1, shifts=None, keep_nan=False, verbose=0):\n global df\n assert max_shift >= 1\n cols = None\n# df = globals()[\"df\"]\n shifts = shifts or range(1, max_shift+1)\n for ishift in shifts:\n if cols is not None:\n shift_cols = cols\n# cols = [\"{}_shift_{:02d}\".format(col, ishift) for col in shift_cols]\n cols = get_shifted_col_names(shift_cols, shift=ishift)\n \n df[cols] = df.groupby(group_cols)[shift_cols].shift()\n \n if not keep_nan:\n df = df[df[cols].notnull().all(1)]\n if verbose:\n print(\"ishift: {} df.shape: {}\".format(ishift, df.shape))\n# del globals()[\"df\"]\n# del df\n \n df.rename(columns={\"centroid_x\": \"centroid_x_shift_00\", \"centroid_y\": \"centroid_y_shift_00\"}, inplace=True)\n return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def merge(scenes, frames, agents, max_shift=50, verbose=False):\n global df\n df = scenes.merge(frames, on = \"scene_db_id\")\n df = df.merge(agents, on=\"frame_db_id\")\n\n df[\"nframes\"] = df.groupby([\"scene_db_id\", \"track_id\"])[\"scene_db_id\"].transform(\"count\")\n df = df[df[\"nframes\"] > max_shift]\n \n shift_cols = [\n# \"centroid_x\", \"centroid_y\",\n \"yaw\",\n \"velocity_x\",\"velocity_y\",\n \"nagents\",\"nlights\",\n 'extent_x', 'extent_y','extent_z',\n 'label_probabilities_PERCEPTION_LABEL_UNKNOWN',\n 'label_probabilities_PERCEPTION_LABEL_CAR', \n 'label_probabilities_PERCEPTION_LABEL_CYCLIST', \n 'label_probabilities_PERCEPTION_LABEL_PEDESTRIAN',\n ]\n \n df[shift_cols] = df[shift_cols].astype(np.float32, copy=True)\n df[[\"scene_db_id\", \"track_id\"]] = df[[\"scene_db_id\", \"track_id\"]].astype(np.float32, copy=True)\n\n shape0 = df.shape\n df = df_shifter(group_cols=[\"scene_db_id\", \"track_id\"], shift_cols=shift_cols, shifts=[max_shift],\n keep_nan=True, verbose=verbose)\n df = df_shifter(group_cols=[\"scene_db_id\", \"track_id\"], shift_cols=[\"centroid_x_shift_00\",\"centroid_y_shift_00\"],\n max_shift=max_shift, keep_nan=False, verbose=verbose)\n \n if verbose:\n print(\"SHAPE0:\", shape0)\n print(\"SHAPE1:\", df.shape)\n print(\"Nulls ratio:\", 1-len(df)/shape0[0])\n \n return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_all(max_shift=50, max_len=5e6, verbose=0):\n dfs = None\n scenes = SCENES[np.random.permutation(len(SCENES))]\n for scene in scenes:\n SCENES_FILE,FRAMES_FILE,AGENTS_FILE = get_scene_path(scene)\n \n scenes = reader(SCENES_FILE)\n \n frames = reader(FRAMES_FILE)\n# frames[\"frame_rank\"] = frames.groupby(\"scene_db_id\").scene_db_id.cumcount()\n frames[\"nagents\"] = frames[\"agent_index_interval_end\"] - frames[\"agent_index_interval_start\"]\n frames[\"nlights\"] = frames[\"traffic_light_faces_index_interval_end\"\n ] - frames[\"traffic_light_faces_index_interval_start\"]\n \n agents = reader(AGENTS_FILE)\n agents.rename(columns = {\"agent_id\": \"agent_db_id\"}, inplace=True)\n \n df = merge(scenes, frames, agents, max_shift=max_shift, verbose=verbose)\n \n dfs = df if dfs is None else dfs.append(df)\n dfs.reset_index(inplace=True, drop=True)\n \n if len(dfs) > max_len:\n break\n \n return dfs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Increase max_len for better results --> memory overflow risk !!!\n# Locally, I used max_len=12e6\ndf = read_all(max_shift=50, verbose=0, max_len=1e5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.isnull().any(1).sum()/len(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = sorted(df.columns[df.columns.str.startswith(\"centroid_\") & ~df.columns.isin(TRAIN_COLS)])\nXTARGET_COLS = [col for col in temp if \"_x_\" in col][::-1]\nYTARGET_COLS = [col for col in temp if \"_y_\" in col][::-1]\nXTARGET_COLS,YTARGET_COLS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nX = scaler.fit_transform(df[TRAIN_COLS].values.astype(\"float32\", copy=False))\nTARGET = np.stack([\n df[XTARGET_COLS].values.astype(\"float32\", copy=False) - df[[\"centroid_x_shift_50\"]].values.astype(\"float32\"),\n df[YTARGET_COLS].values.astype(\"float32\", copy=False) - df[[\"centroid_y_shift_50\"]].values.astype(\"float32\"),\n],\n axis=1,\n)\nX.shape, TARGET.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define the model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torch import nn, optim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SimpleNet(nn.Module):\n def __init__(self):\n super().__init__()\n self.net = nn.Sequential(\n nn.Linear(len(TRAIN_COLS), 124), nn.ReLU(), nn.Dropout(0.2),\n nn.Linear(124, 512), nn.ReLU(), nn.Dropout(0.2),\n nn.Linear(512, 2048), nn.ReLU(),nn.Dropout(0.2),\n nn.Linear(2048, 1024), nn.ReLU(),nn.Dropout(0.2),\n )\n \n self.xynet = nn.Linear(1024, 300)\n \n self.cnet = nn.Sequential(\n nn.Linear(1024, 512), nn.ReLU(),nn.Dropout(0.2),\n nn.Linear(512, 3),\n )\n \n def forward(self, x):\n features = self.net(x)\n xy = self.xynet(features)\n c = self.cnet(features)\n \n return c,xy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef shapefy(xy_pred, xy):\n NDIM = 3\n xy_pred = xy_pred.view((-1,2, NDIM, 50))\n xy = xy[:,:, None].repeat([1,1, NDIM, 1])\n return xy_pred, xy\n\ndef LyftLoss(c, xy_pred, xy):\n xy_pred, xy = shapefy(xy_pred, xy)\n \n c = torch.softmax(c, dim=1)\n \n l = torch.sum(torch.square(xy_pred-xy), dim=(1,3))/2\n \n # The LogSumExp trick for better numerical stability\n # https://en.wikipedia.org/wiki/LogSumExp\n m = l.min(dim=1).values\n l = torch.exp(m[:, None]-l)\n \n l = m - torch.log(torch.sum(l*c, dim=1))\n l = torch.mean(l)\n return l\n\n\ndef MSE(xy_pred, xy):\n xy_pred, xy = shapefy(xy_pred, xy)\n return torch.mean(torch.sum(torch.square(xy_pred-xy), 3))\n\ndef MAE(xy_pred, xy):\n xy_pred, xy = shapefy(xy_pred, xy)\n return torch.mean(torch.sum(torch.abs(xy_pred-xy), 3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_set, valid_set = train_test_split(np.arange(len(X)).reshape((-1,1)),\n np.arange(len(X)), test_size = .20, random_state=177)[2:]\nlen(train_set), len(valid_set)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nnet = SimpleNet().to(device)\ncriterion = LyftLoss\noptimizer = optim.Adam(net.parameters(), lr = 5e-4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"%%time\n\nsel = np.arange(len(train_set))\nbatch_size = 5000\nK = len(sel)//(5*batch_size)\nEPOCHS = 1 # For demo only, please choose a right value by yourself, You may need to enable GPU\n\nfor epoch in tqdm(list(range(EPOCHS))) :\n net.train()\n np.random.shuffle(sel)\n l,mse,mae, icount = 0.,0.,0., 0\n ibatch = 0\n for ibatch in tqdm(list(range(0, len(sel), batch_size)), leave = ibatch >= len(sel) - batch_size) :\n s = sel[ibatch:ibatch+batch_size]\n xb, yb = torch.from_numpy(X[s]).to(device), torch.from_numpy(TARGET[s]).to(device)\n \n optimizer.zero_grad()\n c,o = net(xb)\n loss = criterion(c, o, yb)\n loss.backward()\n optimizer.step()\n \n with torch.no_grad():\n l += loss.item()\n mse += MSE(o,yb).item()\n mae += MAE(o,yb).item()\n \n icount += 1\n \n if not (icount)%K:\n with torch.no_grad():\n s_valid = np.random.choice(valid_set, 100000, replace = False)\n l_valid, mse_valid, mae_valid, valid_count = 0.,0.,0., 0\n b = 10000\n for i_valid in range(0, len(s), b):\n s = s_valid[i_valid:i_valid+b]\n xb, yb = torch.from_numpy(X[s]).to(device), torch.from_numpy(TARGET[s]).to(device)\n\n c,o = net(xb)\n l_valid += criterion(c, o, yb)\n mse_valid += MSE(o,yb).item()\n mae_valid += MAE(o,yb).item()\n\n valid_count += 1\n print(\"[{}-{}] loss: ({:0.5f}, {:0.5f}) rmse: ({:0.5f}, {:0.5f}) mae: ({:0.5f}, {:0.5f})\".format(\n epoch,ibatch, l/K, l_valid/valid_count, np.sqrt(mse/K), np.sqrt(mse_valid/valid_count),\n mae/K, mae_valid/valid_count))\n \n l,mse,mae, icount = 0.,0.,0., 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(net.state_dict(), \"simple_net_00.pth\")\nwith open(\"scaler_simple_net_00.bin\", \"wb\") as f:\n pickle.dump(scaler, f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict with the trained model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Instead of using this untrained model, I will be using the one I trained locally. I trained it with the same script as the one in this kernel up to 10 epochs. The checkpoints is from the last epoch, not the best one ! To put it in a nutshell, you can even do better than this baseline score by adjusting things as you want !","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading locally pretrained weights and scaler\nnet = SimpleNet().to(device)\nnet.load_state_dict(torch.load(\"../input/neural-net-on-lyft-tabular-data/simple_net_00.pth\", map_location=device))\nnet = net.eval()\n\nwith open(\"../input/neural-net-on-lyft-tabular-data/scaler_net_00.bin\", \"rb\") as f:\n scaler = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n\ndf_sub = pd.read_csv(\"../input/lyft-test-set-as-csv/Lyft_test_set.csv\")\ndf_sub.rename(\n columns=dict(zip([col.replace(\"_shift_50\", \"\") for col in TRAIN_COLS], TRAIN_COLS)), inplace=True\n)\nprint(\"df_test:\", df_sub.shape)\ndf_sub.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_sub = df_sub[TRAIN_COLS].values.astype(\"float32\")\nX_sub = scaler.transform(X_sub)\nX_sub.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_colnames():\n xcols = [\"coord_x{}{}\".format(step, rank) for step in range(3) for rank in range(50)]\n ycols = [\"coord_y{}{}\".format(step, rank) for step in range(3) for rank in range(50)]\n cols = [\"timestamp\", \"track_id\"] + [\"conf_0\", \"conf_1\", \"conf_2\"] + xcols + ycols\n return cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nb=1000\npreds = []\ncs = []\nwith torch.no_grad(): \n for icount in tqdm(list(range(0, len(X_sub), b))):\n xb = torch.from_numpy(X_sub[icount:icount+b])\n c, yb = net(xb)\n c = torch.softmax(c, dim=1)\n cs.append(c.cpu().numpy())\n preds.append(yb.cpu().numpy())\npreds = np.vstack(preds)\ncs = np.vstack(cs)\npreds.shape, cs.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = make_colnames()\nsub = pd.DataFrame(np.hstack([cs, preds]), columns = cols[2:])\nsub[[\"timestamp\", \"track_id\"]] = df_sub[[\"timestamp\", \"track_id\"]].astype(int)\nsub = sub[cols]\nprint(\"sub.shape:\", sub.shape)\nsub.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nsub.to_csv(\"submission.csv\", index=False, float_format=\"%.5f\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How to improve results","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Our current result is just a baseline since I trained the model for just 10 epochs and I pick a simple feed forward network for demo purposes. So here are some ideas to improve results:\n\n* Change **architecture**\n* Build more features (I use no **agent historical data** for instant nor any **'ego'** related feature)\n* Train on more data by building better data loading pipeline\n* Train for **more epochs**\n* Cross validation\n* Checkpoint the **best model(s)**\n* Try **time-series** related models\n* ...","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
It's all for now, thanks for reading
\n
@kkiller
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}