{"cells":[{"metadata":{},"cell_type":"markdown","source":"This is an inference kernel. Please [find the training kernel here](https://www.kaggle.com/kneroma/lgbm-on-lyft-tabular-data-training).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Diving far into the zarr file format and the Lyft L5kit github repos, I finally succeeded in converting the competition's dataset into **csv** files on which we could run classical models.\n\nFor those who are interrested, [the csv dataset looks like this one](https://www.kaggle.com/kneroma/lyft-motion-prediction-autonomous-vehicles-as-csv). I've not uploaded the whole dataset for now. Stay tuned !\n\nFinally, let's recall that [this notebook of mine could also help you in stepping far into Zarr files and the Lyft L5kit dataset format.](https://www.kaggle.com/kneroma/zarr-files-and-l5kit-data-for-dummies)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"For the prediction, you can [find the test set as csv here](https://www.kaggle.com/kneroma/lyft-test-set-as-csv). ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"

Please, don't mind upovting the datasets in order to make them more visibe for all of us.

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nimport re,json\nimport itertools as it\nfrom pathlib import Path\n\nimport lightgbm as lgb\n\npd.options.display.max_columns=305","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading the test set as CSV","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Here, I'm gonna load the test, it contains `71122` rows as expected\ndf = pd.read_csv(\"../input/lyft-test-set-as-csv/Lyft_test_set.csv\")\nprint(\"df.shape:\", df.shape)\ndf.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"> Some of the columns are self-explaining, for the others, please refer to the corresponding dataset for more details.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading the LGBM models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model_name(filename):\n return re.search(\"^(lgbm_[x,y]_shift_\\d+)\", filename).group(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_models(path):\n models = {}\n path = Path(path)\n for model in path.glob(\"lgbm*\"):\n model_name = get_model_name(model.stem)\n shift = int(model_name.split(\"shift_\")[1])\n meta = path.joinpath(\"meta_shift_{:02d}.json\".format(shift))\n with meta.open() as f:\n train_cols = json.load(f)[\"TRAIN_COLS\"]\n models[model_name] = {\"model\": model.as_posix(), \"train_cols\": train_cols}\n return models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"models = get_models(\"../input/lyft-models/lgbm_06\")\nlen(models)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"next(iter(models.items()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I've trained **100** LGBM models : one for each of the *50 time horizons*x*2 space dimension*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The whole training took about one hour and the prediction step is even fatster.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make prediction for the test set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_colnames():\n xcols = [\"coord_x{}{}\".format(step, rank) for step in range(3) for rank in range(50)]\n ycols = [\"coord_y{}{}\".format(step, rank) for step in range(3) for rank in range(50)]\n cols = [\"timestamp\", \"track_id\"] + [\"conf_0\", \"conf_1\", \"conf_2\"] + list(it.chain(*zip(xcols, ycols)))\n return cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(models, df):\n sub = np.empty((len(df), 305))\n sub.fill(np.nan)\n sub = pd.DataFrame(sub, columns = make_colnames())\n sub[[\"timestamp\", \"track_id\"]] = df[[\"timestamp\", \"track_id\"]]\n sub[\"conf_0\"] = 1.0\n \n for shift in range(1, 51):\n for suffix in [\"x\", \"y\"]:\n model_info = models[\"lgbm_{}_shift_{:02d}\".format(suffix, shift)]\n \n model = lgb.Booster(model_file= model_info[\"model\"])\n pred = model.predict(df[model_info[\"train_cols\"]])\n \n sub[\"coord_{}0{}\".format(suffix, shift-1)] = pred\n\n if not shift%10:\n print(\"shift: {}\".format(shift))\n \n sub.fillna(0., inplace=True)\n \n return sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = predict(models, df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.iloc[:50, :105]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Getting such a score with no GPU computation nor image processing is just beautiful. More again, my LGBM are not well trained and I **zero** features ! Needless to say that there still room for improvements !","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I will be publishing my training dataset and the whole conversion process by soon. For now, I need some cleaning and refacto for my messy code :) .","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
@Kkiller
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}