{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Submitting a model\n\nThis notebook will guide you through the model submission process. To start, if you're in the notebook view page, click \"... -> Copy and Edit\" on the right.\n\n\n## Step 0: Install dependencies\n\nWe need to install our dependencies here. Just run the following cell:"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/dependencies/dotmap-1.3.23-py3-none-any.whl","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 1: Make a submission package\n\nFirst, save the model checkpoint you want to submit under `attachments`, along with any other files that will be loaded during test time. That is in the repository root:\n\n```bash\n(base) poesia@node01-ccncluster:/data/poesia/riiid/ednet-challenge$ cp path/to/model.ckpt attachments/\n```\n\nThen, run:\n\n```bash\n(base) poesia@node01-ccncluster:/data/poesia/riiid/ednet-challenge$ ./make_submission_package.sh\n```\n\nThis will create a file called `submission.tar.gz`.\n\n## Step 2: Upload the submission package as data in this notebook\n\nIf the submission package is on CCN, first you'll need to get it on your local machine. `scp` will do the trick.\n\nHaving the package, on the right side of this screen, click on \"Add data\", and then \"Upload\". Under name, put \"Submission N\", where N is a number. It can be any number, but each time you submit it needs to be different. Then, upload `submission.tar.gz` and click on \"Create\". Kaggle will take a few seconds to upload it and then download it here in the notebook.\n\nIf all went well, on the right, inside the `input` folder, you'll see `submission-n` (this might take a minute - go drink water). In the cell below, change `n` and run it."},{"metadata":{"trusted":true},"cell_type":"code","source":"SUBMISSION_DIR = 'submission-4'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 3: Set up the Python Path\n\nKaggle has automatically unpacked `submission.tar.gz` under `/kaggle/input`. Now, we'll just add that to our Python Path, so that we can use code from the repository. Additionally, we'll do the same for a Kaggle-specific package with the submission API. For that, just run the cell below."},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\n\nsys.path.append('/kaggle/input/riiid-test-answer-prediction')\nsys.path.append('/kaggle/input/' + SUBMISSION_DIR)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 4: Initialize the test environment\n\nJust run the cell below. You can run this only once per notebook run; to run it again, you need to restart the notebook entirely."},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 5: Load your model\n\nThis part might need some work if it's the first time you submit a model with this kind of agent. We now need to load the model in a wrapper that unifies the API and makes it easy to submit it. In the repository, the wrapper interface is located in `ednet/common/submission.py`. Each agent needs to implement a class that has that interface. The example below loads a DKTAgent. This part will be model-specific, but shouldn't be too different from DKT.\n\nAlso, you probably want to load your model on a GPU (or TPU if you're fancy). To make one available, enable \"Accelerator\" on the right side of the screen, and properly load your model on that device."},{"metadata":{"trusted":true},"cell_type":"code","source":"from ednet.dkt.agents.dkt import DKTSubmissionWrapper\nagent = DKTSubmissionWrapper(f'/kaggle/input/{SUBMISSION_DIR}/attachments/dkt-test.ckpt')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 6: Run on the test set!\n\nFinally! Just run the cell below to run your model on the test set.\n\nImportant: note that you almost certainly do not want to pass empty dicts to the last parameters here. Instead, you'll want to save them from the dataset after the model trains (probably as a pickle file), attach that and load it."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\npd.options.mode.chained_assignment = None\n\nfrom ednet.common.submission import TestSetRunner\n\nrunner = TestSetRunner(agent,\n                       f'/kaggle/input/{SUBMISSION_DIR}/attachments/cache_temporal_feat_splits_train_250k_valid_25k_cv1.pickle',\n                       iter_test,\n                       {},\n                       {},\n                       {})\nrunner.run(env)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 7: Submit the notebook\n\nIf all went well, you just ran your code on the example test set. Now it's time to make an actual submission.  First, **make sure Internet is turned off for the notebook** (on the right). Click \"Save Version\" above, give your version a name, then go to the competition's page, and \"Submit predictions\". Choose the right notebook, and you're done! If your model works, submission takes a considerably long time (at least 3-4 hours, even for a trivial baseline model), so go do something else and check your results later in the day or tomorrow."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}