{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle API Leak Probe\nIn Kaggle's API example notebook [here][1], all of `level_group = '0-4'` comes **before** `level_group = '5-12'` which comes **before** `level_group = '13-22'`. This is the **correct** ordering in time.\n\nDuring submission, we have noticed that the Kaggle API gives us the levels **backward**. First we see all `level_group = '13-22'`, then we see all `level_group = '5-12'`, then we see all `level_group = '0-4'`. This is **incorrect** ordering in time. This means we can build features from the future before predicting the present.\n\n## Kaggle and Host have been notified\nWe brought this to the attention of competition hosts and Kaggle in discussion [here][2]. \n\n## Example Notebook to demonstrate leak\nKaggle needed an example of the leak, so we made this notebook. This submission notebook by default will submit all zeros and score LB 0.226. This submission remembers whenever it sees `level_group = '5-12'` and `level_group = '13-22'`. The first time it observes one leak (i.e. seeing a user `level_group = '0-4'` **AFTER** it sees either `5-12` or `13-22`), it starts predicting 1s thereafter. We see that this notebook scores LB 0.414 (in version 1) which proves that is **observes at least 1 leak** on public LB. (because LB 0.414 is the LB score of submitting all 1s. Furthermore we know the leak occurs very early since we achieved full LB 0.414)\n\nThe reason i know that all the (public LB) data is leaked is because I have a trained an XGB model which uses features from the future. The features from the future change both my CV by +0.012 and LB by +0.012. This indicates that every single user is leaking. (because if only some users were leaking then LB would not boost the same as CV. Note in this competition my CV and LB are always exactly the same).\n\n# UPDATE: \nKaggle has fixed the time ordering of Kaggle API. See discussion [here][2]. If we fork and submit this notebook now, it should score LB 0.226. If it scores better than LB 0.226, please notify Kaggle.\n\n[1]: https://www.kaggle.com/code/philculliton/basic-submission-demo\n[2]: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd","metadata":{"papermill":{"duration":0.023295,"end_time":"2022-06-03T21:13:10.412151","exception":false,"start_time":"2022-06-03T21:13:10.388856","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-02-11T01:29:51.822447Z","iopub.execute_input":"2023-02-11T01:29:51.823320Z","iopub.status.idle":"2023-02-11T01:29:51.853859Z","shell.execute_reply.started":"2023-02-11T01:29:51.823214Z","shell.execute_reply":"2023-02-11T01:29:51.852761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import jo_wilder\nenv = jo_wilder.make_env()\niter_test = env.iter_test()","metadata":{"papermill":{"duration":0.036198,"end_time":"2022-06-03T21:13:10.454318","exception":false,"start_time":"2022-06-03T21:13:10.41812","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-02-11T01:29:51.858563Z","iopub.execute_input":"2023-02-11T01:29:51.858972Z","iopub.status.idle":"2023-02-11T01:29:51.881300Z","shell.execute_reply.started":"2023-02-11T01:29:51.858931Z","shell.execute_reply":"2023-02-11T01:29:51.879936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"counter = 0\nd1 = {}; d2 = {}; d3 = {}\nLEAK = False\n\n# The API will deliver two dataframes in this specific order,\n# for every session+level grouping (one group per session for each checkpoint)\nfor (sample_submission, test) in iter_test:\n    \n    # ANALZE EACH TEST USER TO SEE IF SEEN BEFORE\n    grp = test.level_group.values[0]\n    s = test.session_id.values[0]\n    if grp=='0-4': \n        d1[s]=1\n        if (s in d2)|(s in d3): \n            LEAK = True\n    if grp=='5-12': \n        d2[s]=1\n        if (s in d3): \n            LEAK = True\n    if grp=='13-22': \n        d3[s]=1\n    \n    if counter == 0:\n        print(sample_submission.head())\n        print(test.head())\n        print(test.shape)\n        \n    ## users make predictions here using the test data\n    if LEAK:\n        sample_submission['correct'] = 1\n    else:\n        sample_submission['correct'] = 0\n    \n    ## env.predict appends the session+level sample_submission to the overall\n    ## submission\n    env.predict(sample_submission)\n    counter += 1","metadata":{"papermill":{"duration":0.337707,"end_time":"2022-06-03T21:13:10.798069","exception":false,"start_time":"2022-06-03T21:13:10.460362","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-02-11T01:29:51.885028Z","iopub.execute_input":"2023-02-11T01:29:51.885729Z","iopub.status.idle":"2023-02-11T01:29:51.991790Z","shell.execute_reply.started":"2023-02-11T01:29:51.885693Z","shell.execute_reply":"2023-02-11T01:29:51.990573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## the end result is a submission file containing all test session predictions\n! head submission.csv","metadata":{"papermill":{"duration":0.767504,"end_time":"2022-06-03T21:13:11.572788","exception":false,"start_time":"2022-06-03T21:13:10.805284","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-02-11T01:29:51.993985Z","iopub.execute_input":"2023-02-11T01:29:51.994398Z","iopub.status.idle":"2023-02-11T01:29:53.038793Z","shell.execute_reply.started":"2023-02-11T01:29:51.994363Z","shell.execute_reply":"2023-02-11T01:29:53.037564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}