{"cells":[{"metadata":{},"cell_type":"markdown","source":"When you press \"Commit\", you execute the code with **public** test dataset. But when you hit \"Submit\", in the background, the same code is run against **private** test dataset. \n\nThus we can find out more about the private (hidden) test set. Which actually defines the prizes.  \n\nThe approach is straightforward:\n - ask a binary question about the test dataset (eg. whether it's longer than 1000)\n - if the condition doesn't hold - you submit the sample submission file and get zero on Public LB\n - if the condition holds - you make the script fail (raise an error). Thus after submitting you'll see an error and conclude that that binary condition holds for the hidden test dataset. "},{"metadata":{"trusted":false},"cell_type":"code","source":"import sys\nimport pandas as pd ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"code","source":"sample_sub = pd.read_csv('../input/tensorflow2-question-answering/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sample_sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test_df = pd.read_json('../input/tensorflow2-question-answering/simplified-nq-test.jsonl',\n                      lines=True, orient='records')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this example we'll ask whether the hidden test data set is longer than 1000. "},{"metadata":{"trusted":false},"cell_type":"code","source":"if len(test_df) >= 1000:\n    raise ValueError(\"We'll never see this message again\")\nelse:\n    sample_sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When you **Commit** this notebook, the condition doesn't hold (for **public** 692-long test dataset).\nBut when you **Submit**, you'll actually see the Kernel fail. Thus we conclude that the **private** test dataset is longer that 1000."},{"metadata":{},"cell_type":"markdown","source":"Actually, you can spend a lot of submissions, getting one bit at a time :)\n - Is the median number of long answer candidates larger than the same in the training set?\n - Are questions longer that some threshold?\n - Are there more paragraphs in the hidden test set?\n - etc.\n\nGood luck! Do share your findings with the community! And bear in mind that you still need to be smart and train cool models to win. \n\nPS. Yes, this is close to cheating :) but considered fine for Kaggle competitions I guess.\n\n<img src=\"https://habrastorage.org/webt/ip/fa/wk/ipfawkx9ate-z15kmh6mjevd87e.jpeg\" width=60% />"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}