{"cells":[{"metadata":{"_uuid":"eafc803e3323023c2f6d4e42c7d7db7c2c60e0d1"},"cell_type":"markdown","source":"# Submission distributions\n\nOne of the interesting things I've noticed about this competition is how different my prediction of f1 score is from the LB score, in this kernel I wanted to examine this a bit further\n\n**TL; DR:** the training set and test set have slightly different distrubutions of target values. You should take this into account when you train your model"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb0d783c0e343566cfbf668c7f23b2df7492bc4d"},"cell_type":"code","source":"from sklearn.metrics import f1_score","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bc0b4fe81c9d561121daaf2cd7043b7e464ad490"},"cell_type":"code","source":"df_train.target.value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"963117b86a0550f1b35e54b0a85ed9446b78a061"},"cell_type":"code","source":"len(df_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"364dc73b4c42b8b6fe9c85cac9999aa5d23b6682"},"cell_type":"markdown","source":"First, lets do the dumbest thing and label everything as positive in our training set, then see what f1 score we get"},{"metadata":{"trusted":true,"_uuid":"74f203be4ddfde40a794a077ff10135a6c2645bb"},"cell_type":"code","source":"f1_score(df_train.target.values, np.ones(len(df_train)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b80057c5ba6dbebf3dc52fe2fc2fce978e6c8b8"},"cell_type":"markdown","source":"Let's now do the same for the test set. I submitted this in version 1 of this kernel"},{"metadata":{"trusted":true,"_uuid":"9056a01c58374a7ae60deecee3d3341b35a798ab"},"cell_type":"code","source":"df_test['prediction'] = 1\ndf_test = df_test[['qid', 'prediction']]\n\ndf_test.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e35ebc67421f9b62811b8acefc4a503e153c04e0"},"cell_type":"markdown","source":"This gets a score of `0.113` - which obviously means the distribution of target values is different between the test and training sets, but how different?\n\nwell since the f1 score is defined as\n\n$$f1 = {{2 * precision * recall}\\over{precision + recall}}$$\n\nwhich is the same as\n\n$$ f1 = {{2 TP}\\over{2TP + FP + FN}}$$\n\nbecause we labelled everything as positive, we know $FP = \\sum - TP$ (where $\\sum$ is the total number of results) and $FN = 0$, this becomes\n\n$$ f1 = {{2 TP}\\over{TP + \\sum}}$$\n\nafter some rearraging we get\n\n$${TP\\over{\\sum}} = {{f1}\\over{2 - f1}}$$\n\nie we can relate the ratio of positive labells to the f1 score\n\nDoing this we get"},{"metadata":{"trusted":true,"_uuid":"6fbb4f18ce3f6a885d49eb8654635316578dcf58"},"cell_type":"code","source":"positive_ratio = 0.113 / (2 - 0.113)\n\nprint(positive_ratio)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d300c20987d4ba5e8c8741357424a0209ccb363"},"cell_type":"markdown","source":"the training set positive ratio is `0.06187` as shown above\n\nThis isn't a huge difference, but if there's a similar difference between the test set and the private test set it could really mix up the leaderboard!\n\n\n## Appendix: sanity check :)"},{"metadata":{"trusted":true,"_uuid":"d38b258222a68281d9adc873f69711abe2eb6ee8"},"cell_type":"code","source":"0.11653 / (2 - 0.11653)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ffaf41a1aa2088e23571841ba311a56280e5c68"},"cell_type":"markdown","source":"phew :)"},{"metadata":{"trusted":true,"_uuid":"80f4743c3bddc234bbecdbcc20cf5fec828cc542"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}