{"nbformat_minor": 1, "nbformat": 4, "cells": [{"cell_type": "markdown", "source": ["On this notebook I will create submissions with all the labels equal to a category to check the distribution of the labels in the public test set."], "metadata": {"_cell_guid": "070227e6-16b9-4936-85dd-b5a7fd1033ba", "_uuid": "f0a1e0514050296a5722a05fe68ae4e9e2cffe76"}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt \n", "import seaborn as sns\n", "import plotly.graph_objs as go\n", "\n", "%matplotlib inline"], "metadata": {"_cell_guid": "74577e88-3ea4-4687-9279-53ec95414151", "_uuid": "07f9a353e30d3f657f1782415bde554b74ba4211"}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["submission = pd.read_csv('../input/sample_submission.csv')"], "metadata": {"_cell_guid": "66425e32-a097-44b5-adee-d2d3659f7990", "_uuid": "3462a446bea405c4d79ffe454c1ff426065eb84d", "collapsed": true}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["label_names = ['yes', 'no', 'up', 'down', 'left', 'right', \n", "               'on', 'off', 'stop', 'go', 'silence', 'unknown']\n", "len(label_names)"], "metadata": {"_cell_guid": "2775c07c-a3b9-45f1-81e6-d27796d99e6c", "_uuid": "5b106dcd9bd107adbc4c0b248cc42bc524817cb0"}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["new_submission_path_list = []\n", "for label in label_names:\n", "    submission.label = label\n", "    new_submission_path = 'all_%s.csv.gz' % label\n", "    new_submission_path_list.append(new_submission_path)\n", "    submission.to_csv(new_submission_path, index=False, compression='gzip')"], "metadata": {"_cell_guid": "ad778190-be87-4f18-addf-4076d83c7802", "_uuid": "aeb1ad37e366227eb55df7536cc0cf6d421ef0ae", "collapsed": true}}, {"cell_type": "markdown", "source": ["Let's check that the files have been correctly created."], "metadata": {"_cell_guid": "a37371ae-94e5-4e6c-83d8-5d991cc03322", "_uuid": "40a88560ae0bb1466aa5bad8868bc2a2fe4f2dc5"}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["for label, new_submission_path in zip(label_names, new_submission_path_list):\n", "    print(new_submission_path)\n", "    submission = pd.read_csv(new_submission_path)\n", "    unique_labels = submission.label.unique()\n", "    assert len(unique_labels) == 1\n", "    assert unique_labels[0] == label\n", "    print(submission.label.unique())\n", "    print()"], "metadata": {"_cell_guid": "a3591c9c-834e-4c1a-a6be-af1beb1bd205", "_uuid": "0fc3d59143a75375119c6be82b889fc4bdb4c43a", "collapsed": true, "scrolled": true}}, {"cell_type": "markdown", "source": ["Now I have to make the submisions and save the scores, let's prepare a dictionary for saving the scores."], "metadata": {"_cell_guid": "f9462aef-099a-4c5c-942f-1258a50c95f9", "_uuid": "6676ba961b173b1ac0ff7a2d80bf35628f14218c"}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["score_dict = {label:None for label in label_names}\n", "score_dict"], "metadata": {"_cell_guid": "33eab47c-4a01-4088-8ef5-57b045386bbd", "_uuid": "7cdd84e24b4c20e7aed8efdc008cf836e6bfbe81", "collapsed": true}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["score_dict = {\n", "    'down': 0.06,\n", "    'go': 0.08,\n", "    'left': 0.08,\n", "    'no': 0.07,\n", "    'off': 0.07,\n", "    'on': 0.08,\n", "    'right': 0.07,\n", "    'silence': 0.09,\n", "    'stop': 0.08,\n", "    'unknown': 0.09,\n", "    'up': 0.08,\n", "    'yes': 0.08\n", "}"], "metadata": {"_cell_guid": "777f4118-da0f-4b73-9d25-af414ff1331b", "_uuid": "f6601fbbf83dcb03c2d96d8e3abcbf8be98ac1ae", "collapsed": true}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["label_scores = [score_dict[label] for label in label_names]\n", "np.sum(label_scores)"], "metadata": {"_cell_guid": "e367f01f-651e-4c7c-b691-cb8f2ed6e5d7", "_uuid": "7d13ad04b25b86e32624e014b12864df38a87a21"}}, {"cell_type": "markdown", "source": ["Quite a lot information is loss because of the truncation of the score. There are 12 labels and the mean expected loss is 0.5, so 12*0.5=6 ~ 7. It seems reasonable"], "metadata": {"_cell_guid": "53c503e4-02b4-43a5-b7ac-15cf8e4772e7", "_uuid": "b384e7ec7c12dbd12749c8848990eca5bb5df9b9"}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["plt.figure(figsize=(15, 6))\n", "plt.title('Frequency of the labels on the public test set')\n", "plt.bar(np.arange(len(label_scores)), label_scores, tick_label=label_names);"], "metadata": {"_cell_guid": "96e9c306-d7ff-4b7f-bfe8-2d2687cb859b", "_uuid": "1112bd1ef84e85357568243fae6c27cac5790cef"}}, {"cell_type": "markdown", "source": ["We can see that there are differences between the labels. This could be caused by random splitting of the test set, and if that is the cause maybe it wil be possible to estimate the real size of the test set.\n", "\n", "However I think that the differences are small, so I don't think there's need for oversampling the classes during training."], "metadata": {}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": ["plt.figure(figsize=(15, 6))\n", "sns.distplot(label_scores)\n", "plt.title('Frequency histogram')\n", "plt.xlabel('Frequency');"], "metadata": {}}, {"outputs": [], "execution_count": null, "cell_type": "code", "source": [], "metadata": {"collapsed": true}}], "metadata": {"kernelspec": {"display_name": "Python 3", "name": "python3", "language": "python"}, "language_info": {"name": "python", "pygments_lexer": "ipython3", "version": "3.6.3", "mimetype": "text/x-python", "nbconvert_exporter": "python", "file_extension": ".py", "codemirror_mode": {"name": "ipython", "version": 3}}}}