{"metadata": {"kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}, "language_info": {"pygments_lexer": "ipython2", "mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.12", "codemirror_mode": {"version": 2, "name": "ipython"}, "file_extension": ".py", "name": "python"}}, "cells": [{"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "# Real test set size estimation\n\nIn this notebook I'm going to try to estimate the real size of the test set. We know the followign facts:\n\n> To deter hand labeling, we have supplemented the test set with car images that are ignored in scoring.\n\nAnd also in the leaderboard says the following:\n\n> This leaderboard is calculated with approximately 25% of the test data.\nThe final results will be based on the other 75%, so the final standings may be different.", "metadata": {"_uuid": "8e399c6b234ed1fbe42587c0c9354fbc4017cc0a"}}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "## Strategy\nI have made a submission using a mask that covers the full image. The score was 0.361.\n\nNow I'm going to make random submissions setting some of the masks to zero and leaving only 20% of the full masks. I will collect the scores and hopefully that will have information to estimate the size of the real test set.", "metadata": {"_uuid": "47e95b609295aee39534d336796c98c456da89c0"}}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "## Simulating the experiments\nFirst of all let's make simulations to see if different test sizes will produce different score distributions.", "metadata": {"_uuid": "e2f3ac1d6b57d47a1f01540f7652a804f2d5b0cf"}}, {"metadata": {"_uuid": "134af5aafd8fdd10b3b510f7ae5ac92dfdf26087", "collapsed": true}, "cell_type": "code", "outputs": [], "source": "import numpy as np\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nimport seaborn as sns\n\n%matplotlib inline", "execution_count": 3}, {"metadata": {"_uuid": "19b6750805b363003953d6baa3b6cfcd0ac5c70d", "collapsed": true}, "cell_type": "code", "outputs": [], "source": "def simulate_n_submissions(n_submissions, public_ratio, full_mask_ratio, score_scale=0.361):\n    \"\"\"\n    Simulates the submissions that we are performing on the public leaderboard\n    \n    Parameters\n    ----------\n    n_submissions : int\n        Number of submissions to be made\n    public_ratio : float\n        Fraction of the test set that belongs to public leaderboard\n    full_mask_ratio : float\n        Fraction of the submission that will use full_mask\n    score_scale : float\n        A factor for scaling the score. Typical value is 0.361, which is the score for using\n        full mask on all the submission instances\n    \"\"\"\n    # Create the synthetic test set\n    test_size = int(1e5)\n    test_set = np.concatenate((np.ones(int(test_size*public_ratio)), \n                               np.zeros(int(test_size*(1-public_ratio)))))\n    test_set /= np.sum(test_set)\n    # Do the submissions and collect the scores\n    score_list = []\n    for _ in tqdm(range(n_submissions)):\n        sampling_mask = (np.random.rand(test_size) < full_mask_ratio).astype(np.int)\n        score = np.sum(sampling_mask*test_set)*score_scale\n        \n        score_list.append(score)\n    return score_list", "execution_count": 4}, {"metadata": {"_uuid": "b67108e75db644aff6c827b1f5a725e0c0febdfe"}, "cell_type": "code", "outputs": [], "source": "plt.figure(figsize=(12, 6))\nscore_list = []\np_range = [0.01, 0.05, 0.5]\nlabels = ['test set public ratio= %.2f' % i for i in p_range]\nfor p in p_range:\n    score_list.append(simulate_n_submissions(1000, public_ratio=p, full_mask_ratio=0.2, score_scale=0.361))\n    \nfor values, label in zip(score_list, labels):\n    sns.distplot(values, label=label)\nplt.xlabel('Submission score')\nplt.legend();", "execution_count": 9}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "The plot above shows that there are big differences in the score distributions if the real test size changes. The bigger the test size the narrower the distribution.", "metadata": {"_uuid": "0ee524d36e5552fb78a09c0a69528e25e56d4854"}}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "## Comparing the real data with the simulations\nI have made 25 random submissions, let's compare the distribution of my submission with the simulated scores.", "metadata": {"_uuid": "46ff57aec97665f6daf43c5a7286134f6e096f37"}}, {"metadata": {"_uuid": "a39be5601dd7ed3e53fe364946a1a163a7d9af8f", "collapsed": true}, "cell_type": "code", "outputs": [], "source": "submission_scores = [0.075, 0.074, 0.068, 0.076, 0.069, 0.075,0.07,0.071,0.076,0.071,0.071,0.07,0.073,0.075,0.076,0.065,0.072,0.077,0.063,0.074,0.067,0.068,0.065,0.069]", "execution_count": 8}, {"metadata": {"_uuid": "0c20f06f86b28a2292b274ac832bbf16df95df86"}, "cell_type": "code", "outputs": [], "source": "plt.figure(figsize=(12, 6))\nscore_list = [submission_scores]\np_range = [0.01, 0.05, 0.5]\nlabels = ['real submission'] + ['test set public ratio= %.3f' % i for i in p_range]\nfor p in p_range:\n    score_list.append(simulate_n_submissions(1000, public_ratio=p, full_mask_ratio=0.2, score_scale=0.361))\n    \nfor values, label in zip(score_list, labels):\n    sns.distplot(values, label=label)\nplt.xlabel('Submission score')\nplt.legend();", "execution_count": 10}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "The above plot is showing that the submission distribution is very similar from having a test set public ratio of 0.5 and 0.05.\n\nLet's try smaller values, and get more samples for higher precision.", "metadata": {"_uuid": "0f13c4304a415ceb7526be8844f0668e46058831"}}, {"metadata": {"_uuid": "fa87e503485322afa75b573f335aa7c6437b8cfd"}, "cell_type": "code", "outputs": [], "source": "plt.figure(figsize=(12, 6))\nscore_list = [submission_scores]\np_range = [ 0.005, 0.01, 0.02]\nlabels = ['real submission'] + ['test set public ratio= %.3f' % i for i in p_range]\nfor p in p_range:\n    score_list.append(simulate_n_submissions(10000, public_ratio=p, full_mask_ratio=0.2, score_scale=0.361))\n    \nfor values, label in zip(score_list, labels):\n    sns.distplot(values, label=label)\nplt.xlabel('Submission score')\nplt.legend();", "execution_count": 14}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "The most similar distribution is the one that has a public of 0.01.  \n**This means that only 1% of the test set belongs to the public set.**", "metadata": {"_uuid": "592f0fff3d90c568b978856948b7f48cf37bcb56"}}, {"cell_type": "markdown", "outputs": [], "execution_count": null, "source": "## Reflexions\nFrom my experience in other competitions the test set usually is similar in size to the train set.\n\nIn this competition we have 5.088 images on train and 100.064 on test. If the real test set has the same size of the train set and the public test set is 25% of the total test set we get that the ratio between public test set and total test set is 1.27%. This is very close to our estimate. \n\n** So I think that the real test set has the same size as the train set. And this means that 95% of the test set won't be used.\n**", "metadata": {"_uuid": "ca360d60cfc400b0f0f3f7ebba6b1f0f7e88b99f"}}], "nbformat": 4, "nbformat_minor": 1}