{"cells":[{"metadata":{"_uuid":"00a703deab625f80a35f1121f3ab159b256dec15"},"cell_type":"markdown","source":"The notebook https://www.kaggle.com/alijs1/target-variable-some-interesting-insights and discussion https://www.kaggle.com/c/avito-demand-prediction/discussion/58948 inspired me to look more into the target variable. Below are my findings."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"1e20cf5d56eab24869b296f5bdfac981c24cdbe0"},"cell_type":"code","source":"import numpy  as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.rcParams[\"figure.figsize\"] = (20, 7)","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"c2131d801860805725290b41cf3892efe9e83a96"},"cell_type":"markdown","source":"## Some data preparation"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.csv\")","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"befbfa8600d1aa443ad183df0428dbab6cb8fbf7"},"cell_type":"markdown","source":"Let's set `na` params to a blank string value. This way they'll be included in `groupby`."},{"metadata":{"trusted":true,"_uuid":"949c37bb3cf622b078ec714f1b961aca0a239336"},"cell_type":"code","source":"params = [\"param_1\", \"param_2\", \"param_3\"]\n\ndf[params] = df[params].fillna(\"\")\ndf[params].isnull().any()","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"d20fa9497b7dcc08f56aa4ad9192ab0b732baebf"},"cell_type":"markdown","source":"Next, let's do the group by counts. Note the `item_types` variable here as it will be something we'll use later."},{"metadata":{"trusted":true,"_uuid":"a21d1d60b74c0db4a1521ab4ec90d95cad0557ef"},"cell_type":"code","source":"item_types = [\"parent_category_name\", \"category_name\"] + params\n\ndfg = df.groupby(item_types + [\"deal_probability\"]) \\\n        .size() \\\n        .reset_index(name=\"count\")\ndfg.head()","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"7af307b295a3d8d96559c14f15f4ec76dfa62403"},"cell_type":"markdown","source":"## Verifying services `param_2` binning\n\nThe notebook mentioned above gave us a quick look to see the pattern. In this section, let's take a closer look. "},{"metadata":{"trusted":true,"_uuid":"a712368b9befbd41b31ffce34f44c52a0396b740","collapsed":true},"cell_type":"code","source":"services = dfg[dfg[\"parent_category_name\"] == \"Услуги\"].copy()","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"2cc51b7a91f417571184b36a230687744b54faa2"},"cell_type":"markdown","source":"First let's calculate the diffs in deal probabilities by item types. This way we can see the pattern without doing mental subtractions."},{"metadata":{"trusted":true,"_uuid":"1582238ffa3d669bbf4ee7580decf85532286e85"},"cell_type":"code","source":"services[\"deal_probability_diff\"] = services.groupby(item_types)[\"deal_probability\"].diff()\nservices[services[\"param_2\"] != \"\"].head(n=15)","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"88fb83d905b02c8210d7b161a93ee5d9b0466af9"},"cell_type":"markdown","source":"For each item type, let's take the:\n* `dp_diff` - mean of `deal_probability_diff`. this should just be equal to all the `deal_probability_diff` in that group, ignoring `np.nan` and rounding artifacts\n* `N` - number of unique deal probabilities. ie, this is the number of bins alijs was talking about"},{"metadata":{"trusted":true,"_uuid":"83f48b79bca00007c92d606374e0e3de5f31858a"},"cell_type":"code","source":"dpdiff_stats = services.groupby(item_types)[\"deal_probability_diff\"].agg([\"mean\", \"size\"]).reset_index()\ndpdiff_stats.rename(columns={\"size\": \"N\", \"mean\": \"dp_diff\"}, inplace=True)\ndpdiff_stats[dpdiff_stats[\"param_2\"] != \"\"]","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"1e80cf64bf82147446582c6c48cfd386ec9f0b8f"},"cell_type":"markdown","source":"We see that for the most part `dp_diff = 1 / (N - 1)` which is what we expect.\n\nThere are cases where N is lower that expected. Below is an example when the true N is 8 (`(1 / 0.14286) + 1`)."},{"metadata":{"trusted":true,"_uuid":"e3845cf3674ae75da7a7019cd59e1e6deb2242a3"},"cell_type":"code","source":"dpdiff_stats[dpdiff_stats[\"param_2\"] == \"Ремонт часов\"]","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"d650765ffde52722282a30b03220df4cfdb809cf"},"cell_type":"markdown","source":"I wonder why `N=8` was chosen when not all bins are used. Maybe it makes sense when also counting the items in `test.csv` (or even the active files)?"},{"metadata":{"trusted":true,"_uuid":"76c28ca64c2dfa96095c6a2036cd5f66a9fa36dc"},"cell_type":"code","source":"dfg[dfg[\"param_2\"] == \"Ремонт часов\"]","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"db9d98b90f8e31ce852c0b9ece520193678711a0"},"cell_type":"markdown","source":"## What about the service items with missing param_2?\n\nSo we saw that `deal_probability` in services were binned by `param_2`. However, there are service items with blank `param_2`.\n\nIn this section, we'll see how those are binned."},{"metadata":{"trusted":true,"_uuid":"8fe9a027b37e05e829cb7de0904790a9102f7a27"},"cell_type":"code","source":"dpdiff_stats[dpdiff_stats[\"param_2\"] == \"\"]","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"e9a0370e309d22c7455f2e9a33e3558e0a05fff1"},"cell_type":"markdown","source":"The answer is they are binned by `param_1`. And if that is absent, then they are binned by `category_name` instead. This is consistent with binning with `param_2` because `param_3` is absent.\n\nThe next question is: \"Does this hold for other parent categories?\""},{"metadata":{"_uuid":"eba2b1b625252c48aa0e5c5edd5752994a8b73dd"},"cell_type":"markdown","source":"## Looking at the whole training data\n\nSo far we've only been looking at the services parent category. Time to check the rest.\n"},{"metadata":{"_uuid":"f11d3ae98998fac3ea822d93d23791e0b5f6c48b"},"cell_type":"markdown","source":"### deal_probability_diff"},{"metadata":{"_uuid":"b4caef092feb93d3cb9c15a21fed0b06df191d62","trusted":true},"cell_type":"code","source":"dfg[\"deal_probability_diff\"] = dfg.groupby(item_types)[\"deal_probability\"].diff()\ndfg.head()","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"70c60c7410bf44c128f798091efaa19d3431846a"},"cell_type":"markdown","source":"That doesn't look promising."},{"metadata":{"_uuid":"db15282431d397b47beae0fe4f7aee82e0566689"},"cell_type":"markdown","source":"### dp_diff_stats"},{"metadata":{"trusted":true,"_uuid":"e924b4580e175531f4d712a8a210499ce4552762"},"cell_type":"code","source":"dp_diff_stats = dfg.groupby(item_types)[\"deal_probability_diff\"].agg([\"mean\", \"size\"]).reset_index()\ndp_diff_stats.rename(columns={\"size\": \"N\", \"mean\": \"dp_diff\"}, inplace=True)\ndp_diff_stats.head(n=15)","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"396b9bc891ca87c0416ea3926f7c5dde10293c89"},"cell_type":"markdown","source":"Two observations:\n* There are much more bins here.\n* `dp_diff` and `N` don't seem to match\n\nIt could be that what we observed for services don't apply to other parent_categories. Or, as we've seen before, we're not seeing a lot of the bins and the N we counted is less than the true N. Let's proceed as if the pattern in services do apply to other parent_categories.\n\n### expected_N\n\nLet's calculate:\n* `expected_df_diff` - the `dp_diff` that we expect given the `N` that we see\n* `expected_N` - the `N` that we expect given the `dp_diff` that we see"},{"metadata":{"trusted":true,"_uuid":"574af5afa1f6340b142e7712dc845d2180fd88bc"},"cell_type":"code","source":"dp_diff_stats.dropna(inplace=True)\ndp_diff_stats[\"expected_df_diff\"] = 1 / (dp_diff_stats[\"N\"]  - 1)\ndp_diff_stats[\"expected_N\"] = (1 / dp_diff_stats[\"dp_diff\"]) + 1\ndp_diff_stats.head(n=15)","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"b7cd06dc6a375a9a41017a382031bbed57ff79b5"},"cell_type":"markdown","source":"Our `expected_N` is always greater than or equal to `N`."},{"metadata":{"trusted":true,"_uuid":"b28fe851ad9286d701f795f6f7769803e52f58cc"},"cell_type":"code","source":"(dp_diff_stats[\"expected_N\"] >= dp_diff_stats[\"N\"]).all()","execution_count":21,"outputs":[]},{"metadata":{"_uuid":"ba717d48e7c4f75a596b7a02ff0c1f6a075d6d7c"},"cell_type":"markdown","source":"This means that a lot of times not all bins are used and `N` should be larger that we counted."},{"metadata":{"_uuid":"b2124a0011af0ac4d26bcd10f2c9acd960faa924"},"cell_type":"markdown","source":"## Calculating true N for an example\n\nTo explain further what I was trying to say in the previous section, let's look at an example."},{"metadata":{"trusted":true,"_uuid":"3cc337298a91ccac480ccad3bdd9648ee0c0b489"},"cell_type":"code","source":"example = dfg[(dfg[\"category_name\"] == \"Игры, приставки и программы\") & (dfg[\"param_1\"] == \"\")]\nexample","execution_count":22,"outputs":[]},{"metadata":{"_uuid":"e426d6932ad9f08f807e2d6e01c123c39eaa6fa8"},"cell_type":"markdown","source":"We see that the diffs are inconsistent and we don't even have a `deal_probability` equal to 1!\n\nWe only see 5 bins here. Clearly, that's not the true N. Let's try to work out the true N. We can do this by trying out different N values and calculating their bins until we find something that matches the bins we already see.\n\n### Calculating bins from N\n\nSo, first let's have a way to calculate bins given N:"},{"metadata":{"trusted":true,"_uuid":"f80cd60dd49367210f624883e7aee10dbb8ca915"},"cell_type":"code","source":"def calculate_bins(N):\n    return [\n        round(n / float(N - 1), 5)\n        for n in range(N)\n    ]\n\ndef print_bins(N):\n    print(f\"For N={N}: {calculate_bins(N)}\")\n\nprint_bins(5)","execution_count":23,"outputs":[]},{"metadata":{"_uuid":"460e6511aded13e9e04bbe35da53bc88a9b12ed2"},"cell_type":"markdown","source":"### Try N=7 and N=8\n\nOkay so what N values do we try? Let's start with the `expected_N` in `dp_diff_stats`"},{"metadata":{"trusted":true,"_uuid":"bb6482f8dfb48b5b6816515c9be02256dfff8b0e"},"cell_type":"code","source":"dp_diff_stats[\n    (dp_diff_stats[\"category_name\"] == \"Игры, приставки и программы\") &\n    (dp_diff_stats[\"param_1\"]       == \"\")\n]","execution_count":24,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"054263feedec6cbbd1d8db0e08f3639ee3096909"},"cell_type":"code","source":"print_bins(7)\nprint_bins(8)","execution_count":25,"outputs":[]},{"metadata":{"_uuid":"fdd0c1d7dc74886a1d07f847b6d1ae5c928c2452"},"cell_type":"markdown","source":"Not very helpful. Those aren't enough bins. We need to be smarter than this.\n\n### GCD / LCM\n\nWhat we're trying to achieve feels like a GCD / LCM problem. Let's try to frame the problem this way.\n\nSay we're getting multiples of 25 from 0 to 100. That would be [0, 25, 50, 75, 100]. Those multiples are like bins for N = 5 and diff = 25.\n\nWhat if we're given a series of multiples but with missing values [12, 20, 48, 84, 96, 100]. We don't even know how many missing values there are and where they're situated in the series. How do we go about finding them?\n\nTo start with, we know the series always starts with 0 and end at 100 so now we have [0, 12, 20, 48, 84, 96, 100].\n\nAh! We just have to find the gcd of the whole list!"},{"metadata":{"trusted":true,"_uuid":"3bf6f3e2db8af0574ee6edf6ac3438a7221e4bc0"},"cell_type":"code","source":"import math\n\ndef gcd_arr(arr):\n    diff = arr[0]\n    for num in arr[1:]:\n        diff = math.gcd(diff, num)\n    return diff\n\ndiff = gcd_arr([0, 12, 20, 48, 84, 96, 100])\ndiff","execution_count":213,"outputs":[]},{"metadata":{"_uuid":"3234522e4afac597d822f93642f81a3966b93f77"},"cell_type":"markdown","source":"From diff, we can now calculate N and generate the whole series."},{"metadata":{"trusted":true,"_uuid":"642e6221130c86ee5aefeeb294ca89879226ef05"},"cell_type":"code","source":"%pprint\nprint(f\"N = {int(100 / diff)}\")\nprint(f\"Series: {[ num for num in range(0, 101, diff)]}\")","execution_count":222,"outputs":[]},{"metadata":{"_uuid":"bfdc1625658338e87fe7b03c3d6bc48dedfb660e"},"cell_type":"markdown","source":"Okay so now how do we apply this to our original puzzle.\n\n### Going back\n\nThis will be hard as `deal_probability` is a float and most probably have been rounded. We can't easily find the GCD for them.\n\nActually, scratch that. I hope you enjoyed the last subsection's math lesson because I won't be using it and instead I'll just brute force for N:"},{"metadata":{"trusted":true,"_uuid":"52dd3f0714a3026bd00d0ed39c34d971c3ef44d5"},"cell_type":"code","source":"from tqdm import tqdm\n\ndef brute_force_N(probs):\n    for N in tqdm(range(2, 100000)):\n        bins = calculate_bins(N)\n        \n        # this `in` comparison could fail because of rounding issues\n        # use math.isclose instead?\n        if all([ prob in bins for prob in probs]):\n            return N\n\nbrute_force_N([0, 0.06322, 0.18389, 0.34615, 0.55800, 0.76786])","execution_count":46,"outputs":[]},{"metadata":{"_uuid":"6f414fb98d383a1b3613a8332c7d7f2250b39c54"},"cell_type":"markdown","source":"That's a large N.\n\nSo there are 2 possibilities we must consider:\n1. The binning pattern we saw in services does not apply to other parent categories\n2. The binning pattern we saw in services does apply but other parent categories just have large N and we are not seeing most of the bins\n\nI'm not sure which one is correct."},{"metadata":{"_uuid":"5e635bad28e4721c4f35a3e30bc00e22b9eff3f0"},"cell_type":"markdown","source":"## Other observations\n\nFor extra points"},{"metadata":{"trusted":true,"_uuid":"5ff210b43e904b0bc4697d762ac9b9ec2f2e6b70","collapsed":true},"cell_type":"code","source":"aggs = []\nfor col in [\n    \"region\",\n    \"city\",\n    \"parent_category_name\",\n    \"category_name\",\n    \"param_1\"\n]:\n    agg = df.groupby(col)[\"deal_probability\"].agg([\"nunique\", \"count\"]).reset_index(drop=True)\n    agg[\"column\"] = col\n    aggs.append(agg)\n\naggs = pd.concat(aggs)\naggs.head()","execution_count":40,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"455a0f94df0bfe5fd2cdfbff65d9c31f50713ba2","collapsed":true},"cell_type":"code","source":"sns.lmplot(\n    data=aggs,\n    x=\"count\", y=\"nunique\", col=\"column\",\n    fit_reg=False, sharex=False, sharey=False\n);","execution_count":41,"outputs":[]},{"metadata":{"_uuid":"b8a9d0491fad43fdefb2e035f164ab3cc89e7178"},"cell_type":"markdown","source":"For location-based columns, there is a visible relationship between the number of items and the number of unique `deal_probability` values. The more items, the more likely a city/region is going to have item types with more bins I guess.\n\nNo relation is seen for columns regarding the item's type. From what we know, number of unique `deal_probability` values is related to the number of subcategories it has.\n\n"},{"metadata":{"_uuid":"754bd5fbb561addabec89d342bd45dcdc233aec5"},"cell_type":"markdown","source":"Next we look at region's item count vs unique `deal_probability` count scatterplot again but this time we separate for each `parent_category_name`."},{"metadata":{"trusted":true,"_uuid":"a1a8388141632f8f08766057c7e386206e8442d3","collapsed":true},"cell_type":"code","source":"agg = df.groupby([\"region\", \"parent_category_name\"])[\"deal_probability\"].agg([\"nunique\", \"count\"]).reset_index()\nagg.head()","execution_count":57,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"900c7ee8c9e8858ebf6b51a92c79a190ad16cab2","collapsed":true},"cell_type":"code","source":"sns.lmplot(\n    data=agg,\n    x=\"count\", y=\"nunique\", col=\"parent_category_name\",\n    fit_reg=False, sharex=False, sharey=False\n);","execution_count":61,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"0d2ae88648ed4ea2926c7233ec898b102a460b14"},"cell_type":"markdown","source":"Trending up is expected but what's with that 2nd to the last graph?"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"ef995339eaee3369fd3875fbd939bf3c39aaf37f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"bf310ecd5be0270bd0a069bce9f97eecae6f16ad"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c29c0ec98dd475118f0d87ff4770f81762d8e8f7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"0c06559e27a2990f6b5a9b3ba8b62303241cb6df"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"5ddeb28b388cef7828937201fb8e0aa82b1aae0b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"0c5cafec252dadafb05e0f1053f89bb72a06a006"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"4b6f0dbcf1c43ba91f3eb0801a594f15e9f94388"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"9abcebd99bdd434c3ac2d8d98a20371df401184c"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"fe9e5e7d4592fea8c05d1fc0a3501e94abc9f440"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"0f239fe91643a7dab42e6fe7a42ec0d0e6040ec5"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"a83628efde15d61bbe52ec10eb5f7584cb534775"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"6d646a1dc4b20aac0bae7e625525bd49214f18c7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"6dff6f4142ca108f8259781d779e3d2d5e3b1bfa"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}