{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "802f857e-9c60-4ed0-8c80-26e928368916"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.cm as cm\n",
        "import os\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.cross_validation import KFold\n",
        "from sklearn.metrics import log_loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "27bec405-98d2-4546-acfa-046232905675"
      },
      "source": [
        "# Phone brand and device model based benchmarks\n",
        "\n",
        "The task in this competition is to predict a user's age and gender group using information about their mobile phone model and usage patterns.\n",
        "\n",
        "Since many of the devices have no events data it's interesting to know how much information we can extract from phone brand and device model only. This notebook contains some benchmark approaches of inferring age and gender from data contained in `phone_brand_device_model.csv`.\n",
        "\n",
        "CV scores of different approaches obtained by 10-fold cross-validation are as follows:\n",
        "\n",
        "* 2.485 - 1/n_classes benchmark\n",
        "* 2.428 - class probabilities benchmark\n",
        "* 2.420 - predicting gender from phone brand\n",
        "* 2.417 - predicting gender from device model\n",
        "* 2.402 - predict gender-age group from phone brand\n",
        "* 2.395 - predict gender-age group from device model\n",
        "* 2.391 - mean of last two\n",
        "\n",
        "Some notes from the exploratory analysis:\n",
        "\n",
        "* Some device models could belong to different brands. So the correct way to label encode device model is to concatenate with brand first.\n",
        "* `phone_brand_device_model.csv` contains double entries for 529 device_ids. Most of these duplicate rows are identical and can be safely dropped, but six devices actually have different information in their rows. Of these 1 belongs to the train set and 5 to the test set.\n",
        "\n",
        "## Demographic data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "142a8ec5-658c-4deb-9000-0361086ec1a9"
      },
      "outputs": [],
      "source": [
        "gatrain = pd.read_csv('../input/gender_age_train.csv')\n",
        "gatest = pd.read_csv('../input/gender_age_test.csv')\n",
        "gatrain.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "96e6c39d-048f-49ef-9f71-820a8f73f927"
      },
      "source": ""
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7e9a51f4-06f4-4013-a1f4-fabf1e3985f4"
      },
      "outputs": [],
      "source": [
        "gatrain.shape[0] - gatrain.device_id.nunique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fe8a7c3b-dd75-4dd0-be59-ddd182ff7935"
      },
      "outputs": [],
      "source": [
        "gatest.shape[0] - gatest.device_id.nunique()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d04ef37f-04ec-4022-9e6e-15103e4139fe"
      },
      "source": [
        "None, great.\n",
        "\n",
        "### Age and gender distributions\n",
        "\n",
        "Counts of target groups"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "076af2be-ed13-47b5-832c-7ab9432a0a35"
      },
      "outputs": [],
      "source": [
        "gatrain.group.value_counts().sort_index(ascending=False).plot(kind='barh')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7a3208d0-deb7-4ddd-acc4-31cece6e43df"
      },
      "source": [
        "Gender counts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8fd992fb-eeae-4504-858a-a4935466ff20"
      },
      "outputs": [],
      "source": [
        "gatrain.gender.value_counts().plot(kind='barh')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1de1d364-67f4-43b0-855e-f2f3c71f4229"
      },
      "source": [
        "Age distribution, lines denote target groups."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6b878b9e-3d7a-42ab-891b-3ea282404198"
      },
      "outputs": [],
      "source": [
        "c = gatrain.groupby(['age','gender']).size().unstack().reindex(index=np.arange(gatrain.age.min(), gatrain.age.max()+1)).fillna(0)\n",
        "ax1, ax2 = c.plot(kind='bar',figsize=(12,6),subplots=True);\n",
        "ax1.vlines(np.array([23,26,28,32,42])-0.5,0,1800,alpha=0.5,linewidth=1,color='r')\n",
        "ax2.vlines(np.array([22,26,28,31,38])-0.5,0,3000,alpha=0.5,linewidth=1,color='r')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "46ef2f0c-e660-4d33-a8e7-f94a444fdf71"
      },
      "source": [
        "### 1/n_classes benchmark"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "51374dea-1c7f-4935-bb48-7ee1d9b56740"
      },
      "outputs": [],
      "source": [
        "letarget = LabelEncoder().fit(gatrain.group.values)\n",
        "y = letarget.transform(gatrain.group.values)\n",
        "n_classes = len(letarget.classes_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "30427b3c-b215-4ec3-8eda-c01c1a5b19de"
      },
      "outputs": [],
      "source": [
        "pred = np.ones((gatrain.shape[0],n_classes))/n_classes\n",
        "log_loss(y, pred)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "eaedbfdd-0da5-4f04-ba02-be82844c02c7"
      },
      "source": [
        "### Class probabilities benchmark\n",
        "\n",
        "Best we can do without any features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5c349e2b-7485-48f1-8a2f-d02f2003f089"
      },
      "outputs": [],
      "source": [
        "kf = KFold(gatrain.shape[0], n_folds=10, shuffle=True, random_state=0)\n",
        "pred = np.zeros((gatrain.shape[0],n_classes))\n",
        "for itrain, itest in kf:\n",
        "    train = gatrain.iloc[itrain,:]\n",
        "    test = gatrain.iloc[itest,:]\n",
        "    ytrain, ytest = y[itrain], y[itest]\n",
        "    prob = train.groupby('group').size()/train.shape[0]\n",
        "    pred[itest,:] = prob.values\n",
        "print(log_loss(y, pred))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "46d85d13-0ddc-4650-9a26-4f5e092ab1e9"
      },
      "source": [
        "## Phone brand and model data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "660f1591-1c91-42d3-89e6-fa810a6db960"
      },
      "outputs": [],
      "source": [
        "phone = pd.read_csv('../input/phone_brand_device_model.csv',encoding='utf-8')\n",
        "phone.head(3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3c88c165-2540-43c4-b45e-a653944311aa"
      },
      "outputs": [],
      "source": [
        "print('{} rows'.format(phone.shape[0]))\n",
        "print(\"unique values:\")\n",
        "for c in phone.columns:\n",
        "    print('{}: {}'.format(c, phone[c].nunique()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6d39fce0-99e1-45f8-a224-7681c974b8c4"
      },
      "source": [
        "### Duplicate devide_ids"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "71f46e90-e13f-4940-9d01-fcb823167efe"
      },
      "outputs": [],
      "source": [
        "dup = phone.groupby('device_id').size()\n",
        "dup = dup[dup>1]\n",
        "dup.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d0b96448-a7f8-485a-99cc-81dcdc4f68ea"
      },
      "outputs": [],
      "source": [
        "dup.value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cac31443-a1f8-45e4-8197-426cc29a9cc5"
      },
      "source": [
        "529 device ids have duplicate entries in phone dataframe. Are duplicate rows identical or different?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "324e9052-7237-42de-ace7-a41ff0e8564d"
      },
      "outputs": [],
      "source": [
        "dup = phone.loc[phone.device_id.isin(dup.index)]\n",
        "first = dup.groupby('device_id').first()\n",
        "last = dup.groupby('device_id').last()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "81bcdfa3-ca86-426c-a7ff-60d2d2b72820"
      },
      "outputs": [],
      "source": [
        "diff = (first != last).sum(axis=1).nonzero()\n",
        "pd.concat((first.iloc[diff], last.iloc[diff]),axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "adbd5efc-a0dc-428d-8154-26e1f68bed17"
      },
      "source": [
        "Looks like 6 devices with duplicate rows have different values for brand and model. Are they in train or in test?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f3db0f32-2fb9-4a31-9e4b-a878f566a48f"
      },
      "outputs": [],
      "source": [
        "dev = _.index\n",
        "print(\"in train: {}\".format(sum(gatrain.device_id.isin(dev))))\n",
        "print(\"in test: {}\".format(sum(gatest.device_id.isin(dev))))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1457eba1-550b-4303-8c10-efd9e0dcfcb8"
      },
      "source": [
        "For now just drop duplicates."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f4272389-a8b6-4e23-b1c4-9acb2bafb927"
      },
      "outputs": [],
      "source": [
        "phone = phone.drop_duplicates('device_id', keep='first')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0cabd67c-ed13-48bc-b88b-787af733c0ad"
      },
      "source": [
        "### Any models that can belong to different brands?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "84edb57b-a15b-4c26-ad84-b5fe531cb7a3"
      },
      "outputs": [],
      "source": [
        "c = phone.groupby('device_model')['phone_brand'].apply(pd.Series.nunique)\n",
        "c.value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cf875c4d-148c-438a-95ef-98e2114d72bc"
      },
      "source": [
        "Some device models can belong to more than one brand. So the correct way to label-encode device models is probably to concatenate with brand first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "20a83998-ace6-4864-b331-216c04cde4c6"
      },
      "outputs": [],
      "source": [
        "lebrand = LabelEncoder().fit(phone.phone_brand)\n",
        "phone['brand'] = lebrand.transform(phone.phone_brand)\n",
        "m = phone.phone_brand.str.cat(phone.device_model)\n",
        "lemodel = LabelEncoder().fit(m)\n",
        "phone['model'] = lemodel.transform(m)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "09cc7a16-24ff-4ee7-951b-b406b035d3e5"
      },
      "source": [
        "### Brands and models popularity"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0704ff83-1f00-483d-8af3-efa98e6b6f64"
      },
      "outputs": [],
      "source": [
        "c = phone.brand.value_counts()\n",
        "s = c.cumsum()/c.sum()\n",
        "fig, (ax1,ax2) = plt.subplots(1,2,sharey=True,figsize=(10,4))\n",
        "ax1.plot(np.arange(s.shape[0])+1, s.values*100)\n",
        "ax1.set_xlim(0,30)\n",
        "ax1.set_xlabel('N most popular brands')\n",
        "ax1.set_title('% of devices');\n",
        "c = phone.model.value_counts()\n",
        "s = c.cumsum()/c.sum()\n",
        "ax2.plot(np.arange(s.shape[0])+1, s.values*100)\n",
        "ax2.set_xlim(0,400)\n",
        "ax2.set_xlabel('N most popular models')\n",
        "ax2.set_title('% of devices');"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0a534929-e6d7-4586-bf6b-a28a31b6f62d"
      },
      "source": [
        "9 most popular brands account for 90% of devices. For models the number to account for 90% of devices is 373.\n",
        "\n",
        "### Gender ratios by phone brand"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c96276b2-3675-44ed-a4da-57b24a68054d"
      },
      "outputs": [],
      "source": [
        "train = gatrain.merge(phone[['device_id','brand','model']], how='left',on='device_id')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "59df4958-0845-42e1-a6dc-40145b836564"
      },
      "outputs": [],
      "source": [
        "def plot_by(df, cat, by, perc = 0.95):\n",
        "    # Find popular categories\n",
        "    c = df[by].value_counts().cumsum()/df.shape[0]\n",
        "    take = c[c<=perc].index\n",
        "    # Pool rare categories into 'other' cat\n",
        "    gr = df[by].copy()\n",
        "    gr[~(df[by].isin(take))] = 'other'\n",
        "    # Count target classes in groups\n",
        "    c = df.groupby([gr,cat]).size().unstack().fillna(0)\n",
        "    total = c.sum(axis=1)\n",
        "    meanprobs = c.sum(axis=0).cumsum()/df.shape[0]\n",
        "    # Transform to probabilities\n",
        "    sortcol = c.columns[int(np.floor((c.shape[1]-1)/2))]\n",
        "    c = c.div(c.sum(axis=1), axis='index')\n",
        "    # Cumsum for stacked bars\n",
        "    c = c.cumsum(axis=1).sort_values(by=sortcol)\n",
        "    total = total.loc[c.index]\n",
        "    # Prepare plot data\n",
        "    left = np.array([0, *(total.iloc[:-1].cumsum().values)])\n",
        "    ticks = left + 0.5*total.values\n",
        "    colors = cm.rainbow(np.linspace(0.1,0.9,num=c.shape[1]))\n",
        "    fig, ax = plt.subplots(figsize=(10,5))\n",
        "    for (i,col) in enumerate(c.columns[::-1]):\n",
        "        height = c[col].values\n",
        "        ax.bar(left, height, total.values,label=col,color=colors[i],zorder = c.shape[0]+i)\n",
        "    for p in meanprobs.iloc[:-1]:\n",
        "        ax.hlines(p, 0, total.sum(),\n",
        "                  color='w',alpha=0.6,zorder=c.shape[0]*10,linewidth=1)\n",
        "    # legend placement needs more work\n",
        "    ax.legend(bbox_to_anchor=(1.1, 0.8),title=cat)\n",
        "    ax.set_xticks(ticks)\n",
        "    ax.set_xticklabels(total.index, rotation='vertical')\n",
        "    ax.set_xlabel(by)\n",
        "    ax.set_xlim(0,left[-1]+total.values[-1])\n",
        "    return ax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d97356da-bacc-4c4b-9947-3433398c2742"
      },
      "source": [
        "Bar widths represent group counts, ratio of genders in each group is shown. The line represents overall genders ratio."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a379b3ae-2ccb-44eb-a6a5-2fb260281b31"
      },
      "outputs": [],
      "source": [
        "ax = plot_by(train, 'gender','brand')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "65e648c6-f7dc-405c-8287-f7f0418a7b6b"
      },
      "outputs": [],
      "source": [
        "# brands preferred by female users\n",
        "lebrand.classes_[[15, 7, 13]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "74cce9c3-9b1f-4c0a-ba1b-64aa1b884d80"
      },
      "source": [
        "What about models?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "27d354f3-e3a7-42cf-8501-b982b651a7ca"
      },
      "outputs": [],
      "source": [
        "ax = plot_by(train, 'gender','model')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d296a641-8472-478e-bc26-ee89d25ea048"
      },
      "source": [
        "Way too many device models here...\n",
        "\n",
        "### Age distributions by phone brand\n",
        "\n",
        "Color = gender"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5357ece8-5f1c-49da-b165-f5e2963e4f87"
      },
      "outputs": [],
      "source": [
        "brands = train.brand.value_counts().index[:6]\n",
        "c = train[train.brand.isin(brands)]\n",
        "g = sns.FacetGrid(c, hue='gender', col='brand',col_wrap=2, aspect=2, xlim=(15,60))\n",
        "g.map(sns.distplot, 'age', bins=np.arange(15,70)+0.5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f2773207-9285-4ef0-8488-982df0d0df8d"
      },
      "source": [
        "Mostly look very similar.\n",
        "\n",
        "### Age distributions by phone model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1eb0467e-b592-456b-8492-f6f0a7d8fcb1"
      },
      "outputs": [],
      "source": [
        "models = train.model.value_counts().index[:6]\n",
        "c = train[train.model.isin(models)]\n",
        "g = sns.FacetGrid(c, hue='gender', col='model',col_wrap=2, aspect=2, xlim=(15,60))\n",
        "g.map(sns.distplot, 'age', bins=np.arange(15,70)+0.5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3b5a7d36-f12b-4a45-96f7-9e6857018c3c"
      },
      "source": [
        "So do these.\n",
        "\n",
        "### Benchmark: predict gender from phone brand"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d4c2da62-8253-4e21-8baa-9cc4e730d48f"
      },
      "outputs": [],
      "source": [
        "ptrain = gatrain.merge(phone[['device_id','brand','model']], how='left',on='device_id')\n",
        "ptrain.head(3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "31b5f19f-1f52-4bf7-a1ca-34ee08a7c144"
      },
      "outputs": [],
      "source": [
        "class GenderProb(object):\n",
        "    def __init__(self, prior_weight=10.):\n",
        "        self.prior_weight = prior_weight\n",
        "    \n",
        "    def fit(self, df, by):\n",
        "        self.by = by\n",
        "        self.label = 'pF_' + by\n",
        "        self.prior = (df['gender']=='F').mean()\n",
        "        # fit age groups\n",
        "        prob = train.groupby(['gender','group']).size()\n",
        "        pF = prob['F']/prob['F'].sum()\n",
        "        pM = prob['M']/prob['M'].sum()\n",
        "        self.agegroups = pd.concat((pF,pM),axis=0)\n",
        "        # fit gender probs by grouping column\n",
        "        c = (df[['gender']]=='F').groupby(df[by]).agg(['sum','count'])\n",
        "        s = c[('gender','sum')]\n",
        "        n = c[('gender','count')]\n",
        "        prob = (s + self.prior*self.prior_weight) / (n + self.prior_weight)\n",
        "        self.prob = pd.DataFrame(prob, columns=[self.label])\n",
        "        return self\n",
        "    \n",
        "    def predict_proba(self, df):\n",
        "        pF = df[[self.by]].merge(self.prob, how='left', \n",
        "                                left_on=self.by, right_index=True).fillna(self.prior)[self.label]\n",
        "        pF = np.reshape(pF.values,(pF.shape[0],1))\n",
        "        pred = np.zeros((df.shape[0],12))\n",
        "        pred += self.agegroups.values\n",
        "        pred[:,:6] *= pF\n",
        "        pred[:,6:] *= 1-pF\n",
        "        return pred"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f5dd1d39-eae9-4998-8f71-64bfb6315b10"
      },
      "outputs": [],
      "source": [
        "kf = KFold(ptrain.shape[0], n_folds=10, shuffle=True, random_state=0)\n",
        "pred = np.zeros((ptrain.shape[0],n_classes))\n",
        "for itrain, itest in kf:\n",
        "    train = ptrain.iloc[itrain,:]\n",
        "    test = ptrain.iloc[itest,:]\n",
        "    ytrain, ytest = y[itrain], y[itest]\n",
        "    clf = GenderProb().fit(train,'brand')\n",
        "    pred[itest,:] = clf.predict_proba(test)\n",
        "print(log_loss(y, pred))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7deaeae7-5fd6-464d-ab92-90223ff4071b"
      },
      "source": [
        "Wrap the scoring in a function to try different values for prior_weight."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1f7c7acc-61ef-4335-a914-1bbf705944d4"
      },
      "outputs": [],
      "source": [
        "def score(ptrain, by, prior_weight=10.):\n",
        "    kf = KFold(ptrain.shape[0], n_folds=10, shuffle=True, random_state=0)\n",
        "    pred = np.zeros((ptrain.shape[0],n_classes))\n",
        "    for itrain, itest in kf:\n",
        "        train = ptrain.iloc[itrain,:]\n",
        "        test = ptrain.iloc[itest,:]\n",
        "        ytrain, ytest = y[itrain], y[itest]\n",
        "        clf = GenderProb(prior_weight=prior_weight).fit(train,by)\n",
        "        pred[itest,:] = clf.predict_proba(test)\n",
        "    return log_loss(y, pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "99bbc941-9cdf-42a0-8b66-f9885573d80a"
      },
      "outputs": [],
      "source": [
        "weights = [0.1, 0.5, 1., 5., 10., 20.,40.]\n",
        "res = [score(ptrain,'brand',prior_weight=w) for w in weights]\n",
        "plt.plot(weights, res)\n",
        "plt.title('Best score {:.5f} at prior_weight = {}'.format(np.min(res),weights[np.argmin(res)]))\n",
        "plt.xlabel('prior_weight')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7bc05c74-5071-46a4-8a38-40c54d060193"
      },
      "source": [
        "### Benchmark: predict gender from device model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ef985ad3-7a82-4e2a-ae44-e0f1814c4cd9"
      },
      "outputs": [],
      "source": [
        "weights = [0.1, 0.5, 1., 5., 10., 20.,40.]\n",
        "res = [score(ptrain,'model',prior_weight=w) for w in weights]\n",
        "plt.plot(weights, res)\n",
        "plt.title('Best score {:.5f} at prior_weight = {}'.format(np.min(res),weights[np.argmin(res)]))\n",
        "plt.xlabel('prior_weight')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "090b4e51-a7f5-43a3-a4d7-895b5d945b6a"
      },
      "source": [
        "An improvement of about 0.01 when going from predicting class probabilities to predicting gender based on device model.\n",
        "\n",
        "### Benchmark: predict gender-age group from phone brand"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "db18b958-6b84-4a67-a5be-58e50de6c74d"
      },
      "outputs": [],
      "source": [
        "class GenderAgeGroupProb(object):\n",
        "    def __init__(self, prior_weight=10.):\n",
        "        self.prior_weight = prior_weight\n",
        "    \n",
        "    def fit(self, df, by):\n",
        "        self.by = by\n",
        "        #self.label = 'pF_' + by\n",
        "        self.prior = df['group'].value_counts().sort_index()/df.shape[0]\n",
        "        # fit gender probs by grouping column\n",
        "        c = df.groupby([by, 'group']).size().unstack().fillna(0)\n",
        "        total = c.sum(axis=1)\n",
        "        self.prob = (c.add(self.prior_weight*self.prior)).div(c.sum(axis=1)+self.prior_weight, axis=0)\n",
        "        return self\n",
        "    \n",
        "    def predict_proba(self, df):\n",
        "        pred = df[[self.by]].merge(self.prob, how='left', \n",
        "                                left_on=self.by, right_index=True).fillna(self.prior)[self.prob.columns]\n",
        "        pred.loc[pred.iloc[:,0].isnull(),:] = self.prior\n",
        "        return pred.values\n",
        "    \n",
        "def score(ptrain, by, prior_weight=10.):\n",
        "    kf = KFold(ptrain.shape[0], n_folds=10, shuffle=True, random_state=0)\n",
        "    pred = np.zeros((ptrain.shape[0],n_classes))\n",
        "    for itrain, itest in kf:\n",
        "        train = ptrain.iloc[itrain,:]\n",
        "        test = ptrain.iloc[itest,:]\n",
        "        ytrain, ytest = y[itrain], y[itest]\n",
        "        clf = GenderAgeGroupProb(prior_weight=prior_weight).fit(train,by)\n",
        "        pred[itest,:] = clf.predict_proba(test)\n",
        "    return log_loss(y, pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b590adc6-9699-4c32-b150-dec8c4b40336"
      },
      "outputs": [],
      "source": [
        "weights = [0.5, 1., 5., 10., 20.,40.,100.]\n",
        "res = [score(ptrain,'brand',prior_weight=w) for w in weights]\n",
        "plt.plot(weights, res)\n",
        "plt.title('Best score {:.5f} at prior_weight = {}'.format(np.min(res),weights[np.argmin(res)]))\n",
        "plt.xlabel('prior_weight')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "193fff64-0fcc-4c1e-9588-f6a2f1839e40"
      },
      "source": [
        "### Benchmark: predict gender-age group from device model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "006fb690-63c2-4638-97d8-281e8e306388"
      },
      "outputs": [],
      "source": [
        "weights = [0.5, 1., 5., 10., 20.,40.,100.]\n",
        "res = [score(ptrain,'model',prior_weight=w) for w in weights]\n",
        "plt.plot(weights, res)\n",
        "plt.title('Best score {:.5f} at prior_weight = {}'.format(np.min(res),weights[np.argmin(res)]))\n",
        "plt.xlabel('prior_weight')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "657088b3-5845-49ff-9465-854937ef41dd"
      },
      "source": [
        "What if we combine predictions from phone brand and device model?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b886cf64-bfb3-400a-b277-5a62d47cbd88"
      },
      "outputs": [],
      "source": [
        "kf = KFold(ptrain.shape[0], n_folds=10, shuffle=True, random_state=0)\n",
        "predb = np.zeros((ptrain.shape[0],n_classes))\n",
        "predm = np.zeros((ptrain.shape[0],n_classes))\n",
        "for itrain, itest in kf:\n",
        "    train = ptrain.iloc[itrain,:]\n",
        "    test = ptrain.iloc[itest,:]\n",
        "    ytrain, ytest = y[itrain], y[itest]\n",
        "    clf = GenderAgeGroupProb(prior_weight=40.).fit(train,'brand')\n",
        "    predb[itest,:] = clf.predict_proba(test)\n",
        "    clf = GenderAgeGroupProb(prior_weight=40.).fit(train,'model')\n",
        "    predm[itest,:] = clf.predict_proba(test)\n",
        "log_loss(y, 0.5*(predb+predm))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f9c61de3-2f7d-492a-9397-dfaf41861165"
      },
      "source": [
        "Down ot 2.391.\n",
        "\n",
        "## Make a submission"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "85c28fec-9904-4927-925e-b55fb0a71443"
      },
      "outputs": [],
      "source": [
        "ptest = gatest.merge(phone[['device_id','brand','model']], how='left',on='device_id')\n",
        "ptest.head(3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "530500cc-8f52-4e54-b2d3-0b4498910965"
      },
      "outputs": [],
      "source": [
        "clf = GenderAgeGroupProb(prior_weight=40.).fit(ptrain,'brand')\n",
        "predb = clf.predict_proba(ptest)\n",
        "clf = GenderAgeGroupProb(prior_weight=40.).fit(ptrain,'model')\n",
        "predm = clf.predict_proba(ptest)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4a46b4c5-5d87-4398-8d82-1cacba603088"
      },
      "outputs": [],
      "source": [
        "pd.DataFrame(0.5*(predb+predm), \n",
        "             index = ptest.device_id, \n",
        "             columns=letarget.classes_).to_csv('pbm_subm.csv', index=True)"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}