{
  "metadata": {
    "kernelspec": {
      "name": "python"
    },
    "language_info": {
      "name": "python",
      "version": "3.5.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5fa8ad88-346c-e417-7b53-9e6fcb765ecd",
        "_active": false
      },
      "source": "**Hypothesis I**:  the reason why user clicks the ad is ONLY based on doc content and the access context, without caring about the other ads in the same display.\n\n**Hypothesis II**:  the reason why user clicks the ad is not only based on the doc content and the access context, but also considering the other ads in the same display.\n\nTODO: validate which hypothesis is established. \n\nWe have checked that display_id in clicks_test.csv is ordered. This means we do not need groupby op clicks_test.csv to collect the ads for that display.\nSame situation is in clicks_train.csv.\n\nFor each display, we construct an instance for learning, namely,\n$$ (X) \\rightarrow y $$\nFeature set $X$ is the content of the display, such as document content, user access content, and ad content. The first question is: what is the output label $y$? The naive answer is the click or not for a single ad, like the original presentation in clicks_train.csv. This is consistent with Hypothesis I. So the data set for training is organized as:\n$$(D, U, A) \\rightarrow 1/0$$\nwhere $D, U, A$ are content of document, user and ad respectively. \n\nThe second answer to the first question is more reasonable. The output label is a sequence of click or not (of course only one click is allowed) for all ads in the same display.\nEquivalently, the output is a number to identify which ad is among the ads in the same display. \nThis is consistent with Hypothesis II. This model leaves a problem that display contains varied number of ads. Since we only care maximum 12 ads in test set, we can pad NULL ads for displays with less than 12 ads.\nSo the data set for training is organized as:\n$$(D,U,A_1,A_2,\\ldots,A_{12}) \\rightarrow n \\qquad \\qquad \\text{(1)}$$\nwhere $n$ refers to the clicked $A_i$. \n\nBut wait a minute. Let's check what the test set asks for. It asks for the rank of the ads in the same display. That is:\n$$(D,U,A_1,A_2,\\ldots,A_{12}) \\rightarrow \\text{rank of } A_1 \\ldots, A_{12} \\qquad \\text{(2)}$$\nWhile in the train set the non-click ads have no further differential information with the respect to its sibling ads.  \nOnly one clicked ad stands out of the ads in the same display.\nIn this way, we might guess that the train set does not provide enough information for training to answer the test question.\nUnless we can do some approximations.\n\nFor the test question, we first ask the trained machine to answer which one is the best guess for user click.\nFor example, $A_i$ is the answer among the test case.\nThen we eliminate $A_i$ from the test case, and ask the trained machine.\nSuppose $A_j$ is the answer the the second round for the test case.\nThen $A_j$ is the 2nd rank among the displayed ads.\nIn this way, problem (2) can be solved by solving problem (1).\n\n**So we mapped the Challenge to a 12-class classification problem.**\nLet's check how large the train and test set are.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "09f66385-202d-90f3-277b-3ebdd8bfd6d2",
        "_active": true
      },
      "outputs": [],
      "source": "import pandas as pd\n\ndfTrain = pd.read_csv('../input/clicks_train.csv')\nprint('there are ', len(dfTrain), ' rows in the original train set.')\ngpTrain = dfTrain.groupby('display_id')\nprint('there are ', len(gpTrain), ' displays in the train set.')\n\ndfTest = pd.read_csv('../input/clicks_test.csv')\nprint('there are ', len(dfTest), ' rows in the original test set.')\ngpTest = dfTest.groupby('display_id')\nprint('there are ', len(gpTest), ' displays in the test set.')\ncountTestInst = 0\nfor disp in gpTest:\n    countTestInst += len(disp[1]) - 1\nprint('there are ', countTestInst, ' test instances all together.')",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4f2ab026-ac96-055c-e463-0409555aebfd",
        "_active": false
      },
      "source": "Finally, we have 16M train cases and 25M (derived from 6M) test instances of 12-class classification problem.\nIt is obvious a big data learning problem, not even considering the dimension of features of the data set.",
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f7248c68-a56d-b29e-f88f-13327e0d5c65",
        "_active": false
      },
      "source": "## Deriving features of $D$ and $U$ from display_id",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "d3747a05-5ab5-17cd-2f65-65c9ac86eabb",
        "_active": false
      },
      "outputs": [],
      "source": "import numpy as np\n# the original platform fields mixed with char datatype, cast to int64\ndfEvent = pd.read_csv('../input/events.csv', index_col='display_id')\nprint(dfEvent.dtypes)\ndfEvent.head(5)",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0eb23b3e-4ef3-e868-d62c-c6eacc98b721",
        "_active": false
      },
      "source": "The Dtypewarning message suggests some data cleaning jobs we need to do. We put all such cleaning jobs described in later section.\nRight now, let's assume that very data is correct.\n\ndisplay_id defines uuid, doc_id and other context of such display, such as timestamp, platform and geo.\ngeo feature can be further decoded as country, state and DMA features.\n\nDon't forget to make sure display_id is unique so that it can be used as index of the dataframe.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "de37537b-59ee-480c-7c69-90c26d923df8",
        "_active": false
      },
      "outputs": [],
      "source": "print(len(dfEvent.index.unique()))\nprint(len(dfEvent))\nprint('the number of docs defined in events ', len(dfEvent['document_id'].unique()))\nprint('the number of uuid defined in events ', len(dfEvent['uuid'].unique()))",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2d9da7ca-3008-4d42-be0d-9cb84c0fdb98",
        "_active": false
      },
      "source": "### Document content features",
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8b5df40b-4da1-f376-f660-1dd2a38f6f27",
        "_active": false
      },
      "source": "From doc_id, we can derive a lot of content features for the doc, which should be features $D$.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "eba8b677-29bd-cc38-8edc-6f799e40b9f3",
        "_active": false
      },
      "outputs": [],
      "source": "dfDocMeta = pd.read_csv('../input/documents_meta.csv')\nprint(dfDocMeta.dtypes)\nprint('the unique doc_id number ', len(dfDocMeta['document_id'].unique()))\nprint('the rows of doc_meta ', len(dfDocMeta))\n# if the above two are equal, we are safe to make doc_id as index\ndfDocMeta.set_index(dfDocMeta.document_id, inplace=True)\ndfDocMeta.head(5)",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "181d000c-b836-4eb0-625b-93aed4be46f2",
        "_active": false
      },
      "source": "This table means that ~3M doc have meta infor.\nSo the features and demension of meta is:\n$$Meta = [srcid, pubid, pubtime], |Meta|=3$$\nLet's see what topics information is provided for docs.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "a0b77e2c-6867-fd61-2622-76efe13c60ec",
        "_active": false
      },
      "outputs": [],
      "source": "dfDocTopic = pd.read_csv('../input/documents_topics.csv')\nprint(dfDocTopic.dtypes)\ndfDocTopic.head(5)",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "68a77b29-a0ae-1ca6-9f57-18a1aa6c9d93",
        "_active": false
      },
      "source": "Let's check how many topics and docs are described in this file.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "44bf971b-9c47-7952-524f-b4282eaa7efd",
        "_active": false
      },
      "outputs": [],
      "source": "print('the number of unique topic_id', len(dfDocTopic['topic_id'].unique()))\nprint('the number of unique doc_id', len(dfDocTopic['document_id'].unique()))",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bd40e87a-46f7-0066-aedb-a4bbffcbdb0b",
        "_active": false
      },
      "source": "This means 2.5M docs have been cated into 300 topics, and the same doc_id might be assigned several topic_ids and confidence_levels.\nLet's check if the number of topic_ids is same for every doc_id.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "52ab6c4c-bd31-9c0b-2b9c-8d9bf72c6784",
        "_active": false
      },
      "outputs": [],
      "source": "gpDocTopic = dfDocTopic.groupby('document_id')\nprint(gpDocTopic['topic_id'].count().describe())",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2c678789-aeeb-d36a-7a07-5479ca1da450",
        "_active": false
      },
      "source": "We have problem here.\nThere are as much as 39 topic_ids has been assigned to the doc_id, and as min as 1 topic_id.\nWe choose 7 topic_ids as the regular number for each doc_id.\nFill in NULL if the doc_id has less than 7 topic_ids, and truncate the 7 with higher confidence_level for the doc_id with more than 7 topic_ids.\nSo the topic feature should be:\n$$Top =[(topicid,conflevl)_1, \\ldots, (topicid,conflevl)_7], |Top|=14$$\nNote, there are about 3M docs have meta info, but only 2.5M docs have been assigned topic feature.\n\nNow let's move on entity features of doc.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "41083454-3cae-cdd7-33fa-1bd18336ab28",
        "_active": false
      },
      "outputs": [],
      "source": "dfDocEnt = pd.read_csv('../input/documents_entities.csv')\nprint(dfDocEnt.dtypes)\ndfDocEnt.head(5)",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5dd62bae-08f4-3d05-5c11-9270c37b8b8d",
        "_active": false
      },
      "source": "Similar analysis to dfDocTopic, we have:",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "99e5f7a9-cf06-7cd6-8edc-df4df488244a",
        "_active": false
      },
      "outputs": [],
      "source": "gpDocEnt = dfDocEnt.groupby('document_id')\nprint(gpDocEnt['entity_id'].count().describe())",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "de6bd0ca-66cb-6c70-ab3c-a948fbceb3b2",
        "_active": false
      },
      "source": "So we have 1.3M entities to be assiged as the features of 1.8M docs.\nWe choose 4 (entity, confidence) pairs as the entity features for doc:\n$$Ent =[(entityid,conflevl)_1, \\ldots, (entityid,conflevl)_4],|Ent|=8$$\n\nNow let's check cat feature of docs.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "9fb4caad-a694-6d18-538f-202191f52b28",
        "_active": false
      },
      "outputs": [],
      "source": "dfDocCat = pd.read_csv('../input/documents_categories.csv')\nprint(dfDocCat.dtypes)\nprint(dfDocCat.head(5))\ngpDocCat = dfDocCat.groupby('document_id')\nprint(gpDocCat['category_id'].count().describe())",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5100700b-8df6-49db-fa94-157a936dbf33",
        "_active": false
      },
      "source": "It is easy to have:\n$$Cat =[(catid,conflevl)_1, \\ldots, (catid,conflevl)_2],|Cat|=4$$\nSo, the features of doc content can be defined as:\n$$D = Meta \\cup Ent \\cup Top \\cup Cat \\qquad \\qquad \\text{(4)}$$\nand the demension of $|D|=29$.",
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9667cf01-8946-6d27-3dfe-0accd5a38cbe",
        "_active": false
      },
      "source": "### User features",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "05802c3b-7e37-e92e-511b-f3cc0479690e",
        "_active": false
      },
      "outputs": [],
      "source": "dfPageView = pd.read_csv('../input/page_views_sample.csv')\nprint(dfPageView.dtypes)\ndfPageView.head(5)",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a87932a5-6dcf-aa4d-d04f-658262a780fa",
        "_active": false
      },
      "source": "Since this is a sample from page_views.csv, we cannot analyse the distribution of fields. \nThe (uuid, doc_id) pair defines the user features when accessing the doc.\nBut how does it relate to the same pair defined in events.csv?",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "68bb9600-4ef5-73e1-f616-e75e1913a0a6",
        "_active": false
      },
      "outputs": [],
      "source": "for i in range(5):\n    pv = dfPageView.iloc[i]\n    print(pv[0])\n    print(dfEvent[(dfEvent.uuid==pv[0]) & (dfEvent.document_id==pv[1])])\n    #print(timestamp,platform,geo)",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "50fbe289-c3a4-ade4-6b26-1c62d814ad9a",
        "_active": false
      },
      "source": "We see that not all (uuid,doc_id) pairs are included in events.csv.\nLet's do some reverse check.\nSince PageView is not a complete set, so we have to iterate all over Events for possible matchings.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 115,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "31f2ff85-67ca-04eb-6161-ed9aa4c57d0e",
        "_active": false
      },
      "outputs": [],
      "source": "countUnmatched = 0\ncountMatched = 0\ncount = 0\nfor i in range(len(dfEvent)):\n    ev = dfEvent.iloc[i]\n    matchedPV = dfPageView[(dfPageView.uuid==ev[0]) & (dfPageView.document_id==ev[1]) & (dfPageView.timestamp==ev[2]) & (dfPageView.platform==ev[3]) & (dfPageView.geo_location==ev[4])]\n    if len(matchedPV) == 0:\n        # this should not happen for page_view.csv, the complete set\n        countUnmatched += 1\n        continue\n    elif len(matchedPV) > 1:\n        print('multiple matched found in pageview!')\n        print(i, ev)\n        print(matchedPV.head(10))\n        break\n    countMatched += 1\n    #print(i,ev)\n    #print(matchedPV.head(0))\n    if countMatched == 5 :\n        break\nprint('we have found matched pv ', countMatched)\nprint('and unmatched pv in sample_pageview ', countUnmatched, ' in the first ',i+1,' rows of events.csv.')",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c85c8204-643f-559f-7288-0d1c0f8ba3a0",
        "_active": false,
        "collapsed": false
      },
      "source": "We can see that page_view only provides one extra feature for user access context, traffic_source. \nBut we have to spend a lot of cost to seek that trafic.\nOur first try will drop the heavy feature. This means we do not use the page_view.csv file.\nIf we have not got acceptable rank in the competition, then we add the traffic_source feature.\nSo, we have user features:\n$$\nU = [timestamp, platform, cn, state, DMA, (traffic_source)], |U|=5 or 6\n\\qquad \\qquad \\text{(5)}\n$$",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 116,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "5703e16b-0ec3-038f-373d-a848b2261a9c",
        "_active": false
      },
      "outputs": [],
      "source": "dfPromotedCont = pd.read_csv('../input/promoted_content.csv')\nprint(dfPromotedCont.dtypes)\nprint(dfPromotedCont.head(5))",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1c916f03-1b69-1a50-789e-7a020cf8342a",
        "_active": false
      },
      "source": "Let's check if ad_id can be used as index.",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "24a01a8a-8571-1be2-1273-3bd085bf3a5c",
        "_active": false
      },
      "outputs": [],
      "source": "gpAdid = dfPromotedCont.groupby('ad_id')\nprint('group number of ad_id ', len(gpAdid))\nprint('unique number of promoted_content ', len(dfPromotedCont['ad_id'].unique()))",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "caee2d56-dd6c-61c5-798b-6e2cdf2437b6",
        "_active": false
      },
      "source": "So it is safe to",
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 118,
      "metadata": {
        "collapsed": false,
        "_cell_guid": "c4c35124-5f8f-efbd-15bb-be6617ccda59",
        "_active": false
      },
      "outputs": [],
      "source": "dfPromotedCont = pd.read_csv('../input/promoted_content.csv', index_col=0)\nprint(dfPromotedCont.dtypes)\nprint(dfPromotedCont.head(5))",
      "execution_state": "busy"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1d90e594-8199-09f6-c3c8-9fa2827af597",
        "_active": false,
        "collapsed": false
      },
      "source": "It is easy to know that for each ad, we have features for (disp_id, doc_id): \n$$A = [campid, advid]$$\nSo for problem (1), we have 12 ads, and therefor 24 features for ads.\n\nIn summary, for problem (1), we have altogether 29+6+24=59 features, and one 12-class output label.",
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "69760e5f-9757-0bdd-182f-0ba7a175619c",
        "_active": false
      },
      "source": "## Data cleaning",
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ff235867-3be5-b48a-4b0a-a579f4a7f929",
        "_active": false
      },
      "source": "1. events.csv: platform column. correct value: int 1, 2, 3. error values: '1','2','3','\\\\N'. \nProcessing: eliminate rows with '\\\\N';\n'1'->1\n'2'->2\n'3'->3\n\n2. \n\n\n",
      "outputs": []
    }
  ]
}