{"cells":[{"metadata":{"_uuid":"5e9d0ed870c39115405471f2c277e03528cc52d7"},"cell_type":"markdown","source":"# Exploratory analysis with visualizations 📊\n\n## Table of content\n\n* [1. Importing librairies](#Importing-librairies)\n* [2. Functions for visualization](#Functions-for-visualization)\n* [3. Functions for data processing](#Functions-for-data-processing)\n* [4. Loading data](#Loading-data)\n* [5. Data processing](#Data-processing)\n* [6. Data analysis](#Data-analysis)\n* [7. Data visualization](#Data-visualization)\n\n"},{"metadata":{"_uuid":"355116b4660bcb55b7de7bf04d1c4de5b23971c7"},"cell_type":"markdown","source":"### Importing librairies"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"collapsed":true},"cell_type":"code","source":"import pandas as pd\nimport dask.dataframe as dd\nimport time\n\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport matplotlib.pylab as pylab\nimport seaborn as sns","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"e6c337f3d80ba41d8c81b03d75f1d7cdc3ecf063"},"cell_type":"markdown","source":"### Functions for visualization\n\nI get these functions from [this kernel](https://www.kaggle.com/aditi2009/titanic-data-science-solution)."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"f4a7092de730433f9b917f99ff81c7867b4a8aed"},"cell_type":"code","source":"def plot_categories(df, cat, target, **kwargs):\n    row = kwargs.get('row', None)\n    col = kwargs.get('col', None)\n    facet = sns.FacetGrid(df, row=row, col=col, size=4, aspect=2)\n    facet.map(sns.barplot, cat, target)\n    facet.add_legend()\n    plt.show()\n\ndef plot_distribution(df, var, target, **kwargs):\n    row = kwargs.get('row', None)\n    col = kwargs.get('col', None)\n    facet = sns.FacetGrid(df, hue=target, size=4, aspect=4, row=row, col=col)\n    facet.map(sns.kdeplot, var, shade=True)\n    facet.set(xlim=(0, df[var].max()))\n    facet.add_legend()\n    plt.show()\n\ndef plot_correlation_map(df):\n    corr = df.corr()\n    _, ax = plt.subplots(figsize=(12, 10))\n    cmap = sns.diverging_palette(220, 10, as_cmap=True)\n    _ = sns.heatmap(\n        corr,\n        cmap=cmap,\n        square=True,\n        cbar_kws={'shrink': .9},\n        ax=ax,\n        annot=True,\n        annot_kws={'fontsize': 12}\n    )\n    plt.show()\n\ndef describe_more(df):\n    var = [];\n    l = [];\n    t = []\n    for x in df:\n        var.append(x)\n        l.append(len(pd.value_counts(df[x])))\n        t.append(df[x].dtypes)\n    levels = pd.DataFrame({'Variable': var, 'Levels': l, 'Datatype': t})\n    levels.sort_values(by='Levels', inplace=True)\n    return levels","execution_count":28,"outputs":[]},{"metadata":{"_uuid":"32664f108ffc577cd4cda52512ee6280d4774495"},"cell_type":"markdown","source":"### Functions for data processing"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"def dataPreProcessTime(df):\n    df['click_time'] = pd.to_datetime(df['click_time'])\n    df['click_hour'] = df['click_time'].apply(lambda x: x.strftime('%H')).astype(int)\n\n    return df\n\ndef dataPreProcess(df):\n    df = dataPreProcessTime(df)\n    df = df.fillna(0)\n    return df","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"44448486bca17c41b7482c982c7b4320bdca6a81"},"cell_type":"markdown","source":"### Loading data\n\nI use the code that I shared in [this post](https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51809#295230)."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"db2f8b7dd2af210151f129bd1ea025bdd062f1d8"},"cell_type":"code","source":"path = '../input/'\ntrain = dd.read_csv(path + \"train.csv\")\nfreq = 0.02\ntrain = train.random_split([freq, 1-freq], random_state=42)[0]\ntrain = train.compute()\n\ntrain.columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'attributed_time', 'is_attributed']","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"005c605479416a00d59d8b2f0ad64e317229ec67"},"cell_type":"markdown","source":"### Data processing"},{"metadata":{"trusted":true,"_uuid":"d5555b159f51b20080318689f0a4fb7ccf55f623"},"cell_type":"code","source":"train = dataPreProcess(train)","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"432b7d05b7e76f5dddb4565e62bbb7ea4d5da945"},"cell_type":"markdown","source":"### Data analysis"},{"metadata":{"trusted":true,"_uuid":"3ebaa9c58d7a8d3ae999efcd95926750d5e4fa48"},"cell_type":"code","source":"train.head()","execution_count":13,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2a0cebea9f526abb4d0eba02ef41238e54267cb"},"cell_type":"code","source":"train.shape","execution_count":14,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"46eed558187f774da8e3ac2c082a9a77947f246f"},"cell_type":"code","source":"describe_more(train)","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"0d3642edba38b728ae43ca73e810eb63d1be0f4b"},"cell_type":"markdown","source":"### Data visualization"},{"metadata":{"trusted":true,"_uuid":"99500b8b53d5eef73973daecc9032e063f415687"},"cell_type":"code","source":"plot_categories( train , cat = 'click_hour' , target = 'is_attributed' )","execution_count":23,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76f45dda6caf9bdb2ab07b47040af46d21facfca"},"cell_type":"code","source":"plot_distribution( train , var = 'os' , target = 'is_attributed' )","execution_count":30,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b15f282ffad7e927c78ecc058e7bcaae4b6a5aa"},"cell_type":"code","source":"plot_distribution( train , var = 'channel' , target = 'is_attributed' )","execution_count":31,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1372824c3c1f3373eb44af7805456baf13419745"},"cell_type":"code","source":"plot_distribution( train , var = 'app' , target = 'is_attributed' )","execution_count":32,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"346b62fed66efa90c3a38ac6383ef552a492e507"},"cell_type":"code","source":"plot_correlation_map(train)","execution_count":27,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}