{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nplt.rcParams[\"figure.figsize\"] = (16,9)\nimport seaborn as sns\nimport scipy\nimport os\nprint(os.listdir(\"../input\"))","execution_count":1,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"954ad8eae3624458476b09c68009ddf6f13d72cf"},"cell_type":"code","source":"df = pd.read_csv('../input/train.csv')\ndf.info()","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df.head()","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"1eda020330bdf1c838cf07a027caddd6860a06c5"},"cell_type":"markdown","source":"## Deal probability\nThe histogram of target values show a clear abundance of low values, making this dataset a pretty umbalanced one."},{"metadata":{"trusted":true,"_uuid":"83bdab0340232491b4282ac7b47a71a2311cb8c1"},"cell_type":"code","source":"hist_kws={\"alpha\": 0.3}\nsns.distplot(df.deal_probability, hist_kws=hist_kws)\nplt.title('Deal Probability distribution')\nplt.margins(0.02)\nplt.show()","execution_count":4,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76e602b20a456815a0114a757f850cbab527acae","collapsed":true},"cell_type":"markdown","source":"## Price"},{"metadata":{"trusted":true,"_uuid":"5836bbc66f7ee7940131b9ba0c80c3bf2f782723","collapsed":true},"cell_type":"code","source":"# We sample the dataframe & limit the extreme prices (+ no NaN !)\ndf_dropped = df.loc[df.index[df.price < 500000]].sample(n=50000)","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"6bad53383be42f9b016d7edf80f52fc32e0f64d5"},"cell_type":"markdown","source":"It appears that there is little impact of price on deal probability seeing that:\n- huge confidence intervals for linear regression\n- statistically significant p value for a pearson correlation close to 0\n\nBesides, the slopes we see for the linear regressions may be explained by:\n- the scatter motive looks like a grid (indicating a non uniform distribution of values after our sampling)\n- a higher concentration of low/low values (same as above)"},{"metadata":{"trusted":true,"_uuid":"50cf0133e265d9462093c87f69835cfb42e47b2b"},"cell_type":"code","source":"sns.lmplot('price', 'deal_probability', hue='user_type', data=df_dropped, fit_reg=True, size=10, aspect=2, scatter_kws={'s': 10, 'alpha':0.3})\nplt.xlabel('Ad price')\nplt.ylabel('Deal Probability')\nplt.margins(0.01)\nplt.show()\nprint(f'Pearson correlation : {scipy.stats.pearsonr(df_dropped.price.values, df_dropped.deal_probability.values)}')","execution_count":6,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"20b5c63295fbd3531d5f8e055796e15329bef511"},"cell_type":"markdown","source":"The histogram of the full dataset prices confirms the hypothesis of many low prices offers"},{"metadata":{"trusted":true,"_uuid":"3304c8f74e74e076aa49a9d2d619d346a188cf30"},"cell_type":"code","source":"sns.distplot(df.price.dropna(), kde=False)\nplt.margins(0.02)\nplt.show()","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"38cfe5f20e7dae5b1c3b5d4f499175a9d9bab86e"},"cell_type":"markdown","source":"A kernel density estimation shows more clearly peaks on X.10k prices, which probably correspond with psychological thresholds."},{"metadata":{"trusted":true,"_uuid":"f590f7621a7c3da6e257721b1ce1e8f8c59bee14"},"cell_type":"code","source":"sns.kdeplot(df_dropped.price)\nplt.margins(0.02)\nplt.show()","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"722e659f0235d28441a83af92f2493b4862ad2f4"},"cell_type":"markdown","source":"We can try to trim even further the price."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"54d8caae7c16f2980684dc42590c1819a8e23ebc"},"cell_type":"code","source":"df_trimmed = df.loc[df.index[df.price < 40000]].sample(n=50000)","execution_count":9,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"671bb56fd6feaef0271f95208590f13dcb167c5e"},"cell_type":"code","source":"sns.kdeplot(df_trimmed.price)\nplt.margins(0.02)\nplt.show()","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"fd1db30ad8816bceb8efd0b2efb60a833c8e4769"},"cell_type":"markdown","source":"## User type EDA\nWe colored our scatter plot above by `user_type` (Company, Private, Shop).\nThe Private type is much more present in the dataset."},{"metadata":{"trusted":true,"_uuid":"57a88c2ac0e63a8033f1a3b18ad322139047bc70"},"cell_type":"code","source":"print('Counts:\\n')\nfor user_type in df.user_type.unique():\n    print(f\"{user_type} users : {df[df.user_type == user_type].shape[0]}\")","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"c4d6b9079a006eab0eb5c18b9ac8d4d7998aa7a6"},"cell_type":"markdown","source":"We may want to know if a user_type leans towards to a specific range of prices."},{"metadata":{"trusted":true,"_uuid":"c57f2485dcd45b758a9e086aa210a87228da7ed5"},"cell_type":"code","source":"sns.violinplot('user_type', 'price', data=df_dropped)","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"3f3b363530c74ae2ea100158e3c45adf914aa1b4"},"cell_type":"markdown","source":"As we have huge tails on our violin plot, we may want to use again our trimmed dataframe (price < 40k). Private & Company user_type seem to have fairly similar price distribution"},{"metadata":{"trusted":true,"_uuid":"c6b7efce92d1d7dadb9de203743ab1513a0855e1"},"cell_type":"code","source":"sns.violinplot('user_type', 'price', data=df_trimmed)","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"0c1581bdad474de51a79bd2026d12ae919b8048d"},"cell_type":"markdown","source":"The KDE of `deal_probability` behave the same way as the price : Shop `user_type` are a little bit different than the other two."},{"metadata":{"trusted":true,"_uuid":"548940ba7109c36bb3bda1e3c61c7646e3082c12"},"cell_type":"code","source":"hist_kws={\"alpha\": 0.2}\nsns.distplot(df[df.user_type == 'Private']['deal_probability'], label='Private', hist_kws=hist_kws)\nsns.distplot(df[df.user_type == 'Company']['deal_probability'], label='Company', hist_kws=hist_kws)\nsns.distplot(df[df.user_type == 'Shop']['deal_probability'], label='Shop', hist_kws=hist_kws)\nsns.distplot(df.deal_probability, label='All', hist_kws=hist_kws)\nplt.title('Deal Probability distribution')\nplt.legend()\nplt.margins(0.02)\nplt.show()","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"520dc46ebf807c80d35c79369fd4bedda99db030"},"cell_type":"markdown","source":"## Categorical features"},{"metadata":{"trusted":true,"_uuid":"186dd7bd188a2f1d6b4e937eb8b80428448c66b6"},"cell_type":"code","source":"print('Counts\\n')\nprint(f'region {len(df.region.unique())}')\nprint(f'city {len(df.city.unique())}')\nprint(f'parent cat {len(df.parent_category_name.unique())}')\nprint(f'cat {len(df.category_name.unique())}')","execution_count":15,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"431991a17884b29d6833c13b5c2c48e11c5d3700"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}