{"cells":[{"metadata":{"_cell_guid":"9b5b84e6-84b7-4de2-b317-7c85836ab855","_uuid":"86f77cfde4efa7b2b700cfbaacff4448304757d4"},"cell_type":"markdown","source":"Hello !\n\nThis is just a getting started iPython Notebook. I am pretty new to this so trying my best to do exploratory analysis on this HUGE dataset. Any comments and feedback is welcome.\n\nAfter loading the data, First I will try to understand the data by doing some exploratory analysis. Then I will try to create a benchmark model after some feature engineering(if required)."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport matplotlib.pyplot as plt\nfrom matplotlib import style\nimport seaborn as sns\n%matplotlib inline\n# style.use('fivethirtyeight')\nprint(os.listdir(\"../input\"))","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"ee7e1e0c-283e-4784-a0bf-7a07bf5d799a","_uuid":"a9a79776fbad5d4e343bb7819c6f211b7e481f40"},"cell_type":"markdown","source":"## Read the data into a pandas dataframe\nSince the kernel can't handle the volume of data in train.csv. I am using pandas **nrows** parameter to load ~10M rows in training dataset."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"df_train_sample = pd.read_csv(\"../input/train_sample.csv\")\ndf_train = pd.read_csv(\"../input/train.csv\", nrows=10000000) # this might take a while\n#df_test = pd.read_csv(\"../input/test.csv\")","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"6f519510-357e-4245-a04c-00c8b6c0da7f","_uuid":"c7b97f3f8182d8f8afb3108433eb53b0eaeb388b"},"cell_type":"markdown","source":"## Stare at the data."},{"metadata":{"_cell_guid":"9f0d90a3-187a-4514-846f-10278276b71e","_uuid":"667bf483a362576e79478b3a757285f04a219134","scrolled":false,"trusted":true},"cell_type":"code","source":"a = df_train.head(2).values\na.shape","execution_count":27,"outputs":[]},{"metadata":{"_cell_guid":"39750cc8-08d9-4bb9-a71b-d65d341df95b","_uuid":"fd7c764a5af38c4f73b4f42442c69c6569d4d381"},"cell_type":"markdown","source":"## Check for Null data"},{"metadata":{"_cell_guid":"1d1c9d9b-1195-43bc-8277-eedcc97e5d02","_uuid":"7774f91335905cdc633f7a5e74c117c05b292c73","scrolled":false,"trusted":true},"cell_type":"code","source":"df_train.isnull().sum()","execution_count":12,"outputs":[]},{"metadata":{"_cell_guid":"4bc71d13-5d31-405d-b243-ed03fa9637a4","_uuid":"fb7387b68b422eb0eec608080c710c05e10d40ed"},"cell_type":"markdown","source":"Damn! For 10M rows selected, approx < 1% of the data has *attributed_time* as not null. It seems that the data is quite imbalanced.\nGood new is that there are not any null values for other attributes."},{"metadata":{"_cell_guid":"79a065b7-e10f-4b02-b12a-93028b4d1b7a","_uuid":"5e937f66918c651167954483c5c78cf802935347"},"cell_type":"markdown","source":"## Check datatypes."},{"metadata":{"_cell_guid":"3e8a96c8-3306-4918-ae08-80369882fdd9","_uuid":"331d66476825c1071714039079cc6ddf2be7428b","scrolled":false,"trusted":true},"cell_type":"code","source":"df_train.dtypes","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"035b00c4-d8b7-4a2d-8ed5-1bd242fcc11e","_uuid":"dfe6724efb118f570d3c0de8e32fe83382d97e96"},"cell_type":"markdown","source":"## Convert click_time & attributed_time to timestamp datatype."},{"metadata":{"_cell_guid":"44e949fa-e2b7-4b13-8ca4-030a09e966e5","_uuid":"ff38494b813fc1a8e49b66cd585c9386ca156f1a","collapsed":true,"trusted":true},"cell_type":"code","source":"df_train['click_time'] = pd.to_datetime(df_train['click_time'])\ndf_train['attributed_time'] = pd.to_datetime(df_train['attributed_time'])","execution_count":14,"outputs":[]},{"metadata":{"_cell_guid":"22fc145a-c78e-4227-b218-db149559d1b3","_uuid":"8c1af800924533b7fc8d6e54d948eed97903d774"},"cell_type":"markdown","source":"## Check if *attributed_time* and *is_attributed* are in sync. \n\nif *is_attributed* = 0 then *attributed_time* = NaN\n\nif *is_attributed* = 1 then *attributed_time* = (some time value)\n"},{"metadata":{"_cell_guid":"aa062677-08a9-470b-a103-1ea2bcf27c38","_uuid":"068564b695e2170b14c35f617c4d6dfcc1371f13","scrolled":true,"trusted":true},"cell_type":"code","source":"df_train.groupby('is_attributed').agg({'attributed_time':'unique'})","execution_count":15,"outputs":[]},{"metadata":{"_cell_guid":"6ac35eaa-102c-4bba-b027-dc5429032b62","_uuid":"13543a90fc9dfa3a0877c28d933ac28437822792"},"cell_type":"markdown","source":"Looks like it is quite well balanced."},{"metadata":{"_cell_guid":"feaa0c9b-5abe-4c77-bea4-9ee9dd577ad3","_uuid":"1f89e368c37b561321ad4bcdee2200b821cd3cd0"},"cell_type":"markdown","source":"## Distribution of *is_attributed* variable"},{"metadata":{"_cell_guid":"39231837-12e8-4a90-be89-ed853cba4e98","_uuid":"92ca1915c0cde97da277771ffec3c086ffefa69d","scrolled":true,"trusted":true},"cell_type":"code","source":"df_plot1 = df_train.groupby(df_train['is_attributed'])['is_attributed'].count()\ndf_plot1.plot.bar(fontsize=8, legend=True)","execution_count":16,"outputs":[]},{"metadata":{"_cell_guid":"08e49db0-8dfe-4c08-a476-38031ece25f6","_uuid":"c1b5d16e69560d4c64acffc9b30e367baaf61b01"},"cell_type":"markdown","source":"Looks like 99% of the data has target variable as False."},{"metadata":{"_cell_guid":"780abeb5-cf54-4c4a-a322-7f6d803ef0df","_uuid":"cb289492c4644489525c9544610ee9476fdbeece"},"cell_type":"markdown","source":"## Time difference when user clicked on the ad and when user downloaded it."},{"metadata":{"_cell_guid":"2a1ad9ff-1ff3-48ff-87b7-42e478510ed9","_uuid":"896e1d62cd78dc46ad2cf985f6d57b9ae261572b","scrolled":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"df_train['days_to_download'] = df_train['attributed_time'] - df_train['click_time']","execution_count":17,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"0892b3bd36344a0692c28c4a615bf4a7205eb48a"},"cell_type":"code","source":"days_to_download = df_train[~df_train['days_to_download'].isnull()]['days_to_download']\nhours_to_download = days_to_download.dt.components.hours\nhours_to_download[hours_to_download != 0].plot(kind='hist' ,bins=10)","execution_count":23,"outputs":[]},{"metadata":{"_uuid":"b86eec897337e56e3866c45dc3a6307b2c6d41c3"},"cell_type":"markdown","source":"It shows that most of the users downloaded the app within 24 hours of clicking on the ad."},{"metadata":{"_cell_guid":"b5615aa4-3f21-4524-9cae-49e96f6bc959","_uuid":"45339d0b4b9240d7c100f6b4fee5197bb58860f5","scrolled":true,"trusted":true},"cell_type":"code","source":"df_train[df_train['is_attributed']==1].head()","execution_count":19,"outputs":[]},{"metadata":{"_cell_guid":"87576f7b-046f-4d22-bf69-849cb2075802","_uuid":"199b5bd598fb4e9c1f148c4497fe5d45c6de4bc4"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"4b0ee60f-699a-4764-9248-92ddc4ce60aa","_uuid":"60b173ceb80d7209308d152e79452abb947907cd","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}