{"cells":[{"metadata":{"_uuid":"474b398ee987b28048101ad2a89afea5789b60c3"},"cell_type":"markdown","source":"**Etracting Patterns from the Timestamp using Sin and Cosine**\n\nConverting the timestamp using sin and cosine can make sense if patterns occur 24/7. \n\nIn this way, 23:59 is very close to 00:00, as it should be.\n\nWe map each cyclical variable onto a circle such that the lowest value for that variable appears right next to the largest value. We compute the x- and y- component of that point using sin and cos trigonometric functions.  Here's what it looks like for the \"hours\" variable. Zero (midnight) is on the right, and the hours increase counterclockwise around the circle. \n\nSource: David Kaleko (http://blog.davidkaleko.com/feature-engineering-cyclical-features.html)"},{"metadata":{"_uuid":"57035ae9d6fdcc29fad232dd241d4ae595320609"},"cell_type":"markdown","source":"![title](http://i65.tinypic.com/2akh56x.jpg)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#import only a few rows\ntrain = pd.read_csv('../input/train.csv', nrows=1000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"149b2ab48db6f09c78feb75603c2a48b4bed0b46"},"cell_type":"code","source":"train.columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'attributed_time', 'is_attributed']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"34a9c4e5efde28b24cbe1aeee4c7d2abb5d6d6b4"},"cell_type":"markdown","source":"**STEP 1: Breaking Up Date Data into Multiple Features**\n\nThe goal is to break up the timestamp into seperate columns of year, month, day, hour and minute."},{"metadata":{"trusted":true,"_uuid":"0906989436f9767471b04320a700d57f621fd3e2"},"cell_type":"code","source":"df = pd.DataFrame(data=train, columns=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98b3c3acfab8fbd971203bb4107ab96425e8c6e2"},"cell_type":"code","source":"df.click_time = pd.to_datetime(df.click_time)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b0575f28babf27bd78deff4e028595e9bf6f5a1"},"cell_type":"code","source":"df['new_formatted_date'] = df.click_time.dt.strftime('%d/%m/%y %H:%M')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b336643d9e6454fa4cf2e31905e09fd20782de83"},"cell_type":"code","source":"df.new_formatted_date.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca9a0ac5676acb7c13a55f549ce855b523abfb8b"},"cell_type":"code","source":"# pandas.Series.dt\ndf['month'] = df.click_time.dt.month\ndf['day'] = df.click_time.dt.day\ndf['year'] = df.click_time.dt.year\ndf['hour'] = df.click_time.dt.hour\ndf['minute'] = df.click_time.dt.minute\ndf.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da43c39a4db12ae8f11b8a7923e6d1340f75d465"},"cell_type":"code","source":"print('Unique values of month:', df.month.unique())\nprint('Unique values of day:', df.day.unique())\nprint('Unique values of year:', df.year.unique())\nprint('Unique values of hour:', df.hour.unique())\nprint('Unique values of minute:', df.minute.unique())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dc080b88b921d6e6f9b725ee0e76234c0c72e71"},"cell_type":"markdown","source":"**The Magic :-) **\n\nOnly day, hour and minute are relevant in this dataset."},{"metadata":{"trusted":true,"_uuid":"a573808939ace2ca3a38bffcf56df230ed49c8b7"},"cell_type":"code","source":"# Day\ndf['day_sin'] = np.sin(df.day*(2.*np.pi/30))\ndf['day_cos'] = np.cos(df.day*(2.*np.pi/30))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"adf01485e510b0e3bfa469d4317a069fcc7a94f6"},"cell_type":"code","source":"# Hour\ndf['hour_sin'] = np.sin(df.day*(2.*np.pi/24))\ndf['hour_cos'] = np.cos(df.day*(2.*np.pi/24))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5704ac2605d875b7cd858f1e6d4914b0f13621fa"},"cell_type":"code","source":"# Minute\ndf['minute_sin'] = np.sin(df.day*(2.*np.pi/60))\ndf['minute_cos'] = np.cos(df.day*(2.*np.pi/60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba6f383fadc4088bf679ac3b6132454a978ded31"},"cell_type":"code","source":"# Concatenate\nconcatenated = pd.concat([train, df], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92b577b9180d5a0b1f16e9267cb7f497fa8bf5ce"},"cell_type":"code","source":"# Define X\nX = concatenated[['ip', 'app', 'device', 'os', 'channel', 'day_sin','day_cos', 'hour_sin', 'hour_cos',\n                  'minute_sin', 'minute_cos']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b0aded1f46f9edaa3e3e868471b127763f56ec0"},"cell_type":"code","source":"# Now we have timestamp with sin and cosine:\nX.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8bdc9dfd1504fcb78b23b3c112db1fc0f125fb08"},"cell_type":"markdown","source":"I stop here. The rest is easy.\n\nI tested this approach against Bojan Tunguz's post and did NOT get a better accuracy, but it might work in another case.\nhttps://www.kaggle.com/tunguz/xgboost-starter"},{"metadata":{"trusted":true,"_uuid":"bfd695666f7ce77252fd1f7a6eab4c54ba598a8a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}