{"cells":[{"metadata":{"_uuid":"2f5f179859daca4e3df9bab2a8d6356223bf20c8"},"cell_type":"markdown","source":"# Web Traffic Time Series Forecasting\n\nThis competition focuses on the problem of forecasting the future values of multiple time series, as it has always been one of the most challenging problems in the field. More specifically, we aim the competition at testing state-of-the-art methods designed by the participants, on the problem of forecasting future web traffic for approximately 145,000 Wikipedia articles."},{"metadata":{"trusted":false,"_uuid":"7f7d148c21e0d10777598140c134a3bb56910a76"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport datetime\nfrom matplotlib import pyplot as plt\n%matplotlib inline\n\npd.options.display.max_rows = 10\npd.options.display.max_colwidth = 100\npd.options.display.max_columns = 600\nfrom tqdm import tqdm\nimport gc\nfrom tqdm import tqdm_notebook\nfrom sklearn.linear_model import HuberRegressor\nfrom sklearn.model_selection import cross_val_predict, KFold\nfrom sklearn.decomposition import PCA\nimport pandas_profiling\nfrom keras.layers.normalization import BatchNormalization\n\nfrom keras.models import Sequential, Model\n\nfrom keras.layers import Input, Embedding, Dense, Activation, Dropout, Flatten\n\nfrom keras import regularizers \n\nimport keras\nfrom collections import Counter\nimport re\nfrom sklearn.preprocessing import MinMaxScaler, StandardScaler\nfrom sklearn.model_selection import GroupKFold\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d136b7164f4da98300efa3b80739572fe90914a"},"cell_type":"markdown","source":"# Data Analysis and Basi EDA, Feature Engineering"},{"metadata":{"trusted":false,"_uuid":"0ac876a562000bfbbbf98f2fef1edc52278223c8"},"cell_type":"code","source":"df_train = pd.read_csv('../input/train_1.csv')\ndf_train.shape\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"4deaaf43011e4552ed2550fd4e357b3342f04bba"},"cell_type":"code","source":"df_train.info()\n# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in tqdm_notebook(df.columns):\n        col_type = df[col].dtype\n\n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df\n\ndf_train = reduce_mem_usage(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f907c8cae0d5adef7784e2e3bac3c98e99c129d9"},"cell_type":"code","source":"profile = pandas_profiling.ProfileReport(df_train.sample(50000))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1ee58b6f24acd3b9f43eec4d76e4abed52898d05"},"cell_type":"markdown","source":"# Language in Pages\nOne thing that might be interesting to look it is how the different languages used in Wikipedia might affect the dataset. I'll use a simple regular expression to search for the language code in the wikipedia URL. There are also a number of non-wikipedia URLs that will fail the regex search. These are wikimedia pages, so I'll give them the code 'na' since I haven't determined their language. Many of these will be things like images that do not really have a language."},{"metadata":{"trusted":false,"_uuid":"3e914cea38f26ec02f8cc7e4c1c4c7c1cff04e90"},"cell_type":"code","source":"def get_language(page):\n    res = re.search('[a-z][a-z].wikipedia.org',page)\n    if res:\n        return res[0][0:2]\n    return 'na'\n\ndf_train['lang'] = df_train.Page.map(get_language)\n\n\nprint(Counter(df_train.lang))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9023a1f5f32e2e2dcd9a2da428d71a6ae14c4be9"},"cell_type":"code","source":"lang_sets = {}\nlang_sets['en'] = df_train[df_train.lang=='en'].iloc[:,0:-1]\nlang_sets['ja'] = df_train[df_train.lang=='ja'].iloc[:,0:-1]\nlang_sets['de'] = df_train[df_train.lang=='de'].iloc[:,0:-1]\nlang_sets['na'] = df_train[df_train.lang=='na'].iloc[:,0:-1]\nlang_sets['fr'] = df_train[df_train.lang=='fr'].iloc[:,0:-1]\nlang_sets['zh'] = df_train[df_train.lang=='zh'].iloc[:,0:-1]\nlang_sets['ru'] = df_train[df_train.lang=='ru'].iloc[:,0:-1]\nlang_sets['es'] = df_train[df_train.lang=='es'].iloc[:,0:-1]\n\nsums = {}\nfor key in lang_sets:\n    sums[key] = lang_sets[key].iloc[:,1:].sum(axis=0) / lang_sets[key].shape[0]\ndays = [r for r in range(sums['en'].shape[0])]\n\nfig = plt.figure(1,figsize=[10,10])\nplt.ylabel('Views per Page')\nplt.xlabel('Day')\nplt.title('Pages in Different Languages')\nlabels={'en':'English','ja':'Japanese','de':'German',\n        'na':'Media','fr':'French','zh':'Chinese',\n        'ru':'Russian','es':'Spanish'\n       }\n\nfor key in sums:\n    plt.plot(days,sums[key],label = labels[key] )\n    \nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b03c4d1e55e997021e96764807264bd13417de6"},"cell_type":"markdown","source":"English shows a much higher number of views per page, as might be expected since Wikipedia is a US-based site. There is a lot more structure here than I would have expected. The English and Russian plots show very large spikes around day 400 (around August 2016), with several more spikes in the English data later in 2016. My guess is that this is the effect of both the Summer Olympics in August and the election in the US.\n\nThere's also a strange feature in the English data around day 200.\n\nThe Spanish data is very interesting too. There is a clear periodic structure there, with a ~1 week fast period and what looks like a significant dip around every 6 months or so."},{"metadata":{"_uuid":"f979dc31c47e22191d6a62e70ce8874962912c4e"},"cell_type":"markdown","source":"# Periodic Structure and FFTs\nSince it looks like there is some periodic structure here, I will plot each of these separately so that the scale is more visible. Along with the individual plots, I will also look at the magnitude of the Fast Fourier Transform (FFT). Peaks in the FFT show us the strongest frequencies in the periodic signal."},{"metadata":{"trusted":false,"_uuid":"019c8fa7e83503b86e2ba0434058cc904bf8d439"},"cell_type":"code","source":"from scipy.fftpack import fft\ndef plot_with_fft(key):\n\n    fig = plt.figure(1,figsize=[15,5])\n    plt.ylabel('Views per Page')\n    plt.xlabel('Day')\n    plt.title(labels[key])\n    plt.plot(days,sums[key],label = labels[key] )\n    \n    fig = plt.figure(2,figsize=[15,5])\n    fft_complex = fft(sums[key])\n    fft_mag = [np.sqrt(np.real(x)*np.real(x)+np.imag(x)*np.imag(x)) for x in fft_complex]\n    fft_xvals = [day / days[-1] for day in days]\n    npts = len(fft_xvals) // 2 + 1\n    fft_mag = fft_mag[:npts]\n    fft_xvals = fft_xvals[:npts]\n        \n    plt.ylabel('FFT Magnitude')\n    plt.xlabel(r\"Frequency [days]$^{-1}$\")\n    plt.title('Fourier Transform')\n    plt.plot(fft_xvals[1:],fft_mag[1:],label = labels[key] )\n    # Draw lines at 1, 1/2, and 1/3 week periods\n    plt.axvline(x=1./7,color='red',alpha=0.3)\n    plt.axvline(x=2./7,color='red',alpha=0.3)\n    plt.axvline(x=3./7,color='red',alpha=0.3)\n\n    plt.show()\n\nfor key in sums:\n    plot_with_fft(key)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c35035a9c64985fa123b07b2fa755c31a6f157ff"},"cell_type":"markdown","source":"# Some individual Entries\n"},{"metadata":{"trusted":false,"_uuid":"f9bc87a68e547d3120d2a345ea9870f0a72b2e52"},"cell_type":"code","source":"def plot_entry(key,idx):\n    data = lang_sets[key].iloc[idx,1:]\n    fig = plt.figure(1,figsize=(10,5))\n    plt.plot(days,data)\n    plt.xlabel('day')\n    plt.ylabel('views')\n    plt.title(df_train.iloc[lang_sets[key].index[idx],0])\n    \n    plt.show()\n\nidx = [10, 50, 100, 250,500,750,1000,1500,2000,3000,4000,5000]\nfor i in idx:\n    plot_entry('en',i)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"155817d89d93376afe9926bf23f0f78646d0b272"},"cell_type":"markdown","source":"# Lets look at the top pages\n"},{"metadata":{"trusted":false,"_uuid":"171c0fc3bea13b672290f3a7e579f4063b4e9786"},"cell_type":"code","source":"npages = 5\ntop_pages = {}\nfor key in lang_sets:\n    sum_set = pd.DataFrame(lang_sets[key][['Page']])\n    sum_set['total'] = lang_sets[key].sum(axis=1)\n    sum_set = sum_set.sort_values('total',ascending=False)\n    top_pages[key] = sum_set.index[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ae103b4d0951cb8fe79e23743a521699ab4434e"},"cell_type":"markdown","source":"# Making Predictions\nThe baseline given so far is for guessing 0 views for everything. There are a few other easy benchmarks we can try out:\n\nAverage number of views for that page (constant value)\nLinear regression\nMore complicated regression curves\nBut we've also seen that the data for related topics seems to be correlated and that topics that are in the news get a lot of traffic, so this maybe points to some ways to improve things. Unfortunately, with the different languages, training a model to identify related topics may be quite difficult. However, trying to cluster similar topics using just the data rather than the page name might help us out a bit. We could potentially smooth out some of the unpredictable spikes and could maybe use high-view pages to reduce effects of statistical fluctuations on low-view pages.\n\nI also wonder if something like a recurrent neural net might help out here if we want to try out more complicated methods.\n\nAs an example, I look at ARIMA models for a small set of pages.\n\n# ARIMA Models\nStatsmodels also includes things like ARMA and ARIMA models that can be used to make predictions from time series. This data is not necessarily very stationary and often has strong periodic effects, so these may not necessarily work very well. I'll look at ARIMA predictions for the same set of very high viewcount pages."},{"metadata":{"trusted":false,"_uuid":"15c639f84e20a47ef05e0b4afea66bc032164e2a"},"cell_type":"code","source":"from statsmodels.tsa.arima_model import ARIMA\nimport warnings\n\ncols = df_train.columns[1:-1]\nfor key in top_pages:\n    data = np.array(df_train.loc[top_pages[key],cols],'f')\n    result = None\n    with warnings.catch_warnings():\n        warnings.filterwarnings('ignore')\n        try:\n            arima = ARIMA(data,[2,1,4])\n            result = arima.fit(disp=False)\n        except:\n            try:\n                arima = ARIMA(data,[2,1,2])\n                result = arima.fit(disp=False)\n            except:\n                print(df_train.loc[top_pages[key],'Page'])\n                print('\\tARIMA failed')\n    #print(result.params)\n    pred = result.predict(2,599,typ='levels')\n    x = [i for i in range(600)]\n    i=0\n\n    plt.plot(x[2:len(data)],data[2:] ,label='Data')\n    plt.plot(x[2:],pred,label='ARIMA Model')\n    plt.title(df_train.loc[top_pages[key],'Page'])\n    plt.xlabel('Days')\n    plt.ylabel('Views')\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec69423da95e47c5ac6f740183fb981f297ae8f1"},"cell_type":"markdown","source":"# The Final Model\n\nBased on the Solutions provided by Arthur Suilin,CPMP and thousandvoices, Although they are 1st,2nd,3rd place, my model recreation would have got me 9th place. although its just a simple implementation!\n\nBased on the following:\nThe NN is a feedforward network with 200, 200, 100, 200 cells in each layer. Input is concatenated again with the output of the first layer. I don't know why but this boosted accuracy. Activation is relu except for last one which is linear. I used dropout of 0.5 for almost all layers (0.5 was selected by CV). I also used a batch normalization for the middle layer. Model is compiled with adam optimizer and the loss function defined above.I used 5 fold Cv for NN Batch size was huge: 4096. I found that increasing batch size was both decreasing training time, and improving accuracy. Training went for 200 epochs, computing actual training SMAPE every 10 epochs across all folds, and using median of all predictions of best epoch for submission. For each fold I trained 1 model, and took the median of their predictions"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"cb66719f46f1b96dcd5e660fb1acb1f4844384eb"},"cell_type":"code","source":"def init():\n    np.random.seed = 0\n    \ninit()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"f97ae64d298b4c5e99d660318574eadeb7d9f432"},"cell_type":"code","source":"def smape(y_true, y_pred):\n    denominator = (np.abs(y_true) + np.abs(y_pred)) / 2.0\n    diff = np.abs(y_true - y_pred) / denominator\n    diff[denominator == 0] = 0.0\n    return np.nanmean(diff)\n\ndef smape2D(y_true, y_pred):\n    return smape(np.ravel(y_true), np.ravel(y_pred))\n    \ndef smape_mask(y_true, y_pred, threshold):\n    denominator = (np.abs(y_true) + np.abs(y_pred)) \n    diff = np.abs(y_true - y_pred) \n    diff[denominator == 0] = 0.0\n    \n    return diff <= (threshold / 2.0) * denominator","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"fd437ed67cfe8176155b03ccc94ea2a629e9d31f"},"cell_type":"code","source":"max_size = 181 # number of days in 2015 with 3 days before end\n\noffset = 1/2\n\ntrain_all = pd.read_csv(\"../input/train_2.csv\")\ntrain_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"2ba2ba6865df65c71fb241324044ae15ae568418"},"cell_type":"code","source":"all_page = train_all.Page.copy()\ntrain_key = train_all[['Page']].copy()\ntrain_all = train_all.iloc[:,1:] * offset \ntrain_all.head()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"d24b99daf42cfd311546d81c9a6c4a32a6fb738f"},"cell_type":"code","source":"def get_date_index(date, train_all=train_all):\n    for idx, c in enumerate(train_all.columns):\n        if date == c:\n            break\n    if idx == len(train_all.columns):\n        return None\n    return idx","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"db599afefa861f295260ad2b214d645b96d4b152"},"cell_type":"code","source":"get_date_index('2016-09-13')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"2c12ff0577dd6622e2fe4377d166a19ad1eaf155"},"cell_type":"code","source":"get_date_index('2016-09-10')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"047220e5d0b949807f56593514e88606c196c3df"},"cell_type":"code","source":"train_all.shape[1] - get_date_index('2016-09-10')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e62ddeb9fe75fb2b6278ad0f9d97726521fc2a85"},"cell_type":"code","source":"get_date_index('2017-09-10') - get_date_index('2016-09-10')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"da39f186967b0494f5b5627347279f955f3a7404"},"cell_type":"code","source":"train_end = get_date_index('2016-09-10') + 1\ntest_start = get_date_index('2016-09-13')\n\ntrain = train_all.iloc[ : , (train_end - max_size) : train_end].copy().astype('float32')\ntest = train_all.iloc[:, test_start : (63 + test_start)].copy().astype('float32')\ntrain = train.iloc[:,::-1].copy().astype('float32')\n\ntrain_all = train_all.iloc[:,-(max_size):].astype('float32')\ntrain_all = train_all.iloc[:,::-1].copy().astype('float32')\n\ntest_3_date = test.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"15ccb657333d90408cd247074b7152354ce10dcc"},"cell_type":"code","source":"train_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"87828f9c48fa0aa2fdb2167d54683b94bfe996eb"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f83c7d3190947255e8b9584f60bcab57601a68a4"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"796dcedb5a069ef9991490afabf04d9356c26edb"},"cell_type":"code","source":"data = [page.split('_') for page in tqdm(train_key.Page)]\n\naccess = ['_'.join(page[-2:]) for page in data]\n\nsite = [page[-3] for page in data]\n\npage = ['_'.join(page[:-3]) for page in data]\npage[:2]\n\ntrain_key['PageTitle'] = page\ntrain_key['Site'] = site\ntrain_key['AccessAgent'] = access\ntrain_key.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1bf5f696e96f3c4a144a5dc0fb93370a192a0ed6"},"cell_type":"code","source":"train_norm = np.log1p(train).astype('float32')\ntrain_norm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"63d95f756089791ba6859068a3ddbc7294fd2429"},"cell_type":"code","source":"train_all_norm = np.log1p(train_all).astype('float32')\ntrain_all_norm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"6f70837a59482c902eeba60b8f9bee93c0d153b0"},"cell_type":"code","source":"first_day = 1 # 2016-09-13 is a Tuesday\ntest_columns_date = list(test.columns)\ntest_columns_code = ['w%d_d%d' % (i // 7, (first_day + i) % 7) for i in range(63)]\ntest.columns = test_columns_code\n\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"6d99880ffcad6a4809cee5c0e0ce5c94bf0bf6b6"},"cell_type":"code","source":"test.fillna(0, inplace=True)\n\ntest['Page'] = all_page\ntest.sort_values(by='Page', inplace=True)\ntest.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"266d3dae6635041330fc30804e1170a049b2fc53"},"cell_type":"code","source":"test = test.merge(train_key, how='left', on='Page', copy=False)\n\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"334f5aeac78b55182970c47c7eadbc2fe9ea8e32"},"cell_type":"code","source":"test_all_id = pd.read_csv('../input/key_2.csv')\n\ntest_all_id['Date'] = [page[-10:] for page in tqdm(test_all_id.Page)]\ntest_all_id['Page'] = [page[:-11] for page in tqdm(test_all_id.Page)]\ntest_all_id.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"0c0d4543023c663b50db3fdcb84571c00b1de8f1"},"cell_type":"code","source":"test_all = test_all_id.drop('Id', axis=1)\ntest_all['Visits_true'] = np.NaN\n\ntest_all.Visits_true = test_all.Visits_true * offset\ntest_all = test_all.pivot(index='Page', columns='Date', values='Visits_true').astype('float32').reset_index()\n\ntest_all['2017-11-14'] = np.NaN\ntest_all.sort_values(by='Page', inplace=True)\ntest_all.reset_index(drop=True, inplace=True)\n\ntest_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1d5321f48386d957659d2dce9c1cbacc412c9de0"},"cell_type":"code","source":"test_all.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"97578fcfc8f3a96a91f1d942ce571e92f2ab5808"},"cell_type":"code","source":"test_all_columns_date = list(test_all.columns[1:])\nfirst_day = 2 # 2017-13-09 is a Wednesday\ntest_all_columns_code = ['w%d_d%d' % (i // 7, (first_day + i) % 7) for i in range(63)]\ncols = ['Page']\ncols.extend(test_all_columns_code)\ntest_all.columns = cols\ntest_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"bbef3cc8cbd68f642b451d17552a75719fbf381c"},"cell_type":"code","source":"test_all = test_all.merge(train_key, how='left', on='Page')\ntest_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"6fe41c289ba74d479df5b86f0ce3f0f9f4626a24"},"cell_type":"code","source":"y_cols = test.columns[:63]\ny_cols","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"1f6adb571b32402420dd7b72a3f7d97f54d3f669"},"cell_type":"code","source":"test = test.reset_index()\ntest_all = test_all.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"efe241445c54a9272db3775f2a28132b79a2602c"},"cell_type":"code","source":"test_all.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e9e92a45fd8c5991cf41787980e781f1ac0a416a"},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ccc45a18ce69d5c04356706abd9582efa74549b0"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7a7fd8da6cbee3796b49133f2b2f4974434b6f3f"},"cell_type":"code","source":"test_all = test_all[test.columns].copy()\ntest_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5d7463ba210b59227f08094db9512d179c75dc53"},"cell_type":"code","source":"train_cols = ['d_%d' % i for i in range(train_norm.shape[1])]\nlen(train_cols)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"cf1e1df1572feb9df2d1f9a299b1c3df68eb3f8a"},"cell_type":"code","source":"train_norm.columns = train_cols\ntrain_all_norm.columns = train_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"04e65f46f739a7be05a2463bdc03c9041932654e"},"cell_type":"code","source":"train_norm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8efe79fbaf97ab0b2332228bf6be4df6c6913f84"},"cell_type":"code","source":"all(test[:test_all.shape[0]].Page == test_all.Page)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"d6d10310088b1f6bdfdddb23c2fee1686cd1e2f7"},"cell_type":"code","source":"sites = train_key.Site.unique()\nsites","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"67ff93a3a5f157bea41895628c8f433796c42dfb"},"cell_type":"code","source":"test_site = pd.factorize(test.Site)[0]\ntest['Site_label'] = test_site\ntest_all['Site_label'] = test_site[:test_all.shape[0]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"57301d0102d3d24d4371a8a0b55a5fddb993fdc0"},"cell_type":"code","source":"accesses = train_key.AccessAgent.unique()\naccesses","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"2a94f50dc1e1856eaddb5f795cbe04bbde91de36"},"cell_type":"code","source":"test_access = pd.factorize(test.AccessAgent)[0]\ntest['Access_label'] = test_access\ntest_all['Access_label'] = test_access[:test_all.shape[0]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"bece12da235e4757e29b68f8804b5c6bc0df0aa9"},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"340dbc0c2fbf74bc1ebb2fef95eaadcf8c76052b"},"cell_type":"code","source":"test_all.shape","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"3f244054e312350777a9c073d67838e7a439af08"},"cell_type":"code","source":"test0 = test.copy()\ntest_all0 = test_all.copy()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"454f8cb6917e6c5942cc334034d193c21f08252e"},"cell_type":"code","source":"y_norm_cols = [c+'_norm' for c in y_cols]\ny_pred_cols = [c+'_pred' for c in y_cols]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8f417800fa45a81e2d7cc9fc9680d7c041e8c272"},"cell_type":"code","source":"# all visits is median\ndef add_median(test, train,\n               train_key, periods, max_periods, first_train_weekday):\n    train =  train.iloc[:,:7*max_periods]\n    \n    df = train_key[['Page']].copy()\n    df['AllVisits'] = train.median(axis=1).fillna(0)\n    test = test.merge(df, how='left', on='Page', copy=False)\n    test.AllVisits = test.AllVisits.fillna(0).astype('float32')\n    \n    for site in sites:\n        test[site] = (1 * (test.Site == site)).astype('float32')\n    \n    for access in accesses:\n        test[access] = (1 * (test.AccessAgent == access)).astype('float32')\n\n    for (w1, w2) in periods:\n        \n        df = train_key[['Page']].copy()\n        c = 'median_%d_%d' % (w1, w2)\n        df[c] = train.iloc[:,7*w1:7*w2].median(axis=1, skipna=True) \n        test = test.merge(df, how='left', on='Page', copy=False)\n        test[c] = (test[c] - test.AllVisits).fillna(0).astype('float32')\n\n    for c_norm, c in zip(y_norm_cols, y_cols):\n        test[c_norm] = (np.log1p(test[c]) - test.AllVisits).astype('float32')\n\n    gc.collect()\n\n    return test\n\nmax_periods = 16\nperiods = [(0,1), (1,2), (2,3), (3,4), \n           (4,5), (5,6), (6,7), (7,8),\n           ]\n\n\nsite_cols = list(sites)\naccess_cols = list(accesses)\n\ntest, test_all = test0.copy(), test_all0.copy()\n\nfor c in y_pred_cols:\n    test[c] = np.NaN\n    test_all[c] = np.NaN\n\ntest1 = add_median(test, train_norm, \n                   train_key, periods, max_periods, 3)\n\ntest_all1 = add_median(test_all, train_all_norm, \n                       train_key, periods, max_periods, 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9d941732de1cfcf13ee074e10b10bb8593471797"},"cell_type":"code","source":"num_cols = (['median_%d_%d' % (w1,w2) for (w1,w2) in periods])\n\nimport keras.backend as K\n\ndef smape_error(y_true, y_pred):\n    return K.mean(K.clip(K.abs(y_pred - y_true),  0.0, 1.0), axis=-1)\n\n\ndef get_model(input_dim, num_sites, num_accesses, output_dim):\n    \n    dropout = 0.5\n    regularizer = 0.00004\n    main_input = Input(shape=(input_dim,), dtype='float32', name='main_input')\n    site_input = Input(shape=(num_sites,), dtype='float32', name='site_input')\n    access_input = Input(shape=(num_accesses,), dtype='float32', name='access_input')\n    \n    \n    x0 = keras.layers.concatenate([main_input, site_input, access_input])\n    x = Dense(200, activation='relu', \n              kernel_initializer='lecun_uniform', kernel_regularizer=regularizers.l2(regularizer))(x0)\n    x = Dropout(dropout)(x)\n    x = keras.layers.concatenate([x0, x])\n    x = Dense(200, activation='relu', \n              kernel_initializer='lecun_uniform', kernel_regularizer=regularizers.l2(regularizer))(x)\n    x = BatchNormalization(beta_regularizer=regularizers.l2(regularizer),\n                           gamma_regularizer=regularizers.l2(regularizer)\n                          )(x)\n    x = Dropout(dropout)(x)\n    x = Dense(100, activation='relu', \n              kernel_initializer='lecun_uniform', kernel_regularizer=regularizers.l2(regularizer))(x)\n    x = Dropout(dropout)(x)\n\n    x = Dense(200, activation='relu', \n              kernel_initializer='lecun_uniform', kernel_regularizer=regularizers.l2(regularizer))(x)\n    x = Dropout(dropout)(x)\n    x = Dense(output_dim, activation='linear', \n              kernel_initializer='lecun_uniform', kernel_regularizer=regularizers.l2(regularizer))(x)\n\n    model =  Model(inputs=[main_input, site_input, access_input], outputs=[x])\n    model.compile(loss=smape_error, optimizer='adam')\n    return model\n\ngroup = pd.factorize(test1.Page)[0]\n\nn_bag = 20\nkf = GroupKFold(n_bag)\nbatch_size=4096\n\n#print('week:', week)\ntest2 = test1\ntest_all2 = test_all1\nX, Xs, Xa, y = test2[num_cols].values, test2[site_cols].values, test2[access_cols].values, test2[y_norm_cols].values\nX_all, Xs_all, Xa_all, y_all = test_all2[num_cols].values, test_all2[site_cols].values, test_all2[access_cols].values, test_all2[y_norm_cols].fillna(0).values\n\ny_true = test2[y_cols]\ny_all_true = test_all2[y_cols]\n\nmodels = [get_model(len(num_cols), len(site_cols), len(access_cols), len(y_cols)) for bag in range(n_bag)]\n\nprint('offset:', offset)\nprint('batch size:', batch_size)\n\n\nbest_score = 100\nbest_all_score = 100\n\nsave_pred = 0\nsaved_pred_all = 0\n\nfor n_epoch in range(10, 201, 10):\n    print('************** start %d epochs **************************' % n_epoch)\n\n    y_pred0 = np.zeros((y.shape[0], y.shape[1]))\n    y_all_pred0 = np.zeros((n_bag, y_all.shape[0], y_all.shape[1]))\n    for fold, (train_idx, test_idx) in enumerate(kf.split(X, y, group)):\n        print('train fold', fold, end=' ')    \n        model = models[fold]\n        X_train, Xs_train, Xa_train, y_train = X[train_idx,:], Xs[train_idx,:], Xa[train_idx,:], y[train_idx,:]\n        X_test, Xs_test, Xa_test, y_test = X[test_idx,:], Xs[test_idx,:], Xa[test_idx,:], y[test_idx,:]\n\n        model.fit([ X_train, Xs_train, Xa_train],  y_train, \n                  epochs=10, batch_size=batch_size, verbose=0, shuffle=True, \n                  #validation_data=([X_test, Xs_test, Xa_test],  y_test)\n                 )\n        y_pred = model.predict([ X_test, Xs_test, Xa_test], batch_size=batch_size)\n        y_all_pred = model.predict([X_all, Xs_all, Xa_all], batch_size=batch_size)\n\n        y_pred0[test_idx,:] = y_pred\n        y_all_pred0[fold,:,:]  = y_all_pred\n\n        y_pred += test2.AllVisits.values[test_idx].reshape((-1,1))\n        y_pred = np.expm1(y_pred)\n        y_pred[y_pred < 0.5 * offset] = 0\n        res = smape2D(test2[y_cols].values[test_idx, :], y_pred)\n        y_pred = offset*((y_pred / offset).round())\n        res_round = smape2D(test2[y_cols].values[test_idx, :], y_pred)\n\n        y_all_pred += test_all2.AllVisits.values.reshape((-1,1))\n        y_all_pred = np.expm1(y_all_pred)\n        y_all_pred[y_all_pred < 0.5 * offset] = 0\n        res_all = smape2D(test_all2[y_cols], y_all_pred)\n        y_all_pred = offset*((y_all_pred / offset).round())\n        res_all_round = smape2D(test_all2[y_cols], y_all_pred)\n        print('smape train: %0.5f' % res, 'round: %0.5f' % res_round,\n              '     smape LB: %0.5f' % res_all, 'round: %0.5f' % res_all_round)\n\n    #y_pred0  = np.nanmedian(y_pred0, axis=0)\n    y_all_pred0  = np.nanmedian(y_all_pred0, axis=0)\n\n    y_pred0  += test2.AllVisits.values.reshape((-1,1))\n    y_pred0 = np.expm1(y_pred0)\n    y_pred0[y_pred0 < 0.5 * offset] = 0\n    res = smape2D(y_true, y_pred0)\n    print('smape train: %0.5f' % res, end=' ')\n    y_pred0 = offset*((y_pred0 / offset).round())\n    res_round = smape2D(y_true, y_pred0)\n    print('round: %0.5f' % res_round)\n\n    y_all_pred0 += test_all2.AllVisits.values.reshape((-1,1))\n    y_all_pred0 = np.expm1(y_all_pred0)\n    y_all_pred0[y_all_pred0 < 0.5 * offset] = 0\n    #y_all_pred0 = y_all_pred0.round()\n    res_all = smape2D(y_all_true, y_all_pred0)\n    print('     smape LB: %0.5f' % res_all, end=' ')\n    y_all_pred0 = offset*((y_all_pred0 / offset).round())\n    res_all_round = smape2D(y_all_true, y_all_pred0)\n    print('round: %0.5f' % res_all_round, end=' ')\n    if res_round < best_score:\n        print('saving')\n        best_score = res_round\n        best_all_score = res_all_round\n        test.loc[:, y_pred_cols] = y_pred0\n        test_all.loc[:, y_pred_cols] = y_all_pred0\n    else:\n        print()\n    print('*************** end %d epochs **************************' % n_epoch)\nprint('best saved LB score:', best_all_score)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f305aff175c838b01a580780f810dc9969232259"},"cell_type":"markdown","source":"# Conclusion:\n\nThe Three approaches all involved ensembling techniques, and i was fascinated as all three were quite different from each other.\nRecieved a score of 37.58649 which would place me in 9th position, which iam pretty happy about.\n\n\nI did try a ensemble technique but it took too long to run(2 days) there-fore stopped it.Above is the simple implementation of the Keras model of the 2nd place solution.His discussion was really helpful.\n\nThis Course was an informative journey, and was very pleased to be a part of."}],"metadata":{"anaconda-cloud":{},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}