{"cells":[{"metadata":{"_uuid":"dc4b79f66583a8b6ab7b73d4df5348fe7154bcdb"},"cell_type":"markdown","source":"## (Updated) Keras + CNN: Use time series data as is \nI previously posted [Keras + RNN(GRU) to handle passbands as timeseries](https://www.kaggle.com/higepon/keras-rnn-gru-to-handle-passbands-as-timeseries). Although it's natural to use GRU for time series input, it turns out that GRU is relatively slow to train and it was painful.  [Mithrillion](https://www.kaggle.com/mithrillion) kindly advised me to use CNN instead. Here I put up relatively simple CNN based model.\n\nValidation accuracy is not very high for this model, but I think this woudl be great starter who wants to use CNN.\nI'd appreciate your feedback!\n\nFYI: I got great improvement on my validation accuracy with this setup + alpha. I think this is generally right direction."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport sys\nimport gc\nimport tensorflow as tf\nimport keras.backend as K\nfrom keras import regularizers\nfrom keras.utils import to_categorical\nfrom keras.callbacks import EarlyStopping, TensorBoard\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D, Conv1D, MaxPooling1D, Lambda\nfrom keras.layers import GRU, Dense, Activation, Dropout, concatenate, Input, BatchNormalization\nfrom keras.callbacks import ReduceLROnPlateau,ModelCheckpoint\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import StratifiedKFold\nfrom collections import Counter\nimport matplotlib.pyplot as plt\nimport warnings\nimport os\nimport pickle\nimport time\nfrom tensorflow.python.client import timeline\nimport re\nimport time\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/training_set.csv')\ntrain.head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2997834c213eebccf21642bd7925640bbf8a4987"},"cell_type":"markdown","source":"## Standardize input\nNeural networks works better when inputs are standardized."},{"metadata":{"trusted":true,"_uuid":"db8b93d64f6fd3fb21b599ac6214bfbcbafa10a8"},"cell_type":"code","source":"ss1 = StandardScaler()\ntrain[['mjd', 'flux', 'flux_err']] = ss1.fit_transform(train[['mjd', 'flux', 'flux_err']])\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a178e24b339680969e4db784760abc513ca425b"},"cell_type":"markdown","source":"## Sort\nSort train data before we group them just in case."},{"metadata":{"trusted":true,"_uuid":"aff66c2966807ec57a2efff93305fc8ce109ebaa"},"cell_type":"code","source":"train = train.sort_values(['object_id', 'passband', 'mjd'])\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1286b07fa0df84ae3cfb407c2e86de2a52da99c"},"cell_type":"markdown","source":"## Time series transformation\nThis maybe not be very easy to understand, but basically we are transforming train data into 2D data [num_passbands, len(flux) + len(flux_err) + len(detected)] as below. So we can say, for each object_id we have one monotone image which has width=num_passbands, height=len(flux) + len(flux_err) + len(detected."},{"metadata":{"trusted":true,"_uuid":"37b8a2172838daa7bb794139f92f28edcdfc4772"},"cell_type":"code","source":"train_timeseries = train.groupby(['object_id', 'passband'])['flux', 'flux_err', 'detected'].apply(lambda df: df.reset_index(drop=True)).unstack()\ntrain_timeseries.fillna(0, inplace=True)\n\n# rename column names\ntrain_timeseries.columns = ['_'.join(map(str,tup)).rstrip('_') for tup in train_timeseries.columns.values]\ntrain_timeseries.head(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bcd9d1fb56a4a067dda569c9b7a1f72fab4a6003"},"cell_type":"code","source":"num_columns = len(train_timeseries.columns)\nnum_columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c1a24ee2a13070053326620f852fe63c8adde52c"},"cell_type":"markdown","source":"We reshape the data into [None, num_columns, num_passbands."},{"metadata":{"trusted":true,"_uuid":"1e53efc5d3a5dcde3d5c7ab1f5ebf4d0c80be68d"},"cell_type":"code","source":"X_train = train_timeseries.values.reshape(-1, 6, num_columns).transpose(0, 2, 1)\nX_train","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c82a356f2b86c3ab7e03087821fbb2f4327382bb"},"cell_type":"markdown","source":"## Load metadata and construct target value"},{"metadata":{"trusted":true,"_uuid":"99c88e771495afe4c9830278942478b643c09ea0"},"cell_type":"code","source":"meta_train = pd.read_csv('../input/training_set_metadata.csv')\nmeta_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d4fc85bbc113ddd788c294fcca4a6165aa0e578"},"cell_type":"code","source":"classes = sorted(meta_train.target.unique())\nclasses","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a03ce2aa033d7668659269cc4a3f0cb4a60a6f3"},"cell_type":"code","source":"class_map = dict()\nfor i,val in enumerate(classes):\n    class_map[val] = i\nclass_map","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"38bc9a0c344ada28bb50f796a1cfa1bd9191d997"},"cell_type":"code","source":"train_timeseries0 = train_timeseries.reset_index()\ntrain_timeseries0 = train_timeseries0[train_timeseries0.passband == 0]\ntrain_timeseries0.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7f88e50112e2c4be735e7b180e017bf7503222c"},"cell_type":"code","source":"merged_meta_train = train_timeseries0.merge(meta_train, on='object_id', how='left')\nmerged_meta_train.fillna(0, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7eab962e2700d89085908ee2445a8c2c6789b942"},"cell_type":"code","source":"y = merged_meta_train.target\nclasses = sorted(y.unique())\n\n# Taken from Giba's topic : https://www.kaggle.com/titericz\n# https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n# with Kyle Boone's post https://www.kaggle.com/kyleboone\nclass_weight = {\n    c: 1 for c in classes\n}\nfor c in [64, 15]:\n    class_weight[c] = 2\n\nprint('Unique classes : ', classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c688c25aaaa21ee15cec4cb40282f3c21bc5dd9"},"cell_type":"code","source":"targets = merged_meta_train.target\ntarget_map = np.zeros((targets.shape[0],))\ntarget_map = np.array([class_map[val] for val in targets])\nY = to_categorical(target_map)\nY.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48f3a97cac6042032c210f18bdbb8380c2b4ae70"},"cell_type":"code","source":"Y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9e855853651cce035a68140147319fc1a271654"},"cell_type":"code","source":"def multi_weighted_logloss(y_ohe, y_p):\n    \"\"\"\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"\n    classes = [6, 15, 16, 42, 52, 53, 62, 64, 65, 67, 88, 90, 92, 95]\n    class_weight = {6: 1, 15: 2, 16: 1, 42: 1, 52: 1, 53: 1, 62: 1, 64: 2, 65: 1, 67: 1, 88: 1, 90: 1, 92: 1, 95: 1}\n    # Normalize rows and limit y_preds to 1e-15, 1-1e-15\n    y_p = np.clip(a=y_p, a_min=1e-15, a_max=1-1e-15)\n    # Transform to log\n    y_p_log = np.log(y_p)\n    # Get the log for ones, .values is used to drop the index of DataFrames\n    # Exclude class 99 for now, since there is no class99 in the training set \n    # we gave a special process for that class\n    y_log_ones = np.sum(y_ohe * y_p_log, axis=0)\n    # Get the number of positives for each class\n    nb_pos = y_ohe.sum(axis=0).astype(float)\n    # Weight average and divide by the number of positives\n    class_arr = np.array([class_weight[k] for k in sorted(class_weight.keys())])\n    y_w = y_log_ones * class_arr / nb_pos    \n    loss = - np.sum(y_w) / np.sum(class_arr)\n    return loss\n\ndef plot_loss_acc(history):\n    plt.plot(history.history['loss'][1:])\n    plt.plot(history.history['val_loss'][1:])\n    plt.title('model loss')\n    plt.ylabel('val_loss')\n    plt.xlabel('epoch')\n    plt.legend(['train','Validation'], loc='upper left')\n    plt.show()\n    \n    plt.plot(history.history['acc'][1:])\n    plt.plot(history.history['val_acc'][1:])\n    plt.title('model Accuracy')\n    plt.ylabel('val_acc')\n    plt.xlabel('epoch')\n    plt.legend(['train','Validation'], loc='upper left')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3b8e55f2963f9829e63398cdca40eb2226737339"},"cell_type":"markdown","source":"## Actual CNN begins here"},{"metadata":{"trusted":true,"_uuid":"9e4c6561753c3d8ed065aeff593663ea6231430f"},"cell_type":"code","source":"batch_size = 256\n\ndef weight_variable(shape, name=None):\n    return np.random.normal(scale=.01, size=shape)\n\ndef build_model():\n    input = Input(shape=(X_train.shape[1], 6), dtype='float32', name='input0')\n    output = Conv1D(256,\n                 kernel_size=80,\n                 strides=4,\n                 padding='same',\n                 kernel_initializer='glorot_uniform',\n                 kernel_regularizer=regularizers.l2(l=0.0001))(input)\n    output = BatchNormalization()(output)\n    output = Activation('relu')(output)\n    output = MaxPooling1D(pool_size=4, strides=None)(output)\n    output = Conv1D(256,\n                 kernel_size=3,\n                 strides=1,\n                 padding='same',\n                 kernel_initializer='glorot_uniform',\n                 kernel_regularizer=regularizers.l2(l=0.0001))(output)\n    output = BatchNormalization()(output)\n    output = Activation('relu')(output)\n    output = MaxPooling1D(pool_size=4, strides=None)(output)\n    output = Lambda(lambda x: K.mean(x, axis=1))(output) # Same as GAP for 1D Conv Layer\n    output = Dense(len(classes), activation='softmax')(output)\n    model = Model(inputs=input, outputs=output)\n    return model\n\n# https://www.kaggle.com/c/PLAsTiCC-2018/discussion/69795\ndef mywloss(y_true,y_pred):  \n    yc=tf.clip_by_value(y_pred,1e-15,1-1e-15)\n    loss=-(tf.reduce_mean(tf.reduce_mean(y_true*tf.log(yc),axis=0)/wtable))\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f38c69351f1579285c7787d717ffba849c513dc"},"cell_type":"code","source":"epochs = 1000\ny_count = Counter(target_map)\nwtable = np.zeros((len(classes),))\nfor i in range(len(classes)):\n    wtable[i] = y_count[i] / target_map.shape[0]\n\ny_map = target_map\ny_categorical = Y\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)\nstart = time.time()\nclfs = []\noof_preds = np.zeros((len(X_train), len(classes)))\n\nmodel_file = \"model.weigths\"\n\nfor fold_, (trn_, val_) in enumerate(folds.split(y_map, y_map)):\n    checkPoint = ModelCheckpoint(model_file, monitor='val_loss',mode = 'min', save_best_only=True, verbose=0)\n\n    x_train, y_train = X_train[trn_], Y[trn_]\n    x_valid, y_valid = X_train[val_], Y[val_]\n    \n    model = build_model()    \n    optimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999)\n    stopping = EarlyStopping(monitor='val_loss', patience=60, verbose=0, mode='auto')\n\n    model.compile(loss=mywloss, optimizer=optimizer, metrics=['accuracy'])\n    history = model.fit(x_train, y_train,\n                    validation_data=[x_valid, y_valid], \n                    epochs=epochs,\n                        batch_size=batch_size,\n                    shuffle=False,verbose=1,callbacks=[checkPoint, stopping])           \n    plot_loss_acc(history)\n    \n    print('Loading Best Model')\n    model.load_weights(model_file)\n    # # Get predicted probabilities for each class\n    oof_preds[val_, :] = model.predict(x_valid,batch_size=batch_size)\n    print(multi_weighted_logloss(y_valid, model.predict(x_valid,batch_size=batch_size)))\n    clfs.append(model)\n    \nprint('MULTI WEIGHTED LOG LOSS : %.5f ' % multi_weighted_logloss(Y,oof_preds))\n\nelapsed_time = time.time() - start\nprint(\"elapsed_time:\", elapsed_time)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cdce546445171fbad2aafc613f669fc17e2ccea8"},"cell_type":"markdown","source":"## Ideas for improvement\n- We need to find a proper size of the model, I think it's currently overfitting.\n    -  Or more data, I believe we can't use data augmantation though\n- Add more derived features."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}