{"cells":[{"metadata":{"_uuid":"14d580cec5d864524656207b21e77a63c2944916"},"cell_type":"markdown","source":"# Time Series CNN"},{"metadata":{"_uuid":"247249e86dcf9afc0a59aafba0d17df8848ce5e3"},"cell_type":"markdown","source":"## Overview\n\nIn the following notebook you can see the CNN that was used as part of the 3rd place solution in the 2018 PLAsTiCC kaggle challenge!\n\nOur CNN is a Fully 1D Convolutional Neural Network with 256 * 8,5,3 convolution kernels followed by a GlobalMaxPulling , this FCN is close to the one described here [\"Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline\"]( https://arxiv.org/pdf/1611.06455.pdf)\n\nTo the output of the last layer is concatenated with the output of a small MLP whose inputs are some meta features which would be described later.\n\nThe inputs to the convolution layer is an 18 channels 128 long vectors:\n\n1. 6 channels are built by linear interpolation of the Flux time series - one channel to each passband\n2. 6 channels are built by linear interpolation of Flux*detected time series\n3. 6 channels describe the distance of the current sampling point to the nearest valid sampling point in the corresponding channel (i.e. this channel have a high value if the interpolation is done with very distant points)\n\nThere were 3 different NNs with different meta features as inputs to the MLP:\n\n1. Minimal - only 4 meta features:  hostgal_photoz, distmod, delta mjd, Flux std (the two later ones are designed to keep the normalization data) \n2. Minimal + 16 best features from @mamas's model\n3.  Minimal + 15 best features from @nyanp's model\n\nFor augmentation during training we used:\n\n\n**Before interpolation**\n\n1. Deletion - up to 30% of the time samples in the training set where deleted (the deletion was done over all the set, not per object)\n2. Noise - random value proportional to  flux_err\n\n**After interpolation**\n\n1. Cyclic shift\n2.  Skew - every flux channel was multiplied by 1+k where k is a small random number\n\t\n    "},{"metadata":{"_uuid":"1e53b8f7907ddeb2843611aff0d234c71e273b26"},"cell_type":"markdown","source":"## Extra files\n\nI already prepared and uploaded the following files:\n\n* Nyanp's and Mamas's extra features files\n* 4fold training and validation files with extra features\n* Extra features for test meta\n* Best weights for the 3 NN\n\nThe notebook I used to prepare the files is also uploaded and you can download it to understand how the files were prepared\n\nAnother file which is worthwhile downloading and inspecting is generators.py where the timeseries are interpolated and augmented on the fly. \n\n\n"},{"metadata":{"_uuid":"ceea10799f7e13a702233227cf1be96442108b35"},"cell_type":"markdown","source":"## How to use\n\nTo control the notebooks activity and parameters change the values in the \"Running Params\" block below\n\n### Training\nTo train a network from scratch set pre_weight_ext=None and test_train_switch='Train'\nTo get the best training weights I used a few rounds of training,:\n\n1.  starting with low deletion rate and noise and saving weights\n2. Retrain with more deletion and noise with the previous weight as a starting point\n3. Repeat 2 with more noise and deletion\n\t\nI used 3-4 loops to get to the best weights \n\n### Testing \nset test_train_switch='Test'\n\nset submit_file to save a submitable file\n\n### Using your features\nYou can use your features instead of nyanp's or mamas's.\n\nJust normelize then use the numbers 0..n as colums heads, and upload as csv\n\nFrom my experience too many features degrade the score, use 8-20 extra features (choose your best)\n\n"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"da114038e3216d5841c6297ed58a542816c014d5"},"cell_type":"code","source":"\nimport os \nimport gc\nimport sys\nsys.path.append('../input/plasticc-extra')\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm, tqdm_notebook\nimport pandas as pd \nimport time\nimport pdb\nfrom tqdm import tqdm\nfrom keras.models import Model\nfrom keras import optimizers\nfrom keras import layers\nfrom keras.layers import Activation,Flatten, Dense, Dropout,Conv1D, GlobalMaxPooling1D,LeakyReLU,BatchNormalization,Input,ReLU\nfrom keras.layers.merge import concatenate\nimport keras\nfrom keras.activations import sigmoid, softmax\nfrom generators import load_process,prepare_process\nfrom generators import calculate_inputs\nimport tensorflow as tf\nimport pickle\nfrom keras import regularizers\nfrom keras.utils import to_categorical\nimport concurrent.futures\nfrom multiprocessing import Process, Queue, current_process, freeze_support\n#make wider graphs\n\nplt.figure(figsize=(12,5))\n\n# the following two lines are for changing imported functions, and not needing to restart kernel to use their updated version\n%load_ext autoreload\n%autoreload 2\n%matplotlib inline\n\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\n\nDATA_DIR = '../input/PLAsTiCC-2018/'\nmy_DATA_DIR='../input/plasticc-extra/'\ncolor_map = {0:'g', 1:'r', 2:'c', 3:'m', 4:'y', 5:'k'}\ngalactic_targets = np.array([6,16,53,65,92])\nextragalactic_targets = np.array([15,42,52,62,64,67,88,90,95])\nkw = [1,1,1,1,1,2,1,1,1,2,1,1,1,1]\nreal_targets = np.append(galactic_targets,extragalactic_targets)\nTargets = np.append(real_targets,99)\n\ngalactic=np.array([1,0,1,0,0,1,0,0,1,0,0,0,1,0])            \nex_galactic = 1-galactic\nnum_samples=128","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4250e417c5ec33e0760b3eaa0a008cb23b96e130"},"cell_type":"code","source":"#################################################################\n#  Running Params\n################################################################\ntest_train_switch=None      # Values : 'Train', 'Test', None (if none, use on training set to calculate CV)\nSubmit_name=None   # if not None will submit\nmeta_type='Nyanp'           # meta features type: None, 'Mamas', 'Nyanp' \nfolds=range(4)\nmax_workers=3               # for multiprocessing\n##########################################\n# Test Parameters\nweight_ext='best'        # traing weight named: model16fr{0}{1}_{2}.weights (if training set weights_ext=post_weights_ext)\n                         # {0} - number of extraf features 4 if not added features, 19 - added nyanps, 20 - added mamas\n                         # {1} - pre_weights_ext\n                         # {2} -  fold number\nsubmit_name=None     # Submit file name. if None - don't submit\n##########################################\n# Training parameters\npre_weight_ext=None      # if not None - use pre trasined weights\npost_weight_ext='a'      # must not be None - used for naming training result\ntraining_trails = 4\naug1={'shift':32, 'skew':0.05,'sample':0.85,'noise':0.3} #Augmentation values for corse training\naug2={'shift':128, 'skew':0.05,'sample':0.85,'noise':0.3} #Augmentation values for fine training\n                                                            # shift: control cyclic shift\n                                                            # skew: every channel is multiplied by (1+rand.norm(skew))\n                                                            # sample: determin how which precentage of the data set would be used\n                                                            #         the other samples will be deleted\n                                                            # noise: added noise as proportion of flux_err\n# learning rates for corse training\nlr_corse=np.array([1e-3,1e-3,1e-3,3e-4,3e-4,3e-4,1e-4,1e-4,1e-4,3e-4,3e-4,3e-4,1e-4,1e-4,1e-4,3e-5,3e-5,3e-5,\n             1e-3,3e-4,1e-4,3e-5,3e-5,3e-5,1e-5,1e-5,1e-5,3e-6,3e-6,3e-6,1e-5,1e-5,1e-5,3e-6,\n                                     3e-6,3e-6,1e-5,1e-5,1e-5,3e-6,3e-6,3e-6,3e-6,3e-6,3e-6]) \n# learning rates for fine training\nlr_fine=np.array([3e-6,3e-6,3e-6,3e-5,3e-6,3e-6,3e-5,3e-6,3e-4,3e-5,3e-6,3e-6,3e-5,3e-6,3e-6,3e-5,3e-6,3e-6])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e4eabe3b22356bbece155227ede86f6021a8bc0"},"cell_type":"code","source":"\ndf_timeseries=pd.read_csv(DATA_DIR+'training_set.csv')\ndf_training_meta=[]\ndf_validation_meta=[]\nfor i in folds:\n    df_training_meta.append(pd.read_csv('{}training_meta{}.csv'.format(my_DATA_DIR,i)).sort_values('object_id').reset_index(drop=True))\n    df_validation_meta.append(pd.read_csv('{}validation_meta{}.csv'.format(my_DATA_DIR,i)).sort_values('object_id').reset_index(drop=True))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e17e42255b0296921a74c61ca64a76a75e16e4b1"},"cell_type":"code","source":"training_meta=pd.read_csv(DATA_DIR+'training_set_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"284d82d2c76fe41aa505aa7b087272d653f92fae"},"cell_type":"code","source":"tar_dict=dict(zip(real_targets,range(len(real_targets))))\nty=to_categorical(training_meta.target.map(tar_dict))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9a5445f8edb1c912a7321a97075e7c0fdae164b"},"cell_type":"code","source":"wtable=ty.sum(axis=0)/ty.shape[0]\nwtable\nwtable=wtable/kw\nwtable=wtable/wtable.sum()\nwtable\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f5d9c6116f27fb4aa846f8f207a9d2d63f2133f"},"cell_type":"code","source":"def mywloss(y_true,y_pred):\n    \n    yc=tf.clip_by_value(y_pred,1e-15,1-1e-15)\n    loss=-(tf.reduce_mean(tf.reduce_mean(y_true*tf.log(yc),axis=0)/wtable))\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3632ce5212f8609b60b1de2d3a65120521bc401"},"cell_type":"code","source":"def score(y_pred,y_true):\n    y=np.clip(y_pred,1e-3,1)\n    for i in range(y.shape[0]):\n        y[i,...]=y[i,...]/y[i,...].sum()\n    return -(np.mean(np.mean(y_true*np.log(y),axis=0)/wtable))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a23160db93eb37cf49972d7c42ff7be03e353d4"},"cell_type":"code","source":"# This layer if forcing galactic/extra galactic object to output 0\n\nfrom keras import backend as K\nfrom keras.engine.topology import Layer\n\nclass MySwitch(Layer):\n\n    def __init__(self ,split,**kwargs):\n        self.split=split\n        super(MySwitch, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert isinstance(input_shape, list)\n        assert self.split<=input_shape[1][-1]\n        self.nsplit=input_shape[1][-1]\n        # Create a trainable weight variable for this layer.\n        super(MySwitch, self).build(input_shape)  # Be sure to call this at the end\n\n    def call(self, x):\n        assert isinstance(x, list)\n        x0=tf.equal(x[0],tf.zeros_like(x[0]))\n        x1=tf.logical_not(x0)\n        xt0=tf.tile(x0,[1,self.split])\n        xt1=tf.tile(x1,[1,self.nsplit-self.split])\n        xf=tf.concat([xt0,xt1],axis=-1)\n        return tf.where(xf,x[1],-1e4*tf.ones_like(x[1]))\n\n    def compute_output_shape(self, input_shape):\n        assert isinstance(input_shape, list)\n        shape_a, shape_b = input_shape\n        return (shape_b[0],shape_b[1])\n","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"0e445f8d15e264abc8f3d32d66bc12e842e1e504"},"cell_type":"code","source":"def build_model16():\n    input_timeseries = Input(shape=(num_samples, 6,),name='input_timeseries')\n    input_timeseries0 = Input(shape=(num_samples, 6,),name='input_timeseries0')\n    input_timeseriese = Input(shape=(num_samples, 6,),name='input_timeseriese')\n    input_meta = Input(shape=(6,),name='input_meta')\n    input_gal = Input(shape=(1,),name='input_gal')\n    _series=concatenate([input_timeseries,input_timeseries0,input_timeseriese])\n    x = Conv1D(256,8,padding='same',name='Conv1')(_series)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.1)(x)\n    x = Dropout(0.2)(x)\n    x = Conv1D(256,5,padding='same',name='Conv2')(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.1)(x)\n    x = Dropout(0.2)(x)\n    x = Conv1D(256,3,padding='same',name='Conv5')(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.1)(x)\n    x = GlobalMaxPooling1D()(x)\n    x1 = Dense(16,activation='relu',name='dense0')(input_meta)\n    x1 = Dense(32,activation='relu',name='dense1')(x1)\n    xc = concatenate([x,x1],name='concat')\n    x = Dense(256,activation='relu',name='features')(xc)\n    x = Dense(real_targets.shape[0],name='bout')(x)\n    x = MySwitch(galactic_targets.shape[0])([input_gal,x])\n    out = Activation('softmax',name='out')(x)\n    model=Model([input_timeseries,input_timeseries0,input_timeseriese,input_meta,input_gal],out)\n    return model\nbuild_model16().summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"159906c7d0b2c12da91324bd540d4f1dab7ad01a"},"cell_type":"code","source":"# build MLP for extra data and concat to CNN output \ndef build_top(ntop,convout,n_first=64):\n    input_meta = Input(shape=(ntop,),name='input_meta')\n    input_gal = Input(shape=(1,),name='input_gal')\n    x = BatchNormalization()(convout)\n    x = LeakyReLU(alpha=0.1)(x)\n    x = GlobalMaxPooling1D()(x)\n    x1 = Dense(n_first,activation='relu',name='dense0')(input_meta)\n    x1 = Dropout(0.2)(x1)  #was 0.2\n    x1 = BatchNormalization()(x1)\n    x1 = Dense(64,activation='relu',name='dense2')(x1)\n    xc = concatenate([x,x1],name='concat')\n    x = Dense(256,activation='relu',name='features')(xc)\n    x = Dense(real_targets.shape[0],name='bout')(x)\n    x = MySwitch(galactic_targets.shape[0])([input_gal,x])\n    out = Activation('softmax',name='out')(x)\n    return [input_meta,input_gal],out\n","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"76312b206aae34708a3778806e6932f487332556"},"cell_type":"code","source":"# Change the Original CNN extra data MLP\ndef model_retop(model,ntop,last_conv_layer_name='Conv5',n_first=64):\n    convout = model.get_layer(name=last_conv_layer_name).output\n    inputs=[]\n    for inp in model.inputs:\n        if 'timeseries' in inp.name:\n            inputs.append(inp)\n    dinputs,out=build_top(ntop,convout,n_first=n_first)\n    inputs=inputs+dinputs\n    topmodel=Model(inputs,out)\n    return topmodel\nmodel_retop(build_model16(),8).summary()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"79237ef704eb1bc54555658e1f787cbcbc538b72"},"cell_type":"code","source":"# output from verious layers in the model\ndef model_tap(model,last_conv_layer_name='Conv5'):\n    convout = model.get_layer(name=last_conv_layer_name).output\n    x = BatchNormalization()(convout)\n    x = LeakyReLU(alpha=0.1)(x)\n    out2 = GlobalMaxPooling1D()(x)\n    out=model.get_layer(name='out').output\n    out1=model.get_layer(name='out').input\n    return Model(model.inputs,outputs=[out,out1,out2])\nmodel_tap(build_model16()).summary()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3ea2adc304355d779d1598c38ecc5468538f8e37"},"cell_type":"code","source":"opt=optimizers.Adam()\n\nopt.lr=1e-3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3c4c5c5a2fba1da4ce1a3be4698bb855b2160b4"},"cell_type":"code","source":"meta_cols=['hostgal_photoz','dmjd','std_flux','distmod']\nif (meta_type=='Nyanp'):\n    nyanp_train=pd.read_csv('{}nyanp_train.csv'.format(my_DATA_DIR)).set_index('object_id')\n    nyanp_train=nyanp_train.rename(columns=int)\n    nyanp_train.head()\n    df_add=nyanp_train\n    meta_cols.extend(range(15))\nelif (meta_type=='Mamas'):\n    mamas_train=pd.read_csv('{}mamas_train.csv'.format(my_DATA_DIR)).set_index('object_id')\n    mamas_train=mamas_train.rename(columns=int)\n    mamas_train.head()\n    df_add=mamas_train\n    meta_cols.extend(range(16))\nmeta_size=len(meta_cols)\ncol_weights=np.ones(meta_size)\n    \nif meta_type:\n    for i in folds:\n            df_validation_meta[i]=df_validation_meta[i].merge(df_add,left_on='object_id',right_index=True,how='left').sort_values('object_id').reset_index(drop=True).copy()\n            df_training_meta[i]=df_training_meta[i].merge(df_add,left_on='object_id',right_index=True,how='left').sort_values('object_id').reset_index(drop=True).copy()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"0b0d1ba33138a214f917ea52d5847b62cd632a1d"},"cell_type":"code","source":"# Training\nif test_train_switch=='Train':\n    opt=optimizers.Adam()\n    opt.lr=1e-3\n\n\n    myhist={'val_loss':[],\n            'val_categorical_accuracy':[],\n            'loss':[],\n            'categorical_accuracy':[],\n            'lr':[],\n            'score':[]}\n\n    # Loop over all Folds\n    fold=tqdm_notebook(folds,desc='fold:')\n    for f in fold:\n        smin1=100\n        # Prepare validation data\n        validate_timeseries,validate_timeseries0,validate_void,validate_meta,validate_switch,validate_y=\\\n            calculate_inputs(df_timeseries,df_validation_meta[f],meta_cols,col_weights,real_targets,length=num_samples,aug=None,return_y=True)\n        tq=tqdm_notebook(range(training_trails),desc='full:')\n        # Corse loop\n        for p in tq:\n            weights_arr=[]\n            val_loss_arr=[]\n\n            lr=lr_corse\n            smin=100\n            executor=concurrent.futures.ProcessPoolExecutor(max_workers=max_workers)\n            futurs=[]\n            modelb = build_model16()\n            model=model_retop(modelb,meta_size)\n            if pre_weight_ext:\n                model.load_weights('{}model16fr{}{}_{}.weights'.format(my_DATA_DIR,meta_size,pre_weight_ext,f))\n            model.compile(opt,loss=mywloss,metrics=['categorical_accuracy'])\n            aug_vals=aug1\n            for k in range(lr.shape[0]):\n                futurs.append(executor.submit(calculate_inputs,df_timeseries,df_training_meta[f],\\\n                                              meta_cols,col_weights,real_targets,length=num_samples,aug=aug_vals,return_y=True))\n            tq1=tqdm_notebook(zip(lr,concurrent.futures.as_completed(futurs)),leave=False,total=lr.shape[0])\n            # Epoch loop - use different learning rates\n            for opt.lr,future in tq1:\n                sr,sr0,srv,train_meta,train_switch,train_y=future.result(timeout=200)\n                _=gc.collect()\n                history=model.fit([sr,sr0,srv,train_meta,train_switch],train_y,batch_size=64,epochs=1,\n                                validation_data=([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],validate_y),\n                                                    verbose=0 )\n                y=model.predict([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],batch_size=64)\n\n                s=score(y,validate_y)\n                myhist['val_loss'].extend(history.history['val_loss'])\n                myhist['val_categorical_accuracy'].extend(history.history['val_categorical_accuracy'])\n                myhist['loss'].extend(history.history['loss'])\n                myhist['categorical_accuracy'].extend(history.history['categorical_accuracy'])\n                myhist['lr'].append(opt.lr)\n                myhist['score'].append(s)\n                val_loss_arr.append(s)\n                weights_arr.append(model.get_weights())\n                if s<smin:\n                    smin=s\n                    weights_min=model.get_weights()\n                tq1.set_postfix(lr=opt.lr,current_score=s,loss=history.history['loss'][0],val_loss=history.history['val_loss'][0])\n                tq.set_postfix(temp_min=smin,total_min=min(smin,smin1))\n            executor.shutdown(wait=True)\n\n            lr=lr_fine\n            aug_vals=aug2\n            m=np.argsort(val_loss_arr)\n            # Use the average of 5 best weights for next step\n            new_weights=[]\n            for ii in range(len(weights_arr[0])):\n                new_weights.append(np.stack([weights_arr[m[kk]][ii] for kk in range(5) ],axis=0).mean(axis=0))\n\n            executor=concurrent.futures.ProcessPoolExecutor(max_workers=max_workers)\n            futurs=[]\n\n            modelb = build_model16()\n            model=model_retop(modelb,meta_size)        \n            model.set_weights(new_weights)\n            model.compile(opt,loss=mywloss,metrics=['categorical_accuracy'])\n            weights_arr=[]\n            val_loss_arr=[]\n            # fine training loop\n            for k in range(lr.shape[0]):\n                futurs.append(executor.submit(calculate_inputs,df_timeseries,df_training_meta[f],\\\n                                              meta_cols,col_weights,real_targets,length=num_samples,aug=aug_vals,return_y=True))\n            tq2=tqdm_notebook(zip(lr,concurrent.futures.as_completed(futurs)),leave=False,total=lr.shape[0])\n            for opt.lr,future in tq2:\n\n                sr,sr0,srv,train_meta,train_switch,train_y=future.result(timeout=200)\n                _=gc.collect()\n                history=model.fit([sr,sr0,srv,train_meta,train_switch],train_y,batch_size=256,epochs=1,\n                                validation_data=([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],validate_y),\n                                                    verbose=0 )\n                y=model.predict([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],batch_size=64)\n                s=score(y,validate_y)\n\n                myhist['val_loss'].extend(history.history['val_loss'])\n                myhist['val_categorical_accuracy'].extend(history.history['val_categorical_accuracy'])\n                myhist['loss'].extend(history.history['loss'])\n                myhist['categorical_accuracy'].extend(history.history['categorical_accuracy'])\n                myhist['lr'].append(opt.lr)\n                myhist['score'].append(s)\n                val_loss_arr.append(s)\n                weights_arr.append(model.get_weights())\n                if s<smin:\n                    smin=s\n                    weights_min=model.get_weights()\n                tq2.set_postfix(lr=opt.lr,current_score=s,loss=history.history['loss'][0],val_loss=history.history['val_loss'][0])\n                tq.set_postfix(temp_min=smin,total_min=min(smin,smin1))\n            executor.shutdown(wait=True)\n            if smin<smin1:\n                smin1=smin\n                weights_min1=weights_min.copy()\n\n            m=np.argsort(val_loss_arr)\n            new_weights=[]\n            for ii in range(len(weights_arr[0])):\n                new_weights.append(np.stack([weights_arr[m[kk]][ii] for kk in range(5) ],axis=0).mean(axis=0))\n            model.set_weights(new_weights)\n            y=model.predict([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],batch_size=64)\n            s=score(y,validate_y)\n            if s<smin1:\n                smin1=s\n                weights_min1=new_weights.copy()\n        model.set_weights(weights_min1)\n        model.save_weights('model16fr{}{}_{}.weights'.format(meta_size,post_weight_ext,f))\n        y=model.predict([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],batch_size=64)\n\n        score(y,validate_y)\n\n    plt.plot(myhist['loss'])\n    plt.plot(myhist['val_loss'])\n    plt.plot(myhist['score'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40a2f6d1e66a907dae63eb76e8f3219ec2ae05c6"},"cell_type":"code","source":"#Calculate CV\nys=[]\nyv=[]\n\nmodelb = build_model16()\nmodel=model_retop(modelb,meta_size)\n\nfor f in folds:\n    validate_timeseries,validate_timeseries0,validate_void,validate_meta,validate_switch,validate_y=\\\n        calculate_inputs(df_timeseries,df_validation_meta[f],meta_cols,col_weights,real_targets,length=num_samples,aug=None,return_y=True)\n\n    model.load_weights('{}model16fr{}{}_{}.weights'.format(my_DATA_DIR,meta_size,weight_ext,f))\n    \n    y=model.predict([validate_timeseries,validate_timeseries0,validate_void,\n                                              validate_meta,validate_switch],batch_size=64)\n\n    score(y,validate_y)\n    ys.append(y.copy())\n    yv.append(validate_y.copy())\nscore(np.concatenate(ys),np.concatenate(yv))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"baa54c65fa2f64de68b6585979473783324ad6fe"},"cell_type":"code","source":"# if Traning, let's see the CV before training\nif (test_train_switch=='Train') and (pre_weight_ext):\n    ys=[]\n    yv=[]\n\n    modelb = build_model16()\n    model=model_retop(modelb,meta_size)\n\n    for f in folds:\n        validate_timeseries,validate_timeseries0,validate_void,validate_meta,validate_switch,validate_y=\\\n            calculate_inputs(df_timeseries,df_validation_meta[f],meta_cols,col_weights,real_targets,length=num_samples,aug=None,return_y=True)\n\n        model.load_weights('model16fr{}{}_{}.weights'.format(meta_size,pre_weight_ext,f))\n\n        y=model.predict([validate_timeseries,validate_timeseries0,validate_void,\n                                                  validate_meta,validate_switch],batch_size=64)\n\n        score(y,validate_y)\n        ys.append(y.copy())\n        yv.append(validate_y.copy())\n    score(np.concatenate(ys),np.concatenate(yv))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc722600edc626374fbaf3a80161c95e6b5d410e"},"cell_type":"code","source":"# Run on test\nmeta_cols=['hostgal_photoz','dmjd','std_flux','distmod']\nif test_train_switch=='Test':\n    df_test_meta=pd.read_csv(DATA_DIR+'test_set_metadata.csv')\n    if (meta_type=='Nyanp'):\n        nyanp_test=pd.read_csv('{}nyanp_test.csv'.format(my_DATA_DIR)).set_index('object_id')\n        nyanp_test=nyanp_test.rename(columns=int)\n        nyanp_test.head()\n        added_cols=range(15)\n        meta_cols.extend(added_cols)\n    elif (meta_type=='Mamas'):\n        mamas_test=pd.read_csv('{}mamas_test.csv'.format(my_DATA_DIR)).set_index('object_id')\n        mamas_test=mamas_test.rename(columns=int)\n        mamas_test.head()\n        added_cols=range(16)\n        meta_cols.extend(added_cols)\n\n    meta_size=len(meta_cols)\n    col_weights=np.ones(meta_size)\n\n    models=[]\n    res_dfs=[]\n    res_dfps=[]\n    queue_length=max_workers\n    chunksize=500000\n    for f in folds:\n        modelb = build_model16()\n        modelp=model_retop(modelb,meta_size)\n        model=model_tap(modelp)\n        modelp.load_weights('{}model16fr{}{}_{}.weights'.format(my_DATA_DIR,meta_size,weight_ext,f))\n        models.append(model)\n        res_dfs.append([])\n        res_dfps.append([])\n    timeseries_file=DATA_DIR+'test_set.csv'\n    df_test_meta=pd.read_csv(DATA_DIR+'test_set_metadata.csv')\n    df_test_extra = pd.read_csv(my_DATA_DIR+'test_extra.csv')\n    df_test_meta=df_test_meta.merge(df_test_extra,on='object_id',how='left')\n    if (meta_type=='Mamas'):\n        df_test_meta=df_test_meta.merge(mamas_test[[i for i in added_cols]],left_on='object_id',right_index=True,how='left').sort_values('object_id')\n    elif (meta_type=='Nyanp'):\n        df_test_meta=df_test_meta.merge(nyanp_test[[i for i in added_cols]],left_on='object_id',right_index=True,how='left').sort_values('object_id')\n    in_queue =  Queue(queue_length)\n    out_queue =  Queue(queue_length)\n    print('start')\n    Process(target=load_process, args=(df_test_meta,timeseries_file,in_queue,queue_length,chunksize)).start()\n    for q in range(queue_length):\n        Process(target=prepare_process,args=(meta_cols,col_weights,real_targets,in_queue,out_queue,num_samples)).start()\n    print('running')\n    tq=tqdm_notebook(desc='objects:',total=df_test_meta.shape[0])\n    for st in range(queue_length):\n        for object_id,timeseries,timeseries0,void,meta,sw in iter(out_queue.get, 'STOPED'):\n            for f in folds:\n                [y,yp,yf]=models[f].predict([timeseries,timeseries0,void,meta,sw],batch_size=64)\n                res_dfs[f].append(pd.DataFrame(index=object_id, data=y, columns=['class_%d' % k for k in real_targets]))\n                res_dfps[f].append(pd.DataFrame(index=object_id, data=yp, columns=['class_%d' % k for k in real_targets]))\n            tq.update(meta.shape[0])\n    df_res=[]\n    df_resp=[]\n    for f in range(4):\n        df_res.append(pd.concat(res_dfs[f]).sort_index())\n        df_resp.append(pd.concat(res_dfps[f]).sort_index())\n\n\n    for f in range(4):\n        df_res[f].to_csv('pred16r{}{}_{}.csv'.format(meta_size,weight_ext,f))\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04a96dcf29c48d7e3e9ba089a5eeb3e3b2253b3c"},"cell_type":"code","source":"if submit_name:\n    for f in folds:\n        df_res[f]['class_99']=(1-df_res[f].max(axis=1))\n    for f in folds:\n        m=df_res[f]['class_99'].mean()\n        df_res[f]['class_99']=df_res[f]['class_99']/m/8\n        df_res[f]=df_res[f].div(df_res[f].sum(axis=1), axis=0)\n    df_rest=df_res[0]\n    for f in folds[1:]:\n        df_rest=df_rest+df_res[f]\n    df_rest.head()\n    df_rest=df_rest.clip(1e-3,1)\n    df_rest=df_rest.div(df_rest.sum(axis=1), axis=0)\n    df_rest.sample(20)\n    df_rest.to_csv('{}.csv'.format(External_Dir,submit_name),float_format='%.4f')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b9ba35ad55ebae6b4ccd7b38d07d784050c368a"},"cell_type":"markdown","source":"## Preparing the data\n\n### The code below is only for reference!","attachments":{}},{"metadata":{"trusted":true,"_uuid":"93660c593b5097a6b1328d61f5b004c0021fbab0"},"cell_type":"code","source":"def calc_extra(df_timeseries):\n    gp=df_timeseries.groupby('object_id')\n    dfe=(gp['mjd'].max()-gp['mjd'].min()).rename('dmjd').reset_index()\n    dfe['dmjd']=dfe['dmjd']/1000\n    dfe['std_flux']=gp.flux.std().reset_index().flux/1000\n    return dfe\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ddf9df8f9c81cff304b8e087223d8fce5af04a7"},"cell_type":"markdown","source":"## 4fold split + Calaculate extra meta"},{"metadata":{"trusted":true,"_uuid":"681085d8b134daf70c36e3a62f2b73906f96d2f1"},"cell_type":"code","source":"if False:     # We don't want to run the code below\n    foldn=4\n    df_group=[]\n    df_group_meta=[]\n    df_validate_meta=[]\n    df_train_meta=[]\n    df_train_l=[]\n    for i in range(foldn):\n        df_group.append([])\n        df_group_meta.append([])\n        df_validate_meta.append(None)\n        df_train_meta.append(None)\n        df_train_l.append([])\n    df_training=pd.read_csv(DATA_DIR+'training_set.csv')\n    df_training_meta=pd.read_csv(DATA_DIR+'training_set_metadata.csv')\n    extra = calc_extra(df_training)\n    df_training_metas=df_training_meta.merge(extra,on='object_id',how='left').sample(frac=1).copy()\n\n    for target in real_targets:\n        df = df_training_metas[df_training_metas.target==target].copy()\n        for i in range(foldn):\n            df_group[i].append(df.iloc[int(i*df.shape[0]/foldn):int(((i+1)*df.shape[0])/foldn)])\n\n    for i in range(foldn):\n        df_group_meta[i]=pd.concat(df_group[i])\n\n\n    for i in range(foldn):\n        for j in range(foldn):\n            if (i==j):\n                df_validate_meta[i]=df_group_meta[j].copy()\n            else:\n                df_train_l[i].append(df_group_meta[j].copy())\n\n        df_train_meta[i]=pd.concat(df_train_l[i])\n        df_train_meta[i].shape\n        df_validate_meta[i].shape\n        df_train_meta[i].to_csv('{}training_meta{}.csv'.format(DATA_DIR,i),index=False)\n        df_validate_meta[i].to_csv('{}validation_meta{}.csv'.format(DATA_DIR,i),index=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6210de1728bca49c92a73df0958ea8b4b50a20e"},"cell_type":"markdown","source":"# Calculate extra features for test set"},{"metadata":{"trusted":true,"_uuid":"09bdbfdc338c289449dc2f79d813951bd7b20959"},"cell_type":"code","source":"if False:     # We don't want to run the code below\n    chunksize=5000000\n    df_test_meta=pd.read_csv(DATA_DIR+'test_set_metadata.csv')\n    extras=[]\n    df_test_extras=[]\n    for chunk in tqdm_notebook(pd.read_csv(DATA_DIR+'test_set.csv', chunksize=chunksize)):\n        first_id=chunk.head(1)['object_id'].values[0]\n        last_id=chunk.tail(1)['object_id'].values[0]\n        select=chunk['object_id'].isin([first_id,last_id])\n        extras.append(chunk[select].copy())\n        mid_chunk=chunk[~select].sort_values(['object_id','mjd']).copy()\n        if mid_chunk.shape[0]>0:\n            df_test_extras.append(calc_extra(mid_chunk))\n    mid_chunk=pd.concat(extras)\n    df_test_extras.append(calc_extra(mid_chunk))\n    df_test_extra=pd.concat(df_test_extras).sort_values('object_id')\n    df_test_extra.sample(10)\n    df_test_extra.shape\n    df_test_extra.to_csv(DATA_DIR+'test_extra.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}