{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":10384,"databundleVersionId":120379,"sourceType":"competition"},{"sourceId":212561,"sourceType":"datasetVersion","datasetId":81769}],"dockerImageVersionId":18199,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**UPDATE:**\n\nI made a couple of changes since submitting my first set of predictions, which include scaling the flux observations and doing per-channel convolutions. Together, these two tweaks really improved my model, and ** halfed the test loss from 24 to 12**. I've shared the updates in Bold below (Also updated the TOC headers)","metadata":{"_uuid":"4aec2f48f28631a15d7fcd976b533a4b42217fe2"}},{"cell_type":"markdown","source":"Hey guys!\nThis is my first real kernel for a Kaggle competition, so feedback is greatly appreciated!","metadata":{"_uuid":"fa00d65a0e757b6e8531e5781ee40c2b1a3c3853"}},{"cell_type":"markdown","source":"# Table of Contents\n\n- [Introduction](#Introduction)\n- [Approach](#Approach)\n- [Implementation](#Implementation)\n- [DMDT Images](#DMDT-Images)\n    - [Generating the Images **UPDATED**](#Generating-the-Images)\n    - [Loading the Images](#Loading-the-Images)\n- [Training and Test set](#Training-and-Test-set)\n- [The CNN Model Architecture **UPDATED**](#The-CNN-Model-Architecture)\n- [Custom Loss Function](#Custom-Loss-Function)\n- [Training and Evaluation](#Training-and-Evaluation)","metadata":{"_uuid":"dbfddbc5096b829222f7b548fa75b1aec455eb6f"}},{"cell_type":"markdown","source":"# Introduction\n\nA little bit about myself - by day, I'm just a regular software engineer, trying to eek my way into data science. But by night, I'm a huge astrophysics/cosmology buff. I try to keep up with what's happening by reading news, watching videos, and even going through some lectures as well as reading a few papers. So, this competition was literally the icing on the cake for me! It was fascinating to go through the two example kernels - to learn about the [motivation and the astronomical background](https://www.kaggle.com/michaelapers/the-plasticc-astronomy-starter-kit) (super fun stuff!) as well to look at a [sample approach taken by astronomers](https://www.kaggle.com/michaelapers/the-plasticc-astronomy-classification-demo). \n\nIt was equally (if not more) fun to look at the different kernels posted by everyone here. I'm yet to go through a large chunk of them, so its possible my approach has already been shared by someone, but in general I found some of these particularly interesting/useful - \n\n- [The Astronomical (complete) EDA - PLAsTiCC dataset](https://www.kaggle.com/danilodiogo/the-astronomical-complete-eda-plasticc-dataset) - this does a better job at EDA than I could've done.\n- [All Classes Light Curve Characteristics](https://www.kaggle.com/mithrillion/all-classes-light-curve-characteristics)\n- [Using CNNs in time series analysis](https://www.kaggle.com/hrmello/using-cnns-in-time-series-analysis) - this one in particular since my approach was inspired by it.\n\nI did look at a few basic options such Random Forests/XGBoost, Logistic Regression, etc. But the CNN-based approach seemed to interest me, given that I have some prior experience working with CNNs. Plus, the fact that I sort of started to get into it right away. A short google search yielded a paper that uses CNNs to classify light curves, which I describe below. \n","metadata":{"_uuid":"1d3c8aa637c3872a26b87273b583cde465fd5461"}},{"cell_type":"markdown","source":"# Approach\n\nBased on [Mahabal, Ashish, et al. \"Deep-learnt classification of light curves.\" ](https://arxiv.org/pdf/1709.06257.pdf)\n\n\nI came across this paper and I liked it because it directly tackles the problem at hand - classifying light curves based on time series data is a hard problem. Firstly, because there is often an irregular gap in the observations due to a variety of reasons. Secondly, and this is especially true with LSST, there is always the scope of coming across objects that you've never seen before (hence the training set isn't a good representation of the test set). This makes computing features challenging. Generic statistical features don't yield high accuracy. Sometimes, domain-level features are employed (the paper gives an example feature - 'fading profile of a single peaked fast transient'. These features are useful only for certain classes of objects and don't generalize well.\n\nThe authors present a different approach - to use neural networks to classify objects, since neural networks are good at extracting features from data. In particular, they use Convolutional Neural Networks, which have proven their metal in image classification. You can read more about CNNs [here](https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/). \n\nIn short, the idea is to encode the flux values into a matrix (or a tensor), and use that as input to a CNN (which I implement using Keras). \n\nThe encoding approach is interesting, the idea is to capture changes in flux values at different magnitude and time scales. For each pair of points on the light curve, compute the difference in magnitude $dm$ and time $dt$ and then put them into bins. Each bin contains counts of all pairs of points that fall within it.  The resulting matrix/image is referred in the paper as a _dmdt_ image. There are $p = {n \\choose 2}$ such pairs for n points on the light curve. The sample bins used in the paper (as well my code) are \n$$dm = \\pm [0, 0.1, 0.2, 0.3, 0.5, 1, 1.5, 2, 2.5, 3, 5, 8]$$ \nand \n$$dt = [\\frac{1}{145}, \\frac{2}{145}, \\frac{3}{145}, \\frac{4}{145}, \\frac{1}{25}, \\frac{2}{25}, \\frac{3}{25}, 1.5, 2.5, 3.5, 4.5, 5.5, 7, 10, 20, 30, 60, 90, 120, 240, 600, 960, 2000, 4000]$$\n\nThe bins are in approximately in a semi-logarithmic fashion, so that smaller changes in magnitude and time are spread over many bins, whereas the corresponding larger changes are clubbed together. The flux counts are normalized and stretched to fit between 0 and 255 - $${norm}_{bin} = \\lfloor{\\frac{255 * {count}_{bin}}{p} + 0.99999}\\rfloor$$\n\nI also tried to create a kernel just to visualize these images and do some basic EDA stuff, but I haven't got around to finishing that, you can find the kernel here - [Plasticc DMDT EDA](https://www.kaggle.com/pankajb64/plasticc-dmdt-eda)\n\nThe authors talk about a single light curve, and hence creating a single 2D matrix, but in our case we have 6 different passbands, so 6 such matrices per object. The simplest approach, which is also the one that I use here, is to simply stack these images to create a 6-channel image and feed that to the CNN. Its not ideal, since different bands are measured at different time instants, and the total number of measurements vary across bands. On the other hand, treating each band as an individual input to the CNN isn't a great idea either (though I haven't tried it so can't be sure) since the different bands combine to uniquely identify the objects, and each band may potentially contain different information. Its not clear to me what the best way to deal with this, so feedback is greatly appreciated.\n\nI also implemented the weighted multi-class log loss used in this competition as a custom loss function, since I wasn't sure categorical cross-entropy takes class imbalance into account. In my case I set the weight of each class to be 1, since I obviously don't know the actual weights used by the compeition judges.\n\nSo without further ado, let's just get to the code!","metadata":{"_uuid":"8b6332fc2c0b9773be791d8396c00e61768699fc"}},{"cell_type":"markdown","source":"# Implementation","metadata":{"_uuid":"6befde97937c38ef7410363490a89dd0c788745b"}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nimport pickle\nimport multiprocessing\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport keras.backend as K #to define custom loss function\nimport tensorflow as tf #We'll use tensorflow backend here\nimport dask.dataframe as dd\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tnrange, tqdm_notebook\nfrom collections import OrderedDict\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPooling2D, Dropout, Dense, Flatten, Input, AveragePooling1D, Reshape, DepthwiseConv2D, SeparableConv2D\nfrom keras.optimizers import Adam, SGD\nfrom keras.callbacks import ReduceLROnPlateau,ModelCheckpoint,EarlyStopping\nfrom sklearn.model_selection import StratifiedShuffleSplit\nfrom datetime import datetime\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import MinMaxScaler\n\nprint(os.listdir(\"../input/\"))","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/PLAsTiCC-2018/training_set.csv')","metadata":{"_uuid":"daecdc4e92a586d95cd3fad7a738ac52b7cd8037","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(5)","metadata":{"_uuid":"6859aad5643c5fb69c054604c1cfd82b6b303e46","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DMDT Images","metadata":{"_uuid":"0fbf0439de22ec0595e88f252f320be267a5ed31"}},{"cell_type":"markdown","source":"### Generating the Images\n\nAs mentioned earlier, in this kernel, I'm not doing EDA, but just getting straight to how to generate the dmdt images, and run the model using them.\n\nThe functions below generate dmdt images (or \"dmdtize\" the input as I say). I decided to get around the large number of rows by saving the image for each object in a different file, and combining them while loading. That way, I didn't have to re-process images if I happened to stop the run mid-way.\n\n**UPDATE**:\n\nDuring my post-submission analysis, I got a chance to re-evaluate a larger set of dmdt images, and I realized a bulk of them looked very similar even though they were from different classes. So I decided to scale the observations for each individual object to be in the range defined by the flux magnitude bins, i.e. between -8 and 8.","metadata":{"_uuid":"6f7846292dffc73b95fb2a7b94aff16a51b35f71"}},{"cell_type":"code","source":"def dmdtize_single_band(df, dm_bins, dt_bins, col):\n    n_points = df.shape[0]\n    dmdt_img = np.zeros((len(dm_bins), len(dt_bins)), dtype='int')\n    for i in range(n_points):\n        for j in range(i+1, n_points):\n            dmi = df.iloc[i][col]\n            dmj = df.iloc[j][col]\n            dti = df.iloc[i]['mjd']\n            dtj = df.iloc[j]['mjd']\n            \n            dm = dmj - dmi if dtj > dti else dmi - dmj\n            dt = abs(dtj - dti)\n            \n            dm_idx = min(np.searchsorted(dm_bins, dm), len(dm_bins)-1)\n            dt_idx = min(np.searchsorted(dt_bins, dt), len(dt_bins)-1)\n            \n            dmdt_img[dm_idx, dt_idx] += 1\n    return dmdt_img","metadata":{"_uuid":"3abc3464dd70912c94866219a5c568afa7084da2","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dmdtize_single_object(args):\n    (df, object_id, base_dir) = args\n    key = '{}/{}_dmdt.pkl'.format(base_dir, object_id)\n    if os.path.isfile(key):\n        return\n    num_bands = 6\n    dm_bins = [-8, -5, -3, -2.5, -2, -1.5, -1, -0.5, -0.3, -0.2, -0.1, 0, 0.1, 0.2, 0.3, 0.5, 1, 1.5, 2, 2.5, 3, 5, 8]\n    dt_bins = [1/145, 2/145, 3/145, 4/145, 1/25, 2/25, 3/25, 1.5, 2.5, 3.5, 4.5, 5.5, 7, 10, 20, 30, 60, 90, 120, 240, 600, 960, 2000, 4000]\n    dmdt_img = np.zeros((len(dm_bins), len(dt_bins), num_bands), dtype='int')\n    \n    mms = MinMaxScaler(feature_range=(-8, 8))\n    df['local_tr_flux'] = mms.fit_transform(df['flux'].values.reshape(-1,1))\n    \n    max_points = 0\n    for band_idx in range(num_bands):\n        df_band = df.loc[df['passband'] == band_idx]\n        dmdt_img[:, :, band_idx] = dmdtize_single_band(df_band, dm_bins, dt_bins, 'local_tr_flux')\n        if band_idx == 0 or df_band.shape[0] > max_points:\n            max_points = df_band.shape[0] #store max points to scale the image later\n    \n    max_pairs = (max_points*(max_points-1))//2\n    dmdt_img = np.floor(255*dmdt_img/max_pairs + 0.99999).astype('int')\n    with open(key, 'wb') as f:\n        pickle.dump(dmdt_img, f)        ","metadata":{"_uuid":"6f084d72fd167ed204c267d55c3eb198e5e95537","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dmdtize(df, base_dir='train'):\n    objects = df['object_id'].drop_duplicates().values\n    nobjects = len(objects)\n    dmdt_img_dict = {}\n    pool = multiprocessing.Pool()\n    df_args = []\n    for obj in objects:\n        df_obj = df.loc[df['object_id'] == obj]\n        df_args.append((df_obj, obj, base_dir))\n    pool.map(dmdtize_single_object, df_args)\n    pool.terminate()","metadata":{"_uuid":"6d96b536eb7bee7035e1424bb2811e0a94e7fc13","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To generate/save the dmdt images, convert thes cell below to code and execute. It takes a while (it took me about 20 minutes on a 16-core machine, using all cores), so I've attached the pre-processed images as a dataset, so they can be loaded up easily.","metadata":{"_uuid":"b5cfd70db611deda57a5becc5cabc97e2d1f440d"}},{"cell_type":"markdown","source":"```\ndmdtize(df)\n```","metadata":{"_uuid":"4e8f9597f7f7933928e327a0a738025d9b055a1f"}},{"cell_type":"markdown","source":"### Loading the Images\n\nThe cells below load the dmdt images from the dataset.","metadata":{"_uuid":"0b035a6fa7f5b91ce66aedfdb2c54a118840ed54"}},{"cell_type":"code","source":"objects = df['object_id'].drop_duplicates().values","metadata":{"_uuid":"634bc3f85159742502089874717862d2eaeef6f6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dmdt_images(objects, base_dir='train'):\n    dmdt_img_dict = OrderedDict()\n    for obj in objects:\n        key = '{}/{}_dmdt.pkl'.format(base_dir, obj)\n        if os.path.isfile(key):\n            with(open(key, 'rb')) as f:\n                dmdt_img_dict[obj] = pickle.load(f)\n    return dmdt_img_dict","metadata":{"_uuid":"4966d5dad48ee6336c054543a5c948a047dac9c4","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dmdt_img_dict = load_dmdt_images(objects, '../input/plasticc_dmdt_images/train/data1/plasticc/input/train')","metadata":{"_uuid":"d33903aa9215f1828cf856f803d18bccb189a548","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training and Test set\n\nThe images are already scaled and pre-processed, so we can just feed them to the CNN. Lets create the input and output vectors for training and test.","metadata":{"_uuid":"b5cf88b3e2cd7348157846d25b62dc1671dbd111"}},{"cell_type":"code","source":"X = np.array(list(dmdt_img_dict.values()), dtype='int')\n\ndf_meta = pd.read_csv('../input/PLAsTiCC-2018/training_set_metadata.csv')\nlabels = pd.get_dummies(df_meta.loc[df_meta['object_id'].isin(dmdt_img_dict.keys()) , 'target'])\n\ny = labels.values","metadata":{"_uuid":"df5cce2d431a1220ab8dc12ec232600bef453a43","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_meta","metadata":{"_uuid":"b0befe3bb0d724329a6a0c894b96a44ce15e999d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels","metadata":{"_uuid":"3eab61887d48788fe3e07325394e6e4c3359ee63","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#TODO split X and y into train/test set. (Maybe a val set ?)\nsplitter = StratifiedShuffleSplit(n_splits=1, test_size=0.2)\nsplits = list(splitter.split(X, y))[0]\ntrain_ind, test_ind = splits","metadata":{"_uuid":"0165c5ce25c7dd4db8ca2aab74a55838dc324d9e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X[train_ind]\nX_test  = X[test_ind]\n\ny_train = y[train_ind]\ny_test  = y[test_ind]","metadata":{"_uuid":"bc30adb31c32d705b3b8da640f3c35c4a3bc4025","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(y_train.shape, y_test.shape)","metadata":{"_uuid":"4f870ea3a2f7c8e9f0430b367d828a768d486456","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"6000 images isn't nearly enough for training a CNN (unless you're doing transfer learning), but we'll have to make do with what we have. Again, feedback appreciated.","metadata":{"_uuid":"1e66ab09477a76cbfb0abf63945942453038fb0e"}},{"cell_type":"markdown","source":"## The CNN Model Architecture\n\nThe CNN Architecture is outined below. Now, I'm definitely not an expert in building models, but I tried a few different variations and found this to be most well-suited.\n\nA couple of things things I noted -\n- The dimensions of the input image are unusually small even for a CNN (a typical image size is 256x256 per channel whereas we have a 23x24) and my gut feeling was that the first Convolutional layer should not be max pooled. This turned out to be right - I got better training layer in abscence of the max pooling layer.\n- I didn't try a whole bunch of activations, but using an Exponential Linear Unit (ELU) worked better than ordinary Rectilinear Unit (RELU). This is probably because the internal layers have negative values.\n\n**UPDATE:**\nAfter visualizing a few sample images for each class, I noticed that the lack of synchronicity between the passband observations meant that especially for events like supernovae, only some of the bands would capture the event. The remaining bands are essentially noise for the purpose of classification.  In this case a regular convolution kernel involving multiplication across channels might not be able to capture the features. My hunch was if we try to instead convolve individual features separately, we'd stand a better chance to capture those features.\nLuckily, I found Keras had a layer called *Depthwise Convolution* which does exactly what we need. At some point we need to combine those features together, and that's where a *Separable Convolution* comes in, which is essentially Depthwise convolution followed by a regular pointwise convolution. See [Keras Documentation](https://keras.io/layers/convolutional/) for more information about the various Convolutional Layers.\n","metadata":{"_uuid":"4c6ef68244900ae0840e7f7c57f740398a8eb484"}},{"cell_type":"code","source":"def build_model(n_dm_bins=23, n_dt_bins=24, n_passbands=6, n_classes=14):\n    model = Sequential()\n    model.add(DepthwiseConv2D(kernel_size=2, strides=1, depth_multiplier=8,\n                     padding=\"valid\", activation=\"elu\",\n                     input_shape=(n_dm_bins, n_dt_bins, n_passbands)))\n    model.add(Dropout(0.1))\n    model.add(SeparableConv2D(48, kernel_size=2, strides=1, depth_multiplier=2,\n                    padding=\"valid\", activation=\"elu\"))\n    model.add(MaxPooling2D(pool_size=2))\n    model.add(Dropout(0.1))\n    model.add(Flatten())\n    model.add(Dense(32, activation=\"elu\"))\n    model.add(Dropout(0.25))\n    model.add(Dense(n_classes, activation=\"softmax\"))\n    print(model.summary())\n    return model","metadata":{"_uuid":"51ccf188ef97a138de85a67e5166ff9e322e74e7","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Custom Loss Function\n\nThe code below implements the weighted multi class log loss functions. This is similar to the one defined in the Evaluation section of the project description with one caveat - I have used the proportion of objects in a class instead of the actual counts, since I found it gave more readable loss values.\n\nA couple of Keras-specific things - \n- A custom loss function defined in Keras this way must only methods from the keras.backend interface to process its arguments (and not use numpy directly) to ensure that it can be smoothly translated into the appropriate backend (Tensorflow or theano) when compiling the model. \n- I set Keras to use float32 as the default variable dtype, since float 64 wasn't working for some reason.","metadata":{"_uuid":"27dc0d25123685a653ac89dc4d18dff4e7e229e8"}},{"cell_type":"code","source":"n_classes=14\n#assumes weights to be all ones as actual weights are hidden\n#UPDATE - settings weights for classes 15 (idx=1) and 64(idx=7) to 2 based on LB probing post \n#https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153\nweights = np.ones(n_classes, dtype='float32') \nweights[1], weights[7] = 2, 2\nepsilon = 1e-7\n#number of objects per class\nclass_counts = df_meta.groupby('target')['object_id'].count().values \n#proportion of objects per class\nclass_proportions = class_counts/np.max(class_counts)\n#set backend to float 32\nK.set_floatx('float32')","metadata":{"_uuid":"5b3903329e56325b1d69b51a9029b1b6129c15ca","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#weighted multi-class log loss\ndef weighted_mc_log_loss(y_true, y_pred):\n    y_pred_clipped = K.clip(y_pred, epsilon, 1-epsilon)\n    #true labels weighted by weights and percent elements per class\n    y_true_weighted = (y_true * weights)/class_proportions\n    #multiply tensors element-wise and then sum\n    loss_num = (y_true_weighted * K.log(y_pred_clipped))\n    loss = -1*K.sum(loss_num)/K.sum(weights)\n    \n    return loss","metadata":{"_uuid":"80c2fa279b890449a6cb8a01f7dbe913e5d48672","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is just small test of the loss function to see if its compiles and runs fine.","metadata":{"_uuid":"e4a3e290c48f058097f66c332bd0d0535dceb1f1"}},{"cell_type":"code","source":"y_true = K.variable(np.eye(14, dtype='float32'))\ny_pred = K.variable(np.eye(14, dtype='float32'))\nres = weighted_mc_log_loss(y_true, y_pred)\nK.eval(res)","metadata":{"_uuid":"63f0edc11a09dcbd2a4fa270470e18c34c3370a5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training and Evaluation\n\nLets build the model now.","metadata":{"_uuid":"ca77c4b740eb8c605f24a13044aa4e9d8f5dcbfc"}},{"cell_type":"code","source":"model = build_model()","metadata":{"_uuid":"6a55b64691516beb10d09ba06a6609013d88c483","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I also print the model summary to get an idea of the number of parameters to be trained. 50000 is a lot of parameters for just 6k input rows, but its nothing compared to the CNNs used for image classification. I found higher number of parameters gave worse results on test.\n\nLets compile the model now, I used Adam as the optimizer based on the paper, but I increased the learning rate to 0.002 since it yielded faster convergence.","metadata":{"_uuid":"0181f91a3fcebea3139c2948f117d6901e2f41c2"}},{"cell_type":"code","source":"model.compile(loss=weighted_mc_log_loss, optimizer=Adam(lr=0.002), metrics=['accuracy'])","metadata":{"_uuid":"0dcf3b58a02b8ccf8edaf4a84fea202fd8b2368f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Training for 20 epochs (I didn't do a validation set since the training set is already too small). I found that higher epochs led to overfitting.","metadata":{"_uuid":"365a9f4d9033a96514161f9930aa341e3b7726e9"}},{"cell_type":"code","source":"checkPoint = ModelCheckpoint(\"./keras.model\",monitor='val_loss',mode = 'min', save_best_only=True, verbose=1)","metadata":{"_uuid":"62bdae5e8f6370fa299ec6f6fb72017e8dd7b8e1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ReduceLRWithEarlyStopping(ReduceLROnPlateau):\n    def __init__(self, *args, **kwargs):\n        super(ReduceLRWithEarlyStopping, self).__init__(*args, **kwargs)\n        self.stopped_epoch = 0\n\n    def on_epoch_end(self, epoch, logs=None):\n        super(ReduceLRWithEarlyStopping, self).on_epoch_end(epoch, logs)\n        old_lr = float(K.get_value(self.model.optimizer.lr))\n        if self.wait >= self.patience and old_lr <= self.min_lr:\n            # Stop training early\n            self.stopped_epoch = epoch\n            self.model.stop_training = True\n\n    def on_train_end(self, logs=None):\n        super(ReduceLRWithEarlyStopping, self).on_epoch_end(logs)\n        if self.stopped_epoch > 0 and self.verbose > 0:\n            print('Epoch %05d: early stopping' % (self.stopped_epoch + 1))","metadata":{"_uuid":"841f7c95d9c8fc7df7afd58a9a9f12712a85b9d4","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"reduce_lr = ReduceLRWithEarlyStopping(monitor='val_loss', factor=0.2,\n                              patience=5, min_lr=0.00001)","metadata":{"_uuid":"389ea729853a0e31556bed93527bcf8867cd039e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_train, batch_size=32, epochs=100, validation_data=[X_test, y_test],shuffle=True,verbose=1,callbacks=[checkPoint, reduce_lr])","metadata":{"_uuid":"6c6e22a858acba60135b6ef64a34bd74c6b191fd","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_loss_acc(history):\n    plt.plot(history['loss'][1:])\n    plt.plot(history['val_loss'][1:])\n    plt.title('model loss')\n    plt.ylabel('val_loss')\n    plt.xlabel('epoch')\n    plt.legend(['train','Validation'], loc='upper left')\n    plt.show()\n    \n    plt.plot(history['acc'][1:])\n    plt.plot(history['val_acc'][1:])\n    plt.title('model Accuracy')\n    plt.ylabel('val_acc')\n    plt.xlabel('epoch')\n    plt.legend(['train','Validation'], loc='upper left')\n    plt.show()","metadata":{"_uuid":"90b175133bec0d7ebf8fb0812f7029f18e7d310c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_loss_acc(history.history)","metadata":{"_uuid":"ca4ff4335e0775f550d5312b4d3a97eae3e7b7ac","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets evaluate it on the test set now.","metadata":{"_uuid":"2e32e528b8f5c94fceaeb38d618fb4b88248c1bc"}},{"cell_type":"code","source":"loss_acc = model.evaluate(X_test, y_test, batch_size=32)\nprint(loss_acc)","metadata":{"_uuid":"d1f5f6d4f41efb8475a58d9882a91053422c095a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"~67% accuracy on the training set, and ~55% on the test is good but not great, so really any feedback here is truly appreciated.\n\nLets try and look at some of the predictions.","metadata":{"_uuid":"0175af53770517c2ceb675b580536e509376af9d"}},{"cell_type":"code","source":"y_pred_test = model.predict(X_test)","metadata":{"_uuid":"077aff3808a1d8a10f329e0f98d522b2d5d4cd78","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classes = np.sort(df_meta['target'].drop_duplicates())","metadata":{"_uuid":"d5d124b97c372f81dabd19c922ee7abbc0f064e6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_meta_test = df_meta.iloc[test_ind]\ndf_meta_test['pred_label'] = classes[np.argmax(y_pred_test, axis=1)]","metadata":{"_uuid":"fe174f31e195e814edea67a02ea2524f885e3a1a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_meta_test.loc[df_meta_test.target == 15]","metadata":{"_uuid":"48d1132f0038c6636a5c7e643fd4fa3e8786b250","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_meta_test.loc[(df_meta_test.target == df_meta_test.pred_label) & (df_meta_test.target != 16) & (df_meta_test.target != 92) & (df_meta_test.target != 88)]","metadata":{"_uuid":"6ef8d5c83eb22bbd612b9414631130707b4e4a2f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Just eyeballing the predictions, it doesn't seem like there's any particular class for which its doing good/bad, its just doing decent on average, which is perhaps an effect of the loss function. More investigation is needed here.\n\nFor now, we'll save our model so we can reuse it later.","metadata":{"_uuid":"0d86dfabe82bcfa6ab7d8fba0d6f060cda44517f"}},{"cell_type":"code","source":"time_stamp = datetime.now().strftime('%Y_%m_%d_%H_%M_%S')\nsave_file = 'model_{}.h5'.format(time_stamp)\nmodel.save(save_file)","metadata":{"_uuid":"04b7d7d0da3e2be87fd4fc9c43a06f3e1da9c2a9","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's all for now! Feedback appreciated!","metadata":{"_uuid":"b314e9a73ef8702cea8c52a3c168e7237fd9fe9e","trusted":true}},{"cell_type":"code","source":"","metadata":{"_uuid":"e2a1cea478046e54d9bcb0f5cd8f5f415d105827","trusted":true},"execution_count":null,"outputs":[]}]}