{"cells":[{"metadata":{"_uuid":"4ec806e6a8343e92315251edd716347d441fe341"},"cell_type":"markdown","source":"Adversarial Validation CNN of PTP for VSB Power, Paul Nussbaum, Feb 2019"},{"metadata":{"_uuid":"ec32cca05ac73eefdc6d650e2f278cf2a3007562"},"cell_type":"markdown","source":"\nAn Adversarial Validation approach may be useful when the TEST set may be very different from the TRAINING set. Simply choosing a subset of the training set to perform validation may not yield the best results. \n\nTraditional methods for creation of a validation set include stratified k-fold cross validation, stratified percentage split, and a simple percentage split as included in the \"fit\" function \"validation_split\" parameter.\n\nThis Python Jupyter Notebook demonstrates a different kind of validation split, popular in several Kaggle competitions, called the Adversarial Validation approach. In this approach, we create a machine learning algorithm to distinguish between the training set and the testing set. We then use that algorithm to find those training set examples that \"most resemble\" testing set examples, an we use those as our validation set. We then train our regognition algorithm as we normally would. \n\nIn this example, I will use the same Peak to Peak (PTP) feature from three phases of non-overlapping windows of data, and the same Convolutional Neural Network (CNN) to both create the Adversarial Validation Set as well as to recognize when a fault has taken place in the VSB Power competition.\n\nIt works as follows:\n* first load all the training and testing data into one huge training set X, and replace y with 0 or 1 for train or test, respectively\n* use this huge training set to learn the difference between training and testing examples\n* use the learnings to classify the training set\n* sort the training by the \"similar to the test set\" score of y\n* finally, keep the first n% most test like training samples as the validation group for traditional training.\n* thanks go to http://fastml.com/adversarial-validation-part-one/ - Posted by Zygmunt Z. 2016-05-23\n* thanks go to adversarial_validation_and_lb_shakeup posted by Olivier https://www.kaggle.com/ogrellier/adversarial-validation-and-lb-shakeup\n* thanks go to http://manishbarnwal.com/blog/2017/02/15/introduction_to_adversarial_validation/ by Manish Barnwal"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# LOADING UP PYTHON COMPONENTS\nimport numpy as np # linear algebra\nfrom numpy import sort\nfrom scipy import stats, histogram\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport pyarrow.parquet as pq\nimport os\nprint(os.listdir(\"../input\"))\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom keras.layers import * # Keras is the most friendly Neural Network library, this Kernel use a lot of layers classes\nfrom keras.models import Model, Sequential, load_model\nfrom tqdm import tqdm # Processing time measurement\nfrom keras import backend as K # The backend give us access to tensorflow operations and allow us to create the Attention class\nfrom keras import optimizers # Allow us to access the Adam class to modify some parameters\nfrom keras.callbacks import * # This object helps the model to train in a smarter way, avoiding overfitting\n\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold # Used to use Kfold to train our model\nfrom sklearn.model_selection import train_test_split \nfrom sklearn import preprocessing\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17dfb321bbb0e77c32590921c7938fa7f87f7a4f"},"cell_type":"code","source":"# This function copied from https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694\n#    courtesy Bruno Aquino - 5-fold LSTM Attention (fully commented)\n# It is the official metric used in this competition\n# below is the declaration of a function used inside the keras model, calculation with K (keras backend / thensorflow)\ndef matthews_correlation(y_true, y_pred):\n    '''Calculates the Matthews correlation coefficient measure for quality\n    of binary classification problems.\n    '''\n    y_pred_pos = K.round(K.clip(y_pred, 0, 1))\n    y_pred_neg = 1 - y_pred_pos\n\n    y_pos = K.round(K.clip(y_true, 0, 1))\n    y_neg = 1 - y_pos\n\n    tp = K.sum(y_pos * y_pred_pos)\n    tn = K.sum(y_neg * y_pred_neg)\n\n    fp = K.sum(y_neg * y_pred_pos)\n    fn = K.sum(y_pos * y_pred_neg)\n\n    numerator = (tp * tn - fp * fn)\n    denominator = K.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))\n\n    return numerator / (denominator + K.epsilon())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c66b44ecdcd9f4c28e6cc907c67abb9d6d86c6b9"},"cell_type":"code","source":"# Some global parameters\nfilename = '../input/train.parquet'\n# number of non-overlaping windows...\nwindows = 64 # 312.5 us at 50Hz or 12,500 samples\nwin_size = int(800000 / windows)\n# number of features per time slice\nwin_feat = 1\nphases = 3\n# total number of features extracted from the signal\nnumber_of_params = int(phases * win_feat)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9f173e6790485f28fb034f212e1dd5e7966209e"},"cell_type":"code","source":"# FEATURE EXTRACTION FUNCTION\n# This function chops up three phases of data at a time into equal sized non-overlapping windows.\n# For each phase and each window, it extracts the number of features (in this case, one feature called PTP, which is max - min)\ndef extract_features(begin_col, loc_num_to_use, filename) :  \n    # Use \"Batching\" to avoid RAM limitations of Kaggle\n    # loc_num_to_use must be a multiple of 3, and so does batch\n    batch = 300 \n    num_batches = int((loc_num_to_use) / batch)\n    remainder = int((loc_num_to_use) % batch)\n    ### Create a pandas data frame\n    loc_X = np.zeros((int(loc_num_to_use), windows, win_feat, phases))\n    if num_batches > 0:\n        for ix in range (num_batches) :\n            # load a batch of signals\n            x1 = pq.read_pandas(filename, columns=[str(ix * batch + j + begin_col) for j in range(batch)]).to_pandas().values.transpose()\n            # now look at all three phases at once\n            for i in range (0, batch, phases) :\n                idx = ix * batch + i\n                for k in range (windows) :\n                    # start and end of window in signal data\n                    win_start = k * win_size\n                    win_end = win_start + win_size - 1\n\n                    # THIS PHASE is Phase 1 or Phase 2 for other phases\n                    min0 =  float(x1[i,win_start:win_end].min())\n                    max0 =  float(x1[i,win_start:win_end].max())\n                    ptp0 =  max0 - min0\n                    \n                    loc_X[idx    , k, 0, 0] = ptp0 \n                    loc_X[idx + 1, k, 0, 1] = ptp0\n                    loc_X[idx + 2, k, 0, 2] = ptp0\n                    \n                    # PHASE 1 is this phase or Phase 2 for other phases\n                    min0 =  float(x1[i + 1,win_start:win_end].min())\n                    max0 =  float(x1[i + 1,win_start:win_end].max())\n                    ptp0 =  max0 - min0\n\n                    loc_X[idx    , k, 0, 2] = ptp0\n                    loc_X[idx + 1, k, 0, 0] = ptp0\n                    loc_X[idx + 2, k, 0, 1] = ptp0\n                    \n                    # PHASE 2 is this phase or Phase 1 for other phases\n                    min0 =  float(x1[i + 2,win_start:win_end].min())\n                    max0 =  float(x1[i + 2,win_start:win_end].max())\n                    ptp0 =  max0 - min0\n                    \n                    loc_X[idx    , k, 0, 1] = ptp0  \n                    loc_X[idx + 1, k, 0, 2] = ptp0 \n                    loc_X[idx + 2, k, 0, 0] = ptp0 \n    # Here we process what is left over from all of the Batches\n    if remainder > 0 :\n        ix = num_batches\n        if 1 : # dummy indent so I don't have to keep changing indent on cut and paste\n            # load a batch of signals\n            x1 = pq.read_pandas(filename, columns=[str(ix * batch + j + begin_col) for j in range(remainder)]).to_pandas().values.transpose()\n            for i in range (0,remainder, phases) :\n                idx = ix * batch + i\n                for k in range (windows) :\n                    # start and end of window in signal data\n                    win_start = k * win_size\n                    win_end = win_start + win_size - 1\n\n                    # THIS PHASE is Phase 1 or Phase 2 for other phases\n                    min0 =  float(x1[i,win_start:win_end].min())\n                    max0 =  float(x1[i,win_start:win_end].max())\n                    ptp0 =  max0 - min0\n                    \n                    loc_X[idx    , k, 0, 0] = ptp0 \n                    loc_X[idx + 1, k, 0, 1] = ptp0\n                    loc_X[idx + 2, k, 0, 2] = ptp0\n                    \n                    # PHASE 1 is this phase or Phase 2 for other phases\n                    min0 =  float(x1[i + 1,win_start:win_end].min())\n                    max0 =  float(x1[i + 1,win_start:win_end].max())\n                    ptp0 =  max0 - min0\n\n                    loc_X[idx    , k, 0, 2] = ptp0\n                    loc_X[idx + 1, k, 0, 0] = ptp0\n                    loc_X[idx + 2, k, 0, 1] = ptp0\n                    \n                    # PHASE 2 is this phase or Phase 1 for other phases\n                    min0 =  float(x1[i + 2,win_start:win_end].min())\n                    max0 =  float(x1[i + 2,win_start:win_end].max())\n                    ptp0 =  max0 - min0\n                    \n                    loc_X[idx    , k, 0, 1] = ptp0  \n                    loc_X[idx + 1, k, 0, 2] = ptp0 \n                    loc_X[idx + 2, k, 0, 0] = ptp0 \n\n    return loc_X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"821b054e8a8c272e5c5296dad122439f32215538"},"cell_type":"code","source":"# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-    CREATE ADVERSARIAL DATA SET\n\n# TRAINING DATA\n# USE THIS VALUE TO RUN IT ON THE WHOLE TRAINING SET\nnum_train = 8712\n# HERE IS A SMALLER VALUE FOR SPEED\n# num_train = 300\n\nprint('loading training set')\nX_train = extract_features(0, num_train, '../input/train.parquet')\nprint (X_train.shape)\nprint('swapping features and phases')\nX_train = np.reshape(X_train[:,:,0:win_feat,0:phases], \n                     ((num_train),windows,phases,win_feat))\nprint(X_train.shape)\n\n# TEST DATA\n# USE THIS VALUE TO RUN IT ON THE WHOLE TEST SET\nnum_test = 20337\n# HERE IS A SMALLER VALUE FOR SPEED\n# num_test = 600\n\nprint('loading testing set')\nX_test = extract_features(8712, num_test, '../input/test.parquet')\nprint(X_test.shape)\nprint('swapping features and phases')\nX_test = np.reshape(X_test[:,:,0:win_feat,0:phases], \n                     ((num_test),windows,phases,win_feat))\nprint(X_test.shape)\n\nprint('combining the two sets')\nX = np.concatenate((X_train,X_test))\nprint (X.shape)\nprint('creating target set')\ny = np.zeros((num_train+num_test))\ny[num_train:num_train + num_test] = 1\nprint (y.shape)\n### 13.3 GB peak 13.4 GB","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d57042e2e2b13c6cec6ea21c275c04052eb84957","scrolled":false},"cell_type":"code","source":"# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-    CREATE ADVERSARIAL CNN\n# This uses a Convolutional Neural Network to learn the difference between testing and training data\nCNN_scale = 50\n\nclass_model = Sequential()\nclass_model.add(Conv2D(CNN_scale, (3,phases), strides = 3, activation='relu', input_shape=(windows,phases,win_feat)))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.25))\nclass_model.add(Conv2D(CNN_scale, (3,1), strides = 3, activation='relu' ))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.5))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.5))\nclass_model.add(Flatten())\nclass_model.add(Dense(CNN_scale, activation='relu'))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.5))\nclass_model.add(Dense(CNN_scale, activation='relu'))\nclass_model.add(Dense(1, activation='sigmoid', name='output'))\n\nclass_model.compile(loss='binary_crossentropy', optimizer='adam')\n# Diplay the model summary\nprint(\"model summary\")\nclass_model.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb749997738cc12f14085afdec43eb44f1e27735"},"cell_type":"code","source":"# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-    TRAIN ADVERSARIAL CNN\nearlystopper = EarlyStopping(patience=25,\n                             verbose=1) \ncheckpointer = ModelCheckpoint('adv_classifier', verbose=1, save_best_only=True, monitor='val_loss', mode='min')\n# results = class_model.fit(X, y, validation_split=0.2, batch_size=25, epochs=300, \n#                     callbacks=[earlystopper, checkpointer])\n# Now using stratified validation split (same percentage of t, activation='relu'rue and false in both training and validation)\nresults = class_model.fit(X, y, validation_split = 0.33, \n                          batch_size=32, epochs = 300, \n                          callbacks=[earlystopper, checkpointer])\n### 13.8 GB peak 14 GB","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e0e22abc50c77823ca1ea3d0f0794fca32f2a1f9"},"cell_type":"code","source":"# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-    USE ADVERSARIAL CNN to CLASSIFY TRAINING DATA\ny_liketest = class_model.predict(X_train)\nprint(y_liketest.shape)\n# also load the training set's correct classifications in the same sequence\n# The second row of three shows the three phases of a power line that has a fault\nmeta_train = pd.read_csv('../input/metadata_train.csv')\n# correct classifications from training set\ny_train = np.zeros((num_train))\nfor i in range(0, int(num_train)):\n    y_train[i] = meta_train.target[i]\nprint(y_train.shape)\n### 13.8 GB ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e439184391fe0af5ad43512d8fc003853f9d84c1"},"cell_type":"code","source":"# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-    SORT TRAINING DATA BY TEST-LIKENESS (AND SORT Y)\ny_sorted = np.sort(y_liketest, axis=0)\nplt.plot(y_sorted)\n# see what value would split the data 80% / 20%\neighty = int(num_train * .8)\ntwenty = num_train - eighty\nthresh80 = y_sorted[eighty,0]\n# copy dummy data just to dimension the arrays\nX = X_train[0:eighty]\nX_valid = X_train[0:twenty]\ny = y_train[0:eighty]\ny_valid = y_train[0:twenty]\n# now put the TRAIN data that most resembles the TEST data into the validation set, and use the others for training\ntidx = 0\nvidx = 0\nfor i in range(num_train) :\n    if (y_liketest[i] >= thresh80) and (vidx < twenty) :\n        X_valid[vidx] = X_train[i]\n        y_valid[vidx] = y_train[i]\n        vidx = vidx + 1\n    elif tidx < eighty : \n        X[tidx] = X_train[i]\n        y[tidx] = y_train[i]\n        tidx = tidx + 1\n    else :\n        X_valid[vidx] = X_train[i]\n        y_valid[vidx] = y_train[i]\n        vidx = vidx + 1\n\nprint(X.shape, X_valid.shape, y.shape, y_valid.shape)\n        \nnum_to_use = num_train\n# Quick and dirty RAM cleanup\ny_liketest = 5\n\n### 13.8 GB ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ed6147ab2c51dda60499c8756f91e45c00e3cfd"},"cell_type":"code","source":"# quick and dirty RAM cleanup\nmeta_train = 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69c6aaf5574a5d3cfe114628684624ae591da68e"},"cell_type":"code","source":"#Show stats of TEST data\nprint(\"Was Adversarial creation of a Validation set worth it? Let's compare feature means and variances...\")\nprint (\"\")\nprint(\"X mean (unscaled)\")\nXmin = X.mean(axis=0).min()\nXmax = X.mean(axis=0).max()\nprint(\"X mean ranges from \", Xmin,\" to \", Xmax, \" for a span of \", Xmax - Xmin)\nprint (\"X Variance (unscaled)\")\nXmin = X.var(axis=0).min()\nXmax = X.var(axis=0).max()\nprint(\"X var ranges from \", Xmin,\" to \", Xmax, \" for a span of \", Xmax - Xmin)\nprint(\" \")\nprint(\"Validation mean (unscaled)\")\nXmin = X_valid.mean(axis=0).min()\nXmax = X_valid.mean(axis=0).max()\nprint(\"Validation mean ranges from \", Xmin,\" to \", Xmax, \" for a span of \", Xmax - Xmin)\nprint (\"Validation Variance (unscaled)\")\nXmin = X_valid.var(axis=0).min()\nXmax = X_valid.var(axis=0).max()\nprint(\"Validation var ranges from \", Xmin,\" to \", Xmax, \" for a span of \", Xmax - Xmin)\nprint(\"\")\n#Show stats of TEST data\nprint(\"TEST SET mean (unscaled)\")\nXmin = X_test.mean(axis=0).min()\nXmax = X_test.mean(axis=0).max()\nprint(\"TEST SET mean ranges from \", Xmin,\" to \", Xmax, \" for a span of \", Xmax - Xmin)\nprint (\"TEST SET Variance (unscaled)\")\nXmin = X_test.var(axis=0).min()\nXmax = X_test.var(axis=0).max()\nprint(\"TEST SET var ranges from \", Xmin,\" to \", Xmax, \" for a span of \", Xmax - Xmin)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85b0bc469f5a1f51dfaabd697c6aeabf53251ca5"},"cell_type":"code","source":"# Will use the same CNN to categorize the data\nCNN_scale = 50\n\nclass_model = Sequential()\nclass_model.add(Conv2D(CNN_scale, (3,phases), strides = 3, activation='relu', input_shape=(windows,phases,win_feat)))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.25))\nclass_model.add(Conv2D(CNN_scale, (3,1), strides = 3, activation='relu' ))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.5))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.5))\nclass_model.add(Flatten())\nclass_model.add(Dense(CNN_scale, activation='relu'))\nclass_model.add(BatchNormalization())\nclass_model.add(Dropout(0.5))\nclass_model.add(Dense(CNN_scale, activation='relu'))\nclass_model.add(Dense(1, activation='sigmoid', name='output'))\n\nclass_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[matthews_correlation])\n# Diplay the model summary\nprint(\"model summary\")\nclass_model.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82a6a52776433bff1f23a1d94ada9f30a7cdd998","scrolled":false},"cell_type":"code","source":"### Train the residual compressor\nearlystopper = EarlyStopping(patience=25,\n                             verbose=1) \ncheckpointer = ModelCheckpoint('VSB_classifier', verbose=1, save_best_only=True, monitor='val_matthews_correlation', mode='max')\n# results = class_model.fit(X, y, validation_split=0.2, batch_size=25, epochs=300, \n#                     callbacks=[earlystopper, checkpointer])\n# Now using stratified validation split (same percentage of t, activation='relu'rue and false in both training and validation)\nresults = class_model.fit(X, y, \n                          validation_data=[X_valid, y_valid], \n                          batch_size=32, epochs = 300, \n                          callbacks=[earlystopper, checkpointer])\n### 13.2 GB peak 13.5 GB","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f92c3ec6072d77d86dbd3798c0e6f9fab3b0979"},"cell_type":"code","source":"\nclass_model = load_model('VSB_classifier', custom_objects={'matthews_correlation':matthews_correlation})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"34fa400fc2d83ba9fb12adee2f2058d3e2ef927c"},"cell_type":"code","source":"# X = X_raw\n# y = y_raw\nX = X_train\ny = y_train\n# Quick and dirty RAM cleanup\nX_train = 5\ny_train = 5\nX_valid = 5\ny_valid = 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40fe41a0652eeca370dda60dd836a8b95e0bd087"},"cell_type":"code","source":"pred_y = class_model.predict(X)\npredicted_y = pred_y[:,0] #np.concatenate(pred_y)[0]\npredicted_y.shape\n### 13.2 GB ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"59daebdaacb08e0ada1d809e349456e5966e70ba"},"cell_type":"code","source":"# This function copied from https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694\n#    courtesy Bruno Aquino - 5-fold LSTM Attention (fully commented)\n# The output of this kernel must be binary (0 or 1), but the output of the NN Model is float (0 to 1).\n# So, find the best threshold to convert float to binary is crucial to the result\n# this piece of code is a function that evaluates all the possible thresholds from 0 to 1 by 0.01\ndef threshold_search(y_true, y_proba):\n    best_threshold = 0\n    best_score = 0\n    for threshold in tqdm([i * 0.01 for i in range(100)]):\n        score = K.eval(matthews_correlation(y_true.astype(np.float64), (y_proba > threshold).astype(np.float64)))\n        if score > best_score:\n            best_threshold = threshold\n            best_score = score\n    search_result = {'threshold': best_threshold, 'matthews_correlation': best_score}\n    return search_result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16b647c090c3499b717d75f2f2f0a42f449c0a17"},"cell_type":"code","source":"result = threshold_search(y, predicted_y)\n### 13.2 GB peak 13.2 GB","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7dd5e32e1075c430739e1f1374e0b13695cfabb1"},"cell_type":"code","source":"best_thresh = result['threshold']\nBest_MCC = result['matthews_correlation']\nprint('best MCC of ', Best_MCC, ' using best threshold ', best_thresh)\npredicted_y = pred_y\n# quick and dirty RAM cleanup\npred_y = 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65055b4d4e0aabb311ca529d446a4b61e9addb60"},"cell_type":"code","source":"y = predicted_y\n# See how many doubles and triples we got on the test data\n# How often did we predict power line faults occur on all three phases simultaneously, versus on fewer than all 3?\ntriples = 0\ndoubles = 0\nsingles = 0\nfor i in range(0,int(num_to_use),3) : \n    y[i] = int(y[i] > best_thresh) \n    y[i+1] = int(y[i+1] > best_thresh)\n    y[i+2] = int(y[i+2] > best_thresh)\n    num_phases_faulty = y[i] + y[i+1] + y[i+2]\n    if (num_phases_faulty == 3):\n        triples = triples + 1\n    elif (num_phases_faulty == 2):\n        doubles = doubles + 1\n    elif (num_phases_faulty == 1):\n        singles = singles + 1\n\nprint('triples', triples, 'doubles', doubles, 'singles', singles)\nprint('sanity check: ', 'total faults', y.sum(), ' sum of above ', 3 * triples + 2 * doubles + singles)\n### 13.3 GB ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"716855cef61e0136a7b8201ef6ba02e23040c69d"},"cell_type":"code","source":"X = X_test\nnum_to_use = num_test\nprint(X.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d558a4cb925297416573c43e2d29dcf252bcc0a9"},"cell_type":"code","source":"# quick and dirty RAM cleanup\nX_unscaled = 5\nprint (X.shape)\n### 12.8 GB","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8412016cd69798748e74be9f364fa7705b920cc9"},"cell_type":"code","source":"y = class_model.predict(X)\n\n# How often did we predict power line faults occur on all three phases simultaneously, versus on fewer than all 3?\ntriples = 0\ndoubles = 0\nsingles = 0\nfor i in range(0,int(num_to_use),3) : \n    y[i] = int(y[i] > best_thresh) \n    y[i+1] = int(y[i+1] > best_thresh)\n    y[i+2] = int(y[i+2] > best_thresh)\n    num_phases_faulty = y[i] + y[i+1] + y[i+2]\n    if (num_phases_faulty == 3):\n#        print('triple',meta_train.signal_id[i], meta_train.phase[i] )\n        triples = triples + 1 \n    elif (num_phases_faulty == 2):\n#        print('double',meta_train.signal_id[i], meta_train.phase[i])\n        doubles = doubles + 1\n    elif (num_phases_faulty == 1):\n#        print('single',meta_train.signal_id[i], meta_train.phase[i])\n        singles = singles + 1\n\nprint('triples', triples, 'doubles', doubles, 'singles', singles)\nprint('sanity check: ', 'total faults', y.sum(), ' sum of above ', 3 * triples + 2 * doubles + singles)\n# plt.plot(meta_train.target)\n### 12.2 GB","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"df30f3c42f11c39ab8ad134b238395d3b5ddbd7c"},"cell_type":"code","source":"meta_test = pd.read_csv('../input/metadata_test.csv')\noutput = pd.DataFrame({\"signal_id\":meta_test.signal_id[0:int(num_to_use)]})\n# Use this one for NN \noutput[\"target\"] = pd.Series(y[:,0]) \n# Use this one for Random Forest\n# output[\"target\"] = pd.Series(y[:]) \n\noutput['signal_id'] = output['signal_id'].astype(np.int64)\noutput['target'] = output['target'].astype(np.int64)\noutput.to_csv(\"submission.csv\", index=False)\noutput\n### 13.2 GB","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}