{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Keras MLP with Extended Training Data\n\nThis kernel tries a simple multi-layer NN with just the tabular data. No images here. \n\nIt extends the training data by taking all the data per patient, and creating all possible combinations of them. The thinking is that the goal here is to predict the FVC values per every week from -12 to 133 from just the based FVC value and image. You are given a set of training data with the base FVC value, and number of FVC values for the following weeks (or weeks before). In the test scenario, you must then predict the FVC for all the weeks, given just one week and its FVC.\n\nThe extended training data here is taking each FVC instance per patient, and creating a new row where that weeks FVC is the base and all the other given FVC/week combinations are the target. It gives about 12k rows of training data vs the 1.5k or so in the original. \n\nMuch clever, such data? Don't know, the results score at around position 500, but maybe I am just bad at it :)\n\nI have also experimented with combining this with my 3D CNN model, but that combination did not come up as very useful. This MLP kernel alone scored better than that combination, so something wrong there I guess. But the idea there was for me to train the CNN for images separately, and then combine with this MLP for experiments, since the images would just repeat for all the extended data generated here, making it very resource intensive to train with 12k images that just repeat (although my CNN kernel adds augmentation so its not 100% the same). I have a kernel that does that (combines this MLP with the CNN), but I'd rather not spam too many kernels with almost the same code. \n\nHope you find something interesting in this.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold, GroupShuffleSplit\n\nimport tensorflow\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.datasets import mnist\nfrom tensorflow.keras.layers import Dense, Flatten, Input\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, BatchNormalization, Dropout, Activation\nfrom tensorflow.keras.layers import Conv3D, MaxPooling3D\nfrom tensorflow.keras import layers\n\nfrom tensorflow.keras.callbacks import (ModelCheckpoint, LearningRateScheduler,\n                             EarlyStopping, ReduceLROnPlateau,CSVLogger)\n\nfrom tensorflow.keras import backend as K\n\nimport matplotlib.pylab as plt\n\nfrom skimage.io import imread\nfrom skimage.transform import resize\nfrom keras.utils import Sequence\nimport math\n\n\nfrom tqdm.auto import tqdm\ntqdm.pandas()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# A Few Configuration Variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#which columns from the train and test dataframe to use for the model (the Dense layers..)\ndense_cols = ['base_fvc', 'base_week', 'pct', 'age', 'gender_female', 'gender_male', 'smoking_status', 'target_week']\nbatch_size = 16\nepochs = 30\n#how many splits to do on the training data, or how many cross-validation rounds to use\nN_SPLITS = 10\n#my_test_pct is a percentage of values left out of test/validation data to compare the final model against known results\nmy_test_pct = 0.05\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The preprocessed data is in a [dataset](https://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed) I previously uploaded. This is where it mounts:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = \"/kaggle/input/osic-pulmonary-fibrosispreprocessed/dataset\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Brief Overview of the Data\n\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df_train_orig = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv\")\ndf_train = pd.read_csv(f\"{DATA_DIR}/df_train_scaled_continous_smoke.csv\").drop(\"Unnamed: 0\", axis=1)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case I used a continous value for the smoking feature, although you can say it is categorical. But 0 for non-smoking, 1 for currently smoking, and 0.5 for used to smoke.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[\"SmokingStatus\"].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#the original competition data, not preprocessed or anything else like that\ndf_test_orig = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/test.csv\")\ndf_test_orig.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"List of patients in the training set. This can be useful, for example, to make patient-grouped data-splits.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_ids = df_train[\"Patient\"].unique()\npatient_ids.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extend Training Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we create new rows for all combinations of base-target FVC values that can be created from the train and test sets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_ids = df_train[\"Patient\"].unique()\ntraining_rows = []\npatient_count = 0\nfor patient_id in tqdm(patient_ids):\n    df_patient = df_train[df_train[\"Patient\"] == patient_id]\n    patient_row_count = 0\n    for idx, row in df_patient.iterrows():\n        row_fvc = row[\"FVC\"]\n        for idx2, row2 in df_patient.iterrows():\n            if row2[\"FVC\"] == row_fvc:\n                continue\n            patient_row_count += 1\n            training_row = {}\n            training_row[\"patient_id\"] = row[\"Patient\"]\n            training_row[\"base_fvc\"] = row_fvc\n            training_row[\"base_week\"] = row[\"Weeks\"]\n            training_row[\"pct\"] = row[\"Percent\"]\n            training_row[\"age\"] = row[\"Age\"]\n            training_row[\"gender_female\"] = row[\"Sex_Female\"]\n            training_row[\"gender_male\"] = row[\"Sex_Male\"]\n            training_row[\"smoking_status\"] = row[\"SmokingStatus\"]\n            training_row[\"target_week\"] = row2[\"Weeks\"]\n            training_row[\"target_fvc\"] = row2[\"fvc_raw\"]\n            training_rows.append(training_row)\n    print(f\"created {patient_row_count} instances for patient {patient_id}\")\n    patient_count += 1\nprint(f\"processed {patient_count} patients\")\nprint(f\"rows before: {df_train.shape}\")\nprint(f\"extended rows: {len(training_rows)}\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_new_train = pd.DataFrame(training_rows)\ndf_new_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_new_train.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_cols = [col for col in df_new_train.columns if col != \"target_fvc\"]\ndf_x = df_new_train[x_cols]\ndf_y = df_new_train[\"target_fvc\"]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_x.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_y.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing the Test Set\n\nUnlike my CNN model, I will submit this just as an example for the competition results. For that, the test set will be different than what is given in the downloadable data. So I cannot rely on only the preprocessed test set but have to preprocess the live test set here. \n\nWhen the kernel is submitted for the competition, Kaggle runs the kernel again but with the much larger and real test set. You can actually take the trained model weights from above and build a kernel to do only predictions with the pre-trained model. Running it with the given test set of 5 rows takes about 4 minutes. \n\nRunning a prediction version only with the actual submission data takes somewhere around 2 hours, so thats about the scale difference between the two as well.. And also why I believe you must have internet off, so you cannot upload the real test set and look at it yourself outside the submission. And why there is no execution log to debug errors, you could see information about the actual test set in that log.\n\nIn any case, the functions to do the preprocessing:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler, LabelEncoder, OneHotEncoder, MultiLabelBinarizer\nimport pickle\n\n#scale numerical data to 0-1, one-hot categoricals. one-hot smoking if desired, else just 0, 0.5, and 1.\ndef scale_data(df_train, df_test, scale_smoking):\n    df_train[\"train\"] = 1\n    df_test[\"train\"] = 0\n    df_full = pd.concat([df_train, df_test], axis=0)\n    df_full[\"fvc_raw\"] = df_full[\"FVC\"]\n    df_full = label_encode_data(df_full, scale_smoking)\n    df_full = scale_minmax(df_full, scale_smoking)\n    df_full = onehot_data(df_full, scale_smoking)\n\n    df_train = df_full[df_full[\"train\"] == 1].drop(\"train\", axis=1)\n    df_test = df_full[df_full[\"train\"] == 0].drop(\"train\", axis=1)\n    return df_train, df_test\n\ndef scale_minmax(df_full, scale_smoking):\n    minmax = MinMaxScaler()\n    scale_cols = [\"Weeks\", \"FVC\", \"Percent\", \"Age\"]\n    if scale_smoking:\n        scale_cols.append(\"SmokingStatus\")\n    df_full[scale_cols] = minmax.fit_transform(df_full[scale_cols])\n    with open(\"minmax.pkl\", \"wb\") as f:\n        pickle.dump(minmax, f)\n    return df_full\n\ndef label_encode_data(df_full, scale_smoking):\n    le = LabelEncoder()\n    #fit first to ensure range 0-2 starting from never smoked to currently smoked, so value represent scale of current smoking\n    le.fit([\"Never smoked\", \"Ex-smoker\", \"Currently smokes\"])\n    if scale_smoking:\n        df_full[\"SmokingStatus\"] = le.transform(df_full[\"SmokingStatus\"])\n    with open(\"smoking_labelencoder.pkl\", \"wb\") as f:\n        pickle.dump(le, f)\n\n    # le = LabelEncoder()\n    # #fit first to ensure range 0-2 starting from never smoked to currently smoked, so value represent scale of current smoking\n    # le.fit(df_full[\"Sex\"])\n    # df_full[\"Sex\"] = le.transform(df_full[\"Sex\"])\n    # with open(\"gender_labelencoder.pkl\", \"wb\") as f:\n    #     pickle.dump(le, f)\n    return df_full\n\n#https://blog.cambridgespark.com/robust-one-hot-encoding-in-python-3e29bfcec77e\ndef onehot_data(df_full, scale_smoking):\n    cols_to_encode = [\"Sex\"]\n    if not scale_smoking:\n        cols_to_encode.append(\"SmokingStatus\")\n    df_full = pd.get_dummies(df_full, columns=cols_to_encode)\n    return df_full\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, run the actual scaling on the test data. Cannot use the pre-processed one from my dataset as the actual competition submission has a different test dataset.\n\nSo the following processes the test data into the same format as the training data. In the test data there is only one FVC per patient, so it cannot be extended (and shouldn't, its the test set after all..).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_scaled__, df_test_scaled = scale_data(df_train_orig, df_test_orig, True)\ndf_test = df_test_scaled","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_scaled","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_ids = df_test[\"Patient\"].unique()\ntest_rows = []\npatient_count = 0\nfor patient_id in tqdm(patient_ids):\n    df_patient = df_test[df_test[\"Patient\"] == patient_id]\n    patient_row_count = 0\n    for idx, row in df_patient.iterrows():\n        row_fvc = row[\"FVC\"]\n        patient_row_count += 1\n        test_row = {}\n        test_row[\"patient_id\"] = row[\"Patient\"]\n        test_row[\"base_fvc\"] = row_fvc\n        test_row[\"base_week\"] = row[\"Weeks\"]\n        test_row[\"pct\"] = row[\"Percent\"]\n        test_row[\"age\"] = row[\"Age\"]\n        test_row[\"gender_female\"] = row[\"Sex_Female\"]\n        test_row[\"gender_male\"] = row[\"Sex_Male\"]\n        test_row[\"smoking_status\"] = row[\"SmokingStatus\"]\n        test_rows.append(test_row)\n    print(f\"created {patient_row_count} instances for patient {patient_id}\")\n    patient_count += 1\nprint(f\"processed {patient_count} patients\")\nprint(f\"rows before: {df_test.shape}\")\nprint(f\"extended rows: {len(test_rows)}\")\n    #break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Custom Keras Sequence Generator\n\nHere I define a Keras Sequence generator to provide batches. You don't really need this for this tabular dataset, as the data does not need to be augmented in any way. So it would actually be much simpler to use just basic Keras functionality and provide the training data as is.\n\nHowever, I also wanted to try to combine this with my [3D CNN kernel] model, in which case I needed a generator to provide both the augmented 3D images and the tabular data rows. So doing the generator here allows simple combination of the two when needed. I did another kernel doing that, but its nothing special and scores less than this one so leaving it unpublished for now.\n\nFirst, a small utility function to shuffle x and y at the end of each training epoch:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def unison_shuffled_copies(a, b):\n    assert len(a) == len(b)\n    p = np.random.permutation(len(a))\n    new_a = a.iloc[p]\n    new_b = b.iloc[p]\n    return new_a, new_b","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And the generator itself:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#https://stackoverflow.com/questions/49404993/keras-how-to-use-fit-generator-with-multiple-inputs\nclass MySequence3D(Sequence):\n\n    def __init__(self, x_set, y_set, batch_size, mode=\"train\", augment=True):\n        self.x, self.y = x_set, y_set\n        self.batch_size = batch_size\n        self.mode = mode\n        self.max_idx = math.ceil(len(x_set)/batch_size)\n        self.augment = augment\n\n    def __len__(self):\n        #TODO: check is correct\n        return int(np.ceil(len(self.x) / float(self.batch_size)))\n\n    def __getitem__(self, idx):\n        idx2 = idx % self.max_idx\n        start = idx2 * self.batch_size\n        end = min(start + batch_size, len(self.x))\n        batch_x = self.x.iloc[start : end]\n        batch_y = self.y.iloc[start : end]\n        \n        next_batch_num = []\n        for index, row in batch_x.iterrows():\n            #print(row)\n            nums = [row[col] for col in dense_cols]\n            nums = np.array(nums)\n            file_name = row[\"patient_id\"]\n            next_batch_num.append(nums)\n        np_y = np.array(batch_y)\n        np_x_num = np.array(next_batch_num)\n        del next_batch_num\n        del batch_y\n        #print(f\"loaded shape: {np_x.shape}, batch={idx}\")\n        result = np_x_num, np_y\n\n        #print(f\"shapes: {result[0].shape}, {result[1].shape}\")\n        return result\n\n    def on_epoch_end(self):\n        self.x, self.y = unison_shuffled_copies(self.x, self.y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions to Create and Train the Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model():\n    num_input = Input(shape=(len(dense_cols,)))\n    dense = Dense(200, activation='relu')(num_input)\n    dense = BatchNormalization()(dense)\n    dense = Dropout(0.45)(dense)\n    dense = Dense(200, activation='relu')(dense)\n    dense = BatchNormalization()(dense)\n    dense = Dropout(0.35)(dense)\n    #dense = Dense(50, activation='relu')(num_input)\n    final_dense = Dense(1)(dense)\n    model = keras.Model(\n        inputs=[num_input],\n        outputs=[final_dense],\n    )\n    adam = tensorflow.keras.optimizers.Adam(lr=0.001)\n\n    model.compile(loss='mean_squared_error',\n                  optimizer=adam,  #keras.optimizers.SGD(lr=0.01),\n                  metrics=[tf.keras.metrics.RootMeanSquaredError(name='rmse')])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit_model(model, callbacks_list, df_x_train, df_y_train, df_x_val, df_y_val):\n    train_gen = MySequence3D(df_x_train, df_y_train, batch_size, augment=True)\n    valid_gen = MySequence3D(df_x_val, df_y_val, batch_size, augment=False)\n\n    #the total number of images we have:\n    train_size = df_x_train.shape[0]\n    #train_steps is how many steps per epoch Keras runs the genrator. One step is batch_size*images\n    train_steps = train_size/batch_size\n    #use 2* number of images to get more augmentations in. some do, some dont. up to you\n    train_steps = int(train_steps) #2* seems to break this?\n    #same for the validation set\n    valid_size = df_x_val.shape[0]\n    valid_steps = valid_size/batch_size\n    valid_steps = int(valid_steps) #again, no 2*?    \n\n    fit_history = model.fit_generator(\n            train_gen,\n            steps_per_epoch=train_steps,\n            epochs = epochs,\n            validation_data=valid_gen,\n            validation_steps=valid_steps,\n            callbacks=callbacks_list,\n        use_multiprocessing=False,\n        workers=2,\n        verbose = 1\n    )\n    return fit_history\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom sklearn.model_selection import train_test_split\n\n# create callbacks list\ndef create_callbacks(idx):\n    checkpoint = ModelCheckpoint(f'../working/weights_best_{idx}.h5', monitor='val_loss', verbose=1, \n                                 save_best_only=True, mode='min', save_weights_only = True)\n    reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3,\n                                       verbose=1, mode='auto', epsilon=0.0001)\n    early = EarlyStopping(monitor=\"val_loss\", \n                          mode=\"min\", \n                          patience=22)\n\n    csv_logger = CSVLogger(filename='../working/training_log.csv',\n                           separator=',',\n                           append=True)\n\n    callbacks_list = [checkpoint, csv_logger, early]\n    return callbacks_list\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions to Make Predictions Using Trained Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_predictions(model):\n    predictions = []\n    for target_week in tqdm(range(-12,134)):\n        for row in test_rows:\n            row[\"target_week\"] = (target_week+12)/(133+12)\n            tab_data = [row[col] for col in dense_cols]\n            tab_data = np.array(tab_data)\n            patient_id = row[\"patient_id\"]\n            tabs = np.array([tab_data])\n            pred = model.predict([tabs])\n            predictions.append(pred)\n            #print(f\"target week: {row['target_week']}, pred: {pred}\")\n    predictions = np.array(predictions)\n    predictions = predictions.flatten()\n    return predictions\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_predictions_dict(model, test_rows):\n    if isinstance(test_rows, pd.DataFrame):\n        test_rows = [row.to_dict() for (idx, row) in test_rows.iterrows()]\n    predictions = {}\n    col_names = []\n    for target_week in tqdm(range(-12,134)):\n        for idx, row in enumerate(test_rows):\n            row[\"target_week\"] = (target_week+12)/(133+12)\n            tab_data = [row[col] for col in dense_cols]\n            tab_data = np.array(tab_data)\n            patient_id = row[\"patient_id\"]\n            tabs = np.array([tab_data])\n            pred = model.predict([tabs])\n            col_name = f\"{patient_id}_{target_week}\"\n            col_names.append(col_name)\n            predictions[col_name] = pred.flatten()[0]\n            #print(f\"target week: {row['target_week']}, pred: {pred}\")\n        \n    return predictions, col_names","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_predictions_my_test(model, test_rows):\n    test_rows = [row.to_dict() for (idx, row) in test_rows.iterrows()]\n    predictions = []\n    for idx, row in tqdm(enumerate(test_rows), total=len(test_rows)):\n#        row[\"target_week\"] = (target_week+12)/(133+12)\n        tab_data = [row[col] for col in dense_cols]\n        tab_data = np.array(tab_data)\n        patient_id = row[\"patient_id\"]\n        tabs = np.array([tab_data])\n        pred = model.predict([tabs])\n        predictions.append(pred.flatten()[0])\n        #print(f\"target week: {row['target_week']}, pred: {pred}\")\n        \n    return predictions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Splitting the Data into N-Folds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"indices = np.arange(df_x.shape[0])\n# create a set of indexes for a separate, final, test set\n#https://github.com/scikit-learn/scikit-learn/issues/9193\ntrain_indices, my_test_indices = next(GroupShuffleSplit(test_size=my_test_pct, random_state=8).split(indices, groups=df_x[\"patient_id\"]))\n#train_inds, test_inds = next(GroupShuffleSplit().split(X, groups=groups))\nmy_test_X = df_x.iloc[my_test_indices]\nmy_test_y = df_y.iloc[my_test_indices]\n\nfull_indices = indices\n#set only the 90% original data as train/test data for N-splits. In the end we shall test on the valid in\nindices = train_indices\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Actually Training the Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# First, create a set of indexes of the 5 folds\n#train_indices, valid_indices = train_test_split(indexes, test_size=0.10, random_state=8, stratify=df_x[\"patient_id\"])\n\n#splits = list(StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=2020).split(indices, df_x.iloc[indices][\"patient_id\"]))\nsplits = list(GroupKFold(n_splits=N_SPLITS).split(indices, groups=df_x.iloc[indices][\"patient_id\"]))\npreds_val = []\ny_val = []\npreds_test = []\npreds_my_test = []\ncol_names = []\nfit_histories = []\n# If you dont know, enumerate(['a', 'b', 'c']) returns [(0, 'a'), (1, 'b'), (2, 'c')]\nfor idx, (train_idx, val_idx) in enumerate(splits):\n    K.clear_session() # start Keras from clean state in each iteration\n    print(\"Beginning fold {}\".format(idx+1))\n    # use the indexes to extract the folds in the train and validation data\n    train_X, train_y, val_X, val_y = df_x.iloc[train_idx], df_y.iloc[train_idx], df_x.iloc[val_idx], df_y.iloc[val_idx]\n    # instantiate the model for this fold\n    model = create_model()\n    callbacks = create_callbacks(idx)\n    # Train, train, train\n    history = fit_model(model, callbacks, train_X, train_y, val_X, val_y)\n    fit_histories.append(history)\n    #model.fit(train_X, train_y, batch_size=128, epochs=50, validation_data=[val_X, val_y], callbacks=callbacks)\n    # loads the best weights saved by the checkpoint\n    model.load_weights(f'weights_best_{idx}.h5')\n    # Add the predictions of the validation to the list preds_val\n    preds_val.append(model.predict(val_X[dense_cols], batch_size=512))\n    # and the val true y\n    y_val.append(val_y)\n#    pred_test = make_predictions(model)\n    pred_test, col_names = make_predictions_dict(model, test_rows)\n    preds_test.append(pred_test)\n    pred_test = make_predictions_my_test(model, my_test_X)\n    preds_my_test.append(pred_test)\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(preds_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# A Look at the Predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#this would be the \"fake\" Kaggle test set that comes with the downloadable data\ndf_test_predictions = pd.DataFrame(preds_test)\ndf_test_predictions = df_test_predictions[col_names] #to get sorted order","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_predictions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And `preds_my_test` is my own test set split from the original training data, so the actual results are known and comparable. It is some percentage (5%) of the 12k rows. For specific set of patients, where the patient_id does not overlap with the data used in training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(preds_my_test[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_my_test_mean = np.mean(preds_my_test, axis=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What is the MSE to the actual data, if we average all folds?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\n\nmean_squared_error(my_test_y, preds_my_test_mean, squared=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_my_test_mean.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions = pd.DataFrame()\nfor idx, mtp in enumerate(preds_my_test):\n    df_mytest_predictions[f\"{idx+1}\"] = mtp\n#df_mytest_predictions = df_test_predictions[col_names] #to get sorted order\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How big is the diff between the min and max prediction per row made by the different folds?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions_diff = pd.DataFrame()\nfor idx, mtp in enumerate(preds_my_test):\n    df_mytest_predictions_diff[f\"{idx+1}\"] = np.abs(np.array(mtp) - my_test_y.values)\n#df_mytest_predictions = df_test_predictions[col_names] #to get sorted order\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Raw predictions for my test set:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions.T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions.T.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above shows the mean, str, etc over the N (10 here) folds for all patients in the test set.\n\nBy using describe() on describe(), we can get aggregated statistics on the overall dataset (so mean over all patients means, etc):","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions.T.describe().T.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Above was for raw predictions, the same stats for the diff: ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions_diff.T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions_diff.T.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mytest_predictions_diff.T.describe().T.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plotting and Visualizing the Training History","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_loss_and_accuracy(fit_history, n=0):\n    plt.clf()\n    plt.plot(fit_history.history['rmse'][n:])\n    plt.plot(fit_history.history['val_rmse'][n:])\n    plt.title('model rmse')\n    plt.ylabel('rmse')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'test'], loc='upper left')\n    plt.show()\n    plt.clf()\n    #loss is same as rmse in this case, so just not plotting the same twice. so commented out.\n    # summarize history for loss\n    #plt.plot(fit_history.history['loss'][n:])\n    #plt.plot(fit_history.history['val_loss'][n:])\n    #plt.title('model loss')\n    #plt.ylabel('loss')\n    #plt.xlabel('epoch')\n    #plt.legend(['train', 'test'], loc='upper left')\n    #plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx, fit_history in enumerate(fit_histories):\n    print(f\"fold {idx}\")\n#    plot_loss_and_accuracy_last(fit_history)\n    plot_loss_and_accuracy(fit_history)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx, fit_history in enumerate(fit_histories):\n    print(f\"fold {idx}\")\n#    plot_loss_and_accuracy_last(fit_history)\n    plot_loss_and_accuracy(fit_history, 4)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx, fit_history in enumerate(fit_histories):\n    print(f\"fold {idx}\")\n#    plot_loss_and_accuracy_last(fit_history)\n    plot_loss_and_accuracy(fit_history, 8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will not see the actual test data, ever, but we can always take a look and see that the predictions over the fake test data at least look reasonable. This should (more likely) translate into a successful submissions:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_predictions.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"descriptions = df_test_predictions.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"descriptions.max(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"descriptions.T.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#my_test_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_predictions.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above shape shows we have made predictions for all 5 fake test patients, for all 10 folds:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#this shold show each 5 fake patients has number of predictions matching weeks -12...133:\ndf_test_predictions.shape[1]/5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculating Submission Confidence\n\nBesides the raw FVC values prediction, the competition requires a confidence value per prediction.\n\nIn this case, I take the N-folds and pick the range between the min and max value predicted per test set row as the confidence. So if out of 10 folds, the highest prediction for row 1 is 3100 and the lowest prediction of the 10 folds for row 1 is 2900, the confidence becomes 3000-2900=100.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_confidence = pd.DataFrame(df_test_predictions.max()-df_test_predictions.min())\ndf_confidence.columns = [\"Confidence\"]\ndf_confidence","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am using the average of all folds as the prediction here:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_test_mean = np.mean(df_test_predictions, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_test_mean.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_test_mean.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Building the Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_my_submission = preds_test_mean.to_frame()\ndf_my_submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#this sets the header correct in the generated CSV file, and merges the confidence correct\ndf_my_submission.index.names = ['Patient_Week']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_my_submission.columns = [\"FVC\"]\ndf_my_submission[\"FVC\"] = df_my_submission[\"FVC\"].astype(int)\ndf_my_submission = df_my_submission.join(df_confidence, how=\"outer\")\ndf_my_submission[\"Confidence\"] = df_my_submission[\"Confidence\"].astype(int)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_my_submission.to_csv(\"submission.csv\", index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_my_submission.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}