{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Intuition of Data Augmentation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Set-up\n\nIn this challenge, each patient is given with specific and time-varying informations. For the time being, we will only focus on tabular data. \n\nMore precisely in the training dataset for a patient $i$ at week $t$ we are given time varying informations $x_{it}$ and patient-specific traits $p_i$. \n* $x_{it}$ encompasses: `Percent`, `FVC`, and `Week`, i.e $$ x_{it} = (\\text{FVC}_{it},\\text{Percent}_{it}, \\text{Weeks}_{it})  $$\n* $p_{i}$ encompasses: `Sex`, `Age`, and `Smoking Status`\n\nBut in the test dataset, we can observe patient-specific traits $p_i$, and time-varying features for only one week, let's say $\\tau_i$, that is we only know $x_{i \\tau_i}$.\n\n### Augmentation Strategy\n\nTherefore to line-up with the test dataset set-up, we can augment the dataset with this strategy.Now we are going to devise a dataset with a three-level key: $i$ for patient level, $t$ for week level and $\\tau$ for base week. We will keep using patient-specific trait $p_i$ and time-varying $x_{it}^{\\tau}$ where:\n$$  x_{it}^{\\tau} = (\\text{FVC}_{i \\tau},\\text{Percent}_{i \\tau}, \\text{Weeks}_{i \\tau}, \\text{Weeks}_{it} - \\text{Weeks}_{i \\tau})$$\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport random\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nfrom sklearn.model_selection import KFold,StratifiedKFold\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow.keras.backend as K\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed=2020):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    \nseed_everything(42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = \"../input/osic-pulmonary-fibrosis-progression\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df_tr = pd.read_csv(f\"{ROOT}/train.csv\")\nchunk = pd.read_csv(f\"{ROOT}/test.csv\")\nte = pd.read_csv(f\"{ROOT}/sample_submission.csv\", usecols=['Patient_Week'])\n#===\nprint(\"Naive doublon handling...\")\nchunk.drop_duplicates(keep=False, inplace=True, subset=['Patient'])\ndf_tr.drop_duplicates(keep=False, inplace=True, subset=['Patient','Weeks'])\n#===","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"te['Patient'] = te['Patient_Week'].apply(lambda x:x.split('_')[0])\nte['Weeks'] = te['Patient_Week'].apply(lambda x: int(x.split('_')[-1]))\npiv = df_tr[['Patient','Weeks','FVC','Percent']].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_tr.shape, chunk.shape, te.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Rename columns for pivot dataframes\")\nren_dct = {\"Weeks\":\"base_Weeks\",\"FVC\":\"base_FVC\",\"Percent\":\"base_Percent\"}\ndf_tr = df_tr.rename(columns=ren_dct)\nchunk = chunk.rename(columns=ren_dct)\nprint(\"Test handling...\")\nte = te.merge(chunk, on=\"Patient\", how=\"left\")\ndel chunk\nprint(\"Train handling...\")\nWEEKS = df_tr.base_Weeks.unique()\nCHUNKS = []\nfor week in tqdm(WEEKS):\n    tp = piv.merge(df_tr.loc[df_tr.base_Weeks==week], on=\"Patient\", how=\"inner\")\n    CHUNKS.append(tp)\ntr = pd.DataFrame()\ntr = tr.append(CHUNKS)\nprint(\"original training dataset\", df_tr.shape)\nprint(\"augmented training dataset\", tr.shape)\ndel WEEKS, CHUNKS, df_tr, piv\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FE","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"te['Percent'] = te['base_Percent']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr.shape, te.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr[\"CLUSTER\"]=tr.Patient.astype('category').cat.codes\ntr[\"wk1\"] = tr[\"Weeks\"]\ntr[\"wk2\"] = tr[\"Weeks\"] - tr[\"base_Weeks\"]\nte[\"wk1\"] = te[\"Weeks\"]\nte[\"wk2\"] = te[\"Weeks\"] - te[\"base_Weeks\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE = []\nCATCOLS = [\"Sex\",\"SmokingStatus\"]\nfor col in CATCOLS:\n    for mod in tr[col].unique():\n        FE.append(mod)\n        tr[mod] = (tr[col] == mod).astype(int)\n        te[mod] = (te[col] == mod).astype(int)\n#=================\nNUMCOLS = [\"base_Weeks\",\"base_FVC\",\"wk1\",\"wk2\",\"Age\",\"base_Percent\"] #,\"Percent\"\nFE += NUMCOLS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(FE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def metric( trueFVC, predFVC, predSTD ):\n    \n    clipSTD = np.clip( predSTD, 70 , 9e9 )  \n    \n    deltaFVC = np.clip( np.abs(trueFVC-predFVC), 0 , 1000 )  \n\n    return np.mean( -1*(np.sqrt(2)*deltaFVC/clipSTD) - np.log( np.sqrt(2)*clipSTD ) )\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import Ridge, ElasticNet\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.ensemble import RandomForestClassifier\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"C1, C2 = tf.constant(70, dtype='float32'), tf.constant(1000, dtype=\"float32\")\n#=============================#\ndef score(y_true, y_pred):\n    tf.dtypes.cast(y_true, tf.float32)\n    tf.dtypes.cast(y_pred, tf.float32)\n    sigma = y_pred[:, 2] - y_pred[:, 0]\n    fvc_pred = y_pred[:, 1]\n    \n    #sigma_clip = sigma + C1\n    sigma_clip = tf.maximum(sigma, C1)\n    delta = tf.abs(y_true[:, 0] - fvc_pred)\n    delta = tf.minimum(delta, C2)\n    sq2 = tf.sqrt( tf.dtypes.cast(2, dtype=tf.float32) )\n    metric = (delta / sigma_clip)*sq2 + tf.math.log(sigma_clip* sq2)\n    return K.mean(metric)\n#============================#\ndef qloss(y_true, y_pred):\n    # Pinball loss for multiple quantiles\n    qs = [0.2, 0.50, 0.8]\n    q = tf.constant(np.array([qs]), dtype=tf.float32)\n    e = y_true - y_pred\n    v = tf.maximum(q*e, (q-1)*e)\n    return K.mean(v)\n#=============================#\ndef mloss(_lambda):\n    def loss(y_true, y_pred):\n        return _lambda * qloss(y_true, y_pred) + (1 - _lambda)*score(y_true, y_pred)\n    return loss\n#=================\ndef make_model(nh):\n    z = L.Input((nh,), name=\"Patient\")\n    x = L.Dense(100, activation=\"relu\", name=\"d1\")(z)\n    x = L.Dense(100, activation=\"relu\", name=\"d2\")(x)\n    x = L.Dense(100, activation=\"relu\", name=\"d3\")(x)\n    p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)\n    p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n    preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), \n                     name=\"preds\")([p1, p2])\n    \n    model = M.Model(z, preds, name=\"CNN\")\n    #model.compile(loss=qloss, optimizer=\"adam\", metrics=[score])\n    model.compile(loss=mloss(0.8), optimizer=tf.keras.optimizers.Adam(lr=0.1, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.01, amsgrad=False), metrics=[score])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y = tr['FVC'].values\nz = tr[FE].values\nze = te[FE].values\ncl = tr['CLUSTER'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sc = MinMaxScaler()\nz = sc.fit_transform(z)\nze = sc.transform(ze)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NFOLD = 10\n#kf = KFold(n_splits=NFOLD)\nkf = StratifiedKFold(n_splits=NFOLD)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nnh = len(FE)\nBATCH_SIZE=500\npe = np.zeros((ze.shape[0], 3))\npred = np.zeros((z.shape[0], 3))\n\ncnt = 0\nEPOCHS = 250#500\nfor tr_idx, val_idx in kf.split(z, cl):\n    cnt += 1\n    print(f\"FOLD {cnt}\")\n    net = make_model(nh)\n    ckpt = ModelCheckpoint(\"w.h5\", monitor='val_score', verbose=0, save_best_only=True,mode='min')\n    net.fit(z[tr_idx], y[tr_idx], batch_size=BATCH_SIZE, epochs=EPOCHS, \n            validation_data=(z[val_idx], y[val_idx]), verbose=0, callbacks=[ckpt]) #\n    net = make_model(nh)\n    net.load_weights(\"w.h5\")\n    print(\"train\", net.evaluate(z[tr_idx], y[tr_idx], verbose=0, batch_size=BATCH_SIZE))\n    print(\"val\", net.evaluate(z[val_idx], y[val_idx], verbose=0, batch_size=BATCH_SIZE))\n    print(\"predict val...\")\n    pred[val_idx] = net.predict(z[val_idx], batch_size=BATCH_SIZE, verbose=0)\n    print(\"predict test...\")\n    pe += net.predict(ze, batch_size=BATCH_SIZE, verbose=0) / NFOLD\n#==============","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"oof\", metric( y, pred[:, 1], pred[:, 2] - pred[:, 0] ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sigma_opt = mean_absolute_error(y, pred[:, 1])\nunc = pred[:,2] - pred[:, 0]\nsigma_mean = np.mean(unc)\nprint(sigma_opt, sigma_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"idxs = np.random.randint(0, y.shape[0], 50)\nplt.plot(y[idxs], label=\"ground truth\")\nplt.plot(pred[idxs, 0], label=\"q25\")\nplt.plot(pred[idxs, 1], label=\"q50\")\nplt.plot(pred[idxs, 2], label=\"q75\")\nplt.legend(loc=\"best\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(unc)\nplt.title(\"uncertainty in prediction\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"te['FVC'] = pe[:, 1]\nte['Confidence'] = pe[:, 2] - pe[:, 0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm = te[['Patient_Week','FVC','Confidence']].copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"otest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\nfor i in range(len(otest)):\n    subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]\n    subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm.describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}