{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Annotation\nThis notebook branched from EfficientNets + Quantile Regression (Inference)(https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference)"},{"metadata":{},"cell_type":"markdown","source":"## Overview & Remarks\n\n- Just some experiments I did with efficientnets b0-b7 and blending predictions\n- Best LB of efficientnets was around -0.6922\n- Tried blending efficientnets b0-b7 in a single run but due to out-of-memory errors, it was not successful\n    - you may find the code to perform the mean blend here as well\n- EfficientNets are trained for 30 or 50 epochs with modified callbacks and training parameters\n- More models are being experimented currently. Will update this notebook when I have better results!"},{"metadata":{},"cell_type":"markdown","source":"## Acknowledgements\n\n- Michael Kazachok's Linear Decay (based on ResNet CNN)\n    - Model that uses images can be found at: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn\n- Ulrich GOUE's Osic-Multiple-Quantile-Regression-Starter\n    - Model that uses tabular data can be found at: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter\n- Replaced Michael's model with EfficientNets B0, B2, B4\n- I only tweaked the parameters for the models "},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install ../input/kerasapplications/keras-team-keras-applications-3b180cb -f ./ --no-index\n!pip install ../input/efficientnet/efficientnet-1.1.0/ -f ./ --no-index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport pydicom\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\n\nimport tensorflow as tf\nimport random\nfrom tqdm.notebook import tqdm # show the progress bar of process\nfrom sklearn.model_selection import train_test_split, KFold\nfrom sklearn.metrics import mean_absolute_error\nfrom tensorflow_addons.optimizers import RectifiedAdam\nfrom tensorflow.keras import Model\nimport tensorflow.keras.backend as K\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nfrom tensorflow.keras.optimizers import Nadam\n\ndef seed_everything(seed=2020):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_everything(42)\n\n# import os\n# import cv2\n# import pydicom\n# import pandas as pd\n# import numpy as np \n# import tensorflow as tf \n# import matplotlib.pyplot as plt \n# import random\n# from tqdm.notebook import tqdm \n# from sklearn.model_selection import train_test_split, KFold\n# from sklearn.metrics import mean_absolute_error\n# from tensorflow_addons.optimizers import RectifiedAdam\n# from tensorflow.keras import Model\n# import tensorflow.keras.backend as K\n# import tensorflow.keras.layers as L\n# import tensorflow.keras.models as M\n# from tensorflow.keras.optimizers import Nadam\n# import seaborn as sns\n# from PIL import Image\n\n# def seed_everything(seed=2020):\n#     random.seed(seed)\n#     os.environ['PYTHONHASHSEED'] = str(seed)\n#     np.random.seed(seed)\n#     tf.random.set_seed(seed)\n    \n# seed_everything(42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tensorflow実行時に必要な分だけGPUメモリを確保する設定\n# デフォルトだとGPUのメモリを全て使う設定になっている\n# https://qiita.com/namakemono/items/12ad8a9f6d0561929056\n\nconfig = tf.compat.v1.ConfigProto()\nconfig.gpu_options.allow_growth = True\nsession = tf.compat.v1.Session(config=config)\n\n# config = tf.compat.v1.ConfigProto()\n# config.gpu_options.allow_growth = True\n# session = tf.compat.v1.Session(config=config)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('SmokingStatus').count()['FVC']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Linear Decay (based on EfficientNets)"},{"metadata":{},"cell_type":"markdown","source":"データの整形"},{"metadata":{"trusted":true},"cell_type":"code","source":"# def get_tab(df):\n#     vector = [(df.Age.values[0] - 30) / 30] \n    \n#     if df.Sex.values[0] == 'male':\n#        vector.append(0)\n#     else:\n#        vector.append(1)\n    \n#     if df.SmokingStatus.values[0] == 'Never smoked':\n#         vector.extend([0,0])\n#     elif df.SmokingStatus.values[0] == 'Ex-smoker':\n#         vector.extend([1,1])\n#     elif df.SmokingStatus.values[0] == 'Currently smokes':\n#         vector.extend([0,1])\n#     else:\n#         vector.extend([1,0])\n#     return np.array(vector) \n\ndef get_tab(df):\n    vector = [(df.Age.values[0] - 30) / 30]\n    \n    if df.Sex.values[0] == 'male':\n        vector.append(0)\n    else:\n        vector.append(1)\n    \n    if df.SmokingStatus.values[0] == 'Never smoked':\n        vector.extend([0,0])\n    elif df.SmokingStatus.values[0] == 'Ex-smoker':\n        vector.extend([1,1])\n    elif df.SmokingStatus.values[0] == 'Currently smokes':\n        vector.extend([0,1])\n    else:\n        vector.extend([1,0])\n        \n    print(f'=====vector=====\\n {vector}')\n    \n    return np.array(vector)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['Age'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(train.Age.values[0]-30) / 30","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"A = {}\nTAB = {}\nP = []\n# i:index, p:patient_id\n# To use tqdm, we can check the process bar\nfor i, p in tqdm(enumerate(train.Patient.unique())):\n    sub = train.loc[train.Patient==p, :]\n    fvc = sub.FVC.values\n    weeks = sub.Weeks.values\n    c = np.vstack([weeks, np.ones(len(weeks))]).T # 配列を縦方向に連結\n    a, b = np.linalg.lstsq(c, fvc)[0] # lstsq：最小二乗法を計算。a：係数行列、b：独立変数\n    A[p] = a\n    TAB[p] = get_tab(sub)\n    P.append(p)\n\n# A = {} \n# TAB = {} \n# P = []\n# for i, p in tqdm(enumerate(train.Patient.unique())):\n#     sub = train.loc[train.Patient == p, :] \n#     fvc = sub.FVC.values\n#     weeks = sub.Weeks.values\n#     c = np.vstack([weeks, np.ones(len(weeks))]).T\n#     a, b = np.linalg.lstsq(c, fvc)[0]\n    \n#     A[p] = a\n#     TAB[p] = get_tab(sub)\n#     P.append(p)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CNN for coeff prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_img(path):\n    d = pydicom.dcmread(path)\n    return cv2.resize(d.pixel_array / 2**11, (512, 512))\n\n# def get_img(path):\n#     d = pydicom.dcmread(path)\n#     return cv2.resize(d.pixel_array / 2**11, (512, 512))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Under cell, the auther indicate two PatientId as BAD_ID['ID00011637202177653955184', 'ID00052637202186188008618']"},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the bat id's\ntrain[train['Patient']=='ID00011637202177653955184']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the bat id's\ntrain[train['Patient']== 'ID00052637202186188008618']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.utils import Sequence\n\nclass IGenerator(Sequence):\n    \n    # Why this author set the two 'PatientId' as BAD_ID...?\n    BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618']\n    def __init__(self, keys, a, tab, batch_size=32):\n        self.keys = [k for k in keys if k not in self.BAD_ID]\n        # 謎変数１\n        self.a = a\n        # 謎変数２\n        self.tab = tab\n        self.batch_size = batch_size\n        \n        # Get the CT picture for each number\n        # それぞれのPatientIdに対応したCT画像を取得\n        self.train_data = {}\n        for p in train.Patient.values:\n            self.train_data[p] = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/')\n    \n    def __len__(self):\n        return 1000\n    \n    def __getitem__(self, idx):\n        x = []\n        a, tab = [], []\n        # Get the PatientId's for the number of batch_size\n        # バッチサイズ分key（PatientId）を取得\n        # この辺何してんだ？\n        keys = np.random.choice(self.keys, size = self.batch_size)\n        print(f'=====keys=====\\n {keys}')\n        for k in keys:\n            try:\n                i = np.random.choice(self.keys, size = self.batch_size)[0]\n                img = get_img(f'../input/osic-pulmonary-fibrosis-progression/train/{k}/{i}')\n                x.append(img)\n                a.append(self.a[k])\n                tab.append(self.tab[k])\n            except:\n                print(k, i)\n        x, a, tab = np.array(x), np.array(a), np.array(tab)\n        # expand_dims: https://numpy.org/doc/stable/reference/generated/numpy.expand_dims.html\n        x = np.expand_dims(x, axis=-1)\n        print(f'=====expand_dims:=====\\n {x}')\n        return [x, tab], a\n\n# from tensorflow.keras.utils import Sequence\n\n# class IGenerator(Sequence):\n#     BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618']\n#     def __init__(self, keys, a, tab, batch_size=32):\n#         self.keys = [k for k in keys if k not in self.BAD_ID]\n#         self.a = a\n#         self.tab = tab\n#         self.batch_size = batch_size\n        \n#         self.train_data = {}\n#         for p in train.Patient.values:\n#             self.train_data[p] = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/')\n    \n#     def __len__(self):\n#         return 1000\n    \n#     def __getitem__(self, idx):\n#         x = []\n#         a, tab = [], [] \n#         keys = np.random.choice(self.keys, size = self.batch_size)\n#         for k in keys:\n#             try:\n#                 i = np.random.choice(self.train_data[k], size=1)[0]\n#                 img = get_img(f'../input/osic-pulmonary-fibrosis-progression/train/{k}/{i}')\n\n#                 x.append(img)\n#                 a.append(self.a[k])\n#                 tab.append(self.tab[k])\n#             except:\n#                 print(k, i)\n       \n#         x,a,tab = np.array(x), np.array(a), np.array(tab)\n#         x = np.expand_dims(x, axis=-1)\n#         return [x, tab] , a","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.layers import (\n    Dense, Dropout, Activation, Flatten, Input, BatchNormalization, GlobalAveragePooling2D, Add,\n    Conv2D, AveragePooling2D, LeakyReLU, Concatenate\n)\n# import efficientnet.tfkeras as efn\nimport efficientnet.tfkeras as efn\n\ndef get_efficientnet(model, shape):\n    models_dict = {\n        'bo': efn.EfficientNetB0(input_shape=shape, weights=None, include_top=False),\n        'b1': efn.EfficientNetB1(input_shape=shape, weights=None, include_top=False),\n        'b2': efn.EfficientNetB2(input_shape=shape, weights=None, include_top=False),\n        'b3': efn.EfficientNetB3(input_shape=shape, weights=None, include_top=False),\n        'b4': efn.EfficientNetB4(input_shape=shape, weights=None, include_top=False),\n        'b5': efn.EfficientNetB5(input_shape=shape, weights=None, include_top=False),\n        'b6': efn.EfficientNetB6(input_shape=shape, weights=None, include_top=False),\n        'b7': efn.EfficientNetB7(input_shape=shape, weights=None, include_top=False)\n    }\n    return models_dict[model]\n\ndef build_model(shape=(512, 512, 1), model_class=None):\n    # 入力層\n    inp = Input(shape=shape)\n    base = get_efficientnet(model_class, shape)\n    x = base(inp)\n    x = GlobalAveragePooling2D()(x)\n    inp2 = Input(shape=(4,))\n    x2 = tf.keras.layers.GaussianNoise(0.2)(inp2)\n    x = Concatenate()([x, x2])\n    x = Dropout(0.5)(x)\n    x = Dense(1)(x)\n    model = Model([inp, inp2], x)\n    \n    weights = [w for w in os.listdir('../input/osic-model-weights') if model_class in w][0]\n    model.load_weights('../input/osic-model-weights/' + weights)\n    return model\n\nmodel_classes = ['b5'] #['b0', 'b1', 'b2', 'b3', 'b4', 'b5', 'b6', 'b7']\nmodels = [build_model(shape=(512, 512, 1), model_class=m) for m in model_classes]\nprint(f'=====models=====\\n {models}')\nprint('Number of models: '+ str(len(models)))\n    \n\n# from tensorflow.keras.layers import (\n#     Dense, Dropout, Activation, Flatten, Input, BatchNormalization, GlobalAveragePooling2D, Add, Conv2D, AveragePooling2D, \n#     LeakyReLU, Concatenate \n# )\n# import efficientnet.tfkeras as efn\n\n# def get_efficientnet(model, shape):\n#     models_dict = {\n#         'b0': efn.EfficientNetB0(input_shape=shape,weights=None,include_top=False),\n#         'b1': efn.EfficientNetB1(input_shape=shape,weights=None,include_top=False),\n#         'b2': efn.EfficientNetB2(input_shape=shape,weights=None,include_top=False),\n#         'b3': efn.EfficientNetB3(input_shape=shape,weights=None,include_top=False),\n#         'b4': efn.EfficientNetB4(input_shape=shape,weights=None,include_top=False),\n#         'b5': efn.EfficientNetB5(input_shape=shape,weights=None,include_top=False),\n#         'b6': efn.EfficientNetB6(input_shape=shape,weights=None,include_top=False),\n#         'b7': efn.EfficientNetB7(input_shape=shape,weights=None,include_top=False)\n#     }\n#     return models_dict[model]\n\n# def build_model(shape=(512, 512, 1), model_class=None):\n#     inp = Input(shape=shape)\n#     base = get_efficientnet(model_class, shape)\n#     x = base(inp)\n#     x = GlobalAveragePooling2D()(x)\n#     inp2 = Input(shape=(4,))\n#     x2 = tf.keras.layers.GaussianNoise(0.2)(inp2)\n#     x = Concatenate()([x, x2]) \n#     x = Dropout(0.5)(x) \n#     x = Dense(1)(x)\n#     model = Model([inp, inp2] , x)\n    \n#     weights = [w for w in os.listdir('../input/osic-model-weights') if model_class in w][0]\n#     model.load_weights('../input/osic-model-weights/' + weights)\n#     return model\n\n# model_classes = ['b5'] #['b0','b1','b2','b3',b4','b5','b6','b7']\n# models = [build_model(shape=(512, 512, 1), model_class=m) for m in model_classes]\n# print('Number of models: ' + str(len(models)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntr_p, vl_p = train_test_split(P, shuffle=True, train_size=0.8)\n\n\n\n# from sklearn.model_selection import train_test_split \n\n# tr_p, vl_p = train_test_split(P, \n#                               shuffle=True, \n#                               train_size= 0.8) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"raw","source":"sns.distplot(list(A.values()));"},{"metadata":{"trusted":true},"cell_type":"code","source":"def score(fvc_true, fvc_pred, sigma):\n    sigma_clip = np.maximum(sigma, 70) # changed from 70, trie 66.7 too\n    delta = np.abs(fvc_true - fvc_pred)\n    delta = np.minimum(delta, 1000)\n    sq2 = np.sqrt(2)\n    metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2)\n    return np.mean(metric)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subs = []\nfor model in models:\n    metric = []\n    for q in tqdm(range(1, 10)):\n        m = []\n        for p in vl_p:\n            x = []\n            tab = []\n            \n            # BAD_ID再登場\n            if p in ['ID00011637202177653955184', 'ID00052637202186188008618']:\n                continue\n            \n            # フィルターをかけて画像を取得\n            ldir = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/')\n            for i in ldir:\n                print(f'=====inti[:-4]:\\n {int(i[:-4])}')\n                if int(i[:-4]) / len(ldir) < 0.8 and int(i[:-4] / len(ldir)) > 0.15:\n                    x.append(get_img(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/{i}'))\n                    tab.append(get_tab(train.loc[train.Patient==p, :]))\n            if len(x) < 1:\n                continue\n            tab = np.array(tab)\n            \n            x = np.expand_dims(x, axis=-1)\n            _a = model.predict([x, tab])\n            a = np.quantile(_a, q/10)\n            \n            percent_true = train.Percent.values[train.Patient==p]\n            fvc_true = train.FVC.values[train.Patient==p]\n            weeks_true = train.Weeks.values[train.Patient==p]\n            \n            fvc = a*(weeks_truue - weeks_true[0]) + fvc_true[0]\n            percent = percent_true[0] - a*abs(weeks_true - weeks_true[0])\n            m.append(score(fvc_true, fvc, percent))\n        print(np.mean(m))\n        metric.append(np.mean(m))\n    \n    q = (nnp.argmin(metric) + 1)/10\n    \n    sub = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/sample_submission.csv')\n    test = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n    A_test, B_test, P_test, W, FVC = {}, {}, {}, {}, {}\n    STD, WEEK = {}, {}\n    for p in test.Patient.unique():\n        x = []\n        tab = []\n        ldir = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/test/{p}/')\n        for i in ldir:\n            if int(i[:-4])/len(ldir) < 0.8 and int(i[:-4])/len(ldir) > 0.15:\n                x.append(get_img(f'../input/osic-pulmonary-fibrosis-progression/test/{p}/{i}'))\n                tab.append(get_tab(test.loc[test.Patient==p, :]))\n            \n            if len(x)<1:\n                continue\n            tab = np.array(tab)\n            \n            x = np.expand_dims(x, axis=-1)\n            _a = model.predict([x, tab])\n            a = np.quantile(_a, q)\n            \n            A_test[p] = a\n            B_test[p] = test.FVC.values[test.Patient==p] - a*test.Weeks.values[test.Patient==p]\n            P_test[p] = test.Percent.values[test.Patient==p]\n            WEEK[p] = test.Weeks.values[test.Patient==p]\n            \n        for k in sub.Patient_Week.values:\n            p, w= k.split('_')\n            w = int(w)\n            \n            fvc = A_test[p]*w + B_test[p]\n            sub.loc[sub.Patient_Week==k, 'FVC'] = fvc\n            sub.loc[sub.Patient_Week==k, 'Confidence'] = (\n                P_test[p] - A_test[p]*abs(WEEK[p]-a)\n            )\n            \n            _sub = sub[['Patient_Week', 'FVC', 'Confidence']].copy()\n            subs.append(_sub)\n            \n            \n    \n    \n# subs = []\n# for model in models:\n#     metric = []\n#     for q in tqdm(range(1, 10)):\n#         m = []\n#         for p in vl_p:\n#             x = [] \n#             tab = [] \n            \n#             # BAD_ID 再登場\n#             if p in ['ID00011637202177653955184', 'ID00052637202186188008618']:\n#                 continue\n\n#             ldir = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/')\n#             for i in ldir:\n#                 if int(i[:-4]) / len(ldir) < 0.8 and int(i[:-4]) / len(ldir) > 0.15:\n#                     x.append(get_img(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/{i}')) \n#                     tab.append(get_tab(train.loc[train.Patient == p, :])) \n            \n#             if len(x) <= 1:\n#                 continue\n#             tab = np.array(tab) \n            \n            \n\n#             x = np.expand_dims(x, axis=-1) \n#             _a = model.predict([x, tab]) \n#             a = np.quantile(_a, q / 10)\n\n#             percent_true = train.Percent.values[train.Patient == p]\n#             fvc_true = train.FVC.values[train.Patient == p]\n#             weeks_true = train.Weeks.values[train.Patient == p]\n\n#             fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0]\n#             percent = percent_true[0] - a * abs(weeks_true - weeks_true[0])\n#             m.append(score(fvc_true, fvc, percent))\n#         print(np.mean(m))\n#         metric.append(np.mean(m))\n\n#     q = (np.argmin(metric) + 1)/ 10\n\n#     sub = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/sample_submission.csv') \n#     test = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv') \n#     A_test, B_test, P_test,W, FVC= {}, {}, {},{},{} \n#     STD, WEEK = {}, {} \n#     for p in test.Patient.unique():\n#         x = [] \n#         tab = [] \n#         ldir = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/test/{p}/')\n#         for i in ldir:\n#             if int(i[:-4]) / len(ldir) < 0.8 and int(i[:-4]) / len(ldir) > 0.15:\n#                 x.append(get_img(f'../input/osic-pulmonary-fibrosis-progression/test/{p}/{i}')) \n#                 tab.append(get_tab(test.loc[test.Patient == p, :])) \n#         if len(x) <= 1:\n#             continue\n#         tab = np.array(tab) \n\n#         x = np.expand_dims(x, axis=-1) \n#         _a = model.predict([x, tab]) \n#         a = np.quantile(_a, q)\n#         A_test[p] = a\n#         B_test[p] = test.FVC.values[test.Patient == p] - a*test.Weeks.values[test.Patient == p]\n#         P_test[p] = test.Percent.values[test.Patient == p] \n#         WEEK[p] = test.Weeks.values[test.Patient == p]\n\n#     for k in sub.Patient_Week.values:\n#         p, w = k.split('_')\n#         w = int(w) \n\n#         fvc = A_test[p] * w + B_test[p]\n#         sub.loc[sub.Patient_Week == k, 'FVC'] = fvc\n#         sub.loc[sub.Patient_Week == k, 'Confidence'] = (\n#             P_test[p] - A_test[p] * abs(WEEK[p] - w) \n#     ) \n\n#     _sub = sub[[\"Patient_Week\",\"FVC\",\"Confidence\"]].copy()\n#     subs.append(_sub)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Averaging Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"N = len(subs)\nsub = subs[0].copy() # ref\nsub[\"FVC\"] = 0\nsub[\"Confidence\"] = 0\nfor i in range(N):\n    sub[\"FVC\"] += subs[0][\"FVC\"] * (1/N)\n    sub[\"Confidence\"] += subs[0][\"Confidence\"] * (1/N)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sub[[\"Patient_Week\",\"FVC\",\"Confidence\"]].to_csv(\"submission_img.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"img_sub = sub[[\"Patient_Week\",\"FVC\",\"Confidence\"]].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Osic-Multiple-Quantile-Regression"},{"metadata":{"trusted":false},"cell_type":"code","source":"ROOT = \"../input/osic-pulmonary-fibrosis-progression\"\nBATCH_SIZE=128\n\ntr = pd.read_csv(f\"{ROOT}/train.csv\")\ntr.drop_duplicates(keep=False, inplace=True, subset=['Patient','Weeks'])\nchunk = pd.read_csv(f\"{ROOT}/test.csv\")\n\nprint(\"add infos\")\nsub = pd.read_csv(f\"{ROOT}/sample_submission.csv\")\nsub['Patient'] = sub['Patient_Week'].apply(lambda x:x.split('_')[0])\nsub['Weeks'] = sub['Patient_Week'].apply(lambda x: int(x.split('_')[-1]))\nsub =  sub[['Patient','Weeks','Confidence','Patient_Week']]\nsub = sub.merge(chunk.drop('Weeks', axis=1), on=\"Patient\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"tr['WHERE'] = 'train'\nchunk['WHERE'] = 'val'\nsub['WHERE'] = 'test'\ndata = tr.append([chunk, sub])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(tr.shape, chunk.shape, sub.shape, data.shape)\nprint(tr.Patient.nunique(), chunk.Patient.nunique(), sub.Patient.nunique(), \n      data.Patient.nunique())\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"data['min_week'] = data['Weeks']\ndata.loc[data.WHERE=='test','min_week'] = np.nan\ndata['min_week'] = data.groupby('Patient')['min_week'].transform('min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"base = data.loc[data.Weeks == data.min_week]\nbase = base[['Patient','FVC']].copy()\nbase.columns = ['Patient','min_FVC']\nbase['nb'] = 1\nbase['nb'] = base.groupby('Patient')['nb'].transform('cumsum')\nbase = base[base.nb==1]\nbase.drop('nb', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"data = data.merge(base, on='Patient', how='left')\ndata['base_week'] = data['Weeks'] - data['min_week']\ndel base","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"COLS = ['Sex','SmokingStatus'] #,'Age'\nFE = []\nfor col in COLS:\n    for mod in data[col].unique():\n        FE.append(mod)\n        data[mod] = (data[col] == mod).astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#\ndata['age'] = (data['Age'] - data['Age'].min() ) / ( data['Age'].max() - data['Age'].min() )\ndata['BASE'] = (data['min_FVC'] - data['min_FVC'].min() ) / ( data['min_FVC'].max() - data['min_FVC'].min() )\ndata['week'] = (data['base_week'] - data['base_week'].min() ) / ( data['base_week'].max() - data['base_week'].min() )\ndata['percent'] = (data['Percent'] - data['Percent'].min() ) / ( data['Percent'].max() - data['Percent'].min() )\nFE += ['age','percent','week','BASE']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"tr = data.loc[data.WHERE=='train']\nchunk = data.loc[data.WHERE=='val']\nsub = data.loc[data.WHERE=='test']\ndel data","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"tr.shape, chunk.shape, sub.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"C1, C2 = tf.constant(70, dtype='float32'), tf.constant(1000, dtype=\"float32\")\n\ndef score(y_true, y_pred):\n    tf.dtypes.cast(y_true, tf.float32)\n    tf.dtypes.cast(y_pred, tf.float32)\n    sigma = y_pred[:, 2] - y_pred[:, 0]\n    fvc_pred = y_pred[:, 1]\n    \n    #sigma_clip = sigma + C1\n    sigma_clip = tf.maximum(sigma, C1)\n    delta = tf.abs(y_true[:, 0] - fvc_pred)\n    delta = tf.minimum(delta, C2)\n    sq2 = tf.sqrt( tf.dtypes.cast(2, dtype=tf.float32) )\n    metric = (delta / sigma_clip)*sq2 + tf.math.log(sigma_clip* sq2)\n    return K.mean(metric)\n\ndef qloss(y_true, y_pred):\n    # Pinball loss for multiple quantiles\n    qs = [0.2, 0.50, 0.8]\n    q = tf.constant(np.array([qs]), dtype=tf.float32)\n    e = y_true - y_pred\n    v = tf.maximum(q*e, (q-1)*e)\n    return K.mean(v)\n\ndef mloss(_lambda):\n    def loss(y_true, y_pred):\n        return _lambda * qloss(y_true, y_pred) + (1 - _lambda)*score(y_true, y_pred)\n    return loss\n\ndef make_model(nh):\n    z = L.Input((nh,), name=\"Patient\")\n    x = L.Dense(100, activation=\"relu\", name=\"d1\")(z)\n    x = L.Dense(100, activation=\"relu\", name=\"d2\")(x)\n    #x = L.Dense(100, activation=\"relu\", name=\"d3\")(x)\n    p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)\n    p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n    preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), \n                     name=\"preds\")([p1, p2])\n    \n    model = M.Model(z, preds, name=\"CNN\")\n    #model.compile(loss=qloss, optimizer=\"adam\", metrics=[score])\n    model.compile(loss=mloss(0.8), optimizer=tf.keras.optimizers.Adam(lr=0.1, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.01, amsgrad=False), metrics=[score])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"y = tr['FVC'].values\nz = tr[FE].values\nze = sub[FE].values\nnh = z.shape[1]\npe = np.zeros((ze.shape[0], 3))\npred = np.zeros((z.shape[0], 3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"net = make_model(nh)\nprint(net.summary())\nprint(net.count_params())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"NFOLD = 5 # originally 5\nkf = KFold(n_splits=NFOLD)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\ncnt = 0\nEPOCHS = 800\nfor tr_idx, val_idx in kf.split(z):\n    cnt += 1\n    print(f\"FOLD {cnt}\")\n    net = make_model(nh)\n    net.fit(z[tr_idx], y[tr_idx], batch_size=BATCH_SIZE, epochs=EPOCHS, \n            validation_data=(z[val_idx], y[val_idx]), verbose=0) #\n    print(\"train\", net.evaluate(z[tr_idx], y[tr_idx], verbose=0, batch_size=BATCH_SIZE))\n    print(\"val\", net.evaluate(z[val_idx], y[val_idx], verbose=0, batch_size=BATCH_SIZE))\n    print(\"predict val...\")\n    pred[val_idx] = net.predict(z[val_idx], batch_size=BATCH_SIZE, verbose=0)\n    print(\"predict test...\")\n    pe += net.predict(ze, batch_size=BATCH_SIZE, verbose=0) / NFOLD","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sigma_opt = mean_absolute_error(y, pred[:, 1])\nunc = pred[:,2] - pred[:, 0]\nsigma_mean = np.mean(unc)\nprint(sigma_opt, sigma_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"idxs = np.random.randint(0, y.shape[0], 100)\nplt.plot(y[idxs], label=\"ground truth\")\nplt.plot(pred[idxs, 0], label=\"q25\")\nplt.plot(pred[idxs, 1], label=\"q50\")\nplt.plot(pred[idxs, 2], label=\"q75\")\nplt.legend(loc=\"best\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(unc.min(), unc.mean(), unc.max(), (unc>=0).mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.hist(unc)\nplt.title(\"uncertainty in prediction\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# PREDICTION\nsub['FVC1'] = 1.*pe[:, 1]\nsub['Confidence1'] = pe[:, 2] - pe[:, 0]\nsubm = sub[['Patient_Week','FVC','Confidence','FVC1','Confidence1']].copy()\nsubm.loc[~subm.FVC1.isnull()].head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"subm.loc[~subm.FVC1.isnull(),'FVC'] = subm.loc[~subm.FVC1.isnull(),'FVC1']\nif sigma_mean<70:\n    subm['Confidence'] = sigma_opt\nelse:\n    subm.loc[~subm.FVC1.isnull(),'Confidence'] = subm.loc[~subm.FVC1.isnull(),'Confidence1']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"subm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"subm.describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"otest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\nfor i in range(len(otest)):\n    subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]\n    subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"subm[[\"Patient_Week\",\"FVC\",\"Confidence\"]].to_csv(\"submission_regression.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"reg_sub = subm[[\"Patient_Week\",\"FVC\",\"Confidence\"]].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble (Simple Blend)"},{"metadata":{"trusted":false},"cell_type":"code","source":"df1 = img_sub.sort_values(by=['Patient_Week'], ascending=True).reset_index(drop=True)\ndf2 = reg_sub.sort_values(by=['Patient_Week'], ascending=True).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df = df1[['Patient_Week']].copy()\ndf['FVC'] = 0.25*df1['FVC'] + 0.75*df2['FVC']\ndf['Confidence'] = 0.25*df1['Confidence'] + 0.75*df2['Confidence']\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}