{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview & Remarks\n\nDirect fork from here: https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference\n\n**Diff: \"Cleaned up\" code for my own understanding**\n\n- Just some experiments I did with efficientnets b0-b7 and blending predictions\n- Best LB of efficientnets was around -0.6922\n- Tried blending efficientnets b0-b7 in a single run but due to out-of-memory errors, it was not successful\n    - you may find the code to perform the mean blend here as well\n- EfficientNets are trained for 30 or 50 epochs with modified callbacks and training parameters\n- More models are being experimented currently. Will update this notebook when I have better results!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\n- Michael Kazachok's Linear Decay (based on ResNet CNN)\n    - Model that uses images can be found at: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn\n- Ulrich GOUE's Osic-Multiple-Quantile-Regression-Starter\n    - Model that uses tabular data can be found at: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter\n- Replaced Michael's model with EfficientNets B0, B2, B4\n- I only tweaked the parameters for the models ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install ../input/kerasapplications/keras-team-keras-applications-3b180cb -f ./ --no-index\n!pip install ../input/efficientnet/efficientnet-1.1.0/ -f ./ --no-index","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport datetime\nfrom functools import lru_cache\nimport cv2\nimport pydicom\nimport pandas as pd\nimport numpy as np \nimport tensorflow as tf \nimport matplotlib.pyplot as plt \nimport random\nfrom tqdm.notebook import tqdm\nfrom sklearn.model_selection import train_test_split, KFold\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.preprocessing import MinMaxScaler\nfrom tensorflow_addons.optimizers import RectifiedAdam\nfrom tensorflow.keras import Model\nimport tensorflow.keras.backend as K\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nfrom tensorflow.keras.optimizers import Nadam\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\nfrom colorama import Fore, Back, Style\n\ndef seed_everything(seed=2020):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_everything(42)\nROOT = \"../input/osic-pulmonary-fibrosis-progression/\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"config = tf.compat.v1.ConfigProto()\nconfig.gpu_options.allow_growth = True\nsession = tf.compat.v1.Session(config=config)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(os.path.join(ROOT, 'train.csv'))\ntest = pd.read_csv(os.path.join(ROOT, 'test.csv'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Linear Decay (based on EfficientNets)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_agss_vector(df):\n    \n    \"\"\"agss = age, gender, smokingstatus\"\"\"\n    \n    normalized_age = [(df.Age.values[0] - 30) / 30] \n\n    gender = [0 if df.Sex.values[0] == 'male' else 1]\n    \n    if df.SmokingStatus.values[0] == 'Never smoked':\n        smoking_status = [0, 0]\n    elif df.SmokingStatus.values[0] == 'Ex-smoker':\n        smoking_status = [1, 1]\n    elif df.SmokingStatus.values[0] == 'Currently smokes':\n        smoking_status = [0, 1]\n    else:\n        smoking_status = [1, 0]\n\n    vector = normalized_age + gender + smoking_status\n    return np.array(vector)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sample_best_fit_line_weeks_vs_fvc():\n    \n    patient = train.Patient.sample().iloc[0]\n    sub = train.loc[train.Patient == patient, :]\n    fvc = sub.FVC.values\n    weeks = sub.Weeks.values\n    vals = np.c_[weeks, np.ones(len(weeks))]  # column-wise stack\n    \n    # see example https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html\n    m, c = np.linalg.lstsq(vals, fvc, rcond=-1)[0]\n    \n    print(f\"Patient number: {patient}\")\n    print(\"FVC\", fvc)\n    print(\"Weeks\", weeks)\n    print(vals)\n    print(f\"gradient: {m:.2f}\\nintercept: {c:.2f}\")\n    print()\n    _ = plt.plot(weeks, fvc, 'o', label='Original data', markersize=10)\n    _ = plt.plot(weeks, m * weeks + c, 'r', label='Fitted line')\n    _ = plt.legend()\n    _ = plt.xlabel(\"Weeks\"), plt.ylabel(\"FVC\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# gradient = rate of decay in FVC values\nsample_best_fit_line_weeks_vs_fvc()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gradients = {} \nagss_vectors = {} \npatients = []\n\nfor i, patient_id in enumerate(train.Patient.unique()):\n    sub = train.loc[train.Patient == patient_id, :] \n    fvc = sub.FVC.values\n    weeks = sub.Weeks.values\n    c = np.c_[weeks, np.ones(len(weeks))]\n    gradient, intercept = np.linalg.lstsq(c, fvc, rcond=-1)[0]\n    \n    gradients[patient_id] = gradient\n    agss_vectors[patient_id] = get_agss_vector(sub)\n    patients.append(patient_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CNN for coeff prediction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_img(path):\n    d = pydicom.dcmread(path)\n    return cv2.resize(d.pixel_array / 2**11, (512, 512))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# sample\n_ = plt.imshow(get_img(os.path.join(ROOT, \"train\", \"ID00007637202177411956430\", \"1.dcm\")))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.layers import (\n    Input,\n    Activation,\n    LeakyReLU,\n    Dropout,\n    BatchNormalization,\n    Dense,\n    Conv2D, \n    AveragePooling2D,\n    GlobalAveragePooling2D,\n    Add,\n    Flatten,\n    Concatenate,\n)\nimport efficientnet.tfkeras as efn\n\ndef get_efficientnet(model, shape):\n    models_dict = {\n        'b0': efn.EfficientNetB0(input_shape=shape, weights=None, include_top=False),\n        'b1': efn.EfficientNetB1(input_shape=shape, weights=None, include_top=False),\n        'b2': efn.EfficientNetB2(input_shape=shape, weights=None, include_top=False),\n        'b3': efn.EfficientNetB3(input_shape=shape, weights=None, include_top=False),\n        'b4': efn.EfficientNetB4(input_shape=shape, weights=None, include_top=False),\n        'b5': efn.EfficientNetB5(input_shape=shape, weights=None, include_top=False),\n        'b6': efn.EfficientNetB6(input_shape=shape, weights=None, include_top=False),\n        'b7': efn.EfficientNetB7(input_shape=shape, weights=None, include_top=False),\n    }\n    return models_dict[model]\n\ndef build_model(shape=(512, 512, 1), model_class=None):\n    \n    img_inp = Input(shape=shape, name=\"image_input\")\n    base = get_efficientnet(model_class, shape)\n    x = base(img_inp)\n    img_outp = GlobalAveragePooling2D()(x)\n    \n    # AGSS = Age + Gender + SmokingStatus\n    agss_inp = Input(shape=(4,), name=\"age_gender_smokingsstatus_input\")\n    agss_outp = tf.keras.layers.GaussianNoise(0.2)(agss_inp)\n    \n    x = Concatenate()([img_outp, agss_outp]) \n    x = Dropout(0.5)(x) \n    output = Dense(1)(x)\n    \n    model = Model([img_inp, agss_inp] , output)\n    weights = [w for w in os.listdir('../input/osic-model-weights') if model_class in w]\n    assert len(weights) == 1, \"More than one model weights match the 'model_class' substring\"\n    model.load_weights('../input/osic-model-weights/' + weights[0])\n    \n    return model\n\nmodel_classes = ['b5']  # ['b0','b1','b2','b3',b4','b5','b6','b7']\nmodels = [build_model(shape=(512, 512, 1), model_class=m) for m in model_classes]\nprint('Number of models: ' + str(len(models)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"models[0].summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.utils.plot_model(\n    models[0], \n    to_file='model.png',\n    show_shapes=False, \n    show_layer_names=True,\n    rankdir='TB',\n    expand_nested=False, \n    dpi=120,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patients, validation_patients = train_test_split(patients, shuffle=True, train_size=0.8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(list(gradients.values()));","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DFs = {\n    \"train\": train,\n    \"test\": test,\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fetch_images(patient_id, root=ROOT):\n    image_files = os.listdir(os.path.join(root, f'train/{patient_id}/'))\n    images = read_images_in_middle_of_scan(image_files, patient_id)\n    return images\n\ndef read_images_in_middle_of_scan(image_files, patient_id, lower=0.15, upper=0.8):\n    images = []\n    for filename in image_files:\n        file_no, _ = os.path.splitext(filename) # cut out '.dcm' file extension\n        file_no = int(file_no)\n        is_img_slice_in_middle = lower < file_no / len(image_files) < upper\n        if is_img_slice_in_middle:\n            image_filepath = os.path.join(ROOT, f'train/{patient_id}/{filename}')\n            images.append(get_img(image_filepath))\n    return images\n\ndef create_agss_vec_mat(patient_df, num_rows):\n    agss_vector = get_agss_vector(patient_df)\n    agss_matrix = np.array([agss_vector] * num_rows)\n    return agss_vector, agss_matrix\n\ndef filter_df_with_patient_id(df, patient_id, patient_col=\"Patient\"):\n    return df.loc[df[patient_col] == patient_id, :]\n\ndef pred_fvc(x, m, c):\n    \"\"\"\n    x --> weeks from base week\n    m --> gradient i.e. rate of FVC decay (would be -ve for a patient with disease)\n    c --> base week FVC\n    \"\"\"\n    return m * x + c\n\ndef pred_confidence(base_percent, m, gap_in_weeks):\n    \"\"\"\n    Predict confidence AKA \"std deviation\". Lower val means high confidence in predicted FVC.\n    base_percent --> percentage in the base week\n    m --> gradient i.e. rate of FVC decay (would be -ve for a patient with disease)\n    gap_in_weeks --> just the gap irrespective of whether in the past or future\n    \"\"\"\n    \n    # the formula takes into account that as prediction moves away from the base week,\n    # confidence drops (value gets bigger since m is or would be for most -ve)\n    return base_percent - m * abs(gap_in_weeks)\n\ndef score(fvc_true, fvc_pred, sigma):\n    sigma_clip = np.maximum(sigma, 70) # changed from 70, trie 66.7 too\n    delta = np.abs(fvc_true - fvc_pred)\n    delta = np.minimum(delta, 1000)\n    sq2 = np.sqrt(2)\n    metric = (delta / sigma_clip) * sq2 + np.log(sigma_clip * sq2)\n    return np.mean(metric)\n\n@lru_cache(1000)\ndef make_model_pred(df_name, patient_id, model_idx):\n    global DFs\n    df = DFs[df_name]\n    patient_df = df[df.Patient == patient_id]\n    images = fetch_images(patient_id)\n    images = np.expand_dims(images, axis=-1)\n    agss_vector, agss_matrix = create_agss_vec_mat(patient_df, num_rows=images.shape[0])\n    return models[model_idx].predict([images, agss_matrix])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calc_patient_score(df_name, patient_id, quantile, model_idx, return_extra_vals=False):\n    global DFs\n    df = DFs[df_name]\n    patient_df = df[df.Patient == patient_id]\n    assert not patient_df.empty\n    \n    # model predicts for each image + agss_vector input\n    gradients = make_model_pred(df_name, patient_id, model_idx)\n    \n    if gradients is None:\n        return  # if no valid images in range, it will be None\n    gradient = np.quantile(gradients, quantile)  # gradient @ quantile from gradients\n\n    percent_true = patient_df.Percent.values\n    fvc_true = patient_df.FVC.values\n    weeks_true = patient_df.Weeks.values\n    base_week = base_weeks_test[patient_id]\n\n    predicted_fvc = pred_fvc(x=(weeks_true - weeks_true[0]), \n                             m=gradient, \n                             c=fvc_true[0],\n                            )\n    predicted_confidence = pred_confidence(base_percent=percent_true[0], \n                                 m=gradient, \n                                 gap_in_weeks=(weeks_true - weeks_true[0]),\n                                )\n    patient_score = score(fvc_true, predicted_fvc, predicted_confidence)\n    if not return_extra_vals:\n        return patient_score\n    else:\n        return patient_score, gradient, fvc_predict, confidence","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subs = []\nstart = datetime.datetime.now()\nfor model_idx in range(len(models)):\n    quantile_means = []\n    quantiles = np.arange(0.1, 1.0, 0.05)\n    for quantile in quantiles:\n        \n        print(f\"Quantile: {quantile:.2f}\", end=\" -->  \")\n        patient_scores_per_quantile = []\n        \n        for patient_id in validation_patients:\n            if patient_id in ['ID00011637202177653955184', 'ID00052637202186188008618']:\n                continue\n            one_patient_score_per_quantile = calc_patient_score(\"train\", \n                                                                patient_id, \n                                                                quantile, model_idx,\n                                                               )\n            if one_patient_score_per_quantile is not None:\n                patient_scores_per_quantile.append(one_patient_score_per_quantile)\n\n        mean_quantile_score = np.mean(patient_scores_per_quantile)\n        print(f\"Patient scores mean for quantile {quantile:.2f}: {mean_quantile_score:.4f}\")\n        quantile_means.append(mean_quantile_score)\n\n    sub = pd.read_csv(os.path.join(ROOT,'sample_submission.csv'))\n    test = pd.read_csv(os.path.join(ROOT,'test.csv'))\n\n    ## quantile with the smallest mean -> smallest error\n    lowest_quantile_mean_idx = np.argmin(quantile_means)\n    lowest_quantile = (lowest_quantile_mean_idx + 1) / 10\n\n    gradient_test, calc_fvc_base_test, percent_test, base_weeks_test = {}, {}, {}, {}\n    \n    # this loop defines base parameters for each patient needed to calculate week-by-week prediction\n    for patient_id in test.Patient.unique():\n        _, gradient, *_ = calc_patient_score(\"test\", \n                                             patient_id, \n                                             lowest_quantile,\n                                             model_idx,\n                                             return_extra_vals=True,\n                                            )  # only gradient needed\n        patient_df = test[test.Patient == patient_id]\n        \n        # test assumption: df will have 1 row since test set\n        assert patient_df.shape[0] == 1\n        \n        gradient_test[patient_id] = gradient  # prediction value of the model\n        \n        # pred of FVC at week 0 itself. Other weeks will be predicted using this as base\n        calc_fvc_base_test[patient_id] = (patient_df.FVC.values - \n                                          gradient * patient_df.Weeks).values[0]  \n\n        percent_test[patient_id] = patient_df.Percent.values[0]\n        base_weeks_test[patient_id] = patient_df.Weeks.values[0]\n\n    # this loop predicts values (FVC and confidence) for each patient's each week\n    for k in sub.Patient_Week.values:\n        \n        patient_id, week_no = k.split('_')\n        week_no = int(week_no)\n        \n        gradient = gradient_test[patient_id]\n        base_fvc = calc_fvc_base_test[patient_id]\n        base_percent = percent_test[patient_id]\n        base_week = base_weeks_test[patient_id]\n        gap_from_base_week = base_week - week_no\n        \n        predicted_fvc = pred_fvc(week_no, m=gradient, c=base_fvc)\n        predicted_conf = pred_confidence(base_percent,\n                                         m=gradient,\n                                         gap_in_weeks=gap_from_base_week,\n                                        )\n        \n        sub.loc[sub.Patient_Week==k, 'FVC'] = predicted_fvc\n        sub.loc[sub.Patient_Week==k, 'Confidence'] = predicted_conf\n    \n    sub_ = sub[[\"Patient_Week\", \"FVC\", \"Confidence\"]].copy()\n    subs.append(sub_)\nend = datetime.datetime.now()\nprint(end - start)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Averaging Predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"N = len(subs)\nsub = subs[0].copy() # ref\nsub[\"FVC\"] = 0\nsub[\"Confidence\"] = 0\nfor i in range(N):\n    sub[\"FVC\"] += subs[0][\"FVC\"] * (1/N)\n    sub[\"Confidence\"] += subs[0][\"Confidence\"] * (1/N)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_sub = sub[[\"Patient_Week\",\"FVC\",\"Confidence\"]].copy()\nimg_sub.to_csv(\"submission_img.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Osic-Multiple-Quantile-Regression","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 128\n\ntr = pd.read_csv(f\"{ROOT}/train.csv\")\ntr.drop_duplicates(keep=False, inplace=True, subset=['Patient', 'Weeks'])\nchunk = pd.read_csv(f\"{ROOT}/test.csv\")\nsub = pd.read_csv(f\"{ROOT}/sample_submission.csv\")\n\nsub[['Patient', 'Weeks']] = sub['Patient_Week'].str.split(\"_\", expand=True)\nsub =  sub[['Patient','Weeks','Confidence','Patient_Week']]\nsub = sub.merge(chunk.drop('Weeks', axis=1), on=\"Patient\")\n\ntr['WHERE'] = 'train'\nchunk['WHERE'] = 'val'\nsub['WHERE'] = 'test'\ndata = tr.append([chunk, sub])\n\nnames = [\"train\", \"val\", \"test\", \"combined\"]\nfor i, df in enumerate([tr, chunk, sub, data]):\n    df_shape_in_blue = (Fore.BLUE, df.shape, Style.RESET_ALL)\n    uniq_p_in_green = (Fore.GREEN, df.Patient.nunique(), Style.RESET_ALL)\n    print(names[i], \"-> shape\", *df_shape_in_blue, \" -> unique patients\", *uniq_p_in_green)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# add minimum week for all patients. The actual one. Submission (called test here)\n# contains all possible weeks. But that is just necessary for predictions.\n# The actual one is the no. of weeks before/after the CT-Scan, the patient went for FVC measurement\ndata['min_week'] = data['Weeks']\ndata.loc[data.WHERE=='test', 'min_week'] = np.nan\ndata['min_week'] = data.groupby('Patient')['min_week'].transform('min')\n\nbase = data.loc[data.Weeks == data.min_week, ['Patient','FVC']].drop_duplicates()\ndata = data.merge(base, how=\"left\", on=\"Patient\")\ndata.rename({\"FVC_x\": \"FVC\", \"FVC_y\": \"min_FVC\"}, axis=1, inplace=True)\ndata['from_base_week'] = data['Weeks'].astype(int) - data['min_week']\ndel base\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.get_dummies(data, columns=[\"Sex\", \"SmokingStatus\"], prefix=\"\", prefix_sep=\"\", )\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def min_max_scaler(df, col):\n    scaler = MinMaxScaler()\n    col_matrix = np.expand_dims(df.loc[:, col].values, axis=-1)  # sklearn-requirement\n    scaler.fit(col_matrix)\n    return scaler.transform(col_matrix).squeeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['age'] = min_max_scaler(data, \"Age\")\ndata['BASE'] = min_max_scaler(data, \"min_FVC\")\ndata['week'] = min_max_scaler(data, \"from_base_week\")\ndata['percent'] = min_max_scaler(data, \"Percent\")\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr = data.loc[data.WHERE=='train']\nchunk = data.loc[data.WHERE=='val']\nsub = data.loc[data.WHERE=='test']\ndel data\n\ntr.shape, chunk.shape, sub.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"SIGMA_LOWER_LIMIT = tf.constant(70, dtype='float32')\nMAX_ABS_ERROR = tf.constant(1000, dtype=\"float32\")\nQUANTILES = tf.constant(np.array([[0.2, 0.5, 0.8]]), dtype=tf.float32)\n\ndef score(y_true, y_pred):\n    tf.dtypes.cast(y_true, tf.float32)\n    tf.dtypes.cast(y_pred, tf.float32)\n    sigma = y_pred[:, 2] - y_pred[:, 0]\n    fvc_pred = y_pred[:, 1]\n    \n    sigma_clip = tf.maximum(sigma, SIGMA_LOWER_LIMIT)\n    delta = tf.abs(y_true[:, 0] - fvc_pred)\n    delta = tf.minimum(delta, MAX_ABS_ERROR)\n    sq2 = tf.sqrt( tf.dtypes.cast(2, dtype=tf.float32))\n    \n    metric = (delta / sigma_clip) * sq2 + tf.math.log(sigma_clip * sq2)\n    return K.mean(metric)\n\n# The pinball loss function is a metric used to assess the accuracy of a quantile forecast. \n\ndef quantile_loss(y_true, y_pred):\n    # Pinball loss for multiple quantiles\n    e = y_true - y_pred\n    v = tf.maximum(QUANTILES * e, (QUANTILES - 1) * e)\n    return K.mean(v)\n\ndef mloss(_lambda):\n    def loss(y_true, y_pred):\n        return _lambda * quantile_loss(y_true, y_pred) + (1 - _lambda) * score(y_true, y_pred)\n    return loss\n\ndef make_model(nh):\n    z = L.Input((nh,), name=\"Patient\")\n    x = L.Dense(100, activation=\"relu\", name=\"d1\")(z)\n    x = L.Dense(100, activation=\"relu\", name=\"d2\")(x)\n    p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)  # 2 different activations\n    p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n    \n    # lambda layer takes in this case one input x (a list of outputs [p1, p2])\n    # keep output of linear activation (p1 i.e. x[0]) \n    # add cumsum of relu activation (p2) to p1 --> axis=1 means add horizontally (values of same sample)\n    preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), name=\"preds\")([p1, p2])\n    \n    model = M.Model(z, preds, name=\"CNN\")\n    model.compile(loss=mloss(0.8), \n                  optimizer=tf.keras.optimizers.Adam(lr=0.1, \n                                                     beta_1=0.9,\n                                                     beta_2=0.999, \n                                                     epsilon=None, \n                                                     decay=0.01, \n                                                     amsgrad=False,\n                                                    ), \n                  metrics=[score],\n                 )\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FEATURE_COLS = tr.columns[tr.columns.get_loc(\"from_base_week\") + 1:].tolist()\nFEATURE_COLS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = tr['FVC'].astype(np.float32).values\nX = tr[FEATURE_COLS].values\ntest = sub[FEATURE_COLS].values\nnum_features = X.shape[1]\npred_test = np.zeros((test.shape[0], 3))\npred_val = np.zeros((X.shape[0], 3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net = make_model(num_features)\nnet.summary()  # each input datapoint will have 3 output values (3 quantiles)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.utils.plot_model(\n    net, \n    to_file='model.png',\n    show_shapes=False, \n    show_layer_names=True,\n    rankdir='TB',\n    expand_nested=False, \n    dpi=120,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NFOLD = 5\nkf = KFold(n_splits=NFOLD)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nEPOCHS = 800\n\nfor fold_no, (tr_idx, val_idx) in enumerate(kf.split(X), start=1):\n    \n    print(f\"FOLD {fold_no}\")\n    \n    net = make_model(num_features)\n    \n    X_train, y_train = X[tr_idx], y[tr_idx]\n    X_val, y_val = X[val_idx], y[val_idx]\n    \n    net.fit(x=X_train, \n            y=y_train, \n            batch_size=BATCH_SIZE, \n            epochs=EPOCHS, \n            validation_data=(X_val, y_val),\n            verbose=0,\n           )\n    \n    print(\"train\", net.evaluate(X_train, y_train, verbose=0, batch_size=BATCH_SIZE))\n    print(\"val\", net.evaluate(X_val, y_val, verbose=0, batch_size=BATCH_SIZE))\n    \n    print(\"predict val...\")\n    pred_val[val_idx] = net.predict(X_val, batch_size=BATCH_SIZE, verbose=0)\n    \n    print(\"predict test...\")\n    fold_prediction = net.predict(test, batch_size=BATCH_SIZE, verbose=0)\n    fold_prediction_normalized = fold_prediction / NFOLD\n    pred_test += fold_prediction_normalized","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# prediction for each data point consists of 3 values (i.e. 3 quartiles)\n\nsigma_opt = mean_absolute_error(y, pred_val[:, 1])\nunc = pred_val[:, 2] - pred_val[:, 0]\nsigma_mean = np.mean(unc)\nprint(sigma_opt, sigma_mean)\n\nidxs = np.random.randint(0, y.shape[0], 100)\nplt.figure(figsize=(10, 8))\nplt.plot(y[idxs], label=\"ground truth\")\nplt.plot(pred_val[idxs, 0], label=\"q25\")\nplt.plot(pred_val[idxs, 1], label=\"q50\")\nplt.plot(pred_val[idxs, 2], label=\"q75\")\nplt.legend(loc=\"best\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(unc.min(), unc.mean(), unc.max(), (unc>=0).mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(unc)\nplt.title(\"uncertainty in prediction\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREDICTION\nsub['FVC1'] = 1. * pred_test[:, 1]\nsub['Confidence1'] = pred_test[:, 2] - pred_test[:, 0]\nsubm = sub[['Patient_Week', 'FVC', 'Confidence', 'FVC1', 'Confidence1']].copy()\nassert subm.FVC1.isna().sum() == 0\nsubm.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm.loc[:, 'FVC'] = subm.loc[:, 'FVC1']\nif sigma_mean < 70:\n    subm['Confidence'] = sigma_opt\nelse:\n    subm.loc[:, 'Confidence'] = subm.loc[:,'Confidence1']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm.describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"otest = pd.read_csv(os.path.join(ROOT, 'test.csv'))\n\nfor i in range(len(otest)):\n    \n    patient_week = otest.Patient[i] + '_' + str(otest.Weeks[i])\n    is_patient_week_row = subm['Patient_Week'] == patient_week\n    \n    subm.loc[is_patient_week_row, 'FVC'] = otest.FVC[i]\n    subm.loc[is_patient_week_row, 'Confidence'] = 0.1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reg_sub = subm[[\"Patient_Week\",\"FVC\",\"Confidence\"]].copy()\nreg_sub.to_csv(\"submission_regression.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble (Simple Blend)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df1 = img_sub.sort_values(by=['Patient_Week'], ascending=True).reset_index(drop=True)\ndf2 = reg_sub.sort_values(by=['Patient_Week'], ascending=True).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df1[['Patient_Week']].copy()\ndf['FVC'] = 0.2666 * df1['FVC'] + 0.2444 * df2['FVC']\ndf['Confidence'] = 0.2666 * df1['Confidence'] + 0.7444 * df2['Confidence']\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}