{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Probabilistic Machine Learning: A Different Approach\nI'm very grateful to have entered in this competition, as it drove me to learn new things I wouldn't otherwise.\nAs I wrote in other notebooks, my first objective was to learn how to implement AutoEncoders, and discover why generative models are so hype.\nAfter reading papers, tutorials, experimenting, analyzing the (poor) results achieved, I started asking questions.\nWhy isn't the model learning good representation of the latent space? Btw, what the heck is a latent space? Why are generative models so cool? How do they actually work?\n\nThe search for these (and many more answers) drove me to Variational AutoEncoders (which I will share my implementation here later), but most importantly, drove me to [Probabilistic Machine Learning](https://www.nature.com/articles/nature14541), [Bayesian Inference](https://en.wikipedia.org/wiki/Bayesian_inference), and the very cool field of [Probabilistic Programming](https://en.wikipedia.org/wiki/Probabilistic_programming). As I started to deep dive into papers and books about those subjects, I realized that OSIC Pulmonary Fibrosis problem screams for applying Probabilistic Machine Learning. Mindlessly applying Deep Learning to solve this problem reminded of an oldie but goldie: **if all you have is a hammer, everything looks like a nail**.\n\nI'm so excited about my \"discovery\" of this new field, that I could go on and on writing about what I learned over the past month.\nBut instead, I will leave some nice references that helped me gain insight about **discriminative vs generative machine learning**, **deterministic vs stochastic algorithms**, **bayesian vs frequentist approach**, and dive into a demonstration (there are many many more great readings, these are just some to get started with varied level of depth):\n- [Probabilistic machine learning and artificial intelligence](https://www.nature.com/articles/nature14541)\n- [Automating Inference, Learning, and Design using Probabilistic Programming](https://www.robots.ox.ac.uk/~twgr/assets/pdf/rainforth2017thesis.pdf)\n- [Probabilistic Programming and Bayesian Methods for Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers)\n- [Probabilistic Models of Cognition](http://probmods.org/)\n- [Machine Learning: A Probabilistic Perspective](https://doc.lagout.org/science/Artificial%20Intelligence/Machine%20learning/Machine%20Learning_%20A%20Probabilistic%20Perspective%20%5BMurphy%202012-08-24%5D.pdf)\n\nMy idea is to break the problem in 3 sub-problems:\n1. Define and train a simple probabilistic model to make inferences about FVC using only tabular data (baby steps)\n2. Define and train an advanced probabilistic model (a Variational AutoEncoder) to learn latent features from the CT scans\n3. Combine the 2 models together\n\n## 1.1. Tools\nIMHO the mathematical tools needed are no more advanced than the tools used in Deep Learning. Unfortunately, most of the books and papers are overloaded with hardcore math that may discourage at first. But there are good exceptions (some cited above, like the excelent open books Probabilistic Programming and Bayesian Methods for Hackers, or the Probabilistic Models of Cognition), and I think everyone can learn.\n\nIn terms of Probabilistic Programming Languages, there are several options. I'd say some of the obvious choices would be those 3:\n<img src=\"https://i.ibb.co/10BYkX6/Untitled-2.jpg\" alt=\"drawing\" width=\"600\"/>\n\nAs a PyTorch user (always found TF too verbose to my taste), my natural choice was [Pyro](http://pyro.ai/). However, to my surprise, I discovered that I'd have to install it. PyMC3 and TFP work out-of-the-box in Kaggle Kernels, but Pyro doesn't. As this competition do not allow internet, I will use PyMC3. But most importantly, **Kaggle please add Pyro to kernels**!\n\n## 1.2. Statistical Modeling Process\nIn the course [Bayesian Statistics: Techniques and Models](https://www.coursera.org/learn/mcmc-bayesian-statistics), prof Matthew Heiner summarizes the modeling process as having 8 steps:\n1. Understand the problem\n2. Plan and collect data\n3. Explore the data\n4. Postulate the model\n5. Fit the model\n6. Check the model\n7. Iterate\n8. Use the model\n\nWe understand the problem well enoguh, and data is already collected. So, let's do a quick exploration of the tabular data and then postulate a model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. Exploring the data\n\"In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\"\n\nIn this first notebook, we will use only tabular data. Let's see this **decline in lung function** for 3 different patients.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\ntrain[train['Patient']=='ID00419637202311204720264'].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def chart(patient_id, ax):\n    data = train[train['Patient'] == patient_id]\n    x = data['Weeks']\n    y = data['FVC']\n    ax.set_title(patient_id)\n    ax = sns.regplot(x, y, ax=ax, ci=None, line_kws={'color':'red'})\n    \n\nf, axes = plt.subplots(1, 3, figsize=(15, 5))\nchart('ID00007637202177411956430', axes[0])\nchart('ID00009637202177434476278', axes[1])\nchart('ID00010637202177584971671', axes[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The decline in lung capacity is very clear. We see, though, they are very different from patient to patient.\n\n\"Many large data sets are in fact large collections of small data sets. For example, in areas such as **personalized medicine** and recommendation systems, there might be a large amount of data, but there is still a relatively **small amount of data for each patient** or client, respectively. To **customize predictions for each person** it becomes necessary to **build a model for each person** — with its inherent **uncertainties** — and to couple these models together in a hierarchy so that **information can be borrowed from other similar people**. We call this the **personalization of models**, and it is naturally implemented using **hierarchical Bayesian approaches** (...).\" (Ghahramani, 2015)\n\nThis is exactly what we will try here. Moving on.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 3. Postulate the model\nTime to be creative. There are a multitude of ways we could model this tabular dataset. Some of the tools we could use:\n- Hidden Markov Models\n- Gaussian Processes\n- Variational Auto Encoders\n\nAs I am learning, I will try first the simplest possible model: a **linear regression**.\nHowever, we will sophisticate a little bit. Here are our assumptions:\n- Every patient has unique linear regression parameters ($\\alpha$ and $\\beta$). So, by inferring the right parameters, we will be able to predict the line(s) for each patient, thus being able to predict his FVC in any week\n- However, these parameters are not completely independent. There is an underlying model that governs them for all patients.\n- Both $\\alpha$ and $\\beta$ are normally distributed with different means and variances\n- These means and variances are functions of the baseline measure (baseline week, FVC and Percent), and patient's age, sex and smoking status\n- In the next notebook, we will sophisticate even further, by assuming the parameters are also function of latent variables learned from the CT scans. But that will come later. Baby steps :)\n\nOur model is represented by the following Bayesian Network:\n\n<img src=\"https://i.ibb.co/DCxKbdT/Asset-1-2x-100.jpg\" alt=\"drawing\" width=\"600\"/>\n\nLet me explain the logic behind this model:\n- $FVC_{ij}$ is the observed variable we are interested in. At any week $j$, $-12 \\leq j \\leq 133$, the FVC of the patient $i$ is presumed to be normally distributed with mean $\\alpha_i + \\beta_i i$ and $\\sigma_i^2$ (the confidence asked)\n- $\\alpha_i$, the intercept of the decline function for each patient $i$, logically is a function of $FVC_i^b$ (the baseline measurement for patient $i$) and $w_i^b$ (the week when the baseline FVC was measured). We assume it is normally distributed with mean $FVC_i^b + w_i^b \\beta^{int}$ and variance $\\sigma_{int}^2$ (int is superscript, but I couldn't get latex to behave rs)\n- $\\beta_i$, the slope of the decline function for each patient $i$, logically is a function of $A_i$ (patient's age), sex and smoking status. We assume it is normally distributed with mean $\\alpha^s + A_i \\beta_c^s$, with variance $\\sigma_s^2$ (again, s should be superscript). We considered 6 different $\\beta_c^s$: for women who currently smoke, men who currently smoke, women ex-smokers, men ex-mokers, women who never smoked and men who never smoked.\n- For now, to simplify, we left Percent random variable out. We will include in a second version.\n- Finally, we know nothing about the priors $\\beta^{int}$, $\\alpha^s$, $\\sigma_i$, $\\sigma^{int}$ and $\\sigma^s$. We will model the first 2 as normals, and the last 3 as half-normals.\n\nMathematically, the model specification is\n$$\nFVC_{ij} \\sim \\mathcal{N}(\\alpha_i + j \\beta_i, \\sigma_i) \\\\\n\\sigma_i \\sim |\\mathcal{N}(0, 200)| \\\\\n\\alpha_i \\sim \\mathcal{N}(FVC_i^b + w_i^b \\beta^{int}, \\sigma^{int}) \\\\\n\\beta_i \\sim \\mathcal{N}(\\alpha^s + A_i \\beta_c^s, \\sigma^s)\\\\\n\\beta^{int} \\sim \\mathcal{N}(0, 100) \\\\\n\\sigma^{int} \\sim |\\mathcal{N}(0, 100)| \\\\\n\\beta_c^s \\sim \\mathcal{N}(0, 100) \\\\\n\\alpha^s \\sim \\mathcal{N}(0, 100) \\\\\n\\sigma^s \\sim |\\mathcal{N}(0, 100)|\n$$","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Kaggle, please add Pyro/PyTorch support!\nimport pymc3 as pm\nimport theano\nimport arviz as az\nfrom sklearn import preprocessing","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.1. Simple data prep","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Very simple pre-processing: adding patient class\ndef patient_class(row):\n    if row['Sex'] == 'Male':\n        if row['SmokingStatus'] == 'Currently smokes':\n            return 0\n        elif row['SmokingStatus'] == 'Ex-smoker':\n            return 1\n        elif row['SmokingStatus'] == 'Never smoked':\n            return 2\n    else:\n        if row['SmokingStatus'] == 'Currently smokes':\n            return 3\n        elif row['SmokingStatus'] == 'Ex-smoker':\n            return 4\n        elif row['SmokingStatus'] == 'Never smoked':\n            return 5\n\ntrain['Class'] = train.apply(patient_class, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Very simple pre-processing: adding FVC and week baselines\naux = train[['Patient', 'Weeks']].groupby('Patient')\\\n    .min().reset_index()\naux = pd.merge(aux, train[['Patient', 'Weeks', 'FVC']], how='left', \n               on=['Patient', 'Weeks'])\naux = aux.groupby('Patient').mean().reset_index()\naux['Weeks'] = aux['Weeks'].astype(int)\naux['FVC'] = aux['FVC'].astype(int)\ntrain = pd.merge(train, aux, how='left', on='Patient', suffixes=('', '_base'))\ntrain[train['Patient']=='ID00419637202311204720264'].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Very simple pre-processing: creating patient indexes\nle = preprocessing.LabelEncoder()\ntrain['PatientID'] = le.fit_transform(train['Patient'])\n\npatients = train[['Patient', 'PatientID', 'Age', 'Class', 'Weeks_base', 'FVC_base']].drop_duplicates()\nfvc_data = train[['Patient', 'PatientID', 'Weeks', 'FVC']]\n\npatients.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fvc_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2. Modeling in PyMC3\nProbabilistic Programming Languages are very very very cool :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"FVC_b = patients['FVC_base'].values\nw_b = patients['Weeks_base'].values\nage = patients['Age'].values\npatient_class = patients['Class'].values\n\nt = fvc_data['Weeks'].values\nFVC_obs = fvc_data['FVC'].values\npatient_id = fvc_data['PatientID'].values\n\nwith pm.Model() as hierarchical_model:\n    # Hyperpriors for Alpha\n    beta_int = pm.Normal('beta_int', 0, sigma=100)\n    sigma_int = pm.HalfNormal('sigma_int', 100)\n    \n    # Alpha\n    mu_alpha = FVC_b + beta_int * w_b\n    alpha = pm.Normal('alpha', mu=mu_alpha, sigma=sigma_int, \n                      shape=train['Patient'].nunique())\n    \n    # Hyperpriors for Beta\n    sigma_s = pm.HalfNormal('sigma_s', 100)\n    alpha_s = pm.Normal('alpha_s', 0, sigma=100)\n    beta_cs = pm.Normal('beta_cs', 0, sigma=100, shape=6)\n    \n    # Beta\n    mu_beta = alpha_s + age * beta_cs[patient_class]\n    beta = pm.Normal('beta', mu=mu_beta, sigma=sigma_s,\n                     shape=train['Patient'].nunique())\n    \n    # Model variance\n    sigma = pm.HalfNormal('sigma', 200)\n    \n    # Model estimate\n    FVC_est = alpha[patient_id] + beta[patient_id] * t\n    \n    # Data likelihood\n    FVC_like = pm.Normal('FVC_like', mu=FVC_est,\n                          sigma=sigma, observed=FVC_obs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Fit the model\nJust press the inference button (TM)! :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Inference button (TM)!\nwith hierarchical_model:\n    trace = pm.sample(2000, tune=2000, target_accept=.9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We just sampled 4000 different models that explain the data! Very cool! :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 5. Check the model\nLet's see the generative model we've created.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with hierarchical_model:\n    pm.traceplot(trace);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Very cool!!! Looks like our model learned personalized alphas and betas for each patient!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 5.1. Checking some patients\nPyMC3 comes with a very powerful visualization tool called [ArviZ](https://arviz-devs.github.io/arviz/index.html). However, I didn't figure out how to use yet... Let's use Matplotlib and Seaborn.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def chart(patient_id, ax):\n    data = train[train['Patient'] == patient_id]\n    x = data['Weeks']\n    y = data['FVC']\n    ax.set_title(patient_id)\n    ax = sns.regplot(x, y, ax=ax, ci=None, line_kws={'color':'red'})\n    \n    x2 = np.arange(-12, 133, step=0.1)\n    \n    pid = patients[patients['Patient'] == patient_id]['PatientID'].values[0]\n    for sample in range(100):\n        alpha = trace['alpha'][sample, pid]\n        beta = trace['beta'][sample, pid]\n        sigma = trace['sigma'][sample]\n        y2 = alpha + beta * x2\n        ax.plot(x2, y2, linewidth=0.1, color='green')\n        y2 = alpha + beta * x2 + sigma\n        ax.plot(x2, y2, linewidth=0.1, color='yellow')\n        y2 = alpha + beta * x2 - sigma\n        ax.plot(x2, y2, linewidth=0.1, color='yellow')\n\nf, axes = plt.subplots(1, 3, figsize=(15, 5))\nchart('ID00007637202177411956430', axes[0])\nchart('ID00009637202177434476278', axes[1])\nchart('ID00010637202177584971671', axes[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I plotted 100 out of the 4000 personalized models each patient has! In green we can see the fitted regression line, in yellow the standard deviation. Let's ensemble all that!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 6. (Iterate and) Use the model\nLet's use our generative model now! Iteration will be left as an exercise to the reader :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 6.1. Simple data prep","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Very simple pre-processing: adding patient class\ndef patient_class(row):\n    if row['Sex'] == 'Male':\n        if row['SmokingStatus'] == 'Currently smokes':\n            return 0\n        elif row['SmokingStatus'] == 'Ex-smoker':\n            return 1\n        elif row['SmokingStatus'] == 'Never smoked':\n            return 2\n    else:\n        if row['SmokingStatus'] == 'Currently smokes':\n            return 3\n        elif row['SmokingStatus'] == 'Ex-smoker':\n            return 4\n        elif row['SmokingStatus'] == 'Never smoked':\n            return 5\n\ntest['Class'] = test.apply(patient_class, axis=1)\ntest = test.rename(columns={'FVC': 'FVC_base', 'Weeks': 'Weeks_base'})\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# prepare submission dataset\nsubmission = []\nfor i, patient in enumerate(test['Patient'].unique()):\n    df = pd.DataFrame(columns=['Patient', 'Weeks', 'FVC'])\n    df['Weeks'] = np.arange(-12, 134)\n    df['Patient'] = patient\n    df['PatientID'] = i\n    df['FVC'] = 0\n    submission.append(df)\n    \nsubmission = pd.concat(submission).reset_index(drop=True)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2. Posterior prediction\nThere are 2 ways of generating predictions on unseen held-out data using PyMC3. The first involves using `theano.shared` variables. It's pretty straightforward, 4-5 lines of code and we are done. I tried that, and although it worked perfectly while I was runnning the notebook, when I submitted Kaggle server complained, outputting **Submission CSV Not Found** error msg.\n\nMotivated by that, we used a 2nd approach, that works! It's a little bit longer than the 4-5 lines of code, but way more educational. The idea is outlined by PyMC3 developers in [this answer from Luciano Paz](https://discourse.pymc.io/t/how-do-we-predict-on-new-unseen-groups-in-a-hierarchical-model-in-pymc3/2571). To predict FVCs on hold-out data, we will **create a 2nd model, using as priors the distributions for the parameters learned on the 1st model**. It's Bayes spirit/philosophy: we keep constantly updating our models as we see more data. :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"FVC_b = test['FVC_base'].values\nw_b = test['Weeks_base'].values\nage = test['Age'].values\npatient_class = test['Class'].values\nt = submission['Weeks'].values\npatient_id = submission['PatientID'].values\n            \nwith pm.Model() as new_model:\n    # Hyperpriors for Alpha\n    beta_int = pm.Normal('beta_int', \n                         trace['beta_int'].mean(), \n                         sigma=trace['beta_int'].std())\n    sigma_int = pm.TruncatedNormal('sigma_int', \n                                   trace['sigma_int'].mean(),\n                                   sigma=trace['sigma_int'].std(),\n                                   lower=0)\n    \n    # Alpha\n    mu_alpha = FVC_b + beta_int * w_b\n    alpha = pm.Normal('alpha', mu=mu_alpha, sigma=sigma_int, \n                      shape=test['Patient'].nunique())\n    \n    # Hyperpriors for Beta\n    sigma_s = pm.TruncatedNormal('sigma_s', \n                                 trace['sigma_s'].mean(),\n                                 sigma=trace['sigma_s'].std(),\n                                 lower=0)\n    alpha_s = pm.Normal('alpha_s', \n                        trace['alpha_s'].mean(), \n                        sigma=trace['alpha_s'].std())\n    cov = np.zeros((6, 6))\n    np.fill_diagonal(cov, trace['beta_cs'].var(axis=0))\n    beta_cs = pm.MvNormal('beta_cs',\n                          mu=trace['beta_cs'].mean(axis=0),\n                          cov=cov,\n                          shape=6)\n    \n    # Beta\n    mu_beta = alpha_s + age * beta_cs[patient_class]\n    beta = pm.Normal('beta', mu=mu_beta, sigma=sigma_s,\n                     shape=test['Patient'].nunique())\n    \n    # Model variance\n    sigma = pm.TruncatedNormal('sigma', \n                               trace['sigma'].mean(),\n                               sigma=trace['sigma'].std(),\n                               lower=0)\n    \n    # Model estimate\n    FVC_est = pm.Normal('FVC_est', mu=alpha[patient_id] + beta[patient_id] * t, \n                        sigma=sigma,\n                        shape=submission.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with new_model:\n    trace2 = pm.sample(2000, tune=2000, target_accept=.9)\n    \ntrace2['FVC_est'].shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There we go! 4000 predictions for each point! Before we merge the predictions and submit, let's take a look at the learned parameters for the 5 test patients:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with new_model:\n    pm.traceplot(trace2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can clearly see very different $\\alpha$'s and $\\beta$'s for each patient, with varied levels of uncertainty! That's exactly what we wanted!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 6.3. Estimating competition metric\nFinally, before submitting, let's estimate the competition metric:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = pd.DataFrame(data=trace2['FVC_est'].T)\nsubmission = pd.merge(submission, preds, left_index=True, \n                      right_index=True)\nsubmission['Patient_Week'] = submission['Patient'] + '_' \\\n    + submission['Weeks'].astype(str)\nsubmission = submission.drop(columns=['FVC', 'PatientID'])\n\nFVC = submission.iloc[:, :-1].mean(axis=1)\nconfidence = submission.iloc[:, :-1].std(axis=1)\nsubmission['FVC'] = FVC\nsubmission['Confidence'] = confidence\nsubmission = submission[['Patient', 'Weeks', 'Patient_Week', \n                         'FVC', 'Confidence']]\n\nsubmission.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = pd.merge(train[['Patient', 'Weeks', 'FVC']], \n                submission.drop(columns=['Patient_Week']),\n                on=['Patient', 'Weeks'], how='left', \n                suffixes=['', '_pred'])\ntemp = temp.dropna()\ntemp = temp.groupby('Patient')\n\n# The metric only uses the last 3 measurements, the most uncertain\ntemp = temp.tail(3)\ntemp.head(20)\ntrain[train['Patient'] == 'ID00419637202311204720264'].head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sigma_clipped = temp['Confidence'].apply(lambda s: max(s, 70))\ndelta = temp.apply(lambda row: min([abs(row['FVC'] - row['FVC_pred']), 1000]), axis=1)\nmetric = -np.sqrt(2) * delta / sigma_clipped - np.log(np.sqrt(2) * sigma_clipped)\nmetric.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.4. Generating final predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = submission[['Patient_Week', 'FVC', 'Confidence']]\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}