{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":20604,"databundleVersionId":1357052,"sourceType":"competition"},{"sourceId":1426653,"sourceType":"datasetVersion","datasetId":804202},{"sourceId":2378330,"sourceType":"datasetVersion","datasetId":492658}],"dockerImageVersionId":29981,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n

🩺OSIC Pulmonary Fibrosis Competition🩺

\n

EDA and Preprocessing

\n

vasanth kumar.ch

\n\n# 1. Introduction\n\n### #1. Why do we do this?\n\n> **Pulmonary = lung | fibrosis = scar tissue => Scarring in the lungs.** Over time, scarring can destroy the normal lung and make it hard for oxygen to get into your blood. [Pulmonary fibrosis isn’t just one disease. It is a family of more than 200 different lung diseases that all look very much alike.](https://www.pulmonaryfibrosis.org/life-with-pf/about-pf)\n\n\n\n### #2. Evaluation\n\n**Laplace Log Likelihood (modified version)**: useful to evaluate a model's confidence in its decisions.\n\nThe error is thresholded at 1000 ml to avoid large errors adversely penalizing results, while the confidence values are clipped at 70 ml to reflect the approximate measurement uncertainty in FVC. The final score is calculated by averaging the metric across all test set Patient_Weeks (three per patient). \n\n*Note that metric values will be negative and higher is better.*\n\n### #3. Code Competition\n\nCode Competitions seem more fair to me. Keep in mind:\n* CPU runtime <= 9 hrs\n* GPU runtime <= 4 hrs\n* NO internet access (for inference, but can be used during training)\n* Submission file name: `submission.csv`\n\n### #4. Competition aim:\n> Predict how severe is going to decline the lung capacity in the lungs based on a CT (Computerised Tomography) scan + metadata of the patient. CTs can show soft tissues, blood vessels, and bones much more detailed than normal X-rays.\n\n**Lung Function**: assessed based on a spirometer output which measures the FVC (forced vital capacity), meaning how much air is exhaled.\n\n\n### Libraries 📚⬇","metadata":{}},{"cell_type":"code","source":"# Regular Imports\nimport os\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.image as mpimg\nfrom tabulate import tabulate\nimport missingno as msno \nfrom IPython.display import display_html\nfrom PIL import Image\nimport gc\nimport cv2\nfrom scipy.stats import pearsonr\n\nimport pydicom # for DICOM images\nfrom skimage.transform import resize\nimport copy\nimport re\n\n# Segmentation\nfrom glob import glob\nfrom mpl_toolkits.mplot3d.art3d import Poly3DCollection\nimport scipy.ndimage\nfrom skimage import morphology\nfrom skimage import measure\nfrom skimage.transform import resize\nfrom sklearn.cluster import KMeans\nfrom plotly import __version__\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nfrom plotly.tools import FigureFactory as FF\nfrom plotly.graph_objs import *\ninit_notebook_mode(connected=True) \n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n\n# Set Color Palettes for the notebook\ncustom_colors = ['#74a09e','#86c1b2','#98e2c6','#f3c969','#f2a553', '#d96548', '#c14953']\nsns.palplot(sns.color_palette(custom_colors))\n\n# Set Style\nsns.set_style(\"whitegrid\")\nsns.despine(left=True, bottom=True)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.rc('xtick',labelsize=11)\nplt.rc('ytick',labelsize=11)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. The Metadata 🗄\n\n### We need to predict:\n* FVC - final 3 values for each patient (only these will be used for the final score)\n* Confidence Value - [a thread about what it is here](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166753). It is measured in ml and it's \"how confident\" are you about the the estimated FVC. High value in confidence means that you are off a lot from the actual FVC, while a very low (or even 0) confidence means you're very sure on the FVC. (if somebody thinks this is not correct, please text me, I might be wrong about this)\n\n> ","metadata":{}},{"cell_type":"code","source":"# Import train + test data\ntrain = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ntest = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/test.csv\")\n\n# Train len\nprint(\"Total Recordings in Train Data: {:,}\".format(len(train)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df1_styler = train.head().style.set_table_attributes(\"style='display:inline'\").set_caption('Head Train Data')\ndf2_styler = test.style.set_table_attributes(\"style='display:inline'\").set_caption('Test Data (rest Hidden)')\n\ndisplay_html(df1_styler._repr_html_() + df2_styler._repr_html_(), raw=True)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.1 Any Missing Values? (❔)\n\n> **📌Remember**: No missing values ^^","metadata":{}},{"cell_type":"code","source":"print(\"Q: Are there any missing values?\", \"\\n\" +\n \"A: {}\".format(train.isnull().values.any()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.2 How Many unique patients? 👩‍⚕️👨‍⚕️\n\n> **📌Remember**: **176** unique patients, with entries between 6 and 10 each.","metadata":{}},{"cell_type":"code","source":"print(\"There are {} unique patients in Train Data.\".format(len(train[\"Patient\"].unique())), \"\\n\")\n\n# Recordings per Patient\ndata = train.groupby(by=\"Patient\")[\"Weeks\"].count().reset_index(drop=False)\n# Sort by Weeks\ndata = data.sort_values(['Weeks']).reset_index(drop=True)\nprint(\"Minimum number of entries are: {}\".format(data[\"Weeks\"].min()), \"\\n\" +\n \"Maximum number of entries are: {}\".format(data[\"Weeks\"].max()))\n\n# Plot\nplt.figure(figsize = (16, 6))\np = sns.barplot(data[\"Patient\"], data[\"Weeks\"], color=custom_colors[2])\n\nplt.title(\"Number of Entries per Patient\", fontsize = 17)\nplt.xlabel('Patient', fontsize=14)\nplt.ylabel('Frequency', fontsize=14)\n\np.axes.get_xaxis().set_visible(False);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.3 Patients Bio: who are they? 🤷‍♂️\n\n> **📌Remember**: \n1. **Age**: As the competition description mentions, patients have between 50 and 90 years old, with the majority being between 60 and 70 yo.\n2. **Gender**: The vast majority of the patients are Males.\n3. **Smoking Status**: The vast majority are Ex-smokers. ~50 never smoked and less than 10 patients are still smoking.","metadata":{}},{"cell_type":"code","source":"# Select unique bio info for the patients\ndata = train.groupby(by=\"Patient\")[[\"Patient\", \"Age\", \"Sex\", \"SmokingStatus\"]].first().reset_index(drop=True)\n\n# Figure\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (16, 6))\n\na = sns.distplot(data[\"Age\"], ax=ax1, color=custom_colors[1], hist=False, kde_kws=dict(lw=6, ls=\"--\"))\nb = sns.countplot(data[\"Sex\"], ax=ax2, palette=custom_colors[2:4])\nc = sns.countplot(data[\"SmokingStatus\"], ax=ax3, palette = custom_colors[4:7])\n\na.set_title(\"Patient Age Distribution\", fontsize=16)\nb.set_title(\"Sex Frequency\", fontsize=16)\nc.set_title(\"Smoking Status\", fontsize=16);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.4 FVC & Percent ⚖\n\n
\n

FVC the recorded lung capacity in ml (how much air you can exhale in a maximal forced expiration effort)

\n

Percent a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics

\n
\n\n> **📌Remember**: \n1. **FVC**: most values lie between 1,000 and 5,000 ml. There are also some very high outliers above 5,000 and little values that lie below 1,000.\n2. **Percent**: more than ~80% of the patients scored below 100%.","metadata":{}},{"cell_type":"code","source":"print(\"Min FVC value: {:,}\".format(train[\"FVC\"].min()), \"\\n\" +\n \"Max FVC value: {:,}\".format(train[\"FVC\"].max()), \"\\n\" +\n \"\\n\" +\n \"Min Percent value: {:.4}%\".format(train[\"Percent\"].min()), \"\\n\" +\n \"Max Percent value: {:.4}%\".format(train[\"Percent\"].max()))\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.distplot(train[\"FVC\"], ax=ax1, color=custom_colors[6], hist=False, kde_kws=dict(lw=6, ls=\"--\"))\nb = sns.distplot(train[\"Percent\"], ax=ax2, color=custom_colors[4], hist=False, kde_kws=dict(lw=6, ls=\"-.\"))\n\na.set_title(\"FVC Distribution\", fontsize=16)\nb.set_title(\"Percent Distribution\", fontsize=16);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.5 Weeks📅\n\n> `Weeks`: the relative number of weeks pre/post the baseline CT (may be negative)","metadata":{}},{"cell_type":"code","source":"print(\"Minimum no. weeks before CT: {}\".format(train['Weeks'].min()), \"\\n\" +\n \"Maximum no. weeks after CT: {}\".format(train['Weeks'].max()))\n\nplt.figure(figsize = (16, 6))\n\na = sns.distplot(train['Weeks'], color=custom_colors[3], hist=False, kde_kws=dict(lw=8, ls=\"--\"))\nplt.title(\"Number of weeks before/after the CT scan\", fontsize = 16)\nplt.xlabel(\"Weeks\", fontsize=14);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.6 Correlations between Variables🧬\n\n> **📌Remember**: \n* There is high correlation between FVC and Percent: when the volume of air increases, the Percent increases as well (when you exhale more, you get closer more to 100%)\n* There is no correlation between FVC/Percent and Age, meaning that Age has no influence on the volume of exhaled air.","metadata":{}},{"cell_type":"code","source":"# Compute Correlation\ncorr1, _ = pearsonr(train[\"FVC\"], train[\"Percent\"])\ncorr2, _ = pearsonr(train[\"FVC\"], train[\"Age\"])\ncorr3, _ = pearsonr(train[\"Percent\"], train[\"Age\"])\nprint(\"Pearson Corr FVC x Percent: {:.4}\".format(corr1), \"\\n\" +\n \"Pearson Corr FVC x Age: {:.0}\".format(corr2), \"\\n\" +\n \"Pearson Corr Percent x Age: {:.2}\".format(corr3))\n\n# Figure\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (16, 6))\n\na = sns.scatterplot(x = train[\"FVC\"], y = train[\"Percent\"], palette=[custom_colors[2], custom_colors[6]],\n hue = train[\"Sex\"], style = train[\"Sex\"], s=100, ax=ax1)\n\nb = sns.scatterplot(x = train[\"FVC\"], y = train[\"Age\"], palette=[custom_colors[2], custom_colors[6]],\n hue = train[\"Sex\"], style = train[\"Sex\"], s=100, ax=ax2)\n\nc = sns.scatterplot(x = train[\"Percent\"], y = train[\"Age\"], palette=[custom_colors[2], custom_colors[6]],\n hue = train[\"Sex\"], style = train[\"Sex\"], s=100, ax=ax3)\n\na.set_title(\"Correlation between FVC and Percent\", fontsize = 16)\na.set_xlabel(\"FVC\", fontsize = 14)\na.set_ylabel(\"Percent\", fontsize = 14)\n\nb.set_title(\"Correlation between FVC and Age\", fontsize = 16)\nb.set_xlabel(\"FVC\", fontsize = 14)\nb.set_ylabel(\"Age\", fontsize = 14)\n\nc.set_title(\"Correlation between Percent and Age\", fontsize = 16)\nc.set_xlabel(\"Percent\", fontsize = 14)\nc.set_ylabel(\"Age\", fontsize = 14);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> This is VERY weird: FVC and Percent are the highest for people that still smoke and the lowest for people that never smoked. HOWEVER, we need to keep in mind that the percentage of people that still smoke is very low. So, we CAN'T conclude that if a person smokes it's highly likely that will have a high FVC.","metadata":{}},{"cell_type":"code","source":"# Figure\nf, (ax1, ax2) = plt.subplots(1,2, figsize = (16, 6))\n\na = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"FVC\"], ax=ax1, palette=custom_colors[0:4])\nb = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"Percent\"], ax=ax2, palette=custom_colors[4:7])\n\na.set_title(\"Mean FVC per Smoking Status\", fontsize=16)\nb.set_title(\"Mean Perc per Smoking Status\", fontsize=16);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## #2.7 Evolution of FVC in Patients 📉","metadata":{}},{"cell_type":"code","source":"# Create Time variable to count in ascending order the times the Patient has done a check in FVC\ndata_time = train.groupby(by=\"Patient\")[\"Weeks\"].count().reset_index()\ntrain[\"Time\"] = 0\n\nfor patient, times in zip(data_time[\"Patient\"], data_time[\"Weeks\"]):\n train.loc[train[\"Patient\"] == patient, 'Time'] = range(1, times+1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For graph purposes, keep only Patients that had a big difference in FVC between Time 1 and last Time\nmin_fvc = train[train[\"Time\"] == 1][[\"Patient\", \"FVC\"]].reset_index(drop=True)\n\nidx = train.groupby([\"Patient\"])[\"Weeks\"].transform(max) == train[\"Weeks\"]\nmax_fvc = train[idx][[\"Patient\", \"FVC\"]].reset_index(drop=True)\n\n# Compute difference and select only top patients with biggest difference\ndata = pd.merge(min_fvc, max_fvc, how=\"inner\", on=\"Patient\")\ndata[\"Dif\"] = data[\"FVC_x\"] - data[\"FVC_y\"]\n\n# Select only top n\nl = list(data.sort_values(\"Dif\", ascending=False).head(100)[\"Patient\"])\nx = train[train[\"Patient\"].isin(l)]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 6))\n\na = sns.lineplot(x = x[\"Time\"], y = x[\"FVC\"], hue = x[\"Patient\"], legend=False,\n palette=sns.color_palette(\"GnBu_d\", 100), size=1)\n\nplt.title(\"Patient FVC decrease on Weeks\", fontsize = 16)\nplt.xlabel(\"Weeks\", fontsize=14)\nplt.ylabel(\"FVC\", fontsize=14);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. DICOM Data 📸\n\n> [What are CT scans capturing?](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/167085)\n\n> **Dicom Files**: [A DICOM file is an image saved in the Digital Imaging and Communications in Medicine (DICOM) format.](https://fileinfo.com/extension/dicom#:~:text=A%20DICOM%20file%20is%20an,linked%20to%20a%20specific%20individual.)","metadata":{}},{"cell_type":"code","source":"# Create base director for Train .dcm files\ndirector = \"../input/osic-pulmonary-fibrosis-progression/train\"\n\n# Create path column with the path to each patient's CT\ntrain[\"Path\"] = director + \"/\" + train[\"Patient\"]\n\n# Create variable that shows how many CT scans each patient has\ntrain[\"CT_number\"] = 0\n\nfor k, path in enumerate(train[\"Path\"]):\n train[\"CT_number\"][k] = len(os.listdir(path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.1 Number of CT scans per Patient👴\n\n> **📌Remember**: Huge imbalance in the number of CT scans: half of the patients have less that 100 photos registered.","metadata":{}},{"cell_type":"code","source":"print(\"Minimum number of CT scans: {}\".format(train[\"CT_number\"].min()), \"\\n\" +\n \"Maximum number of CT scans: {:,}\".format(train[\"CT_number\"].max()))\n\n# Scans per Patient\ndata = train.groupby(by=\"Patient\")[\"CT_number\"].first().reset_index(drop=False)\n# Sort by Weeks\ndata = data.sort_values(['CT_number']).reset_index(drop=True)\n\n# Plot\nplt.figure(figsize = (16, 6))\np = sns.barplot(data[\"Patient\"], data[\"CT_number\"], color=custom_colors[5])\nplt.axvline(x=85, color=custom_colors[2], linestyle='--', lw=3)\n\nplt.title(\"Number of CT Scans per Patient\", fontsize = 17)\nplt.xlabel('Patient', fontsize=14)\nplt.ylabel('Frequency', fontsize=14)\n\nplt.text(86, 850, \"Median=94\", fontsize=13)\n\np.axes.get_xaxis().set_visible(False);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Visualise DICOM Info and Image\n\n> DICOM data can be extracted by using `pydicom.dcmread()`\n\n> [What color maps should you use in medical visualisation?](http://noeskasmit.com/colormaps-in-medical-visualization/)","metadata":{}},{"cell_type":"code","source":"class bcolors:\n OKBLUE = '\\033[96m'\n OKGREEN = '\\033[92m'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://gist.github.com/FedeMiorelli/640bbc66b2038a14802729e609abfe89\n# Create Turbo color map\nturbo_colormap_data = np.array(\n [[0.18995,0.07176,0.23217],\n [0.19483,0.08339,0.26149],\n [0.19956,0.09498,0.29024],\n [0.20415,0.10652,0.31844],\n [0.20860,0.11802,0.34607],\n [0.21291,0.12947,0.37314],\n [0.21708,0.14087,0.39964],\n [0.22111,0.15223,0.42558],\n [0.22500,0.16354,0.45096],\n [0.22875,0.17481,0.47578],\n [0.23236,0.18603,0.50004],\n [0.23582,0.19720,0.52373],\n [0.23915,0.20833,0.54686],\n [0.24234,0.21941,0.56942],\n [0.24539,0.23044,0.59142],\n [0.24830,0.24143,0.61286],\n [0.25107,0.25237,0.63374],\n [0.25369,0.26327,0.65406],\n [0.25618,0.27412,0.67381],\n [0.25853,0.28492,0.69300],\n [0.26074,0.29568,0.71162],\n [0.26280,0.30639,0.72968],\n [0.26473,0.31706,0.74718],\n [0.26652,0.32768,0.76412],\n [0.26816,0.33825,0.78050],\n [0.26967,0.34878,0.79631],\n [0.27103,0.35926,0.81156],\n [0.27226,0.36970,0.82624],\n [0.27334,0.38008,0.84037],\n [0.27429,0.39043,0.85393],\n [0.27509,0.40072,0.86692],\n [0.27576,0.41097,0.87936],\n [0.27628,0.42118,0.89123],\n [0.27667,0.43134,0.90254],\n [0.27691,0.44145,0.91328],\n [0.27701,0.45152,0.92347],\n [0.27698,0.46153,0.93309],\n [0.27680,0.47151,0.94214],\n [0.27648,0.48144,0.95064],\n [0.27603,0.49132,0.95857],\n [0.27543,0.50115,0.96594],\n [0.27469,0.51094,0.97275],\n [0.27381,0.52069,0.97899],\n [0.27273,0.53040,0.98461],\n [0.27106,0.54015,0.98930],\n [0.26878,0.54995,0.99303],\n [0.26592,0.55979,0.99583],\n [0.26252,0.56967,0.99773],\n [0.25862,0.57958,0.99876],\n [0.25425,0.58950,0.99896],\n [0.24946,0.59943,0.99835],\n [0.24427,0.60937,0.99697],\n [0.23874,0.61931,0.99485],\n [0.23288,0.62923,0.99202],\n [0.22676,0.63913,0.98851],\n [0.22039,0.64901,0.98436],\n [0.21382,0.65886,0.97959],\n [0.20708,0.66866,0.97423],\n [0.20021,0.67842,0.96833],\n [0.19326,0.68812,0.96190],\n [0.18625,0.69775,0.95498],\n [0.17923,0.70732,0.94761],\n [0.17223,0.71680,0.93981],\n [0.16529,0.72620,0.93161],\n [0.15844,0.73551,0.92305],\n [0.15173,0.74472,0.91416],\n [0.14519,0.75381,0.90496],\n [0.13886,0.76279,0.89550],\n [0.13278,0.77165,0.88580],\n [0.12698,0.78037,0.87590],\n [0.12151,0.78896,0.86581],\n [0.11639,0.79740,0.85559],\n [0.11167,0.80569,0.84525],\n [0.10738,0.81381,0.83484],\n [0.10357,0.82177,0.82437],\n [0.10026,0.82955,0.81389],\n [0.09750,0.83714,0.80342],\n [0.09532,0.84455,0.79299],\n [0.09377,0.85175,0.78264],\n [0.09287,0.85875,0.77240],\n [0.09267,0.86554,0.76230],\n [0.09320,0.87211,0.75237],\n [0.09451,0.87844,0.74265],\n [0.09662,0.88454,0.73316],\n [0.09958,0.89040,0.72393],\n [0.10342,0.89600,0.71500],\n [0.10815,0.90142,0.70599],\n [0.11374,0.90673,0.69651],\n [0.12014,0.91193,0.68660],\n [0.12733,0.91701,0.67627],\n [0.13526,0.92197,0.66556],\n [0.14391,0.92680,0.65448],\n [0.15323,0.93151,0.64308],\n [0.16319,0.93609,0.63137],\n [0.17377,0.94053,0.61938],\n [0.18491,0.94484,0.60713],\n [0.19659,0.94901,0.59466],\n [0.20877,0.95304,0.58199],\n [0.22142,0.95692,0.56914],\n [0.23449,0.96065,0.55614],\n [0.24797,0.96423,0.54303],\n [0.26180,0.96765,0.52981],\n [0.27597,0.97092,0.51653],\n [0.29042,0.97403,0.50321],\n [0.30513,0.97697,0.48987],\n [0.32006,0.97974,0.47654],\n [0.33517,0.98234,0.46325],\n [0.35043,0.98477,0.45002],\n [0.36581,0.98702,0.43688],\n [0.38127,0.98909,0.42386],\n [0.39678,0.99098,0.41098],\n [0.41229,0.99268,0.39826],\n [0.42778,0.99419,0.38575],\n [0.44321,0.99551,0.37345],\n [0.45854,0.99663,0.36140],\n [0.47375,0.99755,0.34963],\n [0.48879,0.99828,0.33816],\n [0.50362,0.99879,0.32701],\n [0.51822,0.99910,0.31622],\n [0.53255,0.99919,0.30581],\n [0.54658,0.99907,0.29581],\n [0.56026,0.99873,0.28623],\n [0.57357,0.99817,0.27712],\n [0.58646,0.99739,0.26849],\n [0.59891,0.99638,0.26038],\n [0.61088,0.99514,0.25280],\n [0.62233,0.99366,0.24579],\n [0.63323,0.99195,0.23937],\n [0.64362,0.98999,0.23356],\n [0.65394,0.98775,0.22835],\n [0.66428,0.98524,0.22370],\n [0.67462,0.98246,0.21960],\n [0.68494,0.97941,0.21602],\n [0.69525,0.97610,0.21294],\n [0.70553,0.97255,0.21032],\n [0.71577,0.96875,0.20815],\n [0.72596,0.96470,0.20640],\n [0.73610,0.96043,0.20504],\n [0.74617,0.95593,0.20406],\n [0.75617,0.95121,0.20343],\n [0.76608,0.94627,0.20311],\n [0.77591,0.94113,0.20310],\n [0.78563,0.93579,0.20336],\n [0.79524,0.93025,0.20386],\n [0.80473,0.92452,0.20459],\n [0.81410,0.91861,0.20552],\n [0.82333,0.91253,0.20663],\n [0.83241,0.90627,0.20788],\n [0.84133,0.89986,0.20926],\n [0.85010,0.89328,0.21074],\n [0.85868,0.88655,0.21230],\n [0.86709,0.87968,0.21391],\n [0.87530,0.87267,0.21555],\n [0.88331,0.86553,0.21719],\n [0.89112,0.85826,0.21880],\n [0.89870,0.85087,0.22038],\n [0.90605,0.84337,0.22188],\n [0.91317,0.83576,0.22328],\n [0.92004,0.82806,0.22456],\n [0.92666,0.82025,0.22570],\n [0.93301,0.81236,0.22667],\n [0.93909,0.80439,0.22744],\n [0.94489,0.79634,0.22800],\n [0.95039,0.78823,0.22831],\n [0.95560,0.78005,0.22836],\n [0.96049,0.77181,0.22811],\n [0.96507,0.76352,0.22754],\n [0.96931,0.75519,0.22663],\n [0.97323,0.74682,0.22536],\n [0.97679,0.73842,0.22369],\n [0.98000,0.73000,0.22161],\n [0.98289,0.72140,0.21918],\n [0.98549,0.71250,0.21650],\n [0.98781,0.70330,0.21358],\n [0.98986,0.69382,0.21043],\n [0.99163,0.68408,0.20706],\n [0.99314,0.67408,0.20348],\n [0.99438,0.66386,0.19971],\n [0.99535,0.65341,0.19577],\n [0.99607,0.64277,0.19165],\n [0.99654,0.63193,0.18738],\n [0.99675,0.62093,0.18297],\n [0.99672,0.60977,0.17842],\n [0.99644,0.59846,0.17376],\n [0.99593,0.58703,0.16899],\n [0.99517,0.57549,0.16412],\n [0.99419,0.56386,0.15918],\n [0.99297,0.55214,0.15417],\n [0.99153,0.54036,0.14910],\n [0.98987,0.52854,0.14398],\n [0.98799,0.51667,0.13883],\n [0.98590,0.50479,0.13367],\n [0.98360,0.49291,0.12849],\n [0.98108,0.48104,0.12332],\n [0.97837,0.46920,0.11817],\n [0.97545,0.45740,0.11305],\n [0.97234,0.44565,0.10797],\n [0.96904,0.43399,0.10294],\n [0.96555,0.42241,0.09798],\n [0.96187,0.41093,0.09310],\n [0.95801,0.39958,0.08831],\n [0.95398,0.38836,0.08362],\n [0.94977,0.37729,0.07905],\n [0.94538,0.36638,0.07461],\n [0.94084,0.35566,0.07031],\n [0.93612,0.34513,0.06616],\n [0.93125,0.33482,0.06218],\n [0.92623,0.32473,0.05837],\n [0.92105,0.31489,0.05475],\n [0.91572,0.30530,0.05134],\n [0.91024,0.29599,0.04814],\n [0.90463,0.28696,0.04516],\n [0.89888,0.27824,0.04243],\n [0.89298,0.26981,0.03993],\n [0.88691,0.26152,0.03753],\n [0.88066,0.25334,0.03521],\n [0.87422,0.24526,0.03297],\n [0.86760,0.23730,0.03082],\n [0.86079,0.22945,0.02875],\n [0.85380,0.22170,0.02677],\n [0.84662,0.21407,0.02487],\n [0.83926,0.20654,0.02305],\n [0.83172,0.19912,0.02131],\n [0.82399,0.19182,0.01966],\n [0.81608,0.18462,0.01809],\n [0.80799,0.17753,0.01660],\n [0.79971,0.17055,0.01520],\n [0.79125,0.16368,0.01387],\n [0.78260,0.15693,0.01264],\n [0.77377,0.15028,0.01148],\n [0.76476,0.14374,0.01041],\n [0.75556,0.13731,0.00942],\n [0.74617,0.13098,0.00851],\n [0.73661,0.12477,0.00769],\n [0.72686,0.11867,0.00695],\n [0.71692,0.11268,0.00629],\n [0.70680,0.10680,0.00571],\n [0.69650,0.10102,0.00522],\n [0.68602,0.09536,0.00481],\n [0.67535,0.08980,0.00449],\n [0.66449,0.08436,0.00424],\n [0.65345,0.07902,0.00408],\n [0.64223,0.07380,0.00401],\n [0.63082,0.06868,0.00401],\n [0.61923,0.06367,0.00410],\n [0.60746,0.05878,0.00427],\n [0.59550,0.05399,0.00453],\n [0.58336,0.04931,0.00486],\n [0.57103,0.04474,0.00529],\n [0.55852,0.04028,0.00579],\n [0.54583,0.03593,0.00638],\n [0.53295,0.03169,0.00705],\n [0.51989,0.02756,0.00780],\n [0.50664,0.02354,0.00863],\n [0.49321,0.01963,0.00955],\n [0.47960,0.01583,0.01055]])\n\n\n\n\ndef RGBToPyCmap(rgbdata):\n nsteps = rgbdata.shape[0]\n stepaxis = np.linspace(0, 1, nsteps)\n\n rdata=[]; gdata=[]; bdata=[]\n for istep in range(nsteps):\n r = rgbdata[istep,0]\n g = rgbdata[istep,1]\n b = rgbdata[istep,2]\n rdata.append((stepaxis[istep], r, r))\n gdata.append((stepaxis[istep], g, g))\n bdata.append((stepaxis[istep], b, b))\n\n mpl_data = {'red': rdata,\n 'green': gdata,\n 'blue': bdata}\n\n return mpl_data\n\n\nmpl_data = RGBToPyCmap(turbo_colormap_data)\nplt.register_cmap(name='turbo', data=mpl_data, lut=turbo_colormap_data.shape[0])\n\nmpl_data_r = RGBToPyCmap(turbo_colormap_data[::-1,:])\nplt.register_cmap(name='turbo_r', data=mpl_data_r, lut=turbo_colormap_data.shape[0])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/19.dcm\"\ndataset = pydicom.dcmread(path)\n\nprint(bcolors.OKBLUE + \"Patient id.......:\", dataset.PatientID, \"\\n\" +\n \"Modality.........:\", dataset.Modality, \"\\n\" +\n \"Rows.............:\", dataset.Rows, \"\\n\" +\n \"Columns..........:\", dataset.Columns)\n\nplt.figure(figsize = (7, 7))\nplt.imshow(dataset.pixel_array, cmap=\"plasma\")\nplt.axis('off');","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 A breath-in + hold for a Patient\n\n> You can see how the lungs expand image by image.","metadata":{}},{"cell_type":"code","source":"patient_dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\ndatasets = []\n\n# First Order the files in the dataset\nfiles = []\nfor dcm in list(os.listdir(patient_dir)):\n files.append(dcm) \nfiles.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n# Read in the Dataset\nfor dcm in files:\n path = patient_dir + \"/\" + dcm\n datasets.append(pydicom.dcmread(path))\n\n# Plot the images\nfig=plt.figure(figsize=(16, 6))\ncolumns = 10\nrows = 3\n\nfor i in range(1, columns*rows +1):\n img = datasets[i-1].pixel_array\n fig.add_subplot(rows, columns, i)\n plt.imshow(img, cmap=\"plasma\")\n plt.title(i, fontsize = 9)\n plt.axis('off');","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.4 GIF from Images 🌔🌕🌖\n\n> **📌Remember**: Patients have various number of CT scans: the more scans/patient, the more information we have about well ... their lungs. Here we can see that when patients have low number of scans (~ 12) only a quick \"inhale\" is observed, whereas when we have 80+ scans the details are much more enhanced.","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nfrom IPython.display import Image as show_gif\nimport scipy.misc\nimport matplotlib","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_gif(number_of_CT = 87):\n \"\"\"Picks a patient at random and creates a GIF with their CT scans.\"\"\"\n \n # Select one of the patients\n # patient = \"ID00007637202177411956430\"\n patient = train[train[\"CT_number\"] == number_of_CT].sample(random_state=1)[\"Patient\"].values[0]\n \n # === READ IN .dcm FILES ===\n patient_dir = \"../input/osic-pulmonary-fibrosis-progression/train/\" + patient\n datasets = []\n\n # First Order the files in the dataset\n files = []\n for dcm in list(os.listdir(patient_dir)):\n files.append(dcm) \n files.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n # Read in the Dataset from the Patient path\n for dcm in files:\n path = patient_dir + \"/\" + dcm\n datasets.append(pydicom.dcmread(path))\n \n \n # === SAVE AS .png ===\n # Create directory to save the png files\n if os.path.isdir(f\"png_{patient}\") == False:\n os.mkdir(f\"png_{patient}\")\n\n # Save images to PNG\n for i in range(len(datasets)):\n img = datasets[i].pixel_array\n matplotlib.image.imsave(f'png_{patient}/img_{i}.png', img)\n \n \n # === CREATE GIF ===\n # First Order the files in the dataset (again)\n files = []\n for png in list(os.listdir(f\"../working/png_{patient}\")):\n files.append(png) \n files.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n # Create the frames\n frames = []\n\n # Create frames\n for file in files:\n # print(\"../working/png_images/\" + name)\n new_frame = Image.open(f\"../working/png_{patient}/\" + file)\n frames.append(new_frame)\n\n # Save into a GIF file that loops forever\n frames[0].save(f'gif_{patient}.gif', format='GIF',\n append_images=frames[1:],\n save_all=True,\n duration=200, loop=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create and compare GIFs 🌖🌗🌘\n\n*Note: Gifs were exported and changed to .mp4 format so the notebook can load properly. However, the commented line of code works just fine and you can uncomment and run it.*","metadata":{}},{"cell_type":"code","source":"create_gif(number_of_CT=12)\n# create_gif(number_of_CT=30)\n# create_gif(number_of_CT=87)\n\n# print(\"First file len:\", len(os.listdir(\"../working/png_ID00165637202237320314458\")), \"\\n\" +\n# \"Second file len:\", len(os.listdir(\"../working/png_ID00199637202248141386743\")), \"\\n\" +\n# \"Third file len:\", len(os.listdir(\"../working/png_ID00340637202287399835821\")))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 12 CT scans GIF:","metadata":{}},{"cell_type":"code","source":"show_gif(filename=\"./gif_ID00165637202237320314458.gif\", format='png', width=400, height=400)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30 and 87 CT scans GIF:","metadata":{}},{"cell_type":"code","source":"# show_gif(filename=\"./gif_ID00340637202287399835821.gif\", format='png', width=400, height=400)\n# show_gif(filename=\"./gif_ID00199637202248141386743.gif\", format='png', width=400, height=400)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.5 DICOM Lung Mask🎭\n\n
\nReference: DICOM Processing Segmentation Visualization in Python\n
\n\n**Mask on Lungs Purpose:**\n* Segmentation is part of the preprocessing method\n* Has the purpose of auto-detecting the boundaries surrounding a volume of interest (our case is the lungs)\n* Drawbacks: be sure you don't exclude important parts (like lesions)","metadata":{}},{"cell_type":"code","source":"# https://www.raddq.com/dicom-processing-segmentation-visualization-in-python/\n\ndef make_lungmask(img, display=False):\n row_size= img.shape[0]\n col_size = img.shape[1]\n \n mean = np.mean(img)\n std = np.std(img)\n img = img-mean\n img = img/std\n \n # Find the average pixel value near the lungs\n # to renormalize washed out images\n middle = img[int(col_size/5):int(col_size/5*4),int(row_size/5):int(row_size/5*4)] \n mean = np.mean(middle) \n max = np.max(img)\n min = np.min(img)\n \n # To improve threshold finding, I'm moving the \n # underflow and overflow on the pixel spectrum\n img[img==max]=mean\n img[img==min]=mean\n \n # Using Kmeans to separate foreground (soft tissue / bone) and background (lung/air)\n \n kmeans = KMeans(n_clusters=2).fit(np.reshape(middle,[np.prod(middle.shape),1]))\n centers = sorted(kmeans.cluster_centers_.flatten())\n threshold = np.mean(centers)\n thresh_img = np.where(imgrow_size/5 and B[2] **📌Remember**: Lungs are quite visible, but on images at the beginning and towards the end the lung dissapears completely.","metadata":{}},{"cell_type":"code","source":"patient_dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\ndatasets = []\n\n# First Order the files in the dataset\nfiles = []\nfor dcm in list(os.listdir(patient_dir)):\n files.append(dcm) \nfiles.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n# Read in the Dataset\nfor dcm in files:\n path = patient_dir + \"/\" + dcm\n datasets.append(pydicom.dcmread(path))\n \nimgs = []\nfor data in datasets:\n img = data.pixel_array\n imgs.append(img)\n \n \n# Show masks\nfig=plt.figure(figsize=(16, 6))\ncolumns = 10\nrows = 3\n\nfor i in range(1, columns*rows +1):\n img = make_lungmask(datasets[i-1].pixel_array)\n fig.add_subplot(rows, columns, i)\n plt.imshow(img, cmap=\"gray\")\n plt.title(i, fontsize = 9)\n plt.axis('off');","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. 🏃‍♂️RAPIDS - crazy fast🏃‍♂️\n\n> In this part we will use [rapids.ai](https://rapids.ai/) to extract and preprocess the *metadata from the dicom files* as well as the *data available in the train.csv*.","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/cdeotte/rapids\n\nimport sys\n!cp ../input/rapids/rapids.0.14.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Libraries 📚⬇","metadata":{}},{"cell_type":"code","source":"# Imports\nimport cudf\nimport tqdm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.1 Extracting Metadata from DICOM files\n\n> **📌Remember:** the `.get()` function is used overall for .dcm files that don't have all columns available. So, to not skip the entire file, we extract only what's available and fill in the rest with *None*.\n\n*Inspo from [Extract Metadata and Resize Notebook](https://www.kaggle.com/trsekhar123/nb-to-extract-metadata-and-resize-images-train)*","metadata":{}},{"cell_type":"code","source":"def get_observation_data(path):\n \"\"\"Get information from the .dcm files.\n path: complete path to the .dcm file\"\"\"\n\n image_data = pydicom.read_file(path)\n \n # Dictionary to store the information from the image\n observation_data = {\n \"FileNumber\" : path.split(\"/\")[5],\n \"Rows\" : image_data.get(\"Rows\"),\n \"Columns\" : image_data.get(\"Columns\"),\n \"PatientID\" : image_data.get(\"PatientID\"),\n \"BodyPartExamined\" : image_data.get(\"BodyPartExamined\"),\n \"RotationDirection\" : image_data.get(\"RotationDirection\"),\n \"ConvolutionKernel\" : image_data.get(\"ConvolutionKernel\"),\n \"PatientPosition\" : image_data.get(\"PatientPosition\"),\n \"PhotometricInterpretation\" : image_data.get(\"PhotometricInterpretation\"),\n \"Modality\" : image_data.get(\"Modality\"),\n \"StudyInstanceUID\" : image_data.get(\"StudyInstanceUID\"),\n \"PixelPaddingValue\" : image_data.get(\"PixelPaddingValue\"),\n \"SamplesPerPixel\" : image_data.get(\"SamplesPerPixel\"),\n \"BitsAllocated\" : image_data.get(\"BitsAllocated\"),\n \"BitsStored\" : image_data.get(\"BitsStored\"),\n \"HighBit\" : image_data.get(\"HighBit\"),\n \"PixelRepresentation\" : image_data.get(\"PixelRepresentation\"),\n \"RescaleType\" : image_data.get(\"RescaleType\"),\n }\n\n # Integer columns\n int_columns = [\"SliceThickness\", \"KVP\", \"DistanceSourceToDetector\", \n \"DistanceSourceToPatient\", \"GantryDetectorTilt\", \"TableHeight\", \n \"XRayTubeCurrent\", \"GeneratorPower\", \"WindowCenter\", \"WindowWidth\", \n \"SliceLocation\", \"RescaleIntercept\", \"RescaleSlope\"]\n for k in int_columns:\n observation_data[k] = int(image_data.get(k)) if k in image_data else None\n\n # String columns\n str_columns = [\"ImagePositionPatient\", \"ImageOrientationPatient\", \"ImageType\", \"PixelSpacing\"]\n for k in str_columns:\n observation_data[k] = str(image_data.get(k)) if k in image_data else None\n\n \n return observation_data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test the function to see if it works","metadata":{}},{"cell_type":"code","source":"p = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/10.dcm\"\nexample = get_observation_data(p)\n\n# example","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get full paths for the images\npaths = []\nfor path in train[\"Path\"]:\n for doc in os.listdir(path):\n paths.append(path + \"/\" + doc)\n \n# How many paths?\nprint(\"There are {:,} paths in total.\".format(len(paths)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Discussion💬 here: [rapids question](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/172373)*\n\n> ❗ **Note**: Code to be revised; maybe make it faster. Also, I'm using `pd.DataFrame()` as `cudf.DataFrame()` is not working properly (it's exracting only the keys of the dict, not the values themselves)","metadata":{}},{"cell_type":"code","source":"# Create a dictionary with the data\nexceptions = 0\ndicts = []\n\nfor path in tqdm.tqdm(paths):\n # Get info in dict format\n try:\n d = get_observation_data(path)\n dicts.append(d)\n except Exception as e:\n exceptions += 1\n continue\n\n# Convert into a cudf dataframe\n# meta_train_data = cudf.DataFrame(data=dicts, columns=example.keys())\nmeta_train_data = pd.DataFrame(data=dicts, columns=example.keys())\n\n# Export information to a .csv\nmeta_train_data.to_csv(\"meta_train.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Exceptions: {}\".format(exceptions))\nmeta_train_data.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Analyze Metadata\n\n## #1. Missing Values\n\n> **📌Remember:** `PixelPaddingValue` and `Rescale Type` columns are more than 80% empty, so they will be erased. ","metadata":{}},{"cell_type":"code","source":"# Inspect the Missing Columns\nprint(\"---Missing Columns---\", \"\\n\" +\n \"Pixel Padding Value missing percentage: {:.2f}\".format(meta_train_data[\"PixelPaddingValue\"].isna().sum()/\n len(meta_train_data[\"PixelPaddingValue\"])), \"\\n\" +\n \"Rescale Type missing percentage: {:.2f}\".format(meta_train_data[\"RescaleType\"].isna().sum()/\n len(meta_train_data[\"RescaleType\"])), \"\\n\" +\n \"Dist Source to Detector missing percentage: {:.2f}\".format(meta_train_data[\"DistanceSourceToDetector\"].isna().sum()/\n len(meta_train_data[\"DistanceSourceToDetector\"])), \"\\n\" +\n \"Dist Source to Patient missing percentage: {:.2f}\".format(meta_train_data[\"DistanceSourceToPatient\"].isna().sum()/\n len(meta_train_data[\"DistanceSourceToPatient\"])), \"\\n\" +\n \"Gantry Detector Tilt missing percentage: {:.2f}\".format(meta_train_data[\"GantryDetectorTilt\"].isna().sum()/\n len(meta_train_data[\"GantryDetectorTilt\"])), \"\\n\" +\n \"Generator Power missing percentage: {:.2f}\".format(meta_train_data[\"GeneratorPower\"].isna().sum()/\n len(meta_train_data[\"GeneratorPower\"])), \"\\n\" +\n \"Slice Location missing percentage: {:.2f}\".format(meta_train_data[\"SliceLocation\"].isna().sum()/\n len(meta_train_data[\"SliceLocation\"])), \"\\n\" +\n \"Img Position Patient missing percentage: {:.2f}\".format(meta_train_data[\"ImagePositionPatient\"].isna().sum()/\n len(meta_train_data[\"ImagePositionPatient\"])), \"\\n\" +\n \"Img Orientation Patient missing percentage: {:.2f}\".format(meta_train_data[\"ImageOrientationPatient\"].isna().sum()/\n len(meta_train_data[\"ImageOrientationPatient\"])))\n\n# Create figure\nax = msno.matrix(meta_train_data, color=(152/255, 241/255, 198/255), fontsize=10);\nplt.title(\"Metadata Missing Values\", fontsize = 25);","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Dropping the 2 columns*","metadata":{}},{"cell_type":"code","source":"meta_train_data.drop(['PixelPaddingValue', 'RescaleType'], axis=1, inplace=True)\n\n# Export information to a .csv\nmeta_train_data.to_csv(\"meta_train.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **📌Remember:** Now let's handle the other 7 columns and impute appropriately.","metadata":{}},{"cell_type":"markdown","source":"# To Be Continued ...\n\n\n# References📇\n* [What is Pulmonary Fibrosis?](https://www.pulmonaryfibrosis.org/life-with-pf/about-pf)\n* [Confidence Clarification](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166753)\n* [What are CT scans capturing?](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/167085)\n","metadata":{}}]}