{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":20604,"databundleVersionId":1357052,"sourceType":"competition"},{"sourceId":1426653,"sourceType":"datasetVersion","datasetId":804202},{"sourceId":2378330,"sourceType":"datasetVersion","datasetId":492658}],"dockerImageVersionId":29981,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# OSIC Pulmonary Fibrosis Progression Competition: EDA & DICOM Preprocessing\n\n**Overview**:\nThis notebook explores the dataset for the OSIC Pulmonary Fibrosis Progression competition through Exploratory Data Analysis (EDA) and preprocesses DICOM images for downstream modeling (e.g., segmentation or prediction tasks).\n\n**Key Tasks Performed**:\n- **EDA**: Analyze metadata from `train.csv` and DICOM files (e.g., patient demographics, FVC trends, missing values).\n- **DICOM Preprocessing**: Extract, resize, normalize, and segment lung images (including GIF visualization and masking).\n\n**Steps**:\n1. Load and inspect metadata from CSV and DICOM files.\n2. Explore missing values and handle them.\n3. Visualize feature distributions (e.g., age, sex, smoking status, FVC over time).\n4. Preprocess DICOM images (e.g., conversion to PNG/GIF, lung masking for segmentation).\n\n**Table of Contents** (for navigation): \n- [2. Metadata Overview](#2-metadata-overview-) \n- [3. DICOM Image Preprocessing](#3-dicom-image-preprocessing) \n- [4. RAPIDS Metadata Extraction](#4-rapids---crazy-fast) \n- [Summary](#-summary)","metadata":{}},{"cell_type":"code","source":"# Core Imports\nimport os, re, warnings\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport missingno as msno\nfrom PIL import Image\n\n# DICOM Handling\nimport pydicom as dicom # Alias for clarity\n\n# Stats and Correlation\nfrom scipy.stats import pearsonr\n\n# Image Processing and Segmentation\nimport scipy.ndimage as ndi\nfrom skimage import morphology, measure\nfrom sklearn.cluster import KMeans\n\n# Suppress Warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Visualization Style\nsns.set_style(\"whitegrid\")\nsns.despine(left=True, bottom=True)\n\n# Custom Color Palette\ncustom_colors = ['#74a09e', '#86c1b2', '#98e2c6', '#f3c969', '#f2a553', '#d96548', '#c14953']\nsns.palplot(sns.color_palette(custom_colors))","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-08-30T15:54:57.648842Z","iopub.execute_input":"2025-08-30T15:54:57.649152Z","iopub.status.idle":"2025-08-30T15:54:59.294878Z","shell.execute_reply.started":"2025-08-30T15:54:57.649121Z","shell.execute_reply":"2025-08-30T15:54:59.294103Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set tick label sizes for better readability\nplt.rc('xtick', labelsize=11)\nplt.rc('ytick', labelsize=11)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:54:59.787284Z","iopub.execute_input":"2025-08-30T15:54:59.787571Z","iopub.status.idle":"2025-08-30T15:54:59.791226Z","shell.execute_reply.started":"2025-08-30T15:54:59.787547Z","shell.execute_reply":"2025-08-30T15:54:59.790308Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Metadata Overview 🗄 \n\n### Prediction Targets:\n- **FVC (Forced Vital Capacity)**: Predict the final 3 FVC values for each patient (these are used for scoring).\n- **Confidence Value**: Represents uncertainty in the FVC prediction (in ml). See [this discussion](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166753) for details. A high confidence means large deviation from actual FVC; low (or 0) means high certainty. *(Note: This is a simplification.)*\n\n> \"FVC","metadata":{}},{"cell_type":"code","source":"# Load train and test metadata\ntrain = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ntest = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/test.csv\")\n\n# Display basic info\nprint(f\"Total Recordings in Train Data: {len(train):,}\")\nprint(f\"Total Recordings in Test Data: {len(test):,}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:55:03.645828Z","iopub.execute_input":"2025-08-30T15:55:03.646107Z","iopub.status.idle":"2025-08-30T15:55:03.681491Z","shell.execute_reply.started":"2025-08-30T15:55:03.646082Z","shell.execute_reply":"2025-08-30T15:55:03.680745Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Display tables\nfrom IPython.display import display\n\ndisplay(train.head().style.set_caption('Head of Train Data'))\ndisplay(test.head().style.set_caption('Head of Test Data'))","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:55:05.507608Z","iopub.execute_input":"2025-08-30T15:55:05.507898Z","iopub.status.idle":"2025-08-30T15:55:05.614121Z","shell.execute_reply.started":"2025-08-30T15:55:05.507874Z","shell.execute_reply":"2025-08-30T15:55:05.613301Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.1 Checking for Missing Values ❔","metadata":{}},{"cell_type":"code","source":"missing_in_train = train.isnull().values.any()\nprint(f\"Are there any missing values in train? {missing_in_train}\")\nif missing_in_train:\n print(train.isnull().sum()) # Added to show counts if any","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:55:08.386406Z","iopub.execute_input":"2025-08-30T15:55:08.386706Z","iopub.status.idle":"2025-08-30T15:55:08.393126Z","shell.execute_reply.started":"2025-08-30T15:55:08.386679Z","shell.execute_reply":"2025-08-30T15:55:08.392264Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.2 Number of Unique Patients 👩‍⚕️👨‍⚕️","metadata":{}},{"cell_type":"code","source":"print(\"There are {} unique patients in Train Data.\".format(len(train[\"Patient\"].unique())), \"\\n\")\n\n# Recordings per Patient\ndata = train.groupby(by=\"Patient\")[\"Weeks\"].count().reset_index(drop=False)\n# Sort by Weeks\ndata = data.sort_values(['Weeks']).reset_index(drop=True)\nprint(\"Minimum number of entries are: {}\".format(data[\"Weeks\"].min()), \"\\n\" +\n \"Maximum number of entries are: {}\".format(data[\"Weeks\"].max()))\n\n# Plot\nplt.figure(figsize = (16, 6))\np = sns.barplot(data[\"Patient\"], data[\"Weeks\"], color=custom_colors[2])\n\nplt.title(\"Number of Entries per Patient\", fontsize = 17)\nplt.xlabel('Patient ID', fontsize=14)\nplt.ylabel('Number of Weeks/Entries', fontsize=14)\n\nplt.xticks([]); # Hide x-ticks if too many","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:55:09.618248Z","iopub.execute_input":"2025-08-30T15:55:09.618567Z","iopub.status.idle":"2025-08-30T15:55:10.440979Z","shell.execute_reply.started":"2025-08-30T15:55:09.61854Z","shell.execute_reply":"2025-08-30T15:55:10.440219Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.3 Patient demographics\n\n1. **Age**: As the competition description mentions, patients have between 50 and 90 years old, with the majority being between 60 and 70 yo.\n2. **Gender**: The vast majority of the patients are Males.\n3. **Smoking Status**: The vast majority are Ex-smokers. ~50 never smoked and less than 10 patients are still smoking.","metadata":{}},{"cell_type":"code","source":"# Select unique bio info for the patients\ndata = train.groupby(by=\"Patient\")[[\"Patient\", \"Age\", \"Sex\", \"SmokingStatus\"]].first().reset_index(drop=True)\n\n# Figure\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (16, 6))\n\na = sns.kdeplot(data[\"Age\"], ax=ax1, color=custom_colors[1], linewidth=6, linestyle=\"--\")\nb = sns.countplot(data[\"Sex\"], ax=ax2, palette=custom_colors[2:4])\nc = sns.countplot(data[\"SmokingStatus\"], ax=ax3, palette = custom_colors[4:7])\n\na.set_title(\"Patient Age Distribution\", fontsize=16)\nb.set_title(\"Sex Frequency\", fontsize=16)\nc.set_title(\"Smoking Status\", fontsize=16);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:55:13.604034Z","iopub.execute_input":"2025-08-30T15:55:13.604357Z","iopub.status.idle":"2025-08-30T15:55:14.151422Z","shell.execute_reply.started":"2025-08-30T15:55:13.604326Z","shell.execute_reply":"2025-08-30T15:55:14.150458Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.4 FVC & Percent ⚖\n\n
\n

FVC the recorded lung capacity in ml (how much air you can exhale in a maximal forced expiration effort)

\n

Percent a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics

\n
\n\n1. **FVC**: most values lie between 1,000 and 5,000 ml. There are also some very high outliers above 5,000 and little values that lie below 1,000.\n2. **Percent**: more than ~80% of the patients scored below 100%.","metadata":{}},{"cell_type":"code","source":"# Group by SmokingStatus and find max/min FVC and Percent\nsmoking_stats = train.groupby(\"SmokingStatus\")[[\"FVC\", \"Percent\"]].agg([\"min\", \"max\"])\nprint(\"FVC and Percent by Smoking Status:\")\nprint(smoking_stats)\n\n# Optional: Identify which patients have the extreme values\nmax_fvc = train.loc[train[\"FVC\"].idxmax()]\nmin_fvc = train.loc[train[\"FVC\"].idxmin()]\nprint(\"\\nPatient with Max FVC:\", max_fvc[[\"Patient\", \"FVC\", \"SmokingStatus\"]])\nprint(\"Patient with Min FVC:\", min_fvc[[\"Patient\", \"FVC\", \"SmokingStatus\"]])\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.kdeplot(train[\"FVC\"], ax=ax1, color=custom_colors[6], linewidth=6, linestyle=\"--\")\nb = sns.kdeplot(train[\"Percent\"], ax=ax2, color=custom_colors[4], linewidth=6, linestyle=\"-.\")\n\na.set_title(\"FVC Distribution\", fontsize=16)\nb.set_title(\"Percent Distribution\", fontsize=16);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-30T15:58:07.956004Z","iopub.execute_input":"2025-08-30T15:58:07.956342Z","iopub.status.idle":"2025-08-30T15:58:08.425119Z","shell.execute_reply.started":"2025-08-30T15:58:07.95631Z","shell.execute_reply":"2025-08-30T15:58:08.424394Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.5 Weeks📅\n\n> `Weeks`: the relative number of weeks pre/post the baseline CT (may be negative)","metadata":{}},{"cell_type":"code","source":"print(\"Minimum no. weeks before CT: {}\".format(train['Weeks'].min()), \"\\n\" +\n \"Maximum no. weeks after CT: {}\".format(train['Weeks'].max()))\n\nplt.figure(figsize = (16, 6))\n\na = sns.kdeplot(train['Weeks'], color=custom_colors[3], linewidth=8, linestyle=\"--\")\nplt.title(\"Number of weeks before/after the CT scan\", fontsize = 16)\nplt.xlabel(\"Weeks\", fontsize=14);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:56.052052Z","iopub.execute_input":"2025-08-29T07:49:56.052283Z","iopub.status.idle":"2025-08-29T07:49:56.312851Z","shell.execute_reply.started":"2025-08-29T07:49:56.052259Z","shell.execute_reply":"2025-08-29T07:49:56.311975Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.6 Correlations between Variables🧬\n\n* There is high correlation between FVC and Percent: when the volume of air increases, the Percent increases as well (when you exhale more, you get closer more to 100%)\n* There is no correlation between FVC/Percent and Age, meaning that Age has no influence on the volume of exhaled air.","metadata":{}},{"cell_type":"code","source":"# Compute Correlation\ncorr1, _ = pearsonr(train[\"FVC\"], train[\"Percent\"])\ncorr2, _ = pearsonr(train[\"FVC\"], train[\"Age\"])\ncorr3, _ = pearsonr(train[\"Percent\"], train[\"Age\"])\nprint(\"Pearson Corr FVC x Percent: {:.4}\".format(corr1), \"\\n\" +\n \"Pearson Corr FVC x Age: {:.0}\".format(corr2), \"\\n\" +\n \"Pearson Corr Percent x Age: {:.2}\".format(corr3))\n\n# Figure\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (16, 6))\n\na = sns.scatterplot(x = train[\"FVC\"], y = train[\"Percent\"], palette=[custom_colors[2], custom_colors[6]],\n hue = train[\"Sex\"], style = train[\"Sex\"], s=100, ax=ax1)\n\nb = sns.scatterplot(x = train[\"FVC\"], y = train[\"Age\"], palette=[custom_colors[2], custom_colors[6]],\n hue = train[\"Sex\"], style = train[\"Sex\"], s=100, ax=ax2)\n\nc = sns.scatterplot(x = train[\"Percent\"], y = train[\"Age\"], palette=[custom_colors[2], custom_colors[6]],\n hue = train[\"Sex\"], style = train[\"Sex\"], s=100, ax=ax3)\n\na.set_title(\"Correlation between FVC and Percent\", fontsize = 16)\na.set_xlabel(\"FVC\", fontsize = 14)\na.set_ylabel(\"Percent\", fontsize = 14)\n\nb.set_title(\"Correlation between FVC and Age\", fontsize = 16)\nb.set_xlabel(\"FVC\", fontsize = 14)\nb.set_ylabel(\"Age\", fontsize = 14)\n\nc.set_title(\"Correlation between Percent and Age\", fontsize = 16)\nc.set_xlabel(\"Percent\", fontsize = 14)\nc.set_ylabel(\"Age\", fontsize = 14);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:56.314261Z","iopub.execute_input":"2025-08-29T07:49:56.314611Z","iopub.status.idle":"2025-08-29T07:49:57.161509Z","shell.execute_reply.started":"2025-08-29T07:49:56.314574Z","shell.execute_reply":"2025-08-29T07:49:57.160769Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> **Unexpected Observation**: The highest FVC and Percent values are observed in individuals who currently smoke, while the lowest values are seen in those who never smoked. \n> **Important Caveat**: However, the proportion of current smokers in the dataset is very small. Therefore, **we cannot conclude** that smoking directly leads to higher FVC values based on this data alone.","metadata":{}},{"cell_type":"code","source":"# Figure\nf, (ax1, ax2) = plt.subplots(1,2, figsize = (16, 6))\n\na = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"FVC\"], ax=ax1, palette=custom_colors[0:4])\nb = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"Percent\"], ax=ax2, palette=custom_colors[4:7])\n\na.set_title(\"Mean FVC per Smoking Status\", fontsize=16)\nb.set_title(\"Mean Perc per Smoking Status\", fontsize=16);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:57.16276Z","iopub.execute_input":"2025-08-29T07:49:57.162989Z","iopub.status.idle":"2025-08-29T07:49:57.544875Z","shell.execute_reply.started":"2025-08-29T07:49:57.162966Z","shell.execute_reply":"2025-08-29T07:49:57.544076Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.7 Evolution of FVC in Patients 📉","metadata":{}},{"cell_type":"code","source":"# Create Time variable to count in ascending order the times the Patient has done a check in FVC\ndata_time = train.groupby(by=\"Patient\")[\"Weeks\"].count().reset_index()\ntrain[\"Time\"] = 0\n\nfor patient, times in zip(data_time[\"Patient\"], data_time[\"Weeks\"]):\n train.loc[train[\"Patient\"] == patient, 'Time'] = range(1, times+1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:57.546354Z","iopub.execute_input":"2025-08-29T07:49:57.546733Z","iopub.status.idle":"2025-08-29T07:49:57.808601Z","shell.execute_reply.started":"2025-08-29T07:49:57.546694Z","shell.execute_reply":"2025-08-29T07:49:57.807984Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# For graph purposes, keep only Patients that had a big difference in FVC between Time 1 and last Time\nmin_fvc = train[train[\"Time\"] == 1][[\"Patient\", \"FVC\"]].reset_index(drop=True)\n\nidx = train.groupby([\"Patient\"])[\"Weeks\"].transform(max) == train[\"Weeks\"]\nmax_fvc = train[idx][[\"Patient\", \"FVC\"]].reset_index(drop=True)\n\n# Compute difference and select only top patients with biggest difference\ndata = pd.merge(min_fvc, max_fvc, how=\"inner\", on=\"Patient\")\ndata[\"Dif\"] = data[\"FVC_x\"] - data[\"FVC_y\"]\n\n# Select only top n\nl = list(data.sort_values(\"Dif\", ascending=False).head(100)[\"Patient\"])\nx = train[train[\"Patient\"].isin(l)]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:57.809737Z","iopub.execute_input":"2025-08-29T07:49:57.809986Z","iopub.status.idle":"2025-08-29T07:49:57.832882Z","shell.execute_reply.started":"2025-08-29T07:49:57.809956Z","shell.execute_reply":"2025-08-29T07:49:57.832283Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize = (16, 6))\n\na = sns.lineplot(x = x[\"Time\"], y = x[\"FVC\"], hue = x[\"Patient\"], legend=False,\n palette=sns.color_palette(\"GnBu_d\", 100), size=1)\n\nplt.title(\"Patient FVC decrease on Weeks\", fontsize = 16)\nplt.xlabel(\"Weeks\", fontsize=14)\nplt.ylabel(\"FVC\", fontsize=14);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:57.834089Z","iopub.execute_input":"2025-08-29T07:49:57.834427Z","iopub.status.idle":"2025-08-29T07:49:59.152275Z","shell.execute_reply.started":"2025-08-29T07:49:57.834386Z","shell.execute_reply":"2025-08-29T07:49:59.151498Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. DICOM Data 📸 \n\n> [What are CT scans capturing?](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/167085)\n\n> **Dicom Files**: [A DICOM file is an image saved in the Digital Imaging and Communications in Medicine (DICOM) format.](https://fileinfo.com/extension/dicom#:~:text=A%20DICOM%20file%20is%20an,linked%20to%20a%20specific%20individual.)","metadata":{}},{"cell_type":"code","source":"# Create base director for Train .dcm files\ndirector = \"../input/osic-pulmonary-fibrosis-progression/train\"\n\n# Create path column with the path to each patient's CT\ntrain[\"Path\"] = director + \"/\" + train[\"Patient\"]\n\n# Create variable that shows how many CT scans each patient has\ntrain[\"CT_number\"] = 0\n\nfor k, path in enumerate(train[\"Path\"]):\n train[\"CT_number\"][k] = len(os.listdir(path))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:49:59.153621Z","iopub.execute_input":"2025-08-29T07:49:59.153839Z","iopub.status.idle":"2025-08-29T07:50:01.43385Z","shell.execute_reply.started":"2025-08-29T07:49:59.153818Z","shell.execute_reply":"2025-08-29T07:50:01.432895Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.1 Number of CT scans per Patient👴\n\n> Huge imbalance in the number of CT scans: half of the patients have less that 100 photos registered.","metadata":{}},{"cell_type":"code","source":"print(\"Minimum number of CT scans: {}\".format(train[\"CT_number\"].min()), \"\\n\" +\n \"Maximum number of CT scans: {:,}\".format(train[\"CT_number\"].max()))\n\n# Scans per Patient\ndata = train.groupby(by=\"Patient\")[\"CT_number\"].first().reset_index(drop=False)\n# Sort by Weeks\ndata = data.sort_values(['CT_number']).reset_index(drop=True)\n\n# Plot\nplt.figure(figsize = (16, 6))\np = sns.barplot(data[\"Patient\"], data[\"CT_number\"], color=custom_colors[5])\nplt.axvline(x=85, color=custom_colors[2], linestyle='--', lw=3)\n\nplt.title(\"Number of CT Scans per Patient\", fontsize = 17)\nplt.xlabel('Patient', fontsize=14)\nplt.ylabel('Frequency', fontsize=14)\n\nplt.text(86, 850, \"Median=94\", fontsize=13)\n\np.axes.get_xaxis().set_visible(False);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:01.435026Z","iopub.execute_input":"2025-08-29T07:50:01.435285Z","iopub.status.idle":"2025-08-29T07:50:02.266501Z","shell.execute_reply.started":"2025-08-29T07:50:01.435259Z","shell.execute_reply":"2025-08-29T07:50:02.265557Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.2 Visualise DICOM Info and Image\n\n> DICOM data can be extracted by using `dicom.dcmread()`\n\n> [What color maps should you use in medical visualisation?](http://noeskasmit.com/colormaps-in-medical-visualization/)","metadata":{}},{"cell_type":"code","source":"class bcolors:\n OKBLUE = '\\033[96m'\n OKGREEN = '\\033[92m'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:02.268039Z","iopub.execute_input":"2025-08-29T07:50:02.268428Z","iopub.status.idle":"2025-08-29T07:50:02.272747Z","shell.execute_reply.started":"2025-08-29T07:50:02.268388Z","shell.execute_reply":"2025-08-29T07:50:02.271892Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# https://gist.github.com/FedeMiorelli/640bbc66b2038a14802729e609abfe89\n# Create Turbo color map\nturbo_colormap_data = np.array(\n [[0.18995,0.07176,0.23217],\n [0.19483,0.08339,0.26149],\n [0.19956,0.09498,0.29024],\n [0.20415,0.10652,0.31844],\n [0.20860,0.11802,0.34607],\n [0.21291,0.12947,0.37314],\n [0.21708,0.14087,0.39964],\n [0.22111,0.15223,0.42558],\n [0.22500,0.16354,0.45096],\n [0.22875,0.17481,0.47578],\n [0.23236,0.18603,0.50004],\n [0.23582,0.19720,0.52373],\n [0.23915,0.20833,0.54686],\n [0.24234,0.21941,0.56942],\n [0.24539,0.23044,0.59142],\n [0.24830,0.24143,0.61286],\n [0.25107,0.25237,0.63374],\n [0.25369,0.26327,0.65406],\n [0.25618,0.27412,0.67381],\n [0.25853,0.28492,0.69300],\n [0.26074,0.29568,0.71162],\n [0.26280,0.30639,0.72968],\n [0.26473,0.31706,0.74718],\n [0.26652,0.32768,0.76412],\n [0.26816,0.33825,0.78050],\n [0.26967,0.34878,0.79631],\n [0.27103,0.35926,0.81156],\n [0.27226,0.36970,0.82624],\n [0.27334,0.38008,0.84037],\n [0.27429,0.39043,0.85393],\n [0.27509,0.40072,0.86692],\n [0.27576,0.41097,0.87936],\n [0.27628,0.42118,0.89123],\n [0.27667,0.43134,0.90254],\n [0.27691,0.44145,0.91328],\n [0.27701,0.45152,0.92347],\n [0.27698,0.46153,0.93309],\n [0.27680,0.47151,0.94214],\n [0.27648,0.48144,0.95064],\n [0.27603,0.49132,0.95857],\n [0.27543,0.50115,0.96594],\n [0.27469,0.51094,0.97275],\n [0.27381,0.52069,0.97899],\n [0.27273,0.53040,0.98461],\n [0.27106,0.54015,0.98930],\n [0.26878,0.54995,0.99303],\n [0.26592,0.55979,0.99583],\n [0.26252,0.56967,0.99773],\n [0.25862,0.57958,0.99876],\n [0.25425,0.58950,0.99896],\n [0.24946,0.59943,0.99835],\n [0.24427,0.60937,0.99697],\n [0.23874,0.61931,0.99485],\n [0.23288,0.62923,0.99202],\n [0.22676,0.63913,0.98851],\n [0.22039,0.64901,0.98436],\n [0.21382,0.65886,0.97959],\n [0.20708,0.66866,0.97423],\n [0.20021,0.67842,0.96833],\n [0.19326,0.68812,0.96190],\n [0.18625,0.69775,0.95498],\n [0.17923,0.70732,0.94761],\n [0.17223,0.71680,0.93981],\n [0.16529,0.72620,0.93161],\n [0.15844,0.73551,0.92305],\n [0.15173,0.74472,0.91416],\n [0.14519,0.75381,0.90496],\n [0.13886,0.76279,0.89550],\n [0.13278,0.77165,0.88580],\n [0.12698,0.78037,0.87590],\n [0.12151,0.78896,0.86581],\n [0.11639,0.79740,0.85559],\n [0.11167,0.80569,0.84525],\n [0.10738,0.81381,0.83484],\n [0.10357,0.82177,0.82437],\n [0.10026,0.82955,0.81389],\n [0.09750,0.83714,0.80342],\n [0.09532,0.84455,0.79299],\n [0.09377,0.85175,0.78264],\n [0.09287,0.85875,0.77240],\n [0.09267,0.86554,0.76230],\n [0.09320,0.87211,0.75237],\n [0.09451,0.87844,0.74265],\n [0.09662,0.88454,0.73316],\n [0.09958,0.89040,0.72393],\n [0.10342,0.89600,0.71500],\n [0.10815,0.90142,0.70599],\n [0.11374,0.90673,0.69651],\n [0.12014,0.91193,0.68660],\n [0.12733,0.91701,0.67627],\n [0.13526,0.92197,0.66556],\n [0.14391,0.92680,0.65448],\n [0.15323,0.93151,0.64308],\n [0.16319,0.93609,0.63137],\n [0.17377,0.94053,0.61938],\n [0.18491,0.94484,0.60713],\n [0.19659,0.94901,0.59466],\n [0.20877,0.95304,0.58199],\n [0.22142,0.95692,0.56914],\n [0.23449,0.96065,0.55614],\n [0.24797,0.96423,0.54303],\n [0.26180,0.96765,0.52981],\n [0.27597,0.97092,0.51653],\n [0.29042,0.97403,0.50321],\n [0.30513,0.97697,0.48987],\n [0.32006,0.97974,0.47654],\n [0.33517,0.98234,0.46325],\n [0.35043,0.98477,0.45002],\n [0.36581,0.98702,0.43688],\n [0.38127,0.98909,0.42386],\n [0.39678,0.99098,0.41098],\n [0.41229,0.99268,0.39826],\n [0.42778,0.99419,0.38575],\n [0.44321,0.99551,0.37345],\n [0.45854,0.99663,0.36140],\n [0.47375,0.99755,0.34963],\n [0.48879,0.99828,0.33816],\n [0.50362,0.99879,0.32701],\n [0.51822,0.99910,0.31622],\n [0.53255,0.99919,0.30581],\n [0.54658,0.99907,0.29581],\n [0.56026,0.99873,0.28623],\n [0.57357,0.99817,0.27712],\n [0.58646,0.99739,0.26849],\n [0.59891,0.99638,0.26038],\n [0.61088,0.99514,0.25280],\n [0.62233,0.99366,0.24579],\n [0.63323,0.99195,0.23937],\n [0.64362,0.98999,0.23356],\n [0.65394,0.98775,0.22835],\n [0.66428,0.98524,0.22370],\n [0.67462,0.98246,0.21960],\n [0.68494,0.97941,0.21602],\n [0.69525,0.97610,0.21294],\n [0.70553,0.97255,0.21032],\n [0.71577,0.96875,0.20815],\n [0.72596,0.96470,0.20640],\n [0.73610,0.96043,0.20504],\n [0.74617,0.95593,0.20406],\n [0.75617,0.95121,0.20343],\n [0.76608,0.94627,0.20311],\n [0.77591,0.94113,0.20310],\n [0.78563,0.93579,0.20336],\n [0.79524,0.93025,0.20386],\n [0.80473,0.92452,0.20459],\n [0.81410,0.91861,0.20552],\n [0.82333,0.91253,0.20663],\n [0.83241,0.90627,0.20788],\n [0.84133,0.89986,0.20926],\n [0.85010,0.89328,0.21074],\n [0.85868,0.88655,0.21230],\n [0.86709,0.87968,0.21391],\n [0.87530,0.87267,0.21555],\n [0.88331,0.86553,0.21719],\n [0.89112,0.85826,0.21880],\n [0.89870,0.85087,0.22038],\n [0.90605,0.84337,0.22188],\n [0.91317,0.83576,0.22328],\n [0.92004,0.82806,0.22456],\n [0.92666,0.82025,0.22570],\n [0.93301,0.81236,0.22667],\n [0.93909,0.80439,0.22744],\n [0.94489,0.79634,0.22800],\n [0.95039,0.78823,0.22831],\n [0.95560,0.78005,0.22836],\n [0.96049,0.77181,0.22811],\n [0.96507,0.76352,0.22754],\n [0.96931,0.75519,0.22663],\n [0.97323,0.74682,0.22536],\n [0.97679,0.73842,0.22369],\n [0.98000,0.73000,0.22161],\n [0.98289,0.72140,0.21918],\n [0.98549,0.71250,0.21650],\n [0.98781,0.70330,0.21358],\n [0.98986,0.69382,0.21043],\n [0.99163,0.68408,0.20706],\n [0.99314,0.67408,0.20348],\n [0.99438,0.66386,0.19971],\n [0.99535,0.65341,0.19577],\n [0.99607,0.64277,0.19165],\n [0.99654,0.63193,0.18738],\n [0.99675,0.62093,0.18297],\n [0.99672,0.60977,0.17842],\n [0.99644,0.59846,0.17376],\n [0.99593,0.58703,0.16899],\n [0.99517,0.57549,0.16412],\n [0.99419,0.56386,0.15918],\n [0.99297,0.55214,0.15417],\n [0.99153,0.54036,0.14910],\n [0.98987,0.52854,0.14398],\n [0.98799,0.51667,0.13883],\n [0.98590,0.50479,0.13367],\n [0.98360,0.49291,0.12849],\n [0.98108,0.48104,0.12332],\n [0.97837,0.46920,0.11817],\n [0.97545,0.45740,0.11305],\n [0.97234,0.44565,0.10797],\n [0.96904,0.43399,0.10294],\n [0.96555,0.42241,0.09798],\n [0.96187,0.41093,0.09310],\n [0.95801,0.39958,0.08831],\n [0.95398,0.38836,0.08362],\n [0.94977,0.37729,0.07905],\n [0.94538,0.36638,0.07461],\n [0.94084,0.35566,0.07031],\n [0.93612,0.34513,0.06616],\n [0.93125,0.33482,0.06218],\n [0.92623,0.32473,0.05837],\n [0.92105,0.31489,0.05475],\n [0.91572,0.30530,0.05134],\n [0.91024,0.29599,0.04814],\n [0.90463,0.28696,0.04516],\n [0.89888,0.27824,0.04243],\n [0.89298,0.26981,0.03993],\n [0.88691,0.26152,0.03753],\n [0.88066,0.25334,0.03521],\n [0.87422,0.24526,0.03297],\n [0.86760,0.23730,0.03082],\n [0.86079,0.22945,0.02875],\n [0.85380,0.22170,0.02677],\n [0.84662,0.21407,0.02487],\n [0.83926,0.20654,0.02305],\n [0.83172,0.19912,0.02131],\n [0.82399,0.19182,0.01966],\n [0.81608,0.18462,0.01809],\n [0.80799,0.17753,0.01660],\n [0.79971,0.17055,0.01520],\n [0.79125,0.16368,0.01387],\n [0.78260,0.15693,0.01264],\n [0.77377,0.15028,0.01148],\n [0.76476,0.14374,0.01041],\n [0.75556,0.13731,0.00942],\n [0.74617,0.13098,0.00851],\n [0.73661,0.12477,0.00769],\n [0.72686,0.11867,0.00695],\n [0.71692,0.11268,0.00629],\n [0.70680,0.10680,0.00571],\n [0.69650,0.10102,0.00522],\n [0.68602,0.09536,0.00481],\n [0.67535,0.08980,0.00449],\n [0.66449,0.08436,0.00424],\n [0.65345,0.07902,0.00408],\n [0.64223,0.07380,0.00401],\n [0.63082,0.06868,0.00401],\n [0.61923,0.06367,0.00410],\n [0.60746,0.05878,0.00427],\n [0.59550,0.05399,0.00453],\n [0.58336,0.04931,0.00486],\n [0.57103,0.04474,0.00529],\n [0.55852,0.04028,0.00579],\n [0.54583,0.03593,0.00638],\n [0.53295,0.03169,0.00705],\n [0.51989,0.02756,0.00780],\n [0.50664,0.02354,0.00863],\n [0.49321,0.01963,0.00955],\n [0.47960,0.01583,0.01055]])\n\ndef RGBToPyCmap(rgbdata):\n nsteps = rgbdata.shape[0]\n stepaxis = np.linspace(0, 1, nsteps)\n\n rdata=[]; gdata=[]; bdata=[]\n for istep in range(nsteps):\n r = rgbdata[istep,0]\n g = rgbdata[istep,1]\n b = rgbdata[istep,2]\n rdata.append((stepaxis[istep], r, r))\n gdata.append((stepaxis[istep], g, g))\n bdata.append((stepaxis[istep], b, b))\n\n mpl_data = {'red': rdata,\n 'green': gdata,\n 'blue': bdata}\n\n return mpl_data\n\n\nmpl_data = RGBToPyCmap(turbo_colormap_data)\nplt.register_cmap(name='turbo', data=mpl_data, lut=turbo_colormap_data.shape[0])\n\nmpl_data_r = RGBToPyCmap(turbo_colormap_data[::-1,:])\nplt.register_cmap(name='turbo_r', data=mpl_data_r, lut=turbo_colormap_data.shape[0])","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:02.274523Z","iopub.execute_input":"2025-08-29T07:50:02.274798Z","iopub.status.idle":"2025-08-29T07:50:02.325951Z","shell.execute_reply.started":"2025-08-29T07:50:02.274757Z","shell.execute_reply":"2025-08-29T07:50:02.325141Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"path = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/19.dcm\"\ndataset = dicom.dcmread(path)\n\nprint(bcolors.OKBLUE + \"Patient id.......:\", dataset.PatientID, \"\\n\" +\n \"Modality.........:\", dataset.Modality, \"\\n\" +\n \"Rows.............:\", dataset.Rows, \"\\n\" +\n \"Columns..........:\", dataset.Columns)\n\nplt.figure(figsize = (7, 7))\nplt.imshow(dataset.pixel_array, cmap=\"plasma\")\nplt.axis('off');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:02.327214Z","iopub.execute_input":"2025-08-29T07:50:02.327517Z","iopub.status.idle":"2025-08-29T07:50:02.61747Z","shell.execute_reply.started":"2025-08-29T07:50:02.327482Z","shell.execute_reply":"2025-08-29T07:50:02.616553Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.3 A breath-in + hold for a Patient\n\n> You can see how the lungs expand image by image.","metadata":{}},{"cell_type":"code","source":"patient_dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\ndatasets = []\n\n# First Order the files in the dataset\nfiles = []\nfor dcm in list(os.listdir(patient_dir)):\n files.append(dcm) \nfiles.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n# Read in the Dataset\nfor dcm in files:\n path = patient_dir + \"/\" + dcm\n datasets.append(dicom.dcmread(path))\n\n# Plot the images\nfig=plt.figure(figsize=(16, 6))\ncolumns = 10\nrows = 3\n\nfor i in range(1, columns*rows +1):\n img = datasets[i-1].pixel_array\n fig.add_subplot(rows, columns, i)\n plt.imshow(img, cmap=\"plasma\")\n plt.title(i, fontsize = 9)\n plt.axis('off');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:02.618911Z","iopub.execute_input":"2025-08-29T07:50:02.619237Z","iopub.status.idle":"2025-08-29T07:50:06.574685Z","shell.execute_reply.started":"2025-08-29T07:50:02.619204Z","shell.execute_reply":"2025-08-29T07:50:06.573872Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.4 GIF from Images 🌔🌕🌖\n\n>Patients have various number of CT scans: the more scans/patient, the more information we have about well ... their lungs. Here we can see that when patients have low number of scans (~ 12) only a quick \"inhale\" is observed, whereas when we have 80+ scans the details are much more enhanced.","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nfrom IPython.display import Image as show_gif\nimport scipy.misc\nimport matplotlib","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:06.575702Z","iopub.execute_input":"2025-08-29T07:50:06.57594Z","iopub.status.idle":"2025-08-29T07:50:06.579726Z","shell.execute_reply.started":"2025-08-29T07:50:06.5759Z","shell.execute_reply":"2025-08-29T07:50:06.578808Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def create_gif(number_of_CT = 87):\n \"\"\"Selects a patient with the specified number of CT scans and creates a GIF.\"\"\"\n\n # Select patient with exact number of CT scans\n patient_row = train[train[\"CT_number\"] == number_of_CT].sample(random_state=1)\n if patient_row.empty:\n raise ValueError(f\"No patient found with exactly {number_of_CT} CT scans.\")\n patient = patient_row[\"Patient\"].values[0]\n \n # === Read DICOM files ===\n patient_dir = f\"../input/osic-pulmonary-fibrosis-progression/train/{patient}\"\n files = sorted(os.listdir(patient_dir), key=lambda f: int(re.sub(r'\\D', '', f))) # Combined sorting\n\n datasets = []\n for dcm in files:\n path = os.path.join(patient_dir, dcm)\n try:\n datasets.append(dicom.dcmread(path))\n except Exception as e:\n print(f\"Error reading {path}: {e}\")\n continue \n \n \n # === Save as .png ===\n # Create directory to save the png files (if not exists)\n png_dir = f\"png_{patient}\"\n os.makedirs(png_dir, exist_ok=True)\n \n # Save images to PNG\n for i, ds in enumerate(datasets):\n img = ds.pixel_array\n plt.imsave(os.path.join(png_dir, f'img_{i}.png'), img, cmap='gray')\n \n # === Create GIF ===\n png_files = sorted(os.listdir(png_dir), key=lambda f: int(re.sub(r'\\D', '', f)))\n frames = [Image.open(os.path.join(png_dir, f)) for f in png_files]\n\n if not frames:\n raise ValueError(\"No frames generated.\")\n\n frames[0].save(f'gif_{patient}.gif',\n format='GIF',\n append_images=frames[1:],\n save_all=True, \n duration=200,\n loop=0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:06.580901Z","iopub.execute_input":"2025-08-29T07:50:06.581198Z","iopub.status.idle":"2025-08-29T07:50:06.595625Z","shell.execute_reply.started":"2025-08-29T07:50:06.581173Z","shell.execute_reply":"2025-08-29T07:50:06.59482Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Create and compare GIFs 🌖🌗🌘","metadata":{}},{"cell_type":"code","source":"create_gif(number_of_CT=12)\ncreate_gif(number_of_CT=30)\ncreate_gif(number_of_CT=87)\n\nprint(\"First file len:\", len(os.listdir(\"../working/png_ID00165637202237320314458\")), \"\\n\" +\n \"Second file len:\", len(os.listdir(\"../working/png_ID00199637202248141386743\")), \"\\n\" +\n \"Third file len:\", len(os.listdir(\"../working/png_ID00340637202287399835821\")))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:06.596919Z","iopub.execute_input":"2025-08-29T07:50:06.597426Z","iopub.status.idle":"2025-08-29T07:50:21.13435Z","shell.execute_reply.started":"2025-08-29T07:50:06.597386Z","shell.execute_reply":"2025-08-29T07:50:21.133567Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 12 CT scans GIF:","metadata":{}},{"cell_type":"code","source":"show_gif(filename=\"./gif_ID00165637202237320314458.gif\", format='png', width=400, height=400)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:21.135649Z","iopub.execute_input":"2025-08-29T07:50:21.135915Z","iopub.status.idle":"2025-08-29T07:50:21.170534Z","shell.execute_reply.started":"2025-08-29T07:50:21.135878Z","shell.execute_reply":"2025-08-29T07:50:21.168528Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 30 CT scans GIF:","metadata":{}},{"cell_type":"code","source":"show_gif(filename=\"./gif_ID00340637202287399835821.gif\", format='png', width=400, height=400)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:21.172085Z","iopub.execute_input":"2025-08-29T07:50:21.172402Z","iopub.status.idle":"2025-08-29T07:50:21.292053Z","shell.execute_reply.started":"2025-08-29T07:50:21.172371Z","shell.execute_reply":"2025-08-29T07:50:21.29116Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 87 CT scans GIF:","metadata":{}},{"cell_type":"code","source":"show_gif(filename=\"./gif_ID00199637202248141386743.gif\", format='png', width=400, height=400)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:21.293395Z","iopub.execute_input":"2025-08-29T07:50:21.293714Z","iopub.status.idle":"2025-08-29T07:50:21.542951Z","shell.execute_reply.started":"2025-08-29T07:50:21.293672Z","shell.execute_reply":"2025-08-29T07:50:21.542011Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.5 DICOM Lung Mask🎭\n\n
\nReference: DICOM Processing Segmentation Visualization in Python\n
\n\n**Mask on Lungs Purpose:**\n* Segmentation is part of the preprocessing method\n* Has the purpose of auto-detecting the boundaries surrounding a volume of interest (our case is the lungs)\n* Drawbacks: be sure you don't exclude important parts (like lesions)","metadata":{}},{"cell_type":"code","source":"# https://www.raddq.com/dicom-processing-segmentation-visualization-in-python/\n\ndef make_lungmask(img, display=False):\n row_size= img.shape[0]\n col_size = img.shape[1]\n \n mean = np.mean(img)\n std = np.std(img)\n img = img-mean\n img = img/std\n \n # Find the average pixel value near the lungs\n # to renormalize washed out images\n middle = img[int(col_size/5):int(col_size/5*4),int(row_size/5):int(row_size/5*4)] \n mean = np.mean(middle) \n max = np.max(img)\n min = np.min(img)\n \n # To improve threshold finding, moving the underflow and overflow on the pixel spectrum\n img[img==max]=mean\n img[img==min]=mean\n \n # Using Kmeans to separate foreground (soft tissue / bone) and background (lung/air)\n try:\n kmeans = KMeans(n_clusters=2, n_init=10).fit(np.reshape(middle,[np.prod(middle.shape),1]))\n except ValueError:\n return np.zeros_like(img)\n centers = sorted(kmeans.cluster_centers_.flatten())\n threshold = np.mean(centers)\n thresh_img = np.where(imgrow_size/5 and B[2]Lungs are quite visible, but on images at the beginning and towards the end the lung dissapears completely.","metadata":{}},{"cell_type":"code","source":"patient_dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\ndatasets = []\n\n# First Order the files in the dataset\nfiles = []\nfor dcm in list(os.listdir(patient_dir)):\n files.append(dcm) \nfiles.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n# Read in the Dataset\nfor dcm in files:\n path = patient_dir + \"/\" + dcm\n datasets.append(dicom.dcmread(path))\n \nimgs = []\nfor data in datasets:\n img = data.pixel_array\n imgs.append(img)\n \n \n# Show masks\nfig=plt.figure(figsize=(16, 6))\ncolumns = 10\nrows = 3\n\nfor i in range(1, columns*rows +1):\n img = make_lungmask(datasets[i-1].pixel_array)\n fig.add_subplot(rows, columns, i)\n plt.imshow(img, cmap=\"gray\")\n plt.title(i, fontsize = 9)\n plt.axis('off');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:22.822883Z","iopub.execute_input":"2025-08-29T07:50:22.82322Z","iopub.status.idle":"2025-08-29T07:50:36.393384Z","shell.execute_reply.started":"2025-08-29T07:50:22.823187Z","shell.execute_reply":"2025-08-29T07:50:36.392485Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. 🏃‍♂️RAPIDS - crazy fast🏃‍♂️ \n\n> In this part we will use [rapids.ai](https://rapids.ai/) to extract and preprocess the *metadata from the dicom files* as well as the *data available in the train.csv*.","metadata":{}},{"cell_type":"code","source":"# RAPIDS Installation (from https://www.kaggle.com/cdeotte/rapids)\n\nimport sys\n!cp ../input/rapids/rapids.21.06 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/\n\n# Verify\ntry:\n import cudf\n print(\"RAPIDS loaded successfully.\")\nexcept ImportError:\n print(\"RAPIDS failed; falling back to pandas for DataFrame.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:50:36.394916Z","iopub.execute_input":"2025-08-29T07:50:36.395315Z","iopub.status.idle":"2025-08-29T07:52:17.978277Z","shell.execute_reply.started":"2025-08-29T07:50:36.395273Z","shell.execute_reply":"2025-08-29T07:52:17.977411Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Libraries 📚⬇","metadata":{}},{"cell_type":"code","source":"# Import RAPIDS libraries (with fallback)\ntry:\n import cudf\nexcept ImportError:\n cudf = pd # Fallback to pandas if cudf fails\n print(\"Using pandas as fallback for cudf.\")\nimport tqdm","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:52:17.979909Z","iopub.execute_input":"2025-08-29T07:52:17.980266Z","iopub.status.idle":"2025-08-29T07:52:17.984589Z","shell.execute_reply.started":"2025-08-29T07:52:17.980223Z","shell.execute_reply":"2025-08-29T07:52:17.983739Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4.1 Extracting Metadata from DICOM files\n\n> The `.get()` function is used overall for .dcm files that don't have all columns available. So, to not skip the entire file, we extract only what's available and fill in the rest with *None*.\n\n*Inspo from [Extract Metadata and Resize Notebook](https://www.kaggle.com/trsekhar123/nb-to-extract-metadata-and-resize-images-train)*","metadata":{}},{"cell_type":"code","source":"def get_observation_data(path):\n \"\"\"Get information from the .dcm files.\n path: complete path to the .dcm file\"\"\"\n\n image_data = dicom.read_file(path)\n \n # Dictionary to store the information from the image\n observation_data = {\n \"FileNumber\" : path.split(\"/\")[5],\n \"Rows\" : image_data.get(\"Rows\"),\n \"Columns\" : image_data.get(\"Columns\"),\n \"PatientID\" : image_data.get(\"PatientID\"),\n \"BodyPartExamined\" : image_data.get(\"BodyPartExamined\"),\n \"RotationDirection\" : image_data.get(\"RotationDirection\"),\n \"ConvolutionKernel\" : image_data.get(\"ConvolutionKernel\"),\n \"PatientPosition\" : image_data.get(\"PatientPosition\"),\n \"PhotometricInterpretation\" : image_data.get(\"PhotometricInterpretation\"),\n \"Modality\" : image_data.get(\"Modality\"),\n \"StudyInstanceUID\" : image_data.get(\"StudyInstanceUID\"),\n \"PixelPaddingValue\" : image_data.get(\"PixelPaddingValue\"),\n \"SamplesPerPixel\" : image_data.get(\"SamplesPerPixel\"),\n \"BitsAllocated\" : image_data.get(\"BitsAllocated\"),\n \"BitsStored\" : image_data.get(\"BitsStored\"),\n \"HighBit\" : image_data.get(\"HighBit\"),\n \"PixelRepresentation\" : image_data.get(\"PixelRepresentation\"),\n \"RescaleType\" : image_data.get(\"RescaleType\"),\n }\n\n # Integer columns\n int_columns = [\"SliceThickness\", \"KVP\", \"DistanceSourceToDetector\", \n \"DistanceSourceToPatient\", \"GantryDetectorTilt\", \"TableHeight\", \n \"XRayTubeCurrent\", \"GeneratorPower\", \"WindowCenter\", \"WindowWidth\", \n \"SliceLocation\", \"RescaleIntercept\", \"RescaleSlope\"]\n for k in int_columns:\n observation_data[k] = int(image_data.get(k)) if k in image_data else None\n\n # String columns\n str_columns = [\"ImagePositionPatient\", \"ImageOrientationPatient\", \"ImageType\", \"PixelSpacing\"]\n for k in str_columns:\n observation_data[k] = str(image_data.get(k)) if k in image_data else None\n\n \n return observation_data","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:52:17.989833Z","iopub.execute_input":"2025-08-29T07:52:17.990285Z","iopub.status.idle":"2025-08-29T07:52:18.891597Z","shell.execute_reply.started":"2025-08-29T07:52:17.990238Z","shell.execute_reply":"2025-08-29T07:52:18.890717Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Test the function to see if it works","metadata":{}},{"cell_type":"code","source":"p = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/10.dcm\"\nexample = get_observation_data(p)\n\n# example","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:52:18.893315Z","iopub.execute_input":"2025-08-29T07:52:18.89368Z","iopub.status.idle":"2025-08-29T07:52:18.909542Z","shell.execute_reply.started":"2025-08-29T07:52:18.893642Z","shell.execute_reply":"2025-08-29T07:52:18.908904Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get full paths for the images\npaths = []\nfor path in train[\"Path\"]:\n for doc in os.listdir(path):\n paths.append(path + \"/\" + doc)\n \n# How many paths?\nprint(\"There are {:,} paths in total.\".format(len(paths)))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:52:18.910827Z","iopub.execute_input":"2025-08-29T07:52:18.911159Z","iopub.status.idle":"2025-08-29T07:52:19.949957Z","shell.execute_reply.started":"2025-08-29T07:52:18.911124Z","shell.execute_reply":"2025-08-29T07:52:19.949064Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"*Discussion💬 here: [rapids question](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/172373)*\n\n> ❗ **Note**: Code to be revised; maybe make it faster. Also, I'm using `pd.DataFrame()` as `cudf.DataFrame()` is not working properly (it's extracting only the keys of the dict, not the values themselves)","metadata":{}},{"cell_type":"code","source":"# Create a dictionary with the data\nexceptions = 0\ndicts = []\n\nfor path in tqdm.tqdm(paths):\n # Get info in dict format\n try:\n d = get_observation_data(path)\n dicts.append(d)\n except Exception as e:\n exceptions += 1\n continue\n\n# Convert into a DataFrame (use cudf if available, else pd)\ntry:\n meta_train_data = cudf.DataFrame(data=dicts, columns=example.keys())\nexcept NameError: # If cudf not defined\n meta_train_data = pd.DataFrame(data=dicts, columns=example.keys())\n\n# Export information to a .csv\nmeta_train_data.to_csv(\"meta_train.csv\", index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-29T07:52:19.951322Z","iopub.execute_input":"2025-08-29T07:52:19.951726Z","execution_failed":"2025-08-29T07:53:11.29Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Exceptions: {}\".format(exceptions))\nmeta_train_data.head()","metadata":{"trusted":true,"execution":{"execution_failed":"2025-08-29T07:53:11.29Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4.2 Analyze Metadata\n\n### Missing Values\n\n> `PixelPaddingValue` and `Rescale Type` columns are more than 80% empty, so they will be erased. ","metadata":{}},{"cell_type":"code","source":"# Inspect the Missing Columns\nprint(\"---Missing Columns---\", \"\\n\" +\n \"Pixel Padding Value missing percentage: {:.2f}\".format(meta_train_data[\"PixelPaddingValue\"].isna().sum()/\n len(meta_train_data[\"PixelPaddingValue\"])), \"\\n\" +\n \"Rescale Type missing percentage: {:.2f}\".format(meta_train_data[\"RescaleType\"].isna().sum()/\n len(meta_train_data[\"RescaleType\"])), \"\\n\" +\n \"Dist Source to Detector missing percentage: {:.2f}\".format(meta_train_data[\"DistanceSourceToDetector\"].isna().sum()/\n len(meta_train_data[\"DistanceSourceToDetector\"])), \"\\n\" +\n \"Dist Source to Patient missing percentage: {:.2f}\".format(meta_train_data[\"DistanceSourceToPatient\"].isna().sum()/\n len(meta_train_data[\"DistanceSourceToPatient\"])), \"\\n\" +\n \"Gantry Detector Tilt missing percentage: {:.2f}\".format(meta_train_data[\"GantryDetectorTilt\"].isna().sum()/\n len(meta_train_data[\"GantryDetectorTilt\"])), \"\\n\" +\n \"Generator Power missing percentage: {:.2f}\".format(meta_train_data[\"GeneratorPower\"].isna().sum()/\n len(meta_train_data[\"GeneratorPower\"])), \"\\n\" +\n \"Slice Location missing percentage: {:.2f}\".format(meta_train_data[\"SliceLocation\"].isna().sum()/\n len(meta_train_data[\"SliceLocation\"])), \"\\n\" +\n \"Img Position Patient missing percentage: {:.2f}\".format(meta_train_data[\"ImagePositionPatient\"].isna().sum()/\n len(meta_train_data[\"ImagePositionPatient\"])), \"\\n\" +\n \"Img Orientation Patient missing percentage: {:.2f}\".format(meta_train_data[\"ImageOrientationPatient\"].isna().sum()/\n len(meta_train_data[\"ImageOrientationPatient\"])))\n\n# Create figure\nax = msno.matrix(meta_train_data, color=(152/255, 241/255, 198/255), fontsize=10);\nplt.title(\"Metadata Missing Values\", fontsize = 25);","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"execution_failed":"2025-08-29T07:53:11.291Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"*Dropping the 2 columns*","metadata":{}},{"cell_type":"code","source":"meta_train_data.drop(['PixelPaddingValue', 'RescaleType'], axis=1, inplace=True)\n\n# Export information to a .csv\nmeta_train_data.to_csv(\"meta_train.csv\", index=False)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-08-29T07:53:11.291Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# ✅ Summary \n- Performed EDA on metadata (e.g., distributions of age, sex, smoking, FVC trends).\n- Handled missing values and visualized them.\n- Preprocessed DICOM images (conversion to PNG/GIF, normalization, lung masking).\n- Extracted DICOM metadata efficiently *(with RAPIDS)* **using Python tools.**\n\n**Next Steps**:\n- Feature engineering (e.g., derive new features from metadata).\n- Modeling (e.g., use preprocessed images for CNN-based FVC prediction).\n- Experiment with segmentation models for lung volume analysis.","metadata":{}},{"cell_type":"markdown","source":"# References📇\n* [What is Pulmonary Fibrosis?](https://www.pulmonaryfibrosis.org/life-with-pf/about-pf)\n* [Confidence Clarification](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166753)\n* [What are CT scans capturing?](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/167085)\n","metadata":{}}]}