{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Data and imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nfrom scipy import stats\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom collections.abc import Iterable\nimport plotly.graph_objects as go\npd.options.display.max_rows = 100\ntrain = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\nsub = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/sample_submission.csv')\nprint('train shape', train.shape, 'test shape', test.shape)\n\n\n# Plot a timeseries line Weeks x FVC (or %)\ndef plot_timeserie(patient_id, var='FVC'):\n    data = train[train.Patient==patient_id]\n    sex = data.loc[data.index[0], 'Sex']\n    smoke = data.loc[data.index[0], 'SmokingStatus']\n    print(\"Week: {} to {} | Age: {} | Sex: {} | Smoke: {}\".format(\n        data.Weeks.min(), data.Weeks.max(), data.Age.max(), sex, smoke))\n\n    fig, ax = plt.subplots(figsize=(12, 4))\n    p1 = sns.lineplot(x='Weeks', y=var, data=data)\n    for i in data.index:\n        s = \"w{}: {:.0f}\".format(data.loc[i, 'Weeks'], data.loc[i, var])\n        ax.text(data.loc[i, 'Weeks'], data.loc[i, var], s)\n    \n\n# Plot FVC for multiple patients (one line for each)\ndef plot_timeseries(patient_ids, var='FVC'):\n    if not isinstance(patient_ids, Iterable):\n        patient_ids = [patient_ids]\n    \n    plt.figure(figsize=(12, 6))\n    data = train[train.Patient.isin(patient_ids)]\n    p1 = sns.lineplot(x='Weeks', y=var, data=data, hue='Patient')\n    p1.get_legend().remove()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Overview"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patients = train.Patient.unique()\nprint(\"There are\", len(train), \"records in the training set\")\nprint(\"There are\", len(patients), \"unique patients in the training set\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So 1549 exams (FVC measurements) for 176 different patients"},{"metadata":{},"cell_type":"markdown","source":"# 2. FVC Curve for each patient\n\nThe right column has each patient id (last five numbers); double-click to select a single patient, them a single click will add more patients to the visualization. Finally, using double-click again will return to all patients."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# create curves and buttons for menu\ntraces = []\n\nfor i, patient_id in enumerate(patients):\n    tmp = train[train.Patient == patient_id]\n    traces.append(go.Scatter(x=tmp.Weeks, y=tmp.FVC, text=tmp.FVC, mode='lines+markers', name=patient_id[-5:]))\n    vx = [i == j for j in range(len(patients))]\n\n# create plot\nfig = go.Figure()\nfig.add_traces(traces)\nfig.update_layout(title_text=\"FVC Curve\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.1 Outliers\n\nLooking at each patient's individual curve we can see many outliers. There is a high level of uncertainty related to FVC and disease progression. Some examples:"},{"metadata":{"trusted":true},"cell_type":"code","source":"outlier_patients = [\n    \"ID00076637202199015035026\", \"ID00077637202199102000916\", \"ID00082637202201836229724\",\n    \"ID00117637202212360228007\", \"ID00119637202215426335765\", \"ID00126637202218610655908\",\n    \"ID00135637202224630271439\", \"ID00165637202237320314458\", \"ID00170637202238079193844\",\n    \"ID00172637202238316925179\", \"ID00197637202246865691526\", \"ID00218637202258156844710\",\n    \"ID00235637202261451839085\", \"ID00288637202279148973731\", \"ID00323637202285211956970\",\n    \"ID00337637202286839091062\", \"ID00355637202295106567614\"\n]\nfor patient_id in outlier_patients:\n    plot_timeserie(patient_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata = train.groupby('Patient').first()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\n_ = sns.distplot(metadata.Age, bins=20).set_title(\"Age distribution\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\n_ = sns.countplot(metadata.SmokingStatus, hue=metadata.Sex).set_title(\"Smoking status and sex distribution\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(metadata.Sex.value_counts(normalize=False))\nprint(metadata.SmokingStatus.value_counts(normalize=False))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Age distribution according to Smoking Status**\n\nBlue: Ex-Smokers\n\nYellow: Never Smoked\n\nGreen: Currently smokes"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\n_ = sns.distplot(metadata[metadata.SmokingStatus == 'Ex-smoker'].Age, hist=False).set_title(\"Age by Smoking Status\")\n_ = sns.distplot(metadata[metadata.SmokingStatus == 'Never smoked'].Age, hist=False)\n_ = sns.distplot(metadata[metadata.SmokingStatus == 'Currently smokes'].Age, hist=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. FVC Distribution"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\n_ = sns.distplot(train.FVC, bins=20).set_title(\"FVC distribution\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\n_ = sns.distplot(train[train.Sex == 'Male'].FVC, bins=20).set_title(\"Male (blue) vs Female (orange) FVC distribution\")\n_ = sns.distplot(train[train.Sex == 'Female'].FVC, bins=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**FVC distribution according to Smoking Status**\n\nBlue: Ex-Smokers\n\nYellow: Never Smoked\n\nGreen: Currently smokes"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\n_ = sns.distplot(train[train.SmokingStatus == 'Ex-smoker'].FVC, hist=False).set_title(\"FVC by Smoking Status\")\n_ = sns.distplot(train[train.SmokingStatus == 'Never smoked'].FVC, hist=False)\n_ = sns.distplot(train[train.SmokingStatus == 'Currently smokes'].FVC, hist=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Multiple FVC Curves"},{"metadata":{},"cell_type":"markdown","source":"FVC evolution for all 176 patients: one line for each patient"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_timeseries(patients, var='FVC')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Same, but using Percent column"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_timeseries(patients, var='Percent')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Smoking Patients"},{"metadata":{"trusted":true},"cell_type":"code","source":"smoking_patients = train[train.SmokingStatus == 'Currently smokes'].Patient.unique()\nplot_timeseries(smoking_patients, var='FVC')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Trying to group patients by the value of the first FVC measure"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get value of first FVC measure for each patient\nidx = train.groupby('Patient').Weeks.idxmin()\nfirst_fvc = train.loc[idx, ['Patient', 'FVC']]\n# Divide in bins (quantiles)\nbin_edges = stats.mstats.mquantiles(first_fvc.FVC, [0.1*i for i in range(1, 11)])\nbin_edges","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(10):\n    if i == 0:\n        tmp = first_fvc[(first_fvc.FVC <= bin_edges[i])]\n    else:\n        tmp = first_fvc[(first_fvc.FVC <= bin_edges[i]) & (first_fvc.FVC > bin_edges[i-1])]\n    plot_timeseries(tmp.Patient, var='FVC')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}