{"cells":[{"metadata":{"collapsed":true,"pycharm":{"name":"#%% md\n"}},"cell_type":"markdown","source":"# Introduction\n\nIdiopathic interstitial pneumonia (IIP) is a group of interstitial lung diseases (ILD) of\nunknown etiology, differing from each other in the pathomorphological type of non-infectious\ninflammation and fibrosis, mainly in the interstitium of the lung, as well as a variant of\nthe clinical course and prognosis [1]. The differential diagnosis of IIP is largely based\non a set of uniform criteria and guidelines that have been proposed by the American Thoracic\nSociety and the European Respiratory Society (ATS / ERS). Early diagnosis is critical for\nmaking policy decisions about treatment, especially for the idiopathic pulmonary fibrosis\n(IPF), while misdiagnosis can lead to life-threatening complications [2].\n\nIt should be noted that IPF is the most common form of IIP - it accounts for 80–90% of all\ncases of idiopathic pneumonia. The prevalence of IPF reaches about 20 cases per 100 thousand\namong men and 13 among women [1]. Thus, the ratio depending on the gender is 1.5: 1 (male:\nfemale).  The incidence of IPF reaches 11.3 cases/year per 100 thousand in men and 7.1 in\nwomen, increasing with age. Approximately 2/3 of patients with IPF are over 60 years old.\nMortality from IPF is higher in the older age group and averages 3 per 100 thousand population\nand the median survival rate ranges from 2.3 to 5 years [1]. Risk factors include cigarette\nsmoking, certain viral infections, and a family history of the condition [4].\n\nIPF usually manifests itself as progressive dyspnea and an unproductive cough, which is often\nparoxysmal and refractory to antitussives. Deformation of the nail phalanges is observed in\n25-50% of patients. Signs of chronic cor pulmonale (peripheral edema) can be observed in the\nlater stages of the disease [1].\n\nIn most patients, the period from the onset of symptoms to a visit to the doctor is more than\n6 months. The clinical course of IPF is characterized by a gradual deterioration in the\ncondition of patients, but often there is a sharp progression associated with a viral\ninfection, the development of pneumonia or diffuse alveolar damage. On radiography of the\nlungs, peripheral reticular shadows are most often observed, mainly in the basal regions,\nassociated with the formation of cellular changes in the lung tissue and a decrease in the\nvolume of the lower lobes. However, an average of 16% of patients with histologically proven\nIPF may have an unchanged radiographic picture [1].\n\nHigh-resolution computed tomography (HRCT) reveals reticular changes, usually bilateral,\npartly associated with traction bronchiectasis. \"Cell lung\" are often observed. Ground-glass\npatches are less common than reticular changes. Disturbances in architectonics, reflecting\npulmonary fibrosis, are characteristic. Pathological changes are characterized by heterogeneity\nand are localized mainly in the peripheral and basal regions. In a number of studies carried\nout during the treatment of patients, it was found that the \"ground glass\" zones may decrease.\nHowever, the most characteristic is the progression of fibrosis with the formation of a\n\"cell lung\". The accuracy of IPF diagnostics according to HRCT data reaches 90% [1].\n\nCorrect identification of patterns in HRCT images in conjunction with the Fleischner\nSociety guidelines [5] is central to the diagnosis and further management of patients\nwith ILD [2]. The available treatments slow down but do not reverse the disease process,\nso today there is an objective need for methods to accurately detect early interstitial\nchanges before their progression and predict rate of progression. The main problem today is\nthe early diagnosis of IPF. The solution to this particular problem will significantly\nincrease the life expectancy of patients [3].\n\nThe purpose of this work is to predict a patient’s severity of decline in lung function\nbased on a CT scan of their lungs. In this paper in first chapter we will analyse full\nhistory of clinical information of patients and their baseline CT. In the next chapter\nis described methodology of prediction patient's FVC measurement for every possible week.\n\n# 1. Analysis of datasets","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport matplotlib\nimport numpy as np\nfrom matplotlib import pyplot as plt\nimport matplotlib.ticker as ticker\nimport seaborn as sns\nimport math\n\n%matplotlib inline\nmatplotlib.rcParams.update({'font.size': 15})\ncolors = [\"#003f5c\", \"#bc5090\", \"#ffa600\", \"#127681\", \"#ea5455\"]\ncustomPalette = sns.set_palette(sns.color_palette(colors))\nsns.set_style(\"whitegrid\")\nfilepath = '../input/osic-pulmonary-fibrosis-progression/';\ntest = pd.read_csv(filepath + 'test.csv')\ntrain = pd.read_csv(filepath + 'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"pycharm":{"name":"#%% md\n"}},"cell_type":"markdown","source":"First, let's highlight the main features of train dataset.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"def show_dataset_common_info(dataset, label, fignumber):\n    df = dataset[['Patient', 'Sex', 'SmokingStatus']].drop_duplicates()\n\n    _, axes = plt.subplots(1, 2, figsize=(12,5))\n    axes[0].set_title('Fig.%s. Gender and patient habits of %s dataset' % (fignumber, label))\n\n    df.groupby(['SmokingStatus'])['Patient'].count().plot.pie(\n        label='', autopct='%.2f%%', labeldistance=None, ax=axes[1], textprops={'color':\"w\"})\n    axes[1].legend(loc='upper right')\n\n    df.groupby(['Sex'])['Patient'].count().plot.pie(\n        label='', autopct='%.2f%%', labeldistance=None, ax=axes[0], textprops={'color':\"w\"})\n    axes[0].legend(loc='upper right')\n\n    plt.show()\n    print('Total patient count in %s dataset: %s' % (label, df['Patient'].count()))\n\nshow_dataset_common_info(train, 'train', '1')","execution_count":null,"outputs":[]},{"metadata":{"pycharm":{"name":"#%% md\n"}},"cell_type":"markdown","source":"### Intermediate conclusion:\n* In train dataset among patients, there are 3.7 fewer women than men,\n* Most patients are ex-smokers.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"df = train.groupby(['Sex', 'SmokingStatus'])['Patient'].unique().reset_index()\ndf['Patient'] = df['Patient'].apply(lambda x: len(x))\nax = sns.catplot(x=\"SmokingStatus\", y=\"Patient\",\n                 kind=\"bar\", data=df,\n                 hue=\"Sex\", palette=customPalette,\n                 height=4, aspect=2.5)\n\nstart, end = ax.axes[0,0].get_ylim()\n\nplt.title('Fig.2. The number of patients in different smoking status', fontsize=20)\nplt.xlabel('', fontsize=19)\nplt.ylabel('Count', fontsize=19)\nplt.yticks(np.arange(start, end, max(int(math.fabs(end-start)/5),1)))\nax.axes[0,0].yaxis.set_major_formatter(ticker.ScalarFormatter())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* In train dataset among male patients, the ratio between ex-smokers and non-smokers is\nsignificantly higher than that of female patients.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"df = train.groupby(['Sex', 'Age'])['Patient'].unique().reset_index()\ndf['Patient'] = df['Patient'].apply(lambda x: len(x))\ndf = df.rename(columns={'Patient': 'Count'})\n\nfig, axes = plt.subplots(2, 1, figsize=(12,5))\nfig.subplots_adjust(wspace=0.2, hspace=0.2)\nax = sns.barplot(x=\"Age\", y=\"Count\",\n                 hue='Sex', data=df,\n                 palette=customPalette, ax=axes[0])\nax.set_title('Fig.3. Age distribution of patients')\nax.legend(loc='upper right')\nax.set_xlabel('')\nstep = max(math.floor(math.fabs(df['Age'].max()-df['Age'].min())/11),1)\nax.xaxis.set_major_locator(ticker.MultipleLocator(step))\n\ncolor = {\"Male\": \"C1\", \"Female\": \"C0\"}\nfor label in df['Sex'].unique():\n    data = df[df['Sex'] == label].set_index('Count')\n    sns.distplot(data['Age'], label=label, hist=True, kde=True, ax=axes[1], rug=True, bins=10)\n\n    mean = data['Age'].mean()\n    median = data['Age'].median()\n    std = data['Age'].std()\n\n    axes[1].axvline(mean, color=color[label], label=\"Mean\", ls='-')\n    axes[1].axvline(mean+std, color=color[label], label=\"Mean+std\", ls='--')\n    axes[1].axvline(mean-std, color=color[label], label=\"Mean-std\", ls='--')\n    axes[1].axvline(median, color=color[label], label=\"Median\", ls='-.')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* Minimum female age is 49, maximum - 87,\n* Minimum male age is 51, maximum - 83,\n* Mean age is about 66-67 years.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"def _range(df):\n    mini = df.min()\n    maxi = df.max()\n    rang = maxi - mini\n    return rang\n\ndf = train[['Patient','Weeks','FVC','Percent']].groupby(['Patient']).agg([_range]).reset_index()\ndf.columns = [\"\".join(x) for x in df.columns.ravel()]\ndf['Patient'] = df['Patient'].keys()\n\nfig, axes = plt.subplots(1, 2, figsize=(12,5))\nfig.subplots_adjust(wspace=0.2, hspace=0.2)\nss_loc = df.groupby(['Weeks_range'])['Patient'].count().reset_index()\nax = sns.barplot(x='Weeks_range', y='Patient', color=\"C2\", palette=customPalette,\n                 data=ss_loc, ax=axes[0])\nax.set_xlabel('Weeks')\nax.set_ylabel('Count')\nstep = max(math.ceil(math.fabs(ss_loc['Weeks_range'].max()-ss_loc['Weeks_range'].min())/5),1)\nax.xaxis.set_major_locator(ticker.MultipleLocator(step))\n\nsns.distplot(ss_loc['Weeks_range'], hist=True, kde=True, ax=axes[1], bins=5)\naxes[1].set_xlabel('Weeks')\n\nax.set_title('Fig.4. Distribution of FVC observation interval')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* Minimum FVC observation interval 25 weeks, maximum - 63,\n* For most patients, changes of FVC are observed within 50-60 weeks.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(18,8))\nfig.subplots_adjust(wspace=0.2, hspace=0.3)\nlabels = [('FVC_range', 'ml') , ('Percent_range', '%')]\naxes[0,0].set_title('Fig.5. Distribution of change of lung capacity (LC) per obs. interval')\n\nfor i, label in enumerate(labels):\n    ax = sns.barplot(x='Patient', y=label[0], color=\"C2\", palette=customPalette,\n                     data=df, ax=axes[i, 0])\n\n    ax.set_xlabel('Patient #')\n    ax.set_ylabel('Change of LC, %s' % label[1])\n    step = max(math.ceil(math.fabs(df['Patient'].max()-df['Patient'].min())/5),1)\n    ax.xaxis.set_major_locator(ticker.MultipleLocator(step))\n\n    sns.distplot(df[label[0]], hist=True, kde=True, ax=axes[i, 1])\n    axes[i, 1].set_xlabel('Change of LC, %s' % label[1])\n\n    mean = df[label[0]].mean()\n    median = df[label[0]].median()\n    std = df[label[0]].std()\n\n    axes[i, 1].axvline(mean, label=\"Mean\", ls='-')\n    axes[i, 1].axvline(mean+std, label=\"Mean+std\", ls='--')\n    axes[i, 1].axvline(mean-std, label=\"Mean-std\", ls='--')\n    axes[i, 1].axvline(median, label=\"Median\", ls='-.')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* Maximum of lung capacity change inside of interval of observation is more than 1500ml\nand more than 40% of normal capacity,\n* Mean of lung capacity change is about 500ml or 13% of normal capacity.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"def _range(df):\n    mini = df.min()\n    maxi = df.max()\n    rang = maxi - mini\n    return rang\n\ndf = train[['Patient', 'Age', 'FVC', 'Percent']].groupby(['Age', 'Patient']).agg([_range]).reset_index()\ndf.columns = [\"\".join(x) for x in df.columns.ravel()]\n\nfig, axes = plt.subplots(2, 2, figsize=(18,8))\nfig.subplots_adjust(wspace=0.2, hspace=0.2)\nlabels = [('FVC_range', 'ml') , ('Percent_range', '%')]\naxes[0, 0].set_title('Fig.6. Distribution of median change of lung capacity (LC) per patient age')\n\nfor i, label in enumerate(labels):\n    ss_loc = df.groupby(['Age'])[label[0]].median().reset_index()\n    ax = sns.barplot(x='Age', y=label[0], color=\"C2\", palette=customPalette,\n                     data=ss_loc, ax=axes[i, 0])\n    ax.set_ylabel('Change of LC, %s' % label[1])\n    ax.set_xlabel('Age')\n    step = max(math.ceil(math.fabs(ss_loc['Age'].max()-ss_loc['Age'].min())/5),1)\n    ax.xaxis.set_major_locator(ticker.MultipleLocator(step))\n\n    sns.distplot(ss_loc[label[0]], hist=True, kde=True, ax=axes[i, 1], bins=10)\n    axes[i, 1].set_xlabel('Change of LC, %s' % label[1])\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* Median change of lung capacity in absolute value decreases with age,\n which is not observed in relative values.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"df = train[['Patient', 'Age', 'FVC', 'Percent', 'Sex']].groupby(['Age', 'Patient', 'Sex']).agg([_range]).reset_index()\ndf.columns = [\"\".join(x) for x in df.columns.ravel()]\n\nfig, axes = plt.subplots(2, 2, figsize=(18,8))\nfig.subplots_adjust(wspace=0.2, hspace=0.2)\nlabels = [('FVC_range', 'ml') , ('Percent_range', '%')]\naxes[0, 0].set_title('Fig.7. Distribution of median change of lung capacity (LC) per patient age')\n\nfor i, label in enumerate(labels):\n    ss_loc = df.groupby(['Age', 'Sex'])[label[0]].median()\n    ss_rloc = ss_loc.reset_index()\n    ax = sns.barplot(x=\"Age\", y=label[0], data=ss_rloc,\n                     hue=\"Sex\", palette=customPalette, ax=axes[i, 0])\n    ax.set_ylabel('Change of LC, %s' % label[1])\n    ax.set_xlabel('Age')\n    ax.legend(loc='upper right')\n    step = max(math.ceil(math.fabs(ss_rloc['Age'].max()-ss_rloc['Age'].min())/5),1)\n    ax.xaxis.set_major_locator(ticker.MultipleLocator(step))\n    sns.distplot(ss_loc[:,'Female',:], hist=True, kde=True, ax=axes[i, 1], label='Female')\n    sns.distplot(ss_loc[:,'Male',:], hist=True, kde=True, ax=axes[i, 1], label='Male')\n    axes[i, 1].set_xlabel('Change of LC, %s' % label[1])\n\n    mean = ss_loc[:,'Female',:].mean()\n    median = ss_loc[:,'Female',:].median()\n    std = ss_loc[:,'Female',:].std()\n\n    axes[i, 1].axvline(mean, label=\"Mean\", ls='-')\n    axes[i, 1].axvline(mean+std, label=\"Mean+-std\", ls='--')\n    axes[i, 1].axvline(mean-std, ls='--')\n    axes[i, 1].axvline(median, label=\"Median\", ls='-.')\n\n    mean = ss_loc[:,'Male',:].mean()\n    median = ss_loc[:,'Male',:].median()\n    std = ss_loc[:,'Male',:].std()\n\n    axes[i, 1].axvline(mean, ls='-', color=\"C1\")\n    axes[i, 1].axvline(mean+std, ls='--', color=\"C1\")\n    axes[i, 1].axvline(mean-std, ls='--', color=\"C1\")\n    axes[i, 1].axvline(median, ls='-.', color=\"C1\")\n    axes[i, 1].legend(loc='upper right')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* Median change of lung capacity of women has significant fluctuation (more than 30%) compared to the same\nparameter in men, which do not depend of age.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"show_dataset_common_info(test, 'test', '8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* In the test dataset all of patients are men, most of them - ex-smoker.","execution_count":null},{"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1, figsize=(12,5))\ndf = train[['Patient','Weeks','Percent']]\nfor patient in test['Patient'].to_numpy():\n    data = df.groupby(['Patient']).get_group(patient)[['Weeks','Percent']]\n    sns.lineplot(x='Weeks', y='Percent', data=data, palette=customPalette,\n                 label=patient[0:6]+\"***\"+patient[-2:])\naxes.set_ylabel('Change of LC, %')\naxes.set_title('Fig.9. Observations of lung function of patients from test dataset')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate conclusion:\n* All patients from the test dataset show a decrease of lung function depending on time,\n* Patients ID00419637202311204720264 and ID00423637202312137826377 have a sharp decrease\nof lung function in the initial observation period.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# TODO:\n1. Describe patient's CT scan analysis and algorithms of preprocessing.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# References\n\n1. Yu. I. Feshchenko, V. K. Gavrysyuk, N. E. Monogarova Idiopathic interstitial pneumonias: classification, differential diagnosis. Ukrainian pulmonological journal. 2007, no. 2 - p 5-11.\n2. Christe, Andreas MD∗; Peters, Alan A. MD∗ et a. Computer-Aided Diagnosis of Pulmonary Fibrosis Using Deep Learning and CT Images, Investigative Radiology: October 2019 - Volume 54 - Issue 10 - p 627-632. https://doi.org/10.1097/RLI.0000000000000574\n3. Bermejo-Peláez, D., Ash, S.Y., Washko, G.R. et al. Classification of Interstitial Lung Abnormality Patterns with an Ensemble of Deep Convolutional Neural Networks. Sci Rep 10, 338 (2020). https://doi.org/10.1038/s41598-019-56989-5\n4. https://www.nhlbi.nih.gov/health-topics/idiopathic-pulmonary-fibrosis\n5. Lynch DA, Sverzellati N, Travis WD, et al. Diagnostic criteria for idiopathic pulmonary fibrosis: a Fleischner Society White Paper. Lancet Respir Med. 2018;6:138–153.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}