{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\npd.set_option('display.max_rows', 500)\npd.set_option('display.max_columns', 500)\npd.set_option('display.width', 1000)\n\nfrom scipy import stats\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ndf_test = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n\nprint(f'Training Set Shape = {df_train.shape} - Patients = {df_train[\"Patient\"].nunique()}')\nprint(f'Training Set Memory Usage = {df_train.memory_usage().sum() / 1024 ** 2:.2f} MB')\nprint(f'Test Set Shape = {df_test.shape} - Patients = {df_test[\"Patient\"].nunique()}')\nprint(f'Test Set Memory Usage = {df_test.memory_usage().sum() / 1024 ** 2:.2f} MB')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **1. Introduction**\n\nThere are **1549** samples in `train.csv` and they belong to **176** different patients. In `train.csv`, patients have images acquired at different timesteps (`Weeks`). All of the patients don't have FVC measured at time `Week = 0` and their number of FVC measurements vary from patient to patient.\n\nMost of the patients have FVC measurements at 9 different timesteps but this number can change between 6 and 10. Thus, number of samples are not consistent for different patients.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"training_sample_counts = df_train.rename(columns={'Weeks': 'Samples'}).groupby('Patient').agg('count')['Samples'].value_counts()\nprint(f'Training Set FVC Measurements Per Patient \\n{(\"-\") * 41}\\n{training_sample_counts}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are **5** samples in `test.csv` and they belong to **5** different patients. Those samples and patients also exist in `train.csv`. They are the last 5 patients of training set and the samples are first measurements of the patients. This is because `test.csv` is a placeholder and the real test set is hidden. The purpose of placeholder test set is showing the structure of real test set and testing submissions. When the notebook is submitted, it runs with the real test set.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"df_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The simplest way of creating submissions is predicting `FVC` and `Confidence` for all test samples, then creating `Patient_Week` on test set and merging it to `sample_submission.csv`. This way submission file will have all of the predictions from test set regardless of their count. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv( '../input/osic-pulmonary-fibrosis-progression/sample_submission.csv' )\ndf_submission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **2. Laplace Log Likelihood**\n\nPredictions are evaluated with a modified version of the Laplace Log Likelihood. For each sample in test set, an `FVC` and a `Confidence` measure (standard deviation σ) has to be predicted.\n\n`Confidence` values smaller than 70 are clipped.\n\n$\\large \\sigma_{clipped} = max(\\sigma, 70),$\n\nErrors greater than 1000 are also clipped in order to avoid large errors.\n\n$\\large \\Delta = min ( |FVC_{true} - FVC_{predicted}|, 1000 ),$\n\nThe metric is defined as:\n\n$\\Large metric = -   \\frac{\\sqrt{2} \\Delta}{\\sigma_{clipped}} - \\ln ( \\sqrt{2} \\sigma_{clipped} ).$\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## **3. FVC (Forced Vital Capacity)**\n\n`FVC` measurement shows the amount of air a person can forcefully and quickly exhale after taking a deep breath. It is defined as `the recorded lung capacity in ml` under the Data tab. The change in `FVC` over the course of weeks is used for predicting the patients' lung function decline.\n\nEven though the `FVC` predictions smaller than 1000 are clipped, the minimum value in training set is **827**. The maximum `FVC` value in training set is **6399**. The distribution is heavily tailed on the right end because some patients have extremely high `FVC` measurements. However, most of the patients are close to mean `FVC`.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'FVC Statistical Summary\\n{\"-\" * 23}')\n\nprint(f'Mean: {df_train[\"FVC\"].mean():.6}  -  Median: {df_train[\"FVC\"].median():.6}  -  Std: {df_train[\"FVC\"].std():.6}')\nprint(f'Min: {df_train[\"FVC\"].min()}  -  25%: {df_train[\"FVC\"].quantile(0.25)}  -  50%: {df_train[\"FVC\"].quantile(0.5)}  -  75%: {df_train[\"FVC\"].quantile(0.75)}  -  Max: {df_train[\"FVC\"].max()}')\nprint(f'Skew: {df_train[\"FVC\"].skew():.6}  -  Kurtosis: {df_train[\"FVC\"].kurtosis():.6}')\nmissing_values_count = df_train[df_train[\"FVC\"].isnull()].shape[0]\ntraining_samples_count = df_train.shape[0]\nprint(f'Missing Values: {missing_values_count}/{training_samples_count} ({missing_values_count * 100 / training_samples_count:.4}%)')\n\nfig, axes = plt.subplots(ncols=2, figsize=(18, 6), dpi=150)\n\nsns.distplot(df_train['FVC'], label='FVC', ax=axes[0])\nstats.probplot(df_train['FVC'], plot=axes[1])\n\nfor i in range(2):\n    axes[i].tick_params(axis='x', labelsize=12)\n    axes[i].tick_params(axis='y', labelsize=12)\n    axes[i].set_xlabel('')\n    axes[i].set_ylabel('')\n    \naxes[0].set_title(f'FVC Distribution in Training Set', size=15, pad=15)\naxes[1].set_title(f'FVC Probability Plot', size=15, pad=15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Every patients' `FVC` should be plotted individually as a function of time because the competition objective is predicting `FVC` values of 5 different patients over 146 timesteps.\n\nMajority of the patients' conditions got worse over the course of weeks except 1-2%. Increase of `FVC` in that small group may be random because `FVC` fluctuates too much over time in some patients. Those patients may not be responding to treatment very well or they are not getting better for some reason. However, some patients are clearly getting better because their `FVC` are increasing linearly with very few fluctuations.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_fvc(df, patient):\n        \n    df[['Weeks', 'FVC']].set_index('Weeks').plot(figsize=(30, 6), label='_nolegend_')\n    \n    plt.tick_params(axis='x', labelsize=20)\n    plt.tick_params(axis='y', labelsize=20)\n    plt.xlabel('')\n    plt.ylabel('')\n    plt.title(f'Patient: {patient} - {df[\"Age\"].tolist()[0]} - {df[\"Sex\"].tolist()[0]} - {df[\"SmokingStatus\"].tolist()[0]} ({len(df)} Measurements in {(df[\"Weeks\"].max() - df[\"Weeks\"].min())} Weeks Period)', size=25, pad=25)\n    plt.legend().set_visible(False)\n    plt.show()\n\nfor patient, df in list(df_train.groupby('Patient')):\n    \n    df['FVC_diff-1'] = np.abs(df['FVC'].diff(-1))\n    \n    print(f'Patient: {patient} FVC Statistical Summary\\n{\"-\" * 58}')\n    print(f'Mean: {df[\"FVC\"].mean():.6}  -  Median: {df[\"FVC\"].median():.6}  -  Std: {df[\"FVC\"].std():.6}')\n    print(f'Min: {df[\"FVC\"].min()} -  Max: {df[\"FVC\"].max()}')\n    print(f'Skew: {df[\"FVC\"].skew():.6}  -  Kurtosis: {df[\"FVC\"].kurtosis():.6}')\n    print(f'Change Mean: {df[\"FVC_diff-1\"].mean():.6}  - Change Median: {df[\"FVC_diff-1\"].median():.6}  - Change Std: {df[\"FVC_diff-1\"].std():.6}')\n    print(f'Change Min: {df[\"FVC_diff-1\"].min()} -  Change Max: {df[\"FVC_diff-1\"].max()}')\n    print(f'Change Skew: {df[\"FVC_diff-1\"].skew():.6} -  Change Kurtosis: {df[\"FVC_diff-1\"].kurtosis():.6}')\n    \n    plot_fvc(df, patient)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **4. To Be Continued**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}