{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# In addition, import...\nimport matplotlib.pyplot as plt\nfrom sklearn.linear_model import LinearRegression\nimport os, shutil\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction \nAfter reading a task provided by the holders of the competition and having a look on the data I have, I asked myself a question: *are actually all the patients in the data sick?*\n\nLet me explain why this question is important. The thing is that healthy person will most likely have a FVC constant in time. So, meaning, there will be no significant differences in FVCs of a healthy person measured on different weeks. On the other hand, a patient with a defect in lungs will have either decreasing FVC with time (in case of progressing disease), or an increasing FVC with time (in case of remission). The state of patients health wrt pulmonary fibrosis will be further called **health status**. \n\nIf the data of different health status groups are intermixed and one uses them to predict something - this prediction will be unclear. It will predict for some averaged health status which has no information value. Basically, it will predict wrongly for each health status group. In a perfect world one should either predict separately for each health status or include the health status as a further variable which will influence the prediction."},{"metadata":{},"cell_type":"markdown","source":"# 2. Look into the Data\n\nLets (once more ;) ) have a look into the data:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv\")\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data above is tabular. \nLooking at the variables we have and reading their description [here](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/data), first hint towards the health status I would expect to get from the **Percent** variable. For instance, 100% should be a typical average FVC for a person with similar characteristics. So for sure > 100% and < 100% Percent clusters should correspond to different health status. Let us have a look to the **Percent**"},{"metadata":{},"cell_type":"markdown","source":"# 3. Percent clusters\nBelow, lets try to look at the distribution of the variable **Percent**:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate mean percent for each client\nclientsPercent = train.groupby('Patient')[\"Percent\"].mean()\n\n# Plot the distribution of mean percent\nplt.hist(clientsPercent, bins = 60)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One can definitely distinguish two to three clusters:\n* One around 60%\n* One around 80%\n* Potentially one above 100%, but the statistics there is very low to judge\nBefore you ask yourselves if it is not an effect of binning. It isn't. I have tried different binning anf the clusters were distinguishable even for a low amount of bins.\n\nI referring to [this](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/172022) discussion topic, the percent values of the FVC are normal if they are 80% and more and ubnormal to different grade if less. So the first conclusion I could make out of this **Percent** distribution is that there are at least two groups by health status. \n\nFine, with this knowledge one could think about splitting these at least two groups of patients for further training."},{"metadata":{},"cell_type":"markdown","source":"# 4. Evolution of FVC and Percent for Different Health Groups\n\nA very naive expectation would be that the people who are ill and their illness progresses, will have a decreasing with time **FVC** and **Percent**. The opposite evolution is expected for the ill people who are recovering: for them I would expect the **FVC** and **Percent** to grow with time. The healthy or close-to-be healthy people should have a constant **FVC** and **Percent** then.\n\n## 4.1 Rough Estimate: Linear Fit\n\nAs a first rough estimate one might assume the simplest evolution of **FVC** and **Percent** with time: linear evolution. For that one needs to perform a linear fit of **FVC** and **Percent** vs time and observe the slope of the line. Negative evolution (evolving disease) should be represented with a negative slope, positive evolution (recovering) should be represented with a positive slope, stable or healthy state will have (almost) no slope - in ranges of fluctuation.\n\nSo first the function on linear fitting:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define a function lfit, which performs linear fitting and writing of the visualization of the fit to a file\n# The fit is done per patient\ndef lfit (x, y, patient):\n    # Detecting which variable is fitted: fraction (percentage), or some absolute one.\n    # Note: this is a hack which works on this data only.\n    titleAddition = \"\"\n    if y.max() <= 100:\n        titleAddition = \"Percent_\"\n    \n    # Linear firring with scikit learn\n    x = np.array(x).reshape(-1,1)\n    y = np.array(y).reshape(-1,1)\n    lr = LinearRegression()\n    lr.fit(x, y)\n    \n    # Save the fitted plot to a file\n    plt.scatter(x, y, marker = 'o')\n    xForPlot = np.array(list(range(x.min(), x.max() + 1))).reshape(-1,1)\n    plt.plot(xForPlot, lr.coef_ * xForPlot + lr.intercept_, 'r--')\n    plt.savefig(f'/kaggle/working/{titleAddition}{patient.unique()}_linearFit.png')\n    plt.clf()\n    \n    return(float(lr.coef_))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's perform a fit for each client inside the data frame in the mode when one groups by patient. The slopes are written to separate columns of the data. The new variables are called **slopePercent** and **slopeFVC**. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clean the folder to which the plots are written before running a round of fitting for all patients\nfolder = '/kaggle/working/'\nfor filename in os.listdir(folder):\n    file_path = os.path.join(folder, filename)\n    try:\n        if os.path.isfile(file_path) or os.path.islink(file_path):\n            os.unlink(file_path)\n        elif os.path.isdir(file_path):\n            shutil.rmtree(file_path)\n    except Exception as e:\n        print('Failed to delete %s. Reason: %s' % (file_path, e))\n\n# Fit for every client\ntrain[\"slopePercent\"] = train.groupby('Patient').apply(lambda x : lfit(x['Weeks'], x['Percent'], x['Patient'])).reindex(train.Patient).values;\ntrain[\"slopeFVC\"] = train.groupby('Patient').apply(lambda x : lfit(x['Weeks'], x['FVC'], x['Patient'])).reindex(train.Patient).values;","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Visualization of the Percent and FVC slope\nLet us draw the distribution of the slopes of the linear fits of the **FVC** (with time):"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(np.array(train.groupby('Patient')[\"slopeFVC\"].unique().tolist()).ravel(), bins = 60);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The same for the **Percent**:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(np.array(train.groupby('Patient')[\"slopePercent\"].unique().tolist()).ravel(), bins = 60);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One can roughly distinguish three clusters of slopes. Two of them negative, but differently large, third one - more around zero, so \"no slope\", close to horizontal line. This might point to the fact that (as seen before) we have three different dynamics in desease progression. I have given hypotheses that those are most likely \"more ill\" and less severely/stable ill patients."},{"metadata":{},"cell_type":"markdown","source":"# 5. Now this can be used with Plots for the Model!\nHere is how the above derived information can be practically used in this competition:\n* **Enrich Training Data With Health Status**: based on the fits above one can group the data to *health status clusters*\n* **Train the Model separately for each cluster**: as obviously the dynamics of the desease is completely different for the patients of different clusters, it makes sence to train the models for these clusters differently\n* **Let the Image predict the health status!**: but what to do on test data? There we can not perform a fit as there is no FVC dynamics available. The only hint we have is an image. Image should be trained on train data to predict the health status. Once the health status is predicted for a concrete client in the test data, the corresponding model will be chosen and applied to him/her.\n\nOf course, for more precision one can try to fit with another function, or better include a measurement uncertainty of the FVC metric (70 for FVC and correspondingly 70 * sqrt(Percent/FVC) for the Percent. The latter was derived based on simplified calculation of [uncertainty propagation](https://en.wikipedia.org/wiki/Propagation_of_uncertainty)). This will most likely produce more precisely separated clusters by health status."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}