{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Read in the inputs","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.shape # How many rows and columns?","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's check a few rows\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# How many rows for each patient?\ndata.groupby('Patient').size()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like 176 unique patients! Let's try and explain one row in words. If we look at previous display of rows, for example - patient ID00007637202177411956430 , we can say that this person is a male, ex-smoker and 79 years old. What's the percent, weeks and FVC? \n\nWhy is weeks negative? From data description - \"the relative number of weeks pre/post the baseline CT (may be negative)\"\n\nOthers from Data Description:\nFVC - is the recorded lung capacity in ml\n(What's a good FVC value?)\n\nPercent- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics. \n\nWhat's the goal?\nIt seems like we have to predict the lung capacity based on gender, smoking status, age and also use the CT scans I guess?\n\nHow is the model evaluated?\nAs per competition page, \"This competition is evaluated on a modified version of the Laplace Log Likelihood. In medical applications, it is useful to evaluate a model's confidence in its decisions. Accordingly, the metric is designed to reflect both the accuracy and certainty of each prediction.\"\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# How many rows for each smokers vs. ex-smokers?\ndata.groupby('SmokingStatus')['Patient'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like a lot of ex-smokers 118 of 176 i.e., 67%. I wonder if people who never smoked have high lung capacity?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data.groupby('SmokingStatus')['FVC'].mean() \n# Seems odd that the data description page says FVC is \"the recorded lung capacity in ml\".","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.groupby('Weeks')['Patient'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.groupby(['Weeks','SmokingStatus','Sex','Age'])['FVC'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Prep categorical variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\ncat_features = ['Sex','SmokingStatus']\nencoder = LabelEncoder()\n\n# Apply the label encoder to each column\nencoded = data[cat_features].apply(encoder.fit_transform)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data2 = data[['FVC','Percent','Weeks','Age']].join(encoded)\ndata2.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = data2[['SmokingStatus','Age','Sex','Weeks','Percent']]\ny = data2['FVC']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation Metric","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's define a function to calculate the metric\n# I didn't actually use this evaluation but sharing my thoughts\n# def eval_metric(FVC,FVC_Pred,sigma):\n#     sigma_clipped = np.max(sigma,70)\n#     delta = np.min(np.abs(FVC-FVC_Pred),1000)\n#     eval_metric = -np.sqrt(2)*delta/sigma_clipped - np.ln(np.sqrt(2)*sigma_clipped)\n#     return eval_metric\n\n# We need the prediction for FVC_Pred and confidence(sigma I think?)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simple Linear Regression Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt  \nimport seaborn as seabornInstance \nfrom sklearn.model_selection import train_test_split \nfrom sklearn.linear_model import LinearRegression\nfrom sklearn import metrics\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create training, validation, and test splits","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"regressor = LinearRegression()  \nregressor.fit(X_train, y_train) #training the algorithm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#To retrieve the intercept:\nprint(regressor.intercept_)\n#For retrieving the slope:\nprint(regressor.coef_)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = regressor.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})\ndf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df1 = df.head(25)\ndf1.plot(kind='bar',figsize=(16,10))\nplt.grid(which='major', linestyle='-', linewidth='0.5', color='green')\nplt.grid(which='minor', linestyle=':', linewidth='0.5', color='black')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test2 = test[['Percent','Weeks','Age']].join(encoded)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test['Patient_Week'] = test['Patient'].astype(str)+\"_\"+test['Weeks'].astype(str)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.groupby('SmokingStatus')['FVC'].mean() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Apply the label encoder to each column\nencoded = test[cat_features].apply(encoder.fit_transform)\ntest2 = test[['Patient','Percent','Weeks','Age']].join(encoded)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test2.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission[['Patient','Weeks']] = submission.Patient_Week.str.split(\"_\",expand=True,)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = submission.drop('FVC',1)\nsubmission = submission.drop('Confidence',1)\ntest2 = test2.drop('Weeks',1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2 = pd.merge(submission,test2,on='Patient',how='left')\nsubmission2.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X2 = submission2[['SmokingStatus','Age','Sex','Weeks','Percent']]\nsubmission2['FVC'] = regressor.predict(X2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2.groupby(['SmokingStatus','Sex','Age'])['FVC'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2['FVC_Group'] = submission2.groupby(['SmokingStatus','Sex','Age'])['FVC'].transform('mean')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2['Confidence'] = 100*submission2['FVC']/submission2['FVC_Group']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission2.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission3 = submission2[['Patient_Week','FVC','Confidence']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission3.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission3['FVC'] = submission3['FVC'].astype(int)\nsubmission3['Confidence'] = submission3['Confidence'].astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission3.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission3.to_csv(\"/kaggle/working/submission.csv\",index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}