{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We provide an analysis of the meta data in `train.csv`. One of the useful notebooks we referred to was [this EDA](https://www.kaggle.com/piantic/osic-pulmonary-fibrosis-progression-basic-eda), with our **main contribution** being the additional analysis of the data **filtered for unique patients** and the use of corresponding *summary statistics* (e.g. `FVC_mean`, `Percent_mean`) to produce scatter plots and heatmaps. This gives an extra overview of what is going on.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Executive Summary","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" \nWe have 176 patients in the training set `train.csv` (cell 11). The test data `test.csv` is a subset of the training set, containing 5 patients. \n- Categorical data: `Sex`, `Age` and  `Smoking Status`.\n- Continuous/discrete: `Percent`, `Age`, `Weeks`.\n- Response: `FVC`\n    \nThe main observations are:\n- Patients made **6-10 visits** (FVC measurements), with the majority of them having made **9 visits**. (cell 18) \n- The range of `FVC` is **827-6399**, with a median of the standard deviations at 130. (cell 26)\n- Feature/feature relationships\n    - The majority of patients consists of **males** who are **ex-smokers**. (cell 21)\n    - **Male** patients tend to have a higher `Percent` in the first visit, but this difference seems to vanish by the last visit. (cell 23)\n- Feature/response relationships\n    - **Female** patients have a lower mean `FVC` than male patients. (various cells, e.g 31)\n    - `Percent` seems like a good indicator of `FVC` (heatmap and various plots).\n    - `Age` does not seem to affect the mean `FVC` (eg. cell 32).\n    - The time series of measurements for `FVC` do not seem to follow clear upwards or downwards trend. (it may be worth to test for stationarity?)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Importing Packages and Functions","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# OS (working folder) packages\nimport os\n\n# DataFrame packages\nimport pandas as pd\n\n# Visualisation packages\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib.colors import ListedColormap\n\nfrom IPython.display import display, display_html\n\n# Calculation packages \nimport numpy as np\n\nsns.set_style(\"whitegrid\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Kevin - Data functions - filter dataframe according to specific value(s) in column \"str_col\"\ndef filter_data(data, str_col, list_filter):\n    '''Example: filter_data(train_df, 'Patient', ['ID00007637202177411956430'])'''\n    df = data.copy()\n    df = df[df[str_col].isin(list_filter)]\n    return(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting functions are written below, click \"Code\" to reveal.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Kevin - plotting functions\ndef plt_fvc_vs_sex1(df, col_str, xlabel, subtitle, suptitle):\n    '''few-time use only, see in sections'''\n    \n    bins = 30\n    fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (16,6))\n\n    # Plot 1 (Left)\n    sns.distplot(df[col_str], label = 'Male', bins = bins, ax = ax1)\n    ax1.set(xlabel = xlabel, ylabel = \"Density\", title = subtitle + \" (overall)\")\n\n    # Plot 2 (Right)\n    # or use plt.hist() instead of sns.distplot()\n    sns.distplot(filter_data(df, 'Sex', ['Male'])[col_str], label = 'Male', bins = bins, ax = ax2)\n    sns.distplot(filter_data(df, 'Sex', ['Female'])[col_str], label = 'Female', bins = bins, ax = ax2)\n    ax2.legend(loc = 'upper right')\n    ax2.set(xlabel = xlabel, ylabel = \"Density\", title = subtitle + \" (by gender)\")\n\n    # Overall info\n    plt.suptitle(suptitle)\n    plt.show()\n    \ndef plt_fvc_vs_sex2(df1, df2, col_str, xlabel, subtitle1, subtitle2, suptitle):\n    '''few-time use only, see in sections'''\n    \n    bins = 30\n    fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (16,6))\n\n    # plot 1\n    sns.distplot(filter_data(df1, 'Sex', ['Male'])[col_str], label = 'Male', bins = bins, ax = ax1)\n    sns.distplot(filter_data(df1, 'Sex', ['Female'])[col_str], label = 'Female', bins = bins, ax = ax1)\n\n    # plot 1 - legends\n    ax1.legend(loc = 'upper right')\n    ax1.set(xlabel = xlabel, ylabel = \"Density\", title = subtitle1 + \" (by gender)\")\n\n    # plot 2\n    sns.distplot(filter_data(df2, 'Sex', ['Male'])[col_str], label = 'Male', bins = bins, ax = ax2)\n    sns.distplot(filter_data(df2, 'Sex', ['Female'])[col_str], label = 'Female', bins = bins, ax = ax2)\n\n    # plot 2 - legends\n    ax2.legend(loc = 'upper right')\n    ax2.set(xlabel = xlabel, ylabel = \"Density\", title = subtitle2 + \" (by gender)\")\n\n    # Overall plot info\n    plt.suptitle(suptitle)\n    plt.show()\n    \n\ndef plt_fvc_vs(df, x, y, z, color_palette):\n    '''Example: plt_fvc_vs(train_df, x = 'Percent', y = 'FVC', z = 'Age', color_palette = 'YlGn')'''\n    \n    my_cmap = ListedColormap(sns.color_palette(color_palette))\n\n    plt.figure(figsize = (15,9))\n    plt.scatter(df[x], df[y], c = df[z], cmap=my_cmap)\n    plt.xlabel(x); plt.ylabel(y); plt.title(y + \" vs. \" + x + \" (\" + z + \" in colours)\")\n    plt.colorbar()\n    plt.show()\n\ndef plt_fvc_vs_sns(df, x, y, z, color_palette):\n    '''Example: plt_fvc_vs_sns(df = train_df, x = 'Percent', y = 'FVC', z = 'Sex', color_palette = 'CMRmap_r')'''\n    df1 = df.copy()\n    df_unique = len(df[z].unique())\n    \n    if (z == \"Sex_Smoke\"):\n        df1 = df1.sort_values(z, ascending = False)\n    \n    sns.lmplot(x = x, y = y, data = df1, hue = z,\n               fit_reg = False, legend = True, palette = sns.color_palette(color_palette, df_unique), height = 6, aspect = 2)\n    ax = plt.gca().set_title(y + \" vs. \" + x + \" (\" + z + \" in colours)\")\n    \n###### Functions for extra plots\ndef plt_fvc_vs_subplot(df, x, y, z, color_palette, ax):\n    '''Example: plt_fvc_vs_subplot(train_df, x = 'Percent', y = 'FVC', z = 'Age', color_palette = 'YlGn', ax = ax1)'''\n    \n    my_cmap = ListedColormap(sns.color_palette(color_palette))\n\n    im = ax.scatter(df[x], df[y], c = df[z], cmap = my_cmap)\n    ax.set(xlabel = x, ylabel = y, title = y + \" vs. \" + x + \" (\" + z + \" in colours)\")\n    plt.colorbar(im, ax = ax)\n    \ndef plt_fvc_vs_compare(df1, df2, x, y, z1, z2, color_palette):\n    '''Example: plt_fvc_compare(first_visit, last_visit, x = 'Percent', y = 'FVC', z = 'Age', color_palette = 'YlGn')'''\n    fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (20,8))\n    plt_fvc_vs_subplot(df1, x = x, y = y, z = z1, color_palette = color_palette, ax = ax1)\n    plt_fvc_vs_subplot(df2, x = x, y = y, z = z2, color_palette = color_palette, ax = ax2)\n    if(z1 == z2):\n        plt.suptitle(y + \" vs. \" + x + \" (\" + z1 + \" in colours) - (first vs. last visit are left/right)\")\n    else:\n        plt.suptitle(y + \" vs. \" + x + \" (\" + z1 + \" and \" + z2 + \" in colours (left/right))\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# List files and folders available\nlist(os.listdir(\"../input/osic-pulmonary-fibrosis-progression\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# 2. Reading in the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ntest_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/test.csv\")\nsample_submission = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train_df.head(6))\ndisplay(test_df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Meta-Data analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Overall Info","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Natalia - Patients\n\n# Number of unique patients\nprint('Number of unique patients in training set: {}'.format(train_df['Patient'].nunique()))\nprint('Number of unique patients in test set: {}'.format(test_df['Patient'].nunique()), '\\n')\n\n# Patient ID\npatient_id_train = set(train_df['Patient'].unique())\npatient_id_test = set(test_df['Patient'].unique())\nprint('Patients in both training and test set:')\nprint(list(patient_id_train.intersection(patient_id_test)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.1.1 Filtering data for unique patients, first/last visits","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Below: We write a data frame patient_info_df, which filters train_df for unique patients.\n# We also add some summary statistics. Click \"Code\" to reveal.","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Natalia - Obtain patient information\n\n# Obtain FVC, Percent, Weeks statistics\npatient_info_df = train_df.groupby(['Patient','Sex','Age','SmokingStatus']).agg({'FVC': ['count','mean','std','min','max'],'Percent': ['mean','std','min','max'],'Weeks': ['min','max']}) \n\n# Rename columns\npatient_info_df.columns = [\"_\".join(x) for x in patient_info_df.columns.ravel()]\npatient_info_df = patient_info_df.rename(columns = {'FVC_count':'Count'})\n\n# Obtain range for weeks\npatient_info_df['Weeks_range'] = patient_info_df['Weeks_max'] - patient_info_df['Weeks_min']\n\n# Reset index\npatient_info_df = patient_info_df.drop(['Weeks_min','Weeks_max'],1).reset_index()\n\n# Combine Sex and SmokingStatus\npatient_info_df = patient_info_df.assign(Sex_Smoke = patient_info_df.Sex.astype(str) + '_' + patient_info_df.SmokingStatus.astype(str))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display dataframe\ndisplay(patient_info_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Below: We filter the dataframe train_df according to the first and last visits.\n# We keep corresponding values. Click \"Code\" to reveal.","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Nikolas - Dataframes with first and last visits only\nfirst_visit = pd.DataFrame(data = None, columns = train_df.columns)\nlast_visit = pd.DataFrame(data = None, columns = train_df.columns)\n\nfor pat in patient_info_df['Patient']: \n     new = filter_data(train_df,'Patient',[pat]).iloc[0]\n     first_visit = first_visit.append(new,ignore_index = True)\n     new = filter_data(train_df,'Patient',[pat]).iloc[-1]\n     last_visit = last_visit.append(new,ignore_index = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(first_visit.head())\ndisplay(last_visit.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Nikolas\npatient_info_df_copy = patient_info_df.copy()\npatient_info_df_copy['Sex'].replace('Female',0,inplace=True)\npatient_info_df_copy['Sex'].replace('Male',1,inplace=True)\n\npatient_info_df_copy['SmokingStatus'].replace('Never smoked', 0, inplace = True)\npatient_info_df_copy['SmokingStatus'].replace('Ex-smoker', 1, inplace = True)\npatient_info_df_copy['SmokingStatus'].replace('Currently smokes', 0, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot of correlation heatmap of feature summaries and responses\nplt.gcf().subplots_adjust(bottom = 0.15)\n\ncorr = patient_info_df_copy.corr()\ntop_corr_features = corr.index\n\nplt.figure(figsize = (12,10))\nax = sns.heatmap(patient_info_df_copy[top_corr_features].corr(),annot=True,cmap=\"RdYlGn\")\n\nbottom, top = ax.get_ylim()\nax.set_ylim(bottom + 0.5, top - 0.5)\nax.set_title(\"Correlation heatmap of feature sumamries and response (filtered for unique patients)\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The map shows that the correlation between age and any of the other fields is non-significant.\n`Sex` is weakly correlated to `SmokingStatus` and `FVC` (but not to the variance of `FVC`).\nUnsurprisingly, `FVC` is also correlated to `Percent`.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.1.2 Visits and Weeks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Nikolas \n\nplt.figure(figsize = (14,6))\nax = sns.countplot(x = \"Weeks\", data = train_df, color = 'lightskyblue')\nax.set_title(\"Histogram of weeks (how many patients visited at particular 'week')\")\n\nfor ind, label in enumerate(ax.get_xticklabels()):\n    if ind % 10 == 0:  # every 10th label is kept\n        label.set_visible(True)\n    else:\n        label.set_visible(False)  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below, we see that Patients made **6-10 visits** (`FVC` measurements), with the majority of them having made **9 visits**. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot number of visits\nplt.figure(figsize = (14,6))\nax = sns.countplot(x = \"Count\", data = patient_info_df, color = 'olivedrab')\nax.set(xlabel = \"Number of visits\", title = \"Number of patient visits\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Natalia - Weeks in which a patient visits (baseline is first FVC measurement)\ntrain_df['Weeks'].describe()\n\n# Plot gaps between visits\n# Notice some patients stop visits at week 27, most go up to week 50+\npatient_weeks = {}\nplt.figure(figsize = (12,10))\n\nfor i,j in enumerate(patient_id_train):\n    \n    weeks = np.array(filter_data(train_df, 'Patient', [j])['Weeks'])\n    weeks = weeks - weeks[0]\n    patient_weeks[j] = weeks\n    plt.plot(weeks)\n    plt.xticks(np.arange(0,10,1))\n    plt.xlabel(\"Visits\")\n    plt.ylabel(\"Weeks\")\n    \n    if i <= 20:\n        print(j, weeks)    \n    else:\n        continue\n\nplt.title(\"Weeks in which a patient visits (baseline is first FVC measurement)\")\nplt.xlabel(\"i-th visit\"), plt.ylabel(\"'Week' corresponding to i-th visit\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.1.3 Categorical Features: Smoking Status and Sex","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Natalia\n# Sex, Smoking Status \ndef display_side_by_side(*args):\n    html_str=''\n    for df in args:\n        html_str+=df.to_html()\n    display_html(html_str.replace('table','table style=\"display:inline\"'),raw=True)\n    \ndisplay_side_by_side(patient_info_df.groupby(['Sex']).count()['Patient'].to_frame(),\n                 patient_info_df.groupby(['SmokingStatus']).count()['Patient'].to_frame(),\n                 patient_info_df.groupby(['Sex','SmokingStatus']).count()['Patient'].to_frame())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also see that the majority of patients consists in **males** who are **ex-smokers**.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Below: Code to create barplot, with analysis of categorical data. Click \"Code\" to reveal.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Extract the data to add to the barplots\nmale_smoking = filter_data(patient_info_df, 'Sex', ['Male']).groupby(['SmokingStatus']).count()['Patient']\nfemale_smoking = filter_data(patient_info_df, 'Sex', ['Female']).groupby(['SmokingStatus']).count()['Patient']\nsmoking_sex1 = filter_data(patient_info_df, 'SmokingStatus', ['Currently smokes']).groupby(['Sex']).count()['Patient']\nsmoking_sex2 = filter_data(patient_info_df, 'SmokingStatus', ['Ex-smoker']).groupby(['Sex']).count()['Patient']\nsmoking_sex3 = filter_data(patient_info_df, 'SmokingStatus', ['Never smoked']).groupby(['Sex']).count()['Patient']\n\n# labels for x and y axis\nsmokingstatus_lab = ['Currently smokes', 'Ex-smoker', 'Never smoked']\nsex_lab = ['Female', 'Male']\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize = (15,6))\n\n# Plot 1\nax1.bar(x = smokingstatus_lab, height = male_smoking, bottom = female_smoking, label = 'Male')\nax1.bar(x = smokingstatus_lab, height = female_smoking, label = 'Female')\n\n# Plot 1 - details\nax1.set(xlabel = \"SmokingStatus\", ylabel = \"Counts\", title = \"Number of smokers, conditioned on Sex\")\nax1.legend()\n\n# Plot 2\nax2.bar(x = sex_lab, height = smoking_sex1, label = 'Currently Smokes')\nax2.bar(x = sex_lab, height = smoking_sex2, bottom = smoking_sex3 + smoking_sex1, label = 'Ex-smoker')\nax2.bar(x = sex_lab, height = smoking_sex3, bottom = smoking_sex1, label = 'Never smoked')\n\n# Plot 2 - details\nax2.set(xlabel = \"Sex\", ylabel = \"Counts\", title = \"Number of sex, conditioned on SmokingStatus\")\nplt.legend()\n\n# Overall info\nfig.suptitle('Comparison of Categorical Data (Filtered for unique patients)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.1.3 Continuous Features: Age and Percent","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Distribution of Percent","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Density plot of percent (not filtered for unique)\nplt_fvc_vs_sex1(train_df, 'Percent', \"Percent of Patient FVC compared to usual FVC\", 'Histogram of Percent',\n               \"Histogram of the distribution of Percent (Not filtered for unique patients)\")\n\n# Density plot of percent (first and last visit)\nplt_fvc_vs_sex2(first_visit, last_visit, 'Percent', \"Percent of Patient FVC compared to usual FVC\",\n                \"Histogram of Percent, first visit\", \"Histogram of Percent, last visit\",\n                \"Comparison of percentage values, for first and last visit\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Distribution of Age","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Density plot of age (filtered for unique)\nplt_fvc_vs_sex1(patient_info_df, 'Age', \"Age (in years)\", 'Histogram of Age',\n               \"Histogram of the distribution of Age (filtered for unique patients)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 FVC General Info","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Nikolas - Time Series of FVC \n\n# most patients have their FVCs not change massively between successive visits\nplt.figure(figsize = (12,8))\nfor pat in patient_id_train:\n       filt = filter_data(train_df, 'Patient',[pat])['FVC'].tolist()\n       plt.plot(range(1,len(filt)+1),filt)\n\nplt.xlabel(\"i-th visit\"); plt.ylabel(\"FVC value\"); plt.title(\"FVC levels by visit\")\nplt.show()  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Histogram/density for FVC (not filtered)\nplt_fvc_vs_sex1(train_df, 'FVC', 'Lung Capacity (ml)', 'Histogram of FVC', \"Overall FVC comparison (not filtered for unique patients)\")\n\n# Histogram/density for mean FVC (filtered)\nplt_fvc_vs_sex1(patient_info_df, 'FVC_mean', 'Lung Capacity (ml)', 'Histogram of FVC_mean', \"Overall FVC comparison (filtered for unique patients)\")\n\n# For Male patients, there seems to be a shift towards the right, while this \n# is not the case for Female Patients","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 FVC vs. The Rest (i.e. response vs feature)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.4.1 Filtered Data (unique patients, first and last visit, i.e. patient_info_df, first_visit, last_visit)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# define color palettes\n# col_list = ['CMRmap_r','gist_ncar_r', 'hsv_r', 'jet_r', 'nipy_spectral_r', 'rainbow_r']\n\ncolor_sex = 'nipy_spectral_r'\ncolor_age = 'nipy_spectral_r'\ncolor_percent = 'gist_ncar_r'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Natalia - 6 Categories\n\nplt_fvc_vs_sns(df = patient_info_df, x = 'Age', y = 'FVC_mean', z = 'Sex_Smoke', color_palette = color_sex)\nplt_fvc_vs_sns(df = patient_info_df, x = 'Percent_mean', y = 'FVC_mean', z = 'Sex_Smoke', color_palette = color_sex)\n# plt_fvc_vs_sns(df = patient_info_df, x = 'Weeks_range', y = 'FVC_mean', z = 'Sex_Smoke', color_palette = 'CMRmap_r')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Above:**\n\nPlot row 1: `Age` does not have a huge impact on `FVC`.\n\nPlot row 2: `Percent` increase seems to lead to `FVC` increase. `Male` is generally higher. `Smoker` is generally higher","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Using FVC mean","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1,2, figsize = (20,8))\n\nplt_fvc_vs_subplot(patient_info_df, x = 'Percent_mean', y = 'FVC_mean', z = 'Age', color_palette = color_age, ax = ax1)\nplt_fvc_vs_subplot(patient_info_df, x = 'Age', y = 'FVC_mean', z = 'Percent_mean', color_palette = color_percent, ax = ax2)\n\nplt.suptitle(\"FVC_mean vs. continuous (summary) statistics (filtered for unique patients)\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Using FVC of First vs. Last Visit","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Kevin - Nikolas Modification\n# Continuous conditions (see \"z = [...]\")\n\n# FVC vs. Age (by Percent)\nplt_fvc_vs_compare(first_visit, last_visit, x = 'Age', y = 'FVC', z1 = 'Percent', z2 = 'Percent', color_palette = color_percent)\n\n# FVC vs. Age (by Weeks)\nplt_fvc_vs_compare(first_visit, last_visit, x = 'Age', y = 'FVC', z1 = 'Weeks', z2 = 'Weeks', color_palette = color_age)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4.2 Unfiltered Data (i.e. using train_df/train_df_cat)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add Sex_smoke status\ntrain_df_cat = train_df.copy()\ntrain_df_cat = train_df_cat.assign(Sex_Smoke = train_df_cat.Sex.astype(str) + '_' + train_df_cat.SmokingStatus.astype(str))\ndisplay(train_df_cat.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt_fvc_vs_compare(train_df, train_df, x = 'Percent', y = 'FVC', z1 = 'Age', z2 = 'Weeks', color_palette = color_age)\nplt_fvc_vs_compare(train_df, train_df, x = 'Age', y = 'FVC', z1 = 'Percent', z2 = 'Weeks', color_palette = color_percent)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Above:**\n\nPlot row 1: The higher the `Percent`, the higher the `FVC` (i.e. possible linear relationship?).\n\nPlot row 2: At any `Age` point, the computation of `Percent` is 'essential'. `Weeks` does not seem to have an impact (yet?).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt_fvc_vs_sns(df = train_df_cat, x = 'Percent', y = 'FVC', z = 'Sex_Smoke', color_palette = color_sex)\nplt_fvc_vs_sns(df = train_df_cat, x = 'Age', y = 'FVC', z = 'Sex_Smoke', color_palette = color_sex)\nplt_fvc_vs_sns(df = train_df_cat, x = 'Weeks', y = 'FVC', z = 'Sex_Smoke', color_palette = color_sex)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 Archived Plots","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Below are previously used plots, which we felt are not needed to present (click \"Code\" and \"Output\" to show).","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Previously used plots:\nimport plotly.express as px\n\n# Nikolas - interactive plots\n\n# Kevin version of Nikolas (static instead of interactive)\nplt_fvc_vs_sns(patient_info_df, x = 'Weeks_range', y = 'Age', z = 'Sex_Smoke', color_palette = 'nipy_spectral_r')\n\n# Age / Weeks scatterplot colored by Sex\nfig = px.scatter(train_df, x=\"Weeks\", y=\"Age\", color='Sex')\nfig.show()\n\n# Age / Weeks scatterplot colored by SmokingStatus\nfig = px.scatter(train_df, x=\"Weeks\", y=\"Age\", color='SmokingStatus')\nfig.show()\n\n\n# Histogram/density for intial and final FVC (filtered) - NOT RELEVANT\nplt_fvc_vs_sex2(first_visit, last_visit, 'FVC', 'Lung Capacity (ml)', 'Histogram with initial FVC ', 'Histogram with final FVC ',\n                'Histograms with intial and final FVC (filtered for unique patients)')\n\nplt_fvc_vs_sns(train_df, x = 'Weeks', y = 'Age', z = 'Sex', color_palette = 'nipy_spectral_r')\nplt_fvc_vs_sns(train_df, x = 'Weeks', y = 'Age', z = 'SmokingStatus', color_palette = 'nipy_spectral_r')\n\nplt_fvc_vs_sns(df = first_visit, x = 'Age', y = 'FVC', z = 'Sex', color_palette = color_sex)\nplt_fvc_vs_sns(df = first_visit, x = 'Age', y = 'FVC', z = 'SmokingStatus', color_palette = color_sex)\n\n# FVC vs. Percent (by Age)\nplt_fvc_vs_compare(first_visit, last_visit, x = 'Percent', y = 'FVC', z1 = 'Age', z2 = 'Age', color_palette = color_age)\n\n# FVC vs. weeks\nplt_fvc_vs_compare(train_df, train_df, x = 'Weeks', y = 'FVC', z1 = 'Percent', z2 = 'Age', color_palette = color_age)\n\n# # FVC vs Percent_mean (by Sex and Smoking Status)\nplt_fvc_vs_sns(df = patient_info_df, x = 'Percent_mean', y = 'FVC_mean', z = 'Sex', color_palette = color_sex)\nplt_fvc_vs_sns(df = patient_info_df, x = 'Percent_mean', y = 'FVC_mean', z = 'SmokingStatus', color_palette = color_sex)\n\n# # FVC vs Age (by Sex and Smoking Status)\nplt_fvc_vs_sns(df = train_df, x = 'Age', y = 'FVC', z = 'Sex', color_palette = color_sex)\nplt_fvc_vs_sns(df = train_df, x = 'Age', y = 'FVC', z = 'SmokingStatus', color_palette = color_sex)\n\n# # FVC vs Weeks (by Sex and Smoking Status)\nplt_fvc_vs_sns(df = train_df, x = 'Weeks', y = 'FVC', z = 'Sex', color_palette = color_sex)\nplt_fvc_vs_sns(df = train_df, x = 'Weeks', y = 'FVC', z = 'SmokingStatus', color_palette = color_sex)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}