{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport glob\n\n# For plots we use plotly express and\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nfrom plotly.colors import n_colors","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**NOTE: The project is currently under development**\n\n**Table of Content**\n\n* Understand train.csv\n* Understand test.csv\n* Understand the dcm file\n\n\n**Some preliminary understanding of the domain we are dealing with:**\n\n**Forced vital capacity (FVC)** is the total amount of air exhaled during the FEV (Force Expiratory Volume) Test. It is important to add that FVC is a calculation of the volume of the lung. Since the amount of air you are able to exhale after a complete inhalation should represents the amount of air that the lung is able to accomodate at a point in time.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Understanding train.csv \n\nFirst let us check and try to understand the data in train.csv","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\nprint(train_df.head())\nprint(train_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's do a brief description of the data\n\nHow many unique patients do we have?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(train_df['Patient'].unique()))\n#print(train_df['Patient'].unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we know that we have 1549 records with 176 unique patients.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df['Weeks'].describe())\n# print(train_df[train_df['Weeks'].isnull()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"According to the data given, we know that the FVC is measured during several weeks interval over the course of 1 - 2 years, hence there is some time series visualization that could be plausible.\n\nHowever we should have at the back of our mind that there is no specific week intervals but maybe we should first check that.\n\nQuestion: What is the frequent week that measurements were taken?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"value_count_week = train_df['Weeks'].value_counts()\n\n\ndata  = go.Bar(\n            x = value_count_week.index,\n            y = value_count_week.values,\n            width = 1.0,\n            marker_color='orangered')\nlayout = go.Layout(\n            height = 500,\n            width = 1200,\n            xaxis=dict(range=[-5, 130], autorange=False, zeroline=False, title=\"Appointment week\"),\n            yaxis=dict(zeroline=False, title='Number of patients on week'),\n            title = \"Which weeks are people most coming in for checkup\",\n        )\nfig  = go.Figure(data=data, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This graph is very interesting as we see that the patients do come in for their follow-up checkups in early weeks. Also, we can see that there is are no record sometimes of the zeroth week i.e first appointment hence first appoint could be some week in the future.\n\nHere are the interesting questions.\n* What are the characteristics of the majority of people up to the 20 weeks of checkup, what characterize the FCV values.\n    - An interesting fact could be that majority stop going for check ups after the 20th week.\n* Is it possible that patients stop coming for checkup after there FCV Value drops to a particular point.\n\nWe need to check if it is possible that the negatives are for the start of those patients that (if yes then we can assume that that is their first apppointment)\n\n* Let us check for how many chekups each unique individual make within the first 20 weeks.\n* Also check appointments count for the various smoker and non-smoker groups","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from plotly.subplots import make_subplots\nfig1 = px.scatter(train_df, x=\"Weeks\", y=\"Patient\", color='SmokingStatus',\n                 title=\"Appointment distribution for all patients\",\n                 labels={\"Weeks\":\"Appointment week (when they showed up)\"})\nfig1.update_yaxes(showticklabels=False)\nfig1.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One critical observation to see from the appointment show up for the various patients is that you will see that the group that currently smokes tried to show up even more often up to the 40week mark compared to other groups","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"total_appointment = train_df.groupby(['Patient', 'SmokingStatus']).size().reset_index(name=\"total_appointment\")\ntotal_appointment_20wks = train_df[train_df[\"Weeks\"] <= 20].groupby(['Patient']).size().reset_index(name=\"appt_within_20_Weeks\")\ntotal_appointment_40wks = train_df[train_df[\"Weeks\"] <= 40].groupby(['Patient']).size().reset_index(name=\"appt_within_40_Weeks\")\n\nfig = make_subplots(rows=1, cols=3,\n                   subplot_titles=(\"Total Appointment within 20 weeks\",\n                                   \"Total Appointment within weeks\",\n                                   \"Total Appointment count all weeks\"))\nfig.add_trace(go.Scatter(\n    x=total_appointment_20wks['Patient'], y=total_appointment_20wks['appt_within_20_Weeks'],\n    mode='markers'\n), row=1, col=1)\nfig.add_trace(go.Scatter(\n    x=total_appointment_40wks['Patient'], y=total_appointment_40wks['appt_within_40_Weeks'],\n    mode='markers'\n), row=1, col=2)\nfig.add_trace(go.Scatter(\n    x=total_appointment['Patient'], y=total_appointment['total_appointment'],\n    mode='markers'\n), row=1, col=3)\nfig.update_xaxes(showticklabels=False, title_text='Patient')\nfig.update_xaxes(title_text='Count appointments')\nfig.update_layout(showlegend=False, title_text=\"Comparing appointments within various week intervals\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total_appontment_20wks = train_df[train_df[\"Weeks\"] <= 20].groupby(['Patient', 'SmokingStatus']).size().reset_index(name=\"appt_within_20_Weeks\")\nfig = px.scatter(total_appontment_20wks, x=\"Patient\", y=\"appt_within_20_Weeks\",\n                 facet_col='SmokingStatus', color_discrete_sequence=[\"coral\"])\nfig.update_xaxes(showticklabels=False, title_text='Patient')\nfig.update_yaxes(title_text='Count appointments')\nfig.show()\n\ntotal_appontment_40wks = train_df[train_df[\"Weeks\"] <= 40].groupby(['Patient', 'SmokingStatus']).size().reset_index(name=\"appt_within_40_Weeks\")\nfig = px.scatter(total_appontment_40wks, x=\"Patient\",\n                 y=\"appt_within_40_Weeks\", facet_col='SmokingStatus',\n                 color_discrete_sequence=[\"goldenrod\"])\nfig.update_xaxes(showticklabels=False, title_text='Patient')\nfig.update_yaxes(title_text='Count appointments')\nfig.show()\n\nfig = px.scatter(total_appointment, x=\"Patient\",\n                 y=\"total_appointment\", facet_col='SmokingStatus',\n                 color_discrete_sequence=[\"blueviolet\"])\nfig.update_xaxes(showticklabels=False, title_text='Patient')\nfig.update_yaxes(title_text='Count appointments')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have a record of usually 9 appontments for each of the ID given in the dataset and more of these appointments occured in the first 20 weeks.\nIt is obvious per according to the images above that their is really no special pattern of appointment for the various smoking status.\nLet's hava a discussion around the appointment weeks and the FVC.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"Weeks\", y=\"FVC\", color='SmokingStatus',\n                 title=\"Appointment distribution based on FCV level\",\n                 labels={\"Weeks\":\"Appointment week (when they showed up)\"})\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"smoking_status = train_df[train_df['SmokingStatus']==\"Ex-smoker\"]\nfig = go.Figure()\n\ncolors = n_colors('rgb(5, 200, 200)', 'rgb(200, 10, 10)', len(train_df['Patient'].unique()), colortype='rgb')\n\nfor (j, color) in zip(train_df['Patient'].unique(), colors):\n    data = smoking_status[smoking_status['Patient'] == j]\n    fig.add_trace(go.Violin(\n        x = data['FVC'], line_color=color\n    ))\nfig.update_layout(\n    showlegend=False, title_text=\"FVC distribution - Ex smokers\"\n)\nfig.update_yaxes(showticklabels=False, title_text='Patient')\nfig.update_traces(orientation='h', side='positive', width=12, points=False)\nfig.update_layout(xaxis_showgrid=False, xaxis_zeroline=False)\n\nfig.show()\n\n\n\nsmoking_status = train_df[train_df['SmokingStatus']==\"Never smoked\"]\nfig = go.Figure()\nfor (j, color) in zip(train_df['Patient'].unique(), colors):\n    data = smoking_status[smoking_status['Patient'] == j]\n    fig.add_trace(go.Violin(\n        x = data['FVC'], line_color=color\n    ))\nfig.update_layout(\n    showlegend=False, title_text=\"FVC distribution - Never smoked patients\"\n)\nfig.update_yaxes(showticklabels=False, title_text='Patient')\nfig.update_traces(orientation='h', side='positive', width=8, points=False)\nfig.update_layout(xaxis_showgrid=False, xaxis_zeroline=False)\nfig.show()\n\n\nsmoking_status = train_df[train_df['SmokingStatus']==\"Currently smokes\"]\nfig = go.Figure()\nfor (j, color) in zip(train_df['Patient'].unique(), colors):\n    data = smoking_status[smoking_status['Patient'] == j]\n    fig.add_trace(go.Violin(\n        x = data['FVC'], line_color=color\n    ))\nfig.update_layout(\n    showlegend=False, title_text=\"FVC distribution - Currently smokes\"\n)\nfig.update_yaxes(showticklabels=False, title_text='Patient')\nfig.update_traces(orientation='h', side='positive', width=8, points=False)\nfig.update_layout(xaxis_showgrid=False, xaxis_zeroline=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see obvious irregularity in the FVC distribution over the course of their appointment weeks, which is not very visible for Ex smokers and those that have never done smoking before.\n\nAlso, some interesting observations to corraborate the evaluation of appointment weeks above, we see that the data present quite a number of smokers appointment after the 10th week, We may need to go back and recalculate how many appointment ***after the first appointment rather than from point 0***","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.violin(train_df, y=\"FVC\", x=\"SmokingStatus\", box=True, # draw box plot inside the violin\n                points='all', # can be 'outliers', or False\n               )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see a clear disparity between the FVC values of the smokers and the other groups, although, something is a little fishy in the violin plot for the never smoked individuals, Why would some values be close to the FVC of smokers?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"smoking_status = train_df[train_df['SmokingStatus']==\"Ex-smoker\"]\nfig = go.Figure()\n\ncolors = n_colors('rgb(5, 200, 200)', 'rgb(200, 10, 10)', len(train_df['Patient'].unique()), colortype='rgb')\n\nfor (j, color) in zip(train_df['Patient'].unique(), colors):\n    data = smoking_status[smoking_status['Patient'] == j]\n    fig.add_trace(go.Violin(\n        x = data['Percent'], line_color=color\n    ))\nfig.update_layout(\n    showlegend=False, title_text=\"Percent distribution - Ex smokers\"\n)\nfig.update_yaxes(showticklabels=False, title_text='Patient')\nfig.update_traces(orientation='h', side='positive', width=12, points=False)\nfig.update_layout(xaxis_showgrid=False, xaxis_zeroline=False)\n\nfig.show()\n\n\n\nsmoking_status = train_df[train_df['SmokingStatus']==\"Never smoked\"]\nfig = go.Figure()\nfor (j, color) in zip(train_df['Patient'].unique(), colors):\n    data = smoking_status[smoking_status['Patient'] == j]\n    fig.add_trace(go.Violin(\n        x = data['Percent'], line_color=color\n    ))\nfig.update_layout(\n    showlegend=False, title_text=\"Percent distribution - Never smoked patients\"\n)\nfig.update_yaxes(showticklabels=False, title_text='Patient')\nfig.update_traces(orientation='h', side='positive', width=8, points=False)\nfig.update_layout(xaxis_showgrid=False, xaxis_zeroline=False)\nfig.show()\n\n\nsmoking_status = train_df[train_df['SmokingStatus']==\"Currently smokes\"]\nfig = go.Figure()\nfor (j, color) in zip(train_df['Patient'].unique(), colors):\n    data = smoking_status[smoking_status['Patient'] == j]\n    fig.add_trace(go.Violin(\n        x = data['Percent'], line_color=color\n    ))\nfig.update_layout(\n    showlegend=False, title_text=\"Percent distribution - Currently smokes\"\n)\nfig.update_yaxes(showticklabels=False, title_text='Patient')\nfig.update_traces(orientation='h', side='positive', width=8, points=False)\nfig.update_layout(xaxis_showgrid=False, xaxis_zeroline=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The percentage distribution also corraborate the PVC distribution well, the graph is very similar in distribution.\n\nAlthough we do have a clear observation of difference in FVC distribution clearly seperating the smokers and non-smokers/ex-smokers, we don't yet see anything that differentiates the non-smokers and ex-smokers, let's try some mean, median scaling over the Percent and FVC and see what the distribution looks like.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"statistical_df = train_df.groupby(['Patient', 'SmokingStatus', 'Age', 'Sex']).agg(\n            {'FVC': ['mean', 'median', 'min', 'max']})\nstatistical_df.columns = ['fvc_mean', 'fvc_median', 'fvc_min', 'fvc_max']\nstatistical_df = statistical_df.reset_index()\nstatistical_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.box(statistical_df, x=\"Sex\", y=\"fvc_mean\", color=\"SmokingStatus\")\nfig.update_layout(\n    showlegend=True, title_text=\"Mean FVC distribution among smokers\"\n)\nfig.show()\n\nfig = px.box(statistical_df, x=\"Sex\", y=\"fvc_median\", color=\"SmokingStatus\")\nfig.update_layout(\n    showlegend=True, title_text=\"Median FVC distribution among smokers\"\n)\nfig.show()\n\nfig = px.box(statistical_df, x=\"Sex\", y=\"fvc_min\", color=\"SmokingStatus\")\nfig.update_layout(\n    showlegend=True, title_text=\"Minimum FVC distribution among smokers\"\n)\nfig.show()\nfig = px.box(statistical_df, x=\"Sex\", y=\"fvc_max\", color=\"SmokingStatus\")\nfig.update_layout(\n    showlegend=True, title_text=\"Maximum FVC distribution among smokers\"\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I think it is safe to assume that your lung capacity can actually improve as an ex-smoker because most FVC distributions of never smoked individuals and ex-smokers usually fall into the same distribution according to the data given.\n\nThe distibution accross the mean, median, min and max all point to the same observation above.\n\nLet us check age distribution\n\nWe should also see the general age distribution that is provided by the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.violin(statistical_df, x='Sex', y=\"Age\", color=\"SmokingStatus\", box=True, points='all')\nfig.update_layout(\n    showlegend=True, title_text=\"Age distribution\"\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let us look at the FVC distribution over age for the various smoking status and sex.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(statistical_df, x='fvc_mean', y='Age', color='Sex', facet_col='SmokingStatus', facet_col_wrap=4)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['Patient'] == \"ID00336637202286801879145\"]['Weeks'].diff()\n# We got to find the time lapsed difference for each of the persons","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are safe when it comes to the weeks column, it seems all weeks were reported for all the patients, but what is it with the negative week numbers?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['Patient'] == 'ID00007637202177411956430'].shape\n#Something to keep in mind, You need to validate that the number of test recorded is the same and the numbe of dcm file provided for each of the Patient ID","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Credit:\n\nI took a lot of inspiration for this work from an interesting kaggler - [Laura Fink](https://www.kaggle.com/allunia), people you should check her page out, she did a lot of awesome stuffs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# smoking_status = train_df[train_df['SmokingStatus']==\"Currently smokes\"]\n# fig = go.Figure()\n# for j in train_df['Patient'].unique():\n#     data = smoking_status[smoking_status['Patient'] == j]\n#     fig.add_trace(go.Violin(\n#         x = data['Weeks'], y=data['FVC']\n#     ))\n# fig.update_layout(\n#     showlegend=False, title_text=\"FVC distribution - Currently smokes\"\n# )\n\n# fig.show()\n\n\n# for j in train_df['Patient'].unique():\n#     data = smoking_status[smoking_status['Patient'] == j]\n#     fig.add_trace(go.Scatter(\n#         x = data['Weeks'], y=data['FVC']\n#     ))\n# fig.update_layout(\n#     showlegend=False, title_text=\"FVC distribution - Ex smokers\"\n# )\n\n# fig.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}