{"cells":[{"metadata":{},"cell_type":"markdown","source":"Hello everyone!\n\nI recently joined a competition and decided to explore the table data provided. I think it can also be helpful in solving and understanding the problem. Wanna share my notes.\n\nIt is my first experience applying theory in statistic. So correct me if i make something wrong.\n\nHope this kernel will be helpful for you!","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom scipy import stats\nfrom scipy.stats import norm\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class config:\n    data_root = '../input/siim-isic-melanoma-classification'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Read data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(config.data_root + '/train.csv')\ntest = pd.read_csv(config.data_root + '/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Size of train data: ', train.shape)\nprint('Size of test data: ', test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's look at data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Cases of melanoma in train data',len(train[train.target == 1]) / len(train) * 100,'%')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Male in train data:',len(train[train.sex == 'male'])/len(train))\nprint('Male in test data:',len(test[test.sex == 'male'])/len(test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Unique patients in train data:', len(train.patient_id.unique())/len(train))\nprint('Unique patients in test data:', len(test.patient_id.unique())/len(test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Matual patients count',len(set(list(train.patient_id.unique())).intersection(list((test.patient_id.unique())))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We do not have very many unique patients and all patients in the train and test are different.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We may also notice that skipped age and gender are found together:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train.sex.isnull()].head(8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train[(train.sex.isnull()) & (train.age_approx.isnull())]) == len(train[train.sex.isnull()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only two people need this information","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[(train.sex.isnull()) & (train.age_approx.isnull())].patient_id.unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Anatom_site_general_challenge feature","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train[train.anatom_site_general_challenge.isnull()])/len(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.anatom_site_general_challenge.unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, we replace missing values with mean value and random value. \n\nFor missing site we create new value  - unknown","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train['anatom_site_general_challenge'] = train['anatom_site_general_challenge'].fillna('unknown')\ntest['anatom_site_general_challenge'] = test['anatom_site_general_challenge'].fillna('unknown')\n\ntrain['age_approx'] = train['age_approx'].fillna(train['age_approx'].mean())\ntest['age_approx'] = test['age_approx'].fillna(test['age_approx'].mean())\n\ntrain['sex'] = train['sex'].fillna('female')\ntest['sex'] = test['sex'].fillna('male')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train.isnull().any(1)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test[test.isnull().any(1)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (10,5))\nplt.subplot(1,2,1)\nsns.distplot(train.age_approx,fit = norm)\nplt.title('Age distribution in train data')\nplt.subplot(1,2,2)\nsns.distplot(test.age_approx,fit = norm)\nplt.title('Age distribution in test data')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Overall, the age distribution in the data is close to normal distribution.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Сreate table with information about each patient:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patients_age = pd.DataFrame(train.groupby(['patient_id'])['age_approx'].max())\npatients_age.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patients_disease = pd.DataFrame(train.groupby(['patient_id'])['target'].sum())\npatients_disease.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patients_gender = pd.DataFrame(train.groupby(['patient_id'])['sex'].max())\npatients_gender.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patients_history = patients_age.merge(patients_disease, on = 'patient_id')\npatients_history = patients_history.merge(patients_gender, on = 'patient_id')\npatients_history.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How are age and disease risk related?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"So, we have table where is information about patient. \nI want to test the effect of age on melanoma risk.\n\nFirst check age distribution in groups with people without melanoma and with this disease.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (10,5))\nplt.subplot(1,2,1)\nsns.distplot(patients_history[patients_history.target == 0].age_approx,fit = norm)\nplt.title('Benign cases')\nplt.subplot(1,2,2)\nsns.distplot(patients_history[patients_history.target != 0].age_approx,fit = norm)\nplt.title('Melanoma')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Q-Q Plot\nhttps://en.wikipedia.org/wiki/Q–Q_plot","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (10,5))\nplt.subplot(1,2,1)\nstats.probplot(patients_history[patients_history.target == 0].age_approx, plot=plt)\nplt.subplot(1,2,2)\nstats.probplot(patients_history[patients_history.target != 0].age_approx, plot=plt)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('M0 = ',patients_history[patients_history.target == 0].age_approx.mean())\nprint('M1 = ',patients_history[patients_history.target != 0].age_approx.mean())\n\nprint('D0 = ',patients_history[patients_history.target == 0].age_approx.var())\nprint('D1 = ',patients_history[patients_history.target != 0].age_approx.var())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Student's t-test\n\nhttps://en.wikipedia.org/wiki/Student%27s_t-test\n\nSample distributions are normal and we can use Student's t-test. We test hypothesis 0 that the means in these samples are equal.\n\nH0: M0=M1\n\nH1: M0!=M1\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.stats import ttest_ind","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"t,p = ttest_ind(patients_history[patients_history.target == 0].age_approx,patients_history[patients_history.target != 0].age_approx)\n\nprint('p-value = ', p)\nprint('p-value < 0.05 - ', p<0.05)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"p-value < 0.05 means that we reject the hypothesis of equality of means. Indeed, age is one of the susceptible risk factors, so we observe statistically significant differences.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Test connection between the location of a malignant mole and age","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I think that some areas of the body may be more at risk for sunburn. Thus, the age of patients with disease at different sites may differ.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train.target == 1].groupby(['anatom_site_general_challenge'])['target'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 15))\n\nfor i,site in enumerate(list(train.anatom_site_general_challenge.unique())):\n    plt.subplot(3,3,i+1)\n    sns.distplot(train[(train.anatom_site_general_challenge == site) & (train.target == 1)].age_approx,fit = norm)\n    plt.title('Age distribution with {} melanoma'.format(site))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I will use only four sites: 'head/neck', 'upper extremity','lower extremity', 'torso'","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"head_age = train[(train.target == 1) & (train.anatom_site_general_challenge == 'head/neck')].age_approx\nupper_age = train[(train.target == 1) & (train.anatom_site_general_challenge == 'upper extremity')].age_approx\nlower_age = train[(train.target == 1) & (train.anatom_site_general_challenge == 'lower extremity')].age_approx\ntorso_age = train[(train.target == 1) & (train.anatom_site_general_challenge == 'torso')].age_approx","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## One-way ANOVA test\n\n> The one-way ANOVA tests the null hypothesis that two or more groups have the same population mean. The test is applied to samples from two or more groups, possibly with different sizes.\n\nhttps://en.wikipedia.org/wiki/One-way_analysis_of_variance","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"(F,p_value) = stats.f_oneway(head_age,upper_age,lower_age,torso_age)\nprint('\\nF = {:.2f}, p-value = {:.4f}'.format(F,p_value)) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"p - value < 0.05 indicates that there are statistically significant differences between at least two groups.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Visualisation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"v = [head_age,upper_age,lower_age,torso_age]\navg = [a.mean() for a in v]\nerr = [1.96 * (np.std(a)/np.sqrt(len(a))) for a in v]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figsize = (30,45)\nplt.errorbar(x = ['head/neck','upper extremity','lower extremity','torso'], y = avg, yerr = err, color=\"black\", capsize=3, marker=\"s\", markersize=5, mfc=\"red\", mec=\"black\",fmt = 'o')\nplt.title('The age of the patient with melanoma at various sites')\nplt.grid()\nplt.xlabel('Site')\nplt.ylabel('Age')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***Multiple comparisons problem*** occurs when we compare many pairs of samples\n\nhttps://en.wikipedia.org/wiki/Multiple_comparisons_problem\n\n### Bonferroni correction \n\nhttps://en.wikipedia.org/wiki/Bonferroni_correction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"p = 0.05\nn = 4\n\nc = (n-1)*n/2\np = p / c\n\nprint(\"new p = {:.10f}\".format(p))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site_target = train[train.target == 1][(train.anatom_site_general_challenge == 'head/neck') | (train.anatom_site_general_challenge == 'upper extremity') | (train.anatom_site_general_challenge == 'lower extremity') | (train.anatom_site_general_challenge == 'torso')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"###  TukeyHSD\n\nhttps://en.wikipedia.org/wiki/Tukey%27s_range_test","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import statsmodels.stats.multicomp as multi\n\ntest = multi.MultiComparison(site_target['age_approx'], site_target['anatom_site_general_challenge'])\nres = test.tukeyhsd(alpha = p)\nsummary = res.summary()\nsummary","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that the age is different in the two pairs of samples. It can be assumed that the site plays an important role in the appearance of melanoma.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}