{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nimport pandas as pd\nimport numpy  as np\nimport plotly.express    as px\nimport matplotlib.pyplot as plt\nfrom sklearn.utils       import resample\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model    import LogisticRegression\nfrom sklearn.metrics         import roc_auc_score, accuracy_score\nfrom IPython.display         import YouTubeVideo","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# What is Melanoma?\n> A quick overview","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"YouTubeVideo('mkYBxfKDyv0', width=800, height=450)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Prevalence and Mortality Rate\n- Between 2 to 3 million non-melanoma skin cancers and 132,000 melanoma skin cancers occur globally each year.\n- Melanoma is the deadliest form of skin cancer, although far less prevalent than non-melanoma skin cancers.\n- The incidence of melanoma has more than doubled in the last 30 years.\n- Caught early, we can have as much as 98% treatment success, but this rate drops dramatically as the cancer progresses.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![Melanoma Treatment Success by stage](https://www.curemelanoma.org/assets/Uploads/_resampled/ResizedImageWzQwMCwyOTZd/MRA18574-5YearSurviveFig-V1-Hex.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> Risk Factors\n- Family history\n- Personal history\n- Light hair color\n- Light eye color\n- High freckle density\n- Imunosupression\n- Multiple moles","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> Symptons\n- Bleeding\n- Itching\n- Scabbing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> Examination guidelines - when thinking about data augmentation for the images, make sure to bear these guidelines in mind!\n- (A) Assymetry\n- (B) Borders\n- (C) Color\n- (D) Diameter\n- (E) Elevation and Evolution","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"YouTubeVideo('hXYd0WRhzN4', width=800, height=450)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read files\ntrain_df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest_df  = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How many images do we have per patient?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Training set contains {} images of {} unique patients, resulting in a ratio of {} images per patient'.format(train_df.shape[0],\n                                                                                                                     train_df.patient_id.nunique(),\n                                                                                                                     round(train_df.shape[0] / train_df.patient_id.nunique(),2)\n                                                                                                                    ))\n\nprint('Testing  set contains {} images of {}  unique patients, resulting in a ratio of {} images per patient'.format(test_df.shape[0],\n                                                                                                                 test_df.patient_id.nunique(),\n                                                                                                                 round(test_df.shape[0] / test_df.patient_id.nunique(),2)\n                                                                                                                ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Do we have missing data?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> Yes, in both training and testing sets.\n\n> For the training set, missingness occurs mostly in the `diagnosis` column, but also present in the `age_approx` and `anatom_site_general_challenge`columns.\nNote that for the `diagnosis` column, I'm considering the value 'unknown' as missingness.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Encode 'unknowns' as NaNs\ntrain_df['diagnosis'] = train_df.diagnosis.apply(lambda x: np.nan if x == 'unknown' else x)\n\nlabels_df = pd.DataFrame(train_df.benign_malignant.value_counts()).reset_index()\nlabels_df.columns = ['Label','Count']\n\n# Create dataframe counting NaN values per column\nnan_df = pd.DataFrame(train_df.isna().sum()).reset_index()\nnan_df.columns  = ['Column', 'NaN_Count']\nnan_df['NaN_Count'] = nan_df['NaN_Count'].astype('int')\nnan_df['NaN_%'] = round(nan_df['NaN_Count']/train_df.shape[0] * 100,1)\nnan_df['Type']  = 'Missingness'\nnan_df.sort_values('NaN_%', inplace=True)\n\n\n# Add completeness\nfor i in range(nan_df.shape[0]):\n    complete_df = pd.DataFrame([nan_df.loc[i,'Column'],train_df.shape[0] - nan_df.loc[i,'NaN_Count'],100 - nan_df.loc[i,'NaN_%'], 'Completeness']).T\n    complete_df.columns  = ['Column','NaN_Count','NaN_%','Type']\n    complete_df['NaN_%'] = complete_df['NaN_%'].astype('int')\n    complete_df['NaN_Count'] = complete_df['NaN_Count'].astype('int')\n    nan_df = nan_df.append(complete_df, sort=True)\n    \n    \n# Missingness Plot\nfig = px.bar(nan_df,\n             x='Column',\n             y='NaN_%',\n             title='Missingness on the Training Set',\n             color='Type',\n             template='plotly_dark',\n             opacity = 0.6,\n             color_discrete_sequence=['#dbdbdb','#38cae0'])\n\nfig.update_xaxes(title='Column Name')\nfig.update_yaxes(title='NaN Percentage')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count NaNs\ntrain_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> For the testing set, we only have missing values in the `anatom_site_general_challenge` column.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"labels_df = pd.DataFrame(train_df.benign_malignant.value_counts()).reset_index()\nlabels_df.columns = ['Label','Count']\n\n# Create dataframe counting NaN values per column\nnan_df = pd.DataFrame(test_df.isna().sum()).reset_index()\nnan_df.columns  = ['Column', 'NaN_Count']\nnan_df['NaN_Count'] = nan_df['NaN_Count'].astype('int')\nnan_df['NaN_%'] = round(nan_df['NaN_Count']/test_df.shape[0] * 100,1)\nnan_df['Type']  = 'Missingness'\nnan_df.sort_values('NaN_%', inplace=True)\n\n\n# Add completeness\nfor i in range(nan_df.shape[0]):\n    complete_df = pd.DataFrame([nan_df.loc[i,'Column'],test_df.shape[0] - nan_df.loc[i,'NaN_Count'],100 - nan_df.loc[i,'NaN_%'], 'Completeness']).T\n    complete_df.columns  = ['Column','NaN_Count','NaN_%','Type']\n    complete_df['NaN_%'] = complete_df['NaN_%'].astype('int')\n    complete_df['NaN_Count'] = complete_df['NaN_Count'].astype('int')\n    nan_df = nan_df.append(complete_df, sort=True)\n    \n    \n# Missingness Plot\nfig = px.bar(nan_df,\n             x='Column',\n             y='NaN_%',\n             title='Missingness on the Testing Set',\n             color='Type',\n             template='plotly_dark',\n             opacity = 0.6,\n             color_discrete_sequence=['#dbdbdb','#38cae0'])\n\nfig.update_xaxes(title='Column Name')\nfig.update_yaxes(title='NaN Percentage')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count NaNs\ntest_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How are the labels distributed?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> Very imbalanced, 1.8% (or 584 images) belong to the positive class.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Summarise data\ncount_df = labels_df.iloc[::-1]\n\n# Create annotations\nannotations = [dict(\n            y=count_df.loc[i,'Label'],\n            x=count_df.loc[i,'Count'] + 1000,\n            text=str(round(count_df.loc[i,'Count']/train_df.shape[0]*100,1))+'%',\n            font=dict(\n            size=14,\n            color=\"#000000\"\n            ),\n            bordercolor=\"#c7c7c7\",\n            borderwidth=1,\n            borderpad=4,\n            bgcolor=\"#ffffff\",\n            opacity=0.95,\n            showarrow=False,\n        ) for i in range(count_df.shape[0])]\n\n\n\nfig = px.bar(labels_df,\n             y = 'Label',\n             x = 'Count',\n             title       = 'Label Distribution',\n             template    = 'plotly_dark',\n             orientation = 'h',\n             opacity     = 0.7,\n             color       = 'Label',\n             color_discrete_sequence = ['#38cae0','#d1324d'] \n            )\n\n\nfig.update_layout(showlegend=False, annotations = annotations)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How much predictive power does the age, anatomical site and gender carry?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> They carry moderate predictive power.. a dull logistic regression with only those three variables was able to achieve **0.66 AUROC**.\n\n> From inspecting the regression coefficients, it is certainly worth considering all variables when creating your own validation set, as they carry some predictive power.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.histogram(train_df,\n             x     = 'age_approx',\n             color = 'target',\n             color_discrete_sequence = ['#38cae0','#d1324d'],\n             barnorm  = 'fraction',\n             template = 'plotly_dark',\n             opacity  = 0.7,\n             title    = 'Impact of Age across Diagnosis'\n            )\n\nfig.update_xaxes(title = 'Approximated Age', tickvals = list(range(0,91,5)))\nfig.update_yaxes(title = 'Percentage of class total')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> From the plot, we can see that age is correlated with melanoma, this is in line with scientific findings.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"parallel_df = train_df.copy()\n\nundersampled_df = pd.concat([parallel_df.query(\"target == 1\"),resample(parallel_df.query(\"target == 0\"),\n                                                                       replace   = False,\n                                                                       n_samples = 584,\n                                                                       random_state = 451)\n                            ],axis=0)\n\n\nkeep_list = ['sex','age_approx','anatom_site_general_challenge','target']\nfig = px.parallel_categories(undersampled_df[keep_list],\n                              color=\"target\",\n                              template='plotly_dark',\n                              labels={\"age_approx\": \"Approximate Age\",\"sex\": \"Sex\", 'anatom_site_general_challenge':'Anatomical Site','target':'Melanoma'},\n                              color_continuous_scale=['#dbdbdb','#38cae0'],\n                              title='Categorical Flow'\n                             )\n\nfig.update_layout(showlegend=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> A few things we can observe:\n- Sex doesn't seem to play a major role, males appear to be more likely to develop melanoma;\n- Younger age groups seem almost unaffected, but have fewer observations as well;\n- Some anatomical sites are more likely to develop this type of cancer, such Head/Neck\n- Oral/genital and Pals/Soles have very little observations to infer anything.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def prepare_dataframe(df):\n    df['sex'] = np.where(df['sex'] == 'female',1,0)\n    df = pd.concat([df.drop('anatom_site_general_challenge',axis=1), pd.get_dummies(df['anatom_site_general_challenge'])],axis=1)\n    df = df.drop(['benign_malignant','image_name','patient_id','diagnosis'],axis=1)\n    df.loc[df['age_approx'].isnull(),'age_approx'] = df['age_approx'].median()\n    \n    return(df)\n\ndef evaluate_predictions(preds, test_labels):\n    '''\n    Evaluate Predictions Function\n    Returns accuracy and auc of the model\n    '''\n    auroc = roc_auc_score(test_labels.astype('uint8'), preds)\n    accur = accuracy_score(test_labels.astype('uint8'), preds >= 0.5)\n    print('Accuracy: ' + str(accur))\n    print('AUC: ' + str(auroc))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split data\ntrain, probe = train_test_split(prepare_dataframe(train_df),\n                                test_size = 0.3,\n                                stratify = train_df['target'],\n                                random_state = 451\n                               )\n\ntrain_y = train.pop('target')\nprobe_y = probe.pop('target')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"logit_model = LogisticRegression(random_state=451, solver='lbfgs', max_iter=1000)\nlogit_model.fit(train, train_y)\n\nlogit_preds = logit_model.predict_proba(probe)\nevaluate_predictions(logit_preds[:,1], probe_y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From these three variables alone we are able to achieve **0.66 AUROC!**\n\nNow let's see what we can learn from the coeficients.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.bar(y = logit_model.coef_.tolist()[0],\n       x = probe.columns.tolist(),\n       template = 'plotly_dark',\n       title = 'Logistic Regression Coefficient Values',\n       color = logit_model.coef_.tolist()[0],\n       color_continuous_scale = ['#d1285b','#28b5d1'],\n       opacity = 0.7\n      )\n\nfig.update_yaxes(title = 'Coefficient Value')\nfig.update_xaxes(title = 'Variable Name')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> We can note a few things:\n- Metadata was very usefull for melanoma classification;\n- Palms/Soles and Oral/Genital may appear very relevant, but have very little data points and little statistical significance.\n- Lower extremity and Torso are less likely to develop melanoma, since they are not as exposed as other regions, though it is not impossible;\n- Head/Neck carries a moderate likelyhood of melanoma with statiscal significance.\n- Age seems small, but this is due to unit of input - the older, the more likely to develop melanoma (this is supported scientifically as well)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### What does the Images look like?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Benign class","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_multiple_images(image_dataframe, rows = 4, columns = 4, figsize = (16, 20), resize=(1024,1024), preprocessing=None, label = 0):\n    '''\n    Plots Multiple Images\n    Reads, resizes, applies preprocessing if desired and plots multiple images from a given dataframe\n    '''\n    query_string    = 'target == {}'.format(label)\n    image_dataframe = image_dataframe.query(query_string).reset_index(drop=True)\n    fig = plt.figure(figsize=figsize)\n    ax  = []\n    base_path = '../input/siim-isic-melanoma-classification/jpeg/train/'\n    \n    for i in range(rows * columns):\n        img = plt.imread(base_path + image_dataframe.loc[i,'image_name'] + '.jpg')\n        img = cv2.resize(img, resize)\n        \n        if preprocessing:\n            img = preprocessing(img)\n        \n        ax.append(fig.add_subplot(rows, columns, i+1) )\n        plot_title = \"Image {}: {}\".format(str(i+1), 'Benign' if label == 0 else 'Malignant') \n        ax[-1].set_title(plot_title)\n        plt.imshow(img, alpha=1, cmap='gray')\n    \n    plt.show()\n\n\nplot_multiple_images(train_df, label = 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Malignant class","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_multiple_images(train_df, label = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> These images are pretty well behaved.\nFrom my field experience, I've had contact with many dermatologists, and seldom is the case where the acquisition protocol is so well established. Usually the images are far more spontaneous, with considerably differences regarding brightness, distance, camera type and so on.\n\n> If you are considering external datasets, which I recommend given the small amount of images, I suggest having a closer look to how the images were acquired.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Closing Remarks","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> External datasets may be crucial in order to reach a reasonable amount of **Malignant cases**, note however, that some datasets don't follow the same image acquisition protocol as this one, so I suggest ISIC datasets from previous years.\n\n> **Age and anatomical site** are very usefull and are certainly advised to be considered when making your splits and models.\n\n> Many obervations are from the same patient, and thus the patient_id might be useful for good CV splits as well.\n\n> Images show a moderate variation, due to skin type, hair, locale , which could mean that the combo of **segmentation + classification** could be usefull for this competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## References","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"- https://www.who.int/news-room/q-a-detail/ultraviolet-(uv)-radiation-and-skin-cancer\n- https://www.curemelanoma.org/\n- https://www.wcrf.org/dietandcancer/cancer-trends/skin-cancer-statistics","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Recommended resources","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"External datasets from ISIC:\n- [ISIC 2019](https://www.kaggle.com/andrewmvd/isic-2019); (Note that the 2019 version contains both 2018 and 2017 datasets as well)\n- [ISIC 2018](https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000);\n- [ISIC 2017](https://www.kaggle.com/wanderdust/skin-lesion-analysis-toward-melanoma-detection).\n\nAlso suggest looking at this [thread](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296) for some info on the competition dataset.\nThere is some overlap with these images and the competition training set, however, as we have only 584 images of the positive class, gathering relevant external data will certainly be crucial for this competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now let's kick some cancer ass.\n\nIf you have any questions, let me know!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}