{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## About this notebook\n\nIn this notebook , I will start with complete explanation of everything you need know related to Skin Cancer(Melanoma) and its detection and I will built on that to explain the dataset and perform extensive EDA.\n\n**This kernel will be a work in Progress,and I will keep on updating it as the competition progresses**\n\n**<span style=\"color:Red\">If you find this kernel useful, Please consider Upvoting it, it motivates me to write more Quality content**\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Domain Knowledge\n\nBefore starting with indepth EDA, i would like to shed some light on the domain knowledge of Skin Cancer by answering few questions related to skin cancer.\n\n### Q1) What is the motivation behind this competition?\n\nSkin cancer is the most prevalent type of cancer. Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n\nCurrently, dermatologists evaluate every one of a patient's moles to identify outlier lesions or “ugly ducklings” that are most likely to be melanoma. Existing AI approaches have not adequately considered this clinical frame of reference. Dermatologists could enhance their diagnostic accuracy if detection algorithms take into account “contextual” images within the same patient to determine which images represent a melanoma. If successful, classifiers would be more accurate and could better support dermatological clinic work.\n\n### Q2) What is Skin Cancer?\n\nCancer is a disease in which normal cells undergo change, grow and duplicate without normal control. **Skin cancer** is an abnormal growth of skin cells. This implies that the abnormal cells grow in a disorganized fashion, invade the encompassing tissue and disrupt normal tissue function. Most often develops on skin that has soaked up the sun for years.\n\n\n### Q3) What are the types of Skin Cancer?\n\nThere are three major types of skin cancers: **basal cell carcinoma (BCC)**, **squamous cell carcinoma (SCC)**, and **melanoma**. The first two skin cancers are grouped together as **non-melanoma** skin cancers. Other unusual types of skin cancer include Merkel cell tumors and dermatofibrosarcoma protruberans.\n\n\n![](https://miiskin.com/wp-content/uploads/2019/08/types-of-skin-cancer.jpg)\n\n\n### Q4) What is Melanoma Skin Cancer?\n\n**Melanoma Skin Cancer** is the deadliest form of cancer. It originates in the pigment-producing melanocytes in the basal layer of the epidermis. Clinically, it can be identified through a skin biopsy.\n\n**Melanoma Skin Cancer** is life-threatening and can grow very quickly. It develops when unrepaired DNA damage to the skin cells triggers mutations that lead the skin cells to multiply rapidly and form malignant tumours.\n\n**Melanoma Skin Cancer** often resembles or may develop from moles. Brown or black skin lesions but can appear as pink, red, blue or white. These cancerous growths have irregular symmetry, edges that may be scalloped or notched and are larger in diameter (more than 7mm). There may be spread of pigment from the border of a spot into the surrounding skin. Melanoma skin cancer may be accompanied by both change in sensation (itchiness, pain) and surface of the mole (oozing. scaliness, bleeding).\n\n\n### Q5) Skin Cancer Signs: The ABCDEs of Melanoma\n\n\n![](https://i1.wp.com/cancerworld.info/wp-content/uploads/2015/10/ABCDE-Melanoma-Skin-Cancer-Pictures-Images-Sign-Symptoms.jpg)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import IFrame, YouTubeVideo\nYouTubeVideo('hXYd0WRhzN4',width=800, height=500)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Q6) What are the causes of Skin Cancer?\n\nMore than 90% of the cases are caused by excessive exposure to ultraviolet radiations emitted by the sun. This excessive emission occurs due to the thinning of the ozone layer which of course acts as a filter to the harmful ultraviolet radiations. People with light skin colour, hazel or blue eyes, extra sensitive skin are exposed more towards the risk.\n\nWhen ultraviolet radiations contribute mostly to this cause, other causes include smoking, having tobacco, sometimes genetic syndromes like moles that increase drastically, chronic wounds that are non-healing, Xrays, artificial methods of getting a tan like tanning beds, use of medicines that decrease immunity etc.\n\nSmoking tobacco not only increases the chances of causing other types of cancer but also it contributes to an effective cause of skin cancer.  Moles, abnormal growth marks since birth, birthmarks are not exactly cancerous but if at all there is any kind of growth in them, it is a cause to worry.  Tanning beds are very common to get an artificial, cosmetic tan and as much as they are successful in providing tan, there are approximately 90% cases in which skin cancer occurs after this treatment.\n\nAny wound that has been active for more than the stipulated time, blister-like red-pinkish mark or any rash that is growing every day, needs to be consulted.\n\nIt is said that use of sunscreens is an effective way of preventing skin cancer; however, the effectiveness cannot be measured. The changing lifestyle and more exposure to the harmful rays, all this drastically contributes to causing skin cancer.\n\n\n\n### Q7) How the dataset look likes?\n\n**Files**\n* `train.csv` - the training set\n* `test.csv` - the test set\n* `sample_submission.csv` - a sample submission file in the correct format\n\n**Columns**\n* `image_name` - unique identifier, points to filename of related DICOM image\n* `patient_id` - unique patient identifier\n* `sex` - the sex of the patient (when unknown, will be blank)\n* `age_approx` - approximate patient age at time of imaging\n* `anatom_site_general_challenge` - location of imaged site\n* `diagnosis` - detailed diagnosis information (train only)\n* `benign_malignant` - indicator of malignancy of imaged lesion\n* `target` - binarized version of the target variable\n\n\n### Q8) What should I expect the data format to be?\n\n* The images are provided in DICOM format. This can be accessed using commonly-available libraries like [pydicom](https://pydicom.github.io/), and contains both image and metadata. It is a commonly used medical imaging data format.\n* Images are also provided in JPEG and TFRecord format (in the `jpeg` and `tfrecords` directories, respectively). Images in TFRecord format have been resized to a uniform 1024x1024.\n* Metadata is also provided outside of the DICOM format, in CSV files.\n\n\n### Q9) What do we need to predict?\n\n* We are predicting a binary target for each image. Your model should predict the probability (floating point) that the lesion in the image is **malignant** (the target).\n* In the training data, train.csv, the value 0 denotes **benign**, and 1 indicates **malignant**. \n* Predictions should be floating point values between `0.0` and `1.0`, with `0.5` as a binary decision threshold.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2. Getting started with the SIIM-ISIC Melanoma Classification dataset\n\n\nThis notebook shows a few methods to load and display images from the SIIM-ISIC Melanoma Classification challenge dataset. \n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#plotly\n!pip install --upgrade pip --quiet\n!pip install chart_studio --quiet\n\nimport os\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n#pydicom\nimport pydicom\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n\n# Settings for pretty nice plots\nplt.style.use('fivethirtyeight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input/siim-isic-melanoma-classification/')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Loading Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_PATH = '../input/siim-isic-melanoma-classification'\n\n\n\nprint('Reading data...')\ntrain = pd.read_csv(f'{BASE_PATH}/train.csv')\ntest = pd.read_csv(f'{BASE_PATH}/test.csv')\nsubmission = pd.read_csv(f'{BASE_PATH}/sample_submission.csv')\nprint('Reading data completed')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train.head())\nprint(\"Shape of train :\", train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(test.head())\nprint(\"Shape of test :\", test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Checking for Null values\n\n**train**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = train.isnull().sum().sort_values(ascending = False)\npercent = (train.isnull().sum()/train.isnull().count()*100).sort_values(ascending = False)\nmissing_train_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**test**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = test.isnull().sum().sort_values(ascending = False)\npercent = (test.isnull().sum()/test.isnull().count()*100).sort_values(ascending = False)\nmissing_test_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Basic EDA\n\n### Let's Start with distribution of variables in train.csv\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.1checking for `benign_malignant` distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_count(df, feature, title='', size=2.5):\n    f, ax = plt.subplots(1,1, figsize=(4*size,3*size))\n    total = float(len(df))\n    sns.countplot(df[feature],order = df[feature].value_counts().index, palette='Set2')\n    plt.title(title)\n    for p in ax.patches:\n        height = p.get_height()\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(100*height/total),\n                ha=\"center\") \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_count(train, 'benign_malignant')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2 checking for `gender` distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_count(train, 'sex')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.3 checking for `anatom_site_general_challenge` distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nplot_count(train, 'anatom_site_general_challenge')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 checking for `diagnosis` distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain['diagnosis'].value_counts(normalize=True).sort_values().iplot(kind='barh',\n                                                      xTitle='Percentage', \n                                                      linecolor='black', \n                                                      opacity=0.7,\n                                                      color='blue',\n                                                      theme='pearl',\n                                                      bargap=0.2,\n                                                      gridcolor='white',\n                                                      title='Distribution in the training set'\n                                                    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Let's check for relative distribution of `sex` and `target`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_relative_distribution(df, feature, hue, title='', size=2):\n    f, ax = plt.subplots(1,1, figsize=(4*size,3*size))\n    total = float(len(df))\n    sns.countplot(x=feature, hue=hue, data=df, palette='Set2')\n    plt.title(title)\n    for p in ax.patches:\n        height = p.get_height()\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(100*height/total),\n                ha=\"center\") \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_relative_distribution(\n    df=train,\n    feature='sex',\n    hue='benign_malignant',\n    title = 'relative count plot of sex with benign_malignant',\n    size=2.8\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Let's check for relative distribution of `anatom_site_general_challenge` and `benign_malignant`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_relative_distribution(\n    df=train,\n    feature='anatom_site_general_challenge',\n    hue='benign_malignant',\n    title = 'relative count plot of anatom_site_general_challenge with benign_malignant',\n    size=3\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.5 Age distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train['age_approx'].iplot(\n    kind='hist',\n    bins=30,\n    color='blue',\n    xTitle='Age',\n    yTitle='Count',\n    title='Age Distribution'\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Randomly displaying few images\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\nfrom PIL import Image, ImageDraw\n\n\ndef display_images(images, title=None): \n    f, ax = plt.subplots(5,3, figsize=(18,22))\n    if title:\n        f.suptitle(title, fontsize = 30)\n\n    for i, image_id in enumerate(images):\n        image_path = os.path.join(BASE_PATH, f'jpeg/train/{image_id}.jpg')\n        image = Image.open(image_path)\n        \n        ax[i//3, i%3].imshow(image) \n        image.close()       \n        ax[i//3, i%3].axis('off')\n\n        benign_malignant = train[train['image_name'] == image_id]['benign_malignant'].values[0]\n        ax[i//3, i%3].set_title(f\"image_name: {image_id}\\nSource: {benign_malignant}\", fontsize=\"15\")\n\n    plt.show() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.1 Displaying benign images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"benign = train[train.benign_malignant == 'benign'].sample(n=15, random_state=42)\ndisplay_images(benign.image_name.values, title = 'benign images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Displaying malignant images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"malignant = train[train.benign_malignant == 'malignant'].sample(n=15, random_state=42)\ndisplay_images(malignant.image_name.values, title='malignant images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Displaying benign images for sex == 'female'","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"female_patients = train[train.sex == 'female']\nbenign = female_patients[female_patients.benign_malignant == 'benign'].sample(n=15, random_state=42)\ndisplay_images(benign.image_name.values, title='benign images for female patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.4 Displaying malignant images for sex == 'female'","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"malignant = female_patients[female_patients.benign_malignant == 'malignant'].sample(n=15, random_state=42)\ndisplay_images(malignant.image_name.values, title='malignant images for female patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.5 Displaying benign images for sex == 'male'","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"male_patients = train[train.sex == 'male']\nbenign = male_patients[male_patients.benign_malignant == 'benign'].sample(n=15, random_state=42)\ndisplay_images(benign.image_name.values, title='benign images for male patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.6 Displaying malignant images for sex == 'male'","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"malignant = male_patients[male_patients.benign_malignant == 'malignant'].sample(n=15, random_state=42)\ndisplay_images(malignant.image_name.values, title='malignant images for male patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.7 Displaying images based on `anatom_site_general_challenge`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"anatom_sites = [ site for site in list(train.anatom_site_general_challenge.unique()) if type(site) != float ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for site in anatom_sites[:2]:\n    site_df = train[train.anatom_site_general_challenge == site].sample(n=15, random_state=42)\n    display_images(site_df.image_name.values, title = f'patient images for anatom_site == {site}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Simple SEResnext Baseline","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade pip --quiet\n!pip install wtfml --quiet\n!pip install pretrainedmodels --quiet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport torch\nimport albumentations\n\nimport torch.nn as nn\nfrom sklearn import metrics\nfrom sklearn import model_selection\nfrom torch.nn import functional as F\n\nfrom wtfml.utils import EarlyStopping\nfrom wtfml.engine import Engine\nfrom wtfml.data_loaders.image import ClassificationLoader\n\nimport pretrainedmodels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_PATH = \"../input/siim-isic-melanoma-classification\"\nDATA_PATH = \"../input/siic-isic-224x224-images/test/\"\nMODEL_PATH = \"../input/melanoma-resnext50\"\n\ndf_test = pd.read_csv(f\"{BASE_PATH}/test.csv\")\n\ndevice = torch.device(\"cuda\")\nmean = (0.485, 0.456, 0.406)\nstd = (0.229, 0.224, 0.225)\naug = albumentations.Compose([\n    albumentations.Normalize(mean, std, max_pixel_value=255.0, always_apply=True)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SEResnext50_32x4d(nn.Module):\n    def __init__(self, pretrained='imagenet'):\n        super(SEResnext50_32x4d, self).__init__()\n        \n        self.model = pretrainedmodels.__dict__[\"se_resnext50_32x4d\"](pretrained=None)\n        self.out = nn.Linear(2048, 1)\n    \n    def forward(self, image, targets):\n        bs, _, _, _ = image.shape\n        \n        x = self.model.features(image)\n        x = F.adaptive_avg_pool2d(x, 1)\n        x = x.reshape(bs, -1)\n        \n        out = self.out(x)\n        loss = nn.BCEWithLogitsLoss()(out, targets.reshape(-1, 1).type_as(out))\n\n        return out, loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ENSEMBLES = [\n    {'model': SEResnext50_32x4d(pretrained=None), 'state_dict': f'{MODEL_PATH}/model_0.bin', 'weight': 1},\n    {'model': SEResnext50_32x4d(pretrained=None), 'state_dict': f'{MODEL_PATH}/model_1.bin', 'weight': 1},\n    {'model': SEResnext50_32x4d(pretrained=None), 'state_dict': f'{MODEL_PATH}/model_2.bin', 'weight': 1},\n    {'model': SEResnext50_32x4d(pretrained=None), 'state_dict': f'{MODEL_PATH}/model_3.bin', 'weight': 1},\n    {'model': SEResnext50_32x4d(pretrained=None), 'state_dict': f'{MODEL_PATH}/model_4.bin', 'weight': 1},\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"models = []\n\nfor ensemble in ENSEMBLES:\n    model = ensemble['model']\n    model_path = ensemble['state_dict']\n    model.load_state_dict(torch.load(model_path))\n    model.to(device)\n    models.append(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = df_test.image_name.values.tolist()\nimages = [os.path.join(DATA_PATH, i + \".png\") for i in images]\ntargets = np.zeros(len(images))\n\ntest_dataset = ClassificationLoader(\n    image_paths=images,\n    targets=targets,\n    resize=None,\n    augmentations=aug,\n)\n\ntest_loader = torch.utils.data.DataLoader(\n    test_dataset, batch_size=48, shuffle=False, num_workers=4\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = []\n\nfor model in models:\n    prediction = Engine.predict(test_loader, model, device=device)\n    prediction = np.vstack((prediction)).ravel()\n    predictions.append(prediction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = np.mean(predictions, axis=0)\nsample = pd.read_csv(f\"{BASE_PATH}/sample_submission.csv\")\nsample.loc[:, \"target\"] = predictions\nsample.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## References\n\n* https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154519\n* https://youtu.be/WaCFd-vL4HA","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### <p><span style=\"color:green\">This Kernel is work in progress, will update soon :) </br></span></p>\n\n### <p><span style=\"color:red\">Ending note: <br>Please upvote this kernel if you like it . It motivates me to produce more quality content :)</br></span></p>","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}