{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Analysis of SIIM-ISIC Melanoma Classification Metadata and Images\n\n# Introduction\n\n## The Competition\n\nSkin cancer is common cancer type and despite beign mostly non malignant, due to high case numbers it's pretty serious diasease and can lead serious cases if not detected, treated in time. It's usually diagnosed by eye for primarily and followed by further clinical analysis if needed. Even though the rares outcome is called melanoma it's the most deadly one, so early detection is pretty important. For this task using computer aided diagnosis might be helpful for primarily steps and early detections. Better detection might save thousands of lives.\n\nThis competition might help reaching that goal and I hope it can help people around the world...\n\n## Updates:\n\n### 23/07/2020:\n- Added adversarial validation,\n- Updated metadata by removing biased features,\n- Created simplier machine learning model.\n\n### 25/07/2020:\n- Added deep learning part\n- Included EfficientNet modelling\n- Ensembled metadata and EffNet predictions\n\n### 01/01/2020:\n- Added external [notebook with past years tabular data here](https://www.kaggle.com/datafan07/eda-modelling-of-the-external-data-inc-ensemble)\n- Small fixes\n\n\n## About the Notebook\n\nFirst of all this is **pretty early version of this notebook**, I decided to start part by part before I fully commit my submission, so for now this notebook covers such as:\n\n- EDA of the metadata,\n- Extracting basic image attributes like image size, colors etc.\n- Creating new features from existing data,\n- Design a machine learning model by using these simple features\n- Make predictions using our model and tabular data\n- Deep learning part will be added in future...\n\nI think using metadata for understanding the problem is really important and plus side is we can use it to improve our scores, for now we only going to use tabular data for submissions. This way we can see it's power and it can help us with future CNN modelling. This notebook going to try answer questions like these:\n\n- How's the data looking?\n- Do we have complete dataset?\n- How's the target distribution looking? Is it balanced?\n- What are the effects of scan site on outcome?\n- Does age effects skin lesion type?\n- Is there difference between female and male patients in terms of target?\n- How many unique patient data we have and how many scans they had? Is it important?\n- Is image quality, colors, size have meaningful impact on the outcome?\n- Can we see similar observations when we analyse both train and test dataset, if not why?\n- And much more...\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# First Impressions and Getting Tools Ready\n\nLet's buckle up and get our tools ready for our work! We start with importing neccesary libraries. Since we going to do mostly EDA our libraries are going to be related with tabular data and visualization.","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# loading packages\n\nimport pandas as pd\nimport numpy as np\n\n#\n\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\n\n#\n\nimport seaborn as sns\nimport plotly.express as px\n\n#\n\nimport os\nimport random\nimport re\nimport math\nimport time\n\nfrom tqdm import tqdm\nfrom tqdm.keras import TqdmCallback\n\n\nfrom pandas_summary import DataFrameSummary\n\nimport warnings\n\n\nwarnings.filterwarnings('ignore') # Disabling warnings for clearer outputs\n\n\n\nseed_val = 42\nrandom.seed(seed_val)\nnp.random.seed(seed_val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We set some custom styling with our notebook for aesthetics...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting color palette.\norange_black = [\n    '#fdc029', '#df861d', '#FF6347', '#aa3d01', '#a30e15', '#800000', '#171820'\n]\n\n# Setting plot styling.\nplt.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting file paths for our notebook:\n\nbase_path = '/kaggle/input/siim-isic-melanoma-classification'\ntrain_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train/'\ntest_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/test/'\nimg_stats_path = '/kaggle/input/melanoma2020imgtabular'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading the Data\n\nWe'll continue by loading metadata we're given. Train data has 8 features, 33126 observations and Test data 5 features, 10982 observations.\n\n#### Train Dataset Consists Of:\n\n1. image name -> the filename of specific image for the train set\n2. patient_id -> identifies the unique patient\n3. sex -> gender of the patient\n4. age_approx -> approx age of the patient at time of scanning\n5. anatom_site_general_challenge -> location of the scan site\n6. diagnosis -> information about the diagnosis\n7. benign_malignant - indicates scan result if it's malignant or benign\n8. target -> same as above but better for modelling since it's binary\n\nAnd the next dataset we going to inspect test. It has same features as train set except for scan results, well that's why it's test set right?!\n\n#### Test Dataset Consists Of:\n\n1. image name -> the filename of specific image for the train set\n2. patient_id -> identifies the unique patient\n3. sex -> gender of the patient\n4. age_approx -> approx age of the patient at time of scanning\n5. anatom_site_general_challenge -> location of the scan site","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading train and test data.\n\ntrain = pd.read_csv(os.path.join(base_path, 'train.csv'))\ntest = pd.read_csv(os.path.join(base_path, 'test.csv'))\nsample = pd.read_csv(os.path.join(base_path, 'sample_submission.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking train and test columns/rows.\n\nprint(\n    f'Train data has {train.shape[1]} features, {train.shape[0]} observations and Test data {test.shape[1]} features, {test.shape[0]} observations.\\nTrain features are:\\n{train.columns.tolist()}\\nTest features are:\\n{test.columns.tolist()}'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Renaming train/test columns:\n\ntrain.columns = [\n    'img_name', 'id', 'sex', 'age', 'location', 'diagnosis',\n    'benign_malignant', 'target'\n]\ntest.columns = ['img_name', 'id', 'sex', 'age', 'location']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Taking 5 random samples from the train data:\n\ntrain.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Taking 5 random samples from the test data:\n\ntest.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Missing Values\n\nWe have small portion of missing values for age and sex I think there is no harm if we impute them with the most frequent ones, meanwhile body parts missing on both datasets, we better be set 'unknown' for missing values for this one...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Checking missing values:\n\ndef missing_percentage(df):\n\n    total = df.isnull().sum().sort_values(\n        ascending=False)[df.isnull().sum().sort_values(ascending=False) != 0]\n    percent = (df.isnull().sum().sort_values(ascending=False) / len(df) *\n               100)[(df.isnull().sum().sort_values(ascending=False) / len(df) *\n                     100) != 0]\n    return pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n\n\nmissing_train = missing_percentage(train)\nmissing_test = missing_percentage(test)\n\nfig, ax = plt.subplots(1, 2, figsize=(16, 6))\n\nsns.barplot(x=missing_train.index,\n            y='Percent',\n            data=missing_train,\n            palette=orange_black,\n            ax=ax[0])\n\nsns.barplot(x=missing_test.index,\n            y='Percent',\n            data=missing_test,\n            palette=orange_black,\n            ax=ax[1])\n\nax[0].set_title('Train Data Missing Values')\nax[1].set_title('Test Data Missing Values')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Checking Variables Before Imputing\n\nJust wanted to check variable distribution before we impute the missing ones. Looks like our assumptions were ok, we can continue with imputing...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 9))\n\n# Creating a grid:\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Gender Distribution')\n\nsns.countplot(train.sex.sort_values(ignore_index=True),\n              alpha=0.9,\n              ax=ax1,\n              color='#fdc029',\n              label='Train')\nsns.countplot(test.sex.sort_values(ignore_index=True),\n              alpha=0.7,\n              ax=ax1,\n              color='#171820',\n              label='Test')\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Plot the countplot.\n\nsns.countplot(train.location,\n              alpha=0.9,\n              ax=ax2,\n              color='#fdc029',\n              label='Train',\n              order=train['location'].value_counts().index)\nsns.countplot(test.location,\n              alpha=0.7,\n              ax=ax2,\n              color='#171820',\n              label='Test',\n              order=test['location'].value_counts().index), ax2.set_title(\n                  'Anatom Site Distribution')\n\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Age Distribution')\n\n# Plot the histogram.\n\nsns.distplot(train.age, ax=ax3, label='Train', color='#fdc029')\nsns.distplot(test.age, ax=ax3, label='Test', color='#171820')\n\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imputing Missing Data\n\nLet's fill the missing values with appropriate methods.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filling missing anatom site values with 'unknown' tag:\n\nfor df in [train, test]:\n    df['location'].fillna('unknown', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Double checking:\n\nids_train = train.location.values\nids_test = test.location.values\nids_train_set = set(ids_train)\nids_test_set = set(ids_test)\n\nlocation_not_overlap = list(ids_train_set.symmetric_difference(ids_test_set))\nn_overlap = len(location_not_overlap)\nif n_overlap == 0:\n    print(\n        f'There are no different body parts occuring between train and test set...'\n    )\nelse:\n    print('There are some not overlapping values between train and test set!')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filling age and sex with appropriate values.\n\ntrain['sex'].fillna(train['sex'].mode()[0], inplace=True)\n\ntrain['age'].fillna(train['age'].median(), inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking missing value counts:\n\nprint(\n    f'Train missing value count: {train.isnull().sum().sum()}\\nTest missing value count: {train.isnull().sum().sum()}'\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploring the Data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Scans by Anatom Site\n\nGood... It looks like both datasets shared scanned body parts similary. Let's check it further.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train data:\n\ncntstr = train.location.value_counts().rename_axis('location').reset_index(\n    name='count')\n\nfig = px.treemap(cntstr,\n                 path=['location'],\n                 values='count',\n                 color='count',\n                 color_continuous_scale=orange_black,\n                 title='Scans by Anatom Site General Challenge - Train Data')\n\nfig.update_traces(textinfo='label+percent entry')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Test data:\n\ncntste = test.location.value_counts().rename_axis('location').reset_index(\n    name='count')\n\nfig = px.treemap(cntste,\n                 path=['location'],\n                 values='count',\n                 color='count',\n                 color_continuous_scale=orange_black,\n                 title='Scans by Anatom Site General Challenge - Test Data')\n\nfig.update_traces(textinfo='label+percent entry')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Body Part Ratio by Gender and Target\n\nLooks like some body parts are more likely to be malignant, head/neck comes first with followed by oral/genital and upper extremity. Scanned body part locations are similar in order between males and females with small differences on distribution.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 9))\n# Creating a grid\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[1, :2])\n# Set the title.\nax1.set_title('Scanned Body Parts - Female')\n\n# Plot:\n\nsns.countplot(\n    train[train['sex'] == 'female'].location.sort_values(ignore_index=True),\n    alpha=0.9,\n    ax=ax1,\n    color='#fdc029',\n    label='Female',\n    order=train['location'].value_counts().index)\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[1, 2:])\n\n# Set the title.\n\nax2.set_title('Scanned Body Parts - Male')\n\n# Plot.\n\nsns.countplot(\n    train[train['sex'] == 'male'].location.sort_values(ignore_index=True),\n    alpha=0.9,\n    ax=ax2,\n    color='#171820',\n    label='Male',\n    order=train['location'].value_counts().index)\n\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[0, :])\n\n# Set the title.\n\nax3.set_title('Malignant Ratio Per Body Part')\n\n# Plot.\n\nloc_freq = train.groupby('location')['target'].mean().sort_values(\n    ascending=False)\nsns.barplot(x=loc_freq.index, y=loc_freq, palette=orange_black, ax=ax3)\n\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# A General Look With Sunburst Chart\n\nSunburst chart is pretty cool looking fella I'd say. It also giving lots of basic information to us. Let's see...\n\n- Only 2% of our targets are malignant\n- On malignant images males are dominant with 62% \n- Gender wise benign images are more balance 52-48% male female ratio\n- Malignant image scan locations differs based on the patients gender:\n    - Meanwhile the torso is most common location in males it's almost half of the scans meanwhile in females it's 39%\n    - Lower extremity is more common with female scans than males 18% males vs 26% females\n    - Again upper extremity malignant scans is common with females than males (23- 17%)\n- Benign image scan locations more similar between male and female patients.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting interactive sunburst:\n\nfig = px.sunburst(data_frame=train,\n                  path=['benign_malignant', 'sex', 'location'],\n                  color='sex',\n                  color_discrete_sequence=orange_black,\n                  maxdepth=-1,\n                  title='Sunburst Chart Benign/Malignant > Sex > Location')\n\nfig.update_traces(textinfo='label+percent parent')\nfig.update_layout(margin=dict(t=0, l=0, r=0, b=0))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Age and Scan Result Relations\n\nAge looks pretty decent factor on scan result. Getting malignant scan result with elderly age is more possible than young patients. There is spike for both genders after age of 85, if we look distribution of ages there isn't much of 80+ patients and it can be the reason of this spike but we can safely say it's more likely to be malignant scan after age of 60. We see small bump on age 15-20 for females, again it depends on the scan numbers but still, poor souls...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting age vs sex vs target:\n\nfig, ax = plt.subplots(1, 2, figsize=(16, 6))\nsns.lineplot(x='age',\n             y='target',\n             data=train,\n             ax=ax[0],\n             hue='sex',\n             palette=orange_black[:2],\n             ci=None)\nsns.boxplot(x='benign_malignant',\n            y='age',\n            data=train,\n            ax=ax[1],\n            hue='sex',\n            palette=orange_black)\n\nplt.legend(loc='lower right')\n\nax[0].set_title('Malignant Scan Frequency by Age')\nax[1].set_title('Scan Results by Age and Sex')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Age Round Two\n\nWanted to double check age distributions after our previous observations. Age seems evenly distributed on both train and test datasets, we can see small bumps at age 75+ and around 40, these worth investigating.\n\nWe can see again older people are more likely to get malignant scan results. One last thing about age distributions, we see more female patients in younger ages this trend changes with the older patients...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\n# Plotting age dist vs target and age dist vs datasets\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Age Distribution by Scan Outcome')\n\n# Plot\n\nax1.legend()\n\nsns.kdeplot(train[train['target'] == 0]['age'],\n            shade=True,\n            ax=ax1,\n            color='#171820',\n            label='Benign')\nsns.kdeplot(train[train['target'] == 1]['age'],\n            shade=True,\n            ax=ax1,\n            color='#fdc029',\n            label='Malignant')\n\n# Customizing second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Age Distribution by Train/Test Observations')\n\n# Plot.\n\nsns.kdeplot(train.age, label='Train', shade=True, ax=ax2, color='#171820')\nsns.kdeplot(test.age, label='Test', shade=True, ax=ax2, color='#fdc029')\n\nax2.legend()\n\n# Customizing third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Age Distribution by Gender')\n\n# Plot\n\nsns.distplot(train[train.sex == 'female'].age,\n             ax=ax3,\n             label='Female',\n             color='#fdc029')\nsns.distplot(train[train.sex == 'male'].age,\n             ax=ax3,\n             label='Male',\n             color='#171820')\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Unique Patients and Their Scan Images\n\nIt looks like we have multiple scan images per patient, actual unique patient counts are much lower than images on both datasets. We can get more information about patients age like when he had his first scan and his last scan. We can get interesting insights like:\n\n- Most of the malignant results are found around first 20 scans. Of course there can be control scans after the diagnosis...\n- Scan numbers are similar in first 100 scans but we have 200+ scan images for **one particular patient** in dataset, it's pretty interesting since we don't have this case in our training data. We should be careful about this and it can effect our model.\n- Most of the malignant cases are under 20 images but in general we can say it's more likely to be malignant result if there are more scan images...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\n    f'Number of unique Patient ID\\'s in train set: {train.id.nunique()}, Total: {train.id.count()}\\nNumber of unique Patient ID\\'s in test set: {test.id.nunique()}, Total: {test.id.count()}'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['age_min'] = train['id'].map(train.groupby(['id']).age.min())\ntrain['age_max'] = train['id'].map(train.groupby(['id']).age.max())\n\ntest['age_min'] = test['id'].map(test.groupby(['id']).age.min())\ntest['age_max'] = test['id'].map(test.groupby(['id']).age.max())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['n_images'] = train.id.map(train.groupby(['id']).img_name.count())\ntest['n_images'] = test.id.map(test.groupby(['id']).img_name.count())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Number of Scans Distribution by Scan Outcome')\n\n# Plot\n\nsns.kdeplot(train[train['target'] == 0]['n_images'],\n            shade=True,\n            ax=ax1,\n            color='#171820',\n            label='Benign')\nsns.kdeplot(train[train['target'] == 1]['n_images'],\n            shade=True,\n            ax=ax1,\n            color='#fdc029',\n            label='Malignant')\n\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Number of Scans Distribution by Train/Test Observations')\n\n# Plot\n\nsns.kdeplot(train.n_images, label='Train', shade=True, ax=ax2, color='#171820')\nsns.kdeplot(test.n_images, label='Test', shade=True, ax=ax2, color='#fdc029')\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Malignant Scan Result Frequency by Number of Scans')\n\n# Plot\n\nz = train.groupby('n_images')['target'].mean()\nsns.lineplot(x=z.index, y=z, color='#171820', ax=ax3)\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Diagnosis Distribution\n\nThis part we can't use in our model but it's giving us some insights about this disease so we can inspect that too. You can see the details below:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"diag = train.diagnosis.value_counts()\nfig = px.pie(diag,\n             values='diagnosis',\n             names=diag.index,\n             color_discrete_sequence=orange_black,\n             hole=.4)\nfig.update_traces(textinfo='percent+label', pull=0.05)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading Image Meta Features\n\nThis is the part where we get basic info directly from images themselves.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting image sizes by using os:\n\nfor data, location in zip([train, test], [train_img_path, test_img_path]):\n    images = data['img_name'].values\n    sizes = np.zeros(images.shape[0])\n    for i, path in enumerate(tqdm(images)):\n        sizes[i] = os.path.getsize(os.path.join(location, f'{path}.jpg'))\n\n    data['image_size'] = sizes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Sizes\n\nWe can see some kind of relation between size and target, but is it meaningful? Too soon to say...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting image sizes:\n\nfig, ax = plt.subplots(1, 2, figsize=(16, 6))\n\nsns.kdeplot(train[train['target'] == 0]['image_size'],\n            shade=True,\n            ax=ax[0],\n            color='#171820',\n            label='Benign')\nsns.kdeplot(train[train['target'] == 1]['image_size'],\n            shade=True,\n            ax=ax[0],\n            color='#fdc029',\n            label='Malignant')\n\nsns.kdeplot(train.image_size,\n            label='Train',\n            shade=True,\n            ax=ax[1],\n            color='#171820')\nsns.kdeplot(test.image_size,\n            label='Test',\n            shade=True,\n            ax=ax[1],\n            color='#fdc029')\n\nax[0].set_title('Scan Image Size Distribution by Scan Outcome')\nax[1].set_title('Scan Image Size Distribution by Train/Test Observations')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Getting Image Attributes\n\nYou can get these attributes by using the code below, I commented it out here and imported it as a data becasue it's time consuming process.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#from keras.preprocessing import image\n#\n# for data, location in zip([train, test],[train_img_path, test_img_path]):\n#    images = data['img_name'].values\n#    reds = np.zeros(images.shape[0])\n#    greens = np.zeros(images.shape[0])\n#    blues = np.zeros(images.shape[0])\n#    mean = np.zeros(images.shape[0])\n#    x = np.zeros(images.shape[0], dtype=int)\n#    y = np.zeros(images.shape[0], dtype=int)\n#    for i, path in enumerate(tqdm(images)):\n#        img = np.array(image.load_img(os.path.join(location, f'{path}.jpg')))\n#\n#        reds[i] = np.mean(img[:,:,0].ravel())\n#        greens[i] = np.mean(img[:,:,1].ravel())\n#        blues[i] = np.mean(img[:,:,2].ravel())\n#        mean[i] = np.mean(img)\n#        x[i] = img.shape[1]\n#        y[i] = img.shape[0]\n#\n#    data['reds'] = reds\n#    data['greens'] = greens\n#    data['blues'] = blues\n#    data['mean_colors'] = mean\n#    data['width'] = x\n#    data['height'] = y\n#\n#train['total_pixels']= train['width']*train['height']\n#test['total_pixels']= test['width']*test['height']\n#train['res'] = train['width'].astype(str) + 'x' + train['height'].astype(str)\n#test['res'] = test['width'].astype(str) + 'x' + test['height'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading color data:\n\ntrain_attr = pd.read_csv(\n    os.path.join(img_stats_path, 'train_mean_colorres.csv'))\ntest_attr = pd.read_csv(os.path.join(img_stats_path, 'test_mean_colorres.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_attr.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.concat([train, train_attr], axis=1)\ntest = pd.concat([test, test_attr], axis=1)\n\ntrain['res'] = train['width'].astype(str) + 'x' + train['height'].astype(str)\ntest['res'] = test['width'].astype(str) + 'x' + test['height'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Colors and Their Effects on Results","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=3, nrows=3, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('RGB Channels of Benign Images')\n\n# Plot.\n\nsns.distplot(train[train['target'] == 0].reds,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='red',\n             kde=True,\n             ax=ax1,\n             label='Reds')\nsns.distplot(train[train['target'] == 0].greens,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='green',\n             kde=True,\n             ax=ax1,\n             label='Greens')\nsns.distplot(train[train['target'] == 0].blues,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='blue',\n             kde=True,\n             ax=ax1,\n             label='Blues')\n\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[1, :2])\n\n# Set the title.\n\nax2.set_title('RGB Channels of Malignant Images')\n\n# Plot\n\nsns.distplot(train[train['target'] == 1].reds,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='red',\n             kde=True,\n             ax=ax2,\n             label='Reds')\nsns.distplot(train[train['target'] == 1].greens,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='green',\n             kde=True,\n             ax=ax2,\n             label='Greens')\nsns.distplot(train[train['target'] == 1].blues,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='blue',\n             kde=True,\n             ax=ax2,\n             label='Blues')\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[:, 2])\n\n# Set the title.\n\nax3.set_title('Mean Colors by Train/Test Images')\n\n# Plot\n\nsns.kdeplot(train.mean_colors,\n            shade=True,\n            label='Train',\n            ax=ax3,\n            color='#171820',\n            vertical=True)\nsns.kdeplot(test.mean_colors,\n            shade=True,\n            label='Test',\n            ax=ax3,\n            color='#fdc029',\n            vertical=True)\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How are the Image Sizes Affecting Targets in Our Data\n\nWe have important observation here, you can see whole 1920x1080 set in test data which is not present in train data. That can have huge impact on final scores, mind that in your models. You might want to leave out image size related info in your models or regularize your models to smooth that effect. It can cause overfitting because of high correlation between image sizes and target, but these correlation might not be the case in test set (most likely) so keep that in mind.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=3, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Scan Image Resolutions of Train Set')\n\n# Plot.\n\ntres = train.res.value_counts().rename_axis('res').reset_index(name='count')\ntres = tres[tres['count'] > 10]\nsns.barplot(x='res', y='count', data=tres, palette=orange_black, ax=ax1)\nplt.xticks(rotation=20)\n\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Scan Image Resolutions of Test Set')\n\n# Plot\n\nteres = test.res.value_counts().rename_axis('res').reset_index(name='count')\nteres = teres[teres['count'] > 10]\nsns.barplot(x='res', y='count', data=teres, palette=orange_black, ax=ax2)\nplt.xticks(rotation=20)\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Scan Image Resolutions by Target')\n\n# Plot.\n\nsns.countplot(x='res',\n              hue='benign_malignant',\n              data=train,\n              order=train.res.value_counts().iloc[:12].index,\n              palette=orange_black,\n              ax=ax3)\nax3.legend()\n\n# Customizing the last grid.\n\nax4 = fig.add_subplot(grid[2, :])\n\n# Set the title.\n\nax4.set_title('Malignant Scan Result Frequency by Image Resolution')\n\n# Plot.\n\nres_freq = train.groupby('res')['target'].mean()\nres_freq = res_freq[(res_freq > 0) & (res_freq < 1)]\nsns.lineplot(x=res_freq.index, y=res_freq, palette=orange_black, ax=ax4)\nax4.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The Mysterious Images\n\nThe name 'Mystery' comes from Chris Deotte from the comments down below and I decided to investigate them further. In last part we found out a new set of images with the resolution of 1920x1080 and they aren't present in train data at all. So we can assume these images weren't selected randomly for this competition. Down below I'm going to compare them with the rest of data.\n\n* It looks like without the 1920x1080 set mean colors are much more similar between train and test.\n* Again image size distribution gets closer between train and test without the mystery set.\n\n\nGonna check other features and add them here soon...","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 14))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=3, nrows=3, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('RGB Channels of Train Images With \"Mysterious\" Set')\n\n# Plot.\n\nsns.distplot(train.reds,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='red',\n             kde=True,\n             ax=ax1,\n             label='Reds')\nsns.distplot(train.greens,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='green',\n             kde=True,\n             ax=ax1,\n             label='Greens')\nsns.distplot(train.blues,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='blue',\n             kde=True,\n             ax=ax1,\n             label='Blues')\n\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[1, :2])\n\n# Set the title.\n\nax2.set_title('RGB Channels of Test Images Without \"Mysterious\" Set')\n\n# Plot\n\nsns.distplot(test[test['res'] != '1920x1080'].reds,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='red',\n             kde=True,\n             ax=ax2,\n             label='Reds')\nsns.distplot(test[test['res'] != '1920x1080'].greens,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='green',\n             kde=True,\n             ax=ax2,\n             label='Greens')\nsns.distplot(test[test['res'] != '1920x1080'].blues,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='blue',\n             kde=True,\n             ax=ax2,\n             label='Blues')\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[:, 2])\n\n# Set the title.\n\nax3.set_title('Mean Colors by Train/Test Images Without \"Mysterious\" Set')\n\n# Plot\n\nsns.kdeplot(train.mean_colors,\n            shade=True,\n            label='Train',\n            ax=ax3,\n            color='#171820',\n            vertical=True)\nsns.kdeplot(test[test['res'] != '1920x1080'].mean_colors,\n            shade=True,\n            label='Test',\n            ax=ax3,\n            color='#fdc029',\n            vertical=True)\nax3.legend()\n\n# Customizing the last grid.\n\nax2 = fig.add_subplot(grid[2, :2])\n\n# Set the title.\n\nax2.set_title('RGB Channels of \"Mysterious\" Set')\n\n# Plot\n\nsns.distplot(test[test['res'] == '1920x1080'].reds,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='red',\n             kde=True,\n             ax=ax2,\n             label='Reds')\nsns.distplot(test[test['res'] == '1920x1080'].greens,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='green',\n             kde=True,\n             ax=ax2,\n             label='Greens')\nsns.distplot(test[test['res'] == '1920x1080'].blues,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             },\n             color='blue',\n             kde=True,\n             ax=ax2,\n             label='Blues')\nax2.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\n# Plotting age dist vs target and age dist vs datasets\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Scan Image Size Distribution by Train/Test Observations')\n\n# Plot\n\nax1.legend()\n\nsns.kdeplot(train['image_size'],\n            shade=True,\n            ax=ax1,\n            color='#171820',\n            label='Train')\nsns.kdeplot(test['image_size'],\n            shade=True,\n            ax=ax1,\n            color='#fdc029',\n            label='Test')\n\n# Customizing second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Scan Image Size Distribution Without \"Mysterious Set\"')\n\n# Plot.\n\nsns.kdeplot(train.image_size,\n            label='Train',\n            shade=True,\n            ax=ax2,\n            color='#171820')\nsns.kdeplot(test[test['res'] != '1920x1080'].image_size,\n            label='Test',\n            shade=True,\n            ax=ax2,\n            color='#fdc029')\nax2.legend()\n\n# Customizing third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Image Size Distribution of Mysterious Images')\n\n# Plot\n\nsns.distplot(test[test['res'] == '1920x1080'].image_size,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.9\n             },\n             color='#FF6347',\n             kde=True,\n             ax=ax3,\n             label='Mysterious Images')\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I was curious about if these 1920x1080 images belong to high scan patients including 200+ one but it seems these observations are grouped around 10 scans, so it makes things more interesting...","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\n# Plotting age dist vs target and age dist vs datasets\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Number of Images Distribution by Train/Test Observations')\n\n# Plot\n\nax1.legend()\n\nsns.kdeplot(train['n_images'],\n            shade=True,\n            ax=ax1,\n            color='#171820',\n            label='Train')\nsns.kdeplot(test['n_images'],\n            shade=True,\n            ax=ax1,\n            color='#fdc029',\n            label='Test')\n\n# Customizing second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Scan Image Size Distribution Without \"Mysterious Set\"')\n\n# Plot.\n\nsns.kdeplot(train.n_images,\n            label='Train',\n            shade=True,\n            ax=ax2,\n            color='#171820')\nsns.kdeplot(test[test['res'] != '1920x1080'].n_images,\n            label='Test',\n            shade=True,\n            ax=ax2,\n            color='#fdc029')\nax2.legend()\n\n# Customizing third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Number of Images Distribution of Mysterious Images')\n\n# Plot\n\nsns.distplot(test[test['res'] == '1920x1080'].n_images,\n             hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.9\n             },\n             color='#FF6347',\n             kde=True,\n             ax=ax3,\n             label='Mysterious Images')\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we checking this mystery patch of images, let's check other features about them too maybe we can find other factors effecting this test sampling...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20, 6))\n\nsns.kdeplot(test[test['res'] != '1920x1080'].age,\n            shade=True,\n            label='Without Mystery Set',\n            color='#171820',\n            )\nsns.kdeplot(test[test['res'] == '1920x1080'].age,\n            shade=True,\n            label='With Mystery Set',\n            color='#fdc029',\n            )\n\nplt.legend(loc='upper right')\n\nax.set_title('Age Distribution With/Without Mysterious Set')\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like our 1920x1080 set images consisting little bit younger patients than the rest. Interesting...","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Visual Inspection of Mysterious Image Set\n\nThis is subjective, but when we look at both samples we can see that 1920x1080 images are coming from a 'imaging device' with black circle around the images? In general this isn't the case with the rest of the test image samples... Maybe we can use similar images from previous competitions for predicting this set? I don't know yet but worth to consider I guess...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mystery = test[test['res'] == '1920x1080']\nmystimages = mystery['img_name'].values\n\nnonmystery = test[test['res'] != '1920x1080']\nnonmystimages = nonmystery['img_name'].values\n\nrandom_myst_images = [np.random.choice(mystimages+'.jpg') for i in range(12)]\nrandom_nmyst_images = [np.random.choice(nonmystimages+'.jpg') for i in range(12)]\n\n# Location of test images\nimg_dir = '../input/siim-isic-melanoma-classification/jpeg/test'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nfor i in range(12):\n    \n    plt.subplot(3, 4, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_myst_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \nplt.suptitle('Sample Images From Mysterious Test Set', fontsize=14)\nplt.tight_layout()   \n  ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nfor i in range(12):\n    \n    plt.subplot(3, 4, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_nmyst_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off') \n    \nplt.suptitle('Sample Images From Rest of the Test Set', fontsize=14, y=1.05)\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correlations Between Features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display numerical correlations between features on heatmap.\n\nsns.set(font_scale=1.1)\ncorrelation_train = train[['target','age','age_min',\n 'age_max',\n 'n_images',\n 'image_size',\n 'reds',\n 'greens',\n 'blues', \n 'width',\n 'height',\n ]].corr()\nmask = np.triu(correlation_train.corr())\nplt.figure(figsize=(16, 6))\nsns.heatmap(correlation_train,\n            annot=True,\n            fmt='.1f',\n            cmap='coolwarm',            \n            mask=mask,\n            linewidths=1,\n            cbar=False)\n\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling Based on Tabular Meta Features","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Getting Landscape Attributes from Images\n\nThanks to this great dataset by Marcelo Kittlein [here.](https://www.kaggle.com/kittlein/landscape)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading lanscape data\n\ntrain40 = pd.read_csv('../input/melanoma2020imgtabular/train40Features.csv')\ntest40 = pd.read_csv('../input/melanoma2020imgtabular/test40Features.csv')\n\ntrainmet = pd.read_csv('../input/melanoma2020imgtabular/trainMetrics.csv')\ntestmet = pd.read_csv('../input/melanoma2020imgtabular/testMetrics.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dropping duplicate data from lanscape dataset\n\ntrain40.drop(['sex', 'age_approx', 'anatom_site_general_challenge'],\n             axis=1,\n             inplace=True)\n\ntest40.drop(['sex', 'age_approx', 'anatom_site_general_challenge'],\n            axis=1,\n            inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# merging both datasets\n\ntrain = pd.concat([train, train40, trainmet], axis=1)\ntest = pd.concat([test, test40, testmet], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking out new dataset\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Getting Data Ready For ML Algorithms","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# getting dummy variables for gender on train set\n\nsex_dummies = pd.get_dummies(train['sex'], prefix='sex')\ntrain = pd.concat([train, sex_dummies], axis=1)\n\n# getting dummy variables for gender on test set\n\nsex_dummies = pd.get_dummies(test['sex'], prefix='sex')\ntest = pd.concat([test, sex_dummies], axis=1)\n\n# dropping not useful columns\n\ntrain.drop(['sex','res','img_name','id','diagnosis','benign_malignant'], axis=1, inplace=True)\ntest.drop(['sex','res','img_name','id'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# getting dummy variables for location on train set\n\nanatom_dummies = pd.get_dummies(train['location'], prefix='anatom')\ntrain = pd.concat([train, anatom_dummies], axis=1)\n\n# getting dummy variables for location on test set\n\nanatom_dummies = pd.get_dummies(test['location'], prefix='anatom')\ntest = pd.concat([test, anatom_dummies], axis=1)\n\n# dropping not useful columns\n\ntrain.drop('location', axis=1, inplace=True)\ntest.drop('location', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading Modelling Tools","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading modelling libraries\n\nimport xgboost as xgb\n\nfrom sklearn.model_selection import StratifiedKFold, train_test_split, cross_val_score, cross_validate\nfrom sklearn.metrics import roc_auc_score, roc_curve","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dividing train set and labels for modelling\n\nX = train.drop('target', axis=1)\ny = train.target","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setting Cross-Validation and Hold-out Set\n\nCross validation might be enough but I wanted to test our model on data which it never seen before.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# taking holdout set for validating with stratified y\n\nX_train, X_test, y_train, y_test = train_test_split(X,\n                                                    y,\n                                                    test_size=0.2,\n                                                    stratify=y,\n                                                    random_state=42)\n\n# 5 fold stratify for cv\n\ncv = StratifiedKFold(5, shuffle=True, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# setting model hyperparameters, didn't include fine tuning here because of timing reasons...\n\nxg = xgb.XGBClassifier(\n    n_estimators=750,\n    min_child_weight=0.81,\n    learning_rate=0.025,\n    max_depth=2,\n    subsample=0.80,\n    colsample_bytree=0.42,\n    gamma=0.10,\n    random_state=42,\n    n_jobs=-1,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"estimators = [xg]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cross validation scheme\n\ndef model_check(X_train, y_train, estimators, cv):\n    model_table = pd.DataFrame()\n\n    row_index = 0\n    for est in estimators:\n\n        MLA_name = est.__class__.__name__\n        model_table.loc[row_index, 'Model Name'] = MLA_name\n\n        cv_results = cross_validate(est,\n                                    X_train,\n                                    y_train,\n                                    cv=cv,\n                                    scoring='roc_auc',\n                                    return_train_score=True,\n                                    n_jobs=-1)\n\n        model_table.loc[row_index,\n                        'Train roc Mean'] = cv_results['train_score'].mean()\n        model_table.loc[row_index,\n                        'Test roc Mean'] = cv_results['test_score'].mean()\n        model_table.loc[row_index, 'Test Std'] = cv_results['test_score'].std()\n        model_table.loc[row_index, 'Time'] = cv_results['fit_time'].mean()\n\n        row_index += 1\n\n    model_table.sort_values(by=['Test roc Mean'],\n                            ascending=False,\n                            inplace=True)\n\n    return model_table","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Results Based on Meta Features\n\nResults are encouraging! It seems little bit overfitting but we might fix that in future by fine tuning. Let's leave it like that for now...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# display cv results\n\nraw_models = model_check(X_train, y_train, estimators, cv)\ndisplay(raw_models)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fitting train data\n\nxg.fit(X_train, y_train)\n\n# predicting on holdout set\nvalidation = xg.predict_proba(X_test)[:, 1]\n\n# checking results on validation set\nroc_auc_score(y_test, validation)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Meta Feature Importances\n\nImage size seems pretty important on our model, but don't forget this can be misleading for final scoring. Don't forget about missing image sizes in test set and size correlation with targets in train data!","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# finding feature importances and creating new dataframe basen on them\n\nfeature_importance = xg.get_booster().get_score(importance_type='weight')\n\nkeys = list(feature_importance.keys())\nvalues = list(feature_importance.values())\n\nimportance = pd.DataFrame(data=values, index=keys,\n                          columns=['score']).sort_values(by='score',\n                                                         ascending=False)\nfig, ax = plt.subplots(figsize=(16, 10))\nsns.barplot(x=importance.score.iloc[:20],\n            y=importance.index[:20],\n            orient='h',\n            palette='Reds_r')\nax.set_title('Feature Importances')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# First Step: Creating Meta Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# predicting on test set\n\npredictions = xg.predict_proba(test)[:, 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating submission df\n\n\nmeta_df = pd.DataFrame(columns=['image_name', 'target'])\n\n# assigning predictions on submission df\n\nmeta_df['image_name'] = sample['image_name']\nmeta_df['target'] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating submission csv file\n\nmeta_df.to_csv('meta_with_img_data.csv', header=True, index=False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The .csv file above scores ~85% on public LB but our inspections on train/test data shows it might be overfitting due to image differences, in next step we going try spot these features and make it little more robust...","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Adversarial Validation\n\nAlright, since we have high doubts for train test sampling wanted to implement what is called 'Adversarial Validation'. For this we going to replace our targets for both datasets (0 for train and 1 for test), then we going build a classifier which tries to predict which observation belongs to train and which one belongs to test set. If datasets randomly selected from similar roots it should be really hard for the classifier to separate them. But if there is systematic selection differences between train and test sets then classifier should be able to capture this trend. So we want our models score lower for the next section because higher detection rate means higher difference between train and test datasets, so let's get started...\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"adv_train = train.copy()\nadv_train.drop('target', axis=1, inplace=True)\nadv_test = test.copy()\n\nadv_train['dataset_label'] = 0\nadv_test['dataset_label'] = 1\n\nadv_master = pd.concat([adv_train, adv_test], axis=0)\n\nadv_X = adv_master.drop('dataset_label', axis=1)\nadv_y = adv_master['dataset_label']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"adv_X_train, adv_X_test, adv_y_train, adv_y_test = train_test_split(adv_X,\n                                                    adv_y,\n                                                    test_size=0.4,\n                                                    stratify=adv_y,\n                                                    random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xg_adv = xgb.XGBClassifier(\n    random_state=42,\n    n_jobs=-1,\n)\n\n# Fitting train data\n\nxg_adv.fit(adv_X_train, adv_y_train)\n\n# Predicting on holdout set\nvalidation = xg_adv.predict_proba(adv_X_test)[:,1]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_roc_feat(y_trues, y_preds, labels, est, x_max=1.0):\n    fig, ax = plt.subplots(1,2, figsize=(16,6))\n    for i, y_pred in enumerate(y_preds):\n        y_true = y_trues[i]\n        fpr, tpr, thresholds = roc_curve(y_true, y_pred)\n        auc = roc_auc_score(y_true, y_pred)\n        ax[0].plot(fpr, tpr, label='%s; AUC=%.3f' % (labels[i], auc), marker='o', markersize=1)\n\n    ax[0].legend()\n    ax[0].grid()\n    ax[0].plot(np.linspace(0, 1, 20), np.linspace(0, 1, 20), linestyle='--')\n    ax[0].set_title('ROC curve')\n    ax[0].set_xlabel('False Positive Rate')\n    ax[0].set_xlim([-0.01, x_max])\n    _ = ax[0].set_ylabel('True Positive Rate')\n    \n    \n    feature_importance = est.get_booster().get_score(importance_type='weight')\n\n    keys = list(feature_importance.keys())\n    values = list(feature_importance.values())\n\n    importance = pd.DataFrame(data=values, index=keys,\n                          columns=['score']).sort_values(by='score',\n                                                         ascending=False)\n    \n    sns.barplot(x=importance.score.iloc[:20],\n            y=importance.index[:20],\n            orient='h',\n            palette='Reds_r', ax=ax[1])\n    ax[1].set_title('Feature Importances')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## First Results\n\nWell... It seems our model can seperate train and test set pretty good. This is not good, when we look at our ROC Curve it almost scores 90%, whe should dig this further... So to find the reason behind it we check our model's feature importances and we see image related features are really affecting it. So this confirms our findings in EDA part that train test sets are selected systematically to some degree... In next part we gonna drop some of these features to see if we can rebuild more robust model...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_roc_feat(\n    [adv_y_test],\n    [validation],\n    ['Baseline'],\n    xg_adv\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let's drop image size and number related features to see if it's increase the randomness...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"adv_X.drop(['n_images', 'image_size','width','height','total_pixels','reds','blues','greens','mean_colors', 'age_min', 'age_max'], axis=1, inplace=True)\n\n\nadv_X_train, adv_X_test, adv_y_train, adv_y_test = train_test_split(adv_X,\n                                                    adv_y,\n                                                    test_size=0.4,\n                                                    stratify=adv_y,\n                                                    random_state=42)\n\n# fitting train data\n\nxg_adv.fit(adv_X_train, adv_y_train)\n\n# predicting on holdout set\nvalidation = xg_adv.predict_proba(adv_X_test)[:,1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Well yeah, it did! The more close our AUC 50% the better, so for now we can continue what we have for now.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_roc_feat(\n    [adv_y_test],\n    [validation],\n    ['Baseline'],\n    xg_adv\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.drop(['n_images', 'image_size','width','height','total_pixels','reds','blues','greens','mean_colors', 'age_min', 'age_max'], axis=1, inplace=True)\n#X_test.drop(['n_images', 'image_size','width','height','total_pixels','reds','blues','greens','mean_colors', 'age_min', 'age_max'], axis=1, inplace=True)\n\ntest.drop(['n_images', 'image_size','width','height','total_pixels','reds','blues','greens','mean_colors', 'age_min', 'age_max'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simplified Meta Predictions\n\nWe're going build our model with less biased features now. I wouldn't recommend submitting these predictions since you have limited submissions. These are for ensembling...","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"xg= xgb.XGBClassifier(\n    n_estimators=750,\n    learning_rate=0.015,\n    min_child_weight= 218,\n    max_delta_step= 4,\n    max_depth= 2,\n    subsample= 0.751,\n    colsample_bytree= 0.77,\n    gamma= 24,\n    reg_lambda= 11,\n    random_state=42,\n    n_jobs=-1,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# display cv results\n\nraw_models = model_check(X_train, y_train, [xg], cv)\ndisplay(raw_models)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fitting train data\n\nxg.fit(X_train, y_train)\n\npredictions = xg.predict_proba(test)[:, 1]\n\nmeta_df = pd.DataFrame(columns=['image_name', 'target'])\n\n# assigning predictions on submission df\n\nmeta_df['image_name'] = sample['image_name']\nmeta_df['target'] = predictions\n\n# creating submission csv file\n\nmeta_df.to_csv('meta_simplified_img_data.csv', header=True, index=False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# End of Tabular Data Part\n\n### This simple approach including basic info as tabular data with good old ML algoithms gave me LB score of 0.8484! \n\n### Update: Since we spotted some important differences between train and test set I wanted to create less biased version of the meta .csv file so I added it here, it score less but would perform better on the ensembles you make...\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Machine Learning to Neural Networks\n\nThis part we gonna train more complicated models by using images themselves. For this part I was inspired by AgentAuers's 'Incredible TPUs' [here](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once). It's a great notebook and you should check that, again thanks for AgentAuers for letting me use some of his code as baseline for this part of the notebook! Also thanks to Chris Deotte for great datasets with tfrecords! \n\nWe start by importing neccesary packages and setting random seed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing packages\n\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom kaggle_datasets import KaggleDatasets\n\ntf.random.set_seed(seed_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading image storage buckets\n\nGCS_PATH = KaggleDatasets().get_gcs_path('melanoma-384x384')\n\nfilenames_train = np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec'))\nfilenames_test = np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting TPU as main device for training, if you get warnings while working with tpu's ignore them.\n\nDEVICE = 'TPU'\nif DEVICE == 'TPU':\n    print('connecting to TPU...')\n    try:        \n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print('Could not connect to TPU')\n        tpu = None\n\n    if tpu:\n        try:\n            print('Initializing  TPU...')\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print('TPU initialized')\n        except _:\n            print('Failed to initialize TPU!')\n    else:\n        DEVICE = 'GPU'\n\nif DEVICE != 'TPU':\n    print('Using default strategy for CPU and single GPU')\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == 'GPU':\n    print('Num GPUs Available: ',\n          len(tf.config.experimental.list_physical_devices('GPU')))\n\nprint('REPLICAS: ', strategy.num_replicas_in_sync)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we set config for our next steps. You can play with these but mind the memory sizes with the batches & image sizes.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# you can edit these settings.\n\ncfg = dict(\n           batch_size=32,\n           img_size=384,\n    \n           lr_start=0.000005,\n           lr_max=0.00000125,\n           lr_min=0.000001,\n           lr_rampup=5,\n           lr_sustain=0,\n           lr_decay=0.8,\n           epochs=12,\n    \n           transform_prob=1.0,\n           rot=180.0,\n           shr=2.0,\n           hzoom=8.0,\n           wzoom=8.0,\n           hshift=8.0,\n           wshift=8.0,\n    \n           optimizer='adam',\n           label_smooth_fac=0.05,\n           tta_steps=20\n            \n        )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift,\n            width_shift):\n    \n    ''' Settings for image preparations '''\n\n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear = math.pi * shear / 180.\n\n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1], dtype='float32')\n    zero = tf.constant([0], dtype='float32')\n    rotation_matrix = tf.reshape(\n        tf.concat([c1, s1, zero, -s1, c1, zero, zero, zero, one], axis=0),\n        [3, 3])\n\n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape(\n        tf.concat([one, s2, zero, zero, c2, zero, zero, zero, one], axis=0),\n        [3, 3])\n\n    # ZOOM MATRIX\n    zoom_matrix = tf.reshape(\n        tf.concat([\n            one / height_zoom, zero, zero, zero, one / width_zoom, zero, zero,\n            zero, one\n        ],\n                  axis=0), [3, 3])\n\n    # SHIFT MATRIX\n    shift_matrix = tf.reshape(\n        tf.concat(\n            [one, zero, height_shift, zero, one, width_shift, zero, zero, one],\n            axis=0), [3, 3])\n\n    return K.dot(K.dot(rotation_matrix, shear_matrix),\n                 K.dot(zoom_matrix, shift_matrix))\n\n\ndef transform(image, cfg):\n    \n    ''' This function takes input images of [: , :, 3] sizes and returns them as randomly rotated, sheared, shifted and zoomed. '''\n\n    DIM = cfg['img_size']\n    XDIM = DIM % 2  # fix for size 331\n\n    rot = cfg['rot'] * tf.random.normal([1], dtype='float32')\n    shr = cfg['shr'] * tf.random.normal([1], dtype='float32')\n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['hzoom']\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['wzoom']\n    h_shift = cfg['hshift'] * tf.random.normal([1], dtype='float32')\n    w_shift = cfg['wshift'] * tf.random.normal([1], dtype='float32')\n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot, shr, h_zoom, w_zoom, h_shift, w_shift)\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat(tf.range(DIM // 2, -DIM // 2, -1), DIM)\n    y = tf.tile(tf.range(-DIM // 2, DIM // 2), [DIM])\n    z = tf.ones([DIM * DIM], dtype='int32')\n    idx = tf.stack([x, y, z])\n\n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM // 2 + XDIM + 1, DIM // 2)\n\n    # FIND ORIGIN PIXEL VALUES\n    idx3 = tf.stack([DIM // 2 - idx2[0, ], DIM // 2 - 1 + idx2[1, ]])\n    d = tf.gather_nd(image, tf.transpose(idx3))\n\n    return tf.reshape(d, [DIM, DIM, 3])\n\ndef prepare_image(img, cfg=None, augment=True):\n    \n    ''' This function loads the image, resizes it, casts a tensor to a new type float32 in our case, transforms it using the function just above, then applies the augmentations.'''\n    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.image.resize(img, [cfg['img_size'], cfg['img_size']],\n                          antialias=True)\n    img = tf.cast(img, tf.float32) / 255.0\n\n    if augment:\n        if cfg['transform_prob'] > tf.random.uniform([1], minval=0, maxval=1):\n            img = transform(img, cfg)\n\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n\n    return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These functions below for reading labeled tfrecords.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'image_name': tf.io.FixedLenFeature([], tf.string),\n        'patient_id': tf.io.FixedLenFeature([], tf.int64),\n        'sex': tf.io.FixedLenFeature([], tf.int64),\n        'age_approx': tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis': tf.io.FixedLenFeature([], tf.int64),\n        'target': tf.io.FixedLenFeature([], tf.int64),\n        #'width': tf.io.FixedLenFeature([], tf.int64),\n        #'height': tf.io.FixedLenFeature([], tf.int64)\n    }\n\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'image_name': tf.io.FixedLenFeature([], tf.string),\n        'patient_id': tf.io.FixedLenFeature([], tf.int64),\n        'sex': tf.io.FixedLenFeature([], tf.int64),\n        'age_approx': tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    return example['image'], example['image_name']\n\ndef count_data_items(filenames):\n    n = [\n        int(re.compile(r'-([0-9]*)\\.').search(filename).group(1))\n        for filename in filenames\n    ]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getTrainDataset(files, cfg, augment=True, shuffle=True):\n    \n    ''' This function reads the tfrecord train images, shuffles them, apply augmentations to them and prepares the data for training. '''\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n\n    if shuffle:\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n\n    ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    ds = ds.repeat()\n    if shuffle:\n        ds = ds.shuffle(2048)\n    ds = ds.map(lambda img, label:\n                (prepare_image(img, augment=augment, cfg=cfg), label),\n                num_parallel_calls=AUTO)\n    ds = ds.batch(cfg['batch_size'] * strategy.num_replicas_in_sync)\n    ds = ds.prefetch(AUTO)\n    return ds\n\ndef getTestDataset(files, cfg, augment=False, repeat=False):\n    \n    ''' This function reads the tfrecord test images and prepares the data for predicting. '''\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    if repeat:\n        ds = ds.repeat()\n    ds = ds.map(read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    ds = ds.map(lambda img, idnum:\n                (prepare_image(img, augment=augment, cfg=cfg), idnum),\n                num_parallel_calls=AUTO)\n    ds = ds.batch(cfg['batch_size'] * strategy.num_replicas_in_sync)\n    ds = ds.prefetch(AUTO)\n    return ds\n\ndef get_model():\n    \n    ''' This function gets the layers inclunding efficientnet ones. '''\n    \n    model_input = tf.keras.Input(shape=(cfg['img_size'], cfg['img_size'], 3),\n                                 name='img_input')\n\n    dummy = tf.keras.layers.Lambda(lambda x: x)(model_input)\n\n    outputs = []\n\n    x = efn.EfficientNetB3(include_top=False,\n                           weights='noisy-student',\n                           input_shape=(cfg['img_size'], cfg['img_size'], 3),\n                           pooling='avg')(dummy)\n    x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n    outputs.append(x)\n\n    x = efn.EfficientNetB4(include_top=False,\n                           weights='noisy-student',\n                           input_shape=(cfg['img_size'], cfg['img_size'], 3),\n                           pooling='avg')(dummy)\n    x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n    outputs.append(x)\n\n    x = efn.EfficientNetB5(include_top=False,\n                           weights='noisy-student',\n                           input_shape=(cfg['img_size'], cfg['img_size'], 3),\n                           pooling='avg')(dummy)\n    x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n    outputs.append(x)\n\n    model = tf.keras.Model(model_input, outputs, name='aNetwork')\n    model.summary()\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def compileNewModel(cfg):\n    \n    ''' Configuring the model with losses and metrics. '''    \n    \n    with strategy.scope():\n        model = get_model()\n\n    with strategy.scope():\n        model.compile(optimizer=cfg['optimizer'],\n                      loss=[\n                          tf.keras.losses.BinaryCrossentropy(\n                              label_smoothing=cfg['label_smooth_fac']),\n                          tf.keras.losses.BinaryCrossentropy(\n                              label_smoothing=cfg['label_smooth_fac']),\n                          tf.keras.losses.BinaryCrossentropy(\n                              label_smoothing=cfg['label_smooth_fac'])\n                      ],\n                      metrics=[tf.keras.metrics.AUC(name='auc')])\n    return model\n\ndef getLearnRateCallback(cfg):\n    \n    ''' Using callbacks for learning rate adjustments. '''\n    \n    lr_start = cfg['lr_start']\n    lr_max = cfg['lr_max'] * strategy.num_replicas_in_sync * cfg['batch_size']\n    lr_min = cfg['lr_min']\n    lr_rampup = cfg['lr_rampup']\n    lr_sustain = cfg['lr_sustain']\n    lr_decay = cfg['lr_decay']\n\n    def lrfn(epoch):\n        if epoch < lr_rampup:\n            lr = (lr_max - lr_start) / lr_rampup * epoch + lr_start\n        elif epoch < lr_rampup + lr_sustain:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_rampup -\n                                                lr_sustain) + lr_min\n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback\n\ndef learnModel(model, ds_train, stepsTrain, cfg, ds_val=None, stepsVal=0):\n    \n    ''' Fitting things together for training '''\n    \n    callbacks = [getLearnRateCallback(cfg)]\n\n    history = model.fit(ds_train,\n                        validation_data=ds_val,\n                        verbose=1,\n                        steps_per_epoch=stepsTrain,\n                        validation_steps=stepsVal,\n                        epochs=cfg['epochs'],\n                        callbacks=callbacks)\n\n    return history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we train our model, takes a while but at the end we'll have strong model to make predictions!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# getting train data\n\nds_train = getTrainDataset(\n    filenames_train, cfg).map(lambda img, label: (img, (label, label, label)))\nstepsTrain = count_data_items(filenames_train) / \\\n    (cfg['batch_size'] * strategy.num_replicas_in_sync)\n\n# compiling and training model\n\nmodel = compileNewModel(cfg)\nlearnModel(model, ds_train, stepsTrain, cfg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we make predictions using the model we trained. Then we blend them for each EffNet by taking mean. We create csv file for each prediction including blended one.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"steps = count_data_items(filenames_test) / \\\n    (cfg['batch_size'] * strategy.num_replicas_in_sync)\n\nz = np.zeros((cfg['batch_size'] * strategy.num_replicas_in_sync))\n\n# loading test data\n\nds_testAug = getTestDataset(\n    filenames_test, cfg, augment=True,\n    repeat=True).map(lambda img, label: (img, (z, z, z)))\n\n# test time augmentations for predictions (20 in our case, you can increase it a little in cfg) and taking mean of them\n\nprobs = model.predict(ds_testAug, verbose=1, steps=steps * cfg['tta_steps'])\nprobs = np.stack(probs)\nprobs = probs[:, :count_data_items(filenames_test) * cfg['tta_steps']]\nprobs = np.stack(np.split(probs, cfg['tta_steps'], axis=1), axis=1)\nprobs = np.mean(probs, axis=1)\n\ntest = pd.read_csv(os.path.join(base_path, 'test.csv'))\n\ny_test_sorted = np.zeros((3, probs.shape[1]))\ntest = test.reset_index()\ntest = test.set_index('image_name')\n\ni = 0\nds_test = getTestDataset(filenames_test, cfg)\nfor img, imgid in tqdm(iter(ds_test.unbatch())):\n    imgid = imgid.numpy().decode('utf-8')\n    y_test_sorted[:, test.loc[imgid]['index']] = probs[:, i, 0]\n    i += 1\n\n    \n# creating .csv files for each effnet\n    \nfor i in range(y_test_sorted.shape[0]):\n    submission = sample\n    submission['target'] = y_test_sorted[i]\n    submission.to_csv('submission_model_%s.csv' % i, index=False)\n\n# blending effnets into a single .csv file    \n\nsubmission = sample\nsubmission['target'] = np.mean(y_test_sorted, axis=0)\nsubmission.to_csv('blended_effnets.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensembling With Meta\n\nHere's the last step. We'll use our blended predictions created by training images and simply metadata created by using tabular data. We ensemble them together with weights and make our final predictions. Feel free to experiment with ensembling. This basic blending increased my LB score a little, you can change lots of things in previous steps to do some experiments, it's fun!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading recently created .csv files from working directory\n\neffnet = pd.read_csv('./blended_effnets.csv')\nmeta = pd.read_csv('./meta_simplified_img_data.csv')\n\n\nsample['target'] = (\n                           \n                           effnet['target'] * 0.9 +\n                           meta['target'] * 0.1 \n                          \n                          )\n\n# final submissions\n\nsample.to_csv('ensembled.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission and Some Notes\n\nAgain I wanted to thank to kaggle community for inspiring me for this approach. I'm still learning and working on these notebooks helping me a lot, I hope these can be helpful you too! About the notebook itself next step would be implementing a cross validation scheme and test your predictions on it, if you find sweet spot that both increases your CV and LB scores then you are on the right track! But I'll leave this notebook here for now, thank you all for reading!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n<div align='center'><font size='6' color='#000000'>Final Words</font></div>\n\n<hr>\n\n<div align='center'><font size='4' color='#000000'>This notebook still in progress and if you have any feedbacks please leave me a comment I'll be reading them for sure and if you liked my work please don't forget to leave an upvote. Thank you for reading!</font></div>\n\n<hr>\n\n## Update: Added new notebook here using our EffNet predictions here and ensembling them with external tabular data including extra malignant examples. You can find it here [Analysis of External and Upsampled Metadata for SIIM-ISIC Melanoma Classification](https://www.kaggle.com/datafan07/eda-modelling-of-the-external-data-inc-ensemble). Happy coding all!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}