{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Analysis of External and Upsampled Metadata for SIIM-ISIC Melanoma Classification\n### Introduction\n\n#### This notebook is part of the project I started here: [Analysis of Melanoma Metadata and EffNet Ensemble](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble), more like complementary work of the tabular part. The main objective of this notebook is using past years competitions data as well as using upsampling with the great datasets uploaded by Chris Deotte [explained here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169139), so what we have in this data:\n\n* Image Names, Sex, Age, Anatom Sites, Width/Height of the Images and Targets of:\n    * Current 2020 set with 33k examples including 584 malignant examples,\n    * 2017/18-2019 set with 25k examples with 4522 malignant examples,\n    * 580 completely new malignant examples coming from Chris Deotte's Malignant TFRecords 15-29\n    \n#### So what are we trying to achieve is getting more solid predictions solely based on metadata, with the help of upsampled data. So we can use these results in our ensembles to get stronger final products for final predictions...\n\n#### I'm gonna try to keep this work as simple as possible and similar to my main work so you can compare them easily.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Loading the Data","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\n\nimport seaborn as sns\nimport plotly.express as px\n\nimport os\nimport random\nimport re\nimport math\nimport time\n\nimport warnings\n\nwarnings.filterwarnings('ignore') # Disabling warnings for clearer outputs\n\n\nseed_val = 42\nrandom.seed(seed_val)\nnp.random.seed(seed_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Setting color palette.\nblack_red = [\n    '#1A1A1D', '#4E4E50', '#C5C6C7', '#6F2232', '#950740', '#C3073F'\n]\n\n# Setting plot styling.\nplt.style.use('fivethirtyeight')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# loading datasets\n\ntrain = pd.read_csv('../input/melanomaextendedtabular/external_upsampled_tabular.csv')\ntest = pd.read_csv('../input/melanomaextendedtabular/test_tabular.csv')\nsample = pd.read_csv('../input/melanomaextendedtabular/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking column names\n\nprint(\n    f'Train data has {train.shape[1]} features, {train.shape[0]} observations and Test data {test.shape[1]} features, {test.shape[0]} observations.\\nTrain features are:\\n{train.columns.tolist()}\\nTest features are:\\n{test.columns.tolist()}'\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# renaming column names for easier use\n\ntrain.columns = [\n    'img_name',  'sex', 'age', 'location', 'target','width','height'\n]\n\ntest.columns = ['img_name', 'sex', 'age', 'location','width','height']\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Missing Values\n\nLooks like our external data has some missing values, which is expected. Body part missing numbers increased around three times, missing ages increased about four times and gender missing rates didn't change much but small increase... [2020 values here](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble#Missing-Values).","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Checking missing values:\n\ndef missing_percentage(df):\n\n    total = df.isnull().sum().sort_values(\n        ascending=False)[df.isnull().sum().sort_values(ascending=False) != 0]\n    percent = (df.isnull().sum().sort_values(ascending=False) / len(df) *\n               100)[(df.isnull().sum().sort_values(ascending=False) / len(df) *\n                     100) != 0]\n    return pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n\n\nmissing_train = missing_percentage(train)\nmissing_test = missing_percentage(test)\n\nfig, ax = plt.subplots(1, 2, figsize=(16, 6))\n\nsns.barplot(x=missing_train.index,\n            y='Percent',\n            data=missing_train,\n            palette=black_red,\n            ax=ax[0])\n\nsns.barplot(x=missing_test.index,\n            y='Percent',\n            data=missing_test,\n            palette=black_red,\n            ax=ax[1])\n\nax[0].set_title('Train Data Missing Values')\nax[1].set_title('Test Data Missing Values')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Distribution Differences\n\nMeanwhile age and gender distributions seems pretty similar with external data anatom site part has some differences: palms/soles, oral/genital and head/neck parts increased a lot in train data we might have to check these further...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 9))\n\n# Creating a grid:\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Gender Distribution')\n\nsns.countplot(train.sex.sort_values(ignore_index=True),\n              alpha=0.9,\n              ax=ax1,\n              color='#C3073F',\n              label='Train')\nsns.countplot(test.sex.sort_values(ignore_index=True),\n              alpha=0.7,\n              ax=ax1,\n              color='#1A1A1D',\n              label='Test')\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Plot the countplot.\n\nsns.countplot(train.location,\n              alpha=0.9,\n              ax=ax2,\n              color='#C3073F',\n              label='Train',\n              order=train['location'].value_counts().index)\nsns.countplot(test.location,\n              alpha=0.7,\n              ax=ax2,\n              color='#1A1A1D',\n              label='Test',\n              order=test['location'].value_counts().index), ax2.set_title(\n                  'Anatom Site Distribution')\n\nax2.legend()\nplt.xticks(rotation=20)\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Age Distribution')\n\n# Plot the histogram.\n\nsns.distplot(train.age, ax=ax3, label='Train', color='#C3073F')\nsns.distplot(test.age, ax=ax3, label='Test', color='#1A1A1D')\n\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imputing Missing Values\n\nThis time I decided to replace missing values with fixed values instead of replacing them with most frequent ones, since we have higher miss ratio...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filling missing  values with 'unknown' and '-1' tags:\n\nfor df in [train, test]:\n    df['location'].fillna('unknown', inplace=True)\n    \ntrain['sex'].fillna('unknown', inplace=True)\n\ntrain['age'].fillna(-1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Anatom Site Differences\n\nIt seems there are some differences between 2020 data and external data in terms of body part values. Looks like external data has more detailed approach for torso, we can fix that by merging all torso parts to 2020 data version which is torso for all...(Duh!)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Double checking:\n\nids_train = train.location.values\nids_test = test.location.values\nids_train_set = set(ids_train)\nids_test_set = set(ids_test)\n\nlocation_not_overlap = list(ids_train_set.symmetric_difference(ids_test_set))\nn_overlap = len(location_not_overlap)\nif n_overlap == 0:\n    print(\n        f'There are no different body parts occuring between train and test set...'\n    )\nelse:\n    print('There are some non-overlapping values between train and test set!\\n')\n    print(f'Different ones are:\\n{pd.Series(np.setdiff1d((train.location.value_counts().index), pd.Series(test.location.value_counts().index)))}')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# merging detailed torso approach to torso only\ntrain.replace(['anterior torso','lateral torso','posterior torso'], 'torso', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Body Part Ratio by Gender and Target\n\nWe have very different malignant ratio for external data it seems. You can check [2020 values here](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble#Body-Part-Ratio-by-Gender-and-Target). This is big difference but might be useful for unseen test data...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 9))\n# Creating a grid\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[1, :2])\n# Set the title.\nax1.set_title('Scanned Body Parts - Female')\n\n# Plot:\n\nsns.countplot(\n    train[train['sex'] == 'female'].location.sort_values(ignore_index=True),\n    alpha=0.9,\n    ax=ax1,\n    color='#C3073F',\n    label='Female',\n    order=train['location'].value_counts().index)\nax1.legend()\nplt.xticks(rotation=20)\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[1, 2:])\n\n# Set the title.\n\nax2.set_title('Scanned Body Parts - Male')\n\n# Plot.\n\nsns.countplot(\n    train[train['sex'] == 'male'].location.sort_values(ignore_index=True),\n    alpha=0.9,\n    ax=ax2,\n    color='#1A1A1D',\n    label='Male',\n    order=train['location'].value_counts().index)\n\nax2.legend()\nplt.xticks(rotation=20)\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[0, :])\n\n# Set the title.\n\nax3.set_title('Malignant Ratio Per Body Part')\n\n# Plot.\n\nloc_freq = train.groupby('location')['target'].mean().sort_values(\n    ascending=False)\nsns.barplot(x=loc_freq.index, y=loc_freq, palette=black_red, ax=ax3)\n\nax3.legend()\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Sunburst Chart\n\n\n- We almost doubled our malignant examples with upsampling and external data,\n- Malignant images became more balanced 56% male to 44% female. It was 62% vs 38% in 2020 data,\n- Gender wise benign images has same ratio as 2020,\n- Malignant image scan locations differs based on the patients gender:\n    - Torso still most common location in males even with 3% decrease; meanwhile female malignant torso scans decreased like 6% with external data,\n    - Lower extremity still more common with female scans than males 12% males vs 26% females (it was 18% vs 26% in 2020 data)\n    - Again upper extremity malignant scans are more common with females than males with 15% males vs 20% females (used to be 17% - 23%)\n    - Head/Neck malignant scans increased with external data on both genders.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plotting interactive sunburst:\n\nfig = px.sunburst(data_frame=train,\n                  path=['target', 'sex', 'location'],\n                  color='sex',\n                  color_discrete_sequence=black_red,\n                  maxdepth=-1,\n                  title='Sunburst Chart Benign/Malignant > Sex > Location')\n\nfig.update_traces(textinfo='label+percent parent')\nfig.update_layout(margin=dict(t=0, l=0, r=0, b=0))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Age and Scan Result Relations\n\nAge distribution seems little bit different with external data added. In general we have older patients, benign scan age distributions seems more closer to each other than 2020 data, meanwhile malignant difference stay at same levels between genders. We can say generally examples which missing gender values are also missing age values too. Lastly we still have age effect on malignant scans with the difference visible on age below 0 because of the imputing method we choose.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plotting age vs sex vs target:\n\nfig, ax = plt.subplots(1, 2, figsize=(16, 6))\nsns.lineplot(x='age',\n             y='target',\n             data=train,\n             ax=ax[0],\n             hue='sex',\n             palette=black_red[2:5],\n             ci=None)\nsns.boxplot(x='target',\n            y='age',\n            data=train,\n            ax=ax[1],\n            hue='sex',\n            palette=black_red[2:5]\n           )\n\nplt.legend(loc='upper right')\n\nax[0].set_title('Malignant Scan Frequency by Age')\nax[1].set_title('Scan Results by Age and Sex')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Age \n\nAge distribution seems similar with external data, you can [you can compare them here](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble#Age-Round-Two).","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\n# Plotting age dist vs target and age dist vs datasets\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Age Distribution by Scan Outcome')\n\n# Plot\n\nax1.legend()\n\nsns.kdeplot(train[train['target'] == 0]['age'],\n            shade=True,\n            ax=ax1,\n            color='#1A1A1D',\n            label='Benign')\nsns.kdeplot(train[train['target'] == 1]['age'],\n            shade=True,\n            ax=ax1,\n            color='#C3073F',\n            label='Malignant')\n\n# Customizing second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Age Distribution by Train/Test Observations')\n\n# Plot.\n\nsns.kdeplot(train.age, label='Train', shade=True, ax=ax2, color='#1A1A1D')\nsns.kdeplot(test.age, label='Test', shade=True, ax=ax2, color='#C3073F')\n\nax2.legend()\n\n# Customizing third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Age Distribution by Gender')\n\n# Plot\n\nsns.distplot(train[train.sex == 'female'].age,\n             ax=ax3,\n             label='Female',\n             color='#C3073F')\nsns.distplot(train[train.sex == 'male'].age,\n             ax=ax3,\n             label='Male',\n             color='#1A1A1D')\nax3.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Resolutions \n\nWe had decent observations in the previous notebook about image sizes so wanted to check them here with external data again. With the new examples we increased variety of the image sizes (with high number of 1024x1024 images coming in) but we can see 1920x1080 images in test set still seperated, so we should be really careful about image sizes in our models...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# getting temporary resolution feature\n\ntrain['res']= train['width'].astype(str)+'x'+train['height'].astype(str)\ntest['res']= test['width'].astype(str)+'x'+test['height'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a customized chart and giving in figsize etc.\n\nfig = plt.figure(constrained_layout=True, figsize=(20, 12))\n\n# Creating a grid\n\ngrid = gridspec.GridSpec(ncols=4, nrows=3, figure=fig)\n\n# Customizing the first grid.\n\nax1 = fig.add_subplot(grid[0, :2])\n\n# Set the title.\n\nax1.set_title('Scan Image Resolutions of Train Set')\n\n# Plot.\n\ntres = train.res.value_counts().rename_axis('res').reset_index(name='count')\ntres = tres[tres['count'] > 10]\nsns.barplot(x='res', y='count', data=tres, palette=black_red, ax=ax1)\nplt.xticks(rotation=60)\n\nax1.legend()\n\n# Customizing the second grid.\n\nax2 = fig.add_subplot(grid[0, 2:])\n\n# Set the title.\n\nax2.set_title('Scan Image Resolutions of Test Set')\n\n# Plot\n\nteres = test.res.value_counts().rename_axis('res').reset_index(name='count')\nteres = teres[teres['count'] > 10]\nsns.barplot(x='res', y='count', data=teres, palette=black_red, ax=ax2)\nplt.xticks(rotation=30)\nax2.legend()\n\n# Customizing the third grid.\n\nax3 = fig.add_subplot(grid[1, :])\n\n# Set the title.\n\nax3.set_title('Scan Image Resolutions by Target')\n\n# Plot.\n\nsns.countplot(x='res',\n              hue='target',\n              data=train,\n              order=train.res.value_counts().iloc[:15].index,\n              palette=black_red[2:4],\n              ax=ax3)\nax3.legend()\n\n# Customizing the last grid.\n\nax4 = fig.add_subplot(grid[2, :])\n\n# Set the title.\n\nax4.set_title('Malignant Scan Result Frequency by Image Resolution')\n\n# Plot.\n\nres_freq = train.groupby('res')['target'].mean()\nres_freq = res_freq[(res_freq > 0) & (res_freq < 1)]\nsns.lineplot(x=res_freq.index, y=res_freq, color='#C3073F', ax=ax4)\nax4.legend()\nplt.xticks(rotation=60)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# getting rid of temporary features\n\ntrain.drop(['res'], axis=1, inplace=True)\ntest.drop(['res'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling the Tabular Data\n\nThis part is pretty straight and very similar to work we have in previous notebook. First we get dummies for categorical data and drop redundant ones, then we import neccesary packages, split our data, set model parameters,cross validate and so on...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#creating dummy variables for categorical sex data\n\nsex_dummies = pd.get_dummies(train['sex'], prefix='sex')\ntrain = pd.concat([train, sex_dummies], axis=1)\n\nsex_dummies = pd.get_dummies(test['sex'], prefix='sex')\ntest = pd.concat([test, sex_dummies], axis=1)\n\ntrain.drop(['sex'], axis=1, inplace=True)\ntest.drop(['sex'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# getting dummy variables for location on train set\n\nanatom_dummies = pd.get_dummies(train['location'], prefix='anatom')\ntrain = pd.concat([train, anatom_dummies], axis=1)\n\n# getting dummy variables for location on test set\n\nanatom_dummies = pd.get_dummies(test['location'], prefix='anatom')\ntest = pd.concat([test, anatom_dummies], axis=1)\n\n# dropping useless columns\n\ntrain.drop('location', axis=1, inplace=True)\ntest.drop(['location'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dropping redundant columns for both dataset\n\nfor df in [train, test]:\n    df.drop('img_name', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# importing basic modelling stuff\n\nimport xgboost as xgb\n\nfrom sklearn.model_selection import StratifiedKFold, train_test_split, cross_val_score, cross_validate\nfrom sklearn.metrics import roc_auc_score, roc_curve","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating train variables\n\nX = train.drop('target', axis=1)\ny = train.target","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# taking 15% of the training data as holdout\n\nX_train, X_test, y_train, y_test = train_test_split(X,\n                                                y,\n                                                test_size=0.15,\n                                                stratify=y,\n                                                random_state=42)\n\n# 5 fold stratify for cv\n\ncv = StratifiedKFold(5, shuffle=True, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# setting model hyperparameters, didn't include fine tuning here because of timing reasons...\n\nxg = xgb.XGBClassifier(\n    n_estimators=750,\n    min_child_weight=0.81,\n    learning_rate=0.025,\n    max_depth=2,\n    subsample=0.80,\n    colsample_bytree=0.42,\n    gamma=0.10,\n    random_state=42,\n    n_jobs=-1,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"estimators = [xg]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# cross validation scheme\n\ndef model_check(X_train, y_train, estimators, cv):\n    model_table = pd.DataFrame()\n\n    row_index = 0\n    for est in estimators:\n\n        MLA_name = est.__class__.__name__\n        model_table.loc[row_index, 'Model Name'] = MLA_name\n\n        cv_results = cross_validate(est,\n                                    X_train,\n                                    y_train,\n                                    cv=cv,\n                                    scoring='roc_auc',\n                                    return_train_score=True,\n                                    n_jobs=-1)\n\n        model_table.loc[row_index,\n                        'Train roc Mean'] = cv_results['train_score'].mean()\n        model_table.loc[row_index,\n                        'Test roc Mean'] = cv_results['test_score'].mean()\n        model_table.loc[row_index, 'Test Std'] = cv_results['test_score'].std()\n        model_table.loc[row_index, 'Time'] = cv_results['fit_time'].mean()\n\n        row_index += 1\n\n    model_table.sort_values(by=['Test roc Mean'],\n                            ascending=False,\n                            inplace=True)\n\n    return model_table","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# First Results\n\nHmmm... I feel like there's overfitting, not directly based on cross validation but from the experiences based on past notebook, we should check feature importances and then apply adversarial validation if needed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# display cv results\n\nraw_models = model_check(X_train, y_train, estimators, cv)\ndisplay(raw_models.style.background_gradient(cmap='twilight', axis=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fitting train data\n\nxg.fit(X_train, y_train)\n\n# predicting on holdout set\nvalidation = xg.predict_proba(X_test)[:, 1]\n\n# checking results on validation set\nroc_auc_score(y_test, validation)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# finding feature importances and creating new dataframe basen on them\n\nfeature_importance = xg.get_booster().get_score(importance_type='weight')\n\nkeys = list(feature_importance.keys())\nvalues = list(feature_importance.values())\n\nimportance = pd.DataFrame(data=values, index=keys,\n                          columns=['score']).sort_values(by='score',\n                                                         ascending=False)\nfig, ax = plt.subplots(figsize=(16, 10))\nsns.barplot(x=importance.score.iloc[:20],\n            y=importance.index[:20],\n            orient='h',\n            palette='Reds_r')\nax.set_title('Feature Importances')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Adversarial Validation\n\nJust quoting from the previous notebook:\n\n> Alright, since we have high doubts for train test sampling wanted to implement what is called 'Adversarial Validation'. For this we going to replace our targets for both datasets (0 for train and 1 for test), then we going build a classifier which tries to predict which observation belongs to train and which one belongs to test set. If datasets randomly selected from similar roots it should be really hard for the classifier to separate them. But if there is systematic selection differences between train and test sets then classifier should be able to capture this trend. So we want our models score lower for the next section because higher detection rate means higher difference between train and test datasets, so let's get started...\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating adversarial training set\n\nadv_train = train.copy()\nadv_train.drop('target', axis=1, inplace=True)\nadv_test = test.copy()\n\nadv_train['dataset_label'] = 0\nadv_test['dataset_label'] = 1\n\nadv_master = pd.concat([adv_train, adv_test], axis=0)\n\nadv_X = adv_master.drop('dataset_label', axis=1)\nadv_y = adv_master['dataset_label']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# holdout set for adv\n\nadv_X_train, adv_X_test, adv_y_train, adv_y_test = train_test_split(adv_X,\n                                                    adv_y,\n                                                    test_size=0.4,\n                                                    stratify=adv_y,\n                                                    random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"xg_adv = xgb.XGBClassifier(\n    random_state=42,\n    n_jobs=-1,\n)\n\n# Fitting train data\n\nxg_adv.fit(adv_X_train, adv_y_train)\n\n# Predicting on holdout set\nvalidation = xg_adv.predict_proba(adv_X_test)[:,1]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_roc_feat(y_trues, y_preds, labels, est, x_max=1.0):\n    \n    \"\"\" A function for displaying roc/auc curve and feature importances. \"\"\"\n    \n    fig, ax = plt.subplots(1,2, figsize=(16,6))\n    for i, y_pred in enumerate(y_preds):\n        y_true = y_trues[i]\n        fpr, tpr, thresholds = roc_curve(y_true, y_pred)\n        auc = roc_auc_score(y_true, y_pred)\n        ax[0].plot(fpr, tpr, label='%s; AUC=%.3f' % (labels[i], auc), marker='o', markersize=1)\n\n    ax[0].legend()\n    ax[0].grid()\n    ax[0].plot(np.linspace(0, 1, 20), np.linspace(0, 1, 20), linestyle='--')\n    ax[0].set_title('ROC curve')\n    ax[0].set_xlabel('False Positive Rate')\n    ax[0].set_xlim([-0.01, x_max])\n    _ = ax[0].set_ylabel('True Positive Rate')\n    \n    \n    feature_importance = est.get_booster().get_score(importance_type='weight')\n\n    keys = list(feature_importance.keys())\n    values = list(feature_importance.values())\n\n    importance = pd.DataFrame(data=values, index=keys,\n                          columns=['score']).sort_values(by='score',\n                                                         ascending=False)\n    \n    sns.barplot(x=importance.score.iloc[:20],\n            y=importance.index[:20],\n            orient='h',\n            palette=black_red, ax=ax[1])\n    ax[1].set_title('Feature Importances')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation Results\n\nWow... So with the help of \"sex_unknown\", \"height\" and \"width\" features our model perfectly predicted what data is coming from external train data or original test set. We should take care of these and see if it helps...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_roc_feat(\n    [adv_y_test],\n    [validation],\n    ['Baseline'],\n    xg_adv\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dropping features for better randomness\n\nadv_X.drop(['sex_unknown', 'height', 'width'], axis=1, inplace=True)\n\n\nadv_X_train, adv_X_test, adv_y_train, adv_y_test = train_test_split(adv_X,\n                                                    adv_y,\n                                                    test_size=0.4,\n                                                    stratify=adv_y,\n                                                    random_state=42)\n\n# fitting train data\n\nxg_adv.fit(adv_X_train, adv_y_train)\n\n# predicting on holdout set\nvalidation = xg_adv.predict_proba(adv_X_test)[:,1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation Without Some Features\n\nSo when we drop these features highly important on seperating train and test set we got much random results, maybe it's not perfect but seems something we can work with...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_roc_feat(\n    [adv_y_test],\n    [validation],\n    ['Baseline'],\n    xg_adv\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dropping features from original train set\n\nX_train.drop(['sex_unknown', 'width','height'], axis=1, inplace=True)\n\ntest.drop(['width','height'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simplified Version of the Extended Tabular Results\n\nAlright! Without the distinctive features we lost some auc score but I feel it's much more regularized and can be used for our testing predictions plus ensembles...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# display cv results\n\nraw_models = model_check(X_train, y_train, [xg], cv)\ndisplay(raw_models.style.background_gradient(cmap='twilight', axis=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fitting and predicting\n\nxg.fit(X_train, y_train)\n\npredictions = xg.predict_proba(test)[:, 1]\n\nmeta_df = pd.DataFrame(columns=['image_name', 'target'])\n\n# assigning predictions on submission df\n\nmeta_df['image_name'] = sample['image_name']\nmeta_df['target'] = predictions\n\n# creating submission csv file\n\nmeta_df.to_csv('external_tabular_predicts.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = xg.predict_proba(test)[:, 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble and Submit\n\nWe don't need to run whole model again here so I'm going to load EffNet predictions from previous [notebook here](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble#Machine-Learning-to-Neural-Networks).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading predictions from csv file and ensemble them\n\neffnet = pd.read_csv('../input/blendedeffnet/blended_effnets.csv')\n\nmeta = pd.read_csv('./external_tabular_predicts.csv')\n\n\nsample['target'] = (\n                           effnet['target'] * 0.9 +\n                           meta['target'] * 0.1 \n                          \n                          )\n\nsample.to_csv('external_meta_ensembled.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# display auc distribution\n\nfig, ax = plt.subplots(figsize=(16,6))\nsns.distplot(sample['target'], hist_kws={\n                 'rwidth': 0.75,\n                 'edgecolor': 'black',\n                 'alpha': 0.3\n             }, color='#C3073F')\nax.set_title('Final Predictions')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Final Words\n\n### This ensembling increased my public LB score by small margin in my private tests compared to previous 2020 meta blend. I hope it works for you too!\n\n#### Again this is complementary work of the what I started at [Analysis of Melanoma Metadata and EffNet Ensemble](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble), I didn't want to pile it on with more EDA and modelling again, hope it helps you as much as it did to me! I'm open to feedbacks so you can leave a comment if you like and if you enjoyed my work it please don't forget to vote, happy coding!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **CatBoost**\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom scipy.stats import skew\nimport xgboost as xgb\nfrom sklearn.ensemble import ExtraTreesClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import LogisticRegression\nfrom math import sqrt\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import KFold\nfrom catboost import CatBoostClassifier\nfrom lightgbm import LGBMClassifier\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cb = CatBoostClassifier(\n#     iterations=2000,\n#     learning_rate=0.02,\n#     depth=5,\n#     l2_leaf_reg=15,\n#     bootstrap_type='Bernoulli',\n#     subsample=0.8,\n#     #scale_pos_weight=,\n#     eval_metric='AUC',\n#     od_type='Iter',\n#     allow_writing_files=False,\n#     random_seed=42)\n#parameters in CatBoostClassifier\n#By default, CatBoost builds 1000 trees. The number of iterations can be decreased to speed up the training.\n#In most cases, the optimal depth ranges from 4 to 10. Values in the range from 6 to 10 are recommended.\n#The type of the overfitting detector to use.\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cb.fit(X_train, y_train)\n# predictions = cb.predict_proba(X_train)[:, 1]\n# predictions\n# df = pd.DataFrame(data=predictions)\n# df.to_csv('X_CatBoost_pred.csv', header=False, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# raw_models = model_check(X_train, y_train, [cb], cv)\n# display(raw_models.style.background_gradient(cmap='twilight', axis=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **XGBoost**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# predictions_x = xg.predict_proba(X_train)[:, 1]\n# predictions_x\n# df = pd.DataFrame(data=predictions_x)\n# df.to_csv('X_XGboost_pred.csv', header=False, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# len(predictions_x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading neccesary packages for modelling.\n\nfrom sklearn.model_selection import cross_val_score, KFold, cross_validate\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.linear_model import ElasticNetCV, LassoCV, RidgeCV, TweedieRegressor\nfrom sklearn.experimental import enable_hist_gradient_boosting\nfrom sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor\nfrom sklearn.svm import SVR\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.metrics import mean_squared_error\nfrom xgboost import XGBRegressor\nfrom lightgbm import LGBMRegressor\nfrom mlxtend.regressor import StackingCVRegressor # This is for stacking part works well with sklearn and others.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kf = KFold(5, random_state=42)\nalphas_alt = [15.5, 15.6, 15.7, 15.8, 15.9, 15, 15.1, 15.2, 15.3, 15.4, 15.5]\nalphas2 = [\n    5e-05, 0.0001, 0.0002, 0.0003, 0.0004, 0.0005, 0.0006, 0.0007, 0.0008\n]\ne_alphas = [\n    0.0001, 0.0002, 0.0003, 0.0004, 0.0005, 0.0006, 0.0007\n]\ne_l1ratio = [0.8, 0.85, 0.9, 0.95, 0.99, 1]\n\n# ridge_cv\n\nridge = make_pipeline(RobustScaler(), RidgeCV(\n    alphas=alphas_alt,\n    cv=kf,\n))\n\n# lasso_cv\n\nlasso = make_pipeline(\n    RobustScaler(),\n    LassoCV(max_iter=1e7, alphas=alphas2, random_state=42, cv=kf))\n\n# elasticnet_cv\n\nelasticnet = make_pipeline(\n    RobustScaler(),\n    ElasticNetCV(max_iter=1e7,\n                 alphas=e_alphas,\n                 cv=kf,\n                 random_state=42,\n                 l1_ratio=e_l1ratio))\n\n# svr\n\nsvr = make_pipeline(RobustScaler(),\n                    SVR(C=21, epsilon=0.0099, gamma=0.00017, tol=0.000121))\n\n# gradientboosting\n\ngbr = GradientBoostingRegressor(n_estimators=3500,\n                                learning_rate=0.0161,\n                                max_depth=4,\n                                max_features='sqrt',\n                                min_samples_leaf=17,\n                                loss='ls',\n                                random_state=42)\n\n# lightgbm\nlightgbm = LGBMRegressor(objective='regression',\n                         n_estimators=3500,\n                         num_leaves=5,\n                         learning_rate=0.00721,\n                         max_bin=163,\n                         bagging_fraction=0.35711,\n                         n_jobs=-1,\n                         bagging_seed=42,\n                         feature_fraction_seed=42,\n                         bagging_freq=7,\n                         feature_fraction=0.1294,\n                         min_data_in_leaf=8)\n\n# xgboost\n\nxgboost = xgb.XGBClassifier(\n    n_estimators=750,\n    min_child_weight=0.81,\n    learning_rate=0.025,\n    max_depth=2,\n    subsample=0.80,\n    colsample_bytree=0.42,\n    gamma=0.10,\n    random_state=42,\n    n_jobs=-1,\n)\n\n\n# hist gradient boosting regressor\n\nhgrd= HistGradientBoostingRegressor(    loss= 'least_squares',\n    max_depth= 2,\n    min_samples_leaf= 40,\n    max_leaf_nodes= 29,\n    learning_rate= 0.15,\n    max_iter= 225,\n                                    random_state=42)\n\n# tweedie regressor\n \ntweed = make_pipeline(RobustScaler(),TweedieRegressor(alpha=0.005))\n\n\n# stacking regressor\n\nstack_gen = StackingCVRegressor(regressors=(ridge, lasso, elasticnet, gbr,\n                                            xgboost, lightgbm,hgrd, tweed),\n                                meta_regressor=xgboost,\n                                use_features_in_secondary=True)\n\n#cat boost\ncb = CatBoostClassifier(\n    iterations=2000,\n    learning_rate=0.02,\n    depth=5,\n    l2_leaf_reg=15,\n    bootstrap_type='Bernoulli',\n    subsample=0.8,\n    #scale_pos_weight=,\n    eval_metric='AUC',\n    od_type='Iter',\n    allow_writing_files=False,\n    random_seed=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cross validation scheme\n\ndef model_check(X, y, estimators, cv):\n    model_table = pd.DataFrame()\n\n    row_index = 0\n    for est, label in zip(estimators, labels):\n\n        MLA_name = label\n        model_table.loc[row_index, 'Model Name'] = MLA_name\n\n        cv_results = cross_validate(est,\n                                    X,\n                                    y,\n                                    cv=cv,\n                                    scoring='roc_auc',\n                                    return_train_score=True,\n                                    n_jobs=-1)\n\n        model_table.loc[row_index,\n                        'Train roc Mean'] = cv_results['train_score'].mean()\n        model_table.loc[row_index,\n                        'Test roc Mean'] = cv_results['test_score'].mean()\n        model_table.loc[row_index, 'Test Std'] = cv_results['test_score'].std()\n        model_table.loc[row_index, 'Time'] = cv_results['fit_time'].mean()\n\n        row_index += 1\n\n    model_table.sort_values(by=['Test roc Mean'],\n                            ascending=False,\n                            inplace=True)\n\n    return model_table\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nestimators = [ridge, lasso, elasticnet, gbr, xgboost, lightgbm, svr, hgrd, tweed,cb]\nlabels = [\n    'Ridge', 'Lasso', 'Elasticnet', 'GradientBoostingRegressor',\n    'XGBRegressor', 'LGBMRegressor', 'SVR', 'HistGradientBoostingRegressor','TweedieRegressor','CatBoostRegressor'\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Executing cross validation.\n\nraw_models = model_check(X_train, y_train, estimators, kf)\ndisplay(raw_models.style.background_gradient(cmap='summer_r'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\nfrom tqdm.keras import TqdmCallback\nfrom sklearn.linear_model import LogisticRegression","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_total = pd.DataFrame()\nestimators = [cb,xg]\nfor i in estimators:\n    i.fit(X_train, y_train)\n    predictions = i.predict_proba(X_train)[:, 1]\n    df = pd.DataFrame(data=predictions)\n    df_total = pd.concat([df_total, df], axis=1)\n    \n    \nestimators = [ridge, lasso, elasticnet, gbr, lightgbm, svr, hgrd, tweed]\n\n    \nfor i in tqdm(estimators):\n    i.fit(X_train, y_train)\n    predictions = i.predict(X_train)\n    \n#     s=(50078,2)\n#     s=np.ones(s)\n#     s[:,1]=predictions\n    \n    #clf = LogisticRegression(random_state=42).fit(s, y_train)\n    #predictions = clf.predict_proba(s)[:, 1]\n    df = pd.DataFrame(data=predictions)\n    df_total = pd.concat([df_total, df], axis=1)\n    \n    \n# predictions\n# df = pd.DataFrame(data=predictions)\n# df.to_csv('X_CatBoost_pred.csv', header=False, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_total","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dfff = df_total.copy()\ndfff.columns = [\"cb\",\"xg\",\"ridge\", \"lasso\", \"elasticnet\", \"gbr\", \"lightgbm\", \"svr\", \"hgrd\", \"tweed\"]\ndf_total = dfff","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xg_stack = xgb.XGBClassifier(\n    n_estimators=750,\n    min_child_weight=0.81,\n    learning_rate=0.025,\n    max_depth=2,\n    subsample=0.80,\n    colsample_bytree=0.42,\n    gamma=0.10,\n    random_state=42,\n    n_jobs=-1,\n)\nxg_stack.fit(df_total,y_train)\npredict_stack = xg_stack.predict_proba(df_total)[:, 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_stack_test = pd.DataFrame(data=predict_stack)\ndf_stack_test.to_csv('stack_pred.csv', header=False, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.DataFrame()\nestimators = [cb,xg]\nfor i in estimators:\n    predictions = i.predict_proba(test)[:, 1]\n    df = pd.DataFrame(data=predictions)\n    df_test = pd.concat([df_test, df], axis=1)\n    print(len(predictions))\n    \n    \nestimators = [ridge, lasso, elasticnet, gbr, lightgbm, svr, hgrd, tweed]\n    \nfor i in tqdm(estimators):\n    predictions = i.predict(X_train)\n    s=(50078,2)\n    s=np.ones(s)\n    s[:,1]=predictions\n    \n    clf = LogisticRegression(random_state=42).fit(s, y_train)\n    predictions = i.predict(test)\n    \n    s=(10982,2)\n    s=np.ones(s)\n    s[:,1]=predictions\n    \n    predictions = clf.predict_proba(s)[:, 1]\n    df = pd.DataFrame(data=predictions)\n    df_test = pd.concat([df_test, df], axis=1)\n    #print(len(predictions))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dfff_test = df_test.copy()\ndfff_test.columns = [\"cb\",\"xg\",\"ridge\", \"lasso\", \"elasticnet\", \"gbr\", \"lightgbm\", \"svr\", \"hgrd\", \"tweed\"]\ndf_test = dfff_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sorted(df_test[\"tweed\"])[-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.head(30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading recently created .csv files from working directory\n\neffnet = pd.read_csv('../input/blended-cnn/blended_effnets.csv')\nmeta = pd.read_csv('../input/stack-pred/stack_pred.csv',names=\"r\")\n\n\nsample['target'] = (\n                           \n                           effnet['target'] * 0.9 +\n                           meta['r'] * 0.025 +\n                           0.01*df_test['cb']+\n                           0.01*df_test['xg']+\n                           0.01*df_test['ridge']+\n                           0.01*df_test['lasso']+ \n                           0.01*df_test['elasticnet']+ \n                           0.01*df_test['gbr']+\n                           0.005*df_test['lightgbm']+\n                           0.005*df_test['hgrd']+\n                           0.005*df_test['tweed']\n                          )\n\n# final submissions\n\nsample.to_csv('ensembled.csv', header=True, index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}