{"cells":[{"metadata":{},"cell_type":"markdown","source":"# SIIM-ISIC Melanoma Classification","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Introduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Melanoma is a deadly disease, but if caught early, most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will improve dermatologists' diagnostic accuracy. Better detection of melanoma has the opportunity to positively impact millions of people.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This competition provides both image data (in DIOCOM, JPEG and TFRecord formats) and tabular data about each sample.\n\nIn this notebook, we are going to build two models that utilize both image and tabular data e ensemble the results. This is a starter code and if you have any suggestions or questions, please let me know in the comments.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Evaluation Metrics \n\nThe metric in evaluation for this competition is AUC, that for [Area Under the ROC Curve](http://developers.google.com/machine-learning/crash-course/classification/roc-and-auc).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Data Exploration","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Loading packages","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport missingno as msno\nimport pydicom as dcm\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n\nplt.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Loading the tabular data","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"PATH = \"/kaggle/input/siim-isic-melanoma-classification/\"\n\ntrain_df = pd.read_csv(os.path.join(PATH, 'train.csv'))\ntest_df = pd.read_csv(os.path.join(PATH, 'test.csv'))\nsample_submission_df = pd.read_csv(os.path.join(PATH, 'sample_submission.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Train shape: {train_df.shape}\")\nprint(f\"Test shape: {test_df.shape}\")\nprint(f\"Sample submission shape: {sample_submission_df.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Change columns names\nnew_names = ['image_name', 'ID', 'sex', 'age', 'anatomy', 'diagnosis', 'benign_malignant', 'target']\ntrain_df.columns = new_names\ntest_df.columns = new_names[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Missing data","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Visualizing the missing values\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\nmsno.matrix(train_df, ax=ax1, fontsize=10)\nmsno.matrix(test_df, ax=ax2, fontsize=10)\n\nax1.set_title('Train Missing Values Map', fontsize=15)\nax2.set_title('Test Missing Values Map', fontsize=15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df.shape)\ntrain_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(test_df.shape)\ntest_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Patients Distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Unique IDs\nprint(f\"The total patient IDs are {train_df.ID.count()}, from those the unique IDs are {train_df.ID.value_counts().shape[0]}.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of images by ID\npatients_train = train_df.groupby('ID')['image_name'].count().reset_index()\npatients_test = test_df.groupby('ID')['image_name'].count().reset_index()\n\nf, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))\nsns.distplot(patients_train.image_name, kde=False, bins=50, ax=ax1)\nsns.distplot(patients_test.image_name, kde=False, bins=50, ax=ax2)\n\nax1.set_title('Images by Patient Distribution - Train')\nax2.set_title('Images by Patient Distribution - Test')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"'Sex' Variable","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Diagnosis and target\nplt.figure(figsize=(8, 5))\nsns.countplot(data=train_df, x='sex')\nplt.title('Gender Distribuition')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Filling sex with the mode.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sex_mode = train_df.sex.mode()[0]\ntrain_df.sex.fillna(sex_mode, inplace=True)\nprint('Mode:', sex_mode)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 'Age' Feature","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Age Variable\nplt.figure(figsize=(10, 5))\nsns.distplot(train_df.age)\nplt.title('Age Distribuition')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.age.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filling the age variable with the median: 50 years.\nage_median = train_df.age.median()\ntrain_df.age.fillna(age_median, inplace=True)\nprint('Median:', age_median)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Age distribution and sex\nplt.figure(figsize=(15, 5))\nsns.distplot(train_df[train_df.sex == 'male']['age'])\nsns.distplot(train_df[train_df.sex == 'female']['age'])\nplt.title('Age Distribution by Gender')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 'Anatomy' Feature","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.countplot(data=train_df, x='anatomy')\nplt.title('Scanned Body Parts')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.anatomy.value_counts(dropna=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking the 'benign_malignant' variable of NaN values in 'anatomy'\ntrain_df[train_df.anatomy.isnull()]['benign_malignant'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the most frequent 'benign_malignant' is benign in torso, we can set the NaN Values in 'anatomy' as torso.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.anatomy.fillna('torso', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking the test data\ntest_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Age variable of NaN values in 'anatomy' variable \ntest_df[test_df.anatomy.isnull()]['age'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The majority of the people with missing anatomy have 70 yo, so we'll use the anatomy with the biggest frequency for age 70.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"age_nan = test_df[test_df['age'] == 70]['anatomy'].value_counts().reset_index()['index'][0]\ntest_df.anatomy.fillna(age_nan, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Genders by Anatomy\nplt.figure(figsize=(15, 5))\nsns.countplot(data=train_df, x='anatomy', hue='sex')\nplt.title('Anatomy Distribution by Genders')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 'Diagnosis'Feature\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.countplot(data=train_df, x='diagnosis')\nplt.title('Diagnosis Distribuiton')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Diagnosis and target\nplt.figure(figsize=(15, 5))\nsns.countplot(data=train_df, x='benign_malignant', hue='diagnosis')\nplt.title('Diagnosis Distribution by Target')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Diagnosis and sex\nplt.figure(figsize=(15, 5))\nsns.countplot(data=train_df, x='diagnosis', hue='sex')\nplt.title('Diagnosis Distribution by Genders')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Target","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Target\ntrain_df.target.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(data=train_df, x='benign_malignant')\nplt.title('Target Distribuition')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Genders by target\nplt.figure(figsize=(10, 5))\nsns.countplot(data=train_df, x='benign_malignant', hue='sex')\nplt.title('Genders Distribuition by Target')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Anatomy by target\nplt.figure(figsize=(15, 5))\nsns.countplot(data=train_df, x='anatomy', hue='benign_malignant')\nplt.title('Anatomy Distribuition by Target')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image Data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First, we are going to show the DICOM images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### DICOM Images","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_dicom_images(data):\n    img_data = list(data.T.to_dict().values())\n    f, ax = plt.subplots(3,3, figsize=(16,18))\n    for i,data_row in enumerate(img_data):\n        patientImage = data_row['image_name']+'.dcm'\n        imagePath = os.path.join(PATH,\"train/\",patientImage)\n        data_row_img_data = dcm.read_file(imagePath)\n        modality = data_row_img_data.Modality\n        age = data_row_img_data.PatientAge\n        sex = data_row_img_data.PatientSex\n        data_row_img = dcm.dcmread(imagePath)\n        ax[i//3, i%3].imshow(data_row_img.pixel_array, cmap=plt.cm.gray) \n        ax[i//3, i%3].axis('off')\n        ax[i//3, i%3].set_title(f\"ID: {data_row['image_name']}\\nAge: {age} Sex: {sex}\\nDiagnosis: {data_row['diagnosis']}\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Selecting images with target=0 (benign).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_dicom_images(train_df[train_df.target == 0].sample(9))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Selecting images with target=1 (malignant).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_dicom_images(train_df[train_df.target == 1].sample(9))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Extracting data from DICOM images","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def extract_DICOM_attributes(folder):\n    images = list(os.listdir(os.path.join(PATH, folder)))\n    df = pd.DataFrame()\n    for image in images:\n        image_name = image.split(\".\")[0]\n        dicom_file_path = os.path.join(PATH,folder,image)\n        dicom_file_dataset = dcm.read_file(dicom_file_path)\n        study_date = dicom_file_dataset.StudyDate\n        modality = dicom_file_dataset.Modality\n        age = dicom_file_dataset.PatientAge\n        sex = dicom_file_dataset.PatientSex\n        body_part_examined = dicom_file_dataset.BodyPartExamined\n        patient_orientation = dicom_file_dataset.PatientOrientation\n        photometric_interpretation = dicom_file_dataset.PhotometricInterpretation\n        rows = dicom_file_dataset.Rows\n        columns = dicom_file_dataset.Columns\n\n        df = df.append(pd.DataFrame({'image_name': image_name, \n                        'dcm_modality': modality,'dcm_study_date':study_date, 'dcm_age': age, 'dcm_sex': sex,\n                        'dcm_body_part_examined': body_part_examined,'dcm_patient_orientation': patient_orientation,\n                        'dcm_photometric_interpretation': photometric_interpretation,\n                        'dcm_rows': rows, 'dcm_columns': columns}, index=[0]))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = extract_DICOM_attributes('train')\ntrain_dicom_df = train_df.merge(df_train, on='image_name')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dicom_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dicom_df.to_csv('train_dicom_df.csv', header=True, index=False) # todo","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# todo","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### JPEG Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Getting the images attributes. \nThe code bellow its commented and imported it as a data becasue it's time consuming process.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#train_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train/'\n#test_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/test/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#from tqdm import tqdm\n#from tqdm.keras import TqdmCallback\n#from keras.preprocessing import image\n#\n#for data, location in zip([train_df, test_df],[train_img_path, test_img_path]):\n#    images = data['image_name'].values\n#    reds = np.zeros(images.shape[0])\n#    greens = np.zeros(images.shape[0])\n#    blues = np.zeros(images.shape[0])\n#    mean = np.zeros(images.shape[0])\n#    x = np.zeros(images.shape[0], dtype=int)\n#    y = np.zeros(images.shape[0], dtype=int)\n#    for i, path in enumerate(tqdm(images)):\n#        img = np.array(image.load_img(os.path.join(location, f'{path}.jpg')))\n#\n#        reds[i] = np.mean(img[:,:,0].ravel())\n#        greens[i] = np.mean(img[:,:,1].ravel())\n#        blues[i] = np.mean(img[:,:,2].ravel())\n#        mean[i] = np.mean(img)\n#        x[i] = img.shape[1]\n#        y[i] = img.shape[0]\n#\n#    data['reds'] = reds\n#    data['greens'] = greens\n#    data['blues'] = blues\n#    data['mean_colors'] = mean\n#    data['width'] = x\n#    data['height'] = y\n#\n#train_df['total_pixels']= train_df['width']*train_df['height']\n#test_df['total_pixels']= test_df['width']*test_df['height']\n#train_df['res'] = train_df['width'].astype(str) + 'x' + train_df['height'].astype(str)\n#test_df['res'] = test_df['width'].astype(str) + 'x' + test_df['height'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Save the files\n#train_df.to_csv('train_atr.csv', index=False)\n#test_df.to_csv('test_atr.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_IMAGES = \"/kaggle/input/imagesatr\"\n\ntrain_df = pd.read_csv(os.path.join(PATH_IMAGES, 'train_atr.csv'))\ntest_df = pd.read_csv(os.path.join(PATH_IMAGES, 'test_atr.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# todo","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building Our Models","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Modelling Based on Tabular data - XGBoost","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading sample submission data\nPATH = \"/kaggle/input/siim-isic-melanoma-classification/\"\n\nsample_submission_df = pd.read_csv(os.path.join(PATH, 'sample_submission.csv'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One Hot Encoding for categorical features: sex and anatomy.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting dummy variables for gender on train set\nsex_dummies = pd.get_dummies(train_df.sex, prefix='sex')\ntrain_df = pd.concat([train_df, sex_dummies], axis=1)\n\n# Now, on test set\nsex_dummies = pd.get_dummies(test_df.sex, prefix='sex')\ntest_df = pd.concat([test_df, sex_dummies], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting dummy variables for anatomy on train set\nanatomy_dummies = pd.get_dummies(train_df.anatomy, prefix='anatomy')\ntrain_df = pd.concat([train_df, anatomy_dummies], axis=1)\n\n# Now, on test set\nanatomy_dummies = pd.get_dummies(test_df.anatomy, prefix='anatomy')\ntest_df = pd.concat([test_df, anatomy_dummies], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Removing white space \ntrain_df.columns = train_df.columns.str.replace(' ', '_')\ntrain_df.columns = train_df.columns.str.replace('/', '_')\n\ntest_df.columns = test_df.columns.str.replace(' ', '_')\ntest_df.columns = test_df.columns.str.replace('/', '_')\n\n# Dropping not usefull columns\ntrain_df.drop(['image_name', 'ID','sex', 'anatomy', 'diagnosis', 'benign_malignant', 'res'], axis=1, inplace=True)\ntest_df.drop(['image_name', 'ID','sex', 'anatomy', 'res'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost as xgb\n\nfrom sklearn.model_selection import StratifiedKFold, train_test_split, cross_val_score, cross_validate\nfrom sklearn.metrics import roc_auc_score\n\nX = train_df.drop(['target'], axis=1)\ny = train_df.target","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Spliting data\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Fit model on all training data\nmodel = xgb.XGBClassifier()\nmodel.fit(X_train, y_train)\n\nvalidation = model.predict_proba(X_test)[:, 1]\n\nroc_auc_score(y_test, validation)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ROC_AUC score: 0.834315","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\n# plotando a curva ROC\ny_pred_proba = model.predict_proba(X_test)[::,1]\nfpr, tpr, _ = metrics.roc_curve(y_test,  y_pred_proba)\nauc = metrics.roc_auc_score(y_test, y_pred_proba)\nplt.plot(fpr,tpr,label=\"data 1, auc=\"+str(auc))\nplt.plot([0, 1], [0, 1], color='red', lw=2, linestyle='--')\nplt.legend(loc=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from xgboost import plot_importance\nmodel = xgb.XGBClassifier()\nmodel.fit(X, y)\n# plot feature importance\nplot_importance(model)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Prediction 1 - XGBoost for the sample submission based on tabular data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prediction 1\npredictions = model.predict_proba(test_df)\nmetadata_df = pd.DataFrame(columns=['image_name', 'target'])\n\nmetadata_df['image_name'] = sample_submission_df['image_name']\nmetadata_df['target'] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making the prediction\nmodel.fit(X_train, y_train)\npredictions = model.predict_proba(test_df)[:, 1]\n\nmetadata_df = pd.DataFrame(columns=['image_name', 'target'])\nmetadata_df['image_name'] = sample_submission_df['image_name']\nmetadata_df['target'] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating submission csv file\nmetadata_df.to_csv('submission_tabular.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Modelling Based on Image Data - EfficientNetB3","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Using neural networks for prediction based on image data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport re\n\nimport numpy as np\nimport pandas as pd\nimport math\n\nfrom matplotlib import pyplot as plt\n\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\n\nimport efficientnet.tfkeras as efn\n\nfrom kaggle_datasets import KaggleDatasets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting TPU as main device for training\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# For tf.dataset\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\n\n# Configuration\nDEBUG = False\nN_FOLD = 4\nEPOCHS = 1 if DEBUG else 7\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [1024, 1024]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\ntest_files = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['image_name']\n    return image, idnum\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    return dataset\n\ndef get_test_dataset(test_files, ordered=False):\n    dataset = load_dataset(test_files, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Prediction 2 - Modelling image data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model():\n    \n    with strategy.scope():\n        model = tf.keras.Sequential([\n            efn.EfficientNetB3(\n                input_shape=(*IMAGE_SIZE, 3),\n                weights=None,\n                include_top=False\n            ),\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Inference\nfrom tqdm import tqdm\n\npred_df = pd.DataFrame()\n\ntk0 = tqdm(range(N_FOLD), total=N_FOLD)\n\nfor fold in tk0:\n    num_test = count_data_items(test_files)\n    test_ds = get_test_dataset(test_files, ordered=True)\n    test_images_ds = test_ds.map(lambda image, idnum: image)\n    model = get_model()\n    probabilities = model.predict(test_images_ds)\n    test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(num_test))).numpy().astype('U')\n    _pred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\n    pred_df = pd.concat([pred_df, _pred_df])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_pred_df = pred_df.groupby('image_name', as_index=False).mean()\nmean_pred_df.columns = ['image_name', 'target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_pred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating submission csv file\nmean_pred_df.to_csv('submission_image.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Ensemble","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Final step. We'll use our blended predictions created by training images and simply metadata created by using tabular data. We ensemble them together with weights and make our final predictions. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_sub = pd.read_csv('/kaggle/working/submission_image.csv')\ntabular_sub = pd.read_csv('/kaggle/working/submission_tabular.csv')\ntabular_sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = image_sub.copy()\nsubmission.target = 0.9 * image_sub.target.values + 0.1 * tabular_sub.target.values\nsubmission.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is not the final version. \n\nI would like to thank the community who helped me complete this challenge, especially to the competitors below:\n\n**References**\n\nhttps://www.kaggle.com/gpreda/siim-isic-melanoma-classification-eda\n\nhttps://www.kaggle.com/andradaolteanu/siim-melanoma-competition-eda-augmentations\n\nhttps://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble/data\n\nhttps://www.kaggle.com/parulpandey/melanoma-classification-eda-starter\n\nhttps://www.kaggle.com/forwet/tensorflow-transfer-learning-melanoma\n\nhttps://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma\n\nhttps://www.kaggle.com/cdeotte/image-and-tabular-data-0-915\n\nhttps://www.kaggle.com/kittlein/xgboost-tabular-data-ml-cv-86-lb-787\n\nhttps://www.kaggle.com/zainahmad/eda-melanoma-classification-using-tensorflow\n\nhttps://www.kaggle.com/redwankarimsony/melanoma-eda-efficentnets-densenet-ensemble#4.-TPU-Setup-Code\n\nhttps://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\nhttps://www.kaggle.com/tunguz/melanoma-classification-eda-and-modeling\n\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}