{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":20270,"databundleVersionId":1222630,"sourceType":"competition"},{"sourceId":33105,"databundleVersionId":3519178,"sourceType":"competition"},{"sourceId":643971,"sourceType":"datasetVersion","datasetId":319080}],"dockerImageVersionId":30177,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\n\n\n## The overview of the buidling of metaclassfier:\n\n<div style=\"background-color: #086887; padding: 10px; border-left: 10px solid #083A87;\"> \n    The architecture of a meta-learning model consists of two primary components: the base models, often termed as Level-0 models, and the meta-model, known as the Level-1 model. Here’s a detailed breakdown:\n\n- **Level-0 Models (Base Models):** These are models trained on the dataset, generating predictions that are then used as inputs for the Level-1 model. Each of these models works independently on the training data to provide a unique set of predictions.\n\n- **Level-1 Model (Meta-Model):** This model is trained on the outputs of the Level-0 models, learning how to optimally combine their predictions to improve the final decision-making process.\n\nIt's important to highlight the potential risk of overfitting when the Level-0 models' predictions serve as inputs for the Level-1 model. Extreme caution is required to ensure that the meta-model does not merely memorize specific patterns from the base models' outputs but rather learns a generalized strategy for combining their predictions.\n\nThe meta-learning framework necessitates that each Level-0 model makes predictions on every single data fold, while learning from the remaining folds, ensuring comprehensive exposure to the dataset. This iterative process enables each base model to develop robust predictive capabilities that contribute to the effectiveness of the Level-1 model. Below is an illustration representing the structure of meta-learning, detailing how the Level-0 models interact with and inform the Level-1 model.\n    \n</div>\n\n ## Demonstration  \n\n<br>\n<img src= \"https://yanpuli.github.io/images/stacking.jpg\" alt =\"TPA MAY\" style='width: 800px;'>\n\n\n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Import Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\nfrom PIL import Image\nimport tensorflow as tf  # Ensure TensorFlow is imported\n\nfrom tensorflow.keras import layers, Sequential, Model\nfrom tensorflow.keras.layers import (\n    Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization,\n    Activation, MaxPooling2D, AveragePooling2D, GlobalAveragePooling2D,\n    GlobalMaxPooling2D, Input\n)\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.applications import VGG19, ResNet50\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\nimport warnings  # Ensure warnings is imported\n\n# Configurations and Initializations\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\ntf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)\nwarnings.filterwarnings('ignore')\nSEED = 123\nnp.random.seed(SEED)\n\n# GPU Configuration\ngpus = tf.config.experimental.list_physical_devices('GPU')\nif gpus:\n    try:\n        for gpu in gpus:\n            tf.config.experimental.set_memory_growth(gpu, True)\n    except RuntimeError as e:\n        print(\"RuntimeError in setting memory growth:\", e)\n\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-06T15:49:01.276163Z","iopub.execute_input":"2024-04-06T15:49:01.277697Z","iopub.status.idle":"2024-04-06T15:49:14.902765Z","shell.execute_reply.started":"2024-04-06T15:49:01.277632Z","shell.execute_reply":"2024-04-06T15:49:14.901925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pathlib\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport os\nimport PIL\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom tensorflow.keras.optimizers import Adam,SGD # - Works\nimport random\nfrom glob import glob\nimport seaborn as sns\nfrom tensorflow.keras.losses import SparseCategoricalCrossentropy\nimport matplotlib.pyplot as plt\nimport matplotlib.image as img\nimport warnings\nwarnings.filterwarnings('ignore')\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\ntf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)\n\nfrom keras.callbacks import ReduceLROnPlateau","metadata":{"execution":{"iopub.status.busy":"2024-06-17T08:31:34.816886Z","iopub.execute_input":"2024-06-17T08:31:34.817630Z","iopub.status.idle":"2024-06-17T08:31:40.890312Z","shell.execute_reply.started":"2024-06-17T08:31:34.817538Z","shell.execute_reply":"2024-06-17T08:31:40.889554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gpus = tf.config.experimental.list_physical_devices('GPU')\nprint(gpus)\ntry:\n    tf.config.experimental.set_memory_growth = True\nexcept Exception as ex:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:14.916198Z","iopub.execute_input":"2024-04-06T15:49:14.91648Z","iopub.status.idle":"2024-04-06T15:49:14.947352Z","shell.execute_reply.started":"2024-04-06T15:49:14.916441Z","shell.execute_reply":"2024-04-06T15:49:14.946443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare dataset\n","metadata":{}},{"cell_type":"code","source":"! rm -rf /kaggle/working/data/","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:14.984722Z","iopub.execute_input":"2024-04-06T15:49:14.985034Z","iopub.status.idle":"2024-04-06T15:49:16.089269Z","shell.execute_reply.started":"2024-04-06T15:49:14.984991Z","shell.execute_reply":"2024-04-06T15:49:16.088186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_dir_train = pathlib.Path(\"../input/skin-cancer9-classesisic/Skin cancer ISIC The International Skin Imaging Collaboration/Train/\")\ndata_dir_test = pathlib.Path(\"../input/skin-cancer9-classesisic/Skin cancer ISIC The International Skin Imaging Collaboration/Test/\")","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:16.090892Z","iopub.execute_input":"2024-04-06T15:49:16.091194Z","iopub.status.idle":"2024-04-06T15:49:16.096476Z","shell.execute_reply.started":"2024-04-06T15:49:16.09116Z","shell.execute_reply":"2024-04-06T15:49:16.095598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_count_train = len(list(data_dir_train.glob('*/*.jpg')))\nprint(image_count_train)\nimage_count_test = len(list(data_dir_test.glob('*/*.jpg')))\nprint(image_count_test)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:16.097943Z","iopub.execute_input":"2024-04-06T15:49:16.098365Z","iopub.status.idle":"2024-04-06T15:49:16.937062Z","shell.execute_reply.started":"2024-04-06T15:49:16.098318Z","shell.execute_reply":"2024-04-06T15:49:16.936156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 32\nimg_height = 180\nimg_width = 180\nrnd_seed = 123\nrandom.seed(rnd_seed)\n\ntrain_ds = tf.keras.preprocessing.image_dataset_from_directory(\n  data_dir_train,\n  validation_split=0.2,\n  subset=\"training\",\n  seed=123,\n  image_size=(img_height, img_width),\n  batch_size=batch_size)\n\nval_ds = tf.keras.preprocessing.image_dataset_from_directory(\n  data_dir_train,\n  validation_split=0.2,\n  subset=\"validation\",\n  seed=123,\n  image_size=(img_height, img_width),\n  batch_size=batch_size)\n\ntest_ds = tf.keras.preprocessing.image_dataset_from_directory(\n  data_dir_test,\n  validation_split=0.9,\n  subset=\"validation\",\n  seed=123,\n  image_size=(img_height, img_width),\n  batch_size=batch_size)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:16.938527Z","iopub.execute_input":"2024-04-06T15:49:16.939229Z","iopub.status.idle":"2024-04-06T15:49:22.497272Z","shell.execute_reply.started":"2024-04-06T15:49:16.939176Z","shell.execute_reply":"2024-04-06T15:49:22.496434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class_names = train_ds.class_names\nprint(class_names)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:22.500216Z","iopub.execute_input":"2024-04-06T15:49:22.500503Z","iopub.status.idle":"2024-04-06T15:49:22.50558Z","shell.execute_reply.started":"2024-04-06T15:49:22.500465Z","shell.execute_reply":"2024-04-06T15:49:22.504438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\ntrain_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)\nval_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:22.506864Z","iopub.execute_input":"2024-04-06T15:49:22.507107Z","iopub.status.idle":"2024-04-06T15:49:22.529996Z","shell.execute_reply.started":"2024-04-06T15:49:22.507076Z","shell.execute_reply":"2024-04-06T15:49:22.529154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_augmentation = keras.Sequential(\n  [\n    layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\", \n                                                 input_shape=(img_height, \n                                                              img_width,\n                                                              3)),\n    layers.experimental.preprocessing.RandomRotation(0.2),\n    layers.experimental.preprocessing.RandomZoom(0.2),\n  ]\n)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:22.531449Z","iopub.execute_input":"2024-04-06T15:49:22.531736Z","iopub.status.idle":"2024-04-06T15:49:22.747361Z","shell.execute_reply.started":"2024-04-06T15:49:22.53169Z","shell.execute_reply":"2024-04-06T15:49:22.746609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_classes = len(class_names)\ntotal = 0\nall_count = []\nclass_name = []\nfor i in range(num_classes):\n  count = len(list(data_dir_train.glob(class_names[i]+'/*.jpg')))\n  total += count\nprint(\"total training image count = {} \\n\".format(total))\nprint(\"-------------------------------------\")\nfor i in range(num_classes):\n  count = len(list(data_dir_train.glob(class_names[i]+'/*.jpg')))\n  print(\"Class name = \",class_names[i])\n  print(\"count      = \",count)\n  print(\"proportion = \",count/total)\n  print(\"-------------------------------------\")\n  all_count.append(count)\n  class_name.append(class_names[i])\n\ntemp_df = pd.DataFrame(list(zip(all_count, class_name)), columns = ['count', 'class_name'])\nsns.barplot(data=temp_df, y=\"count\", x=\"class_name\")\nplt.xticks(rotation=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:22.749129Z","iopub.execute_input":"2024-04-06T15:49:22.749468Z","iopub.status.idle":"2024-04-06T15:49:23.153807Z","shell.execute_reply.started":"2024-04-06T15:49:22.749423Z","shell.execute_reply":"2024-04-06T15:49:23.152963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Augmentor : Class balance","metadata":{}},{"cell_type":"code","source":"!pip install Augmentor","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:23.155328Z","iopub.execute_input":"2024-04-06T15:49:23.156048Z","iopub.status.idle":"2024-04-06T15:49:36.861727Z","shell.execute_reply.started":"2024-04-06T15:49:23.155996Z","shell.execute_reply":"2024-04-06T15:49:36.860856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_to_training_dataset = '../input/skin-cancer9-classesisic/Skin cancer ISIC The International Skin Imaging Collaboration/Train/'\nimport Augmentor\nfor i in class_names:\n    p = Augmentor.Pipeline(path_to_training_dataset + i, output_directory='/kaggle/working/data/'+i+'/output/')\n    p.rotate(probability=0.7, max_left_rotation=10, max_right_rotation=10)\n    p.sample(1000) ","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:49:36.863381Z","iopub.execute_input":"2024-04-06T15:49:36.863696Z","iopub.status.idle":"2024-04-06T15:55:40.07531Z","shell.execute_reply.started":"2024-04-06T15:49:36.863661Z","shell.execute_reply":"2024-04-06T15:55:40.074483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output_dir = pathlib.Path('/kaggle/working/data/')\nimage_count_train = len(list(output_dir.glob('*/output/*.jpg')))\nprint(image_count_train)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:55:40.076865Z","iopub.execute_input":"2024-04-06T15:55:40.077737Z","iopub.status.idle":"2024-04-06T15:55:40.341209Z","shell.execute_reply.started":"2024-04-06T15:55:40.077687Z","shell.execute_reply":"2024-04-06T15:55:40.340307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_classes = len(class_names)\ntotal = 0\nall_count = []\nclass_name = []\n\nfor i in range(num_classes):\n  count = len(list(output_dir.glob(class_names[i]+'/output/*.jpg')))\n  total += count\nprint(\"total training image count = {} \\n\".format(total))\nprint(\"-------------------------------------\")\nfor i in range(num_classes):\n  count = len(list(output_dir.glob(class_names[i]+'/output/*.jpg')))\n  print(\"Class name = \",class_names[i])\n  print(\"count      = \",count)\n  print(\"proportion = \",count/total)\n  print(\"-------------------------------------\")\n  all_count.append(count)\n  class_name.append(class_names[i])\n\n\ntemp_df = pd.DataFrame(list(zip(all_count, class_name)), columns = ['count', 'class_name'])\nsns.barplot(data=temp_df, y=\"count\", x=\"class_name\")\nplt.xticks(rotation=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:55:40.342244Z","iopub.execute_input":"2024-04-06T15:55:40.342464Z","iopub.status.idle":"2024-04-06T15:55:40.716138Z","shell.execute_reply.started":"2024-04-06T15:55:40.342437Z","shell.execute_reply":"2024-04-06T15:55:40.71531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = tf.keras.preprocessing.image_dataset_from_directory(\n  output_dir,\n  seed=123,\n  validation_split = 0.2,\n  subset = 'training',\n  image_size=(img_height, img_width),\n  batch_size=batch_size)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:55:40.717318Z","iopub.execute_input":"2024-04-06T15:55:40.717578Z","iopub.status.idle":"2024-04-06T15:55:41.236211Z","shell.execute_reply.started":"2024-04-06T15:55:40.717545Z","shell.execute_reply":"2024-04-06T15:55:41.235387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_ds = tf.keras.preprocessing.image_dataset_from_directory(\n  output_dir,\n  seed=123,\n  validation_split = 0.2,\n  subset = 'validation',\n  image_size=(img_height, img_width),\n  batch_size=batch_size)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:55:41.23736Z","iopub.execute_input":"2024-04-06T15:55:41.237592Z","iopub.status.idle":"2024-04-06T15:55:41.715477Z","shell.execute_reply.started":"2024-04-06T15:55:41.237561Z","shell.execute_reply":"2024-04-06T15:55:41.714666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling\n## 1. Custom model\n","metadata":{}},{"cell_type":"code","source":"num_classes = 9\nmodel = Sequential([layers.experimental.preprocessing.Rescaling(1.0/255,input_shape=(img_height,img_width,3))])\n\nmodel.add(Conv2D(32, 3,padding=\"same\",activation='relu'))\nmodel.add(MaxPool2D())\n\nmodel.add(Conv2D(64, 3,padding=\"same\",activation='relu'))\nmodel.add(MaxPool2D())\n\nmodel.add(Conv2D(128, 3,padding=\"same\",activation='relu'))\nmodel.add(MaxPool2D())\nmodel.add(Dropout(0.15))\n\nmodel.add(Conv2D(256, 3,padding=\"same\",activation='relu'))\nmodel.add(MaxPool2D())\nmodel.add(Dropout(0.20))\n\nmodel.add(Conv2D(512, 3,padding=\"same\",activation='relu'))\nmodel.add(MaxPool2D())\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(1024,activation=\"relu\"))\nmodel.add(Dense(units=num_classes, activation= 'softmax'))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-06T15:55:41.716777Z","iopub.execute_input":"2024-04-06T15:55:41.717065Z","iopub.status.idle":"2024-04-06T15:55:41.867043Z","shell.execute_reply.started":"2024-04-06T15:55:41.717023Z","shell.execute_reply":"2024-04-06T15:55:41.866338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\n\nopt = Adam(lr=0.001)\nmodel.compile(\n    optimizer='adam',\n    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),\n    metrics=['accuracy']\n)\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T15:55:41.868119Z","iopub.execute_input":"2024-04-06T15:55:41.868343Z","iopub.status.idle":"2024-04-06T15:55:41.893086Z","shell.execute_reply.started":"2024-04-06T15:55:41.868316Z","shell.execute_reply":"2024-04-06T15:55:41.892011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(\n  train_ds,\n  validation_data=val_ds,\n  epochs=35\n)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-06T15:55:41.894278Z","iopub.execute_input":"2024-04-06T15:55:41.894564Z","iopub.status.idle":"2024-04-06T16:05:40.728079Z","shell.execute_reply.started":"2024-04-06T15:55:41.894526Z","shell.execute_reply":"2024-04-06T16:05:40.72721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"acc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\n\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs_range = range(35)\n\nplt.figure(figsize=(8, 8))\nplt.subplot(1, 2, 1)\nplt.plot(epochs_range, acc, label='Training Accuracy')\nplt.plot(epochs_range, val_acc, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.title('Training and Validation Accuracy')\n\nplt.subplot(1, 2, 2)\nplt.plot(epochs_range, loss, label='Training Loss')\nplt.plot(epochs_range, val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.title('Training and Validation Loss')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T16:05:40.736577Z","iopub.execute_input":"2024-04-06T16:05:40.736876Z","iopub.status.idle":"2024-04-06T16:05:41.160068Z","shell.execute_reply.started":"2024-04-06T16:05:40.736842Z","shell.execute_reply":"2024-04-06T16:05:41.159146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. InceptionResNetV2 model","metadata":{}},{"cell_type":"markdown","source":"### Setup and Model Configuration\n- **Model Foundation**: Utilizes InceptionResNetV2, a powerful pre-trained model known for its high performance in image classification tasks, setting `include_top=False` to customize the top layers for the specific task.\n- **Data Augmentation**: Implements on-the-fly data augmentation (random flips and rotations) directly in the model pipeline, enhancing the training data's diversity and aiding in the model's generalization capabilities.\n\n### Model Assembly\n- **Sequential API**: Constructs the model layer by layer in a sequential manner, starting with data augmentation, followed by rescaling, the base InceptionResNetV2 model, global average pooling, dropout for regularization, and dense layers for final classification.\n- **Output Layer**: Ends with a dense layer having a softmax activation function to output probabilities across the nine classes.\n\n### Training Setup\n- **Optimizer**: Adopts `AdamW` from TensorFlow Addons, a variant of the Adam optimizer with weight decay, helping in regularizing the model and potentially improving its generalization.\n- **Compilation**: The model is compiled with the chosen optimizer, loss function (`SparseCategoricalCrossentropy`), and evaluation metric (`accuracy`).\n- **Initial Training**: Conducts a 15-epoch training phase, where the pre-trained layers are frozen to ensure that their learned features are not distorted by the new dataset initially.\n\n### Fine-tuning Phase\n- **Unfreeze Layers**: Post initial training, the base model layers are unfrozen to allow for fine-tuning, which adapts the pre-trained features to the new data more effectively.\n- **Selective Training**: Freezes the first 100 layers while leaving the rest trainable, which is a strategic choice to fine-tune only the more abstract layers of the network.\n- **Re-compilation**: The model is recompiled with a reduced learning rate to prevent large updates that could harm the learned features, thus ensuring a refined tuning of the model.\n\n### Execution and Outcomes\n- The training process involves two phases: initial training to stabilize the new top layers and fine-tuning to adjust the deep layers of the model to the specific task.\n- The approach leverages the strengths of transfer learning and fine-tuning, aiming to achieve high accuracy by adapting a powerful pre-trained network to a new task with relatively few data.\n","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers, Sequential\nfrom tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout\nfrom tensorflow.keras.losses import SparseCategoricalCrossentropy\nfrom tensorflow.keras.applications.inception_resnet_v2 import InceptionResNetV2\nimport tensorflow_addons as tfa\n\n# Define the number of classes\nnum_classes = 9\nimg_height = 180\nimg_width = 180\n\n# Load the InceptionResNetV2 model, pre-trained on ImageNet data\nbase_model = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(img_height, img_width, 3))\n\n# Freeze the layers of the base model\nfor layer in base_model.layers:\n    layer.trainable = False\n\n# Add data augmentation\ndata_augmentation = Sequential([\n    layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),\n    layers.experimental.preprocessing.RandomRotation(0.2),\n])\n\n# Create the model\nmodel2 = Sequential([\n    # Preprocessing layer\n    data_augmentation,\n    layers.experimental.preprocessing.Rescaling(1.0/255),\n    base_model,\n    GlobalAveragePooling2D(),\n    Dropout(0.3),\n    Dense(1024, activation='relu'),\n    Dense(num_classes, activation='softmax')\n])\n\nopt = tfa.optimizers.AdamW(learning_rate=1e-3, weight_decay=1e-4)\n\n# Compile the model\nmodel2.compile(optimizer=opt,\n               loss=SparseCategoricalCrossentropy(),\n               metrics=['accuracy'])\n\n# Train the model\nepochs = 15\nhistory2 = model2.fit(\n    train_ds,\n    validation_data=val_ds,\n    epochs=epochs\n)\n\n# Fine-tuning\n# Unfreeze the base model\nbase_model.trainable = True\n\n# You might want to experiment with which layers to freeze/unfreeze\nfor layer in base_model.layers[:100]:  # Fine-tune from this layer onwards\n    layer.trainable = False\n\n# Compile the model with a lower learning rate\nmodel2.compile(optimizer=tfa.optimizers.AdamW(learning_rate=1e-5, weight_decay=1e-4),\n               loss=SparseCategoricalCrossentropy(),\n               metrics=['accuracy'])\n\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-06T16:05:41.161728Z","iopub.execute_input":"2024-04-06T16:05:41.162352Z","iopub.status.idle":"2024-04-06T16:15:34.381962Z","shell.execute_reply.started":"2024-04-06T16:05:41.162301Z","shell.execute_reply":"2024-04-06T16:15:34.381073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Continue training for more epochs\nfine_tune_epochs = 35\ntotal_epochs = epochs + fine_tune_epochs\n\nhistory_fine = model2.fit(\n    train_ds,\n    validation_data=val_ds,\n    epochs=total_epochs,\n    initial_epoch=history2.epoch[-1]\n)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-06T16:15:34.383538Z","iopub.execute_input":"2024-04-06T16:15:34.384336Z","iopub.status.idle":"2024-04-06T17:09:10.986869Z","shell.execute_reply.started":"2024-04-06T16:15:34.384285Z","shell.execute_reply":"2024-04-06T17:09:10.986123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Correct the epochs_range to match the length of the history data\n\n\nacc = history_fine.history['accuracy']\nval_acc = history_fine.history['val_accuracy']\n\nloss = history_fine.history['loss']\nval_loss = history_fine.history['val_loss']\nepochs_range = range(36)\n\nplt.figure(figsize=(8, 8))\n\n# Plot training and validation accuracy\nplt.subplot(1, 2, 1)\nplt.plot(epochs_range, acc, label='Training Accuracy')\nplt.plot(epochs_range, val_acc, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.title('Training and Validation Accuracy')\n\n# Plot training and validation loss\nplt.subplot(1, 2, 2)\nplt.plot(epochs_range, loss, label='Training Loss')\nplt.plot(epochs_range, val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.title('Training and Validation Loss')\n\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:09:10.988238Z","iopub.execute_input":"2024-04-06T17:09:10.988498Z","iopub.status.idle":"2024-04-06T17:09:11.339417Z","shell.execute_reply.started":"2024-04-06T17:09:10.988465Z","shell.execute_reply":"2024-04-06T17:09:11.338668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. ResNet101V2","metadata":{}},{"cell_type":"markdown","source":"## Configs","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers, Sequential\nfrom tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout\nfrom tensorflow.keras.applications import ResNet101V2\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.losses import SparseCategoricalCrossentropy\n\n# Define the number of classes, image height and width\nnum_classes = 9\nimg_height = 180\nimg_width = 180\n\n# Load the ResNet152 model pre-trained on ImageNet data\nbase_model = ResNet101V2(weights='imagenet', include_top=False, input_shape=(img_height, img_width, 3))\n\n# Freeze the layers of the base model\nfor layer in base_model.layers:\n    layer.trainable = False\n\n# Enhanced data augmentation\ndata_augmentation = Sequential([\n    layers.RandomFlip(\"horizontal_and_vertical\"),\n    layers.RandomRotation(0.2),\n    layers.RandomZoom(0.2),\n    layers.RandomTranslation(height_factor=0.1, width_factor=0.1)\n])\n\n# Create the model\nmodel3 = Sequential([\n    data_augmentation,\n    layers.Rescaling(1.0/255),\n    base_model,\n    GlobalAveragePooling2D(),\n    Dropout(0.3),\n    Dense(1024, activation='relu'),\n    Dropout(0.3),\n    Dense(512, activation='relu'),\n    Dense(num_classes, activation='softmax')\n])\n\n# Compile the model with initial settings\nopt = tfa.optimizers.AdamW(learning_rate=1e-3, weight_decay=1e-4)\nmodel3.compile(optimizer=opt,\n               loss=SparseCategoricalCrossentropy(),\n               metrics=['accuracy'])\n\n# Train the model initially with frozen ResNet152 layers\nepochs = 15\nhistory3 = model3.fit(\n    train_ds,\n    validation_data=val_ds,\n    epochs=epochs\n)\n\n# Fine-tuning: Unfreeze some or all of the layers in the base model\nbase_model.trainable = True\n\n# It's often beneficial to fine-tune only the top layers of the base model initially\n# and gradually include more layers. Here, we choose to train the top 50 layers.\nfor layer in base_model.layers[:-50]:\n    layer.trainable = False\n\n# Re-compile the model for fine-tuning with a lower learning rate\nmodel3.compile(optimizer=tfa.optimizers.AdamW(learning_rate=1e-5, weight_decay=1e-4),\n               loss=SparseCategoricalCrossentropy(),\n               metrics=['accuracy'])\n\n# Continue training the model for fine-tuning\nfine_tune_epochs = 35\ntotal_epochs = epochs + fine_tune_epochs\n\nhistory_fine3 = model3.fit(\n    train_ds,\n    validation_data=val_ds,\n    epochs=total_epochs,\n    initial_epoch=history3.epoch[-1]\n)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:09:11.340597Z","iopub.execute_input":"2024-04-06T17:09:11.340833Z","iopub.status.idle":"2024-04-06T17:45:44.335914Z","shell.execute_reply.started":"2024-04-06T17:09:11.340803Z","shell.execute_reply":"2024-04-06T17:45:44.334866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndef plot_fine_tuning_history(history_fine):\n    # Extract the accuracy and loss for training and validation sets\n    acc = history_fine.history['accuracy']\n    val_acc = history_fine.history['val_accuracy']\n    loss = history_fine.history['loss']\n    val_loss = history_fine.history['val_loss']\n\n    epochs_range = range(len(acc))\n\n    plt.figure(figsize=(16, 8))\n\n    # Plot training and validation accuracy\n    plt.subplot(1, 2, 1)\n    plt.plot(epochs_range, acc, label='Training Accuracy')\n    plt.plot(epochs_range, val_acc, label='Validation Accuracy')\n    plt.xlabel('Epoch')\n    plt.ylabel('Accuracy')\n    plt.title('Training and Validation Accuracy')\n    plt.legend(loc='lower right')\n\n    # Plot training and validation loss\n    plt.subplot(1, 2, 2)\n    plt.plot(epochs_range, loss, label='Training Loss')\n    plt.plot(epochs_range, val_loss, label='Validation Loss')\n    plt.xlabel('Epoch')\n    plt.ylabel('Loss')\n    plt.title('Training and Validation Loss')\n    plt.legend(loc='upper right')\n\n    plt.show()\n\n# Use the function with the fine-tuning history object\nplot_fine_tuning_history(history_fine3)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:45:44.340127Z","iopub.execute_input":"2024-04-06T17:45:44.340347Z","iopub.status.idle":"2024-04-06T17:45:44.67139Z","shell.execute_reply.started":"2024-04-06T17:45:44.340321Z","shell.execute_reply":"2024-04-06T17:45:44.670697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. DenseNet121\nThis code was adapted from [this](https://www.kaggle.com/code/sghoang/skin-cancer-classification-densenet121-and-aug#Step-10-:-Model-Architecture) notebook.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications.resnet import preprocess_input as resnet_preprocess_input\nfrom tensorflow.keras.applications import DenseNet201\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout\n\n# DenseNet201\nmodel_denseNet= Sequential()\nmodel_denseNet.add(DenseNet201(include_top=False, weights='imagenet', input_shape=(img_height, img_width, 3)))\nmodel_denseNet.add(Flatten())\nmodel_denseNet.add(Dropout(0.5))  # Add a Dropout layer with a dropout rate of 0.5\nmodel_denseNet.add(Dense(1024, activation='relu'))\nmodel_denseNet.add(Dropout(0.5))  # Add a Dropout layer with a dropout rate of 0.5\nmodel_denseNet.add(Dense(512, activation='relu'))\nmodel_denseNet.add(Dense(num_classes, activation='softmax'))\n\nmodel_denseNet.summary()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile model\nopt = SGD(learning_rate=0.001, momentum=0.9)\nmodel_denseNet.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\n\n# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc',\n                                            patience=3,\n                                            verbose=1,\n                                            factor=0.5,\n                                            min_lr=0.00001)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"epochs = 50\nbatch_size = 32\nhistory_denseNet = model.fit(\n    train_ds,\n    validation_data=val_ds,\n    epochs=total_epochs,\n    callbacks=learning_rate_reduction\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"acc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\n\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs = range(len(acc))\n\nplt.figure(figsize=(16, 10))\nplt.subplot(1, 2, 1)\nplt.style.use('seaborn')\nplt.plot(epochs, acc, label='Train accuracy')\nplt.plot(epochs, val_acc, label='Val accuracy')\nplt.title('Training and validation accuracy')\nplt.legend()\n\nplt.subplot(1, 2, 2)\nplt.style.use('seaborn')\nplt.plot(epochs, loss, label='Train loss')\nplt.plot(epochs, val_loss, label='Val loss')\nplt.title('Training and validation loss')\nplt.legend()\nplt.show()\n\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Layer 0 Training Results","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Training Accuracy\nplt.figure(figsize=(12, 6))\nplt.plot(history.history['accuracy'], label='Custom Model', color='blue')\nplt.plot(history_fine.history['accuracy'], label='InceptionResNetV2', color='green')\nplt.plot(history_fine3.history['accuracy'], label='ResNet101V2', color='red')\nplt.plot(history_denseNet.history['accuracy'],label=\"DenseNet121\",color='black')\nplt.title('Training Accuracy of Models')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\nplt.legend()\nplt.tight_layout()\nplt.show()\n\n# Training Loss\nplt.figure(figsize=(12, 6))\nplt.plot(history.history['loss'], label='Custom Model', color='blue')\nplt.plot(history_fine.history['loss'], label='InceptionResNetV2', color='green')\nplt.plot(history_fine3.history['loss'], label='ResNet101V2', color='red')\nplt.plot(history_denseNet.history['loss'], label='DenseNet121', color='black')\nplt.title('Training Loss of Models')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.tight_layout()\nplt.show()\n\n# Validation Accuracy\nplt.figure(figsize=(12, 6))\nplt.plot(history.history['val_accuracy'], label='Custom Model', color='blue', linestyle='dashed')\nplt.plot(history_fine.history['val_accuracy'], label='InceptionResNetV2', color='green', linestyle='dashed')\nplt.plot(history_fine3.history['val_accuracy'], label='ResNet101V2', color='red', linestyle='dashed')\nplt.plot(history_denseNet.history['val_accuracy'], label='DenseNet121', color='black', linestyle='dashed')\nplt.title('Validation Accuracy of Models')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\nplt.legend()\nplt.tight_layout()\nplt.show()\n\n# Validation Loss\nplt.figure(figsize=(12, 6))\nplt.plot(history.history['val_loss'], label='Custom Model', color='blue', linestyle='dashed')\nplt.plot(history_fine.history['val_loss'], label='InceptionResNetV2', color='green', linestyle='dashed')\nplt.plot(history_fine3.history['val_loss'], label='ResNet101V2', color='red', linestyle='dashed')\nplt.plot(history_denseNet.history['val_loss'], label='DenseNet121', color='black', linestyle='dashed')\nplt.title('Validation Loss of Models')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:45:44.672511Z","iopub.execute_input":"2024-04-06T17:45:44.672758Z","iopub.status.idle":"2024-04-06T17:45:46.084107Z","shell.execute_reply.started":"2024-04-06T17:45:44.672725Z","shell.execute_reply":"2024-04-06T17:45:46.08336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Meta Learning\n## 5.1 Preprocessing for Meta learning\n\n","metadata":{}},{"cell_type":"code","source":"# Assume predict function returns the model's predictions for the given dataset\ndef train(model, dataset):\n    # Ensure the dataset is batched appropriately\n    predictions = model.predict(dataset)\n    return predictions\n\n# Generate predictions for each model\ntrain_model = train(model, train_ds)  # Replace 'model' with your actual model variable\ntrain_model2 = train(model2, train_ds)\ntrain_model3 = train(model3, train_ds)\ntrain_model_denseNet=train(model_denseNet,train_ds)\n\n# Create a DataFrame for the meta-learner's training data\nimport pandas as pd\n\nMetainput_train2 = pd.DataFrame({\n    'Custom_model_pred': train_model.ravel(),\n    'model2_pred': train_model2.ravel(),\n    'model3_pred': train_model3.ravel(),\n    'model_denseNet_pred': train_model_denseNet.ravel()\n})\n\n# Now Metainput_train2 contains the predictions from your three models\n# which will be used as features for training the meta-learner\n\nMetainput_train2.tail()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:45:46.085183Z","iopub.execute_input":"2024-04-06T17:45:46.085422Z","iopub.status.idle":"2024-04-06T17:46:55.913559Z","shell.execute_reply.started":"2024-04-06T17:45:46.085375Z","shell.execute_reply":"2024-04-06T17:46:55.912712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Assume predict function returns the model's predictions for the given dataset\ndef test(model, dataset):\n    # Ensure the dataset is batched appropriately\n    predictions = model.predict(dataset)\n    return predictions\n\n# Generate predictions for each model\ntest_model = test(model, test_ds)  # Replace 'model' with your actual model variable\ntest_model2 = test(model2, test_ds)\ntest_model3 = test(model3, test_ds)\ntest_model_denseNet=test(model_denseNet,test_ds)\n\n# Create a DataFrame for the meta-learner's training data\nimport pandas as pd\n\nMetainput_test2 = pd.DataFrame({\n    'Custom_model_pred': test_model.ravel(),\n    'model2_pred': test_model2.ravel(),\n    'model3_pred': test_model3.ravel(),\n    'model_denseNet_pred': test_model_denseNet.ravel()\n})\n\n# Now Metainput_train2 contains the predictions from your three models\n# which will be used as features for training the meta-learner\n\nMetainput_test2.tail()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:46:55.914797Z","iopub.execute_input":"2024-04-06T17:46:55.915048Z","iopub.status.idle":"2024-04-06T17:47:08.687031Z","shell.execute_reply.started":"2024-04-06T17:46:55.915018Z","shell.execute_reply":"2024-04-06T17:47:08.686273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2  Meta-learning classifier- LogisticRegression\n","metadata":{}},{"cell_type":"code","source":"# Assuming you use the predictions as both features and pseudo-labels\nimport numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import classification_report, accuracy_score\n\n# Generate pseudo-labels based on the highest prediction score across models\npseudo_labels = np.argmax(Metainput_train2.values, axis=1)\n\n# Initialize the meta-learner model\nmeta_learner = LogisticRegression()\n\n# Train the meta-learner using the predictions as features and the pseudo-labels as targets\nmeta_learner.fit(Metainput_train2, pseudo_labels)\n\n# Evaluate the meta-learner on some metric, perhaps through cross-validation or on a separate validation set\n\ntest_pseudo_labels = np.argmax(Metainput_test2.values, axis=1)\n\n# Evaluate the meta-learner on the test data\ntest_predictions = meta_learner.predict(Metainput_test2)\n\n# Calculate accuracy or other metrics to evaluate performance\naccuracy = accuracy_score(test_pseudo_labels, test_predictions)\nprint(f\"Meta-learner accuracy on test data: {accuracy}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:08.688302Z","iopub.execute_input":"2024-04-06T17:47:08.688595Z","iopub.status.idle":"2024-04-06T17:47:11.059119Z","shell.execute_reply.started":"2024-04-06T17:47:08.688556Z","shell.execute_reply":"2024-04-06T17:47:11.05806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3  Meta-learning classifier- RandomForestClassifier","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\n\n# Initialize the RandomForestClassifier\nrandom_forest_model = RandomForestClassifier(n_estimators=100, max_depth=None, random_state=42)\n\n# Train the RandomForestClassifier\nrandom_forest_model.fit(Metainput_train2, pseudo_labels)\n\n# Make predictions on the test set\ntest_predictions = random_forest_model.predict(Metainput_test2)\n\ntest_labels = np.argmax(Metainput_test2.values, axis=1)\n\n\n# Evaluate the model's performance\ntest_accuracy = accuracy_score(test_labels, test_predictions)\nprint(f\"Test accuracy: {test_accuracy}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:11.061331Z","iopub.execute_input":"2024-04-06T17:47:11.062091Z","iopub.status.idle":"2024-04-06T17:47:16.952246Z","shell.execute_reply.started":"2024-04-06T17:47:11.062032Z","shell.execute_reply":"2024-04-06T17:47:16.951474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.4  Meta-learning classifier- DecisionTreeClassifier","metadata":{}},{"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\nfrom sklearn.metrics import accuracy_score\nimport numpy as np\n\n# Initialize the Decision Tree Classifier\ndecision_tree_model = DecisionTreeClassifier(random_state=42)\n\n# Train the model\ndecision_tree_model.fit(Metainput_train2, pseudo_labels)\n\n# Generate predictions on the test set\ndt_test_predictions = decision_tree_model.predict(Metainput_test2)\n\n# Assuming you have pseudo labels for your test set\ntest_pseudo_labels = np.argmax(Metainput_test2.values, axis=1)\n\n# Evaluate the model's performance\ndt_test_accuracy = accuracy_score(test_pseudo_labels, dt_test_predictions)\nprint(f\"Decision Tree test accuracy: {dt_test_accuracy}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:16.953443Z","iopub.execute_input":"2024-04-06T17:47:16.953692Z","iopub.status.idle":"2024-04-06T17:47:17.151092Z","shell.execute_reply.started":"2024-04-06T17:47:16.953661Z","shell.execute_reply":"2024-04-06T17:47:17.150282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.5 Meta-learning classifier- GradientBoostingClassifier","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import GradientBoostingClassifier\n\ngbm_model = GradientBoostingClassifier(random_state=42)\ngbm_model.fit(Metainput_train2, pseudo_labels)\ngbm_predictions = gbm_model.predict(Metainput_test2)\ngbm_accuracy = accuracy_score(test_pseudo_labels, gbm_predictions)\nprint(f\"GBM test accuracy: {gbm_accuracy}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:17.15219Z","iopub.execute_input":"2024-04-06T17:47:17.152411Z","iopub.status.idle":"2024-04-06T17:47:40.569601Z","shell.execute_reply.started":"2024-04-06T17:47:17.152368Z","shell.execute_reply":"2024-04-06T17:47:40.568803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.6 Meta-learning classifier- XGBClassifier ","metadata":{}},{"cell_type":"code","source":"from xgboost import XGBClassifier\n\nxgb_model = XGBClassifier(random_state=42)\nxgb_model.fit(Metainput_train2, pseudo_labels)\nxgb_predictions = xgb_model.predict(Metainput_test2)\nxgb_accuracy = accuracy_score(test_pseudo_labels, xgb_predictions)\nprint(f\"XGBoost test accuracy: {xgb_accuracy}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:40.570916Z","iopub.execute_input":"2024-04-06T17:47:40.57121Z","iopub.status.idle":"2024-04-06T17:47:54.711309Z","shell.execute_reply.started":"2024-04-06T17:47:40.571165Z","shell.execute_reply":"2024-04-06T17:47:54.710546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"12\"></a><h2></h2>\n<div style=\"color:white;display:fill;border-radius:8px;\n            background-color:#DAA520;font-size:150%;\n            font-family:Nexa;letter-spacing:0.5px\">\n    <p style=\"padding: 8px;color:black;\"><b>5.4 |  Meta-learning classifier- DecisionTreeClassifier","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Model names\nmodels = ['Logistic Regression', 'Random Forest', 'Decision Tree', 'Gradient Boosting', 'XGBoost']\n\n# Corresponding accuracies\naccuracies = [accuracy * 100, test_accuracy* 100, dt_test_accuracy* 100, gbm_accuracy* 100, xgb_accuracy* 100]\n\n# Colors for each bar\ncolors = ['blue', 'green', 'red', 'purple', 'orange']\n\nplt.figure(figsize=(10, 6))\n\n# Create bar chart\nplt.bar(models, accuracies, color=colors)\n\nplt.title('Model Performance Comparison')\nplt.xlabel('Models')\nplt.ylabel('Accuracy')\nplt.xticks(rotation=45)\nplt.ylim(0, 100)  # Assuming accuracy is between 0 and 1\n\n# Display the plot\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:54.712694Z","iopub.execute_input":"2024-04-06T17:47:54.713022Z","iopub.status.idle":"2024-04-06T17:47:54.877381Z","shell.execute_reply.started":"2024-04-06T17:47:54.712983Z","shell.execute_reply":"2024-04-06T17:47:54.87655Z"},"trusted":true},"execution_count":null,"outputs":[]}]}