{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os, json, cv2, math, re\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split, StratifiedKFold, KFold\nimport random\n\n#model imports (keras/tensorflow)\nimport tensorflow as tf\nimport keras\nfrom keras import layers, models\nfrom keras.optimizers import Adam\nfrom keras.layers import Conv2D, MaxPooling2D, BatchNormalization, GlobalAveragePooling2D\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\nimport tensorflow.keras.backend as K\nimport tensorflow_addons as tfa\n\nfrom kaggle_datasets import KaggleDatasets\nfrom functools import partial\n\nprint(\"Tensorflow version \" + tf.__version__)\n\nos.system('pip install /kaggle/input/kerasapplications -q')\nos.system('pip install /kaggle/input/efficientnet-keras-source-code/ -q --no-deps')\n\nimport efficientnet.tfkeras as efn","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Notes\n\n- If you want to learn more about Tensorflow + Computer Vision check out [@dimitreoliveira](https://www.kaggle.com/dimitreoliveira) I learned a ton of tips/tricks from his notebooks.\n- Baseline model trained using TPU's and resized 1200 x 1200 images stored as TFrecords.\n\n- Find a custom loss function that maximizes F1-score - [TF Add-ons](https://stackoverflow.com/questions/59496936/how-to-use-tensorflow-addons-metrics-correctly-in-functional-api), [Kaggle Notebook](https://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric)\n\n- Next up I need to edit the load_dataset function as I no longer need to perform augmentations in this notebook. (they are done during tfrecord creation)\n\n### Version's \n\n- V1: Baseline\n- V2: Changed Seed\n- V5: Leaderboard Rescore and using val_f1_score for CV (CV: 0.60391 -- LB: 0.630)\n- V7: Pre-Augment Data on Tfrec creation, Heavy Augs (CV: 0.70128 -- LB: 0.785)\n- V9: Efficientnet B4 five-fold (LB: 0.745)\n- V11: seed test (CV: 0.689)","metadata":{}},{"cell_type":"markdown","source":"### Seed\n\n- Setting seed for reproduciblity.","metadata":{}},{"cell_type":"code","source":"SEED = 1002\n\ndef seed_everything(seed):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    random.seed(seed)    \n\nseed_everything(SEED)  ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Detecting TPU's\n\n- Number of replicas will be 8 if the TPU's are correctly initialized, but will output 1 if not correctly connected. ","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Variables","metadata":{}},{"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH = KaggleDatasets().get_gcs_path('plant-pathology-2021-tfrecords-1200-x-1200')\n\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [512, 512]\nTARGET_SIZE = 512\nCLASSES = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11']\nNUM_OF_CLASSES = len(CLASSES)\nEPOCHS = 15\nDROPOUT_RATE = 0.2 \nAUG_BATCH = BATCH_SIZE","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Splitting TFRecords\n\nThe number on the end of each tfrecord file corresponds to the number of images in that tfrecord.\n\nExample: 'gs://kds-100c2bc3bab7e1f77f19378980a417f43e62119932994bd622dc7cb4/Id_train01-1427.tfrec' (1427 imgs)","metadata":{}},{"cell_type":"code","source":"#this function counts number of images in all TFRecords\ndef count_data_items(filenames):\n    n = [int(re.compile(r'-([0-9]*)\\.').search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nALL_TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/*.tfrec')\nNUM_ALL_TRAINING_IMAGES = count_data_items(ALL_TRAINING_FILENAMES)\n\n#reading train metadata\ntrain = pd.read_csv('../input/plant-pathology-2021-tfrecords-1200-x-1200/train.csv')\n\nprint(f'GCS: train images: {NUM_ALL_TRAINING_IMAGES}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train_test_split\n\nUsing this train_test_split to train baseline model. Once other model types and variables have been experimented with I plan to see if five-fold cross validation combination increases performance. \n\nNote: When I created the TFrecords, I split them into 50 files so that I could experiment with different sizes of Training and Validation datasets. All tfrecords are created using a stratified split (equal distribution of classes in each TFrecord).\n\nSomething to note here is that since I have pre-augmented the images it looks like I have very few images in validation compared to training set. Will have to experiment to see if this discrepency makes a difference.","metadata":{}},{"cell_type":"code","source":"def split_validation_set():\n    TRAINING_FILENAMES = []\n    VALIDATION_FILENAMES = []\n\n    for file_name in ALL_TRAINING_FILENAMES:\n        #using regex to get second last number in file\n        result = re.findall('[0-9]+', file_name)[-2]\n        if result[0] == \"0\":\n            result = result[1:]\n            \n        #checking if the filenumber is marked as a validation file\n        if int(result) in np.unique(train.loc[train.validation == 1].file.values).tolist():\n            VALIDATION_FILENAMES.append(file_name)\n        else:\n            TRAINING_FILENAMES.append(file_name)\n            \n    return TRAINING_FILENAMES,VALIDATION_FILENAMES\n            \n#assingning list of filenames to variable names\nTRAINING_FILENAMES,VALIDATION_FILENAMES = split_validation_set()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TRAINING_FILENAMES, VALIDATION_FILENAMES = train_test_split(\n#     ALL_TRAINING_FILENAMES,\n#     train_size= 0.90, test_size=0.10,\n#     random_state=SEED,\n# )\n\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\n\nprint(\"Training Images: {}  Validation Image: {}\".format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES))\nprint(\"Training Percent: {:.2f}  Validation Percent: {:.2f}\".format((NUM_TRAINING_IMAGES/NUM_ALL_TRAINING_IMAGES),\n                                                           (NUM_VALIDATION_IMAGES/NUM_ALL_TRAINING_IMAGES)))\n\nSTEPS_PER_EPOCH =  NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Functions\n\nThe following functions are how I am reading the data from the TF records. Casting each pixel value to a floating point and dividing them by 255 is a great way to increase training time.\n\nNOTE: Need to always be wary of order of color channels. RGB or BGR? \n\nRead more here --> [image-read-and-resize-with-opencv-tensorflow-and-pil](https://towardsdatascience.com/image-read-and-resize-with-opencv-tensorflow-and-pil-3e0f29b992be)","metadata":{}},{"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3) #decoding jpeg-encoded img to uint8 tensor\n    image = tf.cast(image, tf.float32) / 255.0 #cast int val to float so we can normalize pixels\n    image = tf.image.resize(image, [*IMAGE_SIZE]) #precautionary as all imgs should be 512x512\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) #resizing to split channels\n    \n    #CHECK IF THE IMAGES ARE RGB OR BGR\n    \n    return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_tfrecord(example, labeled=True):\n    \"\"\"\n        1. Parse data based on the 'TFREC_FORMAT' map.\n        2. Decode image.\n        3. If 'labeled' returns (image, label) if not (image, name).\n    \"\"\"\n    if labeled:\n        TFREC_FORMAT = {\n            'image': tf.io.FixedLenFeature([], tf.string), \n            'target': tf.io.FixedLenFeature([], tf.int64), \n        }\n    else:\n        TFREC_FORMAT = {\n            'image': tf.io.FixedLenFeature([], tf.string), \n            'image_name': tf.io.FixedLenFeature([], tf.string), \n        }\n    example = tf.io.parse_single_example(example, TFREC_FORMAT)\n    image = decode_image(example['image'])\n    if labeled:\n        label_or_name = tf.cast(example['target'], tf.int32)\n    else:\n        label_or_name =  example['image_name']\n    return image, label_or_name","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    \"\"\"\n        Create a Tensorflow dataset from TFRecords.\n    \"\"\"\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE)\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(lambda x: read_tfrecord(x, labeled=labeled), num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data augmentation\n\nI will experiment with many different augmentation types, but for a baseline I am going to use simple flips and rotations. \n\nMaybe some cutmix/mixup in later experiments. ","metadata":{}},{"cell_type":"code","source":"def simple_data_augmenter(image, label):\n    # Thanks to the dataset.prefetch(AUTO) statement in the following function this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part of the TPU while the TPU itself is computing gradients.\n    \n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32) #random int and rotating img based on result\n    \n    if p_rotate > .75:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .25:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n    \n    image = tf.image.random_flip_up_down(image)\n    image = tf.image.random_flip_left_right(image)\n    \n    \n    return image, label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Getting Dataset Functions","metadata":{}},{"cell_type":"code","source":"def get_training_dataset(dataset, do_aug=True, do_onehot=False):\n    #dataset = dataset.map(simple_data_augmenter, num_parallel_calls=AUTOTUNE)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.batch(AUG_BATCH)\n    if do_onehot: dataset = dataset.map(onehot, num_parallel_calls=AUTOTUNE) #onehot happens in do_aug as well \n    dataset = dataset.unbatch()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(dataset, do_onehot=True):\n    dataset = dataset.batch(BATCH_SIZE)\n    if do_onehot: dataset = dataset.map(onehot, num_parallel_calls=AUTOTUNE) # we must use one hot like augmented train data\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTOTUNE) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef onehot(image,label):\n    CLASSES = NUM_OF_CLASSES\n    return image,tf.one_hot(label,CLASSES)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Building Model\n\n- I am going to start by training some small EfficientNet models as these are relatively small but powerful models that often perform well in Computer Vision Problems.","metadata":{}},{"cell_type":"code","source":"#This simplecustom loss function has worked well for me in the past so I am going to start with this\ncatcross_loss = tf.keras.losses.CategoricalCrossentropy(from_logits=False, \n                                               label_smoothing=0.1, \n                                               name='categorical_crossentropy' ) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_model():\n    model = models.Sequential()\n    \n    model.add(efn.EfficientNetB5(include_top = False, weights = 'noisy-student', \n                              input_shape = (TARGET_SIZE, TARGET_SIZE, 3)))\n    \n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(DROPOUT_RATE))\n    model.add(layers.Dense(NUM_OF_CLASSES, activation = \"softmax\"))# 12 is the dimensionality of the output space \"12 classes\"\n\n    model.compile(optimizer = 'adam',\n                  loss = catcross_loss, #use sparse_catgeorical_crossentropy if not one_hot_encoding\n                  metrics = [\"acc\", tfa.metrics.F1Score(\n                    num_classes = NUM_OF_CLASSES, \n                    average = 'weighted')])\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Creating the model in strategy.scope() as I am training the model on TPU's","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    model = create_model()\n\nmodel.save('./EffNet_untrained_TPU_model.h5')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Callbacks\n\n- Using a custom learning rate scheduler that is a cosine decay w/ a warmup period. I used a custom LR scheduler because I wanted to be able to update on every step rather than every epoch. This is beneficial for the warmup period.","metadata":{}},{"cell_type":"code","source":"# LR_START = 0.000007\n# LR_MAX = 0.00007\n# LR_RAMPUP_EPOCHS = 3\n# WARMUP_STEPS = LR_RAMPUP_EPOCHS * (NUM_TRAINING_IMAGES//BATCH_SIZE)\n# TOTAL_STEPS = EPOCHS * (NUM_TRAINING_IMAGES//BATCH_SIZE)\n\n# def lrfn_step(step):\n#     if step < WARMUP_STEPS:\n#         lr = (LR_MAX - LR_START) / WARMUP_STEPS * step + LR_START\n#     else:\n#         progress = (step - WARMUP_STEPS) / (TOTAL_STEPS - WARMUP_STEPS)\n#         lr = LR_MAX * (0.5 * (1.0 + tf.math.cos(np.pi * ((1.0 * progress) % 1.0))))\n#     return lr\n\n\n\n# class CustomCallback(keras.callbacks.Callback):\n#     def __init__(self, schedule):\n#         super(CustomCallback, self).__init__()\n#         self.schedule = schedule\n#         self.epoch = 0\n        \n#     def on_train_batch_begin(self, batch, logs=None):\n#         actual_step = (self.epoch*STEPS_PER_EPOCH) + batch\n#         # Call schedule function to get the scheduled learning rate.\n#         scheduled_lr = self.schedule(actual_step)\n#         # Set the value back to the optimizer before this epoch starts\n#         tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_lr)\n#         if batch == 0:\n#             print(\"--Learning Rate: {:.6f} --\".format(scheduled_lr))\n        \n#     def on_epoch_end(self, epoch, logs=None):\n#         self.epoch+=1\n        \n    \n\n# #visualizing Learning Rate Schedule\n# rng = [i for i in range(TOTAL_STEPS)]\n# y = [lrfn_step(tf.cast(x, tf.float32)) for x in rng]\n\n# sns.set(style='whitegrid')\n# fig, ax = plt.subplots(figsize=(20, 6))\n# plt.plot(rng, y)\n\n# print(f'{TOTAL_STEPS} total steps and {NUM_TRAINING_IMAGES//BATCH_SIZE} steps per epoch')\n# print(f'Learning rate schedule: {y[0]:.3g} to {max(y):.3g} to {y[-1]:.3g}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Experimental Cosine Annealing LR Scheduler\n\nNote V14: this is specific to the following for a single_fold model trained on all data.\n\n- 9825 total steps \n- 1700 rampup steps\n- LR_MAX = 7\n\nNote V15: Specific to a five-fold model notebook\n- 1965 total steps \n- 348 rampup steps\n- LR_MAX = 7\n\n","metadata":{}},{"cell_type":"code","source":"TOTAL_STEPS = EPOCHS * (NUM_TRAINING_IMAGES//BATCH_SIZE)\nLR_RAMPUP_STEPS = 348\nLR_START = 1\nLR_MAX = 7\n\ndef lrfn_step(step):\n    if step <= LR_RAMPUP_STEPS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_STEPS * step + LR_START\n    else:\n        step = step/105\n        lr = LR_MAX + tf.math.cos(step - np.pi*(tf.math.floor(step/np.pi))) - tf.math.floor(step/np.pi)\n    return lr*1/100000\n\nclass CustomCallback(keras.callbacks.Callback):\n    def __init__(self, schedule):\n        super(CustomCallback, self).__init__()\n        self.schedule = schedule\n        self.epoch = 0\n        \n    def on_train_batch_begin(self, batch, logs=None):\n        actual_step = (self.epoch*STEPS_PER_EPOCH) + batch\n        # Call schedule function to get the scheduled learning rate.\n        scheduled_lr = self.schedule(actual_step)\n        # Set the value back to the optimizer before this epoch starts\n        tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_lr)\n        if batch == 0:\n            print(\"--Learning Rate: {:.6f} --\".format(scheduled_lr))\n        \n    def on_epoch_end(self, epoch, logs=None):\n        self.epoch+=1\n        \n#visualizing Learning Rate Schedule\n# rng = [i for i in range(TOTAL_STEPS)]\n# y = [lrfn_step(tf.cast(x, tf.float32)) for x in rng]\n\n# sns.set(style='whitegrid')\n# fig, ax = plt.subplots(figsize=(20, 6))\n# plt.plot(rng, y)\n\n# print(f'{TOTAL_STEPS} total steps and {NUM_TRAINING_IMAGES//BATCH_SIZE} steps per epoch')\n# print(f'Learning rate schedule: {y[0]:.3g} to {max(y):.3g} to {y[-1]:.3g}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Using a ModelCheckpoint callback that saves best_weights_only, this reduces time taken between each epoch as file to save is much smaller\n\n- Also created an early_stopping callback function that will stop the training cycle if there are no improvements in model on validation dataset in three consecutive epoch cycles.","metadata":{}},{"cell_type":"code","source":"model_save = ModelCheckpoint('./Effnet_TPU_Model_best_weights.h5', \n                             save_best_only = True, \n                             save_weights_only = True,\n                             monitor = 'val_f1_score', #note: set to val_f1_score\n                             mode = 'max',\n                             verbose = 1)\n\nmy_early_stopper = EarlyStopping(monitor = 'val_acc', min_delta = 0.001, \n                           patience = 6, mode = 'max', verbose = 1,\n                           restore_best_weights = False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training Model\n\n- NOTE: Setting do_onehot to \"True\" as we need to one_hot_encode the labels for the categorical crossentropy function that we are using.","metadata":{}},{"cell_type":"code","source":"# history = model.fit(x = get_training_dataset(load_dataset(TRAINING_FILENAMES), do_onehot=True),\n#                     epochs = EPOCHS,\n#                     steps_per_epoch = STEPS_PER_EPOCH,\n#                     validation_steps = VALID_STEPS,\n#                     validation_data = get_validation_dataset(load_dataset(VALIDATION_FILENAMES)),\n#                     callbacks = [CustomCallback(lrfn_step), model_save, my_early_stopper],\n#                     verbose = 1,\n#                    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualizing Model History\n\n- We can visualize the accuracy of the model on the training and validation datasets, as well as their loss over each epoch cycle.","metadata":{}},{"cell_type":"code","source":"# plt.figure(figsize=(13, 5))\n# plt.plot(history.history['loss'])\n# plt.plot(history.history['val_loss'])\n# plt.title(\"Model Loss\")\n# plt.xlabel('Epochs')\n# plt.ylabel('Loss')\n# plt.legend(['Train', 'Test'])\n# plt.ylim(ymax = 2, ymin = 0)\n# plt.grid()\n# plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plt.figure(figsize=(13, 5))\n# plt.plot(history.history['acc'])\n# plt.plot(history.history['val_acc'])\n# plt.title('Model Accuracy')\n# plt.xlabel('Epochs')\n# plt.ylabel('Accuracy')\n# plt.legend(['Train','Test'])\n# plt.grid()\n# plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ### Training a five_fold model\n \n Second option: Training a five-fold model.","metadata":{}},{"cell_type":"code","source":"TRAIN_ROUND = 0\nMODELS = []\n\nwith strategy.scope():\n    for val in range(0,5):\n\n        TRAINING_FILENAMES_SPLIT = []\n        VALIDATION_FILENAMES_SPLIT = []\n        \n        TRAINING_FILENAMES_SPLIT = TRAINING_FILENAMES[val:val+9]\n        #had to add extra square brackets as there is only one tfrec file\n        VALIDATION_FILENAMES_SPLIT = [VALIDATION_FILENAMES[val]]\n        \n        #the cosine annealing function is not super reproducable yet so...\n        NUM_VALIDATION_IMAGES_SPLIT = count_data_items(VALIDATION_FILENAMES_SPLIT)\n        NUM_TRAINING_IMAGES_SPLIT = count_data_items(TRAINING_FILENAMES_SPLIT)\n\n        STEPS_PER_EPOCH =  NUM_TRAINING_IMAGES_SPLIT // BATCH_SIZE\n        VALID_STEPS = NUM_VALIDATION_IMAGES_SPLIT // BATCH_SIZE\n        WARMUP_STEPS = LR_RAMPUP_STEPS * (STEPS_PER_EPOCH)\n        TOTAL_STEPS = EPOCHS * (STEPS_PER_EPOCH)\n        \n        #fitting each model fold\n        print(\"TRAINING MODEL: {}\".format(TRAIN_ROUND))\n        \n        MODELS.append(create_model())\n\n        MODELS[TRAIN_ROUND].fit(x=get_training_dataset(load_dataset(TRAINING_FILENAMES_SPLIT), do_onehot=True),\n                                    epochs = EPOCHS,\n                                    steps_per_epoch = STEPS_PER_EPOCH,\n                                    validation_steps = VALID_STEPS,\n                                    validation_data=get_validation_dataset(load_dataset(VALIDATION_FILENAMES_SPLIT)),\n                                    callbacks = [CustomCallback(lrfn_step), my_early_stopper],\n                                    verbose=1,\n                                   )\n        MODELS[TRAIN_ROUND].save_weights('Model_{}_best_weights.h5'.format(TRAIN_ROUND))\n        TRAIN_ROUND+=1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Feel free to comment below with any questions/concerns! Constructive criticism is welcomed!","metadata":{}}]}