{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nThis kernel is designed to work as the training environment for my final degree project, where I intend to create a Deep Learning model able to diagnose illnesses on vegetables, specifically working with the [plant pathology 2021 dataset](https://www.kaggle.com/c/plant-pathology-2021-fgvc8), and therefore can also be seen as one more attempt on the overall competition.\n\nFor this reason, the code present here will be only responsible for training and testing the models, as these tasks require heavy computations capabilities for which TPUs are needed. The models will then be saved into .h5 files.\n\nFor the rest of the tasks such as data analysis, dataset division and results analysis, they all will be performed locally. All the code related to the utilities functions that implement these parts can be found at github.\n\nCheck out the project at my [github repo](https://github.com/gfelis/TFG).","metadata":{}},{"cell_type":"markdown","source":"### EfficientNet requires special installation","metadata":{}},{"cell_type":"code","source":"!pip install -q efficientnet","metadata":{"execution":{"iopub.status.busy":"2021-08-04T15:20:40.806604Z","iopub.execute_input":"2021-08-04T15:20:40.807022Z","iopub.status.idle":"2021-08-04T15:20:50.450882Z","shell.execute_reply.started":"2021-08-04T15:20:40.806979Z","shell.execute_reply":"2021-08-04T15:20:50.449716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Python built-in libraries\nimport random\nimport os\n\n# Third party libraries\nimport numpy as np\nimport cv2\nimport pandas as pd\nimport tensorflow as tf\nimport warnings\nfrom keras.callbacks import CSVLogger\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import train_test_split\nimport tensorflow.keras.layers as L\n\nimport efficientnet.tfkeras as efn\nfrom tensorflow.keras.applications import DenseNet121\n\nfrom tqdm import tqdm\ntqdm.pandas()\n\ntf.random.set_seed(0)\n\n\n# HYPERPARAMS\n\nPATH = '../input/plant-pathology-2021-fgvc8/'\nTRAINDIR = PATH + 'train_images/'\nTESTDIR = PATH + 'test_images/'\nTRAIN_CSV = PATH + 'train.csv'\nSUBMISSION = PATH + 'sample_submission.csv'\n\nEPOCHS = 15\nSAMPLE_LEN = 100","metadata":{"execution":{"iopub.status.busy":"2021-08-04T15:21:02.747830Z","iopub.execute_input":"2021-08-04T15:21:02.748209Z","iopub.status.idle":"2021-08-04T15:21:03.062156Z","shell.execute_reply.started":"2021-08-04T15:21:02.748172Z","shell.execute_reply":"2021-08-04T15:21:03.061242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting a loaded model to TFLite format\n\nfrom tensorflow import lite\nmodel = tf.keras.models.load_model('../input/densenet-2dataaug-model/dense_net_joint_2daug.h5')\nconverter = lite.TFLiteConverter.from_keras_model(model)\ntflite_model = converter.convert()\nwith open('model.tflite', 'wb') as f:\n  f.write(tflite_model)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T15:24:17.290834Z","iopub.execute_input":"2021-08-04T15:24:17.291200Z","iopub.status.idle":"2021-08-04T15:25:44.689968Z","shell.execute_reply.started":"2021-08-04T15:24:17.291172Z","shell.execute_reply":"2021-08-04T15:25:44.688876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the dataset\n\nThis first set of functions are taken from *utils.py* and are related to data loading as well as to the partition of the whole dataset into a train and test sets.\n\nLater on, we will also create a validation set. The difference between the test and validation sets is that the validation set is going to be used by the model building algorithm at the end of each epoch during its training phase, so it can keep track of how close each epoch gets to optimal weights. On the other hand, the test set is going to be used once the model is completely built to try its overall performance.\n\nThis has been the final approach because the provided images for testing only consist of 3 samples, as the competition has a hidden test set that will be only provided once the submission has been done, consisting of 5 thousand extra images.\n\nTo summarize it all, we will use a 15% of the total dataset as a validation set and a 10% as the test set.","metadata":{}},{"cell_type":"code","source":"# To reproduce the same random partition of the dataset into train and test among different experiments\ndef seed_reproducer(seed=2021):\n    np.random.seed(seed)\n    random.seed(seed)\n\ndef load_split_dataset(frac: float=0.1, data: pd.DataFrame=None) -> \"tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]\":\n    dataset = pd.read_csv(TRAIN_CSV)\n    if data is not None:\n        dataset = data\n    seed_reproducer()\n    state = random.randint(0, 10000)\n    test = dataset.sample(frac=frac, random_state=state).reset_index()\n    train = dataset\n    for index in test['index'].values:\n        train = train.drop([index])\n    train = train.reset_index(drop=True)\n    test = test.drop(columns=['index'])\n    return dataset, train, test","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:29.670027Z","iopub.execute_input":"2021-07-15T22:24:29.670444Z","iopub.status.idle":"2021-07-15T22:24:29.67859Z","shell.execute_reply.started":"2021-07-15T22:24:29.670395Z","shell.execute_reply":"2021-07-15T22:24:29.677543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the dataset file and split them into train and test sets\ndata, train, test = load_split_dataset()\n\n# This condition doesn't ensure the function is properly implemented, it's necessary but not sufficient\nassert(len(data) == len(test) + len(train))\n\n# We can check the dataset\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:30.820775Z","iopub.execute_input":"2021-07-15T22:24:30.82113Z","iopub.status.idle":"2021-07-15T22:24:32.920715Z","shell.execute_reply.started":"2021-07-15T22:24:30.821097Z","shell.execute_reply":"2021-07-15T22:24:32.919908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Removing duplicates\n\nTo remove duplicated images, we use duplicates.csv file, which contains 62 sequences of duplicates found with image_hash, this list has been taken [from this notebook](https://www.kaggle.com/nickuzmenkov/pp2021-duplicates-revealing). For each duplicate sequence:\n\nWe leave only one sample if all duplicates share the same labels, and we will delete all duplicates if at least one of them is labeled differently, because in that case we can't know which one is the correct label.","metadata":{}},{"cell_type":"code","source":"with open('../input/pp2021duplicatesrevealing/duplicates.csv', 'r') as file:\n    duplicates = [x.strip().split(',') for x in file.readlines()]\n\ndef eliminate_duplicates(dataframe):\n    init_len = len(dataframe)\n    \n    for row in duplicates:\n        sizes = set()\n        for img in row:\n            labels = dataframe.loc[dataframe['image'] == img]['labels'].values\n            sizes.add(len(labels))\n        if len(sizes) == 1:\n            for img in row[1:]:\n                indexName = dataframe[dataframe['image'] == img].index\n                dataframe.drop(indexName, inplace=True)\n        else:\n            for img in row:\n                indexName = dataframe[dataframe['image'] == img].index\n                dataframe.drop(indexName, inplace=True)\n    print(f'Dropping {init_len - len(dataframe)} duplicate samples.')\n    \n        \n\neliminate_duplicates(data)\n\n# We split the dataset again, but now taking into account the dropped rows\ndata, train, test = load_split_dataset(data=data)","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:32.921983Z","iopub.execute_input":"2021-07-15T22:24:32.922441Z","iopub.status.idle":"2021-07-15T22:24:35.5987Z","shell.execute_reply.started":"2021-07-15T22:24:32.92238Z","shell.execute_reply":"2021-07-15T22:24:35.597633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Normalising the dataset\n\nThe following functions are also taken from *utils.py*, they are used to change the labels representation in the pandas dataframe. \n\nInitially the dataset looked like this:\n\n>|***image***    |***labels***   |  \n|---|---|  \n|e88d1bbd624e9c34.jpg   |powdery_mildew   |  \n|8002cb321f8bfcdf.jpg   |scab frog_eye_leaf_spot complex   |  \n| ...  | ...  |  \n\n\n## Disjoint normalisation\n\nThe first type of normalisation has been named as *disjoint* because it keeps the original 12 labels as 12 unique classes.\n\nNormalising it this way transforms it to:\n\n> |***image***    |***scab***    |***healthy***  |***frog_eye_leaf_spot***    |  ***rust***    |***complex***    |***powdery_mildew***    |***scab frog_eye_leaf_spot***    |***scab frog_eye_leaf_spot complex***    |***frog_eye_leaf_spot complex***    |***rust frog_eye_leaf_spot***   |***rust complex***    |***powdery_mildew complex***    | \n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|e88d1bbd624e9c34.jpg|0|0|0|0|0|1|0|0|0|0|0|0|\n|8002cb321f8bfcdf.jpg|0|0|0|0|0|0|0|1|0|0|0|0|\n|...|...|...|...|...|...|...|...|...|...|...|...|...|\n\n## Joint normalisation\n\nThis normalisation joins the labels into 6 basic labels, removing the possibility of having spaces to separate different diseases.\n\nAccording to the competition's information, different diseases are separated by spaces, that's why this normalisation is also considered and seen as the most accurate one.\n\nThis normalisation would transform it into the following way:\n\n> |***image***    |***scab***    |***healthy***  |***frog_eye_leaf_spot***    |  ***rust***    |***complex***    |***powdery_mildew***    |\n|---|---|---|---|---|---|---|\n|e88d1bbd624e9c34.jpg|0|0|0|0|0|1|\n|8002cb321f8bfcdf.jpg|1|0|1|0|1|0|\n|...|...|...|...|...|...|...|\n\n","metadata":{}},{"cell_type":"code","source":"# Here we have the 2 types of normalisations\n\ndef normalise_from_dataset_disjoint(dataset: pd.DataFrame) -> pd.DataFrame:\n    columns = ['image']\n    labels = dataset['labels'].value_counts().index.tolist()\n        \n    columns.extend(labels)\n    data = []\n\n    for image, label in zip(dataset['image'], dataset['labels']):\n        labelpos = columns.index(label)\n        row = [image]\n        for _ in labels: row.append(0)\n        row[labelpos] =  1\n        data.append(row)\n    \n    return pd.DataFrame(data, columns=columns)\n\ndef normalise_from_dataset_joint(dataset: pd.DataFrame) -> pd.DataFrame:\n    columns = ['image']\n    labels = dataset['labels'].value_counts().index.tolist()\n    basic_labels = set()   \n    for label in labels:\n        for word in label.split():\n            basic_labels.add(word)\n\n    columns.extend(basic_labels)\n    data = []\n\n    for image, labels in zip(dataset['image'], dataset['labels']):\n\n        row = [image]\n        real_labels = labels.split()\n        for _ in basic_labels: row.append(0)\n        for real_label in real_labels:\n            labelpos = columns.index(real_label)\n            row[labelpos] =  1\n        data.append(row)\n    \n    return pd.DataFrame(data, columns=columns)","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:37.900067Z","iopub.execute_input":"2021-07-15T22:24:37.90057Z","iopub.status.idle":"2021-07-15T22:24:37.911309Z","shell.execute_reply.started":"2021-07-15T22:24:37.900539Z","shell.execute_reply":"2021-07-15T22:24:37.910002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Normalising with the joint approach\nnorm_train = normalise_from_dataset_joint(train)\nnorm_test = normalise_from_dataset_joint(test)\n\n# The assertion should still hold\nassert(len(data) == len(norm_train) + len(norm_test))\n\n# If we are lucky enough we can check that there are rows with multiple diseases\nnorm_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:40.029524Z","iopub.execute_input":"2021-07-15T22:24:40.029898Z","iopub.status.idle":"2021-07-15T22:24:40.305777Z","shell.execute_reply.started":"2021-07-15T22:24:40.029863Z","shell.execute_reply":"2021-07-15T22:24:40.304771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle's TPU configuration\n\nAs mentioned before, the sole purpose of this kernel is to perform the computation-intensive tasks that require Tensor Process Units (TPUs), this is why we won't dig into data analysis or data preprocessing and we'll get right to the training part. \n\nThe following piece of code allows the connection of the Kaggle Kernel to the available TPUs or GPUs.","metadata":{}},{"cell_type":"code","source":"# Allows prefetching of data in the input pipeline for each step of the training process, \n# tuning the values of the optimization algorithm dynamically at runtime\nAUTO = tf.data.experimental.AUTOTUNE\n\nprint('Using tensorflow %s' % tf.__version__)\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n    print('Running on TPUv3-8')\nexcept:\n    tpu = None\n    tf.keras.mixed_precision.set_global_policy('mixed_float16')\n    strategy = tf.distribute.get_strategy()\n    print('Running on GPU with mixed precision')\n\n# The batch size refers to the number of samples utilized in each iteration of an epoch\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nprint('Number of replicas:', strategy.num_replicas_in_sync)\nprint('Batch size: %.i' % BATCH_SIZE)\n\n","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:41.795068Z","iopub.execute_input":"2021-07-15T22:24:41.795615Z","iopub.status.idle":"2021-07-15T22:24:47.448814Z","shell.execute_reply.started":"2021-07-15T22:24:41.795573Z","shell.execute_reply":"2021-07-15T22:24:47.447684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tensorflow set up\n\nIn this section we must prepare the data that we have already gathered in the terms that Tensorflow is going to need it, such as *numpy* dataclases, *tensorflow* images or *tensorflow* dataset, as well as adjusting the path where our images are saves in Google Cloud Storage.","metadata":{}},{"cell_type":"code","source":"# The dataset is stored at google cloud's storage buckets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('plant-pathology-2021-fgvc8')\n\n\n#Be careful with this variables, adapt them to the first and last columns names in the dataset, it will cause a compilation error if they don't match\nfirst_label = 'frog_eye_leaf_spot'\nlast_label = 'rust'\n\ndef format_path(st):\n    return GCS_DS_PATH + '/train_images/' + st\n\n# For the moment we will only use the test_paths to predict on our model, \n# test_labels is going to be used for validation at the end\ntest_paths = norm_test['image'].apply(format_path).values\ntest_labels = np.float32(norm_test.loc[:, first_label:last_label].values)\n\ntrain_paths = norm_train['image'].apply(format_path).values\ntrain_labels = np.float32(norm_train.loc[:, first_label:last_label].values)\n\n# Similar to the function we build in utils.py, scikit library provides us \n# a function to split data into train and test, used to set up the validation set\ntrain_paths, valid_paths, train_labels, valid_labels =\\\ntrain_test_split(train_paths, train_labels, test_size=0.15, random_state=2020)\n\n\ndef decode_image(filename, label=None, image_size=(512, 512)):\n    bits = tf.io.read_file(filename)\n    image = tf.image.decode_jpeg(bits, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image, image_size)\n    \n    if label is None:\n        return image\n    else:\n        return image, label\n\n# For the moment we will only use 2 data augmentation techniques\ndef data_augment(image, label=None):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    #TODO\n    #image = tf.image.random_crop()\n    #image = tf.image.random_brightness\n    #image = tf.image.random_contrast\n    \n    if label is None:\n        return image\n    else:\n        return image, label\n    \n# Create Dataset objects\ntrain_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths, train_labels))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls=AUTO)\n    .repeat()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((valid_paths, valid_labels))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(test_paths)\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n)\n","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:24:59.1781Z","iopub.execute_input":"2021-07-15T22:24:59.178485Z","iopub.status.idle":"2021-07-15T22:24:59.877086Z","shell.execute_reply.started":"2021-07-15T22:24:59.178453Z","shell.execute_reply":"2021-07-15T22:24:59.875945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the learning rate function","metadata":{}},{"cell_type":"code","source":"def build_lrfn(lr_start=0.00001, lr_max=0.00005, \n               lr_min=0.00001, lr_rampup_epochs=5, \n               lr_sustain_epochs=0, lr_exp_decay=.8):\n    lr_max = lr_max * strategy.num_replicas_in_sync\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) *\\\n                 lr_exp_decay**(epoch - lr_rampup_epochs\\\n                                - lr_sustain_epochs) + lr_min\n        return lr\n    return lrfn\n\nlrfn = build_lrfn()\nSTEPS_PER_EPOCH = train_labels.shape[0] // BATCH_SIZE\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\n","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:25:02.815358Z","iopub.execute_input":"2021-07-15T22:25:02.815724Z","iopub.status.idle":"2021-07-15T22:25:02.823586Z","shell.execute_reply.started":"2021-07-15T22:25:02.815694Z","shell.execute_reply":"2021-07-15T22:25:02.822368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building up the model: DenseNet 121","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    model_dense = tf.keras.Sequential([DenseNet121(input_shape=(512, 512, 3),\n                                             weights='imagenet',\n                                             include_top=False),\n                                 L.GlobalAveragePooling2D(),\n                                 L.Dense(train_labels.shape[1],\n                                         activation='softmax')])\n        \n    model_dense.compile(optimizer='adam',\n                  loss = 'categorical_crossentropy',\n                  metrics=['categorical_accuracy'])\n    model_dense.summary()\n    \n# To save the model training history for later reviews\ndense_csv_logger = CSVLogger('dense_net_joint_2daug.log', separator=',', append=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-04T20:11:52.6772Z","iopub.execute_input":"2021-07-04T20:11:52.677764Z","iopub.status.idle":"2021-07-04T20:12:16.565731Z","shell.execute_reply.started":"2021-07-04T20:11:52.677719Z","shell.execute_reply":"2021-07-04T20:12:16.564555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training the model","metadata":{}},{"cell_type":"code","source":"history_dense = model_dense.fit(train_dataset,\n                    epochs=EPOCHS,\n                    callbacks=[lr_schedule, dense_csv_logger],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset)\n\n# Saving the model\nmodel_dense.save('dense_net_joint_2daug_dedup.h5')","metadata":{"execution":{"iopub.status.busy":"2021-07-04T20:12:16.567602Z","iopub.execute_input":"2021-07-04T20:12:16.567985Z","iopub.status.idle":"2021-07-04T21:34:49.416368Z","shell.execute_reply.started":"2021-07-04T20:12:16.567949Z","shell.execute_reply":"2021-07-04T21:34:49.415419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building up the model: Efficient Net","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    model_efn = tf.keras.Sequential([efn.EfficientNetB7(input_shape=(512, 512, 3),\n                                                    weights='imagenet',\n                                                    include_top=False),\n                                 L.GlobalAveragePooling2D(),\n                                 L.Dense(train_labels.shape[1],\n                                         activation='softmax')])\n    \n    \n        \n    model_efn.compile(optimizer='adam',\n                  loss = 'categorical_crossentropy',\n                  metrics=['categorical_accuracy'])\n    efn_csv_logger = CSVLogger('efn_joint_2daug.log', separator=',', append=False)\n    model_efn.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-15T22:25:50.111683Z","iopub.execute_input":"2021-07-15T22:25:50.112303Z","iopub.status.idle":"2021-07-15T22:26:04.495257Z","shell.execute_reply.started":"2021-07-15T22:25:50.112258Z","shell.execute_reply":"2021-07-15T22:26:04.494157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"history_efn = model_efn.fit(train_dataset,\n                    epochs=EPOCHS,\n                    callbacks=[lr_schedule, efn_csv_logger],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset)\n\n# Saving the model\nmodel_efn.save('efn_joint_2daug.h5')","metadata":{"execution":{"iopub.status.busy":"2021-06-16T16:01:29.046519Z","iopub.execute_input":"2021-06-16T16:01:29.047206Z","iopub.status.idle":"2021-06-16T17:20:36.915138Z","shell.execute_reply.started":"2021-06-16T16:01:29.047164Z","shell.execute_reply":"2021-06-16T17:20:36.914123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the model: Efficient Noisy Student","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    model_efnns = tf.keras.Sequential([efn.EfficientNetB7(input_shape=(512, 512, 3),\n                                                    weights='noisy-student',\n                                                    include_top=False),\n                                 L.GlobalAveragePooling2D(),\n                                 L.Dense(train_labels.shape[1],\n                                         activation='softmax')])\n    \n    \n        \n    model_efnns.compile(optimizer='adam',\n                  loss = 'categorical_crossentropy',\n                  metrics=['categorical_accuracy'])\n    model_efnns.summary()\n    efnns_csv_logger = CSVLogger('efnns_joint_2daug.log', separator=',', append=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-16T17:26:55.835892Z","iopub.execute_input":"2021-06-16T17:26:55.836265Z","iopub.status.idle":"2021-06-16T17:27:47.623518Z","shell.execute_reply.started":"2021-06-16T17:26:55.836236Z","shell.execute_reply":"2021-06-16T17:27:47.622443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"history_efnns = model_efnns.fit(train_dataset,\n                    epochs=EPOCHS,\n                    callbacks=[lr_schedule, efnns_csv_logger],\n                    steps_per_epoch=STEPS_PER_EPOCH,\n                    validation_data=valid_dataset)\n\n# Saving the model\nmodel_efn.save('efnns_joint_2daug.h5')","metadata":{"execution":{"iopub.status.busy":"2021-06-16T17:28:22.877304Z","iopub.execute_input":"2021-06-16T17:28:22.877676Z","iopub.status.idle":"2021-06-16T18:41:09.301492Z","shell.execute_reply.started":"2021-06-16T17:28:22.877642Z","shell.execute_reply":"2021-06-16T18:41:09.30071Z"},"trusted":true},"execution_count":null,"outputs":[]}]}