{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Recursion Cellular Image Classification\n\n---\n### Universidade de Brasília\n\nCIC0193 - Fundamentos de Sistemas Inteligentes\n\nProf.: Vinicius Borges\n\nAluno: Pedro Lucas Silva Haga Torres\n\nMatrícula: 16/0141575\n\n##### Atividade IV - Redes Neurais Convolucionais\n\n---\nThis is an assignment for the *Fundamentos de Sistemas Inteligentes* (Fundaments of Inteligent Systems) course at Universidade de Brasília (University of Brasília). All of the code and documentation is in english, the only exception being the header above, which identifies myself as an student undertaking the course mentioned previously.","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import glob\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport tensorflow as  tf\nimport tensorflow_addons as tfa\n\nfrom keras import backend as K\nfrom sklearn.metrics import classification_report\nfrom sklearn.model_selection import train_test_split\n\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom tensorflow.keras.metrics import CategoricalAccuracy\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:10:05.148701Z","iopub.execute_input":"2021-10-24T21:10:05.149231Z","iopub.status.idle":"2021-10-24T21:10:05.154975Z","shell.execute_reply.started":"2021-10-24T21:10:05.149195Z","shell.execute_reply":"2021-10-24T21:10:05.154192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data analysis","metadata":{}},{"cell_type":"code","source":"# Open train dataframe and print it\n# train_df = pd.read_csv(r\"../input/recursion-cellular-image-classification/train.csv\")\n\n# train_df","metadata":{"execution":{"iopub.status.busy":"2021-10-20T18:01:13.87597Z","iopub.execute_input":"2021-10-20T18:01:13.876704Z","iopub.status.idle":"2021-10-20T18:01:13.977223Z","shell.execute_reply.started":"2021-10-20T18:01:13.876623Z","shell.execute_reply":"2021-10-20T18:01:13.975451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the only usefull information in the training dataframe are the labels (`sirna`). We can't get to the images using this dataframe, so we'll have to use the information in the images'path to get to their labels. What identifies a label in this dataframe is the experiment, plate and well, so we'll use this information (which, according to the documentation is also in the image path) to label the images.","metadata":{}},{"cell_type":"markdown","source":"### Getting the data we need\n\nSince the directory structure don't give us much beyond labeling the images as train and test, we'll start by getting the path to all images using `glob` and saving it to a Pandas dataframe.","metadata":{}},{"cell_type":"code","source":"# # Set path to get training images\n# path = r\"../input/recursion-cellular-image-classification/train/*/*/*.png\"\n\n# # Save path to all training images in a Pandas dataframe using glob\n# df = pd.DataFrame(glob.glob(path), columns=[\"image_path\"])\n\n# # Print dataframe\n# df","metadata":{"execution":{"iopub.status.busy":"2021-10-20T18:01:13.978529Z","iopub.execute_input":"2021-10-20T18:01:13.979427Z","iopub.status.idle":"2021-10-20T18:01:49.738532Z","shell.execute_reply.started":"2021-10-20T18:01:13.97938Z","shell.execute_reply":"2021-10-20T18:01:49.737603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Getting images' info based on their path\n\nAccording to the documentation, we can find the experiment, plate and well in the path or in the image's name, so we're going to use this in order to properly label the images.","metadata":{}},{"cell_type":"code","source":"# # Get EXPERIMENT from image's path - according to the documentation\n# df['experiment'] = df['image_path'].str.split(\"/\").str[4]\n\n# # Get PLATE from image's path\n# df['plate'] = df['image_path'].str.split(\"/\").str[5].str.split(\"Plate\").str[1]\n\n# # Cast 'plate' values to int\n# df['plate'] = df['plate'].astype(int)\n\n# # Get WELL from image's path\n# df['well'] = df['image_path'].str.split(\"/\").str[6].str.split(\"_\").str[0]\n\n# # Print dataframe to check if process went well\n# df","metadata":{"execution":{"iopub.status.busy":"2021-10-20T18:01:49.740126Z","iopub.execute_input":"2021-10-20T18:01:49.740336Z","iopub.status.idle":"2021-10-20T18:01:57.847698Z","shell.execute_reply.started":"2021-10-20T18:01:49.74031Z","shell.execute_reply":"2021-10-20T18:01:57.846803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Labeling the images\n\nThis is probably not the most efficient way to do this, but we're going to get the labels by searching the training dataframe using the experiment, plate and well that we got from the images' path. This will be saved on a list, first, and then it'll be added on the dataframe with the images' path.","metadata":{}},{"cell_type":"code","source":"# # Create an empty list to save the labels\n# sirna_list = []\n\n# # Iterate through the dataframe using 'itertuples' and search on the training dataframe\n# # for each image's label\n# for t in df.itertuples(index=False):\n#     sirna_list.append(train_df.loc[(t[1] == train_df['experiment']) &\n#                                 (t[2] == train_df['plate']) &\n#                                 (t[3] == train_df['well'])]['sirna'].values)\n\n# # Add new column to dataframe containing the labels accquired\n# df['sirna'] = sirna_list\n\n# # The labels acquired come in the form of a series, we're getting the labels themselves\n# # or, if the image is not labeled, we get an empty list, so we're replacing it with NaN\n# df['sirna'] = df['sirna'].apply(lambda x: np.nan if len(x) == 0 else x[0])\n\n# # Print the dataframe with the labels\n# df","metadata":{"execution":{"iopub.status.busy":"2021-10-20T18:01:57.84894Z","iopub.execute_input":"2021-10-20T18:01:57.849259Z","iopub.status.idle":"2021-10-20T19:43:59.250244Z","shell.execute_reply.started":"2021-10-20T18:01:57.849227Z","shell.execute_reply":"2021-10-20T19:43:59.249414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Removing images without a label\n\nWe can see that we have a few unlabeled images, so we're going to remove them using Pandas' `dropna()`.","metadata":{}},{"cell_type":"code","source":"# # Check for NaNs\n# print(df.info(), end=\"\\n\\n\")\n\n# # Remove NaNs\n# df.dropna(inplace=True)\n\n# # Check final product\n# df.info()","metadata":{"execution":{"iopub.status.busy":"2021-10-20T19:44:19.175444Z","iopub.execute_input":"2021-10-20T19:44:19.175812Z","iopub.status.idle":"2021-10-20T19:44:19.429474Z","shell.execute_reply.started":"2021-10-20T19:44:19.175765Z","shell.execute_reply":"2021-10-20T19:44:19.428539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check number of classes and data distribuition","metadata":{}},{"cell_type":"code","source":"# Plot labels count\n# df['sirna'].value_counts().plot(kind='bar', figsize=(14, 7))","metadata":{"execution":{"iopub.status.busy":"2021-10-20T19:43:59.841211Z","iopub.execute_input":"2021-10-20T19:43:59.841824Z","iopub.status.idle":"2021-10-20T19:44:19.172611Z","shell.execute_reply.started":"2021-10-20T19:43:59.841778Z","shell.execute_reply":"2021-10-20T19:44:19.171684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we have a lot of classes (1108, to be precise) but the data is mostly balanced, with around 400 instances for each class.","metadata":{}},{"cell_type":"markdown","source":"### Save dataframe containing the images' path and label","metadata":{}},{"cell_type":"code","source":"# Save dataframe for later use and to avoid the costly method of acquiring the labels\n# df.to_csv(r\"/kaggle/working/train_dataframe.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T19:44:20.069158Z","iopub.execute_input":"2021-10-20T19:44:20.069407Z","iopub.status.idle":"2021-10-20T19:44:23.017417Z","shell.execute_reply.started":"2021-10-20T19:44:20.069374Z","shell.execute_reply":"2021-10-20T19:44:23.016529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Classification task","metadata":{}},{"cell_type":"code","source":"# Open the dataframe with the images' path and labels\ndf = pd.read_csv(r\"../input/rcic-edited-dataframe/train_dataframe.csv\")\n\ndf","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:10:13.602196Z","iopub.execute_input":"2021-10-24T21:10:13.602452Z","iopub.status.idle":"2021-10-24T21:10:14.587481Z","shell.execute_reply.started":"2021-10-24T21:10:13.602423Z","shell.execute_reply":"2021-10-24T21:10:14.586807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stratified split between train, validation and test\nWe're going to use transfer learning, so, for training the top layers, I used the more common split of 70-20-10 (training, validation and test, respectively, in percentage). But, after consideration, since there are more than 400k images, I choose a split of 90-5-5 to train the whole model.\n\nSince `random_state` is defined, we can repeat the experiment any number of times and get the same results. I also stratified the split according to the labels' distribution.","metadata":{}},{"cell_type":"code","source":"x_train, x_test, y_train, y_test = train_test_split(df, df['sirna'], test_size=0.1,\n                                                    random_state=31415,\n                                                    stratify=df['sirna'])\n\nx_val, x_test, y_val, y_test = train_test_split(x_test, x_test['sirna'], test_size=0.5,\n                                                random_state=31415,\n                                                stratify=x_test['sirna'])\n\nprint(\"Train data's shape:      {}; {}\".format(x_train.shape, y_train.shape))\nprint(\"Validation data's shape: {}; {}\".format(x_val.shape, y_val.shape))\nprint(\"Test data's shape:       {}; {}\".format(x_test.shape, y_test.shape))","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:10:16.517754Z","iopub.execute_input":"2021-10-24T21:10:16.518281Z","iopub.status.idle":"2021-10-24T21:10:17.898289Z","shell.execute_reply.started":"2021-10-24T21:10:16.518203Z","shell.execute_reply":"2021-10-24T21:10:17.897462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dependencies\n#### Metrics\n\nSince Keras/TensorFlow don't have the necessary metrics to evaluate a multiclass task (such as precision, recall and specificity), we're going to use both **micro** and **macro** averaged F1-score and categorical accuracy (available in TF/Keras) as our metrics. For micro/macro F1, we're using [TensorFlow Addons](https://www.tensorflow.org/addons) library.","metadata":{}},{"cell_type":"code","source":"# Multiclass F1-score MICRO Avg.\nmicro_f1 = tfa.metrics.F1Score(\n    num_classes=1108,\n    average='micro',\n    name=\"Micro F1\",\n)\n\n# Multiclass F1-score MACRO Avg.\nmacro_f1 = tfa.metrics.F1Score(\n    num_classes=1108,\n    average='macro',\n    name=\"Macro F1\",\n)","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:10:24.103432Z","iopub.execute_input":"2021-10-24T21:10:24.103735Z","iopub.status.idle":"2021-10-24T21:10:26.628854Z","shell.execute_reply.started":"2021-10-24T21:10:24.103702Z","shell.execute_reply":"2021-10-24T21:10:26.628089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image data generators\nGiven the time available for the task, I didn't considered using data augmentation (DA), because I didn't know much about the underlying details of the problem and what methods were appropriate. So, the top layers were trained without DA.\n\nI added DA. after peeking the work of the competition's winner, so I'm using almost the same method as them. This means that I'm using DA in the training of all layers (just to be clear, this step wasn't taken for the top layers' training).\n\nThe images are resized to 224x224 px because it's the input size for EfficientNet-B0. Batch size is 512 in order to try and accelerate training and also because there is enough memory for the task (1024 works, but raises warnings). Once again we're seeding the RNG, so experiments should be consistent between runs.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 224\nBATCH_SIZE = 512\n\ntrain_dataGen = ImageDataGenerator(\n    rescale=1./255, rotation_range=90, horizontal_flip=True, vertical_flip=True)\n\ntrain_generator = train_dataGen.flow_from_dataframe(\n    dataframe=x_train, x_col='image_path', class_mode='categorical', seed=31415,\n    y_col='sirna', target_size=(IMG_SIZE, IMG_SIZE), batch_size=BATCH_SIZE)\n\nval_generator = train_dataGen.flow_from_dataframe(\n    dataframe=x_val, x_col='image_path', class_mode='categorical', seed=31415,\n    y_col='sirna', target_size=(IMG_SIZE, IMG_SIZE), batch_size=BATCH_SIZE)\n\ntest_generator = train_dataGen.flow_from_dataframe(\n    dataframe=x_test, x_col='image_path', class_mode='categorical', seed=31415,\n    y_col='sirna', target_size=(IMG_SIZE, IMG_SIZE), batch_size=BATCH_SIZE)","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:10:26.630285Z","iopub.execute_input":"2021-10-24T21:10:26.631112Z","iopub.status.idle":"2021-10-24T21:29:43.612575Z","shell.execute_reply.started":"2021-10-24T21:10:26.631074Z","shell.execute_reply":"2021-10-24T21:29:43.611837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Network instatiation\n\nThe chosen architecture was EfficientNet-B0. Since I'm submitting this work as an assignment for an university course, it was chosen in order to minimize training time (as I only had a week to dedicate to this assignment). This means that EfficientNet-B0 is the smallest network of its family and should not have a great classification performance.\n\nIn summary, the EfficientNet architecture was proposed by Tan and Le who are (or were) both researchers at Google. Its idea was to study the growth of a CNN parameters (number of layers, filters and input size) as an optimization problem. This can be used with more traditional architectures such as ResNet, Inception, GoogLeNet, etc. in order to optimize them, or, to create a brand new architecture - EfficientNet - and scale its growth towards better performance with a reduced number of trainable parameters. For more information, check the link to the paper hosted in arXiv below:\n\nhttps://arxiv.org/abs/1905.11946\n\ntl;dr: EfficientNet-B0's performance is close to DenseNet-201 and ResNet-152 in ImageNet, while having way less parameters (about half of DenseNet's, and 1/6 of ResNet's).","metadata":{}},{"cell_type":"code","source":"# # Load EfficientNet pre-trained w/ ImageNet\n# base_model = EfficientNetB0(include_top=False, weights=\"imagenet\")\n\n# # Rebuild top\n# avg = layers.GlobalAveragePooling2D(name=\"avg_pool\")(base_model.output)\n# norm = layers.BatchNormalization()(avg)\n# dropout = layers.Dropout(0.3, name=\"top_dropout\")(norm)\n# output = layers.Dense(1108, activation=\"softmax\", name=\"pred\")(dropout)\n\n# model = tf.keras.Model(base_model.input, output, name=\"EfficientNet-B0\")\n\n# # Freeze the pretrained weights\n# for layer in base_model.layers:\n#     layer.trainable = False\n\n# # Optimizer setup\n# optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n\n# # Metrics\n# metrics = [CategoricalAccuracy(name='Categorical Accuracy'),\n#            micro_f1, macro_f1]\n\n# # Compile model\n# model.compile(optimizer=optimizer, loss=\"categorical_crossentropy\",\n#               metrics=metrics)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T20:54:24.724314Z","iopub.execute_input":"2021-10-20T20:54:24.724602Z","iopub.status.idle":"2021-10-20T20:54:26.170545Z","shell.execute_reply.started":"2021-10-20T20:54:24.724573Z","shell.execute_reply":"2021-10-20T20:54:26.169799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Top layers' training","metadata":{}},{"cell_type":"code","source":"# # Checkpoint to save network's best weights\n# checkpoint = tf.keras.callbacks.ModelCheckpoint(\n#     \"./effNet-B0_{epoch:02d}\",\n#     monitor='val_loss', verbose=1, save_best_only=True,\n#     save_weights_only=True, mode='min')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Top layers training\n\n# # Epochs to train the top layers: min:8; max:80\n# history = model.fit(train_generator, validation_data=val_generator, epochs=3,\n#                     callbacks=[checkpoint])","metadata":{"execution":{"iopub.status.busy":"2021-10-20T21:32:16.177613Z","iopub.execute_input":"2021-10-20T21:32:16.178593Z","iopub.status.idle":"2021-10-21T01:21:35.967565Z","shell.execute_reply.started":"2021-10-20T21:32:16.178543Z","shell.execute_reply":"2021-10-21T01:21:35.965799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Save model's current weights\n# model.save_weights(r\"./effNetB0_topTrained_weights\")\n\n# # Save whole model\n# model.save(r\"./effNetB0_topTrained_model\")","metadata":{"execution":{"iopub.status.busy":"2021-10-21T01:22:29.117831Z","iopub.execute_input":"2021-10-21T01:22:29.118093Z","iopub.status.idle":"2021-10-21T01:23:12.145438Z","shell.execute_reply.started":"2021-10-21T01:22:29.118063Z","shell.execute_reply":"2021-10-21T01:23:12.144637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### All layers' training","metadata":{}},{"cell_type":"code","source":"# Unfreeze all layer's pretrained weights\n# for layer in model.layers:\n#     layer.trainable = True","metadata":{"execution":{"iopub.status.busy":"2021-10-22T19:56:44.584238Z","iopub.execute_input":"2021-10-22T19:56:44.58498Z","iopub.status.idle":"2021-10-22T19:56:44.598046Z","shell.execute_reply.started":"2021-10-22T19:56:44.584937Z","shell.execute_reply":"2021-10-22T19:56:44.597368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load whole model previously trained\n# model = keras.models.load_model(r\"../input/rcictfmodel-final/effNet-B0_13\")\n\n# Change optimizer's parameters (if needed)\n# K.set_value(model.optimizer.learning_rate, 1e-3)\n# K.set_value(model.optimizer.beta_1, 0.9)\n\n# Verify changes to optimizer's parameters\n# print(model.optimizer.learning_rate)\n# print(model.optimizer.beta_1)","metadata":{"execution":{"iopub.status.busy":"2021-10-22T19:55:28.865969Z","iopub.execute_input":"2021-10-22T19:55:28.866238Z","iopub.status.idle":"2021-10-22T19:56:44.582552Z","shell.execute_reply.started":"2021-10-22T19:55:28.866207Z","shell.execute_reply":"2021-10-22T19:56:44.581811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checkpoint to save model after each epoch of training\n# checkpoint = tf.keras.callbacks.ModelCheckpoint(\n#     \"./effNet-B0_{epoch:02d}\",\n#     monitor='val_loss', verbose=1, save_best_only=False,\n#     save_weights_only=False, mode='min')","metadata":{"execution":{"iopub.status.busy":"2021-10-22T19:56:44.599447Z","iopub.execute_input":"2021-10-22T19:56:44.599746Z","iopub.status.idle":"2021-10-22T19:56:44.608499Z","shell.execute_reply.started":"2021-10-22T19:56:44.599687Z","shell.execute_reply":"2021-10-22T19:56:44.607667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train the whole model\n# history = model.fit(train_generator, validation_data=val_generator, epochs=13,\n#                     callbacks=[checkpoint], initial_epoch=10)","metadata":{"execution":{"iopub.status.busy":"2021-10-22T19:57:50.549406Z","iopub.execute_input":"2021-10-22T19:57:50.550126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Open dataframes containing previous training results\nhistory_df1 = pd.read_csv(r\"../input/rcictfmodel-te02/effNetB0_history.csv\")\nhistory_df2 = pd.read_csv(r\"../input/rcictf-modelte04/effNetB0_history.csv\")\nhistory_df3 = pd.read_csv(r\"../input/rcictfmodelte07/effNetB0_history.csv\")\nhistory_df4 = pd.read_csv(r\"../input/rcictfmodelte10/effNetB0_history.csv\")\nhistory_df5 = pd.read_csv(r\"../input/rcictfmodel-final/effNetB0_history.csv\")\n\n# Append newer epochs training values to 1st dataframe\nhistory_df1 = history_df1.append(history_df2, ignore_index=True)\nhistory_df1 = history_df1.append(history_df3, ignore_index=True)\nhistory_df1 = history_df1.append(history_df4, ignore_index=True)\nhistory_df1 = history_df1.append(history_df5, ignore_index=True)\n\n# Save metrics' history as a CSV file\nhistory_df1.to_csv(\"./effNetB0_history.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2021-10-22T19:52:42.464052Z","iopub.status.idle":"2021-10-22T19:52:42.465072Z","shell.execute_reply.started":"2021-10-22T19:52:42.464837Z","shell.execute_reply":"2021-10-22T19:52:42.464862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_df1.to_csv(\"./effNetB0_history.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:39:25.037843Z","iopub.execute_input":"2021-10-24T21:39:25.038406Z","iopub.status.idle":"2021-10-24T21:39:25.044949Z","shell.execute_reply.started":"2021-10-24T21:39:25.038366Z","shell.execute_reply":"2021-10-24T21:39:25.044204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_df1","metadata":{"execution":{"iopub.status.busy":"2021-10-24T22:13:17.758430Z","iopub.execute_input":"2021-10-24T22:13:17.759266Z","iopub.status.idle":"2021-10-24T22:13:17.779505Z","shell.execute_reply.started":"2021-10-24T22:13:17.759222Z","shell.execute_reply":"2021-10-24T22:13:17.778770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# summarize history for accuracy\nplt.plot(history_df1['Categorical Accuracy'])\nplt.plot(history_df1['val_Categorical Accuracy'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='upper right')\nplt.show()\n\n# summarize history for loss\nplt.plot(history_df1['loss'])\nplt.plot(history_df1['val_loss'])\nplt.title('Model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['Train', 'Validation'], loc='upper right')\nplt.show()\n\n# summarize history for MICRO F1-score\nplt.plot(history_df1['Micro F1'])\nplt.plot(history_df1['val_Micro F1'])\nplt.title('Model F1-Score (micro avg.)')\nplt.ylabel('F1-Score (micro avg.)')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='upper right')\nplt.show()\n\n# summarize history for MACRO F1-score\nplt.plot(history_df1['Macro F1'])\nplt.plot(history_df1['val_Macro F1'])\nplt.title('Model F1-score (macro avg.)')\nplt.ylabel('F1-score (macro avg.)')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='upper right')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-24T22:34:09.262473Z","iopub.execute_input":"2021-10-24T22:34:09.263027Z","iopub.status.idle":"2021-10-24T22:34:10.090397Z","shell.execute_reply.started":"2021-10-24T22:34:09.262984Z","shell.execute_reply":"2021-10-24T22:34:10.089662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing the model\nI don't expect much from it, I'm just hoping it's on par with the validation metrics.","metadata":{}},{"cell_type":"code","source":"# Load whole model previously trained\nmodel = keras.models.load_model(r\"../input/rcictfmodel-final/effNet-B0_13\")","metadata":{"execution":{"iopub.status.busy":"2021-10-24T21:41:07.799427Z","iopub.execute_input":"2021-10-24T21:41:07.799889Z","iopub.status.idle":"2021-10-24T21:41:17.378272Z","shell.execute_reply.started":"2021-10-24T21:41:07.799850Z","shell.execute_reply":"2021-10-24T21:41:17.377488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predicts labels\ntest_predictions = model.predict(test_generator, verbose=1)\n\nprint(classification_report(test_generator.labels,\n                            test_predictions.argmax(1), zero_division=0))","metadata":{"execution":{"iopub.status.busy":"2021-10-24T22:00:10.366629Z","iopub.execute_input":"2021-10-24T22:00:10.367053Z","iopub.status.idle":"2021-10-24T22:05:25.675949Z","shell.execute_reply.started":"2021-10-24T22:00:10.367020Z","shell.execute_reply":"2021-10-24T22:05:25.675227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\n\nGiven both validation and test results, I'm not going to bother submitting mine, as it would be just a waste of computer cycles. I'll just leave here what I would have done if I had the time to start over:\n\n1. I would have done a better job with image preprocessing (I'd resize instead of just crop with Keras) and data augmentation (I'd increase the number of training instances 3 or 4 fold.\n2. I'd have used EfficientNet-B2, at the very least, but I would try using B3 or higher with a smaller batch size (128, at most).\n3. Of course, I would train the model for more epochs and using SGD with Nesterov, to compare with Adam.\n\nWith this 3 steps, I think I could have done a better job, without the need to go overboard with ensemble, for instance (remember, this is an undergrad assingment). I'm not satisfied with this results, but it is what I could do with the time I had.","metadata":{}}]}