{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cassava Plant Leaf Disease Classification "},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"## About Cassava Plants\n\nAs the second-largest provider of carbohydrates in Africa, cassava is a key food security crop grown by smallholder farmers because it can withstand harsh conditions. At least 80% of household farms in Sub-Saharan Africa grow this starchy root, but viral diseases are major sources of poor yields."},{"metadata":{},"cell_type":"markdown","source":"## Importing Libraries and Data"},{"metadata":{"trusted":false},"cell_type":"code","source":"import os\nimport glob\nimport json\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport tensorflow as tf\nimport tensorflow.keras as k\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom keras import backend as K \nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.optimizers import RMSprop, Adam, Nadam\nfrom tensorflow.keras.applications import VGG16\nfrom tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras.applications import EfficientNetB4\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import GlobalAveragePooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\n\n#install efficientnet that has noisy-student weights\n!pip install -U efficientnet\nimport efficientnet.keras as efn\n\n#Bi-Tempered Logistic Loss\ncwd = os.getcwd()\nos.chdir('../input/bitempered-logistic-loss-direct-upload')\nfrom tf_bi_tempered_loss import BiTemperedLogisticLoss\nos.chdir(cwd)\n\n# remove warnings (most of them are not critical and meant for debugging tensorflow)\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Mixed precision to speedup training.\n**You Need GPU compute capability of at least 7.0, Remove the below cell if that isn't the case.For more information refer __[this](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html)__**"},{"metadata":{"trusted":false},"cell_type":"code","source":"from tensorflow.keras.mixed_precision import experimental as mixed_precision\npolicy = mixed_precision.Policy('mixed_float16')\nmixed_precision.set_policy(policy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"work_dir = '../input/cassava-leaf-disease-classification/'\nos.listdir(work_dir) \ntrain_path = '/kaggle/input/cassava-leaf-disease-classification/train_images/'\n\ndata = pd.read_csv(work_dir + 'train.csv')\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"**Let's convert the labels to meaningful class names for better understanding.**"},{"metadata":{"trusted":false},"cell_type":"code","source":"f = open(work_dir + 'label_num_to_disease_map.json')\nreal_labels = json.load(f)\nreal_labels = {int(k):v for k,v in real_labels.items()}\n\ndata['class_name'] = data['label'].map(real_labels)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's see what classes are present in the database.**"},{"metadata":{"trusted":false},"cell_type":"code","source":"pd.DataFrame(real_labels.items(),columns=['Labels','Disease Name'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's have a looking on the distribution of the classes in the dataset**"},{"metadata":{"trusted":false},"cell_type":"code","source":"sns.set_style(\"whitegrid\")\nplt.figure(figsize=(15,6))\nsns.countplot(data[\"class_name\"], edgecolor=\"black\", palette=\"mako\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"perc_CMD = (data['label'].value_counts()[3]/np.sum(data['label'].value_counts()))*100\nprint(perc_CMD)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Key Insights:\n- Cassava Mosaic Disease (CMD) is the dominant class in the training set constituting **61.5%** of the total data.\n- The dataset is very skewed so there might be a need of upsampling or undersampling."},{"metadata":{},"cell_type":"markdown","source":"### Experiment 1\n\n**The test data for the competetion is private and only available after submitting out kernel. Lets test out whether the test set is skewed as well. For this we will predict all the data to be the dominant class: CMD (3).**"},{"metadata":{"trusted":false},"cell_type":"code","source":"# TEST_DIR = 'test_images/'\n# test_images = os.listdir(TEST_DIR)\n# predictions = []\n\n# for image in test_images:\n# predictions.append(3)\n\n# sub = pd.DataFrame({'image_id': test_images, 'label': predictions})\n# sub.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Surprisingly The score came out to be 0.60,This means even the test set is very much skewed, CMD seems to be a very common disease**\n\n\n\n****"},{"metadata":{},"cell_type":"markdown","source":"### Let's Have a look at the images of different classes\n**Label 0 : Cassava Bacterial Blight (CBB)**"},{"metadata":{"trusted":false},"cell_type":"code","source":"df0 = data[data[\"label\"] == 0]\nfiles = df0[\"image_id\"].sample(4).tolist()\n\nplt.figure(figsize=(15,5))\nindex = 0\nfor file in files:\n image = Image.open(train_path + file)\n plt.subplot(1, 4, index + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n index += 1\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Label 1: Cassava Brown Streak Disease (CBSD)**"},{"metadata":{"trusted":false},"cell_type":"code","source":"df0 = data[data[\"label\"] == 1]\nfiles = df0[\"image_id\"].sample(4).tolist()\n\nplt.figure(figsize=(15,5))\nindex = 0\nfor file in files:\n image = Image.open(train_path + file)\n plt.subplot(1, 4, index + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n index += 1\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Label 2: Cassava Green Mottle (CGM)**"},{"metadata":{"trusted":false},"cell_type":"code","source":"df0 = data[data[\"label\"] == 2]\nfiles = df0[\"image_id\"].sample(4).tolist()\n\nplt.figure(figsize=(15,5))\nindex = 0\nfor file in files:\n image = Image.open(train_path + file)\n plt.subplot(1, 4, index + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n index += 1\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Label 3: Cassava Mosaic Disease (CMD)**"},{"metadata":{"trusted":false},"cell_type":"code","source":"df0 = data[data[\"label\"] == 3]\nfiles = df0[\"image_id\"].sample(4).tolist()\n\nplt.figure(figsize=(15,5))\nindex = 0\nfor file in files:\n image = Image.open(train_path + file)\n plt.subplot(1, 4, index + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n index += 1\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Label 4: Healthy**"},{"metadata":{"trusted":false},"cell_type":"code","source":"df0 = data[data[\"label\"] == 4]\nfiles = df0[\"image_id\"].sample(4).tolist()\n\nplt.figure(figsize=(15,5))\nindex = 0\nfor file in files:\n image = Image.open(train_path + file)\n plt.subplot(1, 4, index + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n index += 1\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note:** There are many noisy labels in our dataset. To mitigate this we will be using label smoothing and Bi-Tempered Logistic loss, as we will see in the later parts of the notebook."},{"metadata":{"trusted":false},"cell_type":"code","source":"# this is important, adjust image size to fit your memory.\nIMG_SIZE = 512\nsize = (IMG_SIZE,IMG_SIZE)\n# number of classes\nn_CLASS = 5\n# choose batch size according to available VRAM\nBATCH_SIZE = 16 #reduce this if you get a OOM(out of memory error)\n\n# number of epochs\nEPOCHS = 7","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image Augumentation\n\n#### What is it?\nGenerally in cases where the trainset is really small or mostly similar, we can articficially generate more train and test images by slightly changing the color, random rotation, shifts, shear and flips etc. Be careful not to change the image too much as it can be misleading for our model.\nImage augumentation on small datasets boosts the models performace.\n\n#### Why is it important in this case?\nThis dataset is not very small but not very large either. Mostly I use this because this gave a boost in Leaderboard score and Cross Validation accuracy. This can be experimental as well. It is a good practice to do image augumentation for better generalization."},{"metadata":{},"cell_type":"markdown","source":"**Install ImageDataAugumentor**\n\nThis is only one time step if on a local machine, If on kaggle kernel/colab notebook, do this every new session."},{"metadata":{"trusted":false},"cell_type":"code","source":"!pip install git+https://github.com/mjkvaak/ImageDataAugmentor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will use **Albumentations** library, It is a very popular image augumentation library.\nFor a quickstart guide follow [this](https://analyticsindiamag.com/hands-on-guide-to-albumentation/)"},{"metadata":{"trusted":false},"cell_type":"code","source":"from ImageDataAugmentor.image_data_augmentor import *\nimport albumentations as A\n\n\n# add as many alteration you want here, dont meddle with validation data much \n# as we won't be doing TTA(test time augumentation)for now. TTA seems to imrove LB score, but I will skip this here.\n\ntrain_augmentations = A.Compose([\n A.Transpose(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.HueSaturationValue(\n hue_shift_limit=0.2,\n sat_shift_limit=0.2,\n val_shift_limit=0.2,\n p=0.5),\n A.RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1),\n contrast_limit=(-0.1, 0.1),\n p=0.5),\n A.CoarseDropout(p=0.5),\n A.Cutout(p=0.5),\n A.Flip(p=0.5),\n A.ShiftScaleRotate(p=0.5),\n A.HueSaturationValue(p=0.5, hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2),\n A.RandomBrightnessContrast(p=0.5, brightness_limit=(-0.2,0.2), contrast_limit=(-0.2, 0.2)),\n A.RandomCrop(IMG_SIZE, IMG_SIZE, p=1),\n# A.Normalize(\n# mean=[0.485, 0.456, 0.406],\n# std=[0.229, 0.224, 0.225],\n# max_pixel_value=255.0,\n# p=1.0)\n ], p=1)\n\nval_augmentations = A.Compose([\n A.CenterCrop(IMG_SIZE, IMG_SIZE, p=1),\n# A.Normalize(\n# mean=[0.485, 0.456, 0.406],\n# std=[0.229, 0.224, 0.225],\n# max_pixel_value=255.0,\n# p=1.0)\n ], p=1)\n\n# Train and validation data Generators\ndatagen_train = ImageDataAugmentor(augment=train_augmentations)\ndatagen_val = ImageDataAugmentor(augment=val_augmentations)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now let's feed data to above generator**"},{"metadata":{"trusted":false},"cell_type":"code","source":"def DataGenerator(train,val):\n train_set = datagen_train.flow_from_dataframe(train,\n directory = train_path,\n seed=7,\n x_col = 'image_id',\n y_col = 'class_name',\n target_size = size,\n class_mode = 'categorical',\n # interpolation = 'nearest', #avoid this when using albumentation(interpolation integer error)\n shuffle = True,\n batch_size = BATCH_SIZE)\n\n val_set = datagen_val.flow_from_dataframe(val,\n directory = train_path,\n seed=7,\n x_col = 'image_id',\n y_col = 'class_name',\n target_size = size,\n class_mode = 'categorical',\n # interpolation = 'nearest',\n batch_size = BATCH_SIZE)\n return train_set,val_set","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now let's check some images after augumentation**"},{"metadata":{"trusted":false},"cell_type":"code","source":"train_temp = data.iloc[:10]\nval_temp = data.iloc[-10:]\ntrain_set_temp,val_set_temp = DataGenerator(train_temp,val_temp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_images, _ = next(train_set_temp)\n\nplt.figure(figsize=(14, 14))\nfor i in range(9):\n image = train_images[i]\n ax = plt.subplot(3, 3, i + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Note:** While using this to display images beware that your batchsize is less than or equal to the total subplots in the above code."},{"metadata":{"trusted":false},"cell_type":"code","source":"val_images, _ = next(val_set_temp)\n\nplt.figure(figsize=(14, 14))\nfor i in range(9):\n image = train_images[i]\n ax = plt.subplot(3, 3, i + 1)\n plt.imshow(image)\n plt.axis(\"off\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Building (Transfer Learning)"},{"metadata":{},"cell_type":"markdown","source":"### Label Smoothing\n\nAs I pointed out earlier these data labels are noisy. Label smooting is a technique useed to cope up with such datasets. We will also see other solutions such as using Bi-Tempered Logistic loss as loss function in place of the categorical cross entropy that we use in general.\n\nLabel smoothing can be used with the inbuilt Categorical_crossentropy function that keras has.\n\n#### How Label smoothing works?\nBasically we say that we arent 100% confident in our labels. For example: For a digit recognition task, an image of 9 will have a one-hot encoded vector as follows [0,0,0,0,0,0,0,0,1]. This makes sense as well.\nLet's say we aren't sure that our labels are correct in all the datapoints. lets say we use label smoothing with a value of 0.11\n\nnew_labels = onehot_labels * (1 - label_smoothing) + label_smoothing / num_classes\n\nnew_labels = [0,0,0,0,0,0,0,0,1]*(0.89)+0.012\nnew_labels = [0.012,0.012,0.012,0.012,0.012,0.012,0.012,0.012,0.902]\n\nThis means that we are 90% confident in our labels.\nThe general rule of thumb is to take label smoothing as 1/n_classes. but this is value that must be derived after several experiments.\n\nFor more info on label smoothing : Refer [this](https://towardsdatascience.com/what-is-label-smoothing-108debd7ef06)\n\n### K- Fold Cross Validation\n\nWe will also use K-Fold cross validation to train our model. We will use the stratifiedKfold function because of the skew in out data. This will preserve the ratio of each class in each fold.\n\n\n\n\nFor more infomation refer [this](https://medium.com/datadriveninvestor/k-fold-cross-validation-6b8518070833)"},{"metadata":{},"cell_type":"markdown","source":"**Let's create all our models here.**"},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_model_resnet50():\n \n model = Sequential()\n model.add(ResNet50(input_shape = (IMG_SIZE, IMG_SIZE, 3), include_top = False, weights = 'imagenet'))\n model.add(GlobalAveragePooling2D())\n model.add(Dense(512, activation = 'relu', bias_regularizer=tf.keras.regularizers.L1L2(l1=0.01, l2=0.001)))\n model.add(Dropout(0.7))\n model.add(Dense(n_CLASS, activation = 'softmax',dtype='float32')) #this is very important to use mixed_precision\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_model_vgg16():\n \n model = Sequential()\n model.add(VGG16(input_shape = (IMG_SIZE, IMG_SIZE, 3), include_top = False, weights = 'imagenet'))\n model.add(GlobalAveragePooling2D())\n model.add(Dense(512, activation = 'relu', bias_regularizer=tf.keras.regularizers.L1L2(l1=0.01, l2=0.001)))\n model.add(Dropout(0.5))\n model.add(Dense(n_CLASS, activation = 'softmax',dtype='float32')) #this is very important to use mixed_precision\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_model_effnetb4():\n \n model = Sequential()\n model.add(efn.EfficientNetB4(input_shape = (IMG_SIZE, IMG_SIZE, 3), include_top = False, weights = 'noisy-student'))\n model.add(GlobalAveragePooling2D())\n model.add(Dense(256, activation = 'relu', bias_regularizer=tf.keras.regularizers.L1L2(l1=0.01, l2=0.001)))\n model.add(Dropout(0.7))\n model.add(Dense(n_CLASS, activation = 'softmax',dtype='float32')) #this is very important to use mixed_precision\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_model_effnetb0():\n \n model = Sequential()\n model.add(EfficientNetB0(input_shape = (IMG_SIZE, IMG_SIZE, 3), include_top = False, weights = 'imagenet'))\n model.add(GlobalAveragePooling2D())\n# model.add(Dense(64, activation = 'relu', bias_regularizer=tf.keras.regularizers.L1L2(l1=0.01, l2=0.001)))\n model.add(Dropout(0.2))\n model.add(Dense(n_CLASS, activation = 'softmax',dtype='float32')) #this is very important to use mixed_precision\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Crossentropy Loss Function\n\nCross Entropy is defined as:\n\n
\n\nIn our Case n = 5. To understand this more deeply refer [This](https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e)"},{"metadata":{"trusted":false},"cell_type":"code","source":"loss = tf.keras.losses.CategoricalCrossentropy(from_logits = True,\n label_smoothing=0.2,\n name='categorical_crossentropy' )\n# loss = BiTemperedLogisticLoss(t1=0.6, t2=1.4)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"n_splits = 5\nfold_no = 0\n\nkfold = StratifiedKFold(n_splits=n_splits, shuffle=True,random_state=7)\nfor train_index, val_index in kfold.split(data,data['class_name']):\n print(f\"Fold number : {fold_no+1}/{n_splits}\")\n train,val = data.iloc[list(train_index)],data.iloc[list(val_index)]\n train_set,val_set = DataGenerator(train,val)\n \n # useful when predicting, we can ensemble best model from each fold. \n \n # change models here, rest all is same\n model = create_model_effnetb4() \n \n \n model_name = 'effnetb4'\n fold_name = '-fold.h5'\n filepath = model_name + str(fold_no+1) + fold_name\n \n # Adding callbacks\n # Earlystopping to save time\n es = EarlyStopping(monitor='val_loss', mode='min', patience=3,\n restore_best_weights=True, verbose=1)\n\n # Save the model with the minimum validation loss(best model for each fold)\n checkpoint_cb = ModelCheckpoint(filepath,\n save_best_only=True,\n monitor = 'val_loss',\n mode='min')\n\n # reduce learning rate\n reduce_lr = ReduceLROnPlateau(monitor = 'val_loss',\n factor = 0.3,\n patience = 2,\n min_lr = 1e-6,\n mode = 'min',\n verbose = 1)\n callbacks=[es, checkpoint_cb, reduce_lr]\n \n # We use the Adam optimizer\n model.compile(optimizer = Adam(learning_rate = 1e-3),\n loss = loss,\n metrics = ['categorical_accuracy']) \n \n history = model.fit(x = train_set,\n validation_data = val_set,\n epochs= EPOCHS,batch_size=BATCH_SIZE,\n callbacks=callbacks)\n fold_no += 1\n K.clear_session()\n if (fold_no) == n_splits:\n print(\"Training Finished\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference & Submission"},{"metadata":{},"cell_type":"markdown","source":"Make another Notebook for inference as Internet isn't allowed in submission notebook."},{"metadata":{"trusted":false},"cell_type":"code","source":"# models = []\n# image_size = 512\n# n_splits=2\n# model_name = 'effnetb4'\n# fold_name = '-fold.h5'\n# cwd = os.getcwd()\n# os.chdir('../input/newcassavadataset')\n# for i in range(n_splits):\n# effnetb4 = keras.models.load_model(model_name + str(i+1) + fold_name)\n# models.append(effnetb4)\n# os.chdir(cwd)\n# TEST_DIR = '../input/cassava-leaf-disease-classification/test_images/'\n# test_images = os.listdir(TEST_DIR)\n# preds = []\n# results = []\n\n# for image_id in test_images:\n# image = Image.open(os.path.join('../input/cassava-leaf-disease-classification', \"test_images\", image_id))\n# image = image.resize((image_size, image_size))\n# image = np.expand_dims(image, axis = 0)\n# for model in models:\n# p = model.predict(image)\n# preds.append(np.argmax(p))\n# print(p)\n# res = max(set(preds), key = preds.count)\n# results.append(res)\n\n# sub = pd.DataFrame({'image_id': test_images, 'label': results})\n# sub.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion and Results"},{"metadata":{},"cell_type":"markdown","source":"| Model Name | Image Size | Loss Metric | Label Smoothing | Validation Set Score | Leaderboard Score | Remarks | K-fold|\n|-|-|-|-|-|-|-|-|\n| Naive Model (Predict all images as dominant class) | 380 | N/A | N/A | N/A | 0.605 | Test set is skewed too | N/A |\n| ResNet-50 (imagenet weights) | 380 | Cross-Entropy | 0.0001 | 0.8745 | 0.605 | kaggle terminated after 9hrs, image size increase might help, freezing layers can be experimented with. | Test_split=0.2 |\n| ResNet-50 (imagenet) | 380 | Cross-Entropy | 0.0001 | Not Documented | 0.161 | this dataset is very different from imagenet database, freezing layers is useless. | test_split=0.2 |\n| VGG16 (imagenet) | 380 | Cross-Entropy | 0.0001 | 0.8550 | Not Evaluated | Kaggle terminated after 9hrs | test_split=0.2 |\n| EfficientNet B4 (imagenet) | 380 | Cross-Entropy | 0.1 | Not Documented | 0.830 | using 512 might be better. | test_split=0.25 |\n| EfficientNet B4 (imagenet) | 512 | Cross-Entropy | 0.2 | 0.8944 | 0.859 | Can be improved with kfold and little more augumentation | test_split=0.2 |\n| EfficientNet B4 (imagenet) | 512 | Cross-Entropy | 0.2 | 0.9010 | 0.874 | kaggle terminated after 9 hrs, submitted the checkpoint which gave this score | test_split=0.2 |\n| EfficientNet B4 (imagenet) | 512 | Cross-Entropy | 0.1 | Not documented | 0.832 | Using kfold takes a lot of time and training it on kaggle is hard | 4-Fold (No Ensembling) |\n| EfficientNet B0 (imagenet) | 512 | Cross-Entropy | 0.1 | Not Documented | 0.885 (top) | the lighter models work better, but will stick to B4 with kfold | N/A |\n| EfficientNet B4 (imagenet) |512 |Cross-Entropy |0.2 |Not Documented | 0.888 | The 3 fold process stopped midway,I used the first checkpointed mmodel to predict which actually gave a very good result |1/2 |"},{"metadata":{},"cell_type":"markdown","source":"## Future Work"},{"metadata":{},"cell_type":"markdown","source":"- Try noisy student weights instead of imagenet weights.\n- calculate mean and standard deviation manually for this dataset.\n- RAdam+Lookahead optimizer instead of Adam\n- Bi-Tempered Logistic Loss and Focal Cosine Loss instead of categorical cross entropy.\n- Ensemble different models.\n- Test Time Augumentation (TTA) is very optional as discussion threads say its useful but can mess things up.\n- Use the 2019 competetion model to improve the performance"},{"metadata":{},"cell_type":"markdown","source":"## References\n\n* [This awesome notebook that taught me Augumentations in Keras(Definitely Check this out)](https://www.kaggle.com/junyingsg/end-to-end-cassava-disease-classification-in-keras/notebook#Model-Building-and-Selection)\n* [Cross Entropy Made Easy](https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e)\n* [Kfold Reference](https://medium.com/datadriveninvestor/k-fold-cross-validation-6b8518070833)\n* [Label Smoothing](https://towardsdatascience.com/what-is-label-smoothing-108debd7ef06)\n* [More Information on Mixed Precision](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}