{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Herbarium via ResNet50 and 3-step classification\nOur data consists out of input images of Herbariums, each of which belongs to one of 309 families, one of 3,677 geni and one of 32,093 categories. Thus it is a good idea to first guess an images family, then its genus and finally infer from this its category.\n\nIn this notebook, we\n* crop the input images, because Herbariums seem to show pathologic margins that do not contain useful information,\n* use ResNet50 in order to 'preprocess' our input images,\n* use a Dense Neural Network after that to first make a guess for the family of the input,\n* then make a guess for the genus of the input via another Dense layer whose inputs are the guessed family and the ResNet50 output,\n* finally guess the category of the input via another Dense layer, whose inputs are the guessed family and genus as well as the output of ResNet50,\n* activate training for genus and category, only after the network is trained well on for the family, then genus, and only in the end start to train inference of category.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We start by defining the function to load the data. Set debug=False in order to use the full data set, otherwise only a part of the 309 families will be used.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"batchsize=256\nlearning_rate=0.01\nepochs=5\nshape = (200, 136, 3) # original height = 1000, width= 682    219673 667    212347 676    190476\ndebug=True\naugment_data = False\nimport numpy as np, pandas as pd, tensorflow as tf \nimport time, os, json, codecs\nfrom sklearn.model_selection import train_test_split\nt_start = time.time()\nif not os.path.isdir('weights'):\n    os.makedirs('weights')\ndef load_data():\n    with codecs.open('/kaggle/input/herbarium-2020-fgvc7/nybg2020/train/metadata.json','r',encoding='utf-8',errors='ignore') as f:\n        train_meta = json.load(f)\n    with codecs.open('/kaggle/input/herbarium-2020-fgvc7/nybg2020/test/metadata.json','r',encoding='utf-8',errors='ignore') as f:\n        test_meta = json.load(f)\n    train_annotations = pd.DataFrame(train_meta['annotations'])\n    categories = pd.DataFrame(train_meta['categories'])\n    categories.columns = ['family', 'genus', 'category_id', 'category_name']\n    train_images = pd.DataFrame(train_meta['images'])\n    train_images.columns = ['file_name', 'height', 'image_id','license','width']\n    X_test = pd.DataFrame(test_meta['images'])\n    X_test.columns = ['file_name', 'height', 'image_id','license','width']\n    X_test = X_test[['image_id','file_name']]\n    regions = pd.DataFrame(train_meta['regions'])\n    regions.columns=['region_id','name']\n    Xorig = train_annotations.merge(categories,on='category_id', how=\"left\"\n                                     ).merge(train_images, on=\"image_id\", how=\"outer\"\n                                            ).merge(regions, on=\"region_id\", how=\"outer\")\n    X = Xorig[['file_name','family','genus','category_id']]\n    \n    name_list = X['family'].unique().tolist()\n    X.loc[:,'family'] = X['family'].map(lambda x:name_list.index(x))\n    genus_list = X['genus'].unique().tolist()\n    X.loc[:,'genus'] = X['genus'].map(lambda x:genus_list.index(x))\n    if debug:\n        X=X[X['family']>290]\n    return X.astype({'family':'int16','genus':'int16','category_id':'int16'}), X_test\n\nX,X_test=load_data()\nnmb_cat = X['category_id'].max()+1\nnmb_gen = X['genus'].max()+1\nnmb_fam = X['family'].max()+1\nX_train, X_dev = train_test_split(X, test_size=0.05, shuffle=True, random_state=13)\ndel X","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Augmentation\nThe data augmentation here is a little tricky, because we **first** want to crop away the pathological margins of the images, and then apply the data augmentation (rotation, shifts, zooms). This is particular important here, as we have very few data for some categories (for some, there are only ~5 images).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator\nimport matplotlib.pyplot as plt\ngnrt1 = ImageDataGenerator(dtype='uint8')\ngnrt2 = ImageDataGenerator(rotation_range=35,featurewise_center=False,\n                                      featurewise_std_normalization=False,\n                                      width_shift_range=0.1,\n                                      height_shift_range=0.1,\n                                      zoom_range=0.1,horizontal_flip=True,\n                                      dtype='uint8')\n    \ndef crop(batch_x):\n    cut1 = int(0.1*batch_x.shape[1])\n    cut2 = int(0.05*batch_x.shape[2])\n    return batch_x[:,cut1:-cut1,cut2:-cut2]\n\ndef crop_generator(batches,test=False):\n    while True:\n        if test:\n            batch_x = next(batches)\n            yield next(gnrt2.flow(crop(batch_x),batch_size=batchsize))\n        else:\n            batch_x, batch_y = next(batches)\n            yield (next(gnrt2.flow(crop(batch_x),batch_size=batchsize)), batch_y)\n            \n    \ni=0\nfor x, y in crop_generator(gnrt1.flow_from_dataframe(\n                                    dataframe=X_train[:2], directory='../input/herbarium-2020-fgvc7/nybg2020/train/',\n                                    x_col=\"file_name\", y_col=['family','genus','category_id'], class_mode=\"multi_output\",\n                                    target_size=(shape[0],shape[1]),batch_size=batchsize,\n                                    validate_filenames=False, verbose=False)):\n    plt.imshow(x.astype('uint8')[1])\n    i=i+1\n    if i==1:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The model\nWe use tensorflow.keras and import ResNet50 as a 'preprocessing layer', that we will not train to keep the model as simple as possible. It can be seen as a transformation of the input image that extracts features usefull in human vision of images.\n\nIt is followed by the three Dense layers 'family', 'genus' and 'category' with skip connections.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.layers import Dense, Input, concatenate, Flatten\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.utils import plot_model\n\ndef create_model():\n    actual_shape = (crop(np.zeros((1,shape[0],shape[1],shape[2]))).shape)[1:]\n    i = Input(actual_shape)\n    x = ResNet50(weights='imagenet', include_top=False, input_shape=actual_shape, pooling='max')(i)\n    x = Flatten()(x)\n    o1 = Dense(nmb_fam, name=\"family\", activation='softmax')(x)\n    o2 = concatenate([x,o1])\n    o2 = Dense(nmb_gen, name=\"genus\", activation='softmax')(o2)\n    o3 = concatenate([x,o1,o2])\n    o3 = Dense(nmb_cat, name=\"category_id\", activation='softmax')(o3)\n    model = Model(inputs=i,outputs=[o1,o2,o3])\n    model.layers[1].trainable = False\n    model.get_layer('genus').trainable = False\n    model.get_layer('category_id').trainable = False\n    return model\n\ndef compile(model,learning_rate=0.005):\n    model.compile(optimizer=Adam(learning_rate=0.005),loss=[\"sparse_categorical_crossentropy\",\n                                     \"sparse_categorical_crossentropy\",\n                                     \"sparse_categorical_crossentropy\"],\n                                metrics=['accuracy'])\n    \n\n\nTRAINSTEPS = (X_train.shape[0]//batchsize)+1\nVALSTEPS = (X_dev.shape[0]//batchsize)+1\n\ndef train(ep,initial_epoch=0):\n    return model.fit_generator(   gnrt1.flow_from_dataframe(\n                                    dataframe=X_train, directory='../input/herbarium-2020-fgvc7/nybg2020/train/',\n                                    x_col=\"file_name\", y_col=['family','genus','category_id'], class_mode=\"multi_output\",\n                                    target_size=(shape[0],shape[1]),batch_size=batchsize,\n                                    validate_filenames=False, verbose=False),\n                    validation_data=gnrt1.flow_from_dataframe(dataframe=X_dev, directory='../input/herbarium-2020-fgvc7/nybg2020/train/',\n                                    x_col=\"file_name\", y_col=['family','genus','category_id'], class_mode=\"multi_output\",\n                                    target_size=(shape[0],shape[1]),batch_size=batchsize,\n                                    validate_filenames=False, verbose=False),\n                    epochs=ep+initial_epoch,max_queue_size=30, workers=16, #use_multiprocessing=True,\n                               initial_epoch=initial_epoch,\n                               steps_per_epoch=TRAINSTEPS,\n                               validation_steps=VALSTEPS\n                   )\nmodel = create_model()\ncompile(model,learning_rate)\n#model.summary()\nplot_model(model, show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training\nAt first only the 'family' layer is trained. As soon as its validation accuracy crosses the threshold of 70%,, we start to train also the 'genus' layer. When family is over 90%, we stop to train it.\nSimilarly, we start training 'category' only when the validation accuracy of 'genus' goes over 70%.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"t_before = time.time()\nfor i in range(epochs):\n    hist = train(1,i)\n    print(\"Time used for epoch {}: {} min\".format(i+1,int((time.time()-t_before)/60)))\n    gacc = hist.history['genus_accuracy'][0]\n    facc = hist.history['family_accuracy'][0]\n    if facc > 0.9:\n        model.get_layer(\"family\").trainable=False\n        print(\"Stopped training family.\")\n        compile(model,learning_rate)\n    if facc > 0.7:\n        model.get_layer(\"genus\").trainable=True\n        print(\"Training genus now.\")\n        compile(model,learning_rate)\n    if gacc > 0.9:\n        model.get_layer(\"genus\").trainable=False\n        print(\"Stopped training genus.\")\n        compile(model,learning_rate)\n    if gacc >0.7:\n        model.get_layer(\"category_id\").trainable = True\n        print(\"Training category now.\")\n        compile(model,learning_rate)\nfilename=\"weights.h5\" #filename = 'weights/weights_{}.h5'.format(time.strftime(\"%Y%m%d-%H%M%S\"))\nmodel.save_weights(filename)\nprint(\"Weights saved to {}\".format(filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Making the predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(X_test):\n    STEPS_PREDICT = (X_test.shape[0]//batchsize)+1\n    predictions = model.predict_generator(crop_generator(gnrt1.flow_from_dataframe(\n                                    dataframe=X_test, directory='../input/herbarium-2020-fgvc7/nybg2020/test/',\n                                    x_col=\"file_name\",class_mode=None,\n                                    target_size=(shape[0],shape[1]),batch_size=batchsize,\n                                    validate_filenames=False, verbose=True),True),\n                                     steps=STEPS_PREDICT, workers=8, use_multiprocessing=True)\n\n    #submission = pd.DataFrame()\n    #submission['Id'] = X_test['image_id']\n    #submission['Predicted'] = predictions[2].argmax(axis=1)\n    #submission.to_csv('submission.csv', index=False)\n#print(\"Submission file written. Total time elapsed: {} minutes\".format((time.time()-t_start)//60))\npredictions =predict(X_test)\n\nfor i in range(100):\n        if y_test[i] == np.argmax(predictions[i]):\n            pass\n        else:\n            print(\"Element:\", i, \" -- [Value, Predicted] = [\", y_test[i], ',', np.argmax(predictions[i]),']')\n\nfig = plt.figure()\nplt.imshow(X_test[92], cmap='gray', interpolation='none')\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}