{"cells":[{"metadata":{},"cell_type":"markdown","source":"# iNaturalist 2019 EDA + DL\n\nAs part of the FGVC6 workshop at CVPR 2019, Kaggle is conducting the iNat Challenge 2019, the large scale species classification competition, sponsored by Microsoft. It is estimated that the natural world contains several million species of plants and animals. Without expert knowledge, many of these species are extremely difficult to accurately classify due to their visual similarity. The goal of this competition is to push the state of the art in automatic image classification for real world data that features a large number of fine-grained categories.\n\nThis Kernel will use the idea of \"Transfer Learning\", various pre-trained model will be used be used for the problem of multiclassification.\n\nKudos and main ideas / reference: \n\n- [hsinwenchang/keras-data-augmentation-visualize](https://www.kaggle.com/hsinwenchang/keras-data-augmentation-visualize/notebook)\n- [ateplyuk/inat2019-starter-keras-efficientnet](https://www.kaggle.com/ateplyuk/inat2019-starter-keras-efficientnet)"},{"metadata":{},"cell_type":"markdown","source":"![](https://www.bahai.org/chrome/img/beliefs/nature-feature-img.jpg?f0550045)\n\n[image-source](https://www.bahai.org/chrome/img/beliefs/nature-feature-img.jpg?f0550045)"},{"metadata":{},"cell_type":"markdown","source":"### Loading the necessary libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nprint(os.listdir(\"../input\"))\n\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nimport json\nfrom keras.models import Sequential, Model\nfrom keras.layers import Dense, Flatten, Activation, Dropout, GlobalAveragePooling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras import optimizers, applications\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, TensorBoard, EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"ann_file = '../input/train2019.json'\nwith open(ann_file) as data_file:\n        train_anns = json.load(data_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_anns_df = pd.DataFrame(train_anns['annotations'])[['image_id','category_id']]\ntrain_img_df = pd.DataFrame(train_anns['images'])[['id', 'file_name']].rename(columns={'id':'image_id'})\ndf_train_file_cat = pd.merge(train_img_df, train_anns_df, on='image_id')\ndf_train_file_cat['category_id']=df_train_file_cat['category_id'].astype(str)\ndf_train_file_cat.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_file_cat.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(df_train_file_cat['category_id'].unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Example of images for category_id = 400\nimg_names = df_train_file_cat[df_train_file_cat['category_id']=='400']['file_name'][:30]\n\nplt.figure(figsize=[15,15])\ni = 1\nfor img_name in img_names:\n    img = cv2.imread(\"../input/train_val2019/%s\" % img_name)[...,[2, 1, 0]]\n    plt.subplot(6, 5, i)\n    plt.imshow(img)\n    i += 1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Validation data"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_ann_file = '../input/val2019.json'\nwith open(valid_ann_file) as data_file:\n        valid_anns = json.load(data_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_anns_df = pd.DataFrame(valid_anns['annotations'])[['image_id','category_id']]\nvalid_anns_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_img_df = pd.DataFrame(valid_anns['images'])[['id', 'file_name']].rename(columns={'id':'image_id'})\nvalid_img_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_valid_file_cat = pd.merge(valid_img_df, valid_anns_df, on='image_id')\ndf_valid_file_cat['category_id']=df_valid_file_cat['category_id'].astype(str)\ndf_valid_file_cat.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nb_classes = 1010\nbatch_size = 128\nimg_size = 150\nnb_epochs = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Oversampling"},{"metadata":{"trusted":true},"cell_type":"code","source":"#from imblearn.over_sampling import RandomOverSampler\n\n#ros = RandomOverSampler(random_state=0)\n#X_resampled, y_resampled = ros.fit_resample(df_train_file_cat[[\"image_id\", \"file_name\"]], df_train_file_cat[\"category_id\"])\n\n#train_df = pd.DataFrame(X_resampled, columns=[\"image_id\", \"file_name\"])\n#train_df[\"category_id\"] = y_resampled","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"here I applied Data Augmentation technic from [Udacity](https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l05c04_exercise_flowers_with_data_augmentation_solution.ipynb#scrollTo=UOoVpxFwVrWy) as following:\n- random 45 degree rotation\n- random zoom of up to 50%\n- random horizontal flip\n- width shift of 0.15\n- height shfit of 0.15"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain_datagen=ImageDataGenerator(rescale=1./255, rotation_range=45, \n                    width_shift_range=.15, \n                    height_shift_range=.15, \n                    horizontal_flip=True, \n                    zoom_range=0.5)\n\ntrain_generator=train_datagen.flow_from_dataframe(\n    dataframe=df_train_file_cat,\n    directory=\"../input/train_val2019\",\n    x_col=\"file_name\",\n    y_col=\"category_id\",\n    batch_size=batch_size,\n    shuffle=True,\n    class_mode=\"categorical\",    \n    target_size=(img_size,img_size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# udacity_intro_to_tensorflow_for_deep_learning/l05c04_exercise_flowers_with_data_augmentation_solution.ipynb#scrollTo=jqb9OGoVKIOi\n# This function will plot images in the form of a grid with 1 row and 5 columns where images are placed in each column.\ndef plotImages(images_arr):\n    fig, axes = plt.subplots(1, 5, figsize=(20,20))\n    axes = axes.flatten()\n    for img, ax in zip( images_arr, axes):\n        ax.imshow(img)\n    plt.tight_layout()\n    plt.show()\n    \n    \naugmented_images = [train_generator[0][0][0] for i in range(5)]\nplotImages(augmented_images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntest_datagen = ImageDataGenerator(rescale=1./255)\n\nvalid_generator=test_datagen.flow_from_dataframe(    \n    dataframe=df_valid_file_cat,    \n    directory=\"../input/train_val2019\",\n    x_col=\"file_name\",\n    y_col=\"category_id\",\n    batch_size=batch_size,\n    shuffle=False,\n    class_mode=\"categorical\",    \n    target_size=(img_size,img_size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model\n\nI have applied various transfer learning models in order to see which has the best performance, since the numerous categorical data and the little number of cases "},{"metadata":{"trusted":true},"cell_type":"code","source":"#from keras.applications.vgg16 import VGG16\n#from keras.applications.inception_v3 import InceptionV3\n#from keras.applications.inception_resnet_v2 import InceptionResNetV2\n#from keras.applications.nasnet import NASNetLarge\n#from keras.applications.densenet import DenseNet121\nfrom keras.applications.xception import Xception\n\n#model = VGG16(weights='imagenet', include_top=False, input_shape=(img_size, img_size, 3))\nmodel = Xception(weights='imagenet', include_top=False, input_shape=(img_size, img_size, 3))\nmodel_name = \"Xception\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Adding custom layers \nmodel_final = Sequential()\nmodel_final.add(model)\nmodel_final.add(Flatten())\nmodel_final.add(Dense(1024, activation='relu'))\nmodel_final.add(Dropout(0.5))\nmodel_final.add(Dense(nb_classes, activation='softmax'))\n\nmodel_final.compile(optimizers.rmsprop(lr=0.0001, decay=1e-5),loss='categorical_crossentropy',metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Callbacks\n\ncheckpoint = ModelCheckpoint(model_name, monitor='val_loss', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1)\nearly = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_final.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nhistory = model_final.fit_generator(generator=train_generator, \n                    steps_per_epoch=80,\n                    validation_data=valid_generator,\n                    validation_steps=40,\n                    epochs=nb_epochs,\n                    callbacks = [checkpoint, early],                \n                    verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('history.json', 'w') as f:\n    json.dump(history.history, f)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df[['loss', 'val_loss']].plot()\nhistory_df[['acc', 'val_acc']].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ann_file = '../input/test2019.json'\nwith open(test_ann_file) as data_file:\n        test_anns = json.load(data_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_img_df = pd.DataFrame(test_anns['images'])[['id', 'file_name']].rename(columns={'id':'image_id'})\ntest_img_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntest_generator = test_datagen.flow_from_dataframe(      \n    \n        dataframe=test_img_df,    \n    \n        directory = \"../input/test2019\",    \n        x_col=\"file_name\",\n        target_size = (img_size,img_size),\n        batch_size = 1,\n        shuffle = False,\n        class_mode = None\n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\npredict_valid=model_final.predict_generator(valid_generator, steps = np.ceil(valid_generator.samples / valid_generator.batch_size), verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_valid_class=np.argmax(predict_valid,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(predict_valid_class)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\n\n#print(classification_report(valid_generator.classes, predict_valid_class))\nprint(accuracy_score(valid_generator.classes, predict_valid_class))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntest_generator.reset()\npredict=model_final.predict_generator(test_generator, steps = len(test_generator.filenames), verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_class_indices=np.argmax(predict,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = (train_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\npredictions = [labels[k] for k in predicted_class_indices]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sam_sub_df = pd.read_csv('../input/kaggle_sample_submission.csv')\nsam_sub_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sam_sub_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames=test_generator.filenames\nresults=pd.DataFrame({\"file_name\":filenames,\n                      \"predicted\":predictions})\ndf_res = pd.merge(test_img_df, results, on='file_name')[['image_id','predicted']]\\\n    .rename(columns={'image_id':'id'})\n\ndf_res.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_res.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}