{"cells":[{"metadata":{},"cell_type":"markdown","source":"Firstly, I'd like to say that I'm novice in machine learning and don't know how this problem can be resolved. However I'm just curious how to work with such amount of data and if I can create something working. Here I'm going to build ML model using Keras.\n\nJust to save your time. I have managed to create model which achieved 0.0022% of accuracy after 8 hours of learning."},{"metadata":{},"cell_type":"markdown","source":"# Env initialization\n\nHere standart env initialization plus extra package imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.\n\nimport json\nimport seaborn as sns\n\nfrom tensorflow import keras\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.applications.resnet50 import ResNet50\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout, Conv2D, MaxPooling2D\nfrom tensorflow.keras.optimizers import Adam, RMSprop\n\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All parameters in one place:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_images_dir = '/kaggle/input/herbarium-2020-fgvc7/nybg2020/train/'\ntest_images_dir = '/kaggle/input/herbarium-2020-fgvc7/nybg2020/test/'\n\ntrain_metadata_file_path = '/kaggle/input/herbarium-2020-fgvc7/nybg2020/train/metadata.json'\ntest_metadata_file_path = '/kaggle/input/herbarium-2020-fgvc7/nybg2020/test/metadata.json'\n\nnum_classes = 32093 + 1\nbatch_size = 16\n\nsteps_per_epoch = int(num_classes / batch_size)\n\nimg_height = 1000\nimg_width = 661\n\nepochs_num = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data preprocessing\nFirst of all we have to look inside metadata files located in `train` and `test` subdirectories.\n\n## train/metadata.json\nLet's find out what's in that file. For some reason loading data from the file causes problems. We can avoid them by specifyin encoding explicitly and setting `errors='ignore'` option."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"with open(train_metadata_file_path, 'r', encoding='utf-8', errors='ignore') as f:\n    train_metadata_json = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After loading data in memory we can have a look on it."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Let's see presented keys\ntrain_metadata_json.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are no interesting information under 'info', 'licenses' keys. I'm going to create separate DataFrames for the rest keys in metadata file and merge them together nto one DataFrame."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create Pandas DataFrame per each data type\ntrain_metadata = pd.DataFrame(train_metadata_json['annotations'])\n\ntrain_categories = pd.DataFrame(train_metadata_json['categories'])\ntrain_categories.columns = ['family', 'genus', 'category_id', 'category_name']\n\ntrain_images = pd.DataFrame(train_metadata_json['images'])\ntrain_images.columns = ['file_name', 'height', 'image_id', 'license', 'width']\n\ntrain_regions = pd.DataFrame(train_metadata_json['regions'])\ntrain_regions.columns = ['region_id', 'region_name']\n\n#Combine DataFrames\ntrain_data = train_metadata.merge(train_categories, on='category_id', how='outer')\ntrain_data = train_data.merge(train_images, on='image_id', how='outer')\ntrain_data = train_data.merge(train_regions, on='region_id', how='outer')\n\n#Remove NaN values\ntrain_data = train_data.dropna()\n\n# Update data types\ntrain_data = train_data.astype({'category_id': 'int32',\n                                'id': 'int32',\n                                'image_id': 'int32',\n                                'region_id': 'int32',\n                                'height': 'int32',\n                                'license': 'int32',\n                                'width': 'int32'})\n\ntrain_data.info()\n\n#Save DataFrame for future usage.\ntrain_data.to_csv('train_data.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_categories\ndel train_images\ndel train_regions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## test/metadata.json\n\nLet's do the same with test/metadata.json file"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(test_metadata_file_path, 'r', encoding='utf-8', errors='ignore') as f:\n    test_metadata_json = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_metadata_json.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test metadata file contains only three entries:\n* images\n* info\n* licenses\n\nThat way we're interested only in `images`. Let's create DataFrame for it."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data = pd.DataFrame(test_metadata_json['images'])\n\ntest_data = test_data.astype({'height': 'int32',\n                              'id': 'int32',\n                              'license': 'int32',\n                              'width': 'int32'})\n\ntest_data.to_csv('test_data.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generators"},{"metadata":{},"cell_type":"markdown","source":"Dataset contain lots of images and quite heavy. It's impossible to load all images into memory for model fitting. As a workaround I'll try to use ImageDataGenerator from keras. It loads data 'on the fly' and shouldn't consumes a lot of memory."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen_without_augmentation = ImageDataGenerator(rescale=1./255)\ndatagen_with_augmentation = ImageDataGenerator(rescale=1./255, \n                                               featurewise_center=False,\n                                               samplewise_center=False,\n                                               featurewise_std_normalization=False,\n                                               samplewise_std_normalization=False,\n                                               zca_whitening=False,\n                                               rotation_range = 10,\n                                               zoom_range = 0.1,\n                                               width_shift_range=0.1,\n                                               height_shift_range=0.1,\n                                               horizontal_flip=True,\n                                               vertical_flip=False)\n\ntrain_datagen = datagen_with_augmentation.flow_from_dataframe(dataframe=train_data, \n                                                                 directory=train_images_dir, \n                                                                 x_col='file_name', \n                                                                 y_col='category_id',\n                                                                 class_mode=\"raw\",\n                                                                 batch_size=batch_size,\n                                                                 color_mode = 'rgb',\n                                                                 target_size=(img_height,img_width)\n                                                             )\n\nval_datagen = datagen_without_augmentation.flow_from_dataframe(dataframe=train_data, \n                                                                 directory=train_images_dir, \n                                                                 x_col='file_name', \n                                                                 y_col='category_id',\n                                                                 class_mode=\"raw\",\n                                                                 batch_size=batch_size,\n                                                                 color_mode = 'rgb',\n                                                                 target_size=(img_height,img_width))\n\n#test_datagen = datagen_without_augmentation.flow_from_dataframe(dataframe=test_data,\n#                                                               directory=test_images_dir,\n#                                                               x_col='file_name',\n#                                                               color_mode = 'rgb',\n#                                                               class_mode=None,\n#                                                               target_size=(img_height,img_width))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another problem I came across is that Keras expects targets as 'one hot' encoded vectors. But there are 32093 different categories. Storing such sparse vectors for more than 1 million images... takes too much memory. Unfortunately, I wasn't able to find any existing setting for solving that problem. I decided to wrap up ImageDataGenerator inside my own generator. That way I can retrieve portion of data posthandle it and yield as ImageDataGenerator does."},{"metadata":{"trusted":true},"cell_type":"code","source":"def generator_wrapper(generator, num_of_classes):\n    for (X_vals, y_vals) in generator:\n        Y_categorical = to_categorical(y_vals, num_classes=num_of_classes)\n        \n        yield (X_vals, Y_categorical)        \n        \ntrain_datagen_wrapper = generator_wrapper(train_datagen, num_classes)\nval_datagen_wrapper = generator_wrapper(val_datagen, num_classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{},"cell_type":"markdown","source":"I wasn't going to create model which can perform well on such amount of data just because I don't have enough computation resources. So here very simple model just to check that learning works."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(64, kernel_size=5, activation='relu', input_shape=(img_height, img_width, 3), padding='Same', strides=2))\nmodel.add(Conv2D(64, kernel_size=5, activation='relu', padding='Same', strides=2))\nmodel.add(MaxPooling2D(2, 2))\nmodel.add(Conv2D(128, kernel_size=3, activation='relu', padding='Same', strides=2))\nmodel.add(MaxPooling2D(2, 2))\nmodel.add(Dropout(0.25))\nmodel.add(Conv2D(128, kernel_size=3, activation='relu', padding='Same', strides=2))\nmodel.add(MaxPooling2D(2, 2))\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\nmodel.add(Dense(num_classes / 100))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(num_classes, activation='softmax'))\n\noptimizer = RMSprop(lr=0.001)\n\nmodel.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\nstart = time.time()\n\n# history = model.fit_generator(train_datagen_wrapper, \n#                               epochs=epochs_num, \n#                               validation_data=val_datagen_wrapper, \n#                               steps_per_epoch=steps_per_epoch, \n#                               validation_steps=steps_per_epoch)\n# \n\nend = time.time()\n\nprint(f\"\\nLearning took {end - start}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After 5 epochs and almost 8 hours of learning model reached 0.0022% of accuracy. Not so good but not as bad for such simple model and 32k different classes. Hope this might be helpful."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}