{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from fastai.vision.all import *\nimport os\nimport shutil\nimport cv2\nfrom sklearn.model_selection import train_test_split\nimport random","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With more than 2.5m images at 1000px it is impossible to quickly train models to try out ideas. What I want to be able to do is prototype models using a small subset of the data and then run the model on the full dataset *once* to generate a submission, since I don't want to use up hours and hours of GPU time for no reason. The challenge is ensuring that the subset is a good enough representation of the full dataset so that the modelling techniques used can be applied to the final run.\n\nI have been experimenting with a couple of different ways to achieve this as you will see below. For reference with around 20k 256px images I can train a resnet18 model in just under 2 mins/epoch on Colab Pro, so keep that in mind when choosing how many images to include in your sample.\n\nHopefully this is helpful and please let me know in the comments if you have any feedback/other ideas!","metadata":{}},{"cell_type":"code","source":"# Code via: https://www.kaggle.com/muhammadzubairkhan92/herbarium-2021-exploratory-data-analysis\nPATH_BASE = \"../input/herbarium-2021-fgvc8/\"\nPATH_TRAIN = os.path.join(PATH_BASE, \"train/\")\nPATH_TRAIN_META = os.path.join(PATH_TRAIN, \"metadata.json\")\n\nwith open(PATH_TRAIN_META) as json_file:\n    metadata = json.load(json_file)\n    \nids = []\ncategories = []\npaths = []\n\nfor annotation, image in zip(metadata[\"annotations\"], metadata[\"images\"]):\n    assert annotation[\"image_id\"] == image[\"id\"]\n    ids.append(image[\"id\"])\n    categories.append(annotation[\"category_id\"])\n    paths.append(image[\"file_name\"])\n        \ndf_meta = pd.DataFrame({\"id\": ids, \"category_id\": categories, \"file_name\": paths})\n\nd_categories = {category[\"id\"]: category[\"name\"] for category in metadata[\"categories\"]}\nd_families = {category[\"id\"]: category[\"family\"] for category in metadata[\"categories\"]}\nd_orders = {category[\"id\"]: category[\"order\"] for category in metadata[\"categories\"]}\n\ndf_meta[\"category_name\"] = df_meta[\"category_id\"].map(d_categories)\ndf_meta[\"family_name\"] = df_meta[\"category_id\"].map(d_families)\ndf_meta[\"order_name\"] = df_meta[\"category_id\"].map(d_orders)\n\ndf_meta.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Option 1: Sample based on order_name\n\nWhilst there are almost 65000 different herb species in the dataset, these can be grouped into just 81 different `order_name`s:","metadata":{}},{"cell_type":"code","source":"df_meta.groupby('order_name')['file_name'].count().sort_values().reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some of these have just one unique `category_id`, whilst others have thousands:","metadata":{}},{"cell_type":"code","source":"df_meta.groupby('order_name').agg({'category_id': 'nunique'}).sort_values('category_id').reset_index().rename(columns={'category_id':'unique_categories'})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"My idea was to train models to predict `order_name` instead of `category_id` using a subset of the data. This is obviously an easier problem than the actual task, but it could work assuming there are some similarities between species in each `order_name`.","metadata":{}},{"cell_type":"code","source":"# Code via: https://www.kaggle.com/muhammadzubairkhan92/herbarium-2021-exploratory-data-analysis\ndef visualize_train_batch(paths, categories, families, orders):\n    plt.figure(figsize=(16, 16))\n    \n    for ind, info in enumerate(zip(paths, categories, families, orders)):\n        path, category, family, order = info\n        \n        plt.subplot(1, 3, ind + 1)\n        \n        image = cv2.imread(os.path.join(PATH_TRAIN, path))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        plt.imshow(image)\n        \n        plt.title(\n            f\"FAMILY: {family} ORDER: {order}\\n{category}\", \n            fontsize=10,\n        )\n        plt.axis(\"off\")\n    \n    plt.show()\n    \ndef visualize_by_order(df, _order=None):\n    tmp = df.sample(3)\n    if _order is not None:\n        tmp = df[df[\"order_name\"] == _order].sample(3)\n\n    visualize_train_batch(\n        tmp[\"file_name\"].tolist(), \n        tmp[\"category_name\"].tolist(),\n        tmp[\"family_name\"].tolist(),\n        tmp[\"order_name\"].tolist(),\n    )\n    \ndef visualize_by_family(df, _family=None):\n    tmp = df.sample(3)\n    if _family is not None:\n        tmp = df[df[\"family_name\"] == _family].sample(3)\n\n    visualize_train_batch(\n        tmp[\"file_name\"].tolist(), \n        tmp[\"category_name\"].tolist(),\n        tmp[\"family_name\"].tolist(),\n        tmp[\"order_name\"].tolist(),\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_by_order(df_meta, 'Acorales')\nvisualize_by_order(df_meta, 'Myrtales')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can experiment with this more yourself, but my initial thoughts are that there are enough similarities between herbs in each `order_name` that this method could work.\n\nWe can take a stratified sample of the original dataset by grouping on `order_name`. The following code ensures that examples of each `order_name` are included, and limits the maximum number of each class to 300 (you can adjust this yourself to get a larger sample if you like).","metadata":{}},{"cell_type":"code","source":"max_n = 300\n\n# https://stackoverflow.com/questions/44114463/stratified-sampling-in-pandas\nsample = df_meta.groupby('order_name', group_keys=False).apply(lambda x: x.sample(min(len(x), max_n)))\n\nprint(len(sample))\nsample['order_name'].value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can copy the files we need over to a new folder in our working directory:","metadata":{}},{"cell_type":"code","source":"os.mkdir('./sample')\ntarget_dir = '/kaggle/working/sample'\n\nfiles = sample['file_name'].unique()\n\nfor file in files:\n    shutil.copy(os.path.join(PATH_TRAIN, file), target_dir)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`fastai` has a `resize_images` function that will resize all of the images in a folder for us:","metadata":{}},{"cell_type":"code","source":"resize_images(target_dir, max_size=256, dest=f'sample_resized')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can now zip the file and delete the original images (best for working on an external notebook server e.g. Colab). If you are jumping straight into modelling from here, you can skip this step.","metadata":{}},{"cell_type":"code","source":"# http://www.seanbehan.com/how-to-use-python-shutil-make_archive-to-zip-up-a-directory-recursively-including-the-root-folder/\ndef make_archive(source, destination):\n        base = os.path.basename(destination)\n        name = base.split('.')[0]\n        format = base.split('.')[1]\n        archive_from = os.path.dirname(source)\n        archive_to = os.path.basename(source.strip(os.sep))\n        print(source, destination, archive_from, archive_to)\n        shutil.make_archive(name, format, archive_from, archive_to)\n        shutil.move('%s.%s'%(name,format), destination)\n\nmake_archive('/kaggle/working/sample_resized', '/kaggle/working/sample_resized.zip')\nsample.to_csv('/kaggle/working/sample.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Delete original size sample images\nshutil.rmtree('/kaggle/working/sample')\n\n# Can keep this one if you are going straight into model building, otherwise better to delete before saving version/output\nshutil.rmtree('/kaggle/working/sample_resized')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Option 2: Sample the most common categories\nThis is obviously a harder problem to solve than before, since we will be predicting `category_id` instead of `order_name`. It will be more representative of the final task, however.","metadata":{}},{"cell_type":"code","source":"num_cats = 5000\n\nmost_freq = df_meta['category_id'].value_counts().head(num_cats).reset_index()\ntop_cats = df_meta[df_meta['category_id'].isin(most_freq['index'].unique())]\n\nprint(len(top_cats))\ntop_cats['category_id'].value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To cut this down further, you can either use `train_test_split` to generate a stratified sample, or randomly pick N categories from `top_cats`. We could of course only select a small `num_cats` above to get a reasonable dataset size straight away, but the risk is that we might not get enough variety in the images if all of the most common `category_id`s are similar looking plants.","metadata":{}},{"cell_type":"code","source":"# Note: if you make train_size too small you will lose some categories, since there won't be enough examples\nsample2, _ = train_test_split(top_cats, train_size=0.1, stratify=top_cats['category_id'])\n\nprint(len(sample2))\nsample2['category_id'].value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_cats_rand = 500\n\nif num_cats_rand > num_cats:\n    num_cats_rand = num_cats\n    \ncats = random.sample(list(top_cats['category_id'].unique()), num_cats_rand)\n\nsample3 = df_meta[df_meta['category_id'].isin(cats)]\n\nprint(len(sample3))\nsample3['category_id'].value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can copy the files and resize using the same process as before.","metadata":{}}]}