{"metadata": {"kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}, "language_info": {"version": "3.6.3", "pygments_lexer": "ipython3", "codemirror_mode": {"version": 3, "name": "ipython"}, "mimetype": "text/x-python", "name": "python", "file_extension": ".py", "nbconvert_exporter": "python"}}, "nbformat_minor": 1, "nbformat": 4, "cells": [{"metadata": {"_uuid": "7efaaa323da478613572a6c3ecf4067c98149858", "_cell_guid": "dcc89c68-90c8-41a5-9d54-a39e08615024", "collapsed": true}, "cell_type": "markdown", "source": ["# Data visualization kernel\n", "\n", "Here is a new image-based competition hosted by a french e-commerce company *Cdiscount*. \n", "\n", "Dataset announced features: \n", "- Almost 9 million products: half of the current catalogue\n", "- More than 15 million images at 180x180 resolution\n", "- More than 5000 categories: yes this is quite an extreme multi-class classification!\n", "\n", "**Let's explore this in details**\n", "\n", "## Content\n", "\n", "- First images in train and test datasets\n", "- Random item access\n", "- Explore categories\n", "\n", "\n", "PS. Thanks to [this](https://www.kaggle.com/inversion/processing-bson-files) and [this](https://www.kaggle.com/bguberfain/just-showing-a-few-images) very helpful kernels !\n"]}, {"metadata": {"_uuid": "2d9e7efed1f9cff63413b9c83128f76ccac345ba", "_cell_guid": "ac48f0d2-88ff-4c3e-86f4-7391ef31f3fd", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["import os\n", "import sys\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import bson\n", "import cv2\n", "import matplotlib.pyplot as plt"]}, {"metadata": {"_uuid": "8a253017f259f70c781c911c3c534ba8ece025e2", "_cell_guid": "92a319a1-586d-4f12-a4cb-b440c2d158ab", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["INPUT_PATH = os.path.join('..', 'input')\n", "CATEGORY_NAMES_DF = pd.read_csv(os.path.join(INPUT_PATH, 'category_names.csv'))\n", "TRAIN_DB = bson.decode_file_iter(open(os.path.join(INPUT_PATH, 'train.bson'), 'rb'))\n", "TEST_DB = bson.decode_file_iter(open(os.path.join(INPUT_PATH, 'test.bson'), 'rb'))"]}, {"metadata": {"_uuid": "b1c216948cdd46b0748e7e9d5a52833512809ef8", "_cell_guid": "91cc08bf-60b0-4f97-b40e-28ae9c57abc5"}, "cell_type": "markdown", "source": ["## First images in train and test datasets"]}, {"metadata": {"_uuid": "6502d8a9dadadb949d8988b0201e06344168779a", "_cell_guid": "632cd84c-2573-49cb-ae5e-4f9e3bb6e60a"}, "cell_type": "markdown", "source": ["As it is said in data description page,  `TRAIN_DB` contains a list of 7,069,896 dictionaries, one per product. \n", "Each dictionary contains :\n", "- product id (key: _id)\n", "- the category id of the product (key: category_id), \n", "- 1-4 images, stored in a list (key: imgs).     \n", "\n", "Let's look at the first item:"]}, {"metadata": {"_uuid": "37c5525cc2cfa1030b2923c77810da7713d35159", "_cell_guid": "e63f42ed-692f-4093-93cb-14e3bee8f4fe", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["for item in TRAIN_DB:\n", "    break\n", "print(type(item), list(item.keys()))\n", "print(item['_id'], len(item['imgs']), item['category_id'],)"]}, {"metadata": {"_uuid": "31127d3e39bf59f70a16ee5096f3f2da8b5c0ac1", "_cell_guid": "05052407-c4aa-4e51-a660-ce6eedd30a45", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def decode(data):\n", "    arr = np.asarray(bytearray(data), dtype=np.uint8)\n", "    img = cv2.imdecode(arr, cv2.IMREAD_COLOR)\n", "    return cv2.cvtColor(img, cv2.COLOR_BGR2RGB) \n", "\n", "import io\n", "from PIL import Image\n", "\n", "def decode_pil(data):\n", "    return Image.open(io.BytesIO(data))\n", "\n", "for img_dict in item['imgs']:\n", "    img = decode(img_dict['picture'])\n", "    plt.figure()\n", "    plt.imshow(img)"]}, {"metadata": {"_uuid": "10c925d3dc10619032d9e088a28efd9d8d24595b", "_cell_guid": "6d5e25bb-07ec-4ccc-b0d7-f2f81de88622"}, "cell_type": "markdown", "source": ["Table `CATEGORY_NAMES_DF` shows the hierarchy of product classification. \n", "- category_id has 3 category tags of different levels\n", "\n", "Using `category_id` field we can associate image to 3 levels of category tags, labels. Thus, previous image is characterized as "]}, {"metadata": {"_uuid": "876b8f43c3cbd9e4ee1c14f69d3845049ac098a2", "_cell_guid": "7db703fe-bc95-418d-8064-eee9170c2a04", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["level_tags = CATEGORY_NAMES_DF.columns[1:]\n", "CATEGORY_NAMES_DF[CATEGORY_NAMES_DF['category_id'] == item['category_id']][level_tags]"]}, {"metadata": {"_uuid": "45b7f1589c9f6abdce566b0ecc6b170314fd72a3", "_cell_guid": "da3f1ade-0759-4fb2-a672-3675c41b0d3f"}, "cell_type": "markdown", "source": ["Let's see some more images :"]}, {"metadata": {"_uuid": "eeff292bc9915020d2e8165b21c0458f09aad879", "_cell_guid": "20d2c3a1-fc39-442f-8b61-7d2992641ec1", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# Method to compose a single image from 1 - 4 images\n", "def decode_images(item_imgs):\n", "    nx = 2 if len(item_imgs) > 1 else 1\n", "    ny = 2 if len(item_imgs) > 2 else 1\n", "    composed_img = np.zeros((ny * 180, nx * 180, 3), dtype=np.uint8)\n", "    for i, img_dict in enumerate(item_imgs):\n", "        img = decode(img_dict['picture'])\n", "        h, w, _ = img.shape        \n", "        xstart = (i % nx) * 180\n", "        xend = xstart + w\n", "        ystart = (i // nx) * 180\n", "        yend = ystart + h\n", "        composed_img[ystart:yend, xstart:xend] = img\n", "    return composed_img"]}, {"metadata": {"_uuid": "6b7bcadbacca0509487e20cf20a00afec227993f", "_cell_guid": "06f316b9-4e19-487a-82f7-37fbdcfd323d", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["max_counter = 15\n", "counter = 0\n", "n = 4\n", "for item in TRAIN_DB:    \n", "    if counter % n == 0:\n", "        plt.figure(figsize=(14, 6))\n", "    \n", "    mask = CATEGORY_NAMES_DF['category_id'] == item['category_id']    \n", "    plt.subplot(1, n, counter % n + 1)\n", "    cat_levels = CATEGORY_NAMES_DF[mask][level_tags].values.tolist()[0]\n", "    cat_levels = [c[:25] for c in cat_levels]\n", "    title = str(item['category_id']) + '\\n'\n", "    title += '\\n'.join(cat_levels)\n", "    plt.title(title)\n", "    plt.imshow(decode_images(item['imgs']))\n", "    plt.axis('off')\n", "    \n", "    counter += 1\n", "    if counter == max_counter:\n", "        break"]}, {"metadata": {"_uuid": "bfc8be622f54c6e45517c97829d489d93cb9411c", "_cell_guid": "5f787997-3c8a-4028-aa44-87169643fd0f"}, "cell_type": "markdown", "source": ["So, in train dataset we have products indexed by `_id`, belong to a `category_id` and described by 1-4 images.\n", "\n", "Now, let's quickly take a look to test products:"]}, {"metadata": {"_uuid": "5d1476f9bd31be8bac017fac9c5014033ecefdc9", "_cell_guid": "c60e13e4-2682-4167-b57d-ab0ea73c9a86", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["for item in TEST_DB:\n", "    break\n", "print(type(item), list(item.keys()))\n", "print(item['_id'], len(item['imgs']))"]}, {"metadata": {"_uuid": "5c26e0138131cc46e5b31e72ae533c1a4f761af9", "_cell_guid": "dfaebd45-619e-4406-a585-00ce1a6c643b", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["max_counter = 15\n", "counter = 0\n", "n = 4\n", "for item in TEST_DB:    \n", "    if counter % n == 0:\n", "        plt.figure(figsize=(14, 6))\n", "    \n", "    plt.subplot(1, n, counter % n + 1)\n", "    title = str(item['_id'])\n", "    plt.title(title)\n", "    plt.imshow(decode_images(item['imgs']))\n", "    plt.axis('off')\n", "    \n", "    counter += 1\n", "    if counter == max_counter:\n", "        break"]}, {"metadata": {"_uuid": "ae22e16bf163b656cf847d488d20edfb163dd82a", "_cell_guid": "c82dd499-f704-4f53-8378-e81dfab93a34"}, "cell_type": "markdown", "source": ["## Random item access"]}, {"metadata": {"_uuid": "0bb9c4a8c6ac5d8656a3d208c66e3d4209c9d84d", "_cell_guid": "13a62826-e323-45f1-b991-0d11810e8713"}, "cell_type": "markdown", "source": ["Let's make a random access to products by maping each product byte offset and length. \n", "\n", "Following code creates a dictionary with key indexing item `_id` and values `(offset, length)`. It takes around 3 mins to execute."]}, {"metadata": {"_uuid": "74262951e1910dc8fcdb7b3d55aa3511a593c389", "_cell_guid": "53a272ba-4b52-4009-b324-3a7e70172cef", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["import struct\n", "from tqdm import tqdm_notebook\n", "\n", "num_dicts = 7069896 # according to data page\n", "length_size = 4\n", "IDS_MAPPING = {}\n", "\n", "with open(os.path.join(INPUT_PATH, 'train.bson'), 'rb') as f, tqdm_notebook(total=num_dicts) as bar:\n", "    item_data = []\n", "    offset = 0\n", "    while True:        \n", "        bar.update()\n", "        f.seek(offset)\n", "        \n", "        item_length_bytes = f.read(length_size)     \n", "        if len(item_length_bytes) == 0:\n", "            break                \n", "        # Decode item length:\n", "        length = struct.unpack(\"<i\", item_length_bytes)[0]\n", "        \n", "        f.seek(offset)\n", "        item_data = f.read(length)\n", "        assert len(item_data) == length, \"%i vs %i\" % (len(item_data), length)\n", "        \n", "        # Check if we can decode\n", "        item = bson.BSON.decode(item_data)\n", "        \n", "        IDS_MAPPING[item['_id']] = (offset, length)        \n", "        offset += length            \n", "            \n", "def get_item(item_id):\n", "    assert item_id in IDS_MAPPING\n", "    with open(os.path.join(INPUT_PATH, 'train.bson'), 'rb') as f:\n", "        offset, length = IDS_MAPPING[item_id]\n", "        f.seek(offset)\n", "        item_data = f.read(length)\n", "        return bson.BSON.decode(item_data)"]}, {"metadata": {"_uuid": "6ccfa75f5dbaf1013242c5e6f069604cb8878d8d", "_cell_guid": "edc5efe5-798f-4f4d-94f2-36a3c7fce8be"}, "cell_type": "markdown", "source": ["Display for example a item with `_id=1234`  "]}, {"metadata": {"_uuid": "27c188b4c451d070f11ef933aab7b564fb0ad43e", "_cell_guid": "b6cf0670-1247-4da6-a65d-12ec2de01f09", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["item = get_item(1234)\n", "\n", "mask = CATEGORY_NAMES_DF['category_id'] == item['category_id']    \n", "cat_levels = CATEGORY_NAMES_DF[mask][level_tags].values.tolist()[0]\n", "cat_levels = [c[:25] for c in cat_levels]\n", "title = str(item['category_id']) + '\\n'\n", "title += '\\n'.join(cat_levels)\n", "plt.title(title)\n", "plt.imshow(decode_images(item['imgs']))\n", "_ = plt.axis('off')"]}, {"metadata": {"_uuid": "03930e319c80108d1458bf380a08480b0d1a8930", "_cell_guid": "e55f6881-ab13-4d24-8290-cec913e38854", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_uuid": "f661dd5731c657ba5c75ecb2923f8e77ccb0b33f", "_cell_guid": "24ccf0c8-92fe-4b9c-b04d-dbe576f9eb71"}, "cell_type": "markdown", "source": ["## Explore categories\n", "\n", "Let's inspect categories and their relationship to images. We have \n", "- 5270 unique categories\n", "- 49 unique level 1 categories\n", "- 483 unique level 2 categories\n", "- 5263 unique level 3 categories\n"]}, {"metadata": {"_uuid": "4570f4dc27ab7ac612532cda04d901d9ce307631", "_cell_guid": "51931177-9a43-4da2-bccd-9cc9d08f81d8", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["print(\"Unique categories: \", len(CATEGORY_NAMES_DF['category_id'].unique()))\n", "print(\"Unique level 1 categories: \", len(CATEGORY_NAMES_DF['category_level1'].unique()))\n", "print(\"Unique level 2 categories: \", len(CATEGORY_NAMES_DF['category_level2'].unique()))\n", "print(\"Unique level 3 categories: \", len(CATEGORY_NAMES_DF['category_level3'].unique()))"]}, {"metadata": {"_uuid": "62703882aa68acb91eb0434e72dfe4a563eb67ff", "_cell_guid": "993fb902-09ea-4b29-bb12-391b05172842"}, "cell_type": "markdown", "source": ["So as it was asked in comments (by @microland) we can observe that there are items with different 'category_id' but the same 'category_level3':"]}, {"metadata": {"_uuid": "1a726ebf3fedaae6ddd4be2fe0d028a7bce092b6", "_cell_guid": "a7df71f7-ed14-4bd4-88fe-4eb8c0d0cca1", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["gb = CATEGORY_NAMES_DF.groupby('category_level3')\n", "cnt = gb.count()\n", "cnt[cnt['category_id'] > 1]"]}, {"metadata": {"_uuid": "02f8cede7709bc7847235583f08817d5ff0e69f6", "_cell_guid": "06b49d9c-e526-47a1-8e00-32aa468078db", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["gb.get_group(cnt[cnt['category_id'] > 1].index.values[0])"]}, {"metadata": {"_uuid": "921124fdae056748992956352b22b290b5ece830", "_cell_guid": "ff65b030-d7fc-4278-89ae-d5ea7a7443dd", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["import seaborn as sns"]}, {"metadata": {"_uuid": "e754a6aa5220ee07fdd641bd8fca676fd4cbb237", "_cell_guid": "6ad77691-4d4b-498d-b4f2-170ccb5d0c11"}, "cell_type": "markdown", "source": ["Here is the histogram of level 1 categories"]}, {"metadata": {"_uuid": "367450b2d4ac228d917f28217d24970084fe1f21", "_cell_guid": "93811104-a35b-4dd6-bd6d-99ced8d4fea5", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["plt.figure(figsize=(12,12))\n", "_ = sns.countplot(y=CATEGORY_NAMES_DF['category_level1'])"]}, {"metadata": {"_uuid": "aef8d3d5e1f75c786b8ea6edef6a679d1a6c510c", "_cell_guid": "0e455f2b-8182-4cb8-8e19-0c8498c1954b"}, "cell_type": "markdown", "source": ["Level 2 and 3 categories are distributed as follows:"]}, {"metadata": {"_uuid": "47d11228db0f87173e7c7e7a77f9a453a90a8bfc", "_cell_guid": "aeade098-3c6f-4341-9f55-5f216da0e30c", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["cat_level2_counts = CATEGORY_NAMES_DF.groupby('category_level2')['category_level2'].count()\n", "print(cat_level2_counts.describe())\n", "print(\"Level 2 the most frequent category: \", cat_level2_counts.argmax())"]}, {"metadata": {"_uuid": "e3b2c4034fffd31419ce1c1a425cf8a1e0593b3e", "_cell_guid": "b509a860-47dc-4ecf-ab89-5ad230c38cef", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["cat_level3_counts = CATEGORY_NAMES_DF.groupby('category_level3')['category_level3'].count()\n", "print(cat_level3_counts.describe())\n", "print(\"Level 3 the most frequent category: \", cat_level3_counts.argmax())"]}, {"metadata": {"_uuid": "0ad9cf74a00c48e5837e2ba6aa45f0605fa47bc8", "_cell_guid": "d4946f7f-592b-4ddf-9302-3059e6dd3632", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_uuid": "707162c8c55285712d377a72d494aa4261a4fd0f", "_cell_guid": "1b4b0846-1b0c-4e00-a8fd-902e14a70586"}, "cell_type": "markdown", "source": ["Now, let's create training data table `_id`, `category_id`:"]}, {"metadata": {"_uuid": "7cf541cff0cf0e09fac2c3d338589e346cfff83c", "_cell_guid": "06756932-93b1-4267-aa8b-d6343ae2c1be", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["from tqdm import tqdm_notebook\n", "\n", "num_dicts = 7069896 # according to data page\n", "prod_to_category = [None] * num_dicts\n", "\n", "with tqdm_notebook(total=num_dicts) as bar:        \n", "    TRAIN_DB = bson.decode_file_iter(open(os.path.join(INPUT_PATH, 'train.bson'), 'rb'))\n", "\n", "    for i, item in enumerate(TRAIN_DB):\n", "        bar.update()\n", "        prod_to_category[i] = (item['_id'], item['category_id'])"]}, {"metadata": {"_uuid": "408b940f164d5af55a3549effd9b6ec6c33c273a", "_cell_guid": "f682c3a9-6add-4dc8-97fd-88666c7e6e26", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["TRAIN_CATEGORIES_DF = pd.DataFrame(prod_to_category, columns=['_id', 'category_id'])\n", "TRAIN_CATEGORIES_DF.head()"]}, {"metadata": {"_uuid": "5036399d477125794b809a4f96bd2005fd1ea5d1", "_cell_guid": "2266962e-6f5e-4a0c-a22c-718c818616dc"}, "cell_type": "markdown", "source": ["We have in training datasets : \n", "- 5270 unique categories in 7069896 entries\n", "- 1 most frequent category (found 79640 times) : MUSIQUE (en.: music)\n", "- 31 less frequent categories (found 12 times) : PUERICULTURE (en.: childcare),  APICULTURE (en.: beekeeping), SPORT/BASEBALL/BLOUSON DE BASEBALL - VESTE DE BASEBALL, ..."]}, {"metadata": {"_uuid": "553017d5b2a08cbd2f4c1a937ee9ef9721687ad8", "_cell_guid": "08e31dd9-3a42-477a-8644-09cf886813df", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["print(\"Unique categories: %i in %i entries\" % (len(TRAIN_CATEGORIES_DF['category_id'].unique()), len(TRAIN_CATEGORIES_DF)))"]}, {"metadata": {"_uuid": "ac6cbde94a426f4825ed85d349b99ff09094d06d", "_cell_guid": "b88363c7-6d7e-4c26-aed4-961e986676f5", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["train_categories_gb = TRAIN_CATEGORIES_DF.groupby('category_id')\n", "train_categories_count = train_categories_gb['category_id'].count()\n", "print(train_categories_count.describe())"]}, {"metadata": {"_uuid": "57c99dca4372edb2be719e51bfd358587d049b43", "_cell_guid": "b503a419-1dee-4af5-81d7-c9eddfdb11da", "collapsed": true, "_kg_hide-input": false}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["most_freq_cats = train_categories_count[train_categories_count == train_categories_count.max()]\n", "less_freq_cats = train_categories_count[train_categories_count == train_categories_count.min()]\n", "\n", "print(\"Most frequent category: \", CATEGORY_NAMES_DF[CATEGORY_NAMES_DF['category_id'].isin(most_freq_cats.index)].values)\n", "print(\"Less frequent category: \", CATEGORY_NAMES_DF[CATEGORY_NAMES_DF['category_id'].isin(less_freq_cats.index)].values)"]}, {"metadata": {"_uuid": "63be882cda0ebd4985281434b2c665c66461f816", "_cell_guid": "356985b9-10bf-4452-adf6-60034be2ec67"}, "cell_type": "markdown", "source": ["Let's display some of these most frequent category items ('MUSIQUE' 'CD' 'CD POP ROCK - CD ROCK INDE')"]}, {"metadata": {"_uuid": "4b77677f9633d51a7a5c7d25eed2db2b57298433", "_cell_guid": "4aa29026-0c00-4dcf-a4d0-8873ea49ebe0", "collapsed": true, "scrolled": false}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["most_freq_cat = most_freq_cats.index[0]\n", "\n", "plt.figure(figsize=(16, 4))\n", "mask = CATEGORY_NAMES_DF['category_id'] == most_freq_cat    \n", "cat_levels = CATEGORY_NAMES_DF[mask][level_tags].values.tolist()[0]\n", "title = str(most_freq_cat) + '\\n'\n", "title += '\\n'.join(cat_levels)\n", "plt.suptitle(title)\n", "\n", "most_freq_cat_ids = train_categories_gb.get_group(most_freq_cat)['_id']\n", "max_counter = 50\n", "counter = 0\n", "n = 10\n", "for item_id in most_freq_cat_ids.values[:max_counter]:    \n", "    if counter > 0 and counter % n == 0:\n", "        plt.figure(figsize=(14, 6))\n", "    \n", "    item = get_item(item_id)\n", "    \n", "    mask = CATEGORY_NAMES_DF['category_id'] == item['category_id']    \n", "    plt.subplot(1, n, counter % n + 1)\n", "    plt.imshow(decode_images(item['imgs']))\n", "    plt.axis('off')\n", "    \n", "    counter += 1\n", "    if counter == max_counter:\n", "        break"]}, {"metadata": {"_uuid": "a330f73b4ef11f06a3f9795d4410c57765837ae3", "_cell_guid": "a37e536f-df95-45f2-9d09-ebf0b7772f3d"}, "cell_type": "markdown", "source": ["Let's display some of these 31 less frequent categories:"]}, {"metadata": {"_uuid": "eed2b6474547349c1f05580adc9c653c3eddff3c", "_cell_guid": "7ac40a01-9e6b-466d-b3ef-0def5616efde", "collapsed": true, "scrolled": false}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["for less_freq_cat in less_freq_cats.index:\n", "    less_freq_cat_ids = train_categories_gb.get_group(less_freq_cat)['_id']\n", "    counter = 0\n", "    n = 12\n", "    \n", "    plt.figure(figsize=(16, 4))\n", "    mask = CATEGORY_NAMES_DF['category_id'] == less_freq_cat    \n", "    cat_levels = CATEGORY_NAMES_DF[mask][level_tags].values.tolist()[0]\n", "    title = str(less_freq_cat) + '\\n'\n", "    title += '\\n'.join(cat_levels)\n", "    plt.suptitle(title)\n", "\n", "    for item_id in less_freq_cat_ids.values:    \n", "        if counter > 0 and counter % n == 0:\n", "            plt.figure(figsize=(16, 4))\n", "\n", "        item = get_item(item_id)\n", "\n", "        mask = CATEGORY_NAMES_DF['category_id'] == item['category_id']    \n", "        plt.subplot(1, n, counter % n + 1)\n", "        plt.imshow(decode_images(item['imgs']))\n", "        plt.axis('off')\n", "\n", "        counter += 1        "]}, {"metadata": {"_uuid": "7575ca9f220ce09e9303323abd850d0416797583", "_cell_guid": "07ac193c-26b6-4a83-a4b1-addd08e994ac"}, "cell_type": "markdown", "source": ["Here is a plot of sorted category counts "]}, {"metadata": {"_uuid": "ff2ec16ca9bde0d0a63c2361b3b16f31fef858b2", "_cell_guid": "2a8746b1-7ac4-46a6-9f40-36e4567dc54d", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["sorted_train_categories_count = sorted(train_categories_count.values)\n", "index_8000 = np.where(np.array(sorted_train_categories_count) > 8000)[0][0]\n", "\n", "plt.figure(figsize=(12, 6))\n", "plt.title(\"Sorted category counts\")\n", "_ = plt.plot(sorted_train_categories_count, '*-')\n", "\n", "plt.figure(figsize=(12, 6))\n", "plt.subplot(121)\n", "plt.title(\"Sorted category counts < %i\" % index_8000)\n", "_ = plt.plot(sorted_train_categories_count[:index_8000], '*-')\n", "\n", "plt.subplot(122)\n", "plt.title(\"Sorted category counts > %i\" % index_8000)\n", "_ = plt.plot(sorted_train_categories_count[index_8000:], '*-')"]}, {"metadata": {"_uuid": "8fc538530eccb3fd380858436c347cbc4589211b", "_cell_guid": "e46677be-f80c-4b7d-ade3-63acc86a0074"}, "cell_type": "markdown", "source": ["The goal of the competition is to predict `category_id` by image. We need to predict a number, e.g. `1000010653` by an image. \n"]}, {"metadata": {"_uuid": "0b7ae0381bed49418f04e6a88163902d095f223a", "_cell_guid": "675f5f69-82c5-41c1-b351-16e2815d7158", "collapsed": true}, "cell_type": "markdown", "source": ["## Category_count vs Image_count\n", "\n", "We can display a relationship between numbers of categories that are presented by an amount of images: i.e. 10 categories are presented in the training dataset by 100 images. See [this](https://datascience.stackexchange.com/questions/11777/what-is-the-distribution-of-categories-in-imagenet-training-set-ilsvrc2012) for more details. "]}, {"metadata": {"_uuid": "518189474242f71fd6feb9d2dc71a22747adf5db", "_cell_guid": "4be177ca-f72e-4b1c-a395-3fc6a5699f53", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["plt.figure(figsize=(12, 6))\n", "plt.title('Category_count vs Image_count')\n", "bin_size = 25\n", "plt.hist(train_categories_count, bins=range(0, int(1e4), bin_size))\n", "plt.xlabel('Amount of available images')\n", "_ = plt.ylabel('Number of classes')"]}, {"metadata": {"_uuid": "b740b260ae202ca951651504bba48772ca0aa12a", "_cell_guid": "b88ae2c0-b325-4e18-808d-586050e7bc89", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_uuid": "fd2c20210254a9d80b536893c0fb8e60ecefa109", "_cell_guid": "7eb8d6a8-062c-4779-a68a-c7a9ad0f20d3", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_uuid": "95e244d135110c6ae048a72dd8cd42d41ee00fd4", "_cell_guid": "8e7060ee-67a0-407b-a3d8-26928626ae11", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_uuid": "f3d4233549ec97b75f168997916b91b5399fb13f", "_cell_guid": "fad98fb7-6cf1-4d30-a346-9c7f1dc6a91f"}, "cell_type": "markdown", "source": []}]}