{"cells":[{"metadata":{},"cell_type":"markdown","source":"#### UPDATE\nMy current project requires annotation of images, and I needed a refresher on the code. I found this kernel from last year and remembered how fun this competition was!\n\nAlso, I don't like seeing a kernel of mine with 102 votes and only a silver medal:)"},{"metadata":{"_uuid":"46cc0a817aeb796710494fc47c68ebaebe861acc"},"cell_type":"markdown","source":"Let's explore the various data files and get a feel for the images and the data. I'll use OpenCV to read the images and hvplot for the charts. Hvplot is cool in that it's similar to Pandas plotting but with a Bokeh backend to make interactive charts.\n\n\n### Files\nThere are several files provided here. The Data tab for the competition provides a nice overview of each file you see below.  Note that the train images and the image urls aren't part of the kernel files. The Data tab has more information on how to get them."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"import os\nfrom glob import glob\nimport numpy as np\nimport pandas as pd\nfrom bq_helper import BigQueryHelper\nfrom dask import bag, diagnostics \nfrom urllib import request\nimport cv2\nimport missingno as msno\nimport hvplot.pandas  # custom install\nfrom matplotlib import pyplot as plt\n%matplotlib inline\n\nfiles = glob('../input/inclusive-images-challenge/*.csv')\nfor f in files:\n    df = pd.read_csv(f, nrows=5)\n    display(f, df.head()) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c43b9e947ead019b00fd970bec429aa70e9c80cf"},"cell_type":"markdown","source":"### Images\nLet's use the data to annotate some train images and see what we're trying to do. You'll see there is some complexity here, both with the labels and the boxes."},{"metadata":{"trusted":true,"_uuid":"bc2f51372fcbcd9ade1b8a2446a366ac450f2ccf","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train = pd.read_csv('../input/inclusive-images-challenge/train_human_labels.csv', usecols=['ImageID', 'LabelName'])\n\ndescrips =  pd.read_csv('../input/inclusive-images-challenge/class-descriptions.csv', names=['LabelName', 'Description'])\ntrain = train.merge(descrips, how='left', on='LabelName')\ntrain.head(9)\n\nopen_images = BigQueryHelper(active_project=\"bigquery-public-data\", dataset_name=\"open_images\")\nquery = \"\"\"\n            SELECT image_id, original_url \n            FROM `bigquery-public-data.open_images.images` \n            WHERE image_id IN UNNEST(['0199bc3e1db115d0',\n                                      '4fa8054781a4c382',\n                                      '51c5d8d5d9cd87ca',\n                                      '9ec02b5c0315fcd1',\n                                      'b37f763ae67d0888',\n                                      'ddcb4b7478e9917b'])\n        \"\"\"\nurls = open_images.query_to_pandas_safe(query)\n\nboxes = pd.read_csv('../input/inclusive-images-challenge/train_bounding_boxes.csv')\nboxes = boxes[boxes.ImageID.isin(urls.image_id.tolist())].sort_values('ImageID')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1bc19448aedb468a595e0eaabf2a708a6582fcdd","_kg_hide-input":true},"cell_type":"code","source":"imlist = urls.image_id.tolist()\nfiles = ['../input/inclusive-images-challenge/openimages-support/ims/{}.jpg'.format(i) for i in imlist]\nfig, ax = plt.subplots() \nfig.set_size_inches((15,15))\nax.set_axis_off()\nfor n, (file, image) in enumerate(zip(files, imlist)):\n    a = fig.add_subplot(2, 3, n + 1)\n    req = request.urlopen(urls.original_url[n])\n    arr = np.asarray(bytearray(req.read()), dtype=np.uint8)\n    img = cv2.imdecode(arr, 1) \n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    h0, w0 = img.shape[:2]\n    scale = 1024/max(h0, w0)\n    h, w = int(round(h0*scale)), int(round(w0*scale))\n    img = cv2.resize(img, (h, w), interpolation = cv2.INTER_AREA)\n    imboxes = boxes[boxes.ImageID == image]\n    for idx,row in imboxes.iterrows():\n        img = cv2.rectangle(img, (int(row.XMin*h), int(row.YMin*w)),\\\n                    (int(row.XMax*h), int(row.YMax*w)), (0,255,0), 4)\n    label = train.loc[train.ImageID == image, 'Description'].str.cat(sep = ', ')\n    plt.title(label, fontsize=10)\n    plt.axis('off')\n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d8bdf403bbb6f7303673902bb34d6e46038e670"},"cell_type":"markdown","source":"Looking at these images we can see that getting all of the right labels might be quite challenging! Here's part of a graphic from the Open Images website showing the hierarchy of labels for the bounding box set (which is only 600 labels). You can find the full graphic of labels and a corresponding json file on the site.\n\n![Circle](https://storage.googleapis.com/openimages/web/images/v2-bbox_labels_vis_screenshot.png)\n\n"},{"metadata":{"_uuid":"23918880c99386bfa4ee07d408f77a34026d9077"},"cell_type":"markdown","source":"\nThe images are clearly of different dimensions. We can use the test images to explore since they're already in the container. I'll use Dask to parallelize the operation and speed things up. \n\nThere are several different image sizes with a max of 1024 pixels for each dimension, as mentioned by the host. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2498092c2bbf80bde8e1b710dc3b72988eb2db4b"},"cell_type":"code","source":"# get image dimensions\ndef get_dims(file):\n    img = cv2.imread(file)\n    h,w = img.shape[:2]\n    return h,w\n\n# parallelize\nfilepath = '../input/inclusive-images-challenge/stage_1_test_images/'\nfilelist = [filepath + f for f in os.listdir(filepath)]\ndimsbag = bag.from_sequence(filelist).map(get_dims)\nwith diagnostics.ProgressBar():\n    dims = dimsbag.compute()\n    \ndim_df = pd.DataFrame(dims, columns=['height', 'width'])\nsizes = dim_df.groupby(['height', 'width']).size().reset_index().rename(columns={0:'count'})\nsizes.hvplot.scatter(x='height', y='width', size='count', xlim=(0,1200), ylim=(0,1200), grid=True, xticks=2, \n        yticks=2, height=500, width=600).options(scaling_factor=0.1, line_alpha=1, fill_alpha=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc01167d646e036f60d70c80c595d35120e7d8ba"},"cell_type":"markdown","source":"### Labels\nLet's end with a deeper look at labels. I'll use the human labels to get an idea of how many images and labels we have. Note that even though these are images from the Bounding Box set in Open Images, the labels in our scope are far more diverse."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f796814a5ff19416642127c944f2654ca35550b9"},"cell_type":"code","source":"print('{} images with {} unique labels'.format(train.ImageID.nunique(), train.LabelName.nunique()))\ntrain.head(9)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25082440f1536cd264d5b6d646e32284932ce6b7"},"cell_type":"markdown","source":"Here's a look at label frequencies for the top 48 labels. The top 3 are Person, Clothing, and Human Face. There's a long tail on the distribution, meaning that most categories are infrequent. Note that these are the translated labels, not the coded labels, and might not match 1-1."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d344b94b6e5057128444f21ab77571efb99baee9"},"cell_type":"code","source":"### not usually necessary, but helps with flaky notebook plotting\nimport holoviews as hv\nhv.extension('bokeh')\n###\n\ndcounts = train.Description.value_counts(normalize=True)\ndcounts_df = pd.DataFrame({'label': dcounts.index.tolist(), 'pct_of_images': dcounts})\ndcounts_df.reset_index(drop=True, inplace=True)\ndcounts_df[0:48].hvplot.bar(x='label', y='pct_of_images', invert=True, flip_yaxis=True, \n                            height=600, width=600, ylim=(0,0.12))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49c4cdbd5c64f5d30fd8685e308142771b1d38a3"},"cell_type":"markdown","source":"Let's take a closer look at how many labels are in each image. It looks like some images have a lot of labels! Most have ten or fewer though as seen in the histogram. Note, you can zoom in on the histogram to get a closer look at specific ranges."},{"metadata":{"trusted":true,"_uuid":"0d8c4534d5b4ce8cddc8b4d4d3e89df04097bb0f","_kg_hide-input":true},"cell_type":"code","source":"images = train.groupby('ImageID').count()\nimages.columns = ['LabelCount', 'DescriptionCount']\ndisplay(images.sort_values('LabelCount', ascending=False).head(10))\nimages.hvplot.hist('LabelCount', bins=50, height=400, width=600)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c1465d25dda164da4f609948f611b43e27015f1"},"cell_type":"markdown","source":"Finally, to look at correlation, we can use hierarchical clustering to see how often labels appear together. I'm using a shortcut here so these results may only be approximate. Here are the top 48 most frequent labels."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"bacd3bdbb0e97b080d10ef8d57a76f99cc7a3795","_kg_hide-input":true},"cell_type":"code","source":"trainmain = train[train.Description.isin(dcounts_df.loc[0:48, 'label'])]\ntrainpiv = trainmain.pivot_table(index='ImageID', columns='Description', aggfunc='size')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf54efa76015e7709f401c6a85bf33a9d74fdb1c"},"cell_type":"code","source":"trainpiv.isnull().corr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a39e7682ebdace00cd731123dce5bb9d8d326204"},"cell_type":"markdown","source":"That's all for now, good luck!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}