{"cells":[{"metadata":{"_uuid":"3e4b1076ca11e5ff93b121a7decb3042b72aa04c","_cell_guid":"a329e32d-61b9-486e-b237-4ae0296198b6"},"cell_type":"markdown","source":"# Intro\n\nThis notebook shows some characteristics of the images and labels used for the two contests. I've taken a small subset of training images and files from the Open Images dataset and put them here: [Excerpt from OpenImages 2020 Train.](https://www.kaggle.com/jpmiller/excerpt-from-openimages-2020-train)\n\nSome specific objectives:\n\n* Get a feel for the the images and the objects/segments they contain.\n* Implement some basic object detection.\n* Look at label counts image sizes, and object relationships.\n\n\nOn the technical side, there are some things you might find useful:\n\n* Modeling with large datasets in Kaggle notebooks\n* Making interactive plots with hvplot\n* Visualizing graph networks with networkX\n","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"! conda install -y hvplot","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nfrom pathlib import Path\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport hvplot.pandas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Images and annotations\nAnnotations and classes are different for the two competitions. The detection dataset has more classes of objects and more objects per image most of the time. In other words, images common to both challenges will have more boxes than masks.\n\nHere are a few images containing boxes and segment masks along with labels. I limited it to 6 for display purposes. It's easy enough to browse through hundreds and get a more complete idea.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = Path('../input/excerpt-from-openimages-2020-train')\nim_list = sorted(data_dir.glob('train_00_part/*.jpg'))\nmask_list = sorted(data_dir.glob('train-masks-f/*.png'))\nboxes_df = pd.read_csv(data_dir/'oidv6-train-annotations-bbox.csv')\n\nnames_ = ['LabelName', 'Label']\nlabels =  pd.read_csv(data_dir/'class-descriptions-boxable.csv', names=names_)\n\nim_ids = [im.stem for im in im_list]\ncols = ['ImageID', 'LabelName', 'XMin', 'YMin', 'XMax', 'YMax']\nboxes_df = boxes_df.loc[boxes_df.ImageID.isin(im_ids), cols] \\\n                   .merge(labels, how='left', on='LabelName')\nboxes_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Annotate and plot\ncols, rows  = 3, 2\nplt.figure(figsize=(20,30))\n\n\nfor i,im_file in enumerate(im_list[9:15], start=1):\n    df = boxes_df.query('ImageID == @im_file.stem').copy()\n    img = cv2.imread(str(im_file))\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    \n    # Add boxes\n    h0, w0 = img.shape[:2]\n    coords = ['XMin', 'YMin', 'XMax', 'YMax']\n    df[coords] = (df[coords].to_numpy() * np.tile([w0, h0], 2)).astype(int)\n\n    for tup in df.itertuples():\n        cv2.rectangle(img, (tup.XMin, tup.YMin), (tup.XMax, tup.YMax),\n                      color=(0,255,0), thickness=2)\n        cv2.putText(img, tup.Label, (tup.XMin+2, tup.YMax-2),\n                    fontFace=cv2.FONT_HERSHEY_DUPLEX,\n                    fontScale=1, color=(0,255,0), thickness=2)\n    \n    # Add segmentation masks\n    mask_files = [m for m in mask_list if im_file.stem in m.stem]    \n    mask_master = np.zeros_like(img)\n    np.random.seed(10)\n    for m in mask_files:\n        mask = cv2.imread(str(m))\n        mask = cv2.resize(mask, (w0,h0), interpolation = cv2.INTER_AREA)\n        color = np.random.choice([0,255], size=3)\n        mask[np.where((mask==[255, 255, 255]).all(axis=2))] = color\n        mask_master = cv2.add(mask_master, mask)\n    img = cv2.addWeighted(img,1, mask_master,0.5, 0)    \n    \n    plt.subplot(cols, rows, i)    \n    plt.axis('off')\n    plt.imshow(img)\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Object detection demo\nBelow is a simple demo that detects objects. I'm using YOLOv3 as implemented in opencv (yes, opencv has a function for object detection). Weights and the network config come from @pjreddie's darknet repo at https://github.com/pjreddie/darknet.\n\nIf you are space constrained, you might try loading images from their URLs. You can download and resize them in a loop before saving to a hard drive. Alternately, you could pull images in batches and feed them to the dataloader of your model. It's probably super-slow, but you could do it all in RAM (I think - never tried it).\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"urls = pd.read_csv(data_dir/\"image_ids_and_rotation.csv\", \n                   usecols=['ImageID', 'OriginalURL'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classes = np.loadtxt(data_dir/\"openimages.names\", dtype=np.str, delimiter=\"\\n\")\nnet = cv2.dnn.readNet(str(data_dir/\"yolov3-openimages.weights\"), str(data_dir/\"yolov3-openimages.cfg\"))\n\nlayer_names = net.getLayerNames()\noutputlayers = [layer_names[i[0] - 1] for i in net.getUnconnectedOutLayers()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here again is the indoor scene from above. This time the boxes are produced from object detection and not from the box file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nfrom skimage import io\n\nim_url = urls.loc[urls.ImageID==im_list[11].stem, 'OriginalURL'].squeeze()\nimg = io.imread(im_url)\n\nheight,width,channels = img.shape\n\n# Make a blob array and run it through the network\nblob = cv2.dnn.blobFromImage(img,0.00392,(416,416),(0,0,0),True,crop=False)\nnet.setInput(blob)\nouts = net.forward(outputlayers)\n\n# Get confidence scores and objects\nclass_ids=[]\nconfidences=[]\nboxes=[]\nfor out in outs:\n    for detection in out:\n        scores = detection[5:]\n        class_id = np.argmax(scores)\n        confidence = scores[class_id]\n        if confidence > 0.2:   # threshold\n            print(confidence)\n            center_x= int(detection[0]*width)\n            center_y= int(detection[1]*height)\n            w = int(detection[2]*width)\n            h = int(detection[3]*height)\n            x=int(center_x - w/2)\n            y=int(center_y - h/2)\n            boxes.append([x,y,w,h]) #put all rectangle areas\n            confidences.append(float(confidence)) #how confidence was that object detected and show that percentage\n            class_ids.append(class_id) #name of the object tha was detected\n            \n# Non-max suppression\nindexes = cv2.dnn.NMSBoxes(boxes,confidences,0.4,0.6)\nprint(indexes, boxes, class_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"font = cv2.FONT_HERSHEY_DUPLEX\nfor i in range(len(boxes)):\n#     if i in indexes:\n        x,y,w,h = boxes[i]\n        label = str(classes[class_ids[i]])\n        cv2.rectangle(img, (x,y), (x+w,y+h), (255,255,0), 2)\n        cv2.putText(img, label, (x,y+30), font, 2, (255,255,0), 2)\n        \nplt.clf()\nplt.figure(figsize=(10,15))\nplt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pretty cool overall. You can see there are fewer objects detected by this network than appear in the ground truth. YoloV3 has a hard time detecting and delivering objects close together with overlapping boxes. It's a great option for say, flying a drone around and spotting cars, whereas R-CNNs will probably work better for this case.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Label counts per image\n\nNow for some EDA. The distributions are interesting. Some of the images carry dozens or even hundreds of annotations. There's a long tail out past 700 although most images are more like 50 or fewer. Unique annotations are far fewer with many of the highly-labeled pictures being something like a skyscraper wth 102 windows (real example).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"annotations = boxes_df.groupby('ImageID').agg(\n                        box_count=('LabelName', 'size'),\n                        box_unique=('LabelName', 'nunique')\n                        )\n\npd.options.display.float_format = '{:,.1f}'.format\nannotations.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all = annotations.hvplot.hist('box_count', width=600, bins=30)\nunique = annotations.hvplot.hist('box_unique', width=600)\n(all + unique).cols(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's another look at the number of boxes per image with the largest 1% removed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"onepct = annotations.box_count.quantile(0.99)\nannotations.query('box_count < @onepct').box_count.value_counts(normalize=True) \\\n    .sort_index().hvplot.bar(xticks=list(range(0,60,10)), width=600,\n                            line_alpha=0, xlabel='objects per image',\n                            ylabel='fraction of images')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's the skyscraper.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(boxes_df.loc[boxes_df.ImageID==\"fe7c6f7d298893da\"] \\\n         .groupby(['ImageID', 'Label'])['LabelName'].size()\n     )\n\nim_file = \"../input/excerpt-from-openimages-2020-train/train_00_part/fe7c6f7d298893da.jpg\"\nim = cv2.imread(im_file)\nplt.imshow(im)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image size\nThe data set is huge! Knowing image sizes can give an idea of the impact of size reduction. Here is how the test image sizes are distributed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image\nfrom dask import bag, diagnostics\n\n\ndef faster_get_dims(file):\n    dims = Image.open(file).size\n    return dims\n\ndfile_list = glob.glob('../input/open-images-object-detection-rvc-2020/test/*.jpg')\nprint(f\"Getting dimensions for {len(dfile_list)} files.\")\n\n# parallelize\ndfile_bag = bag.from_sequence(dfile_list).map(faster_get_dims)\nwith diagnostics.ProgressBar():\n    dims_list = dfile_bag.compute()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = pd.DataFrame(dims_list, columns=['width', 'height'])\ncounts = sizes.groupby(['width', 'height']).agg(count=('width', 'size')) \\\n              .reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nplot_opts = dict(xlim=(0,1200), \n                 ylim=(0,1200), \n                 grid=True, \n                 xticks=[250, 682, 768, 1024], \n                 yticks=[250, 682, 768, 1024], \n                 height=500, \n                 width=550\n                 )\n\nstyle_opts = dict(scaling_factor=0.2,\n                  line_alpha=1,\n                  fill_alpha=0.1\n                  )\n\ncounts.hvplot.scatter(x='width', y='height', size='count', **plot_opts) \\\n             .options(**style_opts)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Distribution of object labels\n\nHere's a chart showing the frequency of the various types of objects. This is for detection, and for the train set, which will be different for instance segmentation and maybe for the test set. Overall though the data will mostly be pictures of \"people with faces, wearing clothes, and standing near trees\":)\n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels = boxes_df[['ImageID', 'LabelName']].merge(labels, how='left', on='LabelName')\ntrain_labels.Label.value_counts(normalize=True)[:45] \\\n            .hvplot.bar(width=650, height=350, rot=60, line_alpha=0,\n                        title='Label Frequencies',\n                        ylabel='fraction of all objects')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Hierarchy of objects\n\nThe Description page on the website has great information on the objects and how they relate. The picture below gives an idea of the relationships. A more complete picture appears on the website.\n\n\n![hier](https://storage.googleapis.com/openimages/web/images/v2-bbox_labels_vis_screenshot.png)\n\nYou can also see relationships in our data with the file called 'oidv6-relationship-triplets.csv'. It look like this.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"relations = pd.read_csv(data_dir/'oidv6-relationship-triplets.csv')\nrelations = relations.merge(labels, how='left', left_on='LabelName1', right_on='LabelName') \\\n                     .merge(labels, how='left', left_on='LabelName2', right_on='LabelName',\n                            suffixes=['1', '2']) \\\n                     .loc[:, ['Label1', 'RelationshipLabel', 'Label2']] \\\n                     .dropna() \\\n                     .sort_values('RelationshipLabel') \\\n                     .reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" Mapping the entire network is quite complex. Here's a map for only two entities, boy and girl, and all the things to which they connect in the images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import networkx as nx\n\nkids = relations.query('Label1==\"Girl\" or Label1==\"Boy\"')\nG = nx.from_pandas_edgelist(kids, 'Label1', 'Label2', 'RelationshipLabel')\n\n\ngraph_opts = dict(arrows=False,\n                  node_size=5,\n                  width=0.5,\n                  alpha=0.8,\n                  font_size=10,\n                  font_color='darkblue',\n                  edge_color='gray'\n                \n                 )\n\nfig= plt.figure(figsize=(12,10))\nnx.draw_spring(G, with_labels=True, **graph_opts)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There's a lot more to explore before and during model building. I hope this gives a good start!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}