{"cells":[{"metadata":{},"cell_type":"markdown","source":" #                                      **IMAGE PROCESSING / OBJECT DETECTION**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> # Features Overview\n\nThe Cloud AutoML Vision Object Detection release includes the following features:\n\n* **Object localization** - Detects multiple objects in an image and provides information about the object and where the object was found in the image.\n\n* **API/UI** - Provides an API and custom user interface for importing your dataset from a Google Cloud Storage hosted CSV file and training images, for adding and removing annotations from imported images, for training and reviewing model evaluation metrics, and for using your model with online prediction.\n\nAutoML Vision Edge now allows you to export your custom AutoML Vision Object Detection trained models.\n\n* AutoML Vision Edge allows you to train and deploy low-latency, high accuracy models optimized for edge devices.\n* With TensorFlow Lite, Core ML, and container export formats, AutoML Vision Edge supports a variety of devices.\n* Hardware architectures supported: Edge TPUs, ARM and NVIDIA.\n* To build an application on iOS or Android devices you can use AutoML Vision Edge in ML Kit. This solution is available via Firebase and offers an end-to-end development flow for\n* creating and deploying custom models to mobile devices using ML Kit client libraries.\n\n<img src=\"https://cloud.google.com/vision/automl/object-detection/docs/images/index_test_image.png\" alt=\"drawing\" width=\"50%\"/>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction:\nThis exploratory notebook aims to accomplish the following things\n* get a feel for the images and to objects/segments they contain\n* Look at image sizes\n* Look at the frequency of objects\n* Explore the hierarchy of objects in the dataset","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"! conda install -y hvplot","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport cv2\nfrom pathlib import Path\nimport numpy as np\nimport pandas as pd\nimport hvplot.pandas\nfrom dask import bag, diagnostics\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Images and Annotations\nBelow are some examples of images with both boxes and masks. Note that the annotations and classes are different for the two competitions. The detection dataset has more classes of objects and more objects per images most of the time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = Path('../input/excerpt-from-openimages-2020-train')\nim_list = sorted(data_dir.glob('train_00_part/*.jpg'))\nmask_list = sorted(data_dir.glob('train-masks-f/*.png'))\nboxes_df = pd.read_csv(data_dir/'oidv6-train-annotations-bbox.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"im_ids = [im.stem for im in im_list]\ncols = ['ImageID', 'LabelName','XMin','YMin','XMax', 'YMax']\nboxes_df2 = boxes_df.loc[boxes_df.ImageID.isin(im_ids), cols]\n\nnames_ = ['LabelName', 'Label']\nlabels = pd.read_csv(data_dir/'class-descriptions-boxable.csv', names=names_)\n\nboxes_df2 = boxes_df2.merge(labels, how='left', on='LabelName')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols, rows = 3,2\nplt.figure(figsize=(20,30))\n\nfor i,im_file in enumerate(im_list[9:15], start=1):\n    df = boxes_df2.query('ImageID == @im_file.stem').copy()\n    img = cv2.imread(str(im_file))\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    \n    # Add boxes\n    h0, w0 = img.shape[:2]\n    coords = ['XMin', 'YMin', 'XMax', 'YMax']\n    df[coords] = (df[coords].to_numpy() * np.tile([w0, h0], 2)).astype(int)\n    \n    for tup in df.itertuples():\n        cv2.rectangle(img, (tup.XMin, tup.YMin), (tup.XMax, tup.YMax),\n                      color=(0,255,140), thickness=2)\n        cv2.putText(img, tup.Label, (tup.XMin+2, tup.YMax-2),\n                    fontFace=cv2.FONT_HERSHEY_SIMPLEX,\n                    fontScale=1, color=(255,0,0), thickness=2)\n        \n        #add segmentation mask\n    mask_files = [m for m in mask_list if im_file.stem in m.stem]    \n    mask_master = np.zeros_like(img)\n    np.random.seed(10)\n    for m in mask_files:\n        mask = cv2.imread(str(m))\n        mask = cv2.resize(mask, (w0,h0), interpolation = cv2.INTER_AREA)\n        color = np.random.choice([0,255], size=3)\n        mask[np.where((mask==[255, 255, 255]).all(axis=2))] = color\n        mask_master = cv2.add(mask_master, 0.5, 0)\n        \n    img = cv2.addWeighted(img, 1, mask_master, 0.5, 0)\n    \n    #Plotting images\n    plt.subplot(cols, rows, i)\n    plt.axis('off')\n    plt.imshow(img)\n    \n#print images \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"annotations = boxes_df.groupby('ImageID').agg(\n    box_count =('LabelName', 'size'),\n    box_unique= ('LabelName','nunique'))\n\npd.options.display.float_format = '{:,.1f}'.format\nannotations.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"annotations = boxes_df.groupby('ImageID').agg(\n    box_count =('LabelName', 'size'),\n    box_unique= ('LabelName','nunique'))\n\npd.options.display.float_format = '{:,.1f}'.format\nannotations.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all = annotations.hvplot.hist('box_count',width=600, bins=30)\nunique = annotations.hvplot.hist('box_unique', width=600)\n(all + unique).cols(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's another look at the number of boxes per images with the largest 1% removed","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"onepct = annotations.box_count.quantile(0.99)\nannotations.query('box_count < @onepct').box_count.value_counts(normalize=True) \\\n.sort_index().hvplot.bar(xticks=list(range(0,60,10)),width=600,\n                        line_alpha=0, xlabel='objects per image',\n                        ylabel='fraction of image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"Here's the skyscraper","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(boxes_df2.loc[boxes_df2.ImageID == 'fe7c6f7d298893da']\\\n     .groupby(['ImageID', 'Label'])['Label'].size())\n\nim_file = \"../input/excerpt-from-openimages-2020-train/train_00_part/fe7c6f7d298893da.jpg\"\nim = cv2.imread(im_file)\nplt.imshow(im)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Size\n\nThe dataset is huge! knowing image sizes can give an idea of the impact of size reduction. Here is now the test image sizes ae distributed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image\nfrom dask import bag, diagnostics\n\n\ndef faster_get_dims(file):\n    dims = Image.open(file).size\n    return dims\n\ndfile_list = glob.glob('../input/open-images-object-detection-rvc-2020/test/*.jpg')\nprint(f\"Getting dimensions for {len(dfile_list)} files.\")\n\n# parallelize\ndfile_bag = bag.from_sequence(dfile_list).map(faster_get_dims)\nwith diagnostics.ProgressBar():\n    dims_list = dfile_bag.compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = pd.DataFrame(dims_list, columns = ['width', 'height'])\ncounts = sizes.groupby(['width', 'height']).agg(count=('width', 'size'))\\\n.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_opts = dict(xlim=(0,1200),\n                 ylim = (0,1200),\n                 grid=True,\n                 xticks=[250,682,768,1024],\n                 yticks=[250,682,768,1024],\n                 height=500,\n                 width=550\n                )\n\nstyle_opts = dict(scaling_factor=0.2,\n                 line_alpha=1,\n                 fill_alpha=0.1\n                 )\n\ncounts.hvplot.scatter(x='width', y='height',size='count', **plot_opts).options(**style_opts)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Distribution of Object labels\nHere's a chart showing the frequency of the varouos types of objects. This is for detection, and for the train set, which will be different for instance segmentation and maybe for the test set. Overall thogh the datawill mostly be the pictures of \n* PEOPLE WITH FACES\n* WEARING VLOTHES\n* STANDING NEAR TREES, BUILDING\n*","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels = boxes_df[['ImageID','LabelName']].merge(labels,how='left',on='LabelName')\ntrain_labels.Label.value_counts(normalize=True)[:45]\\\n.hvplot.bar(width=650, height=350, rot=60, line_alpha=0,\n           title='Label Frequencies',\n           ylabel = 'fraction of all objetcts')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"relations = pd.read_csv(data_dir/'oidv6-relationship-triplets.csv')\nrelations = relations.merge(labels, how='left', left_on='LabelName1', right_on='LabelName') \\\n                     .merge(labels, how='left', left_on='LabelName2', right_on='LabelName',\n                            suffixes=['1', '2']) \\\n                     .loc[:, ['Label1', 'RelationshipLabel', 'Label2']] \\\n                     .dropna() \\\n                     .sort_values('RelationshipLabel') \\\n                     .reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import networkx as nx\n\nkids = relations.query('Label1==\"Girl\" or Label1==\"Boy\"')\nG = nx.from_pandas_edgelist(kids, 'Label1', 'Label2', 'RelationshipLabel')\n\n\ngraph_opts = dict(arrows=False,\n                  node_size=5,\n                  width=0.5,\n                  alpha=0.8,\n                  font_size=10,\n                  font_color='darkblue',\n                  edge_color='gray'\n                \n                 )\n\nfig= plt.figure(figsize=(12,10))\nnx.draw_spring(G, with_labels=True, **graph_opts)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank You","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}