{"cells":[{"metadata":{"_cell_guid":"96eeb801-2fd0-4cff-b6e7-1d6b1ee8d4ea","_uuid":"d8fd3dc80176c4d40042f2fa78d7f9f2fd80421a"},"cell_type":"markdown","source":"### Intro\nIn this kernel I explore the dataset for the CVPR competition. We are given images from videos produced as a car drives around and records activity from the car's point of view. My primary purpose is to see what's in the images and get a general feel for the objects we are asked to segment. \n\nI'll also create a data frame of labels for the train set that can be used for more in-depth analysis. Here in this kernel we'll just look at counts and distributions for the train set."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"trusted":true},"cell_type":"code","source":"import os\nimport random\nimport numpy as np\nimport pandas as pd \nfrom skimage import io\nfrom PIL import Image\nfrom IPython.display import display\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom dask.array.image import imread\nfrom dask import bag, threaded\nfrom dask.diagnostics import ProgressBar","execution_count":11,"outputs":[]},{"metadata":{"_cell_guid":"3edef9f1-b51a-40d9-b558-11785f705a07","_uuid":"51cf2d82e40e4ec2203210f0b4b9d631ecd2773d"},"cell_type":"markdown","source":"### Files\nFirst let's quickly look at the default file structure. There are 3 directories and a sample submission."},{"metadata":{"_cell_guid":"9d19cec1-e70f-42b5-a54a-2091ac4270cd","_uuid":"1eeffa47f8ce341801e580dba7705e7d3042570d","trusted":true},"cell_type":"code","source":"os.listdir('../input')","execution_count":12,"outputs":[]},{"metadata":{"_cell_guid":"1e0245d7-a394-4b4e-abf8-f13b57afcd63","_uuid":"90411e8ad325487e35fabd8dee365a3c01b73aa6"},"cell_type":"markdown","source":"Let's look first at the training images...."},{"metadata":{"_cell_guid":"10309bae-0552-41e1-80e5-f9a2327c003f","_uuid":"2c5ce290f8330b3448c2fdd88138bf95e3053fd8","trusted":true},"cell_type":"code","source":"def filecheck(dir):\n    dir_size = 0\n    filelist = os.listdir(dir)\n    filelist.sort()\n    print(dir)\n    for i,name in enumerate(filelist):\n        dir_size += os.path.getsize(os.path.join(dir, name))\n    print(\"{:.1f} GB of {} files\".format(dir_size/1024/1024/1024, i))\n    print(\"showing sample files\")\n    print(\"\\n\".join(filelist[300:306]) + \"\\n\")\n\ndirs = [\"../input/train_color\",\"../input/train_label\", \"../input/test\"]\n\nfor d in dirs[0:2]:\n    filecheck(d)","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"b23c1eac-d842-477c-a448-7d9fd61546cd","_uuid":"501f69c2b84343d2f235402cee950016c7eb39b9"},"cell_type":"markdown","source":"92.3GB of image data for the train set images! The filenames are interesting. The prefixes of the files might represent dates the pictures were taken as in 170927 means year (20)17, month 9, and day 27.  The middle string looks like it represents times and/or frame numbers. You can see that each instance from the middle set of numbers is listed twice - once for Camera 5 and once for Camera 6. \n\n[Recovering the Videos](https://www.kaggle.com/andrewrib/recovering-the-videos) has a cool example of stringing together sequential frames into a video from one of the cameras.\n\nBriefly looking at the test set files we see a more cyrptic naming convention. Our host recently uploaded a [Test Video List](https://www.kaggle.com/c/8899/download/test_video_list_and_name_mapping.zip) file that enables time sequencing of test images."},{"metadata":{"_cell_guid":"0b15aaad-9127-403f-ab50-6e2b021dc818","_uuid":"62c9f1ffb9a3059811e66839d22ea712cd73847d","trusted":true},"cell_type":"code","source":"j = os.listdir(dirs[2])\nprint(\"\\n\".join(j[0:6]))\nprint(\"{} files\".format(len(j)))","execution_count":14,"outputs":[]},{"metadata":{"_cell_guid":"c0b16bd0-229e-4003-bca6-4ff018460a0c","_uuid":"865ab3232bd0d9f88b84cdfccb350588bca96e6d"},"cell_type":"markdown","source":"### Images\nJumping back to the training images - let's look at an image with labels."},{"metadata":{"_uuid":"04db4a27c162288f4b1c800bb7b6dca078b86ad4","scrolled":true,"_cell_guid":"29d82aea-5349-41dd-83bf-f60448ee3aca","_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"im = Image.open(\"../input/train_color/170908_061523257_Camera_5.jpg\")\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_061523257_Camera_5_instanceIds.png\")) // 1000\ntlabel[tlabel != 0] = 255\n# plt.imshow(Image.blend(im, Image.fromarray(tlabel).convert('RGB'), alpha=0.4))\nplt.imshow(im)\ndisplay(plt.show())","execution_count":15,"outputs":[]},{"metadata":{"_cell_guid":"da914ff2-d70a-4763-a692-40a83725c43e","_uuid":"df5f062271c3d2dbbc41e285072b0ffd00cf1b65"},"cell_type":"markdown","source":"OK, so there are some vehicles and such on the road, just as we would expect. To see what those things are, we can look into the png file and extract the values emedded inside. From the Data page....\n\n#### The training images labels are encoded in a format mixing spatial and label/instance information:\n\n* All the images are the same size (width, height) of the original images\n* Pixel values indicate both the label and the instance.\n* Each label could contain multiple object instances.\n* int(PixelValue / 1000) is the label (class of object)\n* PixelValue % 1000 is the instance id\n* For example, a pixel value of 33000 means it belongs to label 33 (a car), is instance #0, while the pixel value of 33001 means it also belongs to class 33 (a car) , and is instance #1. These represent two different cars in an image.\n\nClever, eh? Let's look."},{"metadata":{"_cell_guid":"fa192f95-72af-42c6-92a2-b42d896521db","_uuid":"8f187e774949022a6dcb012708ffca8e3be3b35e","trusted":true},"cell_type":"code","source":"# cutting off everything after class 65, see note below\nclassdict = {0:'others', 1:'rover', 17:'sky', 33:'car', 34:'motorbicycle', 35:'bicycle', 36:'person', 37:'rider', 38:'truck', 39:'bus', 40:'tricycle', 49:'road', 50:'siderwalk', 65:'traffic_cone'}\n\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_061523257_Camera_5_instanceIds.png\"))\ncls = np.unique(tlabel)//1000\nunique, counts = np.unique(cls, return_counts=True)\nd = dict(zip(unique, counts))\ndf = pd.DataFrame.from_dict(d, orient='index').transpose()\ndf.rename(columns=classdict, inplace=True)\ndf","execution_count":16,"outputs":[]},{"metadata":{"_cell_guid":"1861fe10-d08e-49ed-8d13-792e4b854c43","_uuid":"b597fcd26754be12a7cd484eedcdbff2f860222c"},"cell_type":"markdown","source":"According to the data we have 5 cars, a bus, a tricycle and a traffic cone (see note below for 'traffic cone'). OK, sure... Let's also look at Camera 6 for the same instance.\n"},{"metadata":{"_kg_hide-input":true,"_cell_guid":"db4eaf65-2e5c-459b-8b02-a44fbc1934b5","_uuid":"5331f8da5ec156248636f47e03975d1a9b3580f3","trusted":true},"cell_type":"code","source":"im = Image.open(\"../input/train_color/170908_073302618_Camera_6.jpg\")\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_073302618_Camera_6_instanceIds.png\"))//1000\ntlabel[tlabel != 0] = 255\nplt.imshow(Image.blend(im, Image.fromarray(tlabel).convert('RGB'), alpha=0.6))\n\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_061523257_Camera_6_instanceIds.png\"))\ncls = np.unique(tlabel)//1000\nunique, counts = np.unique(cls, return_counts=True)\nd = dict(zip(unique, counts))\ndf = pd.DataFrame.from_dict(d, orient='index').transpose()\ndf.rename(columns=classdict, inplace=True)\n\ndisplay(plt.show())\ndf","execution_count":17,"outputs":[]},{"metadata":{"_cell_guid":"0d930d29-2cf3-490c-aaa4-cdb916a82e03","_uuid":"54a10a56b6993b9f30af9cd9dc276daf559a7e03"},"cell_type":"markdown","source":"Camera 6 shows a different view as we might expect. The [First Look](https://www.kaggle.com/aakashnain/firstlook) kernel has more instances of images and masks."},{"metadata":{"_cell_guid":"4e0320d7-bad4-4d21-a346-1bd71860d446","_uuid":"fa2177ede5372cca86fe5acae6b8989abd4a2d71"},"cell_type":"markdown","source":"### Labels (masks)\nLet's now pull labels for the training images and look at some basic stats. I'll write the labels out to csv for everyone's convenience.\n\nNote: The code below can probably be further optimized for better performance. I have it down to less than 1/5 the time of the original code. The latest version uses Dask for multiprocessing. Coding with Dask seems easier than directly using the Multiprocessing package, but not faster, at least the way I'm doing it. "},{"metadata":{"_cell_guid":"63d34401-8043-45d5-8f99-e9d502bff206","_uuid":"71b6e9be6cfe5dc8c524504b3601eb90f2489e64","trusted":true},"cell_type":"code","source":"# dask imread version - in progress\ndef divmil():\n    return 1\n# labels is the image array\nlabels = imread(\"../input/train_label/*.png\")\nprint(labels.shape, labels[0].shape, divmil)\n\n# counts is the array of counts\n    # look only at values between 33000 and 39999\n\n\n# with ProgressBar():\n#     labels.compute()\n","execution_count":21,"outputs":[]},{"metadata":{"_cell_guid":"a61e6252-ef93-4fb3-95c3-79bce133cf2e","_uuid":"8235b886888221a159377b58a5f234bfae90ad2a","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"filenames = os.listdir(dirs[1])\n# filenames = filenames[:100]            # for testing\nfullpaths = [\"../input/train_label/\" + f for f in filenames]\n\n# set up a bag\ndef get_ims(impath):\n    tlabel = io.imread(impath, plugin='pil')\n    cls = np.unique(tlabel)\n    unique,counts = np.unique(cls//1000, return_counts=True)\n    ds = dict(zip(unique, counts))\n    return ds\n\nlabelbag = bag.from_sequence(fullpaths).map(get_ims)\nwith ProgressBar():\n    labels = labelbag.compute()\n    ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"bf085244-1d19-49e8-a533-257a94fda87e","_uuid":"89d29db1198da91100c4053d290028a704284e05","collapsed":true,"trusted":false},"cell_type":"code","source":"    \nlabels_df = pd.DataFrame(labels, index=filenames, dtype='uint8') # dtype not working?\nlabels_df.fillna(value=0, inplace=True)\nlabels_df = labels_df.astype(int)\nlabels_df.rename(columns=classdict, inplace=True)      \nlabels_df.drop(columns=['others', 'rider', 'traffic_cone'], inplace=True)\n\nlabels_df.to_csv('train_labels.csv')\nlabels_df.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"20f9e7c2-4c93-42cd-b370-5cbce79b30bf","_uuid":"a10267fec181e79a797106e49fae285ea2e06399"},"cell_type":"markdown","source":"We can look at the frequency of classes in the images by summing the occurrences across all images."},{"metadata":{"_cell_guid":"2faf7c22-a021-4674-92c4-a0c98884ad68","_uuid":"eab0d33047a74422cdb7635014cdd0aa3035a9f2","collapsed":true,"trusted":false},"cell_type":"code","source":"\nclasses_df = pd.melt(labels_df)\ngroups = classes_df.groupby('variable')\nsums = groups.sum()\n\n\nsns.set(style='whitegrid')\nax = sns.barplot(x=sums.index, y=sums.value, color='steelblue')\nax.set(xlabel='', ylabel='count')\nsns.despine(left=True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6ba68243-f0ae-4016-9819-62791ab633dd","_uuid":"5f48a36fe829a4ad8cbbbac687e54aa43c25319e"},"cell_type":"markdown","source":"The most prevalent class is cars by far, as you might expect. As pointed out in [this discussion](https://www.kaggle.com/c/cvpr-2018-autonomous-driving/discussion/53845), only 7 of the classes will be used for evaluation. These are car, motorbicycle, bicycle, person, truck, bus, and tricycle. Class 65, traffic cone, is actually a false label, It comes from pixel value 65535 which represents the \"ignoring label\".\n\nAnyway, let's look at the differences among images. Here are plots of Total Objects per image and Distinct Classes per image."},{"metadata":{"_cell_guid":"b8562d6b-f817-4a49-9104-665fda900d1a","_uuid":"c1da47688f6da414bd4027d0fd4b8dbd96c3ae62","collapsed":true,"trusted":false},"cell_type":"code","source":"labels_df['objects'] = labels_df.sum(axis=1)\nlabels_df['classes'] = labels_df[labels_df>0].count(axis=1)-1\nlabels_df.clip(lower=0, inplace=True)   # crude fix for when no objects are seen\nlabels_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1ab10eb2-684f-47e1-8b14-de4130f7fca5","_uuid":"9de1d7a26299ed80f00c26dc2d3ecdbd94271a7b","collapsed":true,"trusted":false},"cell_type":"code","source":"plt.figure();\nplt.title(\"Total # of Objects\")\nlabels_df['objects'].plot.hist()\n\nplt.figure();\nplt.title(\"# of Distinct Classes\")\nlabels_df['classes'].value_counts().sort_index().plot.bar(color='steelblue')\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"aef72bc8-40e0-4c7d-bd6c-b663d568e672","_uuid":"66a4046d55c32977271dffc98fce9324918fe015"},"cell_type":"markdown","source":"It's interesting to see quite a difference among the images, expecially for Total Counts.\n\nThere's a lot more that can be done with this data, of course, and I look forward to seeing some great kernels!\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}