{"cells":[{"metadata":{"_cell_guid":"96eeb801-2fd0-4cff-b6e7-1d6b1ee8d4ea","_uuid":"d8fd3dc80176c4d40042f2fa78d7f9f2fd80421a"},"cell_type":"markdown","source":"### Intro\nIn this kernel I explore the dataset for the CVPR competition. We are given images from videos produced as a car drives around and records activity from the car's point of view. My primary purpose is to see what's in the images and get a general feel for the objects we are asked to segment. I'll also create a data frame of labels for the train set that can be used for more in-depth classification.\n\nNew: I've been trying to optimize the code for reading all train images and generating the labels. Maybe you have some ideas to speed it up?."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","collapsed":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"import os\nimport random\nimport numpy as np\nimport pandas as pd \nfrom skimage import io, img_as_float\nfrom numba import jit\nfrom PIL import Image\nimport cv2\nimport tensorflow as tf\nfrom IPython.display import display\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3edef9f1-b51a-40d9-b558-11785f705a07","_uuid":"51cf2d82e40e4ec2203210f0b4b9d631ecd2773d"},"cell_type":"markdown","source":"### Files\nFirst let's quickly look at the default file structure. There are 3 directories and a sample submission."},{"metadata":{"_cell_guid":"9d19cec1-e70f-42b5-a54a-2091ac4270cd","collapsed":true,"_uuid":"1eeffa47f8ce341801e580dba7705e7d3042570d","trusted":false},"cell_type":"code","source":"os.listdir('../input')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1e0245d7-a394-4b4e-abf8-f13b57afcd63","_uuid":"90411e8ad325487e35fabd8dee365a3c01b73aa6"},"cell_type":"markdown","source":"Let's look first at the training images...."},{"metadata":{"_cell_guid":"10309bae-0552-41e1-80e5-f9a2327c003f","collapsed":true,"_uuid":"2c5ce290f8330b3448c2fdd88138bf95e3053fd8","trusted":false},"cell_type":"code","source":"def filecheck(dir):\n    dir_size = 0\n    filelist = os.listdir(dir)\n    filelist.sort()\n    print(dir)\n    for i,name in enumerate(filelist):\n        dir_size += os.path.getsize(os.path.join(dir, name))\n    print(\"{:.1f} GB of {} files\".format(dir_size/1024/1024/1024, i))\n    print(\"showing sample files\")\n    print(\"\\n\".join(filelist[300:306]) + \"\\n\")\n\ndirs = [\"../input/train_color\",\"../input/train_label\", \"../input/test\"]\n\nfor d in dirs[0:2]:\n    filecheck(d)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b23c1eac-d842-477c-a448-7d9fd61546cd","_uuid":"501f69c2b84343d2f235402cee950016c7eb39b9"},"cell_type":"markdown","source":"92.3GB of image data for the train set images! The filenames are interesting. The prefixes of the files, \"170908\" might represent dates the pictures were taken, and the middle string might represent times and/or frame numbers. You can see that each \"instance\" - the middle set of numbers - has images from both Camera 5 and Camera 6. From what I've seen they always come in pairs like that, which probably means there are (at least) two cameras recording at the same time. As expected, each jpg image in the train set has a corresponding png image with the mask of objects to classify. \n\nEdit: This awesome kernel, [Recovering the Videos](https://www.kaggle.com/andrewrib/recovering-the-videos) string together sequential frames into videos.\n\nBriefly looking at the test set files we see a more cyrptic naming convention. Our host refers to a \"test video\" in the Welcome post which suggests these files are also sequential somehow."},{"metadata":{"_cell_guid":"0b15aaad-9127-403f-ab50-6e2b021dc818","collapsed":true,"_uuid":"62c9f1ffb9a3059811e66839d22ea712cd73847d","trusted":false},"cell_type":"code","source":"j = os.listdir(dirs[2])\nprint(\"\\n\".join(j[0:6]))\nprint(\"{} files\".format(len(j)))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c0b16bd0-229e-4003-bca6-4ff018460a0c","_uuid":"865ab3232bd0d9f88b84cdfccb350588bca96e6d"},"cell_type":"markdown","source":"### Images\nJumping back to the training images - let's look at an image with labels."},{"metadata":{"_cell_guid":"29d82aea-5349-41dd-83bf-f60448ee3aca","collapsed":true,"_kg_hide-output":false,"_uuid":"04db4a27c162288f4b1c800bb7b6dca078b86ad4","trusted":false},"cell_type":"code","source":"im = Image.open(\"../input/train_color/170908_061523257_Camera_5.jpg\")\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_061523257_Camera_5_instanceIds.png\")) // 1000\ntlabel[tlabel != 0] = 255\nplt.imshow(Image.blend(im, Image.fromarray(tlabel).convert('RGB'), alpha=0.4))\ndisplay(plt.show())","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"da914ff2-d70a-4763-a692-40a83725c43e","_uuid":"df5f062271c3d2dbbc41e285072b0ffd00cf1b65"},"cell_type":"markdown","source":"OK, so there are some vehicles and such on the road, just as we would expect. To see what those things are, we can look into the png file and extract the values emedded inside. From the Data page....\n\n#### The training images labels are encoded in a format mixing spatial and label/instance information:\n\n* All the images are the same size (width, height) of the original images\n* Pixel values indicate both the label and the instance.\n* Each label could contain multiple object instances.\n* int(PixelValue / 1000) is the label (class of object)\n* PixelValue % 1000 is the instance id\n* For example, a pixel value of 33000 means it belongs to label 33 (a car), is instance #0, while the pixel value of 33001 means it also belongs to class 33 (a car) , and is instance #1. These represent two different cars in an image.\n\nClever, eh? Let's look."},{"metadata":{"_cell_guid":"fa192f95-72af-42c6-92a2-b42d896521db","collapsed":true,"_uuid":"8f187e774949022a6dcb012708ffca8e3be3b35e","trusted":false},"cell_type":"code","source":"classdict = {0:'others', 1:'rover', 17:'sky', 33:'car', 34:'motorbicycle', 35:'bicycle', 36:'person', 37:'rider', 38:'truck', 39:'bus', 40:'tricycle', 49:'road', 50:'siderwalk', 65:'traffic_cone', 66:'road_pile', 67:'fence', 81:'traffic_light', 82:'pole', 83:'traffic_sign', 84:'wall', 85:'dustbin', 86:'billboard', 97:'building', 98:'bridge', 99:'tunnel', 100:'overpass', 113:'vegatation', 161:'car_groups', 162:'motorbicycle_group', 163:'bicycle_group', 164:'person_group', 165:'rider_group', 166:'truck_group', 167:'bus_group', 168:'tricycle_group'}\n\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_061523257_Camera_5_instanceIds.png\"))\ncls = np.unique(tlabel)//1000\nunique, counts = np.unique(cls, return_counts=True)\nd = dict(zip(unique, counts))\ndf = pd.DataFrame.from_dict(d, orient='index').transpose()\ndf.rename(columns=classdict, inplace=True)\ndf","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1861fe10-d08e-49ed-8d13-792e4b854c43","_uuid":"b597fcd26754be12a7cd484eedcdbff2f860222c"},"cell_type":"markdown","source":"According to the data we have 5 cars, a bus, a tricycle and a traffic cone (see note below for 'traffic cone'). OK, sure... Let's also look at Camera 6 for the same instance.\n"},{"metadata":{"_cell_guid":"db4eaf65-2e5c-459b-8b02-a44fbc1934b5","_kg_hide-input":true,"collapsed":true,"_uuid":"5331f8da5ec156248636f47e03975d1a9b3580f3","trusted":false},"cell_type":"code","source":"im = Image.open(\"../input/train_color/170908_073302618_Camera_6.jpg\")\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_073302618_Camera_6_instanceIds.png\"))//1000\ntlabel[tlabel != 0] = 255\nplt.imshow(Image.blend(im, Image.fromarray(tlabel).convert('RGB'), alpha=0.6))\n\ntlabel = np.asarray(Image.open(\"../input/train_label/170908_061523257_Camera_6_instanceIds.png\"))\ncls = np.unique(tlabel)//1000\nunique, counts = np.unique(cls, return_counts=True)\nd = dict(zip(unique, counts))\ndf = pd.DataFrame.from_dict(d, orient='index').transpose()\ndf.rename(columns=classdict, inplace=True)\n\ndisplay(plt.show())\ndf","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0d930d29-2cf3-490c-aaa4-cdb916a82e03","_uuid":"54a10a56b6993b9f30af9cd9dc276daf559a7e03"},"cell_type":"markdown","source":"Camera 6 shows a different view as we might expect. The [First Look](https://www.kaggle.com/aakashnain/firstlook) kernel has more instances of images and masks."},{"metadata":{"_cell_guid":"4e0320d7-bad4-4d21-a346-1bd71860d446","_uuid":"fa2177ede5372cca86fe5acae6b8989abd4a2d71"},"cell_type":"markdown","source":"### Labels (masks)\nLet's now pull labels for the training images and look at some basic stats. (Note: The code below can probably be further optimized for better performance. I have it down to less than 1/2 of the original code. I'll still work with a sample here for now to save time.)"},{"metadata":{"_cell_guid":"a61e6252-ef93-4fb3-95c3-79bce133cf2e","scrolled":true,"collapsed":true,"_uuid":"8235b886888221a159377b58a5f234bfae90ad2a","trusted":false},"cell_type":"code","source":"allfilenames = os.listdir(dirs[1])\nfilenames = random.sample(allfilenames, 1000)\nfullpaths = [\"../input/train_label/\" + f for f in filenames]\nlabarray = np.zeros((len(filenames), 66))\n\n@jit\ndef getcounts():\n    for i,f in enumerate(tqdm(fullpaths)):\n        tlabel = io.imread(f, plugin='pil')\n        cls = np.unique(tlabel)\n        unique,counts = np.unique(cls//1000, return_counts=True)\n        labarray[i, unique] = counts\n\ngetcounts()\nlabels_df = pd.DataFrame(labarray, index=filenames)\nlabels_df = labels_df.loc[:, (labels_df != 0).any(axis=0)]\nlabels_df.rename(columns=classdict, inplace=True)        \nlabels_df.head(6)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"20f9e7c2-4c93-42cd-b370-5cbce79b30bf","_uuid":"a10267fec181e79a797106e49fae285ea2e06399"},"cell_type":"markdown","source":"We can look at the frequency of classes in the images by summing the occurrences across all images."},{"metadata":{"_cell_guid":"2faf7c22-a021-4674-92c4-a0c98884ad68","collapsed":true,"_uuid":"eab0d33047a74422cdb7635014cdd0aa3035a9f2","trusted":false},"cell_type":"code","source":"labels_df.drop('others', axis=1, inplace=True)\nclasses_df = pd.melt(labels_df)\ngroups = classes_df.groupby('variable')\nsums = groups.sum()\n\n\nplt.figure();\nsums.plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6ba68243-f0ae-4016-9819-62791ab633dd","_uuid":"5f48a36fe829a4ad8cbbbac687e54aa43c25319e"},"cell_type":"markdown","source":"The most prevalent class is cars by far, as you might expect. I find it curious that most of the classes are not represented anywhere. Assuming the code is correct, it could be due to the limited sample, or the classes may be extremely rare. \n\nEdit: as pointed out in [this discussion](https://www.kaggle.com/c/cvpr-2018-autonomous-driving/discussion/53845), only 7 of the classes will be used for evaluation. These are car, motorbicycle, bicycle, person, truck, bus, and tricycle. Class 65, traffic cone, is actually a false label, It comes from pixel value 65535 which represents the \"ignoring label\".\n\nAnyway, let's look at the differences among images. Here are histograms of Total Objects per image and Distinct Classes per image."},{"metadata":{"_cell_guid":"b8562d6b-f817-4a49-9104-665fda900d1a","collapsed":true,"_uuid":"c1da47688f6da414bd4027d0fd4b8dbd96c3ae62","trusted":false},"cell_type":"code","source":"labels_df['objects'] = labels_df.sum(axis=1)\nlabels_df['classes'] = labels_df[labels_df>0].count(axis=1)-1\nlabels_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1ab10eb2-684f-47e1-8b14-de4130f7fca5","collapsed":true,"_uuid":"9de1d7a26299ed80f00c26dc2d3ecdbd94271a7b","trusted":false},"cell_type":"code","source":"plt.figure();\nplt.title(\"Total # of Objects\")\nlabels_df['objects'].plot.hist()\n\nplt.figure();\nplt.title(\"# of Distinct Classes\")\nlabels_df['classes'].plot.hist()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"aef72bc8-40e0-4c7d-bd6c-b663d568e672","_uuid":"66a4046d55c32977271dffc98fce9324918fe015"},"cell_type":"markdown","source":"It's interesting to see quite a difference among the images, expecially for total counts. \n\nThere's a lot more that can be done with this data, of course, and I look forward to seeing some great kernels!\n\n"},{"metadata":{"_cell_guid":"d460cfe1-5aa6-43d6-87bc-e37ef588239b","collapsed":true,"_uuid":"d0828858c8edf2519290f7442684581f386e1bac","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}