{"cells":[{"metadata":{},"cell_type":"markdown","source":"This EDA is mainly to investigate the different sizes of images. [ChewZY has made a great Kernel about this](https://www.kaggle.com/chewzy/eda-weird-images-with-new-updates) yet I did want to try some additional things on my side, both for the excercise of actually implementing them myself, and to investigate some issues I am having on my model.\n\n(more content will most likely be added over the next few days, until the end of the competition)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport cv2\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imshow\nfrom PIL import Image\n\nfrom collections import Counter\n\n%matplotlib inline\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the size of the datafrmes we have:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_labels=pd.read_csv('../input/train.csv', dtype=str)\n#Changing the attribute ids into lists instead of str seperated by a ' ' to be able to count them\ntrain_labels['attribute_ids']=train_labels['attribute_ids'].str.split(' ')\ntest_labels=pd.read_csv('../input/sample_submission.csv', dtype=str)\n\nprint('train : \\n', train_labels.head())\nprint('\\ntest : \\n', test_labels.head())\n\nprint('\\ntrain shape: ', len(train_labels))\nprint('\\ntest shape: ', len(test_labels))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A quick look at the firt 5 labels, and the total length of the dataframe"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"labels = pd.read_csv('../input/labels.csv', dtype=str)\nprint('labels : ', '\\n', labels.head())\n\nprint('\\nlabels len :', len(labels))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First look:\n- 109,237 train images\n- 7,7443 test images (keep in mind that the final submission will predict on 5x more)\n- There are 1103 different labels, across 'culture' and 'tag'\n"},{"metadata":{},"cell_type":"markdown","source":"Getting a visual look at the images is always a good idea (keep in mind these are all 3 of the same style, and in no way representative of the entire dataset, which should always be remembered when looking at the top rows of a dataframe, or first couple of images)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Let's show a few images:\nfor i in range(3):\n    name_image=train_labels['id'][i]\n    image = plt.imread('../input/train/'+name_image+'.png')\n    plt.imshow(image)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I wanted to have a look at the distribution of size of the images in the train dataset. The first thing is to look at the distribtion across width and height."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Let's take a look at the sizes of the images:\n\nwidth_list = []\nheight_list = []\nfor i in range(len(train_labels)):\n    name_image=train_labels['id'][i]\n    with Image.open('../input/train/'+name_image+'.png') as img:\n        width, height = img.size\n        #print('width: {} \\nheight: {}'.format(width, height))\n        width_list.append(width)\n        height_list.append(height)\n        \naverage_width = sum(width_list)/len(width_list)\naverage_height = sum(height_list)/len(height_list)\n\nprint('average width: {} and height: {}'.format(average_width, average_height))\n\nfig, ax =plt.subplots(1,2, figsize=(15, 8))\n\nsns.distplot(width_list, ax=ax[0])\nax[0].set_title('Image width')\nsns.distplot(height_list, ax=ax[1])\nax[1].set_title('Image height')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The biggest images seem to go all the way to over 5500 pixels wide, and 7500 pixel tall! It might be worth taking a look at those images in more detail to see what they are of.\n\nWe can also note that the highest value for width is superior than for height, the y scale is different, so the visual comparaison of the two graphs should be done carefully."},{"metadata":{},"cell_type":"markdown","source":"What I was most interested in was the general shape of the images rather than width or height individually."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image_ratio_list = [int(x)/int(y) for x,y in zip(height_list, width_list)]\nmean_ratio = sum(image_ratio_list)/len(image_ratio_list)\nprint('mean ratio (height/width) of images is: ', mean_ratio)\n\nplt.subplots(figsize=(20, 8))\nsns.distplot(image_ratio_list, bins=100)\n#plt.axvline(mean_ratio,color='orange', label='mean')\nplt.axvline(x=1, color='red', label='x=1')\nplt.title('image ratio (height/width) distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This graph shows the distribution of images ratios. We can clearly see that a bit above and below 1 (the red line) there are pics of values. This clearly means most of the images are of slight rectangular shape, meaning resizing into a square might not be the ideal option for the data generators when feeding the neural network. We can also see that there are images that are up to 25 times higher than wide. (in order to see the wider than high images, we would need to get the inverse of this ratio, as all these images get squeezed near zero)."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}