{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"7102be26-0a75-3c16-a061-5ebc223d411d"},"source":"*Poonam Ligade*\n\n*16th March 2017*\n\n\nHi Kagglers,\n\nIn this notebook I am trying to capture various aspects of data which are images.\n\nWe have to create an algorithm which  accurately classifies women's cervix type based on images.\n\nSo its clearly multiclass classification problem where 3 classes are\n\n 1.  Type_1 \n 2.  Type_2 \n 3. Type_3\n\nThis will eventually prevent ineffective treatments for wrongly identified cervix types."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"254a6152-da08-36c3-2d2a-634b184705a5"},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom scipy.misc import imread\nfrom glob import glob\nimport random\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input/train\"]).decode('utf-8'))\n\n# Any results you write to the current directory are saved as output."},{"cell_type":"markdown","metadata":{"_cell_guid":"fa1ee2bd-ee35-2217-fda8-03e4bfe0e04d"},"source":"**Visualization**\n-----------------\n\nLet's visualize all 3 types of images"},{"cell_type":"markdown","metadata":{"_cell_guid":"7cdec286-149b-a947-8739-a9410c1cc644"},"source":"Type 1"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"014c88ba-4504-0ad5-77de-f88bc7fb0549"},"outputs":[],"source":"#fig = plt.figure(figsize=(12,8))\nsize = 256, 256\n\nim=Image.open('../input/train/Type_1/10.jpg')\n#im.thumbnail(size, Image.ANTIALIAS)\n#print (im.format, im.size, im.mode)\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"e05793e2-23b0-a7ea-c207-327164c212ad"},"source":"Type 2"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8460e9ca-73d8-22c6-37d8-bb7bff71eb14"},"outputs":[],"source":"im=Image.open('../input/train/Type_2/100.jpg')\nim.thumbnail(size, Image.ANTIALIAS)\n#print (im.format, im.size, im.mode)\nim"},{"cell_type":"markdown","metadata":{"_cell_guid":"888e19cd-13bf-6715-5f97-5f1c1ea30e73"},"source":"Type 3"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ac193f40-2d7e-884e-bbf2-7b26ac0e9357"},"outputs":[],"source":"im=Image.open('../input/train/Type_3/1000.jpg')\nim.thumbnail(size, Image.ANTIALIAS)\n#print (im.format, im.size, im.mode)\nim"},{"cell_type":"markdown","metadata":{"_cell_guid":"cd94d6c5-c075-6065-6711-e168ed7905a8"},"source":"**Image Statistics**\n--------------------\n\nNow lets look at some of image statistics in train, additional and test folders."},{"cell_type":"markdown","metadata":{"_cell_guid":"a02700b2-ed80-a212-713c-ac1a6cae1578"},"source":"**Number of Images**\n--------------------"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"99a242ae-e573-82b9-aea1-1e13c93ccb29"},"outputs":[],"source":"sub_folders = check_output([\"ls\", \"../input/train/\"]).decode(\"utf8\").strip().split('\\n')\ncount_dict = {}\nfor sub_folder in sub_folders:\n    num_of_files = len(check_output([\"ls\", \"../input/train/\"+sub_folder]).decode(\"utf8\").strip().split('\\n'))\n    print(\"{0} photos of cervix type {1} \".format(num_of_files, sub_folder))\n\n    count_dict[sub_folder] = num_of_files\n    \nplt.figure(figsize=(12,4))\nsns.barplot(list(count_dict.keys()), list(count_dict.values()), alpha=0.8)\nplt.xlabel('Cervix types', fontsize=12)\nplt.ylabel('Number of Images in train', fontsize=12)\nplt.title(\"train dataset\")\n\nplt.show()\n    "},{"cell_type":"markdown","metadata":{"_cell_guid":"6b52b373-5eb2-cda4-89e8-5f446b4c1b10"},"source":"Clearly Type 2 has almost double number of images than Type 3 and 3 times more than Type 1 .\n\nLets look at the additional folder"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3de2ceef-8d62-ef30-230a-a957692001e1"},"outputs":[],"source":"sub_folders = check_output([\"ls\", \"../input/additional/\"]).decode(\"utf8\").strip().split('\\n')\ncount_dict = {}\nfor sub_folder in sub_folders:\n    num_of_files = len(check_output([\"ls\", \"../input/additional/\"+sub_folder]).decode(\"utf8\").strip().split('\\n'))\n    print(\"{0} photos of cervix type {1} \".format(num_of_files, sub_folder))\n\n    count_dict[sub_folder] = num_of_files\n    \nplt.figure(figsize=(12,4))\nsns.barplot(list(count_dict.keys()), list(count_dict.values()), alpha=0.8)\nplt.xlabel('Cervix types', fontsize=12)\nplt.ylabel('Number of Images in additional', fontsize=12)\nplt.title(\"Additional dataset\")\nplt.show()\n    "},{"cell_type":"markdown","metadata":{"_cell_guid":"5f8319ef-c1eb-f07f-587b-53e0c5dd3a78"},"source":"Type_2 has got many more images than rest of the two. Distribution is similar to that of train datset.\n\nLets look at the test folder"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"2b5cb2b4-156a-b98f-e6f0-bbed5c410dff"},"outputs":[],"source":"num_test_files = len(check_output([\"ls\", \"../input/test/\"]).decode(\"utf8\").strip().split('\\n'))\nprint(\"Number of test images present :\", num_test_files)"},{"cell_type":"markdown","metadata":{"_cell_guid":"2fa2458c-1cbc-1100-40c3-645fea4dd072"},"source":"**Sizes of images**\n-------------------"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"6e25542f-5a95-2f6c-798c-4c8a88f4fc18"},"outputs":[],"source":"train_path = \"../input/train/\"\nsub_folders = check_output([\"ls\", train_path]).decode(\"utf8\").strip().split('\\n')\ndifferent_file_sizes = {}\nfor sub_folder in sub_folders:\n    file_names = check_output([\"ls\", train_path+sub_folder]).decode(\"utf8\").strip().split('\\n')\n    for file_name in file_names:\n        im_array = imread(train_path+sub_folder+\"/\"+file_name)\n        size = \"_\".join(map(str,list(im_array.shape)))\n        different_file_sizes[size] = different_file_sizes.get(size,0) + 1\n\nplt.figure(figsize=(12,4))\nsns.barplot(list(different_file_sizes.values()), list(different_file_sizes.keys()), alpha=0.8)\nplt.ylabel('Image size', fontsize=12)\nplt.xlabel('Number of Images in train', fontsize=12)\nplt.title(\"Image sizes present in train dataset\")\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"2d2671ec-75ad-bdba-4c89-0988d8b66551"},"source":"There are  8 different sizes are available in training data.\n\n3264_2448_3 is the most common image size available,  followed by 4128_3096_3.\n\n3088_4128_3 is the largest size of the available images in train set and 3264_2448_3 is the smallest one."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ca73957a-2c92-a6b4-8847-24f3197266f3"},"outputs":[],"source":"test_path = \"../input/test/\"\nfile_names = check_output([\"ls\", test_path]).decode(\"utf8\").strip().split('\\n')\ndifferent_file_sizes = {}\nfor file_name in file_names:\n        size = \"_\".join(map(str,list(imread(test_path+file_name).shape)))\n        different_file_sizes[size] = different_file_sizes.get(size,0) + 1\n\nplt.figure(figsize=(12,4))\nsns.barplot(list(different_file_sizes.keys()), list(different_file_sizes.values()), alpha=0.8)\nplt.xlabel('File size', fontsize=12)\nplt.ylabel('Number of Images in test', fontsize=12)\nplt.xticks(rotation='vertical')\nplt.title(\"Image size present in test dataset\")\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"3cfbcc93-7ad3-97dd-c126-c68561f439f9"},"source":"Test set also has similar size distribution"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f63db9c8-b79b-8d53-343c-6dcd00fcc3b3"},"outputs":[],"source":"additional_path = \"../input/additional/\"\nsub_folders = check_output([\"ls\", additional_path]).decode(\"utf8\").strip().split('\\n')\ndifferent_file_sizes = {}\ncorrupted_images=['2845.jpg','5892.jpg','5893.jpg']\nfor sub_folder in sub_folders:\n    file_names = check_output([\"ls\", additional_path+sub_folder]).decode(\"utf8\").strip().split('\\n')\n    if(sub_folder=='Type_2'):\n        try:\n            file_names.remove('5892.jpg')\n            file_names.remove('5893.jpg')\n            file_names.remove('2845.jpg')\n        except ValueError:\n            pass\n    for file_name in file_names:\n        im_array = imread(additional_path+sub_folder+\"/\"+file_name)\n        size = \"_\".join(map(str,list(im_array.shape)))\n        different_file_sizes[size] = different_file_sizes.get(size,0) + 1\n\nplt.figure(figsize=(12,4))\nsns.barplot(list(different_file_sizes.values()), list(different_file_sizes.keys()), alpha=0.8)\nplt.ylabel('Image size', fontsize=12)\nplt.xlabel('Number of Images in additional', fontsize=12)\nplt.title(\"Image sizes present in additional dataset\")\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"a1c496e3-d5ae-54a7-f3a0-facb96fdba7f"},"source":" few images in additional dataset are corrupt as mentioned in [this][1] kernel.\nWe have removed them to get statistics.\n\n\nMore to come. Please upvote if you like it.\n\n  [1]: https://www.kaggle.com/aamaia/intel-mobileodt-cervical-cancer-screening/three-empty-images-in-additional-7z"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8917a18b-0b6a-89ea-bbe2-8471ddbfc1b3"},"outputs":[],"source":""}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}