{"cells":[{"metadata":{"_cell_guid":"7102be26-0a75-3c16-a061-5ebc223d411d"},"cell_type":"markdown","source":"*Poonam Ligade*\n\n*16th March 2017*\n\n\nHi Kagglers,\n\nIn this notebook I am trying to capture various aspects of data which are images.\n\nWe have to create an algorithm which  accurately classifies women's cervix type based on images.\n\nSo its clearly multiclass classification problem where 3 classes are\n\n 1.  Type_1 \n 2.  Type_2 \n 3. Type_3\n\nThis will eventually prevent ineffective treatments for wrongly identified cervix types.","execution_count":null},{"metadata":{"_cell_guid":"254a6152-da08-36c3-2d2a-634b184705a5","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\n#from scipy.misc import imread\nfrom glob import glob\nimport random\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input/train/\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fa1ee2bd-ee35-2217-fda8-03e4bfe0e04d"},"cell_type":"markdown","source":"**Visualization**\n-----------------\n\nLet's visualize all 3 types of images","execution_count":null},{"metadata":{"_cell_guid":"7cdec286-149b-a947-8739-a9410c1cc644"},"cell_type":"markdown","source":"Type 1","execution_count":null},{"metadata":{"_cell_guid":"014c88ba-4504-0ad5-77de-f88bc7fb0549","trusted":true},"cell_type":"code","source":"#fig = plt.figure(figsize=(12,8))\nsize = 256, 256\n\nim=Image.open('../input/train/train/Type_1/10.jpg')\nim.thumbnail(size, Image.ANTIALIAS)\n#print (im.format, im.size, im.mode)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e05793e2-23b0-a7ea-c207-327164c212ad"},"cell_type":"markdown","source":"Type 2","execution_count":null},{"metadata":{"_cell_guid":"8460e9ca-73d8-22c6-37d8-bb7bff71eb14","trusted":true},"cell_type":"code","source":"im=Image.open('../input/train/train/Type_2/100.jpg')\nim.thumbnail(size, Image.ANTIALIAS)\n#print (im.format, im.size, im.mode)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"888e19cd-13bf-6715-5f97-5f1c1ea30e73"},"cell_type":"markdown","source":"Type 3","execution_count":null},{"metadata":{"_cell_guid":"ac193f40-2d7e-884e-bbf2-7b26ac0e9357","trusted":true},"cell_type":"code","source":"im=Image.open('../input/train/train/Type_3/1000.jpg')\nim.thumbnail(size, Image.ANTIALIAS)\n#print (im.format, im.size, im.mode)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cd94d6c5-c075-6065-6711-e168ed7905a8"},"cell_type":"markdown","source":"**Image Statistics**\n--------------------\n\nNow lets look at some of image statistics in train, additional and test folders.","execution_count":null},{"metadata":{"_cell_guid":"a02700b2-ed80-a212-713c-ac1a6cae1578"},"cell_type":"markdown","source":"**Number of Images**\n--------------------","execution_count":null},{"metadata":{"_cell_guid":"99a242ae-e573-82b9-aea1-1e13c93ccb29","trusted":true},"cell_type":"code","source":"sub_folders = check_output([\"ls\", \"../input/train/\"]).decode(\"utf8\").strip().split('\\n')\ncount_dict = {}\nfor sub_folder in sub_folders:\n    num_of_files = len(check_output([\"ls\", \"../input/train/\"+sub_folder]).decode(\"utf8\").strip().split('\\n'))\n    print(\"{0} photos of cervix type {1} \".format(num_of_files, sub_folder))\n\n    count_dict[sub_folder] = num_of_files\n    \nplt.figure(figsize=(12,4))\nsns.barplot(list(count_dict.keys()), list(count_dict.values()), alpha=0.8)\nplt.xlabel('Cervix types', fontsize=12)\nplt.ylabel('Number of Images in train', fontsize=12)\nplt.title(\"train dataset\")\n\nplt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6b52b373-5eb2-cda4-89e8-5f446b4c1b10"},"cell_type":"markdown","source":"Clearly Type 2 has almost double number of images than Type 3 and 3 times more than Type 1 .\n\nLets look at the additional folder","execution_count":null},{"metadata":{"_cell_guid":"3de2ceef-8d62-ef30-230a-a957692001e1","trusted":true},"cell_type":"code","source":"sub_folders = check_output([\"ls\", \"../input/additional/\"]).decode(\"utf8\").strip().split('\\n')\ncount_dict = {}\nfor sub_folder in sub_folders:\n    num_of_files = len(check_output([\"ls\", \"../input/additional/\"+sub_folder]).decode(\"utf8\").strip().split('\\n'))\n    print(\"{0} photos of cervix type {1} \".format(num_of_files, sub_folder))\n\n    count_dict[sub_folder] = num_of_files\n    \nplt.figure(figsize=(12,4))\nsns.barplot(list(count_dict.keys()), list(count_dict.values()), alpha=0.8)\nplt.xlabel('Cervix types', fontsize=12)\nplt.ylabel('Number of Images in additional', fontsize=12)\nplt.title(\"Additional dataset\")\nplt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5f8319ef-c1eb-f07f-587b-53e0c5dd3a78"},"cell_type":"markdown","source":"Type_2 has got many more images than rest of the two. Distribution is similar to that of train datset.\n\nLets look at the test folder","execution_count":null},{"metadata":{"_cell_guid":"2b5cb2b4-156a-b98f-e6f0-bbed5c410dff","trusted":true},"cell_type":"code","source":"num_test_files = len(check_output([\"ls\", \"../input/test/\"]).decode(\"utf8\").strip().split('\\n'))\nprint(\"Number of test images present :\", num_test_files)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2fa2458c-1cbc-1100-40c3-645fea4dd072"},"cell_type":"markdown","source":"**Sizes of images**\n-------------------","execution_count":null},{"metadata":{"_cell_guid":"6e25542f-5a95-2f6c-798c-4c8a88f4fc18","trusted":true},"cell_type":"code","source":"train_path = \"../input/train/\"\nsub_folders = check_output([\"ls\", train_path]).decode(\"utf8\").strip().split('\\n')\ndifferent_file_sizes = {}\nfor sub_folder in sub_folders:\n    file_names = check_output([\"ls\", train_path+sub_folder]).decode(\"utf8\").strip().split('\\n')\n    for file_name in file_names:\n        im_array = imread(train_path+sub_folder+\"/\"+file_name)\n        size = \"_\".join(map(str,list(im_array.shape)))\n        different_file_sizes[size] = different_file_sizes.get(size,0) + 1\n\nplt.figure(figsize=(12,4))\nsns.barplot(list(different_file_sizes.values()), list(different_file_sizes.keys()), alpha=0.8)\nplt.ylabel('Image size', fontsize=12)\nplt.xlabel('Number of Images in train', fontsize=12)\nplt.title(\"Image sizes present in train dataset\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2d2671ec-75ad-bdba-4c89-0988d8b66551"},"cell_type":"markdown","source":"There are  8 different sizes are available in training data.\n\n3264_2448_3 is the most common image size available,  followed by 4128_3096_3.\n\n3088_4128_3 is the largest size of the available images in train set and 3264_2448_3 is the smallest one.","execution_count":null},{"metadata":{"_cell_guid":"ca73957a-2c92-a6b4-8847-24f3197266f3","trusted":true},"cell_type":"code","source":"test_path = \"../input/test/test/\"\nfile_names = check_output([\"ls\", test_path]).decode(\"utf8\").strip().split('\\n')\ndifferent_file_sizes = {}\nfor file_name in file_names:\n        size = \"_\".join(map(str,list(Image.open(test_path+file_name).size)))\n        different_file_sizes[size] = different_file_sizes.get(size,0) + 1\n\nplt.figure(figsize=(12,4))\nsns.barplot(list(different_file_sizes.keys()), list(different_file_sizes.values()), alpha=0.8)\nplt.xlabel('File size', fontsize=12)\nplt.ylabel('Number of Images in test', fontsize=12)\nplt.xticks(rotation='vertical')\nplt.title(\"Image size present in test dataset\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3cfbcc93-7ad3-97dd-c126-c68561f439f9"},"cell_type":"markdown","source":"Test set also has similar size distribution","execution_count":null},{"metadata":{"_cell_guid":"f63db9c8-b79b-8d53-343c-6dcd00fcc3b3","_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"additional_path = \"../input/additional_Type_1_v2/\"\nsub_folders = check_output([\"ls\", additional_path]).decode(\"utf8\").strip().split('\\n')\ndifferent_file_sizes = {}\n#corrupted_images=['2845.jpg','5892.jpg','5893.jpg']\nfor sub_folder in sub_folders:\n    file_names = check_output([\"ls\", additional_path+sub_folder]).decode(\"utf8\").strip().split('\\n')\n    #if(sub_folder=='Type_2'):\n     #   try:\n      #      file_names.remove('5892.jpg')\n       #     file_names.remove('5893.jpg')\n        #    file_names.remove('2845.jpg')\n        #except ValueError:\n         #   pass\n    for file_name in file_names:\n        im_array = Image.open(additional_path+sub_folder+\"/\"+file_name)\n        size = \"_\".join(map(str,list(im_array.size)))\n        different_file_sizes[size] = different_file_sizes.get(size,0) + 1\n\nplt.figure(figsize=(12,4))\nsns.barplot(list(different_file_sizes.values()), list(different_file_sizes.keys()), alpha=0.8)\nplt.ylabel('Image size', fontsize=12)\nplt.xlabel('Number of Images in additional', fontsize=12)\nplt.title(\"Image sizes present in additional dataset\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a1c496e3-d5ae-54a7-f3a0-facb96fdba7f"},"cell_type":"markdown","source":" few images in additional dataset are corrupt as mentioned in [this][1] kernel.\nWe have removed them to get statistics.\n\n\n\n  [1]: https://www.kaggle.com/aamaia/intel-mobileodt-cervical-cancer-screening/three-empty-images-in-additional-7z","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}