{"cells":[{"metadata":{},"cell_type":"markdown","source":"UPDATE on V3:  \nAdded 2. Display an image for each shape"},{"metadata":{},"cell_type":"markdown","source":"This kernel uses the following kernel code:\n* https://www.kaggle.com/h4211819/image-size-eda\n* https://www.kaggle.com/yangsaewon/basic-eda-train-test-image-distribution-check\n* https://www.kaggle.com/kaerunantoka/extract-image-features\n\nI think that this competition needs to be careful because the target is biased due to the shape of the image.  \nAlso, I think that the information of the image shape of the previous competition will be helpful, but because it can not be read from the kernel, I released the [dataset](https://www.kaggle.com/currypurin/diabetic-retinopathy-detection-image-size) and [discussion](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/99846)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom pathlib import Path\nimport sys\nimport os\nimport pickle\nfrom tqdm import tqdm_notebook as tqdm\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport itertools","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"print(os.listdir('../input'))\nprint(os.listdir('../input/diabetic-retinopathy-detection-image-size'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. present competition"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ntest = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train), len(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#the func is from https://www.kaggle.com/toshik/image-size-and-rate-of-new-whale\ndef get_size_list(targets, dir_target):\n    result = list()\n    for target in tqdm(targets):\n        img = np.array(Image.open(os.path.join(dir_target, target+'.png')))\n        result.append(img.shape)\n    return result\n\n# the func is from https://www.kaggle.com/kaerunantoka/extract-image-features\ndef get_size(file_name_list, dir_target):\n    result = list()\n    #filename = images_path + filename\n    for file_name in tqdm(file_name_list):\n        st = os.stat(f'{dir_target}/{file_name}.png')\n        result.append(st.st_size)\n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train['image_shape'] = get_size_list(train.id_code.tolist(),\n                                     dir_target='../input/aptos2019-blindness-detection/train_images')\ntest['image_shape'] = get_size_list(test.id_code.tolist(),\n                                    dir_target='../input/aptos2019-blindness-detection/test_images')\ntrain['image_size'] = get_size(train.id_code.tolist(),\n                               dir_target='../input/aptos2019-blindness-detection/train_images')\ntest['image_size'] = get_size(test.id_code.tolist(),\n                              dir_target='../input/aptos2019-blindness-detection/test_images')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for df in [train, test]:\n    df['height'] = df['image_shape'].apply(lambda x:x[0])\n    df['width'] = df['image_shape'].apply(lambda x:x[1])\n    df['width_height_ratio'] = df['height'] / df['width']\n    df['width_height_added'] = df['height'] + df['width']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.describe()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(16,10))\nplt.subplot(241)\nplt.hist(train['width'])\nplt.title(\"train width\")\nplt.xlim(200, 4500)\n\nplt.subplot(242)\nplt.hist(test['width'])\nplt.title(\"test width\")\nplt.xlim(200, 4500)\n\nplt.subplot(243)\nplt.hist(train['height'])\nplt.title(\"train height\")\nplt.xlim(200, 3100)\n\nplt.subplot(244)\nplt.hist(test['height'])\nplt.title(\"test height\")\nplt.xlim(200, 3100)\n\nplt.subplot(245)\nplt.hist(train['width_height_ratio'])\nplt.title(\"train width height ratio\")\nplt.xlim(0.6, 1.05)\n\n\nplt.subplot(246)\nplt.hist(test['width_height_ratio'])\nplt.title(\"test width height ratio\")\nplt.xlim(0.6, 1.05)\n\nplt.subplot(247)\nplt.hist(train['width_height_added'])\nplt.title(\"train width height added\")\n\nplt.subplot(248)\nplt.hist(test['width_height_added'])\nplt.title(\"train width height added\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.heatmap(train.corr(), cmap=plt.cm.Blues, annot=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Many feature seems to be correlated with the target."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train_meta = train.groupby(['width', 'height', 'diagnosis']).agg({'diagnosis':'count'}).unstack('diagnosis').fillna(0)\ntrain_meta.columns = [f'{i[0]}_{i[1]}' for i in train_meta.columns]\ntrain_meta['train_count'] = train_meta.sum(axis=1)\n\ntest_meta = test.groupby(['width', 'height']).agg({'id_code':'count'}).rename(columns={'id_code':'pub_test_count'})\ncount_ratio = train_meta.join(test_meta, how='outer')\n\nfor i in range(5):\n    count_ratio.loc[:, f'{i}_ratio'] = count_ratio.iloc[:, i] / count_ratio['train_count']\n\ncount_ratio = count_ratio.fillna(0)\n\ncount_ratio = count_ratio.astype({'diagnosis_0': int, 'diagnosis_1': int, 'diagnosis_2': int,\n                                  'diagnosis_3': int, 'diagnosis_4': int})\ncount_ratio = count_ratio.astype({'train_count': int, 'pub_test_count': int})\n\ncount_ratio.reset_index(inplace=True)\ncount_ratio.set_index(['width', 'height', 'train_count', 'pub_test_count'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_ratio","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Training data has different target distribution for each image shape.\n  * For example 1050x1050 is high ratio of class_0, 2136x3216 is high ratio of class_2\n* If we leave the image shape information in the preprocessed image, there is a possibility of overfitting, so be very careful.\n* Let's look at the images."},{"metadata":{},"cell_type":"markdown","source":"# 2.Display an image for each shape"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def im_show(height, width, num):\n    tmp = train[(train['width'] == width) & (train['height'] == height)].id_code\n    dir_target = '../input/aptos2019-blindness-detection/train_images'\n    id = tmp.values[num]\n    img = Image.open(os.path.join(dir_target, id +'.png'))\n    plt.imshow(img.resize((256, 256)))\n    plt.tick_params(bottom=False,\n                    left=False,\n                    right=False,\n                    top=False,\n                    labelbottom=False,\n                    labelleft=False,\n                    labelright=False,\n                    labeltop=False)\n    value = train.loc[train['id_code'] == id, :].values[0]\n    plt.title(f'({value[4]},{value[5]})->(256,256)\\n {id}, diagnosis:{value[1]}')\n\ndef five_img_plot(height, width):\n    print('-' * 10)\n    print(f'shape({height}, {width})')\n    plt.figure(figsize=(16, 4))\n    for i in range(5):\n        plt.subplot(1,5,i+1)\n        im_show(height, width, i)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"five_img_plot(480, 640)\nfive_img_plot(614, 819)\nfive_img_plot(1050, 1050)\nfive_img_plot(1536, 2048)\nfive_img_plot(1736, 2416)\nfive_img_plot(1958, 2588)\nfive_img_plot(2588, 3388)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The proportion of black area and the tendency of brightness are also likely to be in each image shape.\n* I would like to update this point later.\n* ref : https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/99846#575147"},{"metadata":{},"cell_type":"markdown","source":"# 3.previous competition"},{"metadata":{},"cell_type":"markdown","source":"There are about 90,000 images of [the previous competition](https://www.kaggle.com/c/diabetic-retinopathy-detection) and Kernel can not read all the images. So I run the same code as above, and [dataset](https://www.kaggle.com/currypurin/diabetic-retinopathy-detection-image-size) made public."},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_train = pd.read_csv('../input/diabetic-retinopathy-detection-image-size/pre_train_shape.csv')\npre_test = pd.read_csv('../input/diabetic-retinopathy-detection-image-size/pre_test_shape.csv')\n\nfor df in [pre_train, pre_test]:\n    df['width_height_ratio'] = df['height'] / df['width']\n    df['width_height_added'] = df['height'] + df['width']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(pre_train), len(pre_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The number of images of the last competition is very large."},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(16,10))\nplt.subplot(241)\nplt.hist(pre_train['width'])\nplt.title(\"pre train width\")\nplt.xlim(200, 5500)\n\nplt.subplot(242)\nplt.hist(pre_test['width'])\nplt.title(\"pre test width\")\nplt.xlim(200, 5500)\n\nplt.subplot(243)\nplt.hist(pre_train['height'])\nplt.title(\"pre train height\")\nplt.xlim(200, 4000)\n\nplt.subplot(244)\nplt.hist(pre_test['height'])\nplt.title(\"pre test height\")\nplt.xlim(200, 4000)\n\nplt.subplot(245)\nplt.hist(pre_train['width_height_ratio'])\nplt.title(\"pre train width height ratio\")\nplt.xlim(0.6, 1.05)\n\n\nplt.subplot(246)\nplt.hist(pre_test['width_height_ratio'])\nplt.title(\"pre test width height ratio\")\nplt.xlim(0.6, 1.05)\n\nplt.subplot(247)\nplt.hist(pre_train['width_height_added'])\nplt.title(\"pre train width height added\")\n\nplt.subplot(248)\nplt.hist(pre_test['width_height_added'])\nplt.title(\"pre train width height added\");","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"pre_train.drop('channel', axis=1, inplace=True)\npre_test.drop('channel', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams[\"font.size\"] = 14\n# pre_train.rename(columns={'level': 'diagnosis'}, inplace=True)\nplt.figure(figsize=(16,8))\nplt.subplot(121)\nsns.heatmap(pre_train.corr(), cmap=plt.cm.Blues, annot=True)\nplt.title('previous_competition')\n\nplt.subplot(122)\nsns.heatmap(train.corr(), cmap=plt.cm.Blues, annot=True)\nplt.title('this_competition')\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_train_meta = pre_train.groupby(['width', 'height', 'level']).agg({'level':'count'}).unstack('level').fillna(0)\npre_train_meta.columns = [f'{i[0]}_{i[1]}' for i in pre_train_meta.columns]\npre_train_meta['train_count'] = pre_train_meta.sum(axis=1)\n\npre_test_meta = pre_test.groupby(['width', 'height']).agg({'image':'count'}).rename(columns={'image':'pub_test_count'})\npre_count_ratio = pre_train_meta.join(pre_test_meta, how='outer')\n\nfor i in range(5):\n    pre_count_ratio.loc[:, f'{i}_ratio'] = pre_count_ratio.iloc[:, i] / pre_count_ratio['train_count']\n\npre_count_ratio = pre_count_ratio.fillna(0)\n\npre_count_ratio = pre_count_ratio.astype({'level_0': int, 'level_1': int, 'level_2': int, 'level_3': int, 'level_4': int})\npre_count_ratio = pre_count_ratio.astype({'train_count': int, 'pub_test_count': int})\n\npre_count_ratio.reset_index(inplace=True)\npre_count_ratio.set_index(['width', 'height', 'train_count', 'pub_test_count'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_count_ratio","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4.Number of image and ratio by image shape in previous and present competition"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams[\"font.size\"] = 13\nplt.figure(figsize=(12, 8))\nsns.heatmap(pre_count_ratio.iloc[:, 5:], cmap=plt.cm.Blues)\nplt.xlabel('target')\nplt.ylabel('width - height - number_of_train - number_of_public_test')\nplt.title('Number of image and ratio by image shape in previous competition')\nplt.xticks([0.5, 1.5, 2.5, 3.5, 4.5], ['0',  '1', '2', '3', '4'])\nfor i, j in itertools.product(range(5), range(len(pre_count_ratio))):\n    train_count = pre_count_ratio.index[j][2]\n    if train_count != 0:\n        ratio = np.int(np.round(pre_count_ratio.iloc[j, i+5] * 100))\n        count = pre_count_ratio.iloc[j, i]\n        plt.text(i+0.2, j+0.8, f'{count:>4}', color='k'if ratio < 65 else \"w\")\n        plt.text(i+0.5, j+0.8, f'{ratio:>3}%', color='k'if ratio < 65 else \"w\")\n    elif train_count == 0:\n        plt.text(i+0.5, j+0.8, '-', color='k')\nplt.show()\n\nplt.figure(figsize=(12, 8))\nsns.heatmap(count_ratio.iloc[:, 5:], cmap=plt.cm.Blues)\nplt.xlabel('target')\nplt.ylabel('width - height - number_of_train - number_of_public_test')\nplt.title('Number of image and ratio by image shape in present competition')\nfor i, j in itertools.product(range(5), range(len(count_ratio))):\n    train_count = count_ratio.index[j][2]\n    if train_count != 0:\n        ratio = np.int(np.round(count_ratio.iloc[j, i+5] * 100))\n        count = count_ratio.iloc[j, i]\n        plt.text(i+0.2, j+0.8, f'{count:>4}', color='k'if ratio < 65 else \"w\")\n        plt.text(i+0.5, j+0.8, f'{ratio:>3}%', color='k'if ratio < 65 else \"w\")\n    elif train_count == 0:\n        plt.text(i+0.5, j+0.8, '-', color='k')\n    plt.xticks([0.5, 1.5, 2.5, 3.5, 4.5], ['0',  '1', '2', '3', '4'])\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The bias of the class by the shape of the train image was small in the last competition, but this time the competition is not.\n* In the previous competition, the ranking change on the public and private leaderboard is not large.\n* I think that this bias may cause a large shakeup."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}