{"cells":[{"metadata":{"_uuid":"d9fc623d9468d12790146b51d626c22431b826a2"},"cell_type":"markdown","source":"# Exploration of the data\n\nFirst load some libraries and show what data we have."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport seaborn as sns\nfrom PIL import Image\nimport os\n# ignore warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed85c4fe8a3cad80c60ac5aac6406686dd3dc4ea"},"cell_type":"markdown","source":"Ok, there is a *train* and a *test* folder. let's examine the sizes."},{"metadata":{"trusted":true,"_uuid":"73b564f41d7cf62871eb1c461b9a7ee1e2829d9d"},"cell_type":"code","source":"train_len = len(os.listdir(\"../input/train\"))\ntest_len = len(os.listdir(\"../input/test\"))\nprint(\"Train size is {}, test size is {}\".format(train_len, test_len))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train_labels.csv\")\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"62ee5f5b61f8c5646a2ecf4fde0dca520e374cbe"},"cell_type":"code","source":"negative_cases_train = train_df[train_df[\"label\"] == 0]\npostive_cases_train = train_df[train_df[\"label\"] == 1]\nprint(\"Postive cases {:,}, negative cases {:,} in training set\".format(len(postive_cases_train), len(negative_cases_train)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc9bfe9b7fbf1665c1d361a052fa8bfd43d48843"},"cell_type":"markdown","source":"The label is positive it at least one pixel in the center crop of 32 x 32 pixel is positive. Let's visualize some cases."},{"metadata":{"trusted":true,"_uuid":"fd142daced236035ef5cdbb4209d055ef24493ae"},"cell_type":"code","source":"# now lets write a helper function to show some images\ndef show(df):\n    fig, ax = plt.subplots(2,5, figsize=(20,5))\n    for i, row in enumerate(df.itertuples()):\n        path = os.path.join('../input/train/', row.id)\n        img = Image.open(path+'.tif')\n        w,h = img.size\n        cropped = img.crop((w//2 - 32//2, h//2 - 32//2, w//2 + 32//2, h//2 + 32//2))\n        box = patches.Rectangle((32,32),32,32,linewidth=2,edgecolor='r', facecolor='none')\n        ax[0,i].imshow(img)\n        ax[0,i].add_patch(box)\n        ax[0,i].set_title(\"Label: {}\".format(row.label))\n        ax[1,i].imshow(cropped)\n        ax[0,0].set_ylabel('Sample', size='large')\n        ax[1,0].set_ylabel('Cropped', size='large')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8acac321d295d3e2b05d30802991191b8f4b0d04"},"cell_type":"code","source":"show(negative_cases_train[0:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d81b7fe84a528b06014e628c993850ef90e3bff8"},"cell_type":"code","source":"show(postive_cases_train[0:5])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cec08de7cfcd6a0711d09a7eba965420dc104a9a"},"cell_type":"markdown","source":"It is very hard for the untrained eye to see some pattern. Check some statistics. To load all the images into a dataframe crashes the kernel, that's why only a subset of the data is loaded."},{"metadata":{"trusted":true,"_uuid":"776c13d183d98e4021f6b3a363fc1d7512284ea0"},"cell_type":"code","source":"def load(row):\n    path = os.path.join('../input/train/', row.id)\n    img = Image.open(path+'.tif')\n    a = np.array(img)\n    row['image'] = a\n    row['image_flattened'] = a.flatten()\n    return row","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3fffbf8ed12467b39d9c84b6b871a66a33e81c8a"},"cell_type":"code","source":"def distribution_plot(sample_size=100):\n    neg_sample = train_df.loc[train_df.label == 0][0:sample_size]\n    pos_sample = train_df.loc[train_df.label == 1][0:sample_size]\n    neg_sample = neg_sample.apply(load, axis=1)\n    pos_sample = pos_sample.apply(load, axis=1)\n    print(\"Positive samples size {}\".format(len(pos_sample)))\n    print(\"Negative samples size {}\".format(len(neg_sample)))\n    neg = neg_sample.image_flattened.values.tolist()\n    pos = pos_sample.image_flattened.values.tolist()\n    plt.figure(\"Distribution\")\n    sns.distplot(np.concatenate(neg), label='Negative')\n    sns.distplot(np.concatenate(pos), label='Positive')\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0da994329a15081d62a623779b491b5cb92940cd"},"cell_type":"code","source":"distribution_plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fba6312b054cd4b2fc730c75cbc6c578874ba445"},"cell_type":"code","source":"distribution_plot(1000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0509c87ff484cd3a356cf8aa334c43c03a98a5bf"},"cell_type":"markdown","source":"Most of the values overlap, so no suprise.\n\nAnything else which would be interesting to look at?"},{"metadata":{"trusted":true,"_uuid":"a82d161c34433e762464528aad905f8fe9e765b5"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}