{"cells":[{"metadata":{"_uuid":"a6717103051c3f8b781161d917a4fe7352be60e4"},"cell_type":"markdown","source":"# Introduction\nThis kernel is a based on [this one](https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93) (and the referenced ones).\n\nIt will look in detail at differences between the training and test data pixel value distributions and, in particular, very bright images.\n\n# Highlights\n* The number of pixels with value 255 seems to matter\n* Training and test data have different numbers of very bright images\n* Mean image brightness is not as important"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Load necessary modules\nimport os, cv2\nfrom scipy import stats\nfrom glob import glob \nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm_notebook,trange\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"077d54fd1297b6707ea5f202ec706085599225c6"},"cell_type":"markdown","source":"We will load 50k randomly selected images from the training and test set and compare them."},{"metadata":{"trusted":true,"_uuid":"20444c034b79e9dfca48a4a986069e3880f1a0e3","_kg_hide-input":true},"cell_type":"code","source":"def load_data(N,df):\n    \"\"\" This functions loads N images using the data df\n    \"\"\"\n    # allocate a numpy array for the images (N, 96x96px, 3 channels, values 0 - 255)\n    X = np.zeros([N,96,96,3],dtype=np.uint8) \n    #if we have labels for this data, also get them\n    if 'label' in df.columns:\n        y = np.squeeze(df.as_matrix(columns=['label']))[0:N]\n    else:\n        y = None\n    #read images one by one, tdqm notebook displays a progress bar\n    for i, row in tqdm_notebook(df.iterrows(), total=N):\n        if i == N:\n            break\n        X[i] = cv2.imread(row['path'])\n          \n    return X,y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d72f6d8131277d5a1452d92677bcaaae8117165b","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#set paths to training and test data\npath = \"../input/\" #adapt this path, when running locally\ntrain_path = path + 'train/'\ntest_path = path + 'test/'\n\ndf = pd.DataFrame({'path': glob(os.path.join(train_path,'*.tif'))}) # load the filenames\ndf_test = pd.DataFrame({'path': glob(os.path.join(test_path,'*.tif'))}) # load the test set filenames\ndf['id'] = df.path.map(lambda x: x.split('/')[3].split(\".\")[0]) # keep only the file names in 'id'\nlabels = pd.read_csv(path+\"train_labels.csv\") # read the provided labels\ndf = df.merge(labels, on = \"id\") # merge labels and filepaths\ndf.head(3) # print the first three entrys","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2207ea4a34a42c5173b48e4ba90403fe75d6685","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#shuffle the dataframes to a representative sample\ndf = df.sample(frac=1,random_state = 42).reset_index(drop=True)\ndf_test = df_test.sample(frac=1, random_state = 4242).reset_index(drop=True)\n\n# Load N images from the training set\nN = 50000\nprint(\"Loading training data samples...\")\nX,y = load_data(N=N,df=df) \nprint(\"Loading test data samples...\")\nX_test,_ = load_data(N=N,df=df_test) \nprint(\"Done.\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50a761bd5a4be83eaf540f7602ba8e5a713d59f2"},"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA)\n\nAs we have no labels for the test set, this analysis will first focus on the raw image data.\n\nIn detail, we will now look at:\n* Mean image brightness distribution\n* Individual channel brightness distributions\n* Peculiar images such as very bright ones"},{"metadata":{"_uuid":"c38d6b1820db94afb9dabf3b29f407bec119830e"},"cell_type":"markdown","source":"## Mean image brightness distributions\nLet's start with the mean image brightness and its distribution in the training and test data."},{"metadata":{"trusted":true,"_uuid":"fdc0ab0d697f71bc4a138377b7175028e58f2061","_kg_hide-input":true},"cell_type":"code","source":"nr_of_bins = 256\nfig,axs = plt.subplots(1,2,sharey=True, sharex = False, figsize=(8,2),dpi=150)\ntraining_brightness = np.mean(X,axis=(1,2,3))\ntest_brightness = np.mean(X_test,axis=(1,2,3))\naxs[0].hist(training_brightness, bins=nr_of_bins, density=True)\naxs[1].hist(test_brightness, bins=nr_of_bins, density=True)\naxs[0].legend((\"Mean={:2.2f}|Std={:2.2f}\".format(np.mean(training_brightness),np.std(training_brightness)),),loc=2,prop={'size': 6})\naxs[1].legend((\"Mean={:2.2f}|Std={:2.2f}\".format(np.mean(test_brightness),np.std(test_brightness)),),loc=2,prop={'size': 6})\naxs[0].set_title(\"Mean brightness, training samples\")\naxs[1].set_title(\"Mean brightness, test samples\")\naxs[0].set_xlabel(\"Image mean brightness\")\naxs[1].set_xlabel(\"Image mean brightness\")\naxs[0].set_ylabel(\"Relative frequency\")\naxs[1].set_ylabel(\"Relative frequency\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4f2ecadc6e1e13f11464a83f6081e37bb867586"},"cell_type":"markdown","source":"We can also look at a cumulative distribution."},{"metadata":{"trusted":true,"_uuid":"c7a7735e4a2c7a685f9db41a0a7d6899ac1bb002","_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(6,3),dpi=150)\nplt.hist(training_brightness, bins=nr_of_bins, density=True,cumulative=True, alpha = 0.5)\nplt.hist(test_brightness, bins=nr_of_bins, density=True,cumulative=True, alpha = 0.5);\nplt.legend((\"Training\",\"Test\"),loc=2,prop={'size': 6})\n# axs[1].legend((\"Mean={:2.2f}|Std={:2.2f}\".format(np.mean(test_brightness),np.std(test_brightness)),),loc=2,prop={'size': 6})\nplt.title(\"Cumulative mean brightness, training vs test\")\nplt.xlabel(\"Image mean brightness\")\nplt.ylabel(\"Cumulative frequency\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89691c9618a9b9bfd865ab1da9f2ffa4fba8ea5b"},"cell_type":"markdown","source":"Overall, these look pretty similar, which is good as the test and training set seem to stem from, at least, a similar distribution in that regard. There seem to be a bit more images with a mean brightness of 140 - 180 in the test samples and more very bright ones in the training data. \n\nLet's have a more detailed look at the image data. We will now investigate the distribution of pixel values in each channel separately and all combined."},{"metadata":{"_uuid":"9f0907419bd54f6cae17a469cd054a801db3fdb5","trusted":true,"scrolled":false,"_kg_hide-input":true},"cell_type":"code","source":"nr_of_bins = 256 #each possible pixel value will get a bin in the following histograms\nN_hist = 10000 #we will limit the nr of images we look at, because otherwise this tends to kill the kernel...\nfig,axs = plt.subplots(4,2,sharey=True,figsize=(8,8),dpi=150)\n\n#RGB channels\naxs[0,0].hist(X[0:N_hist,:,:,0].flatten(),bins=nr_of_bins,density=True)\naxs[0,1].hist(X_test[0:N_hist,:,:,0].flatten(),bins=nr_of_bins,density=True)\naxs[1,0].hist(X[0:N_hist,:,:,1].flatten(),bins=nr_of_bins,density=True)\naxs[1,1].hist(X_test[0:N_hist,:,:,1].flatten(),bins=nr_of_bins,density=True)\naxs[2,0].hist(X[0:N_hist,:,:,2].flatten(),bins=nr_of_bins,density=True)\naxs[2,1].hist(X_test[0:N_hist,:,:,2].flatten(),bins=nr_of_bins,density=True)\n\n#All channels\naxs[3,0].hist(X[0:N_hist].flatten(),bins=nr_of_bins,density=True)\naxs[3,1].hist(X_test[0:N_hist].flatten(),bins=nr_of_bins,density=True)\n\n#Set image labels\naxs[0,0].set_title(\"Training samples (N =\" + str(X.shape[0]) + \")\");\naxs[0,1].set_title(\"Test samples (N =\" + str(X_test.shape[0]) + \")\");\n\naxs[0,1].set_ylabel(\"Red\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[1,1].set_ylabel(\"Green\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[2,1].set_ylabel(\"Blue\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[3,1].set_ylabel(\"RGB\",rotation='horizontal',labelpad=35,fontsize=12)\nfor i in range(4):\n    axs[i,0].set_ylabel(\"Relative frequency\")\naxs[3,0].set_xlabel(\"Pixel value\")\naxs[3,1].set_xlabel(\"Pixel value\")\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64979c39dddf575f01912a9b733f0c39fbdae155"},"cell_type":"markdown","source":"These also look relatively similar. There is a difference in the green channels for high pixel values and, note, how both, the training and test set, feature a high number of pixels with values of 255. These are likely artifacts. \n\nLet's have a look at images, that have an unusual number of pixels with values of 255 in particular. They are likely outliers. For the training set, we can also inspect their labels to get more insight.\n\nJust a brief reminder: In the [last kernel](https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93) we found that about 60% of labels are negative overall.\n\nWe will now look at some basic statistics and plot some examples.\n\nFirst off, let's check how many image have more than a specific number of pixels of value 255 in them and, for the training set, how many of those have positive labels."},{"metadata":{"trusted":true,"_uuid":"75ee12a9a3d93f953e7b112e9ea4fa63fbd056ad","_kg_hide-input":true},"cell_type":"code","source":"#first count pixels with value 255 in train and test data per image\nbright_pixels_train = (X == 255).sum(axis=(1,2,3))\nbright_pixels_test = (X_test == 255).sum(axis=(1,2,3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1c2508c9aa40083202a60db044c301655e85048b","_kg_hide-input":true},"cell_type":"code","source":"N_bright_train,N_bright_test,N_bright_positive_labels = [],[],[]\nxtics = range(0,5000,100)\nfor threshold in xtics:\n    #count images with more than threshold 255 pixels\n    N_bright_train.append((bright_pixels_train > threshold).sum() / N) \n    N_bright_test.append((bright_pixels_test > threshold).sum() / N) \n    #count positive samples\n    N_bright_positive_labels.append(y[bright_pixels_train > threshold].sum() / (bright_pixels_train > threshold).sum())\n    \nfig = plt.figure(figsize=(6,3),dpi=150)\nplt.plot(xtics,N_bright_train)\nplt.plot(xtics,N_bright_test)\nplt.plot(xtics,N_bright_positive_labels)\nplt.legend((\"Training images over threshold\",\"Test images over threshold\",\"Positive samples portion (Training images)\"),loc=1,prop={'size': 6})\nplt.title(\"Frequency of bright pixels in training and test with positive rate for training\")\nplt.xlabel(\"Pixel Number Threshold (how many pixels have to be 255)\")\nplt.ylabel(\"Relative frequency\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56313899af34deac47b12b88f3fdf7343e503b3e"},"cell_type":"markdown","source":"So, this is getting quite interesting. Let *B* be the number of pixels with value 255. Then, we can see:\n* There are more images in the training data than in the test data for ~300 < *B* < ~2000 relatively speaking\n* Unlike the rest of the training data, which has ~40% positive samples, images with *B* > 500 have about 32% positive rate only (in the training data)\n* Images with *B* > 1700 are increasingly likely to be labeled negative (in the training data)\n\nLet's have a look at example images with *B* close to 1500 (1475 < *B* < 1500) to see, if they stand out in any way."},{"metadata":{"trusted":true,"_uuid":"edbb4de2bb6b473f6ce28cae7626b0bd0217243d","_kg_hide-input":true},"cell_type":"code","source":"#let's take those images where between 1475 and 1525 pixels have values of 255\nbright_train_imgs = X[np.logical_and(bright_pixels_train > 1475,bright_pixels_train < 1525)] \nbright_test_imgs = X_test[np.logical_and(bright_pixels_test > 1475,bright_pixels_test < 1525)]\n\n#train\nfig = plt.figure(figsize=(8, 5), dpi=100)\nnp.random.seed(100) #we can use the seed to get a different set of random images\nfig.suptitle(\"Images with 1475 < B < 1500 \\n Training samples (N =\" + str(bright_train_imgs.shape[0]) + \")\")\nfor plotNr,idx in enumerate(np.random.randint(0,bright_train_imgs.shape[0],8)):\n    ax = fig.add_subplot(2, 4, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(bright_train_imgs[idx]) #plot image\n    ax.set_title('Label: ' + str(y[idx])) #show the label corresponding to the image\n\n#test\nfig = plt.figure(figsize=(8, 4), dpi=100)\nfig.suptitle(\"Test samples (N =\" + str(bright_test_imgs.shape[0]) + \")\")\nfor plotNr,idx in enumerate(np.random.randint(0,bright_test_imgs.shape[0],8)):\n    ax = fig.add_subplot(2, 4, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(bright_test_imgs[idx]) #plot image\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cea2f77c51b2bd4ef796d4784e45367224a78ae1"},"cell_type":"markdown","source":"So, these look relatively normal to me (no expert though), but some seem to contain areas, where no tissue was (the white blobs afaik). Sometimes these are in the center, but the label was positive, that might be a mislabel. In general, these images should not break things, but one might want to clean the data. Let's look at cases with even brighter images (2400 < *B* < 2600) !"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"61d51f226f89ebafa363c3b3dc86721b74bc25cf","_kg_hide-input":true},"cell_type":"code","source":"#let's take those images where between 2400 and 2600 pixels have values of 255\nbright_train_imgs = X[np.logical_and(bright_pixels_train > 2400,bright_pixels_train < 2600)] \nbright_test_imgs = X_test[np.logical_and(bright_pixels_test > 2400,bright_pixels_test < 2600)]\n\n#train\nfig = plt.figure(figsize=(8, 5), dpi=100)\nnp.random.seed(42) #we can use the seed to get a different set of random images\nfig.suptitle(\"Images with 2400 < B < 2600 \\n Training samples (N =\" + str(bright_train_imgs.shape[0]) + \")\")\nfor plotNr,idx in enumerate(np.random.randint(0,bright_train_imgs.shape[0],8)):\n    ax = fig.add_subplot(2, 4, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(bright_train_imgs[idx]) #plot image\n    ax.set_title('Label: ' + str(y[idx])) #show the label corresponding to the image\n    \n#test\nfig = plt.figure(figsize=(8, 4), dpi=100)\nfig.suptitle(\"Test samples (N =\" + str(bright_test_imgs.shape[0]) + \")\")\nfor plotNr,idx in enumerate(np.random.randint(0,bright_test_imgs.shape[0],8)):\n    ax = fig.add_subplot(2, 4, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(bright_test_imgs[idx]) #plot image\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1841c9866e97af3dc9cc2bfd32770c16562c8ce5"},"cell_type":"markdown","source":"This is where I am mostly at a loss. These are much more likely to be negative samples, however to me they look relatively similar to the ones with *B* =~ 1500. Any experts weighing in on this would be very appreciated. \n\nWe'll return to numbers to avoid being biased by these visual samples. Let's inspect how many pixels are actually 255 and if there are any broken images."},{"metadata":{"trusted":true,"_uuid":"03f03bcd5da3d6e2657f4fb2e8bb57fdbc3bd866","_kg_hide-input":true},"cell_type":"code","source":"#calculate relative and absolute incidence of 255 pixels\nnr_train = (X == 255).sum() / N\nnr_test = (X_test == 255).sum() / N\nfreq_train = nr_train*100 / (96*96*3)\nfreq_test = nr_test*100 / (96*96*3)\n\nprint(\"Nr of pixels with value 255 per image \\n\\\nTraining data - {:.4f}% ; avg. Nr = {:.0f}; maxval = {} \\n\\\nTest - {:.4f}% ; avg. Nr = {:.0f}; maxval = {}\"\\\n.format(freq_train,nr_train,np.max(bright_pixels_train),freq_test,nr_test,np.max(bright_pixels_test)))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5baf5f4e41554d145584ee58e4a09456c74990c9"},"cell_type":"markdown","source":"So, there is a higher average number of very bright pixels in the training than in the test data.\n\nWe will now look at images that have a mean image brightness instead, not ones with many bright pixels. We will start with images that have mean brightness of more than 220."},{"metadata":{"_uuid":"296e1bf788a2ee4f48358f42cfc22a2117870e00","trusted":true,"scrolled":false,"_kg_hide-input":true},"cell_type":"code","source":"#let's take those images with high mean values (> 220)\nbright_train_imgs = X[np.mean(X,axis=(1,2,3)) > 220] \nbright_test_imgs = X_test[np.mean(X_test,axis=(1,2,3)) > 220]\n\n#train\nfig = plt.figure(figsize=(8, 5), dpi=100)\nnp.random.seed(100) #we can use the seed to get a different set of random images\nfig.suptitle(\"Images with mean brightness over 220 \\n Training samples (N =\" + str(bright_train_imgs.shape[0]) + \")\")\nfor plotNr,idx in enumerate(np.random.randint(0,bright_train_imgs.shape[0],8)):\n    ax = fig.add_subplot(2, 4, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(bright_train_imgs[idx]) #plot image\n    ax.set_title('Label: ' + str(y[idx])) #show the label corresponding to the image\n    \n#test\nfig = plt.figure(figsize=(8, 4), dpi=100)\nfig.suptitle(\"Test samples (N =\" + str(bright_test_imgs.shape[0]) + \")\")\nfor plotNr,idx in enumerate(np.random.randint(0,bright_test_imgs.shape[0],8)):\n    ax = fig.add_subplot(2, 4, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(bright_test_imgs[idx]) #plot image\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80bec56a176bd243771b90faf09b1664fcd26cec"},"cell_type":"markdown","source":"So, these look much more broken to my (non-expert) eye. In some cases the center region seems to be completely white and the first one of the displayed test images is defintely broken. Let's look at the labels for these images and how frequent these images are in the test and training set."},{"metadata":{"trusted":true,"_uuid":"aefe74ce225fd66b7df936b3617e311538988bf3","_kg_hide-input":true},"cell_type":"code","source":"training_brightness = np.mean(X,axis=(1,2,3))\ntest_brightness = np.mean(X_test,axis=(1,2,3))\n\nN_bright_train,N_bright_test,N_bright_positive_labels = [],[],[]\nxtics = range(100,255,1)\nfor threshold in xtics:\n    #count images with more than threshold 255 pixels\n    N_bright_train.append((training_brightness > threshold).sum() / N) \n    N_bright_test.append((test_brightness > threshold).sum() / N) \n    #count positive samples\n    N_bright_positive_labels.append(y[bright_pixels_train > threshold].sum() / (bright_pixels_train > threshold).sum())\n    \nfig = plt.figure(figsize=(6,3),dpi=150)\nplt.plot(xtics,N_bright_train)\nplt.plot(xtics,N_bright_test)\nplt.plot(xtics,N_bright_positive_labels)\nplt.legend((\"Training images over threshold\",\"Test images over threshold\",\"Positive samples portion (Training images)\"),loc=1,prop={'size': 6})\nplt.title(\"Frequency of bright pixels in training and test with positive rate for training\")\nplt.xlabel(\"Pixel Number Threshold (how many pixels have to be 255)\")\nplt.ylabel(\"Relative frequency\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0eca6836f2748fb12a246e89617eaff31a326f99"},"cell_type":"markdown","source":"So, once again, the distribution is slightly different. There is a higher portion of images with a high mean brightness in the training data it seems. Interestingly, however, unlike for the images with a high *B*, i.e. with many very bright pixels, images with a high mean brightness don't seem to have different distribution of positive samples."},{"metadata":{"_uuid":"4545aea74d5ed25a3c667d22a64f61be3dce34d9"},"cell_type":"markdown","source":"# Conclusions\nSo, for *B* being the number of pixels with value 255 in an image\n\n* Images with high *B* have a different positive/false distribution than the rest\n* But mean image brightness does not seem to have a strong relationship with the positve/false distribution\n* Images with high *B* and higher mean brightness are more often encountered in the training than in the test set\n\nOverall, it seems there are quite some images with questionable quality in both datasets with very high mean image brightness\nCleaning the data should help and a special treatment of the differently distributed images with high *B* might be warranted."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}