{"cells":[{"metadata":{"_uuid":"a6717103051c3f8b781161d917a4fe7352be60e4"},"cell_type":"markdown","source":"# Acknowledgements\nThis kernel contains excerpts from and was inspired by the following other kernels:\n* https://www.kaggle.com/qitvision/a-complete-ml-pipeline-fast-ai\n* https://www.kaggle.com/CVxTz/cnn-starter-nasnet-mobile-0-9709-lb\n* https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n* https://www.kaggle.com/artgor/simple-eda-and-model-in-pytorch\n\n# Introduction\nThis kernel is an exemplary presentation of a simple exploratory data analysis (EDA) and a tutorial on creating your first model for this challenge. It is targeted at beginners.\n\nRunning it you will be able to \n* take a look at the data for the challenge and its features\n* train a basic convolutional neural network on all of the data (many of the other kernels only train on parts of the data)\n* create your first submission ( LB ~0.93)\n\n# Contents\n1. Useful python modules\n2. The challenge and the data\n3. Loading the data\n4. EDA\n5. Creating a simple keras model\n6. Training and validating the model\n7. Creating a submission"},{"metadata":{"_uuid":"143ed018a1cfecf3438504b2b0d1a63beb053057"},"cell_type":"markdown","source":"# 1. Modules\nWe utilize a variety of modules in this kernel. Most of these provide way more functionality than we need, but they are all handy:\n* [glob](https://docs.python.org/3/library/glob.html) for easily finding matching filenames\n* [numpy](https://www.numpy.org/) - because it is _the_ math module for most things\n* [pandas](https://pandas.pydata.org/) - a powerful module for data structures and analysis\n* [keras](https://keras.io/) - a high-level deep learning API, in our case to ease TensorFlow usage\n* [cv2](https://opencv-python-tutro****als.readthedocs.io/en/latest/py_tutorials/py_setup/py_intro/py_intro.html#intro) - for image processing (we'll just use it for loading images)\n* [tqdm](https://tqdm.github.io/) - a minimalistic yet powerful and easy to use progress bar\n* [matplotlib](https://matplotlib.org/) - a plotting module"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"#Load the modules\nfrom glob import glob \nimport numpy as np\nimport pandas as pd\nimport keras,cv2,os\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, BatchNormalization, Activation\nfrom keras.layers import Conv2D, MaxPool2D\n\nfrom tqdm import tqdm_notebook,trange\nimport matplotlib.pyplot as plt\n\nimport gc #garbage collection, we need to save all the RAM we can","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc88d74857744a4a559f7457caa6b3d8dd8fe958"},"cell_type":"markdown","source":"# 2. The challenge and the data\nTechnically, the goal of this challenge is a binary classification task for images, meaning we want to divide images into two classes. Practically, microscopic images of lymph node tissue with a resolution of 96x96 pixels are provided and we should provide a probability indicating if the images show metastatic cancer tissue in the 32x32 center region of the image.\n\nAbout 220,000 labeled images are provided for training and about 57,000 make up the test set.\n\nLet's start by loading the data and having a look at some of the images."},{"metadata":{"_uuid":"077d54fd1297b6707ea5f202ec706085599225c6"},"cell_type":"markdown","source":"# 3. Loading the data\nWe'll start by creating a pandas data frame containing the path of all the files in the `train_path` folder and then read the matching labels from the provided csv file.\n\n## Load the labels and filenames"},{"metadata":{"_uuid":"d72f6d8131277d5a1452d92677bcaaae8117165b","trusted":true},"cell_type":"code","source":"#set paths to training and test data\npath = \"../input/\" #adapt this path, when running locally\ntrain_path = path + 'train/'\ntest_path = path + 'test/'\n\ndf = pd.DataFrame({'path': glob(os.path.join(train_path,'*.tif'))}) # load the filenames\ndf['id'] = df.path.map(lambda x: x.split('/')[3].split(\".\")[0]) # keep only the file names in 'id'\nlabels = pd.read_csv(path+\"train_labels.csv\") # read the provided labels\ndf = df.merge(labels, on = \"id\") # merge labels and filepaths\ndf.head(3) # print the first three entrys","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e32d37b60a88fb0c038ab2a6938bd46198e79740"},"cell_type":"markdown","source":"## Load the images\n\nNow, we will load some of the images. As [interactive kernels with GPU](https://www.kaggle.com/docs/kernels#the-kernels-environment) currently offer about 14 GB of RAM, we will take care to keep the images in the uint8 format (i.e. pixel values are integers between 0 and 255) to reduce the memory footprint. Processing of the images often requires converting them to float32, which would require additional space.\n\nWe'll declare a function to load a set number of images and then load 10000 images."},{"metadata":{"_uuid":"5087ee421fbaf96cbd18df1467cf14c34b4f7289","trusted":true},"cell_type":"code","source":"def load_data(N,df):\n    \"\"\" This functions loads N images using the data df\n    \"\"\"\n    # allocate a numpy array for the images (N, 96x96px, 3 channels, values 0 - 255)\n    X = np.zeros([N,96,96,3],dtype=np.uint8) \n    #convert the labels to a numpy array too\n    y = np.squeeze(df.as_matrix(columns=['label']))[0:N]\n    #read images one by one, tdqm notebook displays a progress bar\n    for i, row in tqdm_notebook(df.iterrows(), total=N):\n        if i == N:\n            break\n        X[i] = cv2.imread(row['path'])\n          \n    return X,y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2207ea4a34a42c5173b48e4ba90403fe75d6685","trusted":true},"cell_type":"code","source":"# Load 10k images\nN=10000\nX,y = load_data(N=N,df=df) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50a761bd5a4be83eaf540f7602ba8e5a713d59f2"},"cell_type":"markdown","source":"# 4. Exploratory Data Analysis (EDA)\n\nThe purpose of this EDA is to \n* Take a look at the images\n* Understand the distribution of the two classes (no cancer cells / cancer cells)\n* Have a look at some image features (RGB channel distributions, mean brightness)\n\n## Let's plot some example images with their assigned label (0 - no cancer cells, 1 - cancer cells):"},{"metadata":{"_uuid":"ffed68f7b732222097006f1047744950ad085ed8","scrolled":false,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10, 4), dpi=150)\nnp.random.seed(100) #we can use the seed to get a different set of random images\nfor plotNr,idx in enumerate(np.random.randint(0,N,8)):\n    ax = fig.add_subplot(2, 8//2, plotNr+1, xticks=[], yticks=[]) #add subplots\n    plt.imshow(X[idx]) #plot image\n    ax.set_title('Label: ' + str(y[idx])) #show the label corresponding to the image","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7931e9a388a07450ae4a9576dd31ee72aea86ccc"},"cell_type":"markdown","source":"So, for most people (like me) there is no easy way to discern which images contain cancer cells. There is a [variety of things](https://www.nature.com/articles/nmeth.4397.pdf) one can look at to get a deeper insight into the data.\n\n## Let's starting looking at the data distribution\n\nWe'll start by looking at how often the classes are represented. "},{"metadata":{"_uuid":"9c0351062237909773100c6e87409fe270379652","trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(4, 2),dpi=150)\nplt.bar([1,0], [(y==0).sum(), (y==1).sum()]); #plot a bar chart of the label frequency\nplt.xticks([1,0],[\"Negative (N={})\".format((y==0).sum()),\"Positive (N={})\".format((y==1).sum())]);\nplt.ylabel(\"# of samples\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"529ea0bdfc9338af404172c436c9a34bf7ab6b2d"},"cell_type":"markdown","source":"So, we have about a 60 / 40 split of negative to positive samples in the data. This is important because it means that a trivial classifier, that just labels every sample as negative, would achieve an accuracy of 60%. Possible countermeasures to avoid a bias in the classifier and improve stability during training are, e.g., [over- and undersampling](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis).\n\n## Let's start looking at each class individually\n\nWe'll now split the data into positive and negative samples to get an idea what makes the classes unique. Such an analysis can often provide insight into possible [feature engineering](https://en.wikipedia.org/wiki/Feature_engineering) possibilities or which data transformations may be useful."},{"metadata":{"_uuid":"6e4d8e67e64d2cdc6ad4098b4a67f93c87d9ec5d","trusted":true},"cell_type":"code","source":"positive_samples = X[y == 1]\nnegative_samples = X[y == 0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c38d6b1820db94afb9dabf3b29f407bec119830e"},"cell_type":"markdown","source":"Now, we will compare the distribution of pixel values for each color channel (RGB) individually and all channels together"},{"metadata":{"_uuid":"9f0907419bd54f6cae17a469cd054a801db3fdb5","trusted":true,"scrolled":false},"cell_type":"code","source":"nr_of_bins = 256 #each possible pixel value will get a bin in the following histograms\nfig,axs = plt.subplots(4,2,sharey=True,figsize=(8,8),dpi=150)\n\n#RGB channels\naxs[0,0].hist(positive_samples[:,:,:,0].flatten(),bins=nr_of_bins,density=True)\naxs[0,1].hist(negative_samples[:,:,:,0].flatten(),bins=nr_of_bins,density=True)\naxs[1,0].hist(positive_samples[:,:,:,1].flatten(),bins=nr_of_bins,density=True)\naxs[1,1].hist(negative_samples[:,:,:,1].flatten(),bins=nr_of_bins,density=True)\naxs[2,0].hist(positive_samples[:,:,:,2].flatten(),bins=nr_of_bins,density=True)\naxs[2,1].hist(negative_samples[:,:,:,2].flatten(),bins=nr_of_bins,density=True)\n\n#All channels\naxs[3,0].hist(positive_samples.flatten(),bins=nr_of_bins,density=True)\naxs[3,1].hist(negative_samples.flatten(),bins=nr_of_bins,density=True)\n\n#Set image labels\naxs[0,0].set_title(\"Positive samples (N =\" + str(positive_samples.shape[0]) + \")\");\naxs[0,1].set_title(\"Negative samples (N =\" + str(negative_samples.shape[0]) + \")\");\naxs[0,1].set_ylabel(\"Red\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[1,1].set_ylabel(\"Green\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[2,1].set_ylabel(\"Blue\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[3,1].set_ylabel(\"RGB\",rotation='horizontal',labelpad=35,fontsize=12)\nfor i in range(4):\n    axs[i,0].set_ylabel(\"Relative frequency\")\naxs[3,0].set_xlabel(\"Pixel value\")\naxs[3,1].set_xlabel(\"Pixel value\")\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64979c39dddf575f01912a9b733f0c39fbdae155"},"cell_type":"markdown","source":"So we can already spot differences in the distributions of all channels individually and together between positive and negative samples: \n* Negative samples seem to have higher, i.e. brighter, pixel values in general and especially in the green color channel. \n* Interestingly, the positive samples have a darker green channel than red and blue while this is not true for the negative samples. However, very dark pixels are for both sample sets mostly only present in the green channel.\n* Furthermore, note the relatively high frequency of the pixel value 255. Looking at the data above we can see, that these can likely be attributed to the bright white image regions present in some images. They seem to be present in both positive and negative samples similarly frequently.\n\nNow, let's switch perspective and look at the distribution of mean image brightness, i.e. mean image pixel values. Note, previously we were averaging over all pixel values in the positive and negative samples. Now, we will take the mean of each individual image and look at that distribution."},{"metadata":{"_uuid":"296e1bf788a2ee4f48358f42cfc22a2117870e00","trusted":true,"scrolled":false},"cell_type":"code","source":"nr_of_bins = 64 #we use a bit fewer bins to get a smoother image\nfig,axs = plt.subplots(1,2,sharey=True, sharex = True, figsize=(8,2),dpi=150)\naxs[0].hist(np.mean(positive_samples,axis=(1,2,3)),bins=nr_of_bins,density=True);\naxs[1].hist(np.mean(negative_samples,axis=(1,2,3)),bins=nr_of_bins,density=True);\naxs[0].set_title(\"Mean brightness, positive samples\");\naxs[1].set_title(\"Mean brightness, negative samples\");\naxs[0].set_xlabel(\"Image mean brightness\")\naxs[1].set_xlabel(\"Image mean brightness\")\naxs[0].set_ylabel(\"Relative frequency\")\naxs[1].set_ylabel(\"Relative frequency\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3671a84db204421a58fca08010e38a99490b1e27"},"cell_type":"markdown","source":"Once again, we can spot pretty obvious differenes between the positive and negative samples. The distribution of mean brightness for the positive samples looks almost like a normal distribution around a brightness of 150. The negative samples, however, seem to follow some bimodal distribution with peaks around 140 and 225. \n\n**Conclusions:** \n* There are some easily spotted differences in the distributions of pixel values and mean image brightness between positive and negative samples. This is good, because whatever model we will use can likely use this. \n* Some of the images seem to contain very bright regions, which are likely artifacts of the recording process. We might have to find a way to deal with them. They are almost equally distributed between positive and negative samples and, hence, probably not easily usable as a feature.\n* We have about 50% more negative than positive samples. This might require adjustments."},{"metadata":{"_uuid":"c3bd89f6d9e00472c46b1c9c617e3d090c9fccce"},"cell_type":"markdown","source":"# Setup the model\nWe will now focus on creating a simple model for this problem. This is usually the point, where you would want to start considering our previous conclusions, but to keep it simple, we will assume, we did not draw any meaningful conclusions. As the data is - in comparison to other Kaggle challenges - relatively well balanced and accesible, this should be ok. You can use the insight to craft a better model later!\n\nLet's start by loading all the data, not just a subset as before. This will likely require a few minutes. However, we only need to do it once."},{"metadata":{"_uuid":"30e69a2a034e279da6eeb0fc09f1a58b58fc0401","trusted":true},"cell_type":"code","source":"N = df[\"path\"].size # get the number of images in the training data set\nX,y = load_data(N=N,df=df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e5b2386785b29f3a332daffa0160d14aa3a562e"},"cell_type":"markdown","source":"We will use the garbage collector and unbind some variables to free up space in our RAM."},{"metadata":{"trusted":true,"_uuid":"e14b07edae9d62cc873b1b10c5288c72bfbf9671"},"cell_type":"code","source":"#Collect garbage\npositives_samples = None\nnegative_samples = None\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c56525b3ee4c5f65255422ec035c057d350de292"},"cell_type":"markdown","source":"Now, we will split the data into a [training and validation set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets). Due to the RAM limitations, we will just do this in-place by specifying an index at which we will split. We'll use 80% of the data for training and 20% to validate that our model can generalize to new data. After that, to avoid any influence of a possible previous sorting of data we will shuffle the data (in-place)."},{"metadata":{"_uuid":"60644d18d00811a12f6270ac42f8c745902018a8","trusted":true},"cell_type":"code","source":"training_portion = 0.8 # Specify training/validation ratio\nsplit_idx = int(np.round(training_portion * y.shape[0])) #Compute split idx\n\nnp.random.seed(42) #set the seed to ensure reproducibility\n\n#shuffle\nidx = np.arange(y.shape[0])\nnp.random.shuffle(idx)\nX = X[idx]\ny = y[idx]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f6bf258977d100a6781f13e9dfe31009fec0bb3"},"cell_type":"markdown","source":"Let's declare our neural network architecture now. This kernel uses [keras](https://keras.io/), which makes it very easy to setup a neural network and start training it.\n\nThe model architecture is taken from [another kernel](https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb). It is a relatively simple convolutional neural network with three blocks of [convolutional layers, batch normalization, pooling and dropout](https://cs231n.github.io/convolutional-networks/)."},{"metadata":{"_uuid":"9434b86b47f6132843b328d4fbf326aad01b24c0","trusted":true},"cell_type":"code","source":"#just some network parameters, see above link regarding the layers for details\nkernel_size = (3,3)\npool_size= (2,2)\nfirst_filters = 32\nsecond_filters = 64\nthird_filters = 128\n\n#dropout is used for regularization here with a probability of 0.3 for conv layers, 0.5 for the dense layer at the end\ndropout_conv = 0.3\ndropout_dense = 0.5\n\n#initialize the model\nmodel = Sequential()\n\n#now add layers to it\n\n#conv block 1\nmodel.add(Conv2D(first_filters, kernel_size, input_shape = (96, 96, 3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(first_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(MaxPool2D(pool_size = pool_size)) \nmodel.add(Dropout(dropout_conv))\n\n#conv block 2\nmodel.add(Conv2D(second_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(second_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(MaxPool2D(pool_size = pool_size))\nmodel.add(Dropout(dropout_conv))\n\n#conv block 3\nmodel.add(Conv2D(third_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(third_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(MaxPool2D(pool_size = pool_size))\nmodel.add(Dropout(dropout_conv))\n\n#a fully connected (also called dense) layer at the end\nmodel.add(Flatten())\nmodel.add(Dense(256, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Dropout(dropout_dense))\n\n#finally convert to values of 0 to 1 using the sigmoid activation function\nmodel.add(Dense(1, activation = \"sigmoid\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"859fd386ac84192fe162ecb33094e586e1a9d6bf"},"cell_type":"markdown","source":"To start training with keras we need to [compile](https://keras.io/models/model/#compile) our model. We will use a batch size of 50, i.e., feed the network 50 images at once. Further, we will use [binary crossentropy](https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a) as loss function and the [Adam optimizer](ruder.io/optimizing-gradient-descent/index.html). We set the [learning rate](https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10) of 0.001 for now.\n\nAs output we will get the classification accuracy of the model."},{"metadata":{"_uuid":"43c6aea09f1541f2411ad651ada5a3552304ac0b","trusted":true},"cell_type":"code","source":"batch_size = 50\n\nmodel.compile(loss=keras.losses.binary_crossentropy,\n              optimizer=keras.optimizers.Adam(0.001), \n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b88d166534707fac6a1553adc0b0b94f5a6baf76"},"cell_type":"markdown","source":"We are now ready to start training our model!"},{"metadata":{"_uuid":"c23406dd90b311f7683e6d5b76a5c929d8d52bc4"},"cell_type":"markdown","source":"# 6. Train and validate the model\nWe will now train the model for three epochs (should take ~20mins). That means the model will have performed a [forward and backward pass](https://cs231n.github.io/optimization-2/) for each image in the training exactly three times.\n\nTo do so, we will split the training data in batches and feed one batch after another into the network. [The batch size is a critical parameter for training a neural network](https://cs231n.github.io/neural-networks-3/#baby). \n\nKeras can do the splitting automatically for you, but, I thought, this way it is more transparent what is happening."},{"metadata":{"_uuid":"594a0db99d65d223695d4dd8d9437d915363e055","trusted":true},"cell_type":"code","source":"#normally you would want to reshuffle the data between epochs, we don't as we split in-place into training/validation\nepochs = 3 #how many epochs we want to perform\nfor epoch in range(epochs):\n    #compute how many batches we'll need\n    iterations = np.floor(split_idx / batch_size).astype(int) #the floor makes us discard a few samples here, I got lazy...\n    loss,acc = 0,0 #we will compute running loss and accuracy\n    with trange(iterations) as t: #display a progress bar\n        for i in t:\n            start_idx = i * batch_size #starting index of the current batch\n            x_batch = X[start_idx:start_idx+batch_size] #the current batch\n            y_batch = y[start_idx:start_idx+batch_size] #the labels for the current batch\n\n            metrics = model.train_on_batch(x_batch, y_batch) #train the model on a batch\n\n            loss = loss + metrics[0] #compute running loss\n            acc = acc + metrics[1] #compute running accuracy\n            t.set_description('Running training epoch ' + str(epoch)) #set progressbar title\n            t.set_postfix(loss=\"%.2f\" % round(loss / (i+1),2),acc=\"%.2f\" % round(acc / (i+1),2)) #display metrics","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd873cb97936e734aca949c2dc7673b53cbcaa8d"},"cell_type":"markdown","source":"Now, to verify that our model also works with data it hasn't seen yet, we will perform a validation epoch, i.e., check the accuracy on the validation set without further training the network. This is achieved using the [`test_on_batch` function](https://keras.io/models/sequential/#test_on_batch)."},{"metadata":{"_uuid":"86b62f51aaedb77b00bfc39a830cd256c7c826d3","trusted":true},"cell_type":"code","source":"#compute how many batches we'll need\niterations = np.floor((y.shape[0]-split_idx) / batch_size).astype(int) #as above, not perfect\nloss,acc = 0,0 #we will compute running loss and accuracy\nwith trange(iterations) as t: #display a progress bar\n    for i in t:\n        start_idx = i * batch_size #starting index of the current batch\n        x_batch = X[start_idx:start_idx+batch_size] #the current batch\n        y_batch = y[start_idx:start_idx+batch_size] #the labels for the current batch\n        \n        metrics = model.test_on_batch(x_batch, y_batch) #compute metric results for this batch using the model\n        \n        loss = loss + metrics[0] #compute running loss\n        acc = acc + metrics[1] #compute running accuracy\n        t.set_description('Running training') #set progressbar title\n        t.set_description('Running validation')\n        t.set_postfix(loss=\"%.2f\" % round(loss / (i+1),2),acc=\"%.2f\" % round(acc / (i+1),2))\n        \nprint(\"Validation loss:\",loss / iterations)\nprint(\"Validation accuracy:\",acc / iterations)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4726acfd86e61d1ad1c65fa0793204c2b128aafc"},"cell_type":"markdown","source":"# 7. Create a submission\nWell, now that we have a trained a model, we can create a submission by predicting the labels of the test data and see, where we are at in the leaderboards!\n\nLet's just first clear up some RAM. The creation of the submission is a modified version of the on presented in [this kernel](https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb)"},{"metadata":{"trusted":true,"_uuid":"3217bef913dde489dd23aebceaea82a4e88a1748"},"cell_type":"code","source":"X = None\ny = None\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d5d71c896efb4018434d00d1aab03d3355cb367"},"cell_type":"code","source":"base_test_dir = path + 'test/' #specify test data folder\ntest_files = glob(os.path.join(base_test_dir,'*.tif')) #find the test file names\nsubmission = pd.DataFrame() #create a dataframe to hold results\nfile_batch = 5000 #we will predict 5000 images at a time\nmax_idx = len(test_files) #last index to use\nfor idx in range(0, max_idx, file_batch): #iterate over test image batches\n    print(\"Indexes: %i - %i\"%(idx, idx+file_batch))\n    test_df = pd.DataFrame({'path': test_files[idx:idx+file_batch]}) #add the filenames to the dataframe\n    test_df['id'] = test_df.path.map(lambda x: x.split('/')[3].split(\".\")[0]) #add the ids to the dataframe\n    test_df['image'] = test_df['path'].map(cv2.imread) #read the batch\n    K_test = np.stack(test_df[\"image\"].values) #convert to numpy array\n    predictions = model.predict(K_test,verbose = 1) #predict the labels for the test data\n    test_df['label'] = predictions #store them in the dataframe\n    submission = pd.concat([submission, test_df[[\"id\", \"label\"]]])\nsubmission.head() #display first lines","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6113e9f267e0902a423490b75fa331f524e2aae"},"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index = False, header = True) #create the submission file","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}