{"cells":[{"metadata":{},"cell_type":"markdown","source":"Started on 24 June 2019"},{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{},"cell_type":"markdown","source":"#### Following from working on [Aerial Cactus Identification][1] problem using CNN, I try to adapt the same approach on this [Histopathologic Cancer Detection][2] problem.\n[1]: https://www.kaggle.com/rhodiumbeng/aerial-cactus-identification-using-cnn\n[2]: https://www.kaggle.com/c/histopathologic-cancer-detection\n#### Unlike the cactus datasets, the size of the data here (essentially the images) are too large to be loaded as a whole into memory for training and prediction. So we have to use the 'flow' functionality from Keras' ImageDataGenerator.\n#### I would like to thank [Marsh][3] for sharing his insightful [kernel][4]. I learned a lot from it.\n[3]: https://www.kaggle.com/vbookshelf\n[4]: https://www.kaggle.com/vbookshelf/cnn-how-to-use-160-000-images-without-crashing\n#### Separately, I also found this wonderful [resource][5] from [Vijayabhaskar J][6] on using \"flow_from_dataframe\" in ImageDataGenerator. Vijaybhaskar wrote this function that got accepted to the official keras-preprocessing git repo. This allows us to input a Pandas dataframe which contains the filenames column and a column which has the class names and directly read the images from the directory with their respective class names mapped. Wonderful!\n[5]: https://medium.com/@vijayabhaskar96/tutorial-on-keras-flow-from-dataframe-1fd4493d237c\n[6]: https://medium.com/@vijayabhaskar96\n#### I had based this kernel very much from the guidance from the above resources."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Examine the data"},{"metadata":{},"cell_type":"markdown","source":"* The images (tif) for the training data and test data are found in the train and test folders. The filenames of the tif image files are used as the unique 'id' in the csv files.\n* 'train_csv' contains the training data ('id' and 'label') and 'sample_submission.csv' contains the test data 'id'."},{"metadata":{"trusted":true},"cell_type":"code","source":"# load data from csv files\ntrain_df = pd.read_csv('../input/train_labels.csv')\ntest_df = pd.read_csv('../input/sample_submission.csv')\nprint(train_df.shape, test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The 'id' in the csv files are without file extension. So we add '.tif' to make them correspond exactly to the image files."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['id'] = train_df['id'].apply(lambda x: x+'.tif')\ntest_df['id'] = test_df['id'].apply(lambda x: x+'.tif')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['label'] = train_df['label'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array\n\ntrain_path = '../input/train/'\ntest_path = '../input/test/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Here are some images from the training data that are labelled as positive, i.e. '1':"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# look at some of the pics from train_df labelled '1'\npositive = train_df[train_df['label']=='1']\nplt.figure(figsize=(15,7))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(load_img(train_path+positive.iloc[i]['id']))\n    plt.title(\"label=%s\" % positive.iloc[i]['label'], y=1)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Here are some images from the training data that are labelled negative, i.e. '0':"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# look at some of the pics from train_df labelled '0'\nnegative = train_df[train_df['label']=='0']\nplt.figure(figsize=(15,7))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(load_img(train_path+negative.iloc[i]['id']))\n    plt.title(\"label=%s\" % negative.iloc[i]['label'], y=1)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Setting up ImageDataGenerator"},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"from keras_preprocessing.image import ImageDataGenerator\ndatagen = ImageDataGenerator(rescale=1./255., validation_split=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set up two data generators; (1) training, (2) validation from train set\nn_x = 96\ntrain_generator = datagen.flow_from_dataframe(dataframe=train_df, \n                                              directory=train_path, \n                                              target_size=(n_x,n_x), \n                                              x_col='id', y_col='label', \n                                              subset='training', \n                                              batch_size=128, seed=12, \n                                              class_mode='categorical')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_generator = datagen.flow_from_dataframe(dataframe=train_df, \n                                              directory=train_path,\n                                              target_size=(n_x,n_x), \n                                              x_col='id', y_col='label', \n                                              subset='validation', \n                                              batch_size=128, seed=12, \n                                              class_mode='categorical')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set up data generator for test set\ntest_datagen = ImageDataGenerator(rescale=1./255.)\ntest_generator = test_datagen.flow_from_dataframe(dataframe=test_df, \n                                                  directory=test_path, \n                                                  target_size=(n_x,n_x), \n                                                  x_col='id', y_col=None, \n                                                  batch_size=1, seed=12, \n                                                  shuffle=False, \n                                                  class_mode=None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define step sizes for model training\nstep_size_train = train_generator.n//train_generator.batch_size\nstep_size_valid = valid_generator.n//valid_generator.batch_size\nstep_size_test = test_generator.n//test_generator.batch_size\nprint(step_size_train, step_size_valid, step_size_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create CNN Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# build the CNN from keras\nfrom keras import models\nfrom keras import layers\n\nmodel = models.Sequential()\nmodel.add(layers.Conv2D(32, kernel_size=5, activation='relu', input_shape=(96, 96, 3)))\nmodel.add(layers.Conv2D(32, kernel_size=5, activation='relu'))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2), strides=2))\nmodel.add(layers.Dropout(rate=0.4))\nmodel.add(layers.Conv2D(64, kernel_size=5, activation='relu'))\nmodel.add(layers.Conv2D(64, kernel_size=5, activation='relu'))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2), strides=2))\nmodel.add(layers.Dropout(rate=0.4))\nmodel.add(layers.Conv2D(128, kernel_size=5, activation='relu'))\nmodel.add(layers.Conv2D(128, kernel_size=5, activation='relu'))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2), strides=2))\nmodel.add(layers.Dropout(rate=0.4))\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(256, activation='relu'))\nmodel.add(layers.Dropout(rate=0.4))\nmodel.add(layers.Dense(256, activation='relu'))\nmodel.add(layers.Dense(2, activation='softmax'))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', \n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Run the model on the train and validation data, and capture metrics history to visualise the performance of the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train and validate the model\nepochs = 20\nhistory = model.fit_generator(generator=train_generator,\n                              steps_per_epoch=step_size_train, \n                              validation_data=valid_generator, \n                              validation_steps=step_size_valid,\n                              epochs=epochs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plot and visualise the training and validation losses\nloss = history.history['loss']\ndev_loss = history.history['val_loss']\nepochs = range(1, len(loss) + 1)\n\nfrom matplotlib import pyplot as plt\nplt.figure(figsize=(15,10))\nplt.plot(epochs, loss, 'bo', label='training loss')\nplt.plot(epochs, dev_loss, 'b', label='validation loss')\nplt.title('Training and Validation Loss')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict on test set\ntest_generator.reset()\npred = model.predict_generator(test_generator, steps=step_size_test, \n                               verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create submission file\nsub = pd.read_csv('../input/sample_submission.csv')\nsub['label'] = pred[:,0]\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# generate submission file in csv format\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}