{"cells":[{"metadata":{"_uuid":"602c58398509eebbe6a9ff18c8e58babe4e03e33"},"cell_type":"markdown","source":"# Introdcution\nI decided to try CNN with Keras to address the problem of histopathologic cancer detection. \nThis kernel is based on following tutorial: \"[Tutorial on Keras flow_from_dataframe](https://medium.com/@vijayabhaskar96/tutorial-on-keras-flow-from-dataframe-1fd4493d237c)\""},{"metadata":{"_uuid":"1ad266c1f751b31d956d8743f8a6404961983202"},"cell_type":"markdown","source":"# Import Data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3098e9dd17b5f917fe47ed4254e8b63671cce2fb"},"cell_type":"markdown","source":"Load the labelled data, each one being composed by the image name without file extension and the label.\nWe adjust the image name to add the file extension."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def append_ext(fn):\n    return fn+\".tif\"\n\ndf=pd.read_csv(\"../input/train_labels.csv\",dtype=str)\ndf[\"id\"]=df[\"id\"].apply(append_ext)\ndf.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec13ff0e1adba4d478e1fd2b6f5ad7da3822c2fb"},"cell_type":"code","source":"# Display the head of the train data frame\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e37100f1cb2343602d80e8ad4976d01f0a849f69"},"cell_type":"markdown","source":"# Check Data"},{"metadata":{"trusted":true,"_uuid":"c7bfdb0fdce580a84aa636fa5653aef1546d3f80"},"cell_type":"code","source":"# Get the labels\nY = df[\"label\"]\n# Display the histogram of labels\nimport seaborn as sns\ng = sns.countplot(Y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4ca52e96f9b42fc29164ac4a704ba180c4a1ad67"},"cell_type":"code","source":"# Count the number of labels\nY.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6edfcb13c93a7a154327a2d63fcf17cf159e73d1"},"cell_type":"markdown","source":"# Data Generator\nDefine the data generator for the classifier fitting, one for the training set and one for the validation set.  These generators will feed the algorithm wit the data from the data frame with the corresponding image files in the training folder.\nThe training will be composed of 80% of the orginal training data and the validation will be 20% of the original training data.  \nOrginal data will be shuffled."},{"metadata":{"trusted":true,"_uuid":"9062c40fb18c9bf4cbaa3851afa4cdc8792acb5b"},"cell_type":"code","source":"df_train = df.sample(frac=1, random_state=1)\ndf_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b8d3002669793c593a5a4a6fcae2b60d495d7a8c"},"cell_type":"code","source":"# Get the labels\nY_train = df_train[\"label\"]\n# Display the histogram of labels\nimport seaborn as sns\ng_train = sns.countplot(Y_train)\n# Count the number of labels\nY_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c873ea81e6bd1a4bb7348f409f6ee8b3898bb13"},"cell_type":"code","source":"crop_dim = (32,32) # (x,y)\nimg_dim = (96,96) # (x,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70c178bc145596b7388eb8b9d873948bcd813bb5"},"cell_type":"code","source":"def crop_center(img, crop_dim, img_dim):\n    # Note: image_data_format is 'channel_last'\n    assert img.shape[2] == 3\n    height, width = img.shape[0], img.shape[1]\n    dx, dy = crop_dim\n    x, y = (int((img_dim[0]-crop_dim[0])/2),int((img_dim[1]-crop_dim[1])/2))\n    return img[y:(y+dy), x:(x+dx), :]\n\n\ndef crop_generator(batches):\n    while True:\n        batch_x, batch_y = next(batches)\n        batch_crops = np.zeros((batch_x.shape[0], crop_dim[0], crop_dim[1], 3))\n        for i in range(batch_x.shape[0]):\n            batch_crops[i] = crop_center(batch_x[i],crop_dim,img_dim)\n        yield (batch_crops, batch_y)\n\n\nfrom keras_preprocessing.image import ImageDataGenerator\ndatagen=ImageDataGenerator(validation_split=0.20,\n                          rescale=1/255.0)\n\ntrain_generator=datagen.flow_from_dataframe(\n        dataframe=df_train,\n        directory=\"../input/train/\",\n        x_col=\"id\",\n        y_col=\"label\",\n        subset=\"training\",\n        batch_size=160,\n        seed=42,\n        shuffle=True,\n        class_mode=\"categorical\",\n        target_size=img_dim)\n\nif 0:\n    train_generator_crops = crop_generator(train_generator)\n\nvalid_generator=datagen.flow_from_dataframe(\n        dataframe=df_train,\n        directory=\"../input/train/\",\n        x_col=\"id\",\n        y_col=\"label\",\n        subset=\"validation\",\n        batch_size=160,\n        seed=42,\n        shuffle=True,\n        class_mode=\"categorical\",\n        target_size=img_dim)\n\nif 0:\n    valid_generator_crops = crop_generator(valid_generator)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d7e3b7e58a73e242a9c00b95941601cf52a6f02"},"cell_type":"markdown","source":"# CNN Model Definition"},{"metadata":{"trusted":true,"_uuid":"895be08ca670536ed29f7a2a8162967113e49303"},"cell_type":"code","source":"# Importing libraries\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Flatten\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\n\n#height, width = crop_dim\nheight, width = img_dim\n\n# Initialising the CNN\nclassifier = Sequential()\n\nclassifier.add(Conv2D(32, (3, 3), input_shape = (height,width,3), activation = 'relu'))\nclassifier.add(MaxPooling2D(pool_size = (2, 2)))\nclassifier.add(Dropout(0.20))\n\nclassifier.add(Conv2D(64, (3, 3),activation = 'relu'))\nclassifier.add(MaxPooling2D(pool_size = (2, 2)))\nclassifier.add(Dropout(0.20))\n\nclassifier.add(Conv2D(128, (3, 3),activation = 'relu'))\nclassifier.add(MaxPooling2D(pool_size = (2, 2)))\nclassifier.add(Dropout(0.20))\n\n# Step 3 - Flattening\nclassifier.add(Flatten())\n\n# Step 4 - Full connection\nclassifier.add(Dense(units = 256, activation = 'relu'))\nclassifier.add(Dropout(0.20))\nclassifier.add(Dense(units = 64, activation = 'relu'))\nclassifier.add(Dropout(0.20))\nclassifier.add(Dense(units = 64, activation = 'relu'))\nclassifier.add(Dropout(0.20))\nclassifier.add(Dense(units = 64, activation = 'relu'))\nclassifier.add(Dropout(0.20))\nclassifier.add(Dense(units = 64, activation = 'relu'))\nclassifier.add(Dropout(0.20))\nclassifier.add(Dense(units = 2, activation = 'softmax')) #softmax for classification","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cfa1856eb7c8d56ffdd53599f71212ce1f33adf1"},"cell_type":"code","source":"# Compiling the CNN\nfrom keras import optimizers\nclassifier.compile(optimizer = 'adam', \n                   loss = 'categorical_crossentropy', \n                   metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89151552580246170210dd6b3cb506a86da893dd"},"cell_type":"markdown","source":"# CNN Model Training"},{"metadata":{"trusted":true,"_uuid":"8e77a17f9f720e9cd7cd2d10629aff23bc31890f"},"cell_type":"code","source":"# Not used at the moment\nif 0:\n    STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\n    STEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\n    history = classifier.fit_generator(generator=train_generator_crops,\n                             steps_per_epoch=STEP_SIZE_TRAIN,\n                             validation_data=valid_generator_crops,\n                             validation_steps=STEP_SIZE_VALID,\n                             epochs=10,\n                             verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0cce243586498923bbbb41f0839df41011275970"},"cell_type":"code","source":"STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\nhistory = classifier.fit_generator(generator=train_generator,\n                         steps_per_epoch=STEP_SIZE_TRAIN,\n                         validation_data=valid_generator,\n                         validation_steps=STEP_SIZE_VALID,\n                         epochs=20,\n                         verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65bc352a0d9c547e4c795728eb5fec2dfe34cc93"},"cell_type":"markdown","source":"# CNN Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"824b29cfb4125a76fd0aa2db3651222d6c5addfb"},"cell_type":"code","source":"# Display model performance over epochs\nfrom matplotlib import pyplot as plt\nfig, ax = plt.subplots(1, 2, figsize=(15,5))\nax[0].set_title('loss')\nax[0].plot(history.epoch, history.history[\"loss\"], label=\"Train loss\")\nax[0].plot(history.epoch, history.history[\"val_loss\"], label=\"Validation loss\")\nax[0].legend()\nax[1].set_title('acc')\nax[1].plot(history.epoch, history.history[\"acc\"], label=\"Train acc\")\nax[1].plot(history.epoch, history.history[\"val_acc\"], label=\"Validation acc\")\nax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe6a306b3d49babe6646d5255583f16dbee41173"},"cell_type":"code","source":"# Not used at the moment\nif 0:\n    score = classifier.evaluate_generator(generator=valid_generator_crops, steps=STEP_SIZE_VALID, workers=1)\n    print('Score: ',score)\n    print('Metrics: ',classifier.metrics_names)\n    classifier.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d19c0164a34d8d5284a10a3f12bc218ec146b4fd"},"cell_type":"code","source":"score = classifier.evaluate_generator(generator=valid_generator, steps=STEP_SIZE_VALID, workers=1)\nprint('Score: ',score)\nprint('Metrics: ',classifier.metrics_names)\nclassifier.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f00d73c11d908a7227b9ba8c012e6abfffbdde38"},"cell_type":"markdown","source":"# Create Submission on Challenge Test Set"},{"metadata":{"trusted":true,"_uuid":"293d733a4d81ac049abf7c9841573637fc6b54ee"},"cell_type":"code","source":"df_test=pd.read_csv(\"../input/sample_submission.csv\",dtype=str)\ndf_test[\"id\"]=df_test[\"id\"].apply(append_ext)\ndf_test.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b6f51310abdb8e84634832e6a9d0cd2420f57a0e"},"cell_type":"code","source":"df_test.isnull().values.any()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c127d8a25f92556a7b8bc1ba6611c1f68fb22e2"},"cell_type":"code","source":"test_datagen=ImageDataGenerator(rescale=1/255.0)\n\ntest_generator=test_datagen.flow_from_dataframe(\n        dataframe=df_test,\n        directory=\"../input/test/\",\n        x_col=\"id\",\n        y_col=None,\n        batch_size=1,\n        seed=42,\n        shuffle=False,\n        class_mode=None,\n        target_size=img_dim)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7ad57b2e0ded69158f9f4c9aa0cccda8ebbd9f7"},"cell_type":"code","source":"def crop_generator_single(batches):\n    while True:\n        batch_x = next(batches)\n        batch_crops = np.zeros((batch_x.shape[0], crop_dim[0], crop_dim[1], 3))\n        for i in range(batch_x.shape[0]):\n            batch_crops[i] = crop_center(batch_x[i],crop_dim,img_dim)\n        yield (batch_crops)\n        \nif 0:\n    test_generator_crops = crop_generator_single(test_generator)\n\nSTEP_SIZE_TEST=test_generator.n//test_generator.batch_size\ntest_generator.reset()\n\nif 0:\n    pred=classifier.predict_generator(test_generator_crops,\n                                      steps=STEP_SIZE_TEST,\n                                      verbose=1)\n\n    \npred=classifier.predict_generator(test_generator,\n                                  steps=STEP_SIZE_TEST,\n                                  verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa0f9d6b52a7c91c4e30e3f2c7f4fa2c6f04f55d"},"cell_type":"code","source":"print(pred.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b8d7d1803edddfb955c094cdc1d896b88d4a17b7"},"cell_type":"code","source":"print(pred[:5,:])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7b7135b2786f9cf3ad86c32a5257362ab116030d"},"cell_type":"code","source":"predicted_class_indices=np.argmax(pred,axis=1)\n\n\nlabels = (train_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\npredictions = [labels[k] for k in predicted_class_indices]\n\nfilenames=test_generator.filenames\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7bad1ee2e8e2efcfb90872abc63cc07f3c7c4ce6"},"cell_type":"code","source":"print(labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a2a795ec322a62df6dc4c67440a1d7638064cba"},"cell_type":"code","source":"print(predictions[:5])\nprint(len(predictions))\n# Display the histogram of labels\nimport seaborn as sns\ng = sns.countplot(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb8ab94f682b52cfe1059eb805f5f207805bbf19"},"cell_type":"code","source":"# Remove the file extension .tif\nprint(len(filenames))\nimport os\nprint(os.path.splitext(os.path.basename(filenames[0]))[0])\nfor i in range(0,len(filenames)):\n    filenames[i] = os.path.splitext(os.path.basename(filenames[i]))[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"179974b69a17b4ed3cbde093eb7f0d1d51d40154"},"cell_type":"code","source":"# Create submission file\nsubmission=pd.DataFrame({\"id\":filenames,\"label\":predictions})\nsubmission.to_csv(\"Submission.csv\",index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}