{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Acknowledgements **\n\nThis kernel was inspired by \n1. The deep learning book by Michael Nielsen particularly because amazing storytelling of some confusing ideas and concepts of Neural Networks. http://neuralnetworksanddeeplearning.com/chap6.html\n2. Andrew ng Deep learning course (Course 1,2,4) on Coursera\n3. Kaggle kernel https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93   "},{"metadata":{},"cell_type":"markdown","source":"**Introduction**\n\nAs a noob, This kernel is not develop some state of the art Neural net and immediately be at the frontiers. The main purpose of this kernel is to focus on fundamentals and that's it. \nFollowing are the steps I think is right for the gradual understanding of the model:\n\n1. Creating a workable CNN using keras\n2. Improving the CNN with tweaking hyperparameters and new architecture (with varying depth) and understand their effect on the network's performance.\n\n3. Introducing Resnets/ Inception net/ and other models and use transfer learning to train our models."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfrom glob import glob\nimport gc\nimport sys\nfrom tqdm import tqdm_notebook, trange\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Step 1. loading the data in the proper format \n* X: input images rgb representation\n* y: labels for the images "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = \"../input/histopathologic-cancer-detection/\"\ntrain_path = path + 'train/'\ntest_path = path + 'test/'\n\ndf = pd.DataFrame({'path': glob(os.path.join(train_path,'*.tif'))})\n\n\ndf['id'] = df.path.map(lambda x: x.split('/')[4].split(\".\")[0]) # keep only the file names in 'id'\nlabels = pd.read_csv(path+\"train_labels.csv\")                   # read the provided labels\ndf = df.merge(labels, on = \"id\")                                # merge labels and filepaths                                           \ndf.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\n\nfrom keras.models import load_model\n\nnew_model = tf.keras.models.load_model('../input/128batch/my_model4.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To get the rgb numeric representation, we will use the cv2 library by opencv for computer vision .cv2.imread takes the input images path and outputs the images in rgb format"},{"metadata":{"trusted":true},"cell_type":"code","source":"#loading the data of N inputs and labels\nimport  cv2                   \ndef load_data(df,N):\n    X= np.zeros([N,96,96,3],dtype= np.uint8)\n    y = np.zeros([N,1], dtype = np.uint8)\n    for i, row in tqdm_notebook(df.iterrows(), total = N):\n        if i == N:\n            break\n        X[i] = cv2.imread(row['path'])\n        y[i] = row['label']\n        \n    \n    return X,y\n\nX,y = load_data(df= df, N = len(df))\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['label'].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Due to limited availability of the ram(13 GB), we dont want to create any new variable ( X_train, X_val, Y_train, Y_test). as that will overflow the RAM. \n1. Instead, we shuffle the data ,so that the distribution of the data is not skewed. We dont want some particular types of image just in training data. That will lead to poor generalization by the network for the test and validation set.\n2. We replace (to deal with limited RAM) the X,y with the new shuffled indexes"},{"metadata":{"trusted":true},"cell_type":"code","source":"training_portion = 0.8 # Specify training/validation ratio\nsplit_idx = int(np.round(training_portion * y.shape[0])) #Compute split idx\n\nnp.random.seed(42) #set the seed to ensure reproducibility\n\n#shuffle\nidx = np.arange(y.shape[0])\nnp.random.shuffle(idx)\nX = X[idx]\ny = y[idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(X.shape[0])\nprint(y.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"# import tensorflow as tf\n# import keras\n# from keras import optimizers\n# import tensorflow.keras.models\n# from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropout, Activation\n# from tensorflow.keras.models import Sequential\n\n# dropout_conv = 0.3\n# dropout_dense = 0.5\n# #creating the new_model \n\n# new_model = Sequential()\n# new_model.add(Conv2D(filters = 32,kernel_size = 3, activation = 'relu', padding = 'same',input_shape= [96,96,3]))\n# new_model.add(MaxPooling2D(pool_size = 2))\n# new_model.add(Dropout(dropout_conv))\n\n# new_model.add(Conv2D(filters = 64,kernel_size = 3, activation = 'relu', padding = 'same'))\n# new_model.add(MaxPooling2D(pool_size = 2))\n# new_model.add(Dropout(dropout_conv))\n\n# new_model.add(Conv2D(filters = 128,kernel_size = 3, activation = 'relu', padding = 'same'))\n# new_model.add(MaxPooling2D(pool_size = (2,2)))\n# new_model.add(Dropout(dropout_conv)) \n    \n# new_model.add(Flatten())\n# new_model.add(Dense(256, activation ='relu'))\n# new_model.add(Dropout(dropout_dense))\n\n# new_model.add(Dense(1,activation = 'sigmoid'))\n\n# new_model.compile(loss = 'binary_crossentropy',\n#              optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False),\n#              metrics = ['accuracy'])\n\n# # new_model.fit(X_train, Y_train, batch_size = 100, epochs = 15, validation_data = (X_test, Y_test))\n# new_model.fit(X[:split_idx], y[:split_idx], epochs=10, batch_size= 32,validation_data = (X[split_idx:], y[split_idx:]))\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training on the 80% of the dataset and validation on the rest. The accuracy and loss calculated are for each mini-batch and not for the whole dataset. \n\n\nLater, after initial run, we can get the correct accuracy and loss for the whole dataset,Now, we are averaging the accuracy for each mini-batch to calculate for the whole epoch. (A rough estimation)"},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 7\nbatch_size = 128\n\nfor epoch in range(epochs):\n    loss, accuracy  = 0,0\n    \n    iterations = int(split_idx/batch_size)\n    \n#     with trange(0,split_idx, batch_size) as t:\n#         for i in t:\n#             start_ind = i\n#             x_batch = X[i:i + batch_size]\n#             y_batch = y[i:i + batch_size]\n            \n            \n\n    with trange(iterations) as t:\n        for i in t:\n            start_ind = i * batch_size\n            x_batch = X[start_ind: start_ind + batch_size]\n            y_batch = y[start_ind: start_ind + batch_size]\n\n            metrics = new_model.train_on_batch(x_batch, y_batch)\n\n            loss = loss + metrics[0]   # calculating loss and accuracy for that mini- batch\n            accuracy = accuracy +metrics[1]\n\n            t.set_description('running_training_epoch '+ str(epoch))\n            t.set_postfix(loss = \"%.2f\" % round(loss/(i+1),2), accuracy = \"%.2f\" % round(accuracy/(i+1),2) )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After training on the training dataset, we trained the weights and biases of the network. Now, we input the validation images into the network and get the output and calculate the cost and accuracy for the validation set\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = X[split_idx:]\ny = y[split_idx:]\n\niterations = int((y.shape[0])/batch_size)\n# iterations = 20\nloss, accuracy = 0,0 \nwith trange(iterations) as t:\n    for i in t:\n\n        start_idx = i * batch_size #starting index of the current batch\n        x_batch = X[start_idx :start_idx + batch_size] #the current batch\n        y_batch = y[start_idx:start_idx + batch_size] #the labels for the current batch\n\n\n        metrics = new_model.test_on_batch(x_batch, y_batch)\n\n        loss = loss + metrics[0]   # calculating loss and accuracy for that mini- batch\n        accuracy = accuracy +metrics[1]\n\n        t.set_description('running_validation ')\n        t.set_postfix(loss = round(loss/(i+1),2), accuracy = round(accuracy/(i+1),2) )\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = None\ny = None\ndf= None\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After training and validation, we input the test image into the network and take the output and upload in kaggle to get the testing accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame({'path': glob(os.path.join(test_path,'*.tif'))})\ntest_df['id'] = test_df['path'].map(lambda x : x.split('/')[4].split('.')[0])\n\ntest_df['image'] = test_df['path'].map(cv2.imread)\nX = np.stack(test_df['image'], axis = 0)\n\nprint(X.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = new_model.predict(X, verbose = 1)\ntest_df['label'] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = None\ngc.collect()\nsubmission = pd.DataFrame()\nsubmission['id'] = test_df['id']\n\nsubmission['label'] = predictions\n\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"submission.csv\", header = True, index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import load_model\n\nnew_model.save('my_model4.h5')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}