{"cells":[{"metadata":{"_cell_guid":"d4c08f48-fe23-4ddb-ac46-d97f05397514","_uuid":"f2156d1dd26a1243e18512002e10872c5bd7271e"},"cell_type":"markdown","source":"# APTOS 2019 Blindness Detection"},{"metadata":{},"cell_type":"markdown","source":"# I. Competition description"},{"metadata":{},"cell_type":"markdown","source":"> Imagine being able to detect blindness before it happened. Millions of people suffer from diabetic retinopathy, the leading cause of blindness among working aged adults. Aravind Eye Hospital in India hopes to detect and prevent this disease among people living in rural areas where medical screening is difficult to conduct. Successful entries in this competition will improve the hospital’s ability to identify potential patients. Further, the solutions will be spread to other Ophthalmologists through the 4th Asia Pacific Tele-Ophthalmology Society (APTOS) Symposium\n\n> Currently, Aravind technicians travel to these rural areas to capture images and then rely on highly trained doctors to review the images and provide diagnosis. Their goal is to scale their efforts through technology; to gain the ability to automatically screen images for disease and provide information on how severe the condition may be.\n\n> In this synchronous Kernels-only competition, you'll build a machine learning model to speed up disease detection. You’ll work with thousands of images collected in rural areas to help identify diabetic retinopathy automatically. If successful, you will not only help to prevent lifelong blindness, but these models may be used to detect other sorts of diseases in the future, like glaucoma and macular degeneration.\n\n> Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). In the event that there is less agreement between the raters than expected by chance, this metric may go below 0. The quadratic weighted kappa is calculated between the scores assigned by the human rater and the predicted scores.\n\n> Images have five possible ratings, 0,1,2,3,4.  Each image is characterized by a tuple (e,e), which corresponds to its scores by Rater A (human) and Rater B (predicted).  The quadratic weighted kappa is calculated as follows. First, an N x N histogram matrix O is constructed, such that O corresponds to the number of images that received a rating i by A and a rating j by B. An N-by-N matrix of weights, w, is calculated based on the difference between raters' scores:\n\n> An N-by-N histogram matrix of expected ratings, E, is calculated, assuming that there is no correlation between rating scores.  This is calculated as the outer product between each rater's histogram vector of ratings, normalized such that E and O have the same sum.\n\nsource: https://www.kaggle.com/c/aptos2019-blindness-detection"},{"metadata":{},"cell_type":"markdown","source":"# II. Summary of my strategy"},{"metadata":{},"cell_type":"markdown","source":"* Images differ in their luminosity and the extent to which they zoom on a particular area of the retina. I pre-process images by selecting a circular area around the image center and I convert images to gray scale\n* I use data augmentation to increase the size and the diversity of the training data. Given that my pre-processing\ncreates circular images, I only allow for image rotation to preserve the circularity of the input data\n* I use an ensemble of N Convolutional Neural Network (CNN) to predict the five possible outcomes. In practice, I use only 2 CNN\n\n### Credits\n* The idea of cicular cropping comes from this notebook: https://www.kaggle.com/ratthachat/aptos-eye-preprocessing-in-diabetic-retinopathy\n* The implementation of an ensemble of N Convolutional Neural Network was greatly inspired by this notebook: https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist\n"},{"metadata":{"_cell_guid":"eb88b372-a6e5-40c8-a1c6-c03799165490","_uuid":"e9aff3cf1bb8daa73bec67b970d12195677679f3"},"cell_type":"markdown","source":"# III. Implementation"},{"metadata":{},"cell_type":"markdown","source":"## III.A. Preliminaries"},{"metadata":{},"cell_type":"markdown","source":"List of imports:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import cohen_kappa_score\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.models import load_model\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2 #!pip3 install --user opencv-python\n%matplotlib inline\nfrom PIL import Image\nfrom os import listdir\nimport os\nimport glob\nfrom os.path import isfile, join","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Options:"},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE = 64 #length side of images after pre-processing (images are reshaped to squares)\nimg_size = IMG_SIZE * IMG_SIZE #size of the vector representing an image\nlist_features_names = [\"pixel\" + str(i) for i in range(0, img_size)] # list of features\nDEBUG = False #To pre-process only a limited amount of images\npre_process_image = False #To pre-process train and test sets\ntrain_model = False #To train the model(s). If false, load model(s) from disk","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Paths:"},{"metadata":{"trusted":true},"cell_type":"code","source":"path_to_data  = '/kaggle/input/aptos2019-blindness-detection'\npath_to_preprocessed_data_model = '/kaggle/input/aptos-2019' #if using pre-processed data and/or model\npath_to_train_img = path_to_data + '/train_images'\npath_to_test_img = path_to_data + '/test_images'\npath_to_train_img_pre = path_to_data + '/train_images_preprocessed'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## III.B. Image pre-processing"},{"metadata":{},"cell_type":"markdown","source":"This block pre-processes images and store the results to disk. Then we can experiment with different CNN settings\nwithout having to go through the entire process again. First of all, let's create a list to store the name of each image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"list_files = listdir(path_to_train_img)\n#X_train.columns\nlist_files = [s for s in list_files if '.png' in s]","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"list_files[0:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Functions for circle cropping"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rgb2gray(rgb):\n    return np.dot(rgb[...,:3], [0.299, 0.587, 0.144])\n\ndef crop_image_from_gray(img,tol=7):\n    \"\"\"\n    Crop out black borders\n    https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping\n    \"\"\"  \n    \n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        mask = gray_img>tol        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0):\n            return img\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n            img = np.stack([img1,img2,img3],axis=-1)\n        return img\n\n\ndef circle_crop(img):   \n    \"\"\"\n    Create circular crop around image centre    \n    \"\"\"    \n    \n    img = cv2.imread(img)\n    img = crop_image_from_gray(img)    \n    \n    height, width, depth = img.shape    \n    \n    x = int(width/2)\n    y = int(height/2)\n    r = np.amin((x,y))\n    \n    circle_img = np.zeros((height, width), np.uint8)\n    cv2.circle(circle_img, (x,y), int(r), 1, thickness=-1)\n    img = cv2.bitwise_and(img, img, mask=circle_img)\n    img = crop_image_from_gray(img)\n    \n    return img \n\ndef circle_crop_v2(img, IMG_SIZE = 512):\n    \"\"\"\n    Create circular crop around image centre\n    \"\"\"\n    img = cv2.imread(img)\n    img = crop_image_from_gray(img)\n\n    #height, width, depth = img.shape\n    #largest_side = np.max((height, width))\n    #img = cv2.resize(img, (largest_side, largest_side))\n    img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))\n\n    height, width, depth = img.shape\n\n    x = int(width / 2)\n    y = int(height / 2)\n    r = np.amin((x, y))\n\n    circle_img = np.zeros((height, width), np.uint8)\n    cv2.circle(circle_img, (x, y), int(r), 1, thickness=-1)\n    img = cv2.bitwise_and(img, img, mask=circle_img)\n    img = crop_image_from_gray(img)\n    # Make sure the image has the right size\n    img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))\n\n    return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Preview of the pre-processing outcome"},{"metadata":{"trusted":true},"cell_type":"code","source":"## try circle crop\nfig = plt.figure(figsize=(25, 16))\nfor (index_img, img_name)  in enumerate(list_files):\n    # DEBUG\n    if index_img > 10:\n        break\n    ax = fig.add_subplot(5,5, index_img+1)\n    path= path_to_train_img + '/' + img_name\n    image = circle_crop_v2(path, IMG_SIZE = IMG_SIZE)\n    image = rgb2gray(image)\n    plt.imshow(image, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train set pre-processing"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Circle cropping\nif pre_process_image == True:\n    print(\"Pre-processing train images\")\n    df_train = pd.read_csv(path_to_data + '/train.csv')\n    df_train.head()\n    image_2D = None #To store images in a vector form\n    # Read files following the order of df_train\n    for (index_img, row) in df_train.iterrows():\n    #for (index_img, img_name) in enumerate(list_files):\n        img_name = row['id_code'] + '.png'\n        # DEBUG\n        if DEBUG==True:\n            if index_img > 10:\n                break\n        path= path_to_train_img + '/' + img_name\n        image = circle_crop_v2(path, IMG_SIZE = IMG_SIZE)\n        image = rgb2gray(image)\n        img_1D_vector = image.reshape(img_size, 1)\n        if index_img == 0:\n            image_2D = img_1D_vector\n        else:\n            image_2D = np.concatenate((image_2D, img_1D_vector), axis=1)\n    print(\"Done\")\n    print(\"Merging df\")\n    data_train_img = pd.DataFrame(data=np.transpose(image_2D), # values\n                  index=list(range(0, image_2D.shape[1])), # 1st column as index\n                  columns=list_features_names) \n\n    df_merged = df_train.join(data_train_img, how='outer')\n    print(df_merged.head())\n    df_merged.to_csv('train_preprocessed.csv', index=False)\n    print(\"Done\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Freeing memory\ndf_merged = None\ndata_train_img = None\ndf_train = None","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test set pre-processing"},{"metadata":{"trusted":true},"cell_type":"code","source":"if pre_process_image == True:\n    print(\"Pre-processing test images\")\n    df_test = pd.read_csv(path_to_data + '/test.csv')\n    df_test.head()\n    ## Apply similar transformation to the test dataset\n    image_2D = None\n    # Read files following the order of df_train\n    for (index_img, row) in df_test.iterrows():\n    #for (index_img, img_name) in enumerate(list_files):\n        img_name = row['id_code'] + '.png'\n        # DEBUG\n        if DEBUG == True:\n            if index_img > 10:\n                break\n        path= path_to_test_img + '/' + img_name\n        image = circle_crop_v2(path, IMG_SIZE = IMG_SIZE)\n        image = rgb2gray(image)\n        img_1D_vector = image.reshape(img_size, 1)\n        if index_img == 0:\n            image_2D = img_1D_vector\n        else:\n            image_2D = np.concatenate((image_2D, img_1D_vector), axis=1)\n    print(\"Done\")\n    print(\"Merging df\")\n    data_test_img = pd.DataFrame(data=np.transpose(image_2D), # values\n              index=list(range(0, image_2D.shape[1])), # 1st column as index\n              columns=list_features_names) \n    df_merged = df_test.join(data_test_img, how='outer')\n    df_merged.head()\n    df_merged.to_csv('test_preprocessed.csv', index=False)\n    print(\"Done\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_merged = None\ndata_test_img = None\ndf_test = None","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"bdb422e2-bdec-444f-97a5-283a1e54bf2c","_uuid":"39b7a31e843bac6b705461bcce89da216b91799e"},"cell_type":"markdown","source":"## IV. Training"},{"metadata":{},"cell_type":"markdown","source":"### IV.A. Loading pre-processed data"},{"metadata":{},"cell_type":"markdown","source":"Having pre-processed the data, let's proceed to training. First of all, let's load the pre-processed data:"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"if pre_process_image == True:\n    train = pd.read_csv('train_preprocessed.csv')\nelse:\n    train = pd.read_csv(path_to_preprocessed_data_model+'/train_preprocessed.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREPARE DATA FOR NEURAL NETWORK\ntrain[\"label\"] = train[\"diagnosis\"]\nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\", \"diagnosis\", \"id_code\"],axis = 1)\n# Normalization:\nX_train = X_train / 255.0\nX_train = X_train.values.reshape(-1,IMG_SIZE,IMG_SIZE,1)\nY_train = to_categorical(Y_train, num_classes = 5)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# PREVIEW IMAGES\nplt.figure(figsize=(15,4.5))\nfor i in range(20):  \n    plt.subplot(3, 10, i+1)\n    plt.imshow(X_train[i].reshape((IMG_SIZE,IMG_SIZE)),cmap=plt.cm.binary)\n    plt.axis('off')\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### IV.B. Data augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# CREATE MORE IMAGES VIA DATA AUGMENTATION\ndatagen = ImageDataGenerator(\n        rotation_range=30) #let's only rotate images for the moment","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREVIEW AUGMENTED IMAGES\nX_train3 = X_train[9,].reshape((1,IMG_SIZE,IMG_SIZE,1))\nY_train3 = Y_train[9,].reshape((1,5))\nplt.figure(figsize=(15,4.5))\nfor i in range(20):  \n    plt.subplot(3, 10, i+1)\n    X_train2, Y_train2 = datagen.flow(X_train3,Y_train3).next()\n    #plt.imshow(X_train2[0].reshape((IMG_SIZE,IMG_SIZE)),cmap=plt.cm.binary)\n    plt.imshow(X_train2[0].reshape((IMG_SIZE,IMG_SIZE)),cmap='gray')\n    plt.axis('off')\n    if i==9: X_train3 = X_train[11,].reshape((1,IMG_SIZE,IMG_SIZE,1))\n    if i==19: X_train3 = X_train[18,].reshape((1,IMG_SIZE,IMG_SIZE,1))\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### IV.C. Build and Train N Convolutional Neural Networks"},{"metadata":{"trusted":true},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 2\nmodel = [0] *nets\nif train_model==True:\n    print(\"Training model(s)\")\n    for j in range(nets):\n        model[j] = Sequential()\n\n        model[j].add(Conv2D(IMG_SIZE, kernel_size = 3, activation='relu', input_shape = (IMG_SIZE, IMG_SIZE, 1)))\n        model[j].add(BatchNormalization())\n        model[j].add(Conv2D(IMG_SIZE, kernel_size = 3, activation='relu'))\n        model[j].add(BatchNormalization())\n        model[j].add(Conv2D(IMG_SIZE, kernel_size = 5, strides=2, padding='same', activation='relu'))\n        model[j].add(BatchNormalization())\n        model[j].add(Dropout(0.4))\n\n        model[j].add(Conv2D(int(IMG_SIZE*2), kernel_size = 3, activation='relu'))\n        model[j].add(BatchNormalization())\n        model[j].add(Conv2D(int(IMG_SIZE*2), kernel_size = 3, activation='relu'))\n        model[j].add(BatchNormalization())\n        model[j].add(Conv2D(int(IMG_SIZE*2), kernel_size = 5, strides=2, padding='same', activation='relu'))\n        model[j].add(BatchNormalization())\n        model[j].add(Dropout(0.4))\n\n        model[j].add(Conv2D(int(IMG_SIZE*4), kernel_size = 4, activation='relu'))\n        model[j].add(BatchNormalization())\n        model[j].add(Flatten())\n        model[j].add(Dropout(0.4))\n        # Five categories\n        model[j].add(Dense(5, activation='softmax'))\n\n        # COMPILE WITH ADAM OPTIMIZER AND CROSS ENTROPY COST\n        model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\n        # DECREASE LEARNING RATE EACH EPOCH\n        annealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)\n        # TRAIN NETWORKS\n        history = [0] * nets\n        epochs = 50\n        for j in range(nets):\n            X_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.1)\n            history[j] = model[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64),\n                epochs = epochs, steps_per_epoch = X_train2.shape[0]//64,  \n                validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n            print(\"CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n                j+1,epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))\n\n        # save model and architecture to single file\n        for j in range(nets):\n            model_name = \"model{}.h5\".format(j)\n            print(model_name)\n            model[j].save(model_name)\n        print(\"Saved model(s) to disk\")\nelse:\n    print(\"Loading model(s) from disk\")\n    for j in range(nets):\n        model_name = \"/model{}.h5\".format(j)\n        # load model\n        model[j] = load_model(path_to_preprocessed_data_model + model_name)\n        # summarize model.\n        model[j].summary()\n    print(\"Done.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### IV.D. Predictions on the train set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ENSEMBLE PREDICTIONS AND SUBMIT\nresults = np.zeros( (X_train.shape[0], 5) ) \nfor j in range(nets):\n    results = results + model[j].predict(X_train)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREVIEW PREDICTIONS\nplt.figure(figsize=(15,6))\nfor i in range(10):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_train[i].reshape((IMG_SIZE,IMG_SIZE)),cmap=plt.cm.binary)\n    plt.title(\"{} ; {}\".format(results[i],train[\"label\"][i]),y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see where the model is strong and where it is weak:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#confusion matrix\nlabels = ['0 - No DR', '1 - Mild', '2 - Moderate', '3 - Severe', '4 - Proliferative DR']\ncnf_matrix = confusion_matrix(train[\"label\"].astype('int'), results.astype('int'))\ncnf_matrix_norm = cnf_matrix.astype('float') / cnf_matrix.sum(axis=1)[:, np.newaxis]\ndf_cm = pd.DataFrame(cnf_matrix_norm, index=labels, columns=labels)\nplt.figure(figsize=(16, 7))\nsns.heatmap(df_cm, annot=True, fmt='.2f')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Train Cohen Kappa score: %.3f\" % cohen_kappa_score(results.astype('int'), train['diagnosis'].astype('int'), weights='quadratic'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model performs well in classifying classes 0 to 2, but does a poor job in finding severe to proliferate diabetic retinopathy.\nUsing higher resolution data, working with RGB images instead of grayscale, different data augmentation settings and different CNN configurations are\nways to improve performance:"},{"metadata":{},"cell_type":"markdown","source":"## V. Predictions on the test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"if pre_process_image == True:\n    test = pd.read_csv('test_preprocessed.csv')\nelse:\n    test = pd.read_csv(path_to_preprocessed_data_model+'/test_preprocessed.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREPARE DATA FOR NEURAL NETWORK\nX_id_code = test[\"id_code\"]\nX_test = test.drop(labels = [\"id_code\"],axis = 1)\n# Normalization:\nX_test = X_test / 255.0\nX_test = X_test.values.reshape(-1,IMG_SIZE,IMG_SIZE,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ENSEMBLE PREDICTIONS AND SUBMIT\nresults = np.zeros( (X_test.shape[0], 5) ) \nfor j in range(nets):\n    results = results + model[j].predict(X_test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREVIEW PREDICTIONS\nplt.figure(figsize=(15,6))\nfor i in range(30):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_test[i].reshape((IMG_SIZE,IMG_SIZE)),cmap=plt.cm.binary)\n    plt.title(\"predict=%d\" % results[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = pd.DataFrame(results)\nX_id_code = X_id_code.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_id_code[\"diagnosis\"] = results[\"Label\"]\nX_id_code = X_id_code.drop(labels = [\"index\"],axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"X_id_code.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_id_code.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":1}