{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{},"cell_type":"markdown","source":"The ResNet family of models was introduced by Kaiming He and al. and was the winner of ILSVCR competition in 2015. The idea of ResNet is use of plenty layers with few parameters. The original trick of the model is introduction of residual unit and skip connection. \n\nThere is several options of ResNet models with different number of layers. The most popular architectures are ResNet18, ResNet34, ResNet50 and ResNet152. \n\nThis notebook was created using kindly some parts of code from Kaushal Shah's Bengali Graphemes: Multi Output ResNet-50. Thank you Kaushual ! The model architecture was changed in order to create ResNet18 model, which is less deep and therefore more light and more fast than ResNet50. "},{"metadata":{},"cell_type":"markdown","source":"# Load libraries "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom tqdm.auto import tqdm\nfrom glob import glob\nimport time, gc\nimport cv2\n\nfrom tensorflow import keras\nfrom keras import layers\nfrom keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D\nfrom keras.models import Model, load_model\nfrom keras.preprocessing import image\nfrom keras.utils import layer_utils\nfrom keras.utils.data_utils import get_file\nfrom keras.applications.imagenet_utils import preprocess_input\nfrom keras.callbacks import ReduceLROnPlateau\nimport pydot\nfrom IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\nfrom keras.utils import plot_model\nfrom keras.initializers import glorot_uniform\nimport scipy.misc\nfrom matplotlib.pyplot import imshow\nfrom matplotlib import pyplot as plt\nfrom sklearn.model_selection import train_test_split\n%matplotlib inline\n\nimport keras.backend as K\nK.set_image_data_format('channels_last')\nK.set_learning_phase(1)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Load the data\ntrain_df_ = pd.read_csv('/kaggle/input/bengaliai-cv19/train.csv')\ntest_df_ = pd.read_csv('/kaggle/input/bengaliai-cv19/test.csv')\nclass_map_df = pd.read_csv('/kaggle/input/bengaliai-cv19/class_map.csv')\nsample_sub_df = pd.read_csv('/kaggle/input/bengaliai-cv19/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Explore the size of loaded DataFrames\nprint(f'Size of training data: {train_df_.shape}')\nprint(f'Size of test data: {test_df_.shape}')\nprint(f'Size of class map: {class_map_df.shape}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create helpful functions for data processing\ndef resize(df, size=64, need_progress_bar=True):\n    \"\"\"Function which resizes the images to 64x64 pixels\n    \n    ARGS :\n    - df : Data frame containing images' pixels values\n    - size : size of target image (64 pixels by default)\n    - need_progress_bar : display progress bar (True by default)\n    \n    OUTPUT:\n    - dataframe of resized images\n    \n    Source kernel: Bengali Graphemes_Multi Output ResNet-50  \n    \"\"\"\n    resized = {}\n    if need_progress_bar:\n        for i in tqdm(range(df.shape[0])):\n            image = cv2.resize(df.loc[df.index[i]].values.reshape(137,236),(size,size))\n            resized[df.index[i]] = image.reshape(-1)\n    else:\n        for i in range(df.shape[0]):\n            image = cv2.resize(df.loc[df.index[i]].values.reshape(137,236),(size,size))\n            resized[df.index[i]] = image.reshape(-1)\n    resized = pd.DataFrame(resized).T\n    return resized\n\ndef get_dummies(df):\n    \"\"\"     \n    Source kernel: Bengali Graphemes_Multi Output ResNet-50 \"\"\"\n    cols = []\n    for col in df:\n        cols.append(pd.get_dummies(df[col].astype(str)))\n    return pd.concat(cols, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Delete the 'grapheme' column which is not useful for further modeling. Change the type of features to uint8.\ntrain_df_ = train_df_.drop(['grapheme'], axis=1, inplace=False)\ntrain_df_[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']] = train_df_[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']].astype('uint8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ResNet's Identity Block\nHere is the visualization identity block:\n\n![](https://raw.githubusercontent.com/Kulbear/deep-learning-coursera/997fdb2e2db67acd45d29ae418212463a54be06d/Convolutional%20Neural%20Networks/images/idblock3_kiank.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def identity_block(X, f, filters, stage, block):\n    \"\"\"\n    ResNet Identity block\n\n    Arguments:\n    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)\n    f -- integer, specifying the shape the CONV's window for the main path\n    filters -- an integer defining the number of filters in the CONV layers of the main path\n    stage -- integer, used to name the layers, depending on their position in the network\n    block -- string/character, used to name the layers, depending on their position in the network\n\n    Returns:\n    X -- output of the identity block, tensor of shape (n_H, n_W, n_C)\n    \n    Source kernel : Bengali Graphemes_ Multi Output ResNet-50. \n    The block was modified to correspond to ResNet18 Architecture.\n    \"\"\"\n\n    # defining name basis\n    conv_name_base = 'res' + str(stage) + block + '_branch'\n    bn_name_base = 'bn' + str(stage) + block + '_branch'\n\n    # Retrieve Filter \n       \n    F = filters\n\n    # Save the input value. You'll need this later to add back to the main path. \n    X_shortcut = X\n\n    # Component of main path\n    X = Conv2D(filters=F, kernel_size=(f, f), strides=(1, 1), padding='same', name=conv_name_base + '2b', kernel_initializer=glorot_uniform(seed=0))(X)\n    X = BatchNormalization(axis=3, name=bn_name_base + '2b')(X)\n    X = Activation('relu')(X)\n\n    # Add shortcut value to main path, and pass it through a RELU activation\n    X = Add()([X, X_shortcut])\n    X = Activation('relu')(X)\n\n    return X","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ResNet's Convolutional Block\nThis is similar to identity block but used when the input and output dimensions don't match up. The difference with the identity block is that there is a CONV2D layer in the shortcut path.\n\nHere is visualization of ResNet's convolution Block: \n\n![](https://raw.githubusercontent.com/Kulbear/deep-learning-coursera/997fdb2e2db67acd45d29ae418212463a54be06d/Convolutional%20Neural%20Networks/images/convblock_kiank.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def convolutional_block(X, f, filters, stage, block, s=2):\n    \"\"\"\n    Implementation of the convolutional block\n\n    Arguments:\n    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)\n    f -- an integer specifying the shape of the middle CONV's window for the main path\n    filters -- an integer defining the number of filters in the CONV layers of the main path\n    stage -- integer, used to name the layers, depending on their position in the network\n    block -- string/character, used to name the layers, depending on their position in the network\n    s -- Integer, specifying the stride to be used\n\n    Returns:\n    X -- output of the convolutional block, tensor of shape (n_H, n_W, n_C)\n    \n    Source kernel : Bengali Graphemes_ Multi Output ResNet-50. \n    The block was modified to correspond to ResNet18 Architecture.\n    \"\"\"\n\n    # defining name basis\n    conv_name_base = 'res' + str(stage) + block + '_branch'\n    bn_name_base = 'bn' + str(stage) + block + '_branch'\n\n    # Retrieve Filters     \n    F = filters\n\n    # Save the input value\n    X_shortcut = X\n\n    # Second component of main path\n    X = Conv2D(filters=F, kernel_size=(f, f), strides=(s, s), padding='same', name=conv_name_base + '2b', kernel_initializer=glorot_uniform(seed=0))(X)\n    X = BatchNormalization(axis=3, name=bn_name_base + '2b')(X)\n    X = Activation('relu')(X)\n\n    ##### SHORTCUT PATH #### \n    X_shortcut = Conv2D(filters=F, kernel_size=(1, 1), strides=(s, s), padding='valid', name=conv_name_base + '1', kernel_initializer=glorot_uniform(seed=0))(X_shortcut)\n    X_shortcut = BatchNormalization(axis=3, name=bn_name_base + '1')(X_shortcut)\n\n    # Final step: Add shortcut value to main path, and pass it through a RELU activation\n    X = Add()([X, X_shortcut])\n    X = Activation('relu')(X)\n\n    return X","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building ResNet-18 model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def ResNet18(input_shape=(64, 64, 1)):\n    \"\"\"\n    Implementation of the popular ResNet50 the following architecture:\n    CONV2D -> BATCHNORM -> RELU -> MAXPOOL -> CONVBLOCK1 -> IDBLOCK1 -> CONVBLOCK2 -> IDBLOCK2\n    -> CONVBLOCK3 -> IDBLOCK3 -> CONVBLOCK4 -> IDBLOCK4 -> AVGPOOL -> TOPLAYERS\n\n    Arguments:\n    input_shape -- shape of the images of the dataset\n    classes -- integer, number of classes\n\n    Returns:\n    model -- a Model() instance in Keras\n    \n    Source kernel : Bengali Graphemes_ Multi Output ResNet-50. \n    The model was modified to correspond to ResNet18 Architecture.\n    \"\"\"\n\n    # Define the input as a tensor with shape input_shape\n    X_input = Input(input_shape)\n\n    # Zero-Padding\n    X = ZeroPadding2D((3, 3))(X_input)\n\n    # Stage 1\n    X = Conv2D(64, (7, 7), strides=(2, 2), name='conv1', kernel_initializer=glorot_uniform(seed=0))(X)\n    X = BatchNormalization(axis=3, name='bn_conv1')(X)\n    X = Activation('relu')(X)\n    X = MaxPooling2D((3, 3), strides=(2, 2))(X)\n    \n    # Stage 2\n    X = convolutional_block(X, f=3, filters=64, stage=2, block='a', s=1)\n    X = identity_block(X, 3, 64, stage=2, block='b')\n\n    # Stage 3\n    X = convolutional_block(X, f=3, filters=128, stage=3, block='a', s=2)\n    X = identity_block(X, 3, 128, stage=3, block='b')\n\n    # Stage 4\n    X = convolutional_block(X, f=3, filters=256, stage=4, block='a', s=2)\n    X = identity_block(X, 3, 256, stage=4, block='b')\n\n    # Stage 5\n    X = X = convolutional_block(X, f=3, filters=512, stage=5, block='a', s=2)\n    X = identity_block(X, 3, 512, stage=5, block='b')\n\n    # AVGPOOL\n    X = AveragePooling2D(pool_size=(2, 2), padding='same')(X)\n\n    # output layers\n    X = Flatten()(X)\n    head_root = Dense(168, activation = 'softmax', kernel_initializer=glorot_uniform(seed=0))(X)\n    head_vowel = Dense(11, activation = 'softmax', kernel_initializer=glorot_uniform(seed=0))(X)\n    head_consonant = Dense(7, activation = 'softmax', kernel_initializer=glorot_uniform(seed=0))(X)\n\n    # Create model\n    model = Model(inputs=X_input, outputs=[head_root, head_vowel, head_consonant], name='ResNet18')\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = ResNet18(input_shape=(64, 64, 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the summary of our models"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Setting training parameters and callbacks"},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE=64\nN_CHANNELS=1\nbatch_size = 512\nepochs = 23","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set a learning rate annealer. Learning rate will be half after 3 epochs if accuracy is not increased\nlearning_rate_reduction_root = ReduceLROnPlateau(monitor='dense_1_accuracy', \n                                            patience=3, \n                                            verbose=1,\n                                            factor=0.5, \n                                            min_lr=0.00001)\nlearning_rate_reduction_vowel = ReduceLROnPlateau(monitor='dense_2_accuracy', \n                                            patience=3, \n                                            verbose=1,\n                                            factor=0.5, \n                                            min_lr=0.00001)\nlearning_rate_reduction_consonant = ReduceLROnPlateau(monitor='dense_3_accuracy', \n                                            patience=3, \n                                            verbose=1,\n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Stop the training if the global loss function stops decreasing (no progress in 10 epochs)\nearly_stopping_cb = keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, \n                                                              restore_best_weights=True, mode=\"min\")  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating multioutput data generator"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MultiOutputDataGenerator(keras.preprocessing.image.ImageDataGenerator):\n\n    def flow(self,\n             x,\n             y=None,\n             batch_size=32,\n             shuffle=True,\n             sample_weight=None,\n             seed=None,\n             save_to_dir=None,\n             save_prefix='',\n             save_format='png',\n             subset=None):\n\n        targets = None\n        target_lengths = {}\n        ordered_outputs = []\n        for output, target in y.items():\n            if targets is None:\n                targets = target\n            else:\n                targets = np.concatenate((targets, target), axis=1)\n            target_lengths[output] = target.shape[1]\n            ordered_outputs.append(output)\n\n\n        for flowx, flowy in super().flow(x, targets, batch_size=batch_size,\n                                         shuffle=shuffle):\n            target_dict = {}\n            i = 0\n            for output in ordered_outputs:\n                target_length = target_lengths[output]\n                target_dict[output] = flowy[:, i: i + target_length]\n                i += target_length\n\n            yield flowx, target_dict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"histories = []\nfor i in range(4):\n    train_df = pd.merge(pd.read_parquet(f'/kaggle/input/bengaliai-cv19/train_image_data_{i}.parquet'), train_df_, on='image_id').drop(['image_id'], axis=1)\n    \n    # Visualize few samples of current training dataset\n    fig, ax = plt.subplots(nrows=3, ncols=4, figsize=(16, 8))\n    count=0\n    for row in ax:\n        for col in row:\n            col.imshow(resize(train_df.drop(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'], axis=1).iloc[[count]], need_progress_bar=False).values.reshape(64, 64))\n            count += 1\n    plt.show()\n    \n    X_train = train_df.drop(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'], axis=1)\n    X_train = resize(X_train)/255\n    \n    # CNN takes images in shape `(batch_size, h, w, channels)`, so reshape the images\n    X_train = X_train.values.reshape(-1, IMG_SIZE, IMG_SIZE, N_CHANNELS)\n    \n    Y_train_root = pd.get_dummies(train_df['grapheme_root']).values\n    Y_train_vowel = pd.get_dummies(train_df['vowel_diacritic']).values\n    Y_train_consonant = pd.get_dummies(train_df['consonant_diacritic']).values\n\n    print(f'Training images: {X_train.shape}')\n    print(f'Training labels root: {Y_train_root.shape}')\n    print(f'Training labels vowel: {Y_train_vowel.shape}')\n    print(f'Training labels consonants: {Y_train_consonant.shape}')\n\n    # Divide the data into training and validation set\n    x_train, x_test, y_train_root, y_test_root, y_train_vowel, y_test_vowel, y_train_consonant, y_test_consonant = train_test_split(X_train, Y_train_root, Y_train_vowel, Y_train_consonant, test_size=0.08, random_state=666)\n    del train_df\n    del X_train\n    del Y_train_root, Y_train_vowel, Y_train_consonant\n\n    # Data augmentation for creating more training data\n    datagen = MultiOutputDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=8,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.15, # Randomly zoom image \n        width_shift_range=0.15,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.15,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\n    # This will just calculate parameters required to augment the given data. This won't perform any augmentations\n    datagen.fit(x_train)\n\n    # Fit the model\n    history = model.fit_generator(datagen.flow(x_train, {'dense_1': y_train_root, 'dense_2': y_train_vowel, 'dense_3': y_train_consonant}, batch_size=batch_size),\n                              epochs = epochs, validation_data = (x_test, [y_test_root, y_test_vowel, y_test_consonant]), \n                              steps_per_epoch=x_train.shape[0] // batch_size, \n                              callbacks=[learning_rate_reduction_root, learning_rate_reduction_vowel, learning_rate_reduction_consonant, early_stopping_cb])\n\n    histories.append(history)\n    \n    # Delete to reduce memory usage\n    del x_train\n    del x_test\n    del y_train_root\n    del y_test_root\n    del y_train_vowel\n    del y_test_vowel\n    del y_train_consonant\n    del y_test_consonant\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\ndef plot_loss(his, title):\n    \"\"\"Function which plots the history of training, in this case the evolution of training and validation loss function.\n     \n    ARGS : \n    - his : keras history object\n    - title : str with title of each plot\n    \n    OUT :\n    - plot of training curve    \n    \"\"\"\n\n    plt.style.use('ggplot')\n    plt.figure()\n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['loss'], label='train_loss')\n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['dense_1_loss'], label='train_root_loss')\n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['dense_2_loss'], label='train_vowel_loss')\n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['dense_3_loss'], label='train_consonant_loss')\n    \n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['val_dense_1_loss'], label='val_train_root_loss')\n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['val_dense_2_loss'], label='val_train_vowel_loss')\n    plt.plot(np.arange(0, len(his.history['loss'])), his.history['val_dense_3_loss'], label='val_train_consonant_loss')\n    \n    plt.title(title)\n    plt.xlabel('Epoch #/' + str(len(his.history['loss'])))\n    plt.ylabel('Loss')\n    plt.legend(loc='upper right')\n    plt.show()\n\ndef plot_acc(his, title):\n    \"\"\"Function which plots the history of training, in this case the evolution of training and validation accuracy.\n     \n    ARGS : \n    - his : keras history object\n    - title : str with title of each plot\n\n    OUT :\n    - plot of training curve    \n    \"\"\"\n    plt.style.use('ggplot')\n    plt.figure()\n    plt.plot(np.arange(0, len(his.history['dense_1_accuracy'])), his.history['dense_1_accuracy'], label='train_root_acc')\n    plt.plot(np.arange(0, len(his.history['dense_1_accuracy'])), his.history['dense_2_accuracy'], label='train_vowel_acc')\n    plt.plot(np.arange(0, len(his.history['dense_1_accuracy'])), his.history['dense_3_accuracy'], label='train_consonant_acc')\n    \n    plt.plot(np.arange(0, len(his.history['dense_1_accuracy'])), his.history['val_dense_1_accuracy'], label='val_root_acc')\n    plt.plot(np.arange(0, len(his.history['dense_1_accuracy'])), his.history['val_dense_2_accuracy'], label='val_vowel_acc')\n    plt.plot(np.arange(0, len(his.history['dense_1_accuracy'])), his.history['val_dense_3_accuracy'], label='val_consonant_acc')\n    plt.title(title)\n    plt.xlabel('Epoch # /' + str(len(his.history['dense_1_accuracy'])))\n    plt.ylabel('Accuracy')\n    plt.legend(loc='upper right')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the learning curves for 4 datasets\nfor dataset in range(4):\n    plot_loss(histories[dataset], f'Training Dataset: {dataset}')\n    plot_acc(histories[dataset], f'Training Dataset: {dataset}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Delete histories to clean the memory\ndel histories\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create dictionnary of predictions\npreds_dict = {\n    'grapheme_root': [],\n    'vowel_diacritic': [],\n    'consonant_diacritic': []\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generate the submission .csv file\ncomponents = ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']\ntarget=[] # model predictions placeholder\nrow_id=[] # row_id place holder\nfor i in range(4):\n    df_test_img = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i)) \n    df_test_img.set_index('image_id', inplace=True)\n\n    X_test = resize(df_test_img, need_progress_bar=False)/255\n    X_test = X_test.values.reshape(-1, IMG_SIZE, IMG_SIZE, N_CHANNELS)\n    \n    preds = model.predict(X_test)\n\n    for i, p in enumerate(preds_dict):\n        preds_dict[p] = np.argmax(preds[i], axis=1)\n\n    for k,id in enumerate(df_test_img.index.values):  \n        for i,comp in enumerate(components):\n            id_sample=id+'_'+comp\n            row_id.append(id_sample)\n            target.append(preds_dict[comp][k])\n    del df_test_img\n    del X_test\n    gc.collect()\n\ndf_sample = pd.DataFrame(\n    {\n        'row_id': row_id,\n        'target':target\n    },\n    columns = ['row_id','target'] \n)\ndf_sample.to_csv('submission.csv',index=False)\ndf_sample.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}