{"cells":[{"metadata":{},"cell_type":"markdown","source":"# More Advanced Kaggle Starter for House of Blocks Kaggle In-Class Competition\n\nThis workbook is a starter code for the [Kaggle In-Class House of Blocks Competition](https://www.kaggle.com/c/applications-of-deep-learning-wustl-fall-2020)  This competition is one of the assignments for [T81-558: Applications of Deep Neural Netw1orks](https://sites.wustl.edu/jeffheaton/t81-558/) at [Washington University in St. Louis](https://www.wustl.edu).\n\nThis notebook gets a better score than my previous starter, by using transfer learning.  The notebook also implements basic feature importance. "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nPATH = \"/kaggle/input/applications-of-deep-learning-wustl-fall-2020/final-kaggle-data/\"\nPATH_TRAIN = os.path.join(PATH, \"train.csv\")\nPATH_TEST = os.path.join(PATH, \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we check versions and if the GPU is available. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# What version of Python do you have?\nimport sys\n\nimport tensorflow.keras\nimport pandas as pd\nimport sklearn as sk\nimport tensorflow as tf\n\nprint(f\"Tensor Flow Version: {tf.__version__}\")\nprint(f\"Keras Version: {tensorflow.keras.__version__}\")\nprint()\nprint(f\"Python {sys.version}\")\nprint(f\"Pandas {pd.__version__}\")\nprint(f\"Scikit-Learn {sk.__version__}\")\nprint(\"GPU is\", \"available\" if tf.test.is_gpu_available() \\\n      else \"NOT AVAILABLE\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we prepare to read the training data (that we have labels for) and the test data that we must predict and send to Kaggle."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(PATH_TRAIN)\ndf_test = pd.read_csv(PATH_TEST)\n\ndf_train = df_train[df_train.id != 1300]\n\ndf_train['filename'] = df_train[\"id\"].astype(str)+\".png\"\ndf_train['stable'] = df_train['stable'].astype(str)\n\ndf_test['filename'] = df_test[\"id\"].astype(str)+\".png\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Perform a basic balance plot.  This data is fairly well balanced."},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndf_train.stable.value_counts().plot(kind='bar')\nplt.title('Labels counts')\nplt.xlabel('Stable')\nplt.ylabel('Count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We want to use early stopping.  To do this, we need a validation set.  We will break the data into 80 percent test data and 20 validation.  Do not confuse this validation data with the test set provided by Kaggle.  This validation set is unique to your program and is just used for early stopping."},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_PCT = 0.9\nTRAIN_CUT = int(len(df_train) * TRAIN_PCT)\n\ndf_train_cut = df_train[0:TRAIN_CUT]\ndf_validate_cut = df_train[TRAIN_CUT:]\n\nprint(f\"Training size: {len(df_train_cut)}\")\nprint(f\"Validate size: {len(df_validate_cut)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we create the generators that will provide the images to the neural network as it is trained.  We normalize the images so that the RGB colors between 0-255 become ratios between 0 and 1.  We also use the **flow_from_dataframe** generator to connect the Pandas dataframe to the actual image files. We see here a straightforward implementation; you might also wish to use some of the image transformations provided by the data generator.\n\nThe **HEIGHT** and **WIDTH** constants specify the dimensions that the image will be scaled (or expanded) to. It is probably not a good idea to expand the images."},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport keras_preprocessing\nfrom keras_preprocessing import image\nfrom keras_preprocessing.image import ImageDataGenerator\n\nWIDTH = 224\nHEIGHT = 224\n\ntraining_datagen = ImageDataGenerator(\n  rescale = 1./255,\n  horizontal_flip=True,\n  #vertical_flip=True,\n  fill_mode='nearest')\n\ntrain_generator = training_datagen.flow_from_dataframe(\n        dataframe=df_train_cut,\n        directory=PATH,\n        x_col=\"filename\",\n        y_col=\"stable\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=32,\n        class_mode='categorical')\n\nvalidation_datagen = ImageDataGenerator(rescale = 1./255)\n\nval_generator = validation_datagen.flow_from_dataframe(\n        dataframe=df_validate_cut,\n        directory=PATH,\n        x_col=\"filename\",\n        y_col=\"stable\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=32,\n        class_mode='categorical')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now create the neural network and fit it.  Some essential concepts are going on here.\n\n* **Batch Size** - The number of training samples that should be evaluated per training step.  Smaller batch sizes, or mini-batches, are generally preferred.\n* **Step** - A training step is one complete run over the batch.  At the end of a step, the weights are updated, and the neural network learns.\n* **Epoch** - An arbitrary point at which to measure results or checkpoint the model.  Generally, an epoch is one complete pass over the training set.  However, when generators are used, the training set size is theoretically infinite. Because of this, we set a **steps_per_epoch** parameter.\n* **validation steps** - The validation set may also be infinite; because of this, we must specify how many steps we wish to validate at the end of each Epoch."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\nfrom tensorflow.keras.applications import MobileNet\nfrom tensorflow.keras.applications.mobilenet import preprocess_input\nfrom tensorflow.keras.layers import Dense,GlobalAveragePooling2D\nfrom tensorflow.keras.models import Model\n\nbase_model=MobileNet(weights='imagenet',include_top=False) \n\nx=base_model.output\nx=GlobalAveragePooling2D()(x)\n#x=Dense(1024,activation='relu')(x) \n#x=Dense(1024,activation='relu')(x) \npreds=Dense(2,activation='softmax')(x)\n\nmodel=Model(inputs=base_model.input,outputs=preds)\n\nfor layer in model.layers[:20]:\n    layer.trainable=False\nfor layer in model.layers[20:]:\n    layer.trainable=True\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_steps = len(df_validate_cut)\nmodel.compile(loss = 'categorical_crossentropy', optimizer='adam')\nmonitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',\n        restore_best_weights=True)\n\nhistory = model.fit(train_generator, epochs=25, steps_per_epoch=250, \n                    validation_data = val_generator, \n                    verbose = 1, validation_steps=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Importance\n\nWe will use the [Grad-CAM algorithm](https://keras.io/examples/vision/grad_cam/) to determine feature importance for a few images."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow import keras\n\n# Display\nfrom IPython.display import Image\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\n\ndef get_img_array(img_path, size):\n    # `img` is a PIL image of size 299x299\n    img = keras.preprocessing.image.load_img(img_path, target_size=size)\n    # `array` is a float32 Numpy array of shape (299, 299, 3)\n    array = keras.preprocessing.image.img_to_array(img)\n    # We add a dimension to transform our array into a \"batch\"\n    # of size (1, 299, 299, 3)\n    array = np.expand_dims(array, axis=0)\n    return array\n\n\ndef make_gradcam_heatmap(\n    img_array, model, last_conv_layer_name, classifier_layer_names\n):\n    # First, we create a model that maps the input image to the activations\n    # of the last conv layer\n    last_conv_layer = model.get_layer(last_conv_layer_name)\n    last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)\n\n    # Second, we create a model that maps the activations of the last conv\n    # layer to the final class predictions\n    classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])\n    x = classifier_input\n    for layer_name in classifier_layer_names:\n        x = model.get_layer(layer_name)(x)\n    classifier_model = keras.Model(classifier_input, x)\n\n    # Then, we compute the gradient of the top predicted class for our input image\n    # with respect to the activations of the last conv layer\n    with tf.GradientTape() as tape:\n        # Compute activations of the last conv layer and make the tape watch it\n        last_conv_layer_output = last_conv_layer_model(img_array)\n        tape.watch(last_conv_layer_output)\n        # Compute class predictions\n        preds = classifier_model(last_conv_layer_output)\n        top_pred_index = tf.argmax(preds[0])\n        top_class_channel = preds[:, top_pred_index]\n\n    # This is the gradient of the top predicted class with regard to\n    # the output feature map of the last conv layer\n    grads = tape.gradient(top_class_channel, last_conv_layer_output)\n\n    # This is a vector where each entry is the mean intensity of the gradient\n    # over a specific feature map channel\n    pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n\n    # We multiply each channel in the feature map array\n    # by \"how important this channel is\" with regard to the top predicted class\n    last_conv_layer_output = last_conv_layer_output.numpy()[0]\n    pooled_grads = pooled_grads.numpy()\n    for i in range(pooled_grads.shape[-1]):\n        last_conv_layer_output[:, :, i] *= pooled_grads[i]\n\n    # The channel-wise mean of the resulting feature map\n    # is our heatmap of class activation\n    heatmap = np.mean(last_conv_layer_output, axis=-1)\n\n    # For visualization purpose, we will also normalize the heatmap between 0 & 1\n    heatmap = np.maximum(heatmap, 0) / np.max(heatmap)\n    return heatmap","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Activation for last CONV layer.\nlast_conv_layer_name = \"conv_pw_13_relu\"\n\n\nclassifier_layer_names = [\n    \"global_average_pooling2d\",\n    \"dense\",\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_grad_cam(filename):\n    img_path = os.path.join(PATH, filename)\n    img_array = get_img_array(img_path, size=(HEIGHT, WIDTH))\n    img_array /= 255.0\n    preds = model.predict(img_array)\n    print(f\"Prediction: {preds}\")\n    #img = Image(img_path)\n    #display(img)\n    heatmap = make_gradcam_heatmap(\n        img_array, model, last_conv_layer_name, classifier_layer_names\n    )\n    plt.matshow(heatmap)\n    plt.show()\n\n    # We load the original image\n    img = keras.preprocessing.image.load_img(img_path)\n    img = keras.preprocessing.image.img_to_array(img)\n\n    # We rescale heatmap to a range 0-255\n    heatmap = np.uint8(255 * heatmap)\n\n    # We use jet colormap to colorize heatmap\n    jet = cm.get_cmap(\"jet\")\n\n    # We use RGB values of the colormap\n    jet_colors = jet(np.arange(256))[:, :3]\n    jet_heatmap = jet_colors[heatmap]\n\n    # We create an image with RGB colorized heatmap\n    jet_heatmap = keras.preprocessing.image.array_to_img(jet_heatmap)\n    jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))\n    jet_heatmap = keras.preprocessing.image.img_to_array(jet_heatmap)\n\n    # Superimpose the heatmap on original image\n    superimposed_img = jet_heatmap * 0.4 + img\n    superimposed_img = keras.preprocessing.image.array_to_img(superimposed_img)\n\n    # Display Grad CAM\n    display(superimposed_img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Unstable\nprocess_grad_cam(\"8.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Stable\nprocess_grad_cam(\"1.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Submission\n\nNow that the neural network is trained; we need to generate a submit CSV file to send to Kaggle.  We will use nearly the same technique to build the submit file.  However, these essential points that we must address:\n\n* We do not want the data generator to create an infinite date like we did when training.  We have a fixed number of cases to score for the Kaggle submit; we only want to process them.\n* We do not want the data generator to randomize the samples' order like it did when training. Therefore we set shuffle to false.\n* We want to always start at the beginning of the data, so we reset the generator.\n\nThese ensure that the predictions align with the id's."},{"metadata":{"trusted":true},"cell_type":"code","source":"submit_datagen = ImageDataGenerator(rescale = 1./255)\n\nsubmit_generator = submit_datagen.flow_from_dataframe(\n        dataframe=df_test,\n        directory=PATH,\n        x_col=\"filename\",\n        batch_size = 1,\n        shuffle = False,\n        target_size=(HEIGHT, WIDTH),\n        class_mode=None)\n\nsubmit_generator.reset()\npred = model.predict(submit_generator,steps=len(df_test))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submit = pd.DataFrame({\"id\":df_test['id'],'stable':pred[:,1].flatten()})\ndf_submit.to_csv(\"/kaggle/working/submit.csv\",index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submit","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}