{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Kaggle Starter Code for House of Blocks Kaggle In-Class Competition\n\nThis workbook is a starter code for the [Kaggle In-Class House of Blocks Competition](https://www.kaggle.com/c/applications-of-deep-learning-wustl-fall-2020)  This competition is one of the assignments for [T81-558: Applications of Deep Neural Netw1orks](https://sites.wustl.edu/jeffheaton/t81-558/) at [Washington University in St. Louis](https://www.wustl.edu).\n\nThis notebook is not a particularly high-scoring model; however, it does demonstrate how to begin the project entirely in Kaggle.  It is also possable to run this project from Google CoLab.  I have a separate starter project for CoLab.  "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nPATH = \"/kaggle/input/applications-of-deep-learning-wustl-fall-2020/final-kaggle-data/\"\nPATH_TRAIN = os.path.join(PATH, \"train.csv\")\nPATH_TEST = os.path.join(PATH, \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we check versions and if the GPU is available. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# What version of Python do you have?\nimport sys\n\nimport tensorflow.keras\nimport pandas as pd\nimport sklearn as sk\nimport tensorflow as tf\n\nprint(f\"Tensor Flow Version: {tf.__version__}\")\nprint(f\"Keras Version: {tensorflow.keras.__version__}\")\nprint()\nprint(f\"Python {sys.version}\")\nprint(f\"Pandas {pd.__version__}\")\nprint(f\"Scikit-Learn {sk.__version__}\")\nprint(\"GPU is\", \"available\" if tf.test.is_gpu_available() \\\n      else \"NOT AVAILABLE\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we prepare to read the training data (that we have labels for) and the test data that we must predict and send to Kaggle."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(PATH_TRAIN)\ndf_test = pd.read_csv(PATH_TEST)\n\ndf_train = df_train[df_train.id != 1300]\n\ndf_train['filename'] = df_train[\"id\"].astype(str)+\".png\"\ndf_train['stable'] = df_train['stable'].astype(str)\n\ndf_test['filename'] = df_test[\"id\"].astype(str)+\".png\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Perform a basic balance plot.  This data is fairly well balanced."},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndf_train.stable.value_counts().plot(kind='bar')\nplt.title('Labels counts')\nplt.xlabel('Stable')\nplt.ylabel('Count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We want to use early stopping.  To do this, we need a validation set.  We will break the data into 80 percent test data and 20 validation.  Do not confuse this validation data with the test set provided by Kaggle.  This validation set is unique to your program and is just used for early stopping."},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_PCT = 0.9\nTRAIN_CUT = int(len(df_train) * TRAIN_PCT)\n\ndf_train_cut = df_train[0:TRAIN_CUT]\ndf_validate_cut = df_train[TRAIN_CUT:]\n\nprint(f\"Training size: {len(df_train_cut)}\")\nprint(f\"Validate size: {len(df_validate_cut)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we create the generators that will provide the images to the neural network as it is trained.  We normalize the images so that the RGB colors between 0-255 become ratios between 0 and 1.  We also use the **flow_from_dataframe** generator to connect the Pandas dataframe to the actual image files. We see here a straightforward implementation; you might also wish to use some of the image transformations provided by the data generator.\n\nThe **HEIGHT** and **WIDTH** constants specify the dimensions that the image will be scaled (or expanded) to. It is probably not a good idea to expand the images."},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport keras_preprocessing\nfrom keras_preprocessing import image\nfrom keras_preprocessing.image import ImageDataGenerator\n\nWIDTH = 150\nHEIGHT = 150\n\ntraining_datagen = ImageDataGenerator(\n  rescale = 1./255,\n  #horizontal_flip=True,\n  #vertical_flip=True,\n  fill_mode='nearest')\n\ntrain_generator = training_datagen.flow_from_dataframe(\n        dataframe=df_train_cut,\n        directory=PATH,\n        x_col=\"filename\",\n        y_col=\"stable\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=126,\n        class_mode='binary')\n\nvalidation_datagen = ImageDataGenerator(rescale = 1./255)\n\nval_generator = validation_datagen.flow_from_dataframe(\n        dataframe=df_validate_cut,\n        directory=PATH,\n        x_col=\"filename\",\n        y_col=\"stable\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=126,\n        class_mode='binary')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now create the neural network and fit it.  Some essential concepts are going on here.\n\n* **Batch Size** - The number of training samples that should be evaluated per training step.  Smaller batch sizes, or mini-batches, are generally preferred.\n* **Step** - A training step is one complete run over the batch.  At the end of a step, the weights are updated, and the neural network learns.\n* **Epoch** - An arbitrary point at which to measure results or checkpoint the model.  Generally, an epoch is one complete pass over the training set.  However, when generators are used, the training set size is theoretically infinite. Because of this, we set a **steps_per_epoch** parameter.\n* **validation steps** - The validation set may also be infinite; because of this, we must specify how many steps we wish to validate at the end of each Epoch."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nmodel = tf.keras.models.Sequential([\n    # Note the input shape is the desired size of the image 300x300 with 3 bytes color\n    # This is the first convolution\n    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(HEIGHT, WIDTH, 3)),\n    tf.keras.layers.MaxPooling2D(2, 2),\n    # The second convolution\n    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.MaxPooling2D(2,2),\n    # The third convolution\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n    tf.keras.layers.Dropout(0.5),\n    tf.keras.layers.MaxPooling2D(2,2),\n    # The fourth convolution\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    # The fifth convolution\n    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2,2),\n    # Flatten the results to feed into a DNN\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dropout(0.5),\n    # 512 neuron hidden layer\n    tf.keras.layers.Dense(512, activation='relu'),\n    # Only 1 output neuron. It will contain a value from 0-1 where 0 for 1 class ('horses') and 1 for the other ('humans')\n    tf.keras.layers.Dense(1, activation='sigmoid')\n])\n\nmodel.summary()\nepoch_steps = 250 # needed for 2.2\nvalidation_steps = len(df_validate_cut)\nmodel.compile(loss = 'binary_crossentropy', optimizer='adam')\nmonitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',\n        restore_best_weights=True)\n\nhistory = model.fit(train_generator, epochs=25, steps_per_epoch=20, \n                    validation_data = val_generator, \n                    verbose = 1, validation_steps=3)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Submission\n\nNow that the neural network is trained; we need to generate a submit CSV file to send to Kaggle.  We will use nearly the same technique to build the submit file.  However, these essential points that we must address:\n\n* We do not want the data generator to create an infinite date like we did when training.  We have a fixed number of cases to score for the Kaggle submit; we only want to process them.\n* We do not want the data generator to randomize the samples' order like it did when training. Therefore we set shuffle to false.\n* We want to always start at the beginning of the data, so we reset the generator.\n\nThese ensure that the predictions align with the id's."},{"metadata":{"trusted":true},"cell_type":"code","source":"submit_datagen = ImageDataGenerator(rescale = 1./255)\n\nsubmit_generator = submit_datagen.flow_from_dataframe(\n        dataframe=df_test,\n        directory=PATH,\n        x_col=\"filename\",\n        batch_size = 1,\n        shuffle = False,\n        target_size=(HEIGHT, WIDTH),\n        class_mode=None)\n\nsubmit_generator.reset()\npred = model.predict(submit_generator,steps=len(df_test))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submit = pd.DataFrame({\"id\":df_test['id'],'stable':pred.flatten()})\ndf_submit.to_csv(\"/kaggle/working/submit.csv\",index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}