{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b19d00d4-df9a-1ecf-a928-a8b28d401c4a"
      },
      "source": [
        "# First Steps with Keras Convolutional Neural Networks - Nature Conservancy Image Recognition Kaggle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c70d9297-bba2-3f19-f59d-abf448e80a1c"
      },
      "source": [
        "**Keras CNN that will yield 95% accuracy on its training data, if you use the full data set, 25+ epochs.** (Here we use a subset, and only a few epochs, for sake of speed.) More training epochs and more + better data --> more accuracy.\n",
        "\n",
        "This CNN was created using public tutorials, but updated to work on the data set for the current project."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c054968d-f3c9-5467-b60c-c7c387e983e6"
      },
      "source": [
        " **For sources, see:**\n",
        "\n",
        " - [http://machinelearningmastery.com/object-recognition-convolutional-neural-networks-keras-deep-learning-library/][1]\n",
        " - [https://medium.com/@waleedka/traffic-sign-recognition-with-tensorflow-629dffc391a6#.v471kaepx][2]\n",
        "\n",
        "\n",
        "  [1]: http://machinelearningmastery.com/object-recognition-convolutional-neural-networks-keras-deep-learning-library/\n",
        "  [2]: https://medium.com/@waleedka/traffic-sign-recognition-with-tensorflow-629dffc391a6#.v471kaepx"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5fba1249-3064-1465-1632-314226a078bb"
      },
      "source": [
        "## Goals\n",
        "This Notebook is posted to give new users a start on using Keras in this competition. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c4e49ed3-71d4-2422-5441-524f3d891e50"
      },
      "source": [
        "## Results\n",
        "This CNN will predict output with high level of accuracy, BUT: all outputs for predictions will be either 1.0 or 0.0\n",
        "\n",
        "Because of the way this Kaggle is scored, the incorrect data points will lead to a large loss. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7ec8f74f-4a44-a763-ff46-e045c74d2cb2"
      },
      "source": [
        "## Next steps\n",
        "Need to find a network model/architecture that will distribute its predictions over the full set of classes, not simply a 0/1 binary prediction. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fccff7ab-cfb6-5f18-d89d-0c5e73b9f0e2"
      },
      "outputs": [],
      "source": [
        "10# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "# For example, here's several helpful packages to load in \n",
        "\n",
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "\n",
        "# imports needed for CNN\n",
        "import csv\n",
        "import cv2\n",
        "import os, glob\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import sklearn\n",
        "from sklearn.model_selection import train_test_split\n",
        "import time\n",
        "from keras.datasets import cifar10\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Dropout\n",
        "from keras.layers import Flatten\n",
        "from keras.constraints import maxnorm\n",
        "from keras.optimizers import SGD\n",
        "from keras.layers.convolutional import Convolution2D\n",
        "from keras.layers.convolutional import MaxPooling2D\n",
        "from keras.layers.convolutional import ZeroPadding2D\n",
        "from keras.utils import np_utils\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "# Input data files are available in the \"../input/\" directory.\n",
        "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
        "\n",
        "from subprocess import check_output\n",
        "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n",
        "\n",
        "# Load the data\n",
        "def load_data(data_dir):\n",
        "    \"\"\"\n",
        "    From: https://medium.com/@waleedka/traffic-sign-recognition-with-tensorflow-629dffc391a6#.v471kaepx\n",
        "    \"\"\"\n",
        "    # Get all subdirectories of data_dir. Each represents a label.\n",
        "    directories = [d for d in os.listdir(data_dir)\n",
        "                   if os.path.isdir(os.path.join(data_dir, d))]\n",
        "    # Loop through the label directories and collect the data in\n",
        "    # two lists, labels and images.\n",
        "    labels = []\n",
        "    images = []\n",
        "\n",
        "    category = 0\n",
        "    for d in directories:\n",
        "        label_dir = os.path.join(data_dir, d)\n",
        "        file_names = [os.path.join(label_dir, f)\n",
        "                      for f in os.listdir(label_dir)\n",
        "                      if f.endswith(\".jpg\")]\n",
        "        \n",
        "        # adding an early stop for sake of speed\n",
        "        stop = 0\n",
        "        for f in file_names:\n",
        "            img = cv2.imread(f)\n",
        "            imresize = cv2.resize(img, (180, 90))\n",
        "            #plt.imshow(imresize)\n",
        "            images.append(imresize)\n",
        "            labels.append(category)\n",
        "            # remove this to use full data set\n",
        "            #if stop > 300:\n",
        "            #   break\n",
        "            #stop += 1\n",
        "            # end early stop\n",
        "            \n",
        "        category += 1\n",
        "\n",
        "    return images, labels\n",
        "\n",
        "data_dir = \"../input/train\"\n",
        "images, labels = load_data(data_dir)\n",
        "\n",
        "# confirm that we have the data\n",
        "print(images[0:10])\n",
        "print(labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b565dfa6-4990-ad9b-df98-d8d0a8f32de8"
      },
      "source": [
        "Cross validate the data, so we can use a test set to check accuracy, before submitting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c1a02d04-3f8d-1ace-965e-2dfe7a484a1a"
      },
      "outputs": [],
      "source": [
        "def cross_validate(Xs, ys):\n",
        "    X_train, X_test, y_train, y_test = train_test_split(\n",
        "            Xs, ys, test_size=0.2, random_state=0)\n",
        "    return X_train, X_test, y_train, y_test\n",
        "\n",
        "X_train, X_test, y_train, y_test = cross_validate(images, labels)\n",
        "\n",
        "# confirm we got our data\n",
        "print(y_test[0:10])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "28ee099a-5b67-977d-6fdf-a989a62e6b4b"
      },
      "source": [
        "Normalize the data and hot encode outputs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d1a770a4-1733-c6b1-b0ed-322cb48f0da6"
      },
      "outputs": [],
      "source": [
        "    # normalize inputs from 0-255 and 0.0-1.0\n",
        "    X_train = np.array(X_train).astype('float32')\n",
        "    X_test = np.array(X_test).astype('float32')\n",
        "    X_train = X_train / 255.0\n",
        "    X_test = X_test / 255.0\n",
        "\n",
        "    # one hot encode outputs\n",
        "    y_train = np.array(y_train)\n",
        "    y_test = np.array(y_test)\n",
        "    y_train = np_utils.to_categorical(y_train)\n",
        "    y_test = np_utils.to_categorical(y_test)\n",
        "    num_classes = y_test.shape[1]\n",
        "    print(\"Data normalized and hot encoded.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "540c1628-46be-6e7e-a354-e95d6a7c1feb"
      },
      "source": [
        "### The code below creates and fits the CNN. It will take a while to load, even with 2 epochs.\n",
        "Please ensure the code below runs, before testing the final section, which will save your file. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5ebbf96b-f057-1dce-e895-82c7bbcd2f9e"
      },
      "source": [
        "Create our CNN Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d76ad8fd-3c40-2196-8c13-0761ef9b47ea"
      },
      "outputs": [],
      "source": [
        "def createCNNModel(num_classes):\n",
        "    \"\"\" Adapted from: # http://machinelearningmastery.com/object-recognition-convolutional-neural-networks-keras-deep-learning-library/\n",
        "# \"\"\"\n",
        "    # Create the model\n",
        "    model = Sequential()\n",
        "    #model.add(Convolution2D(32, 3, 3, input_shape=(90, 180, 3), border_mode='same', activation='relu', W_constraint=maxnorm(3)))\n",
        "    #model.add(Dropout(0.2))\n",
        "    #model.add(Convolution2D(32, 3, 3, activation='relu', border_mode='same', W_constraint=maxnorm(3)))\n",
        "    #model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "    #model.add(Flatten())\n",
        "    #model.add(Dense(512, activation='relu', W_constraint=maxnorm(3)))\n",
        "    #model.add(Dropout(0.5))\n",
        "    #model.add(Dense(num_classes, activation='softmax'))\n",
        "    \n",
        "    model.add(Convolution2D(20, 3, 3, input_shape=(90, 180, 3), activation='relu'))\n",
        "    model.add(ZeroPadding2D((1, 1), dim_ordering='th'))\n",
        "    model.add(Convolution2D(16, 3, 3, activation='relu'))\n",
        "    model.add(Dropout(0.2))\n",
        "#    model.add(ZeroPadding2D((1, 1), dim_ordering='th'))\n",
        "    model.add(Convolution2D(16, 3, 3, activation='relu'))\n",
        "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "    model.add(Flatten())\n",
        "    model.add(Dense(16, activation='relu'))\n",
        "    model.add(Dropout(0.5))\n",
        "    model.add(Dense(16, activation='relu'))\n",
        "    model.add(Dropout(0.5))\n",
        "    model.add(Dense(8, activation='softmax'))    \n",
        "    \n",
        "    # Compile model\n",
        "    epochs = 3  # >>> should be 25+\n",
        "    lrate = 0.01\n",
        "    decay = lrate/epochs\n",
        "    sgd = SGD(lr=lrate, momentum=0.9, decay=decay, nesterov=False)\n",
        "    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n",
        "    print(model.summary())\n",
        "    return model, epochs\n",
        "\n",
        "# create our CNN model\n",
        "model, epochs = createCNNModel(num_classes)\n",
        "print(\"CNN Model created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "253e27db-4e71-575a-1292-aff25a7f1cf8"
      },
      "source": [
        "\n",
        "Fit and run the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f4eb5456-3035-16c9-52f4-c9f161697948"
      },
      "outputs": [],
      "source": [
        "    model.fit(X_train, y_train, validation_data=(X_test, y_test), nb_epoch=epochs, batch_size=16)\n",
        "    # Final evaluation of the model\n",
        "    scores = model.evaluate(X_test, y_test, verbose=0)\n",
        "    print(\"Accuracy: %.2f%%\" % (scores[1]*100))\n",
        "\n",
        "    print(\"done\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7efcb1ee-e311-fa6d-5461-802ee7afa3f8"
      },
      "source": [
        "Save output for upload \n",
        "\n",
        "**Make sure to run the above code first, otherwise the model will not be defined**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6ab65a65-0cfd-ff5e-ce8a-b1f51b668dfa"
      },
      "outputs": [],
      "source": [
        "    from os import listdir\n",
        "    from os.path import isfile, join\n",
        "\n",
        "    prediction_output_list = []  # list of lists, containing logistic regression for each file\n",
        "    fnames = [f for f in listdir(\"../input/test_stg1/\") if isfile(join(\"../input/test_stg1/\", f))]\n",
        "    print(\"Testing File Names:\")\n",
        "    print(fnames)\n",
        "\n",
        "    # early stoppage...\n",
        "    # only do 10\n",
        "    i = 0\n",
        "    for f in fnames:\n",
        "        file_name = \"../input/test_stg1/\" + f\n",
        "        print(\"---Evaluating File at: \" + file_name)\n",
        "        img = cv2.imread(file_name)  \n",
        "        imresize = cv2.resize(img, (180, 90))  # resize so we're always comparing same-sized images\n",
        "        imlist = np.array([imresize])\n",
        "        print(\"Neural Net Prediction:\")\n",
        "        cnn_prediction = model.predict_proba(imlist)\n",
        "        print(cnn_prediction)\n",
        "\n",
        "        # format list for csv output\n",
        "        csv_output_list = []\n",
        "        csv_output_list.append(f)\n",
        "        for elem in cnn_prediction:\n",
        "            for value in elem:\n",
        "                csv_output_list.append(value)\n",
        "\n",
        "        # append filename to make sure we have right format to write to csv\n",
        "        print(\"CSV Output List Formatted:\")\n",
        "        print(csv_output_list)\n",
        "\n",
        "\n",
        "        # and append this file to the output_list (of lists)\n",
        "        prediction_output_list.append(csv_output_list)\n",
        "\n",
        "    #  Commented out for Kaggle, but you can use this to write to a CSV on your own computer.\n",
        "    try:\n",
        "        with open(\"cnn_predictions.csv\", \"w\") as f:\n",
        "            writer = csv.writer(f)\n",
        "            #headers = ['image', 'ALB', 'BET', 'DOL', 'LAG', 'NoF', 'OTHER', 'SHARK', 'YFT']\n",
        "            writer.writerow(['image', 'ALB', 'BET', 'DOL', 'LAG', 'NoF', 'OTHER', 'SHARK', 'YFT'])\n",
        "            writer.writerows(prediction_output_list)\n",
        "    finally:\n",
        "        f.close()\n",
        "   \n",
        "\n",
        "    print(\"done\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dc09f2de-7e7b-b5ed-49cc-336fafe7e05c"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}