{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "1848f0f6-cd34-e41c-cd20-5a1d651546d3" }, "source": [ "## Welcome to the base Tensorflow kernel ##\n", "\n", "\n", "----------\n", "Lets get right into it:\n", "\n", " 1. First, we will be getting the coordinates with the method used in [this kernel][1]\n", " 2. We need to then crop the images so that they are classified and only 32 x 32 per lion\n", " 3. a set of 32 x 32 images **per sea-lion**, **Per: image_id**\n", " 4. One hot encode these\n", " 5. Use Tensorflow to build a model\n", " 6. Train the model\n", " 7. Plot the results\n", "\n", " [1]: https://www.kaggle.com/radustoicescu/noaa-fisheries-steller-sea-lion-population-count/use-keras-to-classify-sea-lions-0-91-accuracy" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9ecbdd29-a86f-925e-f630-aaa50b73fc65" }, "outputs": [], "source": [ "from tensorflow.contrib.learn.python.learn.estimators import model_fn as model_fn_lib\n", "from tensorflow.contrib.learn.python import SKCompat\n", "from sklearn.preprocessing import label_binarize\n", "from tensorflow.contrib import learn\n", "from subprocess import check_output\n", "import tensorflow as tf\n", "import skimage.feature\n", "import numpy as np \n", "import pandas as pd \n", "import cv2\n", "import os\n", "# if anyone knows me, they know my imports\n", "# have to be ascending order\n", "\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "print('# File sizes')\n", "for f in os.listdir('../input'):\n", " if not os.path.isdir('../input/' + f):\n", " print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')\n", " else:\n", " sizes = [os.path.getsize('../input/'+f+'/'+x)/1000000 for x in os.listdir('../input/' + f)]\n", " print(f.ljust(30) + str(round(sum(sizes), 2)) + 'MB' + ' ({} files)'.format(len(sizes)))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cbde99ff-e82d-f9b8-8770-bcb0c6d195e3" }, "source": [ "As you can see we don't exactly have all the training data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "42f615f8-c681-f4e7-4442-e94f12dd04e9" }, "outputs": [], "source": [ "# CREDITS GO TO: Radu Stoicescu\n", "# classes = [\"adult_males\", \"subadult_males\", \"adult_females\", \"juveniles\", \"pups\"]\n", "classes = ['0','1','2','3','4']\n", "file_names = os.listdir(\"../input/Train/\")\n", "file_names = sorted(file_names, key=lambda \n", " item: (int(item.partition('.')[0]) if item[0].isdigit() else float('inf'), item)) \n", "# select a subset of files to run on\n", "file_names = file_names[0:3] #INCREASE FOR YOUR OWN MACHINE\n", "print(file_names)\n", "# dataframe to store results in\n", "coordinates_df = pd.DataFrame(index=file_names, columns=classes)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7d01af6e-b5e4-2480-3e7b-b35fb57ef5b4" }, "outputs": [], "source": [ "# CREDITS GO TO: Radu Stoicescu\n", "for filename in file_names:\n", " # read the Train and Train Dotted images\n", " image_1 = cv2.imread(\"../input/TrainDotted/\" + filename)\n", " image_2 = cv2.imread(\"../input/Train/\" + filename)\n", " cut = np.copy(image_2)\n", " # absolute difference between Train and Train Dotted\n", " image_3 = cv2.absdiff(image_1,image_2)\n", " # mask out blackened regions from Train Dotted\n", " mask_1 = cv2.cvtColor(image_1, cv2.COLOR_BGR2GRAY)\n", " mask_1[mask_1 < 20] = 0\n", " mask_1[mask_1 > 0] = 255\n", " mask_2 = cv2.cvtColor(image_2, cv2.COLOR_BGR2GRAY)\n", " mask_2[mask_2 < 20] = 0\n", " mask_2[mask_2 > 0] = 255\n", " image_3 = cv2.bitwise_or(image_3, image_3, mask=mask_1)\n", " image_3 = cv2.bitwise_or(image_3, image_3, mask=mask_2) \n", " # convert to grayscale to be accepted by skimage.feature.blob_log\n", " image_3 = cv2.cvtColor(image_3, cv2.COLOR_BGR2GRAY)\n", " # detect blobs\n", " blobs = skimage.feature.blob_log(image_3, min_sigma=3, max_sigma=4, num_sigma=1, threshold=0.02)\n", " adult_males = []\n", " subadult_males = []\n", " pups = []\n", " juveniles = []\n", " adult_females = [] \n", " image_circles = image_1.copy()\n", " for blob in blobs:\n", " # get the coordinates for each blob\n", " y, x, s = blob\n", " # get the color of the pixel from Train Dotted in the center of the blob\n", " g,b,r = image_1[int(y)][int(x)][:]\n", " # decision tree to pick the class of the blob by looking at the color in Train Dotted\n", " if r > 200 and g < 50 and b < 50: # RED\n", " adult_males.append((int(x),int(y)))\n", " cv2.circle(image_circles, (int(x),int(y)), 20, (0,0,255), 10) \n", " elif r > 200 and g > 200 and b < 50: # MAGENTA\n", " subadult_males.append((int(x),int(y))) \n", " cv2.circle(image_circles, (int(x),int(y)), 20, (250,10,250), 10)\n", " elif r < 100 and g < 100 and 150 < b < 200: # GREEN\n", " pups.append((int(x),int(y)))\n", " cv2.circle(image_circles, (int(x),int(y)), 20, (20,180,35), 10)\n", " elif r < 100 and 100 < g and b < 100: # BLUE\n", " juveniles.append((int(x),int(y))) \n", " cv2.circle(image_circles, (int(x),int(y)), 20, (180,60,30), 10)\n", " elif r < 150 and g < 50 and b < 100: # BROWN\n", " adult_females.append((int(x),int(y)))\n", " cv2.circle(image_circles, (int(x),int(y)), 20, (0,42,84), 10) \n", " cv2.rectangle(cut, (int(x)-112,int(y)-112),(int(x)+112,int(y)+112), 0,-1)\n", " coordinates_df[\"0\"][filename] = adult_males\n", " coordinates_df[\"1\"][filename] = subadult_males\n", " coordinates_df[\"2\"][filename] = adult_females\n", " coordinates_df[\"3\"][filename] = juveniles\n", " coordinates_df[\"4\"][filename] = pups" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "12f06fe2-1153-95ef-5612-a5b2bb04f2b4" }, "outputs": [], "source": [ "# CREDITS TO HIM\n", "x = []\n", "y = []\n", "for filename in file_names: \n", " image = cv2.imread(\"../input/Train/\" + filename)\n", " for lion_class in classes:\n", " for coordinates in coordinates_df[lion_class][filename]:\n", " thumb = image[coordinates[1]-16:coordinates[1]+16,coordinates[0]-16:coordinates[0]+16,:]\n", " if np.shape(thumb) == (32, 32, 3):\n", " x.append(thumb)\n", " y.append(lion_class)\n", "# Add negs\n", "for i in range(0,np.shape(cut)[0],224):\n", " for j in range(0,np.shape(cut)[1],224): \n", " thumb = cut[i:i+32,j:j+32,:]\n", " if np.amin(cv2.cvtColor(thumb, cv2.COLOR_BGR2GRAY)) != 0:\n", " if np.shape(thumb) == (32,32,3):\n", " x.append(thumb)\n", " y.append(\"5\") \n", "classes.append(\"5\")\n", "x = np.array(x, dtype=np.float32)\n", "y = np.array(y, dtype=np.float32)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0931d62f-179f-b3f3-7b63-3fab594bee7d" }, "source": [ "## Here is where we switch to Tensorflow ##\n", "\n", "We need to convert our labels to numerical values so TF can validate off of those." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9ae33b4f-4f17-060d-1912-d2aaf5644610" }, "outputs": [], "source": [ "#y = label_binarize(y, classes=classes) # Guy in comments gave me this one\n", "tf.logging.set_verbosity(tf.logging.INFO)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b2e85e33-9842-f4c0-6352-b41d8bef3980" }, "source": [ "Now that we've binarized our labels, we need to create a tensorflow model.\n", "------------------------------------------------------------------------\n", "\n", "Our layers will consist of:\n", "\n", " 1. Convolutional (32 5x5 filters) `conv2d()`\n", " 2. Pooling (Max pooling 2x2, stride of 2) `max_pooling2d()`\n", " 3. Convolution (64 5x5 layers) `conv2d()`\n", " 4. Pooling (Max pooling 5x5 filters) `max_pooling2d()`\n", " 5. Dense 1 (512 number of neurons, dropout at 0.5) `dense()`\n", " 6. Dense 2 (6 neurons for each class, classes{adult male to pups} plus negative class) `dense()`\n", "\n", "*Each of these methods accepts a tensor as input and returns a transformed tensor as output. This makes it easy to connect one layer to another: just take the output from one layer-creation method and supply it as input to another.*\n", "\n", "### Tis easy as: ###\n", "\n", "![cnn][1]\n", "\n", "\n", " [1]: https://s18.postimg.org/xlbv52ujd/Screen_Shot_2017-04-21_at_1.17.40_AM.png" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "92521042-eabb-b289-44a1-92682370f062" }, "outputs": [], "source": [ "def CNN_NOAA(features, labels, mode): # create a function to pass to main run\n", " # Input Layer\n", " input_layer = tf.reshape(features, [-1, 32, 32, 3]) #batch, pixles=32x32x3\n", " #Note that we've indicated -1 for batch size, \n", " # which specifies that this dimension should be dynamically\n", " # computed based on the number of input values in features\n", "\n", " # Convolutional Layer #1\n", " conv1 = tf.layers.conv2d(\n", " inputs=input_layer,\n", " filters=32, # feature map 32 x 32 x 32\n", " kernel_size=[5, 5],\n", " padding=\"same\",\n", " activation=tf.nn.relu) # still 32 x 32 x 32 \n", "\n", " # Pooling Layer #1\n", " pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)# 16x16x32\n", "\n", " # Convolutional Layer #2 \n", " conv2 = tf.layers.conv2d(\n", " inputs=pool1,\n", " filters=64, # 16x16x64\n", " kernel_size=[5, 5],\n", " padding=\"same\",\n", " activation=tf.nn.relu)\n", "\n", " # Pooling Layer #2\n", " pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2) # 8x8x64\n", "\n", " # Dense Layer\n", " pool2_flat = tf.reshape(pool2, [-1, 8 * 8 * 64])\n", " dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)\n", " dropout = tf.layers.dropout(\n", " inputs=dense, rate=0.4, training=mode == learn.ModeKeys.TRAIN)\n", "\n", " # Logits Layer\n", " logits = tf.layers.dense(inputs=dropout, units=6)\n", "\n", " loss = None # starts at none, then gets computed each time\n", " train_op = None # starts at none, then gets computed each time\n", "\n", " # Calculate Loss (for TRAIN mode)\n", " if mode != learn.ModeKeys.INFER:\n", " onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=6)\n", " loss = tf.losses.softmax_cross_entropy(\n", " onehot_labels=onehot_labels, logits=logits)\n", "\n", " # Configure the Training Op (for TRAIN mode)\n", " if mode == learn.ModeKeys.TRAIN:\n", " train_op = tf.contrib.layers.optimize_loss(\n", " loss=loss,\n", " global_step=tf.contrib.framework.get_global_step(),\n", " learning_rate=0.00214, # reduced to 0.0014, maybe should go lower...\n", " optimizer=\"SGD\")\n", "\n", " # Generate Predictions\n", " predictions = {\n", " \"classes\": tf.argmax(\n", " input=logits, axis=1),\n", " \"probabilities\": tf.nn.softmax(\n", " logits, name=\"softmax_tensor\")\n", " }\n", " # Return a ModelFnOps object\n", " return model_fn_lib.ModelFnOps(\n", " mode=mode, predictions=predictions, loss=loss, train_op=train_op)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "62bcf1dd-2dee-d665-aac1-7603f175b564" }, "source": [ "## The difficulty of finding a good learning rate ##\n", "\n", "![lrnimg][1]\n", "\n", "\n", " [1]: http://cs231n.github.io/assets/nn3/learningrates.jpeg" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "89d377ad-dfe7-a486-16f6-cc49efea431c" }, "outputs": [], "source": [ "# Load training from x and y\n", "train_data = x # Returns image arrays\n", "train_labels = y # the label array\n", "\n", "#for evaluation we want to grab a photo from the train and test how well we did on it\n", "eval_data = train_data #\n", "eval_label = train_labels #" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5e887a0a-0484-7e33-2543-39e6407efd7e" }, "source": [ "## Estimator ##\n", "\n", "**a TensorFlow class for performing high-level model training, evaluation, and inference) for our model.**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c264950c-d09d-7d98-4ea5-1d9a48488d6a" }, "outputs": [], "source": [ "# its going to write to a model directory (output)\n", "noaa_classifier = SKCompat(learn.Estimator(model_fn=CNN_NOAA))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d3f9208e-0657-2757-9731-989b16c245df" }, "source": [ "## Add what we need to log ##" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "774c0641-ae5a-a960-b023-7bd54e6872fb" }, "outputs": [], "source": [ "# Set up logging for predictions\n", "tensors_to_log = {\"probabilities\": \"softmax_tensor\"}\n", "logging_hook = tf.train.LoggingTensorHook(\n", " tensors=tensors_to_log, every_n_iter=150)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3e60d83f-758d-4d6f-acd7-940dbfe1f1c2" }, "source": [ "## The problem with accuracy development ##\n", "\n", "![accimg][1]\n", "\n", "\n", " [1]: http://cs231n.github.io/assets/nn3/accuracies.jpeg" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6d71c12c-8646-e973-a8c0-9dda403f1812" }, "source": [ "## Train the model ##\n", "We call `fit()` on noaa_classifier" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1cbec54c-7472-646f-aee3-635dc1e324cf" }, "outputs": [], "source": [ "noaa_classifier.fit(\n", " train_data,\n", " train_labels,\n", " batch_size=128,\n", " steps=300, \n", " monitors=[logging_hook]) # Again, on kaggle machine" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e8e4d987-0464-1f89-f51b-eb570357285c" }, "source": [ "## I planned on doing 20,000 steps ##\n", "If I had the time and the computer power I would to 20,000 steps. Anything over 1,000 on Kaggle just hangs.." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0b2d5eac-bdd9-c717-025b-55f8e8600ff1" }, "source": [ "## Evaluating our Model ##\n", "**How well did it do?**\n", "\n", " metric_fn.\n", " \n", "The function that calculates and returns the value of our metric\n", "\n", " prediction_key\n", "\n", "The key of the tensor that contains the predictions returned by the model function." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "41e3dd35-0cc8-f0d4-b57b-85923ef29893" }, "outputs": [], "source": [ "metrics = {\n", " \"accuracy\": # What we're tracking\n", " learn.MetricSpec( # calculation function\n", " metric_fn=tf.metrics.accuracy, prediction_key=\"classes\"), # returns class predctions\n", "}\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "681b59bc-89cd-de85-bf92-95102f95d355" }, "outputs": [], "source": [ "# Evaluate the model and print results\n", "eval_results = noaa_classifier.predict(\n", " eval_data[1], batch_size=128)\n", "print(eval_results)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "204d044c-9225-1465-1558-45cf3dcb8914" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "plt.imshow(eval_data[1])\n", "plt.show() # honestly IDK, but that look like a lion over a log or something" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bda9fc42-4dc6-3944-82ff-92f116724e05" }, "source": [ "**It calculates the probability for a sea-lion type in a cropped 32x32 picture of one that is passed through noaa_classifier.predict**\n", "\n", "**Under \"class\" it shows _**, from 0, 1 , 2 : \"_\" is **___**\n", "\n", "----------\n", "Things to improve:\n", "\n", " 1. Using a larger subset of the train images\n", " 2. more steps in training \n", " 3. choose a model out_dir\n", " 4. Decode the probabilities to give a better response." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "de75d806-867b-2818-a215-03e2fb51d26a" }, "source": [ "## What mistake I saw people getting ##\n", "\n", " NanLossDuringTrainingError: NaN loss during training\n", "\n", "Solution (implemented):\n", "\n", " - I made the learning rate a lot, lot smaller\n", " - Change the predict batch size to the same as the train (familiarity) \n", "\n", "\n", "----------\n", "**Thanks to the user on stack overflow who made a small thing very clear**\n", "\n", "*Gradient blow up*\n", "\n", "*Reason: large gradients throw the learning process off-track.*\n", "\n", "***What you should expect**: Looking at the runtime log, you should look at the loss values per-iteration. You'll notice that the loss starts to grow significantly from iteration to iteration, eventually the loss will be too large to be represented by a floating point variable and it will become nan.*\n", "\n", "***What can you do**: Decrease the base_lr (in the solver.prototxt) by an order of magnitude (at least). If you have several loss layers, you should inspect the log to see which layer is responsible for the gradient blow up and decrease the loss_weight (in train_val.prototxt) for that specific layer, instead of the general base_lr*\n", "\n", "[Link is here][1]\n", "\n", "\n", " [1]: http://stackoverflow.com/questions/33962226/common-causes-of-nans-during-training" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "38413de5-36ee-9ecd-0cda-a953abfcf055" }, "source": [ "Update:\n", "=====\n", "After the error fix, the model did a lot better!\n", "\n", " 'classes': array([2]), 'probabilities': array([[ 0.10903401, 0.0960522 , 0.38835523, 0.11640827, 0.20649746, 0.08365285\n", "\n", "\n", "as you can see, the class was array of 2 with 0.3 probability (not so confident), which is adult_female." ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }