{"nbformat_minor": 1, "cells": [{"source": ["

Overview

\n", "\n", "The Passenger Screening Algorithm Challenge asks the data science community to assist with improving threat detection at US airports while minimizing false positives to avoid long lines and delays. (Can I get an amen!). This notebook is a follow up to my first effort for this contest called [Exploratory Data Analysis and Example Generation](https://www.kaggle.com/jbfarrar/exploratory-data-analysis-and-example-generation) (I'll call it EDA from now on). As I mentioned in the EDA notebook, the HD-AIT system files supplied in this contest range from 10MB to approximately 2GB per subject. In the instructions, the organizers suggest that one may even be able to win the contest with one of the smaller image suites. In that notebook, in addition to a review of the data and its vagueries, I supplied some basic building blocks for a preprocessing pipeline.\n", "\n", "In this notebook, I continue the series with a full preprocessing pipeline using the building blocks from before as well as a first pass through a CNN based on the Alexnet using Tensorflow. Clearly, no one is going to win the contest with this method, but I thought it would be helpful to everyone working on this to have an end to end working pipeline. I hope you find it useful, and if you do, I hope you'll give me an up vote!\n", "\n", "As previously noted, I'm not an expert on these systems or the related scans. If you see something I've misunderstood or you think I've made an error, let me know and I'll correct it. TSA has made it harder for people to get into this contest by disallowing even masked images to be protrayed on Kaggle, so you'll have to put these scripts in your own environment to take them around the track. In any event, I am convinced that data science can improve the predictive veracity of these scans. I'll get off the soap box now and move on.\n", "\n", "To begin I collect all of the imports used in the notebook at the top. It makes it easier when you're converting to a preprocessing script. Make sure to take note of the last import, tsahelper. You will need to install tsahelper and uncomment this line in order for this pipeline to work. The tsahelper package is made from the EDA and is now available as a pip install (no warranties!). \n"], "cell_type": "markdown", "metadata": {"_uuid": "34721fb6a3517d7fd792148f4d8ff219d07a1af7", "_cell_guid": "98bb1381-5677-494f-a133-e1de69a4a9a5"}}, {"outputs": [], "source": ["# import libraries\n", "from __future__ import print_function\n", "from __future__ import division\n", "\n", "import numpy as np \n", "import pandas as pd\n", "import os\n", "import re\n", "\n", "import tensorflow as tf\n", "import tflearn\n", "from tflearn.layers.conv import conv_2d, max_pool_2d\n", "from tflearn.layers.core import input_data, dropout, fully_connected\n", "from tflearn.layers.estimator import regression\n", "from tflearn.layers.normalization import local_response_normalization\n", "\n", "import random\n", "from timeit import default_timer as timer\n", "\n", "#import tsahelper as tsa"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "a7d0bc57938cca7fbc2f50c9c932095473ccd561", "collapsed": true, "_cell_guid": "c061365b-547d-4c75-91fe-351b765bd307"}}, {"source": ["Here I collect the constants all in one place. Once you are running training routinely, you'll want it to be easy to try different parameters. One tricky thing about the preprocessing approach employed here, is determining when a mini batch is complete. There are 182 views of the threat zones per subject (more on the terminology and approach for this covered in detail in the [EDA](https://www.kaggle.com/jbfarrar/exploratory-data-analysis-and-example-generation)), so the batch counts depend on this fact to know when we have a complete minibatch. Note also that FILE_LIST, TRAIN_FILE_LIST, and TEST_FILE_LIST are empty until after preprocessing. More on that below."], "cell_type": "markdown", "metadata": {"_uuid": "20099504158577e68f45bc04f9bc430622f0ca8c", "_cell_guid": "fa1903ad-bc85-4fa6-bf1d-4d3bc61725b1"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# Constants\n", "#\n", "# INPUT_FOLDER: The folder that contains the source data\n", "#\n", "# PREPROCESSED_DATA_FOLDER: The folder that contains preprocessed .npy files \n", "# \n", "# STAGE1_LABELS: The CSV file containing the labels by subject\n", "#\n", "# THREAT_ZONE: Threat Zone to train on (actual number not 0 based)\n", "#\n", "# BATCH_SIZE: Number of Subjects per batch\n", "#\n", "# EXAMPLES_PER_SUBJECT Number of examples generated per subject\n", "#\n", "# FILE_LIST: A list of the preprocessed .npy files to batch\n", "# \n", "# TRAIN_TEST_SPLIT_RATIO: Ratio to split the FILE_LIST between train and test\n", "#\n", "# TRAIN_SET_FILE_LIST: The list of .npy files to be used for training\n", "#\n", "# TEST_SET_FILE_LIST: The list of .npy files to be used for testing\n", "#\n", "# IMAGE_DIM: The height and width of the images in pixels\n", "#\n", "# LEARNING_RATE Learning rate for the neural network\n", "#\n", "# N_TRAIN_STEPS The number of train steps (epochs) to run\n", "#\n", "# TRAIN_PATH Place to store the tensorboard logs\n", "#\n", "# MODEL_PATH Path where model files are stored\n", "#\n", "# MODEL_NAME Name of the model files\n", "#\n", "#----------------------------------------------------------------------------------------\n", "INPUT_FOLDER = 'tsa_datasets/stage1/aps'\n", "PREPROCESSED_DATA_FOLDER = 'tsa_datasets/preprocessed/'\n", "STAGE1_LABELS = 'tsa_datasets/stage1_labels.csv'\n", "THREAT_ZONE = 1\n", "BATCH_SIZE = 16\n", "EXAMPLES_PER_SUBJECT = 182\n", "\n", "FILE_LIST = []\n", "TRAIN_TEST_SPLIT_RATIO = 0.2\n", "TRAIN_SET_FILE_LIST = []\n", "TEST_SET_FILE_LIST = []\n", "\n", "IMAGE_DIM = 250\n", "LEARNING_RATE = 1e-3\n", "N_TRAIN_STEPS = 1\n", "TRAIN_PATH = 'tsa_logs/train/'\n", "MODEL_PATH = 'tsa_logs/model/'\n", "MODEL_NAME = ('tsa-{}-lr-{}-{}-{}-tz-{}'.format('alexnet-v0.1', LEARNING_RATE, IMAGE_DIM, \n", " IMAGE_DIM, THREAT_ZONE )) \n"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "f0fb0e63e4c0ad2be11d01f7a1016689f5e094d8", "collapsed": true, "_cell_guid": "ba6e5601-b4b8-4279-8cd7-298146ef71fb"}}, {"source": ["

The Preprocessor

\n", "\n", "Throughout this notebook, passengers who are being scanned for contraband are referred to as \"subjects\". The preprocessor begins with 3 different ways you can choose to read in a list of subjects. If you are training on the full data set, OPTION 1 (see the comments) is your best bet. If you want to preprocess all subjects for whom you have data, then OPTION 2 is your best choice, and if you are running in a sample or low volume notebook environment, then you can just give a short list of subject IDs for whom you have data loaded using OPTION 3.\n", "\n", "My approach in this notebook is to isolate each individual threat zone from every visible angle and then make features out of each individual threat zone from each angle that a given threat zone is visible. This allows us to train on each threat zone individually from every view in a 2D format. (This is covered in some detail in the [EDA](https://www.kaggle.com/jbfarrar/exploratory-data-analysis-and-example-generation)).\n", "\n", "The preprocessor loops through the data one subject at a time, transforms the images, isolates threat zones, and uses a set of vertices to crop each image to 250x250. Images are saved in minibatches by threat zone, so that they can be read into the trainer.\n", "\n", "Note that the trainer depends upon the threat zone number being present in the minibatch file name. (Not my favorite approach, but it was fastest and easiest given what I was doing). If you have a better idea, pass it along!"], "cell_type": "markdown", "metadata": {"_uuid": "3d9fa6beb4ab838cee9ca1ebb62c2dc99463a609", "_cell_guid": "83de324c-7451-4885-a55a-cac23a78c1d2"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# preprocess_tsa_data(): preprocesses the tsa datasets\n", "#\n", "# parameters: none\n", "#\n", "# returns: none\n", "#---------------------------------------------------------------------------------------\n", "\n", "def preprocess_tsa_data():\n", " \n", " # OPTION 1: get a list of all subjects for which there are labels\n", " #df = pd.read_csv(STAGE1_LABELS)\n", " #df['Subject'], df['Zone'] = df['Id'].str.split('_',1).str\n", " #SUBJECT_LIST = df['Subject'].unique()\n", "\n", " # OPTION 2: get a list of all subjects for whom there is data\n", " #SUBJECT_LIST = [os.path.splitext(subject)[0] for subject in os.listdir(INPUT_FOLDER)]\n", " \n", " # OPTION 3: get a list of subjects for small bore test purposes\n", " SUBJECT_LIST = ['00360f79fd6e02781457eda48f85da90','0043db5e8c819bffc15261b1f1ac5e42',\n", " '0050492f92e22eed3474ae3a6fc907fa','006ec59fa59dd80a64c85347eef810c7',\n", " '0097503ee9fa0606559c56458b281a08','011516ab0eca7cad7f5257672ddde70e']\n", " \n", " # intialize tracking and saving items\n", " batch_num = 1\n", " threat_zone_examples = []\n", " start_time = timer()\n", " \n", " for subject in SUBJECT_LIST:\n", "\n", " # read in the images\n", " print('--------------------------------------------------------------')\n", " print('t+> {:5.3f} |Reading images for subject #: {}'.format(timer()-start_time, \n", " subject))\n", " print('--------------------------------------------------------------')\n", " images = tsa.read_data(INPUT_FOLDER + '/' + subject + '.aps')\n", "\n", " # transpose so that the slice is the first dimension shape(16, 620, 512)\n", " images = images.transpose()\n", "\n", " # for each threat zone, loop through each image, mask off the zone and then crop it\n", " for tz_num, threat_zone_x_crop_dims in enumerate(zip(tsa.zone_slice_list, \n", " tsa.zone_crop_list)):\n", "\n", " threat_zone = threat_zone_x_crop_dims[0]\n", " crop_dims = threat_zone_x_crop_dims[1]\n", "\n", " # get label\n", " label = np.array(tsa.get_subject_zone_label(tz_num, \n", " tsa.get_subject_labels(STAGE1_LABELS, subject)))\n", "\n", " for img_num, img in enumerate(images):\n", "\n", " print('Threat Zone:Image -> {}:{}'.format(tz_num, img_num))\n", " print('Threat Zone Label -> {}'.format(label))\n", " \n", " if threat_zone[img_num] is not None:\n", "\n", " # correct the orientation of the image\n", " print('-> reorienting base image') \n", " base_img = np.flipud(img)\n", " print('-> shape {}|mean={}'.format(base_img.shape, \n", " base_img.mean()))\n", "\n", " # convert to grayscale\n", " print('-> converting to grayscale')\n", " rescaled_img = tsa.convert_to_grayscale(base_img)\n", " print('-> shape {}|mean={}'.format(rescaled_img.shape, \n", " rescaled_img.mean()))\n", "\n", " # spread the spectrum to improve contrast\n", " print('-> spreading spectrum')\n", " high_contrast_img = tsa.spread_spectrum(rescaled_img)\n", " print('-> shape {}|mean={}'.format(high_contrast_img.shape,\n", " high_contrast_img.mean()))\n", "\n", " # get the masked image\n", " print('-> masking image')\n", " masked_img = tsa.roi(high_contrast_img, threat_zone[img_num])\n", " print('-> shape {}|mean={}'.format(masked_img.shape, \n", " masked_img.mean()))\n", "\n", " # crop the image\n", " print('-> cropping image')\n", " cropped_img = tsa.crop(masked_img, crop_dims[img_num])\n", " print('-> shape {}|mean={}'.format(cropped_img.shape, \n", " cropped_img.mean()))\n", "\n", " # normalize the image\n", " print('-> normalizing image')\n", " normalized_img = tsa.normalize(cropped_img)\n", " print('-> shape {}|mean={}'.format(normalized_img.shape, \n", " normalized_img.mean()))\n", "\n", " # zero center the image\n", " print('-> zero centering')\n", " zero_centered_img = tsa.zero_center(normalized_img)\n", " print('-> shape {}|mean={}'.format(zero_centered_img.shape, \n", " zero_centered_img.mean()))\n", "\n", " # append the features and labels to this threat zone's example array\n", " print ('-> appending example to threat zone {}'.format(tz_num))\n", " threat_zone_examples.append([[tz_num], zero_centered_img, label])\n", " print ('-> shape {:d}:{:d}:{:d}:{:d}:{:d}:{:d}'.format(\n", " len(threat_zone_examples),\n", " len(threat_zone_examples[0]),\n", " len(threat_zone_examples[0][0]),\n", " len(threat_zone_examples[0][1][0]),\n", " len(threat_zone_examples[0][1][1]),\n", " len(threat_zone_examples[0][2])))\n", " else:\n", " print('-> No view of tz:{} in img:{}. Skipping to next...'.format( \n", " tz_num, img_num))\n", " print('------------------------------------------------')\n", "\n", " # each subject gets EXAMPLES_PER_SUBJECT number of examples (182 to be exact, \n", " # so this section just writes out the the data once there is a full minibatch \n", " # complete.\n", " if ((len(threat_zone_examples) % (BATCH_SIZE * EXAMPLES_PER_SUBJECT)) == 0):\n", " for tz_num, tz in enumerate(tsa.zone_slice_list):\n", "\n", " tz_examples_to_save = []\n", "\n", " # write out the batch and reset\n", " print(' -> writing: ' + PREPROCESSED_DATA_FOLDER + \n", " 'preprocessed_TSA_scans-tz{}-{}-{}-b{}.npy'.format( \n", " tz_num+1,\n", " len(threat_zone_examples[0][1][0]),\n", " len(threat_zone_examples[0][1][1]), \n", " batch_num))\n", "\n", " # get this tz's examples\n", " tz_examples = [example for example in threat_zone_examples if example[0] == \n", " [tz_num]]\n", "\n", " # drop unused columns\n", " tz_examples_to_save.append([[features_label[1], features_label[2]] \n", " for features_label in tz_examples])\n", "\n", " # save batch. Note that the trainer looks for tz{} where {} is a \n", " # tz_num 1 based in the minibatch file to select which batches to \n", " # use for training a given threat zone\n", " np.save(PREPROCESSED_DATA_FOLDER + \n", " 'preprocessed_TSA_scans-tz{}-{}-{}-b{}.npy'.format(tz_num+1, \n", " len(threat_zone_examples[0][1][0]),\n", " len(threat_zone_examples[0][1][1]), \n", " batch_num), \n", " tz_examples_to_save)\n", " del tz_examples_to_save\n", "\n", " #reset for next batch \n", " del threat_zone_examples\n", " threat_zone_examples = []\n", " batch_num += 1\n", " \n", " # we may run out of subjects before we finish a batch, so we write out \n", " # the last batch stub\n", " if (len(threat_zone_examples) > 0):\n", " for tz_num, tz in enumerate(tsa.zone_slice_list):\n", "\n", " tz_examples_to_save = []\n", "\n", " # write out the batch and reset\n", " print(' -> writing: ' + PREPROCESSED_DATA_FOLDER \n", " + 'preprocessed_TSA_scans-tz{}-{}-{}-b{}.npy'.format(tz_num+1, \n", " len(threat_zone_examples[0][1][0]),\n", " len(threat_zone_examples[0][1][1]), \n", " batch_num))\n", "\n", " # get this tz's examples\n", " tz_examples = [example for example in threat_zone_examples if example[0] == \n", " [tz_num]]\n", "\n", " # drop unused columns\n", " tz_examples_to_save.append([[features_label[1], features_label[2]] \n", " for features_label in tz_examples])\n", "\n", " #save batch\n", " np.save(PREPROCESSED_DATA_FOLDER + \n", " 'preprocessed_TSA_scans-tz{}-{}-{}-b{}.npy'.format(tz_num+1, \n", " len(threat_zone_examples[0][1][0]),\n", " len(threat_zone_examples[0][1][1]), \n", " batch_num), \n", " tz_examples_to_save)\n", "# unit test ---------------------------------------\n", "#preprocess_tsa_data()"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "4abbbad45acbbdfc286ef534d3e82fc77d20309e", "collapsed": true, "_cell_guid": "09468565-2156-4b4e-ae86-a5db779d8ac0"}}, {"source": ["

Train and Test Split

\n", "\n", "The next function takes the full minibatch list and splits it between train and test sets, using the TRAIN_TEST_SPLIT_RATIO. Note that as mentioned above building FILE_LIST searches through the minibatch file name and looks for the string '-tz' + THREAT_ZONE + '-' in the file name. If you used the preprocessor above, it creates the files in that form."], "cell_type": "markdown", "metadata": {"_uuid": "8779f4e2e8c28d9f4fb2fc7bb6318966dcee7242", "_cell_guid": "dce31fc3-bda1-4e67-af33-27c65fc1787d"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# get_train_test_file_list(): gets the batch file list, splits between train and test\n", "#\n", "# parameters: none\n", "#\n", "# returns: none\n", "#\n", "#-------------------------------------------------------------------------------------\n", "\n", "def get_train_test_file_list():\n", " \n", " global FILE_LIST\n", " global TRAIN_SET_FILE_LIST\n", " global TEST_SET_FILE_LIST\n", "\n", " if os.listdir(PREPROCESSED_DATA_FOLDER) == []:\n", " print ('No preprocessed data available. Skipping preprocessed data setup..')\n", " else:\n", " FILE_LIST = [f for f in os.listdir(PREPROCESSED_DATA_FOLDER) \n", " if re.search(re.compile('-tz' + str(THREAT_ZONE) + '-'), f)]\n", " train_test_split = len(FILE_LIST) - \\\n", " max(int(len(FILE_LIST)*TRAIN_TEST_SPLIT_RATIO),1)\n", " TRAIN_SET_FILE_LIST = FILE_LIST[:train_test_split]\n", " TEST_SET_FILE_LIST = FILE_LIST[train_test_split:]\n", " print('Train/Test Split -> {} file(s) of {} used for testing'.format( \n", " len(FILE_LIST) - train_test_split, len(FILE_LIST)))\n", " \n", "# unit test ----------------------------\n", "#get_train_test_file_list()\n", "#print ("], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "e2531b745b931a9c89fd6ed7ce22b0c007959ef6", "collapsed": true, "_cell_guid": "0bb8ac1c-0bff-4e13-b9ad-f8a4a4846dde"}}, {"source": ["

Generating an Input Pipeline

\n", "\n", "The following function reads in a minibatch, extracts features and labels, and then returns the data in a form that can be easily streamed into a tensorfow feed dictionary, or as we will do below, as a feed dictionary to a TFLearn based CNN."], "cell_type": "markdown", "metadata": {"_uuid": "77cf95ca696d55a8f9d7f2458475a44132552088", "_cell_guid": "038c2af5-eb5e-493f-983d-046e2cf14ce1"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# input_pipeline(filename, path): prepares a batch of features and labels for training\n", "#\n", "# parameters: filename - the file to be batched into the model\n", "# path - the folder where filename resides\n", "#\n", "# returns: feature_batch - a batch of features to train or test on\n", "# label_batch - a batch of labels related to the feature_batch\n", "#\n", "#---------------------------------------------------------------------------------------\n", "\n", "def input_pipeline(filename, path):\n", "\n", " preprocessed_tz_scans = []\n", " feature_batch = []\n", " label_batch = []\n", " \n", " #Load a batch of preprocessed tz scans\n", " preprocessed_tz_scans = np.load(os.path.join(path, filename))\n", " \n", " #Shuffle to randomize for input into the model\n", " np.random.shuffle(preprocessed_tz_scans)\n", " \n", " # separate features and labels\n", " for example_list in preprocessed_tz_scans:\n", " for example in example_list:\n", " feature_batch.append(example[0])\n", " label_batch.append(example[1])\n", " \n", " feature_batch = np.asarray(feature_batch, dtype=np.float32)\n", " label_batch = np.asarray(label_batch, dtype=np.float32)\n", " \n", " return feature_batch, label_batch\n", " \n", "# unit test ------------------------------------------------------------------------\n", "#print ('Train Set -----------------------------')\n", "#for f_in in TRAIN_SET_FILE_LIST:\n", "# feature_batch, label_batch = input_pipeline(f_in, PREPROCESSED_DATA_FOLDER)\n", "# print (' -> features shape {}:{}:{}'.format(len(feature_batch), \n", "# len(feature_batch[0]), \n", "# len(feature_batch[0][0])))\n", "# print (' -> labels shape {}:{}'.format(len(label_batch), len(label_batch[0])))\n", " \n", "#print ('Test Set -----------------------------')\n", "#for f_in in TEST_SET_FILE_LIST:\n", "# feature_batch, label_batch = input_pipeline(f_in, PREPROCESSED_DATA_FOLDER)\n", "# print (' -> features shape {}:{}:{}'.format(len(feature_batch), \n", "# len(feature_batch[0]), \n", "# len(feature_batch[0][0])))\n", "# print (' -> labels shape {}:{}'.format(len(label_batch), len(label_batch[0])))"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "424b8b64e2f5a99de1ad9bd1f3d6c5842d412711", "collapsed": true, "_cell_guid": "d4bf4eb6-5e9e-4a84-a93a-9c51547848ca"}}, {"source": ["

Shuffling the Training Set

\n", "\n", "Below we use TFLearn, an abstraction of Tensorflow, to build the convnet. Using TFLearn we can set the fit operation to shuffle rows within a mini batch. This function shuffles the minibatch list, so that in addition to intra-minibatch shuffling, the there is also shuffling of the order the mini batches are fed to the model."], "cell_type": "markdown", "metadata": {"_uuid": "fa94eacc829c48c2481cd0a689929d3528be563e", "_cell_guid": "50449cf1-7007-472b-bfe5-9992ff2b6b22"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# shuffle_train_set(): shuffle the list of batch files so that each train step\n", "# receives them in a different order since the TRAIN_SET_FILE_LIST\n", "# is a global\n", "#\n", "# parameters: train_set - the file listing to be shuffled\n", "#\n", "# returns: none\n", "#\n", "#-------------------------------------------------------------------------------------\n", "\n", "def shuffle_train_set(train_set):\n", " sorted_file_list = random.shuffle(train_set)\n", " TRAIN_SET_FILE_LIST = sorted_file_list\n", " \n", "# Unit test ---------------\n", "#print ('Before Shuffling ->', TRAIN_SET_FILE_LIST)\n", "#shuffle_train_set(TRAIN_SET_FILE_LIST)\n", "#print ('After Shuffling ->', TRAIN_SET_FILE_LIST)"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "d4a534f79d7bfa244a99cf47351011da91e3c5cd", "collapsed": true, "_cell_guid": "d2965a57-128b-4a87-9a44-6dc3c25f95c6"}}, {"source": ["

Defining the Alexnet CNN

\n", "\n", "The Alexnet was first put to the real world test during the ImageNet Large Scale Visual Recognition Challenge in 2012. The performance of this network was a quantum shift for its time as the model achieved a top-5 error of 15.3%, more than 10.8 percentage points ahead of the runner up. The solution is elaborated in [this paper by the original author](https://www.nvidia.cn/content/tesla/pdf/machine-learning/imagenet-classification-with-deep-convolutional-nn.pdf) if you are interested in learning more. \n", "\n", "But in short the network consists of 7 layers, 5 convolutions/maxpools, plus 2 regression layers at the end. The structure of the model looks like this: \n", "\n", "\n", "\n", "Using TFLearn makes this definition is quite intuitive and simple. "], "cell_type": "markdown", "metadata": {"_uuid": "9452ad999e065793b3587fa74db82b466aa43a96", "_cell_guid": "e20bf7a9-9a94-458e-abb3-6377064872f8"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# alexnet(width, height, lr): defines the alexnet\n", "#\n", "# parameters: width - width of the input image\n", "# height - height of the input image\n", "# lr - learning rate\n", "#\n", "# returns: none\n", "#\n", "#-------------------------------------------------------------------------------------\n", "\n", "def alexnet(width, height, lr):\n", " network = input_data(shape=[None, width, height, 1], name='features')\n", " network = conv_2d(network, 96, 11, strides=4, activation='relu')\n", " network = max_pool_2d(network, 3, strides=2)\n", " network = local_response_normalization(network)\n", " network = conv_2d(network, 256, 5, activation='relu')\n", " network = max_pool_2d(network, 3, strides=2)\n", " network = local_response_normalization(network)\n", " network = conv_2d(network, 384, 3, activation='relu')\n", " network = conv_2d(network, 384, 3, activation='relu')\n", " network = conv_2d(network, 256, 3, activation='relu')\n", " network = max_pool_2d(network, 3, strides=2)\n", " network = local_response_normalization(network)\n", " network = fully_connected(network, 4096, activation='tanh')\n", " network = dropout(network, 0.5)\n", " network = fully_connected(network, 4096, activation='tanh')\n", " network = dropout(network, 0.5)\n", " network = fully_connected(network, 2, activation='softmax')\n", " network = regression(network, optimizer='momentum', loss='categorical_crossentropy', \n", " learning_rate=lr, name='labels')\n", "\n", " model = tflearn.DNN(network, checkpoint_path=MODEL_PATH + MODEL_NAME, \n", " tensorboard_dir=TRAIN_PATH, tensorboard_verbose=3, max_checkpoints=1)\n", "\n", " return model"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "6ee8e83298b920a2326e1f7f0221e80778ba49d9", "collapsed": true, "_cell_guid": "0d16a407-4cd6-4fc4-8d50-d960b68b185d"}}, {"source": ["

The Trainer

\n", "\n", "Finally, the trainer is straight forward. Set up the network, loop to read in minibatches for test and train and run the fit method. Note that TFLearn treats each \"minibatch\" as an epoch. For the illustration purposes noted here, its not a big deal, but after may runs it may be quite annoying. \n", "\n", "Up until now, all the work I've personally done has been using the lower level interface. This was my first time trying TFLearn. I liked a lot about TFLearn. Network construction is easy-peasy. I haven't worked with Keras as of yet, but it looks like it may have a few advantages worth considering."], "cell_type": "markdown", "metadata": {"_uuid": "c65052adfa5d68db426118f56a692ecd00ed1618", "_cell_guid": "2c22886a-29f5-4e00-ab8c-563151f20e2c"}}, {"outputs": [], "source": ["#---------------------------------------------------------------------------------------\n", "# train_conv_net(): runs the train op\n", "#\n", "# parameters: none\n", "#\n", "# returns: none\n", "#\n", "#-------------------------------------------------------------------------------------\n", "\n", "def train_conv_net():\n", " \n", " val_features = []\n", " val_labels = []\n", " \n", " # get train and test batches\n", " get_train_test_file_list()\n", " \n", " # instantiate model\n", " model = alexnet(IMAGE_DIM, IMAGE_DIM, LEARNING_RATE)\n", " \n", " # read in the validation test set\n", " for j, test_f_in in enumerate(TEST_SET_FILE_LIST):\n", " if j == 0:\n", " val_features, val_labels = input_pipeline(test_f_in, PREPROCESSED_DATA_FOLDER)\n", " else:\n", " tmp_feature_batch, tmp_label_batch = input_pipeline(test_f_in, \n", " PREPROCESSED_DATA_FOLDER)\n", " val_features = np.concatenate((tmp_feature_batch, val_features), axis=0)\n", " val_labels = np.concatenate((tmp_label_batch, val_labels), axis=0)\n", "\n", " val_features = val_features.reshape(-1, IMAGE_DIM, IMAGE_DIM, 1)\n", "\n", " \n", " \n", " # start training process\n", " for i in range(N_TRAIN_STEPS):\n", "\n", " # shuffle the train set files before each step\n", " shuffle_train_set(TRAIN_SET_FILE_LIST)\n", " \n", " # run through every batch in the training set\n", " for f_in in TRAIN_SET_FILE_LIST:\n", " \n", " # read in a batch of features and labels for training\n", " feature_batch, label_batch = input_pipeline(f_in, PREPROCESSED_DATA_FOLDER)\n", " feature_batch = feature_batch.reshape(-1, IMAGE_DIM, IMAGE_DIM, 1)\n", " #print ('Feature Batch Shape ->', feature_batch.shape) \n", " \n", " # run the fit operation\n", " model.fit({'features': feature_batch}, {'labels': label_batch}, n_epoch=1, \n", " validation_set=({'features': val_features}, {'labels': val_labels}), \n", " shuffle=True, snapshot_step=None, show_metric=True, \n", " run_id=MODEL_NAME)\n", " \n", "# unit test -----------------------------------\n", "#train_conv_net()"], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "56fb6f1219a0fa99ede5fe7b8b63f66deb2c1c28", "collapsed": true, "_cell_guid": "bb2eac78-0b2d-4972-98bf-2d398ce813f1"}}, {"source": ["

Wrap Up!

\n", "\n", "Alright, now its time to cut it loose. I convert this notebook into a script and let the magic begin. So far I have run training against the first three threat zones (1-3), I am currently seeing validation accuracy in the 92-96% range. If you checked out the [EDA](http://s://www.kaggle.com/jbfarrar/exploratory-data-analysis-and-example-generation) you'll recall that the probabilities for the first three zones are 11.6%, 11%, 9.1% respectively. So a model that just predicts \"no contraband\", should perform at 88.4%, 89%, and 90.8%. So while it appears we may be getting some predictive value, much work would be needed to drive those accuracy numbers higher. With the pipeline working, I'm going to fire up a meaningful training run andI will update this section with a fullsome view by threat zone of the accuracy, once I've run enough epochs to have useful view.\n", "\n", "If you've found this helpful, I hope you'll give me an up vote!\n", "\n", "Good Luck!\n", "\n"], "cell_type": "markdown", "metadata": {"_uuid": "ac700f67b44915a0a7f5bc046dde667a582c8ec2", "_cell_guid": "76ddcdd0-d705-4a71-acf8-a4cd08803774"}}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"mimetype": "text/x-python", "version": "3.6.3", "nbconvert_exporter": "python", "file_extension": ".py", "name": "python", "codemirror_mode": {"version": 3, "name": "ipython"}, "pygments_lexer": "ipython3"}}, "nbformat": 4}