{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "This is a script of current my best submission (LB score 1.36086) \nfor the State Farm Distracted Driver Chllange. \nEven though the LB score is much larger than the top kagglers (more than 10 times larger!), \nI am very glad if this script can help you. "
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "My script consists of two sub-scripts, one is for converting images to the numeric features\nby pretrained VGG-16 network, and the other is for making submission with SVC(kernel=\"rbf\")."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Following is the first script to convert images to 4096 numerical features. This script is based on \nZFTurbo's Keras sample script. To make this script work, you need \"vgg16_weights.h5\" file. Download it from here\nhttps://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import numpy as np\nimport cv2\nimport os\nimport glob\nimport h5py\nfrom tqdm import tqdm\nfrom keras.models import Sequential\nfrom keras.layers.core import Flatten, Dense, Dropout\nfrom keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D\nfrom keras.optimizers import SGD\n\ndef VGG_16(weights_path=\"../data/vgg16_weights.h5\"):\n    model = Sequential()\n    model.add(ZeroPadding2D((1,1),input_shape=(3,224,224)))\n    model.add(Convolution2D(64, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(64, 3, 3, activation='relu'))\n    model.add(MaxPooling2D((2,2), strides=(2,2)))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(128, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(128, 3, 3, activation='relu'))\n    model.add(MaxPooling2D((2,2), strides=(2,2)))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(256, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(256, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(256, 3, 3, activation='relu'))\n    model.add(MaxPooling2D((2,2), strides=(2,2)))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(512, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(512, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(512, 3, 3, activation='relu'))\n    model.add(MaxPooling2D((2,2), strides=(2,2)))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(512, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(512, 3, 3, activation='relu'))\n    model.add(ZeroPadding2D((1,1)))\n    model.add(Convolution2D(512, 3, 3, activation='relu'))\n    model.add(MaxPooling2D((2,2), strides=(2,2)))\n    model.add(Flatten())\n    model.add(Dense(4096, activation='relu'))\n    model.add(Dropout(0.5))\n    model.add(Dense(4096, activation='relu'))\n    # model.add(Dropout(0.5))\n    # model.add(Dense(1000, activation='softmax'))\n    assert os.path.exists(weights_path), 'Model weights not found (see \"weights_path\" variable in script).'\n    f = h5py.File(weights_path)\n    for k in range(f.attrs['nb_layers']):\n        if k >= len(model.layers):\n            break\n        g = f['layer_{}'.format(k)]\n        weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]\n        model.layers[k].set_weights(weights)\n    f.close()\n    return model\n\ndef get_im_cv2(path, img_rows, img_cols, color_type=1):\n    # Load as grayscale\n    if color_type == 1:\n        img = cv2.imread(path, 0)\n    elif color_type == 3:\n        img = cv2.imread(path)\n    # Reduce size\n    resized = cv2.resize(img, (img_cols, img_rows))\n    return resized\n\ndef load_train(img_rows, img_cols, j, color_type=1):\n    X_train = []\n    y_train = []\n    print('Read c%d train images'%j)\n    path = os.path.join('..', 'data', 'train', 'c' + str(j), '*.jpg')\n    files = glob.glob(path)\n    for fl in tqdm(files):\n        flbase = os.path.basename(fl)\n        img = get_im_cv2(fl, img_rows, img_cols, color_type)\n        X_train.append(img)\n        y_train.append(j)\n    return np.array(X_train, dtype=np.float32), np.array(y_train)\n\n\ndef load_test(img_rows, img_cols, read_range=[0, 1000], color_type=1):\n    print('Read test images')\n    path = os.path.join('..', 'data', 'test', '*.jpg')\n    files = glob.glob(path)\n    # Sanity check\n    assert(read_range[0] < len(files))\n    assert(read_range[0] < read_range[1])\n    if read_range[1] > len(files):\n        read_range[1] = len(files)\n    files = files[read_range[0]:read_range[1]]\n    X_test = []\n    X_test_id = []\n    total = 0\n    for fl in tqdm(files):\n        flbase = os.path.basename(fl)\n        img = get_im_cv2(fl, img_rows, img_cols, color_type)\n        X_test.append(img)\n        X_test_id.append(flbase)\n    return np.array(X_test, dtype=np.float32), np.array(X_test_id)\n\nif __name__==\"__main__\":\n    # Load model\n    model = VGG_16(\"../data/vgg16_weights.h5\")\n    model.compile(optimizer=\"sgd\", loss=\"categorical_crossentropy\")\n    \n    # Calculate train features and save it\n    features_train = []\n    labels_train = []\n    for j in range(10):\n        # Load train data\n        X_train, y_train = load_train(224, 224, j, color_type=3)\n        # Modify images\n        print(\"Preprocessing images\")\n        X_train[:, :, :, 0] -= 103.939\n        X_train[:, :, :, 1] -= 116.779\n        X_train[:, :, :, 2] -= 123.68\n        X_train = X_train.transpose((0, 3, 1, 2))\n        # Calculate features\n        print(\"Calculate features\")\n        out = model.predict(X_train, verbose=1)\n        features_train.append(out)\n        labels_train.append(y_train)\n        \n    # Save features\n    features_train = np.concatenate(features_train, axis=0)\n    labels_train = np.concatenate(labels_train)\n    np.save(\"../data/features_train2\", features_train)\n    np.save(\"../data/labels_train2\", labels_train)\n    \n    # Calculate test features and save it\n    features_test = []\n    ids = []\n    for i in range(10):\n        print(\"Converting %d th test set\"%i)\n        # Load train data\n        X_test, X_test_id = load_test(224, 224, read_range=[8000*i, 8000*(i+1)], color_type=3)\n        # Modify images\n        print(\"Preprocessing images\")\n        X_test[:, :, :, 0] -= 103.939\n        X_test[:, :, :, 1] -= 116.779\n        X_test[:, :, :, 2] -= 123.68\n        X_test = X_test.transpose((0, 3, 1, 2))\n        # Calculate features\n        print(\"Calculate features\")\n        out = model.predict(X_test, verbose=1)\n        features_test.append(out)\n        ids.append(X_test_id)\n        \n    # Save features\n    features_test = np.concatenate(features_test, axis=0)\n    ids = np.concatenate(ids)\n    np.save(\"../data/features_test2\", features_test)\n    np.save(\"../data/ids_test2\", ids)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Above script saves four files in ../data folder. features_test2.csv, ids_test2.csv, features_train2.csv, labels_train2.csv.\nLater script only use these files to make predictions. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "from __future__ import division\nimport time\nimport numpy as np\nimport pandas as pd\nfrom random import shuffle\nfrom sklearn.metrics import log_loss\nfrom sklearn.svm import SVC\n\n# Record time\nstart = time.time()\n\n# Load dataset\nprint(\"Load dataset\")\ntrain = np.load(\"../data/features_train2.npy\")\nlabels = np.load(\"../data/labels_train2.npy\")\ntest = np.load(\"../data/features_test2.npy\")\nids = np.load(\"../data/ids_test2.npy\")\nload_end = time.time()\nprint(\"Loading took %f seconds\"%(load_end - start))\n\n# Train model\nprint(\"Training model\")\nperm = np.random.permutation(len(train))\ntrain = train[perm]\nlabels = labels[perm]\nmodel = SVC(probability=True)\nmodel.fit(train, labels)\ntrain_end = time.time()\nprint(\"Training took %f seconds\"%(train_end - load_end))\n\n# Predict\nprint(\"Predict labels\")\nprobs = model.predict_proba(test)\npred_end = time.time()\nprint(\"Prediction took %f seconds\"%(pred_end - train_end))\n\n# Save submission\nprint(\"Save submission\")\nsubmission = pd.DataFrame({\"img\":ids})\nnames = [\"c0\", \"c1\", \"c2\", \"c3\", \"c4\", \"c5\", \"c6\", \"c7\", \"c8\", \"c9\"]\nfor i in range(len(names)):\n    submission[names[i]] = probs[:, i]\n\nsubmission.to_csv(\"../submission/submission17.csv\", index=False)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Above script loads previously generated data files and predict labels by SVC with rbf kernel.\nI tried other classifier like shallow neural network or xgboost, but those couldn't give me good results\nbecause the training samples are too small. "
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}