{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Importing Libraries"},{"metadata":{},"cell_type":"markdown","source":"Installing Caffe"},{"metadata":{"trusted":true},"cell_type":"code","source":"!git clone https://github.com/BVLC/caffe","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\nos.environ['CAFFE_ROOT'] = \"/kaggle/working/caffe\"\nos.environ['PYTHONPATH'] = \"/kaggle/working/caffe/python:/kaggle/lib/kagglegym:/kaggle/lib\"\n!echo $PYTHONPATH","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! apt-get update\n# caffe is pre-installed on kaggle. if it wasn't, you can install using this:\n# ! apt install caffe-cuda","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Imoprting other necessary libraries"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Files\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nimport cv2\nimport glob\n\n#DATA\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.preprocessing.text import one_hot\nfrom keras.utils.np_utils import to_categorical\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing data"},{"metadata":{},"cell_type":"markdown","source":"For machine learning subjects we normally split the raw dataset into two sub-dataset, one for training and one for validation. For this dataset we already have a test dataset ready for evalutaion but its used for final score calculated by kaggle, se we still need to write a function to seperate the train dataset into a train and a validation used for our own testing."},{"metadata":{},"cell_type":"markdown","source":"As you see the datas are already seperated into each class: one directory for images with label c0, another directory for images with label c1, etc.\nFor loading the dataset thus we don't need the `.csv` file provided, we can go to the directories one by one and load the images, and for each image we record the parent directory name (c0, c1, ...) as the label for that image. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def _prepareData(path): \n    '''\n    This function splits raw dataset into training and validation sub-datasets.\n    \n    parameters: path(str) of the directory and flag(int) to know if we prepare data of training or testing\n    return: (list) of images of the dataset and the (list) of labels\n    \n    For training:\n    -Read images of every directory and extract all images\n    -Resize to (128,128,3)\n    -Read the directory name and asign as a class\n    '''\n    imgs_list = []\n    labels = []\n    # For each class directory in imgs/train/*\n    for directory in sorted(glob.glob(os.path.join(path, '*')), key = lambda k: k.split(\"/\")[-1]):\n            # Read all the images in this class\n            for img in glob.glob(os.path.join(directory,'*.jpg')):\n                img_cv = cv2.imread(img)\n                img_cv_r = cv2.resize(img_cv,(128,128))   # Resizing images makes them faster to read\n                imgs_list.append(img_cv_r)\n                labels.append(int(directory.split(\"/\")[-1].replace('c','')))  # Reading parent dir name for label\n    \n    # Leaving 80% of train images for training and saving 20% of them for validation\n    X_Train, X_Test, Y_Train, Y_Test = train_test_split(imgs_list, labels, test_size = 0.2)\n    \n    # Keras has API to do one-hot-encoding the categorical datas. \n    # If you don't know what categorical data is, look here:\n    # https://machinelearningmastery.com/how-to-prepare-categorical-data-for-deep-learning-in-python/\n    Y_Train = tf.keras.utils.to_categorical(Y_Train, num_classes=10)\n    Y_Test = tf.keras.utils.to_categorical(Y_Test, num_classes=10)\n\n    return np.array(X_Train), np.array(X_Test), Y_Train, Y_Test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Get Data"},{"metadata":{},"cell_type":"markdown","source":"Here we use the function we just defined above to load the data. Reading can take a few minutes as we are loading 102k jpg files into RAM, that's a downside for loading all the images once into python lists. If you don't like the slow loading time, you can go for the classic way of working with `.csv` files. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Paths\npath_train_images = \"/kaggle/input/state-farm-distracted-driver-detection/imgs/train/\"\npath_submission_images =  \"/kaggle/input/state-farm-distracted-driver-detection/imgs/test/\"\n\n# List of Images for Train and Test\nX_Train, X_Test, Y_Train, Y_Test = _prepareData(path_train_images)\n\nprint(\"Size X_Train: {}, Size Y_Train: {}\".format(len(X_Train),len(Y_Train)))\nprint(\"Size X_Test: {}, Size Y_Test: {}\".format(len(X_Test),len(Y_Test)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check data integrity\n\n### Classes:\n* c0: safe driving\n* c1: texting - right\n* c2: talking on the phone - right\n* c3: texting - left\n* c4: talking on the phone - left\n* c5: operating the radio\n* c6: drinking\n* c7: reaching behind\n* c8: hair and makeup\n* c9: talking to passenger"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(X_Train))\nprint(X_Train[202].shape)\n\nim = X_Train[202]\nRGB_im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\nplt.imshow(RGB_im)\nplt.show()\nprint(\"Class: {}\".format(Y_Train[202]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check data distribution"},{"metadata":{},"cell_type":"markdown","source":"Extracting some data from the `.csv` file to see how much of each class do we have."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_file = pd.read_csv(\"/kaggle/input/state-farm-distracted-driver-detection/driver_imgs_list.csv\")\n# print(data_file.head())\ndata_x = list(pd.unique(data_file['classname']))\n\n# Clustring all images of each class together\ndata_classes = data_file.loc[:,['classname','img']].groupby(by='classname').count().reset_index()\n# print(data_classes)\ndata_y =list(data_classes['img'])\n\n# Plotting them using matplot\nplt.rcParams.update({'font.size': 22})\nplt.figure(figsize=(30,10))\nplt.bar(data_x, data_y, color=['cornflowerblue', 'lightblue', 'steelblue'])  \nplt.ylabel('Count classes')\nplt.title('Classes')\nplt.xticks(rotation=45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Caffe Overview"},{"metadata":{},"cell_type":"markdown","source":"Caffe is a deep learning framework developed by the Berkeley Vision and Learning Center (BVLC). It is written in C++ and has Python and Matlab interfaces.\n\nThere are 4 steps in training a CNN using Caffe:\n\n*   Step 1 - Data preparation: In this step, we get the images and store them in a format that can be used by Caffe. Here we will write a Python script that will handle image storage.\n  \n*   Step 2 - Model definition: In this step, we choose a CNN architecture and we define its parameters in a configuration file with extension `.prototxt`.\n  \n*   Step 3 - Solver definition: The solver is responsible for model optimization. We define the solver parameters in a configuration file with extension `.prototxt`.\n  \n*   Step 4 - Model training: We train the model by executing `caffe` command from the terminal. After training the model, we will get the trained model in a file with extension `.caffemodel`.\n  \nAfter the training phase, we will use the `.caffemodel` trained model to make predictions of new unseen data."},{"metadata":{},"cell_type":"markdown","source":"# Data Preparation"},{"metadata":{},"cell_type":"markdown","source":"Caffe has multiple ways of reading data. Here we tried to read the images as standard Caffe way, which is using efficient LMDB databases. You can find more ways [here](http://caffe.berkeleyvision.org/tutorial/layers.html)."},{"metadata":{"trusted":true},"cell_type":"code","source":"import caffe\nfrom caffe.src.caffe import proto\n\n\ndef make_datum(img, label):\n    \"\"\"\n    For making a LMDB database we first need to make each image into a datum object.\n    This function does that.\n    parameters: \n        img: numpy.ndarray (BGR instead of RGB)\n        label: int\n    \"\"\"\n    #\n    return proto.caffe_pb2.Datum(\n        channels=3,\n        width=128,\n        height=128,\n        label=label,\n        data=np.rollaxis(img, 2).tostring())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import lmdb\n\n\ndef make_lmdb(lmdb_path, x_data, y_data):\n    \"\"\"\n    Get the path for making the database,\n    and then read the dataset images and send them to database one by one\n    \"\"\"\n    in_db = lmdb.open(lmdb_path, map_size=int(1e12))\n    with in_db.begin(write=True) as in_txn:\n        for idx, img in enumerate(x_data):\n            datum = make_datum(img, y_data[idx])  # Making datum object\n            in_txn.put('{:0>5d}'.format(in_idx), datum.SerializeToString())\n            print '{:0>5d}'.format(in_idx) + ':' + img_path\n    in_db.close()\n    return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_lmdb = 'input/train_lmdb'\nval_lmdb = 'input/validation_lmdb'\n\nmake_lmdb(train_lmdb, X_Train, Y_Train)\nmake_lmdb(val_lmdb, X_Test, Y_Test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create architecture"},{"metadata":{},"cell_type":"markdown","source":"Caffe philosophy is expressivity and speed. For that we use text files to define networks, instead of code API like Keras. Coding is possible in Caffe too, but highly discoureged.\n\nAfter deciding on the CNN architecture, we need to define its parameters in a `.prototxt` file. Here is the details of the defined network structure in my git repo."},{"metadata":{},"cell_type":"markdown","source":"## 1. Data Layer"},{"metadata":{},"cell_type":"markdown","source":"Data enters Caffe through data layers: they lie at the bottom of nets. Data can come from efficient databases (LevelDB or LMDB), directly from memory, or, when efficiency is not critical, from files on disk in HDF5 or common image formats.\nParameters we have in data layer:\n* source: the path to the datas it needs to read\n* backend: specifies the data type that we read\n* batch_size: specifies the size of image batches to read at each step\n\n\n    layer {\n      name: “data”\n      type: “Data”\n      include {\n        phase: TRAIN   # Or TEST\n      }\n      data_param {\n        source: \"/kaggle/working/input/train_lmdb\"\n        backend: LMDB\n        batch_size: 265\n      }\n      top: “data”\n      top: “label”\n    }"},{"metadata":{},"cell_type":"markdown","source":"## 2. Convolution layer"},{"metadata":{},"cell_type":"markdown","source":"This layer recieves the data blob from last layer and produces conv1 blob. Convolution layers in neural networks generally convolve the input image with a set of learnable filters, each producing one feature map in the output image.\n\nthis layer produces 20 filters and kernel size is 5 with the stride of 1 done on input. Fillers help us initialize weight and bias values randomly. Here we use Xavier algorithm to automatically initialize weights based on the number of input and output neurons. And for bias we use a simple constant number of zero. `lr_mult` is also the settings for learning rate, here we set the learning rate for weights same as the resolver in runtime and the learning rate for bias twice of that.\n\n\n    layer {\n      name: \"conv1\"\n      type: \"Convolution\"\n      param { lr_mult: 1 }\n      param { lr_mult: 2 }\n      convolution_param {\n        num_output: 20\n        kernel_size: 5\n        stride: 1\n        weight_filler {\n          type: \"xavier\"\n        }\n        bias_filler {\n          type: \"constant\"\n        }\n      }\n      bottom: \"data\"\n      top: \"conv1\"\n    }"},{"metadata":{},"cell_type":"markdown","source":"## 3. Pooling layer"},{"metadata":{},"cell_type":"markdown","source":"We set the `pool` to max so it does max pooling operation on convolution outputs.\n\n    layer {\n      name: \"pool1\"\n      type: \"Pooling\"\n      pooling_param {\n        kernel_size: 2\n        stride: 2\n        pool: MAX\n      }\n      bottom: \"conv1\"\n      top: \"pool1\"\n    }"},{"metadata":{},"cell_type":"markdown","source":"## 4. Dense layer"},{"metadata":{},"cell_type":"markdown","source":"This layer is similar to previous layers too. Dense layers are knows as InnerProduct layers in Caffe. Here we have a dense layer which has 500 output and parameters is same as previous layers explained.\n\n    layer {\n      name: \"ip1\"\n      type: \"InnerProduct\"\n      param { lr_mult: 1 }\n      param { lr_mult: 2 }\n      inner_product_param {\n        num_output: 500\n        weight_filler {\n          type: \"xavier\"\n        }\n        bias_filler {\n          type: \"constant\"\n        }\n      }\n      bottom: \"pool2\"\n      top: \"ip1\"\n    }"},{"metadata":{},"cell_type":"markdown","source":"## 5. ReLU layer"},{"metadata":{},"cell_type":"markdown","source":"Since ReLU is element-wise we can do the operation once and not waste memory. This can be done with defining one name for top and bottom layers. Note that we can not have same names for blob of other layers and this is pecuilar for this layer.\n\n    layer {\n      name: \"relu1\"\n      type: \"ReLU\"\n      bottom: \"ip1\"\n      top: \"ip1\"\n    }\n   \nAfter ReLU we define another Dense layer with `bottom: \"ip1\"` and `top: \"ip2\"`"},{"metadata":{},"cell_type":"markdown","source":"## 6. Loss"},{"metadata":{},"cell_type":"markdown","source":"We define loss as follow:\n\n    layer {\n      name: \"loss\"\n      type: \"SoftmaxWithLoss\"\n      bottom: \"ip2\"\n      bottom: \"label\"\n    }"},{"metadata":{},"cell_type":"markdown","source":"# Caffe Solver"},{"metadata":{},"cell_type":"markdown","source":"The solver is responsible for model optimization. We define the solver's parameters in a `.prototxt` file. You can find my solver here: `CaffeCNN/caffe_models/caffe_model_1/solver_1.prototxt`. Below is a copy of the same.\n\nThis solver computes the accuracy of the model using the validation set every 1000 iterations. The optimization process will run for a maximum of 40000 iterations and will take a snapshot of the trained model every 5000 iterations.\n\n`base_lr`, `lr_policy`, `gamma`, `momentum` and `weight_decay` are hyperparameters that we need to tune to get a good convergence of the model.\n\nI chose `lr_policy: \"step\"` with `stepsize: 2500`, `base_lr: 0.00`1 and `gamma: 0.1`. In this configuration, we will start with a learning rate of 0.001, and we will drop the learning rate by a factor of ten every 2500 iterations.\n\nThere are different strategies for the optimization process. For a detailed explanation, you can read Caffe's [solver documentation](http://caffe.berkeleyvision.org/tutorial/solver.html).\n\n\n    net: \"/kaggle/working/CaffeCNN/caffe_models/caffe_model_1/caffe_model.prototxt\"\n    test_iter: 1000\n    test_interval: 1000\n    base_lr: 0.001\n    lr_policy: \"step\"\n    gamma: 0.1\n    stepsize: 2500\n    display: 50\n    max_iter: 40000\n    momentum: 0.9\n    weight_decay: 0.0005\n    snapshot: 5000\n    snapshot_prefix: \"/kaggle/working/CaffeCNN/caffe_models/caffe_model_1/caffe_model_1\"\n    solver_mode: GPU"},{"metadata":{},"cell_type":"markdown","source":"# Getting model ready"},{"metadata":{},"cell_type":"markdown","source":"I defined the network and solver in git repo, to get them we clone it. We have two `.prototxt` files, one for model and the other for solver."},{"metadata":{"trusted":true},"cell_type":"code","source":"! git clone https://github.com/Sadiqush/CaffeCNN","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_path = \"CaffeCNN/caffe_models/caffe_model_1/caffe_model.prototxt\"\nsolver_path = \"CaffeCNN/caffe_models/caffe_model_1/solver_1.prototxt\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train"},{"metadata":{},"cell_type":"markdown","source":"After defining the model and the solver, we can start training the model by executing the command below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"! caffe train --solver \"/kaggle/working/CaffeCNN/caffe_models/caffe_model_1/solver_1.prototxt\" 2>&1 | tee /kaggle/working/CaffeCNN/caffe_models/caffe_model_1/model_1_train.log","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training logs will be stored `CaffeCNN/caffe_models/caffe_model_1/model_1_train.log`.\n\nDuring the training process, we need to monitor the loss and the model accuracy. We can stop the process at anytime by pressing Ctrl+c. Caffe will take a snapshot of the trained model every 5000 iterations, and store them under `caffe_model_1` folder.\n\nThe snapshots have `.caffemodel` extension. For example, 10000 iterations snapshot will be called: `caffe_model_1_iter_10000.caffemodel`."},{"metadata":{},"cell_type":"markdown","source":"# Prediction"},{"metadata":{},"cell_type":"markdown","source":"Now that we have a trained model, we can use it to make predictions on new unseen data."},{"metadata":{"trusted":true},"cell_type":"code","source":"import caffe\n\nnet = caffe.Net('/kaggle/working/CaffeCNN/caffe_models/caffe_model_1/caffenet_deploy_1.prototxt',\n                '/kaggle/working/CaffeCNN/caffe_models/caffe_model_1/caffe_model_1_iter_500.caffemodel',\n                caffe.TEST)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out = net.forward()\npred_probas = out['prob']\nprint(pred_probas.argmax())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Building Using Transfer Learning"},{"metadata":{},"cell_type":"markdown","source":"Caffe comes with a repository that is used by researchers and machine learning practitioners to share their trained models. This library is called [Model Zoo](https://github.com/BVLC/caffe/wiki/Model-Zoo)."},{"metadata":{},"cell_type":"markdown","source":"Using this command we download the CaffeNet network structure, trained on ImageNet dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget http://dl.caffe.berkeleyvision.org/bvlc_reference_caffenet.caffemodel","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Training"},{"metadata":{},"cell_type":"markdown","source":"After defining the model and the solver, we can start training the model by executing the command below. The model and solver configuration files are stored under `CaffeCNN/caffe_models/caffe_model_2`.\n\nNote that we pass the trained model's weights by using the argument `--weights`."},{"metadata":{"trusted":true},"cell_type":"code","source":"!caffe train --solver=\"/kaggle/working/CaffeCNN/caffe_models/caffe_model_2/solver_2.prototxt\" --weights \"/kaggle/working/caffe/models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel\" 2>&1 | tee \"/kaggle/working/CaffeCNN/caffe_models/caffe_model_2/model_2_train.log\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Prediction is similar to the previous section, the manually defined network."},{"metadata":{},"cell_type":"markdown","source":"## Save submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.to_csv('submission_file.csv',index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}