{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Problem Statment \nGiven a dataset of 2D dashboard camera images, an algorithm needs to be developed to classify each driver's behaviour and determine if they are driving attentively, wearing their seatbelt, or taking a selfie with their friends in the backseat etc..? This can then be used to automatically detect drivers engaging in distracted behaviours from dashboard cameras.\n\nFollowing are needed tasks for the development of the algorithm:\n1. Download and preprocess the driver images\n1. Build and train the model to classify the driver images\n1. Test the model and further improve the model using different techniques.\n### Data Exploration\nThe provided data set has driver images, each taken in a car with a driver doing something in the car (texting, eating, talking on the phone, makeup, reaching behind, etc). This dataset is obtained from Kaggle(State Farm Distracted Driver Detection competition).\n\nFollowing are the file descriptions and URL’s from which the data can be obtained :\n1. imgs.zip - zipped folder of all (train/test) images\n1. sample_submission.csv - a sample submission file in the correct format\n1. driver_imgs_list.csv - a list of training images, their subject (driver) id, and\n1. class id\n1. driver_imgs_list.csv.zip\n1. sample_submission.csv.zip\n\nThe 10 classes to predict are:\n\n1. c0: safe driving\n\n1. c1: texting - right\n\n1. c2: talking on the phone - right\n\n1. c3: texting - left\n\n1. c4: talking on the phone - left\n\n1. c5: operating the radio\n\n1. c6: drinking\n\n1. c7: reaching behind\n\n1. c8: hair and makeup\n\n1. c9: talking to passenger\n\nThere are 102150 total images.\n\n\n### Data Preprocessing\nPreprocessing of data is carried out before model is built and training process is executed. Following are the steps carried out during preprocessing.\n\n1. Initially the images are divided into training and validation sets.\n1. The images are resized to a square images i.e. 64 x 64 (224 x 224 if ram > 32 gb) pixels.\n1. All three channels were used during training process as these are color images.\n1. The images are normalised by dividing every pixel in every image by 255.\n1. To ensure the mean is zero a value of 0.5 is subtracted.\n\n### Implementation\nA standard CNN architecture was initially created and trained. We have created 4 convolutional layers with 4 max pooling layers in between. Filters were increased from 64 to 512 in each of the convolutional layers. Also dropout was used along with flattening layer before using the fully connected layer. Altogether the CNN has 2 fully connected layers. Number of nodes in the last fully connected layer were setup as 10 along with softmax activation function. Relu activation function was used for all other layers.Xavier initialization was used in each of the layers."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport os\nimport pandas as pd\nimport pickle\nimport numpy as np\nimport seaborn as sns\nfrom sklearn.datasets import load_files\nfrom keras.utils import np_utils\nimport matplotlib.pyplot as plt\n# Pretty display for notebooks\n%matplotlib inline\nfrom keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D\nfrom keras.layers import Dropout, Flatten, Dense\nfrom keras.models import Sequential\nfrom keras.utils.vis_utils import plot_model\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.utils import to_categorical\nfrom sklearn.metrics import confusion_matrix\nfrom keras.preprocessing import image                  \nfrom tqdm import tqdm\n\nimport seaborn as sns\nfrom sklearn.metrics import accuracy_score,precision_score,recall_score,f1_score\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Defining the train,test and model directories\nWe will create the directories for train,test and model training paths if not present"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_DIR = os.path.join(os.getcwd(),\"/kaggle/input/state-farm-distracted-driver-detection/imgs\",\"test\")\nTRAIN_DIR = os.path.join(os.getcwd(),\"/kaggle/input/state-farm-distracted-driver-detection/imgs\",\"train\")\nMODEL_PATH = os.path.join(os.getcwd(),\"model\",\"self_trained\")\nPICKLE_DIR = os.path.join(os.getcwd(),\"pickle_files\")\nif not os.path.exists(TEST_DIR):\n    print(\"Testing data does not exists\")\nif not os.path.exists(TRAIN_DIR):\n    print(\"Training data does not exists\")\nif not os.path.exists(MODEL_PATH):\n    print(\"Model path does not exists\")\n    os.makedirs(MODEL_PATH)\n    print(\"Model path created\")\nif not os.path.exists(PICKLE_DIR):\n    os.makedirs(PICKLE_DIR)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Preparation\ncsv files for saving path location of different files and their classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_csv(DATA_DIR,filename):\n    class_names = os.listdir(DATA_DIR)\n    data = list()\n    if(os.path.isdir(os.path.join(DATA_DIR,class_names[0]))):\n        for class_name in class_names:\n            file_names = os.listdir(os.path.join(DATA_DIR,class_name))\n            for file in file_names:\n                data.append({\n                    \"Filename\":os.path.join(DATA_DIR,class_name,file),\n                    \"ClassName\":class_name\n                })\n    else:\n        class_name = \"test\"\n        file_names = os.listdir(DATA_DIR)\n        for file in file_names:\n            data.append(({\n                \"FileName\":os.path.join(DATA_DIR,file),\n                \"ClassName\":class_name\n            }))\n    data = pd.DataFrame(data)\n    data.to_csv(os.path.join(os.getcwd(),\"csv_files\",filename),index=False)\nCSV_FILES_DIR = os.path.join(os.getcwd(),\"csv_files\")\nif not os.path.exists(CSV_FILES_DIR):\n    os.makedirs(CSV_FILES_DIR)\n    \ncreate_csv(TRAIN_DIR,\"train.csv\")\ncreate_csv(TEST_DIR,\"test.csv\")\ndata_train = pd.read_csv(os.path.join(os.getcwd(),\"csv_files\",\"train.csv\"))\ndata_test = pd.read_csv(os.path.join(os.getcwd(),\"csv_files\",\"test.csv\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_train['ClassName'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data exploration\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"nf = data_train['ClassName'].value_counts(sort=False)\nlabels = data_train['ClassName'].value_counts(sort=False).index.tolist()\ny = np.array(nf)\nwidth = 1/1.5\nN = len(y)\nx = range(N)\n\nfig = plt.figure(figsize=(20,15))\nay = fig.add_subplot(211)\n\nplt.xticks(x, labels, size=15)\nplt.yticks(size=15)\n\nay.bar(x, y, width, color=\"blue\")\n\nplt.title('Bar Chart',size=25)\nplt.xlabel('classname',size=15)\nplt.ylabel('Count',size=15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndata_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Observation\n* 22424 Train samples\n* 79726 Test samples\n* The training dataset is well balanced to a great extent and hence we need not do any downsampling of the data"},{"metadata":{},"cell_type":"markdown","source":"### Converting into numerical values\nData preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_list = list(set(data_train['ClassName'].values.tolist()))\nlabels_id = {label_name:id for id,label_name in enumerate(labels_list)}\nprint(labels_id)\ndata_train['ClassName'].replace(labels_id,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(os.path.join(os.getcwd(),\"pickle_files\",\"labels_list.pkl\"),\"wb\") as handle:\n    pickle.dump(labels_id,handle)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = to_categorical(data_train['ClassName'])\nprint(labels.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Further splitting data into training and test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nxtrain,xtest,ytrain,ytest = train_test_split(data_train.iloc[:,0],labels,test_size = 0.2,random_state=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Converting into 64*64 images\nDue to ram limitations"},{"metadata":{"trusted":true},"cell_type":"code","source":"def path_to_tensor(img_path):\n    # loads RGB image as PIL.Image.Image type\n    img = image.load_img(img_path, target_size=(64, 64))\n    # convert PIL.Image.Image type to 3D tensor with shape (224, 224, 3)\n    x = image.img_to_array(img)\n    # convert 3D tensor to 4D tensor with shape (1, 224, 224, 3) and return 4D tensor\n    return np.expand_dims(x, axis=0)\n\ndef paths_to_tensor(img_paths):\n    list_of_tensors = [path_to_tensor(img_path) for img_path in tqdm(img_paths)]\n    return np.vstack(list_of_tensors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom PIL import ImageFile                            \nImageFile.LOAD_TRUNCATED_IMAGES = True                 \n\n# pre-process the data for Keras\ntrain_tensors = paths_to_tensor(xtrain).astype('float32')/255 - 0.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_tensors = paths_to_tensor(xtest).astype('float32')/255 - 0.5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Defining model\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = Sequential()\n# 64 conv2d filters with relu\nmodel.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(64,64,3), kernel_initializer='glorot_normal'))\nmodel.add(MaxPooling2D(pool_size=2)) #Maxpool\nmodel.add(Conv2D(filters=128, kernel_size=2, padding='same', activation='relu', kernel_initializer='glorot_normal'))\nmodel.add(MaxPooling2D(pool_size=2)) #Maxpool\nmodel.add(Conv2D(filters=256, kernel_size=2, padding='same', activation='relu', kernel_initializer='glorot_normal'))\nmodel.add(MaxPooling2D(pool_size=2)) #Maxpool\nmodel.add(Conv2D(filters=512, kernel_size=2, padding='same', activation='relu', kernel_initializer='glorot_normal'))\nmodel.add(MaxPooling2D(pool_size=2)) #Maxpool\nmodel.add(Dropout(0.5))\nmodel.add(Flatten())\nmodel.add(Dense(500, activation='relu', kernel_initializer='glorot_normal'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax', kernel_initializer='glorot_normal'))\n\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_model(model,to_file=os.path.join(MODEL_PATH,\"model_distracted_driver.png\"),show_shapes=True,show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n\nfilepath = os.path.join(MODEL_PATH,\"distracted-{epoch:02d}-{val_accuracy:.2f}.hdf5\")\n\ncheckpoint = ModelCheckpoint(filepath, monitor='val_accuracy', verbose=1, save_best_only=True, mode='max',period=1)\ncallbacks_list = [checkpoint]\nepochs = 20\nmodel_history = model.fit(train_tensors,ytrain,validation_data = (valid_tensors, ytest),epochs=epochs, batch_size=40, shuffle=True,callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model performance graph"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 12))\nax1.plot(model_history.history['loss'], color='b', label=\"Training loss\")\nax1.plot(model_history.history['val_loss'], color='r', label=\"validation loss\")\nax1.set_xticks(np.arange(1, 25, 1))\nax1.set_yticks(np.arange(0, 1, 0.1))\n\nax2.plot(model_history.history['accuracy'], color='b', label=\"Training accuracy\")\nax2.plot(model_history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nax2.set_xticks(np.arange(1, 25, 1))\n\nlegend = plt.legend(loc='best', shadow=True)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_confusion_matrix(confusion_matrix, class_names, figsize = (10,7), fontsize=14):\n    df_cm = pd.DataFrame(\n        confusion_matrix, index=class_names, columns=class_names, \n    )\n    fig = plt.figure(figsize=figsize)\n    try:\n        heatmap = sns.heatmap(df_cm, annot=True, fmt=\"d\")\n    except ValueError:\n        raise ValueError(\"Confusion matrix values must be integers.\")\n    heatmap.yaxis.set_ticklabels(heatmap.yaxis.get_ticklabels(), rotation=0, ha='right', fontsize=fontsize)\n    heatmap.xaxis.set_ticklabels(heatmap.xaxis.get_ticklabels(), rotation=45, ha='right', fontsize=fontsize)\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n    fig.savefig(os.path.join(MODEL_PATH,\"confusion_matrix.png\"))\n    return fig\n\ndef print_heatmap(n_labels, n_predictions, class_names):\n    labels = n_labels #sess.run(tf.argmax(n_labels, 1))\n    predictions = n_predictions #sess.run(tf.argmax(n_predictions, 1))\n\n#     confusion_matrix = sess.run(tf.contrib.metrics.confusion_matrix(labels, predictions))\n    matrix = confusion_matrix(labels.argmax(axis=1),predictions.argmax(axis=1))\n    row_sum = np.sum(matrix, axis = 1)\n    w, h = matrix.shape\n\n    c_m = np.zeros((w, h))\n\n    for i in range(h):\n        c_m[i] = matrix[i] * 100 / row_sum[i]\n\n    c = c_m.astype(dtype = np.uint8)\n\n    \n    heatmap = print_confusion_matrix(c, class_names, figsize=(18,10), fontsize=20)\n\nclass_names = list()\nfor name,idx in labels_id.items():\n    class_names.append(name)\n# print(class_names)\nypred = model.predict(valid_tensors)\n\nprint_heatmap(ytest,ypred,class_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Precision Recall F1 Score\n\nypred_class = np.argmax(ypred,axis=1)\n# print(ypred_class[:10])\nytest = np.argmax(ytest,axis=1)\n\naccuracy = accuracy_score(ytest,ypred_class)\nprint('Accuracy: %f' % accuracy)\n# precision tp / (tp + fp)\nprecision = precision_score(ytest, ypred_class,average='weighted')\nprint('Precision: %f' % precision)\n# recall: tp / (tp + fn)\nrecall = recall_score(ytest,ypred_class,average='weighted')\nprint('Recall: %f' % recall)\n# f1: 2 tp / (2 tp + fp + fn)\nf1 = f1_score(ytest,ypred_class,average='weighted')\nprint('F1 score: %f' % f1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let us check model performance on never seen images"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import load_model\nfrom keras.utils import np_utils\nimport shutil\nBASE_MODEL_PATH = os.path.join(os.getcwd(),\"model\")\nTEST_DIR = os.path.join(os.getcwd(),\"csv_files\",\"test.csv\")\nPREDICT_DIR = os.path.join(os.getcwd(),\"pred_dir\")\nPICKLE_DIR = os.path.join(os.getcwd(),\"pickle_files\")\nJSON_DIR = os.path.join(os.getcwd(),\"json_files\")\nif not os.path.exists(PREDICT_DIR):\n    os.makedirs(PREDICT_DIR)\nelse:\n    shutil.rmtree(PREDICT_DIR)\n    os.makedirs(PREDICT_DIR)\nif not os.path.exists(JSON_DIR):\n    os.makedirs(JSON_DIR)\n\nBEST_MODEL = os.path.join(BASE_MODEL_PATH,\"self_trained\",\"distracted-07-0.99.hdf5\") #loading checkpoint with best accuracy and min epochs\nmodel = load_model(BEST_MODEL)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_test = pd.read_csv(os.path.join(TEST_DIR))\n#testing on the only 10000 images as loading the all test images requires ram>8gb\ndata_test = data_test[:10000] \ndata_test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(os.path.join(PICKLE_DIR,\"labels_list.pkl\"),\"rb\") as handle:\n    labels_id = pickle.load(handle)\nprint(labels_id)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ImageFile.LOAD_TRUNCATED_IMAGES = True  \ntest_tensors = paths_to_tensor(data_test.iloc[:,0]).astype('float32')/255 - 0.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ypred_test = model.predict(test_tensors,verbose=1)\nypred_class = np.argmax(ypred_test,axis=1)\n\nid_labels = dict()\nfor class_name,idx in labels_id.items():\n    id_labels[idx] = class_name\nprint(id_labels)\n\nfor i in range(data_test.shape[0]):\n    data_test.iloc[i,1] = id_labels[ypred_class[i]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#to create a human readable and understandable class_name\nimport json\nclass_name = dict()\nclass_name[\"c0\"] = \"SAFE_DRIVING\"\nclass_name[\"c1\"] = \"TEXTING_RIGHT\"\nclass_name[\"c2\"] = \"TALKING_PHONE_RIGHT\"\nclass_name[\"c3\"] = \"TEXTING_LEFT\"\nclass_name[\"c4\"] = \"TALKING_PHONE_LEFT\"\nclass_name[\"c5\"] = \"OPERATING_RADIO\"\nclass_name[\"c6\"] = \"DRINKING\"\nclass_name[\"c7\"] = \"REACHING_BEHIND\"\nclass_name[\"c8\"] = \"HAIR_AND_MAKEUP\"\nclass_name[\"c9\"] = \"TALKING_TO_PASSENGER\"\n\n\nwith open(os.path.join(JSON_DIR,'class_name_map.json'),'w') as secret_input:\n    json.dump(class_name,secret_input,indent=4,sort_keys=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating the prediction results for the image classification and shifting the predicted images to another folder\n#with renamed filename having the class name predicted for that image using model\nwith open(os.path.join(JSON_DIR,'class_name_map.json')) as secret_input:\n    info = json.load(secret_input)\n\n\nfor i in range(data_test.shape[0]):\n    new_name = data_test.iloc[i,0].split(\"/\")[-1].split(\".\")[0]+\"_\"+info[data_test.iloc[i,1]]+\".jpg\"\n    shutil.copy(data_test.iloc[i,0],os.path.join(PREDICT_DIR,new_name))\n    \n#saving the model predicted results into a csv file\ndata_test.to_csv(os.path.join(os.getcwd(),\"csv_files\",\"short_test_result.csv\"),index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}