{"cells":[{"metadata":{},"cell_type":"markdown","source":"# This is going to be my first ever EDA kernel\n\n**I will gradually update this kernel**"},{"metadata":{},"cell_type":"markdown","source":"<h1 style=\"color:blue;\">\nIf you find this kernel interesting, please drop an  <font color=\"red\">UPVOTE</font>. It motivates me to produce more quality content :)\n</h1>"},{"metadata":{},"cell_type":"markdown","source":"# what to do in this competition?"},{"metadata":{},"cell_type":"markdown","source":"**The dataset contains photos of streets, taken from the roof of a car. We're attempting to predict the position and orientation of all un-masked cars in the test images. we should also provide a confidence score indicating how sure we are of your prediction.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n!pip install -U git+https://github.com/albu/albumentations\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Imports**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nimport json\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imread\nfrom mpl_toolkits.mplot3d import Axes3D\nimport scipy.ndimage as ndimage\nimport cv2\nfrom tqdm import tqdm#_notebook as tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom functools import reduce\nimport os\nfrom sklearn.model_selection import train_test_split\nfrom scipy.optimize import minimize\nimport cv2\nimport glob\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import models\nfrom torchvision import transforms, utils\n\nfrom albumentations import Compose, ShiftScaleRotate, RandomCrop, \\\n    ToGray, CLAHE, GaussNoise, RandomGamma, \\\n    RandomBrightnessContrast, RGBShift, HueSaturationValue,Normalize\nfrom albumentations import *\nfrom albumentations.pytorch import ToTensor\n\n\nfrom tqdm import tqdm_notebook\n\nPATH = '../input/pku-autonomous-driving/'\nos.listdir(PATH)\nfrom skimage.io import imread\nimport os\nfrom math import sin, cos","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATASET_DIR = '/kaggle/input/pku-autonomous-driving/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from albumentations import (HorizontalFlip,OpticalDistortion,VerticalFlip,GridDistortion,RandomBrightnessContrast,OneOf,ElasticTransform,RandomGamma,IAAEmboss,Blur,RandomRotate90,Transpose, ShiftScaleRotate, Normalize, Resize, Compose, GaussNoise)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Officer: Sir have you been drinking?\n\n> “Of course, but not the car” ;)"},{"metadata":{},"cell_type":"markdown","source":"**Before we begin,lets watch a video first**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('<center><iframe width=\"700\" height=\"400\" src=\"https://www.youtube.com/embed/1W9q5SjaJTc\"rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe></center>')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can be seen in the video above - on the front camera of that Self-Driving Car we have **drive net** detecting obstacles the bounding boxes around the cars we have **weight net** detecting the intersection the yellow box around everything, *weight net*  then also detecting traffic lights and traffic signs and light that is classifying traffic light staid correctly is red we also have sign classification going on using **sign net** at the same time *drive net* is detecting pedestrians in the cyan bounding boxes on the far side of the intersection we also have **open road net** tracing out the free space around obstacles on the scene and on top of that we have our object tracking from frame to frame you see the track IDs on the top of each bounding box we also have our **camera base DNN distance estimation** running so you see the distance and metres displayed at the bottom of each box.**clear sight net** is also running in the background assessing whether and how well the cameras can see in our four cameras around perception set up on our embedded EDX platform and all of these rich perception functionality is what our planning and control software is going to use to execute the autonomous driving maneuvers that you will see if you see the video above.\n\n"},{"metadata":{},"cell_type":"markdown","source":"**I would also like you to recommend watching another video**"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('<center><iframe width=\"700\" height=\"400\"  src=\"https://www.youtube.com/embed/GlbZA62eXpU\" rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe></center>')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**From the video above you can see Makers of Baidu's new driverless car believe it's safer than a human driver and hope to have more on the street within the next 10 years. CNN's Matt Rivers reports.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir(PATH))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Directories**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(PATH + 'train.csv')\ntest = pd.read_csv(PATH + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**size of test set**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_size = len(test)\ntest_size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**public lb uses 9% of 2021 samples means**"},{"metadata":{"trusted":true},"cell_type":"code","source":"public_test = int(test_size * 0.09)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(public_test,'samples for testing public lb!!')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*181 samples for for public lb is very very low so DON\"T TRUST YOUR PUBLIC LB SCORE AT ALL!*"},{"metadata":{},"cell_type":"markdown","source":"**Converting ID_60565cf6d.jpg image from train_images directory to 3D**"},{"metadata":{},"cell_type":"markdown","source":"*the code below shows the image as a surface in 3D using MatPlotLib, numpy, and ndimage (for filtering).If the image is grayscale, the gray values of each pixel can determine the height of the surface.It may be necessary to blur (i.e. filter) the image to smooth out spikes that will occur due to adjacent pixels with very different values.The meshgrid function is used to set up a mesh for the surface. The ‘[::-1]’ reverses the meshgrid rows so the surface has the same orientation as the image.*\n\n[source link](https://www.quora.com/How-do-I-convert-a-single-2D-image-to-3D-using-python)"},{"metadata":{"trusted":true},"cell_type":"code","source":"imageFile =  PATH + 'train_images/ID_60565cf6d.jpg'\nmat = imread(imageFile)\nmat = mat[:,:,0] # get the first channel\nrows, cols = mat.shape\nxv, yv = np.meshgrid(range(cols), range(rows)[::-1])\n     \nblurred = ndimage.gaussian_filter(mat, sigma=(5, 5), order=0)\nfig = plt.figure(figsize=(40,40))\n     \nax = fig.add_subplot(221)\nax.imshow(mat, cmap='gray')\n     \nax = fig.add_subplot(222, projection='3d')\nax.elev= 75\nax.plot_surface(xv, yv, mat)\n     \nax = fig.add_subplot(223)\nax.imshow(blurred, cmap='gray')\n     \nax = fig.add_subplot(224, projection='3d')\nax.elev= 75\nax.plot_surface(xv, yv, blurred)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Lets visualize a messy mask from test_masks directory**"},{"metadata":{"trusted":true},"cell_type":"code","source":"imageFile =  PATH + 'test_masks/ID_f64a00c44.jpg'\nmat = imread(imageFile)\nmat = mat[:,:,0] # get the first channel\nrows, cols = mat.shape\nxv, yv = np.meshgrid(range(cols), range(rows)[::-1])\n     \nblurred = ndimage.gaussian_filter(mat, sigma=(5, 5), order=0)\nfig = plt.figure(figsize=(50,50))\n     \nax = fig.add_subplot(221)\nax.imshow(mat, cmap='gray')\n     \nax = fig.add_subplot(222, projection='3d')\nax.elev= 75\nax.plot_surface(xv, yv, mat)\n     \nax = fig.add_subplot(223)\nax.imshow(blurred, cmap='gray')\n     \nax = fig.add_subplot(224, projection='3d')\nax.elev= 75\nax.plot_surface(xv, yv, blurred)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Means and stds of single image"},{"metadata":{"trusted":true},"cell_type":"code","source":"imageFile =  PATH + 'train_images/ID_01021ce21.jpg'\nmat = imread(imageFile)\n\nfrom numpy import asarray\nfrom PIL import Image\n# load image\n#image = mat #Image.open('sydney_bridge.jpg')\n#image = Image.open(PATH + 'train_images/ID_60565cf6d.jpg')\nimage = cv2.imread(PATH + 'train_images/ID_60565cf6d.jpg')\nimage = cv2.resize(image, (1536, 512))    \n#print(image.shape)\npixels = asarray(image)\n# convert from integers to floats\npixels = pixels.astype('float32')\n# calculate per-channel means and standard deviations\nmeans = pixels.mean(axis=(0,1), dtype='float64')\nstds = pixels.std(axis=(0,1), dtype='float64')\nprint('Means: %s, Stds: %s' % (means, stds))\n# per-channel standardization of pixels\npixels = (pixels - means) / stds\n# confirm it had the desired effect\nmeans = pixels.mean(axis=(0,1), dtype='float64')\nstds = pixels.std(axis=(0,1), dtype='float64')\nprint('Means: %s, Stds: %s' % (means, stds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image = cv2.imread(PATH + 'train_images/ID_60565cf6d.jpg').astype(np.float32) / 255\nimg = image\nimg = cv2.resize(img, (1536, 512))\nprint(image.shape)\nprint(image.mean())\nprint(image.std())\n\nimg -= img.mean()\nimg /= img.std()\nprint('-------------------------------')\nprint(img.shape)\nprint(img.mean())\nprint(img.std())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# means and stds (samples vs pixels) from train images directory"},{"metadata":{},"cell_type":"markdown","source":"**stolen from here :** https://www.kaggle.com/phunghieu/cloud-mean-std-calculation"},{"metadata":{"trusted":true},"cell_type":"code","source":"h = 1536\nw = 512\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def compute_sample_mean(im_dir, image_files):\n    \"\"\"\n    Parameters:\n        im_dir: The directory that contains images.\n        image_files: List of image files you need.\n\n    Returns:\n        sample-mean\n    \"\"\"\n\n    if len(image_files) > 0:\n        image_sum = cv2.imread(os.path.join(im_dir, image_files[0]))\n        image_sum = cv2.resize(image_sum, (h, w))  \n        image_sum = cv2.cvtColor(image_sum, cv2.COLOR_BGR2RGB).astype(np.float64)\n\n        for image_file in tqdm_notebook(image_files[1:]):\n            img = cv2.imread(os.path.join(im_dir, image_file))\n            img = cv2.resize(img, (h, w))  \n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n            image_sum += img\n\n        return image_sum/(255*len(image_files))\n\n    return None\n\n\ndef compute_pixel_mean(im_dir, image_files, sample_mean=None):\n    \"\"\"\n    Parameters:\n        im_dir: The directory that contains images.\n        image_files: List of image files you need.\n        sample_mean: sample-mean value.\n\n    Returns:\n        pixel-mean\n    \"\"\"\n\n    if sample_mean is None:\n        sample_mean = compute_sample_mean(im_dir, image_files)\n\n    return np.mean(sample_mean, axis=(0, 1))\n\n\ndef compute_sample_std(im_dir, image_files, sample_mean=None):\n    \"\"\"\n    Parameters:\n        im_dir: The directory that contains images.\n        image_files: List of image files you need.\n        sample_mean: sample-mean value.\n\n    Returns:\n        sample-std\n    \"\"\"\n\n    if len(image_files) > 0:\n        if sample_mean is None:\n            sample_mean = compute_sample_mean(im_dir, image_files)\n\n        square_diff_sum = 0\n        for image_file in tqdm_notebook(image_files):\n            img = cv2.imread(os.path.join(im_dir, image_file))\n            img = cv2.resize(img, (h, w))\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB).astype(np.float64)\n            img /= 255.\n            square_diff_sum += np.square(np.abs(img - sample_mean))\n\n        return np.sqrt(square_diff_sum/len(image_files))\n\n    return 0\n\n\ndef compute_pixel_std(im_dir, image_files, sample_mean=None, pixel_mean=None):\n    \"\"\"\n    Parameters:\n        im_dir: The directory that contains images.\n        image_files: List of image files you need.\n        sample_mean: sample-mean value.\n        pixel_mean: pixel-mean value.\n\n    Returns:\n        pixel-std\n    \"\"\"\n\n    if len(image_files) > 0:\n        if pixel_mean is None:\n            if sample_mean is not None:\n                pixel_mean = compute_pixel_mean(im_dir, image_files, sample_mean)\n            else:\n                pixel_mean = compute_pixel_mean(im_dir, image_files)\n\n        square_diff_sum = 0\n        for image_file in tqdm_notebook(image_files):\n            img = cv2.imread(os.path.join(im_dir, image_file))\n            img = cv2.resize(img, (h, w))\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB).astype(np.float64)\n            img /= 255.\n            square_diff_sum += np.sum(np.square(np.abs(img - pixel_mean)), axis=(0, 1))\n\n        return np.sqrt(square_diff_sum/(len(image_files)*img.shape[0]*img.shape[1]))\n\n    return 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_images_dir = os.path.join(DATASET_DIR, 'train_images')\nimage_files = os.listdir(train_images_dir)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_mean = compute_sample_mean(train_images_dir, image_files)\nsample_std = compute_sample_std(train_images_dir, image_files, sample_mean=sample_mean)\npixel_mean = compute_pixel_mean(train_images_dir, image_files, sample_mean=sample_mean)\npixel_std = compute_pixel_std(train_images_dir, image_files, pixel_mean=pixel_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Sample mean:', sample_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Sample std:', sample_std)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Pixel mean:', pixel_mean)\nprint('Pixel std:', pixel_std)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# means and stds (samples vs pixels) from test images directory"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_images_dir = os.path.join(DATASET_DIR, 'test_images')\nimage_files = os.listdir(train_images_dir)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_mean = compute_sample_mean(train_images_dir, image_files)\nsample_std = compute_sample_std(train_images_dir, image_files, sample_mean=sample_mean)\npixel_mean = compute_pixel_mean(train_images_dir, image_files, sample_mean=sample_mean)\npixel_std = compute_pixel_std(train_images_dir, image_files, pixel_mean=pixel_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Sample mean:', sample_mean)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Sample std:', sample_std)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Pixel mean:', pixel_mean)\nprint('Pixel std:', pixel_std)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now lets visualize a good looking one from test_masks directory**"},{"metadata":{"trusted":true},"cell_type":"code","source":"imageFile =  PATH + 'test_masks/ID_d17f58f89.jpg'\nmat = imread(imageFile)\nmat = mat[:,:,0] # get the first channel\nrows, cols = mat.shape\nxv, yv = np.meshgrid(range(cols), range(rows)[::-1])\n     \nblurred = ndimage.gaussian_filter(mat, sigma=(5, 5), order=0)\nfig = plt.figure(figsize=(50,50))\n     \nax = fig.add_subplot(221)\nax.imshow(mat, cmap='gray')\n     \nax = fig.add_subplot(222, projection='3d')\nax.elev= 75\nax.plot_surface(xv, yv, mat)\n     \nax = fig.add_subplot(223)\nax.imshow(blurred, cmap='gray')\n     \nax = fig.add_subplot(224, projection='3d')\nax.elev= 75\nax.plot_surface(xv, yv, blurred)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* > COMPARE OUTPUT OF ABOVE 2 IMAGES(messy mask and good looking mask) on your own!"},{"metadata":{},"cell_type":"markdown","source":"\n# Expanding out the prediction string for the first vehicle\n\nWe know the order of each value in the prediction string. We can expand it out for the first vehicle and see some statistics for this first vehicle position.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# taken from : https://www.kaggle.com/robikscube/autonomous-driving-introduction-data-review\n\ntrain_expanded = pd.concat([train, train['PredictionString'].str.split(' ', expand=True)], axis=1)\ntrain_expanded = train_expanded.rename(columns={0 : '1_model_type', 1 : '1_yaw', 2 : '1_pitch',\n                                                3 : '1_roll', 4 : '1_x', 5 : '1_y', 6 : '1_z'})\ntrain_expanded.drop('PredictionString', axis=1).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**I have borrowed some code from this beautiful kernel[ Lyft Competition : Understanding the data](https://www.kaggle.com/tarunpaparaju/lyft-competition-understanding-the-data) for generating few plots**"},{"metadata":{},"cell_type":"markdown","source":"# KDE Plot\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"new_train_objects = train_expanded\nplot = sns.jointplot(x=new_train_objects['1_x'][:1000], y=new_train_objects['1_y'][:1000], kind='kde', color='blueviolet')\nplot.set_axis_labels('1_x', '1_y', fontsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Converting center_x,center_y,center_z,roll and pitch values from string to float**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"center_x = []\nab = new_train_objects['1_x']\nfor i in range(len(ab)):\n    center_x.append(float(ab[i])) \n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"center_y = []\nab = new_train_objects['1_y']\nfor i in range(len(ab)):\n    center_y.append(float(ab[i])) ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"center_z = []\nab = new_train_objects['1_z']\nfor i in range(len(ab)):\n    center_z.append(float(ab[i])) \n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pitch = []\nab = new_train_objects['1_pitch']\nfor i in range(len(ab)):\n    pitch.append(float(ab[i])) \n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"roll = []\nab = new_train_objects['1_roll']\nfor i in range(len(ab)):\n    roll.append(float(ab[i])) \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Distribution of center_z**"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(center_z, color='red', ax=ax).set_title('center_z', fontsize=16)\nplt.xlabel('center_z', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Distribution of pitch**"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(pitch, color='black', ax=ax).set_title('pitch', fontsize=16)\nplt.xlabel('pitch', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Distribution of Roll**"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(roll, color='purple', ax=ax).set_title('Roll', fontsize=16)\nplt.xlabel('Roll', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**we can see inconsistency in pitch and roll distribution**"},{"metadata":{},"cell_type":"markdown","source":"# Distributions of center_x and center_y\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nfig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(center_x, color='darkorange', ax=ax).set_title('center_x and center_y', fontsize=16)\nsns.distplot(center_y, color='purple', ax=ax).set_title('center_x and center_y', fontsize=16)\nplt.xlabel('center_x and center_y', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"from the diagram above,\n\n**darkorange = center_x**\n\n**purple = center_y**"},{"metadata":{},"cell_type":"markdown","source":"**Distribution of yaw**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"yaw = []\nab = new_train_objects['1_yaw']\nfor i in range(len(ab)):\n    yaw.append(float(ab[i])) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(yaw, color='darkgreen', ax=ax).set_title('yaw', fontsize=16)\nplt.xlabel('yaw', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#from car_models_json file\ncar_models = ('guangqi-chuanqi-GS4-2015.json', 'sikeda-jingrui.json', 'dazhong-SUV.json', \n'biyadi-qin.json', 'jipu-3.json', 'qirui-ruihu.json', 'leikesasi.json', 'biaozhi-liangxiang.json',\n'rongwei-750.json', 'fengtian-puladuo-06.json', 'changanbenben.json', 'changan-cs5.json',\n'lingmu-SX4-2012.json', 'biaozhi-508.json', 'linken-SUV.json', 'Skoda_Fabia-2011.json', 'dazhong.json',\n'yiqi-benteng-b50.json', 'baoma-X5.json', 'jili-boyue.json', 'biyadi-2x-F0.json', 'biaozhi-3008.json',\n'qiya.json', 'sikeda-SUV.json', '036-CAR01.json', 'fengtian-liangxiang.json', 'jilixiongmao-2015.json',\n'fengtian-SUV-gai.json', 'dongfeng-xuetielong-C6.json', 'fengtian-MPV.json', 'xiandai-suonata.json',\n'baoma-530.json', 'baojun-510.json', 'baoma-330.json', 'biyadi-F3.json', '037-CAR02.json',\n'lingmu-aotuo-2009.json', 'baojun-310-2017.json', 'fengtian-weichi-2006.json', 'bieke-yinglang-XT.json', \n'biaozhi-408.json', 'lufeng-X8.json', 'yingfeinidi-qx80.json', 'changcheng-H6-2016.json', 'fute.json', \n'mazida-6-2015.json', 'bieke.json', 'oubao.json', 'bieke-kaiyue.json', 'feiyate.json', 'haima-3.json',\n'baoshijie-kayan.json', 'yingfeinidi-SUV.json', 'dongfeng-yulong-naruijie.json', \n'dongfeng-fengguang-S560.json', 'dazhongmaiteng.json', 'lingmu-swift.json', 'benchi-ML500.json', \n'sanling-oulande.json', 'bentian-fengfan.json', 'baoshijie-paoche.json', 'MG-GT-2015.json', \n'beiqi-huansu-H3.json', 'biyadi-tang.json', 'jianghuai-ruifeng-S3.json', 'rongwei-RX5.json', \n'dongnan-V3-lingyue-2011.json', 'kaidilake-CTS.json', '019-SUV.json', 'benchi-SUR.json',\n'dongfeng-fengxing-SX6.json', 'dihao-EV.json', 'aodi-a6.json', 'xiandai-i25-2016.json', \n'dongfeng-DS5.json', 'changan-CS35-2012.json', 'benchi-GLK-300.json', 'supai-2016.json', 'aodi-Q7-SUV.json')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**There are actually 3 types of car models provided in car_models_json file.**"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"suv = twox = threex = 0\nfor i in range(len(car_models)):\n    with open('../input/pku-autonomous-driving/car_models_json/'+car_models[i]) as json_file:\n        data = json.load(json_file)\n        print('car_type: '+data['car_type'])\n        if(data['car_type'] == 'SUV'):\n            suv+=1\n        elif(data['car_type'] == '2x'):\n            twox+=1\n        else:\n            threex+=1\n        \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Total SUV:', suv)       \nprint('Total 2x:', twox)  \nprint('Total 3x:', threex)  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Lets Visualize one SUV model,one 2x model and one 3x model car below**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#taken from here  : https://www.kaggle.com/ebouteillon/load-a-3d-car-model\n\nwith open('../input/pku-autonomous-driving/car_models_json/'+car_models[0]) as json_file:\n    data = json.load(json_file)\n    vertices = np.array(data['vertices'])\n    triangles = np.array(data['faces']) - 1\n    plt.figure(figsize=(20,10))\n    ax = plt.axes(projection='3d')\n    ax.set_title('car_type: '+data['car_type'])\n    ax.set_xlim([-3, 3])\n    ax.set_ylim([-3, 3])\n    ax.set_zlim([0, 3])\n    ax.plot_trisurf(vertices[:,0], vertices[:,2], triangles, -vertices[:,1], shade=True, color='green')\n\nwith open('../input/pku-autonomous-driving/car_models_json/'+car_models[1]) as json_file:\n    data = json.load(json_file)\n    vertices = np.array(data['vertices'])\n    triangles = np.array(data['faces']) - 1\n    plt.figure(figsize=(20,10))\n    ax = plt.axes(projection='3d')\n    ax.set_title('car_type: '+data['car_type'])\n    ax.set_xlim([-3, 3])\n    ax.set_ylim([-3, 3])\n    ax.set_zlim([0, 3])\n    ax.plot_trisurf(vertices[:,0], vertices[:,2], triangles, -vertices[:,1], shade=True, color='red')\n    \nwith open('../input/pku-autonomous-driving/car_models_json/'+car_models[3]) as json_file:\n    data = json.load(json_file)\n    vertices = np.array(data['vertices'])\n    triangles = np.array(data['faces']) - 1\n    plt.figure(figsize=(20,10))\n    ax = plt.axes(projection='3d')\n    ax.set_title('car_type: '+data['car_type'])\n    ax.set_xlim([-3, 3])\n    ax.set_ylim([-3, 3])\n    ax.set_zlim([0, 3])\n    ax.plot_trisurf(vertices[:,0], vertices[:,2], triangles, -vertices[:,1], shade=True, color='yellow')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir(PATH + 'camera'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Lets read what inside that camera folder**\n\ninside camera folder we have got camera_intrinsic.txt file,we will read that file in cell below"},{"metadata":{"trusted":true},"cell_type":"code","source":"file=open(PATH + '/camera/camera_intrinsic.txt', \"r\").read().split('\\n')\nfor i in range(len(file)):\n    print(file[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Taken From : https://www.kaggle.com/hocop1/centernet-baseline\n# From camera.zip\ncamera_matrix = np.array([[2304.5479, 0,  1686.2379],\n                          [0, 2305.8757, 1354.9849],\n                          [0, 0, 1]], dtype=np.float32)\ncamera_matrix_inv = np.linalg.inv(camera_matrix)\n\ndef str2coords(s, names=['id', 'yaw', 'pitch', 'roll', 'x', 'y', 'z']):\n    '''\n    Input:\n        s: PredictionString (e.g. from train dataframe)\n        names: array of what to extract from the string\n    Output:\n        list of dicts with keys from `names`\n    '''\n    coords = []\n    for l in np.array(s.split()).reshape([-1, 7]):\n        coords.append(dict(zip(names, l.astype('float'))))\n        if 'id' in coords[-1]:\n            coords[-1]['id'] = int(coords[-1]['id'])\n    return coords\n\ndef get_img_coords(s):\n    '''\n    Input is a PredictionString (e.g. from train dataframe)\n    Output is two arrays:\n        xs: x coordinates in the image\n        ys: y coordinates in the image\n    '''\n    coords = str2coords(s)\n    xs = [c['x'] for c in coords]\n    ys = [c['y'] for c in coords]\n    zs = [c['z'] for c in coords]\n    P = np.array(list(zip(xs, ys, zs))).T\n    img_p = np.dot(camera_matrix, P).T\n    img_p[:, 0] /= img_p[:, 2]\n    img_p[:, 1] /= img_p[:, 2]\n    img_xs = img_p[:, 0]\n    img_ys = img_p[:, 1]\n    img_zs = img_p[:, 2] # z = Distance from the camera\n    return img_xs, img_ys\n\nplt.figure(figsize=(20,20))\nplt.imshow(imread(PATH + 'train_images/' + train['ImageId'][1] + '.jpg'))\nplt.scatter(*get_img_coords(train['PredictionString'][1]), color='red', s=100);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**What's Inside train.csv?**"},{"metadata":{"trusted":true},"cell_type":"code","source":"a = train\na.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We understand ImageId column contains id of each image but why PredictionString?**"},{"metadata":{},"cell_type":"markdown","source":"Lets take a look at first row of PredictionString "},{"metadata":{"trusted":true},"cell_type":"code","source":"a.PredictionString[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**PredictionString containing a collection of poses and confidence scores.**"},{"metadata":{},"cell_type":"markdown","source":"# FOR Understanding data please check [this link](https://www.kaggle.com/c/pku-autonomous-driving/data)"},{"metadata":{},"cell_type":"markdown","source":"**Visualizing first 5 train images from train_images directory**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"i = 0\nfor img in glob.glob(PATH + \"train_images/*.jpg\"):\n    i+=1\n    if(i==6):\n        break\n    img = cv2.imread(img)\n    plt.figure(figsize=(15,8))\n    plt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizing first 5 train_masks from train_masks directory**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"i = 0\nfor img in glob.glob(PATH + \"train_masks/*.jpg\"):\n    i+=1\n    if(i==6):\n        break\n    img = cv2.imread(img)\n    plt.figure(figsize=(15,8))\n    plt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizing first 10 test_images from test_images directory**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"i = 0\nfor img in glob.glob(PATH + \"test_images/*.jpg\"):\n    i+=1\n    if(i==11):\n        break\n    img = cv2.imread(img)\n    plt.figure(figsize=(15,8))\n    plt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizing first 10 test_masks from test_masks directory**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"i = 0\nfor img in glob.glob(PATH + \"test_masks/*.jpg\"):\n    i+=1\n    if(i==11):\n        break\n    img = cv2.imread(img)\n    plt.figure(figsize=(15,8))\n    plt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train = pd.read_csv(PATH + 'train.csv')\ntest = pd.read_csv(PATH + 'sample_submission.csv')\nbad_list = ['ID_1a5a10365',\n'ID_1db0533c7',\n'ID_53c3fe91a',\n'ID_408f58e9f',\n'ID_4445ae041',\n'ID_bb1d991f6',\n'ID_c44983aeb',\n'ID_f30ebe4d4']\ntrain = train.loc[~train['ImageId'].isin(bad_list)]\n# From camera.zip\ncamera_matrix = np.array([[2304.5479, 0,  1686.2379],\n                          [0, 2305.8757, 1354.9849],\n                          [0, 0, 1]], dtype=np.float32)\ncamera_matrix_inv = np.linalg.inv(camera_matrix)\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def imread(path, fast_mode=False):\n    img = cv2.imread(path)\n    if not fast_mode and img is not None and len(img.shape) == 3:\n        img = np.array(img[:, :, ::-1])\n    return img\n\nimg = imread(PATH + 'train_images/ID_8a6e65317' + '.jpg')\nIMG_SHAPE = img.shape\n\nplt.figure(figsize=(15,8))\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**REF : https://www.kaggle.com/hocop1/centernet-baseline**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rotate(x, angle):\n    x = x + angle\n    x = x - (x + np.pi) // (2 * np.pi) * 2 * np.pi\n    return x\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"IMG_WIDTH = 2048\nIMG_HEIGHT = 512\nMODEL_SCALE = 8\n\ndef _regr_preprocess(regr_dict, flip=False):\n    if flip:\n        for k in ['x', 'pitch', 'roll']:\n            regr_dict[k] = -regr_dict[k]\n    for name in ['x', 'y', 'z']:\n        regr_dict[name] = regr_dict[name] / 100\n    regr_dict['roll'] = rotate(regr_dict['roll'], np.pi)\n    regr_dict['pitch_sin'] = sin(regr_dict['pitch'])\n    regr_dict['pitch_cos'] = cos(regr_dict['pitch'])\n    regr_dict.pop('pitch')\n    regr_dict.pop('id')\n    return regr_dict\n\ndef _regr_back(regr_dict):\n    for name in ['x', 'y', 'z']:\n        regr_dict[name] = regr_dict[name] * 100\n    regr_dict['roll'] = rotate(regr_dict['roll'], -np.pi)\n    \n    pitch_sin = regr_dict['pitch_sin'] / np.sqrt(regr_dict['pitch_sin']**2 + regr_dict['pitch_cos']**2)\n    pitch_cos = regr_dict['pitch_cos'] / np.sqrt(regr_dict['pitch_sin']**2 + regr_dict['pitch_cos']**2)\n    regr_dict['pitch'] = np.arccos(pitch_cos) * np.sign(pitch_sin)\n    return regr_dict\n\ndef preprocess_image(img, flip=False):\n    img = img[img.shape[0] // 2:]\n    bg = np.ones_like(img) * img.mean(1, keepdims=True).astype(img.dtype)\n    bg = bg[:, :img.shape[1] // 6]\n    img = np.concatenate([bg, img, bg], 1)\n    img = cv2.resize(img, (IMG_WIDTH, IMG_HEIGHT))\n    if flip:\n        img = img[:,::-1]\n    return (img / 255).astype('float32')\n\ndef get_mask_and_regr(img, labels, flip=False):\n    mask = np.zeros([IMG_HEIGHT // MODEL_SCALE, IMG_WIDTH // MODEL_SCALE], dtype='float32')\n    regr_names = ['x', 'y', 'z', 'yaw', 'pitch', 'roll']\n    regr = np.zeros([IMG_HEIGHT // MODEL_SCALE, IMG_WIDTH // MODEL_SCALE, 7], dtype='float32')\n    coords = str2coords(labels)\n    xs, ys = get_img_coords(labels)\n    for x, y, regr_dict in zip(xs, ys, coords):\n        x, y = y, x\n        x = (x - img.shape[0] // 2) * IMG_HEIGHT / (img.shape[0] // 2) / MODEL_SCALE\n        x = np.round(x).astype('int')\n        y = (y + img.shape[1] // 6) * IMG_WIDTH / (img.shape[1] * 4/3) / MODEL_SCALE\n        y = np.round(y).astype('int')\n        if x >= 0 and x < IMG_HEIGHT // MODEL_SCALE and y >= 0 and y < IMG_WIDTH // MODEL_SCALE:\n            mask[x, y] = 1\n            regr_dict = _regr_preprocess(regr_dict, flip)\n            regr[x, y] = [regr_dict[n] for n in sorted(regr_dict)]\n    if flip:\n        mask = np.array(mask[:,::-1])\n        regr = np.array(regr[:,::-1])\n    return mask, regr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class CarDataset(Dataset):\n    \"\"\"Car dataset.\"\"\"\n\n    def __init__(self, dataframe, root_dir, training=True, transform=True):\n        self.df = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.training = training\n        if self.transform :\n            if self.training :\n                self.aug = Compose([\n                    \n                            #RandomGamma(p=0.2),\n                            #HorizontalFlip(),\n\n                            Cutout(num_holes=20,p=0.1),\n                        \n                            ])\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n        \n        # Get image name\n        idx, labels = self.df.values[idx]\n        img_name = self.root_dir.format(idx)\n        \n        # Read image\n        img0 = imread(img_name, True)\n        img = preprocess_image(img0)\n        img = np.rollaxis(img, 2, 0)\n        # Get mask and regression maps\n        if self.training:\n            mask, regr = get_mask_and_regr(img0, labels)\n            regr = np.rollaxis(regr, 2, 0)\n        else:\n            mask, regr = 0, 0\n        if self.transform:\n            augmented = self.aug(image=img, mask=mask,regr=regr)\n            img, mask,regr = augmented['image'],augmented['mask'] ,augmented['regr']\n            \n        return [img, mask, regr]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_images_dir = PATH + 'train_images/{}.jpg'\ntest_images_dir = PATH + 'test_images/{}.jpg'\n\ndf_train, df_dev = train_test_split(train, test_size=0.1, random_state=63)\ndf_test = test\n\n# Create dataset objects\ntrain_dataset = CarDataset(df_train, train_images_dir)\ndev_dataset = CarDataset(df_dev, train_images_dir)\ntest_dataset = CarDataset(df_test, test_images_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Without HorizontalFlip"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"img, mask, regr = dev_dataset[10]\n\nplt.figure(figsize=(10,10))\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(8,8))\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(8,8))\nplt.imshow(regr[-2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class CarDataset(Dataset):\n    \"\"\"Car dataset.\"\"\"\n\n    def __init__(self, dataframe, root_dir, training=True, transform=True):\n        self.df = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.training = training\n        if self.transform :\n            if self.training :\n                self.aug = Compose([\n                    \n                            #RandomGamma(p=0.2),\n                            HorizontalFlip(),\n\n                         \n                        \n                            ])\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n        \n        # Get image name\n        idx, labels = self.df.values[idx]\n        img_name = self.root_dir.format(idx)\n        \n        # Read image\n        img0 = imread(img_name, True)\n        img = preprocess_image(img0)\n        img = np.rollaxis(img, 2, 0)\n        # Get mask and regression maps\n        if self.training:\n            mask, regr = get_mask_and_regr(img0, labels)\n            regr = np.rollaxis(regr, 2, 0)\n        else:\n            mask, regr = 0, 0\n        if self.transform:\n            augmented = self.aug(image=img, mask=mask,regr=regr)\n            img, mask,regr = augmented['image'],augmented['mask'] ,augmented['regr']\n            \n        return [img, mask, regr]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_images_dir = PATH + 'train_images/{}.jpg'\ntest_images_dir = PATH + 'test_images/{}.jpg'\n\ndf_train, df_dev = train_test_split(train, test_size=0.1, random_state=63)\ndf_test = test\n\n# Create dataset objects\ntrain_dataset = CarDataset(df_train, train_images_dir)\ndev_dataset = CarDataset(df_dev, train_images_dir)\ntest_dataset = CarDataset(df_test, test_images_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# After Applying HorizontalFlip"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"img, mask, regr = dev_dataset[10]\n\nplt.figure(figsize=(10,10))\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(8,8))\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(8,8))\nplt.imshow(regr[-2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# lets try GaussNoise "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import albumentations as A\nA.MultiplicativeNoise()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\n\nclass CarDataset(Dataset):\n    \"\"\"Car dataset.\"\"\"\n\n    def __init__(self, dataframe, root_dir, training=True, transform=True):\n        self.df = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.training = training\n        if self.transform :\n            if self.training :\n                self.aug = Compose([\n                    \n                            #RandomGamma(p=0.2),\n                            #A.ISONoise(),\n                            A.IAAPerspective(p=1,keep_size=False),\n                            A.GaussNoise()\n                         \n                        \n                            ])\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n        \n        # Get image name\n        idx, labels = self.df.values[idx]\n        img_name = self.root_dir.format(idx)\n        \n        # Read image\n        img0 = imread(img_name, True)\n        img = preprocess_image(img0)\n        img = np.rollaxis(img, 2, 0)\n        # Get mask and regression maps\n        if self.training:\n            mask, regr = get_mask_and_regr(img0, labels)\n            regr = np.rollaxis(regr, 2, 0)\n        else:\n            mask, regr = 0, 0\n        if self.transform:\n            augmented = self.aug(image=img, mask=mask,regr=regr)\n            img, mask,regr = augmented['image'],augmented['mask'] ,augmented['regr']\n            \n        return [img, mask, regr]\n    \n    \ntrain_images_dir = PATH + 'train_images/{}.jpg'\ntest_images_dir = PATH + 'test_images/{}.jpg'\n\ndf_train, df_dev = train_test_split(train, test_size=0.1, random_state=63)\ndf_test = test\n\n# Create dataset objects\ntrain_dataset = CarDataset(df_train, train_images_dir)\ndev_dataset = CarDataset(df_dev, train_images_dir)\ntest_dataset = CarDataset(df_test, test_images_dir)\n\nimg, mask, regr = dev_dataset[10]\n\nplt.figure(figsize=(20,10))\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(8,8))\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(8,8))\nplt.imshow(regr[-2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = CarDataset(df_train, train_images_dir, transform=False)\nBATCH_SIZE = 2\n\n# Create data generators - they will produce batches\ntrain_loader = DataLoader(dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2,drop_last=True)\ndev_loader = DataLoader(dataset=dev_dataset, batch_size=2, shuffle=False, num_workers=2,drop_last=True)\ntest_loader = DataLoader(dataset=test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"IF YOU SPOT ANY ERROR,PLEASE LET ME KNOW IN THE COMMENT SECTION AND IT WILL BE HIGHLY APPRECIATED\n\n**Thanks for reading**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}