{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n

PART 1 : Complete EDA - BEGINNER TO ADVANCE LEVEL

\n
\n"},{"metadata":{},"cell_type":"markdown","source":"\n"},{"metadata":{"id":"XYm0qed03A9W"},"cell_type":"markdown","source":"## Data Description\n \n- instant: record index\n\n- dteday : date\n\n- season : season (1:springer, 2:summer, 3:fall, 4:winter)\n\n- yr : year (0: 2011, 1:2012)\n\n- mnth : month ( 1 to 12)\n\n- hr : hour (0 to 23)\n\n- holiday : weather day is holiday or not\n\n- weekday : day of the week\n\n- workingday : if day is neither weekend nor holiday is 1, otherwise is 0.\n\n- weathersit : \n - 1: Clear, Few clouds, Partly cloudy, Partly cloudy\n\n - 2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist\n\n - 3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds\n\n - 4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog\n\n- temp : Normalized temperature in Celsius. The values are divided to 41 (max)\n\n- atemp: Normalized feeling temperature in Celsius. The values are divided to 50 (max)\n\n- hum: Normalized humidity. The values are divided to 100 (max)\n\n- windspeed: Normalized wind speed. The values are divided to 67 (max)\n\n- cnt: count of total rental bikes including both casual and registered\n \n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Library Imports

\n \n
"},{"metadata":{"id":"Cdz4XwaC1jQc","trusted":true},"cell_type":"code","source":"import pandas as pd # Library for Dataframe operations\nimport numpy as np\nimport matplotlib.pyplot as plt # Plotting library\nimport seaborn as sns # Plotting library\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

View at the dataset

\n
"},{"metadata":{"id":"QX0epGOG2PCm","outputId":"5d2236f4-d4c8-4e76-8081-6439215673bc","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"https://raw.githubusercontent.com/dphi-official/Datasets/master/bike_data/bike_train.csv\" )\n\ntrain.head(3).append(train.tail(3))","execution_count":null,"outputs":[]},{"metadata":{"id":"pevFM4s42PLl","outputId":"42bba918-e777-41ba-fd2d-9b2e49cead9e","trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ***Here we observe that there is no missing values in the dataset and we are good to go with preprocessing setps.***"},{"metadata":{"id":"r6CF3ImW2PQ5","outputId":"538385fc-d460-41c1-c410-a1b3994a720f","trusted":true},"cell_type":"code","source":"train['dteday'] = pd.to_datetime(train['dteday'], format = '%d-%m-%Y')\ntrain_sort = train.sort_values(by = 'instant' , ascending= True,ignore_index= True)\ntrain_sort.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ***We convert the dates into date-time format and sort the entire data with respect to date***."},{"metadata":{"id":"ify1mBHN2Pae","outputId":"26c12458-403c-47dc-e3cf-6c921adce3c8","trusted":true},"cell_type":"code","source":"train_sort.describe(percentiles = [.25,.5,.75,.95,.97])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ***The summary of data provides us lot of information such as mean, variance,min ,max etc.***\n* ***By view at the percentiles we get an idea of outliers in the data***"},{"metadata":{"id":"GCaEitIu68vN","trusted":true},"cell_type":"code","source":"# Renaming the column names\ntrain_sort.rename(columns={'instant':'rec_id',\n 'dteday':'datetime',\n 'holiday':'is_holiday',\n 'workingday':'is_workingday',\n 'weathersit':'weather_condition',\n 'hum':'humidity',\n 'mnth':'month',\n 'cnt':'total_count',\n 'hr':'hour',\n 'yr':'year'},inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"4jMNhjj1680w","outputId":"05d55aa2-10b8-4020-c9be-a844bae6b0c7","trusted":true},"cell_type":"code","source":"# Changing datatype of categorical columns \ntrain_sort['is_holiday'] = train_sort['is_holiday'].astype('category')\ntrain_sort['is_workingday'] = train_sort['is_workingday'].astype('category')\ntrain_sort['month'] = train_sort['month'].astype('category')\ntrain_sort['hour'] = train_sort['hour'].astype('category')\ntrain_sort['year'] = train_sort['year'].astype('category')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

Count plots

\n
"},{"metadata":{"id":"higTOWbsZ_l0","outputId":"6f5ab790-7420-4eed-cc27-70e3b1ca2fb2","trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(2, 3, figsize=(17, 10), sharey=False) \n\n\nsns.countplot(x = train_sort['is_holiday'],ax=ax[0,0])\nax[0,0].set_title(\"Count plot of shares on holiday\")\nsns.countplot(x = train_sort['season'], ax = ax[0,1])\nax[0,1].set_title(\"Count plot of shares different seasons\")\nsns.countplot(x = train_sort['is_workingday'], ax = ax[0,2])\nax[0,2].set_title(\"Count plot of shares on working day\")\nsns.countplot(x = train_sort['month'], ax = ax[1,0])\nax[1,0].set_title(\"Count plot of shares on months\")\nsns.countplot(x = train_sort['year'], ax = ax[1,1])\nax[1,1].set_title(\"Count plot of shares on year\")\nsns.countplot(x = train_sort['hour'], ax = ax[1,2])\nax[1,2].set_title(\"Count plot of shares on hours\")\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"id":"uNoSa3BAdg6h","outputId":"42b7a16c-39a0-47ec-8486-c1a2b994a503","trusted":true},"cell_type":"code","source":"\ncolumn = ['temp','atemp','humidity','windspeed','total_count']\n\nsns.pairplot(train_sort[column],height = 1.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

Regression plots

\n
"},{"metadata":{"id":"KIXSRjhry7PL","outputId":"7e3912f0-4b64-4dac-fc57-24d0123f8c59","trusted":true},"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nfig,ax = plt.subplots(2,4,figsize = (22,10))\nsns.regplot(x = 'temp',y = 'total_count',data = train_sort,marker = '+',color = 'g',x_bins = 10,ax=ax[0,0])\nax[0,0].set_title(\"Temperature Vs total count\")\nsns.regplot(x = 'windspeed',y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[0,1],x_bins = 10)\nax[0,1].set_title(\"Windspeed Vs total count\")\nsns.regplot(x = 'humidity',y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[0,2],x_bins = 20)\nax[0,2].set_title(\"humidity Vs total count\")\nsns.regplot(x = 'atemp',y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[0,3],x_bins = 20)\nax[0,3].set_title(\"atemp Vs total count\")\nsns.regplot(x = train_sort['month'].astype(\"int64\"),y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[1,0],x_jitter=0.7,x_bins = 10)\nax[1,0].set_title(\"month Vs total count\")\nsns.regplot(x = train_sort['weekday'].astype(\"int64\"),y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[1,1],x_jitter=0.7,x_bins = 7)\nax[1,1].set_title(\"weekday Vs total count\")\nsns.regplot(x = train_sort['weather_condition'].astype(\"int64\"),y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[1,2],x_jitter=0.5,x_bins = 4)\nax[1,2].set_title(\"weather_condition Vs total count\")\nsns.regplot(x = train_sort['hour'].astype(\"int64\"),y = 'total_count',data = train_sort,marker = '+',color = 'g',ax=ax[1,3],x_bins = 20)\nax[1,3].set_title(\"hour Vs total count\")\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

Regression plots give us an idea about the linear relationship between the target and the feature variable

\n*

Here we have add some noise to the data inorder to plot the categorical and target variable

\n*

For the clear visulization we have created bins from the data

\n "},{"metadata":{"id":"aKOZf_Hp684M","outputId":"064b39dc-1516-42be-b86e-4e9a9c23c410","trusted":true},"cell_type":"code","source":"\ntrain_sort.plot(x = 'total_count',y= 'hour', kind = 'scatter', c = 'is_holiday', colormap = 'viridis',figsize = (15,6))\nplt.title(\"Total count of bike shares Vs holiday\")","execution_count":null,"outputs":[]},{"metadata":{"id":"vHupY7kq68yW","outputId":"7b7f5f7c-e000-42b6-91ca-8cef921bab1a","trusted":true},"cell_type":"code","source":"plt.figure(figsize = (18,7))\nsns.pointplot(x= 'hour', y = 'total_count', hue = 'season' , data = train_sort)\nplt.title(\"Shares in different seasons at different hour\") ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

By the plot we observe that in season_3 there were greater shares as compared to the other seasons

\n*

At hour 17 and 18 the shares were max n all the seasons

\n\n "},{"metadata":{"id":"oR5Br6xE-FvX","outputId":"617df87c-1d6d-4059-cedf-796dcedf146d","trusted":true},"cell_type":"code","source":"plt.figure(figsize = (18,6))\nsns.pointplot(x = 'hour', y= 'total_count', hue = 'weekday' , data = train_sort)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

All days except day_0 and day_6 other days follow the same pattern of shares

\n*

Day_2 , day_3, day_4 the shares of bikes were greater

\n*

Max shares can be observed at hour_7,8 and hour 18 & 17

\n "},{"metadata":{"id":"USVHPeoz-F6l","outputId":"037268b3-d86c-4236-9242-e4411ae2309d","trusted":true},"cell_type":"code","source":"\nfig,ax = plt.subplots(1, 3, figsize=(17, 6), sharey=False)\nsns.barplot(x = 'month',y = 'total_count',hue='season', data= train_sort,ax=ax[0])\nax[0].set_title(\"Barplot of Total count of bike share in seasons\")\nsns.countplot(x = 'season' , data = train_sort, hue = 'is_workingday', palette = 'rocket',ax =ax[1])\nax[1].set_title(\"Total shares vs working day\")\nsns.barplot(x = 'weekday',y = 'total_count', data= train_sort, ax = ax[2])\nax[2].set_title(\"Total bike shares on weekdays\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

In all the four seasons the shares were greater on working day

\n\n "},{"metadata":{},"cell_type":"markdown","source":"
\n

Distribution plots

\n
"},{"metadata":{"id":"7ynaJgq8-F4n","outputId":"37ad7917-a4be-46d2-db36-7c4c2dc6bfe9","trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(2,3,figsize=(17, 8), sharey=False)\nsns.distplot(train_sort['temp'],kde = True,ax = ax[0,0])\nsns.distplot(train_sort['atemp'],kde = True,ax = ax[0,1])\nsns.distplot(train_sort['humidity'],kde = True,ax = ax[0,2])\nsns.distplot(train_sort['windspeed'],kde = True,ax = ax[1,0])\nsns.distplot(train_sort['total_count'],kde = True,ax = ax[1,1])\n#sns.barplot(train_sort['weather_condition'],train_sort['total_count'],ax = ax[1,2])\nsns.countplot(train_sort['weather_condition'],ax = ax[1,2])\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

Temperature , atemp and humidity almost follow normal distribution

\n*

The windspeed and total count shows a right skewness

\n*

The count plot of weather condition shows that dataset has larger datapoints on weather_condition_1 (Sunny day)

\n "},{"metadata":{},"cell_type":"markdown","source":"
\n

Line Charts

\n
"},{"metadata":{},"cell_type":"markdown","source":"
\n

Filtering the data for year 2011 and 2012

\n
\n"},{"metadata":{"id":"0utvfBON54i1","outputId":"c66a5911-f36f-46e1-d987-95c6a359d65a","trusted":true},"cell_type":"code","source":"y_2012 = train_sort.loc[train_sort['year']==1] \ny_2012.groupby('datetime')['total_count'].sum().plot(kind = 'line',figsize = (15,6),color = 'g')\nplt.title(\"Total count of bike shares per day in 2012\")\nplt.show()\ny_2012.groupby('weekday')['total_count'].sum().plot(kind = 'line',figsize = (15,5),color ='g')\nplt.title(\"Total count of bike shares vs weekday\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

From the first plot we observe some kind of fluctuation of bike shares which range from 0 to 9000 in 2012

\n*

The weekday 5 records maximum shares in total

\n*

Weekday 3,4 and 5 are the most shares occur

\n "},{"metadata":{"id":"J55YeYXWz5Ar","outputId":"4be3d0bd-06ee-4016-c673-1ff9a6c98896","trusted":true},"cell_type":"code","source":"y_2011 = train_sort.loc[train_sort['year']==0] \ny_2011.groupby('datetime')['total_count'].sum().plot(kind = 'line',figsize = (15,6),color = 'r')\nplt.title(\"Total count of bike shares per day in 2011\")\nplt.show()\ny_2011.groupby('weekday')['total_count'].sum().plot(kind = 'line',figsize = (15,5),color = 'r')\nplt.title(\"Total count of bike shares vs weekday\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

From the first plot we observe some kind of fluctuation of bike shares which range from 1000 to 6000 in 2011

\n*

The weekday 5 records maximum shares in total

\n*

The shares are less in weekday 3 and 4 in 2011

\n "},{"metadata":{"id":"cqpq7CB78-JD","outputId":"2a15705c-e460-4715-c995-1f1190b7c2dc","trusted":true},"cell_type":"code","source":"\ny_2011.groupby('weather_condition')['total_count'].sum().plot(kind = 'bar',color = 'c',figsize= (12,4))\nplt.title(\"Plot of weather condition Vs total bike shares in year 2011\")","execution_count":null,"outputs":[]},{"metadata":{"id":"jFjynVsJ_dl-","outputId":"ca8cdd88-c6b8-4f58-c638-377568705522","trusted":true},"cell_type":"code","source":"y_2012.groupby('weather_condition')['total_count'].sum().plot(kind = 'bar',color = 'c',figsize= (12,4))\nplt.title(\"Plot of weather condition Vs total bike shares in year 2012\")\n","execution_count":null,"outputs":[]},{"metadata":{"id":"cbNkRjX_KkgQ","outputId":"0641fdc4-d8a6-42c6-bc49-79798554d521","trusted":true},"cell_type":"code","source":"for i in range(1,5):\n \n y_2011.loc[y_2011['season']==i].groupby('datetime')['total_count'].sum().plot(kind = 'line',figsize = (10,4),color = 'b')\n plt.title(\"(2011) Plot of Total bike shares per day in Season \"+str(i))\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n*

We observe various pattern in bike shares at different seasons in year 2011

\n*

In season 2 there is kind of increasing trend in different months

\n\n "},{"metadata":{"id":"TXoNivJXJ3aa","outputId":"ef254f78-efcc-4c18-803d-b76ba729ba11","trusted":true},"cell_type":"code","source":"for i in range(1,5):\n \n y_2012.loc[y_2012['season']==i].groupby('datetime')['total_count'].sum().plot(kind = 'line',figsize = (10,4),color = 'y')\n plt.title(\"(2012) Plot of Total bike shares per day in Season \"+str(i))\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"XfYPG5ik9YhF","outputId":"97899b2b-7b75-4f17-f76f-8fe5b45a0ef7","trusted":true},"cell_type":"code","source":"!pip install plotly --upgrade\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

View at the skewness in data

\n
"},{"metadata":{"id":"yWNu-jqQNWYS","outputId":"2ccfcb9b-1fbf-4eb7-ab94-8ccbdf6eef59","trusted":true},"cell_type":"code","source":"col = train_sort.columns.to_list()\ncol.remove('rec_id')\nsk = pd.DataFrame(train_sort[col].skew(),columns = ['skewness'])\nsk.sort_values(by = 'skewness',ascending = False).reset_index().style.bar( color='#ff6b6b')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*

There is greater skewness in 2 columns

\n*

We may take transformation such as log,sqrt,box-cox to reduce the skewness in data

\n\n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Box Plots of the data

\n
"},{"metadata":{"id":"jDZvGFblRV08","outputId":"568fe06f-e76a-41c7-858d-3df3a4fd2d99","trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3, 3, figsize=(24, 11))\n\n\n\nsns.boxplot(ax=axes[0, 0], data=train_sort, x='weather_condition', y='total_count')\nsns.boxplot(ax=axes[0, 1], data=train_sort, x='year', y='total_count')\nsns.boxplot(ax=axes[0, 2], data=train_sort, x='weekday', y='total_count')\nsns.boxplot(ax=axes[1, 0], data=train_sort, x='is_workingday', y='total_count')\nsns.boxplot(ax=axes[1, 1], data=train_sort, x='is_holiday', y='total_count')\nsns.boxplot(ax=axes[2,0], data=train_sort, x='hour', y='total_count')\nsns.boxplot(ax=axes[1, 2], data=train_sort, x='season', y='total_count')\nsns.boxplot(ax=axes[2, 1], data=train_sort, x='month', y='total_count')\nsns.boxplot(ax=axes[2, 2], data=train_sort, x='is_holiday', y='total_count',hue='year')\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*

We observe lot of outliers in the data it may due to the skewness in dataset

\n*

We can apply outlier detection techniques such as IQR, z-score etc or else we can aslo use caping and triming

\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Quantile - Quantile plots for Normality check

\n
"},{"metadata":{"id":"sL5a3UR4XFJ6","trusted":true},"cell_type":"code","source":"import scipy","execution_count":null,"outputs":[]},{"metadata":{"id":"Lsy0OM88T_89","outputId":"75ba96d7-ddc0-470d-c7b7-59558cbcb846","trusted":true},"cell_type":"code","source":"cols = ['temp','atemp','humidity','windspeed','total_count']\nfor i in cols:\n scipy.stats.probplot(train_sort[i], dist=\"norm\", plot=plt)\n plt.title(\"quantile-quantile plot of \"+i)\n plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

The QQ plot is used check for normality in the distribution using visual method

\n*

A line of 45 degree is drawn accross the data points if most the points lie on the line it is said to follow normal distribution

\n*

The first 3 plots follows normal distribution, anyhow for more accurate results we can perform normal tests such as shapiro wilk test

\n "},{"metadata":{},"cell_type":"markdown","source":"
\n

Correlation plot

\n
"},{"metadata":{"id":"GNwAYleHN3Jt","outputId":"f069aa8a-51e6-4f32-86f9-0561c82331e9","trusted":true},"cell_type":"code","source":"col = train_sort.columns.to_list()\ncol.remove('rec_id')\nplt.figure(figsize = (10,8))\nsns.heatmap(train_sort[col].corr(),cmap = 'Accent',annot = True, square = True,linewidths = 2.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

PIE Charts

\n
"},{"metadata":{"id":"UsupgJM_9I91","outputId":"6802bc18-29e7-46ea-ce75-52f85b92bf0e","trusted":true},"cell_type":"code","source":"import plotly.express as px\nfig = px.sunburst(train_sort, path=['season', 'weather_condition'],title = \"Plot of Season vs weather condition\",color = 'temp')\nfig.show()\npx.pie(names = ['season_1','season_2','season_3','season_4'],values = [372593,736328,842339,683726],hole = 0.4,title = \"Percentage of total shares in various seasons\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## - Inference \n*

The overall data contains greater shares in season 3 (32%)

\n*

Bike share count on season1 is recorded as 14%

\n\n "},{"metadata":{},"cell_type":"markdown","source":"
\n\n

These are the basic to advance level EDA performed on the dataset at first pahse.\nHope you like this approach.

\n

To Be Continued ...

\n

Please provide your appreciation for the efforts..

\n\n \n
"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}