{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-25T11:22:04.042696Z","iopub.execute_input":"2022-07-25T11:22:04.043096Z","iopub.status.idle":"2022-07-25T11:22:04.636651Z","shell.execute_reply.started":"2022-07-25T11:22:04.043062Z","shell.execute_reply":"2022-07-25T11:22:04.635535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>pandas: all you need to know for EDA.</center>\n\nAt the end of this tutorial, you should be able to plot most the graphs you will ever need for you EDA. ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"toc\"></a>\n* [1. Load data ](#1)<br>\n* [2. One liner visualizations](#2)<br>\n    * [2.1. Categorical Variables](#2.1.)<br>\n    * [2.2. Continuous Variables](#2.2.)<br>\n    * [2.3. Visualize multiple variables (multi-variate)](#2.3.)<br>\n* [3. Joining datasets](#3)<br>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1\"></a>\n# **<center><span style=\"color:#FF7B5F;\">1. Load data</span></center>**\nLet's first start by loading the data using pandas.","metadata":{}},{"cell_type":"code","source":"items = pd.read_csv('/kaggle/input/competitive-data-science-predict-future-sales/items.csv')\nsales = pd.read_csv('/kaggle/input/competitive-data-science-predict-future-sales/sales_train.csv')\ntitanic=pd.read_csv('/kaggle/input/titanic/train.csv')\n","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:04.638068Z","iopub.execute_input":"2022-07-25T11:22:04.638335Z","iopub.status.idle":"2022-07-25T11:22:06.808955Z","shell.execute_reply.started":"2022-07-25T11:22:04.638311Z","shell.execute_reply":"2022-07-25T11:22:06.808107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2\"></a>\n# **<center><span style=\"color:#FF7B5F;\">2. One liner visualizations</span></center>**","metadata":{}},{"cell_type":"code","source":"# we can see the data by using .head(number_of_rows_shown)\ntitanic.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:24:26.670557Z","iopub.execute_input":"2022-07-25T11:24:26.670960Z","iopub.status.idle":"2022-07-25T11:24:26.688270Z","shell.execute_reply.started":"2022-07-25T11:24:26.670925Z","shell.execute_reply":"2022-07-25T11:24:26.686644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2.1.\"></a>\n## **<center><span style=\"color:#FF7B5F;\">2.1. Visualizing categorical variables</span></center>**","metadata":{}},{"cell_type":"markdown","source":"To visualize categorical variables, you usually want to see how many records per category we have. You can use bar or hbar for this from the pandas API.","metadata":{}},{"cell_type":"code","source":"titanic['Sex'].value_counts().plot.bar(title = 'gender breadkdown on the Titanic')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:06.835835Z","iopub.execute_input":"2022-07-25T11:22:06.836230Z","iopub.status.idle":"2022-07-25T11:22:07.056002Z","shell.execute_reply.started":"2022-07-25T11:22:06.836200Z","shell.execute_reply":"2022-07-25T11:22:07.054821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you want to see your graphs horizontally, just use __barh__.","metadata":{}},{"cell_type":"code","source":"titanic['Sex'].value_counts().plot.barh(title = 'gender breadkdown on the Titanic')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:07.057140Z","iopub.execute_input":"2022-07-25T11:22:07.057853Z","iopub.status.idle":"2022-07-25T11:22:07.214923Z","shell.execute_reply.started":"2022-07-25T11:22:07.057823Z","shell.execute_reply":"2022-07-25T11:22:07.213858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic['Sex'].value_counts().plot.pie(title = 'gender breadkdown on the Titanic')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:07.216398Z","iopub.execute_input":"2022-07-25T11:22:07.216920Z","iopub.status.idle":"2022-07-25T11:22:07.314092Z","shell.execute_reply.started":"2022-07-25T11:22:07.216889Z","shell.execute_reply":"2022-07-25T11:22:07.312751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2.2.\"></a>\n## **<center><span style=\"color:#FF7B5F;\">2.2. Visualizing continuous variables</span></center>**","metadata":{}},{"cell_type":"markdown","source":"Histograms are a great way to visualize how your continuous data is distributed. ","metadata":{}},{"cell_type":"code","source":"titanic[\"Age\"].plot.hist(bins=50, title = 'Age distribution on the titanic')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:07.316128Z","iopub.execute_input":"2022-07-25T11:22:07.316603Z","iopub.status.idle":"2022-07-25T11:22:07.604691Z","shell.execute_reply.started":"2022-07-25T11:22:07.316558Z","shell.execute_reply":"2022-07-25T11:22:07.603631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic[['Fare']].plot(kind='hist',bins=100,alpha=0.5, title = 'Fare distribution on the titanic')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:07.606541Z","iopub.execute_input":"2022-07-25T11:22:07.606989Z","iopub.status.idle":"2022-07-25T11:22:07.972955Z","shell.execute_reply.started":"2022-07-25T11:22:07.606948Z","shell.execute_reply":"2022-07-25T11:22:07.971862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you want to have a smoother curve you can use kde, which stands for kernel density estimation (which smoothes the curve by estimating the probability distrbution of the variable)","metadata":{}},{"cell_type":"code","source":"titanic[\"Age\"].plot.kde(title = 'age density on the titanic ')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:07.974276Z","iopub.execute_input":"2022-07-25T11:22:07.974570Z","iopub.status.idle":"2022-07-25T11:22:08.185670Z","shell.execute_reply.started":"2022-07-25T11:22:07.974542Z","shell.execute_reply":"2022-07-25T11:22:08.184767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2.3.\"></a>\n## **<center><span style=\"color:#FF7B5F;\">2.3. Visualizing multiple variables</span></center>**","metadata":{}},{"cell_type":"markdown","source":"Often you want to see how two or more variables are distributed compared to each other and see what is the density of points accross two axes. For this, you can use scatter_matrix which does a scatter plot accross two axes. ","metadata":{}},{"cell_type":"code","source":"pd.plotting.scatter_matrix(titanic[['Age', 'Fare', 'SibSp' ]], diagonal='kde')\nplt.suptitle('Age, Fare and SibSp scatter Matrix')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:08.188677Z","iopub.execute_input":"2022-07-25T11:22:08.188987Z","iopub.status.idle":"2022-07-25T11:22:08.728585Z","shell.execute_reply.started":"2022-07-25T11:22:08.188961Z","shell.execute_reply":"2022-07-25T11:22:08.727520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If your data gets too big, you can use an hexagonal plot for which the indicates the density of points. ","metadata":{}},{"cell_type":"code","source":"titanic.plot.hexbin(x=\"Age\", y=\"Fare\", gridsize=10, title = 'density plot of records by Age and fare')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:08.730008Z","iopub.execute_input":"2022-07-25T11:22:08.730523Z","iopub.status.idle":"2022-07-25T11:22:08.975191Z","shell.execute_reply.started":"2022-07-25T11:22:08.730491Z","shell.execute_reply":"2022-07-25T11:22:08.974431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Also you can use seaborn to plot multiple distribution plots depending on a category. ","metadata":{}},{"cell_type":"code","source":"sns.displot(titanic, x=\"Age\", hue=\"Survived\", kind=\"kde\").set(title='Age density split by survived')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:08.976305Z","iopub.execute_input":"2022-07-25T11:22:08.976812Z","iopub.status.idle":"2022-07-25T11:22:09.373476Z","shell.execute_reply.started":"2022-07-25T11:22:08.976756Z","shell.execute_reply":"2022-07-25T11:22:09.372613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Also, parallel coordinate plot can be very useful to visualize clusters of data along a few axis   ","metadata":{}},{"cell_type":"code","source":"pd.plotting.parallel_coordinates(titanic[['Age','Parch', 'SibSp', 'Survived', 'Pclass']], \"Survived\", color=['lightblue', 'blue'])\nplt.suptitle('Parallel coordinate plot')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:09.374880Z","iopub.execute_input":"2022-07-25T11:22:09.375430Z","iopub.status.idle":"2022-07-25T11:22:10.853863Z","shell.execute_reply.started":"2022-07-25T11:22:09.375398Z","shell.execute_reply":"2022-07-25T11:22:10.852818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see here, the range of Age is much bigger than the other features so we can't really observe well the clusters. Let's normalize the features so we can better see the clusters.","metadata":{}},{"cell_type":"code","source":"# normalize numeric data from Titanic\nnumeric_titanic = titanic[['Age','Parch', 'SibSp', 'Survived', 'Pclass']]\nnumeric_titanic = (numeric_titanic-numeric_titanic.mean())/numeric_titanic.std()\nnumeric_titanic['Survived'] = titanic['Survived']\n# parallel coordinate plot with normalized data\npd.plotting.parallel_coordinates(numeric_titanic, \"Survived\", color=['lightblue', 'blue'])\nplt.suptitle('Parallel coordinate plot')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:10.855223Z","iopub.execute_input":"2022-07-25T11:22:10.855633Z","iopub.status.idle":"2022-07-25T11:22:12.270629Z","shell.execute_reply.started":"2022-07-25T11:22:10.855603Z","shell.execute_reply":"2022-07-25T11:22:12.269644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To observe clusters of multivariate data, you can also use pandas.plotting.radviz. ","metadata":{}},{"cell_type":"code","source":"pd.plotting.radviz(titanic[['Age','Parch', 'SibSp', 'Fare', 'Survived', 'Pclass']], \"Survived\", color=['lightblue', 'blue'])\nplt.suptitle('RadViz')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:12.271998Z","iopub.execute_input":"2022-07-25T11:22:12.272273Z","iopub.status.idle":"2022-07-25T11:22:12.764205Z","shell.execute_reply.started":"2022-07-25T11:22:12.272248Z","shell.execute_reply":"2022-07-25T11:22:12.762923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3\"></a>\n# **<center><span style=\"color:#FF7B5F;\">3. Joining Datasets</span></center>**","metadata":{}},{"cell_type":"markdown","source":"Lets join the two datasets, get the category id and then plot the top 10 categories that brought the most revenue. ","metadata":{}},{"cell_type":"code","source":"print(items.shape)\nitems.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:12.766635Z","iopub.execute_input":"2022-07-25T11:22:12.766991Z","iopub.status.idle":"2022-07-25T11:22:12.779240Z","shell.execute_reply.started":"2022-07-25T11:22:12.766960Z","shell.execute_reply":"2022-07-25T11:22:12.778026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sales.shape)\nsales.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:12.780185Z","iopub.execute_input":"2022-07-25T11:22:12.780560Z","iopub.status.idle":"2022-07-25T11:22:12.795271Z","shell.execute_reply.started":"2022-07-25T11:22:12.780530Z","shell.execute_reply":"2022-07-25T11:22:12.794199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, let's calculate the total price per invoice (its just the price * quantity)","metadata":{}},{"cell_type":"code","source":"sales['total_price'] = sales['item_price']* sales['item_cnt_day']","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:22:12.796558Z","iopub.execute_input":"2022-07-25T11:22:12.798026Z","iopub.status.idle":"2022-07-25T11:22:12.824872Z","shell.execute_reply.started":"2022-07-25T11:22:12.797981Z","shell.execute_reply":"2022-07-25T11:22:12.823728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, let's merge the datasets to have the categroy id on the sales dataset\n\nThis would be the equivalent of the SQL query: \n\n```SELECT * \nFROM sales_with_category_id swci\nLEFT JOIN sales s\n    ON swci.item_id = s.id```","metadata":{}},{"cell_type":"code","source":"sales_with_catogory_id = pd.merge(sales, items, left_on='item_id', right_on='item_id', how ='left')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:23:13.414399Z","iopub.execute_input":"2022-07-25T11:23:13.415450Z","iopub.status.idle":"2022-07-25T11:23:13.945440Z","shell.execute_reply.started":"2022-07-25T11:23:13.415411Z","shell.execute_reply":"2022-07-25T11:23:13.944473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's group by category to have the total_price per category. ","metadata":{}},{"cell_type":"code","source":"total_sales_per_category = sales_with_catogory_id[['item_category_id', 'total_price']].groupby('item_category_id').sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:23:15.200611Z","iopub.execute_input":"2022-07-25T11:23:15.201655Z","iopub.status.idle":"2022-07-25T11:23:15.541514Z","shell.execute_reply.started":"2022-07-25T11:23:15.201603Z","shell.execute_reply":"2022-07-25T11:23:15.540158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's plot the top item categories and their associated price","metadata":{}},{"cell_type":"code","source":"total_sales_per_category.sort_values(by='total_price', ascending=False).head(10).plot.bar(title = 'total sales per category')","metadata":{"execution":{"iopub.status.busy":"2022-07-25T11:23:17.129432Z","iopub.execute_input":"2022-07-25T11:23:17.130333Z","iopub.status.idle":"2022-07-25T11:23:17.343970Z","shell.execute_reply.started":"2022-07-25T11:23:17.130294Z","shell.execute_reply":"2022-07-25T11:23:17.342963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hope you liked the article. If you need, feel free to upvote this notebook! Happy learning!!","metadata":{}}]}