{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
Sales Price Prediction Challenge - Step 1 - Load the data
\n
\n\"C1
","metadata":{}},{"cell_type":"markdown","source":"

If you are new to Kaggle or DS in general. Below is a break down of steps you need to follow



\n1. Data Import - Load the data

\n 2. Exploratory Data Analysis - Analyze and treat the data before training the model on it

\n3. Feature engineering - Pick the best variables and the less number of variable the better

\n4. Modelling technique - Find the best modelling technique that suits your problem statement.

\n5. Model evaluation - Apply prediction on test data using trained model

\n6. Iterate - Repeat, repeat and repeat above steps.

\n
\nRemember these steps for now. There are so many variations to it though.

\n\nFor now, we will explore the basic steps and you can explore more in upcoming notebooks.\n\nBut here is the good news !! We will take it sloooowly and to the point :)\nI will comment several codes so you can learn, DIY and if you are stuck, you can always uncomment and see the output.

\nJust remember the objective -> We have to predict future sales! \n\n@Credits : to this beautiful community.
","metadata":{}},{"cell_type":"markdown","source":"

Little bit about the competition


\n
\nBefore we begin to start code our way to this tutorial. its always important to understand the source of data. This data is provided by 1C company which is a russian software firm and we have to predict future sale of the firm based on the data we recieved.

Now that we have our goal fixed. Lets start step by step.

\nFirst things first, lets load the libraries and data. We will use Pandas library for it. Its super easy with Pandas, start by importing the libraries
","metadata":{}},{"cell_type":"code","source":"import pandas as pd ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load items data\nitems = pd.read_csv(\"/kaggle/input/competitive-data-science-predict-future-sales/items.csv\")\n#load items category data\nitemscat = pd.read_csv(\"/kaggle/input/competitive-data-science-predict-future-sales/item_categories.csv\")\n#load shops data\nshops = pd.read_csv(\"/kaggle/input/competitive-data-science-predict-future-sales/shops.csv\")\n#load sales train data \ntrain = pd.read_csv(\"/kaggle/input/competitive-data-science-predict-future-sales/sales_train.csv\")\n#load sales test data \ntest = pd.read_csv(\"/kaggle/input/competitive-data-science-predict-future-sales/test.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

After the data is loaded, we need to understand if the data is loaded properly. How do we check? this is how

","metadata":{}},{"cell_type":"code","source":"\nprint(\"************************* items data ***************************\")\nprint(items.head(5))\n \n# Rest of the part \n# print(\"************************* items category data ***************************\")\n# print(itemscat.head(5))\n \n# print(\"************************* Shops data ***************************\")\n# print(shops.head(5))\n \n# print(\"************************* train data ***************************\")\n# print(train.head(5))\n \n# print(\"*************************test data ***************************\")\n# print(test.head(5))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Alternatively, you can also use shape method to get # of rows and columns","metadata":{}},{"cell_type":"code","source":"#get training data set dimensions\nprint(\"train data shape\", train.shape)\n#get testing data set dimensions\nprint(\"test data shape\", test.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nprint(\"************************* items data ***************************\")\nprint(items.shape)\n \n# Rest of the part \n# print(\"************************* items category data ***************************\")\n# print(itemscat.shape)\n \n# print(\"************************* Shops data ***************************\")\n# print(shops.shape)\n \n# print(\"************************* train data ***************************\")\n# print(train.shape\n \n# print(\"*************************test data ***************************\")\n# print(test.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Now that we have we have loaded the data and briefly looked at it. Let's poke the data further

","metadata":{}},{"cell_type":"code","source":"#get a list of all columns in firs table, Why you ask? so that we can use them for analysis. \nprint(\"********list of columns in items table*********\")\nprint(items.columns.tolist())\n#DIY, get list of rest of the columsn from data set \n#print(\"*********list of columns in items category table.*********\")\n#print(itemscat.columns.tolist())\n#print(\"*********list of columns in shops table.*********\")\n#print( shops.columns.tolist())\n#print(\"*********list of columns in train table.*********\")\n#print(train.columns.tolist())\n#print(\"*********list of columns in test table.*********\")\n#print(test.columns.tolist())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Now lets pick one variable from train dataset and take a look at its details

","metadata":{}},{"cell_type":"code","source":"#lets go back to the loaded datasets \ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

while analyzing any dataset, first look for the rows and columns structure i.e. look specifically at columns names and their data types and how many observations(records) are there. Lets take a look at that first

","metadata":{}},{"cell_type":"code","source":"#total number of records \nprint(\"Total # of records in train dataset \\n\" ,len(train))\n\n#DIY, \nprint(\"Total # of records in items dataset \\n\" ,len(items))\nprint(\"Total # of records in itemscat dataset \\n\" ,len(itemscat))\nprint(\"Total # of records in test dataset \\n\" ,len(test))\n#get datatypes \nprint(\"\\n\\nData types \\n\",train.dtypes)\n#or you could use info method that gives more details than dtypes method\nprint(\"\\n\\nData types \\n\",train.info())\n#DYI, you can experiment with rest of the datasets yourself. should be easy now!!! ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Sometimes you will find a data variable in original dataset in a different form than it currently is.

If you take a look closely at the date variable of train, the date format is object or string e.g. 02.01.2013 so we will convert it to date format to something like this 02-03-2021. Below is how we do it.\n

","metadata":{}},{"cell_type":"code","source":"print(\"Before converting date field\")\nprint(train.head())\ntrain[\"date\"]= pd.to_datetime(train[\"date\"])\nprint(\"After converting date field\")\nprint(train.head())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

hmm... lets see what can we do with this dataset. lets think about it for a second. We have variables like date, date_block_num, shop_id, item_id and item_cnt_day.

lets briefly look at this details of each variable. one of the key information we look at is how many unique values the variable have.

","metadata":{}},{"cell_type":"code","source":"#find all unique values in the variables of this dataset. \nprint(train.date.unique)\n#you see ? We can use the column name in the dataset property itself \n#'date_block_num', 'shop_id', 'item_id', 'item_price', 'item_cnt_day']\n#print(train.date_block_num.unique)\n#print(train.shop_id.unique)\n#print(train.item_price.unique)\n#print(train.item_cnt_day.unique)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

you might also want to check the minimum and maximum values in a variable

","metadata":{}},{"cell_type":"code","source":"print(train.item_price.min())\nprint(train.item_price.max())\n\n#DIY, for other variables and check the minimum and maximum and next we have to understand why","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

what if you have to find all basic statistics like minimum, maximum, mean, mode, median ... all percentiles? this would take forever for all the variables.

\n\nLuckily we have a few functions that provide that statistics. One such method of info(). lets take a look at how it works and what info does it provide.

","metadata":{}},{"cell_type":"code","source":"train.info()\n# alternatively you can also use train.dtypes which will give you almost same info. Check that one too. ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# you can also check individual variable details by using individual variables with describe method like this \ntrain.item_price.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

*** Tips and tricks ***

\n

Did you notice the count for item_price variable ?.e.g. 2.935849e+06 is scientific notation. it isnt really clear what values does it contains

\n

how do we convert this output to a more readable and clean format

\nwe just need to suppress this using one of the pandas display option.

","metadata":{}},{"cell_type":"code","source":"pd.options.display.float_format = '{:.2f}'.format","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.item_price.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n In this kernel, we learned how to ingest the data and look at the basic sanity testing of data.

\n We will work on this one with few more ideas of sanity check on data.

\n Next upcoming is Step 2 EDA kernel in which we will deep dive more in to data we just imported in this kernel. We will continue from here. \n

\n

Stay tuned. :)

","metadata":{}}]}