{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Remember!!

\n\n###
“The goal of forecasting is not to predict the future but to tell you what you need to know to take meaningful action in the present”.
-Paul Saffo-
","metadata":{}},{"cell_type":"markdown","source":"

🦉FUTUR SALES PREDICTION & TIME SERIES ANALYSIS 🦉

\n
","metadata":{}},{"cell_type":"markdown","source":"# 1 | INTRODUCTION\n## **🎯Objective** \nIn this competition, our task is to forecast the total amount of products sold in every shop. So, we'll creat a robust model based on times series.\n# **📚File descriptions**\n* **sales_train.csv** - the training set. Daily historical data from January 2013 to October 2015.\n* **test.csv** - the test set. You need to forecast the sales for these shops and products for November 2015.\n* **sample_submission.csv** - a sample submission file in the correct format.\n* **items.csv** - supplemental information about the items/products.\n* **item_categories.csv** - supplemental information about the items categories.\n* **shops.csv**- supplemental information about the shops.\n\n# **📁Data fields**\n* **ID** - an Id that represents a (Shop, Item) tuple within the test set\n* **shop_id** - unique identifier of a shop\n* **item_id** - unique identifier of a product\n* **item_category_id** - unique identifier of item category\n* **item_cnt_day** - number of products sold. You are predicting a monthly amount of this measure\n* **item_price** - current price of an item\n* **date** - date in format dd/mm/yyyy\n* **date_block_num** - a consecutive month number, used for convenience. January 2013 is 0, February 2013 is 1,..., October 2015 is 33\n* **item_name** - name of item\n* **shop_name** - name of shop\n* **item_category_name** - name of item category\n\n# **📈What is time series analysis?**\nTime series analysis is a specific way of analyzing a sequence of data points collected over an interval of time. In time series analysis, analysts record data points at consistent intervals over a set period of time rather than just recording the data points intermittently or randomly.","metadata":{}},{"cell_type":"markdown","source":"# 2 | DATA LOADING AND OVERIEW\n\n
\n

2.1 | Import Necessary Librairies 📕

\n ","metadata":{}},{"cell_type":"code","source":"!pip install pmdarima","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:45:43.154744Z","iopub.execute_input":"2022-07-07T16:45:43.155131Z","iopub.status.idle":"2022-07-07T16:45:51.953371Z","shell.execute_reply.started":"2022-07-07T16:45:43.1551Z","shell.execute_reply":"2022-07-07T16:45:51.952201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython import display\nfrom wordcloud import WordCloud\nimport datetime \nimport matplotlib as mpl\nfrom statsmodels.tsa.seasonal import seasonal_decompose\nfrom statsmodels.tsa.stattools import adfuller\nfrom statsmodels.graphics.tsaplots import plot_acf, plot_pacf\nfrom pmdarima.arima import ARIMA\nfrom pmdarima.arima import auto_arima\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\nimport warnings\nwarnings.filterwarnings('always')\nwarnings.filterwarnings('ignore')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-07T16:45:51.957889Z","iopub.execute_input":"2022-07-07T16:45:51.958297Z","iopub.status.idle":"2022-07-07T16:45:51.969488Z","shell.execute_reply.started":"2022-07-07T16:45:51.958251Z","shell.execute_reply":"2022-07-07T16:45:51.968216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

2.2 | Loading data 🎨

","metadata":{}},{"cell_type":"code","source":"item_categories=pd.read_csv(\"../input/competitive-data-science-predict-future-sales/item_categories.csv\")\nitems=pd.read_csv(\"../input/competitive-data-science-predict-future-sales/items.csv\")\nshops=pd.read_csv(\"../input/competitive-data-science-predict-future-sales/shops.csv\")\nsales_train=pd.read_csv(\"../input/competitive-data-science-predict-future-sales/sales_train.csv\")\ntest=pd.read_csv(\"../input/competitive-data-science-predict-future-sales/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:45:51.971047Z","iopub.execute_input":"2022-07-07T16:45:51.971476Z","iopub.status.idle":"2022-07-07T16:45:53.221482Z","shell.execute_reply.started":"2022-07-07T16:45:51.97133Z","shell.execute_reply":"2022-07-07T16:45:53.220327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

2.3 | Displaying Data Frames 🧩

","metadata":{}},{"cell_type":"code","source":"styles = [dict(selector=\"caption\", props=[(\"font-size\", \"120%\"),\n (\"font-weight\", \"bold\"),(\"background-color\", \"cyan\"),(\"color\",\"black\"),(\"text-align\",\"center\")])]\n# displaying the DataFrame item_categories\ndisplay.display(item_categories.head(6).style.background_gradient(cmap='Reds').set_properties(**{'font-family': 'Segoe UI'}).hide_index() \n.set_caption('item_categories dataset & summary').set_table_styles(styles))\nitem_categories.info()\n# displaying the DataFrame items\ndisplay.display(items.head(6).style.background_gradient(cmap='Reds').set_properties(**{'font-family': 'Segoe UI'}).hide_index() .set_caption('items dataset & summary').set_table_styles(styles))\nitems.info()\n# displaying the DataFrame shops\ndisplay.display(shops.head(6).style.background_gradient(cmap='Reds').set_properties(**{'font-family': 'Segoe UI'}).hide_index().set_caption('shops dataset & summary').set_table_styles(styles))\nshops.info()\n# displaying the DataFrame sales_train\ndisplay.display(sales_train.head(6).style.background_gradient(cmap='Reds').set_properties(**{'font-family': 'Segoe UI'}).hide_index().set_caption('sales_train dataset & summary').set_table_styles(styles))\nsales_train.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:45:53.223694Z","iopub.execute_input":"2022-07-07T16:45:53.224029Z","iopub.status.idle":"2022-07-07T16:45:53.317483Z","shell.execute_reply.started":"2022-07-07T16:45:53.223999Z","shell.execute_reply":"2022-07-07T16:45:53.316426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Looking 👀 to the results above we see that:\n* All datasets don't have missing values.\n* There are 22170 items and 60 shops.\n* Each item belongs to one of the 84 categories. ","metadata":{}},{"cell_type":"markdown","source":"# 3 | EDA AND TIME SERIES ANALYSIS\n\n
\n

3.1 | Exploratory data analysis 📊

\n
","metadata":{}},{"cell_type":"markdown","source":"
\n

Merging datasets

\n
","metadata":{}},{"cell_type":"code","source":"#merge the dataset \"item_categories\" with the dataset \"items\"\nitems_merged=pd.merge(item_categories,items, how='inner')\n#merge the dataset \"sales_train\" with the dataset \"shops\"\nsales_train_merged=pd.merge(sales_train,shops,on='shop_id')\n#merge the dataset \"sales_train_merged\" with the dataset \"items_merged\"\nsales_train_merged=pd.merge(sales_train_merged,items_merged,on='item_id')\nsales_train_merged.head().style.background_gradient(cmap='Blues').set_properties(**{'font-family': 'Segoe UI'})","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:45:53.31871Z","iopub.execute_input":"2022-07-07T16:45:53.319006Z","iopub.status.idle":"2022-07-07T16:45:54.341045Z","shell.execute_reply.started":"2022-07-07T16:45:53.318979Z","shell.execute_reply":"2022-07-07T16:45:54.339873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

WordCloud of items

\n
","metadata":{}},{"cell_type":"code","source":"# creating the text variable\ntext1 = \" \".join(title for title in sales_train_merged.item_name)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:45:54.3427Z","iopub.execute_input":"2022-07-07T16:45:54.343023Z","iopub.status.idle":"2022-07-07T16:45:54.920475Z","shell.execute_reply.started":"2022-07-07T16:45:54.342993Z","shell.execute_reply":"2022-07-07T16:45:54.919398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating word_cloud with text as argument in .generate() method\n\nword_cloud1 = WordCloud(collocations = False, background_color = 'white',\n width = 2048, height = 1080).generate(text1)\n# saving the image\nword_cloud1.to_file('got.png')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:45:54.922041Z","iopub.execute_input":"2022-07-07T16:45:54.922435Z","iopub.status.idle":"2022-07-07T16:46:21.765895Z","shell.execute_reply.started":"2022-07-07T16:45:54.922399Z","shell.execute_reply":"2022-07-07T16:46:21.764696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the generated Word Cloud\nplt.figure(figsize=[15,10])\nplt.imshow(word_cloud1, interpolation='bilinear')\nplt.axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:21.767208Z","iopub.execute_input":"2022-07-07T16:46:21.767614Z","iopub.status.idle":"2022-07-07T16:46:22.827191Z","shell.execute_reply.started":"2022-07-07T16:46:21.767582Z","shell.execute_reply":"2022-07-07T16:46:22.825525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Number of items in each categorie

\n
","metadata":{}},{"cell_type":"code","source":"result=items_merged['item_category_name'].value_counts().sort_values(ascending=False)[0:20]\nresult.plot(kind='bar',figsize=(12,5),width = 0.8,color=sns.color_palette(\"Spectral\", 9))\nplt.title(\"Number of items in each category\")\nplt.ylabel('Number of items')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:22.828806Z","iopub.execute_input":"2022-07-07T16:46:22.829153Z","iopub.status.idle":"2022-07-07T16:46:23.137058Z","shell.execute_reply.started":"2022-07-07T16:46:22.829119Z","shell.execute_reply":"2022-07-07T16:46:23.13587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**
📌 As we can see : most of items belong to the category kИHO-DVD
**","metadata":{}},{"cell_type":"markdown","source":"
\n

Number of products sold in each category

\n
","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,5))\nplt.title('top categories')\nplt.ylabel('item_cnt_day')\nsales_train_merged.groupby('item_category_name')['item_cnt_day'].sum().sort_values(ascending=False)[0:15].plot(kind='line', marker='*', color='red', ms=10)\nsales_train_merged.groupby('item_category_name')['item_cnt_day'].sum().sort_values(ascending=False)[0:15].plot(kind='bar',color=sns.color_palette(\"inferno_r\", 7))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:23.14074Z","iopub.execute_input":"2022-07-07T16:46:23.141066Z","iopub.status.idle":"2022-07-07T16:46:24.037437Z","shell.execute_reply.started":"2022-07-07T16:46:23.141034Z","shell.execute_reply":"2022-07-07T16:46:24.036332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**
📌 As we can see : most of sold products belong to the category kИHO-DVD
**","metadata":{}},{"cell_type":"markdown","source":"
\n

Number of products sold in each Shop

\n
","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,5))\nplt.title('top shops')\nplt.ylabel('item_cnt_day')\nsales_train_merged.groupby('shop_name')['item_cnt_day'].sum().sort_values(ascending=False)[0:10].plot(kind='line', marker='*', color='red', ms=10)\nsales_train_merged.groupby('shop_name')['item_cnt_day'].sum().sort_values(ascending=False)[0:10].plot(kind='bar',color=sns.color_palette(\"viridis\", 100))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:24.039651Z","iopub.execute_input":"2022-07-07T16:46:24.040479Z","iopub.status.idle":"2022-07-07T16:46:24.927564Z","shell.execute_reply.started":"2022-07-07T16:46:24.040431Z","shell.execute_reply":"2022-07-07T16:46:24.926469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**
📌 As we can see : most of sold products are in the shop Москва МТРЦ \"Афи Молл\"
**","metadata":{}},{"cell_type":"markdown","source":"
\n

item_price in each category

\n
","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,5))\nplt.title('top categories')\nplt.ylabel('item_price')\nsales_train_merged.groupby('item_category_name')['item_price'].mean().sort_values(ascending=True)[70:84].plot(kind='line', marker='*', color='red', ms=10)\nsales_train_merged.groupby('item_category_name')['item_price'].mean().sort_values(ascending=True)[70:84].plot(kind='bar',color=sns.color_palette(\"inferno_r\", 7))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:24.92895Z","iopub.execute_input":"2022-07-07T16:46:24.929308Z","iopub.status.idle":"2022-07-07T16:46:25.794117Z","shell.execute_reply.started":"2022-07-07T16:46:24.929275Z","shell.execute_reply":"2022-07-07T16:46:25.793009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**
📌 As we can see : The category Игровые консоли - PS4 is the most expensive category
**","metadata":{}},{"cell_type":"markdown","source":"
\n

3.2 | Time series analysis ⏱

\n
","metadata":{}},{"cell_type":"markdown","source":"
\n

Visualisation of daily sold products

\n
","metadata":{}},{"cell_type":"code","source":"# Convert the date column to a datetime type\nsales_train['date'] = sales_train['date'].apply(lambda x:datetime.datetime.strptime(x, '%d.%m.%Y'))\n#save a copy of sales_train\ndf_train=sales_train.copy()\n# Set the date column as the index of DataFrame \nsales_train=sales_train.set_index('date')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:25.795905Z","iopub.execute_input":"2022-07-07T16:46:25.796408Z","iopub.status.idle":"2022-07-07T16:46:49.669222Z","shell.execute_reply.started":"2022-07-07T16:46:25.796363Z","shell.execute_reply":"2022-07-07T16:46:49.667986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Plot the time series \n# Use the ggplot style\nplt.style.use('ggplot')\nax2 = sales_train['item_cnt_day'].plot(figsize=(16,6),color='blue')\n# Set the title\nax2.set_title('item_cnt_day plot');\n# Specify the x-axis label \nax2.set_xlabel('Date')\n# Specify the y-axis label \nax2.set_ylabel('item_cnt_day')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:46:49.670526Z","iopub.execute_input":"2022-07-07T16:46:49.670843Z","iopub.status.idle":"2022-07-07T16:47:02.52018Z","shell.execute_reply.started":"2022-07-07T16:46:49.670814Z","shell.execute_reply":"2022-07-07T16:47:02.519015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=2, ncols=3, figsize=(18, 10))\nsales_train.loc['2013','item_cnt_day'].resample('M').plot(ax=ax[0,0])\nsales_train.loc['2014','item_cnt_day'].resample('M').plot(ax=ax[0,1])\nsales_train.loc['2015','item_cnt_day'].resample('M').plot(ax=ax[0,2])\nsales_train.loc['2013','item_cnt_day'].resample('M').sum().plot( ax=ax[1,0],color='blue')\nsales_train.loc['2014','item_cnt_day'].resample('M').sum().plot( ax=ax[1,1],color='red')\nsales_train.loc['2015','item_cnt_day'].resample('M').sum().plot( ax=ax[1,2],color='green')\n \nplt.suptitle('item_cnt_day from 2013 to 2015',fontsize=20)\nplt.subplots_adjust(left=0.1,\n bottom=0.1, \n right=0.9, \n top=0.92, \n wspace=0.4, \n hspace=0.4)\n\nfig.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:02.521734Z","iopub.execute_input":"2022-07-07T16:47:02.522775Z","iopub.status.idle":"2022-07-07T16:47:17.475079Z","shell.execute_reply.started":"2022-07-07T16:47:02.522735Z","shell.execute_reply":"2022-07-07T16:47:17.473969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Visualisation of monthly sold products

\n
","metadata":{}},{"cell_type":"code","source":"df_train['month'] = df_train['date'].dt.to_period('M')\ndf_train['month'] = df_train['month'].astype(str)\ndff2=df_train.copy()\ndf_train['month'] = pd.to_datetime(df_train['month'])","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:17.47663Z","iopub.execute_input":"2022-07-07T16:47:17.477352Z","iopub.status.idle":"2022-07-07T16:47:30.88101Z","shell.execute_reply.started":"2022-07-07T16:47:17.477306Z","shell.execute_reply":"2022-07-07T16:47:30.879809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dff_train = df_train.groupby(['month']).agg({'item_cnt_day':'sum'})\ndff_train['month'] = dff_train.index\ndff_train.drop(['month'],axis=1,inplace=True)\ndff_train.rename(columns = {'item_cnt_day':'item_cnt_month'}, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:30.882757Z","iopub.execute_input":"2022-07-07T16:47:30.883733Z","iopub.status.idle":"2022-07-07T16:47:30.950132Z","shell.execute_reply.started":"2022-07-07T16:47:30.883685Z","shell.execute_reply":"2022-07-07T16:47:30.949035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dff2=dff2.groupby(['month']).agg({'item_cnt_day':'sum'})\ndff2['month'] = dff2.index\ndff2.rename(columns = {'item_cnt_day':'item_cnt_month'}, inplace = True)\n# Get the Peaks and Troughs\ndata = dff2['item_cnt_month'].values\ndoublediff = np.diff(np.sign(np.diff(data)))\npeak_locations = np.where(doublediff == -2)[0] + 1\n\ndoublediff2 = np.diff(np.sign(np.diff(-1*data)))\ntrough_locations = np.where(doublediff2 == -2)[0] + 1\n\n# Draw Plot\nplt.figure(figsize=(17,7))\nplt.plot('month', 'item_cnt_month', data=dff2, color='blue', label='Air Traffic')\nplt.scatter(dff2.month[peak_locations], dff2.item_cnt_month[peak_locations], marker=mpl.markers.CARETUPBASE, color='green', s=100, label='Peaks')\nplt.scatter(dff2.month[trough_locations], dff2.item_cnt_month[trough_locations], marker=mpl.markers.CARETDOWNBASE, color='red', s=100, label='Troughs')\n\n# Annotate\nfor t, p in zip(trough_locations[1::5], peak_locations[::3]):\n plt.text(dff2.month[p], dff2.item_cnt_month[p]+15, dff2.month[p], horizontalalignment='center', color='darkgreen')\n plt.text(dff2.month[t], dff2.item_cnt_month[t]-35, dff2.month[t], horizontalalignment='center', color='darkred')\n\n# Decoration\nxtick_location = dff2.index.tolist()[::6]\nxtick_labels = dff2.month.tolist()[::6]\nplt.xticks(ticks=xtick_location, labels=xtick_labels, rotation=90, fontsize=12, alpha=.7)\nplt.title(\"item_cnt_month (2013 - 2015)\", fontsize=22)\nplt.yticks(fontsize=12, alpha=.7)\n\n# Lighten borders\nplt.gca().spines[\"top\"].set_alpha(.0)\nplt.gca().spines[\"bottom\"].set_alpha(.3)\nplt.gca().spines[\"right\"].set_alpha(.0)\nplt.gca().spines[\"left\"].set_alpha(.3)\n\nplt.legend(loc='upper left')\nplt.grid(axis='y', alpha=.3)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:30.951651Z","iopub.execute_input":"2022-07-07T16:47:30.951967Z","iopub.status.idle":"2022-07-07T16:47:31.418889Z","shell.execute_reply.started":"2022-07-07T16:47:30.951938Z","shell.execute_reply":"2022-07-07T16:47:31.417747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Seasonality, Trend and Noise

\n
","metadata":{}},{"cell_type":"code","source":"decomposition=seasonal_decompose(dff_train, model = 'additive')\n# Extract the trend and seasonal components\ntrend = decomposition.trend\nseasonal = decomposition.seasonal\nsales_decomposed = pd.DataFrame(np.c_[trend, seasonal], index=dff_train.index, columns=['trend', 'seasonal'])","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:31.420448Z","iopub.execute_input":"2022-07-07T16:47:31.420882Z","iopub.status.idle":"2022-07-07T16:47:31.431421Z","shell.execute_reply.started":"2022-07-07T16:47:31.420839Z","shell.execute_reply":"2022-07-07T16:47:31.43032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the values of the sales_decomposed DataFrame\nax = sales_decomposed.plot(figsize=(12, 6), fontsize=15)\n\n# Specify axis labels\nax.set_xlabel('Date', fontsize=15)\nplt.legend(fontsize=15)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:31.432923Z","iopub.execute_input":"2022-07-07T16:47:31.433365Z","iopub.status.idle":"2022-07-07T16:47:31.666251Z","shell.execute_reply.started":"2022-07-07T16:47:31.433324Z","shell.execute_reply":"2022-07-07T16:47:31.665027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# seasonal_decompose for additive model\nseasonal_decompose(dff_train, model = 'additive').plot().set_size_inches(10, 8)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:31.667614Z","iopub.execute_input":"2022-07-07T16:47:31.667934Z","iopub.status.idle":"2022-07-07T16:47:32.288109Z","shell.execute_reply.started":"2022-07-07T16:47:31.667904Z","shell.execute_reply":"2022-07-07T16:47:32.286855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# seasonal_decompose for multiplicative model\nseasonal_decompose(dff_train, model = 'multiplicative').plot().set_size_inches(10, 8)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:32.289595Z","iopub.execute_input":"2022-07-07T16:47:32.290596Z","iopub.status.idle":"2022-07-07T16:47:32.925667Z","shell.execute_reply.started":"2022-07-07T16:47:32.29056Z","shell.execute_reply":"2022-07-07T16:47:32.924399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4 | Futur Sales Forecasting with SARIMA model","metadata":{}},{"cell_type":"markdown","source":"
\n

What is SARIMA?

\n
\nSeasonal Autoregressive Integrated Moving Average, SARIMA or Seasonal ARIMA, is an extension of ARIMA that explicitly supports univariate time series data with a seasonal component.\n\nIt adds three new hyperparameters to specify the autoregression (AR), differencing (I) and moving average (MA) for the seasonal component of the series, as well as an additional parameter for the period of the seasonality.\n
\n

How to Configure SARIMA

\n
\nConfiguring a SARIMA requires selecting hyperparameters for both the trend and seasonal elements of the series.
\n\n
\n

Trend Elements

\n
\n\nThere are three trend elements that require configuration.\n\nThey are the same as the ARIMA model; specifically:\n\n* **p**: Trend autoregression order.\n* **d**: Trend difference order.\n* **q**: Trend moving average order.\n\n
\n

Seasonal Elements

\n
\n\nThere are four seasonal elements that are not part of ARIMA that must be configured; they are:\n\n* **P**: Seasonal autoregressive order.\n* **D**: Seasonal difference order.\n* **Q**: Seasonal moving average order.\n* **m**: The number of time steps for a single seasonal period.","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"
\n

4.1 | Checks for Stationarity of the time serie

\n
","metadata":{}},{"cell_type":"markdown","source":" There are many methods to check whether a time series (direct observations, residuals, otherwise) is stationary or non-stationary. Here, i will use Augmented Dickey Fuller test.
\nThe null hypothesis of the ADF test is that the time series is non-stationary. So, if the p-value of the test is less than the significance level (0.05) then you reject the null hypothesis and infer that the time series is indeed stationary.","metadata":{}},{"cell_type":"code","source":"result = adfuller(dff_train)\nprint('ADF Statistic: %f' % result[0])\nprint('p-value: %f' % result[1])\nprint('Critical Values:')\nfor key, value in result[4].items():\n print('\\t%s: %.3f' % (key, value))","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:32.927503Z","iopub.execute_input":"2022-07-07T16:47:32.927865Z","iopub.status.idle":"2022-07-07T16:47:32.940976Z","shell.execute_reply.started":"2022-07-07T16:47:32.927833Z","shell.execute_reply":"2022-07-07T16:47:32.939795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Since the p-value is greater than the critical value of 0.05, we can statistically confirm that the series is not stationary. To make this time serie stationnary with need to do differencing.","metadata":{}},{"cell_type":"markdown","source":"
\n

4.2 | Differencing

\n
","metadata":{}},{"cell_type":"markdown","source":"Almost by definition, it may be necessary to examine differenced data when we have seasonality. Seasonality usually causes the series to be nonstationary because the average values at some particular times within the seasonal span (months, for example) may be different than the average values at other times.","metadata":{}},{"cell_type":"markdown","source":"
\n

Seasonal differencing

\n
","metadata":{}},{"cell_type":"code","source":"sd_dff_train=dff_train - dff_train.shift(12)\nsd_dff_train = sd_dff_train.dropna()\nsd_dff_train.plot(figsize=(12,6))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:32.942036Z","iopub.execute_input":"2022-07-07T16:47:32.942392Z","iopub.status.idle":"2022-07-07T16:47:33.134524Z","shell.execute_reply.started":"2022-07-07T16:47:32.942332Z","shell.execute_reply":"2022-07-07T16:47:33.133718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#ADF test\nresult = adfuller(sd_dff_train)\nprint('ADF Statistic: %f' % result[0])\nprint('p-value: %f' % result[1])","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:33.13572Z","iopub.execute_input":"2022-07-07T16:47:33.136213Z","iopub.status.idle":"2022-07-07T16:47:33.14622Z","shell.execute_reply.started":"2022-07-07T16:47:33.136183Z","shell.execute_reply":"2022-07-07T16:47:33.145214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" The p-value is less than the critical value of 0.05. Hence we can confirm that the series is now stationary. We can say that D=1.","metadata":{}},{"cell_type":"markdown","source":"
\n

4.3 | Autocorrelation in time series data

\n
","metadata":{}},{"cell_type":"markdown","source":" In order to figure out the parameters p,d,q,P,D,Q of SARIMA model we would need to plot the ACF and PACF plots.
ACF stands for Auto Correlation Function and PACF stands for Partial Auto Correlation Function.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(16, 5))\nplot_acf(dff_train,lags=10, ax=axes[0])\nplot_pacf(dff_train,lags=10, ax=axes[1])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:33.147902Z","iopub.execute_input":"2022-07-07T16:47:33.148894Z","iopub.status.idle":"2022-07-07T16:47:33.397328Z","shell.execute_reply.started":"2022-07-07T16:47:33.14885Z","shell.execute_reply":"2022-07-07T16:47:33.396237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

4.4 | Parameters estimation & model building

\n
","metadata":{}},{"cell_type":"code","source":"model=auto_arima(dff_train, start_p = 0, start_q = 0,D=1, m = 12, seasonal = True, test = \"adf\", trace = True, alpha = 0.05, information_criterion = 'aic', suppress_warnings = True, \n stepwise = True)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:33.398746Z","iopub.execute_input":"2022-07-07T16:47:33.399293Z","iopub.status.idle":"2022-07-07T16:47:36.834526Z","shell.execute_reply.started":"2022-07-07T16:47:33.399242Z","shell.execute_reply":"2022-07-07T16:47:36.833312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" According to the algorithm the best SARIMA model is SARIMA(0,2,1)(0,1,1)[12] ","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:36.842429Z","iopub.execute_input":"2022-07-07T16:47:36.84353Z","iopub.status.idle":"2022-07-07T16:47:36.871979Z","shell.execute_reply.started":"2022-07-07T16:47:36.843469Z","shell.execute_reply":"2022-07-07T16:47:36.870656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### According to the model summary:\n\n * Because the p-value of the Ljung-Box test is greater than 0.05, we cannot reject the null hypothesis that the residuals are independent.\n * Because the p-value of the heteroskedasticity test is grater than 0.05, we fail to reject the null hypothesis the null hypothesis of Homoscedasticity.\n * Because the p-value of the Jarque Bera test is grater than 0.05, we fail to reject the null hypothesis and conclude that the sample data follows normal distribution.\n \n**We conclude that residuals form a white noise, so the the model is good and can be used for prediction.**\n ","metadata":{}},{"cell_type":"markdown","source":"
\n

4.5 | Diagnostics of residuals

\n
","metadata":{}},{"cell_type":"code","source":"model.plot_diagnostics(figsize=(14,10))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:36.87756Z","iopub.execute_input":"2022-07-07T16:47:36.878602Z","iopub.status.idle":"2022-07-07T16:47:37.451135Z","shell.execute_reply.started":"2022-07-07T16:47:36.87854Z","shell.execute_reply":"2022-07-07T16:47:37.450331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

4.6 | Forcasting

\n
","metadata":{}},{"cell_type":"code","source":"prediction, confint = model.predict(n_periods = 6, return_conf_int = True) #95% CI default\nperiod_index = pd.period_range(start = dff_train.index[-1], periods = 6, freq='M')\nforecast = pd.DataFrame({'Predicted item_cnt_month': prediction.round(2)}, index = period_index)\nforecast","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:37.452164Z","iopub.execute_input":"2022-07-07T16:47:37.453125Z","iopub.status.idle":"2022-07-07T16:47:37.471327Z","shell.execute_reply.started":"2022-07-07T16:47:37.453041Z","shell.execute_reply":"2022-07-07T16:47:37.470057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cf= pd.DataFrame(confint)\nprediction_series = pd.Series(prediction,index=period_index)\nplt.figure(figsize=(15, 5))\nplt.plot(dff_train, color='red', label='Actual')\nplt.plot(prediction_series, color='orange', label='Predicted')\nplt.fill_between(prediction_series.index,\n cf[0],\n cf[1],color='grey',alpha=.2, label='Confidence Intervals Area')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:37.472328Z","iopub.execute_input":"2022-07-07T16:47:37.473096Z","iopub.status.idle":"2022-07-07T16:47:37.739488Z","shell.execute_reply.started":"2022-07-07T16:47:37.47306Z","shell.execute_reply":"2022-07-07T16:47:37.738625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5 | Submission","metadata":{}},{"cell_type":"code","source":"train_dfff = df_train.groupby(['shop_id', 'item_id'])['date', 'item_cnt_day'].agg({'item_cnt_day':'sum'})\ntrain_dfff = train_dfff.reset_index()\nprint(train_dfff)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:37.740564Z","iopub.execute_input":"2022-07-07T16:47:37.74163Z","iopub.status.idle":"2022-07-07T16:47:38.066861Z","shell.execute_reply.started":"2022-07-07T16:47:37.741595Z","shell.execute_reply":"2022-07-07T16:47:38.065967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['item_cnt_month'] = (prediction[0].round(2)*len(test)/len(train_dfff))/len(test)\nsubmission = test.drop(['shop_id', 'item_id'], axis = 1)\nprint(submission)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:38.067977Z","iopub.execute_input":"2022-07-07T16:47:38.068996Z","iopub.status.idle":"2022-07-07T16:47:38.079883Z","shell.execute_reply.started":"2022-07-07T16:47:38.06896Z","shell.execute_reply":"2022-07-07T16:47:38.078726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T16:47:38.081361Z","iopub.execute_input":"2022-07-07T16:47:38.081706Z","iopub.status.idle":"2022-07-07T16:47:38.541048Z","shell.execute_reply.started":"2022-07-07T16:47:38.081665Z","shell.execute_reply":"2022-07-07T16:47:38.539767Z"},"trusted":true},"execution_count":null,"outputs":[]}]}