{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Time Series](https://www.kaggle.com/learn/time-series) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/seasonality).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nRun this cell to set everything up!","metadata":{}},{"cell_type":"code","source":"# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.time_series.ex3 import *\n\n# Setup notebook\nfrom pathlib import Path\nfrom learntools.time_series.style import *  # plot style settings\nfrom learntools.time_series.utils import plot_periodogram, seasonal_plot\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport seaborn as sns\nfrom sklearn.linear_model import LinearRegression\nfrom statsmodels.tsa.deterministic import CalendarFourier, DeterministicProcess\n\n\ncomp_dir = Path('../input/store-sales-time-series-forecasting')\n\nholidays_events = pd.read_csv(\n    comp_dir / \"holidays_events.csv\",\n    dtype={\n        'type': 'category',\n        'locale': 'category',\n        'locale_name': 'category',\n        'description': 'category',\n        'transferred': 'bool',\n    },\n    parse_dates=['date'],\n    infer_datetime_format=True,\n)\nholidays_events = holidays_events.set_index('date').to_period('D')\n\nstore_sales = pd.read_csv(\n    comp_dir / 'train.csv',\n    usecols=['store_nbr', 'family', 'date', 'sales'],\n    dtype={\n        'store_nbr': 'category',\n        'family': 'category',\n        'sales': 'float32',\n    },\n    parse_dates=['date'],\n    infer_datetime_format=True,\n)\nstore_sales['date'] = store_sales.date.dt.to_period('D')\nstore_sales = store_sales.set_index(['store_nbr', 'family', 'date']).sort_index()\naverage_sales = (\n    store_sales\n    .groupby('date').mean()\n    .squeeze()\n    .loc['2017']\n)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:24:57.633821Z","iopub.execute_input":"2022-08-14T09:24:57.634193Z","iopub.status.idle":"2022-08-14T09:25:03.140486Z","shell.execute_reply.started":"2022-08-14T09:24:57.634105Z","shell.execute_reply":"2022-08-14T09:25:03.139091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\nExamine the following seasonal plot:","metadata":{}},{"cell_type":"code","source":"X = average_sales.to_frame()\nX[\"week\"] = X.index.week\nX[\"day\"] = X.index.dayofweek\nseasonal_plot(X, y='sales', period='week', freq='day');","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:03.142461Z","iopub.execute_input":"2022-08-14T09:25:03.142915Z","iopub.status.idle":"2022-08-14T09:25:03.739113Z","shell.execute_reply.started":"2022-08-14T09:25:03.142890Z","shell.execute_reply":"2022-08-14T09:25:03.738164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And also the periodogram:","metadata":{}},{"cell_type":"code","source":"plot_periodogram(average_sales);","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:08.518461Z","iopub.execute_input":"2022-08-14T09:25:08.518795Z","iopub.status.idle":"2022-08-14T09:25:08.925634Z","shell.execute_reply.started":"2022-08-14T09:25:08.518770Z","shell.execute_reply":"2022-08-14T09:25:08.924447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1) Determine seasonality\n\nWhat kind of seasonality do you see evidence of? Once you've thought about it, run the next cell for some discussion.","metadata":{}},{"cell_type":"code","source":"# View the solution (Run this cell to receive credit!)\nq_1.check()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:11.074277Z","iopub.execute_input":"2022-08-14T09:25:11.074967Z","iopub.status.idle":"2022-08-14T09:25:11.082500Z","shell.execute_reply.started":"2022-08-14T09:25:11.074934Z","shell.execute_reply":"2022-08-14T09:25:11.081862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\n# 2) Create seasonal features\n\nUse `DeterministicProcess` and `CalendarFourier` to create:\n- indicators for weekly seasons and\n- Fourier features of order 4 for monthly seasons.","metadata":{}},{"cell_type":"code","source":"y = average_sales.copy()\n\n# YOUR CODE HERE\ny = average_sales.copy()\n\nfourier = CalendarFourier(freq='M', order=4)\ndp = DeterministicProcess(\n    index=y.index,\n    constant=True,\n    order=1,\n    seasonal=True,\n    additional_terms=[fourier],\n    drop=True,\n)\nX = dp.in_sample()\n\n# Check your answer\nq_2.check()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:35.336857Z","iopub.execute_input":"2022-08-14T09:25:35.337233Z","iopub.status.idle":"2022-08-14T09:25:37.320419Z","shell.execute_reply.started":"2022-08-14T09:25:35.337206Z","shell.execute_reply":"2022-08-14T09:25:37.319716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_2.hint()\n# q_2.solution()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:40.656716Z","iopub.execute_input":"2022-08-14T09:25:40.657063Z","iopub.status.idle":"2022-08-14T09:25:40.661141Z","shell.execute_reply.started":"2022-08-14T09:25:40.657039Z","shell.execute_reply":"2022-08-14T09:25:40.660053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now run this cell to fit the seasonal model.","metadata":{}},{"cell_type":"code","source":"model = LinearRegression().fit(X, y)\ny_pred = pd.Series(\n    model.predict(X),\n    index=X.index,\n    name='Fitted',\n)\n\ny_pred = pd.Series(model.predict(X), index=X.index)\nax = y.plot(**plot_params, alpha=0.5, title=\"Average Sales\", ylabel=\"items sold\")\nax = y_pred.plot(ax=ax, label=\"Seasonal\")\nax.legend();","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:42.859538Z","iopub.execute_input":"2022-08-14T09:25:42.860052Z","iopub.status.idle":"2022-08-14T09:25:43.202850Z","shell.execute_reply.started":"2022-08-14T09:25:42.860027Z","shell.execute_reply":"2022-08-14T09:25:43.201780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n","metadata":{}},{"cell_type":"markdown","source":"Removing from a series its trend or seasons is called **detrending** or **deseasonalizing** the series.\n\nLook at the periodogram of the deseasonalized series.","metadata":{}},{"cell_type":"code","source":"y_deseason = y - y_pred\n\nfig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, sharey=True, figsize=(10, 7))\nax1 = plot_periodogram(y, ax=ax1)\nax1.set_title(\"Product Sales Frequency Components\")\nax2 = plot_periodogram(y_deseason, ax=ax2);\nax2.set_title(\"Deseasonalized\");","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:45.858005Z","iopub.execute_input":"2022-08-14T09:25:45.858864Z","iopub.status.idle":"2022-08-14T09:25:46.342721Z","shell.execute_reply.started":"2022-08-14T09:25:45.858834Z","shell.execute_reply":"2022-08-14T09:25:46.341634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3) Check for remaining seasonality\n\nBased on these periodograms, how effectively does it appear your model captured the seasonality in *Average Sales*? Does the periodogram agree with the time plot of the deseasonalized series?","metadata":{}},{"cell_type":"code","source":"# View the solution (Run this cell to receive credit!)\nq_3.check()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:48.121824Z","iopub.execute_input":"2022-08-14T09:25:48.122148Z","iopub.status.idle":"2022-08-14T09:25:48.131221Z","shell.execute_reply.started":"2022-08-14T09:25:48.122121Z","shell.execute_reply":"2022-08-14T09:25:48.129854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\nThe *Store Sales* dataset includes a table of Ecuadorian holidays.","metadata":{}},{"cell_type":"code","source":"# National and regional holidays in the training set\nholidays = (\n    holidays_events\n    .query(\"locale in ['National', 'Regional']\")\n    .loc['2017':'2017-08-15', ['description']]\n    .assign(description=lambda x: x.description.cat.remove_unused_categories())\n)\n\ndisplay(holidays)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:49.894319Z","iopub.execute_input":"2022-08-14T09:25:49.894646Z","iopub.status.idle":"2022-08-14T09:25:49.918109Z","shell.execute_reply.started":"2022-08-14T09:25:49.894620Z","shell.execute_reply":"2022-08-14T09:25:49.916821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From a plot of the deseasonalized *Average Sales*, it appears these holidays could have some predictive power.","metadata":{}},{"cell_type":"code","source":"ax = y_deseason.plot(**plot_params)\nplt.plot_date(holidays.index, y_deseason[holidays.index], color='C3')\nax.set_title('National and Regional Holidays');","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:25:51.520838Z","iopub.execute_input":"2022-08-14T09:25:51.521156Z","iopub.status.idle":"2022-08-14T09:25:51.804070Z","shell.execute_reply.started":"2022-08-14T09:25:51.521132Z","shell.execute_reply":"2022-08-14T09:25:51.803026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4) Create holiday features\n\nWhat kind of features could you create to help your model make use of this information? Code your answer in the next cell. (Scikit-learn and Pandas both have utilities that should make this easy. See the `hint` if you'd like more details.)\n","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE\nX_holidays = pd.get_dummies(holidays)\nX2 = X.join(X_holidays, on='date').fillna(0.0)\n\n# Check your answer\nq_4.check()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:26:39.520067Z","iopub.execute_input":"2022-08-14T09:26:39.520496Z","iopub.status.idle":"2022-08-14T09:26:39.542204Z","shell.execute_reply.started":"2022-08-14T09:26:39.520471Z","shell.execute_reply":"2022-08-14T09:26:39.541495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_4.hint()\n#q_4.hint(2)\n# q_4.solution()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:26:43.869803Z","iopub.execute_input":"2022-08-14T09:26:43.870914Z","iopub.status.idle":"2022-08-14T09:26:43.875181Z","shell.execute_reply.started":"2022-08-14T09:26:43.870876Z","shell.execute_reply":"2022-08-14T09:26:43.873952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use this cell to fit the seasonal model with holiday features added. Do the fitted values seem to have improved?","metadata":{}},{"cell_type":"code","source":"model = LinearRegression().fit(X2, y)\ny_pred = pd.Series(\n    model.predict(X2),\n    index=X2.index,\n    name='Fitted',\n)\n\ny_pred = pd.Series(model.predict(X2), index=X2.index)\nax = y.plot(**plot_params, alpha=0.5, title=\"Average Sales\", ylabel=\"items sold\")\nax = y_pred.plot(ax=ax, label=\"Seasonal\")\nax.legend();","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:26:45.389599Z","iopub.execute_input":"2022-08-14T09:26:45.389929Z","iopub.status.idle":"2022-08-14T09:26:45.694416Z","shell.execute_reply.started":"2022-08-14T09:26:45.389904Z","shell.execute_reply":"2022-08-14T09:26:45.693522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\n# (Optional) Submit to Store Sales competition\n\nThis part of the exercise will walk you through your first submission to this course's companion competition: [**Store Sales - Time Series Forecasting**](https://www.kaggle.com/c/29781). Submitting to the competition isn't required to complete the course, but it's a great way to try out your new skills.\n\nThe next cell creates a seasonal model of the kind you've learned about in this lesson for the full *Store Sales* dataset with all 1800 time series.","metadata":{}},{"cell_type":"code","source":"y = store_sales.unstack(['store_nbr', 'family']).loc[\"2017\"]\n\n# Create training data\nfourier = CalendarFourier(freq='M', order=4)\ndp = DeterministicProcess(\n    index=y.index,\n    constant=True,\n    order=1,\n    seasonal=True,\n    additional_terms=[fourier],\n    drop=True,\n)\nX = dp.in_sample()\nX['NewYear'] = (X.index.dayofyear == 1)\n\nmodel = LinearRegression(fit_intercept=False)\nmodel.fit(X, y)\ny_pred = pd.DataFrame(model.predict(X), index=X.index, columns=y.columns)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:26:47.370992Z","iopub.execute_input":"2022-08-14T09:26:47.371908Z","iopub.status.idle":"2022-08-14T09:26:48.340776Z","shell.execute_reply.started":"2022-08-14T09:26:47.371879Z","shell.execute_reply":"2022-08-14T09:26:48.339750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can use this cell to see some of its predictions.\n","metadata":{}},{"cell_type":"code","source":"STORE_NBR = '1'  # 1 - 54\nFAMILY = 'PRODUCE'\n# Uncomment to see a list of product families\n# display(store_sales.index.get_level_values('family').unique())\n\nax = y.loc(axis=1)['sales', STORE_NBR, FAMILY].plot(**plot_params)\nax = y_pred.loc(axis=1)['sales', STORE_NBR, FAMILY].plot(ax=ax)\nax.set_title(f'{FAMILY} Sales at Store {STORE_NBR}');","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:26:51.064452Z","iopub.execute_input":"2022-08-14T09:26:51.064753Z","iopub.status.idle":"2022-08-14T09:26:51.470666Z","shell.execute_reply.started":"2022-08-14T09:26:51.064712Z","shell.execute_reply":"2022-08-14T09:26:51.469857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, this cell loads the test data, creates a feature set for the forecast period, and then creates the submission file `submission.csv`.","metadata":{}},{"cell_type":"code","source":"df_test = pd.read_csv(\n    comp_dir / 'test.csv',\n    dtype={\n        'store_nbr': 'category',\n        'family': 'category',\n        'onpromotion': 'uint32',\n    },\n    parse_dates=['date'],\n    infer_datetime_format=True,\n)\ndf_test['date'] = df_test.date.dt.to_period('D')\ndf_test = df_test.set_index(['store_nbr', 'family', 'date']).sort_index()\n\n# Create features for test set\nX_test = dp.out_of_sample(steps=16)\nX_test.index.name = 'date'\nX_test['NewYear'] = (X_test.index.dayofyear == 1)\n\n\ny_submit = pd.DataFrame(model.predict(X_test), index=X_test.index, columns=y.columns)\ny_submit = y_submit.stack(['store_nbr', 'family'])\ny_submit = y_submit.join(df_test.id).reindex(columns=['id', 'sales'])\ny_submit.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T09:26:53.444729Z","iopub.execute_input":"2022-08-14T09:26:53.445118Z","iopub.status.idle":"2022-08-14T09:26:53.895967Z","shell.execute_reply.started":"2022-08-14T09:26:53.445093Z","shell.execute_reply":"2022-08-14T09:26:53.894630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To test your forecasts, you'll need to join the competition (if you haven't already). So open a new window by clicking on [this link](https://www.kaggle.com/c/29781). Then click on the **Join Competition** button.\n\nNext, follow the instructions below:\n1. Begin by clicking on the **Save Version** button in the top right corner of the window.  This will generate a pop-up window.  \n2. Ensure that the **Save and Run All** option is selected, and then click on the **Save** button.\n3. This generates a window in the bottom left corner of the notebook.  After it has finished running, click on the number to the right of the **Save Version** button.  This pulls up a list of versions on the right of the screen.  Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**.  This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n4. Click on the **Output** tab on the right of the screen.  Then, click on the file you would like to submit, and click on the **Submit** button to submit your results to the leaderboard.\n\nYou have now successfully submitted to the competition!\n\nIf you want to keep working to improve your performance, select the **Edit** button in the top right of the screen. Then you can change your code and repeat the process. There's a lot of room to improve, and you will climb up the leaderboard as you work.\n","metadata":{}},{"cell_type":"markdown","source":"# Keep Going #\n\n[**Use time series as features**](https://www.kaggle.com/ryanholbrook/time-series-as-features) to capture cycles and other kinds of serial dependence.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/time-series/discussion) to chat with other learners.*","metadata":{}}]}