{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Time Series](https://www.kaggle.com/learn/time-series) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/seasonality).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nRun this cell to set everything up!","metadata":{}},{"cell_type":"code","source":"# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.time_series.ex3 import *\n\n# Setup notebook\nfrom pathlib import Path\nfrom learntools.time_series.style import *  # plot style settings\nfrom learntools.time_series.utils import plot_periodogram, seasonal_plot\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport seaborn as sns\nfrom sklearn.linear_model import LinearRegression\nfrom statsmodels.tsa.deterministic import CalendarFourier, DeterministicProcess\n\n\ncomp_dir = Path('../input/store-sales-time-series-forecasting')\n\nholidays_events = pd.read_csv(\n    comp_dir / \"holidays_events.csv\",\n    dtype={\n        'type': 'category',\n        'locale': 'category',\n        'locale_name': 'category',\n        'description': 'category',\n        'transferred': 'bool',\n    },\n    parse_dates=['date'],\n    infer_datetime_format=True,\n)\nholidays_events = holidays_events.set_index('date').to_period('D')\n\nstore_sales = pd.read_csv(\n    comp_dir / 'train.csv',\n    usecols=['store_nbr', 'family', 'date', 'sales'],\n    dtype={\n        'store_nbr': 'category',\n        'family': 'category',\n        'sales': 'float32',\n    },\n    parse_dates=['date'],\n    infer_datetime_format=True,\n)\nstore_sales['date'] = store_sales.date.dt.to_period('D')\nstore_sales = store_sales.set_index(['store_nbr', 'family', 'date']).sort_index()\naverage_sales = (\n    store_sales\n    .groupby('date').mean()\n    .squeeze()\n    .loc['2017']\n)","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:52:45.120892Z","iopub.execute_input":"2022-07-28T02:52:45.121681Z","iopub.status.idle":"2022-07-28T02:52:51.634376Z","shell.execute_reply.started":"2022-07-28T02:52:45.121570Z","shell.execute_reply":"2022-07-28T02:52:51.633191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\nExamine the following seasonal plot:","metadata":{}},{"cell_type":"code","source":"X = average_sales.to_frame()\nX[\"week\"] = X.index.week\nX[\"day\"] = X.index.dayofweek\nseasonal_plot(X, y='sales', period='week', freq='day');","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:52:51.636349Z","iopub.execute_input":"2022-07-28T02:52:51.636690Z","iopub.status.idle":"2022-07-28T02:52:52.393088Z","shell.execute_reply.started":"2022-07-28T02:52:51.636658Z","shell.execute_reply":"2022-07-28T02:52:52.392302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And also the periodogram:","metadata":{}},{"cell_type":"code","source":"plot_periodogram(average_sales);","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:52:52.394225Z","iopub.execute_input":"2022-07-28T02:52:52.394706Z","iopub.status.idle":"2022-07-28T02:52:52.894613Z","shell.execute_reply.started":"2022-07-28T02:52:52.394677Z","shell.execute_reply":"2022-07-28T02:52:52.893326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1) Determine seasonality\n\nWhat kind of seasonality do you see evidence of? Once you've thought about it, run the next cell for some discussion.","metadata":{}},{"cell_type":"code","source":"# View the solution (Run this cell to receive credit!)\nq_1.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:52:52.896824Z","iopub.execute_input":"2022-07-28T02:52:52.897174Z","iopub.status.idle":"2022-07-28T02:52:52.906102Z","shell.execute_reply.started":"2022-07-28T02:52:52.897143Z","shell.execute_reply":"2022-07-28T02:52:52.904972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\n# 2) Create seasonal features\n\nUse `DeterministicProcess` and `CalendarFourier` to create:\n- indicators for weekly seasons and\n- Fourier features of order 4 for monthly seasons.","metadata":{}},{"cell_type":"code","source":"y = average_sales.copy()\n\n# YOUR CODE HERE\nfourier = CalendarFourier(freq='M', order=4)\ndp = DeterministicProcess(\n    index=y.index,\n    constant=True,\n    order=1,\n    seasonal=True,\n    additional_terms=[fourier],\n    drop=True,\n)\nX = dp.in_sample()\n\n# Check your answer\nq_2.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:56:12.837861Z","iopub.execute_input":"2022-07-28T02:56:12.838915Z","iopub.status.idle":"2022-07-28T02:56:15.321017Z","shell.execute_reply.started":"2022-07-28T02:56:12.838857Z","shell.execute_reply":"2022-07-28T02:56:15.319968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\nq_2.hint()\n# q_2.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:56:22.336347Z","iopub.execute_input":"2022-07-28T02:56:22.336729Z","iopub.status.idle":"2022-07-28T02:56:22.345622Z","shell.execute_reply.started":"2022-07-28T02:56:22.336698Z","shell.execute_reply":"2022-07-28T02:56:22.344521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now run this cell to fit the seasonal model.","metadata":{}},{"cell_type":"code","source":"model = LinearRegression().fit(X, y)\ny_pred = pd.Series(\n    model.predict(X),\n    index=X.index,\n    name='Fitted',\n)\n\ny_pred = pd.Series(model.predict(X), index=X.index)\nax = y.plot(**plot_params, alpha=0.5, title=\"Average Sales\", ylabel=\"items sold\")\nax = y_pred.plot(ax=ax, label=\"Seasonal\")\nax.legend();","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:56:28.084035Z","iopub.execute_input":"2022-07-28T02:56:28.084634Z","iopub.status.idle":"2022-07-28T02:56:28.536840Z","shell.execute_reply.started":"2022-07-28T02:56:28.084582Z","shell.execute_reply":"2022-07-28T02:56:28.535373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n","metadata":{}},{"cell_type":"markdown","source":"Removing from a series its trend or seasons is called **detrending** or **deseasonalizing** the series.\n\nLook at the periodogram of the deseasonalized series.","metadata":{}},{"cell_type":"code","source":"y_deseason = y - y_pred\n\nfig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, sharey=True, figsize=(10, 7))\nax1 = plot_periodogram(y, ax=ax1)\nax1.set_title(\"Product Sales Frequency Components\")\nax2 = plot_periodogram(y_deseason, ax=ax2);\nax2.set_title(\"Deseasonalized\");","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:56:51.262849Z","iopub.execute_input":"2022-07-28T02:56:51.263323Z","iopub.status.idle":"2022-07-28T02:56:51.778494Z","shell.execute_reply.started":"2022-07-28T02:56:51.263284Z","shell.execute_reply":"2022-07-28T02:56:51.777207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3) Check for remaining seasonality\n\nBased on these periodograms, how effectively does it appear your model captured the seasonality in *Average Sales*? Does the periodogram agree with the time plot of the deseasonalized series?","metadata":{}},{"cell_type":"code","source":"# View the solution (Run this cell to receive credit!)\nq_3.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:57:01.847671Z","iopub.execute_input":"2022-07-28T02:57:01.848109Z","iopub.status.idle":"2022-07-28T02:57:01.855859Z","shell.execute_reply.started":"2022-07-28T02:57:01.848071Z","shell.execute_reply":"2022-07-28T02:57:01.855081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\nThe *Store Sales* dataset includes a table of Ecuadorian holidays.","metadata":{}},{"cell_type":"code","source":"# National and regional holidays in the training set\nholidays = (\n    holidays_events\n    .query(\"locale in ['National', 'Regional']\")\n    .loc['2017':'2017-08-15', ['description']]\n    .assign(description=lambda x: x.description.cat.remove_unused_categories())\n)\n\ndisplay(holidays)","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:57:30.017794Z","iopub.execute_input":"2022-07-28T02:57:30.018204Z","iopub.status.idle":"2022-07-28T02:57:30.042585Z","shell.execute_reply.started":"2022-07-28T02:57:30.018172Z","shell.execute_reply":"2022-07-28T02:57:30.041474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From a plot of the deseasonalized *Average Sales*, it appears these holidays could have some predictive power.","metadata":{}},{"cell_type":"code","source":"ax = y_deseason.plot(**plot_params)\nplt.plot_date(holidays.index, y_deseason[holidays.index], color='C3')\nax.set_title('National and Regional Holidays');","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:57:46.046934Z","iopub.execute_input":"2022-07-28T02:57:46.047620Z","iopub.status.idle":"2022-07-28T02:57:46.367623Z","shell.execute_reply.started":"2022-07-28T02:57:46.047575Z","shell.execute_reply":"2022-07-28T02:57:46.366308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4) Create holiday features\n\nWhat kind of features could you create to help your model make use of this information? Code your answer in the next cell. (Scikit-learn and Pandas both have utilities that should make this easy. See the `hint` if you'd like more details.)\n","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE\n# Scikit-learn solution\nfrom sklearn.preprocessing import OneHotEncoder\n\nohe = OneHotEncoder(sparse=False)\nX_holidays = pd.DataFrame(\n    ohe.fit_transform(holidays),\n    index=holidays.index,\n    columns=holidays.description.unique(),\n)\nX2 = X.join(X_holidays, on='date').fillna(0.0)\n\n# Check your answer\nq_4.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:58:15.145772Z","iopub.execute_input":"2022-07-28T02:58:15.146189Z","iopub.status.idle":"2022-07-28T02:58:15.178609Z","shell.execute_reply.started":"2022-07-28T02:58:15.146156Z","shell.execute_reply":"2022-07-28T02:58:15.177804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\nq_4.hint()\n#q_4.hint(2)\n# q_4.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:58:20.897739Z","iopub.execute_input":"2022-07-28T02:58:20.898164Z","iopub.status.idle":"2022-07-28T02:58:20.907336Z","shell.execute_reply.started":"2022-07-28T02:58:20.898128Z","shell.execute_reply":"2022-07-28T02:58:20.906165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use this cell to fit the seasonal model with holiday features added. Do the fitted values seem to have improved?","metadata":{}},{"cell_type":"code","source":"model = LinearRegression().fit(X2, y)\ny_pred = pd.Series(\n    model.predict(X2),\n    index=X2.index,\n    name='Fitted',\n)\n\ny_pred = pd.Series(model.predict(X2), index=X2.index)\nax = y.plot(**plot_params, alpha=0.5, title=\"Average Sales\", ylabel=\"items sold\")\nax = y_pred.plot(ax=ax, label=\"Seasonal\")\nax.legend();","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:58:38.074704Z","iopub.execute_input":"2022-07-28T02:58:38.075179Z","iopub.status.idle":"2022-07-28T02:58:38.416435Z","shell.execute_reply.started":"2022-07-28T02:58:38.075140Z","shell.execute_reply":"2022-07-28T02:58:38.415253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\n# (Optional) Submit to Store Sales competition\n\nThis part of the exercise will walk you through your first submission to this course's companion competition: [**Store Sales - Time Series Forecasting**](https://www.kaggle.com/c/29781). Submitting to the competition isn't required to complete the course, but it's a great way to try out your new skills.\n\nThe next cell creates a seasonal model of the kind you've learned about in this lesson for the full *Store Sales* dataset with all 1800 time series.","metadata":{}},{"cell_type":"code","source":"y = store_sales.unstack(['store_nbr', 'family']).loc[\"2017\"]\n\n# Create training data\nfourier = CalendarFourier(freq='M', order=4)\ndp = DeterministicProcess(\n    index=y.index,\n    constant=True,\n    order=1,\n    seasonal=True,\n    additional_terms=[fourier],\n    drop=True,\n)\nX = dp.in_sample()\nX['NewYear'] = (X.index.dayofyear == 1)\n\nmodel = LinearRegression(fit_intercept=False)\nmodel.fit(X, y)\ny_pred = pd.DataFrame(model.predict(X), index=X.index, columns=y.columns)","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:58:47.519173Z","iopub.execute_input":"2022-07-28T02:58:47.519730Z","iopub.status.idle":"2022-07-28T02:58:48.816621Z","shell.execute_reply.started":"2022-07-28T02:58:47.519683Z","shell.execute_reply":"2022-07-28T02:58:48.814942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can use this cell to see some of its predictions.\n","metadata":{}},{"cell_type":"code","source":"STORE_NBR = '1'  # 1 - 54\nFAMILY = 'PRODUCE'\n# Uncomment to see a list of product families\n# display(store_sales.index.get_level_values('family').unique())\n\nax = y.loc(axis=1)['sales', STORE_NBR, FAMILY].plot(**plot_params)\nax = y_pred.loc(axis=1)['sales', STORE_NBR, FAMILY].plot(ax=ax)\nax.set_title(f'{FAMILY} Sales at Store {STORE_NBR}');","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:59:13.895901Z","iopub.execute_input":"2022-07-28T02:59:13.896349Z","iopub.status.idle":"2022-07-28T02:59:14.357423Z","shell.execute_reply.started":"2022-07-28T02:59:13.896311Z","shell.execute_reply":"2022-07-28T02:59:14.356324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, this cell loads the test data, creates a feature set for the forecast period, and then creates the submission file `submission.csv`.","metadata":{}},{"cell_type":"code","source":"df_test = pd.read_csv(\n    comp_dir / 'test.csv',\n    dtype={\n        'store_nbr': 'category',\n        'family': 'category',\n        'onpromotion': 'uint32',\n    },\n    parse_dates=['date'],\n    infer_datetime_format=True,\n)\ndf_test['date'] = df_test.date.dt.to_period('D')\ndf_test = df_test.set_index(['store_nbr', 'family', 'date']).sort_index()\n\n# Create features for test set\nX_test = dp.out_of_sample(steps=16)\nX_test.index.name = 'date'\nX_test['NewYear'] = (X_test.index.dayofyear == 1)\n\n\ny_submit = pd.DataFrame(model.predict(X_test), index=X_test.index, columns=y.columns)\ny_submit = y_submit.stack(['store_nbr', 'family'])\ny_submit = y_submit.join(df_test.id).reindex(columns=['id', 'sales'])\ny_submit.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-28T02:59:17.823329Z","iopub.execute_input":"2022-07-28T02:59:17.823958Z","iopub.status.idle":"2022-07-28T02:59:18.339284Z","shell.execute_reply.started":"2022-07-28T02:59:17.823913Z","shell.execute_reply":"2022-07-28T02:59:18.338251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To test your forecasts, you'll need to join the competition (if you haven't already). So open a new window by clicking on [this link](https://www.kaggle.com/c/29781). Then click on the **Join Competition** button.\n\nNext, follow the instructions below:\n1. Begin by clicking on the **Save Version** button in the top right corner of the window.  This will generate a pop-up window.  \n2. Ensure that the **Save and Run All** option is selected, and then click on the **Save** button.\n3. This generates a window in the bottom left corner of the notebook.  After it has finished running, click on the number to the right of the **Save Version** button.  This pulls up a list of versions on the right of the screen.  Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**.  This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n4. Click on the **Output** tab on the right of the screen.  Then, click on the file you would like to submit, and click on the **Submit** button to submit your results to the leaderboard.\n\nYou have now successfully submitted to the competition!\n\nIf you want to keep working to improve your performance, select the **Edit** button in the top right of the screen. Then you can change your code and repeat the process. There's a lot of room to improve, and you will climb up the leaderboard as you work.\n","metadata":{}},{"cell_type":"markdown","source":"# Keep Going #\n\n[**Use time series as features**](https://www.kaggle.com/ryanholbrook/time-series-as-features) to capture cycles and other kinds of serial dependence.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/time-series/discussion) to chat with other learners.*","metadata":{}}]}