{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport numpy as np\nimport plotly.express as px\nimport plotly.graph_objs as go\nimport xgboost as xgb\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.linear_model import LinearRegression, Lasso\nfrom lightgbm import LGBMRegressor,log_evaluation\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import Pipeline\n\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-13T19:01:54.996634Z","iopub.execute_input":"2022-07-13T19:01:54.997075Z","iopub.status.idle":"2022-07-13T19:01:58.202102Z","shell.execute_reply.started":"2022-07-13T19:01:54.996977Z","shell.execute_reply":"2022-07-13T19:01:58.200660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each store contains many departments, and participants must project the sales **for each department in each store**. To add to the challenge, selected holiday markdown events are included in the dataset. ","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/walmart-recruiting-store-sales-forecasting/train.csv.zip')\ntest = pd.read_csv('../input/walmart-recruiting-store-sales-forecasting/test.csv.zip')\nstores = pd.read_csv('../input/walmart-recruiting-store-sales-forecasting/stores.csv')\nfeatures = pd.read_csv('../input/walmart-recruiting-store-sales-forecasting/features.csv.zip')\nsample_submission = pd.read_csv('../input/walmart-recruiting-store-sales-forecasting/sampleSubmission.csv.zip')","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:58.204279Z","iopub.execute_input":"2022-07-13T19:01:58.204720Z","iopub.status.idle":"2022-07-13T19:01:58.721139Z","shell.execute_reply.started":"2022-07-13T19:01:58.204675Z","shell.execute_reply":"2022-07-13T19:01:58.720030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# INTRODUÇÃO","metadata":{}},{"cell_type":"markdown","source":"Neste notebook será desenvolvida a solução para problema proposto pela empresa Wallmart. O problema consiste em prever as vendas semanais (Weekly_Sales) de cada um dos depastamentos de 45 lojas localizadas em diferentes regiões, a partir de bases de dados disponibilizadas pela competição, contendo informações históricas.\n\n**Analise exploratória:** Inicialmente sera feita uma analise exploratória para entender o tipo de info contida nas bases, capturar insights e embasar todas as decisões tomadas na etapa de tratamentos de dados. Foram disponivbilizadas as seguintes bases:\n- Train.csv: Estes são os dados históricos para treinar o modelo.\n- Teste.csv: Esta base é identica à base Train.csv, mas é referente a um periodo posterior e não possui a variavel resposta \"Weekly_Sales\". Esta é a base que de deveremos escorar para a competição.\n- Store.csv: Este arquivo contem informação anonimizada das 45 lojas,indicando o Tipo e o tamanho da Loja.\n- Features.csv: Este arquivo contem informação adicional relativa às lojas,e condições regionais para as determinadas datas\n \n\n**Tratamento de dados:** A partir das conclusoes da etapa anterior os dados serão tratados criando, transformando, retirando variaveis, e inputando informações vazias quando necessário. \n\n**Modelagem de dados:** Com os dados tratados, inicia-se o processo de modelagem. Neste passo serão utilizadas as seguintes tecnicas:\n- LASSO \n- DECISION TREE\n- RANDOM FOREST\n- LIGHGBM\n\n**Avaliação :** Nesta etapa iremos comparar a performance dos modelos. Aquele com com melhor performance será utilizado para escorar a base Test.csv, e essas infos serão submetidas à competição. \n\n**Implantaçao :** Nesta passo fiz uma sugestão de codigo para implantação do modelo.\n\n**Pontos de melhoria :** Considerações sobre o que poderia ser melhorado.\n","metadata":{}},{"cell_type":"markdown","source":"# ANALISE EXPLORATÓRIA\n","metadata":{}},{"cell_type":"markdown","source":"## Avaliando Base Train","metadata":{}},{"cell_type":"markdown","source":"train.csv\n\nNesta base estão contidas as informações:\n- Store - Numero da Loja\n- Dept - Numero do departamento\n- Date - A semana\n- Weekly_Sales - Vendas semanais para de um  departamento de uma loja\n- IsHoliday - se a semana é uma semana de feriado","metadata":{}},{"cell_type":"code","source":"#Perguntas sobre a base Store\n# 1.Qual tamanho?\n# 2.Como é a base\n# 3.As variaveis possuem missings?\n# 4.Quais conclusões sobre a base","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:58.722374Z","iopub.execute_input":"2022-07-13T19:01:58.722693Z","iopub.status.idle":"2022-07-13T19:01:58.727022Z","shell.execute_reply.started":"2022-07-13T19:01:58.722666Z","shell.execute_reply":"2022-07-13T19:01:58.726129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#1.Tamanho\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:58.728878Z","iopub.execute_input":"2022-07-13T19:01:58.729704Z","iopub.status.idle":"2022-07-13T19:01:58.743272Z","shell.execute_reply.started":"2022-07-13T19:01:58.729672Z","shell.execute_reply":"2022-07-13T19:01:58.742127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#2.Informação\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:58.744841Z","iopub.execute_input":"2022-07-13T19:01:58.745214Z","iopub.status.idle":"2022-07-13T19:01:58.769473Z","shell.execute_reply.started":"2022-07-13T19:01:58.745175Z","shell.execute_reply":"2022-07-13T19:01:58.768370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#2.1 Incluir info de Datas\ntrain['Date'] = pd.to_datetime(train['Date'])\ntrain['Ano'] = (train.Date.dt.year).astype(int)\ntrain['Mes'] = (train.Date.dt.month).astype(int)\ntrain['Dia'] = (train.Date.dt.day).astype(int)\ntrain['SemanaDoAno'] = ((train.Date.dt.isocalendar().week)*1.0).astype(int) \n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:58.770692Z","iopub.execute_input":"2022-07-13T19:01:58.771004Z","iopub.status.idle":"2022-07-13T19:01:59.122926Z","shell.execute_reply.started":"2022-07-13T19:01:58.770975Z","shell.execute_reply":"2022-07-13T19:01:59.121775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.124328Z","iopub.execute_input":"2022-07-13T19:01:59.124773Z","iopub.status.idle":"2022-07-13T19:01:59.132958Z","shell.execute_reply.started":"2022-07-13T19:01:59.124739Z","shell.execute_reply":"2022-07-13T19:01:59.131595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#3.Missings?\ntrain.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.134221Z","iopub.execute_input":"2022-07-13T19:01:59.134612Z","iopub.status.idle":"2022-07-13T19:01:59.154456Z","shell.execute_reply.started":"2022-07-13T19:01:59.134582Z","shell.execute_reply":"2022-07-13T19:01:59.152886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 4.Quais conclusões sobre a base?\nprint(\"As datas vão de: \"+ str(train.Date.min()) +\", Até: \"+ str(train.Date.max()))\nprint(\"Vendas mensais. Mínimo: \" + train.Weekly_Sales.min().astype(str) +\", Máximo: \" + train.Weekly_Sales.max().astype(str))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.156130Z","iopub.execute_input":"2022-07-13T19:01:59.156451Z","iopub.status.idle":"2022-07-13T19:01:59.169424Z","shell.execute_reply.started":"2022-07-13T19:01:59.156422Z","shell.execute_reply":"2022-07-13T19:01:59.167914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 4.Quais conclusões sobre a base?\n# Existe valor negativo no Target, faz sentido?\n# Como está a distribuição?\n\ntrain.Weekly_Sales.describe().T","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.174738Z","iopub.execute_input":"2022-07-13T19:01:59.175153Z","iopub.status.idle":"2022-07-13T19:01:59.207153Z","shell.execute_reply.started":"2022-07-13T19:01:59.175122Z","shell.execute_reply":"2022-07-13T19:01:59.205897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Qual a recorrencia de vendas negativas\nprint(\"Existem \" + str(train[\"Weekly_Sales\"].loc[train.Weekly_Sales<0].count()) + \" casos de vendas negativas, que representam da base toda: \" + str(round(train.loc[train.Weekly_Sales<0].shape[0]/train.shape[0],4)))\n\n#Não parece ser dedada, aparenta ser algum tipo de prejuizo que a loja/departamento podem ter, não é comum mas acontece. ","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.210282Z","iopub.execute_input":"2022-07-13T19:01:59.210643Z","iopub.status.idle":"2022-07-13T19:01:59.243893Z","shell.execute_reply.started":"2022-07-13T19:01:59.210603Z","shell.execute_reply":"2022-07-13T19:01:59.242701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Estes prejuisos estão concentrados em alguma epoca, loja , departamento?\npreju=train.loc[train.Weekly_Sales<0]\npreju_group=preju[[\"Store\",\"Weekly_Sales\"]].groupby([\"Store\"]).count().sort_values(by=\"Weekly_Sales\",ascending=False)\npreju_group[\"percent\"]=preju_group[\"Weekly_Sales\"]/preju_group[\"Weekly_Sales\"].sum()\npreju_group.head(20)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.245459Z","iopub.execute_input":"2022-07-13T19:01:59.246181Z","iopub.status.idle":"2022-07-13T19:01:59.265173Z","shell.execute_reply.started":"2022-07-13T19:01:59.246134Z","shell.execute_reply":"2022-07-13T19:01:59.263988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Os prejuizos não estao concentrados em uma loja especifica","metadata":{}},{"cell_type":"code","source":"preju_group1=preju[[\"Dept\",\"Weekly_Sales\"]].groupby([\"Dept\"]).count().sort_values(by=\"Weekly_Sales\",ascending=False)\npreju_group1[\"percent\"]=preju_group1[\"Weekly_Sales\"]/preju_group1[\"Weekly_Sales\"].sum()\nprint(\"Dos \" + str(len(preju_group1[\"Weekly_Sales\"].unique())) +\" departamentos com algum faturamento negativo, 5 representam \" + str(round(preju_group1[\"Weekly_Sales\"].head(5).sum()/preju_group1[\"Weekly_Sales\"].sum(),2)) + \" do total das recorencias\")\npreju_group1.head(5)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.267076Z","iopub.execute_input":"2022-07-13T19:01:59.267780Z","iopub.status.idle":"2022-07-13T19:01:59.287445Z","shell.execute_reply.started":"2022-07-13T19:01:59.267738Z","shell.execute_reply":"2022-07-13T19:01:59.286363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vale Verificar como está a distribuição das vendas, ao longo do tempo","metadata":{}},{"cell_type":"code","source":"#Plota Graficos de barra com metrica escolhida\nimport matplotlib.pyplot as plt\n%matplotlib inline\ndef plot_por_target(df,var,resposta,metrica,ascending=0,segmentacao=\"\"):\n    \n    base_agrup=df[[var,resposta]].groupby([var],as_index=False).agg({resposta:metrica})\n    if ascending==1:\n        base_agrup=base_agrup.sort_values(by=resposta,ascending=False)\n    \n    \n    if segmentacao!=\"\":\n        data=[]\n        base_agrup=df[[var,segmentacao,resposta]].groupby([var,segmentacao],as_index=False).agg({resposta:metrica})\n        base_agrup[\"media\"]=df[[var]].mean()\n        import plotly.graph_objs as go\n        for i in df[segmentacao].unique():\n            base_agrup_1=base_agrup.loc[base_agrup[segmentacao]==i]\n            trace = go.Bar(\n            x = base_agrup_1[var],\n            y = base_agrup_1[resposta],\n            name = str(i))\n            data.append(trace)\n            layout = go.Layout(barmode = \"group\",\n                               template='plotly_dark',\n                               xaxis_title=var,\n                               yaxis_title=resposta)\n     \n        fig1 = go.Figure(data = data, layout = layout)\n        fig1.update_xaxes(dtick=1)\n\n        return  fig1.show()\n    else:\n        fig,ax = plt.subplots(figsize=(16,8))\n        base_agrup.plot.bar(x=var,y=resposta,ax=ax,label=var)\n        return fig.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.289350Z","iopub.execute_input":"2022-07-13T19:01:59.290096Z","iopub.status.idle":"2022-07-13T19:01:59.307125Z","shell.execute_reply.started":"2022-07-13T19:01:59.290050Z","shell.execute_reply":"2022-07-13T19:01:59.306051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Verifica como foi a soma de vendas nos anos\nplot_por_target(train,\"Ano\",\"Weekly_Sales\",sum,0)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.309368Z","iopub.execute_input":"2022-07-13T19:01:59.309901Z","iopub.status.idle":"2022-07-13T19:01:59.561994Z","shell.execute_reply.started":"2022-07-13T19:01:59.309819Z","shell.execute_reply":"2022-07-13T19:01:59.560863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Num primeiro momento parece que 2011 foi um ano melhor que 2010 e 2012","metadata":{}},{"cell_type":"code","source":"#Verifica como foi a soma de vendas nos meses\nplot_por_target(train,\"Mes\",\"Weekly_Sales\",sum,0,\"Ano\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:01:59.564596Z","iopub.execute_input":"2022-07-13T19:01:59.565059Z","iopub.status.idle":"2022-07-13T19:02:00.747053Z","shell.execute_reply.started":"2022-07-13T19:01:59.565011Z","shell.execute_reply":"2022-07-13T19:02:00.745681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Mas depois de abrir as vendas por mes que faltam informações do começo de 2010 e final de 2012.","metadata":{}},{"cell_type":"code","source":"plot_por_target(train,\"SemanaDoAno\",\"Weekly_Sales\",sum,0,\"Ano\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:00.748969Z","iopub.execute_input":"2022-07-13T19:02:00.749349Z","iopub.status.idle":"2022-07-13T19:02:00.895340Z","shell.execute_reply.started":"2022-07-13T19:02:00.749318Z","shell.execute_reply":"2022-07-13T19:02:00.893888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Quando abrimos a informação de vendas por semana percebemos que (para os casos que possuem informação) há uma tendencia de aumento das vendas no final do ano principalmente entre o Thanksgiving e ao Natal. Após essas datas os níveis voltam a patamares até abaixo da média. Com excessão deste periodo as vendas não aparentam aumentar substancialmente. ","metadata":{}},{"cell_type":"markdown","source":"Como estão as vendas das lojas ao longo dos anos?","metadata":{}},{"cell_type":"code","source":"plot_por_target(train,\"Store\",\"Weekly_Sales\",sum,0,\"Ano\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:00.897101Z","iopub.execute_input":"2022-07-13T19:02:00.897515Z","iopub.status.idle":"2022-07-13T19:02:01.038156Z","shell.execute_reply.started":"2022-07-13T19:02:00.897475Z","shell.execute_reply":"2022-07-13T19:02:01.036549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Existe uma relativa consistencia no valor de vendas das Lojas quando comparamos os 3 anos. Aqui podemos ordenar as lojas por vendas.","metadata":{}},{"cell_type":"markdown","source":"como estão as vendas dos departamentos ao longo dos anos?","metadata":{}},{"cell_type":"code","source":"plot_por_target(train,\"Dept\",\"Weekly_Sales\",sum,0,\"Ano\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:01.041733Z","iopub.execute_input":"2022-07-13T19:02:01.042511Z","iopub.status.idle":"2022-07-13T19:02:01.172060Z","shell.execute_reply.started":"2022-07-13T19:02:01.042467Z","shell.execute_reply":"2022-07-13T19:02:01.170882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Existe uma relativa consistencia no valor de vendas dos departamentos quando comparamos os 3 anos. Aqui podemos ordenar os departamentos por vendas.","metadata":{}},{"cell_type":"code","source":"#Esta função cria um grafico de boxplot \ndef boxplot(df,var,resposta,segmentacao,outliers=True):\n    plt.figure(figsize = (20,6))\n    if segmentacao!=\"\":\n        fig = sns.boxplot(x = segmentacao\n                          ,y = resposta\n                          ,data = df[[segmentacao,resposta,var]]\n                          ,showfliers = outliers\n                          ,hue = var\n                     )\n    else:\n        fig = sns.boxplot(x = var\n                          ,y = resposta\n                          ,data = df[[var,resposta]]\n                          ,showfliers = outliers\n                     )\n    return plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:01.173915Z","iopub.execute_input":"2022-07-13T19:02:01.175052Z","iopub.status.idle":"2022-07-13T19:02:01.183256Z","shell.execute_reply.started":"2022-07-13T19:02:01.175006Z","shell.execute_reply":"2022-07-13T19:02:01.181903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Agora iremos avaliar a variavel IsHoliday para ver como ela influencia nas vendas das lojas.","metadata":{}},{"cell_type":"code","source":"# boxplot(train,\"IsHoliday\",\"Weekly_Sales\",\"Store\",True)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:01.185014Z","iopub.execute_input":"2022-07-13T19:02:01.185368Z","iopub.status.idle":"2022-07-13T19:02:01.201678Z","shell.execute_reply.started":"2022-07-13T19:02:01.185335Z","shell.execute_reply":"2022-07-13T19:02:01.200170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(train,\"IsHoliday\",\"Weekly_Sales\",\"Store\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:01.203448Z","iopub.execute_input":"2022-07-13T19:02:01.205138Z","iopub.status.idle":"2022-07-13T19:02:02.824246Z","shell.execute_reply.started":"2022-07-13T19:02:01.205090Z","shell.execute_reply":"2022-07-13T19:02:02.823119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"De forma geral os Feriados possuem um impacto marginalmente positivo nas vendas das lojas","metadata":{}},{"cell_type":"markdown","source":"Como está a relação de departamentos com o Target?","metadata":{}},{"cell_type":"markdown","source":"Agora iremos avaliar a variavel IsHoliday para ver como ela influencia nas vendas nos departamentos.","metadata":{}},{"cell_type":"code","source":"# boxplot(train,\"IsHoliday\",\"Weekly_Sales\",\"Dept\",True)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:02.825907Z","iopub.execute_input":"2022-07-13T19:02:02.826614Z","iopub.status.idle":"2022-07-13T19:02:02.833609Z","shell.execute_reply.started":"2022-07-13T19:02:02.826566Z","shell.execute_reply":"2022-07-13T19:02:02.831776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Retirando os ouliers\nboxplot(train,\"IsHoliday\",\"Weekly_Sales\",\"Dept\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:02.835664Z","iopub.execute_input":"2022-07-13T19:02:02.836171Z","iopub.status.idle":"2022-07-13T19:02:05.705820Z","shell.execute_reply.started":"2022-07-13T19:02:02.836130Z","shell.execute_reply":"2022-07-13T19:02:05.704620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(train[train[\"Ano\"]==2010],\"IsHoliday\",\"Weekly_Sales\",\"Dept\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:05.707108Z","iopub.execute_input":"2022-07-13T19:02:05.707506Z","iopub.status.idle":"2022-07-13T19:02:08.418004Z","shell.execute_reply.started":"2022-07-13T19:02:05.707473Z","shell.execute_reply":"2022-07-13T19:02:08.416841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(train[train[\"Ano\"]==2011],\"IsHoliday\",\"Weekly_Sales\",\"Dept\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:08.419683Z","iopub.execute_input":"2022-07-13T19:02:08.420017Z","iopub.status.idle":"2022-07-13T19:02:11.153276Z","shell.execute_reply.started":"2022-07-13T19:02:08.419986Z","shell.execute_reply":"2022-07-13T19:02:11.151875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(train[train[\"Ano\"]==2012],\"IsHoliday\",\"Weekly_Sales\",\"Dept\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:11.155081Z","iopub.execute_input":"2022-07-13T19:02:11.155420Z","iopub.status.idle":"2022-07-13T19:02:13.859824Z","shell.execute_reply.started":"2022-07-13T19:02:11.155388Z","shell.execute_reply":"2022-07-13T19:02:13.858517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"De forma geral os feriados não afetam todos os departamentos de forma expressiva. No entando podemos ressaltar o departamento 72 que é altamente impactado pelos feriados (mais do que os outros). Nos anos de 2010 e 2011 este departamento apresentou um distribuição superior  quando comparamos \"feriados\" e não \"feriados\". Observamos 2012, este departamento ficou abaixo do esperado em comparação aos outros anos pois não há na base, o periodo que compreende o Thanxgiving e o Natal. \n","metadata":{}},{"cell_type":"markdown","source":"##  Avaliando a Base Teste e Sample Submission","metadata":{}},{"cell_type":"code","source":"test.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.868280Z","iopub.execute_input":"2022-07-13T19:02:13.868636Z","iopub.status.idle":"2022-07-13T19:02:13.875427Z","shell.execute_reply.started":"2022-07-13T19:02:13.868607Z","shell.execute_reply":"2022-07-13T19:02:13.874329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.877077Z","iopub.execute_input":"2022-07-13T19:02:13.877744Z","iopub.status.idle":"2022-07-13T19:02:13.917493Z","shell.execute_reply.started":"2022-07-13T19:02:13.877701Z","shell.execute_reply":"2022-07-13T19:02:13.916110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"As datas da base teste vão de: \"+ str(test.Date.min()) +\", até: \"+ str(test.Date.max()))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.919407Z","iopub.execute_input":"2022-07-13T19:02:13.919875Z","iopub.status.idle":"2022-07-13T19:02:13.953393Z","shell.execute_reply.started":"2022-07-13T19:02:13.919829Z","shell.execute_reply":"2022-07-13T19:02:13.952116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.955187Z","iopub.execute_input":"2022-07-13T19:02:13.955620Z","iopub.status.idle":"2022-07-13T19:02:13.966950Z","shell.execute_reply.started":"2022-07-13T19:02:13.955575Z","shell.execute_reply":"2022-07-13T19:02:13.965863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.968676Z","iopub.execute_input":"2022-07-13T19:02:13.969259Z","iopub.status.idle":"2022-07-13T19:02:13.978791Z","shell.execute_reply.started":"2022-07-13T19:02:13.969226Z","shell.execute_reply":"2022-07-13T19:02:13.977937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Para submissão do problema será necessário cruzar as features à base teste e depois trazerestas infos na base sample_submission","metadata":{}},{"cell_type":"markdown","source":"## Avaliando a Base Store","metadata":{}},{"cell_type":"code","source":"#Perguntas sobre a base Store\n# 1.Qual tamanho?\n# 2.Como é a base\n# 3.As variaveis possuem missings?\n# 4.Quais conclusões sobre a base","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.980206Z","iopub.execute_input":"2022-07-13T19:02:13.980791Z","iopub.status.idle":"2022-07-13T19:02:13.985533Z","shell.execute_reply.started":"2022-07-13T19:02:13.980761Z","shell.execute_reply":"2022-07-13T19:02:13.984448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#1.Tamanho\nstores.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:13.986909Z","iopub.execute_input":"2022-07-13T19:02:13.987228Z","iopub.status.idle":"2022-07-13T19:02:13.999992Z","shell.execute_reply.started":"2022-07-13T19:02:13.987200Z","shell.execute_reply":"2022-07-13T19:02:13.998741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#2. Como é a base\nstores.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.001802Z","iopub.execute_input":"2022-07-13T19:02:14.002638Z","iopub.status.idle":"2022-07-13T19:02:14.016490Z","shell.execute_reply.started":"2022-07-13T19:02:14.002593Z","shell.execute_reply":"2022-07-13T19:02:14.015202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#3. Missings\nstores.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.017958Z","iopub.execute_input":"2022-07-13T19:02:14.019065Z","iopub.status.idle":"2022-07-13T19:02:14.029549Z","shell.execute_reply.started":"2022-07-13T19:02:14.019018Z","shell.execute_reply":"2022-07-13T19:02:14.028694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nstores.groupby([\"Type\"]).agg({\"Size\":[min,\"mean\", \"median\",max,\"std\"],\"Type\":\"count\"})\n#tipo parece ter a ver com o porte da loja","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.030864Z","iopub.execute_input":"2022-07-13T19:02:14.031596Z","iopub.status.idle":"2022-07-13T19:02:14.066444Z","shell.execute_reply.started":"2022-07-13T19:02:14.031550Z","shell.execute_reply":"2022-07-13T19:02:14.065055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Existe uma segmentação das lojas. Aparentemente pode ter relação com o tamanho, mas deve ser avaliada a relação com as vendas.","metadata":{}},{"cell_type":"markdown","source":"Cruzando train e store para avaliar tamanho e tipo em relação à variavel resposta","metadata":{}},{"cell_type":"code","source":"train_store=pd.merge(train,stores,on=[\"Store\"],how=\"left\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.067924Z","iopub.execute_input":"2022-07-13T19:02:14.071075Z","iopub.status.idle":"2022-07-13T19:02:14.174781Z","shell.execute_reply.started":"2022-07-13T19:02:14.071031Z","shell.execute_reply":"2022-07-13T19:02:14.173383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_store.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.177063Z","iopub.execute_input":"2022-07-13T19:02:14.177407Z","iopub.status.idle":"2022-07-13T19:02:14.195039Z","shell.execute_reply.started":"2022-07-13T19:02:14.177375Z","shell.execute_reply":"2022-07-13T19:02:14.193888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# boxplot(train_store,\"Ano\",\"Weekly_Sales\",\"Type\",True)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.196501Z","iopub.execute_input":"2022-07-13T19:02:14.199139Z","iopub.status.idle":"2022-07-13T19:02:14.204299Z","shell.execute_reply.started":"2022-07-13T19:02:14.199091Z","shell.execute_reply":"2022-07-13T19:02:14.203491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(train_store,\"Ano\",\"Weekly_Sales\",\"Type\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.205632Z","iopub.execute_input":"2022-07-13T19:02:14.206616Z","iopub.status.idle":"2022-07-13T19:02:14.796164Z","shell.execute_reply.started":"2022-07-13T19:02:14.206562Z","shell.execute_reply":"2022-07-13T19:02:14.795033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"O \"Type\" da loja aparenta ter relação com as vendas. Não sabemos exatamente o que é a variavel, mas podemos assumir que A > B > C Quando comparamos as medianas apesar de B apresentar maior aplitude nos outliers.\n","metadata":{}},{"cell_type":"code","source":"boxplot(train_store,\"Type\",\"Weekly_Sales\",\"Size\",False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:14.797676Z","iopub.execute_input":"2022-07-13T19:02:14.797997Z","iopub.status.idle":"2022-07-13T19:02:15.969824Z","shell.execute_reply.started":"2022-07-13T19:02:14.797967Z","shell.execute_reply":"2022-07-13T19:02:15.968784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#boxplot(train_store,\"Type\",\"Weekly_Sales\",\"Size\",True)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:15.973361Z","iopub.execute_input":"2022-07-13T19:02:15.973698Z","iopub.status.idle":"2022-07-13T19:02:15.978429Z","shell.execute_reply.started":"2022-07-13T19:02:15.973667Z","shell.execute_reply":"2022-07-13T19:02:15.977270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"De forma geral os dados indicam que pode haver uma relação entre tamanho x tipo de loja x vendas","metadata":{}},{"cell_type":"markdown","source":"## Avaliando a Base Features\n","metadata":{}},{"cell_type":"markdown","source":"\nfeatures.csv\n\nEste arquivo contém dados adicionais relacionados à loja, departamento e atividade regional para as datas especificadas. Ele contém os seguintes campos:\n\nStore - O número da loja\nDate - A semana\nTemperatura - temperatura média na região\nFuel_Price - custo do combustível na região\nMarkDown1-5 - dados anônimizados relacionados a descontos promocionais que o Walmart está executando. Os MarkDown só estão disponíveis após novembro de 2011 e não estão disponíveis para todas as lojas o tempo todo. Qualquer valor ausente é marcado com um NA.\nCPI - O índice de preços ao consumidor (Inflação)\nDesemprego - A taxa de desemprego\nIsHoliday - Se a semana é uma semana de feriado\nPor conveniência, os quatro feriados se enquadram nas semanas a seguir no conjunto de dados (nem todos os feriados estão nos dados):\n\nSuper Bowl: 12-Feb-10, 11-Feb-11, 10-Feb-12, 8-Feb-13\nLabor Day: 10-Sep-10, 9-Sep-11, 7-Sep-12, 6-Sep-13\nThanksgiving: 26-Nov-10, 25-Nov-11, 23-Nov-12, 29-Nov-13\nChristmas: 31-Dec-10, 30-Dec-11, 28-Dec-12, 27-Dec-13","metadata":{}},{"cell_type":"code","source":"#Perguntas sobre a base Store\n# Qual tamanho?\n# Que informação há lá\n# As variaveis possuem missings?\n# Quais conclusões sobre a base","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:15.979932Z","iopub.execute_input":"2022-07-13T19:02:15.980306Z","iopub.status.idle":"2022-07-13T19:02:15.996875Z","shell.execute_reply.started":"2022-07-13T19:02:15.980275Z","shell.execute_reply":"2022-07-13T19:02:15.995761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"As datas da base features vão de: \"+ str(features.Date.min()) +\", até: \"+ str(features.Date.max()))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:15.998266Z","iopub.execute_input":"2022-07-13T19:02:15.998585Z","iopub.status.idle":"2022-07-13T19:02:16.011538Z","shell.execute_reply.started":"2022-07-13T19:02:15.998556Z","shell.execute_reply":"2022-07-13T19:02:16.010621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:16.012579Z","iopub.execute_input":"2022-07-13T19:02:16.012942Z","iopub.status.idle":"2022-07-13T19:02:16.035378Z","shell.execute_reply.started":"2022-07-13T19:02:16.012909Z","shell.execute_reply":"2022-07-13T19:02:16.034379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Missings da base\nfeatures.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:16.036402Z","iopub.execute_input":"2022-07-13T19:02:16.036835Z","iopub.status.idle":"2022-07-13T19:02:16.047200Z","shell.execute_reply.started":"2022-07-13T19:02:16.036777Z","shell.execute_reply":"2022-07-13T19:02:16.046144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Abaixo quero ter uma visão geral de como se comportam as variaveis contidas na base feature em relação à store, dado que estas variaveis estão associadas com o região de localização das lojas.","metadata":{}},{"cell_type":"code","source":"features[[\"Store\",\"Temperature\",\"Fuel_Price\",\"CPI\",\"Unemployment\"]].groupby([\"Store\"]).\\\n                                                                    agg({\"Temperature\":\"median\",\\\n                                                                        \"Fuel_Price\":\"median\",\\\n                                                                        \"CPI\":\"median\",\\\n                                                                        \"Unemployment\":\"median\"})","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:16.048427Z","iopub.execute_input":"2022-07-13T19:02:16.048732Z","iopub.status.idle":"2022-07-13T19:02:16.076223Z","shell.execute_reply.started":"2022-07-13T19:02:16.048705Z","shell.execute_reply":"2022-07-13T19:02:16.074942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Existem diferenças nas medianas entre as lojas, vale avaliar as distribuições","metadata":{}},{"cell_type":"code","source":"features[\"Anomesdia\"]=pd.to_datetime(features['Date']).dt.strftime('%Y%m%d').astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:16.077754Z","iopub.execute_input":"2022-07-13T19:02:16.078293Z","iopub.status.idle":"2022-07-13T19:02:16.127964Z","shell.execute_reply.started":"2022-07-13T19:02:16.078248Z","shell.execute_reply":"2022-07-13T19:02:16.126827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nos gráficos seguintes avaliaremos como estão distribuidas para cada Loja, as variaveis: CPI, Unemployment, Fuel_Price, Temperature.","metadata":{}},{"cell_type":"code","source":"boxplot(features,\"Store\",'CPI',\"\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:16.129344Z","iopub.execute_input":"2022-07-13T19:02:16.129867Z","iopub.status.idle":"2022-07-13T19:02:16.903685Z","shell.execute_reply.started":"2022-07-13T19:02:16.129829Z","shell.execute_reply":"2022-07-13T19:02:16.902332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(features,\"Store\",'Unemployment',\"\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:16.905248Z","iopub.execute_input":"2022-07-13T19:02:16.905998Z","iopub.status.idle":"2022-07-13T19:02:17.929390Z","shell.execute_reply.started":"2022-07-13T19:02:16.905951Z","shell.execute_reply":"2022-07-13T19:02:17.928630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(features,\"Store\",'Fuel_Price',\"\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:17.930462Z","iopub.execute_input":"2022-07-13T19:02:17.931478Z","iopub.status.idle":"2022-07-13T19:02:18.711360Z","shell.execute_reply.started":"2022-07-13T19:02:17.931440Z","shell.execute_reply":"2022-07-13T19:02:18.710123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxplot(features,\"Store\",'Temperature',\"\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:18.712675Z","iopub.execute_input":"2022-07-13T19:02:18.712993Z","iopub.status.idle":"2022-07-13T19:02:19.475651Z","shell.execute_reply.started":"2022-07-13T19:02:18.712964Z","shell.execute_reply":"2022-07-13T19:02:19.474882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cada loja aparentemente esta localizada em uma região especifica com suas peculiaridades de temperatura, inflação, desemprego e preço de combustível\n\nTemperaturas frias podem ter maior impacto em regiões com medianas de temperatura mais quentes.\n\nFaz sentido testar se variações dentro das regiões (Normalizar por região) possuem correlação com a variavel resposta.","metadata":{}},{"cell_type":"code","source":"%%capture\n#Normaliza os valores para cada agrupamento pelo desvio padrão e imputa missing pela mediana (menor influencia por outliers)\ndef normaliza_dentro_grupo(df,var_num,grupo):\n    df_final=pd.DataFrame()\n    for idx,i in enumerate(df[grupo].unique()):\n        df1=df[df[grupo]==i][var_num].fillna(df[df[grupo]==i].median())\n        df1=(df1-df1.mean())/df1.std()\n        df_final=df_final.append(pd.DataFrame(df1))\n    return df_final.sort_index()\nfeatures[\"Unemployment_norm\"]=normaliza_dentro_grupo(features,\"Unemployment\",\"Store\")\nfeatures[\"CPI_norm\"]=normaliza_dentro_grupo(features,\"CPI\",\"Store\")\nfeatures[\"Temperature_norm\"]=normaliza_dentro_grupo(features,\"Temperature\",\"Store\")\nfeatures[\"Fuel_price_norm\"]=normaliza_dentro_grupo(features,\"Fuel_Price\",\"Store\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:19.476834Z","iopub.execute_input":"2022-07-13T19:02:19.477317Z","iopub.status.idle":"2022-07-13T19:02:20.581005Z","shell.execute_reply.started":"2022-07-13T19:02:19.477288Z","shell.execute_reply":"2022-07-13T19:02:20.579940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Imputando missings pela mediana (menor influencia por outliers)\nfeatures['Temperature'].fillna(features['Temperature'].median(), inplace=True)\nfeatures['Fuel_Price'].fillna(features['Fuel_Price'].median(), inplace=True)\nfeatures['CPI'].fillna(features['CPI'].median(), inplace=True)\nfeatures['Unemployment'].fillna(features['Unemployment'].median(), inplace=True)\nfeatures.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:20.582140Z","iopub.execute_input":"2022-07-13T19:02:20.582407Z","iopub.status.idle":"2022-07-13T19:02:20.596624Z","shell.execute_reply.started":"2022-07-13T19:02:20.582382Z","shell.execute_reply":"2022-07-13T19:02:20.595631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transformando em datetime para cruzar com a base train_store\n\nfeatures[\"Date\"]=pd.to_datetime(features[\"Date\"])\n\n#Cruza a base Trains Store com a base Features\n\nbase_treino=pd.merge(train_store,features.drop(columns=[\"IsHoliday\"],axis=1,errors=\"ignore\"),left_on=[\"Date\",\"Store\"],right_on=[\"Date\",\"Store\"],how=\"left\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:20.598123Z","iopub.execute_input":"2022-07-13T19:02:20.598606Z","iopub.status.idle":"2022-07-13T19:02:20.749304Z","shell.execute_reply.started":"2022-07-13T19:02:20.598566Z","shell.execute_reply":"2022-07-13T19:02:20.748192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Os valores missing nos markdownsserão substituidos pelo valor 0, trataremos como se a ausencia da info indicasse que não houveram de ações promocionais.\nbase_treino['MarkDown1'].fillna(0, inplace=True)\nbase_treino['MarkDown2'].fillna(0, inplace=True)\nbase_treino['MarkDown3'].fillna(0, inplace=True)\nbase_treino['MarkDown4'].fillna(0, inplace=True)\nbase_treino['MarkDown5'].fillna(0, inplace=True)\nbase_treino.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:20.750628Z","iopub.execute_input":"2022-07-13T19:02:20.750969Z","iopub.status.idle":"2022-07-13T19:02:20.804597Z","shell.execute_reply.started":"2022-07-13T19:02:20.750938Z","shell.execute_reply":"2022-07-13T19:02:20.803569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Agora iremos avaliar as variaveis e suas correspondentes normalizadas, da base features em relação às vendas ","metadata":{}},{"cell_type":"code","source":"def scatter(dataset, var1,var1_norm,var2,segmentacao):\n\n    fig, (ax1,ax2) = plt.subplots(1,2, figsize=(16,6))\n\n    if segmentacao==\"\":\n        sns.scatterplot(data=dataset[[var1,var2]], \n                        x=var1, y=var2,ax=ax1)\n\n        sns.scatterplot(data=dataset[[var1_norm,var2]], \n                        x=var1, y=var2,ax=ax2)\n    else:\n\n        sns.scatterplot(data=dataset[[var1,var2,segmentacao]], \n                        x=var1, y=var2, \n                        hue= segmentacao,ax=ax1)\n        sns.scatterplot(data=dataset[[var1_norm,var2,segmentacao]], \n                        x=var1_norm, y=var2, \n                        hue= segmentacao,ax=ax2)\n    return ","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:20.805735Z","iopub.execute_input":"2022-07-13T19:02:20.806318Z","iopub.status.idle":"2022-07-13T19:02:20.814523Z","shell.execute_reply.started":"2022-07-13T19:02:20.806287Z","shell.execute_reply":"2022-07-13T19:02:20.813366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scatter(base_treino, 'Fuel_Price',\"Fuel_price_norm\",\"Weekly_Sales\",\"IsHoliday\")\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:20.815964Z","iopub.execute_input":"2022-07-13T19:02:20.816458Z","iopub.status.idle":"2022-07-13T19:02:41.171766Z","shell.execute_reply.started":"2022-07-13T19:02:20.816421Z","shell.execute_reply":"2022-07-13T19:02:41.170962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fuel_price não apresenta uma correlação evidente com a variavel resposta, assim como sua correspondente normalizada \"Fuel_price_norm\"","metadata":{}},{"cell_type":"code","source":"scatter(base_treino, 'CPI',\"CPI_norm\",\"Weekly_Sales\",\"IsHoliday\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:02:41.173011Z","iopub.execute_input":"2022-07-13T19:02:41.173535Z","iopub.status.idle":"2022-07-13T19:03:01.544604Z","shell.execute_reply.started":"2022-07-13T19:02:41.173492Z","shell.execute_reply":"2022-07-13T19:03:01.543788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CPI não apresenta uma correlação evidente com a variavel resposta, assim como sua correspondente normalizada \"CPI_norm\"","metadata":{}},{"cell_type":"code","source":"scatter(base_treino, 'Temperature',\"Temperature_norm\",\"Weekly_Sales\",\"IsHoliday\")\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:01.545914Z","iopub.execute_input":"2022-07-13T19:03:01.546424Z","iopub.status.idle":"2022-07-13T19:03:21.878455Z","shell.execute_reply.started":"2022-07-13T19:03:01.546394Z","shell.execute_reply":"2022-07-13T19:03:21.877704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#alterado 11/07/2022\n\nTemperatura apresenta uma concentração em uma região especifica (entre 20F e 80F) e pode apresentar alguma peculiaridade não linear que possa influenciar nas vendas. Observando pela Temperatura normalizada não há uma evidente alteração (em comparação com a não normalizada) nas vendas quando as temperaturas ficam muito distantes das médias correspondentes (para cada store).","metadata":{}},{"cell_type":"code","source":"scatter(base_treino,'Unemployment',\"Unemployment_norm\",\"Weekly_Sales\",\"IsHoliday\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:21.879587Z","iopub.execute_input":"2022-07-13T19:03:21.880571Z","iopub.status.idle":"2022-07-13T19:03:41.459062Z","shell.execute_reply.started":"2022-07-13T19:03:21.880538Z","shell.execute_reply":"2022-07-13T19:03:41.457847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unemployment não apresenta uma correlação evidente com a variavel resposta, assim como sua correspondente normalizada \"Unemployment_norm\".","metadata":{}},{"cell_type":"markdown","source":"#Alterado 11/07/2022,\n\nEsperava-se que a normalização das variaveis Unemployment, Fuel_Price, CPI e Temperature, agregando por store, trouxesse algum insight em relação ao comportamento dos clientes frente às variações externas, pois os efeitos estariam diluidos pela visão massificada. Quando abrimos as variaveis por feriado, num primeiro momento tambem não é possivel estabelecer uma relação clara.","metadata":{}},{"cell_type":"markdown","source":"# TRATAMENTO DOS DADOS","metadata":{}},{"cell_type":"markdown","source":"Nos passos seguintes será feita a recategorização das variaveis Store, Dept, e Type ordenando pela variavel resposta. Para Store e Dept","metadata":{}},{"cell_type":"code","source":"def cat_por_target(df,var,resposta,metrica):\n    base_group=((df[[var,resposta]].groupby([var], as_index=False).agg({resposta:metrica}).sort_values(by=resposta,ascending=False)).reset_index(drop=True)).reset_index()\n    retorno = dict((v, k) for k, v in enumerate(base_group[var].astype(str).unique(), 1))\n    return retorno","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:41.460712Z","iopub.execute_input":"2022-07-13T19:03:41.461157Z","iopub.status.idle":"2022-07-13T19:03:41.469755Z","shell.execute_reply.started":"2022-07-13T19:03:41.461118Z","shell.execute_reply":"2022-07-13T19:03:41.468623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"str(cat_por_target(base_treino,\"Dept\",\"Weekly_Sales\",\"median\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:41.471064Z","iopub.execute_input":"2022-07-13T19:03:41.471413Z","iopub.status.idle":"2022-07-13T19:03:41.504698Z","shell.execute_reply.started":"2022-07-13T19:03:41.471384Z","shell.execute_reply":"2022-07-13T19:03:41.503932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Fillna 40 pois qualquer valor missing (se for criado um novo departamento ou se houver algum problema no envio desta variavel) \n# será considerado como havendo valor intermediário de Vendas. \nbase_treino[\"Dept_cat\"]=(base_treino[\"Dept\"].astype(str).map(cat_por_target(base_treino,\"Dept\",\"Weekly_Sales\",\"median\")).fillna(40)).astype(float)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:41.505614Z","iopub.execute_input":"2022-07-13T19:03:41.506483Z","iopub.status.idle":"2022-07-13T19:03:41.796119Z","shell.execute_reply.started":"2022-07-13T19:03:41.506448Z","shell.execute_reply":"2022-07-13T19:03:41.795053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"str(cat_por_target(base_treino,\"Store\",\"Weekly_Sales\",\"median\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:41.797230Z","iopub.execute_input":"2022-07-13T19:03:41.798030Z","iopub.status.idle":"2022-07-13T19:03:41.865980Z","shell.execute_reply.started":"2022-07-13T19:03:41.797996Z","shell.execute_reply":"2022-07-13T19:03:41.864896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Fillna 22 pois qualquer valor missing (se for criado um novo departamento ou se houver algum problema no envio desta variavel) \n# será considerado como havendo valor intermediário de Vendas. \nbase_treino[\"Store_cat\"]=(base_treino[\"Store\"].astype(str).map(cat_por_target(base_treino,\"Store\",\"Weekly_Sales\",\"median\")).fillna(22)).astype(float)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:41.867520Z","iopub.execute_input":"2022-07-13T19:03:41.868083Z","iopub.status.idle":"2022-07-13T19:03:42.170332Z","shell.execute_reply.started":"2022-07-13T19:03:41.868051Z","shell.execute_reply":"2022-07-13T19:03:42.169194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"str(cat_por_target(base_treino,\"Type\",\"Weekly_Sales\",\"median\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:42.171877Z","iopub.execute_input":"2022-07-13T19:03:42.172200Z","iopub.status.idle":"2022-07-13T19:03:42.259335Z","shell.execute_reply.started":"2022-07-13T19:03:42.172172Z","shell.execute_reply":"2022-07-13T19:03:42.257970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Aplica recategorização à variavel Type\n#Neste caso o imputaremos o missing com o tipo de loja mais comum (B) \nbase_treino[\"Type_cat\"]=(base_treino[\"Type\"].astype(str).map(cat_por_target(base_treino,\"Type\",\"Weekly_Sales\",\"median\")).fillna(2)).astype(float)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:42.260884Z","iopub.execute_input":"2022-07-13T19:03:42.261203Z","iopub.status.idle":"2022-07-13T19:03:42.333165Z","shell.execute_reply.started":"2022-07-13T19:03:42.261173Z","shell.execute_reply":"2022-07-13T19:03:42.331991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Transforma a variavel IsHoliday em numérico\nbase_treino['IsHoliday_cat'] = base_treino['IsHoliday'].apply(lambda x: 1 if x else 0)\nbase_treino['IsHoliday_cat'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:42.334700Z","iopub.execute_input":"2022-07-13T19:03:42.335051Z","iopub.status.idle":"2022-07-13T19:03:42.468529Z","shell.execute_reply.started":"2022-07-13T19:03:42.335022Z","shell.execute_reply":"2022-07-13T19:03:42.467404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Estas variavel conta quantos dias faltam para o Thanksgiving e para o Natal\nbase_treino['Dias_p_Thansksgiving'] = (pd.to_datetime(base_treino[\"Ano\"].astype(str)+\"-11-24\", format=\"%Y-%m-%d\") - pd.to_datetime(base_treino[\"Date\"], format=\"%Y-%m-%d\")).dt.days.astype(float)\nbase_treino['Dias_p_Natal'] = (pd.to_datetime(base_treino[\"Ano\"].astype(str)+\"-12-24\", format=\"%Y-%m-%d\") - pd.to_datetime(base_treino[\"Date\"], format=\"%Y-%m-%d\")).dt.days.astype(float)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:03:42.469905Z","iopub.execute_input":"2022-07-13T19:03:42.470218Z","iopub.status.idle":"2022-07-13T19:03:43.199977Z","shell.execute_reply.started":"2022-07-13T19:03:42.470189Z","shell.execute_reply":"2022-07-13T19:03:43.198988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Esta variavel marca uma flag para cada feriado\nbase_treino['SuperBowlWeek'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 6 else 0)\nbase_treino['LaborDay'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 35 else 0)\nbase_treino['Tranksgiving'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 47 else 0)\nbase_treino['Christmas'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 51 else 0)\nbase_treino[\"THX_GVN_TO_XMAS\"]=base_treino['SemanaDoAno'].apply(lambda x: 1 if (x>46) & (x<52)  else 0)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:36:54.174167Z","iopub.execute_input":"2022-07-13T19:36:54.174541Z","iopub.status.idle":"2022-07-13T19:36:54.859469Z","shell.execute_reply.started":"2022-07-13T19:36:54.174511Z","shell.execute_reply":"2022-07-13T19:36:54.858271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Unifica os markdowns em uma variavel só\nbase_treino['MarkdownsSum'] = base_treino['MarkDown1'] + base_treino['MarkDown2'] + base_treino['MarkDown3'] + base_treino['MarkDown4'] + base_treino['MarkDown5'] ","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:36:55.553793Z","iopub.execute_input":"2022-07-13T19:36:55.554202Z","iopub.status.idle":"2022-07-13T19:36:55.568612Z","shell.execute_reply.started":"2022-07-13T19:36:55.554168Z","shell.execute_reply":"2022-07-13T19:36:55.567699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Variaveis que poderiam ter sido inseridas posteriormente 09/07/2022\n\n#base_treino['Tranksgiving'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 47 else 0)#ajuste da semana\n#base_treino['dept_72'] = base_treino['Dept'].apply(lambda x: 1 if x ==72  else 0) #Var com alto impacto em vendas em epoca de Thx givin e Xmas\n#base_treino['dept_35'] = base_treino['Dept'].apply(lambda x: 1 if x ==35  else 0) #Loja com grande quantidade de semanas com prejuizo\n#base_treino['too_hot_too_cold'] = base_treino['Temperature'].apply(lambda x: 1 if ((x >80) |(x<20))  else 0) #tentanto capturar a alguma info nos extremos de temperatura\n\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:36:56.215305Z","iopub.execute_input":"2022-07-13T19:36:56.216080Z","iopub.status.idle":"2022-07-13T19:36:56.219837Z","shell.execute_reply.started":"2022-07-13T19:36:56.216030Z","shell.execute_reply":"2022-07-13T19:36:56.219054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Seleção de Variáveis","metadata":{}},{"cell_type":"markdown","source":"Nesta etapa iremos avaliar a correlação das variaveis entre si e com a variável resposta. Para isso contrui uma função que plota as correlações e aponta as variaveis com alta correlação","metadata":{}},{"cell_type":"code","source":"#Função para plotar a matriz de correlação\ndef print_correlation_matrix(df,explicativas,target,corr_max=0.75):\n\n    sns.set(style=\"white\")\n    corr = round(df[explicativas+[target]].corr(),3)\n    corr_abs = df[explicativas+[target]].corr().abs()\n    upper = corr_abs.where(np.triu(np.ones(corr_abs.shape), k=1).astype(bool))\n    high_corr = [column for column in upper.columns if any(upper[column]>=corr_max)]\n    \n\n    f, ax = plt.subplots(figsize=(21, 21))\n    cmap = sns.diverging_palette(220, 20, as_cmap=True)\n    mask = np.triu(np.ones_like(corr, dtype=bool))\n    \n    \n    sns.heatmap(corr*100,mask=mask, cmap=cmap, vmax=100,vmin=-100, center=0, annot=True,\n                square=True, linewidths=.5, cbar_kws={'shrink': .5},annot_kws={\"size\":12},fmt=\".0f\")\n    sns.set(font_scale=2)\n    plt.show()\n    return high_corr","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:36:57.826312Z","iopub.execute_input":"2022-07-13T19:36:57.826731Z","iopub.status.idle":"2022-07-13T19:36:57.839241Z","shell.execute_reply.started":"2022-07-13T19:36:57.826687Z","shell.execute_reply":"2022-07-13T19:36:57.837353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Ajustado em 09/07/2022\nvars_inicio=list(base_treino.columns)\n#Estou dropando estas vars da lista pois umas eu fiz recategorização e outras não fazem sentido estarem na analise\nretirar = ['Store', 'Dept','IsHoliday', 'Type', 'Ano', 'Date', 'Weekly_Sales', 'Anomesdia']\nvars_explicativas = [e for e in vars_inicio if e not in retirar]\nvars_explicativas","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:36:58.817392Z","iopub.execute_input":"2022-07-13T19:36:58.817881Z","iopub.status.idle":"2022-07-13T19:36:58.827994Z","shell.execute_reply.started":"2022-07-13T19:36:58.817835Z","shell.execute_reply":"2022-07-13T19:36:58.826368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"high_corr=print_correlation_matrix(base_treino,vars_explicativas,'Weekly_Sales',0.75)\nhigh_corr","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:36:59.657975Z","iopub.execute_input":"2022-07-13T19:36:59.658412Z","iopub.status.idle":"2022-07-13T19:37:03.948445Z","shell.execute_reply.started":"2022-07-13T19:36:59.658376Z","shell.execute_reply":"2022-07-13T19:37:03.947403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As variaveis acima estão altamente correlacionadas com outras. Aqui o critério adotado foi manter a variavel que tiver maior correlação com o target.\nObs: Um ponto de melhoria aqui seria automatizar a decisão de dropar, dentre as com alta correlação, a que possuir menor correlação com a var respota.  ","metadata":{}},{"cell_type":"code","source":"#Ajustado 09/07/2022\n\nretirar2=['Dias_p_Thansksgiving', 'Dias_p_Natal',\"SemanaDoAno\",'Anomesdia', 'MarkDown1', 'MarkDown2', 'MarkDown3', 'MarkDown4',\"Size\", 'MarkDown5',\"MarkdownsSum\", 'Unemployment_norm','CPI_norm',\"Fuel_price_norm\",\"Temperature_norm\"]\nvars_explicativas2 = [e for e in vars_explicativas if e not in retirar2]\nvars_explicativas2","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:03.950532Z","iopub.execute_input":"2022-07-13T19:37:03.950879Z","iopub.status.idle":"2022-07-13T19:37:03.959954Z","shell.execute_reply.started":"2022-07-13T19:37:03.950847Z","shell.execute_reply":"2022-07-13T19:37:03.957457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Size é altamente correlacionada com Store e Tipo e será retirada.\nSemanaDoAno é altamente correlacionada com Mes e será retirada.\nDias_p_Thansksgiving e Dias_p_Natal são altamente correlacionada com Mes e dia e serão retirados.\nOs 'MarkDown' de 1 a 5 não possuem correlação significante com 'Weekly_Sales' e possuem muitos missings principalmente antes de 2011-11. Se eu fosse manter os markdowns faria sentido treinar o modelo selecionando a base apartir de 2011-11. No entanto perderiamos informações valiosas das variaveis que possuem maior correlação com o target. Por isso decidi retir-las da base.\nA variavel Unemployment_norm norm possui baixissima correlação com a variavel resposta utilizaremos somente Unemployment.\nA variavel CPI_norm norm possui baixissima correlação com a variavel resposta utilizaremos somente CPI.\nAs variaveis Temperature_norm e Fuel_price_norm possuem baixa correlação com o target, no entanto maior que suas correspondentes não normalizadas(Temperature, Fuel). Pensando pelo lado de implantação, não valeria a pena manter duas variaveis com baixa correlação mas que demandariam implantação de uma base com histórico para calcular média e desvio padrao.\n","metadata":{}},{"cell_type":"code","source":"#Apos a retirada das variaveis\nhigh_corr_after_cut=print_correlation_matrix(base_treino,vars_explicativas2,'Weekly_Sales',0.75)\nhigh_corr_after_cut","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:03.961460Z","iopub.execute_input":"2022-07-13T19:37:03.962168Z","iopub.status.idle":"2022-07-13T19:37:05.508601Z","shell.execute_reply.started":"2022-07-13T19:37:03.962133Z","shell.execute_reply":"2022-07-13T19:37:05.507846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aqui observamos que não há mais nenhuma variavel com correlação acima do limite estipulado então utilizaremos estas variaveis para modelagem.","metadata":{}},{"cell_type":"markdown","source":"# MODELAGEM","metadata":{}},{"cell_type":"markdown","source":"No passo de modelagem vamos utilizar 4 metodos de machine learning para predizer as vendas semanais:\n- LASSO\n- Decision Tree\n- Random Forest\n- LGBM\n\nPor questões de processamento e tempo, o refinamento de hiperparametros será tera de ser simplificado.","metadata":{}},{"cell_type":"markdown","source":"Separaremos a base em modelagem e validação Out Of Time. Utilizaremos a base modelagem para treinar os modelos e avaliaremos o desempenho dos modelos considerando dados de fora do treinaome.","metadata":{}},{"cell_type":"code","source":"modelagem=base_treino[base_treino[\"Anomesdia\"]<20120726]\nvalida_OOT=base_treino[base_treino[\"Anomesdia\"]>=20120726]\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:14.149797Z","iopub.execute_input":"2022-07-13T19:37:14.150185Z","iopub.status.idle":"2022-07-13T19:37:14.223719Z","shell.execute_reply.started":"2022-07-13T19:37:14.150145Z","shell.execute_reply":"2022-07-13T19:37:14.222894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Cria a base para treino e validacao out of time\nX_treino=modelagem[vars_explicativas2].drop(columns=['Weekly_Sales',\"Date\",\"Anomesdia\"],axis=1,errors=\"ignore\")\ny_treino=modelagem[\"Weekly_Sales\"]\nX_val=valida_OOT[vars_explicativas2].drop(columns=['Weekly_Sales',\"Date\",\"Anomesdia\"],axis=1,errors=\"ignore\")\ny_val=valida_OOT[\"Weekly_Sales\"]","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:15.621361Z","iopub.execute_input":"2022-07-13T19:37:15.622106Z","iopub.status.idle":"2022-07-13T19:37:15.693884Z","shell.execute_reply.started":"2022-07-13T19:37:15.622062Z","shell.execute_reply":"2022-07-13T19:37:15.692510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# # LASSO","metadata":{}},{"cell_type":"markdown","source":"    \n Esta função auxilia na avaliação dos modelos utilizando as métricas escolhidas\n Entradas: Nome dado ao modelo, valores reais e valores preditos \n Saídas: Dataframe com nome do modelo, erro médio absoluto e erro médio quadrado","metadata":{}},{"cell_type":"code","source":"def ml_error(model_name, df,y,pred,var_feriado):\n\n    \n    mae = mean_absolute_error(y, pred)\n    rmse = np.sqrt(mean_squared_error(y, pred))  \n    \n    weights = df[var_feriado].apply(lambda x: 1 if x==0 else 5)\n    wmae= np.round(np.sum(weights*abs(y-pred))/(np.sum(weights)), 2)\n    \n    return pd.DataFrame({'Model Name' : model_name,\n                        'MAE' : mae,\n                        'RMSE' : rmse,\n                        'WMAE':wmae}, index = [0])","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:16.930014Z","iopub.execute_input":"2022-07-13T19:37:16.930454Z","iopub.status.idle":"2022-07-13T19:37:16.937783Z","shell.execute_reply.started":"2022-07-13T19:37:16.930417Z","shell.execute_reply":"2022-07-13T19:37:16.936096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"code","source":"#Esta função realiza grid search no modelo escolhido, com os parametros escolhidos para avaliação\ndef modelos(modelo,parametros,X,Y,cv=None):\n    # model\n    mod = modelo\n    mod_grid = GridSearchCV(mod, parametros, cv=cv)\n    mod_grid.fit(X, Y)\n    \n    return mod_grid,mod_grid.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:17.904231Z","iopub.execute_input":"2022-07-13T19:37:17.904855Z","iopub.status.idle":"2022-07-13T19:37:17.911170Z","shell.execute_reply.started":"2022-07-13T19:37:17.904790Z","shell.execute_reply":"2022-07-13T19:37:17.909961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Roda Lasso testando alguns valores de alfa e fazendo cross validation\nscaler = MinMaxScaler()\nscaler.fit(X_treino)\nX_treino1= scaler.transform(X_treino)\n    \nparameters = {'alpha':[0.01,0.05, 0.1 ,0.15,0.175,0.2, 0.5, 2, 5]}\nmodelo_lasso, melhor_param_lasso =modelos(Lasso(),parameters,X_treino1,y_treino,15)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:37:20.740067Z","iopub.execute_input":"2022-07-13T19:37:20.740685Z","iopub.status.idle":"2022-07-13T19:38:07.583588Z","shell.execute_reply.started":"2022-07-13T19:37:20.740636Z","shell.execute_reply":"2022-07-13T19:38:07.582570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"melhor_param_lasso","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:38:07.585628Z","iopub.execute_input":"2022-07-13T19:38:07.586057Z","iopub.status.idle":"2022-07-13T19:38:07.599384Z","shell.execute_reply.started":"2022-07-13T19:38:07.586016Z","shell.execute_reply":"2022-07-13T19:38:07.594751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vars_explicativas2","metadata":{"execution":{"iopub.status.busy":"2022-07-13T20:01:08.475803Z","iopub.execute_input":"2022-07-13T20:01:08.476208Z","iopub.status.idle":"2022-07-13T20:01:08.485664Z","shell.execute_reply.started":"2022-07-13T20:01:08.476177Z","shell.execute_reply":"2022-07-13T20:01:08.484178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"importancia=np.abs(modelo_lasso.best_estimator_.coef_)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T20:05:49.237100Z","iopub.execute_input":"2022-07-13T20:05:49.237546Z","iopub.status.idle":"2022-07-13T20:05:49.243444Z","shell.execute_reply.started":"2022-07-13T20:05:49.237510Z","shell.execute_reply":"2022-07-13T20:05:49.242143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dict(zip(vars_explicativas2, importancia.T))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"importance_df = pd.DataFrame({\n    'feature': X_treino1.columns,\n    'importance': np.abs(modelo_lasso.best_estimator_.coef_)\n}).sort_values('importance', ascending=False)\n    \nplt.figure(figsize=(16,10))\nplt.title('Feature Importance')\nsns.barplot(data=importance_df, x='importance', y='feature')\n\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T20:09:13.363213Z","iopub.execute_input":"2022-07-13T20:09:13.363730Z","iopub.status.idle":"2022-07-13T20:09:13.730748Z","shell.execute_reply.started":"2022-07-13T20:09:13.363687Z","shell.execute_reply":"2022-07-13T20:09:13.729526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred_treino_lasso = modelo_lasso.predict(scaler.transform(X_treino))\ny_pred_val_lasso = modelo_lasso.predict(scaler.transform(X_val))\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:38:07.601034Z","iopub.execute_input":"2022-07-13T19:38:07.601581Z","iopub.status.idle":"2022-07-13T19:38:07.672630Z","shell.execute_reply.started":"2022-07-13T19:38:07.601525Z","shell.execute_reply":"2022-07-13T19:38:07.671400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Result_Lasso=ml_error(\"LASSO_Treino\", X_treino,y_treino,y_pred_treino_lasso,\"IsHoliday_cat\")\nResult_Lasso=Result_Lasso.append(ml_error(\"LASSO_OOT\", X_val,y_val,y_pred_val_lasso,\"IsHoliday_cat\"))\nResult_Lasso","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:38:07.675312Z","iopub.execute_input":"2022-07-13T19:38:07.676013Z","iopub.status.idle":"2022-07-13T19:38:07.901640Z","shell.execute_reply.started":"2022-07-13T19:38:07.675970Z","shell.execute_reply":"2022-07-13T19:38:07.900502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# # DECISION TREE","metadata":{}},{"cell_type":"code","source":"parameters_DT = { 'max_depth': [50,100,150],\n               'min_samples_split': [50,100]\n              #'max_features':[\"log2\"]\n               ,'max_features': [0.5,0.75]}\n\n#Treina o modelo Decision_Tree testando os parametros acima com GridSearch e crossvalidation\nmodelo_DT,melhor_param_DT =modelos(DecisionTreeRegressor(),parameters_DT,X_treino,y_treino,10)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:38:07.902878Z","iopub.execute_input":"2022-07-13T19:38:07.903224Z","iopub.status.idle":"2022-07-13T19:40:33.282287Z","shell.execute_reply.started":"2022-07-13T19:38:07.903194Z","shell.execute_reply":"2022-07-13T19:40:33.281072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"melhor_param_DT","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:40:33.283938Z","iopub.execute_input":"2022-07-13T19:40:33.284376Z","iopub.status.idle":"2022-07-13T19:40:33.292192Z","shell.execute_reply.started":"2022-07-13T19:40:33.284334Z","shell.execute_reply":"2022-07-13T19:40:33.291051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Escora o modelo\ny_pred_treino_DT = modelo_DT.predict(X_treino)\ny_pred_val_DT = modelo_DT.predict(X_val)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:40:33.293860Z","iopub.execute_input":"2022-07-13T19:40:33.294482Z","iopub.status.idle":"2022-07-13T19:40:33.368671Z","shell.execute_reply.started":"2022-07-13T19:40:33.294448Z","shell.execute_reply":"2022-07-13T19:40:33.367879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Avalia o desempenho\nResult_DT=ml_error(\"DT_Treino\", X_treino,y_treino,y_pred_treino_DT,\"IsHoliday_cat\")\nResult_DT=Result_DT.append(ml_error(\"DT_OOT\", X_val,y_val,y_pred_val_DT,\"IsHoliday_cat\"))\nResult_DT","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:40:33.370492Z","iopub.execute_input":"2022-07-13T19:40:33.371328Z","iopub.status.idle":"2022-07-13T19:40:33.530504Z","shell.execute_reply.started":"2022-07-13T19:40:33.371280Z","shell.execute_reply":"2022-07-13T19:40:33.529706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Função para plotar a feature importance\n\ndef plota_feature_importance(X,modelo):\n    \n    importance_df = pd.DataFrame({\n        'feature': X.columns,\n        'importance': modelo.best_estimator_.feature_importances_\n    }).sort_values('importance', ascending=False)\n\n    plt.figure(figsize=(16,10))\n    plt.title('Feature Importance')\n    sns.barplot(data=importance_df, x='importance', y='feature')","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:40:33.531636Z","iopub.execute_input":"2022-07-13T19:40:33.532386Z","iopub.status.idle":"2022-07-13T19:40:33.539361Z","shell.execute_reply.started":"2022-07-13T19:40:33.532343Z","shell.execute_reply":"2022-07-13T19:40:33.537915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Plota Feature Importance do Decisio Tree\nplota_feature_importance(X_treino,modelo_DT)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:40:33.542936Z","iopub.execute_input":"2022-07-13T19:40:33.543492Z","iopub.status.idle":"2022-07-13T19:40:34.133911Z","shell.execute_reply.started":"2022-07-13T19:40:33.543458Z","shell.execute_reply":"2022-07-13T19:40:34.132782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"É possivel observar que as variaveis Dept, Store e Tipo são as que possuem mais importancia para este modelo","metadata":{}},{"cell_type":"markdown","source":"# # RANDOM FOREST","metadata":{}},{"cell_type":"code","source":"#Testei um numero menor de parametros e sem crossvalidation por falta de tempo e pelas limitações do Kaggle\nparameters_RF =   {'n_estimators': [100,150]\n                  ,'max_depth': [None]\n                  ,'max_features': [0.75]\n                  ,\"n_jobs\":[-1]\n                  }\n\n#Treina o modelo Random Forest testando os parametros acima com GridSearch e crossvalidation\nmodelo_RF,melhor_param_RF =modelos(RandomForestRegressor(),parameters_RF,X_treino,y_treino)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:40:34.135745Z","iopub.execute_input":"2022-07-13T19:40:34.136989Z","iopub.status.idle":"2022-07-13T19:53:20.858207Z","shell.execute_reply.started":"2022-07-13T19:40:34.136949Z","shell.execute_reply":"2022-07-13T19:53:20.856681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Escora o modelo\ny_pred_treino_RF = modelo_RF.predict(X_treino)\ny_pred_val_RF = modelo_RF.predict(X_val)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:20.860247Z","iopub.execute_input":"2022-07-13T19:53:20.860742Z","iopub.status.idle":"2022-07-13T19:53:25.524721Z","shell.execute_reply.started":"2022-07-13T19:53:20.860689Z","shell.execute_reply":"2022-07-13T19:53:25.523528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Avalia o desempenho\nResult_RF=ml_error(\"RF_Treino\", X_treino,y_treino,y_pred_treino_RF,\"IsHoliday_cat\")\nResult_RF=Result_RF.append(ml_error(\"RF_OOT\", X_val,y_val,y_pred_val_RF,\"IsHoliday_cat\"))\nResult_RF","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:25.526261Z","iopub.execute_input":"2022-07-13T19:53:25.526706Z","iopub.status.idle":"2022-07-13T19:53:25.689650Z","shell.execute_reply.started":"2022-07-13T19:53:25.526640Z","shell.execute_reply":"2022-07-13T19:53:25.688436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Plota a Feature importance\nplota_feature_importance(X_treino,modelo_RF)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:25.690765Z","iopub.execute_input":"2022-07-13T19:53:25.691370Z","iopub.status.idle":"2022-07-13T19:53:26.440975Z","shell.execute_reply.started":"2022-07-13T19:53:25.691334Z","shell.execute_reply":"2022-07-13T19:53:26.439478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"É possivel observar que as variaveis Dept, Store e Tipo são as que possuem mais importancia para este modelo","metadata":{}},{"cell_type":"markdown","source":"# # LIGHTGBM","metadata":{}},{"cell_type":"code","source":"# Este modelo possui a opção de sumeter uma base de validação OOS, entao criei uma função só para ele\n# Neste caso nao utilizei crossvalidation por falta de tempo e pelas limitações do Kaggle.\ndef mod_LGBM(X_treino,Y_treino,X_teste,Y_teste,cv=None):\n    from lightgbm import LGBMRegressor,log_evaluation\n    from lightgbm import early_stopping\n    parameters = {\n    \"learning_rate\": [0.5,0.6,0.7],\n    \"subsample\":[0.75],\n    \"n_estimators\": [150,300,500],\n    \"max_depth\": [15],\n    \"colsample_bytree\":[0.7,0.9],  \n    \"reg_lambda\":[0,1],\n    \"reg_alpha\": [0,1],\n       \"verbose\":[0]\n#      \"scale_pos_weight\": [15],\n    #\"n_jobs\":[-1]\n    }    \n    eval_set = [(X_treino,Y_treino),(X_teste,Y_teste)]\n        \n    regressor=LGBMRegressor (objective='regression', seed=0,verbose_eval=False)\n\n    LGBM_grid = GridSearchCV(regressor,parameters,cv=cv)\n    LGBM_grid.fit(X_treino,Y_treino\n                  ,eval_set=eval_set\n                  ,eval_metric=['rmse']\n                  ,callbacks=[\n                  early_stopping(stopping_rounds=5,first_metric_only=False)\n                  ]\n                  \n                 )\n\n    LGBM_grid.best_params_\n    \n   \n    return LGBM_grid,LGBM_grid.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:26.442574Z","iopub.execute_input":"2022-07-13T19:53:26.443124Z","iopub.status.idle":"2022-07-13T19:53:26.452907Z","shell.execute_reply.started":"2022-07-13T19:53:26.443088Z","shell.execute_reply":"2022-07-13T19:53:26.451370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Da base de treino  extraí uma porção para servir de validação OOS  \n# Dividindo a base de treino em treino e teste\nfrom sklearn.model_selection import train_test_split\nX_treino1, X_val_OOS, y_treino1, y_val_OOS = train_test_split(X_treino # Variáveis Explicativas\n                                                    ,y_treino,  # Variável Resposta\n                                                    test_size = 0.3, # Proporção entre treino e teste\n                                                    random_state = 0)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:26.454548Z","iopub.execute_input":"2022-07-13T19:53:26.454911Z","iopub.status.idle":"2022-07-13T19:53:26.546311Z","shell.execute_reply.started":"2022-07-13T19:53:26.454878Z","shell.execute_reply":"2022-07-13T19:53:26.545158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture\nmodelo_LGBM,melhor_param_LGBM =mod_LGBM(X_treino1,y_treino1,X_val_OOS,y_val_OOS)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:26.548122Z","iopub.execute_input":"2022-07-13T19:53:26.548476Z","iopub.status.idle":"2022-07-13T19:53:39.311914Z","shell.execute_reply.started":"2022-07-13T19:53:26.548443Z","shell.execute_reply":"2022-07-13T19:53:39.310598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"melhor_param_LGBM","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.313733Z","iopub.execute_input":"2022-07-13T19:53:39.314440Z","iopub.status.idle":"2022-07-13T19:53:39.333653Z","shell.execute_reply.started":"2022-07-13T19:53:39.314406Z","shell.execute_reply":"2022-07-13T19:53:39.331328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Escora Modelo LGBM\ny_pred_treino_LGBM = modelo_LGBM.predict(X_treino)\ny_pred_val_OOS_LGBM = modelo_LGBM.predict(X_val_OOS)\ny_pred_val_LGBM = modelo_LGBM.predict(X_val)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.335735Z","iopub.status.idle":"2022-07-13T19:53:39.336719Z","shell.execute_reply.started":"2022-07-13T19:53:39.336341Z","shell.execute_reply":"2022-07-13T19:53:39.336384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Avalia o desempenho\nResult_LGBM=ml_error(\"LGBM_Treino\", X_treino,y_treino,y_pred_treino_LGBM,\"IsHoliday_cat\")\nResult_LGBM=Result_LGBM.append(ml_error(\"LGBM_OOS\", X_val_OOS,y_val_OOS,y_pred_val_OOS_LGBM,\"IsHoliday_cat\"))\nResult_LGBM=Result_LGBM.append(ml_error(\"LGBM_OOT\", X_val,y_val,y_pred_val_LGBM,\"IsHoliday_cat\"))\nResult_LGBM","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.338929Z","iopub.status.idle":"2022-07-13T19:53:39.339547Z","shell.execute_reply.started":"2022-07-13T19:53:39.339349Z","shell.execute_reply":"2022-07-13T19:53:39.339370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Plota Feature Importance do LightGBM\nplota_feature_importance(X_treino,modelo_LGBM)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.340721Z","iopub.status.idle":"2022-07-13T19:53:39.341378Z","shell.execute_reply.started":"2022-07-13T19:53:39.341169Z","shell.execute_reply":"2022-07-13T19:53:39.341195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No modelo Light GBM podemos observar Dept e Store como as mais importantes, mas diferente de Decision Tree e Random Forest, nesse modelo há uma maior contribuição de fafores externos na perfoemance do modelo,","metadata":{}},{"cell_type":"markdown","source":"## Conclusoes ","metadata":{}},{"cell_type":"markdown","source":"Como observado na matriz de correlação as variaveis Dept, Store e Tipo são as que possuem mais correlação com a variavel resposta e consequentemente são as mais importantes para este modelo, o que indica uma maior relação das vendas com as caracteristicas dos departamentos/lojas do que com fatores externos. No entanto o modelo Light GBM conseguiu uma performance satisfatória explorando melhor os fatores externos (CPI/Fuel_Price/Unemployment/Temperature).","metadata":{}},{"cell_type":"markdown","source":"# AVALIAÇÃO","metadata":{}},{"cell_type":"code","source":"Resultado = pd.concat([Result_Lasso,Result_DT,Result_RF,Result_LGBM],axis = 0)\nResultado","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.342744Z","iopub.status.idle":"2022-07-13T19:53:39.343160Z","shell.execute_reply.started":"2022-07-13T19:53:39.342978Z","shell.execute_reply":"2022-07-13T19:53:39.342997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Random forest aparentemente apresentou o melhor desempenho em comparação com os outros. Mesmo com um indicio de overfit","metadata":{}},{"cell_type":"markdown","source":"Abaixo serão comparados os resultados dos modelos (escora a base_treino) comparados com a variavel resposta","metadata":{}},{"cell_type":"code","source":"#Esta variavel vou utilizar para agrupar a informação por mes\nbase_treino[\"Anomes\"]=pd.to_datetime(base_treino['Date']).dt.strftime('%Y%m').astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.344687Z","iopub.status.idle":"2022-07-13T19:53:39.345076Z","shell.execute_reply.started":"2022-07-13T19:53:39.344895Z","shell.execute_reply":"2022-07-13T19:53:39.344912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_treino[\"mod_lasso\"] = modelo_lasso.predict(scaler.transform(base_treino[vars_explicativas2]))\nbase_treino[\"mod_DT\"] = modelo_DT.predict(base_treino[vars_explicativas2])\nbase_treino[\"mod_RF\"] = modelo_RF.predict(base_treino[vars_explicativas2])\nbase_treino[\"mod_LGBM\"] = modelo_LGBM.predict(base_treino[vars_explicativas2])\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.346849Z","iopub.status.idle":"2022-07-13T19:53:39.347232Z","shell.execute_reply.started":"2022-07-13T19:53:39.347057Z","shell.execute_reply":"2022-07-13T19:53:39.347073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_sales_lasso_mes = base_treino.groupby('Anomes')['mod_lasso'].sum()\npredicted_sales_DT_mes= base_treino.groupby('Anomes')['mod_DT'].sum()\npredicted_sales_RF_mes=base_treino.groupby('Anomes')['mod_RF'].sum()\npredicted_sales_LGBM_mes=base_treino.groupby('Anomesdia')['mod_LGBM'].sum()\nWeekly_SALES_mes=base_treino.groupby('Anomes')['Weekly_Sales'].sum()\n","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.349505Z","iopub.status.idle":"2022-07-13T19:53:39.350708Z","shell.execute_reply.started":"2022-07-13T19:53:39.350421Z","shell.execute_reply":"2022-07-13T19:53:39.350448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,6))\nplt.plot(predicted_sales_lasso_mes.values)\nplt.plot(predicted_sales_DT_mes.values)\nplt.plot(predicted_sales_RF_mes.values)\nplt.plot(Weekly_SALES_mes.values)\n\n# plt.xticks(np.arange(1, 53, step=1), fontsize=16)\nplt.yticks( fontsize=16)\nplt.ylabel('Sales', fontsize=20, labelpad=20)\nplt.xlabel('Meses a partir do inicio', fontsize=20, labelpad=20)\n\nplt.title(\"Comparação dos preditos com o real\", fontsize=24)\nplt.legend(['LASSO', 'Decision Tree', 'Random Forest', \"Real\"], fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.352362Z","iopub.status.idle":"2022-07-13T19:53:39.353279Z","shell.execute_reply.started":"2022-07-13T19:53:39.352978Z","shell.execute_reply":"2022-07-13T19:53:39.353006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_sales_lasso_semana = base_treino.groupby('Date')['mod_lasso'].sum()\npredicted_sales_DT_semana= base_treino.groupby('Date')['mod_DT'].sum()\npredicted_sales_RF_semana=base_treino.groupby('Date')['mod_RF'].sum()\npredicted_sales_LGBM_semana=base_treino.groupby('Date')['mod_LGBM'].sum()\nWeekly_SALES_semana=base_treino.groupby('Date')['Weekly_Sales'].sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.354519Z","iopub.status.idle":"2022-07-13T19:53:39.354898Z","shell.execute_reply.started":"2022-07-13T19:53:39.354708Z","shell.execute_reply":"2022-07-13T19:53:39.354724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,6))\nplt.plot(predicted_sales_lasso_semana.values)\nplt.plot(predicted_sales_DT_semana.values)\nplt.plot(predicted_sales_RF_semana.values)\nplt.plot(Weekly_SALES_semana.values)\n\n# plt.xticks(np.arange(1, 53, step=1), fontsize=16)\nplt.yticks( fontsize=16)\nplt.ylabel('Sales', fontsize=20, labelpad=20)\nplt.xlabel('Semanas a partir do inicio', fontsize=20, labelpad=20)\n\nplt.title(\"Comparação dos preditos com o real\", fontsize=24)\nplt.legend(['LASSO', 'Decision Tree', 'Random Forest', \"Real\"], fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.357340Z","iopub.status.idle":"2022-07-13T19:53:39.358422Z","shell.execute_reply.started":"2022-07-13T19:53:39.358095Z","shell.execute_reply":"2022-07-13T19:53:39.358127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Escoragem da base Test.csv ","metadata":{}},{"cell_type":"code","source":"#Faço esta transformação para a variavel date ficar do mesmo tipo de sua correspondente na variavel feature\ntest['Date'] = pd.to_datetime(test['Date'])","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.360219Z","iopub.status.idle":"2022-07-13T19:53:39.361234Z","shell.execute_reply.started":"2022-07-13T19:53:39.360907Z","shell.execute_reply":"2022-07-13T19:53:39.360937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Cruza base teste com features\nbase_teste=pd.merge(test,features.drop(columns=[\"IsHoliday\"],axis=1)\\\n                    ,left_on=[\"Store\",\"Date\"],right_on=[\"Store\",\"Date\"],how=\"left\").merge(stores,left_on=[\"Store\"],right_on=[\"Store\"],how=\"left\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.362650Z","iopub.status.idle":"2022-07-13T19:53:39.363048Z","shell.execute_reply.started":"2022-07-13T19:53:39.362867Z","shell.execute_reply":"2022-07-13T19:53:39.362886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_teste.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.365386Z","iopub.status.idle":"2022-07-13T19:53:39.366342Z","shell.execute_reply.started":"2022-07-13T19:53:39.366037Z","shell.execute_reply":"2022-07-13T19:53:39.366065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# IMPLANTAÇÃO","metadata":{}},{"cell_type":"markdown","source":"#verifica media e mediana de variaveis para inputação","metadata":{}},{"cell_type":"code","source":"features[[\"Unemployment\",\"CPI\",\"Temperature\",\"Fuel_Price\"]].agg({\"Unemployment\":[\"mean\",\"median\"],\"CPI\":[\"mean\",\"median\"],\"Temperature\":[\"mean\",\"median\"],\"Fuel_Price\":[\"mean\",\"median\"]})","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.368081Z","iopub.status.idle":"2022-07-13T19:53:39.369024Z","shell.execute_reply.started":"2022-07-13T19:53:39.368721Z","shell.execute_reply":"2022-07-13T19:53:39.368749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"O códigos abaixos são uma proposta para facilitar a implantação do modelo. \nPrimeiro criei um classe para transformar JSON em Dataframe. Em seguida criei uma classe para transformar os dados de forma que eles sejam compativeis com o modelo. Depois crio um pipeline que ira pegar a informação crua como input e ira gerar como resultado uma base contendo o escore do modelo\n","metadata":{}},{"cell_type":"code","source":"#Transforma JSON em dataframe\nimport json\nclass JsonToDF:\n    def fit(self, X, y):\n        pass\n    def transform(self,input_data):\n        if type(input_data)==str:\n            json_input = str(input_data)\n            return pd.json_normalize(json.loads(json_input))\n        elif type(input_data) == pd.DataFrame:\n            return input_data","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.370670Z","iopub.status.idle":"2022-07-13T19:53:39.371570Z","shell.execute_reply.started":"2022-07-13T19:53:39.371291Z","shell.execute_reply":"2022-07-13T19:53:39.371319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Trata as variaveis cruas e devolve somente as variaveis de interesse do modelo\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.pipeline import Pipeline\n\nclass Prep(BaseEstimator,TransformerMixin):\n\n    def __init__(self):\n        self.input_vars = ['Date',\n                           'Temperature',\n                           'Fuel_Price',\n                           'CPI', \n                           'Unemployment',\n                           'Dept',\n                           'Store', \n                           'Type',\n                           'IsHoliday']\n        \n    def transform(self, X):\n        \n        if(not isinstance(X, pd.DataFrame)):\n            X = pd.DataFrame(X, columns=self.input_vars)\n            \n            \n        fill_missings = {'SuperBowlWeek':0,\n                             'LaborDay':0,\n                             'Tranksgiving':0,\n                             'Christmas':0,\n                             \"THX_GVN_TO_XMAS\":0,\n                             \n                             'IsHoliday_cat':0,\n                             \"Type_cat\":1,\n                             \"Store_cat\":22,\n                             \"Dept_cat\":40,\n    \n                             \"CPI\":182.7,\n                             \"Temperature\":60.7,\n                             \"Fuel_Price\":3.51,\n                             \"Unemployment\":7.8               \n                            }\n        \n        self.final_vars=[\n           'Mes',\n           'Dia', \n           'Temperature',\n           'Fuel_Price',\n           'CPI', \n           'Unemployment',\n           'Dept_cat', 'Store_cat', 'Type_cat',\n           'IsHoliday_cat', \n           'SuperBowlWeek', 'LaborDay', 'Tranksgiving', 'Christmas',\n           'THX_GVN_TO_XMAS']\n\n        map_Dept={'92': 1, '95': 2, '38': 3, '65': 4, '72': 5, '40': 6, '2': 7,\\\n                  '90': 8, '94': 9, '91': 10, '8': 11, '13': 12, '93': 13, '4': 14,\\\n                  '23': 15, '79': 16, '7': 17, '46': 18, '5': 19, '10': 20, '9': 21,\\\n                  '1': 22, '34': 23, '82': 24, '96': 25, '97': 26, '11': 27, '14': 28,\\\n                  '81': 29, '74': 30, '80': 31, '87': 32, '17': 33, '3': 34, '22': 35,\\\n                  '55': 36, '25': 37, '49': 38, '16': 39, '98': 40, '26': 41, '67': 42, \\\n                  '32': 43, '33': 44, '24': 45, '42': 46, '20': 47, '71': 48, '29': 49, \\\n                  '21': 50, '44': 51, '6': 52, '12': 53, '30': 54, '37': 55, '83': 56,\\\n                  '35': 57, '58': 58, '50': 59, '31': 60, '85': 61, '52': 62, '56': 63, \\\n                  '36': 64, '19': 65, '27': 66, '41': 67, '18': 68, '48': 69, '28': 70, \\\n                  '59': 71, '60': 72, '77': 73, '99': 74, '54': 75, '47': 76, '45': 77, '51': 78, '39': 79, '78': 80, '43': 81}\n\n        map_Store={'20': 1, '14': 2, '4': 3, '2': 4, '13': 5, '10': 6, '6': 7, '27': 8,\\\n                   '19': 9, '11': 10, '23': 11, '39': 12, '28': 13, '24': 14, '1': 15, \\\n                   '18': 16, '35': 17, '41': 18, '31': 19, '22': 20, '12': 21, '32': 22,\\\n                   '8': 23, '26': 24, '40': 25, '17': 26, '21': 27, '34': 28, '45': 29,\\\n                   '25': 30, '15': 31, '29': 32, '7': 33, '9': 34, '16': 35, '3': 36, \\\n                   '5': 37, '36': 38, '43': 39, '37': 40, '42': 41, '30': 42, '33': 43, '44': 44, '38': 45}\n\n        map_Type={'A': 1, 'B': 2, 'C': 3}\n\n        X[\"Dept_cat\"]=X[\"Dept\"].astype(str).map(map_Dept).astype(float)\n        X[\"Store_cat\"]=X[\"Store\"].astype(str).map(map_Store).astype(float)\n        X[\"Type_cat\"]=X[\"Type\"].astype(str).map(map_Type).astype(float)\n        \n        X.fillna(fill_missings, inplace=True)\n\n        X['Date'] = pd.to_datetime(X['Date'])\n        X['Ano'] = (X.Date.dt.year).astype(int)\n        X['Mes'] = (X.Date.dt.month).astype(int)\n        X['Dia'] = (X.Date.dt.day).astype(int)\n        X['SemanaDoAno'] = ((X.Date.dt.isocalendar().week)*1.0).astype(int) \n\n\n        X['IsHoliday_cat'] = X['IsHoliday'].apply(lambda x: 1 if x else 0)\n\n        X['SuperBowlWeek'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 6 else 0)\n        X['LaborDay'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 35 else 0)\n        X['Tranksgiving'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 47 else 0)\n        X['Christmas'] = base_treino['SemanaDoAno'].apply(lambda x: 1 if x == 51 else 0)\n        X[\"THX_GVN_TO_XMAS\"]=base_treino['SemanaDoAno'].apply(lambda x: 1 if (x>46) & (x<52)  else 0)\n        \n        \n#         X['dept_72'] = X['Dept'].apply(lambda x: 1 if x ==72  else 0) #Var com alto impacto de Thx givin e Xmas\n#         X['dept_35'] = X['Dept'].apply(lambda x: 1 if x ==35  else 0) #Loja com grande quantidade de semanas com prejuizo\n#         X['too_hot_too_cold'] = X['Temperature'].apply(lambda x: 1 if ((x >80) |(x<20))  else 0)\n\n        \n        \n\n        return X[self.final_vars]","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.373297Z","iopub.status.idle":"2022-07-13T19:53:39.374199Z","shell.execute_reply.started":"2022-07-13T19:53:39.373909Z","shell.execute_reply":"2022-07-13T19:53:39.373937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Chama as classes\nprep=Prep()\njsontodf=JsonToDF()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.375898Z","iopub.status.idle":"2022-07-13T19:53:39.376979Z","shell.execute_reply.started":"2022-07-13T19:53:39.376671Z","shell.execute_reply":"2022-07-13T19:53:39.376700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Cria o pipeline para tratamento e escoragem\np=Pipeline(steps=[(\"transform\",jsontodf),(\"prep\",prep),(\"modelo\",modelo_RF)])","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.378415Z","iopub.status.idle":"2022-07-13T19:53:39.378823Z","shell.execute_reply.started":"2022-07-13T19:53:39.378631Z","shell.execute_reply":"2022-07-13T19:53:39.378648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submissão","metadata":{}},{"cell_type":"code","source":"#Submete\nsample_submission['Weekly_Sales'] = p.predict(base_teste)\nsample_submission.to_csv('submission.csv',index=False)\nsample_submission.to_csv('/kaggle/working/submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T19:53:39.380826Z","iopub.status.idle":"2022-07-13T19:53:39.381550Z","shell.execute_reply.started":"2022-07-13T19:53:39.381263Z","shell.execute_reply":"2022-07-13T19:53:39.381289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PONTOS DE MELHORIA","metadata":{}},{"cell_type":"markdown","source":"\n- Fazer um modelo para cada tipo (\"Type\") de loja pode melhorar a performance.\n\n- Utilização do cross validation e maior quantidade de parametros para os modelos mais complexos, que demandam maior capadicade computacional e tempo.\n\n- Feriados no geral são muito importantes para as vendas. Faria sentido incluir outros feriados que não constam nas bases e testar\n\n- Realizar uma melhor exploração de Markdowns com mais tempo observando os periodos antes e depois\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}