{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Getting Started on Vertex AI Notebooks #\n\nThis notebook demonstrates how to do the following on Vertex AI, Google's powerful new machine learning platform:\n\n- run the getting started notebook on Vertex AI Notebooks, to load the data, create a model & generate predictions\n- explore explainable AI on Vertex AI to refine your features\n- tune hyperparameters with Vizier\n\nIt is a complement to the [Getting Started with MLB Digital Engagement](https://www.kaggle.com/ryanholbrook/getting-started-with-mlb-player-digital-engagement) tutorial which was designed to be run on Kaggle Notebooks.\n\nThis tutorial uses Cloud Notebooks, a billable component of Google Cloud. Learn more about [Notebooks pricing](https://cloud.google.com/notebooks/pricing).","metadata":{"execution":{"iopub.execute_input":"2021-06-14T18:57:35.372476Z","iopub.status.busy":"2021-06-14T18:57:35.371945Z","iopub.status.idle":"2021-06-14T18:57:35.386745Z","shell.execute_reply":"2021-06-14T18:57:35.384957Z","shell.execute_reply.started":"2021-06-14T18:57:35.372429Z"}}},{"cell_type":"markdown","source":"# Notebook Setup #\n\n\n### 1. Download this Notebook ###\n\nStart by creating your own copy of this notebook. Click the *Copy and Edit* button to the upper right. Now, in the menubar above, click *File -> Download Notebook* and save a copy of the notebook to your computer. We will reupload this in an AI Notebooks instance to take advantage of the Explainable AI service.\n\n### 2. Download Kaggle API Key ###\nWe'll use the Kaggle API to download the competition data to the notebook instance. You'll need a copy of your Kaggle credentials to authenticate your account.\n\nFrom the site header, click on your user profile picture, then on “My Account” from the dropdown menu. This will take you to your account settings at https://www.kaggle.com/account. Scroll down to the section of the page labelled API.\n\nTo create a new token, click on the “Create New API Token” button. This will download a fresh authentication token onto your machine.\n\n### 3. Sign up for Google Cloud Platform ###\nIf you don't have a GCP account already, go to https://cloud.google.com/ and click on “Get Started For Free\". This is a two step sign up process where you will need to provide your name, address and a credit card. The starter account is free and it comes with $300 credit that you can use. For this step you will need to provide a Google Account (i.e. your Gmail account) to sign in.\n\n### 4. Create a Project and Enable the Notebook API ###\nFollow the directions at https://cloud.google.com/notebooks/docs/before-you-begin to setup a notebook project.\n\n### 5. Create a Notebook Instance ###\nNext, go to https://notebook.new. Enter an `Instance name` of your choice and then click the blue **CREATE** button at the end of the page. Be sure to keep the default `TensorFlow Enterprise` environment. You'll be redirected to a page with a list of your notebook instances. It may take a few minutes for the instance you just created to start up.\n\nOnce the notebook instance is running, click `OPEN JUPYTERLAB` just to the right of the instance name. You should be redirected to a JupyterLab environment.\n\n### 6. Upload MLB Notebook and API Key ###\nFrom inside JupyterLab, click the \"Upload Files\" (up arrow) button in the file browser on the left and upload the files `kaggle.json` and `vertex-ai-with-mlb-player-digital-engagement.ipynb`.\n\n### 7. Authenticate Kaggle API and Download MLB Date ###\n\nRun the next cell to download the competition data.","metadata":{}},{"cell_type":"code","source":"%%bash\n\npip install -q kaggle\nmkdir -p /home/jupyter/.kaggle\nmv kaggle.json /home/jupyter/.kaggle/kaggle.json 2>/dev/null\nchmod 600 /home/jupyter/.kaggle/kaggle.json\n\nkaggle competitions download mlb-player-digital-engagement-forecasting\nmkdir -p /home/jupyter/input\nunzip -u mlb-player-digital-engagement-forecasting.zip -d /home/jupyter/input/mlb-player-digital-engagement-forecasting","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 9. Run this Tutorial ###\n\nAfter you've completed the setup steps above in Vertex AI Notebooks, select \"Run\" from the menubar and \"Run Selected Cell and All Below\" to run through the rest of this notebook automatically. Or step through cell-by-cell, if you'd prefer. Note that the Explainable AI (XAI) walkthrough is at the bottom of this notebook.\n\n### 8. Cleaning Up ###\n\nYou'll be billed for any time you keep the notebook instance running. So after you're done, be sure either to stop the notebook instance from the *Notebooks* page, or else to delete the Cloud project you created.","metadata":{}},{"cell_type":"markdown","source":"# MLB Getting Started #","metadata":{}},{"cell_type":"markdown","source":"The first part of this notebook reproduces the data and model setup of the [Getting Started](https://www.kaggle.com/ryanholbrook/getting-started-with-mlb-player-digital-engagement) notebook.","metadata":{}},{"cell_type":"code","source":"import gc\nimport sys\nimport warnings\nfrom joblib import Parallel, delayed\nfrom pathlib import Path\n\nimport ipywidgets as widgets\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom statsmodels.tsa.deterministic import (CalendarFourier,\n                                           CalendarSeasonality,\n                                           CalendarTimeTrend,\n                                           DeterministicProcess)\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom keras.layers.experimental.preprocessing import StringLookup\n\nwarnings.simplefilter(\"ignore\")\n\n# Set Matplotlib defaults\nplt.style.use(\"seaborn-whitegrid\")\nplt.rc(\"figure\", autolayout=True, figsize=(11, 5))\nplt.rc(\n    \"axes\",\n    labelweight=\"bold\",\n    labelsize=\"large\",\n    titleweight=\"bold\",\n    titlesize=14,\n    titlepad=10,\n)\nplot_params = dict(\n    color=\"0.75\",\n    style=\".-\",\n    markeredgecolor=\"0.25\",\n    markerfacecolor=\"0.25\",\n    legend=False,\n)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2021-06-07T23:22:31.432389Z","iopub.status.busy":"2021-06-07T23:22:31.430659Z","iopub.status.idle":"2021-06-07T23:22:37.716704Z","shell.execute_reply":"2021-06-07T23:22:37.716057Z","shell.execute_reply.started":"2021-06-07T23:05:35.708039Z"},"papermill":{"duration":6.311292,"end_time":"2021-06-07T23:22:37.716875","exception":false,"start_time":"2021-06-07T23:22:31.405583","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Helper function to unpack json found in daily data\ndef unpack_json(json_str):\n    return pd.DataFrame() if pd.isna(json_str) else pd.read_json(json_str)\n\n\ndef unpack_data(data, dfs=None, n_jobs=-1):\n    if dfs is not None:\n        data = data.loc[:, dfs]\n    unnested_dfs = {}\n    for name, column in data.iteritems():\n        daily_dfs = Parallel(n_jobs=n_jobs)(\n            delayed(unpack_json)(item) for date, item in column.iteritems())\n        df = pd.concat(daily_dfs)\n        unnested_dfs[name] = df\n    return unnested_dfs","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-06-07T23:22:37.763965Z","iopub.status.busy":"2021-06-07T23:22:37.763324Z","iopub.status.idle":"2021-06-07T23:22:38.003542Z","shell.execute_reply":"2021-06-07T23:22:38.002777Z","shell.execute_reply.started":"2021-06-07T22:44:49.497356Z"},"papermill":{"duration":0.266957,"end_time":"2021-06-07T23:22:38.003781","exception":false,"start_time":"2021-06-07T23:22:37.736824","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_dir = Path('input/mlb-player-digital-engagement-forecasting/')\n\ndf_names = ['seasons', 'teams', 'players', 'awards']\n\nfor name in df_names:\n    globals()[name] = pd.read_csv(data_dir / f\"{name}.csv\")\n\nkaggle_data_tabs = widgets.Tab()\n# Add Output widgets for each pandas DF as tabs' children\nkaggle_data_tabs.children = list([widgets.Output() for df_name in df_names])\n\nfor index in range(0, len(df_names)):\n    # Rename tab bar titles to df names\n    kaggle_data_tabs.set_title(index, df_names[index])\n    \n    # Display corresponding table output for this tab name\n    with kaggle_data_tabs.children[index]:\n        display(eval(df_names[index]))\n\ndisplay(kaggle_data_tabs)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define dataframes to load from training set\ndfs = [\n    'nextDayPlayerEngagement',  # targets\n    'playerBoxScores',  # features\n    # Other dataframes available for features:\n    # 'games',\n    # 'rosters',\n    # 'teamBoxScores',\n    # 'transactions',\n    # 'standings',\n    # 'awards',\n    # 'events',\n    # 'playerTwitterFollowers',\n    # 'teamTwitterFollowers',\n]\n\n# Read training data\ntraining = pd.read_csv(\n    data_dir / 'train.csv',\n    usecols=['date'] + dfs,\n)\n\n# Convert training data date field to datetime type\ntraining['date'] = pd.to_datetime(training['date'], format=\"%Y%m%d\")\ntraining = training.set_index('date').to_period('D')\nprint(training.info())","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-06-07T23:22:38.057824Z","iopub.status.busy":"2021-06-07T23:22:38.057211Z","iopub.status.idle":"2021-06-07T23:24:24.850016Z","shell.execute_reply":"2021-06-07T23:24:24.850544Z","shell.execute_reply.started":"2021-06-07T23:08:32.242092Z"},"papermill":{"duration":106.821184,"end_time":"2021-06-07T23:24:24.850746","exception":false,"start_time":"2021-06-07T23:22:38.029562","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Unpack nested dataframes and store in dictionary `training_dfs`\ntraining_dfs = unpack_data(training, dfs=dfs)\nprint('\\n', training_dfs.keys())","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-06-07T23:24:24.909384Z","iopub.status.busy":"2021-06-07T23:24:24.90433Z","iopub.status.idle":"2021-06-07T23:30:07.355558Z","shell.execute_reply":"2021-06-07T23:30:07.356342Z","shell.execute_reply.started":"2021-06-07T22:45:28.367554Z"},"papermill":{"duration":342.485363,"end_time":"2021-06-07T23:30:07.356822","exception":false,"start_time":"2021-06-07T23:24:24.871459","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results of Explainable AI will be easier to understand if we restrict our analysis to a single player. The next cell has a helper function to load data for only a single player, by default Aaron Judge of the NY Yankees, who had the highest overall engagement during the training period.","metadata":{}},{"cell_type":"code","source":"pids_test = players.playerId.loc[\n    players.playerForTestSetAndFuturePreds.fillna(False)\n].astype(str)\n\ndef make_playerBoxScores(dfs: dict, features):\n    X = dfs['playerBoxScores']\n    X = X[['gameDate', 'playerId'] + features]\n    # Create date index\n    X = X.rename(columns={'gameDate': 'date'})\n    X['date'] = pd.PeriodIndex(X.date, freq='D')\n    # Aggregate multiple games per day by summing\n    X = X.groupby(['date', 'playerId'], as_index=False).sum()\n    return X\n\n\ndef make_targets(training_dfs: dict):\n    Y = training_dfs['nextDayPlayerEngagement'].copy()\n    # Match target dates to feature dates and create date index\n    Y = Y.rename(columns={'engagementMetricsDate': 'date'})\n    Y['date'] = pd.to_datetime(Y['date'])\n    Y = Y.set_index('date').to_period('D')\n    Y.index = Y.index - 1\n    return Y.reset_index()\n\n\ndef join_datasets(dfs):\n    dfs = [x.pivot(index='date', columns='playerId') for x in dfs]\n    df = pd.concat(dfs, axis=1).stack().reset_index('playerId')\n    return df\n\n\ndef make_training_data(training_dfs: dict,\n                       features,\n                       targets,\n                       fourier=4,\n                       player=592450,  # Aaron Judge of the NY Yankees\n                       test_size=30):\n    # Process dataframes\n    X = make_playerBoxScores(training_dfs, features)\n    Y = make_targets(training_dfs)\n    # Merge for processing\n    df = join_datasets([X, Y])\n    # Process\n    df = df.astype({'playerId': str})\n    df = df.astype({name: np.float32 for name in features})\n    # Filter for chosen player\n    df = df.loc[df.playerId == str(player), :].drop('playerId', axis=1)\n    for name in features:\n        df[name] = df[name].fillna(-1)\n    # Restore features and targets\n    X = df[features]\n    Y = df[targets]\n    # Create temporal features\n    fourier_terms = CalendarFourier(freq='A', order=fourier)\n    deterministic = DeterministicProcess(\n        index=X.index,\n        order=0,\n        seasonal=False,\n        additional_terms=[fourier_terms],\n    )\n    X = pd.concat([X, deterministic.in_sample()], axis=1)\n    # Create train / validation splits\n    X_train, X_valid, y_train, y_valid = train_test_split(\n        X,\n        Y,\n        test_size=test_size,\n        shuffle=False,\n    )\n    return X_train, X_valid, y_train, y_valid, deterministic","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-06-07T23:30:07.408558Z","iopub.status.busy":"2021-06-07T23:30:07.407596Z","iopub.status.idle":"2021-06-07T23:30:12.899641Z","shell.execute_reply":"2021-06-07T23:30:12.899104Z","shell.execute_reply.started":"2021-06-07T22:47:57.570731Z"},"papermill":{"duration":5.517467,"end_time":"2021-06-07T23:30:12.899795","exception":false,"start_time":"2021-06-07T23:30:07.382328","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We've picked out a few features from the `playerBoxScores` dataframe, but there are lots more you could try (see the [data documentation](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data) for a complete description). Increase the number of Fourier components to model seasonality with in more detail. You could also look at explanations for other players -- the `players` dataframe can tell you the `playerId` for each player.","metadata":{}},{"cell_type":"code","source":"player = 592450  # Aaron Judge of the NY Yankees\nfeatures = [\n    \"runsScored\",\n    \"hits\",\n    \"rbi\",\n    \"atBats\",\n    \"hitByPitch\",\n    \"saves\",\n    \"homeRuns\",\n    \"stolenBases\",\n    \"strikeOuts\",\n    \"groundIntoTriplePlay\",\n    \"totalBases\",\n]\nfourier = 4  # number of annual seasonal components\n\ntargets = [\"target1\", \"target2\", \"target3\", \"target4\"]\ntest_size = 30\n\nX_train, X_valid, y_train, y_valid, deterministic = make_training_data(\n    training_dfs, \n    features=features, \n    targets=targets,\n    fourier=4,\n    player=592450,\n    test_size=test_size,\n)","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-06-07T23:30:12.959082Z","iopub.status.busy":"2021-06-07T23:30:12.958458Z","iopub.status.idle":"2021-06-07T23:30:13.04453Z","shell.execute_reply":"2021-06-07T23:30:13.044004Z","shell.execute_reply.started":"2021-06-07T22:48:00.781309Z"},"papermill":{"duration":0.121909,"end_time":"2021-06-07T23:30:13.044701","exception":false,"start_time":"2021-06-07T23:30:12.922792","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model #","metadata":{"papermill":{"duration":0.036549,"end_time":"2021-06-07T23:30:47.569193","exception":false,"start_time":"2021-06-07T23:30:47.532644","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Now we'll create a neural network with Keras and fit it to our training data.","metadata":{}},{"cell_type":"code","source":"INPUTS = X_train.shape[-1]\nOUTPUTS = y_train.shape[-1]\nearly_stopping = keras.callbacks.EarlyStopping(patience=10,\n                                               restore_best_weights=True)\n\nmodel = keras.Sequential([\n    layers.InputLayer(name='numpy_inputs', input_shape=(INPUTS,)),\n    layers.Dense(128, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.2),\n    layers.Dense(128, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.2),\n    layers.Dense(128, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.2),\n    layers.Dense(OUTPUTS),\n])\n","metadata":{"execution":{"iopub.execute_input":"2021-06-07T23:30:47.64678Z","iopub.status.busy":"2021-06-07T23:30:47.645816Z","iopub.status.idle":"2021-06-07T23:30:47.72237Z","shell.execute_reply":"2021-06-07T23:30:47.722849Z","shell.execute_reply.started":"2021-06-07T23:06:36.852123Z"},"papermill":{"duration":0.11699,"end_time":"2021-06-07T23:30:47.723031","exception":false,"start_time":"2021-06-07T23:30:47.606041","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss='mae',\n    metrics=['mae'],\n)\nhistory = model.fit(\n    X_train,\n    y_train,\n    validation_data=(X_valid, y_valid),\n    epochs=50,\n    callbacks=[early_stopping]\n)","metadata":{"execution":{"iopub.execute_input":"2021-06-07T23:30:47.809965Z","iopub.status.busy":"2021-06-07T23:30:47.809281Z","iopub.status.idle":"2021-06-07T23:31:32.828239Z","shell.execute_reply":"2021-06-07T23:31:32.828768Z","shell.execute_reply.started":"2021-06-07T23:06:49.511418Z"},"papermill":{"duration":45.069407,"end_time":"2021-06-07T23:31:32.828944","exception":false,"start_time":"2021-06-07T23:30:47.759537","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate #","metadata":{"papermill":{"duration":0.224828,"end_time":"2021-06-07T23:31:33.281108","exception":false,"start_time":"2021-06-07T23:31:33.05628","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"We can get an idea of how well our network was able to fit the data by plotting its predictions.","metadata":{}},{"cell_type":"code","source":"y_fit = model.predict(X_train)\ny_fit = pd.DataFrame(y_fit,\n                     index=y_train.index,\n                     columns=y_train.columns)\ny_eval = model.predict(X_valid)\ny_eval = pd.DataFrame(y_eval,\n                      index=y_valid.index,\n                      columns=y_valid.columns)","metadata":{"execution":{"iopub.execute_input":"2021-06-07T23:31:33.73647Z","iopub.status.busy":"2021-06-07T23:31:33.735508Z","iopub.status.idle":"2021-06-07T23:31:35.133451Z","shell.execute_reply":"2021-06-07T23:31:35.132904Z","shell.execute_reply.started":"2021-06-07T23:07:55.049202Z"},"papermill":{"duration":1.627791,"end_time":"2021-06-07T23:31:35.133596","exception":false,"start_time":"2021-06-07T23:31:33.505805","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(11, 8))\nax = y_eval.plot(ax=ax, subplots=True, color='C0')\nax = y_valid.plot(subplots=True,\n                  sharex=True,\n                  ax=ax,\n                  color='0.25')\nax = y_fit.plot(subplots=True,\n                sharex=True,\n                ax=ax,\n                color='C3')\nax = y_train.plot(subplots=True,\n                  sharex=True,\n                  ax=ax,\n                  color='0.25')","metadata":{"execution":{"iopub.execute_input":"2021-06-07T23:31:35.623176Z","iopub.status.busy":"2021-06-07T23:31:35.622535Z","iopub.status.idle":"2021-06-07T23:31:37.411686Z","shell.execute_reply":"2021-06-07T23:31:37.412191Z","shell.execute_reply.started":"2021-06-07T23:07:57.512927Z"},"papermill":{"duration":2.050107,"end_time":"2021-06-07T23:31:37.41236","exception":false,"start_time":"2021-06-07T23:31:35.362253","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explain #","metadata":{"papermill":{"duration":0.234556,"end_time":"2021-06-07T23:31:37.878617","exception":false,"start_time":"2021-06-07T23:31:37.644061","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Explainable AI on Vertex AI Notebooks lets you compute feature attributions for neural networks. Feature attributions describe the contribution each features makes to the final prediction relative to a baseline. Feature attributions can help you tune your model by indicating which features are important and which are not. Features with little importance you could consider dropping from your feature set.\n\nRead more about Vertex Explainable AI here: [Introduction to Vertex Explainable AI for Vertex AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview). In JupyterLab on Vertex AI Notebooks, you can also review a tutorial on XAI in the `tutorials` > `explainable_ai` > `sdk_tutorial.ipynb` file\n\nNow we can look at explanations using the `explainable_ai_sdk` library. Run the following cell on AI Notebooks with a Cloud TF image to see model explanations.","metadata":{}},{"cell_type":"code","source":"import explainable_ai_sdk\nfrom explainable_ai_sdk.model import configs\nfrom explainable_ai_sdk.metadata.tf.v2 import SavedModelMetadataBuilder\n\nmodel_path = \"working/model\"\nmodel.save(model_path)\n\nbuilder = SavedModelMetadataBuilder(model_path)\nbuilder.set_numeric_metadata(\n    \"numpy_inputs\",\n    input_baselines=[X_train.median().tolist()],  # attributions relative to the median of the target\n    index_feature_mapping=X_train.columns.tolist(),  # the names of each feature\n)\nbuilder.save_metadata(model_path)\n\nexplainer = explainable_ai_sdk.load_model_from_local_path(\n    model_path=model_path,\n    config=configs.SampledShapleyConfig(path_count=20),\n)\n\ninstances = list(X_valid.to_numpy())  # explanations for the validation set\nexplanations = explainer.explain(instances={'numpy_inputs': instances})[0]  # needs to match name of `InputLayer` in the model\n\nexplanations.visualize_attributions()","metadata":{"execution":{"iopub.execute_input":"2021-06-07T23:31:38.825566Z","iopub.status.busy":"2021-06-07T23:31:38.824899Z","iopub.status.idle":"2021-06-07T23:31:38.828073Z","shell.execute_reply":"2021-06-07T23:31:38.827449Z"},"papermill":{"duration":0.243921,"end_time":"2021-06-07T23:31:38.828208","exception":false,"start_time":"2021-06-07T23:31:38.584287","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You should see a Feature Attributions chart above. Each of the attributions is computed on a given set of instances relative to a baseline, in our case the median on the training set. We've computed attributions for the validation set, the last 30 days of the data, but you could try a different set if you like by changing the `instances` parameter. To get attributions relative to another baseline (the mean, say), just change the computation for `input_baselines` above.\n\nOur feature attribution chart suggest that features like `atBats` and `strikeOuts` are important for digital engagement with Aaron Judge, but `stolenBases` much less so. We also see that the model did make use of several of the Fourier features modeling annual seasonality.","metadata":{}},{"cell_type":"markdown","source":"# Hyperparameter Tuning with Vizier #","metadata":{}},{"cell_type":"markdown","source":"In the second half of this notebook, we'll demonstrate **Vertex Vizier**, Vertex AI's hyperparameter optimization service. Among its capabilities are a *Bayesian Optimization* algorithm to search efficiently within a hyperparameter space, *transfer learning* to make use of information from previous hyperparameter studies, and *automated early stopping* when tuning models that train incrementally, like neural nets with stochastic gradient descent or gradient boosted trees. Google Research has a great whitepaper describing the capabilities of Vizier in detail: [Google Vizier: A Service for Black-Box Optimization](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/bcb15507f4b52991a0783013df4222240e942381.pdf). Also see the [Vizier guide](https://cloud.google.com/vertex-ai/docs/vizier/overview) for a nice overview.\n\nWhile in preview, Vertex Vizier is offered at [no charge](https://cloud.google.com/vertex-ai/pricing#vizier).\n\n## Setup ##\n\nFirst, make sure you've run everything in the notebook prior to this up to **Data Pipeline**.\n\nNow run the next cell to enable the Vertex AI API on your project and install the Python client in this notebook instance.","metadata":{}},{"cell_type":"code","source":"%%bash\ngcloud services enable aiplatform.googleapis.com\npip install -q google-cloud-aiplatform","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The next cell imports the Vizier service and defines some helper functions we'll use to run the hyperparameter search","metadata":{}},{"cell_type":"code","source":"import datetime\nimport json\nfrom typing import List, Dict\n\nimport proto\nfrom google.cloud.aiplatform_v1beta1 import VizierServiceClient\n\n\ndef create_study(\n    parameters: List[Dict],\n    metrics: List[Dict],\n    vizier_client,\n    project_id: str,\n    location: str = 'us-central1',\n):\n    parent = f\"projects/{project_id}/locations/{location}\"\n    display_name = \"{}_study_{}\".format(\n        project_id.replace(\"-\", \"\"),\n        datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\"))\n    study = {\n        'display_name': display_name,\n        'study_spec': {\n            # 'ALGORITHM_UNSPECIFIED' means Bayesian optimization\n            # can also be 'GRID_SEARCH' or 'RANDOM_SEARCH'\n            'algorithm': 'ALGORITHM_UNSPECIFIED',\n            'parameters': parameters,\n            'metrics': metrics,\n        }\n    }\n    study = vizier_client.create_study(parent=parent, study=study)\n    return study.name\n\n\ndef params_to_dict(parameters):\n    return {p.parameter_id: p.value for p in parameters}\n\n\ndef run_study(\n    metric_fn,\n    requests: int,\n    suggestions_per_request: int,\n    client_id: str,\n    study_id: str,\n    vizier_client,\n):\n    for k in range(requests):\n        suggest_response = vizier_client.suggest_trials({\n            \"parent\": study_id,\n            \"suggestion_count\": suggestions_per_request,\n            \"client_id\": client_id,\n        })\n        print(f\"Request {k}\")\n        for suggested_trial in suggest_response.result().trials:\n            suggested_params = params_to_dict(suggested_trial.parameters)\n            metric = metric_fn(suggested_params)\n            print(f\"Trial Results: {metric['metric_id']}={metric['value']}\")\n\n            vizier_client.add_trial_measurement({\n                'trial_name': suggested_trial.name,\n                'measurement': {\n                    'metrics': [metric]\n                }\n            })\n\n            response = vizier_client.complete_trial({\n                \"name\": suggested_trial.name,\n                \"trial_infeasible\": False\n            })\n\n\ndef get_optimal_trials(study_id, vizier_client):\n    optimal_trials = vizier_client.list_optimal_trials({'parent': study_id})\n    optimal_params = []\n    for trial in proto.Message.to_dict(optimal_trials)['optimal_trials']:\n        optimal_params.append({p['parameter_id']: p['value'] for p in trial['parameters']})\n    return optimal_params","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we'll create the client that will communicate with the Vizier service. You'll need to provide it with your project id.","metadata":{}},{"cell_type":"code","source":"# Fill in your project ID\nPROJECT_ID = \"mlb-xai\"\n\n# These will be automatically filled in\nREGION = \"us-central1\"\nENDPOINT = REGION + \"-aiplatform.googleapis.com\"\nvizier_client = VizierServiceClient(client_options=dict(api_endpoint=ENDPOINT))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you don't know your project ID, you may be able to retrieve it with this command:","metadata":{}},{"cell_type":"code","source":"!gcloud config list --format 'value(core.project)'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define Hyperparameters ##","metadata":{}},{"cell_type":"markdown","source":"The search space (or feasible space) for each hyperparameter is determined by its *type*. In Vizier, hyperparameters can be one of four types: `DOUBLE` or `INTEGER`, defined by minimum and maximum values, or, `CATEGORICAL` or `DISCRETE`, defined by an enumeration of values. We've created one of each below for illustration. \n\nHyperparameters of numeric type may also have their feasible space [scaled](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/StudySpec#scaletype). It's common to apply logarithmic scaling when you want to search efficiently across orders of magnitude, like for a learning rate between `1e-6` and `1e-1`. See the [`StudySpec` API documentation](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/StudySpec) for more about scaling and other options.","metadata":{}},{"cell_type":"code","source":"# Define the hyperparameter feasible space\nparameters = [{\n    'parameter_id': 'batch_size',\n    'integer_value_spec': {\n        'min_value': 8,\n        'max_value': 128,\n    }\n}, {\n    'parameter_id': 'dropout',\n    'double_value_spec': {\n        'min_value': 0.0,\n        'max_value': 0.5,\n    }\n}, {\n    'parameter_id': 'units',\n    'discrete_value_spec': {\n        'values': [128, 256, 512, 1024]\n    }\n}, {\n    'parameter_id': 'optimizer',\n    'categorical_value_spec': {\n        'values': ['sgd', 'adam'],\n    }\n}]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create Study ##","metadata":{"execution":{"iopub.execute_input":"2021-06-14T18:46:19.220237Z","iopub.status.busy":"2021-06-14T18:46:19.219453Z","iopub.status.idle":"2021-06-14T18:46:19.226889Z","shell.execute_reply":"2021-06-14T18:46:19.225581Z","shell.execute_reply.started":"2021-06-14T18:46:19.220042Z"}}},{"cell_type":"markdown","source":"Now we'll create a *study*. A study conducts *trials* in order to optimize one or more *metrics*. A trial is a selection of hyperparameter values together with the outcome they produce. In our case, the hyperparameters define a neural net architechture and training regimen, and will produce a validation loss, the metric we hope to minimize.","metadata":{}},{"cell_type":"code","source":"metrics = [{\n    'metric_id': 'val_loss',  # the name of the quantity we want to minimize\n    'goal': 'MINIMIZE',  # choose MINIMIZE or MAXIMIZE\n}]\n\n# Call a helper function to create the study\nstudy_id = create_study(\n    parameters=parameters,\n    metrics=metrics,\n    vizier_client=vizier_client,\n    project_id=PROJECT_ID,\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll put our model definition and training inside a function we can pass hyperparameters to.","metadata":{}},{"cell_type":"code","source":"def metric_fn(params):\n    # Parse hyperparameters\n    units = int(params['units'])\n    dropout = params['dropout']\n    optimizer = params['optimizer']\n    batch_size = int(params['batch_size'])\n    # Create and train model\n    INPUTS = X_train.shape[-1]\n    OUTPUTS = y_train.shape[-1]\n    early_stopping = keras.callbacks.EarlyStopping(patience=10,\n                                                   restore_best_weights=True)\n    model = keras.Sequential([\n        layers.InputLayer(name='numpy_inputs', input_shape=(INPUTS, )),\n        layers.Dense(units, activation='relu'),\n        layers.BatchNormalization(),\n        layers.Dropout(dropout),\n        layers.Dense(units, activation='relu'),\n        layers.BatchNormalization(),\n        layers.Dropout(dropout),\n        layers.Dense(units, activation='relu'),\n        layers.BatchNormalization(),\n        layers.Dropout(dropout),\n        layers.Dense(OUTPUTS),\n    ])\n    model.compile(\n        optimizer=optimizer,\n        loss='mae',\n        metrics=['mae'],\n    )\n    model.fit(X_train,\n              y_train,\n              validation_data=(X_valid, y_valid),\n              batch_size=batch_size,\n              epochs=50,\n              callbacks=[early_stopping],\n              verbose=0)\n    # Optimize the metric monitored by `early_stopping` (`val_loss` by default)\n    # The metric needs to be reported in this format\n    return {'metric_id': early_stopping.monitor, 'value': early_stopping.best}","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That done, we'll call another helper function to run the hyperparameter search.","metadata":{}},{"cell_type":"markdown","source":"## Run Study ##","metadata":{}},{"cell_type":"code","source":"run_study(\n    metric_fn,\n    requests=5,\n    # set >1 to get suggestions \"in parallel\", good for distributed training\n    suggestions_per_request=1,\n    # keep the name the same to resume a trial\n    client_id=\"client_1\",\n    study_id=study_id,\n    vizier_client=vizier_client,\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Look at your [**Studies**](https://console.cloud.google.com/vertex-ai/experiments/studies) tab and select your study for a summary of the results. Also be sure to click the **Analysis** tab for the results presented in a parallel coordinates chart.\n\nLet's finish by looking at the best set of hyperparameters we found:","metadata":{}},{"cell_type":"code","source":"print(get_optimal_trials(study_id, vizier_client))","metadata":{},"execution_count":null,"outputs":[]}]}