{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 0. Libraries and Personal Tools","metadata":{}},{"cell_type":"markdown","source":"A single objective for this notebook: deep dive into how the ball position behaves during an event. Is the ball's position as expected or changes from our traditional conception of sports?\n\nThe questions I'll try to answer:\n\n1. How many games, events, and frames do we have available to analyze? \n1. How many events ended up with a team scoring? What was the proportion?\n1. What is the average event duration?\n1. How does ball position change across different stages of the game?\n\n**If you don't want to spend a lot of time reading this nootebook, go directly to the last chart!**","metadata":{}},{"cell_type":"code","source":"import os\nimport sys\nimport datatable as dt\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom matplotlib import rcParams","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:09:04.588255Z","iopub.execute_input":"2022-10-12T20:09:04.589856Z","iopub.status.idle":"2022-10-12T20:09:04.695141Z","shell.execute_reply.started":"2022-10-12T20:09:04.589722Z","shell.execute_reply":"2022-10-12T20:09:04.694393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set the default figure size and theme to display good looking matplotlib plots.\nrcParams[\"figure.figsize\"] = (12, 8)\nplt.style.use(\"fivethirtyeight\")","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:09:05.588676Z","iopub.execute_input":"2022-10-12T20:09:05.589066Z","iopub.status.idle":"2022-10-12T20:09:05.594994Z","shell.execute_reply.started":"2022-10-12T20:09:05.589033Z","shell.execute_reply":"2022-10-12T20:09:05.593809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import custom modules\nfrom visualize_tps_oct22 import *\nfrom utilities_tps_oct22 import *","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:09:43.532592Z","iopub.execute_input":"2022-10-12T20:09:43.533563Z","iopub.status.idle":"2022-10-12T20:09:44.182133Z","shell.execute_reply.started":"2022-10-12T20:09:43.533519Z","shell.execute_reply":"2022-10-12T20:09:44.180914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Get Raw Data","metadata":{}},{"cell_type":"markdown","source":"Kudos to Ayush Bihani! He help me to quickly read all training data.","metadata":{}},{"cell_type":"code","source":"# Author: Author: Ayush Bihani\n# Source: https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/notebook\n\nfile_names = []\nfor dirname, _, filenames in os.walk(\"../input/tabular-playground-series-oct-2022\"):\n    for filename in filenames:\n        file_names.append(os.path.join(dirname, filename))\n        \ntrain_paths = []\nfor k in file_names:\n    if 'train_' in k and 'dtypes' not in k:\n        train_paths.append(k)\n        \ntrain_paths","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:09:47.018011Z","iopub.execute_input":"2022-10-12T20:09:47.018409Z","iopub.status.idle":"2022-10-12T20:09:47.028963Z","shell.execute_reply.started":"2022-10-12T20:09:47.018374Z","shell.execute_reply":"2022-10-12T20:09:47.027661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dtypes_df = pd.read_csv(\"../input/tabular-playground-series-oct-2022/train_dtypes.csv\")\ndtypes = {k: v for (k, v) in zip(dtypes_df.column, dtypes_df.dtype)}\n\ndf = pd.DataFrame()\n\nfor i in range(10):\n    df_aux = pd.read_csv(f\"../input/tabular-playground-series-oct-2022/train_{i}.csv\", dtype=dtypes)\n    df = pd.concat([df, df_aux])\n    \ndf = reduce_mem_usage(df)","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:12:08.231642Z","iopub.execute_input":"2022-10-12T20:12:08.232082Z","iopub.status.idle":"2022-10-12T20:17:35.129340Z","shell.execute_reply.started":"2022-10-12T20:12:08.232041Z","shell.execute_reply":"2022-10-12T20:17:35.127603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Exploratory Data Analysis\n\nLet's quickly analyze the type of data we are dealing, column names and more. ","metadata":{}},{"cell_type":"code","source":"df.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.sample(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.1. About Games, Events and Frames","metadata":{}},{"cell_type":"markdown","source":"First, we want to understand the basic concepts from this data. Our training dataset contains a set of games.\n\nA game in Rocket League takes place between two teams of three players each (it can be 2v2, but this time is only 3v3). To win the game, you need to score as many goals as possible. The exciting part of this game is that you are driving powerful cars and can perform crazy maneuvers. \n\nTo learn more, check my post on [Better Understanding of Rocket League Might Help You](https://www.kaggle.com/competitions/tabular-playground-series-oct-2022/discussion/358801#1979635).","metadata":{}},{"cell_type":"code","source":"# How many games are available in the dataset?\ndf.game_num.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:25:48.082630Z","iopub.execute_input":"2022-10-12T20:25:48.083143Z","iopub.status.idle":"2022-10-12T20:25:48.241328Z","shell.execute_reply.started":"2022-10-12T20:25:48.083107Z","shell.execute_reply":"2022-10-12T20:25:48.240224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Alright, we know there are more than 7,000 games in this dataset. But each game might have more than one event!\n\nAn event is defined by a team scoring a goal—either team A or team B. Of course, there is a possibility that an event ends without a single team scoring. ","metadata":{}},{"cell_type":"code","source":"# How many events are available in the dataset?\ndf.event_id.nunique()\n\n# Follow-up question: On average, how many events a game has?\n# Follow-up question: What proportion of events ends with a score, and what proportion doesn't?","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:29:12.564736Z","iopub.execute_input":"2022-10-12T20:29:12.565823Z","iopub.status.idle":"2022-10-12T20:29:12.673506Z","shell.execute_reply.started":"2022-10-12T20:29:12.565781Z","shell.execute_reply":"2022-10-12T20:29:12.672163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we finally have the lowest granularity level for the training set: a frame.\n\nThe Kaggle team recorded each event with a speed of 10 frames per second. Since we are interested in the result of each event, the event_time is registered as a negative number.\n\nAlso, with this concept in mind, we know that the event duration might be different, even in a single game. \n\nImagine a game in Rocket League is starting. Team A scores, the team takes the ball, manages to trick team B, and scores during the initial 30 seconds. For this hypothetical situation, you could find `game_id` as 0, and `event_id` equals 1001 and 300 frames with a lot of information about what happened during the match. The game continues, but both teams have no score for the remaining time. In these cases, you could find the same `game_id`, and a new `event_id` but a lot more frames since the recording was truncated since there were no goals!","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:42:02.998862Z","iopub.execute_input":"2022-10-12T20:42:02.999288Z","iopub.status.idle":"2022-10-12T20:42:03.009859Z","shell.execute_reply.started":"2022-10-12T20:42:02.999256Z","shell.execute_reply":"2022-10-12T20:42:03.008494Z"}}},{"cell_type":"code","source":"# How many frames per event are available in the dataset?\n(\n    df.groupby(\"event_id\")\n    .event_time\n    .count()\n    .describe()\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:31:17.156244Z","iopub.execute_input":"2022-10-12T20:31:17.156750Z","iopub.status.idle":"2022-10-12T20:31:17.813360Z","shell.execute_reply.started":"2022-10-12T20:31:17.156712Z","shell.execute_reply":"2022-10-12T20:31:17.812194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.3. Event Time Duration\n\nSince we can get the duration of any event, we are interested in segmenting each game and trying to check where the ball was during each part of the event.\n\nWe want to understand how the ball positioning evolves until a team scores (or not!)","metadata":{}},{"cell_type":"code","source":"df_team_scoring = (\n    df[[\"event_id\", \"team_scoring_next\"]]\n    .drop_duplicates()\n    .set_index(\"event_id\")\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:10.664794Z","iopub.execute_input":"2022-10-12T20:43:10.665220Z","iopub.status.idle":"2022-10-12T20:43:13.899528Z","shell.execute_reply.started":"2022-10-12T20:43:10.665182Z","shell.execute_reply":"2022-10-12T20:43:13.898637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# With the following code I'm counting frames for each event, but I also want the flag of team_A|B_scoring_within_10sec\n# for other purposes.\ndf_event_duration = (\n    df_team_scoring\n    .join(\n        df\n        .groupby([\"event_id\", \"team_A_scoring_within_10sec\"])\n        [[\"event_time\"]]\n        .count()\n        .pivot_table(index=\"event_id\", columns=\"team_A_scoring_within_10sec\", values=\"event_time\")\n        .rename(columns={0: \"team_A_no_goal\", 1: \"team_A_goal\"})\n        # divide by 10 to get the number of seconds since event were recorded at 10 frames per second\n        .div(10)\n        )\n    .join(\n        df\n        .groupby([\"event_id\", \"team_B_scoring_within_10sec\"])\n        [[\"event_time\"]]\n        .count()\n        .pivot_table(index=\"event_id\", columns=\"team_B_scoring_within_10sec\", values=\"event_time\")\n        .rename(columns={0: \"team_B_no_goal\", 1: \"team_B_goal\"})\n        # divide by 10 to get the number of seconds since event were recorded at 10 frames per second\n        .div(10)\n        )\n    ).reset_index()\n\n# Remember, I splitted an event between two main segments: before 10 seconds if a team scores and the rest of the game\ndf_event_duration[\"seconds_outside_scoring_range\"] = df_event_duration.apply(lambda x: x[\"team_A_no_goal\"] if x[\"team_scoring_next\"] == \"A\" else x[\"team_B_no_goal\"], axis=1)\ndf_event_duration[\"seconds_within_scoring_range\"] = df_event_duration.apply(lambda x: x[\"team_A_goal\"] if x[\"team_scoring_next\"] == \"A\" else x[\"team_B_goal\"], axis=1)\n\ndf_event_duration = (\n    df_event_duration\n    [[\"event_id\", \"team_scoring_next\", \"seconds_within_scoring_range\", \"seconds_outside_scoring_range\"]]\n    .set_index(\"event_id\")\n    )\n\ndf_event_duration[\"total_duration\"] = df_event_duration[[\"seconds_within_scoring_range\", \"seconds_outside_scoring_range\"]].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:15.242378Z","iopub.execute_input":"2022-10-12T20:43:15.242795Z","iopub.status.idle":"2022-10-12T20:43:19.220631Z","shell.execute_reply.started":"2022-10-12T20:43:15.242764Z","shell.execute_reply":"2022-10-12T20:43:19.219681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# How the event duration distributes? I'm particulary interested in what happened before a score occur, \n# So I filtered the dataset to check the event duration for those seconds after the \"scoring range\"\n\n(\n    df_event_duration\n    [df_event_duration[\"seconds_outside_scoring_range\"] < df_event_duration[\"seconds_outside_scoring_range\"].quantile(0.99)]\n    .seconds_outside_scoring_range\n    .plot.hist(\n        bins=25, \n        title=\"Average time duration of event when outside scoring range (> 10s before next goal)\"\n        )\n    );","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:31.833530Z","iopub.execute_input":"2022-10-12T20:43:31.834208Z","iopub.status.idle":"2022-10-12T20:43:32.156921Z","shell.execute_reply.started":"2022-10-12T20:43:31.834171Z","shell.execute_reply":"2022-10-12T20:43:32.155665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# If we take into account the two big segments of the game (10 seconds before scoring and the rest of the game)\n# We can see the average duration of a Rocket League event is 69 seconds (which is a great number btw)\ndf_event_duration[\"total_duration\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:37.826691Z","iopub.execute_input":"2022-10-12T20:43:37.827541Z","iopub.status.idle":"2022-10-12T20:43:37.839240Z","shell.execute_reply.started":"2022-10-12T20:43:37.827503Z","shell.execute_reply":"2022-10-12T20:43:37.838266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.4. Average Ball Position (Every 10 seconds)","metadata":{}},{"cell_type":"markdown","source":"Ok, enough vague insights, and let's dig into the real deal. \n\nWe can split our events every 10 seconds and try to visualize where the ball has been around (on average). \n\nI'm choosing to create segments of 10 seconds since the challenge is to predict if a team will score within the next 10 seconds, so it makes sense (at least to me) to create these buckets with this time length.","metadata":{}},{"cell_type":"code","source":"df_ball = df[[\n    \"event_id\", \"event_time\", \n    \"ball_pos_x\", \"ball_pos_y\", \"ball_pos_z\", \n    \"ball_vel_x\", \"ball_vel_y\", \"ball_vel_z\", \n    \"team_A_scoring_within_10sec\", \"team_B_scoring_within_10sec\"\n    ]].copy()\n\n# I'm calculating some metrics for later modeling and analysis, hold with me\ndf_ball[\"scoring_within_10sec\"] = df_ball[\"team_A_scoring_within_10sec\"] + df_ball[\"team_B_scoring_within_10sec\"]\ndf_ball[\"time_inverval_before_event_ending\"] = pd.cut(df_ball[\"event_time\"], [-1000] + list(range(-300, 0, 10)) + [0])\ndf_ball[\"ball_vel_scalar\"] = np.sqrt(df_ball[\"ball_vel_x\"]**2 + df_ball[\"ball_vel_y\"]**2 + df_ball[\"ball_vel_z\"]**2)\ndf_ball[\"ball_distance_from_center\"] = np.sqrt(df_ball[\"ball_pos_x\"]**2 + df_ball[\"ball_pos_y\"]**2)\n","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:43.197340Z","iopub.execute_input":"2022-10-12T20:43:43.197731Z","iopub.status.idle":"2022-10-12T20:43:47.535128Z","shell.execute_reply.started":"2022-10-12T20:43:43.197699Z","shell.execute_reply":"2022-10-12T20:43:47.533891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_ball = df_ball[[\n    \"event_id\", \"scoring_within_10sec\", \"event_time\", \"time_inverval_before_event_ending\", \n    \"ball_pos_x\", \"ball_pos_y\", \"ball_pos_z\", \"ball_vel_scalar\", \"ball_distance_from_center\"\n    ]]","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:47.537879Z","iopub.execute_input":"2022-10-12T20:43:47.539758Z","iopub.status.idle":"2022-10-12T20:43:48.491807Z","shell.execute_reply.started":"2022-10-12T20:43:47.539712Z","shell.execute_reply":"2022-10-12T20:43:48.490890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_avg_ball_10s = (\n    df_ball\n    .groupby([\"scoring_within_10sec\", \"event_id\", \"time_inverval_before_event_ending\"])\n    [[\"ball_pos_x\", \"ball_pos_y\", \"ball_pos_z\", \"ball_vel_scalar\", \"ball_distance_from_center\"]]\n    # I'm averaging my metrics but feel free to comment any other method you would have try here\n    .mean()\n    .dropna(how=\"all\")\n    .reset_index()\n    )","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:43:48.493062Z","iopub.execute_input":"2022-10-12T20:43:48.493471Z","iopub.status.idle":"2022-10-12T20:44:02.799521Z","shell.execute_reply.started":"2022-10-12T20:43:48.493412Z","shell.execute_reply":"2022-10-12T20:44:02.798234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Plot a Sample of Average Ball Positions (Color = Velocity Scalar)","metadata":{}},{"cell_type":"code","source":"sample_df = df_avg_ball_10s[\n    (df_avg_ball_10s[\"scoring_within_10sec\"] == 1)\n    & (1 < df_avg_ball_10s[\"ball_distance_from_center\"])\n    ].sample(frac=0.2).copy()\n\nmeta = {\n    \"game_num\": False,\n    \"event_id\": False,\n    \"event_time\": False,\n}\n\nfig, ax = plt.subplots(1, figsize=(12, 8))\nrl_fig, rl_ax = draw_rocket_league_ball(\n    data=sample_df, \n    meta=meta,\n    field=(fig, ax)\n);","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:44:02.801799Z","iopub.execute_input":"2022-10-12T20:44:02.802460Z","iopub.status.idle":"2022-10-12T20:44:03.553802Z","shell.execute_reply.started":"2022-10-12T20:44:02.802396Z","shell.execute_reply":"2022-10-12T20:44:03.552376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nfrom scipy.ndimage import gaussian_filter\n\ndef myplot(x, y, s, bins=1000):\n    \n    heatmap, xedges, yedges = np.histogram2d(x, y, bins=bins)\n    heatmap = gaussian_filter(heatmap, sigma=s)\n\n    extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]\n    \n    return heatmap.T, extent","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:44:14.323065Z","iopub.execute_input":"2022-10-12T20:44:14.324580Z","iopub.status.idle":"2022-10-12T20:44:14.334669Z","shell.execute_reply.started":"2022-10-12T20:44:14.324516Z","shell.execute_reply":"2022-10-12T20:44:14.333094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"N = 9\n\ncats = df_avg_ball_10s[df_avg_ball_10s[\"scoring_within_10sec\"] == 0].time_inverval_before_event_ending.cat.categories.tolist()[-N:-1]\ndf_avg_ball_no10s = df_avg_ball_10s[df_avg_ball_10s[\"scoring_within_10sec\"] == 0].copy()","metadata":{"execution":{"iopub.status.busy":"2022-10-12T20:44:17.085418Z","iopub.execute_input":"2022-10-12T20:44:17.086117Z","iopub.status.idle":"2022-10-12T20:44:17.117840Z","shell.execute_reply.started":"2022-10-12T20:44:17.086076Z","shell.execute_reply":"2022-10-12T20:44:17.116884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Since ploting a bunch of points does't make a lot of sense, I decided to create some heatmaps!","metadata":{}},{"cell_type":"code","source":"# Author: Alejandro\n# Source: https://stackoverflow.com/a/36515364\n\nfigs, axs = plt.subplots(3, 3, figsize=(20, 20))\n\nfor ax, i in zip(axs.flatten(), range(N)):\n    \n    if i == N-1:\n        df_aux = df_avg_ball_10s[\n            (df_avg_ball_10s[\"scoring_within_10sec\"] == 1)\n            & (10 < df_avg_ball_10s[\"ball_distance_from_center\"])\n            ]\n\n        img, extent = myplot(df_aux.ball_pos_y, df_aux.ball_pos_x, 16)\n        ax.imshow(img, extent=extent, origin='lower', cmap=cm.jet)\n        ax.set_title(\"Ball distribution when\\ngoal is scored within 10s\")\n    \n    else:\n        df_aux = df_avg_ball_no10s[\n            (df_avg_ball_no10s[\"scoring_within_10sec\"] == 0)\n            & (10 < df_avg_ball_no10s[\"ball_distance_from_center\"])\n            & (df_avg_ball_no10s[\"time_inverval_before_event_ending\"] == cats[i])\n            ]\n\n        # Generate some test data\n        x = df_aux.ball_pos_y\n        y = df_aux.ball_pos_x\n\n        img, extent = myplot(x, y, 16)\n        ax.imshow(img, extent=extent, origin='lower', cmap=cm.jet)\n        ax.set_title(f\"From {N*10 - (10*i)}s to {N*10 - (10*(i+1))}s\")\n        \n# ===== ATTENTION =====\n# X and Y are inverted to match the previous plot\n# The scoring zone is at left and right (not up/bottom)\n# =====================","metadata":{"execution":{"iopub.status.busy":"2022-10-12T21:01:12.264709Z","iopub.execute_input":"2022-10-12T21:01:12.265125Z","iopub.status.idle":"2022-10-12T21:01:16.774692Z","shell.execute_reply.started":"2022-10-12T21:01:12.265091Z","shell.execute_reply":"2022-10-12T21:01:16.773645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}