{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Episodification of the Test Dataset.\n\n\nIn this notebook, we will try to cluster the test data into episodes/events. The test dataset is nothing but a set of 'events', shuffled. While we may never be able to recover the exact order, we can try to get close enough so that it makes it easier to identify those 10-11% of the instants where a goal is actually scored by any team.\n\nInstead of directly checking similarity between instances, we will forecast the current frame one step ahead, making it more easier to identify the next frame. The player velocities and boosts are unreliable for this, as these are always under the players control, making it near impossible to forecast them. Hence we will be dropping these columns. Instead, we will focus on positions, and boost orb timers, as they are fairly calculatable.\n\n<br >\n<br >\n\nTo forecast the next frame, we will be using the same method as in the [Simulation Notebook](https://www.kaggle.com/code/aatiffraz/prediction-by-simulation-lets-play-rocket-league), where we try to predict the outcome by simulating a very primitive version of the game, so make sure to check that out!\n\nFor checking similarity between two frames, we will use euclidean distance.\n\n$$ sim\\_score_{\\{p, q\\}} =  \\sqrt[2]{ \\sum\\limits_{i \\in \\{ x, y, z \\}}{ (ball\\_pos\\_i_p - ball\\_pos\\_i_q)^2 } + \n\\sum\\limits_{i = 1}^{6}{ \\sum\\limits_{j \\in \\{x, y, z\\}} { ( pi\\_pos\\_j_p - pi\\_pos\\_j_q )^2 } } +\n\\sum\\limits_{i = 1}^{6}{ (boost\\_timer\\_i_p - boost\\_timer\\_j_q)^2 } }$$\n\n<br >\n\nBefore we get started with the test data, let's have a look at the train data first. The computation time is rather large, so we will leverage the power of GPUs (bought down the time taken for one iteration of clustering from about 5 hours to 21 mins :)","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom datetime import datetime\nimport torch\nimport gc\n!pip install plotly\nimport plotly.express as px\nimport plotly\n# plotly.offline.init_notebook_mode(connected=True)\nfrom IPython.display import clear_output","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-08T10:57:47.141207Z","iopub.execute_input":"2022-10-08T10:57:47.141753Z","iopub.status.idle":"2022-10-08T10:57:57.634935Z","shell.execute_reply.started":"2022-10-08T10:57:47.141714Z","shell.execute_reply":"2022-10-08T10:57:57.633874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = None\nif torch.cuda.is_available():\n    device = torch.device('cuda')\nelse:\n    device = torch.device('cpu')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-08T10:45:03.514728Z","iopub.execute_input":"2022-10-08T10:45:03.515541Z","iopub.status.idle":"2022-10-08T10:45:03.522690Z","shell.execute_reply.started":"2022-10-08T10:45:03.515500Z","shell.execute_reply":"2022-10-08T10:45:03.521401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"typedf = pd.read_csv('../input/tabular-playground-series-oct-2022/test_dtypes.csv')\n\ntest_types = {}\nfor _, row in typedf.iterrows():\n    if row.column == 'id': continue\n    test_types[row.column] = row['dtype']\n    \nstart_time = datetime.now()\n\n\ntestdf = pd.read_csv('../input/tabular-playground-series-oct-2022/test.csv', index_col='id').astype(test_types)\ntestensor = torch.Tensor(testdf.values).to(torch.float32).to(device)\ntestensor[torch.isnan(testensor)] = 0.","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-08T09:55:20.620553Z","iopub.execute_input":"2022-10-08T09:55:20.620902Z","iopub.status.idle":"2022-10-08T09:55:30.335551Z","shell.execute_reply.started":"2022-10-08T09:55:20.620874Z","shell.execute_reply":"2022-10-08T09:55:30.334543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"typedf = pd.read_csv('../input/tabular-playground-series-oct-2022/test_dtypes.csv')\n\ntrain_types = {}\nfor _, row in typedf.iterrows():\n    if row.column == 'id': continue\n    train_types[row.column] = row['dtype']\n\ntraindf = pd.read_csv('../input/tabular-playground-series-oct-2022/train_0.csv').astype(train_types)\n\ndata = traindf[['game_num', 'event_id']].groupby('event_id').count()\nprint(f'Total number of events: {len(traindf[\"event_id\"].unique())}')\nprint(f'Average size of an event {data.sum()/len(data)}, largest: {data.max()}')\n\npx.histogram(data['game_num'],\n            title=\"Count vs Size of Events in Train dataset\",\n            labels={\n                \"value\": \"Size of the Event\",\n                \"count\": \"Count\"\n            })","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-10-08T09:55:30.337088Z","iopub.execute_input":"2022-10-08T09:55:30.337458Z","iopub.status.idle":"2022-10-08T09:55:50.236418Z","shell.execute_reply.started":"2022-10-08T09:55:30.337429Z","shell.execute_reply":"2022-10-08T09:55:50.235419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The train dataset (train_0.csv) has about 3000 events, with sizes akin to a poisson distribution. ","metadata":{}},{"cell_type":"code","source":"train_types = {}\nfor _, row in typedf.iterrows():\n    if row.column == 'id': continue\n    if row.column not in testdf.columns: continue\n    train_types[row.column] = row['dtype']\n\n\nREQ_COLS = [0, 1, 2, 6, 7, 8, 13, 14, 15, 20, 21, 22, 27, 28, 29, 34, 35, 36, 41, 42, 43, 48, 49, 50, 51, 52, 53]\n\ntraindf = traindf.loc[:, testdf.columns]\n\nvals = torch.Tensor(traindf.values).to(torch.float32).to(device)\nvals = vals[:, REQ_COLS]\nvals[torch.isnan(vals)] = 0.\nvals1 = torch.concat([vals[1:], vals[-1].reshape((1, vals.shape[1]))], axis=0).to(device)\n\nansvals = torch.sqrt(torch.sum((vals1 - vals)**2, axis=1))\n\nprint(f'Average Distance between two consecutive frames: {ansvals.sum()/len(vals)}')\ncounts = ansvals[ansvals < 40].to(torch.int8).cpu().numpy()\n# bin_count = torch.bincount(ansvals[ansvals < 100].to(torch.int32)).cpu().numpy()\n# torch.cuda.synchronize()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-10-08T09:55:50.239753Z","iopub.execute_input":"2022-10-08T09:55:50.240542Z","iopub.status.idle":"2022-10-08T09:55:51.301030Z","shell.execute_reply.started":"2022-10-08T09:55:50.240494Z","shell.execute_reply":"2022-10-08T09:55:51.299966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(counts, nbins=40, title=\"Distance between consecutive frames in the Train Dataset\",\n                  labels={\n                      \"value\": \"Difference between t, t+1\",\n                  })\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-08T09:55:51.302142Z","iopub.execute_input":"2022-10-08T09:55:51.302535Z","iopub.status.idle":"2022-10-08T09:55:52.387207Z","shell.execute_reply.started":"2022-10-08T09:55:51.302494Z","shell.execute_reply":"2022-10-08T09:55:52.385573Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"5-15 is a fairly small distance between two consecutive frames. Let's move on to the test dataset.","metadata":{}},{"cell_type":"code","source":"del vals, vals1, traindf, testdf, ansvals\n_ = gc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-08T10:47:19.644981Z","iopub.execute_input":"2022-10-08T10:47:19.645784Z","iopub.status.idle":"2022-10-08T10:47:19.689528Z","shell.execute_reply.started":"2022-10-08T10:47:19.645734Z","shell.execute_reply":"2022-10-08T10:47:19.688530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class StepUtil():\n    def __init__(self, device):\n        self.zeros = torch.zeros(6).to(device)\n        \n        self.min_limit = torch.from_numpy(np.array([-85, -105, 0], dtype=np.float32)).to(device)\n        self.max_limit = torch.from_numpy(np.array([85, 105, 40], dtype=np.float32)).to(device)\n\n    def futurize(self, ind):\n        next_row = req_vals[ind].clone().detach()\n        orig_row = testensor[ind]\n\n        # Updating Ball Pos and checking bounds\n        next_row[:3] = next_row[:3] + orig_row[3:6]*0.1\n        next_row[:3] = torch.minimum(torch.maximum(next_row[:3], self.min_limit), self.max_limit)\n\n        #Updating Player Pos and checking bounds\n        for a, b in zip(range(3, 19, 3), range(9, 45, 7)):\n            next_row[a: a+3] = next_row[a: a+3] + 0.1 * orig_row[b: b+3]\n            next_row[a: a+3] = torch.minimum(torch.maximum(next_row[a:a+3], self.min_limit), self.max_limit)\n\n        # Updating boost timers\n        next_row[21:] = torch.minimum(next_row[21:] + 0.11, self.zeros)\n\n        return next_row","metadata":{"execution":{"iopub.status.busy":"2022-10-08T09:55:52.637885Z","iopub.execute_input":"2022-10-08T09:55:52.638204Z","iopub.status.idle":"2022-10-08T09:55:52.650418Z","shell.execute_reply.started":"2022-10-08T09:55:52.638175Z","shell.execute_reply":"2022-10-08T09:55:52.649295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"REQ_COLS = [0, 1, 2, 6, 7, 8, 13, 14, 15, 20, 21, 22, 27, 28, 29, 34, 35, 36, 41, 42, 43, 48, 49, 50, 51, 52, 53]\n\nreq_vals = testensor[:, REQ_COLS].to(device)\n\nindexnp = np.array([i for i in range(testensor.shape[0])], dtype=np.int32)\nepisodes = torch.from_numpy(indexnp).to(device)\n\nerrs = torch.from_numpy(np.zeros(testensor.shape[0])).to(device)\n\nsteputil = StepUtil(device)\nrows = testensor.shape[0]\n\nstart_time = datetime.now()\n\nis_changing = True\nin_iter = 0\n\nwhile is_changing:\n    is_changing = False\n    in_iter += 1\n    print(f'Length of the average event: {episodes.shape[0]/len(episodes.unique())}')\n    print(f'Largest Episode Cluster Id: {episodes.mode().values.item()}, With {torch.sum(episodes == episodes.mode().values).cpu()} frames.')\n\n    for i in range(rows):\n\n        if i % 2000 == 0:\n            clear_output(wait=True)\n            print(f'{in_iter}:{i} in {datetime.now() - start_time}')\n\n        # Take the differences between all frames, and set the difference\n        # from current frame to a large value, then store the distance to closest frames\n        diffs = torch.sqrt(torch.sum((req_vals - steputil.futurize(i))**2, axis=1))\n        diffs[i] = 1e8\n        errs[i] = diffs.min()\n        \n        torch.cuda.synchronize()\n\n    break\n","metadata":{"execution":{"iopub.status.busy":"2022-10-08T09:55:52.651926Z","iopub.execute_input":"2022-10-08T09:55:52.652285Z","iopub.status.idle":"2022-10-08T09:55:53.052092Z","shell.execute_reply.started":"2022-10-08T09:55:52.652252Z","shell.execute_reply":"2022-10-08T09:55:53.050359Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"errs = errs.cpu().numpy()\nfig = px.histogram(errs, nbins=400, title=\"Distance of each frame in the test dataset to it's closest frame\")\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-10-08T10:55:50.870728Z","iopub.execute_input":"2022-10-08T10:55:50.871306Z","iopub.status.idle":"2022-10-08T10:55:51.468837Z","shell.execute_reply.started":"2022-10-08T10:55:50.871262Z","shell.execute_reply":"2022-10-08T10:55:51.467344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So the distances between forecasted frames to their closest frames are much larger. We will use these values to set the threshold for clustering. Assuming there are no standalone events of only a single frame, it will be safe to assume that in the worst cases(collisions etc) the distance between two consecutive frame will not go larger than 80. If they do, we simply let that row be. Otherwise we will merge the cluster of the present event with the closest one. ","metadata":{}},{"cell_type":"code","source":"REQ_COLS = [0, 1, 2, 6, 7, 8, 13, 14, 15, 20, 21, 22, 27, 28, 29, 34, 35, 36, 41, 42, 43, 48, 49, 50, 51, 52, 53]\n\nreq_vals = testensor[:, REQ_COLS].to(device)\n\nindexnp = np.array([i for i in range(testensor.shape[0])], dtype=np.int32)\nepisodes = torch.from_numpy(indexnp).to(device)\n\nerrs = torch.from_numpy(np.zeros(testensor.shape[0])).to(device)\n\nsteputil = StepUtil(device)\nrows = testensor.shape[0]\n\nstart_time = datetime.now()\n\nis_changing = True\nin_iter = 0\n\nwhile is_changing:\n    is_changing = False\n    in_iter += 1\n    print(f'Length of the average event: {episodes.shape[0]/len(episodes.unique())}')\n    print(f'Largest Episode Cluster Id: {episodes.mode().values.item()}, With {torch.sum(episodes == episodes.mode().values).cpu()} frames.')\n    \n    # Keeping in mind Kaggle Notebook limits\n    if in_iter > 10: break\n    for i in range(rows):\n\n        if i % 2000 == 0:\n            clear_output(wait=True)\n            print(f'{in_iter}:{i} in {datetime.now() - start_time}')\n\n        # Take the differences between all frames, and set the difference\n        # from current frame to a large value\n        diffs = torch.sqrt(torch.sum((req_vals - steputil.futurize(i))**2, axis=1))\n        diffs[i] = 1e8\n        \n          # No point in checking within cluster\n        cond = (episodes == episodes[i])\n        diffs[cond] = 1e8\n\n        # If couldn't find any close frames, just skip\n        if diffs.min() > 80. : continue\n\n        is_changing = True\n        \n        # Merge the two clusters\n        # The 'closest_frame' is probably the next timeframe, hence we merge that into\n        # the present cluster, retaining present cluster's id.\n        # Hoping to see if we can de-shuffle the episodes this way as well\n        closest_frame = diffs.argmin()\n        closest_event = (episodes == closest_frame)\n        episodes[closest_event] = episodes[i].item()\n\n        torch.cuda.synchronize()\n\n\n# Average length of each episode\nprint(f'Length of the average event: {episodes.shape[0]/len(episodes.unique())} and number of events: {len(episodes.unique())}')\nprint(f'Largest Episode Id: {episodes.mode().values.item()}, With {torch.sum(episodes == episodes.mode().values).cpu()} instances')","metadata":{"execution":{"iopub.status.busy":"2022-10-08T09:55:53.055759Z","iopub.status.idle":"2022-10-08T09:55:53.056312Z","shell.execute_reply.started":"2022-10-08T09:55:53.056038Z","shell.execute_reply":"2022-10-08T09:55:53.056063Z"},"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bin_count = torch.bincount(episodes).detach().cpu().numpy()\n# The large majority of events are single-frame anyway, so ignore them here\nbin_count = bin_count[((bin_count > 1))]\n\nfig = px.histogram(bin_count,\n                   nbins=100,\n                  labels = {\n                      \"value\": \"Size of an Event\",\n                      \"count\": \"Count of Events\",\n                  },\n                  title=\"Size vs Count of Events formed\")\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-08T09:55:53.057932Z","iopub.status.idle":"2022-10-08T09:55:53.058465Z","shell.execute_reply.started":"2022-10-08T09:55:53.058196Z","shell.execute_reply":"2022-10-08T09:55:53.058220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The distribution seems similar to that of the training data. Let's go ahead and save it to our test data.","metadata":{}},{"cell_type":"code","source":"testdf = pd.read_csv('../input/tabular-playground-series-oct-2022/test.csv', index_col='id').astype(test_types)\ntestdf['event_id'] = episodes.detach().cpu().numpy()\ntestdf.to_csv('clustered_testdata.csv')","metadata":{"execution":{"iopub.status.busy":"2022-10-08T11:17:18.440339Z","iopub.execute_input":"2022-10-08T11:17:18.440812Z","iopub.status.idle":"2022-10-08T11:17:55.260219Z","shell.execute_reply.started":"2022-10-08T11:17:18.440772Z","shell.execute_reply":"2022-10-08T11:17:55.258897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Feel free to comment on what could be improved, or to use it in your notebooks. Please do like the notebook if you found it informative/helpful, as its taken hours and hours to learn pytorch, implement, debug and tune. I'd be very happy if it comes of use to anybody, or even provides them a way to improve on this and come up with something interesting.\n\nThanks for reading!","metadata":{}}]}