{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Big Data Derby 2022</p>\n<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/03%20kentucky%20derby.jpg?raw=true\">\n<a id='top'></a>","metadata":{"execution":{"iopub.status.busy":"2022-08-12T10:32:46.472602Z","iopub.execute_input":"2022-08-12T10:32:46.473067Z","iopub.status.idle":"2022-08-12T10:32:46.478683Z","shell.execute_reply.started":"2022-08-12T10:32:46.473025Z","shell.execute_reply":"2022-08-12T10:32:46.477801Z"}}},{"cell_type":"markdown","source":"## Goal of the Competition\nThe goal of this competition is to analyze horse racing tactics, drafting strategies, and path efficiency. You will develop a model using never-before-released coordinate data along with basic race information.\n\nYour work will help racing horse owners, trainers, and veterinarians better understand how equine performance and welfare fit together. With better data analysis, equine welfare could significantly improve.","metadata":{}},{"cell_type":"markdown","source":"I have to admit that my skill is not enough to meet the goal of this competition. So my objective of joining this is to practice and of course learn from the expert. So if you have any thoughts on how I could improve, I would appreciate if you would fire them up in the comment session.","metadata":{}},{"cell_type":"markdown","source":"## Scoring\nAn entry to the competition consists of a Notebook submission that is evaluated on the following five components, where 0 is the low score and 100 is the high score. Submissions will be judged based on how well they address:\n\n### Innovation (25 Points Total)\n\nIs this a novel way of looking at tracking data? (10 Pts)\nAre the statistical/machine learning approaches using the most up-to-date standards? (5 Pts)\nWill the conclusions challenge the status quo of horse racing methods? (10 Pts)\n\n### Relevance (30 Points Total)\n\nCan the conclusions influence equine welfare, equine performance or rider decision making? (10 Points)\nCan the conclusions be the basis of future research on future (larger, more granular) data sets? (10 Points)\nAre the conclusions something that horse racing participants (e.g. owners, trainers, veterinarians) can understand, digest and debate? (10 Points)\n\n### Competence (25 Points Total)\n\nGiven the data, are the statistical models appropriate? (5 Points)\nAre the conclusions supported by the data? (10 Points)\nIs the analysis accurate? (10 Points)\n\n### Presentation (20 Points Total)\n\nIs the writing clear and free of nomenclature? (5 Points)\nAre the charts and tables provided interesting, visually appealing, and accurate? (5 Points)\nCan the analysis thread be followed throughout the presentation? (10 Points)\n","metadata":{}},{"cell_type":"markdown","source":"With that in mind, let's get started to practice my skill with this data!","metadata":{}},{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Start</p>\n<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/02%20Start%20Horse%20Race.jpg?raw=true\">","metadata":{}},{"cell_type":"markdown","source":"# Acknowledgement\nI am a beginner to this field, so I took [this notebook](#https://www.kaggle.com/code/mattop/big-data-derby-2022-eda) as reference. To objective of writing this notebook is to improve myself and help other beginners to get started. ","metadata":{}},{"cell_type":"markdown","source":"   <a id='table-of-content'></a>\n<div class=\"list-group\" id=\"list-tab\" role=\"tablist\">\n<p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:150%;text-align:center;border-radius:10px 10px;\">TABLE OF CONTENTS</p>   \n    \n* [1. Top](#top)\n* [2. Table of Content](#table-of-content)\n* [3. Import Libraries](#import-libraries)\n* [4. Read Data](#read-data)\n* [5. Tracking](#tracking)\n* [6. Analysis](#analysis)\n* [Work In Progress](#work-in-progress)\n* [End](#end)","metadata":{}},{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Import Libraries</p>\n<a id='import-libraries'></a>","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport plotly.express as px #graphing\nimport plotly.graph_objects as go #graphing\nfrom plotly.subplots import make_subplots #graphing\nimport plotly.figure_factory as ff #graphing\nimport matplotlib.pyplot as plt #graphing\nimport seaborn as sns #graphing\nimport missingno as msno #describe data\nimport os\nsns.set_theme(style=\"darkgrid\")\n\ncolors = [\"#FFFFFF\", \"#6CD4FF\", \"#F7DF00\", \"#E60000\"]\n\nplt.rcParams[\"figure.figsize\"] = (12, 8)\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-13T01:08:15.895101Z","iopub.execute_input":"2022-08-13T01:08:15.895615Z","iopub.status.idle":"2022-08-13T01:08:19.685939Z","shell.execute_reply.started":"2022-08-13T01:08:15.895501Z","shell.execute_reply":"2022-08-13T01:08:19.684842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Read Data</p>\n<a id='read-data'></a>","metadata":{}},{"cell_type":"code","source":"nyra_tracking = pd.read_csv(\"/kaggle/input/big-data-derby-2022/nyra_tracking_table.csv\")\nnyra_start = pd.read_csv(\"/kaggle/input/big-data-derby-2022/nyra_start_table.csv\")\nnyra_race = pd.read_csv(\"/kaggle/input/big-data-derby-2022/nyra_race_table.csv\")\nnyra_2019 = pd.read_csv(\"/kaggle/input/big-data-derby-2022/nyra_2019_complete.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-08-13T01:08:19.688034Z","iopub.execute_input":"2022-08-13T01:08:19.688390Z","iopub.status.idle":"2022-08-13T01:08:44.949891Z","shell.execute_reply.started":"2022-08-13T01:08:19.688357Z","shell.execute_reply":"2022-08-13T01:08:44.948966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_tracking.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:36:59.200440Z","iopub.execute_input":"2022-08-12T11:36:59.200768Z","iopub.status.idle":"2022-08-12T11:36:59.208368Z","shell.execute_reply.started":"2022-08-12T11:36:59.200739Z","shell.execute_reply":"2022-08-12T11:36:59.207221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:36:59.211026Z","iopub.execute_input":"2022-08-12T11:36:59.211372Z","iopub.status.idle":"2022-08-12T11:36:59.220969Z","shell.execute_reply.started":"2022-08-12T11:36:59.211342Z","shell.execute_reply":"2022-08-12T11:36:59.219847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_race.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:36:59.223724Z","iopub.execute_input":"2022-08-12T11:36:59.224091Z","iopub.status.idle":"2022-08-12T11:36:59.233550Z","shell.execute_reply.started":"2022-08-12T11:36:59.224060Z","shell.execute_reply":"2022-08-12T11:36:59.232622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_2019.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:36:59.234668Z","iopub.execute_input":"2022-08-12T11:36:59.235002Z","iopub.status.idle":"2022-08-12T11:36:59.250193Z","shell.execute_reply.started":"2022-08-12T11:36:59.234972Z","shell.execute_reply":"2022-08-12T11:36:59.249371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_tracking.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:36:59.251103Z","iopub.execute_input":"2022-08-12T11:36:59.251894Z","iopub.status.idle":"2022-08-12T11:36:59.270331Z","shell.execute_reply.started":"2022-08-12T11:36:59.251857Z","shell.execute_reply":"2022-08-12T11:36:59.269152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_tracking['track_id'].unique()","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:36:59.271900Z","iopub.execute_input":"2022-08-12T11:36:59.272763Z","iopub.status.idle":"2022-08-12T11:36:59.601599Z","shell.execute_reply.started":"2022-08-12T11:36:59.272720Z","shell.execute_reply":"2022-08-12T11:36:59.600454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Tracking</p>\n<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/01%20Race%20Track.jpg?raw=true\">\n<a id='tracking'></a>","metadata":{}},{"cell_type":"code","source":"for i in ['AQU','BEL','SAR']:\n    if i =='AQU':\n        track = nyra_2019[(nyra_2019[\"track_id\"] == i)&(nyra_2019[\"race_number\"] == 1)]\n    plt.style.use(\"Solarize_Light2\")\n    plt.figure(figsize = (16, 8))\n    sns.scatterplot(data = track, x = \"longitude\", y = \"latitude\", hue = \"jockey\", palette = \"Set2\")\n\n    plt.title(f\"{i}\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-13T01:56:46.835389Z","iopub.execute_input":"2022-08-13T01:56:46.835788Z","iopub.status.idle":"2022-08-13T01:57:21.571616Z","shell.execute_reply.started":"2022-08-13T01:56:46.835754Z","shell.execute_reply":"2022-08-13T01:57:21.570559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Click [here](#https://matplotlib.org/stable/gallery/style_sheets/style_sheets_reference.html) for more options to style your plot.","metadata":{}},{"cell_type":"markdown","source":"**trakus_index** - The common collection of point of the lat / long of the horse in the race passed as an integer. From what we can tell, it's collected every 0.25 seconds.\n<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/07%20trakus%20index.png?raw=true\">\nSo if I understand it right trakus_index is the actual distance in the race","metadata":{}},{"cell_type":"code","source":"# select race\ndf_select = nyra_2019[(nyra_2019.track_id=='SAR') &\n               (nyra_2019.race_date=='2019-09-02') &\n               (nyra_2019.race_number==1)]","metadata":{"execution":{"iopub.status.busy":"2022-08-13T01:48:24.188372Z","iopub.execute_input":"2022-08-13T01:48:24.188792Z","iopub.status.idle":"2022-08-13T01:48:25.061273Z","shell.execute_reply.started":"2022-08-13T01:48:24.188757Z","shell.execute_reply":"2022-08-13T01:48:25.060216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('min: ',df_select['trakus_index'].min())\nprint('max: ',df_select['trakus_index'].max())","metadata":{"execution":{"iopub.status.busy":"2022-08-13T01:53:17.203335Z","iopub.execute_input":"2022-08-13T01:53:17.203782Z","iopub.status.idle":"2022-08-13T01:53:17.211169Z","shell.execute_reply.started":"2022-08-13T01:53:17.203744Z","shell.execute_reply":"2022-08-13T01:53:17.209785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Run progress\nfor i in [1,50,100,150,200,300,400]:\n\n    plt.figure(figsize=(10,7))\n    sns.scatterplot(data=df_select[df_select.trakus_index>i], x='longitude', y='latitude',\n                    hue='jockey',\n                    size='trakus_index')\n    plt.title(f\"trakus_index > {i}\")\n    plt.grid()\n    plt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-08-13T02:01:51.500920Z","iopub.execute_input":"2022-08-13T02:01:51.501325Z","iopub.status.idle":"2022-08-13T02:01:55.831051Z","shell.execute_reply.started":"2022-08-13T02:01:51.501293Z","shell.execute_reply":"2022-08-13T02:01:55.829874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Analysis</p>\n<a id='analysis'></a>\n<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/04%20Analysis.png?raw=true\">","metadata":{}},{"cell_type":"code","source":"nyra_start.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:37:00.009179Z","iopub.status.idle":"2022-08-12T11:37:00.009820Z","shell.execute_reply.started":"2022-08-12T11:37:00.009581Z","shell.execute_reply":"2022-08-12T11:37:00.009603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pd.option_context(\"display.max_rows\", None):\n    print(nyra_start['jockey'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:37:00.010999Z","iopub.status.idle":"2022-08-12T11:37:00.011561Z","shell.execute_reply.started":"2022-08-12T11:37:00.011367Z","shell.execute_reply":"2022-08-12T11:37:00.011388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Odds Calculation","metadata":{}},{"cell_type":"markdown","source":"**odds** - Odds to win the race passed as an integer. Divide by 100 to derive the odds to 1. Example - 1280 would be 12.8-1.","metadata":{}},{"cell_type":"markdown","source":"## Original distribution of odds","metadata":{}},{"cell_type":"code","source":"binwidth = 100\ndata = nyra_start['odds']\nplt.hist(data, bins=range(min(data), max(data) + binwidth, binwidth))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:37:00.012522Z","iopub.status.idle":"2022-08-12T11:37:00.013531Z","shell.execute_reply.started":"2022-08-12T11:37:00.013312Z","shell.execute_reply":"2022-08-12T11:37:00.013334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Divide by 100\nnyra_start['odds'] = nyra_start['odds']/100","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:37:00.014862Z","iopub.status.idle":"2022-08-12T11:37:00.015239Z","shell.execute_reply.started":"2022-08-12T11:37:00.015050Z","shell.execute_reply":"2022-08-12T11:37:00.015069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start['odds']","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:37:00.017128Z","iopub.status.idle":"2022-08-12T11:37:00.017512Z","shell.execute_reply.started":"2022-08-12T11:37:00.017325Z","shell.execute_reply":"2022-08-12T11:37:00.017343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#calculate sum of values by group\ndf_groups = pd.DataFrame(nyra_start.groupby(['jockey'])['odds'].mean())\ndf_plot = df_groups.loc[df_groups['odds']>50]\n\n#create bar plot by group\ndf_plot.sort_values(by=['odds'],ascending=True).plot.barh()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-12T11:37:00.019379Z","iopub.status.idle":"2022-08-12T11:37:00.019778Z","shell.execute_reply.started":"2022-08-12T11:37:00.019580Z","shell.execute_reply":"2022-08-12T11:37:00.019598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The bar chart shows jockey that has odds of more than 50-1. I don't know much about horse racing but is odds more than 50 very common? Appreciate if you would share your knowledge on horse racing in the comment session.","metadata":{}},{"cell_type":"markdown","source":"# Track last position\nCreate a horse rating measuring expected finish position versus actual finish position. How does a horse’s expected finish position change through the running of a race? Does this metric rely solely on a horse’s own position or is it influenced by the position of competitors?","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <p style=\"background-color:#682F2F;font-family:newtimeroman;color:#FFF9ED;font-size:200%;text-align:center;border-radius:50px 50px;\">Work In Progress</p>\n<a id='work-in-progress'></a>\n<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/06%20Work%20in%20progress.png?raw=true\">\n","metadata":{}},{"cell_type":"markdown","source":"<img\nsrc=\"https://github.com/Chee-ChuanFoo/Big-Data-Derby/blob/main/05%20thank%20you.png?raw=true\">\n<a id='end'></a>\n[back to top](#top)\n<br></br>\n[Table of Content](#table-of-content)","metadata":{}}]}