{"cells":[{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:13:59.814370Z","start_time":"2019-01-09T00:13:59.800356Z"},"trusted":true,"_uuid":"bd3ff41db5c9206f7248c24e84408dc23d87abf2","_kg_hide-input":true},"cell_type":"code","source":"from IPython.display import HTML\nstyle = \"\"\"\n\n\"\"\"\nHTML(style)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2cdb1bdea0c57e4e0b8e8c3a55c56a540f0fd979"},"cell_type":"markdown","source":"# Introduction\nThere are many ways to make the punt play safer. Concussions, fortunately, are rare events. On over 6000 punts, there were 37 concussions. With this type of data sparsity, it would be easy to see signals that are not there. To address that problem, I took the following approach.\n\n* Clean and label the data -- This kernal analyzes concussion data from 2016 and 2017. There were rule changes before the 2017 and 2018 season. I identify the concussions covered by the current rules and remove them from the analysis. This cleaning required detailed video analysis and review of the NFL rules. I used the NGS data to produce play diagrams and impact charts to help identify exactly where, when, and who were involved in the concussion. During this cleaning, I found and fixed multiple problems. After this process, I had a root cause determined for each concussion.\n\n* Brainstorm a large list of possible rules -- Thinking about what rules are possible first allowed me to apply these possible rules to each of the videos and determined if a particular rule would prevent a particular concussions. For instance if the problem was a blocking technique, there are multiple possible ways to prevent that technique.\n\n* Visualizations -- Bar charts make differences between small numbers seem more significant. A difference between 2 and 4 concussions out of 6000 punts is completely insignificant but a bar char would make that 4 look twice as important as the 2. For that reason, I have used tables when summarizing concussions. When I used graphs, I focused on visualizations that represented all of the data instead of a summary.\n\n### Audience\nThere is a variety of people who hopefully will be reading this kernal. I tried to put enough data analysis to keep data scientists happy and enough domain knowledge for everyone else. For those that want to skip the analytics, I have included a summary that highlights the key takeaways from each section\n\n\n### Rule Types\nA brief overview of rules that will be evaluated. This overview is deliberately brief. After analyzing the data, I will focus on a few rules. At that point, more details will be provided.\n\n**Reduce Punts**\n* CFL Rouge/Single - https://en.wikipedia.org/wiki/Single_(football)\n* Fair Catch Bonus Yards - A fair catch is treated as a 5 or 10 yard return\n* Encourage teams to keep kicks out of play\n * Out of Bounds Bonus Yards - Converse of previous rule. Out of bound punts are rewarded\n * Reduce touchback distance\n* No Preseason Punts\n* Encourage going for it on 4th down - On missed FG's, keep ball at line of scrimmage\n\n\n**Make Punts Safer**\n* Formation (borrow from Kickoff Rules)\n * Limit receiving team players outside the numbers\n * Require same number of players on each side of the ball\n* Tackling\n * Stricter targeting rules\n* Blocking\n * No Wedge Blocks\n * Expand Blindside block rules\n* Returner Protection\n * Play calls dead early\n * On muffed punts\n * On fair catch\n * CFL buffer rule\n * Fair Catch even when ball bounces\n\n**Eliminate Punts**\n* Automatic Punt - Instead of punting, move the ball as if there was a 40 yard net punt.\n* Half Punt - If team fails on 4th down, treat it like a 20 yard net punt.\n\n### Integrity of the game\nRules can address three phases of the play:\n1. Before\n * Formation rules, illegal player downfield\n * Attempt to prevent bad situations from happening\n * Impacts all plays to prevent bad plays\n2. During\n * Blindside blocks, targeting, landing on the QB, CFL 5 yard buffer\n * Target very specific behavior\n * Biggest impact on both concussions and integrity of game.\n3. End\n * Touchbacks, spot of the ball after missed field goal, CFL rouge\n * Encourages players to avoid bad situations\n * Selectively applied if player chooses it\n \n All other things being equal, a rule that is applied at the end of the play changes the game the least.\n\n\n\n"},{"metadata":{"_uuid":"dd49e94c47df9472a6e0cccb75e598c8d5f7fbcb"},"cell_type":"markdown","source":"# Load Data and Basic EDA"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:02.004484Z","start_time":"2019-01-09T00:13:59.816372Z"},"_kg_hide-input":true,"_uuid":"b055a1d2694473f5fe1399824cc7b6aca8725732","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport glob\nfrom plotly import offline\nimport plotly.graph_objs as go\nimport os\nimport numpy as np\npd.set_option('max.columns', None)\npd.set_option('display.max_rows', 500)\npd.set_option('display.max_columns', 500)\npd.set_option('display.width', 1000)\noffline.init_notebook_mode()\nconfig = dict(showLink=False)\nfrom IPython.display import IFrame\nfrom IPython.display import display\nfrom IPython.display import Image\n\n#Load provided files\nplay_information = pd.read_csv('../input/NFL-Punt-Analytics-Competition/play_information.csv')\nvideo_review = pd.read_csv('../input/NFL-Punt-Analytics-Competition/video_review.csv')\nvideo_footage_injury = pd.read_csv('../input/NFL-Punt-Analytics-Competition/video_footage-injury.csv')\nvideo_footage_control = pd.read_csv('../input/NFL-Punt-Analytics-Competition/video_footage-control.csv')\ngame_data = pd.read_csv('../input/NFL-Punt-Analytics-Competition/game_data.csv')\nplayer_jersey = pd.read_csv('../input/NFL-Punt-Analytics-Competition/player_punt_data.csv')\nplayer_role = pd.read_csv('../input/NFL-Punt-Analytics-Competition/play_player_role_data.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d50cb9634781e2d80c8c0d305cbade6468ec968"},"cell_type":"markdown","source":"### Parse information from play_information.csv\nPlay_information.csv has information on all punts for 2016-2017.\n\nThe play description column on play information has a wealth of information but required some basic text extractions. The play description had the result of the punt, punt distance, return distance, \n\nThe punt location was recorded in the standard team yardline format (ATL 48 or SF 35). For analysis, I converted these into yards from TD.\n\nFor yard metrics, I used the qcut feature that divided the total data into 5 equal groups - 0-20th percentile, 20-40th percentile, etc. I combined that overall data with the concussion data to help see if concussions were more frequent in certain groups. For instance, were there more concussions on the longest punts? The benefit of this technique is that for every variable I get a simple 5 row table. At a quick glance, I can see if the rows are roughly equal or if one row has most of the concussions. For instance if return yards were correlated with concussions, I would expect the 80th-100th percentile group to have the most concussions. "},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.088496Z","start_time":"2019-01-09T00:14:02.191665Z"},"trusted":true,"_uuid":"ec6253875c09b760cef04afb6005c49a61061a52","_kg_hide-input":true},"cell_type":"code","source":"# replace \". \" with \",\" to make the descriptions easier to separate\nplay_information.PlayDescription=play_information.PlayDescription.str.replace('. ',',',regex=False)\n\n#Look for key words. 0 = not found, 1 = found\nplay_information['Center']=np.where(play_information.PlayDescription.str.find('Center')==-1,0,1)\nplay_information['Penalty']=np.where(play_information.PlayDescription.str.find('PENALTY')==-1,0,1)\nplay_information['Downed']=np.where(play_information.PlayDescription.str.find('downed')==-1,0,1)\nplay_information['Touchback']=np.where(play_information.PlayDescription.str.find('Touchback')==-1,0,1)\nplay_information['OutofBounds']=np.where(play_information.PlayDescription.str.find('out of bounds')==-1,0,1)\nplay_information['FairCatch']=np.where(play_information.PlayDescription.str.find('fair catch')==-1,0,1)\nplay_information['TD']=np.where(play_information.PlayDescription.str.find('TOUCHDOWN')==-1,0,1)\nplay_information['Injury']=np.where(play_information.PlayDescription.str.find('njur')==-1,0,1)\n\n#Divide the play description into 4 parts -- Punter, Center, Returner, Extra \nplay_information[['Punter','Center','Returner','Extra']]=play_information.PlayDescription.str.split(',', expand=True, n=3)\nplay_information.PlayDescription=play_information.PlayDescription.str.replace('(Punt formation)','',regex=False)\nplay_information['ReturnYards'] = play_information.Returner.str.extract(r'for.([\\w]+)')\nplay_information['PuntYards'] = play_information.Punter.str.extract(r'punts.([\\w]+)')\n\n#Clean up errors\nplay_information['PuntYards'] = pd.to_numeric(play_information['PuntYards'], errors='coerce')\nplay_information['ReturnYards'] = pd.to_numeric(play_information['ReturnYards'].str.replace('no','0',regex=False), errors='coerce')\nplay_information['PuntYards_qcut'] = pd.qcut(play_information['PuntYards'],5)\nplay_information['ReturnYards_qcut'] = pd.qcut(play_information['ReturnYards'],5)\n\n#Calculate how close the punting team was to a TD when they punted.\n#If the ball is on the Possession team's side, this will calculate the distance to the endzone\n#If the ball is on the receiver team's side, this will return 1.\nplay_information['YardsforTD_possession']=100-pd.to_numeric(play_information.apply(lambda x : x['YardLine'].replace((x['Poss_Team']+' '),''),1), errors='coerce')\nplay_information['YardsforTD_receive']=play_information.YardLine.str.extract(r'([\\d]+)')\nplay_information.YardsforTD_receive = pd.to_numeric(play_information.YardsforTD_receive)\n#Find the larger of the two to get the real value. Probably could have done this in one line, but \n#I think it's clearer this way\nplay_information['YardsforTD']= play_information[['YardsforTD_possession','YardsforTD_receive']].max(axis=1)\nplay_information['YardsforTD_qcut'] = pd.qcut(play_information['YardsforTD'],5)\n#Calculate netyards by starting with punt yards and subtracting either return or touchback\nplay_information['NetYards'] = play_information.PuntYards\nplay_information['NetYards'] = play_information['NetYards'] - play_information.Touchback.apply(lambda x: 20 if x > 0 else 0)\nplay_information['NetYards'] = play_information['NetYards'] - play_information.ReturnYards.apply(lambda x: 0 if pd.isna(x) else x)\nplay_information['NetYards_qcut'] = pd.qcut(play_information['NetYards'],5)\n#Make the score usable\nplay_information[['Score_Home','Score_Visiting']] = play_information.Score_Home_Visiting.str.split('-',expand=True,n=2)\nplay_information['Score_Diff'] = abs(pd.to_numeric(play_information.Score_Home) - pd.to_numeric(play_information.Score_Visiting))\nplay_information['Score_Diff_qcut'] = pd.qcut(play_information['Score_Diff'],4)\n\n#Determine receiving team\nplay_information['Rec_Team'] =play_information.apply(lambda x : x['Home_Team_Visit_Team'].replace(x['Poss_Team'],''),1)\nplay_information['Rec_Team'] = play_information.apply(lambda x : x['Rec_Team'].replace('-',''),1)\n\n#Delete intermediate variables\nplay_information.drop(['Punter','Center','Returner','Extra','YardsforTD_possession','YardsforTD_receive'], axis=1, inplace=True)\n\n#play_information.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66c88a03a6166c22f2bcc58e957b6d5a6444d8ea"},"cell_type":"markdown","source":"## Combine Video footage and video review. \nI am using video injury and video control to get the URL of the plays.\nThe details of the play I will get from play_information."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.100507Z","start_time":"2019-01-09T00:14:04.090498Z"},"trusted":true,"_uuid":"4a1fc4b858e35c0c26ab9911c6512d49e2e837d1","_kg_hide-input":true},"cell_type":"code","source":"#Injury Data -- Need only URL column. All other columns are in play_information\nvideo_footage_injury = video_footage_injury[['season','gamekey','playid','PREVIEW LINK (5000K)']]\nvideo_footage_injury.columns = ['Season','GameKey','PlayID','URL']\n\n#Control data -- Need only URL column. All other columns are in play_information\nvideo_footage_control = video_footage_control[['season','gamekey','playid','Preview Link']]\nvideo_footage_control.columns = ['Season','GameKey','PlayID','URL']\n\n#for injury data, add in video review columns\nvideo = pd.merge(video_footage_injury,video_review, on=['GameKey','PlayID'])\n#video.head()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-06T13:01:04.115122Z","start_time":"2019-01-06T13:01:04.089097Z"},"_uuid":"d29b5ae5eba5730e275fafaa4e0df5046a4036da"},"cell_type":"markdown","source":"### Format Data to Analyze Formations"},{"metadata":{"_uuid":"c1419746c8bc28a6c39bcce57b2ccda2c1f3305e"},"cell_type":"markdown","source":"I created a file to summarize the position data. There are 8 different types of defensive lineman, but they are all essentially the same position. By looking at them at a group, I can better detect patterns.\n\nNote: There is a nice symmetry between the roles. Each role has a counterpart on the other team\n"},{"metadata":{"_uuid":"f85fd5ca9ab051fd7c589681a08e1124626b0e2a"},"cell_type":"markdown","source":"Coverage Members\n\n| Role | Description |\n|----------|----------------------------------------------------------------------------|\n|Gunner | Lines up outside and tries to reach punt returner before the ball arrives|\n|OLine | Offensive Line. 5 members must be on the line of scrimmage |\n|Wing | Line up on the sides of the Oline |\n|Backfield | Off the line of scrimmage |\n|Punter | Lines up 15 yards behind line of scrimmage |\n\nReturning Members\n\n| Role | Description |\n|------------|:-----------------------------------------------------------------------------------|\n|Jammer | Blocks Gunner |\n|DLine | Defensive Line. Line up on line of scrimmage |\n|LineBacker | Lines up behind the line of scrimmage |\n|Front Block | Back with returner to block or catch short punts |\n|Returner | Lines up 40 yards behind line of scrimmage. chooses to return or signal fair catch |\n"},{"metadata":{"_uuid":"3ac90a57ee1ff27fce2e5ed16c32f5f60d610e1a","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"Image(\"../input/images/Players_Roles.png\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae47602f2b741051cdf5dc8df5a67555912090e0"},"cell_type":"markdown","source":"One of the new Kickoff rules deals with formations:\n* Kicking team players must have 5 on either side of the ball, at least 2 between hashmarks and numbers, at least 2 outside numbers\n* Receiving team must have 8 players within the “setup zone” (15-yd zone between 10-25 yards from the ball)\nhttps://operations.nfl.com/the-rules/nfl-video-rulebook/kickoff/\n\nA FG rule also addresses defense formation:\n\n* No more than six Team B (defense) players may be on the line of scrimmage on either side of the snapper at the snap.\nhttps://operations.nfl.com/the-rules/nfl-video-rulebook/illegal-formation/\n\n\nI will use role and side data to determine if particular formations lead to more concussions. In punting terms, limiting the team to 2 outside the numbers would be equivalent of limiting a team to 2 Jammers.\nThe 8 players in the setup zone would also eliminate front blockers and more than 2 gunners. \n\nFor line balance, both FG's and Kickoffs restrict the number of players that can be on one side of the ball."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.591982Z","start_time":"2019-01-09T00:14:04.101509Z"},"trusted":true,"_uuid":"2a7697d9c3c41f8d8878a321b499d21e0b51def7","_kg_hide-input":true},"cell_type":"code","source":"#File I created that adds team (coverage vs. return), side (left, right, center)\n# and Group (Gunner, Jammer, Dline, Wing)\nexpanded_role = pd.read_csv('../input/extradata/Expanded_Roles.csv')\n\n#Players rarely change positions but they frequently change numbers (pre-season vs regular season, 2016 vs 2017)\nplayer_jersey = player_jersey.groupby('GSISID').agg({'Number': ', '.join, \n 'Position': 'first' }).reset_index()\n\n#Merge 3 DFs into one DF\nplayer = pd.merge(player_role,player_jersey,on='GSISID')\nplayer = pd.merge(player,expanded_role,on=\"Role\")\n\n#Formation - How players are lined up\nRoleGroup = player.groupby(['Season_Year','GameKey','PlayID','Cov_Ret','Group']).agg({'Role': 'count'}).reset_index()\nRole = RoleGroup.sort_values(['Season_Year','GameKey','PlayID','Role'])\nRole['RoleCount'] = Role.Group + '-' + Role.Role.map(str)\nRole = Role.groupby(['Season_Year','GameKey','PlayID','Cov_Ret']).agg({'RoleCount': ', '.join}).reset_index()\nRole.columns = ['Season_Year','GameKey','PlayID','Cov_Ret','Formation']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9d33753df347c8c423664bc95dd63bd2838e7fb"},"cell_type":"markdown","source":"### Player Roles\nLet's start with looking for connections between player roles and concussions"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.645033Z","start_time":"2019-01-09T00:14:04.593984Z"},"trusted":true,"_uuid":"49cb7bc453298f83b6763457da2f0daba58f8be0","_kg_hide-input":true},"cell_type":"code","source":"#Add Concussion column to Video to help separate concussion plays\nvideo['Concussion']='YES'\nRole_Con = pd.merge(Role,video,on=['Season_Year','GameKey','PlayID'],how='left')\nRole_Con.Concussion = Role_Con.Concussion.fillna('NO')\n\n#Group by formation. Separate formations by coverage and return teams\nRole_Agg = Role_Con.groupby(['Cov_Ret','Formation','Concussion'])['Season_Year'].agg('count').reset_index()\n\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nRole_Yes = Role_Agg[Role_Agg.Concussion=='YES']\nRole_No = Role_Agg[Role_Agg.Concussion=='NO']\n\nRole_Percent=pd.merge(Role_Yes,Role_No,on=['Cov_Ret','Formation'],suffixes=['_Yes','_No'],how='left')\n\n#The second calculation is a little non-intuitive. I could have hard coded, but I \n# wanted the code to work if the data changed\n# Len(Role_Con) = 2*plays because each play has a coverage formation and return formation\nRole_Percent['Season_Year_Yes_Percent'] = Role_Percent.Season_Year_Yes/len(video)\nRole_Percent.Season_Year_No = Role_Percent.Season_Year_No/(len(Role_Con)/2-len(video))\n\n#High Number indicate bad formations\nRole_Percent['Ratio'] = Role_Percent.Season_Year_Yes_Percent/Role_Percent.Season_Year_No\n\n#Drop unnecessary columns and clean up names\nRole_Percent = Role_Percent[['Cov_Ret','Formation','Season_Year_Yes','Season_Year_No','Season_Year_Yes_Percent','Ratio']]\nRole_Percent.columns = ['Cov_Ret','Formation','Concussion_Count','Not_Concussion_Percent','Concussion_Percent','Ratio']\ndisplay(Role_Percent)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b89cde9e2e34b4251d7ce26517094a6ce7f1795"},"cell_type":"markdown","source":"The coverage team had the same formation for all concussions. This formation is also the most popular formation by far. From this point, I will focus only on the return teams' formation.\nThe return team had a wide variety of formations, but there were a few positions that seemed imporant."},{"metadata":{"ExecuteTime":{"end_time":"2018-12-27T16:48:44.061873Z","start_time":"2018-12-27T16:48:44.048860Z"},"_uuid":"2ae9edd663cddedbe94d8fa52dc423dc040635f1"},"cell_type":"markdown","source":"### Jammers & Front Blockers\nJammers look like they are important. More than 2 appear to cause problems. \n\nFront Blockers look like they might be a problem as well, but the numbers are small (3). Teams rarely use a front blocker. If any rules would increase the use of front blockers, that might be a problem. At the very least, such a rule would need to be evaluated in preseason to determine if it inadvertantly increased concussions.\n\nLet us rerun the analysis focused only on the returning team positions and looking at number in those Roles. I ignore returner because it was 1 in all cases. (theoretically, a team could use two punt returners, but I did not see that formation in the concussion data) "},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.712098Z","start_time":"2019-01-09T00:14:04.647035Z"},"trusted":true,"_uuid":"f9ac01f0ff67e6d3f2db90cc3c808fefb005502b","_kg_hide-input":true},"cell_type":"code","source":"#Fortunately I already have that data. In hindsight, I should have looked at this first. \nRoleGroup_Ret = RoleGroup[(RoleGroup.Cov_Ret=='Return') & (RoleGroup.Group != 'Returner')]\n\n#Very similar to above, but with a few tweaks\nRole_Con = pd.merge(RoleGroup_Ret,video,on=['Season_Year','GameKey','PlayID'],how='left')\nRole_Con.Concussion = Role_Con.Concussion.fillna('NO')\n\n#Group by formation. Separate formations by coverage and return teams\nRole_Agg = Role_Con.groupby(['Group','Role','Concussion'])['Season_Year'].agg('count').reset_index()\nRole_Agg.columns = ['Group','Role','Concussion','Count']\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nRole_Yes = Role_Agg[Role_Agg.Concussion=='YES']\nRole_No = Role_Agg[Role_Agg.Concussion=='NO']\n\nRole_Percent=pd.merge(Role_Yes,Role_No,on=['Group','Role'],suffixes=['_Yes','_No'],how='left')\nRole_Percent['Count_Yes_Percent'] = Role_Percent.Count_Yes/len(video)\nRole_Percent['Count_No_Percent'] = Role_Percent.Count_No/(len(play_information)-len(video))\n\n#High Number indicate bad formations\nRole_Percent['Ratio'] = Role_Percent.Count_Yes_Percent/Role_Percent.Count_No_Percent\nRole_Percent=Role_Percent.rename(columns = {'Role':'Players'})\ndisplay(Role_Percent[['Group','Players','Count_Yes','Count_No','Ratio']])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98b43c7456cfdac8d14d635f324006b1df0d6454"},"cell_type":"markdown","source":"Ratio compares the % of plays in the concussion dataset to the % of plays in the overall dataset. Jammer and Front Block have the most interesting data. A low number of defensive lineman and linebackers would be associated with both additional jammers and using a front blocker. From here on out, I will focus on Jammers."},{"metadata":{"_uuid":"4b25e8ab1d206c1d243a911fdf5b7dc1047e28d5"},"cell_type":"markdown","source":"### Side Data \nIn this section, I'll compare balanced lines (equal number of players on left and right). \nGoing to take Side Data and do Left-Right to get an idea of overloaded/lopsided lines\n* 0 = perfectly balanced\n* +1 more player on left\n* -1 = one more player on right\n\nThis will also remove the problem with center players. A player in the center would be scored a 0 and not effect the balance."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.791174Z","start_time":"2019-01-09T00:14:04.714100Z"},"trusted":true,"_uuid":"c7c8f1827e8849f8a775a52efd714e35a7c2bb20","_kg_hide-input":true},"cell_type":"code","source":"#Calculate how players are lined up\n#Based on above will only look at Dline and Linebacker\nSidePlayer = player[player.Group.isin(['DLine','Linebacker'])]\nSideGroup = SidePlayer.groupby(['Season_Year','GameKey','PlayID','Side']).agg({'Role': 'count'}).reset_index()\nSide_Left = SideGroup[SideGroup.Side=='Left']\nSide_Right = SideGroup[SideGroup.Side=='Right']\nSide_Con = pd.merge(Side_Left,Side_Right,on=['Season_Year','GameKey','PlayID'],how='inner',suffixes=['_Left','_Right'])\nSide_Con['Delta']= Side_Con.Role_Left-Side_Con.Role_Right\nSide_Con = Side_Con[['Season_Year','GameKey','PlayID','Delta']]\nSide_Con = pd.merge(Side_Con,video,on=['Season_Year','GameKey','PlayID'],how='left')\nSide_Con.Concussion = Side_Con.Concussion.fillna('NO')\n\n#Group by formation. Separate formations by coverage and return teams\nSide_Agg = Side_Con.groupby(['Delta','Concussion'])['Season_Year'].agg('count').reset_index()\nSide_Agg.columns = ['Delta','Concussion','Count']\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nSide_Yes = Side_Agg[Side_Agg.Concussion=='YES']\nSide_No = Side_Agg[Side_Agg.Concussion=='NO']\n\nSide_Percent=pd.merge(Side_Yes,Side_No,on=['Delta'],suffixes=['_Yes','_No'],how='left')\nSide_Percent['Count_Yes_Percent'] = Side_Percent.Count_Yes/len(video)\nSide_Percent['Count_No_Percent'] = Side_Percent.Count_No/(len(play_information)-len(video))\n\n#High Number indicate bad formations\nSide_Percent['Ratio'] = Side_Percent.Count_Yes_Percent/Side_Percent.Count_No_Percent\ndisplay(Side_Percent[['Delta','Count_Yes','Count_No','Ratio']])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39056bdf1459e4f4b66fc59054c6a968daf97b76"},"cell_type":"markdown","source":"* Breaking out by Delta is too granular. I will try Delta==0 (completely balanced) and Delta!=0 (not balanced)"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.818200Z","start_time":"2019-01-09T00:14:04.792175Z"},"trusted":true,"_uuid":"f4666fd2b589843f0e0e6c1d083ca06953284886","_kg_hide-input":true},"cell_type":"code","source":"Side_Con['Balanced']=np.where(Side_Con.Delta==0,True,False)\nSide_Agg = Side_Con.groupby(['Balanced','Concussion'])['Season_Year'].agg('count').reset_index()\nSide_Agg.columns = ['Balanced','Concussion','Count']\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nSide_Yes = Side_Agg[Side_Agg.Concussion=='YES']\nSide_No = Side_Agg[Side_Agg.Concussion=='NO']\n\nSide_Percent=pd.merge(Side_Yes,Side_No,on=['Balanced'],suffixes=['_Yes','_No'],how='left')\nSide_Percent['Count_Yes_Percent'] = Side_Percent.Count_Yes/len(video)\nSide_Percent['Count_No_Percent'] = Side_Percent.Count_No/(len(play_information)-len(video))\n\n#High Number indicate bad formations\nSide_Percent['Ratio'] = Side_Percent.Count_Yes_Percent/Side_Percent.Count_No_Percent\ndisplay(Side_Percent[['Balanced','Count_Yes','Count_No','Ratio']])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"950a0e2e9f3c4d34149a9e1ac877ec68ddfe39f7"},"cell_type":"markdown","source":"Balanced looks slightly worse. A rule forcing a non-balanced line would be very different than all other formation rules. \nWith such sparse data, this small difference is not significant."},{"metadata":{"_uuid":"0abd0ff1185c062f4c3dfe2900549fefb7e0f2e8"},"cell_type":"markdown","source":"## Game Data\nA rule could not be based on StadiumType, Turf, GameWeather, Temperature, OutdoorWeather. If a particular turf or weather causes more concussions, then there may be equipment or stadium designs needed."},{"metadata":{"_uuid":"dc7c90bdd1b9734975957a2092f71eca1a065064"},"cell_type":"markdown","source":"# Player Role Summary\nThe number of Jammers, the players who block gunners, appear to be important. Line imbalance does not."},{"metadata":{"ExecuteTime":{"end_time":"2018-12-27T18:19:22.443528Z","start_time":"2018-12-27T18:19:22.439525Z"},"_uuid":"01a24c455c016a07197e4ab486bd0e202a201f33"},"cell_type":"markdown","source":"# EDA - quick look at multiple small tables\nLet's take a quick look at some variables that might be important. I'll just do quick tables because the purpose of this analysis is to get a feel for the data. For numerical values, I'm using qcut with 5 bins. This approach bins the data using all 6000+ punts and then sees how the concussion data. If all the bins are about the same, then the variable is likely not important. If there are wide variations, then it will be worth exploring. "},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:04.898278Z","start_time":"2019-01-09T00:14:04.819201Z"},"trusted":true,"_uuid":"0437c89f956c4cf48bc7082c4a0a4592c3173de0","_kg_hide-input":true},"cell_type":"code","source":"#Combine the data to use\nconc_player = pd.merge(video_review,player, left_on=['Season_Year','GameKey','PlayID','GSISID'],\n right_on=['Season_Year','GameKey','PlayID','GSISID'],how='left')\nconc_player = pd.merge(conc_player,play_information,on=['Season_Year','GameKey','PlayID'] ,how='left')\n#conc_player.head()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:14:05.063437Z","start_time":"2019-01-09T00:14:04.899278Z"},"_uuid":"54883ae3cb49cdd1f966da8961658e9d76566682","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"display(conc_player.groupby('Player_Activity_Derived')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Evenly split between blocking and tackling')\n\ndisplay(conc_player.groupby('Primary_Impact_Type')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Most concussions involve hitting another helmet or body')\n\ndisplay(conc_player.groupby('Score_Diff_qcut')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('This is an unlikely variable, but I can get a sense of what random looks like for this dataset.')\n\ndisplay(conc_player.groupby('YardsforTD_qcut')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Very evenly split. Interesting that we can see that 80% of returns are <15 yards. Only 20% are \"big\" returns. ')\n\ndisplay(conc_player.groupby('NetYards_qcut')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Bad punt/punt coverage has slightly more concussions.')\n\ndisplay(conc_player.groupby('PuntYards_qcut')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('\"Outkicking the coverage\" is associated with long returns. The longest punts are about equivalent for concussions.')\n\ndisplay(conc_player.groupby('Group')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Offensive Line has by far the most concussions. Second is Wing, which is a very similar position. One factor to take into account is that there are 5 OL, 2 Wings, 2 gunners, and 1 Returner on each punt.')\n\ndisplay(conc_player.groupby('Position')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Comparing this table with the previous table, and the obvious question is \"What happened to the offensive line?\" Not a single offensive lineman received a concussion even though 14 players play o-line did. Looking into this in more detailed revealed that to improve punt coverage, players play out of position on the offensive line.')\n\ndisplay(conc_player.groupby('Cov_Ret')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('3x injuries for the coverage team. This finding was definitely unexpected. The difference is very large so should be significant even at these low numbers.')\n\ndisplay(conc_player.groupby('Side')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('No difference seen.')\n\ndisplay(conc_player.groupby('Season_Type')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('There are 4 preseason games and 16 regular season games. On a per game basis, preseason has twice the concussions of regular season.')\n\ndisplay(conc_player.groupby('Season_Year')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Despite the 2017 rule change, no obvious progress')\n\ndisplay(conc_player.groupby('Friendly_Fire')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Most concussions do not involve friendly fire')\n\ndisplay(conc_player.groupby('Penalty')['GSISID'].count().reset_index().rename(columns = {'GSISID':'count'}))\nprint('Over a quarter of concussions are on penalty plays. I will review the video to determine if the penalty and concussion are related. ')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5aad3302ec29a8b937f580dba07af69249037cb"},"cell_type":"markdown","source":"## EDA Summary\nThe quick EDA gave an idea of some variables to keep an eye on. I'll use the videos to validate the data and then revisit. \nThe ones that stood out to me are:\n* **season_type** - preseason has more injuries than regular season\n* **position_group** - Offensive lineman have the most concussions., While they play the offensive line, they are actually linebackers, cornerbacks, receivers, and running backs\n* **coverage team** - players covering punts are 3x more likely to get injured"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-06T20:02:24.294633Z","start_time":"2019-01-06T20:02:24.291630Z"},"_uuid":"a01e582c2ee4b662c79257105202fadcc54f21a9"},"cell_type":"markdown","source":"# Load and Clean NGS Data\nBefore reviewing the videos, I will use NGS data to create diagrams on the plays and identify the speed at impact.\n\n## Load NGS Data\nTo save space only loading plays with concussions. Could save more space by only loading GSISID in concussion but want to see what other players are doing during the play."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:17:04.700851Z","start_time":"2019-01-09T00:14:22.289066Z"},"trusted":true,"_uuid":"e29a79271e050b98981edd43df39765fd8c6ccb5","_kg_hide-input":true},"cell_type":"code","source":"NGS_Key = video[:][['Season_Year','GameKey','PlayID']]\n# gets all csv with NGS in their filename\npath = \"../input/NFL-Punt-Analytics-Competition/\"\nNGS_csvs = [path+file for file in os.listdir(path) if 'NGS' in file]\n\nNGS = pd.DataFrame() #initialize an empty dataframe\n\n# loop to csv then appends it to df\nfor path_csv in NGS_csvs:\n _df = pd.read_csv(path_csv,low_memory=False)\n _df = pd.merge(NGS_Key,_df,how='left', on=['Season_Year','GameKey','PlayID'])\n NGS = NGS.append(_df,ignore_index=True)\n del _df # deletes the _df to free up memory\nNGS = NGS.dropna(subset=['GSISID','x','y','dis','o','dir']) ","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-07T19:16:19.785548Z","start_time":"2019-01-07T19:16:19.560332Z"},"_uuid":"d31d76d941f5c0990147501cdf79fa35353d2137"},"cell_type":"markdown","source":"Note: 2 of the plays do not have NGS data. That will not have a major impact on the analysis\nThe NGS data does include a long timespan for each play. For analysis, I will narrow the time to the line_set to 2 seconds after the conclusion (tackle, out of bounds, fair catch, touchdown, downed punt)"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:17:15.775542Z","start_time":"2019-01-09T00:17:04.702853Z"},"trusted":true,"_uuid":"a247d922dbe58155141e9244216d8e14d9e1e7c2","_kg_hide-input":true},"cell_type":"code","source":"NGS['Time'] = pd.to_datetime(NGS.Time)\nNGS.sort_values(['Season_Year','GameKey','PlayID','Time'])\nNGS['Event'] = NGS.groupby(['Season_Year','GameKey','PlayID','Time'])['Event'].fillna(method='ffill')\nNGS['Event'] = NGS.groupby(['Season_Year','GameKey','PlayID','Time'])['Event'].fillna(method='bfill')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:22:53.980942Z","start_time":"2019-01-09T00:22:53.974936Z"},"trusted":true,"_uuid":"3467d1ef3bdb906e2f157647e6c57075cff0bcfd","_kg_hide-input":true},"cell_type":"code","source":"def probplot(df,groupCol,varCol):\n#if 1==1:\n #df = Tackle[:]\n #groupCol ='GameKey'\n #varCol='DIS_diff'\n g=df[[groupCol,varCol]].dropna()\n h=g.groupby(groupCol)[varCol].rank(pct=True,method='dense').reset_index()\n g=pd.merge(g,h,right_on='index',left_index=True,suffixes=('','_pct'))\n g=g.sort_values([groupCol,varCol])\n traces=[]\n for hue in sorted(g[groupCol].unique()):\n trace = go.Scatter(y=g[g[groupCol]==hue][varCol + '_pct'], x=g[g[groupCol]==hue][varCol],name=str(hue), showlegend=True)\n traces.append(trace)\n data = traces\n layout = go.Layout(\n autosize=False,\n width=500,\n height=500)\n fig = dict(data=data, layout=layout)\n #print(\"\\n\\n\\t\",play_description)\n #print(GameKey,PlayID)\n #print(URL)\n offline.iplot(fig)#, config=config) ","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:17:51.417950Z","start_time":"2019-01-09T00:17:15.784551Z"},"trusted":true,"_uuid":"a24bcc4bff227a3c2884fe056c4e88206e75bc87","_kg_hide-input":true},"cell_type":"code","source":"#Inner Join Player to remove players who were not on punt play and to remove players without NGS data\nTackle = pd.merge(NGS,player,on=['Season_Year','GameKey','PlayID','GSISID'],how='inner')\nTackle = pd.merge(NGS,video[['Season_Year','GameKey','PlayID','GSISID']], \n on=['Season_Year','GameKey','PlayID'],suffixes=['','_Conc'])\n\ndel NGS # deletes the _df to free up memory\n#Run for Debugging\n#GSISIDGroup = player.groupby(['GameKey','PlayID']).agg({'GSISID': 'count'}).reset_index()\n#Note: GameKey=89, PLayID=4662 has 1 GSISID GameKey=319, PlayID=3019 has 16 GSISID\n\n#Copy Concussed GSISID Data to a new column\nTackle['Conc_X'] = np.where(Tackle.GSISID==Tackle.GSISID_Conc,Tackle.x,np.nan)\nTackle['Conc_Y'] = np.where(Tackle.GSISID==Tackle.GSISID_Conc,Tackle.y,np.nan)\nTackle['Conc_Dis'] = np.where(Tackle.GSISID==Tackle.GSISID_Conc,Tackle.dis,np.nan)\nTackle['Conc_Dir'] = np.where(Tackle.GSISID==Tackle.GSISID_Conc,Tackle.dir,np.nan)\n\n#Is there a better way to do this???\n#Copy Conc_ position to all other positions.\n#Note: calling ffill and then bfill on same line did not act as I expected. Groupby only applied to first call\nTackle.Conc_X = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_X'].fillna(method='ffill')\nTackle.Conc_Y = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_Y'].fillna(method='ffill')\nTackle.Conc_X = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_X'].fillna(method='bfill')\nTackle.Conc_Y = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_Y'].fillna(method='bfill')\nTackle.Conc_Dis = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_Dis'].fillna(method='ffill')\nTackle.Conc_Dir = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_Dir'].fillna(method='ffill')\nTackle.Conc_Dis = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_Dis'].fillna(method='bfill')\nTackle.Conc_Dir = Tackle.groupby(['Season_Year','GameKey','PlayID','Time'])['Conc_Dir'].fillna(method='bfill')\n\n#How Far is the player from the Conc_player (Conc_player will always be 0)\nTackle['Distance']= ((Tackle.x-Tackle.Conc_X)**2+(Tackle.y-Tackle.Conc_Y)**2)**0.5\n\n#Get DF in right order\n\n\nTackle.sort_values(['Season_Year','GameKey','PlayID','GSISID','Time'],inplace=True)\nTackle['Speed'] = Tackle.groupby(['Season_Year','GameKey','PlayID','GSISID'])['Distance'].diff(1)\nTackle['Acceleration'] = Tackle.groupby(['Season_Year','GameKey','PlayID','GSISID'])['Speed'].diff(1)\n\n\n#Calculate beginning of each play\nminSeconds = Tackle.groupby(['Season_Year','GameKey','PlayID'])['Time'].min().reset_index()\nminSeconds.columns = ['Season_Year','GameKey','PlayID','PlayStart']\n\n#Merge with Tackle DF and calculate seconds for each play\nTackle = pd.merge(Tackle,minSeconds,on=['Season_Year','GameKey','PlayID'],how='left')\nTackle['seconds'] = (Tackle['Time']-Tackle.PlayStart).dt.total_seconds()\n\n#Sort again\nTackle.sort_values(['Season_Year','GameKey','PlayID','GSISID','seconds'],inplace=True)\n\n#delta seconds & shift_seconds are used to address gaps in NGS data\n#If there is a gap in the previous 2 records, then acceleration will be wrong.\nTackle['delta_seconds'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['seconds'].diff(1)\nTackle['shift_seconds'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['delta_seconds'].shift(1)\n\n\n#Remove bad data rows\nTackle['Good_time'] = np.where((Tackle['delta_seconds']>0.15)|(Tackle['shift_seconds']>0.15),False,True)\n\nTackle['Conc_DIS_diff'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['Conc_Dis'].diff(1)\nTackle['Conc_DIS_diffA'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['Conc_DIS_diff'].shift(1)\nTackle['Conc_DIS_diffB'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['Conc_DIS_diff'].shift(-1)\nTackle['DIS_diff'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['dis'].diff(1)\nTackle['DIS_diffA'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['DIS_diff'].shift(1)\nTackle['DIS_diffB'] = Tackle.groupby(['GameKey','PlayID','GSISID'])['DIS_diff'].shift(-1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67538ae98433a58ad63ac5164e7160d8535fc580"},"cell_type":"markdown","source":"During video analysis, I noted that players appeared to move quickly when they were under piles of other players. To fix this problem, I looked at the change in speed and removed outliers. To do that I used a probplot.\nSince many people are not famiilar with a probplot or qqplot, I'll give a brief overview. You can imagine you take a histogram and pull the right lower corner to the right upper corner. If the distribution is normal, the data will make a perfectly straight line. This is a great tool to compare multiple distributions and to spot outliers. For more info, see https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"21dacb70915af14d9e85821db35321c3a54f9ca8"},"cell_type":"code","source":"Image(\"../input/images2/threegraphs.png\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40f4c03676dfa10b6269a53191d98408c758aa6e"},"cell_type":"markdown","source":"The above image shows the same exact data in 3 different graphs -- probplot, stacked histogram, and boxplot. As can be seen, the prob plot provides more detail than the other two and is able to show many more groups without losing information.\nAs an added bonus, it also shows if the data is normally distributed. The dummy data was randomly generated and that is obvious only from the prob plot."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:01.611790Z","start_time":"2019-01-09T00:17:51.419952Z"},"trusted":true,"_uuid":"a81cf34631f4a9389c34c249cc999728981e46e9","_kg_hide-input":true},"cell_type":"code","source":"probplot(Tackle,'GameKey','DIS_diff')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43a1abfde6a4968d80f9ad65cfe04cec91eff1d6"},"cell_type":"markdown","source":"From the probplot it is obvious that there are severe outliers. I drew the limits at -0,2 and 0.2"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:01.759933Z","start_time":"2019-01-09T00:18:01.613792Z"},"trusted":true,"_uuid":"f57faaf9d2815d0bc7bf5e1f7d45fe233addaac5","_kg_hide-input":true},"cell_type":"code","source":"\nTackle['Good_dis'] = np.where((Tackle.DIS_diff<-0.2) | (Tackle.DIS_diff>0.2) | \n (Tackle.DIS_diffA<-0.2) | (Tackle.DIS_diffA>0.2) | \n (Tackle.DIS_diffB<-0.2) | (Tackle.DIS_diffB>0.2) |\n (Tackle.Conc_DIS_diff<-0.2) | (Tackle.Conc_DIS_diff>0.2) |\n (Tackle.Conc_DIS_diffA<-0.2) | (Tackle.Conc_DIS_diffA>0.2) | \n (Tackle.Conc_DIS_diffB<-0.2) | (Tackle.Conc_DIS_diffB>0.2), False, True)\nTackle = Tackle[Tackle.Good_time & Tackle.Good_dis]\nTackle['Shift_Speed'] = Tackle['Speed'].shift(-1)\nTackle['Hit_Speed'] = np.where(((Tackle['Shift_Speed']/Tackle['Speed']<0) &\n (Tackle['Distance']<5)),abs(Tackle['Speed']),0)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:07.513488Z","start_time":"2019-01-09T00:18:01.760935Z"},"trusted":true,"_uuid":"b65a826a9141606953c28ad8bdef41bdabee20a3","_kg_hide-input":true},"cell_type":"code","source":"probplot(Tackle,'GameKey','DIS_diff')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7106ed2b92a767eafb597f5fb63ac260d60a0c94"},"cell_type":"markdown","source":"After removing the outliers, the prob plots look much cleaner"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:07.638609Z","start_time":"2019-01-09T00:18:07.515490Z"},"trusted":true,"_uuid":"43b23abca030fa8b2721df2b43d9c70ce2912c0c","_kg_hide-input":true},"cell_type":"code","source":"\n#Add Partner where available\nconc_player.Primary_Partner_GSISID = pd.to_numeric(conc_player.Primary_Partner_GSISID,errors='coerce') \nconc_pair = pd.merge(conc_player,player,how='left', left_on=['Season_Year','GameKey','PlayID','Primary_Partner_GSISID'],\n right_on=['Season_Year','GameKey','PlayID','GSISID'], suffixes=('','_Partner'))\n\n#Add in Jammers based on EDA\nRole_Add = Role_Con[Role_Con.Group=='Jammer'][['Season_Year','GameKey','PlayID','Role']].rename(columns = {'Role':'Jammers'})\nconc_pair = pd.merge(conc_pair,Role_Add, on=['Season_Year','GameKey','PlayID'], how='left')\n\n#Add in Balance based on EDA\nconc_pair = pd.merge(conc_pair,Side_Con[['Season_Year','GameKey','PlayID','Balanced']], on=['Season_Year','GameKey','PlayID'], how='left')\n\nconc_pair = pd.merge(conc_pair,video[['Season_Year','GameKey','PlayID','URL']], on=['Season_Year','GameKey','PlayID'], how='left')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf457d9cb3c33f6337dc856926649a838e39dc32"},"cell_type":"markdown","source":"# Videos and Graphs\n\n## Graph Layout\nUse the layout from the starter kernal because it is nice.\n"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:07.958918Z","start_time":"2019-01-09T00:18:07.640611Z"},"_kg_hide-input":true,"_kg_hide-output":false,"_uuid":"467bebb25381ec19657962d0b617633acb076eec","hide_input":true,"trusted":true},"cell_type":"code","source":"def load_layout():\n \"\"\"\n Returns a dict for a Football themed Plot.ly layout \n \"\"\"\n layout = dict(\n title = \"Player Activity\",\n plot_bgcolor='darkseagreen',\n showlegend=False,\n width=640,\n height=400,\n margin = dict(t=1),\n xaxis=dict(\n autorange=False,\n range=[0, 120],\n showgrid=False,\n zeroline=False,\n showline=True,\n linecolor='black',\n linewidth=1,\n mirror=True,\n ticks='',\n tickmode='array',\n tickvals=[10,20, 30, 40, 50, 60, 70, 80, 90, 100, 110],\n ticktext=['Goal', 10, 20, 30, 40, 50, 40, 30, 20, 10, 'Goal'],\n showticklabels=True\n ),\n yaxis=dict(\n title='',\n autorange=False,\n range=[-3.3,56.3],\n showgrid=False,\n zeroline=False,\n showline=True,\n linecolor='black',\n linewidth=1,\n mirror=True,\n ticks='',\n showticklabels=False\n ),\n shapes=[\n dict(\n type='line',\n layer='below',\n x0=0,\n y0=0,\n x1=120,\n y1=0,\n line=dict(\n color='white',\n width=2\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=0,\n y0=53.3,\n x1=120,\n y1=53.3,\n line=dict(\n color='white',\n width=2\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=10,\n y0=0,\n x1=10,\n y1=53.3,\n line=dict(\n color='white',\n width=10\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=20,\n y0=0,\n x1=20,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=30,\n y0=0,\n x1=30,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=40,\n y0=0,\n x1=40,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=50,\n y0=0,\n x1=50,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=60,\n y0=0,\n x1=60,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),dict(\n type='line',\n layer='below',\n x0=70,\n y0=0,\n x1=70,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),dict(\n type='line',\n layer='below',\n x0=80,\n y0=0,\n x1=80,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=90,\n y0=0,\n x1=90,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),dict(\n type='line',\n layer='below',\n x0=100,\n y0=0,\n x1=100,\n y1=53.3,\n line=dict(\n color='white'\n )\n ),\n dict(\n type='line',\n layer='below',\n x0=110,\n y0=0,\n x1=110,\n y1=53.3,\n line=dict(\n color='white',\n width=10\n )\n )\n ]\n )\n return layout\n\nend_where = ((Tackle.Event=='tackle') | \n (Tackle.Event=='punt_downed') |\n (Tackle.Event=='out_of_bounds') |\n (Tackle.Event=='touchdown') |\n (Tackle.Event=='fair_catch'))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:08.067022Z","start_time":"2019-01-09T00:18:08.042999Z"},"_kg_hide-input":true,"_kg_hide-output":false,"_uuid":"cc8dc7fbe48e5e0b6a1838780678dac1682e87cd","trusted":true},"cell_type":"code","source":"# Loading and plotting functions\ndef plot_play(play_number):\n#play_number = 1\n#if play_number!=-1: \n \"\"\"\n Plots player movements on the field for a given game, play, and two players\n \"\"\"\n #print (Primary.loc[play_number])#[['Season_Year','Season_Type','Week','GameKey','PlayID'])\n GSISID = conc_pair.loc[play_number]['GSISID']\n GameKey = conc_pair.loc[play_number]['GameKey']\n PlayID = conc_pair.loc[play_number]['PlayID']\n play_description = conc_pair.iloc[play_number][\"PlayDescription\"]\n URL = conc_pair.iloc[play_number][\"URL\"]\n Second = conc_pair.loc[play_number]['GSISID_Partner']\n \n print ('Season_Type:', conc_pair.iloc[play_number]['Season_Type'],\n ' Season_Year:', conc_pair.iloc[play_number]['Season_Year'],\n ' GameKey:', conc_pair.iloc[play_number]['GameKey'],\n ' PlayID:', conc_pair.iloc[play_number]['PlayID'],\n ' Jammers:', conc_pair.iloc[play_number]['Jammers'],\n ' Balanced:', conc_pair.iloc[play_number]['Balanced'], \n ' Penalty:', conc_pair.iloc[play_number]['Penalty'])\n display ()\n print (\"GSISID:\", GSISID,\n \" Role:\", conc_pair.iloc[play_number]['Group'],\n \" Number:\", conc_pair.iloc[play_number]['Number'],\n \" Position:\", conc_pair.iloc[play_number]['Position'],\n \" Activity:\", conc_pair.iloc[play_number]['Player_Activity_Derived'])\n if pd.isna(Second)==False:\n display()\n print (\"Partner GSISID:\", Second,\n \" Role:\", conc_pair.iloc[play_number]['Group_Partner'],\n \" Number:\", conc_pair.iloc[play_number]['Number_Partner'],\n \" Position:\", conc_pair.iloc[play_number]['Position_Partner'],\n \" Activity:\", conc_pair.iloc[play_number]['Primary_Partner_Activity_Derived'],\n \" Friendly Fire:\", conc_pair.iloc[play_number]['Friendly_Fire'])\n display (play_description)\n\n game_df = Tackle[(Tackle.PlayID==PlayID) & (Tackle.GameKey==GameKey)].sort_values(\"Time\")\n playstart = game_df[game_df.Event == \"line_set\"][\"Time\"].min()\n\n end_where = ((game_df.Event=='tackle') | \n (game_df.Event=='punt_downed') |\n (game_df.Event=='out_of_bounds') |\n (game_df.Event=='touchdown') |\n (game_df.Event=='fair_catch'))\n \n playend = game_df[end_where][\"Time\"].min() + pd.to_timedelta(2, unit='s')\n \n game_df = game_df[(game_df.Time > playstart) & (game_df.Time < playend)]\n \n \n if len(game_df)==0:\n game_df = Tackle[(Tackle.PlayID==PlayID) & (Tackle.GameKey==GameKey)].sort_values(\"Time\")\n #return URL\n #GameKey=str(pd.unique(game_df.GameKey)[0])\n #HomeTeam = pd.unique(game_df.Home_Team_Visit_Team)[0].split(\"-\")[0]\n #VisitingTeam = pd.unique(game_df.Home_Team_Visit_Team)[0].split(\"-\")[1]\n #YardLine = game_df[(game_df.PlayID==PlayID) & (game_df.GSISID==player1)]['YardLine'].iloc[0]\n \n traces=[] \n game_df['Delta'] = game_df.Time - game_df.Time.min()\n game_df.Delta = game_df.Delta.dt.total_seconds() \n \n playerid = int(GSISID)\n playernumber = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Number'].values[0]\n playerGroup = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Group'].values[0]\n trace = go.Scatter(\n x = game_df[game_df.GSISID==playerid].x,\n y = game_df[game_df.GSISID==playerid].y,\n name ='Position: '+str(playerGroup) + ' Number: '+str(playernumber),\n mode='markers',\n marker = dict(\n size = np.minimum(game_df[game_df.GSISID==playerid].Delta+6,10),\n color = 'rgba(255,255,0, .8)'))\n traces.append(trace)\n \n #Partner\n if pd.isna(Second)==False:\n playerid = int(Second)\n playernumber = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Number'].values[0]\n playerGroup = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Group'].values[0]\n trace = go.Scatter(\n x = game_df[game_df.GSISID==playerid].x,\n y = game_df[game_df.GSISID==playerid].y,\n name ='Position: '+str(playerGroup) + ' Number: '+str(playernumber),\n mode='markers',\n marker = dict(\n size = np.minimum(game_df[game_df.GSISID==playerid].Delta+6,10),\n color = 'rgba(0,255,0, .8)'))\n traces.append(trace)\n \n #get coverage \n for playerid in pd.unique(player[(player.PlayID==PlayID) & (player.GameKey==GameKey) & \n (player.Cov_Ret=='Coverage')]['GSISID']):\n playerid = int(playerid)\n playernumber = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Number'].values[0]\n playerGroup = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Group'].values[0]\n trace = go.Scatter(\n x = game_df[game_df.GSISID==playerid].x,\n y = game_df[game_df.GSISID==playerid].y,\n name ='Position: '+str(playerGroup) + ' Number: '+str(playernumber),\n mode='markers',\n marker = dict(\n size = 2,\n color = 'rgba(0, 0, 255, .8)'))\n traces.append(trace)\n \n #get receivers \n for playerid in pd.unique(player[(player.PlayID==PlayID) & (player.GameKey==GameKey) & \n (player.Cov_Ret=='Return')]['GSISID']):\n playerid = int(playerid)\n trace = go.Scatter(\n x = game_df[game_df.GSISID==playerid].x,\n y = game_df[game_df.GSISID==playerid].y,\n name ='Position: '+str(playerGroup) + ' Number: '+str(playernumber),\n mode='markers',\n marker = dict(\n size = 2,\n color = 'rgba(255,0,0, .8)'))\n traces.append(trace)\n\n \n layout = load_layout()\n # layout['title'] = HomeTeam + \\\n # ' vs. ' + VisitingTeam + \\\n # ' Possession: ' + \\\n # YardLine.split(\" \")[0] +'@'+YardLine.split(\" \")[1]\n data = traces\n fig = dict(data=data, layout=layout)\n #print(\"\\n\\n\\t\",play_description)\n #print(GameKey,PlayID)\n #print(URL)\n offline.iplot(fig, config=config)\n \n return URL\n #HTML('')\n","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:08.080037Z","start_time":"2019-01-09T00:18:08.068023Z"},"trusted":true,"_uuid":"a648aed8821a053f24c8e172a4dff005cf9b08b1","_kg_hide-input":true},"cell_type":"code","source":"# Loading and plotting functions\ndef plot_acceleration(play_number):\n#play_number=16\n#if play_number!=-1:\n GSISID = conc_pair.loc[play_number]['GSISID']\n GameKey = conc_pair.loc[play_number]['GameKey']\n PlayID = conc_pair.loc[play_number]['PlayID'] \n \n game_df = Tackle[(Tackle.PlayID==PlayID) & (Tackle.GameKey==GameKey)].sort_values(\"Time\")\n playstart = game_df[game_df.Event == \"line_set\"][\"Time\"].min()\n\n end_where = ((game_df.Event=='tackle') | \n (game_df.Event=='punt_downed') |\n (game_df.Event=='out_of_bounds') |\n (game_df.Event=='touchdown') |\n (game_df.Event=='fair_catch'))\n \n playend = game_df[end_where][\"Time\"].min() + pd.to_timedelta(2, unit='s')\n \n game_df = game_df[(game_df.Time > playstart) & (game_df.Time < playend)] \n \n if len(game_df)==0:\n game_df = Tackle[(Tackle.PlayID==PlayID) & (Tackle.GameKey==GameKey)].sort_values(\"Time\")\n game_df['Delta'] = game_df.Time - game_df.Time.min()\n game_df.Delta = game_df.Delta.dt.total_seconds()\n Fast_Hits = game_df.groupby('GSISID')['Hit_Speed'].max().reset_index()\n Fast_Hits = Fast_Hits[Fast_Hits.Hit_Speed> 0.1]\n if len(Fast_Hits)==0:\n print(\"No Fast Hits\")\n return\n traces=[] \n for playerid in pd.unique(Fast_Hits.GSISID):\n playerid = int(playerid)\n playernumber = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Number'].values[0]\n playerGroup = player[(player.GSISID==playerid)&(player.PlayID==PlayID) & \n (player.GameKey==GameKey)]['Group'].values[0]\n\n trace = go.Scatter(\n x = game_df[game_df.GSISID==playerid].Delta,\n y = game_df[game_df.GSISID==playerid].Hit_Speed*36000/1760,\n name ='Position: '+str(playerGroup) + ' Number: '+str(playernumber),\n mode='lines+markers',\n marker = dict(\n size = 10\n #,color = 'rgba(152, 0, 0, .8)'\n ))\n traces.append(trace)\n layout = go.Layout(\n title='Hit Speed vs Time',\n width=800,\n height=350,\n showlegend=True,\n yaxis=dict(\n title='Hit Speed (MPH)'\n )\n )\n fig = go.Figure(data=traces,layout=layout)\n offline.iplot(fig)#, config=config)\n return\n \n #HTML('')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d79509af09cea75ab88a539eeb07fbc94541e3c6"},"cell_type":"markdown","source":"## Main Graph\nThe code below was integral to my video analyis. This view included all the information I needed to review each of the videos. It shows the time, speed, location, and participants for each concussion. It also shows where everyone else on the field was during the play.\n\nBelow is an example of a play. \n\n### Text\nThe first 4 lines of text give all the key information about the play. Included is the variables I flagged from the EDA above.\n\n### Play Diagram\nThe concussed player is identified in yellow. The partner in green. Red indicates coverage players, blue indicates punters. Each dot is 0.1 seconds apart so dense lines indicate slowness and dotted lines indicate high speed.\n\n### Impact Graph\nFor the concussed player, the distance to all other players was calculated use x,y data. The change in that distance was then used as speed. The speed when distance crossed 0 was then used as the impact speed. One side effect of this approach was that near misses and collisions were both recorded. \n\nDistance Between Players: $ \\sqrt (X_1 - X_2)^2 + (Y_1-Y_2)^2) $\n\nSpeed: $ (D_{t=2}-D_{t=1})/0.1 $\n\nImpact Speed: Speed when Distance = 0\n\nThe speed was converted to miles per hour to give a common frame of reference to the speed. As a quick refresher, 6 MPH is a 10 minute mile or a light jog. 10 MPH is a run or 6 minute mile. 15 MPH and above would be a sprint. Because this speed measures the difference between two elite atheletes, there are values of 30 MPH or above. That would occur when two players were running at full speed directly at each other. \n\nThe time on the x-axis is seconds and 0 is equal to line_set in the NGS data. I used that value because aligned very closely with most of the videos.\n\n### Video\nThe URL and the video are provided so that video can be watched either in the notebook or on a second monitor.\n\n### Running the cell\nEach of the 37 plays could be called by giving the row number. I did this to make sure, I did not overlook any play. In addition a for loop could then run all 37 plays. For submission, I did not run the loop because I did not want to overwhelm reviewers. "},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:18:08.587525Z","start_time":"2019-01-09T00:18:08.082037Z"},"_uuid":"f61881d037ef67430a79ce0edf58bbf3cc925878","scrolled":false,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#for play_no in range(0,36):\nplay_no=12\nURL=plot_play(play_no)\nplot_acceleration(play_no)\nprint(URL)\nHTML(\"\"\"\"\"\".format(URL))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-08T13:32:08.792549Z","start_time":"2019-01-08T13:32:08.789546Z"},"_uuid":"0ae8136aa62d4edfba7d5a307692dd71d12e8e4c"},"cell_type":"markdown","source":"### Speed vs. concussions"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:21:21.450976Z","start_time":"2019-01-09T00:21:21.437964Z"},"trusted":true,"_uuid":"9194f795fa233ee42807fb5e78560358e6b14fef","_kg_hide-input":true},"cell_type":"code","source":"SpeedChart = pd.merge(Tackle[(Tackle.Hit_Speed>0.4)],conc_pair[['Season_Year','GameKey','PlayID','Primary_Partner_GSISID']],\n how='left',left_on=['Season_Year','GameKey','PlayID','GSISID'],\n right_on=['Season_Year','GameKey','PlayID','Primary_Partner_GSISID'])\nSpeedChart['Partner'] = np.where(pd.isna(SpeedChart.Primary_Partner_GSISID),'NotPartner','Partner')\nSpeedChart['Hit_Speed'] = SpeedChart['Hit_Speed']*36000/1760 ","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:58.531318Z","start_time":"2019-01-09T00:25:58.510297Z"},"trusted":true,"_uuid":"b5f8bfbbc180b07162d16736f3582ff3cf291810","_kg_hide-input":true},"cell_type":"code","source":"probplot(SpeedChart,'Partner','Hit_Speed')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18bdff84710ab21e2c7eae64b0e211ef0870fc6b"},"cell_type":"markdown","source":"The probplot show that speed of collisions is not a key differentiator between concussions and not concussions. Collisions that caused concussions and those that did not had the same range of speeds. I would have expected that speed would be an important variable. After looking at the videos with the hit speed chart, I found that even relatively low speed hits can cause concussions. The most consistent problem I found was group tackles. With multiple bodies coliding, one person would get hit in the head. \n\n###Video Review Key Summary\n There were many small groups of reasons for concussions. \n 1. Concussions are covered by current rules,\n 2. Concussions not during a punt return. (Fakes and before the punt return)\n 3. Group Tackles\n 4. Secondary collision. One player would be blocked into another.\n 5. Slipping\n 6. Punt muffs\n 7. Short punts. The returner would catch the ball on the run and be immediately hit.\n \n** Notably, speed of collision and blindside blocks were not a problem. ** \n \n For futher analysis, I grouped the data into three groups of about equal size.\n * Exclude from further analysis - already covered and not punt returns\n * Group Tackles\n * Everything else"},{"metadata":{"_uuid":"fea50f86da885bebc2aa702680c6e8999e84fdd7"},"cell_type":"markdown","source":"# Punt return concussions not covered by existing rules\n* The 2018 rulebook was used to determine if a penalty should have been called. Most of the violations were flagged by the onfield officials and I deferred to their judgment. There is a very fine distinction between a legal block and an illegal block. I was conservative and added calls that were obvious and preferrably were not in the official rules at the time. On one play, I saw what appeared to be a missed illegal block. That block happened way before the punt returner received the ball. Because that one play violated both of my conditions, I felt comfortable removing it.\n\n* 5 other plays were removed because the concussion did not occur on a punt return. One was on a fake. Three were before the punt. One is the punt mentioned above.\n"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.283841Z","start_time":"2019-01-09T00:25:36.257816Z"},"trusted":true,"_uuid":"53687fbe6c636aa4d6481b634e3fe68bb0a4a416","_kg_hide-input":true},"cell_type":"code","source":"bad = pd.read_csv('../input/extradata/Existing_Penalties.csv')\ndisplay(bad)\nExplained = pd.merge(bad,conc_player,on=['Season_Year','GameKey','PlayID'],how='right')\nprint('Table shows specific reason each play was excluded from further analysis')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.289847Z","start_time":"2019-01-09T00:25:36.284842Z"},"trusted":true,"_uuid":"e41e28e83beb858c862d4ff38ef4eebc8b84e48f","_kg_hide-input":true},"cell_type":"code","source":"Explained=Explained[pd.isna(Explained.Reason)]","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.294852Z","start_time":"2019-01-09T00:25:36.291849Z"},"trusted":true,"_uuid":"66039fdce01643c475193612d2aa01a01de5a47e","_kg_hide-input":true},"cell_type":"code","source":"def Display_Explained(strColumn):\n#if 1==1:\n #strColumn = 'Player_Activity_Derived'\n clean = Explained.groupby(strColumn)['GSISID'].count().reset_index().rename(columns = {'GSISID':'Clean'})\n data = conc_player.groupby(strColumn)['GSISID'].count().reset_index().rename(columns = {'GSISID':'Raw'})\n merged = pd.merge(clean,data)\n merged['Removed']=merged.Clean-merged.Raw\n display(merged)\n ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a43c83e62280bec8db3cfcc3d3f8d83433ccc2b"},"cell_type":"markdown","source":"The analysis above is repeated with the cleaned dataset. After cleaning the data, are there different patterns?"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.416970Z","start_time":"2019-01-09T00:25:36.296854Z"},"scrolled":false,"trusted":true,"_uuid":"bba06465f57a3f72ac07740a80b263c394988889","_kg_hide-input":true},"cell_type":"code","source":"Display_Explained('Player_Activity_Derived')\nprint('Tackling is the biggest remaining issue. Looking at videos, the main problem was multiple players tackling. This situation caused players heads to bump into each other. I can not think of a rule that would prevent group tackles and maintain the integrity of the game.')\n\nDisplay_Explained('Primary_Impact_Type')\nprint('Helmet-to-helmet hits were the most reduced by rules changes')\n\nDisplay_Explained('YardsforTD_qcut')\nprint('Almost all concussions occur when the punter is punting for distance')\n\nDisplay_Explained('Cov_Ret')\nprint('The initial analysis of 3x injuries for the coverage team is unchanged.')\n\nDisplay_Explained('Season_Type')\nprint('There problem with preseason games appears even worse after cleaning the data. Remember there are 4x more regular season games than preseason games')\n\nDisplay_Explained('Season_Year')\nprint('2017 now looks much worse than 2016 but the numbers are so low the difference is not meaningful')\n\nDisplay_Explained('Penalty')\nprint('5 concussions occurred on plays that were nullified for other penalties.')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.524073Z","start_time":"2019-01-09T00:25:36.417971Z"},"scrolled":true,"trusted":true,"_uuid":"9f402403f8c90c926e8d589b1dba972ccdab8563","_kg_hide-input":true},"cell_type":"code","source":"#Calculate how players are lined up\n#Based on above will only look at Dline and Linebacker\nSidePlayer = player[player.Group.isin(['DLine','Linebacker'])]\nSideGroup = SidePlayer.groupby(['Season_Year','GameKey','PlayID','Side']).agg({'Role': 'count'}).reset_index()\nSideGroup = pd.merge(bad,SideGroup,on=['Season_Year','GameKey','PlayID'],how='right')\nSideGroup = SideGroup[pd.isna(SideGroup.Reason)]\nSide_Left = SideGroup[SideGroup.Side=='Left']\nSide_Right = SideGroup[SideGroup.Side=='Right']\nSide_Con = pd.merge(Side_Left,Side_Right,on=['Season_Year','GameKey','PlayID'],how='inner',suffixes=['_Left','_Right'])\nSide_Con['Delta']= Side_Con.Role_Left-Side_Con.Role_Right\nSide_Con = Side_Con[['Season_Year','GameKey','PlayID','Delta']]\nSide_Con = pd.merge(Side_Con,video,on=['Season_Year','GameKey','PlayID'],how='left')\nSide_Con.Concussion = Side_Con.Concussion.fillna('NO')\n\n#Group by formation. Separate formations by coverage and return teams\nSide_Agg = Side_Con.groupby(['Delta','Concussion'])['Season_Year'].agg('count').reset_index()\nSide_Agg.columns = ['Delta','Concussion','Count']\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nSide_Yes = Side_Agg[Side_Agg.Concussion=='YES']\nSide_No = Side_Agg[Side_Agg.Concussion=='NO']\n\nSide_Percent=pd.merge(Side_Yes,Side_No,on=['Delta'],suffixes=['_Yes','_No'],how='left')\nSide_Percent['Count_Yes_Percent'] = Side_Percent.Count_Yes/len(video)\nSide_Percent['Count_No_Percent'] = Side_Percent.Count_No/(len(play_information)-len(video))\n\n#High Number indicate bad formations\nSide_Percent['Ratio'] = Side_Percent.Count_Yes_Percent/Side_Percent.Count_No_Percent\nSide_Percent\nSide_Con['Balanced']=np.where(Side_Con.Delta==0,True,False)\nSide_Agg = Side_Con.groupby(['Balanced','Concussion'])['Season_Year'].agg('count').reset_index()\nSide_Agg.columns = ['Balanced','Concussion','Count']\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nSide_Yes = Side_Agg[Side_Agg.Concussion=='YES']\nSide_No = Side_Agg[Side_Agg.Concussion=='NO']\n\nSide_Percent=pd.merge(Side_Yes,Side_No,on=['Balanced'],suffixes=['_Yes','_No'],how='left')\nSide_Percent['Count_Yes_Percent'] = Side_Percent.Count_Yes/len(Explained)\nSide_Percent['Count_No_Percent'] = Side_Percent.Count_No/(len(play_information)-len(Explained))\n\n#High Number indicate bad formations\nSide_Percent['Ratio'] = Side_Percent.Count_Yes_Percent/Side_Percent.Count_No_Percent\nSide_Percent[['Balanced','Count_Yes','Count_No','Ratio']]","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.602148Z","start_time":"2019-01-09T00:25:36.526075Z"},"trusted":true,"_uuid":"647e9c8a29363cb79cd7b2f50ffb394c1b7a859a","_kg_hide-input":true},"cell_type":"code","source":"#Fortunately I already have that data. In hindsight, I should have looked at this first. \nRoleGroup_Ret = RoleGroup[(RoleGroup.Cov_Ret=='Return') & (RoleGroup.Group != 'Returner')]\nRoleGroup_Ret= pd.merge(bad,RoleGroup_Ret,on=['Season_Year','GameKey','PlayID'],how='right')\nRoleGroup_Ret = RoleGroup_Ret[pd.isna(RoleGroup_Ret.Reason)]\n#Very similar to above, but with a few tweaks\nRole_Con = pd.merge(RoleGroup_Ret,video,on=['Season_Year','GameKey','PlayID'],how='left')\nRole_Con.Concussion = Role_Con.Concussion.fillna('NO')\n\n#Group by formation. Separate formations by coverage and return teams\nRole_Agg = Role_Con.groupby(['Group','Role','Concussion'])['Season_Year'].agg('count').reset_index()\nRole_Agg.columns = ['Group','Role','Concussion','Count']\n#Separate Yes and No's and join them to compare them.\n#Many formations had no concussions. We may revisit them later.\nRole_Yes = Role_Agg[Role_Agg.Concussion=='YES']\nRole_No = Role_Agg[Role_Agg.Concussion=='NO']\n\nRole_Percent=pd.merge(Role_Yes,Role_No,on=['Group','Role'],suffixes=['_Yes','_No'],how='left')\nRole_Percent['Count_Yes_Percent'] = Role_Percent.Count_Yes/len(Explained)\nRole_Percent['Count_No_Percent'] = Role_Percent.Count_No/(len(play_information)-len(Explained))\n\n#High Number indicate bad formations\nRole_Percent['Ratio'] = Role_Percent.Count_Yes_Percent/Role_Percent.Count_No_Percent\nRole_Percent[Role_Percent.Group=='Jammer'][['Role','Count_Yes','Count_No','Ratio']].rename(columns = {'Role':'Jammers'})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"761f0902f7743b4a20e497955c5a44978b0dc582"},"cell_type":"markdown","source":"The conclusion about Jammers and line balance are unchanged.\n\n## Types of Punts\nBased on the analysis above, it is now worthwhile to look at the 2 major different types of punts.\n1. Distance Punt - When a team is far from their endzone, the kicker can focus on distance and not worry about kicking the ball into the endzone.\n2. Pinning Punt - As a team approaches their endzone, the kicker focuses on pinning the other team inside their 10 yard line. This punt tends to be higher and shorter and is seldom returned.\n\nAlmost all concussions occur on the distance punt. This is not suprising but has major implications on which rules will help reduce concussions."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-08T21:00:17.925462Z","start_time":"2019-01-08T21:00:17.923460Z"},"_uuid":"5840e489f587593d0f9810cbbad9ef2c913ac3d9"},"cell_type":"markdown","source":"## Clean Data Analysis Summary\nA rule change should target one or more of the following areas:\n1. Tackling\n * Fair Catch bonus yards (5 or 10 yards)\n * Kicking out of bounds bonus (5 or 10 yards)\n * Eliminating Punts\n2. Distance Punts (>65 Yards for a TD)\n3. Jammers\n * Limit teams to 2 Jammers\n4. Coverage Team injuries\n5. Preseason\n * Eliminate punts during preseason\n * Experiment with punt rules during preseason\n\nRecent rule changes have helped. The following areas are lower priority: \n1. Blocking\n * Changes to Blindside block definition\n * Changes to defeneless player definition\n * Wedge block rule from kickoff rules\n * Speed differential between players\n2. Pinning Punts (<65 Yards for a TD)\n * Changes in TouchBack distance\n * Encouraging teams to go for 4th down\n * Remove rule that places missed fields goals at spot of kick.\n3. Line Balance\n * Certain number of players on the line\n * Limit line imbalance\n * Setup Zone rules\n4. Returner Injuries\n * CFL 5 yards buffer zone\n * Fair catch results in play ending\n * Changes in muff rules\n5. Targeting/Helmet to Helmet Blows\n * Any enhancements to the targeting rule"},{"metadata":{"ExecuteTime":{"end_time":"2018-12-28T21:14:06.487519Z","start_time":"2018-12-28T21:14:06.485518Z"},"_uuid":"4029cd729b6d1eb73f77fd3661261f034a9ecb12"},"cell_type":"markdown","source":"## Rules & Rule Impacts"},{"metadata":{"_uuid":"c6ad549e13284fb146c10da0970ac529c1b7d1ef"},"cell_type":"markdown","source":"### Limit Jammers\nMy first thought is to limit jammers to 2. That change would be transparent to the casual fan and would reduce concussions.\nFurther analysis shows that could have a impact on the return game and also calls into question the conclusion that Jammers should be limited."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.647192Z","start_time":"2019-01-09T00:25:36.604150Z"},"trusted":true,"_uuid":"ee2ec895b26858859cd46609eb2e9e92db9de6b5","_kg_hide-input":true},"cell_type":"code","source":"print('Short Returns on Distance Punts')\ngood = pd.merge(bad,Role_Con[(Role_Con.Group == \"Jammer\") & (Role_Con.Concussion==\"YES\")], \n on=['Season_Year','GameKey','PlayID'],how='right')\ngood = good[(pd.isna(good.Reason_x))]\ngood = pd.merge(good,play_information[(play_information.YardsforTD>65)] , on=['Season_Year','GameKey','PlayID'],how='left')\ngood = good[~(pd.isna(good.Season_Type))&(good.ReturnYards<10)]\ndisplay(good.groupby('Role')['ReturnYards'].agg(['mean','count']).reset_index().rename(columns = {'Role':'Jammers', 'mean':'Return Yards'}).dropna())\n#display(good.groupby('Role')['PuntYards'].agg(['mean','count']).reset_index().rename(columns = {'Role':'Jammers', 'mean':'Punt Yards'}).dropna())\n\ngood = pd.merge(Role_Con[Role_Con.Group=='Jammer'],play_information[(play_information.YardsforTD>65)], on=['Season_Year','GameKey','PlayID'])\ngood = good[(good.Role>1)&(good.ReturnYards<10)]\ndisplay(good.groupby('Role')['ReturnYards'].agg(['mean','count']).reset_index().rename(columns = {'Role':'Jammers', 'mean':'Return Yards'}).dropna())\nprint('On short returns, more jammers do not appear to have an impact on concussions, but the numbers are very small')\nprint('')\nprint('Long Returns on Distance Punts')\ngood = pd.merge(bad,Role_Con[(Role_Con.Group == \"Jammer\") & (Role_Con.Concussion==\"YES\")], \n on=['Season_Year','GameKey','PlayID'],how='right')\ngood = good[(pd.isna(good.Reason_x))]\ngood = pd.merge(good,play_information[(play_information.YardsforTD>65)] , on=['Season_Year','GameKey','PlayID'],how='left')\ngood2 = good[:]\ngood = good[~(pd.isna(good.Season_Type))&(good.ReturnYards>10)]\ndisplay(good.groupby('Role')['PuntYards'].agg(['mean','count']).reset_index().rename(columns = {'Role':'Jammers', 'mean':'Punt Yards'}).dropna())\n\ngood = pd.merge(Role_Con[Role_Con.Group=='Jammer'],play_information[(play_information.YardsforTD>65)], on=['Season_Year','GameKey','PlayID'])\ngood = good[(good.Role>1)&(good.ReturnYards>10)]\ndisplay(good.groupby('Role')['ReturnYards'].agg(['mean','count']).reset_index().rename(columns = {'Role':'Jammers', 'mean':'Return Yards'}).dropna())\nprint('On long returns, more jammers appear to have an impact on concussions, but the numbers are very small')\n\ngood2 = good2[(good2.Role>1)]\ndisplay(good2.groupby('Role')['ReturnYards'].agg(['mean','count']).reset_index().rename(columns = {'Role':'Jammers', 'mean':'Return Yards'}).dropna())\nprint('Looking at all of the concussion data, there is a much larger difference in return yards than would be expected')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ff43e542aefffb9a17d5a55e8c65f5a184e70cf5"},"cell_type":"code","source":"probplot(good,'Role','ReturnYards')\nprobplot(good2,'Role','ReturnYards')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72ac8c61437951cb04463bde9c2205a4953b90b0"},"cell_type":"markdown","source":"There is a small but consistent increase in return yards associated with more jammers in the overall data, but a large difference in the concussion data. "},{"metadata":{"_uuid":"19f4cef45019da287caea47577888c9dbd81c8b1"},"cell_type":"markdown","source":"\n\n### Do more jammers correlate to concussions? \nThe concern I have when looking at this data is the difference in return yards in the concussion subsample is very skewed compared to the general population. In the overall data, more jammers are associated with about a 3 yard increase in return yards. In the concussion dataset, the difference is very large. The difference between 2 and 4 jammers is over 20 yards. \n"},{"metadata":{"_uuid":"de71c6b6deb4e84ba004be9af72f8a696840c2c0"},"cell_type":"markdown","source":"### Increase Fair Catches \n\nAward the receiving team 5 yards for a fair catch to decrease returns. This rule would eliminate short, \"useless\" returns. The analysis below focuses only on distant punts."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.661205Z","start_time":"2019-01-09T00:25:36.649194Z"},"trusted":true,"_uuid":"eb148e85600d1968cf65c65bdb84481afc5149b4","_kg_hide-input":true},"cell_type":"code","source":"down = len(play_information[play_information.Downed==1])\ntouchback = len(play_information[play_information.Touchback==1])\nOOB = len(play_information[play_information.OutofBounds==1])\nfaircatch = len(play_information[play_information.FairCatch==1])\nreturns = len(play_information.ReturnYards.dropna())\npunts = len(play_information)\nprint ('Downed: {:.1%} Touchback: {:.1%} OutofBounds: {:.1%} Fair Catch: {:.1%} Returns: {:.1%}'.format(down/punts,touchback/punts,OOB/punts,faircatch/punts,returns/punts))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd92d44795525b3b984cbf8dafce4c7b19887dd2"},"cell_type":"markdown","source":"Today, only 38% of punts are returned. A good rule change will decrease returns but not eliminate them."},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.672216Z","start_time":"2019-01-09T00:25:36.663207Z"},"trusted":true,"_uuid":"c10dce356f85fa509cd5ee2970170ecfc95fb8cb","_kg_hide-input":true},"cell_type":"code","source":"long_punts = play_information[(play_information.YardsforTD>65)]\n#long_punts.ReturnYards = long_punts.ReturnYards.fillna(0)\nlong_punts = long_punts.dropna(subset=['YardsforTD'])","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.679223Z","start_time":"2019-01-09T00:25:36.674218Z"},"trusted":true,"_uuid":"34487650588a108ba6a8ca96392ee61385483cf6","_kg_hide-input":true},"cell_type":"code","source":"long_punts['ten'] = np.where((long_punts['ReturnYards']<10) | (long_punts.FairCatch==1),1,0)\nlong_punts['five'] = np.where((long_punts['ReturnYards']<5) | (long_punts.FairCatch==1),1,0)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.730272Z","start_time":"2019-01-09T00:25:36.680224Z"},"trusted":true,"_uuid":"cb95ab2afe06a7d648527ba2b740f40b48b3fd9d","_kg_hide-input":true},"cell_type":"code","source":"display(long_punts.groupby('FairCatch')['NetYards'].mean().reset_index())\ndisplay(long_punts.groupby('FairCatch')['PuntYards'].mean().reset_index())\nprint('Fair catches occur on shorter punts but result in fewer net yards')\ndisplay(long_punts.groupby('five')['NetYards'].mean().reset_index())\nprint('A five yard bonus results in fair catches and not fair catches having the same net yards')\ndisplay(long_punts.groupby('ten')['NetYards'].mean().reset_index())\nprint('A ten yard bonus makes fair catches better than not fair catches.')\ndisplay(long_punts.groupby('OutofBounds')['NetYards'].mean().reset_index())\nprint('Kicking out of bounds is slightly better than not, but not as good as forcing a fair catch')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-09T00:25:36.736278Z","start_time":"2019-01-09T00:25:36.732274Z"},"trusted":true,"_uuid":"abd082a041c24ee506a24fb7933d96e808fb682a"},"cell_type":"code","source":"print('Average Return Yards: ', play_information.ReturnYards.mean())","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-08T22:11:56.573275Z","start_time":"2019-01-08T22:11:56.568270Z"},"_uuid":"0495bda48b805981a32d13503e4c28a44035cd5f"},"cell_type":"markdown","source":"It is difficult to precisely predict the impact of the rule change. Today, a fair catch results in 4 fewer yards in field position. The 5 yard bonus effectively eliminates the disparity between fair catches and returns. On average, a team could fair catch all punts and not be disadvantaged. \n"},{"metadata":{"_uuid":"f154c8de5c8f41168e660a0729d8f029dc6ba0e4"},"cell_type":"markdown","source":"### Why 5 yards and not 10?\n1. Start small and can always increase. 5 yards would eliminate 30% of returns and 10 yards would eliminate another 30%\n2. There are problems with 10 yards on pinning punts. Today, returners stand on the 10 yards line and ignore everything over their head. With a 10 yards bonus, they could fair catch anything and guarantee their team a position outside the 10 yard line. \n"},{"metadata":{"_uuid":"e96d1c48de30d39d512583a7e7d5a15219f3df3d"},"cell_type":"markdown","source":"### Punt Bonus\nA Punt bonus would decrease returns in the same way as a fair catch bonus would. This rule would be awkward because it would be exactly opposite of the kickoff rule that penalizes out of bound kicks. It would also reward teams for avoiding good returners. The punting team already has two ways to avoid good returners -- kick out of bounds or kick it short. "},{"metadata":{"ExecuteTime":{"end_time":"2019-01-08T21:53:10.450364Z","start_time":"2019-01-08T21:53:10.447362Z"},"_uuid":"7ffdaff6015de31a60c49785d5d60f6dcd35725d"},"cell_type":"markdown","source":"### Eliminating Punts\nEliminating punts even during just the preseason would be a major change to the game. It would be better to use the preseason to validate any punt rule changes.\nThere is no need for that big of a change at this time at the NFL level. High schools and younger teams might want to explore these rules. "},{"metadata":{"_uuid":"81a9f37e8ed954846149e1f3eed65051acb5a885"},"cell_type":"markdown","source":"## Integrity of the Game\n### Fair Catch\nHistorically, the NFL has used small yardage nudges to change behavior. Kickoff touchbacks were increased by 5 yards. Kickoff starting points have moved multiple times in 5 yards increments. In an attempt to decrease long field goal attempts, the ball was placed at the spot of the kick. That rule was essentially an 8 yard \"penalty\" on missed field goals.\n\nAwarding 5 yards to discourage dangerous play is the exact kind of rule change that the NFL has done before. The rule would also be transparent to the casual fan. After a TV timeout, the ball would be advanced 5 yards from where the ball was caught. \n\nCollege football has also paved the way for this rule change by allowing teams to fair catch kickoffs inside the 25 and award a touchback. Likewise, this rule change would set a positive example for college and high school teams. Changes to improve NFL players' safety quickly get moved and expanded to other levels. If the NFL adopted 5 yards for a fair catch, it would not be suprising to see college or high school adopt 10 yards. \n\n### Limit Jammers\nFormation changes are also a popular technique to reduce concussions. Limiting jammers would be consistent with recent kickoff rules. It would be easy to referee. "},{"metadata":{"_uuid":"ca6a8f44e6266147d622f9fafa58e882629bf89d"},"cell_type":"markdown","source":"## Reaction to 5 yards for Fair Catch\nHow may kicking and receiving teams react to the new rule? Would any of these reactions result in more injuries.\nThere are 4 ways the rule may impact strategy\n1. Teams try to punt further to recover the 5 yards. Longer punts are associated with longer returns, so receiving teams may end up returning the ball more often. In addition, receiving teams may use more jammers if they return more balls. Because receiving teams have an effective counter, I believe teams will not consistently go for long punts\n\n2. Teams go for the block more often. If the returner is going to fair catch, send more players after the punt. Very few concussions were associated with blocking for the punt (3 of 37), so any increase in those concussions should be more tha offset with the decrease in concussions from increased fair catches. In addition, teams could not go for an all out block without opening themselves up to a fake.\n\n3. Teams may punt out of bounds more often. Today, kicking out of bounds is slightly worse than a fair catch. With a 5 yard bonus, teams would be encourage to kick out of bounds. Taken to an extreme, this would remove the excitement of punt returns. This rule should be evaluated in the preseason to determine if teams choose to always kick it out of bounds. \n\n4. Pinning a team inside the 10 yard line would be more difficult. The receiver can fair catch the ball at the 5 and get the ball at the 10. Balls that land past the 5 yard line are more difficult to down. Pinning punts are not a problem, so this rule unnecesarily makes them harder. I resisted the temptation to make the rule more complicated to deal with this case."},{"metadata":{"_uuid":"c4342f9b05c14231b2879ce8bc8dd98943fde34f"},"cell_type":"markdown","source":"## Reaction to Limit Jammers\nHow may kicking and receiving teams react to the new rule? Would any of these reactions result in more injuries.\n\n* Limiting jammers would impact all punt plays. It is more likely to have unforseen consequences.\n* With only 2 jammers, teams may use more front blockers. Front blockers are rarely used right now, but they are associated with a higher rate of concussions. A rule limiting jammers would also need to address the number of players near the line of scrimmage. \n* Punts are not distinct plays like kickoffs. On punts, defenses need to protect against fake attempts. Restrictions on the return punt formations might make defenses more vulnerable. If a team lines up in a traditional punt formation, the rule is easy to enforce. What happens if kicking teams explore different formations? Would the receiving team be unable to counter them because of these rule changes? "},{"metadata":{"_uuid":"3c17a302ca472a0b8b72b7fcc480b577a3d4cb4b"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"06663284fbab31d89a936dbac09ce52feb6d3f2d"},"cell_type":"markdown","source":"# Summary"},{"metadata":{"_uuid":"85915aea4698fe8f7691f3703b6d9ed7a1f2472a"},"cell_type":"markdown","source":"In this kernal, I showed an analytical framework for cleaning data and evaluating rule changes to increase player safety. This analysis showed that concussions can be broken up into three roughly equal groups\n1. Covered by existing rules or not punt return related\n2. Group tackling on long punt returns\n3. Long tail of many other factors.\n\nGroup tackling is a fundamental part of football, and there were no obvious technique problems. Any new rule needs to decrease the frequency of long returns. After reviewing 17 rules, there are two leading candidates. Limiting jammers and awarding bonus yards for fair catches both have pros and cons. Because of data sparsity, it is unclear if the connection between jammers and concussion is real. It could easily be a correlation with long returns. Because a limit on jammers would impact all punt players, it should not be implemented until the connection is clearer. Awarding 5 yards for a fair catch is better because it will impact only a subset of punts. It is also a simpler rule to officiate and can be easily adjusted in the future and at different levels. \n"},{"metadata":{"trusted":true,"_uuid":"a66a790aad1d5009c0ef77d14b8ba56d1f7dc8c2"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"hide_input":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"384px"},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":1}