{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.0.5"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<font size=+1>The dataset provided for this competition does not include all special teams plays for the included NFL seasons; specifically it is missing the plays where player or ball tracking data could not be measured. Refer to <a href='http://www.kaggle.com/c/nfl-big-data-bowl-2022/discussion/286330'>this discussion thread</a> for details. Additionally the dataset only includes the seasons 2018-2020.</font>\n\n<font size=+1>This notebook supplements the data Kaggle provided by adding missing plays from 2018-2020 and older seasons going back to 2010.  The data source is <a href='http://github.com/nflverse/nflfastR'>NFLFastR</a>, which obtained the play data via web scraping.  Note that my notebook only includes punt plays, and only a subset of the columns provided by Kaggle. However the data source is rich enough so that all special teams plays and (except for the player and ball tracking data) all of the Kaggle-provided features can be added, using methodology similar to that demonstrated here.</font>","metadata":{}},{"cell_type":"code","source":"rm(list=ls())\n\nlibrary(dplyr)\nlibrary(tidyverse)\n\noptions(dplyr.summarise.inform = FALSE)\noptions(digits=5) \n\nNFL_DATA_PATH=\"../input/nfl-big-data-bowl-2022/\"\n#from https://github.com/nflverse/nflfastR-data/\nSCRAPED_DATA_PATH=\"../input/nfl-play-data-2010-2020/\"\nSCRAPED_PLAY_BASE=paste(SCRAPED_DATA_PATH, \"play_by_play_\", sep=\"\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-25T20:54:27.079807Z","iopub.execute_input":"2021-11-25T20:54:27.081552Z","iopub.status.idle":"2021-11-25T20:54:27.109144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>Load Kaggle data. The filter excludes plays where the punting team retained possession, which are irrelevant to the analyses I am doing for the competition.</font>","metadata":{}},{"cell_type":"code","source":"plays<-read.csv(paste(NFL_DATA_PATH, \"plays.csv\", sep = \"\"))\ngames<-read.csv(paste(NFL_DATA_PATH, \"games.csv\", sep=\"\")) %>% select(gameId, season)\n\n#non special teams results generally indicates a fake or fumble\n\npunt_plays=plays %>% filter(specialTeamsPlayType==\"Punt\" &   specialTeamsResult!=\"Non-Special Teams Result\") \n\npunt_plays_start_and_results<-punt_plays %>%  \n  inner_join(games, by=c(\"gameId\")) %>%\n   select(gameId,playId,possessionTeam,yardlineNumber,yardlineSide,playResult)  \n","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:27.142987Z","iopub.execute_input":"2021-11-25T20:54:27.144436Z","iopub.status.idle":"2021-11-25T20:54:27.349873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>All of the data I load from the external source is transformed to the format and variable names used in the Kaggle data set. The below function assists in transforming data from that source to playResult, which for punts, represents net punt yards.</font>","metadata":{}},{"cell_type":"code","source":"\n#convert data from alternatiave source to kaggle.playResult\ngetScrapedPlayResult <- function (yardline_100, next_play_yardline_100,series_result,posteam,posteamNextPlay)\n{\n  \n  \n  ans = \n    if_else(series_result==\"Safety\", ifelse(posteam==posteamNextPlay, yardline_100, -(100-yardline_100)),\n            if_else(series_result==\"Opp touchdown\", yardline_100 -100, \n                ifelse(series_result== \"Touchdown\", yardline_100,\n                  ifelse(posteam==posteamNextPlay, yardline_100-next_play_yardline_100,\n                         yardline_100-(next_play_yardline_100 + (50-next_play_yardline_100) * 2)))))\n  return(ans)\n}\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-11-25T20:54:27.111721Z","iopub.execute_input":"2021-11-25T20:54:27.113262Z","iopub.status.idle":"2021-11-25T20:54:27.140357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>The below loads the 2018-2020 data from the supplemental data source</font>","metadata":{}},{"cell_type":"code","source":"#add missing data and previous years\nprint(\"Loading 2018 data\")\nseason_2018_scraped<-read.csv(paste(SCRAPED_PLAY_BASE,\"2018.csv\", sep=\"\")) %>% \n  select(playId=play_id, gameId=old_game_id,week, posteam, yardline_100, play_type,\n         series_result,ydsnet, play_type_nfl,side_of_field)  %>% drop_na(play_type, ydsnet)\n\nprint(\"Loading 2019 data\")\n\nseason_2019_scraped<-read.csv(paste(SCRAPED_PLAY_BASE,\"2019.csv\", sep=\"\")) %>% \n  select(playId=play_id, gameId=old_game_id,week, posteam, yardline_100, play_type,\n         series_result,ydsnet, play_type_nfl,side_of_field)  %>%\n          drop_na(play_type, ydsnet)\nprint(\"Loading 2020 data\")\n\nseason_2020_scraped<-read.csv(paste(SCRAPED_PLAY_BASE,\"2020.csv\", sep=\"\")) %>% \n  select(playId=play_id, gameId=old_game_id,week, posteam, yardline_100, play_type,\n         series_result,ydsnet, play_type_nfl,side_of_field)  %>%\n  drop_na(play_type, ydsnet)\n\n\n#season_2018_scraped<-season_2018_scraped%>%filter(!is.na(play_type))\nseason_2018_scraped <-season_2018_scraped %>% mutate(posteamNextPlay=lead(posteam), \n  next_play_type=lead(play_type),next_play_yardline_100 = lead(yardline_100))\n\n\nseason_2019_scraped <-season_2019_scraped %>% mutate(posteamNextPlay=lead(posteam), \n  next_play_type=lead(play_type),next_play_yardline_100 = lead(yardline_100))\n\nseason_2020_scraped <-season_2020_scraped %>% mutate(posteamNextPlay=lead(posteam), \n    next_play_type=lead(play_type),next_play_yardline_100 = lead(yardline_100))\n\nscraped_plays<-season_2018_scraped %>% rbind(season_2019_scraped) %>% rbind(season_2020_scraped) \n\nscraped_plays<- scraped_plays %>% filter(play_type==\"punt\" | play_type_nfl==\"PUNT\")\n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:27.35257Z","iopub.execute_input":"2021-11-25T20:54:27.3541Z","iopub.status.idle":"2021-11-25T20:54:56.15416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>The below converts data from the supplemental data source to Kaggle format, where required. Additionally, in both data sets, the punting and receiving team is set to OAK in all cases where  it shows LV, in order to resolve inconsistencies I observed across the data sets (the Oakland Raiders moved to Las Vegas in 2020).</font>","metadata":{}},{"cell_type":"code","source":"#Below converts all data in scraped source to Kaggle column names and format\n#calculating  where needed.\n\n\nscraped_plays <-  scraped_plays %>% \n  mutate(playResult_Scraped=getScrapedPlayResult(yardline_100,\n          next_play_yardline_100,series_result,posteam, posteamNextPlay))\n\n\n#convert LV and OAK team names all to OAK \n#because there is some inconsistency within/across datasets\npunt_plays_start_and_results <- punt_plays_start_and_results %>% \n        mutate(yardlineSide=ifelse(yardlineSide==\"LV\", \"OAK\", yardlineSide),\n               possessionTeam = ifelse(possessionTeam==\"LV\", \"OAK\", possessionTeam))\n\nscraped_plays<- scraped_plays %>% mutate(posteam=ifelse(posteam==\"LV\", \"OAK\", posteam),\n  side_of_field = ifelse(side_of_field==\"LV\", \"OAK\", side_of_field),\n  yardlineNumber_scraped=ifelse(yardline_100==50, 50, ifelse(posteam==side_of_field, \n      yardline_100 - ((yardline_100 - 50) * 2), yardline_100)))\n","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:56.158021Z","iopub.execute_input":"2021-11-25T20:54:56.160469Z","iopub.status.idle":"2021-11-25T20:54:56.198503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>Below runs a few validation steps and data reviews to check tie outs across the data sets and review the data quality of the external data source. Refer to comments and output for details and one case where the the supplemental source seems to be incorrect for a small number of observations. </font>","metadata":{}},{"cell_type":"code","source":"#below are validation step to ensure that there are tie outs between matching\n#kaggle and scraped games.\ncombined<-punt_plays_start_and_results %>% \n  inner_join(scraped_plays, by=c(\"playId\", \"gameId\"))\n\nprint(ifelse(combined %>% filter(possessionTeam !=posteam) %>% nrow()==0, \n             \"Tie out on possession team\", \n             \"Mismatch on possession team\"))\n\nprint(ifelse(combined %>% filter(yardlineSide !=side_of_field) %>% nrow()==0, \n             \"Tie out on receiving team\", \n             \"Mismatch on receiving team\"))\n\nprint(ifelse(combined %>% filter(yardlineNumber_scraped!=yardlineNumber) %>% nrow()==0, \n             \"Tie out on yardline\", \n             \"Mismatch on yardline\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:56.202163Z","iopub.execute_input":"2021-11-25T20:54:56.204529Z","iopub.status.idle":"2021-11-25T20:54:56.24773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#15 total play result mismatch for three overlapping years.\n#manual review suggests kaggle data is correct, for those 3 years use kaggle data\n#for other years live with low error rate\nerrors<-combined %>% filter(playResult !=playResult_Scraped)\nprint(paste(\"Mismatches on Kaggle/Scraped data, net punt yards, for overlapping games:\", nrow(errors))) \n","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:56.251328Z","iopub.execute_input":"2021-11-25T20:54:56.253654Z","iopub.status.idle":"2021-11-25T20:54:56.277226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#data missing? #when yardlineNumber = 50 Kaggle yardlineSide is known \n#to be is.na, otherwise\n#should be none\nprint(ifelse(combined[rowSums(is.na(combined)) > 0,] %>% \n  filter(yardlineNumber !=50) %>% nrow()==0, \"no missing data\", \"data missing\"))\n\n \n\n#missing data in kaggle?\nin_scraped_not_kaggle<- scraped_plays %>% \n  left_join(punt_plays_start_and_results, by=c(\"playId\", \"gameId\"))\nprint(paste(\"Punts missing in Kaggle data\", in_scraped_not_kaggle%>% filter(is.na(playResult))   \n            %>% summarize(n=n())))\n#640 punts not in kaggle. This is expected per competition admins \n#(and one of the reasons I created this notebook)\n\n#By contrast we expect scraped data to have all punts, including all of the punts in the kaggle data\nin_kaggle_not_scraped<- punt_plays_start_and_results %>% \n  left_join(scraped_plays, by=c(\"playId\", \"gameId\")) %>% filter(is.na(playResult_Scraped)) %>%\n    filter(is.na(yardlineNumber_scraped))\nprint(ifelse(nrow(in_kaggle_not_scraped)==0, \"Scraped data has all games\", \"Scraped data missing games\"))\n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:56.281Z","iopub.execute_input":"2021-11-25T20:54:56.283471Z","iopub.status.idle":"2021-11-25T20:54:56.33778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>Below is step 1 in creating the final data set. For the overlapping seasons and plays, \nKaggle data is retained. The supplemental data is use for missing 2018-2020 punts.  The code runs a few more checks after the missing data is added.</font>\n","metadata":{}},{"cell_type":"code","source":"#now that validation checks have and all punts from 2010-2017. passed populate kaggle data from union of kaggle/scraped\n#and other years from scraped\n\n# use kaggle data for  years 2018-2020 where available, add scraped data for missing\n#games.\n\nadditional_plays <-in_scraped_not_kaggle %>% filter(is.na(playResult)) %>% \n  select(gameId, playId, possessionTeam=posteam,\n      yardlineNumber=yardlineNumber_scraped, yardlineSide=side_of_field, playResult=playResult_Scraped)\n\n#add missing data to Kaggle from Scraped and run two more checks.\n#check s/b and is 0\nprint(ifelse(additional_plays[rowSums(is.na(additional_plays)) > 0,] %>%nrow() ==0,\n        \"New data not missing values\", \"New data missing values\"))\n\npunt_plays_start_and_results <-punt_plays_start_and_results %>% rbind(additional_plays)\n\n#check s/b and is 0\nprint(ifelse(punt_plays_start_and_results %>% group_by(gameId, playId) %>% \n      summarize(n=n()) %>% filter(n>1) %>% nrow()==0, \"data unique on playId/gameId\", \n      \"Uniqueness check failed\"))\n","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:54:56.340481Z","iopub.execute_input":"2021-11-25T20:54:56.341985Z","iopub.status.idle":"2021-11-25T20:54:56.483163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font size=+1>Below is the final step, adding data from the supplemental source for seasons 2010-2017.</font>","metadata":{}},{"cell_type":"code","source":"# finally load punts from earlier seasons\nyears=seq(2017, 2010)\nfor(y in years)\n{\n  print(paste(\"Loading\", y ))\n  \n  year_data<-read.csv(paste(SCRAPED_PLAY_BASE, y, \".csv\", sep=\"\")) %>% \n    select(playId=play_id, gameId=old_game_id,week, posteam, yardline_100, play_type,\n           series_result,ydsnet, play_type_nfl,side_of_field)  %>% drop_na(play_type, ydsnet)\n  \n  year_data <- year_data %>% mutate(posteamNextPlay=lead(posteam), \n                                    next_play_type=lead(play_type),next_play_yardline_100 = lead(yardline_100))\n  \n  year_data <-year_data %>% filter(play_type==\"punt\" | play_type_nfl==\"PUNT\")\n  \n  year_data <-   year_data   %>% \n    mutate(playResult_Scraped=getScrapedPlayResult(yardline_100,\n                                                   next_play_yardline_100,series_result,posteam, posteamNextPlay))\n  \n  \n  year_data <-   year_data  %>% \n    mutate(posteam=ifelse(posteam==\"LV\", \"OAK\", posteam),\n           side_of_field = ifelse(side_of_field==\"LV\", \"OAK\", side_of_field),\n           yardlineNumber_scraped=ifelse(yardline_100==50, 50, ifelse(posteam==side_of_field, \n                                                                      yardline_100 - ((yardline_100 - 50) * 2), yardline_100)))\n  \n  year_data <-   year_data  %>% select(gameId, playId, possessionTeam=posteam,\n                                       yardlineNumber=yardlineNumber_scraped, yardlineSide=side_of_field, \n                                       playResult=playResult_Scraped)\n  \n  punt_plays_start_and_results <-   punt_plays_start_and_results %>% rbind(year_data)\n  }\n    \n   ","metadata":{"execution":{"iopub.status.busy":"2021-11-25T20:56:14.487338Z","iopub.execute_input":"2021-11-25T20:56:14.489652Z","iopub.status.idle":"2021-11-25T20:57:31.24156Z"},"trusted":true},"execution_count":null,"outputs":[]}]}