{"cells":[{"metadata":{"_uuid":"d36b2a86eaf3f010886004134577e93fe7076594"},"cell_type":"markdown","source":"# NFL Game Data Dictionary Creation Code\nby [Bon Crowder](http://boncrowder.com)"},{"metadata":{"_uuid":"87e6ab7c138223a419c34731c85e275d1e0fc00b"},"cell_type":"markdown","source":"This is the creation code for creating and saving a data dictionary for the `game_data.csv` data in the [Kaggle NFL Punt Analytics Competition](https://www.kaggle.com/c/NFL-Punt-Analytics-Competition/data).\n\n* Find the [actual data dictionary here](https://www.kaggle.com/mathfour/nfl-game-data-dictionary?target=_blank)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5830344e0398815e1da0227fe65bc61fa655ff89"},"cell_type":"code","source":"import pandas as pd\npd.options.display.max_columns = 100\npd.options.display.max_rows = 100\npd.options.display.float_format = lambda x: f' {x:,.2f}'\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cde06658fa8053e165cadbf84341f9340b7e7780"},"cell_type":"markdown","source":"## Read in the Data"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"a4dd5563a2f0366835fe717139d87f639f9c89f5"},"cell_type":"code","source":"games = pd.read_csv('../input/game_data.csv', parse_dates=['Game_Date'])\ngames.head(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f24e82aba8f17a164c6842381566a09b8cf5cbc1"},"cell_type":"markdown","source":"## Create the Data Dictionary Shell\nMost of the bits are blank here. In the next step, I'll write in the details."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b037b34fcf99c12c8bf86b442244c364a581000d"},"cell_type":"code","source":"# assign our variable so the rest of the code works nicely\ndata = games\n\n# helper function to count the blanks (not the NaN's, but the actual blank values)\ndef count_blanks(column):\n    count = 0\n    for thing in column:\n        if not thing:\n            count += 1\n    return count\n\n# create the data dictionary directly (mostly empty info at this point)\ndata_dictionary = pd.DataFrame(index=data.columns, columns=['unique', 'missing', 'blank', 'read type',\n                                                            'act type', 'description'])\nfor name in data.columns:\n    data_dictionary.loc[name,'missing'] = data[name].isna().sum()\n    data_dictionary.loc[name, 'unique'] = data[name].nunique()\n    data_dictionary.loc[name,'blank'] = count_blanks(data[name])\n    data_dictionary.loc[name, 'values'] = str(data[name].unique())\n    data_dictionary.loc[name,'act type'] = 'string'\n    data_dictionary.loc[name,'read type'] = str(data[name].dtype)\n    \n# just to make sure it's looking like I want\ndata_dictionary","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66bd5f03c12f120cfea0fd2e3f84c542dee57c1a"},"cell_type":"markdown","source":"## Define the parts of this particular data dictionary\nThis is the part where I had to manually insert the column names into the dtypes they _should_ be in."},{"metadata":{"trusted":true,"_uuid":"668522dd2f6ce8e25da503768d2ac683e0860b3f"},"cell_type":"code","source":"# defining manually which columns should be which kind of data type\n\n# the data that should be dtype string\ndata_strings = ['Season_Type', 'Game_Day','Game_Site', 'Start_Time', \n                'Home_Team', 'HomeTeamCode', 'Visit_Team', 'VisitTeamCode', \n                'Stadium', 'StadiumType', 'Turf', 'GameWeather', 'OutdoorWeather']\ndata_bools = [] # data that should be dtype boolean (none in this dataset)\ndata_ints = ['GameKey', 'Season_Year', 'Week'] # data that should be dtype int\ndata_floats = ['Temperature'] # data that should be dtype float\ndata_dates = ['Game_Date'] # data that should be dtype datetime\n\n# defining manually what each column actually is\ndata_dictionary.loc['GameKey','description'] = (\n            'Looks like just an index from 1 instead of 0')\ndata_dictionary.loc['Season_Year','description'] = (\n            'One of the two years covered in this dataset')\ndata_dictionary.loc['Season_Type','description'] = (\n            'Pre, post or regular season')\ndata_dictionary.loc['Week','description'] = (\n            'The week in which the game is played')\ndata_dictionary.loc['Game_Date','description'] = (\n            'Date on which the game is played')\ndata_dictionary.loc['Game_Day','description'] = (\n            'Day of the week (word) in which the game is played')\ndata_dictionary.loc['Game_Site','description'] = (\n            'City, country or other location of the stadium')\ndata_dictionary.loc['Start_Time','description'] = (\n            'Start of the game (kickoff?) in 24 hour scale')\ndata_dictionary.loc['Home_Team','description'] = (\n            'Full name of the home team including loc and mascot')\ndata_dictionary.loc['HomeTeamCode','description'] = (\n            'Two or three letter team identifier')\ndata_dictionary.loc['Visit_Team','description'] = (\n            'Full name of the visiting team including loc and mascot')\ndata_dictionary.loc['VisitTeamCode','description'] = (\n            'Two or three letter team identifier')\ndata_dictionary.loc['Stadium','description'] = (\n            'Full name of stadium')\ndata_dictionary.loc['StadiumType','description'] = (\n            'Type as well as status of stadium. Many dup\\'s b/c of typos')\ndata_dictionary.loc['Turf','description'] = (\n            'Type of turf. Many dup\\'s b/c of typos')\ndata_dictionary.loc['GameWeather','description'] = (\n            'Sometimes detailed sometimes not. Should be cleaned')\ndata_dictionary.loc['Temperature','description'] = (\n            'Temperature, assumingly in Fahrenheit')\ndata_dictionary.loc['OutdoorWeather','description'] = (\n            'Sometimes detailed sometimes not. Not sure relation to GameWeather')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f756de414f6c2a31ec339c1c5a0c5b260b31001d"},"cell_type":"code","source":"# insert into the dictionary df the types they should be\ndata_dictionary.loc[data_bools,'act type'] = 'bool'\ndata_dictionary.loc[data_ints,'act type'] = 'int'\ndata_dictionary.loc[data_floats,'act type'] = 'float'\ndata_dictionary.loc[data_dates,'act type'] = 'date'\n\ndata_dictionary","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"efa4f40d7b6ba54029a8b82b4f35372f549e5167"},"cell_type":"markdown","source":"## Save data dictionary as a file to use\nYou can save it yourself, or accessed the [ready-made file here](https://www.kaggle.com/mathfour/nfl-game-data-dictionary?target=_blank)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8c3824e25631aafd9e0aa9bd1f9bca781d7a9760"},"cell_type":"code","source":"# save data dictionary to a file for later use if you want\n# data_dictionary.to_csv('../output/NFL_Game_Data_Dictionary.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false}},"nbformat":4,"nbformat_minor":1}