{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook, I'll show you how can can very simply load the whole training data (over 21M rows) in Kaggle kernel, only taking **2 lines of code** and **less than 2 minutes**.\n","metadata":{}},{"cell_type":"markdown","source":"# The problem\nNormally, you should iterate on the train files doing something like this:","metadata":{}},{"cell_type":"code","source":"files = [f'train_{i}.csv' for i in range(10)]\ndf = pd.DataFrame()\nfor file in files:\n    df_2 = pd.read_csv(file, dtype = dtypes)\n    print('File', file, 'read')\n    df = pd.concat([df,df_2], axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:38:26.198677Z","iopub.execute_input":"2022-10-19T11:38:26.199408Z","iopub.status.idle":"2022-10-19T11:38:26.225285Z","shell.execute_reply.started":"2022-10-19T11:38:26.199307Z","shell.execute_reply":"2022-10-19T11:38:26.224018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are two problems with this approach:  \n* Too much memory is allocated, resulting in slowdowns and eventually crashing the environment because of out-of-memory errors.  \n* It's very slow. In the kaggle kernel it takes between five to ten minutes.\n* It's not parallelized. Every file is loaded sequentially only using one CPU core.","metadata":{}},{"cell_type":"markdown","source":"# The solution: Dask","metadata":{}},{"cell_type":"markdown","source":"<p  style=\"align:center\"> <img src=\"https://mma.prnewswire.com/media/1832641/Dask_Logo_lockup_primary_Logo.jpg?p=facebook\" width=\"250\"/></p>\n","metadata":{}},{"cell_type":"markdown","source":"Dask is a flexible open-source Python library for parallel computing. Dask [1] scales Python code from multi-core local machines to large distributed clusters in the cloud. Dask provides a familiar user interface by mirroring the APIs of other libraries in the PyData ecosystem including: Pandas, Scikit-learn and NumPy. It also exposes low-level APIs that help programmers run custom algorithms in parallel.\n\nDask was created by Matthew Rocklin in December 2014 and has over 9.8k stars and 500 contributors on GitHub.\n\nDask is used by retail, financial, governmental organizations, as well as life science and geophysical institutes. Walmart, Wayfair, JDA GrubHub, General Motors, NVIDIA, Harvard Medical School, Capital One  and NASA are among the organizations that use Dask.\n*[source](https://en.wikipedia.org/wiki/Dask_(software))*\n","metadata":{}},{"cell_type":"markdown","source":"# Loading data","metadata":{}},{"cell_type":"code","source":"import dask.dataframe as dd\ndtypes_dict = {\n    'game_num': 'int8', 'event_id': 'int8', 'event_time': 'float16',\n    'ball_pos_x': 'float16', 'ball_pos_y': 'float16', 'ball_pos_z': 'float16',\n    'ball_vel_x': 'float16', 'ball_vel_y': 'float16', 'ball_vel_z': 'float16',\n    'p0_pos_x': 'float16', 'p0_pos_y': 'float16', 'p0_pos_z': 'float16',\n    'p0_vel_x': 'float16', 'p0_vel_y': 'float16', 'p0_vel_z': 'float16',\n    'p0_boost': 'float16', 'p1_pos_x': 'float16', 'p1_pos_y': 'float16',\n    'p1_pos_z': 'float16', 'p1_vel_x': 'float16', 'p1_vel_y': 'float16',\n    'p1_vel_z': 'float16', 'p1_boost': 'float16', 'p2_pos_x': 'float16',\n    'p2_pos_y': 'float16', 'p2_pos_z': 'float16', 'p2_vel_x': 'float16',\n    'p2_vel_y': 'float16', 'p2_vel_z': 'float16', 'p2_boost': 'float16',\n    'p3_pos_x': 'float16', 'p3_pos_y': 'float16', 'p3_pos_z': 'float16',\n    'p3_vel_x': 'float16', 'p3_vel_y': 'float16', 'p3_vel_z': 'float16',\n    'p3_boost': 'float16', 'p4_pos_x': 'float16', 'p4_pos_y': 'float16',\n    'p4_pos_z': 'float16', 'p4_vel_x': 'float16', 'p4_vel_y': 'float16',\n    'p4_vel_z': 'float16', 'p4_boost': 'float16', 'p5_pos_x': 'float16',\n    'p5_pos_y': 'float16', 'p5_pos_z': 'float16', 'p5_vel_x': 'float16',\n    'p5_vel_y': 'float16', 'p5_vel_z': 'float16', 'p5_boost': 'float16',\n    'boost0_timer': 'float16', 'boost1_timer': 'float16', 'boost2_timer': 'float16',\n    'boost3_timer': 'float16', 'boost4_timer': 'float16', 'boost5_timer': 'float16',\n    'player_scoring_next': 'O', 'team_scoring_next': 'O', 'team_A_scoring_within_10sec': 'int8',\n    'team_B_scoring_within_10sec': 'int8'\n}\n\n\ndf = dd.read_csv('../input/tabular-playground-series-oct-2022/train_*.csv', dtype = dtypes_dict)","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:38:26.227066Z","iopub.execute_input":"2022-10-19T11:38:26.227380Z","iopub.status.idle":"2022-10-19T11:38:27.249380Z","shell.execute_reply.started":"2022-10-19T11:38:26.227350Z","shell.execute_reply":"2022-10-19T11:38:27.248187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The read_csv function doesn't load everything into memory. Instead it creates a [task graph].(https://docs.dask.org/en/latest/graphs.html)\nIt saves the columns and divides the datasets in partitions without loading them into memory (for now).","metadata":{}},{"cell_type":"code","source":"df","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:38:27.251211Z","iopub.execute_input":"2022-10-19T11:38:27.251600Z","iopub.status.idle":"2022-10-19T11:38:27.325587Z","shell.execute_reply.started":"2022-10-19T11:38:27.251568Z","shell.execute_reply":"2022-10-19T11:38:27.324939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Every operation in dask is `Delayed` until we execute the `compute()` command.\nFor example, if we want to load the whole dataset into a pandas DataFrame, we can do it the the following command:","metadata":{}},{"cell_type":"code","source":"from datetime import datetime\nstart = datetime.now()\n\ndf = dd.read_csv('../input/tabular-playground-series-oct-2022/train_*.csv', dtype = dtypes_dict)\ndf = df.compute()\n\nend = datetime.now()\ntime_readcsv = end - start\nprint(f'Time taken for reading data: {time_readcsv.seconds} seconds')","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:38:27.327281Z","iopub.execute_input":"2022-10-19T11:38:27.328137Z","iopub.status.idle":"2022-10-19T11:39:52.687577Z","shell.execute_reply.started":"2022-10-19T11:38:27.328075Z","shell.execute_reply":"2022-10-19T11:39:52.686361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"During the execution the code is using all the cores available in the Kaggle kernel.  \n![image.png](attachment:0b0acb53-de6b-4494-903a-7b69ebb3ee8b.png)","metadata":{},"attachments":{"0b0acb53-de6b-4494-903a-7b69ebb3ee8b.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAUoAAABuCAYAAABBY1qRAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAGdYAABnWARjRyu0AACLUSURBVHhe7d0J3G3V/MfxHUkydqUkSWUKiVIpRZPSpFGiaFCiSEIkelHyUpT0QoZokCtFRdIgc2iQxG0ypNE8lVnp/p/36qzHvrt99t7nec6+Pff+f5/Xa7/uc8/ZZ5999l7ru37D2uu3yNwJiiAIgmAo9xv8GwRBEAwhhDIIgqCFEMogCIIWQiiDIAhaCKEMgiBoIYQyCIKghRDKIAiCFkIogyAIWgihDIIgaCGEMgiCoIXeH2H8+9//XlxzzTXpXyy77LLFyiuvXCy66KLp/1X+9re/Ff/+978H//sf97vf/YqHPvShjZ+78847G/dB1/2CYKpMtQ1X+ec//1n84x//KB74wAcWD3nIQwav3htd+I477ijuuuuuYokllige9KAHDd4Zjn3/+te/pr+jL3SAUPbBxA2ee8wxx8ydNWsWIZ5nW3PNNed+9atfnfvf//53sPf/OOyww+61f95WWGGFuccee2w6dhWfW2211eZed911g1fq6bpfEEyVpjasP+gXE2I62Lueu+++e+7RRx+dPrPPPvvUtvmM9+xj33e84x1zJ0Rw8M5wLrroorkTAhl9oSO9uN6sxze96U3FgQceWKy77rrFmWeeWdxyyy3JspwQuuJf//pXsf322xennXZaGg2rPPGJTyxOPfXU4hvf+Mbk5hhrrbVWccABBxQTjaF2xA6CmcKwNrzxxhunfnHQQQdNell1/PGPfywmjIn097e//e3ihhtuSH+3ceGFFxY333zz4H/16Dtf+tKXJi3KoAP36OX4MJodddRRabQ68sgj506I4uCd//Gb3/xm7jbbbJMsxMsvv3zw6j00WXyO5f0J933upZdeOnj1HsKiDGYKTW3szjvvnHv44Yen/sGqG0a2+A4++OC5q6yyytwTTjhh8M69KVuUtqZ9MWfOnHRM+0Zf6MbYLcqJi16ceOKJxc4771zst99+Kb5SZZlllikmGlOKpxjZJlzwwTvNONYWW2yR/v7ud7+b/g2CBQmxwBe96EXFox/96OKyyy4bvDov+sPXvva1Yo011ih23333Yu211y7OP//84vbbbx/sUc9mm22W+se5555b/PnPfx68Oi8Tfb644IIL0t+77bZb+jdoZ+xCefHFFxe33nprugkPfvCDB6/eG64J93z55ZdPrnhXBLWXXnrpFDAPggURAz4jQUKlDq7z17/+9WL99ddPic9NNtkkGQbXX3/9YI96Hv/4xycDZcIaLX74wx8OXp2XCW+uOO+884oXvOAF6fhBN8YqlP/5z3/SzTQSrrjiioNX65GZ23PPPYu99967UVCriK9MuBqRpQsWWFiGv/vd74Zmsq+88soUz3/+859f3P/+909x/sc97nEp/sgibOK5z31uskB5anVx/Msvv7y49NJLk1W7+OKLD14N2hirUHIZiBgr0ZSDcWMENhoaFSV2gmBBQwLlpJNOSn8TtSoSPOecc056b7XVVkuvPeYxj0ni961vfSu1/SaEtTbffPO074033jh49R4IJ5eehbr66qsPXg26MFahvPvuu9PN4FZMx+JzHPPCZP7yNmfOnOLggw8ujjjiiOLlL3958ZznPGewdxDMPFiMn/rUp4ojjzxychNvXHXVVYvjjz++eP3rX18rVsSNxcc1fuQjH5le46qzLrnU+kET5mqKVfLuhMHK/PznP0+ZdC73kksuOXg16MJYhdJNclPFKEeJO1b5yU9+kizGpZZaanLTwD72sY8V++67b3H44YeP5K4Hwfzm17/+dXHUUUcVb33rWye3k08+OSVxCN6b3/zmWmOCuP3lL38p1ltvvWKRRRYZvFqkcBYXnEdFBJsQq2SBlpM65STOpptumv4NujNWoVxsscWKWbNmpZvDIpwqdXPQNKCbbrqpeO9735u+IwhmMtxmM0AIlE1Iap999kmu88Mf/vBkVFTRb4gZkSV0hDJvxO973/te2m677bbBJ+phRGy99dbzJHXKSZwnPOEJ6bWgO2MVSoFnLrGbecUVVwxerYeQvva1r00uSHXiK9f92c9+drHBBhtMbmI2Te6C0Zm705QN12C7TkUKgnEiebnXXnulv0844YT0eGKVn/3sZ8Ull1xS7LrrrskgqG76iiSP/tXGOuusM09S5/vf//5kEqduyl7QzFiFEmuuuWYKFp9yyinFb3/728Gr90aDsM8KK6zQ+BxrV572tKelkVgQe1hm8Fe/+lV6ysHz5jn+EwTzi2c+85nFLrvsUsyePbv4zne+M3j1HgzgX/7yl4tHPOIRyU1/y1veUrsRP5Zi01M94OJL6ohJyqJLEEUSZ+qMXSiJ0Ctf+co0D0wcppqlI2KmKBx66KHFM57xjPQoYzkWM1WMoNttt11x3HHHJVe9Kpas1g996EPpPd8ZQhnMb3g9L3vZy4rHPvaxyaosTwrnDbH6CCE3uw7iJ6ljTuUvfvGLwav16FOSOnj729+eXPott9wykjhTZOxCiR133LF4z3veU5x99tnJ0nv1q1+dsn5eI1Ked5Xsed/73je0UYzKox71qCTMRuRtt902jdyf/OQnU0D7ne98Z3Lds/uisY5DnINgVJ7ylKekWRtnnHFGGrQzjAeWIqtvWKJSm5WIEV6y7zDPKSMWKSZpSpB5mBtttNHgnWBUehFKI6fHF7/5zW8mUeRqcCcOOeSQ5Absv//+ySVgBY4TxyPOJrIbQcWEttpqq+Jd73pXet95EOzImAf3FeL4L33pS9PAbfoQjyvPb2RNymw3IdEpD5Cz402IRYpJmtNMYIllMDV6X48SeV09mb5hGb9xYwpFThLNz+8NgmDhY74IZRAEwYJMmFhBEAQthFAGQRC0EEIZBEHQQghlEARBCyGUQRAELYRQBkEQtBBCGQRB0EIIZRAEQQshlEEQBC2EUAZBELQQQhkEQdBCCGUQBEELIZRBEAQthFAGQRC0MF+XWbvrrruKG264obj22muLX/7yl8Wf/vSnVDpC3RzlaBV6t7BpE2p+q33cVjOkC773YQ972OB/9bg86opfffXVxc0335zO2zlaQNX25Cc/ORVD6xPnoN6PwlJqM6sNVL5uSgtMZ8X2vo5/++23Fz/4wQ9S+WH3y+reisZZQLbr8dSScc2taeozzslCtAsq1kjVjrRjdGmDdYzrOE2U275+6z649lZOdy9XXHHFYvHFFx/svXAzX4RS2QcrjqsIpzbIMJSNUKZB5xzWkSzI+4lPfKI4//zzB69MHXWXV1lllcH/7o1z/cxnPpPq/wxDw3nJS16S6pP00Wg00NNOOy3VUxmGgm4q96200kqDV7rTx/E1KUXc1GGvVtjETjvtVLz4xS/udL1UDjz66KOTUPqc9tE2mM5EXJOrrroqFdRTbTHT1garjOs4bXRp+0svvXRaQb2vtj+TuP87FZTpEZbKBz7wgeIrX/lKqxX4+9//PnVYFtrKK69cuyI5C0OtYpbPdFFPRK2dKhqj81DTZ86cOYNX6yHc+XxYl+OydpyD4x5zzDHF9ddfP3i1HtfYvn5LV+uvz+P/6Ec/Ssd1v3UmBbGe9axnJY8iWygqb7peTcfyveoeKZdArNWa6dt67wOW9ec+97lU+qFamXRYG6xjXMdpIrcL9aXa2r77a99bb7013cuFucRKrxalUenYY49NrldGg19//fWLpz71qanRE0c3RA0dLjXUQFbz+3nPe176fxmdTV1jnxsV1o2CTlx+DBuFnQ9xz9Yv8TNy6uwq4d15553J7THaKo+bUfPEeStwNl2GnQPXdZlllkkdhVvruuV9iNIb3vCG4ulPf3r6fxN9HV/JD9UulWPVeRRzW3755SffY2W6bl5TR2lYHRcD0IknnphKuI7yu2YSulad9VemiyU4ruN0odousN5666W+qF3gpptuSvdXQbTMONv+TKQ3i5KYfP7zn58UkuWWWy41doWVxDZYFIofKZ+pQym2ZIRShpMYKuW52mqr3WuUYmWKxxDaUTdFnLiEjq9Urop3VQtFhTsuhzgqCPvb3va2VNBpqaWWSi6Gz4jhKWYmTmlf537bbbclwdFgu1h1w2A5nHrqqZOWHoEmKgpPzZo1K103/xIO5yCmyPpyDgYQ9aObLK8+j+8aCLEYlNSV9vl8LR7wgAekQdBgdccdd6Tju5dVCIOSrKwn94qbbnBdkGoe1Vl/+oBSzsII+bU2S3Bcx+kCa19YSzgGuc+6/gY2fdWm/xJOgxzhzm3f/XVPF8baVL39IhdOfAk6xz777JNEZ5iAKP5FRAkYiI/kwrgggCrdaVzYYIMNamt7E+orrrgi/a1xvOIVr0hWZB1+CwtM7CyjymRbdbw2LrvsssmYoUa59957J5e3DtbWHnvskQYb/PjHP04i0+Qo9Hl815eoggVSvd8Gm1yiuFrzPcOVU7DfsVgqKnkuSHFJgwnXlaGQ29sWW2yRKoCutdZanX/LuI7ThTw4ub9w31mIw/qs1xg3r3vd69KgClVXJXwWRnoTSqNS7jCsMVmyNlxwNyYj7seiGAcaQBYHFhGBq+OnP/3pZKO0n1G1CQ3GfllInPNUwgIZgl52aVi9w0Qs430WRcbnh4l138dvw/XKHU+8uYrE3xe/+MVk7eqsO+ywwwLnzgkbsLKg/Rx66KGptj1jYBTGdZwu8ADEGzPbbLNNSq424T5KvGpD4K4zjnImfmGiN6F0g7l0Nu4pq7INF77cKbjK47joxMFol+E2cCHqKHde0y24E23Yz9QmENlyfGdUbrnllpQMAfFlUWVhGYb3y2Lt8+JIdfR9/OnAqhGqMUMCOmuXAXamst122yXrr8mT6sK4jtOENputQV7GGmus0em7WLXCK9mqlKibrkc1E+lNKLfeeuvisMMOS5u/70u6WpPjYjqZb8H0bNESibrwQB3iUzlsARZZnXvc9/HFp/KgyDqsYtDKLjeLsQzxZU1io402SpZKX8LQJ2K8r3nNa1LIYjrW37iO0wXxyZzkFBrJwtcFIZYcTpmuRzVTmVFRVx0vB6dBcBZddNHB/6bGKNYkyhathIOkVBvcltz5WZZTdRVZ0OW4nXmLXX8/gWIJZMSIq0LV9/Hh2uZOI84s053hHRi0CDUxlRDLuE+SFixesWFWVFNCaiYjDjvdhB7GdZwu5METBHqU+Kf986DnONrGwsaMEkojUU6k6Eiy3tPNoJniMoo1KTuep7NwMdtuOnGXwMlZcvM/q5ZSVwhP2W0vC0kX7J8tgT/84Q/zNH70fXzo3GLSMA1o9uzZKWtu//POO68466yz0nvctZzxJqAXXnhhcfHFF6f/y7LWZcODmYlYKos0M53Q00xlxgil6UCmrGTBESMhOtPB1Irc+dBmTULQXCcG68Y5lRtBGSJpjujpp5+e/k/cTa6eqiXEqnLOYL2N+kia788uGiu3bM2h7+PDwCZLveGGG6b/c6Vf9apXJffxox/9aPqcWOeOO+44eZ248eeee276W1a3PKUomD8Y4DKMA22lK9qUgTDjs+NKws4U7lOhdDFdYJOaZfTMrwN3wyNz5lpOB5ZenqLUNTbJ5RBTleSAzx9yyCGpwwsL6Og22fGPf/zjxbvf/e7JEdSjjNOJf7LQspD57YRpFAhfHghM3neeZfo+foaY7rvvvmnaUXnWQJ7Ubs5mttoF/r/whS+ka0hAt9pqq/83zw/PJMSqPcMNcUZGQhcYC+Le5WlB40rCziTmu1CKFxIimzjUHhOWxnHHHZc6no4r03nwwQe3TstpgyCMak1mxBjf+MY3Frvttlvq3EbYE044odhrr73Ss8Y273tqhPjI6jvnbbfddlpz28RDc0Bdtr1Lxn0U+j5+GWJHFI8//vj0LPlnP/vZNJGfeGbrxQwDc1sNRu69NtE2VSnohxwPhTZtLYUuVqU5rwydUSCivDTTB20GyzZh5d4LHVW3+SXKM8b1ZoUQzXHNm5uKNVlmscUWS/PIPK3ShHikfYzG042nLoxwoT1dxYKtutOmkniKB9z1cLnvOwya+cksMDLEk4nRMHgBHjPNT3i1YWBkrZpAz4tgaNiErrzmvTrR43nqyx7hrG4nn3xy42fHxYzp2SzAj3zkI+lJGM+Hl2MeozIdaxLipR/84AeLgw46aHJOH1i55oWaJpOnAGksbpanGNxwI1/QDotClpuFy5LhXRicgvsO94EXkNGePUPOtS7HHFmaF110UQqXefiAscCbasLnTWiX4ONtmPq1yy67pKfxPCVHqL1nVsQwwfM9VioSnsmb4xiEPXueH73sg/kulC6Kx9PyZmSxiIKLlsXHBfPoVtc4SRWP6E3VmuQGfPjDH56Mlzonlq6kjmSEeaFHHHFEciOdt/l+4K7Yx2+qe+Ik+B+uj9WkdAout85Snikg7iXcoRNyAVmeoyQXgqkhbLTlllumhFrGPRBWMpDlkJn7xZBwj9w/xk15wRLxzuqgZ44swdUfrQFgahpPQ/960pOelL7TvF6eoAVn6nBMczZNwcub4/BGxLw9d95kAU+H+9yi9ONNBdl5552TOOY4CXPe1JJROwhr0MIXmVGsSR2Y0GWRlVwgjNtvv/29JvxyEZ33AQccUBx44IGTiRFWEgGYCo6Rn/Axh3PU324kzqOxY1VDAX0fvyumbLnO0PHKE9nFrA4//PD0qJ5rb9CS/JE5lxnvMq81mDqsPQtuSKbmNj0MHpZ7o48xFDLVOL3ZEdddd12aXkZQ62Lj5mJqBxKG3OhRPDOfdS6+Z6EVyjJWI5EtzRlR7vOooqMT5kf0RrUmjZB5zqVYjZEyZwKHQTA1FLFVaDAew5vKDXPD85QZrsqolinhy1ltC3lUZw30ffwuEMIzzjgjXSeP5HGfcscSxhB2yc+ia/zixDqsDDuLnsAOc82C8cB4sdCLJKuZHNXEKgPCE0PWHF199dVTH8jtAtVVjIRX9C19qWnqHGNEaIvVWHb1u6A9OY++8gQzSihBLPOEZVhBqOtFm441Cd+V3X2T3WWzu6CjS0RkgTcXdNjKOE0Qnjy30SyAUeO0rMQ8TUOjqwpZ38dvg5VA6HgLXG2Wer4/xE/21LUjjKx0GXNehn/zdC2xMZnWoF+IjsGQZWmAEiLj4UnwvP/970+ucnnQzY8tunfVBxn0S+2l7bFI38mVttBGk6BWIcTcdd5SX4sHzzihNCKUF3Pt+hghpmNN6qh5PiRc9DbXo4xGkJdjYzWVH8XsCouvvKTbqAtPlPd3nOr59338JsQdWeumA0FHy2EWiA2LRUL2tZwBF/Piovs+A9k4l98LusHKFE+se+SVy5sHL4+vVo0T/ZeATXd+LI9GG9HH8yaJI46dE63TmZ7XxIwTyipM+i4W5XStSd8xbAJ1F8Rdyt9Xjtl0RWMsP09tbcyux2Gt2T9jPqLjlen7+E2U15gkgrKV5UZtQMxPQDnHaqdigWYXcGFcnea+Jrd/G0EaJSzDe8qeBve63A8MkI7FAKpzixknZeHLW9mVz7jvHnGWrM0bD4R4+44+49e9CCW3zkK9rACxjq7zrDLli6SDsISacDNcNBcYU5k36WLnrDsIwyixMDHJskU6irVVRuYvJ1ws/98186+xZouMdVu21sr0ffw6XJvyGpMyqNXkGAFliaP6HsphA4I6SrA/aIfrbLqPhyn222+/eQbFJgiU5Gd+kKGarOEVGBC1gbp7ZoAsC1/e6gZwg7O8gcRe3vbcc88UlhEDVSKm68A/Kr0IpQadl+9y4lSfmHWBSJZr7HSxXNyksjVpusAo1iSqLr9OzUrtijlceUAgRLm+yKhwabMI+V2mKbWJgsYqgZRFjwtS/i1l+j5+Ffd9YVpjcmHFQx45xq5dED/3vQ19O6/OxUCpG0AZDdzzunbGAi0Ln+l2hLZrUoaBY/YJL4WhImPeB70IpYteXh2Zy0X82sTShTS/Lk/PceN0yiYcU2wyZ8eNLm2fGYbkTXZNHc9TI12y127Q2WefPTmaaSzlWCCcp3lesroqCw5LpHA5LayRLVIxPbG9YdfO62YH5Ok2PqfRDIsH9X38KuKaXdaYdNxs6datZ8jyyANX3Ty9YDium0dIc1XRunvt+stgZ4gfEWyCB2BSevakFCGre6ouG03ZYxiG8yLSPIqu7SvDy2EcMbS6CPyo9CKURgOrx+QVsV1I2UvCU7fijAukQ5kzJ7OW0ananv2tWpNuVp3r1gUd1XdmzjzzzDSp3LnVNS7CbiqL35aX0eda1tU55srYjzARVZnEujgMDDKsYhBf18VnqqLt+2WBVTbMIu26y9g30ffxM2JdXdeYdM9cO8hg5hIIGfcgx8Gqg1AwHPfA4i36lX5CLHMIpYpwlf4DfVY1Rh5HNfZHiIS5ZL9NEIfPqaFTNwgST/2YodAUX9YftBWiN0rWG8Jk+ig3v+4cpktvVRjF+wgPS1Kj1+mIiUnDArIuiAvHgvz0pz+dOlR5BRKPUlmKq25yasaFkfXK1g5rkms36miUcYGZ8R60z260DuocNTIjrFiI30FEVcYzpYVYw6hs0Ye6ZfQJpBhKxnFMg6qrnOdmc0mMwPYTaNcgubAm7nrd9/p+TzG5tmBJi+G0PSvf9/Gh4brX+VluC4pwzYY1YveM1eg8zBjQftwL5yoRxHIRJzX4usejTk2aKbiW2oE1OjHV6oldj6NPeYosJ0QNeAaa8pM0GVa6a859FQfWb3kbPAqvaSvWDT3llFOSl5DbvXui3Q/7He6hPINj8BbsV+2j7r3v8i/BLhs72pI24Dfz+KqZd+8LfbGWhXWmcj3bWGRCbJr94WniB5x00kmTI08bOohJrh5zanOv3EyTXrPbbUUfj0hOFzfEIrPEu2sm3A30RIOnC+rEgCtz9NFHD/53T1jBUw1NsT6julWLNKA2uMPEKFtlXejz+AYVFofvMBVojz32aB3AWBss+Bx6qWIgkmgwo6EPq2F+oD25LtkDmWo97q7HIW7q7ZSNkP3333+eYnFViBIrtEuf9dCAdpHDJk3wqoguYSPILEf4LXTC69qZRxrLMUoir40aFKxRS3gzjBq/USjL5xgfUzWUmujNosyIGzDLrRxuhBoWp9ABPVjvyRyxkvLFqIO+cwsJGqZrTZbx3UYmHZLlwsKpuoIZo6nMvkA092JYBzaKZ0tVh2ct+51NQWtzz9x4YmpQyNNnypi3ZjQ3uOSscFf6Oj7Bs1CINTtdH1ZojlM14d6ZbCw8o51kKwj5PLSlBVUkMb8tSoaHNsyz8xmhE7NRmvqJ+0ywLL1GgOrcZfdj9913T899d02cEkaf48rz1IRYiByX2+v6m7ZYvb/ZorT5vT6TN1at72eFMlL6il33blFWcbMEl1kacMOIiJs5UzuASyTWw/XLcTzuZ50L0YTjEAFCPKqg+yyx1lCcw7ivW9/HH5U8qDoXMwgIbdvgGQxHv7MZHEe5n9oFdzgPXONqF8SPYDo+ceua5b6vmO9CGQRBsKAxs2U8CIJgBhBCGQRB0EIIZRAEQQshlEEQBC2EUAZBELQQQhkEQdBCCGUQBEELIZRBEAQthFAGQRC0EEIZBEHQQghlEARBCyGUQRAELYRQBkEQtBCrBwXBmLF8mHIJloVrWqE/owtaAs2/M3XJsfybrC5eXWG8jOXTLKNo7VW/w0rl1l9tw2+3tJ4l3eC6+WzT+pL5nOoY93UMoQyCMWPVcQXvrLhtoeEmYQFhUXrDArlKoFjvcaZAiJRwuOSSS5L4KRCnjEgdFvm1mLB1WzMET40li+oOuw4+53pZyLeMz6rv5PN1Javzda7D/hbGtup72/XvQu8rnAfB/zcsfmyhW/VplEiwWG4TakflMqtWhLco7kyAdacm1TXXXJNWO1eyQcmTXMKhDJFXV4q4qfJp5XwixQ5TiMzrqhhUF/tVokKtJtdKTSYFygicagFE2ndbAd11dIwyrjMr1GcIqoHJttJKK6XPKhEz7HtHJWKUQdATVrNnJTU5bfbJ9dJnEkRSqRVCpOaRUgvDIEqK0rHciOSyyy6bVkJXUkJNHSKmLEi16qj/X3XVValagO9gOfqbq04o1b9SHkLJB/vVudm+k4gT0rwpJ+E8WLHq9GR3fjqEUAZBD7BiWDbqvCgjMgz1mJRFUVOqCRaXOjOsM/VvVDMUDyxDkLnx2Zqt4jXv2act4uZ957/VVlulUhxNECIDApe8aj0TMvWnWHrlGkiOz5J2bVTnrKvsKcbomDYWt/PuivgwsVUHqlqGeSqEUAZBD0gmEAdCwCKqg3AQv+WWWy5V5ayDGHI/Tz/99OKCCy4oLrvsslSR8KyzzkqlhMuCyL202Uf8rixM/vaa9whQmyvKGiRgVXe3DgW+HHNYbFW9I8cqFz9z3mKZqjE2VfYktAYR5XWrA0MbBNL5jyOpE0IZBD0hlseqYW1JhFQR12NxchXrkhUQw1O+V2xw1113TdU+VaN84QtfmCzRq6++eh7rkGWWXd1yUsXfSsISnHLN7GGMEtPjQi+xxBLJ3RY3VPv7xBNPLGbPnp2EWXihitf8fm5zW7IlC22ToFYRMvB7DUBdfm8bIZRB0BMEgMXEoqxzG7nBEjcsymGI98k0r7XWWkmMQMR8RolXYlkVYd9JWIkoy41lxSr1us+ME3FDoue3co/PP//8ZPmJNxJB50A4qyVvWbg+p5zudPD7lIAWksibDP0555yTBh9JpWGD0CiEUAZBj4jvsYRYjmXLLydxvN+UFSeOBI61VoZYDivfy+1nVYrP3XjjjckqlbFumqIzVfwmm++RZd5kk02StStzvemmmxabb755ElPTeMqCnhMzdefj2ojDlsXPVp0+BPtK9LBc8+Y8iLVrZH7qOAihDIIekcElhtWkTk7icLvbYmiEiEXGYhNnZKGdffbZSSCGwZojlqwrIuXvuoTJOCGO1ViruKXvNiiYL5nJIs9FrkLchA7K4mcrfz7DLd9pp51SSCJve+21V7Hhhhum623KUtWanQohlEHQI6waYlhO6pSTOAStCfE/SRzJHMkbVhWRkUluypT7Xm67ZAYLs+17pgqRZxWKAw6LIXqdZVxOPDknn2ERViHoO+yww6TwEUKC2DUpYz+/fd11102W67XXXjtyIqhKCGUQ9Ew1qSNL3JbEAUG98sor0/6bbbZZspS23XbbYuONN04imWOWdfiePOGacIhRlrPg48KxxVmJ9zAh87r3y2Il3LDkkkum5E9doquMQUYYYdR4pvOSaTfYeNxxOoRQBkHPlJM65j8SzLYkDrigRMLTOj5fFiKi0yQw4pLc3bXXXjtN+jbx2mt9QIzEQIfNc/Q6sS8LuwHCQCG2WRd7zBB32WtWZt0TQU0IWbAoXX8W9nQIoQyC+UBO6ogx6vgszLZHFQmjjSDm5EdGjHOY8InJyTabx+l7bP72WvXpmHHAMhSbFDOtxgP93+sGharQ5Sz85Zdfnq5JOdkFIumznvox6XzUaT6sVYMTke0yH7SJeNY7CMaMDsoSNEUnZ3V1VNN0dHriJ/FRFkr7m+ZSftabuyquN2fOnGRdsorsxwoTd8ur8pS/J7vr9mNNsuJ8nwnkrEoxQY/5la3TLuTzq3vW23f7HmLnO7i52RKUhCH04oXVz7kmYo+ulzCB6VKSOwYB1rD5o44hHus58KrY+ZyNwEr0mCtq81kDkky5Acpnq7MGRiWEMgjGTJ1QQgKDGEg0EMSyWNUJJWHMSRjCaJMNJjzmBxKI6vcQFkKZF5bIEArfx0IjWKzAUWgSSjhn38cFd572Zc3JeltBiWDV4bwc0795UrzP+ZsFuc466xSrrrpqrUXoOkuKEVafyZvPimeay7nGGmu0LkrShVhmLQgWAFhoNuJJNEa1COcn+VxZxHUC1wTLWfx1Kp/tkxDKIAiCFiKZEwRB0EIIZRAEQQshlEEQBC2EUAZBELQQQhkEQdBIUfwf9w+wDlcvmK8AAAAASUVORK5CYII="}}},{"cell_type":"markdown","source":"Now we have the whole dataset into memory. As you can see, the whole operation only takes 1 minute and 22 seconds.\nIt's much faster thanks to the parallelization. Another advantage is that by using the wildcard `*` we can avoid loops and load every file in just one line.","metadata":{}},{"cell_type":"markdown","source":"# Result\nNow we have the full dataset loaded into pandas. You can perform EDA, run models or anything! Using the full dataset as training will give better result than only using a part of it.\nLet's try running a model.","metadata":{}},{"cell_type":"code","source":"df.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:39:52.688916Z","iopub.execute_input":"2022-10-19T11:39:52.689254Z","iopub.status.idle":"2022-10-19T11:39:52.722055Z","shell.execute_reply.started":"2022-10-19T11:39:52.689228Z","shell.execute_reply":"2022-10-19T11:39:52.721139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to remove the last two columns added by Dask:","metadata":{}},{"cell_type":"code","source":"df = df.iloc[:,:-2]\ndf.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:39:52.723821Z","iopub.execute_input":"2022-10-19T11:39:52.724273Z","iopub.status.idle":"2022-10-19T11:39:55.269230Z","shell.execute_reply.started":"2022-10-19T11:39:52.724234Z","shell.execute_reply":"2022-10-19T11:39:55.267564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Putting it all together","metadata":{}},{"cell_type":"code","source":"import dask.dataframe as dd\ndtypes_dict = {\n    'game_num': 'int8', 'event_id': 'int8', 'event_time': 'float16',\n    'ball_pos_x': 'float16', 'ball_pos_y': 'float16', 'ball_pos_z': 'float16',\n    'ball_vel_x': 'float16', 'ball_vel_y': 'float16', 'ball_vel_z': 'float16',\n    'p0_pos_x': 'float16', 'p0_pos_y': 'float16', 'p0_pos_z': 'float16',\n    'p0_vel_x': 'float16', 'p0_vel_y': 'float16', 'p0_vel_z': 'float16',\n    'p0_boost': 'float16', 'p1_pos_x': 'float16', 'p1_pos_y': 'float16',\n    'p1_pos_z': 'float16', 'p1_vel_x': 'float16', 'p1_vel_y': 'float16',\n    'p1_vel_z': 'float16', 'p1_boost': 'float16', 'p2_pos_x': 'float16',\n    'p2_pos_y': 'float16', 'p2_pos_z': 'float16', 'p2_vel_x': 'float16',\n    'p2_vel_y': 'float16', 'p2_vel_z': 'float16', 'p2_boost': 'float16',\n    'p3_pos_x': 'float16', 'p3_pos_y': 'float16', 'p3_pos_z': 'float16',\n    'p3_vel_x': 'float16', 'p3_vel_y': 'float16', 'p3_vel_z': 'float16',\n    'p3_boost': 'float16', 'p4_pos_x': 'float16', 'p4_pos_y': 'float16',\n    'p4_pos_z': 'float16', 'p4_vel_x': 'float16', 'p4_vel_y': 'float16',\n    'p4_vel_z': 'float16', 'p4_boost': 'float16', 'p5_pos_x': 'float16',\n    'p5_pos_y': 'float16', 'p5_pos_z': 'float16', 'p5_vel_x': 'float16',\n    'p5_vel_y': 'float16', 'p5_vel_z': 'float16', 'p5_boost': 'float16',\n    'boost0_timer': 'float16', 'boost1_timer': 'float16', 'boost2_timer': 'float16',\n    'boost3_timer': 'float16', 'boost4_timer': 'float16', 'boost5_timer': 'float16',\n    'player_scoring_next': 'O', 'team_scoring_next': 'O', 'team_A_scoring_within_10sec': 'int8',\n    'team_B_scoring_within_10sec': 'int8'\n}\n\n\ndf = dd.read_csv('../input/tabular-playground-series-oct-2022/train_*.csv', dtype = dtypes_dict)\ndf = df.compute()\n\ndf = df.iloc[:,:-2]","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:41:41.486936Z","iopub.execute_input":"2022-10-19T11:41:41.487367Z","iopub.status.idle":"2022-10-19T11:43:06.972645Z","shell.execute_reply.started":"2022-10-19T11:41:41.487289Z","shell.execute_reply":"2022-10-19T11:43:06.970796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Saving to feather format\nWe save the result in a feather format that will be later posted as a new dataset.","metadata":{}},{"cell_type":"code","source":"df.reset_index(drop = True).to_feather('train.feather')","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:43:06.974938Z","iopub.execute_input":"2022-10-19T11:43:06.975942Z","iopub.status.idle":"2022-10-19T11:43:15.605655Z","shell.execute_reply.started":"2022-10-19T11:43:06.975892Z","shell.execute_reply":"2022-10-19T11:43:15.604177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Running a LightGBM model\nNow that we have the data loaded, we can run a LightGBM model using the whole training set.","metadata":{}},{"cell_type":"code","source":"import gc\ngc.collect()\nfrom lightgbm import LGBMClassifier\ndf.dropna(inplace = True)\nto_drop = ['game_num','event_id', 'event_time', 'player_scoring_next',\n       'team_scoring_next',\n       'team_B_scoring_within_10sec','team_A_scoring_within_10sec']\ny_b = df['team_B_scoring_within_10sec']\ny_a = df['team_A_scoring_within_10sec']\ndf.drop(to_drop, inplace = True, axis = 1)\nmodel = LGBMClassifier()\nmodel.fit(df, y_a)\nprint('Model A trained')\nimport pandas as pd\nX_test = pd.read_csv('../input/tabular-playground-series-oct-2022/test.csv')\nX_test.drop('id', axis = 1, inplace = True)\ny_a_pred = model.predict_proba(X_test)\ndel model\ngc.collect()\nmodel = LGBMClassifier()\nmodel.fit(df, y_b)\nprint('Model B trained')\ny_b_pred = model.predict_proba(X_test)\n\nsubmission = pd.read_csv('../input/tabular-playground-series-oct-2022/sample_submission.csv', usecols = ['id'])\nsubmission['team_A_scoring_within_10sec'] = y_a_pred[:,1]\nsubmission['team_B_scoring_within_10sec'] = y_b_pred[:,1]\nsubmission.to_csv('submission.csv', index = False)\n\nprint('Submission saved')","metadata":{"execution":{"iopub.status.busy":"2022-10-19T11:57:06.139745Z","iopub.execute_input":"2022-10-19T11:57:06.140142Z","iopub.status.idle":"2022-10-19T11:57:06.148197Z","shell.execute_reply.started":"2022-10-19T11:57:06.140107Z","shell.execute_reply":"2022-10-19T11:57:06.146329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Scoring\nThe current scoring is 0.201, which at the time of writing puts us in the top 40% of the leaderboard. That's not bad for running a model with default parameters on the original dataset.","metadata":{}}]}