{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Reading In Datatable (RID)\n![py_datatable_logo.png](attachment:py_datatable_logo.png)\n\nThe training dataset of this competition is large-ish in nature and the plain vanilla ***pd.read_csv*** will result in an out-of-memory error on Kaggle Notebooks.\n\nThis notebook shows how you can use [Python datatable](https://datatable.readthedocs.io/en/latest/index.html) to read the complete training data and convert it to a pandas dataframe in under a minute.   \nJust reading the dataset from binary format into datatable takes less than a second!\n\nThere are other options to consider as well and some of them have been described in [this notebook](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).\n\nYou can try out or learn more about datatable from [**DatatableTon**: *💯 datatable exercises*](https://github.com/vopani/datatableton)","metadata":{"papermill":{"duration":0.012477,"end_time":"2020-10-08T17:18:38.675317","exception":false,"start_time":"2020-10-08T17:18:38.66284","status":"completed"},"tags":[]},"attachments":{"py_datatable_logo.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAb4AAAGICAYAAAA+roVZAAAACXBIWXMAAAsSAAALEgHS3X78AAAgAElEQVR4nO3dTYxW2Z3f8VPvRXVBAc2L2wSqCpt2Tzsx2FIykUdxYylDhmy6Rko0LY0s04tn7XK2LEwvWGXR9PpZGOQNO8MKiUhpcJRVFqlSonimW2kgbZwhjQNlQ72/jC6c2zzU83b+955zz7nnfD9SazQzwPPc57787/+c87t3YGdnR8Gvc5dvzSilLiilzrIrgOgtKKWu3r54foFd7QeFz7Nzl2/NK6U+TvpHANL0ye2L5+fZ99Wj8HlE0QOSd+32xfMXUv8Rqkbh80QPb95LcuMBtPrx7Yvn7/CLVGcwlQ0N0FzqPwCAF+j4Kkbh84fCByAzw69QLQqfB+cu39qvlHovuQ0HgABQ+PwgtgAgd59foloUPj8Y5gSQY2FLxSh8ftDxAcjd4JeoFoWvYucu3zqjlJpOaqMBdLN4++L5p/w61aLwVY9uD0CObs8DCl/1mN8DkKPwecCTWyqkYwxPktlgAL0s3b54fj+/UPXo+KrFMCeAHN2eJxS+ajHMCSBHjMETCl+16PgA5Oj4PKHwVYQYA4AWxBg8ovBVh24PQI5uzyMKX3WY3wOQo/B5RJyhAsQYALQgxuAZHV81GOYEkKPb84zCVw2GOQHkiDF4RuGrBh0fgBwdn2cUPseIMQBocZcYg38UPvfo9gDk6PYCQOFzj/k9ADnm9wJAnMEhYgwAWjy4ffH8DD+If3R8bjHMCSBHtxcICp9bDHMCyDG/FwgKn1t0fABydHyBoPA5QowBQAtiDAGh8LlDtwcgxzBnQCh87jC/ByDHMGdAiDM4QIwBQAtiDIGh43ODYU4AObq9wFD43GCYE0CO+b3AUPjcoOMDkKPjCwyFzzJiDABaEGMIEIXPPoY5AeQY5gwQhc8+Ch+AHMOcASLOYBExBgAtiDEEio7PLro9ADm6vUBR+OxiNSeAHPN7gaLw2UXHByBHxxcoCp8lOsYwFcXGACiLGEPAKHz20O0ByDHMGTAKnz0UPgA5hjkDRpzBAmIMAFoQYwgcHZ8ddHsAcnR7gRtO/QewhBhDQNZX11L/CeDRP5s5/Ltf/ebzOl0TFn7yo1NJLcRhqNOCc5dvPWVFp1/b29vq+dKzped/fD66s7OzJ+XfAn797G9+qMZGa9dT3FRKzf/kR6fuB/BdnGOosyRiDP5lRe/x77/6/8+Wnk1R9ODTm1MTD2tY9DLvZ53fr37z+ZkAvotzFL7ymN/zbOnx00dbm1sHk/4REITvnjw6UeM9kd3A3/nVbz7fH8B3cYrCVx6Fz6OtzS21urx6NNkfAEE5+c0DB2q+R7LidyGA7+EUha8EHWM4XdsNiMDK8xViJAjC2MjwkyMHJ2PYGdHfzFP4yqHb82x1efVZ0j8AgnHsyD6OxZqg8JVDjMGzjbX140n/AAjG6VNvcSzWBIWvHDo+j1aXV5PddoTnxNFoFndHH8Cn8BVEjMG/1eXVh6n/BgjDNw/t+7KmMYZOon/ANoWvOLo9z1aXV+u8dBwR+fbxN6NY1ZKlg37yo1MLAXwPpyh8xVH4PMpiDDvb23VfOo5I/NnM4ViOxSRep0ThK4AYg3/EGBCKLMYwNTkey/5I4gHbFL5i6PY8I8aAUHz7+JvLEe0MOj50RYzBM2IMCMXbJw4di2RnLKbylgYKXzF0fB4RY0BITh1/M5b9kUS3pyh8csQY/CPGgFBkMYaIdgaFD13R7XlGjAGhIMZQTxQ+OQqfR8QYEBJiDPVE4RMgxuAfMQaEghhDfVH4ZOj2PFt5thzT0nHUGDGG+qLwyRBj8Gh7e1ttbmzGsnQcNUeMob4ofDJ0fB6tr64nu+0IDzGG+qLwGTp3+dZZYgx+Lf9pOaal46gxYgz1RuEzR7fn2fraeixLx1FzxBjqjcJnjvk9jzbWN4gxIBjEGOqNwmfg3OVbM8QY/FpbWSPGgCAQY6g/Cp8Zuj3PiDEgFMQY6o/CZ4b5PY+IMSAkEcUY7qYWY8hR+MzQ8XlEjAEhIcZQfxS+Pogx+EeMAaGILMaQ5PyeovAZYZjTM2IMCEVEMYYHKcYYchS+/hjm9IgYA0ISUYwh2W5PUfh6I8bgHzEGhCKyGEOy83uKwtcX3Z5nxBgQishiDHR86Ir5PY+IMSAkxBjiQeHrjY7PI2IMCAkxhnhQ+LogxuAfMQaEghhDXCh83THM6RkxBoSCGENcKHzdMczpETEGhIQYQ1wofB0QY/CPGANCQYwhPhS+zuj2PCPGgFAQY4gPha8z5vc8IsaAkBBjiA+FrzM6Po+IMSAUQ4MDK8QY4kPh24UYg3/EGBCKowf3Po5oZzDMqVH42jHM6RkxBoTie6e+cSiSnUGMoQWFrx3DnB4RY0BIpr+xf08kO4RurwWFrwUxBv+IMSAUk3tGHxFjiBOF73V0e54RY0AoTh47OBTRzqDja0Hhex3zex4RY0BI3j15JJb5PWIMu1D4XkfH5xExBoQiizGcOLo/lv3BMOcuw0F9G4+IMfinYwzHU/8dnNlZV2rjfz1RG3+/rNTOZqRbacW3j+0ZVF/d3a77duwd2xk+/872T9W9zZRHs+6/KP6z17++AaDwvcIwp2fEGBza/GJZrfznQaW2shWzrJrtY/bQ9ora3Kr9is7pw6NPxobGzgTwVXx6Tyn1U3Xvg7svrvOz158y1PkKw5weEWNwaOv3Sq38pwmltqJZouja9JGRKGIMxw8MPwvga4TivXyRD4WPGEMQVp6tPEr9N3Bm5fZSpFvmxOT44KOpiTgWdB6aHGLq4HWn1b0P5il8L9Hteba6vBLT0vFwbP9BqZ015q4FTh4djeJYPH6AmawuLlD4XmJ+z6MsxrC1uRXL0vGwbHxGJy307onxKI7FEweGHwbwNUJ0msL3Eh2fR6vPV1eS3XjXNr6gkxYYGlQrJw6N1Ob79nL8wPBEuN/Or+QL37nLt+aIMfi1urwa0xPww5HFF3ae0UkLHN0/EsWxODk2qEaGBlgs1tli8oWPbs+/9dU1Ls4ubH5BJy30vek4hjlnDg7zzNvu7lD4mN/z6kWMYWcnlifgh2Xj7+ikhYgxJOFG0oVPxximA/gqySLG4NDWV3TSArHEGEaHBogxdLekZq8n3/HR7XlGjMGRLLSutumkBWKJMRzdxynVAwF25vf8Isbg0MbnDHMKxRJjOHV45MsAvkaoXjyvM/XC934A3yFZxBgc2vxyK9ptcyCmGMORvUM887a7tDs+HWOAR8QYHNn+k1I7z49GuW2OxBJjODhBjKGHRTV7PXtTQ9IdH8OcnhFjcGTrIZ20UCwxhm9OEWPo4eu30Kdc+Oj4PCLG4BAxBrFYYgwnD48sB/A1QvX1+/iSLHzEGPwjxuAQMQaRmGIMU+ODxwL4KiF6EWPIv1eqHR/dnmfEGBwhxiBGjCEJd1o3MtXCx/yeR8QYHCLGIEaMIQk3Wjcy1cJHjMEjYgwOEWMQIcaQjLQ7PmIM/hFjcIQYgxgxhiR8HWPIpdjxMczpGTEGR4gxiBFjSMKd3RuZYuGj4/OIGINDxBjEiDEk4cbujUyq8BFj8I8Yg0PEGESIMSThtRhDLrWOj27PM2IMjhBjECPGkIS2oqcSLHzM73lEjMEhYgxiP/jWHmIM8Wsb5lQJFj5iDB4RY3CIGINIFmM4MjVco2/cHTGGntLu+Igx+EeMwRFiDGLTR0aJMcSvLcaQS6njY5jTM2IMjhBjEHv7rTFiDPHr2O2pxAofHZ9HxBgcIsYg9p1jY8QY4tdxfk+lUviIMfhHjMEhYgwie/cMPh4bGajRN+6MGENPHWMMuVQ6Pro9z4gxOEKMQeydY2NRLAQixtBT16KnEip8zO95RIzBIWIMYt89MR7FQiBiDD11HeZUCRU+YgweEWNwiBiDCDGGZKTd8RFj8I8YgyPEGMSIMSSha4whl0LHxzCnZ8QYHCHGIEaMIQk9uz2VSOGj4/OIGINDxBjEiDEkoef8noq98BFj8G/5j8+5OLtCjEGEGEMSesYYcrF3fHR7nq2trLH4wgViDGLEGJLQt+ipBAof83sebW1uqa2tLRZfuECMQYwYQxL6DnOqmAvfucu39hNj8GttZY3FF64QYxAhxpCM5Ds+uj3Plp8t05W4QIxBjBhDEvrGGHIxFz7m9zzbXN9g8YULxBjEiDEkwWiYU9HxwZX11TViDK4QYxCLJcbw9tFRhri7S7vwnbt86wwxBr9Wnq1wcXZl69HxODfMjZhiDG+MDjCK0lkWY1gw/cOxdnx0e54RY3Bk02gKAy1iiTGcODjMEHd3xt2eirjwMb/nETEGhzbvPYx22xyJJcZw4sAwoyjdGa3mzEVX+HSM4b0AvkqyiDE4tHl/ItptcyCmGMNbU8MMc3aXfMfHMKdnxBgceRFjWGcpu0BMMYbBAcVisc6yGMNTyV+IsfAxzOkZMQZHNv83S9mFYokxzB4aeRTA1wiVqNvLDOzs7CTz6wAAkMob2AEAeIHCBwBICoUPAJAUCh8AICkUPgBAUih8AICkUPgAAEmh8AEAkhLHQ+w8aTQaZw0ekXal2WyKHqcDoDqxnMdcj8xR+ApqNBoz+lE5U33+hew9MleD3RAgYfo8/tTgF7gjfQNAlWLZjqow1FncnEHRyxi/HBFA5Uyf7Rv6eWy6Hcm/0FFR+EoxeQvEUrPZpPAB4TIpGIs1GB40uR5l25F84VMUvlLeN/jL4qeGA6hGo9EwfXdn0Oex3g6T61HyQ5w5Cl8BjUbDdFiBAw0Il+m7O0O/gY1lOypD4SuGAw2oP5Mb2DpMV5huBzfiGqs6i4llXsA5PQxzpsfNQnZRucNvJaNX8WW/6UyHv/hU/6bML/dmcgNbh5tXk+2g6LWg8AnpC860wd+i23spO+FO9/kzP8/yRVV9obrTQ+2/NtiMgdR/q24ajcYZw/M46IIh2A6uRy0Y6pQznd9L/kDTF+h+RU/pDgXmLhj8ySV+z55Sm9+j42tB4ZMjxmCOuVA3WFFcXizTFabbQYyhBYVPjouOOeZCLWNFcXmRxRhMtoNjYRcKnwAXHXPMhTpDF10eMYbEUfhkONDMMRfqBl10ecQYEkfhk+GiY465UMvooq0xOY+JMUSMwmeIi44Yc6H20UWXpJf/mzxcnhhDxCh85rjoGGIu1Bm66PJiOY+JMZRA4TPHRcccc6GWCR5EzG/aGzEGxP/klpZHZvWz0Odg937R0W9YdumpaeHu8btmJ+TPDL/jfKPRMLkj7bdvChMcH2XcN70A6SGs/bv+z9n/fsnw8+d0x93v9zLe10V02Q6rpIs29L42eaBCx/NYn3+S7brj4tgNJcZQZB+HstAmusKnd8ac7jpMDo7Wv7ukD/qrrTvIx9CdnlPMt+Os4byEjc/9sNls9nxjvP5uCxa+0y/0f/0s2ipOel/mFzDR8VHCkskFotFoXBHcNHQzZfg4s+zzft5sNks/Kk5fiOf0f6ZzT6UV+P7iYU59rF8SvHi61S/0v3EzeySfxYt+5SMqLc+GLbWPG41G9j8e6GvljWaz6WWEIprC12g0ssc4zRve0XWTHdg/zf5rNBqfNJvNef3nKpsX0HeVlyq8KO9mcnc6X1UhtkFfmOf1o74quSgXVLboSZXqRPSxekGfMz5IO0rj6Qp9zFyxtG3ZaNH7ugBesNABVhZj0Pt43nDEy9R0y3U2K4LzVRfA2hc+3eFdcVAoftZoNLLhoEuCtxsXPqD1HdVVjwUvZ3KymN4I2FL4BNY3RFc8F+q+J7VgVMGmQr+r5aJQhvRiaRRj0NeUqyVvojvJisedrJiULH7OYwz6enTFcsHrJCuCv7Y1+mCq1otb9EXtvzssFvNVLBvW27EQQNHrW7wFsQ6bxL9tdnFuNBrZ3/tlAN2pyUXI9fztboUWPugO4H4ARU+0kEwQY7hv+EaRok7rolpIRdejOX09cl30Wn2sr4OVqG3h0/Mhv3T8MVOC1+UUOtAajcbVQC7OynAbqu5MxEM2+uJwp+ITt5cQf1dxR6AvTJ/W6FhtZfr7Xqhg+94vsVDN6fye3se/9rSPr+jRBOdqWfgajcalCudDTLqwQjEGXfR83zm3MjlZqu5MpEVvxvEdu1QUXbS+ILq+0ZSQFm7TwlfVfija3TiLYzQajXnP+3hKzyc6V7vCp09Ak5WAVSoyFDcfWNF7YFi8q+6ijH9bfbd4I7CFN7XvovXQV0hFTxU4LkK5EcqJbyBdvlVCX1c/lv49Byo5F2q1uKVlwjU00q7kbCAHWSvTbXhQ4V3xXeFJ7GJBQlkm37/qsLTxXXXLoquQSDuashfTu33+/0U69iLnkJNhzpYFgiGo5Pyt26rOq8K7+Wv6gn6/W5BUn9gzOptyoeAPL73DklxIlvSfv6NDx5I79f2CYT+jbWg2mzNdPmfB8GTOLiJzDkK9c8JudFH/rguSgLl6dePyqcEfNRoC17nJtmNCb5NJJi87Ri45WhVn/ZzrRi9GcvGgCGl3lZ9zVwUPdKhiFaSrt0pIVj0bXY92XVfnQ4sR1abw6YuN6arHuzov0/dipv9MvpLrip4/lAyl3hWe3JcEB8En+oJWqEhkf6/RaCwYFr7Cy5/159w32K7spHQ1R2h60V/Sx0aZ3JDphaVsNsn0c664KHq68Jqeczd1HqvMI7JMFyS5Wtiiit6Y6e2eExTvIkzOnSJzt6b7+CMd7+qrw3U1qPUMder4TB/ZdK3ZbBZeFpvtWGGRlc41mA4z9X2CiiGTk0VUvHfz/UZrffKa3Exkw7RnLHSbVT0g2PcDlU2LaalzroX15+EKYgzK0nbYDnu/IIgxSI850+tq2etRUGsaarG4Re90kwvroqUTULKDJQea6VLpT2wUvQpfXeL7SfHGz7G0NMRaVUHy9mB0ffNnNHRt6ZxTnrs9K9cOQce7JPynrc/vCW4YS1+P9Hm3WObfsKkuqzpNuyRbJ6DpwWu6EjJnsh0PBBfyfqLvTAQX6I8sFgjnT/LRfD4YvdJzzuHzcCX5vSpJj0UXMQajOUOL1yMT/RYSWVGXwmeyg256eCWQZKGJafd1xeLCD5PfTVq8O6mqEHRiesGyMgdWVRft852GglcgXbP42hsXHY1pjOGarWuHPj5MSBepWZ1K0ItPTPbxVYvnrcm+qOQaHnzh03f0JsODNpdct61c7EJycTO9kFnZjqpeXSIoBK6WxJv8rtcsnrxVPRnf5/ye6TbaXFDjq6NRlo9NF8eHz3/T1vXI+g1BGXXo+IwOXstP93YxRFj1CzCrukB7m98TLFyw+dkpdNGuls13JHhyjYsYwwPL74hzMS/rYn9UdRxLPk9R+F4xOZBuevhM45WQZV+AWVBVB1rVJ5D0s5Wt3zXALtrV/J71ZfN9uNqPJv+u7Qutrxyii3+z6hvGUqvLJYIufPpO0GkGrcNnurjo+Bi2CinG4OouLvUu2sViIVfL5nux3tEIRgNsvqzV+s2mi/0hmD6yecNY9Y1/T6F3fD4uAC6G7kyHXGxOsIcUY3BxgU69i3YSYwi4i3YVY6h6mFMFML9X9fRRUMOcqgaFz3QYzdbKMsln2h5P93ECVhVjcHFAp95Fu7o7DrWLdhFjsD205uK3M1m1LP03q54+qvTG30QMHZ/NoQrrQ3c+hlwCW4Dhaty+0nB3Cg8DCLyLdhFjsHntsL5Ax8V2+Jg+8jTX2lOwjywTjEOHPlRRaVeU0AIMk9819CHwTnzGGELtol3FGKq+uKsA1gaYHsdn9XN+y7rk4ca/r5Cf1ekjxuDihKl6yIUYg5vPpot+qQ5dtI+hNRe/na9ohNKrU6t892alHV/IQ53EGIohxvAKMQaZULvoVGMMLvaHq7ejlFFZjCEXZOFLMMZQ9XAtMQYZumg3n2395imyGIPV/SGYPqpapcOcKuCOL9S5BkWM4QViDJ3RRRtyePMUS4zB55xh1Sod5lQBFz7TgkGM4XXEGF7hYQAysXTRscQYXPybIQ5zVhpjyIVa+Cp9HQsxBrHQF2DQRQvE0kUTY+j5b5pOH1XN1cPrewpuVaen17EQYzD/nDoswKCLdvPZoXfRxBi6Mz2Olyy/daOX+zZeuF1EiHEGox1EjKENCzBeoYu2/9l1iDH4mCONKcaQvQT7TNUrLH0IcajT6KWzlj+TGIPdz/G9AIMuWiaWhwFU/cYBFVmMweZLsIMWVOETjJcTY2jHAoyX6KIFIooxVPrGAVWvGMOch30ctNA6vlhiDJV2RSzAcPbZdNGvEGNoV5cYQ3APifYttMLnI8Ywb/iZtsfobZ6AJk9wt/GZdViAUauXZyYYY3BVaH3MkdYlxhDcQ6J9C63wmYyX2xzmPOvgJY+VDh3pC6dJ4UthAYay2EVfqGjoLLW3MZjcaCrhOWe6VN/mMKfptcN41aLDGEPVLxYOXjCFTxcMEza7vUuGf87FsmFb2zFfRaEVnEA2908ro6Jr44P0Bcj02Ch7wfBx3Hv5bD3XZHJhl948+bh5MB1l8T3M6fP4ClZIHZ/pDrKi0WjMGw4xKeEJU9l26JuFXxj+8bJ3uzMl/35ZJkXXlkuGn1dVF+2K6WeX7uD1zYRp9xN0jEF3ez81+KN3hdMyLmIMlV5X6yKkwmd6YS19AdbDWB8b/nHp3Wcl26GLnqQgl73bNb1IWj/RBKMBpT9bHxs/M/zjNjqI/YZ/zueNR6nirIveHcEDkoONMehtMS3M0nC2ixiD6XFjehxGIfQ3sHcyJ7gQtmk0Gtnd/C8Ff8VVduqSPonE9B2n5EJS5Ws/TuviYZPp7zSlO3mxbF80Go0rHo4N08dIFT5eejA9jy6UOFbzGzTT7Qw2xqC3ZcHw8xYlTyVx+HAG08JX6Lypq5BfRNtNdnDcyS5wpgeWnp+a0ztXOmTmatJ3umU7+n6GvvDM6bkF0yHaXNWv/fil/s2vWl6Ba+LjfFjN5LP1BWdOMFfaqsoFAdnxsqCPl6r355T+7EuCc25O/64mQ4KtXMUYCtPHyLxwW6SFxOeK6cx7jUYj+7cveDhnK1fHwqf0iZhdXLO788U+cxDSItHKdbYluwv+tNFoqD4LM0wXlnRT+fuu9NzjLxqNxpK+S97tqT7JTDpRabeaf/aDPpP2ZY4NW130oqAbyo6BX+vjpdtxf6PZbJo+a3FB8JbtacNzrupj1XQY9mqj0ZCey6arLHf7xORmdhdX0Zn7guM8+3P3DM6bou40m03TRWNOhVT4il5EXD5xvMjd1ULBC2qZi3AvtnKPdwQLaVpN9dg2o3nK7OZDX+ylph0uirF1M3G/4DHc6++YFr5an3PCNw70Og5tWhSsCM63w2WspMi57+q8eU/627gS0hxfiDmSIgdaaNth5QKt72CXbPxbLSR34Lafz1qWrf1suxuXXOh8jAT04irGUJXs/Jgr0JW5HOYMaR/bvn4UFkzh00OKDyr6OKMdUGQuRf+dKrbD9CCyWYhtvq7kpvACUdWrUkx+V5tD4DcsXxCMj1k9EmAl+9iH6fa5iDFUJdvGswVHV5w9Ukz/nSr2sYlginBoqzqrWFn0keGfK9NhuN6ORdODyPJCiCsWi7qoIOuO0/UJ/POqH+ari7/N40X63ao4Vk1vWlzEGKqQbeNMiZsh148UC2J4MaTRsKAKn75IX3P4ER/pguH04qa345Oif7+Pm/qEr/z1TfoiPWepQylSkOf0RcaFDwVzXlbvXPVKSRvHvXjxg75Yf2jhsztZ1Meq9Y5GEGNw7aNms1n4HXZVvGNS3zS62scSdHzdNJvNC4KuzFTWpfy1XlFUydPom83mvO4gbA1jZf/Oz5vN5pxeOeflNSP64nS2ZPdVaMGNvriU/ezdsovz93Xx8fbcTH3cf1jyeCl0zOpt/2vLQ66ftPyePt/G4Ep2ozJrYZViJTEGR/tYwuaDzksLMsCuD6ZZfXCV2VHZRe3DZrM50zLkZxLalT5qqCO9rPyMvggUHSJ8oG8EZlqWqZuEiZdc3WFlxa/ZbJ7VF+oiXaX0iRatn/1Uf/aPS3a0N/XN0JmWTsMk7HvN1QmsL04zen9Lj5dS+1ufH0U/O/dAH+tZQZjXv5NpSF56TPh4FNeivpnNts9W3s3kZks6H96RpX1cVOFz3oWBnZ2dkL5PR3pYY0bwFILs7mih08HSEljuxcmbiPVnzxietAt6G9pOLr38ud/czI0q36+l99EZg6J8X/JEC8uf/VT/ph3vnnXoute+eaqD8ZXcuerl+mcMjxer+1v42Qt6v3b8fP00nV775o4092Z4HttwX2+bk/kpg+1wdswJ93EZ1s/5smpR+AAAsKWOz+oEAKAwCh8AICkUPgBAUih8AICkUPgAAEmh8AEAkkLhAwAkhcIHAEgKhQ8AkJRK38B+7vKt0F4cmZSNtfXJx//38bPUfwcn1v7rhFr/n8sRbpkThw8ePHr2B//0rRi25c+P/2np37y9dC+ArxKiBTV7PZiHU+cqK3znLt/Kngv3aVWfh3Zrq+tPlFIH+Gkc2NlcUUrtiW67HFlZfvpodu9vj8awLe8e3lN1D1Efs9cHQvyuVQ510u15tvJsmY7Ela2HFD2Bk0dHh2rzZfs4ui+aTbHN6vtAbaqy8Pl+f1bStre31ebG5rHUfwcndp4/Utt/inDD3Hn3xPihGLZjanzw4ehQkE1NCIJ54/pudHyJWH2+upL6b+DM5pfc8gsMDaqVE4dGavN9ezl5eGQi3G/nXTBvXN+tksKnF7WYvDEcjqwurz7mt3Vk4/MoupeqHN0/Es2x+M2pYebMO3ugZq/beFGvE1V1fAxzera+usbF2YntFbX1+wi3y53vTccxzDkyNPDk4ASJsC6C7fYUhS8NG+sbamdnh8UXLmx9RSctNH1kJIpj8cjeIaJB3QU7v6eqKHw6xjDt+nPQ3cqzlUf8PI5s/B2dtMDk+OCjqYk4pkRPHR45HsDXCNGSmr2efMdHt+fZ6vIKiy9cIcYgQowhCUF3e6qiwsdqTo+yGOgQoSIAAArwSURBVMPW5hZdiQs7zx4TY5AhxpCEoLs9VVHhe7+Cz0AXxBgc2vhiK9ptc4AYQzLS7vjOXb7FMKdnxBgc2vgsikduVSWmGMPMQWIMXSyGHGPIue74GOb0jBiDK9sravsPcW6aIzHFGCbHiDF0EXy3pyoofHR8HhFjcGjzd3TSQrHEGI4fGOaZt90FP7+nXBY+Ygz+EWNwaPMenbRATDGGEweGeeZtZ1mMIfmOj27PM2IMDm1+QSctEFOM4fgBXkHURS2KnnJc+Jjf84gYg0NZjGFnPdrNcyGWGMOhyaEvA/gaoarFMKdyXPiIMXhEjMEhYgwiMcUYjh8Yngzga4Qq7Y6PGIN/xBgcIsYgQowhCbWIMeRcdXwMc3pGjMEVYgxSxBiSUJtuTzksfHR8HhFjcIgYgxgxhiTUZn5PuSh8xBj8I8bgEDEGEWIMSahNjCHnouOj2/OMGINDxBhEiDEkoVZFTzkqfMzveUSMwSFiDGLEGJJQq2FO5ajwEWPwiBiDQ8QYRIgxJCPtjo8Yg3/EGBwixiBCjCEJtYox5Gx3fAxzekaMwRViDFLEGJJQu25POSh8dHweEWNwiBiD2HeOjRFjiF/t5veUzcJHjME/YgwOEWMQ2btn8PHYyECNvnF3xBi6ql2MIWez46Pb84wYg0PEGETeOTYWzUIgYgxd1bLoKcuFj/k9j4gxOESMQey7J8ajWAhEjKGnWg5zKsuFjxiDR8QYHCLGIJLFGI5MxdElEWPoKe2OjxiDf8QYHCLGIDJ9ZJQYQ/xqGWPI2er4GOb0jBiDK8QYpN5+a4wYQ/xq2+0pi4WPjs8jYgwOEWMQI8aQhNrO7ykbhY8Yg3/EGBwixiBCjCEJtY0x5Gx0fHR7nhFjcIgYgwgxhiTUuugpS4WP+T2Ptja3iDG4QoxBjBhDEmo9zKksFT5iDB6trawRY3CFGIMIMYZkpN3xEWPwb/nZMosvXCHGIEKMIQm1jjHkynZ8FD7PNtc3GOZ0ghiDFDGGJNS+21MWCh/zex6tr64RY3CFGIMYMYYk1H5+T5UpfOcu3zpDjMGvlWcrXJxdIcYgQowhCbWPMeTKdHx0e56tr60/T/oHcIkYgwgxhiRE0e2pkoWP+T3Ptja3jiT9A7iy/fQhMQYZYgxJiKLbU0UL37nLt/Yrpd6z/3UgMTg4+Ed+MAc2/n4ium1yKKYYw6nDIwxxd5d8x8cwZwCGR0doS1zY/JKl7AIxxRi+sW+IIe7OshjD0xC/WBFFCx/DnAHYe2Dv8dR/A+t21p8QY5CJJcZwdN/QY2IMXUXT7Sk6vnobGR1Rk1OTS6n/DlZt/cOziLamEjHEGIYG1drZU3sY5uwu7cJHjCEsew/sm5p6c//KwMAAjy6zYeO3dNECMcQYDkwMPv53358cGx2KI47hQBZjWIhpg4rMSNPtBWZi78Se8TfG1frqehZqf5D671HUzub/e0PtfLUyoIa367kF1fur05ND7xwdreUxt2d0YF/2aLLJsUE6vd6i6vZUwcLH/F6ABgcH1fjEePYf3XhRz//LQzWyfVypkXp+fw/+9l/uVd86xO8VuWhiDDnRUCcxBkRt43NiDALDQwMrFL0kRNfxSef4GOZEnLaXlNpZI8Yg8IN/Ek+MAV1FFWPISQsfw5yI0/pvn7BnZf7i5DhzY/GLrttTdHyAtvEZMQahf/WtccLe8Uu78BFjQNQ2f0+MQeDwJGHvBEQXY8hJjly6PcRp4zN2rNCPvj0ezdsY0FWU3Z4SFj7m9xCn9c8esmdl/vV39kTxNgb0FF2MIWdU+IgxIGrEGESIMSQj+Y6PYU7EiRiDGDGGJEQZY8iZFj6GOREnYgxi//bdCRYCxS/abk/R8SF5xBjEvndstGbfGAWkXfiIMSBqxBhEjh8YfkiMIXrRxhhyJkcw3R7iRIxB7C/f2cNCoPhF3e0pw8LH/B7iRIxB7J+fGGMhUPyijTHkehY+YgyIGjEGkTdGB54QY0hC8h0fw5yIEzEGse++NcpCoPhFHWPI9St8DHMiTsQYxIgxJCH6bk/R8SFZxBjEiDEkIe3CR4wBUSPGIEKMIQnRxxhyvY5kuj3EiRiDGDGGJCTR7ak+hY/5PcSJGIMYMYYkRB9jyHUsfMQYEDViDCLEGJKRfMfHMCfiRIxBjBhDEpKIMeS6FT6GOREnYgxixBiSkEy3pyh8SA4xBjFiDElIu/DpGMOUn68DOEaMQYQYQxIepBJjyHU6oun2ECdiDGLEGJKQzGrOHIUP6Vhb/JK9LUOMIQlJDXOq3YVPxxhO+/s6gEObv5vk5zVHjCEZyXd8dHuI09YjYgxCxBiScDelGENud+Ejv4c4bdwjxiBEjCEJyQ1zKjo+JGP9fyyzs2WIMSQhuWFO1Vr4iDEgWjurSm394Rg72BwxhiQkF2PItR7ZdHuI0+b/YccKEWNIQpLdnqLwIQnEGMSIMSQhyfk9lRc+YgyIGjEGEWIMyUi+46PbQ5yIMYgRY0hCkjGGXF74iDEgTsQYxIgxJCHZYU5Fx4foEWMQI8aQhGSHOTODxBgQLWIMYsQYkpBsjCE3SLeHaBFjECPGkISkuz1F4UPUiDGIEWNIQtLze0oXPmIMiBMxBhFiDMmg4wvgOwD2EWMQ++HsOAuB4pd0jCE3+GKiE4gNMQaxvzg5zkKg+CU/zKl04bsawPcA7CLGIDI8NLDyw5PjNfrGKCj5YU6lC98VpdRSAN8FsIMYg9iHf753oGZfGXLJxxhyg7cvnn+qn9xC8UMciDGI/NW7E4///fffoN2LH92e9mJxy+2L57O7gBml1EdKqcUQvhhQ1D713/6BH6+3bAXnv5ge+/I/zr2p/sOPpw6F/F1hDfN72sDOzk4QXwSw5t4H95VS0/ygwGsOsKLzJeIMiMu9D85Q9IA2xBhaUPgQG940ArRjmLMFhQ+x4RF8QDsWtrRgjg/xuPfBfqUUwXXgdVmMYYbf5BU6PsSEYU6gHd3eLhQ+xIRhTqAd83u7UPgQEzo+oB0d3y4UPsSBGAPQCTGGDih8iAXdHtCOYc4OKHyIBfN7QDsKXwfEGVB/xBiATogxdEHHhxjQ7QHt6Pa6oPAhBszvAe1YzdkFQ52ov3sfZKvWptiTQIvZ67xcuAs6PtTbyxgDRQ943U1+j+4ofKg75veAdgxz9kDhQ91R+IB2LGzpgTk+1BcxBqATYgx90PGhzuj2gHZ0e31Q+FBnxBiAdszv9cFQJ+qLGAPQjhhDX3R8qCdiDEAnxBgMUPhQV8zvAe0Y5jRA4UNdUfiAdixsMcAcH+qHGAPQCTEGQ3R8qCO6PaAd3Z4hCh/qiBgD0I75PUMMdaJ+iDEA7YgxGKPjQ70QYwA6IcYgQOFD3TC/B7RjmFOAwoe6ofAB7VjYIsAcH+qDGAPQCTEGITo+1AndHtCObk+Iwoc6IcYAtGN+T4ihTtQHMQagHTEGMTo+1AMxBqATYgwFUPhQF8zvAe0Y5iyAwoe6oPAB7VjYUgBzfAgfMQagE2IMBdHxoQ7o9oB2dHsFUfhQB8QYgHbM7xXEUCfCR4wBaEeMoTA6PoSNGAPQCTGGEih8CN1+9hDQhmHOEih8AFA/LGwpgTk+hO/eBxykwCuLavb6GX6P4uj4UAefsJeAr13hpyiHwoc6uPTiLhfANTV7/Wryv0JJFD6Eb/b6U53lu8beQqKWlFIfqdnrFzgAymOOD/Vy74MZXQR5VBNSsfBiFefLG0CUpZT6R/8SFLBJJ3G8AAAAAElFTkSuQmCC"}}},{"cell_type":"markdown","source":"## Installation\nPython datatable is available by default or can be installed using pip (with internet) or using a wheel file (without internet).","metadata":{}},{"cell_type":"code","source":"# installation with internet\n# !python3 -m pip install pip --upgrade\n# !python3 -m pip install datatable --upgrade","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":false,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-10-08T17:18:38.703648Z","iopub.status.busy":"2020-10-08T17:18:38.70288Z","iopub.status.idle":"2020-10-08T17:18:58.57622Z","shell.execute_reply":"2020-10-08T17:18:58.575394Z"},"papermill":{"duration":19.889466,"end_time":"2020-10-08T17:18:58.576363","exception":false,"start_time":"2020-10-08T17:18:38.686897","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# installation without internet\n# !pip install ../input/python-datatable/datatable-1.0.0-cp37-cp37m-manylinux_2_12_x86_64.whl","metadata":{"execution":{"iopub.status.busy":"2021-07-08T06:31:44.248741Z","iopub.execute_input":"2021-07-08T06:31:44.249163Z","iopub.status.idle":"2021-07-08T06:32:13.293956Z","shell.execute_reply.started":"2021-07-08T06:31:44.249081Z","shell.execute_reply":"2021-07-08T06:32:13.292747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading data in csv format\nFirst let's read the dataset from the raw csv file. Datatable takes less than a minute to read the full dataset and convert it to pandas.\n","metadata":{"papermill":{"duration":0.077718,"end_time":"2020-10-08T17:18:58.735861","exception":false,"start_time":"2020-10-08T17:18:58.658143","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n\n# reading the dataset from raw csv file\nimport datatable as dt\n\ntrain = dt.fread(\"../input/riiid-test-answer-prediction/train.csv\").to_pandas()\n\nprint(train.shape)\n","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":42.032589,"end_time":"2020-10-08T17:19:40.846019","exception":false,"start_time":"2020-10-08T17:18:58.81343","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-08T06:32:49.105356Z","iopub.execute_input":"2021-07-08T06:32:49.105745Z","iopub.status.idle":"2021-07-08T06:34:19.198215Z","shell.execute_reply.started":"2021-07-08T06:32:49.105709Z","shell.execute_reply":"2021-07-08T06:34:19.196758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.execute_input":"2020-10-08T17:19:41.012525Z","iopub.status.busy":"2020-10-08T17:19:41.008302Z","iopub.status.idle":"2020-10-08T17:19:41.035929Z","shell.execute_reply":"2020-10-08T17:19:41.035185Z"},"papermill":{"duration":0.112892,"end_time":"2020-10-08T17:19:41.036056","exception":false,"start_time":"2020-10-08T17:19:40.923164","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading data in jay format\nThe dataset can be saved in binary format (.jay) and datatable can then read the entire dataset in less than a second!\n","metadata":{"papermill":{"duration":0.076957,"end_time":"2020-10-08T17:19:41.189182","exception":false,"start_time":"2020-10-08T17:19:41.112225","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# saving the dataset in .jay (binary format)\ndt.fread(\"../input/riiid-test-answer-prediction/train.csv\").to_jay(\"train.jay\")","metadata":{"execution":{"iopub.execute_input":"2020-10-08T17:19:41.366093Z","iopub.status.busy":"2020-10-08T17:19:41.365241Z","iopub.status.idle":"2020-10-08T17:20:10.959012Z","shell.execute_reply":"2020-10-08T17:20:10.958214Z"},"papermill":{"duration":29.679476,"end_time":"2020-10-08T17:20:10.959155","exception":false,"start_time":"2020-10-08T17:19:41.279679","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# reading the dataset from .jay format\nimport datatable as dt\n\ntrain = dt.fread(\"train.jay\")\n\nprint(train.shape)","metadata":{"execution":{"iopub.execute_input":"2020-10-08T17:20:22.609498Z","iopub.status.busy":"2020-10-08T17:20:22.607689Z","iopub.status.idle":"2020-10-08T17:20:22.619057Z","shell.execute_reply":"2020-10-08T17:20:22.620055Z"},"papermill":{"duration":0.692052,"end_time":"2020-10-08T17:20:22.620267","exception":false,"start_time":"2020-10-08T17:20:21.928215","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.execute_input":"2020-10-08T17:20:22.884967Z","iopub.status.busy":"2020-10-08T17:20:22.883955Z","iopub.status.idle":"2020-10-08T17:20:22.891092Z","shell.execute_reply":"2020-10-08T17:20:22.891793Z"},"papermill":{"duration":0.103631,"end_time":"2020-10-08T17:20:22.891952","exception":false,"start_time":"2020-10-08T17:20:22.788321","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It takes less than a second to read the entire dataset into Python datatable!\n\nConverting it to pandas is fairly fast too, taking another few seconds.","metadata":{"papermill":{"duration":0.100177,"end_time":"2020-10-08T17:20:23.080225","exception":false,"start_time":"2020-10-08T17:20:22.980048","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n\nimport datatable as dt\n\ntrain = dt.fread(\"train.jay\").to_pandas()\n\nprint(train.shape)\n","metadata":{"execution":{"iopub.execute_input":"2020-10-08T17:20:42.313538Z","iopub.status.busy":"2020-10-08T17:20:42.312384Z","iopub.status.idle":"2020-10-08T17:21:04.082438Z","shell.execute_reply":"2020-10-08T17:21:04.083225Z"},"papermill":{"duration":40.900269,"end_time":"2020-10-08T17:21:04.083519","exception":false,"start_time":"2020-10-08T17:20:23.18325","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.execute_input":"2020-10-08T17:21:04.289083Z","iopub.status.busy":"2020-10-08T17:21:04.275561Z","iopub.status.idle":"2020-10-08T17:21:04.308526Z","shell.execute_reply":"2020-10-08T17:21:04.309109Z"},"papermill":{"duration":0.148653,"end_time":"2020-10-08T17:21:04.309287","exception":false,"start_time":"2020-10-08T17:21:04.160634","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While I personally find this the fastest, there are other options to consider as well and some of them have been described in [this notebook](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).\n\nYou can try out or learn more about datatable from [**DatatableTon**: *💯 datatable exercises*](https://github.com/vopani/datatableton)","metadata":{}}]}