{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction\n**Vaex** is an open-source DataFrame library for Python with an API that closely resembles that of Pandas.\n\nNowadays, it becomes increasingly more common to encounter datasets that are larger than the available RAM on a typical laptop or a desktop workstation. Vaex solves this problem rather elegantly by the use of memory mapping and lazy evaluations. As long as your data is stored in a memory mappable file format such as Apache Arrow or HDF5, Vaex will open it instantly, no matter how large it is, or how much RAM your machine has. In fact, the size of the files Vaex can read are only limited by the amount of free hard-disk space you have. If your data is not in a memory-mappable file format (e.g. CSV, JSON), you can easily convert it by using the rich Pandas I/O in combination with Vaex. See this guide on how to do so.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:49:28.216594Z","iopub.execute_input":"2021-06-03T08:49:28.217155Z","iopub.status.idle":"2021-06-03T08:49:28.237762Z","shell.execute_reply.started":"2021-06-03T08:49:28.217031Z","shell.execute_reply":"2021-06-03T08:49:28.236127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install vaex","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:49:28.240009Z","iopub.execute_input":"2021-06-03T08:49:28.240386Z","iopub.status.idle":"2021-06-03T08:49:47.598955Z","shell.execute_reply.started":"2021-06-03T08:49:28.240346Z","shell.execute_reply":"2021-06-03T08:49:47.597662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first step is to convert the data into a memory mappable file format, such as Apache Arrow, Apache Parquet, or HDF5.  Once the data is in a memory mappable format, opening it with Vaex is instant (0.052 seconds!), despite its size of over 100GB on disk:","metadata":{}},{"cell_type":"code","source":"import vaex","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:49:47.600631Z","iopub.execute_input":"2021-06-03T08:49:47.600970Z","iopub.status.idle":"2021-06-03T08:49:49.239857Z","shell.execute_reply.started":"2021-06-03T08:49:47.600934Z","shell.execute_reply":"2021-06-03T08:49:49.239047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# read data using vaex\ndf = vaex.open('../input/riiid-test-answer-prediction/train.csv', convert='train_bigdata.hdf5')","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:49:49.241002Z","iopub.execute_input":"2021-06-03T08:49:49.241441Z","iopub.status.idle":"2021-06-03T08:54:48.083482Z","shell.execute_reply.started":"2021-06-03T08:49:49.241409Z","shell.execute_reply":"2021-06-03T08:54:48.081490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# read data using vaex\ndf = vaex.open('./train_bigdata.hdf5')","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:54:48.087689Z","iopub.execute_input":"2021-06-03T08:54:48.088241Z","iopub.status.idle":"2021-06-03T08:54:48.109066Z","shell.execute_reply.started":"2021-06-03T08:54:48.088200Z","shell.execute_reply":"2021-06-03T08:54:48.107839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You may see the there are files created in output folder. Once it will tak esome time and once done its realy fast to read hdf files.\n\nWhy is it so fast? When you open a memory mapped file with Vaex, there is actually no data reading going on. Vaex only reads the file metadata, such as the location of the data on disk, the data structure (number of rows, number of columns, column names and types), the file description and so on. So what if we want to inspect or interact with the data? Opening a dataset results in a standard DataFrame and inspecting it is as fast as it is trivial.","metadata":{}},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:54:48.110696Z","iopub.execute_input":"2021-06-03T08:54:48.112213Z","iopub.status.idle":"2021-06-03T08:54:48.163149Z","shell.execute_reply.started":"2021-06-03T08:54:48.112173Z","shell.execute_reply":"2021-06-03T08:54:48.161947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type(df)","metadata":{"execution":{"iopub.status.busy":"2021-06-03T08:54:48.164845Z","iopub.execute_input":"2021-06-03T08:54:48.165306Z","iopub.status.idle":"2021-06-03T08:54:48.171531Z","shell.execute_reply.started":"2021-06-03T08:54:48.165259Z","shell.execute_reply":"2021-06-03T08:54:48.170839Z"},"trusted":true},"execution_count":null,"outputs":[]}]}