{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/%E0%A6%AC%E0%A6%BE%E0%A6%82%E0%A6%B2%E0%A6%BE.svg/400px-%E0%A6%AC%E0%A6%BE%E0%A6%82%E0%A6%B2%E0%A6%BE.svg.png)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport PIL.Image\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Typical Training File"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/bengaliai-cv19/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The more interesting Parquet files, Do you know What's Parquet?"},{"metadata":{},"cell_type":"markdown","source":"### Apache Parquet is a columnar binary format that is easy to split into multiple files (easier for parallel loading) and is generally much simpler to deal with than HDF5 (from the library’s perspective).\n\n![](https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fimage.slidesharecdn.com%2Falexlevensonhadoopsummitv3-150609221151-lva1-app6891%2F95%2Fslide-48-1024.jpg&f=1&nofb=1)\n\n###  It is also a common format used by other big data systems like Apache Spark and Apache Impala, and so it is useful to interchange with other systems:\n\n### References - https://www.slideshare.net/alexlevenson/hadoop-summit-2015-performance-optimization-at-scale/46-APACHE_PARQUETColumnar_storage_for_the"},{"metadata":{},"cell_type":"markdown","source":"1. ### So, Parquet is really fast and efficient\n\n"},{"metadata":{},"cell_type":"markdown","source":"You can read Parquet with `read_parquet()` of `pandas`  or `read_parquet()` of `dask`"},{"metadata":{},"cell_type":"markdown","source":"Let's time `pandas` vs `dask` and see how's reading parquet doing "},{"metadata":{"trusted":true},"cell_type":"code","source":"import dask.dataframe as dd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_dd = dd.read_parquet('/kaggle/input/bengaliai-cv19/train_image_data_1.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf = pd.read_parquet('/kaggle/input/bengaliai-cv19/train_image_data_1.parquet')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Class Map"},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map_df = pd.read_csv('/kaggle/input/bengaliai-cv19/class_map.csv')\nclass_map_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map_df.component_type.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### WIP, Hopefully more tomorrow! Cheers! "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}