{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What is Parquet?\n\nParquet is an open source file format built to handle flat columnar storage data formats. Parquet operates well with complex data in large volumes.It is known for its both performant data compression and its ability to handle a wide variety of encoding types. \n\nParquet is an open source file format available to any project in the Hadoop ecosystem. Apache Parquet is designed for efficient as well as performant flat columnar storage format of data compared to row based files like CSV or TSV files.\n\nParquet uses the record shredding and assembly algorithm which is superior to simple flattening of nested namespaces. Parquet is optimized to work with complex data in bulk and features different ways for efficient data compression and encoding types. This approach is best especially for those queries that need to read certain columns from a large table. Parquet can only read the needed columns therefore greatly minimizing the IO.","metadata":{}},{"cell_type":"markdown","source":"# Train File","metadata":{}},{"cell_type":"code","source":"import pandas as pd","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-01-19T19:27:10.616517Z","iopub.execute_input":"2023-01-19T19:27:10.616991Z","iopub.status.idle":"2023-01-19T19:27:10.715913Z","shell.execute_reply.started":"2023-01-19T19:27:10.616895Z","shell.execute_reply":"2023-01-19T19:27:10.714775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndf = pd.read_parquet(\"/kaggle/input/icecube-neutrinos-in-deep-ice/train/batch_1.parquet\")","metadata":{"execution":{"iopub.status.busy":"2023-01-19T19:27:11.347678Z","iopub.execute_input":"2023-01-19T19:27:11.348120Z","iopub.status.idle":"2023-01-19T19:27:14.707926Z","shell.execute_reply.started":"2023-01-19T19:27:11.348082Z","shell.execute_reply":"2023-01-19T19:27:14.706724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-19T19:27:14.710442Z","iopub.execute_input":"2023-01-19T19:27:14.710938Z","iopub.status.idle":"2023-01-19T19:27:14.727025Z","shell.execute_reply.started":"2023-01-19T19:27:14.710895Z","shell.execute_reply":"2023-01-19T19:27:14.725996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-19T19:27:24.835151Z","iopub.execute_input":"2023-01-19T19:27:24.835557Z","iopub.status.idle":"2023-01-19T19:27:24.851807Z","shell.execute_reply.started":"2023-01-19T19:27:24.835521Z","shell.execute_reply":"2023-01-19T19:27:24.850640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission File","metadata":{}},{"cell_type":"code","source":"%%time\nsubmission = pd.read_parquet(\"/kaggle/input/icecube-neutrinos-in-deep-ice/sample_submission.parquet\")\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-01-19T19:30:44.423441Z","iopub.execute_input":"2023-01-19T19:30:44.423879Z","iopub.status.idle":"2023-01-19T19:30:44.444643Z","shell.execute_reply.started":"2023-01-19T19:30:44.423844Z","shell.execute_reply":"2023-01-19T19:30:44.443912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Meta File","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_meta = pd.read_parquet(\"/kaggle/input/icecube-neutrinos-in-deep-ice/train_meta.parquet\")\ntrain_meta","metadata":{"execution":{"iopub.status.busy":"2023-01-19T19:30:47.171606Z","iopub.execute_input":"2023-01-19T19:30:47.171996Z","iopub.status.idle":"2023-01-19T19:30:58.934951Z","shell.execute_reply.started":"2023-01-19T19:30:47.171966Z","shell.execute_reply":"2023-01-19T19:30:58.933995Z"},"trusted":true},"execution_count":null,"outputs":[]}]}