{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Memory Efficient File Format for Fast Data Loading for Images and Tables\nThe goal of this notebook is to convert all the data, especially the images which are 105.100 Files in 86 Folders, in only one `HDF5` file with 86 *groups* and 105.100 *datasets*. Since the size of the `HDF5` file increases rapidly with the number of images, we convert the images in `binary` format at first. The `csv` files will be converted in compressed `Parquett` files.\n\n**NOTE: All the codes are commented out. The final data can be found [here](https://www.kaggle.com/ismailbaris/hum-parquet-hdf5). See section `Read Images` at the end of this notebook to read the images!**\n\n**Links:**\n- **[Link](https://www.kaggle.com/ismailbaris/hum-parquet-hdf5) to the dataset.**\n- [Link](https://databricks.com/glossary/what-is-parquet) to `parquet` homepage.\n- [Link](https://www.hdfgroup.org/solutions/hdf5/) to `hdf5` homepage.\n- [Link](https://www.machinecurve.com/index.php/2020/04/13/how-to-use-h5py-and-keras-to-train-with-data-from-hdf5-files/) to *How to use hdf5 files with keras*.\n\n## What is Parquet?\nApache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. Apache Parquet is designed to be a common interchange format for both batch and interactive workloads. It is similar to other columnar-storage file formats available in Hadoop, namely RCFile and ORC* ([Reference](https://databricks.com/glossary/what-is-parquet)).\n\n## What is HDF5?\n*Utilize the HDF5 high performance data software library and file format to manage, process, and store your heterogeneous data. HDF5 is built for fast I/O processing and storage. HDF® is portable, with no vendor lock-in, and is a self-describing file format, meaning everything all data and metadata can be passed along in one file. There is no limit on the number or size of data objects in the collection, giving great flexibility for big data* ([Reference](https://www.hdfgroup.org/solutions/hdf5/)).","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"Importing necessary packages and defining paths.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"from pathlib import Path\nimport os\nimport pandas as pd\nimport h5py\nimport numpy as np\nfrom PIL import Image\nimport io\n\n# data_path = Path(\"../input/h-and-m-personalized-fashion-recommendations\")  # Path to the H&M data.\n# image_path = Path(\"../input/h-and-m-personalized-fashion-recommendations/images\")  # Path to the images.\n\n# export_path = Path(\"./\")  # Path, where the new files will be exported.\n\n# export_path.mkdir(parents=True, exist_ok=True)","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-13T18:10:28.445433Z","iopub.execute_input":"2022-02-13T18:10:28.446209Z","iopub.status.idle":"2022-02-13T18:10:28.637962Z","shell.execute_reply.started":"2022-02-13T18:10:28.446104Z","shell.execute_reply":"2022-02-13T18:10:28.637274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading Articles\nWe read the `articles.csv` file and determine which articles have an image present.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# articles = pd.read_csv(data_path / \"articles.csv\")","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The next step is to determine all present image names in the directory `images`..","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# file_list = []\n# for p, d, f in os.walk(image_path):\n#     for item in f:\n#         file_list.append(int(item.split(\".\")[0]))","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We add a new column to the data and mark all the article ids which have an image","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# articles[\"image\"] = False\n# articles.loc[articles['article_id'].isin(file_list), \"image\"] = True","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Export the `articles` to the `parquet` file format now.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# articles.to_parquet(os.path.join(export_path, \"articles.parquet.gzip\"), compression='gzip')","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conversion to Parquet\nFirstly, we convert all the remaining data in the `parquet` file format:","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# files = [\"customers\", \"sample_submission\", \"transactions_train\"]\n\n# for item in files:\n#     print(\"\\r> Processing File: \", end=str(item))\n#     dataframe = pd.read_csv(data_path / f\"{item}.csv\")\n#     dataframe.to_parquet(export_path / f\"{item}.parquet.gzip\", compression='gzip')\n#     del dataframe\n\n# print(\"\\n> [Done]\")","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Enhance Transaction Dataset\nSince the information in the file `transaction_train` contains only 5 columns, we will join all the available information from the file `customers` data.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# customers = pd.read_parquet(export_path / \"customers.parquet.gzip\")  # Open the customer dataset.\n# transactions = pd.read_parquet(export_path / \"transactions_train.parquet.gzip\")  # Open the transaction dataset.\n\n# training_data = pd.merge(transactions, customers, on=[\"customer_id\"])\n# del customers\n# del transactions\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Save the new dataset in a new `parquet` file.\n# training_data.to_parquet(export_path / \"train.parquet.gzip\", compression='gzip')","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# training_data.columns","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# del training_data","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convert Images to HDF5\nAlthough the HDF5 file format is very efficient format to read and write image files, the hdf5 file size increases rapidly with the number of images. This happens because the numpy array takes more storage space than the original image files. To overcome this problem, we will store the images as binary files. The strategy is, to create one HDF5 image dataset and group all the subdirectories in that one file. Here is the file structure:\n\n- images.h5 (Created `HDF5` file)\n    - 010 (Group 1: The first three digits of the `article_id`)\n       - 0108775015 (Image 1: `article_id`)\n       - 0108775044 (Image 2: `article_id`)\n       - ...\n    - 010 (Group 2)\n       - 0110065001 (Image 1)\n       - ...\n\nAt first, we will group the images in a directory:","metadata":{}},{"cell_type":"code","source":"# image_dict = dict()\n# for p, d, f in os.walk(image_path):\n#     file_list = list()\n#     cat_id = os.path.basename(p)\n#     for item in f:\n#         file_list.append(os.path.join(p, item))\n\n#     if len(file_list) != 0:\n#         image_dict[cat_id] = file_list","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After that, we start to write the files in the `HDF5` file.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# hf = h5py.File(export_path / \"images.h5\", \"a\")  # Create the HDF5 file as in append mode.\n# \n# for group, image_paths in image_dict.items():  # Iterate over all image groups (first three digits of `article_id`).\n#     print(\"\\r> Processing Group \", end=group)\n# \n#     grp = hf.create_group(group)  # Create a group where the name of the group is the first three digits of `article_id`.\n# \n#     for image_path in image_paths:  # Iterate over all images within that group.\n#         name = os.path.basename(image_path.split(\".\")[0])  # Extract the `article_id` (filename without extension).\n# \n#         with open(image_path, 'rb') as img_f:  # Open the image as python binary.\n#             binary_data = img_f.read()\n# \n#         binary_data_np = np.asarray(binary_data)\n# \n#         dset = grp.create_dataset(name, data=binary_data_np)  # Save the binary array in the group.\n# \n# hf.close()\n# print(\"\\n> [Done]\")","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read Images\nTo read the data back, we use the `visititems` function of `h5py`","metadata":{}},{"cell_type":"code","source":"group = []  # List all groups.\n\ndata = []  # Store all the full data paths (group/file). These are the keys to access the image data.\n\n\ndef func(name, obj):  # Function to recursively store all the keys\n    if isinstance(obj, h5py.Dataset):\n        data.append(name)\n    elif isinstance(obj, h5py.Group):\n        group.append(name)\n\nhf_path = Path(\"../input/hum-parquet-hdf5/hum-data-efc\")\nhf = h5py.File(hf_path / \"images.h5\", 'r')\nhf.visititems(func)  # This operation fills the previously created lists `group` and `data`.","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-13T18:10:43.368123Z","iopub.execute_input":"2022-02-13T18:10:43.368516Z","iopub.status.idle":"2022-02-13T18:11:49.488203Z","shell.execute_reply.started":"2022-02-13T18:10:43.368465Z","shell.execute_reply":"2022-02-13T18:11:49.4871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Print the first dataset.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"print(data[0])","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-13T18:12:00.648888Z","iopub.execute_input":"2022-02-13T18:12:00.649281Z","iopub.status.idle":"2022-02-13T18:12:00.655916Z","shell.execute_reply.started":"2022-02-13T18:12:00.649236Z","shell.execute_reply":"2022-02-13T18:12:00.654655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read the first file as image:","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"hf_data = np.array(hf[data[0]])\nimage = Image.open(io.BytesIO(hf_data))\nprint('Image Size:', image.size)","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-13T18:12:03.655705Z","iopub.execute_input":"2022-02-13T18:12:03.656104Z","iopub.status.idle":"2022-02-13T18:12:03.688992Z","shell.execute_reply.started":"2022-02-13T18:12:03.65605Z","shell.execute_reply":"2022-02-13T18:12:03.687817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can also loop over all images, in order to use it in the training process","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# for ds in data:\n#     hf_ds = np.array(hf[ds])\n#     image = Image.open(io.BytesIO(hf_ds))\n#     print('Image S-ize:', image.size)\nhf.close()","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-13T18:12:10.550907Z","iopub.execute_input":"2022-02-13T18:12:10.551292Z","iopub.status.idle":"2022-02-13T18:12:10.579279Z","shell.execute_reply.started":"2022-02-13T18:12:10.551244Z","shell.execute_reply":"2022-02-13T18:12:10.578232Z"},"trusted":true},"execution_count":null,"outputs":[]}]}