Dealing with large chunked data is one of the challenges of this competition, which may impose an additional barrier for people willing to join. Since the competition was not very active recently, I decided to share quite a simple way of dealing with the provided data without the time-consuming conversion of data into other formats or trying to load everything into the RAM, which is quite challenging, especially at Kaggle.
\nThe chunked data format is not very friendly to Pytorch, and some additional tricks are needed. My approach consists of two things: (1) A dataloader caching the considered chunks and (2) Random Chunked Sampler, which randomly selects a chunk and then goes through all ids in the chunk before selecting a new one. With this sampler, the caching pipeline spends time reading the corresponding chunk only at the first request, and the following requests are finished quickly with reading data from RAM. Meanwhile, when the cache is full, the earliest record is removed, which limits RAM usage. At inference and test, when reading is sequential, no extra modifications are needed to the data sampler. It would be great if the Pytorch team implemented native support of chunked data.
\nThe notebook is available -> here.
", "messages": [ { "id": 2180791, "postDate": "2023-03-14T05:17:36.303Z", "content": "Dealing with large chunked data is one of the challenges of this competition, which may impose an additional barrier for people willing to join. Since the competition was not very active recently, I decided to share quite a simple way of dealing with the provided data without the time-consuming conversion of data into other formats or trying to load everything into the RAM, which is quite challenging, especially at Kaggle.
\nThe chunked data format is not very friendly to Pytorch, and some additional tricks are needed. My approach consists of two things: (1) A dataloader caching the considered chunks and (2) Random Chunked Sampler, which randomly selects a chunk and then goes through all ids in the chunk before selecting a new one. With this sampler, the caching pipeline spends time reading the corresponding chunk only at the first request, and the following requests are finished quickly with reading data from RAM. Meanwhile, when the cache is full, the earliest record is removed, which limits RAM usage. At inference and test, when reading is sequential, no extra modifications are needed to the data sampler. It would be great if the Pytorch team implemented native support of chunked data.
\nThe notebook is available -> here.
", "rawMarkdown": "Dealing with large chunked data is one of the challenges of this competition, which may impose an additional barrier for people willing to join. Since the competition was not very active recently, I decided to share quite a simple way of dealing with the provided data without the time-consuming conversion of data into other formats or trying to load everything into the RAM, which is quite challenging, especially at Kaggle.\n\nThe chunked data format is not very friendly to Pytorch, and some additional tricks are needed. My approach consists of two things: (1) A **dataloader caching the considered chunks** and (2) **Random Chunked Sampler**, which randomly selects a chunk and then goes through all ids in the chunk before selecting a new one. With this sampler, the caching pipeline spends time reading the corresponding chunk only at the first request, and the following requests are finished quickly with reading data from RAM. Meanwhile, when the cache is full, the earliest record is removed, which limits RAM usage. At inference and test, when reading is sequential, no extra modifications are needed to the data sampler. It would be great if the Pytorch team implemented native support of chunked data.\n\nThe notebook is available -> [here](https://www.kaggle.com/code/iafoss/chunk-based-data-loading-with-caching).", "votes": 31 } ], "comments": [], "raw_markdown_by_id": { "2180791": "Dealing with large chunked data is one of the challenges of this competition, which may impose an additional barrier for people willing to join. Since the competition was not very active recently, I decided to share quite a simple way of dealing with the provided data without the time-consuming conversion of data into other formats or trying to load everything into the RAM, which is quite challenging, especially at Kaggle.\n\nThe chunked data format is not very friendly to Pytorch, and some additional tricks are needed. My approach consists of two things: (1) A **dataloader caching the considered chunks** and (2) **Random Chunked Sampler**, which randomly selects a chunk and then goes through all ids in the chunk before selecting a new one. With this sampler, the caching pipeline spends time reading the corresponding chunk only at the first request, and the following requests are finished quickly with reading data from RAM. Meanwhile, when the cache is full, the earliest record is removed, which limits RAM usage. At inference and test, when reading is sequential, no extra modifications are needed to the data sampler. It would be great if the Pytorch team implemented native support of chunked data.\n\nThe notebook is available -> [here](https://www.kaggle.com/code/iafoss/chunk-based-data-loading-with-caching)." } }