I have implemented a class that allows one to read specific subsets of one file (say, multiome data for one specific day and one specific donor) without the need to load all the file into RAM. I used the h5py library, as @alexandervc suggested in another discussion (see: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/344829#1905232).
\nThe class I implemented can be found in this notebook. Comments and suggestions about how to make it more efficient (or whether you find this approach useful) are most welcome.
", "messages": [ { "id": "1907571", "postDate": "08/20/2022 23:02:10", "content": "I have implemented a class that allows one to read specific subsets of one file (say, multiome data for one specific day and one specific donor) without the need to load all the file into RAM. I used the h5py library, as @alexandervc suggested in another discussion (see: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/344829#1905232).
\nThe class I implemented can be found in this notebook. Comments and suggestions about how to make it more efficient (or whether you find this approach useful) are most welcome.
", "rawMarkdown": "I have implemented a class that allows one to read specific subsets of one file (say, multiome data for one specific day and one specific donor) without the need to load all the file into RAM. I used the h5py library, as @alexandervc suggested in another discussion (see: [https://www.kaggle.com/competitions/open-problems-multimodal/discussion/344829#1905232](https://www.kaggle.com/competitions/open-problems-multimodal/discussion/344829#1905232)).\n\nThe class I implemented can be found in this [notebook](https://www.kaggle.com/alekeuro/querying-specific-subsets-of-the-data-without-load). Comments and suggestions about how to make it more efficient (or whether you find this approach useful) are most welcome.", "votes": null }, { "id": "1910968", "postDate": "08/23/2022 20:20:39", "content": "I just implemented a new feature in this DataReader class for columns selection: you can pass in
I have updated the above mentioned notebook.
", "rawMarkdown": "I just implemented a new feature in this `DataReader` class for columns selection: you can pass in \n\n- a list of columns names (if you know them in advance)\n- a list of ints corresponding to indices of desired columns, or \n- you can ask for a random sample of N columns. \n\nI have updated the above mentioned notebook.", "votes": null } ], "comments": [ { "id": 1910968, "author_name": "alekeuro", "author_url": "", "post_date": "08/23/2022 20:20:39", "content": "I just implemented a new feature in this DataReader class for columns selection: you can pass in
I have updated the above mentioned notebook.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1907571": "I have implemented a class that allows one to read specific subsets of one file (say, multiome data for one specific day and one specific donor) without the need to load all the file into RAM. I used the h5py library, as @alexandervc suggested in another discussion (see: [https://www.kaggle.com/competitions/open-problems-multimodal/discussion/344829#1905232](https://www.kaggle.com/competitions/open-problems-multimodal/discussion/344829#1905232)).\n\nThe class I implemented can be found in this [notebook](https://www.kaggle.com/alekeuro/querying-specific-subsets-of-the-data-without-load). Comments and suggestions about how to make it more efficient (or whether you find this approach useful) are most welcome.", "1910968": "I just implemented a new feature in this `DataReader` class for columns selection: you can pass in \n\n- a list of columns names (if you know them in advance)\n- a list of ints corresponding to indices of desired columns, or \n- you can ask for a random sample of N columns. \n\nI have updated the above mentioned notebook." }, "source": "meta" }