I've been following the graphnet-example uploaded by @rasmusrse. I have (unsuccessfully) been trying to train on larger samples of data since, from empirical experience, more data = more performance. However, I've reached a plateau where I am restricted by memory constraints. Has anyone managed to train Graphnet (on Kaggle) with large batches of data?
\nNote: for each batch (parquet train file) the corresponding SQLite db is ~2GB
", "messages": [ { "id": "2163136", "postDate": "02/28/2023 15:59:32", "content": "I've been following the graphnet-example uploaded by @rasmusrse. I have (unsuccessfully) been trying to train on larger samples of data since, from empirical experience, more data = more performance. However, I've reached a plateau where I am restricted by memory constraints. Has anyone managed to train Graphnet (on Kaggle) with large batches of data?
\nNote: for each batch (parquet train file) the corresponding SQLite db is ~2GB
", "rawMarkdown": "I've been following the graphnet-example uploaded by @rasmusrse. I have (unsuccessfully) been trying to train on larger samples of data since, from empirical experience, more data = more performance. However, I've reached a plateau where I am restricted by memory constraints. Has anyone managed to train Graphnet (on Kaggle) with large batches of data? \n\nNote: for each batch (parquet train file) the corresponding SQLite db is ~2GB", "votes": null }, { "id": "2163155", "postDate": "02/28/2023 16:18:08", "content": "You may want to rewrite the dataloader for it to upload the new db and erase the old one from memory each time when previous db is completely processed by batches
\nBut again, as Rasmus has pointed out, that may be a walkaround for some different problem
", "rawMarkdown": "You may want to rewrite the dataloader for it to upload the new db and erase the old one from memory each time when previous db is completely processed by batches\n\nBut again, as Rasmus has pointed out, that may be a walkaround for some different problem", "votes": null }, { "id": "2163163", "postDate": "02/28/2023 16:23:10", "content": "15 batches in SQLite format should take up around 30 gb of disk space - not RAM. Using the example, you should never have more than batch_size*(1 + prefetch_factor) events in memory (because of SQLite) - so I'm surprised that you're running out of memory. Could you elaborate on what you're doing?
", "rawMarkdown": "15 batches in SQLite format should take up around 30 gb of disk space - not RAM. Using the example, you should never have more than batch_size*(1 + prefetch_factor) events in memory (because of SQLite) - so I'm surprised that you're running out of memory. Could you elaborate on what you're doing?", "votes": null }, { "id": "2164196", "postDate": "03/01/2023 11:49:54", "content": "Hello! I have same problem with @alejopaullier, when I try to convert multiple batches into a single database by adjusting batch_id in kaggle env. Even if I convert just one batch, as the code indicates, a memory(RAM) overflow will occur.
\nconvert_to_sqlite(meta_data_path,\n database_path=database_path,\n input_data_folder=input_data_folder,\n batch_ids = [51])
Thanks Rasmus and sorry for not being so clear. When I create a database from many batches the output is larger than what Kaggle supports in the output directory (/kaggle/working/), which is around ~20GB, which limits the size of the database I can create.
Hi @alejopaullier @lfz5cv please try to put them in /tmp. For example,
SAVE_FOLDER = \"/tmp/output/\"\nos.makedirs(SAVE_FOLDER, exist_ok=True)\nThen you can save up to 70GB data on disk.
", "rawMarkdown": "Hi @alejopaullier @lfz5cv please try to put them in `/tmp`. For example,\n\n```\nSAVE_FOLDER = \"/tmp/output/\"\nos.makedirs(SAVE_FOLDER, exist_ok=True)\n```\n\nThen you can save up to 70GB data on disk.", "votes": null }, { "id": "2187301", "postDate": "03/18/2023 15:55:04", "content": "Hello, @alejopaullier, @rasmusrse
\nI have similar problems.
By using the sample code, we can generate the SQLite table separately like:
\nbatch_1.db\nbatch_2.db\n...\nbatch_50.db\n, which table can be saved at the storage and each db is ~2GB.
\nHowever, when we train by using these all batch (1-50), I think we have to merge the db like batch_1_50.db.
\nIn my understanding, we can set the config file like:
config = {\n \"path\": f'{INPUT_DIR}/batch_1_50.db',\n \"inference_database_path\": f'{INPUT_DIR}/batch_51.db',\n ...\n 'train_selection': './train_selection_max_200_pulses_1_50_batch.csv',\n 'validate_selection': './validate_selection_max_200_pulses_51_batch.csv',\n ...\n}\n.
\nTherefore, we have to load batch_1_50.db to the memory which is huge size(2GB * 50 ~ 100GB).
\nI'm wondering we need to prepare more than 100GB size memory.
\nI'm not sure this idea is correct or not.
\nI would appreciate if you or anyone could answer my question.
You may want to rewrite the dataloader for it to upload the new db and erase the old one from memory each time when previous db is completely processed by batches
\nBut again, as Rasmus has pointed out, that may be a walkaround for some different problem
", "votes": null, "replies": [] }, { "id": 2163163, "author_name": "rasmusrse", "author_url": "", "post_date": "02/28/2023 16:23:10", "content": "15 batches in SQLite format should take up around 30 gb of disk space - not RAM. Using the example, you should never have more than batch_size*(1 + prefetch_factor) events in memory (because of SQLite) - so I'm surprised that you're running out of memory. Could you elaborate on what you're doing?
", "votes": null, "replies": [ { "id": 2164196, "author_name": "lfz5cv", "author_url": "", "post_date": "03/01/2023 11:49:54", "content": "Hello! I have same problem with @alejopaullier, when I try to convert multiple batches into a single database by adjusting batch_id in kaggle env. Even if I convert just one batch, as the code indicates, a memory(RAM) overflow will occur.
\nconvert_to_sqlite(meta_data_path,\n database_path=database_path,\n input_data_folder=input_data_folder,\n batch_ids = [51])
Thanks Rasmus and sorry for not being so clear. When I create a database from many batches the output is larger than what Kaggle supports in the output directory (/kaggle/working/), which is around ~20GB, which limits the size of the database I can create.
Hi @alejopaullier @lfz5cv please try to put them in /tmp. For example,
SAVE_FOLDER = \"/tmp/output/\"\nos.makedirs(SAVE_FOLDER, exist_ok=True)\nThen you can save up to 70GB data on disk.
", "votes": null, "replies": [] } ] } ] }, { "id": 2187301, "author_name": "tetsuro731", "author_url": "", "post_date": "03/18/2023 15:55:04", "content": "Hello, @alejopaullier, @rasmusrse
\nI have similar problems.
By using the sample code, we can generate the SQLite table separately like:
\nbatch_1.db\nbatch_2.db\n...\nbatch_50.db\n, which table can be saved at the storage and each db is ~2GB.
\nHowever, when we train by using these all batch (1-50), I think we have to merge the db like batch_1_50.db.
\nIn my understanding, we can set the config file like:
config = {\n \"path\": f'{INPUT_DIR}/batch_1_50.db',\n \"inference_database_path\": f'{INPUT_DIR}/batch_51.db',\n ...\n 'train_selection': './train_selection_max_200_pulses_1_50_batch.csv',\n 'validate_selection': './validate_selection_max_200_pulses_51_batch.csv',\n ...\n}\n.
\nTherefore, we have to load batch_1_50.db to the memory which is huge size(2GB * 50 ~ 100GB).
\nI'm wondering we need to prepare more than 100GB size memory.
\nI'm not sure this idea is correct or not.
\nI would appreciate if you or anyone could answer my question.