I tried to train the graph-net provided by the host on my own.
\nI've copied all the required codes from the git, and now I can train and predict some angles.
\nBut I found that the training is much slower than my LSTM.
To give you some numbers, training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours while training my LSTM through 50 batches with batch_size of 128 took about 3 hours, with the same machine.
\nWell, the experiment is not controlled at all.
\nThis is the first time using pytorch and GNN, so I don't have any sense of time…
Is this long time consumption expectable? Is GNN generally slower than small LSTM?
\nOr does it indicate some potential inefficiency in my code?
I tried to train the graph-net provided by the host on my own.
\nI've copied all the required codes from the git, and now I can train and predict some angles.
\nBut I found that the training is much slower than my LSTM.
To give you some numbers, training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours while training my LSTM through 50 batches with batch_size of 128 took about 3 hours, with the same machine.
\nWell, the experiment is not controlled at all.
\nThis is the first time using pytorch and GNN, so I don't have any sense of time…
Is this long time consumption expectable? Is GNN generally slower than small LSTM?
\nOr does it indicate some potential inefficiency in my code?
At first glance, it seems weird that an LSTM (so a sequential model) outperforms a fully parallelized model in speed.
\nWhat are your machine ressources ? My implementation of graphnet takes several hours to train too.
I'm using a machine with RTX2080.
\nSo, several hours is not a big deal, right?
\"training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours\"
\nWhen you say 50 batches, do you mean the batches in the /train folder in the input data, or do you mean batches of 32 samples?
", "rawMarkdown": "\"training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours\"\n\nWhen you say 50 batches, do you mean the batches in the /train folder in the input data, or do you mean batches of 32 samples?", "votes": null }, { "id": "2151860", "postDate": "02/20/2023 12:35:32", "content": "The earlier one. 50 batch files, each containing the 200,000 events I meant.
", "rawMarkdown": "The earlier one. 50 batch files, each containing the 200,000 events I meant.", "votes": null }, { "id": "2153497", "postDate": "02/21/2023 13:03:53", "content": "@seungmoklee how did you train GraphNet with 50 batch files? At least for me that would mean a ~100GB file that would definitely lead to OOM
", "rawMarkdown": "seungmoklee how did you train GraphNet with 50 batch files? At least for me that would mean a ~100GB file that would definitely lead to OOM", "votes": null }, { "id": "2153554", "postDate": "02/21/2023 13:44:23", "content": "Using pytorch dataloader, I could read one batch, train it, discard the batch from memory, read the next batch, train it, …
", "rawMarkdown": "Using pytorch dataloader, I could read one batch, train it, discard the batch from memory, read the next batch, train it, ...", "votes": null }, { "id": "2153557", "postDate": "02/21/2023 13:45:32", "content": "@alejopaullier You can find an example in the graph-net git.
", "rawMarkdown": "alejopaullier You can find an example in the graph-net git.", "votes": null }, { "id": "2153580", "postDate": "02/21/2023 14:06:51", "content": "Thanks a lot. I went for the SQLite approach mentioned in the graphnet_example notebook which involves creating a huge database
", "rawMarkdown": "Thanks a lot. I went for the SQLite approach mentioned in the graphnet_example notebook which involves creating a huge database", "votes": null }, { "id": "2159340", "postDate": "02/25/2023 17:26:32", "content": "Are the models of comparable size? I think your shared LSTM model was way smaller than graphnet.
", "rawMarkdown": "Are the models of comparable size? I think your shared LSTM model was way smaller than graphnet.", "votes": null }, { "id": "2163149", "postDate": "02/28/2023 16:13:02", "content": "You can with relatively little effort switch from the SQLite backend to the parquet backend in graphnet. We have a ParquetDataset class already - with a few light modifications it'll work with the competition data.
", "rawMarkdown": "You can with relatively little effort switch from the SQLite backend to the parquet backend in graphnet. We have a ParquetDataset class already - with a few light modifications it'll work with the competition data.", "votes": null }, { "id": "2166215", "postDate": "03/02/2023 17:25:45", "content": "@seungmoklee would you mind sharing how to load the parquet files. I've been struggling with how to use the parquet files for this competition
", "rawMarkdown": "seungmoklee would you mind sharing how to load the parquet files. I've been struggling with how to use the parquet files for this competition", "votes": null } ], "comments": [ { "id": 2150506, "author_name": "louisstefanuto", "author_url": "", "post_date": "02/19/2023 10:15:36", "content": "At first glance, it seems weird that an LSTM (so a sequential model) outperforms a fully parallelized model in speed.
\nWhat are your machine ressources ? My implementation of graphnet takes several hours to train too.
I'm using a machine with RTX2080.
\nSo, several hours is not a big deal, right?
\"training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours\"
\nWhen you say 50 batches, do you mean the batches in the /train folder in the input data, or do you mean batches of 32 samples?
", "votes": null, "replies": [ { "id": 2151860, "author_name": "seungmoklee", "author_url": "", "post_date": "02/20/2023 12:35:32", "content": "The earlier one. 50 batch files, each containing the 200,000 events I meant.
", "votes": null, "replies": [ { "id": 2153497, "author_name": "alejopaullier", "author_url": "", "post_date": "02/21/2023 13:03:53", "content": "@seungmoklee how did you train GraphNet with 50 batch files? At least for me that would mean a ~100GB file that would definitely lead to OOM
", "votes": null, "replies": [ { "id": 2153554, "author_name": "seungmoklee", "author_url": "", "post_date": "02/21/2023 13:44:23", "content": "Using pytorch dataloader, I could read one batch, train it, discard the batch from memory, read the next batch, train it, …
", "votes": null, "replies": [ { "id": 2153557, "author_name": "seungmoklee", "author_url": "", "post_date": "02/21/2023 13:45:32", "content": "@alejopaullier You can find an example in the graph-net git.
", "votes": null, "replies": [ { "id": 2153580, "author_name": "alejopaullier", "author_url": "", "post_date": "02/21/2023 14:06:51", "content": "Thanks a lot. I went for the SQLite approach mentioned in the graphnet_example notebook which involves creating a huge database
", "votes": null, "replies": [ { "id": 2163149, "author_name": "rasmusrse", "author_url": "", "post_date": "02/28/2023 16:13:02", "content": "You can with relatively little effort switch from the SQLite backend to the parquet backend in graphnet. We have a ParquetDataset class already - with a few light modifications it'll work with the competition data.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2166215, "author_name": "aneighbourhood", "author_url": "", "post_date": "03/02/2023 17:25:45", "content": "@seungmoklee would you mind sharing how to load the parquet files. I've been struggling with how to use the parquet files for this competition
", "votes": null, "replies": [] } ] } ] }, { "id": 2159340, "author_name": "dipamc77", "author_url": "", "post_date": "02/25/2023 17:26:32", "content": "Are the models of comparable size? I think your shared LSTM model was way smaller than graphnet.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2150371": "I tried to train the graph-net provided by the host on my own.\nI've copied all the required codes from the git, and now I can train and predict some angles.\nBut I found that the training is much slower than my LSTM.\n\nTo give you some numbers, training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours while training my LSTM through 50 batches with batch_size of 128 took about 3 hours, with the same machine.\n\nWell, the experiment is not controlled at all.\nThis is the first time using pytorch and GNN, so I don't have any sense of time...\n\nIs this long time consumption expectable? Is GNN generally slower than small LSTM?\nOr does it indicate some potential inefficiency in my code?", "2150506": "At first glance, it seems weird that an LSTM (so a sequential model) outperforms a fully parallelized model in speed.\nWhat are your machine ressources ? My implementation of graphnet takes several hours to train too.", "2150766": "I'm using a machine with RTX2080.\nSo, several hours is not a big deal, right?", "2151442": "\"training the graph-net through 50 batches of data, with batch_size of 32, took about 5 hours\"\n\nWhen you say 50 batches, do you mean the batches in the /train folder in the input data, or do you mean batches of 32 samples?", "2151860": "The earlier one. 50 batch files, each containing the 200,000 events I meant.", "2153497": "seungmoklee how did you train GraphNet with 50 batch files? At least for me that would mean a ~100GB file that would definitely lead to OOM", "2153554": "Using pytorch dataloader, I could read one batch, train it, discard the batch from memory, read the next batch, train it, ...", "2153557": "alejopaullier You can find an example in the graph-net git.", "2153580": "Thanks a lot. I went for the SQLite approach mentioned in the graphnet_example notebook which involves creating a huge database", "2159340": "Are the models of comparable size? I think your shared LSTM model was way smaller than graphnet.", "2163149": "You can with relatively little effort switch from the SQLite backend to the parquet backend in graphnet. We have a ParquetDataset class already - with a few light modifications it'll work with the competition data.", "2166215": "seungmoklee would you mind sharing how to load the parquet files. I've been struggling with how to use the parquet files for this competition" }, "source": "meta" }