Hello everyone!
\nGraphNeT is an open-source Python framework aimed at providing high quality, user friendly, end-to-end functionality to perform reconstruction tasks at neutrino telescopes using graph neural networks (GNNs). In a recent paper we showcased how a GNN implemented in GraphNeT performs on a series of reconstruction tasks in IceCube.
\nWe've created an example notebook that contains everything you need to install GraphNeT and train/apply this model to the competition data. Since the code is public, you should have every opportunity to tinker with the library, the GNN or to implement a completely different solution using GraphNeT. The notebook contains a pre-trained version of the GNN shown in the paper with a similar training configuration, which has been trained on batches 1 to 50. The notebook is available here.
\nWe've submitted this pre-trained model to the leaderboard using this notebook as \"GraphNeT Baseline\". This submission is not a prize contender.
", "messages": [ { "id": "2128870", "postDate": "02/04/2023 06:38:48", "content": "Hello everyone!
\nGraphNeT is an open-source Python framework aimed at providing high quality, user friendly, end-to-end functionality to perform reconstruction tasks at neutrino telescopes using graph neural networks (GNNs). In a recent paper we showcased how a GNN implemented in GraphNeT performs on a series of reconstruction tasks in IceCube.
\nWe've created an example notebook that contains everything you need to install GraphNeT and train/apply this model to the competition data. Since the code is public, you should have every opportunity to tinker with the library, the GNN or to implement a completely different solution using GraphNeT. The notebook contains a pre-trained version of the GNN shown in the paper with a similar training configuration, which has been trained on batches 1 to 50. The notebook is available here.
\nWe've submitted this pre-trained model to the leaderboard using this notebook as \"GraphNeT Baseline\". This submission is not a prize contender.
", "rawMarkdown": "Hello everyone!\n\n[GraphNeT](https://github.com/graphnet-team/graphnet) is an open-source Python framework aimed at providing high quality, user friendly, end-to-end functionality to perform reconstruction tasks at neutrino telescopes using graph neural networks (GNNs). In a [recent paper](https://iopscience.iop.org/article/10.1088/1748-0221/17/11/P11003) we showcased how a GNN implemented in GraphNeT performs on a series of reconstruction tasks in IceCube. \n\nWe've created an example notebook that contains everything you need to install GraphNeT and train/apply this model to the competition data. Since the code is public, you should have every opportunity to tinker with the library, the GNN or to implement a completely different solution using GraphNeT. The notebook contains a pre-trained version of the GNN shown in the paper with a similar training configuration, which has been trained on batches 1 to 50. The notebook is available [here](https://www.kaggle.com/code/rasmusrse/graphnet-example). \n\nWe've submitted this pre-trained model to the leaderboard using [this notebook](https://www.kaggle.com/code/rasmusrse/graphnet-baseline-submission) as \"GraphNeT Baseline\". **This submission is not a prize contender**.", "votes": null }, { "id": "2128904", "postDate": "02/04/2023 07:03:56", "content": "Thank you for providing the baseline example. I believe it's a great approach that will enhance the competition and eventually improve the accuracy of neutrino reconstruction :) The performance of DybeEdge right out of the box is really impressive. How long does it take to train the model on 50 batches?
", "rawMarkdown": "Thank you for providing the baseline example. I believe it's a great approach that will enhance the competition and eventually improve the accuracy of neutrino reconstruction :) The performance of DybeEdge right out of the box is really impressive. How long does it take to train the model on 50 batches?", "votes": null }, { "id": "2128927", "postDate": "02/04/2023 07:26:48", "content": "Thank you! The training itself should conclude in well under the 12h Kaggle limit if on GPU. However, the conversion of parquet files to sqlite for 50 batches would probably take close to 12 hours. As I point out in the end of the example notebook, one could just modify the dataset class to read the competition parquet files directly instead.
", "rawMarkdown": "Thank you! The training itself should conclude in well under the 12h Kaggle limit if on GPU. However, the conversion of parquet files to sqlite for 50 batches would probably take close to 12 hours. As I point out in the end of the example notebook, one could just modify the dataset class to read the competition parquet files directly instead.", "votes": null }, { "id": "2128983", "postDate": "02/04/2023 08:33:15", "content": "Having baseline is great, early sharing prize was also a great idea imo. idk why @anjum48 was having second thoughts about sharing or not his solution, it will for sure be overruned with only 2 weeks into competition. But 2 first weeks probably is not enough time to build a solid alternatives to existing solutions. Most probable ESP winner just first to semi-implement this same baseline shared by @rasmusrse
", "rawMarkdown": "Having baseline is great, early sharing prize was also a great idea imo. idk why @anjum48 was having second thoughts about sharing or not his solution, it will for sure be overruned with only 2 weeks into competition. But 2 first weeks probably is not enough time to build a solid alternatives to existing solutions. Most probable ESP winner just first to semi-implement this same baseline shared by @rasmusrse", "votes": null }, { "id": "2128995", "postDate": "02/04/2023 08:37:22", "content": "You're totally right. The DynNet baseline is pretty solid IMO too - I have a long list of things that I tried that didn't work. It'll be interesting to see what people come up with
", "rawMarkdown": "You're totally right. The DynNet baseline is pretty solid IMO too - I have a long list of things that I tried that didn't work. It'll be interesting to see what people come up with", "votes": null }, { "id": "2128998", "postDate": "02/04/2023 08:40:15", "content": "just curious, what did you decide on sharing before you notice 1.018 graphnet baseline on LB? And how many graphnet solutions you managed to train over what, a week? given other things that didnt work
", "rawMarkdown": "just curious, what did you decide on sharing before you notice 1.018 graphnet baseline on LB? And how many graphnet solutions you managed to train over what, a week? given other things that didnt work", "votes": null }, { "id": "2129004", "postDate": "02/04/2023 08:47:17", "content": "I knew the Graphnet baseline was going to appear anyway. I saw that commits were being made in the Graphnet repo specific to this competition over the last few days.
\nRegarding other solutions, I have a tonne of other GNNs, a few of which are now starting to surpass DynEdge. But in terms of training speed & simplicity, DynEdge is very strong
", "rawMarkdown": "I knew the Graphnet baseline was going to appear anyway. I saw that commits were being made in the Graphnet repo specific to this competition over the last few days.\n\nRegarding other solutions, I have a tonne of other GNNs, a few of which are now starting to surpass DynEdge. But in terms of training speed & simplicity, DynEdge is very strong", "votes": null }, { "id": "2129015", "postDate": "02/04/2023 08:56:25", "content": "Thank you for sharing, i hope you ll get ESP, well deserved
", "rawMarkdown": "Thank you for sharing, i hope you ll get ESP, well deserved", "votes": null }, { "id": "2129066", "postDate": "02/04/2023 09:38:03", "content": "great sharing
", "rawMarkdown": "great sharing", "votes": null }, { "id": "2129385", "postDate": "02/04/2023 15:34:48", "content": "Roughly how long does a LB submission take with this baseline notebook?
", "rawMarkdown": "Roughly how long does a LB submission take with this baseline notebook?", "votes": null }, { "id": "2129410", "postDate": "02/04/2023 15:59:39", "content": "It takes less than 3 hours for the submission notebook to get scored.
", "rawMarkdown": "It takes less than 3 hours for the submission notebook to get scored.", "votes": null }, { "id": "2129623", "postDate": "02/04/2023 18:56:07", "content": "@rasmusrse I get out of memory error on kaggle with GPU when trying to add in the code for converting to sqlite format below.
\ninput_data_folder = \ngeometry_table = pd.read_csv()\nmeta_data_path = \n\n\ndatabase_path = \nconvert_to_sqlite(meta_data_path,\n database_path=database_path,\n input_data_folder=input_data_folder,\n batch_ids = [])\nNote that if trying CPU run, I get this Import error (condensed):
\nOSError Traceback (most recent call last)\n---> graphnet.data.sqlite.sqlite_utilities create_table\n----> .sqlite_dataset SQLiteDataset\n----> graphnet.data.dataset Dataset, ColumnMissingException\n---> torch_geometric.data Data\n----> torch_geometric.data\n----> .data Data\n----> torch_sparse SparseTensor\n/opt/conda/lib/python3/site-packages/torch_sparse/__init__.py <module>\n spec = cuda_spec cpu_spec\n spec :\n---> torch.ops.load_library(spec.origin)\n/opt/conda/lib/python3/site-packages/torch/_ops.py load_library(self, path)\n \n \n--> ctypes.CDLL(path)\n/opt/conda/lib/python3/ctypes/__init__.py __init__(self, name, mode, handle, use_errno, use_last_error)\n handle :\n--> self._handle = _dlopen(self._name, mode)\n\nOSError: libcusparse.so: cannot shared file: No such file directory\nAny suggestions on fixing it?
\nI suppose CPU run for this preprocessing step would be much better on kaggle anyways, if there's a way to fix that second error. Or a streamlined import with fewer pip installs or manual addition of just create_table might be even better?
\nI would assume that the 32GB memory of CPU notebooks would be more than enough to eliminate the memory error side of things.
", "rawMarkdown": "rasmusrse I get out of memory error on kaggle with GPU when trying to add in the code for converting to sqlite format below.\n\n```python\ninput_data_folder = '/kaggle/input/icecube-neutrinos-in-deep-ice/train'\ngeometry_table = pd.read_csv('/kaggle/input/icecube-neutrinos-in-deep-ice/sensor_geometry.csv')\nmeta_data_path = '/kaggle/input/icecube-neutrinos-in-deep-ice/train_meta.parquet'\n\n#batch_51\ndatabase_path = '/kaggle/working/batch_51'\nconvert_to_sqlite(meta_data_path,\n database_path=database_path,\n input_data_folder=input_data_folder,\n batch_ids = [51])\n```\n\n\nNote that if trying CPU run, I get this Import error (condensed):\n\n```python\nOSError Traceback (most recent call last)\n---> 10 from graphnet.data.sqlite.sqlite_utilities import create_table\n----> 9 from .sqlite_dataset import SQLiteDataset\n----> 7 from graphnet.data.dataset import Dataset, ColumnMissingException\n---> 10 from torch_geometric.data import Data\n----> 4 import torch_geometric.data\n----> 1 from .data import Data\n----> 9 from torch_sparse import SparseTensor\n/opt/conda/lib/python3.7/site-packages/torch_sparse/__init__.py inUpdate: To create a CPU notebook that just does this task, I did the following:
\nThen it ran successfully in 10 minutes (599 seconds) for a single batch [52].
", "rawMarkdown": "Update: To create a CPU notebook that just does this task, I did the following:\n\n* removed graphnet-related pip install and imports\n* import sqlite3\n* manually copied into my notebook three functions from graphnet's [sqlite_utilities.py](https://github.com/graphnet-team/graphnet/blob/57fd0f37b65cfc8f2171b800353e5a4c40e752cb/src/graphnet/data/sqlite/sqlite_utilities.py) file:\n * run_sql_code\n * attach_index\n * create_table\n\nThen it ran successfully in 10 minutes (599 seconds) for a single batch [52].", "votes": null }, { "id": "2129987", "postDate": "02/05/2023 04:32:54", "content": "@rasmusrse Man, having published paper AND code, you are golden! Thanks
", "rawMarkdown": "rasmusrse Man, having published paper AND code, you are golden! Thanks", "votes": null }, { "id": "2130177", "postDate": "02/05/2023 08:50:21", "content": "Hey Robert - glad to hear you got it working. For reference, the error message you show here typically indicates that graphnet has been installed with GPU on a system without one.
", "rawMarkdown": "Hey Robert - glad to hear you got it working. For reference, the error message you show here typically indicates that graphnet has been installed with GPU on a system without one.", "votes": null }, { "id": "2135094", "postDate": "02/08/2023 12:56:58", "content": "If I want to download the dependecies on my on-prem environment, the way to do it is getting them
\nby using:
\nkaggle kernels output rasmusrse/graphnet-baseline-submission -p /path/to/dest
Sorry for stupid question I am very new to kaggle and this is my first competition :)
\nThanks,
Hi @rasmusrse thank you a lot for your great work!! Is there a reason why you use the direction task and not the combination of the azimuth/zenith tasks ? Do you think one approach is better than the other?
", "rawMarkdown": "Hi @rasmusrse thank you a lot for your great work!! Is there a reason why you use the direction task and not the combination of the azimuth/zenith tasks ? Do you think one approach is better than the other?", "votes": null }, { "id": "2164986", "postDate": "03/01/2023 22:46:57", "content": "Thank you for publishing the baseline!
", "rawMarkdown": "Thank you for publishing the baseline!", "votes": null }, { "id": "2171502", "postDate": "03/06/2023 20:53:54", "content": "Hi @rasmusrse I'm getting the following error from the baseline notebook in the cell where the model, build, inference, and data functions are defined:
\nOSError: /opt/conda/lib/python3.7/site-packages/torch_sparse/_version_cuda.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs
Anyone have any idea what's going on?
", "rawMarkdown": "Hi @rasmusrse I'm getting the following error from the baseline notebook in the cell where the model, build, inference, and data functions are defined:\n`OSError: /opt/conda/lib/python3.7/site-packages/torch_sparse/_version_cuda.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs`\n\nAnyone have any idea what's going on?", "votes": null } ], "comments": [ { "id": 2128904, "author_name": "inartimiryasov", "author_url": "", "post_date": "02/04/2023 07:03:56", "content": "Thank you for providing the baseline example. I believe it's a great approach that will enhance the competition and eventually improve the accuracy of neutrino reconstruction :) The performance of DybeEdge right out of the box is really impressive. How long does it take to train the model on 50 batches?
", "votes": null, "replies": [ { "id": 2128927, "author_name": "rasmusrse", "author_url": "", "post_date": "02/04/2023 07:26:48", "content": "Thank you! The training itself should conclude in well under the 12h Kaggle limit if on GPU. However, the conversion of parquet files to sqlite for 50 batches would probably take close to 12 hours. As I point out in the end of the example notebook, one could just modify the dataset class to read the competition parquet files directly instead.
", "votes": null, "replies": [] } ] }, { "id": 2128983, "author_name": "bakeryproducts", "author_url": "", "post_date": "02/04/2023 08:33:15", "content": "Having baseline is great, early sharing prize was also a great idea imo. idk why @anjum48 was having second thoughts about sharing or not his solution, it will for sure be overruned with only 2 weeks into competition. But 2 first weeks probably is not enough time to build a solid alternatives to existing solutions. Most probable ESP winner just first to semi-implement this same baseline shared by @rasmusrse
", "votes": null, "replies": [ { "id": 2128995, "author_name": "anjum48", "author_url": "", "post_date": "02/04/2023 08:37:22", "content": "You're totally right. The DynNet baseline is pretty solid IMO too - I have a long list of things that I tried that didn't work. It'll be interesting to see what people come up with
", "votes": null, "replies": [ { "id": 2128998, "author_name": "bakeryproducts", "author_url": "", "post_date": "02/04/2023 08:40:15", "content": "just curious, what did you decide on sharing before you notice 1.018 graphnet baseline on LB? And how many graphnet solutions you managed to train over what, a week? given other things that didnt work
", "votes": null, "replies": [ { "id": 2129004, "author_name": "anjum48", "author_url": "", "post_date": "02/04/2023 08:47:17", "content": "I knew the Graphnet baseline was going to appear anyway. I saw that commits were being made in the Graphnet repo specific to this competition over the last few days.
\nRegarding other solutions, I have a tonne of other GNNs, a few of which are now starting to surpass DynEdge. But in terms of training speed & simplicity, DynEdge is very strong
", "votes": null, "replies": [ { "id": 2129015, "author_name": "bakeryproducts", "author_url": "", "post_date": "02/04/2023 08:56:25", "content": "Thank you for sharing, i hope you ll get ESP, well deserved
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2129066, "author_name": "moro146", "author_url": "", "post_date": "02/04/2023 09:38:03", "content": "great sharing
", "votes": null, "replies": [] }, { "id": 2129385, "author_name": "roberthatch", "author_url": "", "post_date": "02/04/2023 15:34:48", "content": "Roughly how long does a LB submission take with this baseline notebook?
", "votes": null, "replies": [ { "id": 2129410, "author_name": "rasmusrse", "author_url": "", "post_date": "02/04/2023 15:59:39", "content": "It takes less than 3 hours for the submission notebook to get scored.
", "votes": null, "replies": [] } ] }, { "id": 2129623, "author_name": "roberthatch", "author_url": "", "post_date": "02/04/2023 18:56:07", "content": "@rasmusrse I get out of memory error on kaggle with GPU when trying to add in the code for converting to sqlite format below.
\ninput_data_folder = \ngeometry_table = pd.read_csv()\nmeta_data_path = \n\n\ndatabase_path = \nconvert_to_sqlite(meta_data_path,\n database_path=database_path,\n input_data_folder=input_data_folder,\n batch_ids = [])\nNote that if trying CPU run, I get this Import error (condensed):
\nOSError Traceback (most recent call last)\n---> graphnet.data.sqlite.sqlite_utilities create_table\n----> .sqlite_dataset SQLiteDataset\n----> graphnet.data.dataset Dataset, ColumnMissingException\n---> torch_geometric.data Data\n----> torch_geometric.data\n----> .data Data\n----> torch_sparse SparseTensor\n/opt/conda/lib/python3/site-packages/torch_sparse/__init__.py <module>\n spec = cuda_spec cpu_spec\n spec :\n---> torch.ops.load_library(spec.origin)\n/opt/conda/lib/python3/site-packages/torch/_ops.py load_library(self, path)\n \n \n--> ctypes.CDLL(path)\n/opt/conda/lib/python3/ctypes/__init__.py __init__(self, name, mode, handle, use_errno, use_last_error)\n handle :\n--> self._handle = _dlopen(self._name, mode)\n\nOSError: libcusparse.so: cannot shared file: No such file directory\nAny suggestions on fixing it?
\nI suppose CPU run for this preprocessing step would be much better on kaggle anyways, if there's a way to fix that second error. Or a streamlined import with fewer pip installs or manual addition of just create_table might be even better?
\nI would assume that the 32GB memory of CPU notebooks would be more than enough to eliminate the memory error side of things.
", "votes": null, "replies": [ { "id": 2129688, "author_name": "roberthatch", "author_url": "", "post_date": "02/04/2023 19:54:27", "content": "Update: To create a CPU notebook that just does this task, I did the following:
\nThen it ran successfully in 10 minutes (599 seconds) for a single batch [52].
", "votes": null, "replies": [ { "id": 2130177, "author_name": "rasmusrse", "author_url": "", "post_date": "02/05/2023 08:50:21", "content": "Hey Robert - glad to hear you got it working. For reference, the error message you show here typically indicates that graphnet has been installed with GPU on a system without one.
", "votes": null, "replies": [] } ] } ] }, { "id": 2129987, "author_name": "minhbtnguyen", "author_url": "", "post_date": "02/05/2023 04:32:54", "content": "@rasmusrse Man, having published paper AND code, you are golden! Thanks
", "votes": null, "replies": [] }, { "id": 2135094, "author_name": "riccardogatti", "author_url": "", "post_date": "02/08/2023 12:56:58", "content": "If I want to download the dependecies on my on-prem environment, the way to do it is getting them
\nby using:
\nkaggle kernels output rasmusrse/graphnet-baseline-submission -p /path/to/dest
Sorry for stupid question I am very new to kaggle and this is my first competition :)
\nThanks,
Hi @rasmusrse thank you a lot for your great work!! Is there a reason why you use the direction task and not the combination of the azimuth/zenith tasks ? Do you think one approach is better than the other?
", "votes": null, "replies": [] }, { "id": 2164986, "author_name": "yassinetoughrai", "author_url": "", "post_date": "03/01/2023 22:46:57", "content": "Thank you for publishing the baseline!
", "votes": null, "replies": [] }, { "id": 2171502, "author_name": "samuelzxu", "author_url": "", "post_date": "03/06/2023 20:53:54", "content": "Hi @rasmusrse I'm getting the following error from the baseline notebook in the cell where the model, build, inference, and data functions are defined:
\nOSError: /opt/conda/lib/python3.7/site-packages/torch_sparse/_version_cuda.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs
Anyone have any idea what's going on?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2128870": "Hello everyone!\n\n[GraphNeT](https://github.com/graphnet-team/graphnet) is an open-source Python framework aimed at providing high quality, user friendly, end-to-end functionality to perform reconstruction tasks at neutrino telescopes using graph neural networks (GNNs). In a [recent paper](https://iopscience.iop.org/article/10.1088/1748-0221/17/11/P11003) we showcased how a GNN implemented in GraphNeT performs on a series of reconstruction tasks in IceCube. \n\nWe've created an example notebook that contains everything you need to install GraphNeT and train/apply this model to the competition data. Since the code is public, you should have every opportunity to tinker with the library, the GNN or to implement a completely different solution using GraphNeT. The notebook contains a pre-trained version of the GNN shown in the paper with a similar training configuration, which has been trained on batches 1 to 50. The notebook is available [here](https://www.kaggle.com/code/rasmusrse/graphnet-example). \n\nWe've submitted this pre-trained model to the leaderboard using [this notebook](https://www.kaggle.com/code/rasmusrse/graphnet-baseline-submission) as \"GraphNeT Baseline\". **This submission is not a prize contender**.", "2128904": "Thank you for providing the baseline example. I believe it's a great approach that will enhance the competition and eventually improve the accuracy of neutrino reconstruction :) The performance of DybeEdge right out of the box is really impressive. How long does it take to train the model on 50 batches?", "2128927": "Thank you! The training itself should conclude in well under the 12h Kaggle limit if on GPU. However, the conversion of parquet files to sqlite for 50 batches would probably take close to 12 hours. As I point out in the end of the example notebook, one could just modify the dataset class to read the competition parquet files directly instead.", "2128983": "Having baseline is great, early sharing prize was also a great idea imo. idk why @anjum48 was having second thoughts about sharing or not his solution, it will for sure be overruned with only 2 weeks into competition. But 2 first weeks probably is not enough time to build a solid alternatives to existing solutions. Most probable ESP winner just first to semi-implement this same baseline shared by @rasmusrse", "2128995": "You're totally right. The DynNet baseline is pretty solid IMO too - I have a long list of things that I tried that didn't work. It'll be interesting to see what people come up with", "2128998": "just curious, what did you decide on sharing before you notice 1.018 graphnet baseline on LB? And how many graphnet solutions you managed to train over what, a week? given other things that didnt work", "2129004": "I knew the Graphnet baseline was going to appear anyway. I saw that commits were being made in the Graphnet repo specific to this competition over the last few days.\n\nRegarding other solutions, I have a tonne of other GNNs, a few of which are now starting to surpass DynEdge. But in terms of training speed & simplicity, DynEdge is very strong", "2129015": "Thank you for sharing, i hope you ll get ESP, well deserved", "2129066": "great sharing", "2129385": "Roughly how long does a LB submission take with this baseline notebook?", "2129410": "It takes less than 3 hours for the submission notebook to get scored.", "2129623": "rasmusrse I get out of memory error on kaggle with GPU when trying to add in the code for converting to sqlite format below.\n\n```python\ninput_data_folder = '/kaggle/input/icecube-neutrinos-in-deep-ice/train'\ngeometry_table = pd.read_csv('/kaggle/input/icecube-neutrinos-in-deep-ice/sensor_geometry.csv')\nmeta_data_path = '/kaggle/input/icecube-neutrinos-in-deep-ice/train_meta.parquet'\n\n#batch_51\ndatabase_path = '/kaggle/working/batch_51'\nconvert_to_sqlite(meta_data_path,\n database_path=database_path,\n input_data_folder=input_data_folder,\n batch_ids = [51])\n```\n\n\nNote that if trying CPU run, I get this Import error (condensed):\n\n```python\nOSError Traceback (most recent call last)\n---> 10 from graphnet.data.sqlite.sqlite_utilities import create_table\n----> 9 from .sqlite_dataset import SQLiteDataset\n----> 7 from graphnet.data.dataset import Dataset, ColumnMissingException\n---> 10 from torch_geometric.data import Data\n----> 4 import torch_geometric.data\n----> 1 from .data import Data\n----> 9 from torch_sparse import SparseTensor\n/opt/conda/lib/python3.7/site-packages/torch_sparse/__init__.py in