{ "id": 384833, "title": "Trained GraphNet with and without auxiliary=True", "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/384833", "author_name": "", "post_date": "2023-02-09T17:14:06.122625800Z", "votes": 7, "comment_count": 11, "views": 0, "content": "

I trained the 2 GraphNet models from the example:

\n
    \n
  1. Model 1 was trained with one batch with no auxiliary=True cases and inferred on batch 51 also with no auxiliary=True cases.
    2. \n
  2. Model 2 was trained with one batch (same batch_id as model 1) but this contained both auxiliary types and inferred on batch 51 with also both auxiliary types.
    3. \n
\n

Model 1 Performance:
\n\"\"

\n

Model 2 Performance:
\n\"\"

\n

As you can see, there is barely any difference in performance which makes me wonder if my assumption that training without True makes a difference.

\n

Note: Maybe training with only 1 batch is not representative. Perhaps if a model was trained on several batches with no True can result in a much significant difference in performance.

\n

Edit: Run the same experiment but with larger batches (5) and there is a difference in performance.

", "messages": [ { "id": "2137010", "postDate": "02/09/2023 17:14:06", "content": "

I trained the 2 GraphNet models from the example:

\n
    \n
  1. Model 1 was trained with one batch with no auxiliary=True cases and inferred on batch 51 also with no auxiliary=True cases.
    2. \n
  2. Model 2 was trained with one batch (same batch_id as model 1) but this contained both auxiliary types and inferred on batch 51 with also both auxiliary types.
    3. \n
\n

Model 1 Performance:
\n\"\"

\n

Model 2 Performance:
\n\"\"

\n

As you can see, there is barely any difference in performance which makes me wonder if my assumption that training without True makes a difference.

\n

Note: Maybe training with only 1 batch is not representative. Perhaps if a model was trained on several batches with no True can result in a much significant difference in performance.

\n

Edit: Run the same experiment but with larger batches (5) and there is a difference in performance.

", "rawMarkdown": "I trained the 2 GraphNet models from the example:\n1. Model 1 was trained with one batch with no auxiliary=`True` cases and inferred on batch 51 also with no auxiliary=`True` cases.\n2. Model 2 was trained with one batch (same `batch_id` as model 1) but this contained both auxiliary types and inferred on batch 51 with also both auxiliary types.\n\nModel 1 Performance:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F5e295363960e5394770a55ebdd336801%2FScreen%20Shot%202023-02-09%20at%2014.12.51.png?generation=1675962789188316&alt=media)\n\nModel 2 Performance:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F63c9cad544f826970350e4932ad66971%2FScreen%20Shot%202023-02-09%20at%2010.52.28.png?generation=1675962740252565&alt=media)\n\nAs you can see, there is barely any difference in performance which makes me wonder if my assumption that training without `True` makes a difference.\n\n**Note:** Maybe training with only 1 batch is not representative. Perhaps if a model was trained on several batches with no `True` can result in a much significant difference in performance.\n\n**Edit:** Run the same experiment but with larger batches (5) and there is a difference in performance.", "votes": null }, { "id": "2137077", "postDate": "02/09/2023 18:04:35", "content": "

How about training times? And did you add any logic for events with less than n rows with aux=False?

\n

Thanks for running the experiment, it's interesting.

", "rawMarkdown": "How about training times? And did you add any logic for events with less than n rows with aux=False? \n\nThanks for running the experiment, it's interesting.", "votes": null }, { "id": "2137190", "postDate": "02/09/2023 19:37:10", "content": "

The configuration of the experiments is like in the example. Only events with less than 200 rows are kept. Regarding training times, Model 2 converged faster than Model 1. I think Model 1 took ~2hrs, and Model 2 took ~1.5 hrs

", "rawMarkdown": "The configuration of the experiments is like in the example. Only events with less than 200 rows are kept. Regarding training times, Model 2 converged faster than Model 1. I think Model 1 took ~2hrs, and Model 2 took ~1.5 hrs", "votes": null }, { "id": "2137310", "postDate": "02/09/2023 20:57:00", "content": "

Is that what the scores on the leaderboard represent: Mean angular error?

", "rawMarkdown": "Is that what the scores on the leaderboard represent: Mean angular error?", "votes": null }, { "id": "2137316", "postDate": "02/09/2023 21:01:37", "content": "

Indeed. You can check the competition's metric here and its implementation here. @michaelbarrett

", "rawMarkdown": "Indeed. You can check the competition's metric [here](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/overview/evaluation) and its implementation [here](https://www.kaggle.com/code/sohier/mean-angular-error). @michaelbarrett", "votes": null }, { "id": "2137634", "postDate": "02/10/2023 07:49:48", "content": "

@alejopaullier Interesting test! I would suggest that you repeat this for models that have a larger training budget. I think at least 5 batches (1 million events) is needed. Preferably 2 - 3 million events. Good luck!

", "rawMarkdown": "alejopaullier Interesting test! I would suggest that you repeat this for models that have a larger training budget. I think at least 5 batches (1 million events) is needed. Preferably 2 - 3 million events. Good luck!", "votes": null }, { "id": "2137932", "postDate": "02/10/2023 12:46:04", "content": "

Thanks @rasmusrse I will give it a try. Most difficulties are related to memory constraints. If I am able to train with larger data I'll come back with some updates.

", "rawMarkdown": "Thanks @rasmusrse I will give it a try. Most difficulties are related to memory constraints. If I am able to train with larger data I'll come back with some updates.", "votes": null }, { "id": "2141292", "postDate": "02/12/2023 15:58:34", "content": "

Update:

\n

Trained a model with batches 60 to 64 (5 batches) with only auxiliary=False, same experimental conditions as before, and:

\n
    \n
  1. Inferred on batch 51 with only auxiliary=False. Performance: 1.1
    2. \n
  2. Inferred on batch 51 with both types of auxiliary. Performance: 1.29
    3. \n
\n

There is a difference in performance.

", "rawMarkdown": "**Update:**\n\nTrained a model with batches 60 to 64 (5 batches) with only auxiliary=`False`, same experimental conditions as before, and:\n1. Inferred on batch 51 with only auxiliary=`False`. Performance: `1.1`\n2. Inferred on batch 51 with both types of auxiliary. Performance: `1.29`\n\nThere is a difference in performance.", "votes": null }, { "id": "2141332", "postDate": "02/12/2023 16:54:06", "content": "

my current score and model (which is not graphnet) uses both Auxiliary (True and False). If i keep only one auxiliary i get degraded performance.

", "rawMarkdown": "my current score and model (which is not graphnet) uses both Auxiliary (True and False). If i keep only one auxiliary i get degraded performance.", "votes": null }, { "id": "2142272", "postDate": "02/13/2023 12:22:28", "content": "

@drhabib thanks for the insight. Have you tried training with only False and inferred on only False? Training with only False and prediciting on a dataset that has both auxiliary yields a worse performance. These experiments were done to verify if its worthy trying to train a False/True classifier to get only the False category.

", "rawMarkdown": "drhabib thanks for the insight. Have you tried training with only False and inferred on only False? Training with only False and prediciting on a dataset that has both auxiliary yields a worse performance. These experiments were done to verify if its worthy trying to train a False/True classifier to get only the False category.", "votes": null }, { "id": "2142438", "postDate": "02/13/2023 14:26:40", "content": "

Hi @drhabib, what gpus do you need to train this model?

", "rawMarkdown": "Hi @drhabib, what gpus do you need to train this model?", "votes": null }, { "id": "2142445", "postDate": "02/13/2023 14:32:58", "content": "

i used 1 3090 …

", "rawMarkdown": "i used 1 3090 …", "votes": null } ], "comments": [ { "id": 2137077, "author_name": "roberthatch", "author_url": "", "post_date": "02/09/2023 18:04:35", "content": "

How about training times? And did you add any logic for events with less than n rows with aux=False?

\n

Thanks for running the experiment, it's interesting.

", "votes": null, "replies": [ { "id": 2137190, "author_name": "alejopaullier", "author_url": "", "post_date": "02/09/2023 19:37:10", "content": "

The configuration of the experiments is like in the example. Only events with less than 200 rows are kept. Regarding training times, Model 2 converged faster than Model 1. I think Model 1 took ~2hrs, and Model 2 took ~1.5 hrs

", "votes": null, "replies": [] } ] }, { "id": 2137310, "author_name": "michaelbarrett", "author_url": "", "post_date": "02/09/2023 20:57:00", "content": "

Is that what the scores on the leaderboard represent: Mean angular error?

", "votes": null, "replies": [ { "id": 2137316, "author_name": "alejopaullier", "author_url": "", "post_date": "02/09/2023 21:01:37", "content": "

Indeed. You can check the competition's metric here and its implementation here. @michaelbarrett

", "votes": null, "replies": [] } ] }, { "id": 2137634, "author_name": "rasmusrse", "author_url": "", "post_date": "02/10/2023 07:49:48", "content": "

@alejopaullier Interesting test! I would suggest that you repeat this for models that have a larger training budget. I think at least 5 batches (1 million events) is needed. Preferably 2 - 3 million events. Good luck!

", "votes": null, "replies": [ { "id": 2137932, "author_name": "alejopaullier", "author_url": "", "post_date": "02/10/2023 12:46:04", "content": "

Thanks @rasmusrse I will give it a try. Most difficulties are related to memory constraints. If I am able to train with larger data I'll come back with some updates.

", "votes": null, "replies": [] } ] }, { "id": 2141292, "author_name": "alejopaullier", "author_url": "", "post_date": "02/12/2023 15:58:34", "content": "

Update:

\n

Trained a model with batches 60 to 64 (5 batches) with only auxiliary=False, same experimental conditions as before, and:

\n
    \n
  1. Inferred on batch 51 with only auxiliary=False. Performance: 1.1
    2. \n
  2. Inferred on batch 51 with both types of auxiliary. Performance: 1.29
    3. \n
\n

There is a difference in performance.

", "votes": null, "replies": [ { "id": 2141332, "author_name": "drhabib", "author_url": "", "post_date": "02/12/2023 16:54:06", "content": "

my current score and model (which is not graphnet) uses both Auxiliary (True and False). If i keep only one auxiliary i get degraded performance.

", "votes": null, "replies": [ { "id": 2142272, "author_name": "alejopaullier", "author_url": "", "post_date": "02/13/2023 12:22:28", "content": "

@drhabib thanks for the insight. Have you tried training with only False and inferred on only False? Training with only False and prediciting on a dataset that has both auxiliary yields a worse performance. These experiments were done to verify if its worthy trying to train a False/True classifier to get only the False category.

", "votes": null, "replies": [] }, { "id": 2142438, "author_name": "forcewithme", "author_url": "", "post_date": "02/13/2023 14:26:40", "content": "

Hi @drhabib, what gpus do you need to train this model?

", "votes": null, "replies": [ { "id": 2142445, "author_name": "drhabib", "author_url": "", "post_date": "02/13/2023 14:32:58", "content": "

i used 1 3090 …

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2137010": "I trained the 2 GraphNet models from the example:\n1. Model 1 was trained with one batch with no auxiliary=`True` cases and inferred on batch 51 also with no auxiliary=`True` cases.\n2. Model 2 was trained with one batch (same `batch_id` as model 1) but this contained both auxiliary types and inferred on batch 51 with also both auxiliary types.\n\nModel 1 Performance:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F5e295363960e5394770a55ebdd336801%2FScreen%20Shot%202023-02-09%20at%2014.12.51.png?generation=1675962789188316&alt=media)\n\nModel 2 Performance:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F63c9cad544f826970350e4932ad66971%2FScreen%20Shot%202023-02-09%20at%2010.52.28.png?generation=1675962740252565&alt=media)\n\nAs you can see, there is barely any difference in performance which makes me wonder if my assumption that training without `True` makes a difference.\n\n**Note:** Maybe training with only 1 batch is not representative. Perhaps if a model was trained on several batches with no `True` can result in a much significant difference in performance.\n\n**Edit:** Run the same experiment but with larger batches (5) and there is a difference in performance.", "2137077": "How about training times? And did you add any logic for events with less than n rows with aux=False? \n\nThanks for running the experiment, it's interesting.", "2137190": "The configuration of the experiments is like in the example. Only events with less than 200 rows are kept. Regarding training times, Model 2 converged faster than Model 1. I think Model 1 took ~2hrs, and Model 2 took ~1.5 hrs", "2137310": "Is that what the scores on the leaderboard represent: Mean angular error?", "2137316": "Indeed. You can check the competition's metric [here](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/overview/evaluation) and its implementation [here](https://www.kaggle.com/code/sohier/mean-angular-error). @michaelbarrett", "2137634": "alejopaullier Interesting test! I would suggest that you repeat this for models that have a larger training budget. I think at least 5 batches (1 million events) is needed. Preferably 2 - 3 million events. Good luck!", "2137932": "Thanks @rasmusrse I will give it a try. Most difficulties are related to memory constraints. If I am able to train with larger data I'll come back with some updates.", "2141292": "**Update:**\n\nTrained a model with batches 60 to 64 (5 batches) with only auxiliary=`False`, same experimental conditions as before, and:\n1. Inferred on batch 51 with only auxiliary=`False`. Performance: `1.1`\n2. Inferred on batch 51 with both types of auxiliary. Performance: `1.29`\n\nThere is a difference in performance.", "2141332": "my current score and model (which is not graphnet) uses both Auxiliary (True and False). If i keep only one auxiliary i get degraded performance.", "2142272": "drhabib thanks for the insight. Have you tried training with only False and inferred on only False? Training with only False and prediciting on a dataset that has both auxiliary yields a worse performance. These experiments were done to verify if its worthy trying to train a False/True classifier to get only the False category.", "2142438": "Hi @drhabib, what gpus do you need to train this model?", "2142445": "i used 1 3090 …" }, "source": "meta" }