{ "id": 269977, "title": "Tips for weathering the storm?", "url": "/competitions/g2net-gravitational-wave-detection/discussion/269977", "author_name": "", "post_date": "2021-09-03T00:52:57.596882Z", "votes": 9, "comment_count": 12, "views": 0, "content": "

My models used to be <3min / epoch but now they're +20min / epoch, so I dedicated the day to hacking around making things run faster. Even yet, I feel like if I don't invest 3-6h per day to experimenting, I'm going to get flushed on private lb. But we still have +27 days(!) in this marathon, which of course isn't sustainable. Any tips on weathering the storm?

\n

\"\"

", "messages": [ { "id": "1501127", "postDate": "09/03/2021 00:52:57", "content": "

My models used to be <3min / epoch but now they're +20min / epoch, so I dedicated the day to hacking around making things run faster. Even yet, I feel like if I don't invest 3-6h per day to experimenting, I'm going to get flushed on private lb. But we still have +27 days(!) in this marathon, which of course isn't sustainable. Any tips on weathering the storm?

\n

\"\"

", "rawMarkdown": "My models used to be <3min / epoch but now they're +20min / epoch, so I dedicated the day to hacking around making things run faster. Even yet, I feel like if I don't invest 3-6h per day to experimenting, I'm going to get flushed on private lb. But we still have +27 days(!) in this marathon, which of course isn't sustainable. Any tips on weathering the storm?\n\n![](https://i.imgur.com/ICOPB8n.jpg)", "votes": null }, { "id": "1501441", "postDate": "09/03/2021 08:17:07", "content": "

what's your setup that get 3min/epoch? I'm using colab pro (P100) and I get 25 min/epoch with B0 batch=128 spectrogram size = (69, 129)

", "rawMarkdown": "what's your setup that get 3min/epoch? I'm using colab pro (P100) and I get 25 min/epoch with B0 batch=128 spectrogram size = (69, 129)", "votes": null }, { "id": "1501473", "postDate": "09/03/2021 08:54:52", "content": "

Do all your experiments on small models and using a fraction of the dataset (10-20%). Using this method I managed to get a ResNet50d to perform pretty well on a subset of the data in relatively little compute hours. Once I was ready to train on the full dataset, the same model scored 0.8788

", "rawMarkdown": "Do all your experiments on small models and using a fraction of the dataset (10-20%). Using this method I managed to get a ResNet50d to perform pretty well on a subset of the data in relatively little compute hours. Once I was ready to train on the full dataset, the same model scored 0.8788", "votes": null }, { "id": "1501501", "postDate": "09/03/2021 09:28:22", "content": "

Do you do heavy regularization when you train on only 10% subset? When I first joined this comp, I tried training on only 10% of the data and I found that the model ended up overfit too easily and the performance on cv does not reflect the performance when I train on full-data set at all so it was hard to keep track of what works and what doesn't.

", "rawMarkdown": "Do you do heavy regularization when you train on only 10% subset? When I first joined this comp, I tried training on only 10% of the data and I found that the model ended up overfit too easily and the performance on cv does not reflect the performance when I train on full-data set at all so it was hard to keep track of what works and what doesn't.", "votes": null }, { "id": "1501628", "postDate": "09/03/2021 11:29:20", "content": "

Do you use TPU ?.

", "rawMarkdown": "Do you use TPU ?.", "votes": null }, { "id": "1501690", "postDate": "09/03/2021 12:26:45", "content": "

Local 3090s

", "rawMarkdown": "Local 3090s", "votes": null }, { "id": "1501692", "postDate": "09/03/2021 12:27:52", "content": "

Started with 1d then moved to 2d

", "rawMarkdown": "Started with 1d then moved to 2d", "votes": null }, { "id": "1501726", "postDate": "09/03/2021 12:59:29", "content": "

I use the same params for the 10% and full models. Regularisation is important but I found effective denoising much more effective against over fitting

", "rawMarkdown": "I use the same params for the 10% and full models. Regularisation is important but I found effective denoising much more effective against over fitting", "votes": null }, { "id": "1502605", "postDate": "09/04/2021 13:15:07", "content": "

GCP is your friend I guess. 😄

\n

Joke aside, I guess the best thing to do is to try to experiment on a smallish subset and then run on the full dataset once you are sure. The hard thing is of course to make sure that the smallish subset is representative of the full dataset.

", "rawMarkdown": "GCP is your friend I guess. 😄\n\nJoke aside, I guess the best thing to do is to try to experiment on a smallish subset and then run on the full dataset once you are sure. The hard thing is of course to make sure that the smallish subset is representative of the full dataset.", "votes": null }, { "id": "1527567", "postDate": "09/28/2021 21:50:58", "content": "

Tomorrow, I hope I'll have my first good sleep in a while.

", "rawMarkdown": "Tomorrow, I hope I'll have my first good sleep in a while.", "votes": null }, { "id": "1527974", "postDate": "09/29/2021 08:41:44", "content": "

I can't yet, I also joined the GLR competitions! LOL.

", "rawMarkdown": "I can't yet, I also joined the GLR competitions! LOL.", "votes": null }, { "id": "1527980", "postDate": "09/29/2021 08:48:06", "content": "

@cpmpml good luck to you guys in GLR, on an unrelated note, any guess why GLR lasting only for a month this time? Previous (all of them?) was 3 month if im correct

", "rawMarkdown": "cpmpml good luck to you guys in GLR, on an unrelated note, any guess why GLR lasting only for a month this time? Previous (all of them?) was 3 month if im correct", "votes": null }, { "id": "1559741", "postDate": "10/27/2021 07:10:55", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "votes": null } ], "comments": [ { "id": 1501441, "author_name": "brachester", "author_url": "", "post_date": "09/03/2021 08:17:07", "content": "

what's your setup that get 3min/epoch? I'm using colab pro (P100) and I get 25 min/epoch with B0 batch=128 spectrogram size = (69, 129)

", "votes": null, "replies": [ { "id": 1501692, "author_name": "authman", "author_url": "", "post_date": "09/03/2021 12:27:52", "content": "

Started with 1d then moved to 2d

", "votes": null, "replies": [] } ] }, { "id": 1501473, "author_name": "anjum48", "author_url": "", "post_date": "09/03/2021 08:54:52", "content": "

Do all your experiments on small models and using a fraction of the dataset (10-20%). Using this method I managed to get a ResNet50d to perform pretty well on a subset of the data in relatively little compute hours. Once I was ready to train on the full dataset, the same model scored 0.8788

", "votes": null, "replies": [ { "id": 1501501, "author_name": "brachester", "author_url": "", "post_date": "09/03/2021 09:28:22", "content": "

Do you do heavy regularization when you train on only 10% subset? When I first joined this comp, I tried training on only 10% of the data and I found that the model ended up overfit too easily and the performance on cv does not reflect the performance when I train on full-data set at all so it was hard to keep track of what works and what doesn't.

", "votes": null, "replies": [] }, { "id": 1501726, "author_name": "anjum48", "author_url": "", "post_date": "09/03/2021 12:59:29", "content": "

I use the same params for the 10% and full models. Regularisation is important but I found effective denoising much more effective against over fitting

", "votes": null, "replies": [] } ] }, { "id": 1501628, "author_name": "kingkong153", "author_url": "", "post_date": "09/03/2021 11:29:20", "content": "

Do you use TPU ?.

", "votes": null, "replies": [ { "id": 1501690, "author_name": "authman", "author_url": "", "post_date": "09/03/2021 12:26:45", "content": "

Local 3090s

", "votes": null, "replies": [] } ] }, { "id": 1502605, "author_name": "yassinealouini", "author_url": "", "post_date": "09/04/2021 13:15:07", "content": "

GCP is your friend I guess. 😄

\n

Joke aside, I guess the best thing to do is to try to experiment on a smallish subset and then run on the full dataset once you are sure. The hard thing is of course to make sure that the smallish subset is representative of the full dataset.

", "votes": null, "replies": [] }, { "id": 1527567, "author_name": "authman", "author_url": "", "post_date": "09/28/2021 21:50:58", "content": "

Tomorrow, I hope I'll have my first good sleep in a while.

", "votes": null, "replies": [ { "id": 1527974, "author_name": "cpmpml", "author_url": "", "post_date": "09/29/2021 08:41:44", "content": "

I can't yet, I also joined the GLR competitions! LOL.

", "votes": null, "replies": [] }, { "id": 1527980, "author_name": "bakeryproducts", "author_url": "", "post_date": "09/29/2021 08:48:06", "content": "

@cpmpml good luck to you guys in GLR, on an unrelated note, any guess why GLR lasting only for a month this time? Previous (all of them?) was 3 month if im correct

", "votes": null, "replies": [] } ] }, { "id": 1559741, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 07:10:55", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1501127": "My models used to be <3min / epoch but now they're +20min / epoch, so I dedicated the day to hacking around making things run faster. Even yet, I feel like if I don't invest 3-6h per day to experimenting, I'm going to get flushed on private lb. But we still have +27 days(!) in this marathon, which of course isn't sustainable. Any tips on weathering the storm?\n\n![](https://i.imgur.com/ICOPB8n.jpg)", "1501441": "what's your setup that get 3min/epoch? I'm using colab pro (P100) and I get 25 min/epoch with B0 batch=128 spectrogram size = (69, 129)", "1501473": "Do all your experiments on small models and using a fraction of the dataset (10-20%). Using this method I managed to get a ResNet50d to perform pretty well on a subset of the data in relatively little compute hours. Once I was ready to train on the full dataset, the same model scored 0.8788", "1501501": "Do you do heavy regularization when you train on only 10% subset? When I first joined this comp, I tried training on only 10% of the data and I found that the model ended up overfit too easily and the performance on cv does not reflect the performance when I train on full-data set at all so it was hard to keep track of what works and what doesn't.", "1501628": "Do you use TPU ?.", "1501690": "Local 3090s", "1501692": "Started with 1d then moved to 2d", "1501726": "I use the same params for the 10% and full models. Regularisation is important but I found effective denoising much more effective against over fitting", "1502605": "GCP is your friend I guess. 😄\n\nJoke aside, I guess the best thing to do is to try to experiment on a smallish subset and then run on the full dataset once you are sure. The hard thing is of course to make sure that the smallish subset is representative of the full dataset.", "1527567": "Tomorrow, I hope I'll have my first good sleep in a while.", "1527974": "I can't yet, I also joined the GLR competitions! LOL.", "1527980": "cpmpml good luck to you guys in GLR, on an unrelated note, any guess why GLR lasting only for a month this time? Previous (all of them?) was 3 month if im correct", "1559741": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris" }, "source": "meta" }