Hai,
\nI am pretty new to kaggle. The dataset provided is very big. Even by choosing a few subset (50k) of training examples it takes literally hours to train a model just below the kaggle hour limit. Can someone tell whether I am doing something wrong or is it the way it is supposed to be?
\nSorry if this question does not make any sense to you
@shalvinpshaji The majority of the people aren't using kaggle kernels for this. They use either their own machines, google colab or any other web platform. You can use kaggle kernels, by running it for one epoch, saving the weights and optimizers/schedulers states, and then restart training by loading those values. And you keep on going until you are done with training.
", "rawMarkdown": "@shalvinpshaji The majority of the people aren't using kaggle kernels for this. They use either their own machines, google colab or any other web platform. You can use kaggle kernels, by running it for one epoch, saving the weights and optimizers/schedulers states, and then restart training by loading those values. And you keep on going until you are done with training.", "votes": 5 }, { "id": 1269285, "postDate": "2021-04-10T11:49:37.997Z", "content": "Hai,
\nI am pretty new to kaggle. The dataset provided is very big. Even by choosing a few subset (50k) of training examples it takes literally hours to train a model just below the kaggle hour limit. Can someone tell whether I am doing something wrong or is it the way it is supposed to be?
\nSorry if this question does not make any sense to you
I use Kaggle Kernels.
\nI utilize free TPU. And am usually able to process/infer-on around 40 million images per 9 hour session.
\nI will be releasing a very detailed training notebook soon… so stay tuned!
\nAlso feel free to share a link to your notebook, it will allow others to better offer suggestions/help.
", "rawMarkdown": "I use Kaggle Kernels. \n\nI utilize free TPU. And am usually able to process/infer-on around 40 million images per 9 hour session.\n\nI will be releasing a very detailed training notebook soon... so stay tuned!\n\nAlso feel free to share a link to your notebook, it will allow others to better offer suggestions/help.", "votes": 3, "replies": [ { "id": 1273418, "postDate": "2021-04-14T10:29:13.323Z", "content": "Is the TPU notebook runtime limit 9 hours? I thought it was 3 hours for TPU and 9 hours for GPU.
", "rawMarkdown": "Is the TPU notebook runtime limit 9 hours? I thought it was 3 hours for TPU and 9 hours for GPU." } ] }, { "id": 1270060, "postDate": "2021-04-11T08:36:45.157Z", "content": "Hi,
\nthis is also my first competition and I started with GPUs, which was terribly slow (probably also because of a bad model choice).
\nCheck out this notebook on how to use TPUs in this competition. It was extremly helpful to me.
\nThen training is quite fast compared to before and reading through it I was able write an inference notebook that processes all test data in 10min (+20min for conversion to InChI strings).
will you share the inference part? because there is official example about TPU training but no inference on the whole test dataset.
", "rawMarkdown": "will you share the inference part? because there is official example about TPU training but no inference on the whole test dataset. ", "votes": 1 }, { "id": 1272366, "postDate": "2021-04-13T12:25:17.733Z", "content": "Hi, the original notebook author decided against sharing the inference part and I would like to respect that. But the necessary code blocks are basically all there. Hint: the validation loop is very similar to the final inference loop. I learned a lot by having to write the inference by myself :)
", "rawMarkdown": "Hi, the original notebook author decided against sharing the inference part and I would like to respect that. But the necessary code blocks are basically all there. Hint: the validation loop is very similar to the final inference loop. I learned a lot by having to write the inference by myself :)" } ] }, { "id": 1271073, "postDate": "2021-04-12T09:20:12.060Z", "content": "(For now) I train 10 epochs, 1000 steps each with batch size 512 on the provided TPUs and get training + inference in under 2h, so I think with TPUs it's definitely possible
", "rawMarkdown": "(For now) I train 10 epochs, 1000 steps each with batch size 512 on the provided TPUs and get training + inference in under 2h, so I think with TPUs it's definitely possible", "votes": 2, "replies": [ { "id": 1271920, "postDate": "2021-04-13T04:04:59.883Z", "content": "Hai, are you using pytorch for training, if so could you provide some reference notebooks?
", "rawMarkdown": "Hai, are you using pytorch for training, if so could you provide some reference notebooks?", "votes": 1 }, { "id": 1288291, "postDate": "2021-04-29T20:41:31.013Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1269352, "author_name": "Nuno Ferreira", "author_url": "", "post_date": "2021-04-10T13:16:59.503000", "content": "@shalvinpshaji The majority of the people aren't using kaggle kernels for this. They use either their own machines, google colab or any other web platform. You can use kaggle kernels, by running it for one epoch, saving the weights and optimizers/schedulers states, and then restart training by loading those values. And you keep on going until you are done with training.
", "votes": 5, "replies": [] }, { "id": 1269561, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-04-10T16:47:43.877000", "content": "I use Kaggle Kernels.
\nI utilize free TPU. And am usually able to process/infer-on around 40 million images per 9 hour session.
\nI will be releasing a very detailed training notebook soon… so stay tuned!
\nAlso feel free to share a link to your notebook, it will allow others to better offer suggestions/help.
", "votes": 3, "replies": [ { "id": 1273418, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2021-04-14T10:29:13.323000", "content": "Is the TPU notebook runtime limit 9 hours? I thought it was 3 hours for TPU and 9 hours for GPU.
", "votes": 0, "replies": [] } ] }, { "id": 1270060, "author_name": "Michael Wolff", "author_url": "", "post_date": "2021-04-11T08:36:45.157000", "content": "Hi,
\nthis is also my first competition and I started with GPUs, which was terribly slow (probably also because of a bad model choice).
\nCheck out this notebook on how to use TPUs in this competition. It was extremly helpful to me.
\nThen training is quite fast compared to before and reading through it I was able write an inference notebook that processes all test data in 10min (+20min for conversion to InChI strings).
will you share the inference part? because there is official example about TPU training but no inference on the whole test dataset.
", "votes": 1, "replies": [] }, { "id": 1272366, "author_name": "Michael Wolff", "author_url": "", "post_date": "2021-04-13T12:25:17.733000", "content": "Hi, the original notebook author decided against sharing the inference part and I would like to respect that. But the necessary code blocks are basically all there. Hint: the validation loop is very similar to the final inference loop. I learned a lot by having to write the inference by myself :)
", "votes": 0, "replies": [] } ] }, { "id": 1271073, "author_name": "Max", "author_url": "", "post_date": "2021-04-12T09:20:12.060000", "content": "(For now) I train 10 epochs, 1000 steps each with batch size 512 on the provided TPUs and get training + inference in under 2h, so I think with TPUs it's definitely possible
", "votes": 2, "replies": [ { "id": 1271920, "author_name": "Shalvin P Shaji", "author_url": "", "post_date": "2021-04-13T04:04:59.883000", "content": "Hai, are you using pytorch for training, if so could you provide some reference notebooks?
", "votes": 1, "replies": [] }, { "id": 1288291, "author_name": "", "author_url": "", "post_date": "2021-04-29T20:41:31.013000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1269352": "@shalvinpshaji The majority of the people aren't using kaggle kernels for this. They use either their own machines, google colab or any other web platform. You can use kaggle kernels, by running it for one epoch, saving the weights and optimizers/schedulers states, and then restart training by loading those values. And you keep on going until you are done with training.", "1269285": "Hai,\nI am pretty new to kaggle. The dataset provided is very big. Even by choosing a few subset (50k) of training examples it takes literally hours to train a model just below the kaggle hour limit. Can someone tell whether I am doing something wrong or is it the way it is supposed to be?\nSorry if this question does not make any sense to you ", "1269561": "I use Kaggle Kernels. \n\nI utilize free TPU. And am usually able to process/infer-on around 40 million images per 9 hour session.\n\nI will be releasing a very detailed training notebook soon... so stay tuned!\n\nAlso feel free to share a link to your notebook, it will allow others to better offer suggestions/help.", "1270060": "Hi,\nthis is also my first competition and I started with GPUs, which was terribly slow (probably also because of a bad model choice). \nCheck out this [notebook](https://www.kaggle.com/markwijkhuizen/tensorflow-tpu-training-baseline-lb-16-92) on how to use TPUs in this competition. It was extremly helpful to me.\nThen training is quite fast compared to before and reading through it I was able write an inference notebook that processes all test data in 10min (+20min for conversion to InChI strings).", "1271073": "(For now) I train 10 epochs, 1000 steps each with batch size 512 on the provided TPUs and get training + inference in under 2h, so I think with TPUs it's definitely possible" } }