As the competition end is approaching, one should think about using the last 30h of available TPU quota wisely. As you know, using TFRecords can really boost up training, but sometimes those provided in the original competition dataset are not enough, e.g.:
\nA one-stop solution is from_tensor_slices method, which I came across in many notebooks. This makes it just as easy as feeding a DataFrame into the network but results in slower (really slower) runtime. E.g. training EfficientNetB4 on dataset of 25K 512x512 images takes nearly 6x more time:
| \n | from tensor slices | \nfrom tfrecord files | \n
|---|---|---|
| Max time per epoch | \n996s | \n132s | \n
| Average time per epoch | \n502s | \n81s | \n
On the other hand, serializing this dataset to TFRecords can be done in just 30 lines of code. Taking only 10 minutes on GPU, this step would save you up to a few hours on TPU when training an ensemble or a large model.
\nSo I've decided to create this short notebook as an example, where I take soft targets for this competition and create new TFRecords. Hope this helps someone. Happy coding!
", "messages": [ { "id": "1200250", "postDate": "02/14/2021 14:15:36", "content": "As the competition end is approaching, one should think about using the last 30h of available TPU quota wisely. As you know, using TFRecords can really boost up training, but sometimes those provided in the original competition dataset are not enough, e.g.:
\nA one-stop solution is from_tensor_slices method, which I came across in many notebooks. This makes it just as easy as feeding a DataFrame into the network but results in slower (really slower) runtime. E.g. training EfficientNetB4 on dataset of 25K 512x512 images takes nearly 6x more time:
| \n | from tensor slices | \nfrom tfrecord files | \n
|---|---|---|
| Max time per epoch | \n996s | \n132s | \n
| Average time per epoch | \n502s | \n81s | \n
On the other hand, serializing this dataset to TFRecords can be done in just 30 lines of code. Taking only 10 minutes on GPU, this step would save you up to a few hours on TPU when training an ensemble or a large model.
\nSo I've decided to create this short notebook as an example, where I take soft targets for this competition and create new TFRecords. Hope this helps someone. Happy coding!
", "rawMarkdown": "### Hello! \n\nAs the competition end is approaching, one should think about using the last 30h of available TPU quota wisely. As you know, using TFRecords can really boost up training, but sometimes those provided in the original competition dataset are not enough, e.g.:\n* you need to add more data, but external data is only available in .jpeg format\n* you need to prepare augmented data instead of creating it on the fly\n* you need to perform knowledge distillation on soft labels, etc.\n\nA one-stop solution is `from_tensor_slices` method, which I came across in many notebooks. This makes it just as easy as feeding a DataFrame into the network but results in slower (really slower) runtime. E.g. training EfficientNetB4 on dataset of 25K 512x512 images takes nearly 6x more time:\n\n| | from tensor slices | from tfrecord files |\n| --- | --- | --- |\n| Max time per epoch | 996s | **132s** |\n| Average time per epoch | 502s | **81s** | \n\nOn the other hand, serializing this dataset to TFRecords can be done in just 30 lines of code. Taking only 10 minutes on GPU, this step would save you up to a few hours on TPU when training an ensemble or a large model.\n\nSo I've decided to create **[this short notebook](https://www.kaggle.com/nickuzmenkov/tfrecord-lab)** as an example, where I take [soft targets](https://www.kaggle.com/nickuzmenkov/cassava-leaf-disease-soft-targets-09-model) for this competition and create new TFRecords. Hope this helps someone. Happy coding!", "votes": null }, { "id": "1200361", "postDate": "02/14/2021 15:44:13", "content": "Thank you really much, I will finally give these soft targets a try!
", "rawMarkdown": "Thank you really much, I will finally give these soft targets a try!", "votes": null }, { "id": "1204158", "postDate": "02/15/2021 22:16:02", "content": "Thank you for sharing this tips 👌
", "rawMarkdown": "Thank you for sharing this tips 👌", "votes": null } ], "comments": [ { "id": 1200361, "author_name": "mviola", "author_url": "", "post_date": "02/14/2021 15:44:13", "content": "Thank you really much, I will finally give these soft targets a try!
", "votes": null, "replies": [] }, { "id": 1204158, "author_name": "milobele", "author_url": "", "post_date": "02/15/2021 22:16:02", "content": "Thank you for sharing this tips 👌
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1200250": "### Hello! \n\nAs the competition end is approaching, one should think about using the last 30h of available TPU quota wisely. As you know, using TFRecords can really boost up training, but sometimes those provided in the original competition dataset are not enough, e.g.:\n* you need to add more data, but external data is only available in .jpeg format\n* you need to prepare augmented data instead of creating it on the fly\n* you need to perform knowledge distillation on soft labels, etc.\n\nA one-stop solution is `from_tensor_slices` method, which I came across in many notebooks. This makes it just as easy as feeding a DataFrame into the network but results in slower (really slower) runtime. E.g. training EfficientNetB4 on dataset of 25K 512x512 images takes nearly 6x more time:\n\n| | from tensor slices | from tfrecord files |\n| --- | --- | --- |\n| Max time per epoch | 996s | **132s** |\n| Average time per epoch | 502s | **81s** | \n\nOn the other hand, serializing this dataset to TFRecords can be done in just 30 lines of code. Taking only 10 minutes on GPU, this step would save you up to a few hours on TPU when training an ensemble or a large model.\n\nSo I've decided to create **[this short notebook](https://www.kaggle.com/nickuzmenkov/tfrecord-lab)** as an example, where I take [soft targets](https://www.kaggle.com/nickuzmenkov/cassava-leaf-disease-soft-targets-09-model) for this competition and create new TFRecords. Hope this helps someone. Happy coding!", "1200361": "Thank you really much, I will finally give these soft targets a try!", "1204158": "Thank you for sharing this tips 👌" }, "source": "meta" }