![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F877700%2Ffd51cbde9717caff294d0e20c7a36e50%2Ferror_tmp.png?generation=1608215745737036&alt=media\")
When training with TPUs using tf.data.Dataset, you can automatically cache the training images in memory by calling dataset.cache(). However, this method is not very convenient if you try to run the same code on CPU, since you will quickly run out of memory.
Rather than iteratively resizing and saving the images as pickle/numpy arrays, you can simply specify a local directory where you want to cache your data, e.g. dataset.cache(\"/kaggle/tf_cache\"). This will avoid writing custom pre-processing code, as well as eliminate the CPU bottleneck when reading the images as JPEGs and resizing them.
In my GPU starter notebook, the training time dropped from 1895s for the first epoch to 640s for the subsequent epochs. This is not quite as dramatic as TPU's time reduction (which dropped from 1191s to to 265s), but it ensures a smaller EfficientNet model can be trained in one or two hours.
", "messages": [ { "id": "1112977", "postDate": "12/15/2020 04:29:29", "content": "When training with TPUs using tf.data.Dataset, you can automatically cache the training images in memory by calling dataset.cache(). However, this method is not very convenient if you try to run the same code on CPU, since you will quickly run out of memory.
Rather than iteratively resizing and saving the images as pickle/numpy arrays, you can simply specify a local directory where you want to cache your data, e.g. dataset.cache(\"/kaggle/tf_cache\"). This will avoid writing custom pre-processing code, as well as eliminate the CPU bottleneck when reading the images as JPEGs and resizing them.
In my GPU starter notebook, the training time dropped from 1895s for the first epoch to 640s for the subsequent epochs. This is not quite as dramatic as TPU's time reduction (which dropped from 1191s to to 265s), but it ensures a smaller EfficientNet model can be trained in one or two hours.
", "rawMarkdown": "When training with TPUs using `tf.data.Dataset`, you can automatically cache the training images in memory by calling `dataset.cache()`. However, this method is not very convenient if you try to run the same code on CPU, since you will quickly run out of memory. \n\nRather than iteratively resizing and saving the images as pickle/numpy arrays, you can simply specify a local directory where you want to cache your data, e.g. `dataset.cache(\"/kaggle/tf_cache\")`. This will avoid writing custom pre-processing code, as well as eliminate the CPU bottleneck when reading the images as JPEGs and resizing them. \n\nIn my [GPU starter notebook](https://www.kaggle.com/xhlulu/ranzcr-efficientnet-b3-gpu-starter), the training time dropped from 1895s for the first epoch to 640s for the subsequent epochs. This is not quite as dramatic as [TPU's time reduction](https://www.kaggle.com/xhlulu/ranzcr-efficientnet-b7-tpu-training) (which dropped from 1191s to to 265s), but it ensures a smaller EfficientNet model can be trained in one or two hours.", "votes": null }, { "id": "1113016", "postDate": "12/15/2020 05:34:14", "content": "Great trick. Thanks!
", "rawMarkdown": "Great trick. Thanks!", "votes": null }, { "id": "1113556", "postDate": "12/15/2020 14:51:18", "content": "I have tried in previous competitions and these are my findings:
\nDid you encounter this issue?
", "rawMarkdown": "I have tried in previous competitions and these are my findings:\n\n1. If you just use ds.cache() it would store results in RAM and in this case kernel will go out of memory.\n2. If you use ds.cache('kaggle.tfcache') the cache would be stored in disk and eventually the disk would go out of space.\n \n\nDid you encounter this issue?", "votes": null }, { "id": "1113568", "postDate": "12/15/2020 14:57:46", "content": "You have to store inside /kaggle rather than /kaggle/working, since the latter will be saved as output and is limited to 20gb (and the former has much more space)
", "rawMarkdown": "You have to store inside /kaggle rather than /kaggle/working, since the latter will be saved as output and is limited to 20gb (and the former has much more space)", "votes": null }, { "id": "1113584", "postDate": "12/15/2020 15:03:31", "content": "Yes that is correct I was using the /kaggle/working directory. Thank you for this tip! It will save a lot of my gpu time!
", "rawMarkdown": "Yes that is correct I was using the /kaggle/working directory. Thank you for this tip! It will save a lot of my gpu time!", "votes": null }, { "id": "1113638", "postDate": "12/15/2020 15:51:12", "content": "Hi @xhlulu, thanks for sharing your valuable insight. The \"GPU starter notebook\" is a nice work of arts. Thanks again for sharing.
", "rawMarkdown": "Hi @xhlulu, thanks for sharing your valuable insight. The \"GPU starter notebook\" is a nice work of arts. Thanks again for sharing.", "votes": null }, { "id": "1114047", "postDate": "12/15/2020 23:53:40", "content": "Hey! Thanks for sharing, would this work for data augmented images as well? If i call the cache method with a data augmentation function maped to the dataset, let's say three times, will it generate three sets of augmented images?
", "rawMarkdown": "Hey! Thanks for sharing, would this work for data augmented images as well? If i call the cache method with a data augmentation function maped to the dataset, let's say three times, will it generate three sets of augmented images?", "votes": null }, { "id": "1115107", "postDate": "12/16/2020 01:49:18", "content": "I'm assuming order matters for tensorflow, in which case I think it will depend on when exactly you cache the file. What I do is that I cache it before I call augment, so the augmentation will still be random.
", "rawMarkdown": "I'm assuming order matters for tensorflow, in which case I think it will depend on when exactly you cache the file. What I do is that I cache it before I call augment, so the augmentation will still be random.", "votes": null }, { "id": "1115156", "postDate": "12/16/2020 03:24:04", "content": "This is a great trick to use. Is there any such thing to use in PyTorch as well?
", "rawMarkdown": "This is a great trick to use. Is there any such thing to use in PyTorch as well?", "votes": null }, { "id": "1115244", "postDate": "12/16/2020 05:56:14", "content": "Awesome! For pytorch : loader.dataset.set_use_cache(True)
Just to confirm:
\n| Strategy | \nTime taken | \n
|---|---|
| without cache | \n2728s | \n
| with cache | \n646s | \n
I will update results with the mixed precision as well.
", "rawMarkdown": "Just to confirm:\n\n| Strategy | Time taken |\n| --- | --- |\n| without cache | 2728s |\n| with cache | 646s |\n\nI will update results with the mixed precision as well.", "votes": null }, { "id": "1115585", "postDate": "12/16/2020 11:51:07", "content": "Great stuff!
", "rawMarkdown": "Great stuff!", "votes": null }, { "id": "1116869", "postDate": "12/17/2020 14:24:56", "content": "Thank you @xhlulu for sharing. It saves a lot of GPU time. But, probably, there is a limit to using kaggle disk. On 'HubMAP…' competition, I got the error below
\nafter using
\nos.makedirs('/kaggle/tf_cache')\n....\ndataset.cache('/kaggle/tf_cache')\n
Thanks for the info.
", "rawMarkdown": "Thanks for the info.", "votes": null }, { "id": "1140272", "postDate": "01/05/2021 22:35:48", "content": "@alincijov please how can i use this line of code in pytorch because when i try it it gives error
\n loader.dataset.set_use_cache(True)
Great trick. Thanks!
", "votes": null, "replies": [] }, { "id": 1113556, "author_name": "harveenchadha", "author_url": "", "post_date": "12/15/2020 14:51:18", "content": "I have tried in previous competitions and these are my findings:
\nDid you encounter this issue?
", "votes": null, "replies": [ { "id": 1113568, "author_name": "xhlulu", "author_url": "", "post_date": "12/15/2020 14:57:46", "content": "You have to store inside /kaggle rather than /kaggle/working, since the latter will be saved as output and is limited to 20gb (and the former has much more space)
", "votes": null, "replies": [] }, { "id": 1113584, "author_name": "harveenchadha", "author_url": "", "post_date": "12/15/2020 15:03:31", "content": "Yes that is correct I was using the /kaggle/working directory. Thank you for this tip! It will save a lot of my gpu time!
", "votes": null, "replies": [] } ] }, { "id": 1113638, "author_name": "puzuwe", "author_url": "", "post_date": "12/15/2020 15:51:12", "content": "Hi @xhlulu, thanks for sharing your valuable insight. The \"GPU starter notebook\" is a nice work of arts. Thanks again for sharing.
", "votes": null, "replies": [] }, { "id": 1114047, "author_name": "capiru", "author_url": "", "post_date": "12/15/2020 23:53:40", "content": "Hey! Thanks for sharing, would this work for data augmented images as well? If i call the cache method with a data augmentation function maped to the dataset, let's say three times, will it generate three sets of augmented images?
", "votes": null, "replies": [ { "id": 1115107, "author_name": "xhlulu", "author_url": "", "post_date": "12/16/2020 01:49:18", "content": "I'm assuming order matters for tensorflow, in which case I think it will depend on when exactly you cache the file. What I do is that I cache it before I call augment, so the augmentation will still be random.
", "votes": null, "replies": [] } ] }, { "id": 1115156, "author_name": "sovitrath", "author_url": "", "post_date": "12/16/2020 03:24:04", "content": "This is a great trick to use. Is there any such thing to use in PyTorch as well?
", "votes": null, "replies": [] }, { "id": 1115244, "author_name": "alincijov", "author_url": "", "post_date": "12/16/2020 05:56:14", "content": "Awesome! For pytorch : loader.dataset.set_use_cache(True)
Just to confirm:
\n| Strategy | \nTime taken | \n
|---|---|
| without cache | \n2728s | \n
| with cache | \n646s | \n
I will update results with the mixed precision as well.
", "votes": null, "replies": [] }, { "id": 1115585, "author_name": "saurabhshahane", "author_url": "", "post_date": "12/16/2020 11:51:07", "content": "Great stuff!
", "votes": null, "replies": [] }, { "id": 1116869, "author_name": "isakev", "author_url": "", "post_date": "12/17/2020 14:24:56", "content": "Thank you @xhlulu for sharing. It saves a lot of GPU time. But, probably, there is a limit to using kaggle disk. On 'HubMAP…' competition, I got the error below
\nafter using
\nos.makedirs('/kaggle/tf_cache')\n....\ndataset.cache('/kaggle/tf_cache')\n
Thanks for the info.
", "votes": null, "replies": [] }, { "id": 1140272, "author_name": "mohamed3abdelrazik", "author_url": "", "post_date": "01/05/2021 22:35:48", "content": "@alincijov please how can i use this line of code in pytorch because when i try it it gives error
\n loader.dataset.set_use_cache(True)