Hi there,
\nI have created a notebook that implements a pipeline consisting of a Transformer Decoder and an EfficientNetV2 Encoder (all in Tensorflow).
\nThis was a bit more difficult than I anticipated and I learned a lot (it took all 30 TPU hours this week and I still didn't get inference on the test set working). I just want to take this opportunity and share the knowledge and struggles with everyone.
\nThe full architecture was able to achieve CV=3.71 in 6 epochs (although I loaded the EfficientNetV2 pre-trained model from my other pipeline as a starting point for the encoder)
\nThe learning rate scheduling is difficult to get right and I encountered lots and lots of trouble with spiking loss and weird stuff happening. I think batching by sequence length and using a lower max LR would help considerably.
\nTL;DR ––––––– TF+TPU+EfficientNetV2+Decoder –––> DIFFICULT TO TRAIN … BUT 🔥🔥🧑🚒(me)🔥🔥
\nA caveat: I ran out of TPU hours and was not able to get the test inference to work. Also, the loss/accuracy became unstable during my last training (LR curve should be flatter and lower probably). All this being equal, the validation inference DOES work, and it's very similar to the test inference… so I must just be missing something. But as such, I have not submitted this notebook to the public LB yet. If someone gets it to work please let me know how it does!
\nADDENDUM:
\nFINAL ADDENDUM:
\nd_model -> Image Embedding Depth (208)Br is a token, 191 is a token, etc.)This was just where I had to stop because I ran out of TPU… I assume with a better schedule you could take this a lot further. Even on half-length tokens.
\nI hope this helps! It was an effort and a half to get this to work and I'm certain that I did things that don't make sense or could be better. If you see anything like this please don't hesitate to let me know. I'm all about learning and getting better.
\nThank you for your time!
", "messages": [ { "id": "1322932", "postDate": "05/25/2021 19:47:00", "content": "Hi there,
\nI have created a notebook that implements a pipeline consisting of a Transformer Decoder and an EfficientNetV2 Encoder (all in Tensorflow).
\nThis was a bit more difficult than I anticipated and I learned a lot (it took all 30 TPU hours this week and I still didn't get inference on the test set working). I just want to take this opportunity and share the knowledge and struggles with everyone.
\nThe full architecture was able to achieve CV=3.71 in 6 epochs (although I loaded the EfficientNetV2 pre-trained model from my other pipeline as a starting point for the encoder)
\nThe learning rate scheduling is difficult to get right and I encountered lots and lots of trouble with spiking loss and weird stuff happening. I think batching by sequence length and using a lower max LR would help considerably.
\nTL;DR ––––––– TF+TPU+EfficientNetV2+Decoder –––> DIFFICULT TO TRAIN … BUT 🔥🔥🧑🚒(me)🔥🔥
\nA caveat: I ran out of TPU hours and was not able to get the test inference to work. Also, the loss/accuracy became unstable during my last training (LR curve should be flatter and lower probably). All this being equal, the validation inference DOES work, and it's very similar to the test inference… so I must just be missing something. But as such, I have not submitted this notebook to the public LB yet. If someone gets it to work please let me know how it does!
\nADDENDUM:
\nFINAL ADDENDUM:
\nd_model -> Image Embedding Depth (208)Br is a token, 191 is a token, etc.)This was just where I had to stop because I ran out of TPU… I assume with a better schedule you could take this a lot further. Even on half-length tokens.
\nI hope this helps! It was an effort and a half to get this to work and I'm certain that I did things that don't make sense or could be better. If you see anything like this please don't hesitate to let me know. I'm all about learning and getting better.
\nThank you for your time!
", "rawMarkdown": "Hi there, \n\nI have created a [notebook](https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e) that implements a pipeline consisting of a Transformer Decoder and an EfficientNetV2 Encoder (all in Tensorflow).\n\nThis was a bit more difficult than I anticipated and I learned a lot (it took all 30 TPU hours this week and I still didn't get inference on the test set working). I just want to take this opportunity and share the knowledge and struggles with everyone.\n\nThe full architecture was able to achieve **CV=3.71 in 6 epochs** *(although I loaded the EfficientNetV2 pre-trained model from my other pipeline as a starting point for the encoder)*\n\nThe learning rate scheduling is difficult to get right and I encountered lots and lots of trouble with spiking loss and weird stuff happening. I think batching by sequence length and using a lower max LR would help considerably.\n\n---\n\nI'm an extreme begginer here, but for training of transformers , DeepSpeed and DDP can be used to significantly amp up the training speed. Also I dont think taking RGB values helps in making the algorithm learn something extra and only grayscale pics can help you reduces a bit of computaion time. Please let me know if i was of any help, it would be helpful for me!
", "rawMarkdown": "I'm an extreme begginer here, but for training of transformers , DeepSpeed and DDP can be used to significantly amp up the training speed. Also I dont think taking RGB values helps in making the algorithm learn something extra and only grayscale pics can help you reduces a bit of computaion time. Please let me know if i was of any help, it would be helpful for me!", "votes": null }, { "id": "1325030", "postDate": "05/27/2021 12:55:46", "content": "how could you find spike learning rate is key point,
\nI read \"attention is all you need\" paper but fully ignore this part
\nI waste two weeks time on dig in model structure to find something went wrong
\nThank you for reply in my post to point this part out
I think it's the learning rate is too high. It also might be due to the image captioning flow?
\nWhen I tried freezing the encoder (using a pre-trained one), I noticed it was much more stable.
\nAlso, the d_model size is very small in the above experiment. This might make it unstable. If you use an EfficientNetV1 model the d_model variable will be 5-10 times larger and the Transformer will get waayyyy bigger. Maybe this is impacting it too?
When the competition is over I will opensource my private notebook that does training using a larger image model and a slightly modified transformer from above.
\nGlad it helped!
", "rawMarkdown": "I think it's the learning rate is too high. It also might be due to the image captioning flow?\n\nWhen I tried freezing the encoder (using a pre-trained one), I noticed it was much more stable.\n\nAlso, the **`d_model`** size is very small in the above experiment. This might make it unstable. If you use an EfficientNetV1 model the d_model variable will be 5-10 times larger and the Transformer will get waayyyy bigger. Maybe this is impacting it too?\n\nWhen the competition is over I will opensource my private notebook that does training using a larger image model and a slightly modified transformer from above.\n\nGlad it helped!", "votes": null }, { "id": "1325063", "postDate": "05/27/2021 13:25:52", "content": "RGB does not help over grayscale, but it doesn't really increase the network size that much and it's easier to implement. Print a network summary using one channel vs 3-channel and you'll see the difference is minimal.
\nAs for DeepSpeed and DDP, I haven't heard or read about this. Can you link me to anything specific? I can try to implement it?
\nThanks for your comment!
", "rawMarkdown": "RGB does not help over grayscale, but it doesn't really increase the network size that much and it's easier to implement. Print a network summary using one channel vs 3-channel and you'll see the difference is minimal.\n\nAs for DeepSpeed and DDP, I haven't heard or read about this. Can you link me to anything specific? I can try to implement it?\n\nThanks for your comment!", "votes": null }, { "id": "1325199", "postDate": "05/27/2021 15:08:17", "content": "Deepspeed is Microsoft’s library for training very large distributed models in various ways and implements zero redundancy optimizers and some other fancy things. It’s pytorch only though and won’t be of much use here. DDP is just pytorch native distributed data parallel.
", "rawMarkdown": "Deepspeed is Microsoft’s library for training very large distributed models in various ways and implements zero redundancy optimizers and some other fancy things. It’s pytorch only though and won’t be of much use here. DDP is just pytorch native distributed data parallel.", "votes": null }, { "id": "1325271", "postDate": "05/27/2021 15:59:34", "content": "TF2.X already handles distributed training very efficiently with almost no code change. This wasn't the case with TF1.X
\nSetting up DDP with Pytorch wasn't an easy experience for me (I didn't use Lightning though but vanilla Pytorch instead)
", "rawMarkdown": "TF2.X already handles distributed training very efficiently with almost no code change. This wasn't the case with TF1.X\n\nSetting up DDP with Pytorch wasn't an easy experience for me (I didn't use Lightning though but vanilla Pytorch instead)", "votes": null }, { "id": "1325801", "postDate": "05/28/2021 02:45:26", "content": "Thanks a lot for the great share. Any suggestion for how to match LR if I start train from previously trained weight for lower max LR?
", "rawMarkdown": "Thanks a lot for the great share. Any suggestion for how to match LR if I start train from previously trained weight for lower max LR?", "votes": null }, { "id": "1325848", "postDate": "05/28/2021 04:17:17", "content": "why does transformer better than LSTM in precise?
\nDoes you try vision transformer? I plan to try it asap
Thanks a lot for sharing.
", "rawMarkdown": "Thanks a lot for sharing.", "votes": null }, { "id": "1326759", "postDate": "05/28/2021 16:59:54", "content": "I'm struggling with this now. I think because I am constrained to Kaggle's limits, I lose some information when restarting training. My only suggestion would be to lower the peak LR for each subsequent training session. i.e. increase the value in the rsqrt().
", "rawMarkdown": "I'm struggling with this now. I think because I am constrained to Kaggle's limits, I lose some information when restarting training. My only suggestion would be to lower the peak LR for each subsequent training session. i.e. increase the value in the rsqrt().", "votes": null }, { "id": "1331067", "postDate": "06/01/2021 08:53:46", "content": "Validation take about 30-40 minutes? am I config something wrong?
", "rawMarkdown": "Validation take about 30-40 minutes? am I config something wrong?", "votes": null }, { "id": "1331418", "postDate": "06/01/2021 13:08:10", "content": "The first validation run seems to take longer than the following runs. 30-40 minutes seems excessive though. I found it only added a few extra minutes per epoch.
\nThat being said, the transformer will be inferring MAX_LEN times more frequently during validation/testing. So some increase in timing is expected.
it seem valid step doesn't use TPU , strangely
", "rawMarkdown": "it seem valid step doesn't use TPU , strangely", "votes": null }, { "id": "1332787", "postDate": "06/02/2021 09:52:24", "content": "A great notebook - can be a good guidance for Image Processing using TPUs. As I understand, the image processing is done separately (couldn't see the processing of the original /train and /test images) - how long did that part take?
", "rawMarkdown": "A great notebook - can be a good guidance for Image Processing using TPUs. As I understand, the image processing is done separately (couldn't see the processing of the original /train and /test images) - how long did that part take?", "votes": null }, { "id": "1333495", "postDate": "06/02/2021 18:58:37", "content": "Thanks for sharing your progress throughout the competition! I haven't coded for TPU yet but would like to compare models. My implementation (on GPU) takes MUCH longer than your TPU implementation. Do you know what your training time would look like on GPU?
", "rawMarkdown": "Thanks for sharing your progress throughout the competition! I haven't coded for TPU yet but would like to compare models. My implementation (on GPU) takes MUCH longer than your TPU implementation. Do you know what your training time would look like on GPU?", "votes": null } ], "comments": [ { "id": 1323453, "author_name": "puranjaymishra", "author_url": "", "post_date": "05/26/2021 08:37:06", "content": "I'm an extreme begginer here, but for training of transformers , DeepSpeed and DDP can be used to significantly amp up the training speed. Also I dont think taking RGB values helps in making the algorithm learn something extra and only grayscale pics can help you reduces a bit of computaion time. Please let me know if i was of any help, it would be helpful for me!
", "votes": null, "replies": [ { "id": 1325063, "author_name": "dschettler8845", "author_url": "", "post_date": "05/27/2021 13:25:52", "content": "RGB does not help over grayscale, but it doesn't really increase the network size that much and it's easier to implement. Print a network summary using one channel vs 3-channel and you'll see the difference is minimal.
\nAs for DeepSpeed and DDP, I haven't heard or read about this. Can you link me to anything specific? I can try to implement it?
\nThanks for your comment!
", "votes": null, "replies": [] }, { "id": 1325199, "author_name": "interneuron", "author_url": "", "post_date": "05/27/2021 15:08:17", "content": "Deepspeed is Microsoft’s library for training very large distributed models in various ways and implements zero redundancy optimizers and some other fancy things. It’s pytorch only though and won’t be of much use here. DDP is just pytorch native distributed data parallel.
", "votes": null, "replies": [] }, { "id": 1325271, "author_name": "serigne", "author_url": "", "post_date": "05/27/2021 15:59:34", "content": "TF2.X already handles distributed training very efficiently with almost no code change. This wasn't the case with TF1.X
\nSetting up DDP with Pytorch wasn't an easy experience for me (I didn't use Lightning though but vanilla Pytorch instead)
", "votes": null, "replies": [] } ] }, { "id": 1325030, "author_name": "drzhuzhe", "author_url": "", "post_date": "05/27/2021 12:55:46", "content": "how could you find spike learning rate is key point,
\nI read \"attention is all you need\" paper but fully ignore this part
\nI waste two weeks time on dig in model structure to find something went wrong
\nThank you for reply in my post to point this part out
I think it's the learning rate is too high. It also might be due to the image captioning flow?
\nWhen I tried freezing the encoder (using a pre-trained one), I noticed it was much more stable.
\nAlso, the d_model size is very small in the above experiment. This might make it unstable. If you use an EfficientNetV1 model the d_model variable will be 5-10 times larger and the Transformer will get waayyyy bigger. Maybe this is impacting it too?
When the competition is over I will opensource my private notebook that does training using a larger image model and a slightly modified transformer from above.
\nGlad it helped!
", "votes": null, "replies": [] }, { "id": 1325848, "author_name": "drzhuzhe", "author_url": "", "post_date": "05/28/2021 04:17:17", "content": "why does transformer better than LSTM in precise?
\nDoes you try vision transformer? I plan to try it asap
Thanks a lot for the great share. Any suggestion for how to match LR if I start train from previously trained weight for lower max LR?
", "votes": null, "replies": [ { "id": 1326759, "author_name": "dschettler8845", "author_url": "", "post_date": "05/28/2021 16:59:54", "content": "I'm struggling with this now. I think because I am constrained to Kaggle's limits, I lose some information when restarting training. My only suggestion would be to lower the peak LR for each subsequent training session. i.e. increase the value in the rsqrt().
", "votes": null, "replies": [] } ] }, { "id": 1326422, "author_name": "saivineetha1998", "author_url": "", "post_date": "05/28/2021 13:30:15", "content": "Thanks a lot for sharing.
", "votes": null, "replies": [] }, { "id": 1331067, "author_name": "drzhuzhe", "author_url": "", "post_date": "06/01/2021 08:53:46", "content": "Validation take about 30-40 minutes? am I config something wrong?
", "votes": null, "replies": [ { "id": 1331418, "author_name": "dschettler8845", "author_url": "", "post_date": "06/01/2021 13:08:10", "content": "The first validation run seems to take longer than the following runs. 30-40 minutes seems excessive though. I found it only added a few extra minutes per epoch.
\nThat being said, the transformer will be inferring MAX_LEN times more frequently during validation/testing. So some increase in timing is expected.
it seem valid step doesn't use TPU , strangely
", "votes": null, "replies": [] } ] }, { "id": 1332787, "author_name": "jhasanov", "author_url": "", "post_date": "06/02/2021 09:52:24", "content": "A great notebook - can be a good guidance for Image Processing using TPUs. As I understand, the image processing is done separately (couldn't see the processing of the original /train and /test images) - how long did that part take?
", "votes": null, "replies": [] }, { "id": 1333495, "author_name": "mvenou", "author_url": "", "post_date": "06/02/2021 18:58:37", "content": "Thanks for sharing your progress throughout the competition! I haven't coded for TPU yet but would like to compare models. My implementation (on GPU) takes MUCH longer than your TPU implementation. Do you know what your training time would look like on GPU?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1322932": "Hi there, \n\nI have created a [notebook](https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e) that implements a pipeline consisting of a Transformer Decoder and an EfficientNetV2 Encoder (all in Tensorflow).\n\nThis was a bit more difficult than I anticipated and I learned a lot (it took all 30 TPU hours this week and I still didn't get inference on the test set working). I just want to take this opportunity and share the knowledge and struggles with everyone.\n\nThe full architecture was able to achieve **CV=3.71 in 6 epochs** *(although I loaded the EfficientNetV2 pre-trained model from my other pipeline as a starting point for the encoder)*\n\nThe learning rate scheduling is difficult to get right and I encountered lots and lots of trouble with spiking loss and weird stuff happening. I think batching by sequence length and using a lower max LR would help considerably.\n\n---\n\n