Hey everyone. I was thinking if it is possible to encode the image normally using a ResNet (or something similar) but then feed this representation to a Transformer for decoding. For example, you get a (N, 512, 7, 7) tensor out of the ResNet and then treat it like it is embeddings for 49 (7 * 7) words and feed it to the Transformer to do a kind of translation.
\nIn theory it seems plausible for me but I don't know if there could be any reason to do this.
\nI'll be really glad if you let me know what you think.
Hey everyone. I was thinking if it is possible to encode the image normally using a ResNet (or something similar) but then feed this representation to a Transformer for decoding. For example, you get a (N, 512, 7, 7) tensor out of the ResNet and then treat it like it is embeddings for 49 (7 * 7) words and feed it to the Transformer to do a kind of translation.
\nIn theory it seems plausible for me but I don't know if there could be any reason to do this.
\nI'll be really glad if you let me know what you think.
Of course you can.
\nThe dataset is huge. LSTM is slow to train if you don't have descent hardware like myself.
", "rawMarkdown": "Of course you can. \n\nThe dataset is huge. LSTM is slow to train if you don't have descent hardware like myself.", "votes": null }, { "id": "1249942", "postDate": "03/23/2021 16:35:30", "content": "Exactly. My main problem with LSTM is that it is tooo slow and makes me unable to run many experiments on my available hardware. Good to hear that. I'm gonna start working on it.
", "rawMarkdown": "Exactly. My main problem with LSTM is that it is tooo slow and makes me unable to run many experiments on my available hardware. Good to hear that. I'm gonna start working on it.", "votes": null }, { "id": "1250064", "postDate": "03/23/2021 18:42:01", "content": "Especially if you plan to use TPU, I suspect it would be more efficient with a transformer as it is not sequential as LSTM/GRU
", "rawMarkdown": "Especially if you plan to use TPU, I suspect it would be more efficient with a transformer as it is not sequential as LSTM/GRU", "votes": null }, { "id": "1250186", "postDate": "03/23/2021 20:25:52", "content": "Oh, that's a good point. I did not know about it. Thanks for mentioning!
", "rawMarkdown": "Oh, that's a good point. I did not know about it. Thanks for mentioning!", "votes": null }, { "id": "1250238", "postDate": "03/23/2021 21:20:39", "content": "yes.. you just have to make sure to add positional embedding to image features.. otherwise results will be funny =)
yes, sure. Do you think 1D positional embedding suffices or I must add 2D positional embedding? By 1D I mean I first flatten the 7 by 7 output grid and then treat them like sequential words in a sentence.
", "rawMarkdown": "yes, sure. Do you think 1D positional embedding suffices or I must add 2D positional embedding? By 1D I mean I first flatten the 7 by 7 output grid and then treat them like sequential words in a sentence.", "votes": null } ], "comments": [ { "id": 1249918, "author_name": "serigne", "author_url": "", "post_date": "03/23/2021 16:15:37", "content": "Of course you can.
\nThe dataset is huge. LSTM is slow to train if you don't have descent hardware like myself.
", "votes": null, "replies": [ { "id": 1249942, "author_name": "moeinshariatnia", "author_url": "", "post_date": "03/23/2021 16:35:30", "content": "Exactly. My main problem with LSTM is that it is tooo slow and makes me unable to run many experiments on my available hardware. Good to hear that. I'm gonna start working on it.
", "votes": null, "replies": [] }, { "id": 1250064, "author_name": "ludovick", "author_url": "", "post_date": "03/23/2021 18:42:01", "content": "Especially if you plan to use TPU, I suspect it would be more efficient with a transformer as it is not sequential as LSTM/GRU
", "votes": null, "replies": [] }, { "id": 1250186, "author_name": "moeinshariatnia", "author_url": "", "post_date": "03/23/2021 20:25:52", "content": "Oh, that's a good point. I did not know about it. Thanks for mentioning!
", "votes": null, "replies": [] } ] }, { "id": 1250238, "author_name": "drhabib", "author_url": "", "post_date": "03/23/2021 21:20:39", "content": "yes.. you just have to make sure to add positional embedding to image features.. otherwise results will be funny =)
yes, sure. Do you think 1D positional embedding suffices or I must add 2D positional embedding? By 1D I mean I first flatten the 7 by 7 output grid and then treat them like sequential words in a sentence.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1247826": "Hey everyone. I was thinking if it is possible to encode the image normally using a ResNet (or something similar) but then feed this representation to a Transformer for decoding. For example, you get a (N, 512, 7, 7) tensor out of the ResNet and then treat it like it is embeddings for 49 (7 * 7) words and feed it to the Transformer to do a kind of translation.\nIn theory it seems plausible for me but I don't know if there could be any reason to do this.\nI'll be really glad if you let me know what you think.", "1249918": "Of course you can. \n\nThe dataset is huge. LSTM is slow to train if you don't have descent hardware like myself.", "1249942": "Exactly. My main problem with LSTM is that it is tooo slow and makes me unable to run many experiments on my available hardware. Good to hear that. I'm gonna start working on it.", "1250064": "Especially if you plan to use TPU, I suspect it would be more efficient with a transformer as it is not sequential as LSTM/GRU", "1250186": "Oh, that's a good point. I did not know about it. Thanks for mentioning!", "1250238": "yes.. you just have to make sure to add `positional embedding` to image features.. otherwise results will be funny =)", "1250479": "yes, sure. Do you think 1D positional embedding suffices or I must add 2D positional embedding? By 1D I mean I first flatten the 7 by 7 output grid and then treat them like sequential words in a sentence." }, "source": "meta" }