Hello, have anyone trying to use transformer to solve this challenge?
", "messages": [ { "id": "1503066", "postDate": "09/04/2021 23:42:27", "content": "Hello, have anyone trying to use transformer to solve this challenge?
", "rawMarkdown": "Hello, have anyone trying to use transformer to solve this challenge?", "votes": null }, { "id": "1504036", "postDate": "09/06/2021 02:59:36", "content": "I did consider using a transformer for this competition however i believe it is not suitable for this task mainly due to the main requirement as given in the paper 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale'
\nwhich is you need to train transformers on a very large dataset. You can use pretrained models for this problem however with scarce training samples i dont think transfer learning will be successfull.
Secondly results for transformers when compared with Resnets is not a very big improvement and i believe this competition is more about the data and not the model.
", "rawMarkdown": "I did consider using a transformer for this competition however i believe it is not suitable for this task mainly due to the main requirement as given in the paper 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale'\nwhich is you need to train transformers on a very large dataset. You can use pretrained models for this problem however with scarce training samples i dont think transfer learning will be successfull.\n\nSecondly results for transformers when compared with Resnets is not a very big improvement and i believe this competition is more about the data and not the model.", "votes": null }, { "id": "1504086", "postDate": "09/06/2021 04:33:33", "content": "I also tried a transformer 3D model, but conventional CNN-3D models work well.
", "rawMarkdown": "I also tried a transformer 3D model, but conventional CNN-3D models work well.", "votes": null }, { "id": "1504201", "postDate": "09/06/2021 07:03:12", "content": "I did make a model with the Perceiver Transformer, the issue is they need lots of memory and like @aryamansharma47 wrote also lots of data.
\nI used 3 Steps of convolution + pooling to reduce the memory of the transformer.
Can you share any resource about 3D transformer?
", "rawMarkdown": "Can you share any resource about 3D transformer?", "votes": null }, { "id": "1506069", "postDate": "09/07/2021 19:38:24", "content": "I find some 2D transformer like Vit but i didn't find any 3D. Do you know anything?
", "rawMarkdown": "I find some 2D transformer like Vit but i didn't find any 3D. Do you know anything?", "votes": null }, { "id": "1506089", "postDate": "09/07/2021 20:29:55", "content": "You have to write it yourself. It is not that hard, just project 3D patches into a vector instead of 2D Patches into a Vector.
", "rawMarkdown": "You have to write it yourself. It is not that hard, just project 3D patches into a vector instead of 2D Patches into a Vector.", "votes": null }, { "id": "1506479", "postDate": "09/08/2021 09:13:26", "content": "Did you see any notebook or code about this?
", "rawMarkdown": "Did you see any notebook or code about this?", "votes": null }, { "id": "1506481", "postDate": "09/08/2021 09:15:00", "content": "Do you know any transformer that trained for medical images?
", "rawMarkdown": "Do you know any transformer that trained for medical images?", "votes": null }, { "id": "1507256", "postDate": "09/09/2021 03:05:55", "content": "https://arxiv.org/pdf/2108.09038.pdf
", "rawMarkdown": "https://arxiv.org/pdf/2108.09038.pdf", "votes": null }, { "id": "1507486", "postDate": "09/09/2021 08:40:48", "content": "From this article we can understand it's not good for use Transformer instead of CNN. isn't it?
", "rawMarkdown": "From this article we can understand it's not good for use Transformer instead of CNN. isn't it?", "votes": null }, { "id": "1508202", "postDate": "09/10/2021 02:15:12", "content": "For this competition i believe that may be the case
", "rawMarkdown": "For this competition i believe that may be the case", "votes": null } ], "comments": [ { "id": 1504036, "author_name": "aryamansharma47", "author_url": "", "post_date": "09/06/2021 02:59:36", "content": "I did consider using a transformer for this competition however i believe it is not suitable for this task mainly due to the main requirement as given in the paper 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale'
\nwhich is you need to train transformers on a very large dataset. You can use pretrained models for this problem however with scarce training samples i dont think transfer learning will be successfull.
Secondly results for transformers when compared with Resnets is not a very big improvement and i believe this competition is more about the data and not the model.
", "votes": null, "replies": [ { "id": 1506481, "author_name": "mohammadhosein1998", "author_url": "", "post_date": "09/08/2021 09:15:00", "content": "Do you know any transformer that trained for medical images?
", "votes": null, "replies": [] } ] }, { "id": 1504086, "author_name": "hyeonhoonlee", "author_url": "", "post_date": "09/06/2021 04:33:33", "content": "I also tried a transformer 3D model, but conventional CNN-3D models work well.
", "votes": null, "replies": [ { "id": 1506066, "author_name": "mohammadhosein1998", "author_url": "", "post_date": "09/07/2021 19:37:10", "content": "Can you share any resource about 3D transformer?
", "votes": null, "replies": [] } ] }, { "id": 1504201, "author_name": "theudas", "author_url": "", "post_date": "09/06/2021 07:03:12", "content": "I did make a model with the Perceiver Transformer, the issue is they need lots of memory and like @aryamansharma47 wrote also lots of data.
\nI used 3 Steps of convolution + pooling to reduce the memory of the transformer.
I find some 2D transformer like Vit but i didn't find any 3D. Do you know anything?
", "votes": null, "replies": [ { "id": 1506089, "author_name": "skyyy93", "author_url": "", "post_date": "09/07/2021 20:29:55", "content": "You have to write it yourself. It is not that hard, just project 3D patches into a vector instead of 2D Patches into a Vector.
", "votes": null, "replies": [] }, { "id": 1506479, "author_name": "mohammadhosein1998", "author_url": "", "post_date": "09/08/2021 09:13:26", "content": "Did you see any notebook or code about this?
", "votes": null, "replies": [] } ] }, { "id": 1507256, "author_name": "aryamansharma47", "author_url": "", "post_date": "09/09/2021 03:05:55", "content": "https://arxiv.org/pdf/2108.09038.pdf
", "votes": null, "replies": [ { "id": 1507486, "author_name": "mohammadhosein1998", "author_url": "", "post_date": "09/09/2021 08:40:48", "content": "From this article we can understand it's not good for use Transformer instead of CNN. isn't it?
", "votes": null, "replies": [] }, { "id": 1508202, "author_name": "aryamansharma47", "author_url": "", "post_date": "09/10/2021 02:15:12", "content": "For this competition i believe that may be the case
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1503066": "Hello, have anyone trying to use transformer to solve this challenge?", "1504036": "I did consider using a transformer for this competition however i believe it is not suitable for this task mainly due to the main requirement as given in the paper 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale'\nwhich is you need to train transformers on a very large dataset. You can use pretrained models for this problem however with scarce training samples i dont think transfer learning will be successfull.\n\nSecondly results for transformers when compared with Resnets is not a very big improvement and i believe this competition is more about the data and not the model.", "1504086": "I also tried a transformer 3D model, but conventional CNN-3D models work well.", "1504201": "I did make a model with the Perceiver Transformer, the issue is they need lots of memory and like @aryamansharma47 wrote also lots of data.\nI used 3 Steps of convolution + pooling to reduce the memory of the transformer.", "1506066": "Can you share any resource about 3D transformer?", "1506069": "I find some 2D transformer like Vit but i didn't find any 3D. Do you know anything?", "1506089": "You have to write it yourself. It is not that hard, just project 3D patches into a vector instead of 2D Patches into a Vector.", "1506479": "Did you see any notebook or code about this?", "1506481": "Do you know any transformer that trained for medical images?", "1507256": "https://arxiv.org/pdf/2108.09038.pdf", "1507486": "From this article we can understand it's not good for use Transformer instead of CNN. isn't it?", "1508202": "For this competition i believe that may be the case" }, "source": "meta" }