![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8094969%2F590706064952d89072a4f8e44c0b91c3%2Fgeometry-aware-transformer-diagram.drawio.png?generation=1681980865962692&alt=media\"){kind=spacer}
Here's my competition writeup! Unfortunately I didn't make it into my goal of the top 30, but I am happy with the result nonetheless.
\nI used a single-model geometry-aware transformer. After seeing the performance of the GraphNet baseline, I tried other point-cloud inspired models. I landed on the geometry-aware transformer (from this paper) as the backbone for the model.
\nThe geometry-aware transformer block I used is modified from the PoinTr paper, and uses a radius-graph query (much like the edge-conv blocks in GraphNet), along side multi-head attention. I found that a radius graph performed significantly better than a KNN graph for the edges.
\nThe particular structure I got the best performance out of is below:
It is interesting to me to see that a few of the top solutions were also transformers, with some differences and many more blocks: I only used 4 transformer blocks for my final model, with an embedding dimension of 256 - by the end of the competition I had limited time to train this model - I would have liked to try larger models as well. I think in this respect I was also held back by using the edge-conv structure, which seems to not scale as well as only self-attention.
", "messages": [ { "id": "2228085", "postDate": "04/20/2023 09:00:53", "content": "Here's my competition writeup! Unfortunately I didn't make it into my goal of the top 30, but I am happy with the result nonetheless.
\nI used a single-model geometry-aware transformer. After seeing the performance of the GraphNet baseline, I tried other point-cloud inspired models. I landed on the geometry-aware transformer (from this paper) as the backbone for the model.
\nThe geometry-aware transformer block I used is modified from the PoinTr paper, and uses a radius-graph query (much like the edge-conv blocks in GraphNet), along side multi-head attention. I found that a radius graph performed significantly better than a KNN graph for the edges.
\nThe particular structure I got the best performance out of is below:
It is interesting to me to see that a few of the top solutions were also transformers, with some differences and many more blocks: I only used 4 transformer blocks for my final model, with an embedding dimension of 256 - by the end of the competition I had limited time to train this model - I would have liked to try larger models as well. I think in this respect I was also held back by using the edge-conv structure, which seems to not scale as well as only self-attention.
", "rawMarkdown": "Here's my competition writeup! Unfortunately I didn't make it into my goal of the top 30, but I am happy with the result nonetheless.\n\nI used a single-model geometry-aware transformer. After seeing the performance of the GraphNet baseline, I tried other point-cloud inspired models. I landed on the geometry-aware transformer (from [this paper](https://arxiv.org/abs/2108.08839)) as the backbone for the model.\n\nThe geometry-aware transformer block I used is modified from the PoinTr paper, and uses a radius-graph query (much like the edge-conv blocks in GraphNet), along side multi-head attention. I found that a radius graph performed significantly better than a KNN graph for the edges.\nThe particular structure I got the best performance out of is below:\n\n\n\nIt is interesting to me to see that a few of the top solutions were also transformers, with some differences and many more blocks: I only used 4 transformer blocks for my final model, with an embedding dimension of 256 - by the end of the competition I had limited time to train this model - I would have liked to try larger models as well. I think in this respect I was also held back by using the edge-conv structure, which seems to not scale as well as only self-attention.", "votes": null }, { "id": "2228475", "postDate": "04/20/2023 15:17:44", "content": "Congrats @sebastianvangerwen! Thanks for sharing your solution; it’s interesting to see this transformer network!
", "rawMarkdown": "Congrats @sebastianvangerwen! Thanks for sharing your solution; it’s interesting to see this transformer network!", "votes": null } ], "comments": [ { "id": 2228475, "author_name": "ravishah1", "author_url": "", "post_date": "04/20/2023 15:17:44", "content": "Congrats @sebastianvangerwen! Thanks for sharing your solution; it’s interesting to see this transformer network!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2228085": "Here's my competition writeup! Unfortunately I didn't make it into my goal of the top 30, but I am happy with the result nonetheless.\n\nI used a single-model geometry-aware transformer. After seeing the performance of the GraphNet baseline, I tried other point-cloud inspired models. I landed on the geometry-aware transformer (from [this paper](https://arxiv.org/abs/2108.08839)) as the backbone for the model.\n\nThe geometry-aware transformer block I used is modified from the PoinTr paper, and uses a radius-graph query (much like the edge-conv blocks in GraphNet), along side multi-head attention. I found that a radius graph performed significantly better than a KNN graph for the edges.\nThe particular structure I got the best performance out of is below:\n\n\n\nIt is interesting to me to see that a few of the top solutions were also transformers, with some differences and many more blocks: I only used 4 transformer blocks for my final model, with an embedding dimension of 256 - by the end of the competition I had limited time to train this model - I would have liked to try larger models as well. I think in this respect I was also held back by using the edge-conv structure, which seems to not scale as well as only self-attention.", "2228475": "Congrats @sebastianvangerwen! Thanks for sharing your solution; it’s interesting to see this transformer network!" }, "source": "meta" }