Thanks for hosting the interesting competition, and congratulations to the winners!
\nMy solution is based on an end-to-end graph neural network (GNN). I implemented a 3-layer GraphSage based on PyG. In each layer, I operate graph convolution in both directions of edges by different weights and concatenate the outputs.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9088007%2Fa038962047b9285a904e29d787b3d1ff%2FTPUGraphs-GNN.drawio.png?generation=1702755579495636&alt=media\")
\nI trained the model to minimize pairwise hinge loss using the AdamW optimizer using a cosine annealing scheduler.
\nFor the loss, I used the average of a pairwise hinge loss among different configurations of the same graph and a pairwise hinge loss among all the samples in a batch (including different graphs). For this reason, I didn't use a subgraph but a whole graph as an input to GNN.
Node features include 30 features (including tile and layout configurations) for each of the 6 dimensions. A naive approach to input this to GNN is to simply flatten them (I call this naive model), but I considered the following two disadvantages.
\nTo tackle these issues, I implemented a dimension feature embedding layer using a transformer that handles each dimension as a token. In this layer, I transform (6, 30) input to (6, mid_ch) by a transformer and reduce to (mid_ch) by taking the sum in the token dimension.
\nSince most dimension features are exactly the same (padded ones), I could compute this efficiently by calculating embedding for only unique ones in each batch and copying them.
I trained the model using only the tile dataset.
\nUsing the transformer model, I could easily achieve 0.2 (nearly perfect) in public and private LB. The transformer model was significantly better than the naive approach on the validation Kendall tau score.
I trained the model using the whole layout dataset (random and default of xla and nlp). Also, including the tile dataset enhanced the performance a little.
\nI could not outperform the naive model by the transformer model in the validation score (due to limited time), but it was comparable. My final submission was an ensemble of naive models and transformer models.
\nI acknowledge Preferred Networks, Inc. for allowing me to use computational resources.
\nSource Code: https://github.com/knshnb/kaggle-tpu-graph-5th-place
", "messages": [ { "id": "2529179", "postDate": "11/18/2023 02:39:01", "content": "Thanks for hosting the interesting competition, and congratulations to the winners!
\nMy solution is based on an end-to-end graph neural network (GNN). I implemented a 3-layer GraphSage based on PyG. In each layer, I operate graph convolution in both directions of edges by different weights and concatenate the outputs.
\n
\nI trained the model to minimize pairwise hinge loss using the AdamW optimizer using a cosine annealing scheduler.
\nFor the loss, I used the average of a pairwise hinge loss among different configurations of the same graph and a pairwise hinge loss among all the samples in a batch (including different graphs). For this reason, I didn't use a subgraph but a whole graph as an input to GNN.
Node features include 30 features (including tile and layout configurations) for each of the 6 dimensions. A naive approach to input this to GNN is to simply flatten them (I call this naive model), but I considered the following two disadvantages.
\nTo tackle these issues, I implemented a dimension feature embedding layer using a transformer that handles each dimension as a token. In this layer, I transform (6, 30) input to (6, mid_ch) by a transformer and reduce to (mid_ch) by taking the sum in the token dimension.
\nSince most dimension features are exactly the same (padded ones), I could compute this efficiently by calculating embedding for only unique ones in each batch and copying them.
I trained the model using only the tile dataset.
\nUsing the transformer model, I could easily achieve 0.2 (nearly perfect) in public and private LB. The transformer model was significantly better than the naive approach on the validation Kendall tau score.
I trained the model using the whole layout dataset (random and default of xla and nlp). Also, including the tile dataset enhanced the performance a little.
\nI could not outperform the naive model by the transformer model in the validation score (due to limited time), but it was comparable. My final submission was an ensemble of naive models and transformer models.
\nI acknowledge Preferred Networks, Inc. for allowing me to use computational resources.
\nSource Code: https://github.com/knshnb/kaggle-tpu-graph-5th-place
", "rawMarkdown": "Thanks for hosting the interesting competition, and congratulations to the winners!\n\n## Overview\nMy solution is based on an end-to-end graph neural network (GNN). I implemented a 3-layer GraphSage based on [PyG](https://pytorch-geometric.readthedocs.io/en/latest/). In each layer, I operate graph convolution in both directions of edges by different weights and concatenate the outputs.\n\nI trained the model to minimize pairwise hinge loss using the AdamW optimizer using a cosine annealing scheduler.\nFor the loss, I used the average of a pairwise hinge loss among different configurations of the same graph and a pairwise hinge loss among all the samples in a batch (including different graphs). For this reason, I didn't use a subgraph but a whole graph as an input to GNN.\n\n## Dimension Feature Embed by Transformer\nNode features include 30 features (including tile and layout configurations) for each of the 6 dimensions. A naive approach to input this to GNN is to simply flatten them (I call this naive model), but I considered the following two disadvantages.\n- It drops prior information about feature correspondence across dimensions\n- The output should be invariant to the indexing order of dimensions (I'm not sure if this is exactly correct)\n\nTo tackle these issues, I implemented a dimension feature embedding layer using a transformer that handles each dimension as a token. In this layer, I transform (6, 30) input to (6, mid_ch) by a transformer and reduce to (mid_ch) by taking the sum in the token dimension.\nSince most dimension features are exactly the same (padded ones), I could compute this efficiently by calculating embedding for only unique ones in each batch and copying them.\n\n## Tile Config Dataset\nI trained the model using only the tile dataset.\nUsing the transformer model, I could easily achieve 0.2 (nearly perfect) in public and private LB. The transformer model was significantly better than the naive approach on the validation Kendall tau score.\n\n## Layout Config Dataset\nI trained the model using the whole layout dataset (random and default of xla and nlp). Also, including the tile dataset enhanced the performance a little.\n\nI could not outperform the naive model by the transformer model in the validation score (due to limited time), but it was comparable. My final submission was an ensemble of naive models and transformer models.\n\n## Tips\n- use the same opcode embedding for unary operations such as abs, ceil, cosine, etc.\n- override layout_minor_to_major by layout config features for configurable nodes\n- [DropEdge](https://arxiv.org/abs/1907.10903)\n- apply log transformation to input features\n- oversampling\n- load layout config data by numpy's mmap mode to save RAM\n\n## What Didn't Work\n- graph pooling\n- pretrain on the tile dataset and finetune on the layout dataset\n- graph normalization\n- dropout node\n- GAT, GATv2, GIN\n- fp16\n- pseudo label\n\n## Acknowledgement\nI acknowledge Preferred Networks, Inc. for allowing me to use computational resources.\n\n\nSource Code: https://github.com/knshnb/kaggle-tpu-graph-5th-place", "votes": null }, { "id": "2529222", "postDate": "11/18/2023 04:08:03", "content": "How did you get the number 43?
\nBTW, congrats for winning!
Sorry, 43 was a mistake and it was 30 correctly (I was tired from the hard work…). I extracted 10 that end with _{i} from the node features, 14 dim numbers from the node features, 3 from the tile config, and 3 from the layout config.
Thanks for the write-up and thanks for sharing. Would you share your training code at some point?
", "rawMarkdown": "Thanks for the write-up and thanks for sharing. Would you share your training code at some point?", "votes": null }, { "id": "2529890", "postDate": "11/18/2023 16:15:57", "content": "Congratulations on achieving 5th place. Thanks for sharing info about your solution.
", "rawMarkdown": "Congratulations on achieving 5th place. Thanks for sharing info about your solution.", "votes": null }, { "id": "2530514", "postDate": "11/19/2023 08:04:51", "content": "Thanks! Yeah, I'm going to publish the code later.
", "rawMarkdown": "Thanks! Yeah, I'm going to publish the code later.", "votes": null }, { "id": "2530752", "postDate": "11/19/2023 13:50:05", "content": "Update: Due to limited time, I could only train the final best model for 1.2 epochs out of 2 epochs. After the training of whole epochs finished, this single model achieved 1st place score…😢😢😢
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9088007%2F07b81bbe731009d97fee08622f9997ba%2FF_TAVI6bcAAMHBg.jpeg?generation=1700401008267090&alt=media\")
Hey man, congratulations!! Really smart how you handled the dimension features!
", "rawMarkdown": "Hey man, congratulations!! Really smart how you handled the dimension features!", "votes": null }, { "id": "2531809", "postDate": "11/20/2023 14:54:46", "content": "Congratulations!
\nI have a question about this:
\n\n\nI operate graph convolution in both directions of edges by different weights and concatenate the outputs.
\n
What were the weights you used? What was the intuition behind this decision?
", "rawMarkdown": "Congratulations!\n\nI have a question about this:\n> I operate graph convolution in both directions of edges by different weights and concatenate the outputs.\n\nWhat were the weights you used? What was the intuition behind this decision?", "votes": null }, { "id": "2531918", "postDate": "11/20/2023 16:30:44", "content": "Thanks, and congratulations on your 1st place!
", "rawMarkdown": "Thanks, and congratulations on your 1st place!", "votes": null }, { "id": "2532820", "postDate": "11/21/2023 11:08:15", "content": "Thanks!
\nI mean I concatenated the outputs of two independent graph convolutions. The pseudocode is like this:
self.gconv = SAGEConv(mid_ch, mid_ch // , ...)\nself.rev_gconv =SAGEConv(mid_ch, mid_ch // , ...)\n\nrev_edge_index = torch.flip(edge_index, (,))\nx = torch.cat([self.gconv(x, edge_index), self.rev_gconv(x, rev_edge_index)], dim=-))\nIntuition: If we have an edge (u, v), we want to propagate information not only from u to v but also from v to u. However, the weight should not be equal because it's a directed edge.
", "rawMarkdown": "Thanks!\nI mean I concatenated the outputs of two independent graph convolutions. The pseudocode is like this:\n```python\nself.gconv = SAGEConv(mid_ch, mid_ch // 2, ...)\nself.rev_gconv =SAGEConv(mid_ch, mid_ch // 2, ...)\n\nrev_edge_index = torch.flip(edge_index, (0,))\nx = torch.cat([self.gconv(x, edge_index), self.rev_gconv(x, rev_edge_index)], dim=-1))\n```\n\nIntuition: If we have an edge (u, v), we want to propagate information not only from u to v but also from v to u. However, the weight should not be equal because it's a directed edge.", "votes": null }, { "id": "2532933", "postDate": "11/21/2023 13:00:13", "content": "Awesome! Thanks again. 🥳
", "rawMarkdown": "Awesome! Thanks again. 🥳", "votes": null }, { "id": "2551117", "postDate": "12/06/2023 14:08:58", "content": "I published a source code!
\nhttps://github.com/knshnb/kaggle-tpu-graph-5th-place
thanks for the writeup. i was at the winner call video conference with you today. i am impressed with your work!.
\nI have a question: any idea why the public score of the post submission (2 epoch trained) is much lower than the private one?
\nhow is your validation score? 0.739 and 0.707 is a huge difference
Thanks for the comment. I learned a lot from your presentation!
\nI just focused on improving the validation score and did not care about public and private during the competition. According to the LB probing analysis by @charmq, it seems public data has some graphs that are far from training graphs, which caused high randomness.
\nhttps://twitter.com/charmq00/status/1725675672222503043
\n(Sorry, the above tweet is in Japanese…)
For others, this is the post submission results I mentioned in the winner call.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9088007%2F7da5830169c4933ab603971a32033ca4%2FScreen%20Shot%202023-12-11%20at%200.22.31.png?generation=1702222223063708&alt=media\")
How did you get the number 43?
\nBTW, congrats for winning!
Sorry, 43 was a mistake and it was 30 correctly (I was tired from the hard work…). I extracted 10 that end with _{i} from the node features, 14 dim numbers from the node features, 3 from the tile config, and 3 from the layout config.
Thanks for the write-up and thanks for sharing. Would you share your training code at some point?
", "votes": null, "replies": [ { "id": 2530514, "author_name": "knshnb", "author_url": "", "post_date": "11/19/2023 08:04:51", "content": "Thanks! Yeah, I'm going to publish the code later.
", "votes": null, "replies": [ { "id": 2532933, "author_name": "yassinealouini", "author_url": "", "post_date": "11/21/2023 13:00:13", "content": "Awesome! Thanks again. 🥳
", "votes": null, "replies": [] } ] } ] }, { "id": 2529890, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "11/18/2023 16:15:57", "content": "Congratulations on achieving 5th place. Thanks for sharing info about your solution.
", "votes": null, "replies": [] }, { "id": 2530752, "author_name": "knshnb", "author_url": "", "post_date": "11/19/2023 13:50:05", "content": "Update: Due to limited time, I could only train the final best model for 1.2 epochs out of 2 epochs. After the training of whole epochs finished, this single model achieved 1st place score…😢😢😢
\n
thanks for the writeup. i was at the winner call video conference with you today. i am impressed with your work!.
\nI have a question: any idea why the public score of the post submission (2 epoch trained) is much lower than the private one?
\nhow is your validation score? 0.739 and 0.707 is a huge difference
Thanks for the comment. I learned a lot from your presentation!
\nI just focused on improving the validation score and did not care about public and private during the competition. According to the LB probing analysis by @charmq, it seems public data has some graphs that are far from training graphs, which caused high randomness.
\nhttps://twitter.com/charmq00/status/1725675672222503043
\n(Sorry, the above tweet is in Japanese…)
For others, this is the post submission results I mentioned in the winner call.
\n
Hey man, congratulations!! Really smart how you handled the dimension features!
", "votes": null, "replies": [ { "id": 2531918, "author_name": "knshnb", "author_url": "", "post_date": "11/20/2023 16:30:44", "content": "Thanks, and congratulations on your 1st place!
", "votes": null, "replies": [] } ] }, { "id": 2531809, "author_name": "mattdeakos", "author_url": "", "post_date": "11/20/2023 14:54:46", "content": "Congratulations!
\nI have a question about this:
\n\n\nI operate graph convolution in both directions of edges by different weights and concatenate the outputs.
\n
What were the weights you used? What was the intuition behind this decision?
", "votes": null, "replies": [ { "id": 2532820, "author_name": "knshnb", "author_url": "", "post_date": "11/21/2023 11:08:15", "content": "Thanks!
\nI mean I concatenated the outputs of two independent graph convolutions. The pseudocode is like this:
self.gconv = SAGEConv(mid_ch, mid_ch // , ...)\nself.rev_gconv =SAGEConv(mid_ch, mid_ch // , ...)\n\nrev_edge_index = torch.flip(edge_index, (,))\nx = torch.cat([self.gconv(x, edge_index), self.rev_gconv(x, rev_edge_index)], dim=-))\nIntuition: If we have an edge (u, v), we want to propagate information not only from u to v but also from v to u. However, the weight should not be equal because it's a directed edge.
", "votes": null, "replies": [] } ] }, { "id": 2551117, "author_name": "knshnb", "author_url": "", "post_date": "12/06/2023 14:08:58", "content": "I published a source code!
\nhttps://github.com/knshnb/kaggle-tpu-graph-5th-place