![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10842317%2F8203b36d4bbe981f7d172cd270451c94%2Flayer.drawio.png?generation=1700420261387549&alt=media\")
First of all, thanks for hosting this competition. It was my first Kaggle competition which I entered on a whim because I had some free time. It turned out to be a lot of funstration (fun + frustration). Looking at all the amazing solutions from the other teams, I consider myself incredibly lucky to have ranked this high!
\nEdit: After cleaning up the code I noticed that I forgot to mention some things which I now added. The code is now available here.
\nMy solution is more or less composed of three parts. Minor feature extraction and engineering, tinkering with the graph, e.g. pruning, and training a graph neural network (GNN). The GNN layers are based on the GPS layers, using SAGE convolutions, Linformers and learnable positional encodings. Note that this discussion concerns mainly the layout dataset. The solution for the tile dataset is mentioned briefly at the end.
\nI used all 140 provided input features and used a simple log transform after shifting them such that each feature was at least 1. Additionally, I went through the protocol buffers and extracted the following features:
\nhas_dynamic_com: A flag indicating whether the graph has dynamic computations.is_root_of_com: A flag indicating if a node is the output node of a computation.indices_are_sorted: A flag that I am not sure why I added it.dot operation, I extracted lhs_contracting_dimensions, rhs_contracting_dimensions, lhs_batch_dimensions and rhs_batch_dimensions, which are all integer sequences that I padded to a length of 3, so 12 features in total.gather operation I added the integer sequences offset_dims, collapsed_slice_dims and start_index_map padded to length 3, the single integer index_vector_dim and the sequence gather_slice_sizes padded to length 5.The padding lengths were chosen based on the longest sequences contained in the dataset and I always used -1 as the padding value. Some of these features were useless depending on the applied graph pruning, but I left them in the input anyway. Additionally, while going through the protobufs, I added the input shapes (6D) of the two input arguments for the dot and conv operations as additional features, making sure that they are always ordered in the same way (e.g. lhs, rhs arguments for the dot). This adds 16 dimensions to the input (with sum and products of the shapes). I did this because I thought that it might be difficult for the network to learn the order of the inputs and dimensions which are reduced in these operations given solely the message-passing networks.
\nI also took the 30 dimensional features and added them to the input features once modulus 128 ((x % 128)/128 and once as true divide (x // 128)/10 with some normalization. I did this to make it easier for the network to process the dimensions of the tensors and compare them to the register size of the TPUs.
I used 128-dimensional embeddings for the OP codes. For the positional encodings, I used RWPE described here. I created 16 dimensional PEs with the directed adjacency matrix and 112 using the undirected one, for a total of 128 features. The encoding was always calculated with the full graph, independent of the pruning that was used during the training of the network.
\nI experimented with three versions of pruning/pooling:
\nAddionally, I added a virtual output node, connecting all nodes that produce ouputs.
\nThe GNN consists of SAGEConvolutions and Linformer. The architecture is shown in the figure below. The SAGEConvolutions use both the input and the output nodes with different weights and a message dimension of half the size of the input dimension. The Linformer dimension was set to 128 (or 256 in some experiments). I used Sigmoid Linear Units (SiLU) activation functions and a lot of layer normalisation.
\n
The training was done with Adam and a cosine annealing scheduler. I tried batch sizes of 8, 16 and 32 with 5, 8 or 10 configs and pairwise hinge-loss. I trained on all collections at the same time and then did finetuning on the individual collections. However, I did not have time to train a network on all collections with the merged nodes. I only implemented this towards the very end and trained only one network on the xla:default collection, which gave me the best CV. The final submission was composed of networks trained with my second pruning strategy for xla:random and the nlp collections and the third pruning strategy for xla:default.
The tile network was a simple GNN with 5 SAGEConvolutions, no extra features and no positional encodings.
\nThese are things I tried out but failed or could not evaluate if they had a consistent positive impact on the results.
\nFirst of all, thanks for hosting this competition. It was my first Kaggle competition which I entered on a whim because I had some free time. It turned out to be a lot of funstration (fun + frustration). Looking at all the amazing solutions from the other teams, I consider myself incredibly lucky to have ranked this high!
\nEdit: After cleaning up the code I noticed that I forgot to mention some things which I now added. The code is now available here.
\nMy solution is more or less composed of three parts. Minor feature extraction and engineering, tinkering with the graph, e.g. pruning, and training a graph neural network (GNN). The GNN layers are based on the GPS layers, using SAGE convolutions, Linformers and learnable positional encodings. Note that this discussion concerns mainly the layout dataset. The solution for the tile dataset is mentioned briefly at the end.
\nI used all 140 provided input features and used a simple log transform after shifting them such that each feature was at least 1. Additionally, I went through the protocol buffers and extracted the following features:
\nhas_dynamic_com: A flag indicating whether the graph has dynamic computations.is_root_of_com: A flag indicating if a node is the output node of a computation.indices_are_sorted: A flag that I am not sure why I added it.dot operation, I extracted lhs_contracting_dimensions, rhs_contracting_dimensions, lhs_batch_dimensions and rhs_batch_dimensions, which are all integer sequences that I padded to a length of 3, so 12 features in total.gather operation I added the integer sequences offset_dims, collapsed_slice_dims and start_index_map padded to length 3, the single integer index_vector_dim and the sequence gather_slice_sizes padded to length 5.The padding lengths were chosen based on the longest sequences contained in the dataset and I always used -1 as the padding value. Some of these features were useless depending on the applied graph pruning, but I left them in the input anyway. Additionally, while going through the protobufs, I added the input shapes (6D) of the two input arguments for the dot and conv operations as additional features, making sure that they are always ordered in the same way (e.g. lhs, rhs arguments for the dot). This adds 16 dimensions to the input (with sum and products of the shapes). I did this because I thought that it might be difficult for the network to learn the order of the inputs and dimensions which are reduced in these operations given solely the message-passing networks.
\nI also took the 30 dimensional features and added them to the input features once modulus 128 ((x % 128)/128 and once as true divide (x // 128)/10 with some normalization. I did this to make it easier for the network to process the dimensions of the tensors and compare them to the register size of the TPUs.
I used 128-dimensional embeddings for the OP codes. For the positional encodings, I used RWPE described here. I created 16 dimensional PEs with the directed adjacency matrix and 112 using the undirected one, for a total of 128 features. The encoding was always calculated with the full graph, independent of the pruning that was used during the training of the network.
\nI experimented with three versions of pruning/pooling:
\nAddionally, I added a virtual output node, connecting all nodes that produce ouputs.
\nThe GNN consists of SAGEConvolutions and Linformer. The architecture is shown in the figure below. The SAGEConvolutions use both the input and the output nodes with different weights and a message dimension of half the size of the input dimension. The Linformer dimension was set to 128 (or 256 in some experiments). I used Sigmoid Linear Units (SiLU) activation functions and a lot of layer normalisation.
\n
The training was done with Adam and a cosine annealing scheduler. I tried batch sizes of 8, 16 and 32 with 5, 8 or 10 configs and pairwise hinge-loss. I trained on all collections at the same time and then did finetuning on the individual collections. However, I did not have time to train a network on all collections with the merged nodes. I only implemented this towards the very end and trained only one network on the xla:default collection, which gave me the best CV. The final submission was composed of networks trained with my second pruning strategy for xla:random and the nlp collections and the third pruning strategy for xla:default.
The tile network was a simple GNN with 5 SAGEConvolutions, no extra features and no positional encodings.
\nThese are things I tried out but failed or could not evaluate if they had a consistent positive impact on the results.
\nCongratulations for securing the third position on the LB. Thanks for sharing the writeup on your solution with flowcharts.
", "rawMarkdown": "Congratulations for securing the third position on the LB. Thanks for sharing the writeup on your solution with flowcharts.", "votes": null }, { "id": "2539742", "postDate": "11/27/2023 09:36:56", "content": "Great solution. I have tried to use base Graph Transformers from GraphGPS with no improvements.
", "rawMarkdown": "Great solution. I have tried to use base Graph Transformers from GraphGPS with no improvements.", "votes": null } ], "comments": [ { "id": 2531253, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "11/20/2023 04:02:59", "content": "Congratulations for securing the third position on the LB. Thanks for sharing the writeup on your solution with flowcharts.
", "votes": null, "replies": [] }, { "id": 2539742, "author_name": "belgraviton", "author_url": "", "post_date": "11/27/2023 09:36:56", "content": "Great solution. I have tried to use base Graph Transformers from GraphGPS with no improvements.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2531022": "First of all, thanks for hosting this competition. It was my first Kaggle competition which I entered on a whim because I had some free time. It turned out to be a lot of funstration (fun + frustration). Looking at all the amazing solutions from the other teams, I consider myself incredibly lucky to have ranked this high!\n\n**Edit:** After cleaning up the code I noticed that I forgot to mention some things which I now added. The code is now available [here](https://github.com/jafluri/kaggle_tpu_graph).\n\n## Overview\n\nMy solution is more or less composed of three parts. Minor feature extraction and engineering, tinkering with the graph, e.g. pruning, and training a graph neural network (GNN). The GNN layers are based on the [GPS layers](https://openreview.net/pdf?id=lMMaNf6oxKM), using SAGE convolutions, [Linformers](https://arxiv.org/pdf/2006.04768.pdf) and [learnable positional encodings](https://arxiv.org/pdf/2110.07875.pdf). Note that this discussion concerns mainly the layout dataset. The solution for the tile dataset is mentioned briefly at the end.\n\n## Input Features\n\nI used all 140 provided input features and used a simple log transform after shifting them such that each feature was at least 1. Additionally, I went through the protocol buffers and extracted the following features:\n\n- `has_dynamic_com`: A flag indicating whether the graph has dynamic computations.\n- `is_root_of_com`: A flag indicating if a node is the output node of a computation.\n- `indices_are_sorted`: A flag that I am not sure why I added it.\n- For the `dot` operation, I extracted `lhs_contracting_dimensions`, `rhs_contracting_dimensions`, \n `lhs_batch_dimensions` and `rhs_batch_dimensions`, which are all integer sequences that I padded to a length of 3, so 12 features in total.\n- For the `gather` operation I added the integer sequences `offset_dims`, `collapsed_slice_dims` and \n `start_index_map` padded to length 3, the single integer `index_vector_dim` and the sequence `gather_slice_sizes` padded to length 5.\n\nThe padding lengths were chosen based on the longest sequences contained in the dataset and I always used -1 as the padding value. Some of these features were useless depending on the applied graph pruning, but I left them in the input anyway. Additionally, while going through the protobufs, I added the input shapes (6D) of the two input arguments for the `dot` and `conv` operations as additional features, making sure that they are always ordered in the same way (e.g. lhs, rhs arguments for the `dot`). This adds 16 dimensions to the input (with sum and products of the shapes). I did this because I thought that it might be difficult for the network to learn the order of the inputs and dimensions which are reduced in these operations given solely the message-passing networks.\nI also took the 30 dimensional features and added them to the input features once modulus 128 (`(x % 128)/128` and once as true divide `(x // 128)/10` with some normalization. I did this to make it easier for the network to process the dimensions of the tensors and compare them to the [register size](https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/437673) of the TPUs.\n\n## OP code embeddings and positional encoding\n\nI used 128-dimensional embeddings for the OP codes. For the positional encodings, I used RWPE described [here](https://arxiv.org/pdf/2110.07875.pdf). I created 16 dimensional PEs with the directed adjacency matrix and 112 using the undirected one, for a total of 128 features. The encoding was always calculated with the full graph, independent of the pruning that was used during the training of the network.\n\n## Graph Modifications\n\nI experimented with three versions of pruning/pooling:\n\n- Dropping all nodes and connections besides the configurable one. This results essentially in training an MLP.\n- Dropping all the nodes besides the configurable nodes and their inputs/outputs.\n- Merging all nodes besides the configurable nodes and their inputs/outputs. Two nodes were merged if they had at least one connection and were neither configurable nor an input or output of a configurable node. This was done until no further merging was possible. The merged nodes had a unique OP code but their features are set to zero. \n\nAddionally, I added a virtual output node, connecting all nodes that produce ouputs.\n\n## GNN\n\nThe GNN consists of SAGEConvolutions and Linformer. The architecture is shown in the figure below. The SAGEConvolutions use both the input and the output nodes with different weights and a message dimension of half the size of the input dimension. The Linformer dimension was set to 128 (or 256 in some experiments). I used Sigmoid Linear Units (SiLU) activation functions and a lot of layer normalisation.\n\n\n\nThe training was done with Adam and a cosine annealing scheduler. I tried batch sizes of 8, 16 and 32 with 5, 8 or 10 configs and pairwise hinge-loss. I trained on all collections at the same time and then did finetuning on the individual collections. However, I did not have time to train a network on all collections with the merged nodes. I only implemented this towards the very end and trained only one network on the `xla:default` collection, which gave me the best CV. The final submission was composed of networks trained with my second pruning strategy for `xla:random` and the `nlp` collections and the third pruning strategy for `xla:default`.\n\n# Tile Network\n\nThe tile network was a simple GNN with 5 SAGEConvolutions, no extra features and no positional encodings. \n\n## Other stuff\n\nThese are things I tried out but failed or could not evaluate if they had a consistent positive impact on the results. \n\n- Node dropout and test-time augmentations with the dropout. \n- Using the full graph\n- Transformers instead of Linformers (Memory)\n- Some self-implemented attention with configs and features", "2531253": "Congratulations for securing the third position on the LB. Thanks for sharing the writeup on your solution with flowcharts.", "2539742": "Great solution. I have tried to use base Graph Transformers from GraphGPS with no improvements." }, "source": "meta" }