![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1648129%2F7cf79a021748ba6f4b38529447419992%2Fwriteup_graph_model.png?generation=1700408602690996&alt=media\")
First of all, thanks Kaggle and Google for hosting this great competition. Also, I'd like to thanks my teammates and friends @tomirol and @thanhhau097a, which made this journey even more special.
\nnode_feat to use -1 padding instead of 0train, val, test splits since we found good correlation with LBnode_feat[:134]node_feat[134:] and node_config_featInstanceNorm -> SAGEConv -> SelfChannelAttetion -> CrossConfigAttetion -> +residual -> GELUPairwiseHingeLoss functionWe joined the competition with a bit less than a month to finish, so one of the first problems we tried to tackle was the low efficiency in our training jobs.
\nFor layout, we noticed that only Convolution, Dot and Reshape were configurable nodes. Also, in most cases, the majority of nodes would be identical across the config set. Thus, we opted for a very simple pruning strategy where, for each graph, we would only keep the nodes that were either configurable models themselves or were connected to a configurable node, i.e., input or output to a configurable node. By doing this, we would transform a single big graph into multiple (possibly disconnected) sub-graphs, which was not a problem since the network has a global graph pooling layer in the end that fuses the sub-graphs information. This simple trick reduced 4 times the vRAM usage and sped up training by a factor of 5 in some cases.
Most of the configuration sets for layout contain a lot of duplication. However, the runtime for the duplicated configs can vary quite a bit and make training less stable. Thus, we opted for removing all the duplicated configs for layout.
\nEven with pruning and de-duplication, the RAM usage to load all configs to memory for NLP collection was super high. We circumvent that issue by compressing node_config_feat beforehand and only decompressing it on-the-fly in the dataloader after config sampling. This enabled us to load all data to memory at the beginning of training, which reduced IO/CPU bottlenecks considerably and allowed us to train faster.
The idea behind the compression is that each node_config_feat 6-dim vector (input, output and kernel) can only have 7 possible values (-1, 0, 1, 2, 3, 4, 5) and, thus, can be represented by a single integer in base-7 (from 0 to 7^6).
node_featWe noted that the features in node_feat were 0 padded. Whilst this is not a problem for most features, for others like layout_minor_to_major_* this can be ambiguous since 0 is a valid axis index. Also, the node_config_feat are -1 padded, which makes it incompatible with layout_minor_to_major_* from node_feat. With that in mind, we re-generated node_feat with -1 padded and this allowed us to use a single embedding matrix for both node_feat[134:] and node_config_feat.
For layout, we split node_feat into node_feat[:134] and node_feat[134:] (layout_minor_to_major_*). The former was simply normalised using a StandardScaler, while the latter, along with node_config_feat, was fed into a learned embedding matrix (4 channels). We found that the normalisation is essential here since node_feat has features like *_sum and *_product that can be very high and, consequently, disrupt the optimisation.
For node_opcode, we also used a separate embedding layer with 16 channels. The input to the network is the concatenation of all features aforementioned and, for each graph, we sample on-the-fly 64 (default) or 128 (random) configs to form the input batch. For tile, on the other hand, we opt to use late fusion to integrate config_feat into the network.
Our network architecture was quite simple. We first feed the input features to a Linear block to map it to a 256d embedding vector followed by 2x Conv blocks, global graph mean pooling and a final linear layer.
\nAs for the graph convolutional layer itself, we tried many types but none was better than SAGEConv. In particular, I had good experience with GAT variants in the past but none worked well in this competition. If I were to guess the reason, I'd say that in the other application the graph itself was quite noisy, so attention helped to \"ignore\" the connections that were not meaningful. However, for TPU graphs, all connections are \"real\" and important so graph attention was not that helpful. Nonetheless, we found two other types of attention that were useful: self-channel attention and cross-config attention.
We borrowed the idea from Squeeze-and-Excitation to create a channel-wise attention layer. We first apply a Linear layer to bottleneck the channel dimensions (8x reduction) followed by ReLU. Then, we applied a second linear layer to increase the channels again to the original value followed by sigmoid. We finish by applying element wise multiplication on the obtained feature map and the original input.
The idea behind this is to capture the correlations between channels and use it to suppress less useful ones while enhancing others.
\nAnother dimension that we can exploit attention is the batch plane (cross-configs). We designed a very simple block that allows the model to explicitly \"compare\" each config against the others throughout the network. We found this to be much better than letting the model infer for each config individually and only compare them implicitly via the loss function (PairwiseHingeLoss). The attention code is as follows:
(nn.Module):\n ():\n ().__init__()\n .temperature = nn.Parameter(torch.tensor())\n\n ():\n \n scores = (x / .temperature).softmax(dim=)\n x = x * scores\n x\nBy applying this simple layer after the self-channel attention at every block of the network, it gave us a huge boost for default collections. For inference, we simply use a reasonably large batch size of 128. However, since the prediction depends on the batch, we can leverage it further by applying TTA to generate N (10) permutations of the configs and average the result after sorting it back to the original order.
To create our Linear/Conv blocks we followed the good practices in computer vision. We start by using InstanceNorm to normalise the input feature map, followed by Linear/SAGEConv layer, SelfChannelAttetion and CrossConfigAttetion (we concat the output with its input to preserve the individuality of each sample). Then, we sum the residual connection and finish with GELU and dropout.
Our best single model prediction scored 0.714 (0.748) on private (public) LB. However, since the number of test samples is quite low for some collections, we opted to use ensembles to improve the results and prevent shaking up. Our best result was 0.736 (0.757) LB by using the simple average of 5-10 models for each collection but we sadly didn't select this sub.
\nI made a quick diagram of our network, it's a bit crap but should give a good idea 😅
\n
First of all, thanks Kaggle and Google for hosting this great competition. Also, I'd like to thanks my teammates and friends @tomirol and @thanhhau097a, which made this journey even more special.
\nnode_feat to use -1 padding instead of 0train, val, test splits since we found good correlation with LBnode_feat[:134]node_feat[134:] and node_config_featInstanceNorm -> SAGEConv -> SelfChannelAttetion -> CrossConfigAttetion -> +residual -> GELUPairwiseHingeLoss functionWe joined the competition with a bit less than a month to finish, so one of the first problems we tried to tackle was the low efficiency in our training jobs.
\nFor layout, we noticed that only Convolution, Dot and Reshape were configurable nodes. Also, in most cases, the majority of nodes would be identical across the config set. Thus, we opted for a very simple pruning strategy where, for each graph, we would only keep the nodes that were either configurable models themselves or were connected to a configurable node, i.e., input or output to a configurable node. By doing this, we would transform a single big graph into multiple (possibly disconnected) sub-graphs, which was not a problem since the network has a global graph pooling layer in the end that fuses the sub-graphs information. This simple trick reduced 4 times the vRAM usage and sped up training by a factor of 5 in some cases.
Most of the configuration sets for layout contain a lot of duplication. However, the runtime for the duplicated configs can vary quite a bit and make training less stable. Thus, we opted for removing all the duplicated configs for layout.
\nEven with pruning and de-duplication, the RAM usage to load all configs to memory for NLP collection was super high. We circumvent that issue by compressing node_config_feat beforehand and only decompressing it on-the-fly in the dataloader after config sampling. This enabled us to load all data to memory at the beginning of training, which reduced IO/CPU bottlenecks considerably and allowed us to train faster.
The idea behind the compression is that each node_config_feat 6-dim vector (input, output and kernel) can only have 7 possible values (-1, 0, 1, 2, 3, 4, 5) and, thus, can be represented by a single integer in base-7 (from 0 to 7^6).
node_featWe noted that the features in node_feat were 0 padded. Whilst this is not a problem for most features, for others like layout_minor_to_major_* this can be ambiguous since 0 is a valid axis index. Also, the node_config_feat are -1 padded, which makes it incompatible with layout_minor_to_major_* from node_feat. With that in mind, we re-generated node_feat with -1 padded and this allowed us to use a single embedding matrix for both node_feat[134:] and node_config_feat.
For layout, we split node_feat into node_feat[:134] and node_feat[134:] (layout_minor_to_major_*). The former was simply normalised using a StandardScaler, while the latter, along with node_config_feat, was fed into a learned embedding matrix (4 channels). We found that the normalisation is essential here since node_feat has features like *_sum and *_product that can be very high and, consequently, disrupt the optimisation.
For node_opcode, we also used a separate embedding layer with 16 channels. The input to the network is the concatenation of all features aforementioned and, for each graph, we sample on-the-fly 64 (default) or 128 (random) configs to form the input batch. For tile, on the other hand, we opt to use late fusion to integrate config_feat into the network.
Our network architecture was quite simple. We first feed the input features to a Linear block to map it to a 256d embedding vector followed by 2x Conv blocks, global graph mean pooling and a final linear layer.
\nAs for the graph convolutional layer itself, we tried many types but none was better than SAGEConv. In particular, I had good experience with GAT variants in the past but none worked well in this competition. If I were to guess the reason, I'd say that in the other application the graph itself was quite noisy, so attention helped to \"ignore\" the connections that were not meaningful. However, for TPU graphs, all connections are \"real\" and important so graph attention was not that helpful. Nonetheless, we found two other types of attention that were useful: self-channel attention and cross-config attention.
We borrowed the idea from Squeeze-and-Excitation to create a channel-wise attention layer. We first apply a Linear layer to bottleneck the channel dimensions (8x reduction) followed by ReLU. Then, we applied a second linear layer to increase the channels again to the original value followed by sigmoid. We finish by applying element wise multiplication on the obtained feature map and the original input.
The idea behind this is to capture the correlations between channels and use it to suppress less useful ones while enhancing others.
\nAnother dimension that we can exploit attention is the batch plane (cross-configs). We designed a very simple block that allows the model to explicitly \"compare\" each config against the others throughout the network. We found this to be much better than letting the model infer for each config individually and only compare them implicitly via the loss function (PairwiseHingeLoss). The attention code is as follows:
(nn.Module):\n ():\n ().__init__()\n .temperature = nn.Parameter(torch.tensor())\n\n ():\n \n scores = (x / .temperature).softmax(dim=)\n x = x * scores\n x\nBy applying this simple layer after the self-channel attention at every block of the network, it gave us a huge boost for default collections. For inference, we simply use a reasonably large batch size of 128. However, since the prediction depends on the batch, we can leverage it further by applying TTA to generate N (10) permutations of the configs and average the result after sorting it back to the original order.
To create our Linear/Conv blocks we followed the good practices in computer vision. We start by using InstanceNorm to normalise the input feature map, followed by Linear/SAGEConv layer, SelfChannelAttetion and CrossConfigAttetion (we concat the output with its input to preserve the individuality of each sample). Then, we sum the residual connection and finish with GELU and dropout.
Our best single model prediction scored 0.714 (0.748) on private (public) LB. However, since the number of test samples is quite low for some collections, we opted to use ensembles to improve the results and prevent shaking up. Our best result was 0.736 (0.757) LB by using the simple average of 5-10 models for each collection but we sadly didn't select this sub.
\nI made a quick diagram of our network, it's a bit crap but should give a good idea 😅
\n
Congrats @arc144 and team! Nice to see a brazillian winner 😉
\nThanks for sharing.
Obrigado Giba! 🙇
", "rawMarkdown": "Obrigado Giba! 🙇", "votes": 2 } ] }, { "id": 2531509, "postDate": "2023-11-20T09:18:37.973Z", "content": "Congrats on the 1st place! I am curious: what tool did you use to make this beautiful block diagram in the writeup?
", "rawMarkdown": "Congrats on the 1st place! I am curious: what tool did you use to make this beautiful block diagram in the writeup?", "votes": 3, "replies": [ { "id": 2531680, "postDate": "2023-11-20T12:24:44.493Z", "content": "Thank you Dmitrii! Congratulation to you and your team as well!
\nI used a website called drawIo
Hi Eduardo,
\nI'm so glad to read your write-up. I'm a beginner, so it's very hard to understand it all.
Congratulations for you and your team (Thanh Hau Nguyen and Tomirol) for the amazing achievement.
\nParabéns!
Muito obrigado Marília 🙏
\nLet me know if there is anything I can help clarifying!
", "rawMarkdown": "Muito obrigado Marília 🙏\n\nLet me know if there is anything I can help clarifying!", "votes": 2 } ] }, { "id": 2606250, "postDate": "2024-01-17T14:11:39.653Z", "content": "Congrats on winning the competition! :) Thanks for implementation details. If you could, kindly share training details, ex: GPU used, GPU training time etc.
", "rawMarkdown": "Congrats on winning the competition! :) Thanks for implementation details. If you could, kindly share training details, ex: GPU used, GPU training time etc.", "votes": 1, "replies": [ { "id": 2674796, "postDate": "2024-02-29T14:45:53.720Z", "content": "Sorry for the late response, was away recently. We used a RTX 4090 and it took around 1-2 hours (depends on the collection) to train the models after the optimisations we described in the thread.
", "rawMarkdown": "Sorry for the late response, was away recently. We used a RTX 4090 and it took around 1-2 hours (depends on the collection) to train the models after the optimisations we described in the thread.", "votes": 1 } ] }, { "id": 2561136, "postDate": "2023-12-14T09:53:50.293Z", "content": "congratulations!
", "rawMarkdown": "congratulations!", "votes": 1 }, { "id": 2534259, "postDate": "2023-11-22T14:07:37.950Z", "content": "Great solution. Architecture ideas are quite amazing. Learning embeddings for configs' features is what I was missing :)
", "rawMarkdown": "Great solution. Architecture ideas are quite amazing. Learning embeddings for configs' features is what I was missing :)", "votes": 1 }, { "id": 2533774, "postDate": "2023-11-22T08:10:40.150Z", "content": "Congratulations, first place! What tool did you use to create the graph in the last image?
", "rawMarkdown": "Congratulations, first place! What tool did you use to create the graph in the last image?", "votes": 1, "replies": [ { "id": 2533970, "postDate": "2023-11-22T10:15:30.680Z", "content": "Thank you! I use a tool called drawIo
", "rawMarkdown": "Thank you! I use a tool called [drawIo](https://app.diagrams.net/)", "votes": 1 } ] }, { "id": 2533350, "postDate": "2023-11-21T20:09:01.367Z", "content": "congratulations.
", "rawMarkdown": "congratulations. ", "votes": 1 }, { "id": 2532973, "postDate": "2023-11-21T13:49:32.747Z", "content": "Congratulations and thanks for sharing! The solution helps me capture all the details.
", "rawMarkdown": "Congratulations and thanks for sharing! The solution helps me capture all the details.", "votes": 1 }, { "id": 2532336, "postDate": "2023-11-21T01:15:46.477Z", "content": "Congrats for 1st place. Does an ensemble model use the same weight for each model?
", "rawMarkdown": "Congrats for 1st place. Does an ensemble model use the same weight for each model?", "votes": 1, "replies": [ { "id": 2532736, "postDate": "2023-11-21T09:46:59.803Z", "content": "Thank you! For the ensemble, we train the same model architecture many (5 - 10 usually) times with different random seeds (main impact is in the model's weights initialisation). Then, we just load each model/trained weight individually to make the prediction and average them up
", "rawMarkdown": "Thank you! For the ensemble, we train the same model architecture many (5 - 10 usually) times with different random seeds (main impact is in the model's weights initialisation). Then, we just load each model/trained weight individually to make the prediction and average them up", "votes": 1, "replies": [ { "id": 2533377, "postDate": "2023-11-21T20:51:25.077Z", "content": "Using different weights could sometimes improve accuracy. In another competition, I used the Optuna library to experiment with different weight combinations and found the optimal ones, leading to further accuracy gains. Congratulations again on your excellent solutions!
", "rawMarkdown": "Using different weights could sometimes improve accuracy. In another competition, I used the Optuna library to experiment with different weight combinations and found the optimal ones, leading to further accuracy gains. Congratulations again on your excellent solutions!", "votes": 1 } ] } ] }, { "id": 2532324, "postDate": "2023-11-21T00:34:13.307Z", "content": "Congrats brother on achieving 1st place.
", "rawMarkdown": "Congrats brother on achieving 1st place.", "votes": 1 }, { "id": 2532248, "postDate": "2023-11-20T21:28:00.927Z", "content": "Thanks for sharing, congrats for winning
", "rawMarkdown": "Thanks for sharing, congrats for winning", "votes": 1 }, { "id": 2531240, "postDate": "2023-11-20T03:53:28.827Z", "content": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with flowcharts.
", "rawMarkdown": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with flowcharts. ", "votes": 1 }, { "id": 2531191, "postDate": "2023-11-20T02:28:38.813Z", "content": "Congratulations for your team and learn lots from your solution!
", "rawMarkdown": "Congratulations for your team and learn lots from your solution!", "votes": 1 }, { "id": 2533324, "postDate": "2023-11-21T19:25:23.300Z", "content": "Great competition solution! Your approach, especially using techniques like graph pruning and deduplication in data preprocessing, is quite interesting for achieving high performance. Additionally, your optimization of network architecture and attention mechanisms is very impressive. The results you obtained with ensembling are noteworthy. Congratulations! 🚀
", "rawMarkdown": "Great competition solution! Your approach, especially using techniques like graph pruning and deduplication in data preprocessing, is quite interesting for achieving high performance. Additionally, your optimization of network architecture and attention mechanisms is very impressive. The results you obtained with ensembling are noteworthy. Congratulations! 🚀", "votes": 2 }, { "id": 2533525, "postDate": "2023-11-22T02:21:27.817Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2532621, "postDate": "2023-11-21T07:30:48.087Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2532982, "postDate": "2023-11-21T13:59:45.577Z", "content": "Good job! Thanks for the sharing!
", "rawMarkdown": "Good job! Thanks for the sharing!", "votes": 1 }, { "id": 2532390, "postDate": "2023-11-21T03:07:21.020Z", "content": "congrats! Thanks for sharing
", "rawMarkdown": "congrats! Thanks for sharing", "votes": 1 }, { "id": 2531335, "postDate": "2023-11-20T06:05:09.580Z", "content": "Thanks for sharing！
", "rawMarkdown": "Thanks for sharing！", "votes": 1 } ], "comments": [ { "id": 2537036, "author_name": "Giba", "author_url": "", "post_date": "2023-11-24T17:18:51.283000", "content": "Congrats @arc144 and team! Nice to see a brazillian winner 😉
\nThanks for sharing.
Obrigado Giba! 🙇
", "votes": 2, "replies": [] } ] }, { "id": 2531509, "author_name": "Dmitrii Khizbullin", "author_url": "", "post_date": "2023-11-20T09:18:37.973000", "content": "Congrats on the 1st place! I am curious: what tool did you use to make this beautiful block diagram in the writeup?
", "votes": 3, "replies": [ { "id": 2531680, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2023-11-20T12:24:44.493000", "content": "Thank you Dmitrii! Congratulation to you and your team as well!
\nI used a website called drawIo
Hi Eduardo,
\nI'm so glad to read your write-up. I'm a beginner, so it's very hard to understand it all.
Congratulations for you and your team (Thanh Hau Nguyen and Tomirol) for the amazing achievement.
\nParabéns!
Muito obrigado Marília 🙏
\nLet me know if there is anything I can help clarifying!
", "votes": 2, "replies": [] } ] }, { "id": 2606250, "author_name": "Kilaru Vasudeva", "author_url": "", "post_date": "2024-01-17T14:11:39.653000", "content": "Congrats on winning the competition! :) Thanks for implementation details. If you could, kindly share training details, ex: GPU used, GPU training time etc.
", "votes": 1, "replies": [ { "id": 2674796, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2024-02-29T14:45:53.720000", "content": "Sorry for the late response, was away recently. We used a RTX 4090 and it took around 1-2 hours (depends on the collection) to train the models after the optimisations we described in the thread.
", "votes": 1, "replies": [] } ] }, { "id": 2561136, "author_name": "Filtered", "author_url": "", "post_date": "2023-12-14T09:53:50.293000", "content": "congratulations!
", "votes": 1, "replies": [] }, { "id": 2534259, "author_name": "belgraviton", "author_url": "", "post_date": "2023-11-22T14:07:37.950000", "content": "Great solution. Architecture ideas are quite amazing. Learning embeddings for configs' features is what I was missing :)
", "votes": 1, "replies": [] }, { "id": 2533774, "author_name": "hyohyolim", "author_url": "", "post_date": "2023-11-22T08:10:40.150000", "content": "Congratulations, first place! What tool did you use to create the graph in the last image?
", "votes": 1, "replies": [ { "id": 2533970, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2023-11-22T10:15:30.680000", "content": "Thank you! I use a tool called drawIo
", "votes": 1, "replies": [] } ] }, { "id": 2533350, "author_name": "Md. Amanat Ullah", "author_url": "", "post_date": "2023-11-21T20:09:01.367000", "content": "congratulations.
", "votes": 1, "replies": [] }, { "id": 2532973, "author_name": "Suma Mallapragada", "author_url": "", "post_date": "2023-11-21T13:49:32.747000", "content": "Congratulations and thanks for sharing! The solution helps me capture all the details.
", "votes": 1, "replies": [] }, { "id": 2532336, "author_name": "Min-Hsien Weng", "author_url": "", "post_date": "2023-11-21T01:15:46.477000", "content": "Congrats for 1st place. Does an ensemble model use the same weight for each model?
", "votes": 1, "replies": [ { "id": 2532736, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2023-11-21T09:46:59.803000", "content": "Thank you! For the ensemble, we train the same model architecture many (5 - 10 usually) times with different random seeds (main impact is in the model's weights initialisation). Then, we just load each model/trained weight individually to make the prediction and average them up
", "votes": 1, "replies": [ { "id": 2533377, "author_name": "Min-Hsien Weng", "author_url": "", "post_date": "2023-11-21T20:51:25.077000", "content": "Using different weights could sometimes improve accuracy. In another competition, I used the Optuna library to experiment with different weight combinations and found the optimal ones, leading to further accuracy gains. Congratulations again on your excellent solutions!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2532324, "author_name": "Indra Sonowal", "author_url": "", "post_date": "2023-11-21T00:34:13.307000", "content": "Congrats brother on achieving 1st place.
", "votes": 1, "replies": [] }, { "id": 2532248, "author_name": "Haohan Tsao", "author_url": "", "post_date": "2023-11-20T21:28:00.927000", "content": "Thanks for sharing, congrats for winning
", "votes": 1, "replies": [] }, { "id": 2531240, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-11-20T03:53:28.827000", "content": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with flowcharts.
", "votes": 1, "replies": [] }, { "id": 2531191, "author_name": "yy1165", "author_url": "", "post_date": "2023-11-20T02:28:38.813000", "content": "Congratulations for your team and learn lots from your solution!
", "votes": 1, "replies": [] }, { "id": 2533324, "author_name": "Semih Günak", "author_url": "", "post_date": "2023-11-21T19:25:23.300000", "content": "Great competition solution! Your approach, especially using techniques like graph pruning and deduplication in data preprocessing, is quite interesting for achieving high performance. Additionally, your optimization of network architecture and attention mechanisms is very impressive. The results you obtained with ensembling are noteworthy. Congratulations! 🚀
", "votes": 2, "replies": [] }, { "id": 2533525, "author_name": "", "author_url": "", "post_date": "2023-11-22T02:21:27.817000", "content": "", "votes": 1, "replies": [] }, { "id": 2532621, "author_name": "", "author_url": "", "post_date": "2023-11-21T07:30:48.087000", "content": "", "votes": 1, "replies": [] }, { "id": 2532982, "author_name": "Zhuravlev Viktor", "author_url": "", "post_date": "2023-11-21T13:59:45.577000", "content": "Good job! Thanks for the sharing!
", "votes": 1, "replies": [] }, { "id": 2532390, "author_name": "Phil", "author_url": "", "post_date": "2023-11-21T03:07:21.020000", "content": "congrats! Thanks for sharing
", "votes": 1, "replies": [] }, { "id": 2531335, "author_name": "Haoqin Hong", "author_url": "", "post_date": "2023-11-20T06:05:09.580000", "content": "Thanks for sharing！
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2530875": "First of all, thanks Kaggle and Google for hosting this great competition. Also, I'd like to thanks my teammates and friends @tomirol and @thanhhau097a, which made this journey even more special.\n\n# Context\n- Business context: https://www.kaggle.com/competitions/predict-ai-model-runtime/overview\n- Data context: https://www.kaggle.com/competitions/predict-ai-model-runtime/data\n\n# TLDR:\n- We pruned and compressed the layout graphs in order to increase the efficiency of our experiments\n- We removed duplicated configs for layout\n- We changed the `node_feat` to use -1 padding instead of 0\n- We used the provided `train`, `val`, `test` splits since we found good correlation with LB\n- Data preprocessing:\n - StandardScaler for `node_feat[:134]`\n - Shared learned embedding (4 channels) for `node_feat[134:]` and `node_config_feat`\n - Features are concatenated before first linear\n- Models with the following architecture:\n - Linear on the input features\n - 2x graph conv with attention blocks \n - `InstanceNorm` -> `SAGEConv` -> `SelfChannelAttetion` -> `CrossConfigAttetion` -> `+residual` -> `GELU`\n - Global (graph) mean pooling\n - Linear logit layer\n- `PairwiseHingeLoss` function\n\n# Data preparation\nWe joined the competition with a bit less than a month to finish, so one of the first problems we tried to tackle was the low efficiency in our training jobs.\n\n## Graph pruning\nFor layout, we noticed that only `Convolution`, `Dot` and `Reshape` were configurable nodes. Also, in most cases, the majority of nodes would be identical across the config set. Thus, we opted for a very simple pruning strategy where, for each graph, we would only keep the nodes that were either configurable models themselves or were connected to a configurable node, i.e., input or output to a configurable node. By doing this, we would transform a single big graph into multiple (possibly disconnected) sub-graphs, which was not a problem since the network has a global graph pooling layer in the end that fuses the sub-graphs information. This simple trick reduced 4 times the vRAM usage and sped up training by a factor of 5 in some cases.\n\n## Deduplication\nMost of the configuration sets for layout contain a lot of duplication. However, the runtime for the duplicated configs can vary quite a bit and make training less stable. Thus, we opted for removing all the duplicated configs for layout.\n\n## Compression\nEven with pruning and de-duplication, the RAM usage to load all configs to memory for NLP collection was super high. We circumvent that issue by compressing `node_config_feat` beforehand and only decompressing it on-the-fly in the dataloader after config sampling. This enabled us to load all data to memory at the beginning of training, which reduced IO/CPU bottlenecks considerably and allowed us to train faster.\n\nThe idea behind the compression is that each `node_config_feat` 6-dim vector (input, output and kernel) can only have 7 possible values (-1, 0, 1, 2, 3, 4, 5) and, thus, can be represented by a single integer in base-7 (from 0 to 7^6).\n\n## Changing pad value in `node_feat`\nWe noted that the features in `node_feat` were 0 padded. Whilst this is not a problem for most features, for others like `layout_minor_to_major_*` this can be ambiguous since 0 is a valid axis index. Also, the `node_config_feat` are -1 padded, which makes it incompatible with `layout_minor_to_major_*` from `node_feat`. With that in mind, we re-generated `node_feat` with -1 padded and this allowed us to use a single embedding matrix for both `node_feat[134:]` and `node_config_feat`.\n\n# Data preprocessing\nFor layout, we split `node_feat` into `node_feat[:134]` and `node_feat[134:]` (`layout_minor_to_major_*`). The former was simply normalised using a `StandardScaler`, while the latter, along with `node_config_feat`, was fed into a learned embedding matrix (4 channels). We found that the normalisation is essential here since `node_feat` has features like `*_sum` and `*_product` that can be very high and, consequently, disrupt the optimisation.\n\nFor `node_opcode`, we also used a separate embedding layer with 16 channels. The input to the network is the concatenation of all features aforementioned and, for each graph, we sample on-the-fly 64 (default) or 128 (random) configs to form the input batch. For tile, on the other hand, we opt to use late fusion to integrate `config_feat` into the network.\n\n# Network architecture\nOur network architecture was quite simple. We first feed the input features to a Linear block to map it to a 256d embedding vector followed by 2x Conv blocks, global graph mean pooling and a final linear layer.\n\nAs for the graph convolutional layer itself, we tried many types but none was better than `SAGEConv`. In particular, I had good experience with GAT variants in the past but none worked well in this competition. If I were to guess the reason, I'd say that in the other application the graph itself was quite noisy, so attention helped to \"ignore\" the connections that were not meaningful. However, for TPU graphs, all connections are \"real\" and important so graph attention was not that helpful. Nonetheless, we found two other types of attention that were useful: self-channel attention and cross-config attention.\n\n## Self-Channel Attention\nWe borrowed the idea from Squeeze-and-Excitation to create a channel-wise attention layer. We first apply a Linear layer to bottleneck the channel dimensions (8x reduction) followed by `ReLU`. Then, we applied a second linear layer to increase the channels again to the original value followed by sigmoid. We finish by applying element wise multiplication on the obtained feature map and the original input.\n\nThe idea behind this is to capture the correlations between channels and use it to suppress less useful ones while enhancing others. \n\n## Cross-Config Attention\nAnother dimension that we can exploit attention is the batch plane (cross-configs). We designed a very simple block that allows the model to explicitly \"compare\" each config against the others throughout the network. We found this to be much better than letting the model infer for each config individually and only compare them implicitly via the loss function (`PairwiseHingeLoss`). The attention code is as follows:\n```\nclass CrossConfigAttention(nn.Module):\n def __init__(self):\n super().__init__()\n self.temperature = nn.Parameter(torch.tensor(0.5))\n\n def forward(self, x):\n # x of shape (nb_configs, nb_nodes, nb_features)\n scores = (x / self.temperature).softmax(dim=0)\n x = x * scores\n return x\n```\n\nBy applying this simple layer after the self-channel attention at every block of the network, it gave us a huge boost for default collections. For inference, we simply use a reasonably large batch size of 128. However, since the prediction depends on the batch, we can leverage it further by applying TTA to generate `N` (10) permutations of the configs and average the result after sorting it back to the original order.\n\n## Linear/Conv blocks design\nTo create our Linear/Conv blocks we followed the good practices in computer vision. We start by using `InstanceNorm` to normalise the input feature map, followed by `Linear`/`SAGEConv` layer, `SelfChannelAttetion` and `CrossConfigAttetion` (we concat the output with its input to preserve the individuality of each sample). Then, we sum the residual connection and finish with `GELU` and dropout.\n\n# Ensembling\nOur best single model prediction scored 0.714 (0.748) on private (public) LB. However, since the number of test samples is quite low for some collections, we opted to use ensembles to improve the results and prevent shaking up. Our best result was 0.736 (0.757) LB by using the simple average of 5-10 models for each collection but we sadly didn't select this sub.\n\n# Things that didn't work\n- Train together on both random and default data\n- 2nd level stacking on top of model's embeddings/predictions (worked well locally but not that well on LB)\n- Pseudo labelling test set\n- Finetunning models for specific graphs in test set (worked well locally but not that well on LB)\n- Different ways to sample the configs, e.g., annealing the runtime spacing between configs\n- Other loss functions\n- Gradient accumulation and training with more than one graph at a time\n\n# Sources\n- Our code can be found [in this GitHub repo](https://github.com/thanhhau097/google_fast_or_slow/tree/main)\n- Our work was based/forked on @werus23 [public kernel](https://www.kaggle.com/code/werus23/tile-xla-end-to-end-train-infer) (thank you!)\n- The modified layout dataset with -1 padding can be found [here](https://www.kaggle.com/datasets/tomirol/layout-npz-padding)\n\nI made a quick diagram of our network, it's a bit crap but should give a good idea 😅\n\n", "2537036": "Congrats @arc144 and team! Nice to see a brazillian winner 😉\nThanks for sharing.", "2531509": "Congrats on the 1st place! I am curious: what tool did you use to make this beautiful block diagram in the writeup?", "2530966": "Hi Eduardo,\nI'm so glad to read your write-up. I'm a beginner, so it's very hard to understand it all.\n\nCongratulations for you and your team (Thanh Hau Nguyen and Tomirol) for the amazing achievement.\nParabéns!", "2606250": "Congrats on winning the competition! :) Thanks for implementation details. If you could, kindly share training details, ex: GPU used, GPU training time etc.", "2561136": "congratulations!", "2534259": "Great solution. Architecture ideas are quite amazing. Learning embeddings for configs' features is what I was missing :)", "2533774": "Congratulations, first place! What tool did you use to create the graph in the last image?", "2533350": "congratulations. ", "2532973": "Congratulations and thanks for sharing! The solution helps me capture all the details.", "2532336": "Congrats for 1st place. Does an ensemble model use the same weight for each model?", "2532324": "Congrats brother on achieving 1st place.", "2532248": "Thanks for sharing, congrats for winning", "2531240": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with flowcharts. ", "2531191": "Congratulations for your team and learn lots from your solution!", "2533324": "Great competition solution! Your approach, especially using techniques like graph pruning and deduplication in data preprocessing, is quite interesting for achieving high performance. Additionally, your optimization of network architecture and attention mechanisms is very impressive. The results you obtained with ensembling are noteworthy. Congratulations! 🚀", "2533525": "", "2532621": "", "2532982": "Good job! Thanks for the sharing!", "2532390": "congrats! Thanks for sharing", "2531335": "Thanks for sharing！" } }