![\"Overview\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16611574%2F4548b69d2c2e5cbc11e502a30fece82c%2FPresentation1.jpg?generation=1700528423839754&alt=media\")
Thank you to Kaggle and Google for hosting this great competition. I learned a lot about training graph data models, especially how to deal with memory efficiency and optimize performance.
\nI started by training a modified BERT model by @KSMCG90 using cross-validation and the Lightning AI library. But this model was too demanding on memory for the layout dataset. So, I switched to the GNN (Graph Neural Network) model by @GUSTHEMA for the layout task. Since the tile and layout datasets didn't show any significant correlation, I built two separate models: one BERT model for the tile dataset and one GNN model for the layout dataset.
\nThe solution notebook focuses on training two models and run them efficiently on limited computing resources (4-core CPUs).
\nBusiness context: https://www.kaggle.com/competitions/predict-ai-model-runtime/
\nData context: https://www.kaggle.com/competitions/predict-ai-model-runtime/data
The solution comprised two models: a GNN model for the layout dataset and a modified BERT model for the tile dataset. The inputs were the graph structures generated by an AI model. In this graph, nodes represented tensor operations (e.g., add, max, reshape, convolution) and edges represented tensors.
\n
I chose the residual model for layout configurations because it can measure the difference between predicted and actual runtime time of each layout configuration. The model uses TensorFlow's graph neural network library (TF-GNN) to train efficiently on TPUs. I trained the layout model on TPUs and saved the best model as a pretrained model for prediction. In this notebook, I built four separate models, one for each type of layout dataset: nlp-default, nlp-random, xla-default, and xla-random.
The layout model encodes the layout graph into graph embeddings (node sets, edge sets, and context) and includes 5 hidden layers and a global pooling layer. The model is then trained on 10 epochs, with ListMLE ranking loss as the loss function and ordered pair accuracy (OPA) as the performance metric. The optimizer is AdamW with a learning rate of $10^{-3}$.
Things to do: The model can be improved by adjusting the learning rate with cross validation and by pruning irrelevant context to improve memory efficiency. Adding positional encodings may also help. PyG library could be used to build and train GNN model. x
Similar to layout dataset, the tile data is encoded into graph embeddings that include 9 data columns (node_feat, node_opcode, edge_index, config_runtime, config_runtime_normalizers, edges_adjecency, node_config_feat, node_config_ids and selected_idxs).
The tile model is based on the BERT model and uses Graph Attention Networks (GANs) to give more attention to important parts of the data. This helps improve accuracy by focusing on the most relevant information. The tile model uses a GraphEncoder to encode the tile configuration into graph embeddings and applies a special mask to all layers and heads. This mask is based on the connections between different parts of the graph. The model's loss function compares its predictions to different permutations of the actual results.
\nLastly, the model was wrapped with Lightning AI library to perform cross-validation using five folds of training and validation datasets. The best model from each fold was saved and then used to further train and predict the testing dataset.
\nI tried to make the BERT model work with the layout dataset, but it needed too much memory (>300GB). So, I used the TF-GNN-based residual model that uses less memory (<30GB). However, training the TF-GNN model is time-consuming. For instance, training one epoch on the 'xla-random' dataset required approximately one hour on CPUs. Training for 10 epochs would exceed the competition's time constraints (>10 hours). Despite utilizing GPUs to accelerate the training process, the training time has little/no speedups.
\nTo speed up training, I used a checkpointing technique to save the model's weights after each epoch. This allowed me to resume training, saving time. The best model was saved and used to predict the test datasets.
\nThe infer process for the tile dataset is time-consuming, for example, the model requires approximately 1.81 hours (6,532 seconds) to predict the testing dataset of 844 tile configurations. This is because the infer loop involves running each fold model separately and then merging the results from the five models to get the final output. I tried to speed up the infer loop by using five separate threads to run the inference for each fold's model concurrently. Using a thread pool of three threads, the total infer time is reduced to 58 minutes (3,448 seconds), achieving approximately a 2x speedup on CPUs.
\nThis experiment demonstrates that parallelism techniques can be effectively applied to machine learning models, enabling them to run on standard hardware configurations. This eliminates the need for specialized hardware, making machine learning more accessible to students and researchers with limited computing resources.
\nThe final solution was inspired by the starter notebook by @GUSTHEMA and BERT-like GAT Tile wLayout Dataset by @KSMCG90
\nThank you to Kaggle and Google for hosting this great competition. I learned a lot about training graph data models, especially how to deal with memory efficiency and optimize performance.
\nI started by training a modified BERT model by @KSMCG90 using cross-validation and the Lightning AI library. But this model was too demanding on memory for the layout dataset. So, I switched to the GNN (Graph Neural Network) model by @GUSTHEMA for the layout task. Since the tile and layout datasets didn't show any significant correlation, I built two separate models: one BERT model for the tile dataset and one GNN model for the layout dataset.
\nThe solution notebook focuses on training two models and run them efficiently on limited computing resources (4-core CPUs).
\nBusiness context: https://www.kaggle.com/competitions/predict-ai-model-runtime/
\nData context: https://www.kaggle.com/competitions/predict-ai-model-runtime/data
The solution comprised two models: a GNN model for the layout dataset and a modified BERT model for the tile dataset. The inputs were the graph structures generated by an AI model. In this graph, nodes represented tensor operations (e.g., add, max, reshape, convolution) and edges represented tensors.
\n
I chose the residual model for layout configurations because it can measure the difference between predicted and actual runtime time of each layout configuration. The model uses TensorFlow's graph neural network library (TF-GNN) to train efficiently on TPUs. I trained the layout model on TPUs and saved the best model as a pretrained model for prediction. In this notebook, I built four separate models, one for each type of layout dataset: nlp-default, nlp-random, xla-default, and xla-random.
The layout model encodes the layout graph into graph embeddings (node sets, edge sets, and context) and includes 5 hidden layers and a global pooling layer. The model is then trained on 10 epochs, with ListMLE ranking loss as the loss function and ordered pair accuracy (OPA) as the performance metric. The optimizer is AdamW with a learning rate of $10^{-3}$.
Things to do: The model can be improved by adjusting the learning rate with cross validation and by pruning irrelevant context to improve memory efficiency. Adding positional encodings may also help. PyG library could be used to build and train GNN model. x
Similar to layout dataset, the tile data is encoded into graph embeddings that include 9 data columns (node_feat, node_opcode, edge_index, config_runtime, config_runtime_normalizers, edges_adjecency, node_config_feat, node_config_ids and selected_idxs).
The tile model is based on the BERT model and uses Graph Attention Networks (GANs) to give more attention to important parts of the data. This helps improve accuracy by focusing on the most relevant information. The tile model uses a GraphEncoder to encode the tile configuration into graph embeddings and applies a special mask to all layers and heads. This mask is based on the connections between different parts of the graph. The model's loss function compares its predictions to different permutations of the actual results.
\nLastly, the model was wrapped with Lightning AI library to perform cross-validation using five folds of training and validation datasets. The best model from each fold was saved and then used to further train and predict the testing dataset.
\nI tried to make the BERT model work with the layout dataset, but it needed too much memory (>300GB). So, I used the TF-GNN-based residual model that uses less memory (<30GB). However, training the TF-GNN model is time-consuming. For instance, training one epoch on the 'xla-random' dataset required approximately one hour on CPUs. Training for 10 epochs would exceed the competition's time constraints (>10 hours). Despite utilizing GPUs to accelerate the training process, the training time has little/no speedups.
\nTo speed up training, I used a checkpointing technique to save the model's weights after each epoch. This allowed me to resume training, saving time. The best model was saved and used to predict the test datasets.
\nThe infer process for the tile dataset is time-consuming, for example, the model requires approximately 1.81 hours (6,532 seconds) to predict the testing dataset of 844 tile configurations. This is because the infer loop involves running each fold model separately and then merging the results from the five models to get the final output. I tried to speed up the infer loop by using five separate threads to run the inference for each fold's model concurrently. Using a thread pool of three threads, the total infer time is reduced to 58 minutes (3,448 seconds), achieving approximately a 2x speedup on CPUs.
\nThis experiment demonstrates that parallelism techniques can be effectively applied to machine learning models, enabling them to run on standard hardware configurations. This eliminates the need for specialized hardware, making machine learning more accessible to students and researchers with limited computing resources.
\nThe final solution was inspired by the starter notebook by @GUSTHEMA and BERT-like GAT Tile wLayout Dataset by @KSMCG90
\nThank you for sharing your write-up with Modified BERT GAN and infer loop.
", "rawMarkdown": "Thank you for sharing your write-up with Modified BERT GAN and infer loop.", "votes": null }, { "id": "2533380", "postDate": "11/21/2023 20:53:14", "content": "Thanks for liking my solution. Please feel free to reach out if you have any questions.
", "rawMarkdown": "Thanks for liking my solution. Please feel free to reach out if you have any questions.", "votes": null }, { "id": "2533445", "postDate": "11/21/2023 23:57:57", "content": "I'm just a beginner, therefore it's hard to make questions since all is very new to me. I hope that other experienced kagglers could read it and learn from your approach. Thank you M. Weng.
", "rawMarkdown": "I'm just a beginner, therefore it's hard to make questions since all is very new to me. I hope that other experienced kagglers could read it and learn from your approach. Thank you M. Weng.", "votes": null }, { "id": "2533459", "postDate": "11/22/2023 00:32:18", "content": "I am still learning about GNNs, but I am interested to explore their potential.
\nKaggle provides an excellent platform for learning and experimenting with novel ML models. Most importantly, enjoy the competition and have fun! 😁
", "rawMarkdown": "I am still learning about GNNs, but I am interested to explore their potential.\n\nKaggle provides an excellent platform for learning and experimenting with novel ML models. Most importantly, enjoy the competition and have fun! 😁", "votes": null }, { "id": "2533473", "postDate": "11/22/2023 00:44:17", "content": "The more you grow (on Kaggle) cause you're already experienced, other users will start to pay attention on your work. Unfortunately, our community is young and shallow.
\nSo, they don't read much what contributors/experts are writting. Worst for them, since they lose a precious opportunity to learn from the best professionals. Besides, those that are working, don't have enough time to spend many hours kaggling : )
", "rawMarkdown": "The more you grow (on Kaggle) cause you're already experienced, other users will start to pay attention on your work. Unfortunately, our community is young and shallow. \n\nSo, they don't read much what contributors/experts are writting. Worst for them, since they lose a precious opportunity to learn from the best professionals. Besides, those that are working, don't have enough time to spend many hours kaggling : )", "votes": null }, { "id": "2533502", "postDate": "11/22/2023 01:39:18", "content": "I wrote the solution partly for myself. If I encounter similar problems, I can leverage the knowledge and experience I gained from this competition.
\nThank you for your encouragement! 😀
", "rawMarkdown": "I wrote the solution partly for myself. If I encounter similar problems, I can leverage the knowledge and experience I gained from this competition.\n\nThank you for your encouragement! 😀", "votes": null }, { "id": "2533511", "postDate": "11/22/2023 01:54:22", "content": "\"Better to write for ourselves (and have few audience), than to write for the others (and have No self)\" Cyril Connoly quote.
\nBesides, since everything is on the cloud, anyone/anywhere could read it on the future. Even yourself, can compare how do you think and apply those methods and metrics today and what you could deliver in the future.
", "rawMarkdown": "\"Better to write for ourselves (and have few audience), than to write for the others (and have No self)\" Cyril Connoly quote.\n\nBesides, since everything is on the cloud, anyone/anywhere could read it on the future. Even yourself, can compare how do you think and apply those methods and metrics today and what you could deliver in the future.", "votes": null } ], "comments": [ { "id": 2533176, "author_name": "mpwolke", "author_url": "", "post_date": "11/21/2023 16:37:00", "content": "Thank you for sharing your write-up with Modified BERT GAN and infer loop.
", "votes": null, "replies": [ { "id": 2533380, "author_name": "minhsienweng", "author_url": "", "post_date": "11/21/2023 20:53:14", "content": "Thanks for liking my solution. Please feel free to reach out if you have any questions.
", "votes": null, "replies": [ { "id": 2533445, "author_name": "mpwolke", "author_url": "", "post_date": "11/21/2023 23:57:57", "content": "I'm just a beginner, therefore it's hard to make questions since all is very new to me. I hope that other experienced kagglers could read it and learn from your approach. Thank you M. Weng.
", "votes": null, "replies": [ { "id": 2533459, "author_name": "minhsienweng", "author_url": "", "post_date": "11/22/2023 00:32:18", "content": "I am still learning about GNNs, but I am interested to explore their potential.
\nKaggle provides an excellent platform for learning and experimenting with novel ML models. Most importantly, enjoy the competition and have fun! 😁
", "votes": null, "replies": [ { "id": 2533473, "author_name": "mpwolke", "author_url": "", "post_date": "11/22/2023 00:44:17", "content": "The more you grow (on Kaggle) cause you're already experienced, other users will start to pay attention on your work. Unfortunately, our community is young and shallow.
\nSo, they don't read much what contributors/experts are writting. Worst for them, since they lose a precious opportunity to learn from the best professionals. Besides, those that are working, don't have enough time to spend many hours kaggling : )
", "votes": null, "replies": [ { "id": 2533502, "author_name": "minhsienweng", "author_url": "", "post_date": "11/22/2023 01:39:18", "content": "I wrote the solution partly for myself. If I encounter similar problems, I can leverage the knowledge and experience I gained from this competition.
\nThank you for your encouragement! 😀
", "votes": null, "replies": [ { "id": 2533511, "author_name": "mpwolke", "author_url": "", "post_date": "11/22/2023 01:54:22", "content": "\"Better to write for ourselves (and have few audience), than to write for the others (and have No self)\" Cyril Connoly quote.
\nBesides, since everything is on the cloud, anyone/anywhere could read it on the future. Even yourself, can compare how do you think and apply those methods and metrics today and what you could deliver in the future.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2532327": "Thank you to Kaggle and Google for hosting this great competition. I learned a lot about training graph data models, especially how to deal with memory efficiency and optimize performance. \n\nI started by training a modified BERT model by @KSMCG90 using cross-validation and the [Lightning AI](https://lightning.ai/) library. But this model was too demanding on memory for the layout dataset. So, I switched to the GNN (Graph Neural Network) model by @GUSTHEMA for the layout task. Since the tile and layout datasets didn't show any significant correlation, I built two separate models: one BERT model for the tile dataset and one GNN model for the layout dataset. \n\nThe solution [notebook](https://www.kaggle.com/code/minhsienweng/bertlike-tile-model-gnn-layout-model) focuses on training two models and run them efficiently on limited computing resources (4-core CPUs).\n\n# Context \n\nBusiness context: https://www.kaggle.com/competitions/predict-ai-model-runtime/\nData context: https://www.kaggle.com/competitions/predict-ai-model-runtime/data\n\n# Overview of the Approach\nThe solution comprised two models: a GNN model for the layout dataset and a modified BERT model for the tile dataset. The inputs were the graph structures generated by an AI model. In this graph, nodes represented tensor operations (e.g., add, max, reshape, convolution) and edges represented tensors.\n\n\n \n## TF-GNN Residual Layout model\nI chose the residual model for layout configurations because it can measure the difference between predicted and actual runtime time of each layout configuration. The model uses TensorFlow's graph neural network library ([TF-GNN](https://github.com/tensorflow/gnn)) to train efficiently on TPUs. I trained the layout model on TPUs and saved the best model as a pretrained model for prediction. In this notebook, I built four separate models, one for each type of layout dataset: `nlp-default`, `nlp-random`, `xla-default`, and `xla-random`.\n\nThe layout model encodes the layout graph into graph embeddings (`node sets`, `edge sets`, and `context`) and includes 5 hidden layers and a global pooling layer. The model is then trained on 10 epochs, with `ListMLE` ranking loss as the loss function and ordered pair accuracy (OPA) as the performance metric. The optimizer is AdamW with a learning rate of $10^{-3}$. \n\n*Things to do*: The model can be improved by adjusting the learning rate with cross validation and by pruning irrelevant `context` to improve memory efficiency. Adding [positional encodings](https://github.com/vijaydwivedi75/gnn-lspe) may also help. [PyG](https://pyg.org/) library could be used to build and train GNN model. x\n\n## Modified BERT GAN Tile model\nSimilar to layout dataset, the tile data is encoded into graph embeddings that include 9 data columns (`node_feat`, `node_opcode`, `edge_index`, `config_runtime`, `config_runtime_normalizers`, `edges_adjecency`, `node_config_feat`, `node_config_ids` and `selected_idxs`).\n\nThe tile model is based on the BERT model and uses Graph Attention Networks ([GANs](https://arxiv.org/abs/1710.10903)) to give more attention to important parts of the data. This helps improve accuracy by focusing on the most relevant information. The tile model uses a GraphEncoder to encode the tile configuration into graph embeddings and applies a special mask to all layers and heads. This mask is based on the connections between different parts of the graph. The model's loss function compares its predictions to different permutations of the actual results.\n\nLastly, the model was wrapped with [Lightning AI](https://lightning.ai/) library to perform cross-validation using five folds of training and validation datasets. The best model from each fold was saved and then used to further train and predict the testing dataset.\n\n# Details of the submission\n\n## Training layout model with checkpoints\nI tried to make the BERT model work with the layout dataset, but it needed too much memory (>300GB). So, I used the TF-GNN-based residual model that uses less memory (<30GB). However, training the TF-GNN model is time-consuming. For instance, training one epoch on the 'xla-random' dataset required approximately one hour on CPUs. Training for 10 epochs would exceed the competition's time constraints (>10 hours). Despite utilizing GPUs to accelerate the training process, the training time has little/no speedups. \n\nTo speed up training, I used a checkpointing technique to save the model's weights after each epoch. This allowed me to resume training, saving time. The best model was saved and used to predict the test datasets.\n\n## Parallelising the infer loop of tile model\nThe infer process for the tile dataset is time-consuming, for example, the model requires approximately 1.81 hours (6,532 seconds) to predict the testing dataset of 844 tile configurations. This is because the infer loop involves running each fold model separately and then merging the results from the five models to get the final output. I tried to speed up the infer loop by using five separate threads to run the inference for each fold's model concurrently. Using a thread pool of three threads, the total infer time is reduced to 58 minutes (3,448 seconds), achieving approximately a 2x speedup on CPUs.\n\nThis experiment demonstrates that parallelism techniques can be effectively applied to machine learning models, enabling them to run on standard hardware configurations. This eliminates the need for specialized hardware, making machine learning more accessible to students and researchers with limited computing resources.\n\n\n# Sources\nThe final solution was inspired by the starter notebook by @GUSTHEMA and BERT-like GAT Tile wLayout Dataset by @KSMCG90\n- https://www.kaggle.com/code/gusthema/starter-notebook-fast-or-slow-with-tensorflow-gnn\n- https://www.kaggle.com/code/ksmcg90/bertlike-gat-tile-wlayout-dataset?scriptVersionId=148717592", "2533176": "Thank you for sharing your write-up with Modified BERT GAN and infer loop.", "2533380": "Thanks for liking my solution. Please feel free to reach out if you have any questions.", "2533445": "I'm just a beginner, therefore it's hard to make questions since all is very new to me. I hope that other experienced kagglers could read it and learn from your approach. Thank you M. Weng.", "2533459": "I am still learning about GNNs, but I am interested to explore their potential.\n\nKaggle provides an excellent platform for learning and experimenting with novel ML models. Most importantly, enjoy the competition and have fun! 😁", "2533473": "The more you grow (on Kaggle) cause you're already experienced, other users will start to pay attention on your work. Unfortunately, our community is young and shallow. \n\nSo, they don't read much what contributors/experts are writting. Worst for them, since they lose a precious opportunity to learn from the best professionals. Besides, those that are working, don't have enough time to spend many hours kaggling : )", "2533502": "I wrote the solution partly for myself. If I encounter similar problems, I can leverage the knowledge and experience I gained from this competition.\n\nThank you for your encouragement! 😀", "2533511": "\"Better to write for ourselves (and have few audience), than to write for the others (and have No self)\" Cyril Connoly quote.\n\nBesides, since everything is on the cloud, anyone/anywhere could read it on the future. Even yourself, can compare how do you think and apply those methods and metrics today and what you could deliver in the future." }, "source": "meta" }