Hey Guys,
\nI am a beginner and this competition seems advance to me. But it's interesting so I'll continue this and gain the knowledge along the way. Please share resources which are prerequisite to this competition. Also explain all the technical terminology. Share whatever the knowledge you have regarding this task.
\nThanks
", "messages": [ { "id": "2427233", "postDate": "09/07/2023 06:01:19", "content": "Hey Guys,
\nI am a beginner and this competition seems advance to me. But it's interesting so I'll continue this and gain the knowledge along the way. Please share resources which are prerequisite to this competition. Also explain all the technical terminology. Share whatever the knowledge you have regarding this task.
\nThanks
", "rawMarkdown": "Hey Guys,\n\nI am a beginner and this competition seems advance to me. But it's interesting so I'll continue this and gain the knowledge along the way. Please share resources which are prerequisite to this competition. Also explain all the technical terminology. Share whatever the knowledge you have regarding this task.\n\nThanks", "votes": null }, { "id": "2427244", "postDate": "09/07/2023 06:05:58", "content": "→ Each .npz file contains a graphs with “n” nodes and “m” edges.
\n→ Suppose we compile the graph with “c” different configurations.
\n→ XLA (Accelerated Linear Algebra): is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.
\nKeys:
\nGoal:
\nFor the tile collection, your job is to predict the indices of the best configurations (i.e., ones leading to the smallest d[\"config_runtime\"] / d[\"config_runtime_normalizers\"]).
Understanding Tiles:
\nTile Configuration:
\nFor tile configuration, let's first understand tiling. Imagine an operation that needs to be performed on a large matrix (or tensor). Doing it all at once might be inefficient. Instead, what we do is \"tile\" or break the matrix into smaller chunks (tiles) and perform the operation on these smaller chunks individually or simultaneously (depending on the hardware).
\nA tile configuration, therefore, determines the size of these tiles. In essence, how big each chunk or tile should be when we break up the tensor.
\nWhy is this important?
\nChoosing the right tile size can significantly impact performance. A tile size that's too small might cause overhead due to the frequent need to load and store data, while a tile size that's too large might not fit well in the cache memory, causing cache misses.
\nKey Point: The tile configuration allows Alice to specify the ideal tile size for each fused subgraph in the AI model. This can be crucial for ensuring the efficient use of memory and computational resources.
\nKeys:
\nGoal:
\nFinally, for the layout collections, your job is to predict the order of the indices from best-to-worse configurations (i.e., ones leading to the smallest d[\"config_runtime\"]). We do not have to use runtime normalizers for this task because the runtime variation at the entire program level is very small.
Understanding Layouts:
\nLayout Configuration:
\nWhen we talk about the layout configuration, we are essentially discussing the manner in which tensors are organized in memory. Think of a tensor as a multi-dimensional array (similar to matrices). For instance, for image data in deep learning, we might represent it as a tensor with the dimensions [Batch, Channels, Height, Width] or in shorthand, BCHW.
However, sometimes, due to architectural specificities or optimization purposes, we may want to rearrange this order. For example, a layout might rearrange the tensor dimensions as [Batch, Height, Width, Channels] or BHWC.
Why does this matter?
\nThe order in which these dimensions are stored can significantly affect the speed of tensor operations. Certain operations might be faster on a BCHW layout, while others on a BHWC. This is due to how these operations access memory and the underlying hardware's capability.
\nKey Point: A layout configuration lets Alice specify how the dimensions of a tensor are arranged in memory, which in turn, can optimize specific tensor operations.
\nRef: https://www.youtube.com/watch?v=BT2Cv-Tjq7Q
\nIR: Intermediate Representation of Programming Language
\nLLVM: Converts programming language into IR
\nMLIR: Multi-level intermediate representation
\n→ LLVM and MLIR are both valuable compiler infrastructure projects, but they have different scopes and focuses. LLVM excels in code generation and optimization for a wide range of target architectures, while MLIR offers flexibility and extensibility to represent and optimize code at various levels of abstraction, making it suitable for a broader set of use cases, including machine learning and domain-specific languages.
\nResource: https://www.youtube.com/playlist?list=PLlONLmJCfHTo9WYfsoQvwjsa5ZB6hjOG5
\nMachine Learning Compiler:
\nRef: https://www.youtube.com/watch?v=y7d7CWX_Is8
\nRef: https://www.youtube.com/playlist?list=PLoROMvodv4rPLKxIpqhjhPgdQy7imNkDn
", "rawMarkdown": "## Here is my notion page regarding this competition:\n\n\n→ Each .npz file contains a graphs with “n” nodes and “m” edges.\n\n→ Suppose we compile the graph with “c” different configurations.\n\n→ **XLA (Accelerated Linear Algebra)**: is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.\n\n---\n\n### Tiles:\n\n**Keys:**\n\n1. node_feat:\n 1. float - (n,140)\n2. node_opcode:\n 1. int32 vector - (n, )\n3. edge_index:\n 1. int32 - (m, 2)\n4. config_feat:\n 1. float - (c, 24)\n5. config_runtime:\n 1. int64 vector of length c\n6. config_runtime_normalizers:\n 1. int64 vector of length c\n\n**Goal:**\n\nFor the tile collection, your job is to predict the indices of the best configurations (i.e., ones leading to the smallest `d[\"config_runtime\"] / d[\"config_runtime_normalizers\"]`).\n\n**Understanding Tiles:**\n\nTile Configuration:\n\nFor tile configuration, let's first understand tiling. Imagine an operation that needs to be performed on a large matrix (or tensor). Doing it all at once might be inefficient. Instead, what we do is \"tile\" or break the matrix into smaller chunks (tiles) and perform the operation on these smaller chunks individually or simultaneously (depending on the hardware).\n\nA tile configuration, therefore, determines the size of these tiles. In essence, how big each chunk or tile should be when we break up the tensor.\n\nWhy is this important?\n\nChoosing the right tile size can significantly impact performance. A tile size that's too small might cause overhead due to the frequent need to load and store data, while a tile size that's too large might not fit well in the cache memory, causing cache misses.\n\nKey Point: The tile configuration allows Alice to specify the ideal tile size for each fused subgraph in the AI model. This can be crucial for ensuring the efficient use of memory and computational resources.\n\n---\n\n### Layout:\n\n**Keys:**\n\n1. node_feat:\n 1. same as above\n2. node_opcode:\n 1. same as above\n3. edge_index:\n 1. same as above\n4. node_config_ids:\n 1. int32 vector - (nc, )\n5. node_config_feat:\n 1. float32 - (c, nc, 18)\n6. config_runtime:\n 1. int32 vector - (c, )\n\n**Goal:**\n\nFinally, for the layout collections, your job is to predict the order of the indices from best-to-worse configurations (i.e., ones leading to the smallest `d[\"config_runtime\"]`). We do not have to use runtime normalizers for this task because the runtime variation at the entire program level is very small.\n\n**Understanding Layouts:**\n\nLayout Configuration:\n\nWhen we talk about the layout configuration, we are essentially discussing the manner in which tensors are organized in memory. Think of a tensor as a multi-dimensional array (similar to matrices). For instance, for image data in deep learning, we might represent it as a tensor with the dimensions `[Batch, Channels, Height, Width]` or in shorthand, **BCHW**.\n\nHowever, sometimes, due to architectural specificities or optimization purposes, we may want to rearrange this order. For example, a layout might rearrange the tensor dimensions as `[Batch, Height, Width, Channels]` or **BHWC**.\n\nWhy does this matter?\n\nThe order in which these dimensions are stored can significantly affect the speed of tensor operations. Certain operations might be faster on a BCHW layout, while others on a BHWC. This is due to how these operations access memory and the underlying hardware's capability.\n\nKey Point: A layout configuration lets Alice specify how the dimensions of a tensor are arranged in memory, which in turn, can optimize specific tensor operations.\n\n---\n\n### Learning\n\n1. Computational Graph Compilation\n2. XLA, HLO\n3. Competition Papers\n4. GraphSAGE\n5. Graph NN\n6. Domain Knowledge\n7. Data Understanding\n8. Related Videos: https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436629\n9. Research Paper 1: https://www.cs.ubc.ca/~kevinlb/papers/2015-IJCAI-EPMs.pdf\n10. Research Paper 2: https://arxiv.org/pdf/2308.13490v1.pdf\n11. Research Paper 3: https://sci-hub.se/https://ieeexplore.ieee.org/document/8924184\n12. Research Paper 4: https://arxiv.org/pdf/2008.01040.pdf\n13. Slides 1: https://mangpo.net/talks/2021-xla-autotuning-pact.pdf\n14. YT 1: https://www.youtube.com/watch?v=esD_zvAf49I\n15. YT 2: https://www.youtube.com/watch?v=vYQ37pBNAFI\n16. GNN with PyTorch: https://www.youtube.com/watch?v=-UjytpbqX4A\n\n---\n\n### IR | LLVM | MLIR\n\nRef: https://www.youtube.com/watch?v=BT2Cv-Tjq7Q\n\n**IR**: Intermediate Representation of Programming Language\n\n**LLVM**: Converts programming language into IR\n\n**MLIR**: Multi-level intermediate representation\n\n→ LLVM and MLIR are both valuable compiler infrastructure projects, but they have different scopes and focuses. LLVM excels in code generation and optimization for a wide range of target architectures, while MLIR offers flexibility and extensibility to represent and optimize code at various levels of abstraction, making it suitable for a broader set of use cases, including machine learning and domain-specific languages.\n\nResource: https://www.youtube.com/playlist?list=PLlONLmJCfHTo9WYfsoQvwjsa5ZB6hjOG5\n\n**Machine Learning Compiler:**\n\nRef: https://www.youtube.com/watch?v=y7d7CWX_Is8\n\n---\n\n### Graph Neural Networks\n\nRef: https://www.youtube.com/playlist?list=PLoROMvodv4rPLKxIpqhjhPgdQy7imNkDn", "votes": null }, { "id": "2427354", "postDate": "09/07/2023 07:15:40", "content": "Thanks for sharing your resources
", "rawMarkdown": "Thanks for sharing your resources", "votes": null }, { "id": "2440863", "postDate": "09/15/2023 19:35:27", "content": "Is XLA complier knowledge a must for this competition ?
", "rawMarkdown": "Is XLA complier knowledge a must for this competition ?", "votes": null }, { "id": "2443457", "postDate": "09/17/2023 18:06:45", "content": "Not at all.
", "rawMarkdown": "Not at all.", "votes": null } ], "comments": [ { "id": 2427244, "author_name": "hussamcheema", "author_url": "", "post_date": "09/07/2023 06:05:58", "content": "→ Each .npz file contains a graphs with “n” nodes and “m” edges.
\n→ Suppose we compile the graph with “c” different configurations.
\n→ XLA (Accelerated Linear Algebra): is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.
\nKeys:
\nGoal:
\nFor the tile collection, your job is to predict the indices of the best configurations (i.e., ones leading to the smallest d[\"config_runtime\"] / d[\"config_runtime_normalizers\"]).
Understanding Tiles:
\nTile Configuration:
\nFor tile configuration, let's first understand tiling. Imagine an operation that needs to be performed on a large matrix (or tensor). Doing it all at once might be inefficient. Instead, what we do is \"tile\" or break the matrix into smaller chunks (tiles) and perform the operation on these smaller chunks individually or simultaneously (depending on the hardware).
\nA tile configuration, therefore, determines the size of these tiles. In essence, how big each chunk or tile should be when we break up the tensor.
\nWhy is this important?
\nChoosing the right tile size can significantly impact performance. A tile size that's too small might cause overhead due to the frequent need to load and store data, while a tile size that's too large might not fit well in the cache memory, causing cache misses.
\nKey Point: The tile configuration allows Alice to specify the ideal tile size for each fused subgraph in the AI model. This can be crucial for ensuring the efficient use of memory and computational resources.
\nKeys:
\nGoal:
\nFinally, for the layout collections, your job is to predict the order of the indices from best-to-worse configurations (i.e., ones leading to the smallest d[\"config_runtime\"]). We do not have to use runtime normalizers for this task because the runtime variation at the entire program level is very small.
Understanding Layouts:
\nLayout Configuration:
\nWhen we talk about the layout configuration, we are essentially discussing the manner in which tensors are organized in memory. Think of a tensor as a multi-dimensional array (similar to matrices). For instance, for image data in deep learning, we might represent it as a tensor with the dimensions [Batch, Channels, Height, Width] or in shorthand, BCHW.
However, sometimes, due to architectural specificities or optimization purposes, we may want to rearrange this order. For example, a layout might rearrange the tensor dimensions as [Batch, Height, Width, Channels] or BHWC.
Why does this matter?
\nThe order in which these dimensions are stored can significantly affect the speed of tensor operations. Certain operations might be faster on a BCHW layout, while others on a BHWC. This is due to how these operations access memory and the underlying hardware's capability.
\nKey Point: A layout configuration lets Alice specify how the dimensions of a tensor are arranged in memory, which in turn, can optimize specific tensor operations.
\nRef: https://www.youtube.com/watch?v=BT2Cv-Tjq7Q
\nIR: Intermediate Representation of Programming Language
\nLLVM: Converts programming language into IR
\nMLIR: Multi-level intermediate representation
\n→ LLVM and MLIR are both valuable compiler infrastructure projects, but they have different scopes and focuses. LLVM excels in code generation and optimization for a wide range of target architectures, while MLIR offers flexibility and extensibility to represent and optimize code at various levels of abstraction, making it suitable for a broader set of use cases, including machine learning and domain-specific languages.
\nResource: https://www.youtube.com/playlist?list=PLlONLmJCfHTo9WYfsoQvwjsa5ZB6hjOG5
\nMachine Learning Compiler:
\nRef: https://www.youtube.com/watch?v=y7d7CWX_Is8
\nRef: https://www.youtube.com/playlist?list=PLoROMvodv4rPLKxIpqhjhPgdQy7imNkDn
", "votes": null, "replies": [] }, { "id": 2427354, "author_name": "faysalmiah1721758", "author_url": "", "post_date": "09/07/2023 07:15:40", "content": "Thanks for sharing your resources
", "votes": null, "replies": [] }, { "id": 2440863, "author_name": "riteshbhalerao", "author_url": "", "post_date": "09/15/2023 19:35:27", "content": "Is XLA complier knowledge a must for this competition ?
", "votes": null, "replies": [ { "id": 2443457, "author_name": "mangpophothilimthana", "author_url": "", "post_date": "09/17/2023 18:06:45", "content": "Not at all.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2427233": "Hey Guys,\n\nI am a beginner and this competition seems advance to me. But it's interesting so I'll continue this and gain the knowledge along the way. Please share resources which are prerequisite to this competition. Also explain all the technical terminology. Share whatever the knowledge you have regarding this task.\n\nThanks", "2427244": "## Here is my notion page regarding this competition:\n\n\n→ Each .npz file contains a graphs with “n” nodes and “m” edges.\n\n→ Suppose we compile the graph with “c” different configurations.\n\n→ **XLA (Accelerated Linear Algebra)**: is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.\n\n---\n\n### Tiles:\n\n**Keys:**\n\n1. node_feat:\n 1. float - (n,140)\n2. node_opcode:\n 1. int32 vector - (n, )\n3. edge_index:\n 1. int32 - (m, 2)\n4. config_feat:\n 1. float - (c, 24)\n5. config_runtime:\n 1. int64 vector of length c\n6. config_runtime_normalizers:\n 1. int64 vector of length c\n\n**Goal:**\n\nFor the tile collection, your job is to predict the indices of the best configurations (i.e., ones leading to the smallest `d[\"config_runtime\"] / d[\"config_runtime_normalizers\"]`).\n\n**Understanding Tiles:**\n\nTile Configuration:\n\nFor tile configuration, let's first understand tiling. Imagine an operation that needs to be performed on a large matrix (or tensor). Doing it all at once might be inefficient. Instead, what we do is \"tile\" or break the matrix into smaller chunks (tiles) and perform the operation on these smaller chunks individually or simultaneously (depending on the hardware).\n\nA tile configuration, therefore, determines the size of these tiles. In essence, how big each chunk or tile should be when we break up the tensor.\n\nWhy is this important?\n\nChoosing the right tile size can significantly impact performance. A tile size that's too small might cause overhead due to the frequent need to load and store data, while a tile size that's too large might not fit well in the cache memory, causing cache misses.\n\nKey Point: The tile configuration allows Alice to specify the ideal tile size for each fused subgraph in the AI model. This can be crucial for ensuring the efficient use of memory and computational resources.\n\n---\n\n### Layout:\n\n**Keys:**\n\n1. node_feat:\n 1. same as above\n2. node_opcode:\n 1. same as above\n3. edge_index:\n 1. same as above\n4. node_config_ids:\n 1. int32 vector - (nc, )\n5. node_config_feat:\n 1. float32 - (c, nc, 18)\n6. config_runtime:\n 1. int32 vector - (c, )\n\n**Goal:**\n\nFinally, for the layout collections, your job is to predict the order of the indices from best-to-worse configurations (i.e., ones leading to the smallest `d[\"config_runtime\"]`). We do not have to use runtime normalizers for this task because the runtime variation at the entire program level is very small.\n\n**Understanding Layouts:**\n\nLayout Configuration:\n\nWhen we talk about the layout configuration, we are essentially discussing the manner in which tensors are organized in memory. Think of a tensor as a multi-dimensional array (similar to matrices). For instance, for image data in deep learning, we might represent it as a tensor with the dimensions `[Batch, Channels, Height, Width]` or in shorthand, **BCHW**.\n\nHowever, sometimes, due to architectural specificities or optimization purposes, we may want to rearrange this order. For example, a layout might rearrange the tensor dimensions as `[Batch, Height, Width, Channels]` or **BHWC**.\n\nWhy does this matter?\n\nThe order in which these dimensions are stored can significantly affect the speed of tensor operations. Certain operations might be faster on a BCHW layout, while others on a BHWC. This is due to how these operations access memory and the underlying hardware's capability.\n\nKey Point: A layout configuration lets Alice specify how the dimensions of a tensor are arranged in memory, which in turn, can optimize specific tensor operations.\n\n---\n\n### Learning\n\n1. Computational Graph Compilation\n2. XLA, HLO\n3. Competition Papers\n4. GraphSAGE\n5. Graph NN\n6. Domain Knowledge\n7. Data Understanding\n8. Related Videos: https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436629\n9. Research Paper 1: https://www.cs.ubc.ca/~kevinlb/papers/2015-IJCAI-EPMs.pdf\n10. Research Paper 2: https://arxiv.org/pdf/2308.13490v1.pdf\n11. Research Paper 3: https://sci-hub.se/https://ieeexplore.ieee.org/document/8924184\n12. Research Paper 4: https://arxiv.org/pdf/2008.01040.pdf\n13. Slides 1: https://mangpo.net/talks/2021-xla-autotuning-pact.pdf\n14. YT 1: https://www.youtube.com/watch?v=esD_zvAf49I\n15. YT 2: https://www.youtube.com/watch?v=vYQ37pBNAFI\n16. GNN with PyTorch: https://www.youtube.com/watch?v=-UjytpbqX4A\n\n---\n\n### IR | LLVM | MLIR\n\nRef: https://www.youtube.com/watch?v=BT2Cv-Tjq7Q\n\n**IR**: Intermediate Representation of Programming Language\n\n**LLVM**: Converts programming language into IR\n\n**MLIR**: Multi-level intermediate representation\n\n→ LLVM and MLIR are both valuable compiler infrastructure projects, but they have different scopes and focuses. LLVM excels in code generation and optimization for a wide range of target architectures, while MLIR offers flexibility and extensibility to represent and optimize code at various levels of abstraction, making it suitable for a broader set of use cases, including machine learning and domain-specific languages.\n\nResource: https://www.youtube.com/playlist?list=PLlONLmJCfHTo9WYfsoQvwjsa5ZB6hjOG5\n\n**Machine Learning Compiler:**\n\nRef: https://www.youtube.com/watch?v=y7d7CWX_Is8\n\n---\n\n### Graph Neural Networks\n\nRef: https://www.youtube.com/playlist?list=PLoROMvodv4rPLKxIpqhjhPgdQy7imNkDn", "2427354": "Thanks for sharing your resources", "2440863": "Is XLA complier knowledge a must for this competition ?", "2443457": "Not at all." }, "source": "meta" }