{ "id": 443265, "title": "Understanding the \"layout propagation\" technique", "url": "/competitions/predict-ai-model-runtime/discussion/443265", "author_name": "Shekhovtsov Aleksandr", "post_date": "2023-09-26T09:18:22.232000", "votes": 8, "comment_count": 11, "views": 0, "content": "

I have question about layout configuration. In paper TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs the authors wrote \"The autotuner tunes the input-output layouts of the most layout-performance-critical nodes and propagates layouts from these nodes to others.\" And I heard the same in some videos with @mangpophothilimthana
\nDoes anyone understand how exactly \"layout propagation\" works?
\nEach (non-configurable) node has 6 features related to layout. Is it a layout for an input or output tensor?
\nAnd after we change the layout configurations of the input and output tensors for some configurable node, what kind of propagation is required?

\n

Thanks for the clarifications in advance

", "messages": [ { "id": 2456557, "postDate": "2023-09-26T09:18:22.233Z", "content": "

I have question about layout configuration. In paper TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs the authors wrote \"The autotuner tunes the input-output layouts of the most layout-performance-critical nodes and propagates layouts from these nodes to others.\" And I heard the same in some videos with @mangpophothilimthana
\nDoes anyone understand how exactly \"layout propagation\" works?
\nEach (non-configurable) node has 6 features related to layout. Is it a layout for an input or output tensor?
\nAnd after we change the layout configurations of the input and output tensors for some configurable node, what kind of propagation is required?

\n

Thanks for the clarifications in advance

", "rawMarkdown": "I have question about layout configuration. In paper [TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs](https://arxiv.org/pdf/2308.13490.pdf) the authors wrote \"The autotuner tunes the input-output layouts of the most layout-performance-critical nodes and **propagates layouts from these nodes to others**.\" And I heard the same in some videos with @mangpophothilimthana \nDoes anyone understand how exactly \"layout propagation\" works?\nEach (non-configurable) node has 6 features related to layout. Is it a layout for an input or output tensor?\nAnd after we change the layout configurations of the input and output tensors for some configurable node, what kind of propagation is required?\n\nThanks for the clarifications in advance", "votes": 7 }, { "id": 2465300, "postDate": "2023-10-02T20:07:31.617Z", "content": "

The layout propagation algorithm in the compiler is rather complex, and I don't fully understand it myself (I didn't implement it). The high-level idea is that the compiler (1) selects a convolution, dot, or reshape node and assign the best layout for that node, and (2) propagate the layout from that node as far as possible. The compiler integrates between these two steps until all layouts are assigned. There are a bunch of rules on how a layout is propagated between inputs and output of a node. Some can be found here. For example, an element-wise node will assign the same layout for its inputs and output during propagation.

", "rawMarkdown": "The layout propagation algorithm in the compiler is rather complex, and I don't fully understand it myself (I didn't implement it). The high-level idea is that the compiler (1) selects a convolution, dot, or reshape node and assign the best layout for that node, and (2) propagate the layout from that node as far as possible. The compiler integrates between these two steps until all layouts are assigned. There are a bunch of rules on how a layout is propagated between inputs and output of a node. Some can be found [here](https://github.com/openxla/xla/blob/main/xla/service/layout_assignment.cc). For example, an element-wise node will assign the same layout for its inputs and output during propagation.", "votes": 2, "replies": [ { "id": 2473896, "postDate": "2023-10-08T16:38:40.957Z", "content": "

How does this interact with the inserted copy operations? This is how I understand that it works (please correct me where I'm wrong)

\n

For configurable nodes

\n
    \n
  1. Configurable nodes expect input tensor to be laid out as described by input_layout_<N> fields in the layout config
    2. \n
  2. A copy operation is inserted when the layout an input tensor does not match the config output layout
    3. \n
  3. After inserting the copy operation, the layouts match
    4. \n
  4. The layout of the output tensor of the configurable node is as described by the layout output config
    5. \n
\n

For non-configurable nodes

\n
    \n
  1. Non-configurable nodes expect input tensors to be laid out as described by layout_minor_to_major_<N> fields
    2. \n
  2. Non-configurable nodes do dot change the tensor layout (so the output layout is the same as the input)
    3. \n
  3. If the input tensor does not match the expected input layout, a copy operation is inserted to resolve like for non-configurable nodes
    4. \n
\n

What I don't understand is how this requires any advanced layout propagation, it should be at most a copy at the input and outputs of each configurable nodes.

\n

Thankful if you can provide some insights to this!

", "rawMarkdown": "How does this interact with the inserted copy operations? This is how I understand that it works (please correct me where I'm wrong) \n\n**For configurable nodes**\n1. Configurable nodes expect input tensor to be laid out as described by `input_layout_` fields in the layout config\n1. A copy operation is inserted when the layout an input tensor does not match the config output layout\n2. After inserting the copy operation, the layouts match\n3. The layout of the output tensor of the configurable node is as described by the layout output config\n\n**For non-configurable nodes**\n1. Non-configurable nodes expect input tensors to be laid out as described by `layout_minor_to_major_` fields\n2. Non-configurable nodes do dot change the tensor layout (so the output layout is the same as the input)\n3. If the input tensor does not match the expected input layout, a copy operation is inserted to resolve like for non-configurable nodes\n\nWhat I don't understand is how this requires any advanced layout propagation, it should be at most a copy at the input and outputs of each configurable nodes.\n\nThankful if you can provide some insights to this!", "votes": 2, "replies": [ { "id": 2474046, "postDate": "2023-10-08T21:21:43.403Z", "content": "

Input and output layout configuration for some node don't have to match. Output layout config from previous node (input node) must be the same as input layout for configurable node, otherwise copy operator is needed.

\n

Why \"propagation\" is needed?
\nAssume you have two nodes A -> B (A is input for B). And node A is configurable node. Is it important for node B what you choose as output layout config for A {2,1,-1,-1,-1,-1} or {0,2,-1,-1,-1,-1}? For sure, it is. So, subsequent nodes must to know what choice you made, and that happens via \"layout propagation\"

", "rawMarkdown": "Input and output layout configuration for some node don't have to match. Output layout config from previous node (input node) must be the same as input layout for configurable node, otherwise copy operator is needed.\n\nWhy \"propagation\" is needed?\nAssume you have two nodes A -> B (A is input for B). And node A is configurable node. Is it important for node B what you choose as output layout config for A {2,1,-1,-1,-1,-1} or {0,2,-1,-1,-1,-1}? For sure, it is. So, subsequent nodes must to know what choice you made, and that happens via \"layout propagation\" ", "votes": 1 }, { "id": 2479529, "postDate": "2023-10-12T17:09:24.773Z", "content": "

Exactly, I believe you are 100% correct. And if the layout of an output of a configurable node (A) does not match the expected layout at a non-configurable (B), a copy operation is inserted (just like at the input of node A in case of mismatch). I am assuming that the last 6 node features describe both the expected input layout as well as the layout of the output tensor. So as far as I understand it, you only need to consider the immediate neighborhood around each configurable node, and optionally insert copy operations. But I am just speculating, would be great to get some input from organizers / read mode XLA code…

", "rawMarkdown": "Exactly, I believe you are 100% correct. And if the layout of an output of a configurable node (A) does not match the expected layout at a non-configurable (B), a copy operation is inserted (just like at the input of node A in case of mismatch). I am assuming that the last 6 node features describe both the expected input layout as well as the layout of the output tensor. So as far as I understand it, you only need to consider the immediate neighborhood around each configurable node, and optionally insert copy operations. But I am just speculating, would be great to get some input from organizers / read mode XLA code..." }, { "id": 2479993, "postDate": "2023-10-13T03:57:42.903Z", "content": "

This is correct. However, there is no limit on how far a layout propagation will go. For example, if there is a long chain of element-wise operation nodes, then the layout propagation would propagate through the whole chain until it hits a node whose inputs and output don't have the same shape.

", "rawMarkdown": "This is correct. However, there is no limit on how far a layout propagation will go. For example, if there is a long chain of element-wise operation nodes, then the layout propagation would propagate through the whole chain until it hits a node whose inputs and output don't have the same shape." }, { "id": 2480026, "postDate": "2023-10-13T05:07:10.660Z", "content": "

Thanks Mangpo, now I understand how this mechanism works!

", "rawMarkdown": "Thanks Mangpo, now I understand how this mechanism works!" }, { "id": 2498980, "postDate": "2023-10-25T16:30:44.393Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2499004, "postDate": "2023-10-25T16:55:22.163Z", "content": "

Sorry to break in, but let me confirm what layout_minor_to_major_<N> is. It seems sometimes configurable nodes have non-zero layout_minor_to_major_<N>. In that case, layout_minor_to_major_<N> is overridden by config input_layout_<N> and output_layout_<N>?

", "rawMarkdown": "Sorry to break in, but let me confirm what `layout_minor_to_major_` is. It seems sometimes configurable nodes have non-zero `layout_minor_to_major_`. In that case, `layout_minor_to_major_` is overridden by config `input_layout_` and `output_layout_`?", "votes": 1 }, { "id": 2501778, "postDate": "2023-10-27T16:43:15.760Z", "content": "

@mangpophothilimthana Could you probably answer this…? I'm wondering why layout_minor_to_major_<N> exists for configurable nodes.

", "rawMarkdown": "@mangpophothilimthana Could you probably answer this...? I'm wondering why `layout_minor_to_major_` exists for configurable nodes." }, { "id": 2505910, "postDate": "2023-10-31T00:57:36.210Z", "content": "

I'm not sure either… I think some compiler passes may have decided the layouts for some nodes before the layout assignment pass. But I don't know which one takes priority to be honest. I would guess that input_layout_<N> and output_layout_<N> take priority.

", "rawMarkdown": "I'm not sure either... I think some compiler passes may have decided the layouts for some nodes before the layout assignment pass. But I don't know which one takes priority to be honest. I would guess that `input_layout_` and `output_layout_` take priority." }, { "id": 2506490, "postDate": "2023-10-31T10:39:47.543Z", "content": "

Okay, thanks for your answer.

", "rawMarkdown": "Okay, thanks for your answer." } ] } ] } ], "comments": [ { "id": 2465300, "author_name": "Mangpo Phothilimthana", "author_url": "", "post_date": "2023-10-02T20:07:31.617000", "content": "

The layout propagation algorithm in the compiler is rather complex, and I don't fully understand it myself (I didn't implement it). The high-level idea is that the compiler (1) selects a convolution, dot, or reshape node and assign the best layout for that node, and (2) propagate the layout from that node as far as possible. The compiler integrates between these two steps until all layouts are assigned. There are a bunch of rules on how a layout is propagated between inputs and output of a node. Some can be found here. For example, an element-wise node will assign the same layout for its inputs and output during propagation.

", "votes": 2, "replies": [ { "id": 2473896, "author_name": "cdeln", "author_url": "", "post_date": "2023-10-08T16:38:40.957000", "content": "

How does this interact with the inserted copy operations? This is how I understand that it works (please correct me where I'm wrong)

\n

For configurable nodes

\n
    \n
  1. Configurable nodes expect input tensor to be laid out as described by input_layout_<N> fields in the layout config
    2. \n
  2. A copy operation is inserted when the layout an input tensor does not match the config output layout
    3. \n
  3. After inserting the copy operation, the layouts match
    4. \n
  4. The layout of the output tensor of the configurable node is as described by the layout output config
    5. \n
\n

For non-configurable nodes

\n
    \n
  1. Non-configurable nodes expect input tensors to be laid out as described by layout_minor_to_major_<N> fields
    2. \n
  2. Non-configurable nodes do dot change the tensor layout (so the output layout is the same as the input)
    3. \n
  3. If the input tensor does not match the expected input layout, a copy operation is inserted to resolve like for non-configurable nodes
    4. \n
\n

What I don't understand is how this requires any advanced layout propagation, it should be at most a copy at the input and outputs of each configurable nodes.

\n

Thankful if you can provide some insights to this!

", "votes": 2, "replies": [ { "id": 2474046, "author_name": "Shekhovtsov Aleksandr", "author_url": "", "post_date": "2023-10-08T21:21:43.403000", "content": "

Input and output layout configuration for some node don't have to match. Output layout config from previous node (input node) must be the same as input layout for configurable node, otherwise copy operator is needed.

\n

Why \"propagation\" is needed?
\nAssume you have two nodes A -> B (A is input for B). And node A is configurable node. Is it important for node B what you choose as output layout config for A {2,1,-1,-1,-1,-1} or {0,2,-1,-1,-1,-1}? For sure, it is. So, subsequent nodes must to know what choice you made, and that happens via \"layout propagation\"

", "votes": 1, "replies": [] }, { "id": 2479529, "author_name": "cdeln", "author_url": "", "post_date": "2023-10-12T17:09:24.773000", "content": "

Exactly, I believe you are 100% correct. And if the layout of an output of a configurable node (A) does not match the expected layout at a non-configurable (B), a copy operation is inserted (just like at the input of node A in case of mismatch). I am assuming that the last 6 node features describe both the expected input layout as well as the layout of the output tensor. So as far as I understand it, you only need to consider the immediate neighborhood around each configurable node, and optionally insert copy operations. But I am just speculating, would be great to get some input from organizers / read mode XLA code…

", "votes": 0, "replies": [] }, { "id": 2479993, "author_name": "Mangpo Phothilimthana", "author_url": "", "post_date": "2023-10-13T03:57:42.903000", "content": "

This is correct. However, there is no limit on how far a layout propagation will go. For example, if there is a long chain of element-wise operation nodes, then the layout propagation would propagate through the whole chain until it hits a node whose inputs and output don't have the same shape.

", "votes": 0, "replies": [] }, { "id": 2480026, "author_name": "cdeln", "author_url": "", "post_date": "2023-10-13T05:07:10.660000", "content": "

Thanks Mangpo, now I understand how this mechanism works!

", "votes": 0, "replies": [] }, { "id": 2498980, "author_name": "", "author_url": "", "post_date": "2023-10-25T16:30:44.393000", "content": "", "votes": 0, "replies": [] }, { "id": 2499004, "author_name": "knshnb", "author_url": "", "post_date": "2023-10-25T16:55:22.163000", "content": "

Sorry to break in, but let me confirm what layout_minor_to_major_<N> is. It seems sometimes configurable nodes have non-zero layout_minor_to_major_<N>. In that case, layout_minor_to_major_<N> is overridden by config input_layout_<N> and output_layout_<N>?

", "votes": 1, "replies": [] }, { "id": 2501778, "author_name": "knshnb", "author_url": "", "post_date": "2023-10-27T16:43:15.760000", "content": "

@mangpophothilimthana Could you probably answer this…? I'm wondering why layout_minor_to_major_<N> exists for configurable nodes.

", "votes": 0, "replies": [] }, { "id": 2505910, "author_name": "Mangpo Phothilimthana", "author_url": "", "post_date": "2023-10-31T00:57:36.210000", "content": "

I'm not sure either… I think some compiler passes may have decided the layouts for some nodes before the layout assignment pass. But I don't know which one takes priority to be honest. I would guess that input_layout_<N> and output_layout_<N> take priority.

", "votes": 0, "replies": [] }, { "id": 2506490, "author_name": "knshnb", "author_url": "", "post_date": "2023-10-31T10:39:47.543000", "content": "

Okay, thanks for your answer.

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2456557": "I have question about layout configuration. In paper [TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs](https://arxiv.org/pdf/2308.13490.pdf) the authors wrote \"The autotuner tunes the input-output layouts of the most layout-performance-critical nodes and **propagates layouts from these nodes to others**.\" And I heard the same in some videos with @mangpophothilimthana \nDoes anyone understand how exactly \"layout propagation\" works?\nEach (non-configurable) node has 6 features related to layout. Is it a layout for an input or output tensor?\nAnd after we change the layout configurations of the input and output tensors for some configurable node, what kind of propagation is required?\n\nThanks for the clarifications in advance", "2465300": "The layout propagation algorithm in the compiler is rather complex, and I don't fully understand it myself (I didn't implement it). The high-level idea is that the compiler (1) selects a convolution, dot, or reshape node and assign the best layout for that node, and (2) propagate the layout from that node as far as possible. The compiler integrates between these two steps until all layouts are assigned. There are a bunch of rules on how a layout is propagated between inputs and output of a node. Some can be found [here](https://github.com/openxla/xla/blob/main/xla/service/layout_assignment.cc). For example, an element-wise node will assign the same layout for its inputs and output during propagation." } }