{
  "id": 438819,
  "title": "Understanding the data collection process",
  "url": "/competitions/predict-ai-model-runtime/discussion/438819",
  "author_name": "",
  "post_date": "2023-09-12T16:51:04.226126300Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>If I understand correctly, the runtime statistics for each configuration is collected as follows</p>\n<ol>\n<li>Compile the graph according to the config. This includes kernel tiling.</li>\n<li>Execute the compiled model N times</li>\n</ol>\n<p>My questions are</p>\n<ol>\n<li>Is it possible to read the kernel tiling information in the layout dataset? I looked at <code>note_splits</code>, but that does not seem to be it</li>\n<li>How many times is each model executed? I.e. what is the expected variance of the statistics for each layout configuration?</li>\n</ol>",
  "messages": [
    {
      "id": "2435027",
      "postDate": "09/12/2023 16:51:04",
      "content": "<p>If I understand correctly, the runtime statistics for each configuration is collected as follows</p>\n<ol>\n<li>Compile the graph according to the config. This includes kernel tiling.</li>\n<li>Execute the compiled model N times</li>\n</ol>\n<p>My questions are</p>\n<ol>\n<li>Is it possible to read the kernel tiling information in the layout dataset? I looked at <code>note_splits</code>, but that does not seem to be it</li>\n<li>How many times is each model executed? I.e. what is the expected variance of the statistics for each layout configuration?</li>\n</ol>",
      "rawMarkdown": "If I understand correctly, the runtime statistics for each configuration is collected as follows\n\n1. Compile the graph according to the config. This includes kernel tiling.\n2. Execute the compiled model N times\n\nMy questions are\n\n1. Is it possible to read the kernel tiling information in the layout dataset? I looked at `note_splits`, but that does not seem to be it\n2. How many times is each model executed? I.e. what is the expected variance of the statistics for each layout configuration?",
      "votes": null
    },
    {
      "id": "2443477",
      "postDate": "09/17/2023 18:17:15",
      "content": "<ol>\n<li>No, we didn't include kernel tiling information in the layout dataset.</li>\n<li>Each model was executed 3 times. The variance for the layout dataset is extremely small (&lt;1% variation between min and max).</li>\n</ol>",
      "rawMarkdown": "1. No, we didn't include kernel tiling information in the layout dataset.\n2. Each model was executed 3 times. The variance for the layout dataset is extremely small (<1% variation between min and max).",
      "votes": null
    },
    {
      "id": "2444091",
      "postDate": "09/18/2023 06:42:52",
      "content": "<p>Thanks for you input. Follow up question: Does the node layout configuration have an impact on the compiler decision of the kernel tiling layout? I am asking because if it does, then the predictor essentially needs to learn the inner workings of the XLA compiler, which does seem like very difficult thing to learn well… Otherwise, if the kernel tiling is constant in the layout datasets, this is less of a problem.</p>",
      "rawMarkdown": "Thanks for you input. Follow up question: Does the node layout configuration have an impact on the compiler decision of the kernel tiling layout? I am asking because if it does, then the predictor essentially needs to learn the inner workings of the XLA compiler, which does seem like very difficult thing to learn well... Otherwise, if the kernel tiling is constant in the layout datasets, this is less of a problem.",
      "votes": null
    },
    {
      "id": "2447391",
      "postDate": "09/20/2023 03:50:10",
      "content": "<p>Yes, layout configuration has an impact on the tiling decision! And yes, it is indeed a difficult problem.</p>",
      "rawMarkdown": "Yes, layout configuration has an impact on the tiling decision! And yes, it is indeed a difficult problem.",
      "votes": null
    },
    {
      "id": "2454214",
      "postDate": "09/24/2023 16:44:15",
      "content": "<p>Now I see why you said that the model becomes specific for each version of the compiler in your talk. May I ask with that compiler version did you run the experiments?</p>",
      "rawMarkdown": "Now I see why you said that the model becomes specific for each version of the compiler in your talk. May I ask with that compiler version did you run the experiments?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2443477,
      "author_name": "mangpophothilimthana",
      "author_url": "",
      "post_date": "09/17/2023 18:17:15",
      "content": "<ol>\n<li>No, we didn't include kernel tiling information in the layout dataset.</li>\n<li>Each model was executed 3 times. The variance for the layout dataset is extremely small (&lt;1% variation between min and max).</li>\n</ol>",
      "votes": null,
      "replies": [
        {
          "id": 2444091,
          "author_name": "carldehlin",
          "author_url": "",
          "post_date": "09/18/2023 06:42:52",
          "content": "<p>Thanks for you input. Follow up question: Does the node layout configuration have an impact on the compiler decision of the kernel tiling layout? I am asking because if it does, then the predictor essentially needs to learn the inner workings of the XLA compiler, which does seem like very difficult thing to learn well… Otherwise, if the kernel tiling is constant in the layout datasets, this is less of a problem.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2447391,
              "author_name": "mangpophothilimthana",
              "author_url": "",
              "post_date": "09/20/2023 03:50:10",
              "content": "<p>Yes, layout configuration has an impact on the tiling decision! And yes, it is indeed a difficult problem.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2454214,
                  "author_name": "carldehlin",
                  "author_url": "",
                  "post_date": "09/24/2023 16:44:15",
                  "content": "<p>Now I see why you said that the model becomes specific for each version of the compiler in your talk. May I ask with that compiler version did you run the experiments?</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2435027": "If I understand correctly, the runtime statistics for each configuration is collected as follows\n\n1. Compile the graph according to the config. This includes kernel tiling.\n2. Execute the compiled model N times\n\nMy questions are\n\n1. Is it possible to read the kernel tiling information in the layout dataset? I looked at `note_splits`, but that does not seem to be it\n2. How many times is each model executed? I.e. what is the expected variance of the statistics for each layout configuration?",
    "2443477": "1. No, we didn't include kernel tiling information in the layout dataset.\n2. Each model was executed 3 times. The variance for the layout dataset is extremely small (<1% variation between min and max).",
    "2444091": "Thanks for you input. Follow up question: Does the node layout configuration have an impact on the compiler decision of the kernel tiling layout? I am asking because if it does, then the predictor essentially needs to learn the inner workings of the XLA compiler, which does seem like very difficult thing to learn well... Otherwise, if the kernel tiling is constant in the layout datasets, this is less of a problem.",
    "2447391": "Yes, layout configuration has an impact on the tiling decision! And yes, it is indeed a difficult problem.",
    "2454214": "Now I see why you said that the model becomes specific for each version of the compiler in your talk. May I ask with that compiler version did you run the experiments?"
  },
  "source": "meta"
}