{
  "id": 453985,
  "title": "Evaluation for test dataset same config files",
  "url": "/competitions/predict-ai-model-runtime/discussion/453985",
  "author_name": "",
  "post_date": "2023-11-08T13:35:02.275679600Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>As I explore the data for the test data configurations, I'm wondering whether we should consider configurations with the same ranking values.</p>\n<p>Initially, I noticed that there are instances where configurations are the same. If two configurations are the same, the runtime should be considered with the same rank. Does the evaluation metric disregard the identical ranking, or do we need to predict the ranking for these configurations?</p>\n<p>Furthermore, does Kendall Tau account for identical ranks in the case of the same configurations?</p>\n<p>For example, if the index of config, 23, 49 ,133, 134,205 are the same configurations, do results of this same ranking such as (23,49,133,134,205) or (205,134,49,23,133) cause different results while evaluation?</p>",
  "messages": [
    {
      "id": "2517444",
      "postDate": "11/08/2023 13:35:02",
      "content": "<p>As I explore the data for the test data configurations, I'm wondering whether we should consider configurations with the same ranking values.</p>\n<p>Initially, I noticed that there are instances where configurations are the same. If two configurations are the same, the runtime should be considered with the same rank. Does the evaluation metric disregard the identical ranking, or do we need to predict the ranking for these configurations?</p>\n<p>Furthermore, does Kendall Tau account for identical ranks in the case of the same configurations?</p>\n<p>For example, if the index of config, 23, 49 ,133, 134,205 are the same configurations, do results of this same ranking such as (23,49,133,134,205) or (205,134,49,23,133) cause different results while evaluation?</p>",
      "rawMarkdown": "As I explore the data for the test data configurations, I'm wondering whether we should consider configurations with the same ranking values.\n\nInitially, I noticed that there are instances where configurations are the same. If two configurations are the same, the runtime should be considered with the same rank. Does the evaluation metric disregard the identical ranking, or do we need to predict the ranking for these configurations?\n\nFurthermore, does Kendall Tau account for identical ranks in the case of the same configurations?\n\nFor example, if the index of config, 23, 49 ,133, 134,205 are the same configurations, do results of this same ranking such as (23,49,133,134,205) or (205,134,49,23,133) cause different results while evaluation?",
      "votes": null
    },
    {
      "id": "2517569",
      "postDate": "11/08/2023 15:24:07",
      "content": "<p>\" there are instances where configurations are the same.\"</p>\n<p>it would be better you can list the problematic examples here:<br>\ne.g.  collection xla:random:xxx.npz, 5th and 189th rows of node_config_feature …</p>",
      "rawMarkdown": "\" there are instances where configurations are the same.\"\n\nit would be better you can list the problematic examples here:\ne.g.  collection xla:random:xxx.npz, 5th and 189th rows of node\\_config\\_feature ...",
      "votes": null
    },
    {
      "id": "2517617",
      "postDate": "11/08/2023 16:24:16",
      "content": "<p>I just checked npz files with ['node_config_feat'] and checked with np.unique(np_file['node_config_feat'], axis = 0).</p>\n<p>e.g. collection xla:random:05ae41e26dd3c4c06390371a0423233c, 97th configuration seems to be same with [143, 232, 244, 259, 267, 274, 323, 341, 345, 358, 367, 400, 440, 498, 585, 709, 730, 779, 787, 849, 894, 934]. </p>\n<p>Is there anything wrong with checking the configuration checking? </p>",
      "rawMarkdown": "I just checked npz files with ['node_config_feat'] and checked with np.unique(np_file['node_config_feat'], axis = 0).\n\ne.g. collection xla:random:05ae41e26dd3c4c06390371a0423233c, 97th configuration seems to be same with [143, 232, 244, 259, 267, 274, 323, 341, 345, 358, 367, 400, 440, 498, 585, 709, 730, 779, 787, 849, 894, 934]. \n\nIs there anything wrong with checking the configuration checking?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2517569,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "11/08/2023 15:24:07",
      "content": "<p>\" there are instances where configurations are the same.\"</p>\n<p>it would be better you can list the problematic examples here:<br>\ne.g.  collection xla:random:xxx.npz, 5th and 189th rows of node_config_feature …</p>",
      "votes": null,
      "replies": [
        {
          "id": 2517617,
          "author_name": "sunjongpark",
          "author_url": "",
          "post_date": "11/08/2023 16:24:16",
          "content": "<p>I just checked npz files with ['node_config_feat'] and checked with np.unique(np_file['node_config_feat'], axis = 0).</p>\n<p>e.g. collection xla:random:05ae41e26dd3c4c06390371a0423233c, 97th configuration seems to be same with [143, 232, 244, 259, 267, 274, 323, 341, 345, 358, 367, 400, 440, 498, 585, 709, 730, 779, 787, 849, 894, 934]. </p>\n<p>Is there anything wrong with checking the configuration checking? </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2517444": "As I explore the data for the test data configurations, I'm wondering whether we should consider configurations with the same ranking values.\n\nInitially, I noticed that there are instances where configurations are the same. If two configurations are the same, the runtime should be considered with the same rank. Does the evaluation metric disregard the identical ranking, or do we need to predict the ranking for these configurations?\n\nFurthermore, does Kendall Tau account for identical ranks in the case of the same configurations?\n\nFor example, if the index of config, 23, 49 ,133, 134,205 are the same configurations, do results of this same ranking such as (23,49,133,134,205) or (205,134,49,23,133) cause different results while evaluation?",
    "2517569": "\" there are instances where configurations are the same.\"\n\nit would be better you can list the problematic examples here:\ne.g.  collection xla:random:xxx.npz, 5th and 189th rows of node\\_config\\_feature ...",
    "2517617": "I just checked npz files with ['node_config_feat'] and checked with np.unique(np_file['node_config_feat'], axis = 0).\n\ne.g. collection xla:random:05ae41e26dd3c4c06390371a0423233c, 97th configuration seems to be same with [143, 232, 244, 259, 267, 274, 323, 341, 345, 358, 367, 400, 440, 498, 585, 709, 730, 779, 787, 849, 894, 934]. \n\nIs there anything wrong with checking the configuration checking?"
  },
  "source": "meta"
}