{
  "id": 443415,
  "title": "What is your kendal tau for layour:xla",
  "url": "/competitions/predict-ai-model-runtime/discussion/443415",
  "author_name": "",
  "post_date": "2023-09-27T06:08:57.535962700Z",
  "votes": 7,
  "comment_count": 8,
  "views": 0,
  "content": "<p>When training a model for layout:xla, the Kendal tau coefficient does not increase more than 0.1, although the ranking loss of my model steadily decreases on both training and validation datasets. However, using my model with such a low Kendal-Tau increases the public score by 0.05. Do I head in the right direction? </p>\n<p>Update: I trained my model on the updated model and got about 0.2 ~ 0.22 on the local validation dataset, resulting in 0.217 in the public leaderboard when only using the layout data.</p>",
  "messages": [
    {
      "id": "2457714",
      "postDate": "09/27/2023 06:08:57",
      "content": "<p>When training a model for layout:xla, the Kendal tau coefficient does not increase more than 0.1, although the ranking loss of my model steadily decreases on both training and validation datasets. However, using my model with such a low Kendal-Tau increases the public score by 0.05. Do I head in the right direction? </p>\n<p>Update: I trained my model on the updated model and got about 0.2 ~ 0.22 on the local validation dataset, resulting in 0.217 in the public leaderboard when only using the layout data.</p>",
      "rawMarkdown": "When training a model for layout:xla, the Kendal tau coefficient does not increase more than 0.1, although the ranking loss of my model steadily decreases on both training and validation datasets. However, using my model with such a low Kendal-Tau increases the public score by 0.05. Do I head in the right direction? \n\nUpdate: I trained my model on the updated model and got about 0.2 ~ 0.22 on the local validation dataset, resulting in 0.217 in the public leaderboard when only using the layout data.",
      "votes": null
    },
    {
      "id": "2485471",
      "postDate": "10/17/2023 07:59:51",
      "content": "<p>How do you compute the Kendal Tau correlation metric?</p>\n<p>I'm using the one from scipy.stats and averaging the correlation values from all datasets in layout:xla:default. Is this how the competition score is calculated?</p>\n<p>Thanks,</p>",
      "rawMarkdown": "How do you compute the Kendal Tau correlation metric?\n\nI'm using the one from scipy.stats and averaging the correlation values from all datasets in layout:xla:default. Is this how the competition score is calculated?\n\nThanks,",
      "votes": null
    },
    {
      "id": "2485917",
      "postDate": "10/17/2023 14:26:36",
      "content": "<p>I also performed poorly on the layout:xla dataset</p>",
      "rawMarkdown": "I also performed poorly on the layout:xla dataset",
      "votes": null
    },
    {
      "id": "2487430",
      "postDate": "10/18/2023 15:32:44",
      "content": "<p>I also face the same problem. My current local validation score is around 0.25 ~ 0.3 for layout:default:xla, 0.45 ~ 0.5 for layout:random:xla, 0.35 ~ 0.40 for layout:default:nlp, and 0.65 ~ 0.70 for layout:random:nlp</p>",
      "rawMarkdown": "I also face the same problem. My current local validation score is around 0.25 ~ 0.3 for layout:default:xla, 0.45 ~ 0.5 for layout:random:xla, 0.35 ~ 0.40 for layout:default:nlp, and 0.65 ~ 0.70 for layout:random:nlp",
      "votes": null
    },
    {
      "id": "2487434",
      "postDate": "10/18/2023 15:34:39",
      "content": "<p>I think this thread will help you. <a href=\"https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436977#2425405\" target=\"_blank\">https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436977#2425405</a></p>",
      "rawMarkdown": "I think this thread will help you. https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436977#2425405",
      "votes": null
    },
    {
      "id": "2487589",
      "postDate": "10/18/2023 17:00:33",
      "content": "<p>When you compute the Kendall Tau correlation for layout datasets, do you normalize it such that the correlation is always positive?</p>",
      "rawMarkdown": "When you compute the Kendall Tau correlation for layout datasets, do you normalize it such that the correlation is always positive?",
      "votes": null
    },
    {
      "id": "2491168",
      "postDate": "10/21/2023 11:38:58",
      "content": "<p>How do you validate your models? Do you just run kendal tau over all configurations in the graph? Or do you use a smaller sample since it's computationally expensive to run all of the configurations?</p>",
      "rawMarkdown": "How do you validate your models? Do you just run kendal tau over all configurations in the graph? Or do you use a smaller sample since it's computationally expensive to run all of the configurations?",
      "votes": null
    },
    {
      "id": "2492374",
      "postDate": "10/22/2023 12:54:49",
      "content": "<p>I use sampling. I guess using 24 ~ 100 configs is enough.</p>",
      "rawMarkdown": "I use sampling. I guess using 24 ~ 100 configs is enough.",
      "votes": null
    },
    {
      "id": "2492376",
      "postDate": "10/22/2023 12:55:28",
      "content": "<p>I tried both, but normalizing the output does not work well for my model.</p>",
      "rawMarkdown": "I tried both, but normalizing the output does not work well for my model.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2485471,
      "author_name": "passengerc07",
      "author_url": "",
      "post_date": "10/17/2023 07:59:51",
      "content": "<p>How do you compute the Kendal Tau correlation metric?</p>\n<p>I'm using the one from scipy.stats and averaging the correlation values from all datasets in layout:xla:default. Is this how the competition score is calculated?</p>\n<p>Thanks,</p>",
      "votes": null,
      "replies": [
        {
          "id": 2487434,
          "author_name": "syumei",
          "author_url": "",
          "post_date": "10/18/2023 15:34:39",
          "content": "<p>I think this thread will help you. <a href=\"https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436977#2425405\" target=\"_blank\">https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436977#2425405</a></p>",
          "votes": null,
          "replies": [
            {
              "id": 2487589,
              "author_name": "passengerc07",
              "author_url": "",
              "post_date": "10/18/2023 17:00:33",
              "content": "<p>When you compute the Kendall Tau correlation for layout datasets, do you normalize it such that the correlation is always positive?</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2492376,
                  "author_name": "syumei",
                  "author_url": "",
                  "post_date": "10/22/2023 12:55:28",
                  "content": "<p>I tried both, but normalizing the output does not work well for my model.</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2485917,
      "author_name": "chenboluo",
      "author_url": "",
      "post_date": "10/17/2023 14:26:36",
      "content": "<p>I also performed poorly on the layout:xla dataset</p>",
      "votes": null,
      "replies": [
        {
          "id": 2487430,
          "author_name": "syumei",
          "author_url": "",
          "post_date": "10/18/2023 15:32:44",
          "content": "<p>I also face the same problem. My current local validation score is around 0.25 ~ 0.3 for layout:default:xla, 0.45 ~ 0.5 for layout:random:xla, 0.35 ~ 0.40 for layout:default:nlp, and 0.65 ~ 0.70 for layout:random:nlp</p>",
          "votes": null,
          "replies": [
            {
              "id": 2491168,
              "author_name": "amitaharoni",
              "author_url": "",
              "post_date": "10/21/2023 11:38:58",
              "content": "<p>How do you validate your models? Do you just run kendal tau over all configurations in the graph? Or do you use a smaller sample since it's computationally expensive to run all of the configurations?</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2492374,
                  "author_name": "syumei",
                  "author_url": "",
                  "post_date": "10/22/2023 12:54:49",
                  "content": "<p>I use sampling. I guess using 24 ~ 100 configs is enough.</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2457714": "When training a model for layout:xla, the Kendal tau coefficient does not increase more than 0.1, although the ranking loss of my model steadily decreases on both training and validation datasets. However, using my model with such a low Kendal-Tau increases the public score by 0.05. Do I head in the right direction? \n\nUpdate: I trained my model on the updated model and got about 0.2 ~ 0.22 on the local validation dataset, resulting in 0.217 in the public leaderboard when only using the layout data.",
    "2485471": "How do you compute the Kendal Tau correlation metric?\n\nI'm using the one from scipy.stats and averaging the correlation values from all datasets in layout:xla:default. Is this how the competition score is calculated?\n\nThanks,",
    "2485917": "I also performed poorly on the layout:xla dataset",
    "2487430": "I also face the same problem. My current local validation score is around 0.25 ~ 0.3 for layout:default:xla, 0.45 ~ 0.5 for layout:random:xla, 0.35 ~ 0.40 for layout:default:nlp, and 0.65 ~ 0.70 for layout:random:nlp",
    "2487434": "I think this thread will help you. https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/436977#2425405",
    "2487589": "When you compute the Kendall Tau correlation for layout datasets, do you normalize it such that the correlation is always positive?",
    "2491168": "How do you validate your models? Do you just run kendal tau over all configurations in the graph? Or do you use a smaller sample since it's computationally expensive to run all of the configurations?",
    "2492374": "I use sampling. I guess using 24 ~ 100 configs is enough.",
    "2492376": "I tried both, but normalizing the output does not work well for my model."
  },
  "source": "meta"
}