{
  "id": 581201,
  "title": "Evaluation metrics clarification",
  "url": "/competitions/drw-crypto-market-prediction/discussion/581201",
  "author_name": "",
  "post_date": "2025-05-29T03:13:31.696498300Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am new to this competition, and would like to clarify how to compute the evaluation metric.</p>\n<p>As outlined, the evaluation metric is the correlation between the prediction and the actual \"label\" values. I compute the correlation in my test sample using:<br>\n<code>\ncorr = np.corrcoef(results[['my prediction','label']].T)\n</code><br>\nThe correlation seems unreasonably high (i.e., 0.6 - 0.7). I wonder what did I miss here and how you guys compute the evaluation. A short coding demonstration will be helpful. Appreciated 😃</p>",
  "messages": [
    {
      "id": "3211891",
      "postDate": "05/29/2025 03:13:31",
      "content": "<p>I am new to this competition, and would like to clarify how to compute the evaluation metric.</p>\n<p>As outlined, the evaluation metric is the correlation between the prediction and the actual \"label\" values. I compute the correlation in my test sample using:<br>\n<code>\ncorr = np.corrcoef(results[['my prediction','label']].T)\n</code><br>\nThe correlation seems unreasonably high (i.e., 0.6 - 0.7). I wonder what did I miss here and how you guys compute the evaluation. A short coding demonstration will be helpful. Appreciated 😃</p>",
      "rawMarkdown": "I am new to this competition, and would like to clarify how to compute the evaluation metric.\n\nAs outlined, the evaluation metric is the correlation between the prediction and the actual \"label\" values. I compute the correlation in my test sample using:\n`\ncorr = np.corrcoef(results[['my prediction','label']].T)\n`\nThe correlation seems unreasonably high (i.e., 0.6 - 0.7). I wonder what did I miss here and how you guys compute the evaluation. A short coding demonstration will be helpful. Appreciated 😃",
      "votes": null
    },
    {
      "id": "3211896",
      "postDate": "05/29/2025 03:21:58",
      "content": "<p>You need to split the training set considering time information as past (train) and future (val), rather than splitting it randomly.</p>",
      "rawMarkdown": "You need to split the training set considering time information as past (train) and future (val), rather than splitting it randomly.",
      "votes": null
    },
    {
      "id": "3211908",
      "postDate": "05/29/2025 03:33:28",
      "content": "<p>Thanks! After using this time-series split, the evaluation metric looks normal now. </p>",
      "rawMarkdown": "Thanks! After using this time-series split, the evaluation metric looks normal now.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3211896,
      "author_name": "onurkoc83",
      "author_url": "",
      "post_date": "05/29/2025 03:21:58",
      "content": "<p>You need to split the training set considering time information as past (train) and future (val), rather than splitting it randomly.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3211908,
          "author_name": "shinchen93",
          "author_url": "",
          "post_date": "05/29/2025 03:33:28",
          "content": "<p>Thanks! After using this time-series split, the evaluation metric looks normal now. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3211891": "I am new to this competition, and would like to clarify how to compute the evaluation metric.\n\nAs outlined, the evaluation metric is the correlation between the prediction and the actual \"label\" values. I compute the correlation in my test sample using:\n`\ncorr = np.corrcoef(results[['my prediction','label']].T)\n`\nThe correlation seems unreasonably high (i.e., 0.6 - 0.7). I wonder what did I miss here and how you guys compute the evaluation. A short coding demonstration will be helpful. Appreciated 😃",
    "3211896": "You need to split the training set considering time information as past (train) and future (val), rather than splitting it randomly.",
    "3211908": "Thanks! After using this time-series split, the evaluation metric looks normal now."
  },
  "source": "meta"
}