{
  "id": 328230,
  "title": "Discrepancy between train\\val metrics and leaderboard metrics",
  "url": "/competitions/amex-default-prediction/discussion/328230",
  "author_name": "",
  "post_date": "2022-05-31T13:43:37.892841900Z",
  "votes": 6,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Does anyone else is having such a disparity over train\\val metrics and the leaderboard results?</p>\n<p>Here, over a baseline model, I got like 0.51\\0.49 in train\\val and got like 0.71 in the public leaderboard</p>\n<p>I am using the amex metric avaiable at:<br>\n<a href=\"https://www.kaggle.com/code/inversion/amex-competition-metric-python\" target=\"_blank\">Competions metric note</a><br>\nCheers!</p>",
  "messages": [
    {
      "id": "1806840",
      "postDate": "05/31/2022 13:43:37",
      "content": "<p>Does anyone else is having such a disparity over train\\val metrics and the leaderboard results?</p>\n<p>Here, over a baseline model, I got like 0.51\\0.49 in train\\val and got like 0.71 in the public leaderboard</p>\n<p>I am using the amex metric avaiable at:<br>\n<a href=\"https://www.kaggle.com/code/inversion/amex-competition-metric-python\" target=\"_blank\">Competions metric note</a><br>\nCheers!</p>",
      "rawMarkdown": "Does anyone else is having such a disparity over train\\val metrics and the leaderboard results?\n \nHere, over a baseline model, I got like 0.51\\0.49 in train\\val and got like 0.71 in the public leaderboard\n\nI am using the amex metric avaiable at:\n[Competions metric note](https://www.kaggle.com/code/inversion/amex-competition-metric-python)\nCheers!",
      "votes": null
    },
    {
      "id": "1806946",
      "postDate": "05/31/2022 15:41:39",
      "content": "<p>try resetting the index of the label df and prediction df</p>",
      "rawMarkdown": "try resetting the index of the label df and prediction df",
      "votes": null
    },
    {
      "id": "1807796",
      "postDate": "06/01/2022 10:34:39",
      "content": "<p>hi <a href=\"https://www.kaggle.com/weipengzhang\" target=\"_blank\">@weipengzhang</a> , <br>\nI tried it, didn't work. But thanks for the advice!<br>\ncheers</p>",
      "rawMarkdown": "hi @weipengzhang , \nI tried it, didn't work. But thanks for the advice!\ncheers",
      "votes": null
    },
    {
      "id": "1813071",
      "postDate": "06/06/2022 14:07:05",
      "content": "<p>I had a similar problem using numpy implementation avaiable <a href=\"https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\" target=\"_blank\">here</a>.  In my case, to solve this i just change y_pred vector for probas_pred vector:</p>\n<p>Before:<br>\nAMEX_train = 0.51<br>\nAMEX_val = 0.49</p>\n<p>After:<br>\nAMEX_train = 0.72<br>\nAMEX_val = 0.70</p>\n<p>This model takes 0.716 in public LB.</p>",
      "rawMarkdown": "I had a similar problem using numpy implementation avaiable [here](https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations).  In my case, to solve this i just change y_pred vector for probas_pred vector:\n\nBefore:\nAMEX_train = 0.51\nAMEX_val = 0.49\n\nAfter:\nAMEX_train = 0.72\nAMEX_val = 0.70\n\nThis model takes 0.716 in public LB.",
      "votes": null
    },
    {
      "id": "1813198",
      "postDate": "06/06/2022 15:49:42",
      "content": "<p>Great!, that was the problem! I was using the predict method and the metric asks for probabilities. Solve!<br>\nThanks!</p>",
      "rawMarkdown": "Great!, that was the problem! I was using the predict method and the metric asks for probabilities. Solve!\nThanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1806946,
      "author_name": "weipengzhang",
      "author_url": "",
      "post_date": "05/31/2022 15:41:39",
      "content": "<p>try resetting the index of the label df and prediction df</p>",
      "votes": null,
      "replies": [
        {
          "id": 1807796,
          "author_name": "paulojunqueira",
          "author_url": "",
          "post_date": "06/01/2022 10:34:39",
          "content": "<p>hi <a href=\"https://www.kaggle.com/weipengzhang\" target=\"_blank\">@weipengzhang</a> , <br>\nI tried it, didn't work. But thanks for the advice!<br>\ncheers</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1813071,
      "author_name": "gabrielvinicius",
      "author_url": "",
      "post_date": "06/06/2022 14:07:05",
      "content": "<p>I had a similar problem using numpy implementation avaiable <a href=\"https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\" target=\"_blank\">here</a>.  In my case, to solve this i just change y_pred vector for probas_pred vector:</p>\n<p>Before:<br>\nAMEX_train = 0.51<br>\nAMEX_val = 0.49</p>\n<p>After:<br>\nAMEX_train = 0.72<br>\nAMEX_val = 0.70</p>\n<p>This model takes 0.716 in public LB.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1813198,
          "author_name": "paulojunqueira",
          "author_url": "",
          "post_date": "06/06/2022 15:49:42",
          "content": "<p>Great!, that was the problem! I was using the predict method and the metric asks for probabilities. Solve!<br>\nThanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1806840": "Does anyone else is having such a disparity over train\\val metrics and the leaderboard results?\n \nHere, over a baseline model, I got like 0.51\\0.49 in train\\val and got like 0.71 in the public leaderboard\n\nI am using the amex metric avaiable at:\n[Competions metric note](https://www.kaggle.com/code/inversion/amex-competition-metric-python)\nCheers!",
    "1806946": "try resetting the index of the label df and prediction df",
    "1807796": "hi @weipengzhang , \nI tried it, didn't work. But thanks for the advice!\ncheers",
    "1813071": "I had a similar problem using numpy implementation avaiable [here](https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations).  In my case, to solve this i just change y_pred vector for probas_pred vector:\n\nBefore:\nAMEX_train = 0.51\nAMEX_val = 0.49\n\nAfter:\nAMEX_train = 0.72\nAMEX_val = 0.70\n\nThis model takes 0.716 in public LB.",
    "1813198": "Great!, that was the problem! I was using the predict method and the metric asks for probabilities. Solve!\nThanks!"
  },
  "source": "meta"
}