{
  "id": 346897,
  "title": "Negative score !!!!!!",
  "url": "/competitions/amex-default-prediction/discussion/346897",
  "author_name": "",
  "post_date": "2022-08-21T22:44:49.180603600Z",
  "votes": 6,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hey guys. <br>\nIt may seems funny but do you think it is possible for the Amex-metric to be negative? Cause my recent submissions for today received negative scores! </p>",
  "messages": [
    {
      "id": "1908693",
      "postDate": "08/21/2022 22:44:49",
      "content": "<p>Hey guys. <br>\nIt may seems funny but do you think it is possible for the Amex-metric to be negative? Cause my recent submissions for today received negative scores! </p>",
      "rawMarkdown": "Hey guys. \nIt may seems funny but do you think it is possible for the Amex-metric to be negative? Cause my recent submissions for today received negative scores!",
      "votes": null
    },
    {
      "id": "1908700",
      "postDate": "08/21/2022 23:18:45",
      "content": "<p>A negative score is possible. The metric is <code>0.5 * (G+D)</code> where <code>G = 2*AUC - 1</code>. Therefore if we take a high scoring public notebook and flip the predictions with <code>pred = 1 - pred</code> then <code>AUC &lt; 0.5</code> and the metric will most likely go negative. </p>\n<p>You can try flipping your predictions to get a positive Kaggle LB score.</p>",
      "rawMarkdown": "A negative score is possible. The metric is `0.5 * (G+D)` where `G = 2*AUC - 1`. Therefore if we take a high scoring public notebook and flip the predictions with `pred = 1 - pred` then `AUC < 0.5` and the metric will most likely go negative. \n\nYou can try flipping your predictions to get a positive Kaggle LB score.",
      "votes": null
    },
    {
      "id": "1908869",
      "postDate": "08/22/2022 05:36:17",
      "content": "<p>Thank you! I'm a little bit confused, although I'm getting a high validation value for amex metric (98%), the test score is negative</p>",
      "rawMarkdown": "Thank you! I'm a little bit confused, although I'm getting a high validation value for amex metric (98%), the test score is negative",
      "votes": null
    },
    {
      "id": "1909086",
      "postDate": "08/22/2022 10:09:04",
      "content": "<blockquote>\n  <p>high validation value for amex metric (98%)</p>\n</blockquote>\n<p>probably you have a strong leakage in your validation set, check your features again</p>",
      "rawMarkdown": "> high validation value for amex metric (98%)\n\nprobably you have a strong leakage in your validation set, check your features again",
      "votes": null
    },
    {
      "id": "1910688",
      "postDate": "08/23/2022 15:48:20",
      "content": "<p>Wow, you did it man! I accidentally 'scrambled' my submissions a couple times but only got near-zero scores :)</p>",
      "rawMarkdown": "Wow, you did it man! I accidentally 'scrambled' my submissions a couple times but only got near-zero scores :)",
      "votes": null
    },
    {
      "id": "1911897",
      "postDate": "08/24/2022 11:31:32",
      "content": "<p><a href=\"https://www.kaggle.com/javadkhorramdel\" target=\"_blank\">@javadkhorramdel</a> when I had this problem, it was because the feature ordering for the test set was different compared to the train set i.e. the model \"thinks\" the first column is \"a\" but in reality it is \"b\"</p>",
      "rawMarkdown": "javadkhorramdel when I had this problem, it was because the feature ordering for the test set was different compared to the train set i.e. the model \"thinks\" the first column is \"a\" but in reality it is \"b\"",
      "votes": null
    },
    {
      "id": "1912607",
      "postDate": "08/24/2022 21:17:12",
      "content": "<p>Happened to me too lol</p>",
      "rawMarkdown": "Happened to me too lol",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1908700,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "08/21/2022 23:18:45",
      "content": "<p>A negative score is possible. The metric is <code>0.5 * (G+D)</code> where <code>G = 2*AUC - 1</code>. Therefore if we take a high scoring public notebook and flip the predictions with <code>pred = 1 - pred</code> then <code>AUC &lt; 0.5</code> and the metric will most likely go negative. </p>\n<p>You can try flipping your predictions to get a positive Kaggle LB score.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1908869,
          "author_name": "javadkhorramdel",
          "author_url": "",
          "post_date": "08/22/2022 05:36:17",
          "content": "<p>Thank you! I'm a little bit confused, although I'm getting a high validation value for amex metric (98%), the test score is negative</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1909086,
          "author_name": "imeintanis",
          "author_url": "",
          "post_date": "08/22/2022 10:09:04",
          "content": "<blockquote>\n  <p>high validation value for amex metric (98%)</p>\n</blockquote>\n<p>probably you have a strong leakage in your validation set, check your features again</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1911897,
          "author_name": "pyagoubi",
          "author_url": "",
          "post_date": "08/24/2022 11:31:32",
          "content": "<p><a href=\"https://www.kaggle.com/javadkhorramdel\" target=\"_blank\">@javadkhorramdel</a> when I had this problem, it was because the feature ordering for the test set was different compared to the train set i.e. the model \"thinks\" the first column is \"a\" but in reality it is \"b\"</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1910688,
      "author_name": "scharlesworth",
      "author_url": "",
      "post_date": "08/23/2022 15:48:20",
      "content": "<p>Wow, you did it man! I accidentally 'scrambled' my submissions a couple times but only got near-zero scores :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1912607,
      "author_name": "powercatstats",
      "author_url": "",
      "post_date": "08/24/2022 21:17:12",
      "content": "<p>Happened to me too lol</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1908693": "Hey guys. \nIt may seems funny but do you think it is possible for the Amex-metric to be negative? Cause my recent submissions for today received negative scores!",
    "1908700": "A negative score is possible. The metric is `0.5 * (G+D)` where `G = 2*AUC - 1`. Therefore if we take a high scoring public notebook and flip the predictions with `pred = 1 - pred` then `AUC < 0.5` and the metric will most likely go negative. \n\nYou can try flipping your predictions to get a positive Kaggle LB score.",
    "1908869": "Thank you! I'm a little bit confused, although I'm getting a high validation value for amex metric (98%), the test score is negative",
    "1909086": "> high validation value for amex metric (98%)\n\nprobably you have a strong leakage in your validation set, check your features again",
    "1910688": "Wow, you did it man! I accidentally 'scrambled' my submissions a couple times but only got near-zero scores :)",
    "1911897": "javadkhorramdel when I had this problem, it was because the feature ordering for the test set was different compared to the train set i.e. the model \"thinks\" the first column is \"a\" but in reality it is \"b\"",
    "1912607": "Happened to me too lol"
  },
  "source": "meta"
}