{
  "id": 123209,
  "title": "Probability transformation in imbalanced datasets",
  "url": "/competitions/deepfake-detection-challenge/discussion/123209",
  "author_name": "",
  "post_date": "2019-12-25T18:39:11.609947200Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>My logloss ~.4 and AUC ~0.7 in training set which is unbalanced 0.8 (REAL/FAKE) , in validation set I did unbalanced 0.5 (trying to imitate private set) and my logloss b 0.9 but AUC still 0.7 . And LB supports that result logloss 0.9 . How do you transform? Please :) (I shifted my predictions by 0.3 it helped but still worse then all .5 probs. )</p>",
  "messages": [
    {
      "id": "703184",
      "postDate": "12/25/2019 18:39:11",
      "content": "<p>My logloss ~.4 and AUC ~0.7 in training set which is unbalanced 0.8 (REAL/FAKE) , in validation set I did unbalanced 0.5 (trying to imitate private set) and my logloss b 0.9 but AUC still 0.7 . And LB supports that result logloss 0.9 . How do you transform? Please :) (I shifted my predictions by 0.3 it helped but still worse then all .5 probs. )</p>",
      "rawMarkdown": "My logloss ~.4 and AUC ~0.7 in training set which is unbalanced 0.8 (REAL/FAKE) , in validation set I did unbalanced 0.5 (trying to imitate private set) and my logloss b 0.9 but AUC still 0.7 . And LB supports that result logloss 0.9 . How do you transform? Please :) (I shifted my predictions by 0.3 it helped but still worse then all .5 probs. )",
      "votes": null
    },
    {
      "id": "703246",
      "postDate": "12/25/2019 21:47:27",
      "content": "<p>AUC <code>0.7</code> is promising, definitely better than a random choice. You have a vector of probabilities, and a prior knowledge that its average needs to be <code>0.5</code>. The question is how to transform it, simply shifting the predictions is clearly sub-optimal. One way, which is at least better (may be optimal under some assumptions?), is first to transform your probs by <a href=\"https://en.wikipedia.org/wiki/Logit\">logit</a>, and only then shift by a constant, until you get average of <code>0.5</code> when transforming back with sigmoid. </p>",
      "rawMarkdown": "AUC `0.7` is promising, definitely better than a random choice. You have a vector of probabilities, and a prior knowledge that its average needs to be `0.5`. The question is how to transform it, simply shifting the predictions is clearly sub-optimal. One way, which is at least better (may be optimal under some assumptions?), is first to transform your probs by [logit](https://en.wikipedia.org/wiki/Logit), and only then shift by a constant, until you get average of `0.5` when transforming back with sigmoid.",
      "votes": null
    },
    {
      "id": "703660",
      "postDate": "12/26/2019 12:59:07",
      "content": "<p>I did sigmoid(logit(p)-d) minimizing logloss on 0.5 balanced validation set (by d). is it OK ? and get worse then p-0.3 on leaderboard (first case 0.7 second 0.73).\nBut.. btw first case on 0.5 balanced validation set i get 0.63 second 0.61 that showed that your suggested method is better then simple shift.</p>",
      "rawMarkdown": "I did sigmoid(logit(p)-d) minimizing logloss on 0.5 balanced validation set (by d). is it OK ? and get worse then p-0.3 on leaderboard (first case 0.7 second 0.73).\nBut.. btw first case on 0.5 balanced validation set i get 0.63 second 0.61 that showed that your suggested method is better then simple shift.",
      "votes": null
    },
    {
      "id": "703671",
      "postDate": "12/26/2019 13:29:00",
      "content": "<p>Interesting. What if you optimize closeness to <code>0.5</code> average, instead of the validation log-loss? Additionally, you can do it for the public test set itself, automatically inside the kernel, and not on the validation set. May be this is a good point to note that the private test set is not known to be balanced, so all this balancing is just overfitting to the public, need to keep this in mind.</p>",
      "rawMarkdown": "Interesting. What if you optimize closeness to `0.5` average, instead of the validation log-loss? Additionally, you can do it for the public test set itself, automatically inside the kernel, and not on the validation set. May be this is a good point to note that the private test set is not known to be balanced, so all this balancing is just overfitting to the public, need to keep this in mind.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 703246,
      "author_name": "zaharch",
      "author_url": "",
      "post_date": "12/25/2019 21:47:27",
      "content": "<p>AUC <code>0.7</code> is promising, definitely better than a random choice. You have a vector of probabilities, and a prior knowledge that its average needs to be <code>0.5</code>. The question is how to transform it, simply shifting the predictions is clearly sub-optimal. One way, which is at least better (may be optimal under some assumptions?), is first to transform your probs by <a href=\"https://en.wikipedia.org/wiki/Logit\">logit</a>, and only then shift by a constant, until you get average of <code>0.5</code> when transforming back with sigmoid. </p>",
      "votes": null,
      "replies": [
        {
          "id": 703660,
          "author_name": "jonasmatuzas",
          "author_url": "",
          "post_date": "12/26/2019 12:59:07",
          "content": "<p>I did sigmoid(logit(p)-d) minimizing logloss on 0.5 balanced validation set (by d). is it OK ? and get worse then p-0.3 on leaderboard (first case 0.7 second 0.73).\nBut.. btw first case on 0.5 balanced validation set i get 0.63 second 0.61 that showed that your suggested method is better then simple shift.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 703671,
          "author_name": "zaharch",
          "author_url": "",
          "post_date": "12/26/2019 13:29:00",
          "content": "<p>Interesting. What if you optimize closeness to <code>0.5</code> average, instead of the validation log-loss? Additionally, you can do it for the public test set itself, automatically inside the kernel, and not on the validation set. May be this is a good point to note that the private test set is not known to be balanced, so all this balancing is just overfitting to the public, need to keep this in mind.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "703184": "My logloss ~.4 and AUC ~0.7 in training set which is unbalanced 0.8 (REAL/FAKE) , in validation set I did unbalanced 0.5 (trying to imitate private set) and my logloss b 0.9 but AUC still 0.7 . And LB supports that result logloss 0.9 . How do you transform? Please :) (I shifted my predictions by 0.3 it helped but still worse then all .5 probs. )",
    "703246": "AUC `0.7` is promising, definitely better than a random choice. You have a vector of probabilities, and a prior knowledge that its average needs to be `0.5`. The question is how to transform it, simply shifting the predictions is clearly sub-optimal. One way, which is at least better (may be optimal under some assumptions?), is first to transform your probs by [logit](https://en.wikipedia.org/wiki/Logit), and only then shift by a constant, until you get average of `0.5` when transforming back with sigmoid.",
    "703660": "I did sigmoid(logit(p)-d) minimizing logloss on 0.5 balanced validation set (by d). is it OK ? and get worse then p-0.3 on leaderboard (first case 0.7 second 0.73).\nBut.. btw first case on 0.5 balanced validation set i get 0.63 second 0.61 that showed that your suggested method is better then simple shift.",
    "703671": "Interesting. What if you optimize closeness to `0.5` average, instead of the validation log-loss? Additionally, you can do it for the public test set itself, automatically inside the kernel, and not on the validation set. May be this is a good point to note that the private test set is not known to be balanced, so all this balancing is just overfitting to the public, need to keep this in mind."
  },
  "source": "meta"
}