{
  "id": 182737,
  "title": "Prediction Layer with cumsum function",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/182737",
  "author_name": "",
  "post_date": "2020-09-14T06:09:00.782176400Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi, Keep coming across this last layer in several public notebooks:<br>\n<code>p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)\n p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), \n                     name=\"preds\")([p1, p2])</code></p>\n<p>Can someone please give an explanation for this?</p>\n<p>Thanks</p>",
  "messages": [
    {
      "id": "1009607",
      "postDate": "09/14/2020 06:09:00",
      "content": "<p>Hi, Keep coming across this last layer in several public notebooks:<br>\n<code>p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)\n p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), \n                     name=\"preds\")([p1, p2])</code></p>\n<p>Can someone please give an explanation for this?</p>\n<p>Thanks</p>",
      "rawMarkdown": "Hi, Keep coming across this last layer in several public notebooks:\n`p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)\n p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), \n                     name=\"preds\")([p1, p2])`\n\nCan someone please give an explanation for this?\n\nThanks",
      "votes": null
    },
    {
      "id": "1009826",
      "postDate": "09/14/2020 09:16:51",
      "content": "<p>The idea is that when you estimate a low quantile (e.g. 15th percentile), the median and a high quantile (e.g. 85th percentile), you want to enforce that low quantile &lt;= median &lt;= high quantile. Logically, that should always be the case, but if you estimated each quantile separately with no connection between the loss functions, then you may end up sometimes not getting this ordering.</p>\n<p>If all values are non-negative (as e.g. a ReLU activation ensures), then the cumulative sum ensures that ordering of the values - e.g. let's say a prediction is [2400, 250, 250], then the cumsum is [2400, 2650, 2900].</p>\n<p>Note: Often there's also a prediction of a first value so that this actually becomes e.g. x=[[2300], [100, 250, 250]] and x[0] + cumsum(x[1]) =  [2400, 2650, 2900]. That's really just something to try out to see whether it works.</p>",
      "rawMarkdown": "The idea is that when you estimate a low quantile (e.g. 15th percentile), the median and a high quantile (e.g. 85th percentile), you want to enforce that low quantile <= median <= high quantile. Logically, that should always be the case, but if you estimated each quantile separately with no connection between the loss functions, then you may end up sometimes not getting this ordering.\n\nIf all values are non-negative (as e.g. a ReLU activation ensures), then the cumulative sum ensures that ordering of the values - e.g. let's say a prediction is [2400, 250, 250], then the cumsum is [2400, 2650, 2900].\n\nNote: Often there's also a prediction of a first value so that this actually becomes e.g. x=[[2300], [100, 250, 250]] and x[0] + cumsum(x[1]) =  [2400, 2650, 2900]. That's really just something to try out to see whether it works.",
      "votes": null
    },
    {
      "id": "1011816",
      "postDate": "09/15/2020 17:37:39",
      "content": "<p>Yes <a href=\"https://www.kaggle.com/bjoernholzhauer\" target=\"_blank\">@bjoernholzhauer</a>  you got it. That's was my intuition behind this setup.</p>",
      "rawMarkdown": "Yes @bjoernholzhauer  you got it. That's was my intuition behind this setup.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1009826,
      "author_name": "bjoernholzhauer",
      "author_url": "",
      "post_date": "09/14/2020 09:16:51",
      "content": "<p>The idea is that when you estimate a low quantile (e.g. 15th percentile), the median and a high quantile (e.g. 85th percentile), you want to enforce that low quantile &lt;= median &lt;= high quantile. Logically, that should always be the case, but if you estimated each quantile separately with no connection between the loss functions, then you may end up sometimes not getting this ordering.</p>\n<p>If all values are non-negative (as e.g. a ReLU activation ensures), then the cumulative sum ensures that ordering of the values - e.g. let's say a prediction is [2400, 250, 250], then the cumsum is [2400, 2650, 2900].</p>\n<p>Note: Often there's also a prediction of a first value so that this actually becomes e.g. x=[[2300], [100, 250, 250]] and x[0] + cumsum(x[1]) =  [2400, 2650, 2900]. That's really just something to try out to see whether it works.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1011816,
          "author_name": "ulrich07",
          "author_url": "",
          "post_date": "09/15/2020 17:37:39",
          "content": "<p>Yes <a href=\"https://www.kaggle.com/bjoernholzhauer\" target=\"_blank\">@bjoernholzhauer</a>  you got it. That's was my intuition behind this setup.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1009607": "Hi, Keep coming across this last layer in several public notebooks:\n`p1 = L.Dense(3, activation=\"linear\", name=\"p1\")(x)\n p2 = L.Dense(3, activation=\"relu\", name=\"p2\")(x)\n preds = L.Lambda(lambda x: x[0] + tf.cumsum(x[1], axis=1), \n                     name=\"preds\")([p1, p2])`\n\nCan someone please give an explanation for this?\n\nThanks",
    "1009826": "The idea is that when you estimate a low quantile (e.g. 15th percentile), the median and a high quantile (e.g. 85th percentile), you want to enforce that low quantile <= median <= high quantile. Logically, that should always be the case, but if you estimated each quantile separately with no connection between the loss functions, then you may end up sometimes not getting this ordering.\n\nIf all values are non-negative (as e.g. a ReLU activation ensures), then the cumulative sum ensures that ordering of the values - e.g. let's say a prediction is [2400, 250, 250], then the cumsum is [2400, 2650, 2900].\n\nNote: Often there's also a prediction of a first value so that this actually becomes e.g. x=[[2300], [100, 250, 250]] and x[0] + cumsum(x[1]) =  [2400, 2650, 2900]. That's really just something to try out to see whether it works.",
    "1011816": "Yes @bjoernholzhauer  you got it. That's was my intuition behind this setup."
  },
  "source": "meta"
}