{
  "id": 494235,
  "title": "I wanna extract last non-null value in preprocessing data.",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/494235",
  "author_name": "",
  "post_date": "2024-04-16T13:31:20.969983800Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi, all.<br>\nI wanna extract last or first non-null value in preprocessing.</p>\n<p>How can I do?<br>\ncould you teach me?</p>\n<p>In below code, I coded in handle_dates method, but it didn't work.<br>\n<a href=\"https://www.kaggle.com/code/takumimukaiyama/home-credit-lgb-encoding-idea\" target=\"_blank\">https://www.kaggle.com/code/takumimukaiyama/home-credit-lgb-encoding-idea</a></p>",
  "messages": [
    {
      "id": "2755345",
      "postDate": "04/16/2024 13:31:20",
      "content": "<p>Hi, all.<br>\nI wanna extract last or first non-null value in preprocessing.</p>\n<p>How can I do?<br>\ncould you teach me?</p>\n<p>In below code, I coded in handle_dates method, but it didn't work.<br>\n<a href=\"https://www.kaggle.com/code/takumimukaiyama/home-credit-lgb-encoding-idea\" target=\"_blank\">https://www.kaggle.com/code/takumimukaiyama/home-credit-lgb-encoding-idea</a></p>",
      "rawMarkdown": "Hi, all.\nI wanna extract last or first non-null value in preprocessing.\n\nHow can I do?\ncould you teach me?\n\nIn below code, I coded in handle_dates method, but it didn't work.\nhttps://www.kaggle.com/code/takumimukaiyama/home-credit-lgb-encoding-idea",
      "votes": null
    },
    {
      "id": "2755361",
      "postDate": "04/16/2024 13:37:42",
      "content": "<p>maybe max, min…?</p>",
      "rawMarkdown": "maybe max, min...?",
      "votes": null
    },
    {
      "id": "2755466",
      "postDate": "04/16/2024 14:22:47",
      "content": "<p>in polars u can run pl.col(column).first() or pl.col(column).last() to get the fist or last element in the series. To remove the null or nan or now elements, one option is to apply a filter or even drop the null values by using drop_nulls(). so over all it might look like this,  pl.col(column).drop_nulls().first()<br>\nthis is just a rough idea,  will have run the expression and correct if there are any minor errors and check if it is producing the required result.<br>\nfor reference, polars documentation is of great help and combined with stack overflow helps a lot to solve most of the tasks.</p>",
      "rawMarkdown": "in polars u can run pl.col(column).first() or pl.col(column).last() to get the fist or last element in the series. To remove the null or nan or now elements, one option is to apply a filter or even drop the null values by using drop_nulls(). so over all it might look like this,  pl.col(column).drop_nulls().first()\nthis is just a rough idea,  will have run the expression and correct if there are any minor errors and check if it is producing the required result.\nfor reference, polars documentation is of great help and combined with stack overflow helps a lot to solve most of the tasks.",
      "votes": null
    },
    {
      "id": "2755636",
      "postDate": "04/16/2024 15:52:35",
      "content": "<p>there is a little flaw in my above mentioned code. if the series has only 1 element and it is none than after the drop_nulls() will remove that only element and then, the first() will throw error as there are no elements to take first.<br>\nso better is pl.col(column).drop_nulls().append(None).first()<br>\nthis will return the first non null value and in case if all elements are null than it will return none</p>\n<p>for the last will have to do little more manpulations may be even combining with the when(condition).then(statement).otherwise(statement)</p>",
      "rawMarkdown": "there is a little flaw in my above mentioned code. if the series has only 1 element and it is none than after the drop_nulls() will remove that only element and then, the first() will throw error as there are no elements to take first.\nso better is pl.col(column).drop_nulls().append(None).first()\nthis will return the first non null value and in case if all elements are null than it will return none\n\nfor the last will have to do little more manpulations may be even combining with the when(condition).then(statement).otherwise(statement)",
      "votes": null
    },
    {
      "id": "2756350",
      "postDate": "04/17/2024 02:18:47",
      "content": "<p>pl.col(column).reverse().drop_nulls().append(None).first()</p>",
      "rawMarkdown": "pl.col(column).reverse().drop_nulls().append(None).first()",
      "votes": null
    },
    {
      "id": "2771945",
      "postDate": "04/24/2024 13:19:32",
      "content": "<blockquote>\n  <p>pl.col(col).filter(pl.col(col).is_not_null()).last()</p>\n</blockquote>\n<p>This expression does the job for me - inspiration from this <a href=\"https://stackoverflow.com/questions/75093945/how-to-select-the-last-non-null-value-from-one-column-and-also-the-value-from-an\" target=\"_blank\">stack overflow post</a>.</p>",
      "rawMarkdown": "> pl.col(col).filter(pl.col(col).is_not_null()).last()\n\nThis expression does the job for me - inspiration from this [stack overflow post](https://stackoverflow.com/questions/75093945/how-to-select-the-last-non-null-value-from-one-column-and-also-the-value-from-an).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2755361,
      "author_name": "takumimukaiyama",
      "author_url": "",
      "post_date": "04/16/2024 13:37:42",
      "content": "<p>maybe max, min…?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2755466,
      "author_name": "shreyas9181",
      "author_url": "",
      "post_date": "04/16/2024 14:22:47",
      "content": "<p>in polars u can run pl.col(column).first() or pl.col(column).last() to get the fist or last element in the series. To remove the null or nan or now elements, one option is to apply a filter or even drop the null values by using drop_nulls(). so over all it might look like this,  pl.col(column).drop_nulls().first()<br>\nthis is just a rough idea,  will have run the expression and correct if there are any minor errors and check if it is producing the required result.<br>\nfor reference, polars documentation is of great help and combined with stack overflow helps a lot to solve most of the tasks.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2755636,
          "author_name": "shreyas9181",
          "author_url": "",
          "post_date": "04/16/2024 15:52:35",
          "content": "<p>there is a little flaw in my above mentioned code. if the series has only 1 element and it is none than after the drop_nulls() will remove that only element and then, the first() will throw error as there are no elements to take first.<br>\nso better is pl.col(column).drop_nulls().append(None).first()<br>\nthis will return the first non null value and in case if all elements are null than it will return none</p>\n<p>for the last will have to do little more manpulations may be even combining with the when(condition).then(statement).otherwise(statement)</p>",
          "votes": null,
          "replies": [
            {
              "id": 2756350,
              "author_name": "shreyas9181",
              "author_url": "",
              "post_date": "04/17/2024 02:18:47",
              "content": "<p>pl.col(column).reverse().drop_nulls().append(None).first()</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2771945,
      "author_name": "andreasbis",
      "author_url": "",
      "post_date": "04/24/2024 13:19:32",
      "content": "<blockquote>\n  <p>pl.col(col).filter(pl.col(col).is_not_null()).last()</p>\n</blockquote>\n<p>This expression does the job for me - inspiration from this <a href=\"https://stackoverflow.com/questions/75093945/how-to-select-the-last-non-null-value-from-one-column-and-also-the-value-from-an\" target=\"_blank\">stack overflow post</a>.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2755345": "Hi, all.\nI wanna extract last or first non-null value in preprocessing.\n\nHow can I do?\ncould you teach me?\n\nIn below code, I coded in handle_dates method, but it didn't work.\nhttps://www.kaggle.com/code/takumimukaiyama/home-credit-lgb-encoding-idea",
    "2755361": "maybe max, min...?",
    "2755466": "in polars u can run pl.col(column).first() or pl.col(column).last() to get the fist or last element in the series. To remove the null or nan or now elements, one option is to apply a filter or even drop the null values by using drop_nulls(). so over all it might look like this,  pl.col(column).drop_nulls().first()\nthis is just a rough idea,  will have run the expression and correct if there are any minor errors and check if it is producing the required result.\nfor reference, polars documentation is of great help and combined with stack overflow helps a lot to solve most of the tasks.",
    "2755636": "there is a little flaw in my above mentioned code. if the series has only 1 element and it is none than after the drop_nulls() will remove that only element and then, the first() will throw error as there are no elements to take first.\nso better is pl.col(column).drop_nulls().append(None).first()\nthis will return the first non null value and in case if all elements are null than it will return none\n\nfor the last will have to do little more manpulations may be even combining with the when(condition).then(statement).otherwise(statement)",
    "2756350": "pl.col(column).reverse().drop_nulls().append(None).first()",
    "2771945": "> pl.col(col).filter(pl.col(col).is_not_null()).last()\n\nThis expression does the job for me - inspiration from this [stack overflow post](https://stackoverflow.com/questions/75093945/how-to-select-the-last-non-null-value-from-one-column-and-also-the-value-from-an)."
  },
  "source": "meta"
}