{
  "id": 344612,
  "title": "Hidden column statistics when using .isnull().sum()",
  "url": "/competitions/amex-default-prediction/discussion/344612",
  "author_name": "",
  "post_date": "2022-08-15T23:21:06.713086100Z",
  "votes": null,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Hi there,</p>\n<p>When applying .isnull().sum() to the training dataframe, I get the following shortened response:</p>\n<p>customer_ID          0<br>\nS_2                         0<br>\nP_2                45985<br>\nD_39                       0<br>\nB_1                          0<br>\n                …<br>\nD_141            101548<br>\nD_142        4587043<br>\nD_143           101548<br>\nD_144             40727<br>\nD_145           101548<br>\nLength: 190, dtype: int64</p>\n<p>Is there a way I can force Pandas to return the full list of columns with the respective isnull sums?</p>",
  "messages": [
    {
      "id": "1900334",
      "postDate": "08/15/2022 23:21:06",
      "content": "<p>Hi there,</p>\n<p>When applying .isnull().sum() to the training dataframe, I get the following shortened response:</p>\n<p>customer_ID          0<br>\nS_2                         0<br>\nP_2                45985<br>\nD_39                       0<br>\nB_1                          0<br>\n                …<br>\nD_141            101548<br>\nD_142        4587043<br>\nD_143           101548<br>\nD_144             40727<br>\nD_145           101548<br>\nLength: 190, dtype: int64</p>\n<p>Is there a way I can force Pandas to return the full list of columns with the respective isnull sums?</p>",
      "rawMarkdown": "Hi there,\n\nWhen applying .isnull().sum() to the training dataframe, I get the following shortened response:\n\ncustomer_ID          0\nS_2                         0\nP_2                45985\nD_39                       0\nB_1                          0\n                ...\nD_141            101548\nD_142        4587043\nD_143           101548\nD_144             40727\nD_145           101548\nLength: 190, dtype: int64\n\nIs there a way I can force Pandas to return the full list of columns with the respective isnull sums?",
      "votes": null
    },
    {
      "id": "1900336",
      "postDate": "08/15/2022 23:24:29",
      "content": "<p>I would recommend to loop over columns and print it</p>\n<pre><code>for col in list(df):\n    print(f'Number of NaNs for {col} -', df[col].isna().sum())\n</code></pre>\n<p>or</p>\n<pre><code>print({ col: df[col].isna().sum() for col in list(df)})\n</code></pre>\n<p>Or use pd.set_option('display.max_rows', None)</p>\n<p>Or use .to_string()</p>",
      "rawMarkdown": "I would recommend to loop over columns and print it\n```\n\nfor col in list(df):\n    print(f'Number of NaNs for {col} -', df[col].isna().sum())\n\n```\n\nor\n\n```\nprint({ col: df[col].isna().sum() for col in list(df)})\n\n```\n\nOr use pd.set_option('display.max_rows', None)\n\nOr use .to_string()",
      "votes": null
    },
    {
      "id": "1900341",
      "postDate": "08/15/2022 23:46:39",
      "content": "<p>you can use this code</p>\n<blockquote>\n  <p>obj = train.isnull().sum()<br>\n  for key,value in obj.iteritems():<br>\n      print(key,\",\",value)</p>\n</blockquote>",
      "rawMarkdown": "you can use this code\n> obj = train.isnull().sum()\nfor key,value in obj.iteritems():\n    print(key,\",\",value)",
      "votes": null
    },
    {
      "id": "1900351",
      "postDate": "08/16/2022 00:04:27",
      "content": "<p>Thank you very much!</p>",
      "rawMarkdown": "Thank you very much!",
      "votes": null
    },
    {
      "id": "1900353",
      "postDate": "08/16/2022 00:04:54",
      "content": "<p>Thanks a lot! </p>",
      "rawMarkdown": "Thanks a lot!",
      "votes": null
    },
    {
      "id": "1900357",
      "postDate": "08/16/2022 00:10:15",
      "content": "<p>You may also use<br>\n<code>df.isnull().sum().to_frame()</code><br>\nto get a new null_df dataframe and display it as whole</p>",
      "rawMarkdown": "You may also use\n`df.isnull().sum().to_frame()`\nto get a new null_df dataframe and display it as whole",
      "votes": null
    },
    {
      "id": "1900365",
      "postDate": "08/16/2022 00:33:36",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!",
      "votes": null
    },
    {
      "id": "1900445",
      "postDate": "08/16/2022 03:48:40",
      "content": "<p>use the below code -<br>\n<strong>pd.set_option('display.max_rows', None)</strong><br>\n<strong>df.isnull().sum()</strong></p>",
      "rawMarkdown": "use the below code -\n**pd.set_option('display.max_rows', None)**\n**df.isnull().sum()**",
      "votes": null
    },
    {
      "id": "1900838",
      "postDate": "08/16/2022 09:34:56",
      "content": "<p>You can convert this into a dataframe using .to_frame and display it using ipython.display</p>",
      "rawMarkdown": "You can convert this into a dataframe using .to_frame and display it using ipython.display",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1900336,
      "author_name": "kyakovlev",
      "author_url": "",
      "post_date": "08/15/2022 23:24:29",
      "content": "<p>I would recommend to loop over columns and print it</p>\n<pre><code>for col in list(df):\n    print(f'Number of NaNs for {col} -', df[col].isna().sum())\n</code></pre>\n<p>or</p>\n<pre><code>print({ col: df[col].isna().sum() for col in list(df)})\n</code></pre>\n<p>Or use pd.set_option('display.max_rows', None)</p>\n<p>Or use .to_string()</p>",
      "votes": null,
      "replies": [
        {
          "id": 1900353,
          "author_name": "antoniointini1980",
          "author_url": "",
          "post_date": "08/16/2022 00:04:54",
          "content": "<p>Thanks a lot! </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1900341,
      "author_name": "naiborhujosua",
      "author_url": "",
      "post_date": "08/15/2022 23:46:39",
      "content": "<p>you can use this code</p>\n<blockquote>\n  <p>obj = train.isnull().sum()<br>\n  for key,value in obj.iteritems():<br>\n      print(key,\",\",value)</p>\n</blockquote>",
      "votes": null,
      "replies": [
        {
          "id": 1900351,
          "author_name": "antoniointini1980",
          "author_url": "",
          "post_date": "08/16/2022 00:04:27",
          "content": "<p>Thank you very much!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1900357,
      "author_name": "imeintanis",
      "author_url": "",
      "post_date": "08/16/2022 00:10:15",
      "content": "<p>You may also use<br>\n<code>df.isnull().sum().to_frame()</code><br>\nto get a new null_df dataframe and display it as whole</p>",
      "votes": null,
      "replies": [
        {
          "id": 1900365,
          "author_name": "antoniointini1980",
          "author_url": "",
          "post_date": "08/16/2022 00:33:36",
          "content": "<p>Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1900445,
      "author_name": "nitinchoudhary012",
      "author_url": "",
      "post_date": "08/16/2022 03:48:40",
      "content": "<p>use the below code -<br>\n<strong>pd.set_option('display.max_rows', None)</strong><br>\n<strong>df.isnull().sum()</strong></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1900838,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "08/16/2022 09:34:56",
      "content": "<p>You can convert this into a dataframe using .to_frame and display it using ipython.display</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1900334": "Hi there,\n\nWhen applying .isnull().sum() to the training dataframe, I get the following shortened response:\n\ncustomer_ID          0\nS_2                         0\nP_2                45985\nD_39                       0\nB_1                          0\n                ...\nD_141            101548\nD_142        4587043\nD_143           101548\nD_144             40727\nD_145           101548\nLength: 190, dtype: int64\n\nIs there a way I can force Pandas to return the full list of columns with the respective isnull sums?",
    "1900336": "I would recommend to loop over columns and print it\n```\n\nfor col in list(df):\n    print(f'Number of NaNs for {col} -', df[col].isna().sum())\n\n```\n\nor\n\n```\nprint({ col: df[col].isna().sum() for col in list(df)})\n\n```\n\nOr use pd.set_option('display.max_rows', None)\n\nOr use .to_string()",
    "1900341": "you can use this code\n> obj = train.isnull().sum()\nfor key,value in obj.iteritems():\n    print(key,\",\",value)",
    "1900351": "Thank you very much!",
    "1900353": "Thanks a lot!",
    "1900357": "You may also use\n`df.isnull().sum().to_frame()`\nto get a new null_df dataframe and display it as whole",
    "1900365": "Thank you!",
    "1900445": "use the below code -\n**pd.set_option('display.max_rows', None)**\n**df.isnull().sum()**",
    "1900838": "You can convert this into a dataframe using .to_frame and display it using ipython.display"
  },
  "source": "meta"
}