{ "id": 193411, "title": "rapids questions", "url": "/competitions/riiid-test-answer-prediction/discussion/193411", "author_name": "", "post_date": "2020-10-27T00:49:58.263934900Z", "votes": 3, "comment_count": 3, "views": 0, "content": "

Does anybody know how to do cumum by group and tail and head by group, where a specified number of rows within each group are returned, using cudf? I looked at the api and those methods don't exist. Cumsum is being worked on, but was curious if anyone had any ideas on how to use existing methods to achieve the same results.

", "messages": [ { "id": "1061367", "postDate": "10/27/2020 00:49:58", "content": "

Does anybody know how to do cumum by group and tail and head by group, where a specified number of rows within each group are returned, using cudf? I looked at the api and those methods don't exist. Cumsum is being worked on, but was curious if anyone had any ideas on how to use existing methods to achieve the same results.

", "rawMarkdown": "Does anybody know how to do cumum by group and [tail](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.tail.html) and [head](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.head.html) by group, where a specified number of rows within each group are returned, using cudf? I looked at the [api](https://docs.rapids.ai/api/cudf/stable/api.html?highlight=groupby#groupby) and those methods don't exist. [Cumsum is being worked on](https://github.com/rapidsai/cudf/issues/1298), but was curious if anyone had any ideas on how to use existing methods to achieve the same results.", "votes": null }, { "id": "1077067", "postDate": "11/13/2020 08:00:00", "content": "

How about these??

\n

`

\n
import cudf\nfrom numba import cuda\n\n#pandas\ndf = pd.DataFrame({'group' : [1,1,2,3,3,3,3,2,2,4,4,4] , 'value' : [1,2,1,1,3,2,4,2,1,3,2,1]} )\ndf[\"cumcount_pd\"] = df.groupby(\"group\").cumcount()\ndf[\"cumsum_pd\"] = df.groupby(\"group\")[\"value\"].cumsum()\nprint(df)\n\n\n#cudf\ndf = cudf.DataFrame(df)\n\n# Define a function to apply to each group\ndef cudf_cum(value, cumcount_cudf, cumsum_cudf):\n\n    for i in range(cuda.threadIdx.x, len(value), cuda.blockDim.x):\n        #print('thread_id:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x, 'array size:', value.size, 'block threads:', cuda.blockDim.x)\n\n\n        cumcount_cudf[i] = i\n        cumsum_cudf[i] = 0\n        for j in range(i+1):\n\n            cumsum_cudf[i] += value[j]\n\n\n\ndf = df.groupby(\"group\", method=\"cudf\").apply_grouped(cudf_cum,\n                          incols=[\"value\"],\n                          outcols={'cumcount_cudf': np.int64,\n                                    'cumsum_cudf': np.int64,\n                                   },\n                          # threads per block\n                          tpb=8)\n\n######################\nhead_cudf = df.groupby(\"group\").nth(0) #head\ntail_cudf = df.groupby(\"group\").nth(-1) #tail\n
\n

`

", "rawMarkdown": "How about these??\n\n`\n\n import cudf\n from numba import cuda\n\n #pandas\n df = pd.DataFrame({'group' : [1,1,2,3,3,3,3,2,2,4,4,4] , 'value' : [1,2,1,1,3,2,4,2,1,3,2,1]} )\n df[\"cumcount_pd\"] = df.groupby(\"group\").cumcount()\n df[\"cumsum_pd\"] = df.groupby(\"group\")[\"value\"].cumsum()\n print(df)\n\n\n #cudf\n df = cudf.DataFrame(df)\n\n # Define a function to apply to each group\n def cudf_cum(value, cumcount_cudf, cumsum_cudf):\n\n for i in range(cuda.threadIdx.x, len(value), cuda.blockDim.x):\n #print('thread_id:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x, 'array size:', value.size, 'block threads:', cuda.blockDim.x)\n \n \n cumcount_cudf[i] = i\n cumsum_cudf[i] = 0\n for j in range(i+1):\n\n cumsum_cudf[i] += value[j]\n\n \n\n df = df.groupby(\"group\", method=\"cudf\").apply_grouped(cudf_cum,\n incols=[\"value\"],\n outcols={'cumcount_cudf': np.int64,\n 'cumsum_cudf': np.int64,\n },\n # threads per block\n tpb=8)\n\n ######################\n head_cudf = df.groupby(\"group\").nth(0) #head\n tail_cudf = df.groupby(\"group\").nth(-1) #tail\n\n\n`", "votes": null }, { "id": "1077168", "postDate": "11/13/2020 09:38:28", "content": "

this cumsum code is very slow…

", "rawMarkdown": "this cumsum code is very slow...", "votes": null }, { "id": "1079176", "postDate": "11/15/2020 17:57:04", "content": "

Cool, thanks. I ended up with sql.

", "rawMarkdown": "Cool, thanks. I ended up with sql.", "votes": null } ], "comments": [ { "id": 1077067, "author_name": "nakayamar", "author_url": "", "post_date": "11/13/2020 08:00:00", "content": "

How about these??

\n

`

\n
import cudf\nfrom numba import cuda\n\n#pandas\ndf = pd.DataFrame({'group' : [1,1,2,3,3,3,3,2,2,4,4,4] , 'value' : [1,2,1,1,3,2,4,2,1,3,2,1]} )\ndf[\"cumcount_pd\"] = df.groupby(\"group\").cumcount()\ndf[\"cumsum_pd\"] = df.groupby(\"group\")[\"value\"].cumsum()\nprint(df)\n\n\n#cudf\ndf = cudf.DataFrame(df)\n\n# Define a function to apply to each group\ndef cudf_cum(value, cumcount_cudf, cumsum_cudf):\n\n    for i in range(cuda.threadIdx.x, len(value), cuda.blockDim.x):\n        #print('thread_id:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x, 'array size:', value.size, 'block threads:', cuda.blockDim.x)\n\n\n        cumcount_cudf[i] = i\n        cumsum_cudf[i] = 0\n        for j in range(i+1):\n\n            cumsum_cudf[i] += value[j]\n\n\n\ndf = df.groupby(\"group\", method=\"cudf\").apply_grouped(cudf_cum,\n                          incols=[\"value\"],\n                          outcols={'cumcount_cudf': np.int64,\n                                    'cumsum_cudf': np.int64,\n                                   },\n                          # threads per block\n                          tpb=8)\n\n######################\nhead_cudf = df.groupby(\"group\").nth(0) #head\ntail_cudf = df.groupby(\"group\").nth(-1) #tail\n
\n

`

", "votes": null, "replies": [ { "id": 1077168, "author_name": "nakayamar", "author_url": "", "post_date": "11/13/2020 09:38:28", "content": "

this cumsum code is very slow…

", "votes": null, "replies": [] }, { "id": 1079176, "author_name": "calebeverett", "author_url": "", "post_date": "11/15/2020 17:57:04", "content": "

Cool, thanks. I ended up with sql.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1061367": "Does anybody know how to do cumum by group and [tail](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.tail.html) and [head](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.head.html) by group, where a specified number of rows within each group are returned, using cudf? I looked at the [api](https://docs.rapids.ai/api/cudf/stable/api.html?highlight=groupby#groupby) and those methods don't exist. [Cumsum is being worked on](https://github.com/rapidsai/cudf/issues/1298), but was curious if anyone had any ideas on how to use existing methods to achieve the same results.", "1077067": "How about these??\n\n`\n\n import cudf\n from numba import cuda\n\n #pandas\n df = pd.DataFrame({'group' : [1,1,2,3,3,3,3,2,2,4,4,4] , 'value' : [1,2,1,1,3,2,4,2,1,3,2,1]} )\n df[\"cumcount_pd\"] = df.groupby(\"group\").cumcount()\n df[\"cumsum_pd\"] = df.groupby(\"group\")[\"value\"].cumsum()\n print(df)\n\n\n #cudf\n df = cudf.DataFrame(df)\n\n # Define a function to apply to each group\n def cudf_cum(value, cumcount_cudf, cumsum_cudf):\n\n for i in range(cuda.threadIdx.x, len(value), cuda.blockDim.x):\n #print('thread_id:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x, 'array size:', value.size, 'block threads:', cuda.blockDim.x)\n \n \n cumcount_cudf[i] = i\n cumsum_cudf[i] = 0\n for j in range(i+1):\n\n cumsum_cudf[i] += value[j]\n\n \n\n df = df.groupby(\"group\", method=\"cudf\").apply_grouped(cudf_cum,\n incols=[\"value\"],\n outcols={'cumcount_cudf': np.int64,\n 'cumsum_cudf': np.int64,\n },\n # threads per block\n tpb=8)\n\n ######################\n head_cudf = df.groupby(\"group\").nth(0) #head\n tail_cudf = df.groupby(\"group\").nth(-1) #tail\n\n\n`", "1077168": "this cumsum code is very slow...", "1079176": "Cool, thanks. I ended up with sql." }, "source": "meta" }