Is it possible to parallelize the processing of a list of dataframes received from a polars partition_by? Taking into account the fact that polars already parallelizes calculations, but as I understand it in this context, polars will not parallelize calculations between processing individual dataframes from the list. I would like to speed up dataframe processing.
\ndef processing_df(df):\n ...\n ...\n return result # np.array for example\n\ndf_gr = df_sourse.partition_by(groups=\"group_col\", maintain_order=True)\n\nresults = []\nfor df in df_gr:\n res = processing_df(df)\n results.append(res)\nP.S. I tried use this approaches, but processes were not running.
", "messages": [ { "id": "2149895", "postDate": "02/18/2023 19:11:49", "content": "Is it possible to parallelize the processing of a list of dataframes received from a polars partition_by? Taking into account the fact that polars already parallelizes calculations, but as I understand it in this context, polars will not parallelize calculations between processing individual dataframes from the list. I would like to speed up dataframe processing.
\ndef processing_df(df):\n ...\n ...\n return result # np.array for example\n\ndf_gr = df_sourse.partition_by(groups=\"group_col\", maintain_order=True)\n\nresults = []\nfor df in df_gr:\n res = processing_df(df)\n results.append(res)\nP.S. I tried use this approaches, but processes were not running.
", "rawMarkdown": "Is it possible to parallelize the processing of a list of dataframes received from a polars partition_by? Taking into account the fact that polars already parallelizes calculations, but as I understand it in this context, polars will not parallelize calculations between processing individual dataframes from the list. I would like to speed up dataframe processing.\n\n```\ndef processing_df(df):\n ...\n ...\n return result # np.array for example\n\ndf_gr = df_sourse.partition_by(groups=\"group_col\", maintain_order=True)\n\nresults = []\nfor df in df_gr:\n res = processing_df(df)\n results.append(res)\n```\n\nP.S. I tried use [this approaches](https://www.machinelearningplus.com/python/parallel-processing-python/), but processes were not running.", "votes": null }, { "id": "2149899", "postDate": "02/18/2023 19:14:24", "content": "It is impossible to cope only with aggregation functions, since I use not only them
", "rawMarkdown": "It is impossible to cope only with aggregation functions, since I use not only them", "votes": null }, { "id": "2149961", "postDate": "02/18/2023 20:26:52", "content": "To parallelize the processing of multiple partitioned dataframes using Polars partition_by function, you can make use of Python's built-in multiprocessing module.
\nHere's a high-level overview of the approach:
\nCreate a list of the partitioned dataframes using the partition_by function.
\nSplit the list of dataframes into smaller chunks, based on the number of CPU cores available or some other factor.
\nUse the multiprocessing.Pool class to create a pool of worker processes.
\nUse the Pool.map method to apply a function to each chunk of dataframes in parallel.
\nFinally, use the concat method to combine the results from the processed dataframes into a single dataframe.
Here's some sample code that demonstrates this approach:
\n polars pl\n multiprocessing mp\n\n\ndf = pl.DataFrame({\n : [, , , , , ],\n : [, , , , , ]\n})\n\n\npartitions = df.partition_by()\n\n\n ():\n \n result = partitions.select(, ).groupby().()\n result\n\n\nchunk_size = (partitions) // mp.cpu_count()\npartition_chunks = [partitions[i:i+chunk_size] i (, (partitions), chunk_size)]\n\n\npool = mp.Pool()\n\n\nresults = pool.(process_partitions, partition_chunks)\n\n\nfinal_result = pl.concat(results)\nIn this example, the process_partitions function is applied to each chunk of partitions in parallel using the Pool.map method. The resulting dataframes are concatenated into a single dataframe using the concat method.
", "rawMarkdown": "To parallelize the processing of multiple partitioned dataframes using Polars partition_by function, you can make use of Python's built-in multiprocessing module.\n\nHere's a high-level overview of the approach:\n\nCreate a list of the partitioned dataframes using the partition_by function.\nSplit the list of dataframes into smaller chunks, based on the number of CPU cores available or some other factor.\nUse the multiprocessing.Pool class to create a pool of worker processes.\nUse the Pool.map method to apply a function to each chunk of dataframes in parallel.\nFinally, use the concat method to combine the results from the processed dataframes into a single dataframe.\n\nHere's some sample code that demonstrates this approach:\n```python\n\nimport polars as pl\nimport multiprocessing as mp\n\n# create a DataFrame\ndf = pl.DataFrame({\n 'foo': ['A', 'A', 'B', 'B', 'C', 'C'],\n 'bar': [1, 2, 3, 4, 5, 6]\n})\n\n# partition the DataFrame by 'foo'\npartitions = df.partition_by('foo')\n\n# define a function to process a chunk of partitions\ndef process_partitions(partitions):\n # do some processing on the partitions\n result = partitions.select('foo', 'bar').groupby('foo').sum()\n return result\n\n# split the list of partitions into chunks\nchunk_size = len(partitions) // mp.cpu_count()\npartition_chunks = [partitions[i:i+chunk_size] for i in range(0, len(partitions), chunk_size)]\n\n# create a pool of worker processes\npool = mp.Pool()\n\n# apply the function to each chunk of partitions in parallel\nresults = pool.map(process_partitions, partition_chunks)\n\n# concatenate the results into a single dataframe\nfinal_result = pl.concat(results)\n```\n\n\nIn this example, the process_partitions function is applied to each chunk of partitions in parallel using the Pool.map method. The resulting dataframes are concatenated into a single dataframe using the concat method.", "votes": null }, { "id": "2150346", "postDate": "02/19/2023 07:09:55", "content": "Thanks you very much for the answer! I tried to use this method, but the functions were not executed. I now tried the method with joblib Parallel, delayed - this way it worked
\nfrom joblib import Parallel, delayed\nimport polars as pl\n\ndf = pl.DataFrame({\n 'foo': ['A', 'A', 'B', 'B', 'C', 'C'],\n 'bar': [1, 2, 3, 4, 5, 6]\n})\n\n# partition the DataFrame by 'foo'\npartitions = df.partition_by('foo')\n\n# define a function to process a chunk of partitions\ndef process_partitions(partitions):\n # do some processing on the partitions\n result = [part.select('foo', 'bar').groupby('foo').sum() for part in partitions]\n return result\n\n# split the list of partitions into chunks\nchunk_size = len(partitions) // mp.cpu_count()\npartition_chunks = [partitions[i:i+chunk_size] for i in range(0, len(partitions), chunk_size)]\n\nres = Parallel(\n n_jobs=5\n)(\n delayed(process_partitions)(chunk) for chink in partition_chunks\n)\nIt is impossible to cope only with aggregation functions, since I use not only them
", "votes": null, "replies": [] }, { "id": 2149961, "author_name": "soheiltehranipour", "author_url": "", "post_date": "02/18/2023 20:26:52", "content": "To parallelize the processing of multiple partitioned dataframes using Polars partition_by function, you can make use of Python's built-in multiprocessing module.
\nHere's a high-level overview of the approach:
\nCreate a list of the partitioned dataframes using the partition_by function.
\nSplit the list of dataframes into smaller chunks, based on the number of CPU cores available or some other factor.
\nUse the multiprocessing.Pool class to create a pool of worker processes.
\nUse the Pool.map method to apply a function to each chunk of dataframes in parallel.
\nFinally, use the concat method to combine the results from the processed dataframes into a single dataframe.
Here's some sample code that demonstrates this approach:
\n polars pl\n multiprocessing mp\n\n\ndf = pl.DataFrame({\n : [, , , , , ],\n : [, , , , , ]\n})\n\n\npartitions = df.partition_by()\n\n\n ():\n \n result = partitions.select(, ).groupby().()\n result\n\n\nchunk_size = (partitions) // mp.cpu_count()\npartition_chunks = [partitions[i:i+chunk_size] i (, (partitions), chunk_size)]\n\n\npool = mp.Pool()\n\n\nresults = pool.(process_partitions, partition_chunks)\n\n\nfinal_result = pl.concat(results)\nIn this example, the process_partitions function is applied to each chunk of partitions in parallel using the Pool.map method. The resulting dataframes are concatenated into a single dataframe using the concat method.
", "votes": null, "replies": [ { "id": 2150346, "author_name": "dmdgik", "author_url": "", "post_date": "02/19/2023 07:09:55", "content": "Thanks you very much for the answer! I tried to use this method, but the functions were not executed. I now tried the method with joblib Parallel, delayed - this way it worked
\nfrom joblib import Parallel, delayed\nimport polars as pl\n\ndf = pl.DataFrame({\n 'foo': ['A', 'A', 'B', 'B', 'C', 'C'],\n 'bar': [1, 2, 3, 4, 5, 6]\n})\n\n# partition the DataFrame by 'foo'\npartitions = df.partition_by('foo')\n\n# define a function to process a chunk of partitions\ndef process_partitions(partitions):\n # do some processing on the partitions\n result = [part.select('foo', 'bar').groupby('foo').sum() for part in partitions]\n return result\n\n# split the list of partitions into chunks\nchunk_size = len(partitions) // mp.cpu_count()\npartition_chunks = [partitions[i:i+chunk_size] for i in range(0, len(partitions), chunk_size)]\n\nres = Parallel(\n n_jobs=5\n)(\n delayed(process_partitions)(chunk) for chink in partition_chunks\n)\n