![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16265464%2F2fc86baa7bd1b04d5edb2b1a86d73243%2Fphoto_2024-06-18_10-31-10.jpg?generation=1718696219062757&alt=media\")
Faced a problem with RAM usage.
\nAll information about any patent is contained in parquet in the patent_data/ folder. When working with test data, we will not be able to upload all the information about each unique patent into RAM (we will not have enough of RAM). Therefore, 'as I understand it' we will have to work with patents step by step (we have processed 1 patent -> move to the next one). I don't quite understand how RAM works.
\nHere's a link to a notebook https://www.kaggle.com/code/qurusx/memory-issue where I elementary read information about each parquet in patent_data/. Why is RAM constantly increasing, eventually we end up with an excess of available RAM if we read the parquet -> delete the reference (variable) to it -> do rubbish collection (gc.collect). The most weighty file is 2023_6.parquet (1.09 GB In folder -> 11.3 GB RAM), so any parquet can be loaded into RAM.
Is there any explanation for this? (RAM work in Python)
\nUsing only pandas (without polars and other libraries), is it possible to get information about the needed patent faster and less memory consuming than next method. Get publication_number in test.csv -> get publication_date in patent_metadata.parquet -> get info in patent_data/publication_date.parquet (Load full parquet)?
\nThanks for any help, explanations, links
Faced a problem with RAM usage.
\nAll information about any patent is contained in parquet in the patent_data/ folder. When working with test data, we will not be able to upload all the information about each unique patent into RAM (we will not have enough of RAM). Therefore, 'as I understand it' we will have to work with patents step by step (we have processed 1 patent -> move to the next one). I don't quite understand how RAM works.
\nHere's a link to a notebook https://www.kaggle.com/code/qurusx/memory-issue where I elementary read information about each parquet in patent_data/. Why is RAM constantly increasing, eventually we end up with an excess of available RAM if we read the parquet -> delete the reference (variable) to it -> do rubbish collection (gc.collect). The most weighty file is 2023_6.parquet (1.09 GB In folder -> 11.3 GB RAM), so any parquet can be loaded into RAM.
Is there any explanation for this? (RAM work in Python)
\nUsing only pandas (without polars and other libraries), is it possible to get information about the needed patent faster and less memory consuming than next method. Get publication_number in test.csv -> get publication_date in patent_metadata.parquet -> get info in patent_data/publication_date.parquet (Load full parquet)?
\nThanks for any help, explanations, links
Garbage collector in Python doesn't always fully release unused memory. There are ways to deal with it, e.g.
\nlibc = ctypes.CDLL(\"libc.so.6\") and then libc.malloc_trim(0)
You may read the files by chunks so that they definitely fit into the memory. Even better yet, you may read only required columns, that will improve both memory usage and performance.
", "rawMarkdown": "Garbage collector in Python doesn't always fully release unused memory. There are ways to deal with it, e.g. \nlibc = ctypes.CDLL(\"libc.so.6\") and then libc.malloc_trim(0)\n\nYou may read the files by chunks so that they definitely fit into the memory. Even better yet, you may read only required columns, that will improve both memory usage and performance.", "votes": null }, { "id": "2878601", "postDate": "06/19/2024 04:49:45", "content": "Thanks, I'll try to apply the memory tip.
\nIf you mean reading the parquet line by line or chunk by chunk, it will take a long time to find the right line. I usually load the whole parquet, set 'publication_number' as an index and quickly find the required row by index. Doesn't seem to be a faster way?
\nAs for columns, I want to work not only with title and abstract. As far as I know, loading only 'publication_number' column and making it an index, we can't load the rest of information (claims, description etc.) for the line we NEED. That's why we have to load the whole parquet.
", "rawMarkdown": "Thanks, I'll try to apply the memory tip.\n\nIf you mean reading the parquet line by line or chunk by chunk, it will take a long time to find the right line. I usually load the whole parquet, set 'publication_number' as an index and quickly find the required row by index. Doesn't seem to be a faster way?\n\nAs for columns, I want to work not only with title and abstract. As far as I know, loading only 'publication_number' column and making it an index, we can't load the rest of information (claims, description etc.) for the line we NEED. That's why we have to load the whole parquet.", "votes": null }, { "id": "2878909", "postDate": "06/19/2024 08:35:01", "content": "Try using polars it will reduce the memory ,As I faced this issue at the beginning of the competition , And of course this competition needs a good knowledge of how to deal with memory
", "rawMarkdown": "Try using polars it will reduce the memory ,As I faced this issue at the beginning of the competition , And of course this competition needs a good knowledge of how to deal with memory", "votes": null }, { "id": "2878917", "postDate": "06/19/2024 08:42:40", "content": "For example, there is a way to split publication_date.parquet into smaller subfiles and save them, and refer to the dictionary of patent->filename.
\npublication_number -> dict_patent2file -> smaller_target_file
", "rawMarkdown": "For example, there is a way to split publication_date.parquet into smaller subfiles and save them, and refer to the dictionary of patent->filename.\n\npublication_number -> dict_patent2file -> smaller_target_file", "votes": null } ], "comments": [ { "id": 2878565, "author_name": "apparition", "author_url": "", "post_date": "06/19/2024 04:26:48", "content": "Garbage collector in Python doesn't always fully release unused memory. There are ways to deal with it, e.g.
\nlibc = ctypes.CDLL(\"libc.so.6\") and then libc.malloc_trim(0)
You may read the files by chunks so that they definitely fit into the memory. Even better yet, you may read only required columns, that will improve both memory usage and performance.
", "votes": null, "replies": [ { "id": 2878601, "author_name": "qurusx", "author_url": "", "post_date": "06/19/2024 04:49:45", "content": "Thanks, I'll try to apply the memory tip.
\nIf you mean reading the parquet line by line or chunk by chunk, it will take a long time to find the right line. I usually load the whole parquet, set 'publication_number' as an index and quickly find the required row by index. Doesn't seem to be a faster way?
\nAs for columns, I want to work not only with title and abstract. As far as I know, loading only 'publication_number' column and making it an index, we can't load the rest of information (claims, description etc.) for the line we NEED. That's why we have to load the whole parquet.
", "votes": null, "replies": [] } ] }, { "id": 2878909, "author_name": "mahmoudsaber", "author_url": "", "post_date": "06/19/2024 08:35:01", "content": "Try using polars it will reduce the memory ,As I faced this issue at the beginning of the competition , And of course this competition needs a good knowledge of how to deal with memory
", "votes": null, "replies": [] }, { "id": 2878917, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "06/19/2024 08:42:40", "content": "For example, there is a way to split publication_date.parquet into smaller subfiles and save them, and refer to the dictionary of patent->filename.
\npublication_number -> dict_patent2file -> smaller_target_file
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2877119": "Faced a problem with RAM usage.\n\n# Get information about each patent\nAll information about any patent is contained in parquet in the patent_data/ folder. When working with test data, we will not be able to upload all the information about each unique patent into RAM (we will not have enough of RAM). Therefore, 'as I understand it' we will have to work with patents step by step (we have processed 1 patent -> move to the next one). I don't quite understand how RAM works.\n\n# Notebook issue\nHere's a link to a notebook [https://www.kaggle.com/code/qurusx/memory-issue](url) where I elementary read information about each parquet in patent_data/. Why is RAM constantly increasing, eventually we end up with an excess of available RAM if we read the parquet -> delete the reference (variable) to it -> do rubbish collection (gc.collect). The most weighty file is 2023_6.parquet (1.09 GB In folder -> 11.3 GB RAM), so any parquet can be loaded into RAM.\n\n# Questions\nIs there any explanation for this? (RAM work in Python)\nUsing only pandas (without polars and other libraries), is it possible to get information about the needed patent faster and less memory consuming than next method. Get publication_number in test.csv -> get publication_date in patent_metadata.parquet -> get info in patent_data/publication_date.parquet (Load full parquet)?\n**Thanks for any help, explanations, links**", "2878565": "Garbage collector in Python doesn't always fully release unused memory. There are ways to deal with it, e.g. \nlibc = ctypes.CDLL(\"libc.so.6\") and then libc.malloc_trim(0)\n\nYou may read the files by chunks so that they definitely fit into the memory. Even better yet, you may read only required columns, that will improve both memory usage and performance.", "2878601": "Thanks, I'll try to apply the memory tip.\n\nIf you mean reading the parquet line by line or chunk by chunk, it will take a long time to find the right line. I usually load the whole parquet, set 'publication_number' as an index and quickly find the required row by index. Doesn't seem to be a faster way?\n\nAs for columns, I want to work not only with title and abstract. As far as I know, loading only 'publication_number' column and making it an index, we can't load the rest of information (claims, description etc.) for the line we NEED. That's why we have to load the whole parquet.", "2878909": "Try using polars it will reduce the memory ,As I faced this issue at the beginning of the competition , And of course this competition needs a good knowledge of how to deal with memory", "2878917": "For example, there is a way to split publication_date.parquet into smaller subfiles and save them, and refer to the dictionary of patent->filename.\n\npublication_number -> dict_patent2file -> smaller_target_file" }, "source": "meta" }