I understand that my Kaggle Hard drive space is limited to 5GB.\nAs you can see in the below image its stops by only executing 13-15 species.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3004733%2F4f7a128272a431a9905da7874bd1426b%2FScreenshot%202020-08-04%20at%2011.20.15%20PM.png?generation=1596563454787054&alt=media\")
I am working to create a Dataset with noise removed using this code . I am already using 5-sliced Data (of original data) tried running in different notebooks. Even tried to slice the data into 15 equal quantity dataset even then it is going out of memory. I understand that my Kaggle Hard drive space is limited to 5GB.\nAs you can see in the below image its stops by only executing 13-15 species.\n
Also in other competition I faced similar problem. \nAny guidance or help from this amazing platform will be great.\nThank you in advance.
", "messages": [ { "id": "958028", "postDate": "08/04/2020 17:46:39", "content": "I am working to create a Dataset with noise removed using this code . I am already using 5-sliced Data (of original data) tried running in different notebooks. Even tried to slice the data into 15 equal quantity dataset even then it is going out of memory. I understand that my Kaggle Hard drive space is limited to 5GB.\nAs you can see in the below image its stops by only executing 13-15 species.\n
Also in other competition I faced similar problem. \nAny guidance or help from this amazing platform will be great.\nThank you in advance.
", "rawMarkdown": "I am working to create a Dataset with noise removed using this [code](https://www.kaggle.com/jainarindam/imp-remove-background-dead-noise) . I am already using 5-sliced Data (of original data) tried running in different notebooks. Even tried to slice the data into 15 equal quantity dataset even then it is going out of memory. `I understand that my Kaggle Hard drive space is limited to 5GB. `\nAs you can see in the below image its stops by only executing 13-15 species.\n\n\nAlso in other competition I faced similar problem. \nAny guidance or help from this amazing platform will be great.\nThank you in advance.", "votes": null }, { "id": "958068", "postDate": "08/04/2020 18:36:07", "content": "try colab pro to have better memory
", "rawMarkdown": "try colab pro to have better memory", "votes": null }, { "id": "958071", "postDate": "08/04/2020 18:39:54", "content": "I don't have colab pro but still can try Colab. Thanks for suggestion @doanquanvietnamca \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3004733%2F0702f455c48a7d30a196d6c892d27364%2FScreenshot%202020-08-05%20at%2012.08.15%20AM.png?generation=1596566382654941&alt=media\")
have trick to use 25gb GPU with colab. You can load from COLAB PRO notebook, then change it and run. It will not save your code but you can download it.
", "rawMarkdown": "have trick to use 25gb GPU with colab. You can load from COLAB PRO notebook, then change it and run. It will not save your code but you can download it.", "votes": null }, { "id": "958089", "postDate": "08/04/2020 18:59:05", "content": "@doanquanvietnamca pardon me but I didn't understand your trick.\nHowever I found this blog https://towardsdatascience.com/upgrade-your-memory-on-google-colab-for-free-1b8b18e8791d \nI hope almighty colab will save me now 😎
", "rawMarkdown": "doanquanvietnamca pardon me but I didn't understand your trick.\nHowever I found this blog https://towardsdatascience.com/upgrade-your-memory-on-google-colab-for-free-1b8b18e8791d \nI hope almighty colab will save me now 😎", "votes": null }, { "id": "958091", "postDate": "08/04/2020 19:00:36", "content": "It's fixed. I think cannot use
", "rawMarkdown": "It's fixed. I think cannot use", "votes": null }, { "id": "958100", "postDate": "08/04/2020 19:04:14", "content": "Oh! @doanquanvietnamca Can you provide more details on your trick. Sorry I didn't understand it
", "rawMarkdown": "Oh! @doanquanvietnamca Can you provide more details on your trick. Sorry I didn't understand it", "votes": null }, { "id": "959846", "postDate": "08/06/2020 00:53:21", "content": "pandas rolling has a potential memory leak. Try to use scipy.ndimage.maximum_filter1d.
", "rawMarkdown": "pandas rolling has a potential memory leak. Try to use scipy.ndimage.maximum_filter1d.", "votes": null }, { "id": "960100", "postDate": "08/06/2020 06:20:54", "content": "@qiuosier Thanks for the alternate functions :)\nbtw it doesn’t necessarily mean “potential memory leak. It could also mean that, there are some objects that are still not cleaned up by Garbage Cleaner (GC).\nEdit:\nThe GC seems to be working fine, but it’s not able to clean up the objects as fast as it’s required in this case.
A memory leak on rolling max was confirmed on v1.0.1 and fixed on v1.0.4. You can get more details here:\nhttps://github.com/pandas-dev/pandas/issues/32266
\n\nIt is \"potential\" because I don't know which version you are using. If you are using any version before the fix, your problem is very likely caused by the memory leak.\nGC is pretty reliable and fast. In this case, GC won't clean up the objects even if you stop the training. You should be able to observe that by checking the memory usage.
", "rawMarkdown": "A memory leak on rolling max was confirmed on v1.0.1 and fixed on v1.0.4. You can get more details here:\n[https://github.com/pandas-dev/pandas/issues/32266](https://github.com/pandas-dev/pandas/issues/32266)\n\nIt is \"potential\" because I don't know which version you are using. If you are using any version before the fix, your problem is very likely caused by the memory leak.\nGC is pretty reliable and fast. In this case, GC won't clean up the objects even if you stop the training. You should be able to observe that by checking the memory usage.", "votes": null } ], "comments": [ { "id": 958068, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "08/04/2020 18:36:07", "content": "try colab pro to have better memory
", "votes": null, "replies": [ { "id": 958071, "author_name": "jainarindam", "author_url": "", "post_date": "08/04/2020 18:39:54", "content": "I don't have colab pro but still can try Colab. Thanks for suggestion @doanquanvietnamca \n
have trick to use 25gb GPU with colab. You can load from COLAB PRO notebook, then change it and run. It will not save your code but you can download it.
", "votes": null, "replies": [] }, { "id": 958089, "author_name": "jainarindam", "author_url": "", "post_date": "08/04/2020 18:59:05", "content": "@doanquanvietnamca pardon me but I didn't understand your trick.\nHowever I found this blog https://towardsdatascience.com/upgrade-your-memory-on-google-colab-for-free-1b8b18e8791d \nI hope almighty colab will save me now 😎
", "votes": null, "replies": [] }, { "id": 958091, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "08/04/2020 19:00:36", "content": "It's fixed. I think cannot use
", "votes": null, "replies": [] }, { "id": 958100, "author_name": "jainarindam", "author_url": "", "post_date": "08/04/2020 19:04:14", "content": "Oh! @doanquanvietnamca Can you provide more details on your trick. Sorry I didn't understand it
", "votes": null, "replies": [] } ] }, { "id": 959846, "author_name": "qiuosier", "author_url": "", "post_date": "08/06/2020 00:53:21", "content": "pandas rolling has a potential memory leak. Try to use scipy.ndimage.maximum_filter1d.
", "votes": null, "replies": [ { "id": 960100, "author_name": "jainarindam", "author_url": "", "post_date": "08/06/2020 06:20:54", "content": "@qiuosier Thanks for the alternate functions :)\nbtw it doesn’t necessarily mean “potential memory leak. It could also mean that, there are some objects that are still not cleaned up by Garbage Cleaner (GC).\nEdit:\nThe GC seems to be working fine, but it’s not able to clean up the objects as fast as it’s required in this case.
A memory leak on rolling max was confirmed on v1.0.1 and fixed on v1.0.4. You can get more details here:\nhttps://github.com/pandas-dev/pandas/issues/32266
\n\nIt is \"potential\" because I don't know which version you are using. If you are using any version before the fix, your problem is very likely caused by the memory leak.\nGC is pretty reliable and fast. In this case, GC won't clean up the objects even if you stop the training. You should be able to observe that by checking the memory usage.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "958028": "I am working to create a Dataset with noise removed using this [code](https://www.kaggle.com/jainarindam/imp-remove-background-dead-noise) . I am already using 5-sliced Data (of original data) tried running in different notebooks. Even tried to slice the data into 15 equal quantity dataset even then it is going out of memory. `I understand that my Kaggle Hard drive space is limited to 5GB. `\nAs you can see in the below image its stops by only executing 13-15 species.\n\n\nAlso in other competition I faced similar problem. \nAny guidance or help from this amazing platform will be great.\nThank you in advance.", "958068": "try colab pro to have better memory", "958071": "I don't have colab pro but still can try Colab. Thanks for suggestion @doanquanvietnamca \n", "958081": "have trick to use 25gb GPU with colab. You can load from COLAB PRO notebook, then change it and run. It will not save your code but you can download it.", "958089": "doanquanvietnamca pardon me but I didn't understand your trick.\nHowever I found this blog https://towardsdatascience.com/upgrade-your-memory-on-google-colab-for-free-1b8b18e8791d \nI hope almighty colab will save me now 😎", "958091": "It's fixed. I think cannot use", "958100": "Oh! @doanquanvietnamca Can you provide more details on your trick. Sorry I didn't understand it", "959846": "pandas rolling has a potential memory leak. Try to use scipy.ndimage.maximum_filter1d.", "960100": "qiuosier Thanks for the alternate functions :)\nbtw it doesn’t necessarily mean “potential memory leak. It could also mean that, there are some objects that are still not cleaned up by Garbage Cleaner (GC).\nEdit:\n`The GC seems to be working fine, but it’s not able to clean up the objects as fast as it’s required in this case.`", "960822": "A memory leak on rolling max was confirmed on v1.0.1 and fixed on v1.0.4. You can get more details here:\n[https://github.com/pandas-dev/pandas/issues/32266](https://github.com/pandas-dev/pandas/issues/32266)\n\nIt is \"potential\" because I don't know which version you are using. If you are using any version before the fix, your problem is very likely caused by the memory leak.\nGC is pretty reliable and fast. In this case, GC won't clean up the objects even if you stop the training. You should be able to observe that by checking the memory usage." }, "source": "meta" }