Dear Competition Managers,
\n\nAny chance you could please expose the following for the submission runs:\n1- GPU compute usage,\n2- CPU compute usage,\n3- RAM usage\n4- GPU RAM usage\n5- HDD usage
\n\nfor submissions.
\n\nI don't need more than that to understand the issue that I'm facing. My code doesn't store arrays in memory and I actively collect garbage to make sure that I keep the memory in check.
\n\nIt would be amazing if you can do that every 5 mins for example to show over time and you could use max.
\n\nAnother thing, I have ran this submission before and it worked with no issue. The only difference is something related to the calculation that I use for labeling. It shouldn't affect the memory usage.
\n\nCould you please help?
\n\nRegards,\nAmro
", "messages": [ { "id": 755120, "postDate": "2020-02-24T13:11:16.690Z", "content": "We have faced the same error recently, for 3 consecutive submissions. It disappeared after few changes, but it was not clear what caused it and why it disappeared. The line of debugging for us was running the kernel in interactive mode. It shows resources utilization, so can be helpful. In our case we saw RAM usage growing over time, but the changes that we made did not solve it, so I am still clueless what was it.
", "rawMarkdown": "We have faced the same error recently, for 3 consecutive submissions. It disappeared after few changes, but it was not clear what caused it and why it disappeared. The line of debugging for us was running the kernel in interactive mode. It shows resources utilization, so can be helpful. In our case we saw RAM usage growing over time, but the changes that we made did not solve it, so I am still clueless what was it. ", "votes": 1, "replies": [ { "id": 755123, "postDate": "2020-02-24T13:21:49.543Z", "content": "I did that and I'm pretty sure that I don't use more than 12GB of memory in the interactive runs. I assumed that the public validation set is equivalent in size to the public test set used for evaluating the scores.
", "rawMarkdown": "I did that and I'm pretty sure that I don't use more than 12GB of memory in the interactive runs. I assumed that the public validation set is equivalent in size to the public test set used for evaluating the scores." }, { "id": 755129, "postDate": "2020-02-24T13:27:29.457Z", "content": "Thanks @zaharch Appreciate your feedback. Hopefully competition managers have something to say.
", "rawMarkdown": "Thanks @zaharch Appreciate your feedback. Hopefully competition managers have something to say." } ] }, { "id": 755106, "postDate": "2020-02-24T12:47:11.763Z", "content": "Dear Competition Managers,
\n\nAny chance you could please expose the following for the submission runs:\n1- GPU compute usage,\n2- CPU compute usage,\n3- RAM usage\n4- GPU RAM usage\n5- HDD usage
\n\nfor submissions.
\n\nI don't need more than that to understand the issue that I'm facing. My code doesn't store arrays in memory and I actively collect garbage to make sure that I keep the memory in check.
\n\nIt would be amazing if you can do that every 5 mins for example to show over time and you could use max.
\n\nAnother thing, I have ran this submission before and it worked with no issue. The only difference is something related to the calculation that I use for labeling. It shouldn't affect the memory usage.
\n\nCould you please help?
\n\nRegards,\nAmro
", "rawMarkdown": "Dear Competition Managers,\n\nAny chance you could please expose the following for the submission runs:\n1- GPU compute usage,\n2- CPU compute usage,\n3- RAM usage\n4- GPU RAM usage\n5- HDD usage \n\nfor submissions. \n\nI don't need more than that to understand the issue that I'm facing. My code doesn't store arrays in memory and I actively collect garbage to make sure that I keep the memory in check.\n\nIt would be amazing if you can do that every 5 mins for example to show over time and you could use max.\n\nAnother thing, I have ran this submission before and it worked with no issue. The only difference is something related to the calculation that I use for labeling. It shouldn't affect the memory usage.\n\nCould you please help?\n\nRegards,\nAmro", "votes": 1 }, { "id": 755532, "postDate": "2020-02-24T21:31:42.657Z", "content": "Same. For each batch I have cleared using pytorch empty_cache, gc.collect(). Still having issues of running out of memory.
", "rawMarkdown": "Same. For each batch I have cleared using pytorch empty_cache, gc.collect(). Still having issues of running out of memory." }, { "id": 755121, "postDate": "2020-02-24T13:19:50.070Z", "rawMarkdown": "", "isDeleted": true }, { "id": 755108, "postDate": "2020-02-24T12:49:07.403Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 755120, "author_name": "nosound", "author_url": "", "post_date": "2020-02-24T13:11:16.690000", "content": "We have faced the same error recently, for 3 consecutive submissions. It disappeared after few changes, but it was not clear what caused it and why it disappeared. The line of debugging for us was running the kernel in interactive mode. It shows resources utilization, so can be helpful. In our case we saw RAM usage growing over time, but the changes that we made did not solve it, so I am still clueless what was it.
", "votes": 1, "replies": [ { "id": 755123, "author_name": "Amro", "author_url": "", "post_date": "2020-02-24T13:21:49.543000", "content": "I did that and I'm pretty sure that I don't use more than 12GB of memory in the interactive runs. I assumed that the public validation set is equivalent in size to the public test set used for evaluating the scores.
", "votes": 0, "replies": [] }, { "id": 755129, "author_name": "Amro", "author_url": "", "post_date": "2020-02-24T13:27:29.457000", "content": "Thanks @zaharch Appreciate your feedback. Hopefully competition managers have something to say.
", "votes": 0, "replies": [] } ] }, { "id": 755532, "author_name": "Thomas Sherk", "author_url": "", "post_date": "2020-02-24T21:31:42.657000", "content": "Same. For each batch I have cleared using pytorch empty_cache, gc.collect(). Still having issues of running out of memory.
", "votes": 0, "replies": [] }, { "id": 755121, "author_name": "", "author_url": "", "post_date": "2020-02-24T13:19:50.070000", "content": "", "votes": 0, "replies": [] }, { "id": 755108, "author_name": "", "author_url": "", "post_date": "2020-02-24T12:49:07.403000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "755120": "We have faced the same error recently, for 3 consecutive submissions. It disappeared after few changes, but it was not clear what caused it and why it disappeared. The line of debugging for us was running the kernel in interactive mode. It shows resources utilization, so can be helpful. In our case we saw RAM usage growing over time, but the changes that we made did not solve it, so I am still clueless what was it. ", "755106": "Dear Competition Managers,\n\nAny chance you could please expose the following for the submission runs:\n1- GPU compute usage,\n2- CPU compute usage,\n3- RAM usage\n4- GPU RAM usage\n5- HDD usage \n\nfor submissions. \n\nI don't need more than that to understand the issue that I'm facing. My code doesn't store arrays in memory and I actively collect garbage to make sure that I keep the memory in check.\n\nIt would be amazing if you can do that every 5 mins for example to show over time and you could use max.\n\nAnother thing, I have ran this submission before and it worked with no issue. The only difference is something related to the calculation that I use for labeling. It shouldn't affect the memory usage.\n\nCould you please help?\n\nRegards,\nAmro", "755532": "Same. For each batch I have cleared using pytorch empty_cache, gc.collect(). Still having issues of running out of memory.", "755121": "", "755108": "" } }