I've a working inference kernel, I've updated/uploaded the weights of my model and I got \"Submission CSV Not Found\" error. When I look at diff between yesterday working version and today's broken version I can confirm that only path to weights has been updated. Am I alone to get this issue today? I've tried to re-submit and I've got the same error. It does not make sense, weights update cannot break the pipeline and the submission.csv and I can see it's generated on commit.
", "messages": [ { "id": 1183015, "postDate": "2021-02-02T17:31:03.437Z", "content": "I've a working inference kernel, I've updated/uploaded the weights of my model and I got \"Submission CSV Not Found\" error. When I look at diff between yesterday working version and today's broken version I can confirm that only path to weights has been updated. Am I alone to get this issue today? I've tried to re-submit and I've got the same error. It does not make sense, weights update cannot break the pipeline and the submission.csv and I can see it's generated on commit.
", "rawMarkdown": "I've a working inference kernel, I've updated/uploaded the weights of my model and I got \"**Submission CSV Not Found**\" error. When I look at diff between yesterday working version and today's broken version I can confirm that only path to weights has been updated. Am I alone to get this issue today? I've tried to re-submit and I've got the same error. It does not make sense, weights update cannot break the pipeline and the submission.csv and I can see it's generated on commit. ", "votes": 11 }, { "id": 1215663, "postDate": "2021-02-23T20:59:18.750Z", "content": "Hi @mpware I just experienced the exact same issue that you described…only change in my kernel is that it is a new weights file that is loaded…a follow up version with yet another weights file is working.
\nDoes not seem to happen often though…first time for me.
", "rawMarkdown": "Hi @mpware I just experienced the exact same issue that you described...only change in my kernel is that it is a new weights file that is loaded...a follow up version with yet another weights file is working.\n\nDoes not seem to happen often though...first time for me. \n", "votes": 1, "replies": [ { "id": 1215731, "postDate": "2021-02-23T22:35:49.193Z", "content": "@rsmits Are you using multiple workers? Or multiprocessing? It was the root cause of my problem. It looks RasterIO does not work well with multiple workers, it fails silently (not always) and the submission.csv file is not generated. I've been able to reproduce on commit. I'm using workers=0 now and it works.
Hi @mpware I'am using the default workers setting which I believe is 0. So that's likely not the cause of my issue…
\nI'am retrying it with the model file again uploaded. So far so good..another model file from the same training run (different fold) did work so I thought I could try that.
\nSeems to keep working although it will be a few hours before I can see the final end result.
@mpware Hope, so you will resolve this soon.
", "rawMarkdown": "@mpware Hope, so you will resolve this soon.\n\n", "votes": 2 }, { "id": 1183959, "postDate": "2021-02-03T10:15:03.067Z", "content": "Hope you will resolve soon.
", "rawMarkdown": "Hope you will resolve soon. ", "votes": 2 }, { "id": 1183525, "postDate": "2021-02-03T04:14:54.847Z", "content": "Today I've succeeded in submitting my kernel to get a score, so there seems to be little Kaggle scoring issue.
", "rawMarkdown": "Today I've succeeded in submitting my kernel to get a score, so there seems to be little Kaggle scoring issue.", "votes": 2, "replies": [ { "id": 1183910, "postDate": "2021-02-03T09:45:03.157Z", "content": "Thanks, I was also able to have successfull submissions in the mean time. However, it looks I'm able to reproduce the problem now on commit. Kernel succeeds (but dies indeed) and logs report:
\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/asynchronous/channels.py\", line 46, in get_msg\nreturn await self._recv()\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/asynchronous/channels.py\", line 37, in _recv\nreturn self.session.deserialize(smsg)\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/session.py\", line 921, in deserialize\nraise ValueError(\"Duplicate Signature: %r\" % signature\nI suspect a concurrent threading issue with RasterIO. I'm trying to use the 2 CPUs to speed up processing but it fails on some race conditions. I'm using thread lock but it does not seem to work as expected. If I commit again then it works. Same issue might happen on submit.
", "rawMarkdown": "Thanks, I was also able to have successfull submissions in the mean time. However, it looks I'm able to reproduce the problem now on commit. Kernel succeeds (but dies indeed) and logs report:\n\n```\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/asynchronous/channels.py\", line 46, in get_msg\nreturn await self._recv()\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/asynchronous/channels.py\", line 37, in _recv\nreturn self.session.deserialize(smsg)\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/session.py\", line 921, in deserialize\nraise ValueError(\"Duplicate Signature: %r\" % signature\n```\nI suspect a concurrent threading issue with RasterIO. I'm trying to use the 2 CPUs to speed up processing but it fails on some race conditions. I'm using thread lock but it does not seem to work as expected. If I commit again then it works. Same issue might happen on submit.", "votes": 4 }, { "id": 1187188, "postDate": "2021-02-05T09:17:19.277Z", "content": "Confirmed, so it was me. No problem with Kaggle environment.
", "rawMarkdown": "Confirmed, so it was me. No problem with Kaggle environment.", "votes": 3 }, { "id": 1191868, "postDate": "2021-02-08T18:27:03.333Z", "content": "RasterIO documentation claims it would work:
\nhttps://rasterio.readthedocs.io/en/latest/topics/concurrency.html
with rasterio.open(outfile, \"w\", **src.profile) as dst:\n windows = [window for ij, window in dst.block_windows()]\n\n # We cannot write to the same file from multiple threads\n # without causing race conditions. To safely read/write\n # from multiple threads, we use a lock to protect the\n # DatasetReader/Writer\n read_lock = threading.Lock()\n write_lock = threading.Lock()\n\n def process(window):\n with read_lock:\n src_array = src.read(window=window)\nBut for some unknown reason it fails randomly. Anyone experimenting similar issue?
\nThis guy had similar problem but not with rasterio:
\nhttps://www.kaggle.com/product-feedback/164002
@mpware Will we get same error in case of oom ? I get sub csv error after 45 to 50 min .
\n. If there is an oom we get with in 30 min or after this much time only?
I'm having the same problem. Have you solved it? how?
", "rawMarkdown": "I'm having the same problem. Have you solved it? how?", "replies": [ { "id": 1277349, "postDate": "2021-04-18T16:30:55.277Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1211404, "postDate": "2021-02-20T07:39:40.203Z", "content": "I think the problem is with your end. Check again once.
", "rawMarkdown": "I think the problem is with your end. Check again once." }, { "id": 1253543, "postDate": "2021-03-26T21:05:35.537Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1215663, "author_name": "Robin Smits", "author_url": "", "post_date": "2021-02-23T20:59:18.750000", "content": "Hi @mpware I just experienced the exact same issue that you described…only change in my kernel is that it is a new weights file that is loaded…a follow up version with yet another weights file is working.
\nDoes not seem to happen often though…first time for me.
", "votes": 1, "replies": [ { "id": 1215731, "author_name": "MPWARE", "author_url": "", "post_date": "2021-02-23T22:35:49.193000", "content": "@rsmits Are you using multiple workers? Or multiprocessing? It was the root cause of my problem. It looks RasterIO does not work well with multiple workers, it fails silently (not always) and the submission.csv file is not generated. I've been able to reproduce on commit. I'm using workers=0 now and it works.
Hi @mpware I'am using the default workers setting which I believe is 0. So that's likely not the cause of my issue…
\nI'am retrying it with the model file again uploaded. So far so good..another model file from the same training run (different fold) did work so I thought I could try that.
\nSeems to keep working although it will be a few hours before I can see the final end result.
@mpware Hope, so you will resolve this soon.
", "votes": 2, "replies": [] }, { "id": 1183959, "author_name": "Adnan Zaidi", "author_url": "", "post_date": "2021-02-03T10:15:03.067000", "content": "Hope you will resolve soon.
", "votes": 2, "replies": [] }, { "id": 1183525, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-02-03T04:14:54.847000", "content": "Today I've succeeded in submitting my kernel to get a score, so there seems to be little Kaggle scoring issue.
", "votes": 2, "replies": [ { "id": 1183910, "author_name": "MPWARE", "author_url": "", "post_date": "2021-02-03T09:45:03.157000", "content": "Thanks, I was also able to have successfull submissions in the mean time. However, it looks I'm able to reproduce the problem now on commit. Kernel succeeds (but dies indeed) and logs report:
\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/asynchronous/channels.py\", line 46, in get_msg\nreturn await self._recv()\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/asynchronous/channels.py\", line 37, in _recv\nreturn self.session.deserialize(smsg)\nFile \"/opt/conda/lib/python3.7/site-packages/jupyter_client/session.py\", line 921, in deserialize\nraise ValueError(\"Duplicate Signature: %r\" % signature\nI suspect a concurrent threading issue with RasterIO. I'm trying to use the 2 CPUs to speed up processing but it fails on some race conditions. I'm using thread lock but it does not seem to work as expected. If I commit again then it works. Same issue might happen on submit.
", "votes": 4, "replies": [] }, { "id": 1187188, "author_name": "MPWARE", "author_url": "", "post_date": "2021-02-05T09:17:19.277000", "content": "Confirmed, so it was me. No problem with Kaggle environment.
", "votes": 3, "replies": [] }, { "id": 1191868, "author_name": "MPWARE", "author_url": "", "post_date": "2021-02-08T18:27:03.333000", "content": "RasterIO documentation claims it would work:
\nhttps://rasterio.readthedocs.io/en/latest/topics/concurrency.html
with rasterio.open(outfile, \"w\", **src.profile) as dst:\n windows = [window for ij, window in dst.block_windows()]\n\n # We cannot write to the same file from multiple threads\n # without causing race conditions. To safely read/write\n # from multiple threads, we use a lock to protect the\n # DatasetReader/Writer\n read_lock = threading.Lock()\n write_lock = threading.Lock()\n\n def process(window):\n with read_lock:\n src_array = src.read(window=window)\nBut for some unknown reason it fails randomly. Anyone experimenting similar issue?
\nThis guy had similar problem but not with rasterio:
\nhttps://www.kaggle.com/product-feedback/164002
@mpware Will we get same error in case of oom ? I get sub csv error after 45 to 50 min .
\n. If there is an oom we get with in 30 min or after this much time only?
I'm having the same problem. Have you solved it? how?
", "votes": 0, "replies": [ { "id": 1277349, "author_name": "", "author_url": "", "post_date": "2021-04-18T16:30:55.277000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1211404, "author_name": "Himanshu Mehndiratta", "author_url": "", "post_date": "2021-02-20T07:39:40.203000", "content": "I think the problem is with your end. Check again once.
", "votes": 0, "replies": [] }, { "id": 1253543, "author_name": "", "author_url": "", "post_date": "2021-03-26T21:05:35.537000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1183015": "I've a working inference kernel, I've updated/uploaded the weights of my model and I got \"**Submission CSV Not Found**\" error. When I look at diff between yesterday working version and today's broken version I can confirm that only path to weights has been updated. Am I alone to get this issue today? I've tried to re-submit and I've got the same error. It does not make sense, weights update cannot break the pipeline and the submission.csv and I can see it's generated on commit. ", "1215663": "Hi @mpware I just experienced the exact same issue that you described...only change in my kernel is that it is a new weights file that is loaded...a follow up version with yet another weights file is working.\n\nDoes not seem to happen often though...first time for me. \n", "1185435": "@mpware Hope, so you will resolve this soon.\n\n", "1183959": "Hope you will resolve soon. ", "1183525": "Today I've succeeded in submitting my kernel to get a score, so there seems to be little Kaggle scoring issue.", "1277347": "I'm having the same problem. Have you solved it? how?", "1211404": "I think the problem is with your end. Check again once.", "1253543": "" } }