{
  "id": 244135,
  "title": "3 days of submissions to find the error",
  "url": "/competitions/siim-covid19-detection/discussion/244135",
  "author_name": "",
  "post_date": "2021-06-05T10:37:58.246944900Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>hello,</p>\n<p>in the last 3 days (over 15 submissions) I was searching for the error in my code,</p>\n<p>there are two runs for each kernel - first it runs with public test data, then it runs in hidden test data, they should be similar, so code which works for first run should also run for the second run</p>\n<p>however, in the first run my kernel is always good but at the end I see status \"Submission CSV Not Found\", which means the same code is able to generate submission.csv for the first run but not for the second</p>\n<p>I started debugging by executing this at the end of my kernel:</p>\n<p><code>copyfile(\"../input/siim-covid19-detection/sample_submission.csv\", \"submission.csv\")\n</code></p>\n<p>This way I could check is my code crashing or not (if not there will be 0.050 score for sample submission).</p>\n<p>Soon I realized my code is crashing not in the PyTorch stuff but in the image resizing. In this competiton we must generate images for our model, because the original images are in dicom format. I was thinking previously to put dicom code into PyTorch Dataset but first I wanted to make it simple, so I followed the trend to just convert dicom into jpg (or png).</p>\n<p>After 3 days of debugging and comparing to public kernels I think the reason why my code is crashing is that instead using \"/kaggle/tmp/test/\" to store generated images I used \"images/\" folder. For some reason placing images into local directory works correctly but on second run it crashes. Why? Do you have any ideas?</p>\n<p>Maybe the second set is bigger and there is out of space? But this is unlikely, I just store 128x128 jpegs in my current code.</p>\n<p>What is the difference between first and second run?</p>",
  "messages": [
    {
      "id": "1336980",
      "postDate": "06/05/2021 10:37:58",
      "content": "<p>hello,</p>\n<p>in the last 3 days (over 15 submissions) I was searching for the error in my code,</p>\n<p>there are two runs for each kernel - first it runs with public test data, then it runs in hidden test data, they should be similar, so code which works for first run should also run for the second run</p>\n<p>however, in the first run my kernel is always good but at the end I see status \"Submission CSV Not Found\", which means the same code is able to generate submission.csv for the first run but not for the second</p>\n<p>I started debugging by executing this at the end of my kernel:</p>\n<p><code>copyfile(\"../input/siim-covid19-detection/sample_submission.csv\", \"submission.csv\")\n</code></p>\n<p>This way I could check is my code crashing or not (if not there will be 0.050 score for sample submission).</p>\n<p>Soon I realized my code is crashing not in the PyTorch stuff but in the image resizing. In this competiton we must generate images for our model, because the original images are in dicom format. I was thinking previously to put dicom code into PyTorch Dataset but first I wanted to make it simple, so I followed the trend to just convert dicom into jpg (or png).</p>\n<p>After 3 days of debugging and comparing to public kernels I think the reason why my code is crashing is that instead using \"/kaggle/tmp/test/\" to store generated images I used \"images/\" folder. For some reason placing images into local directory works correctly but on second run it crashes. Why? Do you have any ideas?</p>\n<p>Maybe the second set is bigger and there is out of space? But this is unlikely, I just store 128x128 jpegs in my current code.</p>\n<p>What is the difference between first and second run?</p>",
      "rawMarkdown": "hello,\n\nin the last 3 days (over 15 submissions) I was searching for the error in my code,\n\nthere are two runs for each kernel - first it runs with public test data, then it runs in hidden test data, they should be similar, so code which works for first run should also run for the second run\n\nhowever, in the first run my kernel is always good but at the end I see status \"Submission CSV Not Found\", which means the same code is able to generate submission.csv for the first run but not for the second\n\nI started debugging by executing this at the end of my kernel:\n\n`copyfile(\"../input/siim-covid19-detection/sample_submission.csv\", \"submission.csv\")\n`\n\nThis way I could check is my code crashing or not (if not there will be 0.050 score for sample submission).\n\nSoon I realized my code is crashing not in the PyTorch stuff but in the image resizing. In this competiton we must generate images for our model, because the original images are in dicom format. I was thinking previously to put dicom code into PyTorch Dataset but first I wanted to make it simple, so I followed the trend to just convert dicom into jpg (or png).\n\nAfter 3 days of debugging and comparing to public kernels I think the reason why my code is crashing is that instead using \"/kaggle/tmp/test/\" to store generated images I used \"images/\" folder. For some reason placing images into local directory works correctly but on second run it crashes. Why? Do you have any ideas?\n\nMaybe the second set is bigger and there is out of space? But this is unlikely, I just store 128x128 jpegs in my current code.\n\nWhat is the difference between first and second run?",
      "votes": null
    },
    {
      "id": "1337122",
      "postDate": "06/05/2021 12:35:30",
      "content": "<p>so for what reason don't you switch to /kaggle/tmp/test or  /kaggle/working/test ?</p>",
      "rawMarkdown": "so for what reason don't you switch to /kaggle/tmp/test or  /kaggle/working/test ?",
      "votes": null
    },
    {
      "id": "1337126",
      "postDate": "06/05/2021 12:37:53",
      "content": "<p>I switched to it, but first I needed to find out that was the reason.<br>\nMy question is why I can't use local folder instead these ones.</p>",
      "rawMarkdown": "I switched to it, but first I needed to find out that was the reason.\nMy question is why I can't use local folder instead these ones.",
      "votes": null
    },
    {
      "id": "1337215",
      "postDate": "06/05/2021 13:34:59",
      "content": "<p>what do you mean local folder? where did you run your code?  what is the error message?</p>",
      "rawMarkdown": "what do you mean local folder? where did you run your code?  what is the error message?",
      "votes": null
    },
    {
      "id": "1337226",
      "postDate": "06/05/2021 13:40:31",
      "content": "<p>Please read my whole post. </p>\n<p>There is no error message visible during second run of the kernel, because we can see only output of original, first run (on public data). So the only message I see is submission error \"Submission CSV Not Found\".</p>\n<p>After 3 days I found that the problem was place when I put generated images. When I used folder called \"images\" it had some problems, when instead I used folder \"/kaggle/tmp/test\" it does work. However in the first run (with public data) it works in both cases. That's why this is strange.</p>",
      "rawMarkdown": "Please read my whole post. \n\nThere is no error message visible during second run of the kernel, because we can see only output of original, first run (on public data). So the only message I see is submission error \"Submission CSV Not Found\".\n\nAfter 3 days I found that the problem was place when I put generated images. When I used folder called \"images\" it had some problems, when instead I used folder \"/kaggle/tmp/test\" it does work. However in the first run (with public data) it works in both cases. That's why this is strange.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1337122,
      "author_name": "dragonzhang",
      "author_url": "",
      "post_date": "06/05/2021 12:35:30",
      "content": "<p>so for what reason don't you switch to /kaggle/tmp/test or  /kaggle/working/test ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1337126,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "06/05/2021 12:37:53",
          "content": "<p>I switched to it, but first I needed to find out that was the reason.<br>\nMy question is why I can't use local folder instead these ones.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1337215,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "06/05/2021 13:34:59",
          "content": "<p>what do you mean local folder? where did you run your code?  what is the error message?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1337226,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "06/05/2021 13:40:31",
          "content": "<p>Please read my whole post. </p>\n<p>There is no error message visible during second run of the kernel, because we can see only output of original, first run (on public data). So the only message I see is submission error \"Submission CSV Not Found\".</p>\n<p>After 3 days I found that the problem was place when I put generated images. When I used folder called \"images\" it had some problems, when instead I used folder \"/kaggle/tmp/test\" it does work. However in the first run (with public data) it works in both cases. That's why this is strange.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1336980": "hello,\n\nin the last 3 days (over 15 submissions) I was searching for the error in my code,\n\nthere are two runs for each kernel - first it runs with public test data, then it runs in hidden test data, they should be similar, so code which works for first run should also run for the second run\n\nhowever, in the first run my kernel is always good but at the end I see status \"Submission CSV Not Found\", which means the same code is able to generate submission.csv for the first run but not for the second\n\nI started debugging by executing this at the end of my kernel:\n\n`copyfile(\"../input/siim-covid19-detection/sample_submission.csv\", \"submission.csv\")\n`\n\nThis way I could check is my code crashing or not (if not there will be 0.050 score for sample submission).\n\nSoon I realized my code is crashing not in the PyTorch stuff but in the image resizing. In this competiton we must generate images for our model, because the original images are in dicom format. I was thinking previously to put dicom code into PyTorch Dataset but first I wanted to make it simple, so I followed the trend to just convert dicom into jpg (or png).\n\nAfter 3 days of debugging and comparing to public kernels I think the reason why my code is crashing is that instead using \"/kaggle/tmp/test/\" to store generated images I used \"images/\" folder. For some reason placing images into local directory works correctly but on second run it crashes. Why? Do you have any ideas?\n\nMaybe the second set is bigger and there is out of space? But this is unlikely, I just store 128x128 jpegs in my current code.\n\nWhat is the difference between first and second run?",
    "1337122": "so for what reason don't you switch to /kaggle/tmp/test or  /kaggle/working/test ?",
    "1337126": "I switched to it, but first I needed to find out that was the reason.\nMy question is why I can't use local folder instead these ones.",
    "1337215": "what do you mean local folder? where did you run your code?  what is the error message?",
    "1337226": "Please read my whole post. \n\nThere is no error message visible during second run of the kernel, because we can see only output of original, first run (on public data). So the only message I see is submission error \"Submission CSV Not Found\".\n\nAfter 3 days I found that the problem was place when I put generated images. When I used folder called \"images\" it had some problems, when instead I used folder \"/kaggle/tmp/test\" it does work. However in the first run (with public data) it works in both cases. That's why this is strange."
  },
  "source": "meta"
}