I am facing the submission scoring error . the weird issue i am facing is if i remove a code line the submission is working fine and getting scored (~0.06). the code line is a preprocessing line image = 255 - image. adding this line is giving me the submission scoring error , just removing it is getting my submission scored but , my model is not trained to work on the unprocessed dataset
`def image_transform(df, size=64):\n resized = {}\n for i in range(df.shape[0]):
\n\n image = df.loc[df.index[i]].values.reshape(137,236).astype(np.uint8)\n image = np.array(image)\n image = 255 - image #### the line causing issue . if i remove it , the submission is getting scored\n image = crop_resize(image)\n #print(\"the img shape \",image.max())\n resized[df.index[i]] = image.reshape(-1)`\ni have checked the kernel for memory issue by passing and running train data ant it worked perfect . i am dong the del operation as suggested by many kernels and that prevented this kernel from memory issue . some help needed in resolving the submission scoring error in my case . \nhttps://www.kaggle.com/yuvaramsingh/kernel7d6a6a6624?scriptVersionId=27692506
", "messages": [ { "id": "730366", "postDate": "01/27/2020 11:43:50", "content": "I am facing the submission scoring error . the weird issue i am facing is if i remove a code line the submission is working fine and getting scored (~0.06). the code line is a preprocessing line image = 255 - image. adding this line is giving me the submission scoring error , just removing it is getting my submission scored but , my model is not trained to work on the unprocessed dataset
`def image_transform(df, size=64):\n resized = {}\n for i in range(df.shape[0]):
\n\n image = df.loc[df.index[i]].values.reshape(137,236).astype(np.uint8)\n image = np.array(image)\n image = 255 - image #### the line causing issue . if i remove it , the submission is getting scored\n image = crop_resize(image)\n #print(\"the img shape \",image.max())\n resized[df.index[i]] = image.reshape(-1)`\ni have checked the kernel for memory issue by passing and running train data ant it worked perfect . i am dong the del operation as suggested by many kernels and that prevented this kernel from memory issue . some help needed in resolving the submission scoring error in my case . \nhttps://www.kaggle.com/yuvaramsingh/kernel7d6a6a6624?scriptVersionId=27692506
", "rawMarkdown": "I am facing the submission scoring error . the weird issue i am facing is if i remove a code line the submission is working fine and getting scored (~0.06). the code line is a preprocessing line `image = 255 - image`. adding this line is giving me the submission scoring error , just removing it is getting my submission scored but , my model is not trained to work on the unprocessed dataset\n\n`def image_transform(df, size=64):\n resized = {}\n for i in range(df.shape[0]):\n \n image = df.loc[df.index[i]].values.reshape(137,236).astype(np.uint8)\n image = np.array(image)\n image = 255 - image #### the line causing issue . if i remove it , the submission is getting scored\n image = crop_resize(image)\n #print(\"the img shape \",image.max())\n resized[df.index[i]] = image.reshape(-1)`\n\ni have checked the kernel for memory issue by passing and running train data ant it worked perfect . i am dong the del operation as suggested by many kernels and that prevented this kernel from memory issue . some help needed in resolving the submission scoring error in my case . \nhttps://www.kaggle.com/yuvaramsingh/kernel7d6a6a6624?scriptVersionId=27692506", "votes": null }, { "id": "734468", "postDate": "02/01/2020 14:20:59", "content": "You do remind me of this! I've tested my code and I'm having the same problem. Don't know why but at least we know what the cause is.
", "rawMarkdown": "You do remind me of this! I've tested my code and I'm having the same problem. Don't know why but at least we know what the cause is.", "votes": null }, { "id": "734739", "postDate": "02/01/2020 23:56:59", "content": "I am facing the exact same error. \nthanks for giving the hint!\nNow, we need to find a solution
", "rawMarkdown": "I am facing the exact same error. \nthanks for giving the hint!\nNow, we need to find a solution", "votes": null }, { "id": "734754", "postDate": "02/02/2020 00:47:38", "content": "crop_resize needs the image to be of type np.float32 is that the problem?
", "rawMarkdown": "crop_resize needs the image to be of type np.float32 is that the problem?", "votes": null }, { "id": "734859", "postDate": "02/02/2020 05:24:15", "content": "In my case, my kernel runs for around 18mins and then get the submission scoring error, which almost costs the same time I've test on my train data. If np.float32 is the cause, I think it will give the error as the kernel started.
i suggest simple load for the cpu. you can push all the intensity, resize task to the gpu, e.g:
\n\n```
\n\nclass Net(nn.Module):
\n\ndef forward(self, x):\n batch_size,C,H,W = x.shape\n if (H,W) !=(68,118):\n x = F.interpolate(x,size=(68,118), mode='bilinear',align_corners=False) #e.g. resize\n\n x = 1 - x # e.g. intensity invert\n x = x.repeat(1,3,1,1) #e.g. gray to rgb\n...
\n\n```
", "rawMarkdown": "i suggest simple load for the cpu. you can push all the intensity, resize task to the gpu, e.g:\n\n```\n\nclass Net(nn.Module):\n \n def forward(self, x):\n batch_size,C,H,W = x.shape\n if (H,W) !=(68,118):\n x = F.interpolate(x,size=(68,118), mode='bilinear',align_corners=False) #e.g. resize\n \n x = 1 - x # e.g. intensity invert\n x = x.repeat(1,3,1,1) #e.g. gray to rgb\n...\n\n```", "votes": null }, { "id": "735022", "postDate": "02/02/2020 12:00:17", "content": "I don't know why but set threshold in crop_resize to a lower value such as 60 instead of the original value 80 worked for me. I've tried around 10 times to locate the problem and hope this can help. @yuvaramsingh @ilu000
When I had an error with cropresize my kernal ran for 12 minutes before erroring out.\nI looked into it more and crop_resize can actually take int and float.\nCropresize errors out if the input isn't normalized correctly so your solution should work. But if your using the 128x128 preprocessing then (looking at the implementation from the main topic) there is another step to normalizing: (im*(255.0/im.max())).astype(np.uint8)
Thanks a lot for the info, @syoya1997 \nI am going to try it out now.
\n\nI thought about exactly that case this night. It would mean that the test images contain graphemes with a very low intensity. Thus, normalizing the data per image might be superior. I am only puzzled why it works in the original kernel.
\n\nEdit: Indeed, this seems to fix the error. My test submission worked well with the adjusted threshold value. Thanks again!
", "rawMarkdown": "Thanks a lot for the info, @syoya1997 \nI am going to try it out now.\n\nI thought about exactly that case this night. It would mean that the test images contain graphemes with a very low intensity. Thus, normalizing the data per image might be superior. I am only puzzled why it works in the original kernel.\n\nEdit: Indeed, this seems to fix the error. My test submission worked well with the adjusted threshold value. Thanks again!", "votes": null }, { "id": "735142", "postDate": "02/02/2020 16:43:38", "content": "You're right. Normalizing before setting the threshold might be useful. Glad this fixes the error for you. 😄
", "rawMarkdown": "You're right. Normalizing before setting the threshold might be useful. Glad this fixes the error for you. 😄", "votes": null }, { "id": "741014", "postDate": "02/10/2020 04:59:29", "content": "@syoya1997 that worked . changing the threshold from 80 to 60 did the trick . still i am not sure on why it worked. is it due to computation, precision , or something else . but anyways , made my first submission . thanks
\n\n`def crop_resize(img0, size=SIZE, pad=16):
\n\nymin,ymax,xmin,xmax = bbox(img0[5:-5,5:-5] > 60)##### changing threshold from 80 to 60 did the trick . \nxmin = xmin - 13 if (xmin > 13) else 0\nymin = ymin - 10 if (ymin > 10) else 0\nxmax = xmax + 13 if (xmax < WIDTH - 13) else WIDTH\nymax = ymax + 10 if (ymax < HEIGHT - 10) else HEIGHT\nimg = img0[ymin:ymax,xmin:xmax]\nimg[img < 28] = 0\nlx, ly = xmax-xmin,ymax-ymin\nl = max(lx,ly) + pad\nimg = np.pad(img, [((l-ly)//2,), ((l-lx)//2,)], mode='constant')\nreturn cv2.resize(img,(size,size))`\nthanks @hengck23 will try this also . but, my main doubt was why adding a simple numpy subtraction is causing submission scoring error . still, now it works after changing the threshold . its not the correct answer for the question but , it works now .
", "rawMarkdown": "thanks @hengck23 will try this also . but, my main doubt was why adding a simple numpy subtraction is causing submission scoring error . still, now it works after changing the threshold . its not the correct answer for the question but , it works now .", "votes": null }, { "id": "741046", "postDate": "02/10/2020 05:38:04", "content": "I think the reason is that some images in test set have really low image density, which means that when you select threshold 80, it could not crop any valid image and then the error occurs. Just decreasing the threshold help solve the problem.
", "rawMarkdown": "I think the reason is that some images in test set have really low image density, which means that when you select threshold 80, it could not crop any valid image and then the error occurs. Just decreasing the threshold help solve the problem.", "votes": null }, { "id": "748752", "postDate": "02/18/2020 00:00:11", "content": "Thanks for your hint. I also confirm this issue on that 255-image line. I spent a week to check almost everything until I located this line.\nAnyway, thanks for your hint. Now it's time to move forward...
I suspect the line should read image = image/255. The values in the array are either 255 or 0, this changes all the values of 255 to 1, and zero remains zero. Your equation does not normalize the values, i.e., 255 = 0 and 0=255, it reverses the image.
@syoya1997. Don't you think we need to retrain the model on the new threshold ( i.e. 60), instead directly testing it on test images as the saved model (if you did it) was trained with image threshold of 80 {in my case}\nSTATUS OF MY SUB: Notebook Threw Exception
", "rawMarkdown": "syoya1997. Don't you think we need to retrain the model on the new threshold ( i.e. 60), instead directly testing it on test images as the saved model *(if you did it)* was trained with image threshold of 80 *{in my case}*\nSTATUS OF MY SUB: *Notebook Threw Exception*", "votes": null }, { "id": "763341", "postDate": "03/04/2020 11:42:26", "content": "You helped me a lot. Thank you
", "rawMarkdown": "You helped me a lot. Thank you", "votes": null } ], "comments": [ { "id": 734468, "author_name": "syoya1997", "author_url": "", "post_date": "02/01/2020 14:20:59", "content": "You do remind me of this! I've tested my code and I'm having the same problem. Don't know why but at least we know what the cause is.
", "votes": null, "replies": [] }, { "id": 734739, "author_name": "ilu000", "author_url": "", "post_date": "02/01/2020 23:56:59", "content": "I am facing the exact same error. \nthanks for giving the hint!\nNow, we need to find a solution
", "votes": null, "replies": [] }, { "id": 734754, "author_name": "greatgamedota", "author_url": "", "post_date": "02/02/2020 00:47:38", "content": "crop_resize needs the image to be of type np.float32 is that the problem?
", "votes": null, "replies": [ { "id": 734859, "author_name": "syoya1997", "author_url": "", "post_date": "02/02/2020 05:24:15", "content": "In my case, my kernel runs for around 18mins and then get the submission scoring error, which almost costs the same time I've test on my train data. If np.float32 is the cause, I think it will give the error as the kernel started.
When I had an error with cropresize my kernal ran for 12 minutes before erroring out.\nI looked into it more and crop_resize can actually take int and float.\nCropresize errors out if the input isn't normalized correctly so your solution should work. But if your using the 128x128 preprocessing then (looking at the implementation from the main topic) there is another step to normalizing: (im*(255.0/im.max())).astype(np.uint8)
i suggest simple load for the cpu. you can push all the intensity, resize task to the gpu, e.g:
\n\n```
\n\nclass Net(nn.Module):
\n\ndef forward(self, x):\n batch_size,C,H,W = x.shape\n if (H,W) !=(68,118):\n x = F.interpolate(x,size=(68,118), mode='bilinear',align_corners=False) #e.g. resize\n\n x = 1 - x # e.g. intensity invert\n x = x.repeat(1,3,1,1) #e.g. gray to rgb\n...
\n\n```
", "votes": null, "replies": [ { "id": 741016, "author_name": "yuvaramsingh", "author_url": "", "post_date": "02/10/2020 05:02:18", "content": "thanks @hengck23 will try this also . but, my main doubt was why adding a simple numpy subtraction is causing submission scoring error . still, now it works after changing the threshold . its not the correct answer for the question but , it works now .
", "votes": null, "replies": [] } ] }, { "id": 735022, "author_name": "syoya1997", "author_url": "", "post_date": "02/02/2020 12:00:17", "content": "I don't know why but set threshold in crop_resize to a lower value such as 60 instead of the original value 80 worked for me. I've tried around 10 times to locate the problem and hope this can help. @yuvaramsingh @ilu000
Thanks a lot for the info, @syoya1997 \nI am going to try it out now.
\n\nI thought about exactly that case this night. It would mean that the test images contain graphemes with a very low intensity. Thus, normalizing the data per image might be superior. I am only puzzled why it works in the original kernel.
\n\nEdit: Indeed, this seems to fix the error. My test submission worked well with the adjusted threshold value. Thanks again!
", "votes": null, "replies": [] }, { "id": 735142, "author_name": "syoya1997", "author_url": "", "post_date": "02/02/2020 16:43:38", "content": "You're right. Normalizing before setting the threshold might be useful. Glad this fixes the error for you. 😄
", "votes": null, "replies": [] } ] }, { "id": 741014, "author_name": "yuvaramsingh", "author_url": "", "post_date": "02/10/2020 04:59:29", "content": "@syoya1997 that worked . changing the threshold from 80 to 60 did the trick . still i am not sure on why it worked. is it due to computation, precision , or something else . but anyways , made my first submission . thanks
\n\n`def crop_resize(img0, size=SIZE, pad=16):
\n\nymin,ymax,xmin,xmax = bbox(img0[5:-5,5:-5] > 60)##### changing threshold from 80 to 60 did the trick . \nxmin = xmin - 13 if (xmin > 13) else 0\nymin = ymin - 10 if (ymin > 10) else 0\nxmax = xmax + 13 if (xmax < WIDTH - 13) else WIDTH\nymax = ymax + 10 if (ymax < HEIGHT - 10) else HEIGHT\nimg = img0[ymin:ymax,xmin:xmax]\nimg[img < 28] = 0\nlx, ly = xmax-xmin,ymax-ymin\nl = max(lx,ly) + pad\nimg = np.pad(img, [((l-ly)//2,), ((l-lx)//2,)], mode='constant')\nreturn cv2.resize(img,(size,size))`\nI think the reason is that some images in test set have really low image density, which means that when you select threshold 80, it could not crop any valid image and then the error occurs. Just decreasing the threshold help solve the problem.
", "votes": null, "replies": [] }, { "id": 763341, "author_name": "egm108", "author_url": "", "post_date": "03/04/2020 11:42:26", "content": "You helped me a lot. Thank you
", "votes": null, "replies": [] } ] }, { "id": 748752, "author_name": "ivanwang2016", "author_url": "", "post_date": "02/18/2020 00:00:11", "content": "Thanks for your hint. I also confirm this issue on that 255-image line. I spent a week to check almost everything until I located this line.\nAnyway, thanks for your hint. Now it's time to move forward...
I suspect the line should read image = image/255. The values in the array are either 255 or 0, this changes all the values of 255 to 1, and zero remains zero. Your equation does not normalize the values, i.e., 255 = 0 and 0=255, it reverses the image.
@syoya1997. Don't you think we need to retrain the model on the new threshold ( i.e. 60), instead directly testing it on test images as the saved model (if you did it) was trained with image threshold of 80 {in my case}\nSTATUS OF MY SUB: Notebook Threw Exception
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "730366": "I am facing the submission scoring error . the weird issue i am facing is if i remove a code line the submission is working fine and getting scored (~0.06). the code line is a preprocessing line `image = 255 - image`. adding this line is giving me the submission scoring error , just removing it is getting my submission scored but , my model is not trained to work on the unprocessed dataset\n\n`def image_transform(df, size=64):\n resized = {}\n for i in range(df.shape[0]):\n \n image = df.loc[df.index[i]].values.reshape(137,236).astype(np.uint8)\n image = np.array(image)\n image = 255 - image #### the line causing issue . if i remove it , the submission is getting scored\n image = crop_resize(image)\n #print(\"the img shape \",image.max())\n resized[df.index[i]] = image.reshape(-1)`\n\ni have checked the kernel for memory issue by passing and running train data ant it worked perfect . i am dong the del operation as suggested by many kernels and that prevented this kernel from memory issue . some help needed in resolving the submission scoring error in my case . \nhttps://www.kaggle.com/yuvaramsingh/kernel7d6a6a6624?scriptVersionId=27692506", "734468": "You do remind me of this! I've tested my code and I'm having the same problem. Don't know why but at least we know what the cause is.", "734739": "I am facing the exact same error. \nthanks for giving the hint!\nNow, we need to find a solution", "734754": "crop_resize needs the image to be of type np.float32 is that the problem?", "734859": "In my case, my kernel runs for around 18mins and then get the submission scoring error, which almost costs the same time I've test on my train data. If `np.float32` is the cause, I think it will give the error as the kernel started.", "734874": "i suggest simple load for the cpu. you can push all the intensity, resize task to the gpu, e.g:\n\n```\n\nclass Net(nn.Module):\n \n def forward(self, x):\n batch_size,C,H,W = x.shape\n if (H,W) !=(68,118):\n x = F.interpolate(x,size=(68,118), mode='bilinear',align_corners=False) #e.g. resize\n \n x = 1 - x # e.g. intensity invert\n x = x.repeat(1,3,1,1) #e.g. gray to rgb\n...\n\n```", "735022": "I don't know why but set threshold in `crop_resize` to a lower value such as 60 instead of the original value 80 worked for me. I've tried around 10 times to locate the problem and hope this can help. @yuvaramsingh @ilu000", "735095": "When I had an error with cropresize my kernal ran for 12 minutes before erroring out.\nI looked into it more and crop_resize can actually take int and float.\nCropresize errors out if the input isn't normalized correctly so your solution should work. But if your using the 128x128 preprocessing then (looking at the implementation from the main topic) there is another step to normalizing: ```(im*(255.0/im.max())).astype(np.uint8)```", "735102": "Thanks a lot for the info, @syoya1997 \nI am going to try it out now.\n\nI thought about exactly that case this night. It would mean that the test images contain graphemes with a very low intensity. Thus, normalizing the data per image might be superior. I am only puzzled why it works in the original kernel.\n\nEdit: Indeed, this seems to fix the error. My test submission worked well with the adjusted threshold value. Thanks again!", "735142": "You're right. Normalizing before setting the threshold might be useful. Glad this fixes the error for you. 😄", "741014": "syoya1997 that worked . changing the threshold from 80 to 60 did the trick . still i am not sure on why it worked. is it due to computation, precision , or something else . but anyways , made my first submission . thanks\n\n`def crop_resize(img0, size=SIZE, pad=16):\n \n ymin,ymax,xmin,xmax = bbox(img0[5:-5,5:-5] > 60)##### changing threshold from 80 to 60 did the trick . \n xmin = xmin - 13 if (xmin > 13) else 0\n ymin = ymin - 10 if (ymin > 10) else 0\n xmax = xmax + 13 if (xmax < WIDTH - 13) else WIDTH\n ymax = ymax + 10 if (ymax < HEIGHT - 10) else HEIGHT\n img = img0[ymin:ymax,xmin:xmax]\n img[img < 28] = 0\n lx, ly = xmax-xmin,ymax-ymin\n l = max(lx,ly) + pad\n img = np.pad(img, [((l-ly)//2,), ((l-lx)//2,)], mode='constant')\n return cv2.resize(img,(size,size))`", "741016": "thanks @hengck23 will try this also . but, my main doubt was why adding a simple numpy subtraction is causing submission scoring error . still, now it works after changing the threshold . its not the correct answer for the question but , it works now .", "741046": "I think the reason is that some images in test set have really low image density, which means that when you select threshold 80, it could not crop any valid image and then the error occurs. Just decreasing the threshold help solve the problem.", "748752": "Thanks for your hint. I also confirm this issue on that `255-image` line. I spent a week to check almost everything until I located this line.\nAnyway, thanks for your hint. Now it's time to move forward...", "748760": "I suspect the line should read `image = image/255`. The values in the array are either 255 or 0, this changes all the values of 255 to 1, and zero remains zero. Your equation does not normalize the values, i.e., 255 = 0 and 0=255, it reverses the image.", "753154": "syoya1997. Don't you think we need to retrain the model on the new threshold ( i.e. 60), instead directly testing it on test images as the saved model *(if you did it)* was trained with image threshold of 80 *{in my case}*\nSTATUS OF MY SUB: *Notebook Threw Exception*", "763341": "You helped me a lot. Thank you" }, "source": "meta" }