![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F421965%2F620bd7cd6018dba46d3dce82022a940a%2Fdownload.png?generation=1592109019364555&alt=media\")
I was facing a strange issue where occasionally my validation loss went to NaN (train loss was fine). I tracked the issue down to theses images in all 4 folders (there may be more):
\n\nI'm still not sure why they were causing NaNs but my guess is that the poor quality was causing something in my model to explode
\n\n
It looks like there is an underlying image in each of these but maybe a payload has been applied twice or something?
", "messages": [ { "id": 885004, "postDate": "2020-06-13T19:43:10.987Z", "content": "I was facing a strange issue where occasionally my validation loss went to NaN (train loss was fine). I tracked the issue down to theses images in all 4 folders (there may be more):
\n\nI'm still not sure why they were causing NaNs but my guess is that the poor quality was causing something in my model to explode
\n\n
It looks like there is an underlying image in each of these but maybe a payload has been applied twice or something?
", "rawMarkdown": "I was facing a strange issue where occasionally my validation loss went to NaN (train loss was fine). I tracked the issue down to theses images in all 4 folders (there may be more):\n\nI'm still not sure why they were causing NaNs but my guess is that the poor quality was causing something in my model to explode\n\n\n\nIt looks like there is an underlying image in each of these but maybe a payload has been applied twice or something?\n", "votes": 9 }, { "id": 886365, "postDate": "2020-06-15T01:40:25.733Z", "content": "I met this problem before when I was using RMSprop, changed to Adam , then the problem gone.
", "rawMarkdown": "I met this problem before when I was using RMSprop, changed to Adam , then the problem gone.", "votes": 3, "replies": [ { "id": 886489, "postDate": "2020-06-15T04:37:48.493Z", "content": "I think you're right. Changing some of my optimiser settings seems to solve the NaNs for now
", "rawMarkdown": "I think you're right. Changing some of my optimiser settings seems to solve the NaNs for now" } ] }, { "id": 896318, "postDate": "2020-06-22T04:33:48.243Z", "content": "Hello there,\nIt's exam period in my University, which implied quite an important workload, hence my distance.\nRobga, you are abolutely right. Our goal with ALASKA (#1 but especially this contest) is to move \"into the wilderness\" of image that you can get from random source.\nTherefore we included images that are quite blurred as well as noisy nighty shot.\nBelieve me, if you download image from FlickR (for instance) at random 95% of image are just fine but some images are really of poor quality .... :)\nThe ALASKA dataset mimic the presence of such images. There are far from being a majority but for sure you can hardly handle NaN for such pictures.\nEdgar is also right, such image come from a combinaison of nighty picture without denoising but with harsh edge sharpening....
", "rawMarkdown": "Hello there,\nIt's exam period in my University, which implied quite an important workload, hence my distance.\nRobga, you are abolutely right. Our goal with ALASKA (#1 but especially this contest) is to move \"into the wilderness\" of image that you can get from random source.\nTherefore we included images that are quite blurred as well as noisy nighty shot.\nBelieve me, if you download image from FlickR (for instance) at random 95% of image are just fine but some images are really of poor quality .... :)\nThe ALASKA dataset mimic the presence of such images. There are far from being a majority but for sure you can hardly handle NaN for such pictures.\nEdgar is also right, such image come from a combinaison of nighty picture without denoising but with harsh edge sharpening....", "votes": 4, "replies": [ { "id": 896334, "postDate": "2020-06-22T05:01:19.950Z", "content": "Thanks for the clarification Remi!
", "rawMarkdown": "Thanks for the clarification Remi!", "votes": 1 } ] }, { "id": 885510, "postDate": "2020-06-14T09:21:29.983Z", "content": "It seems a deliberate choice. \"we also tried to mimic a realistic dataset with ... as well as an important ratio of 13.8% of night or indoor pictures with ISO larger than 1000,\". Once you realise these are crops of large images, and you add in sharpening, it can look as above. The question is whether your model works better without such images.
\n\nThere are some other odd images though, like test/3176.jpg which looks like a payload itself.
", "rawMarkdown": "It seems a deliberate choice. \"we also tried to mimic a realistic dataset with ... as well as an important ratio of 13.8% of night or indoor pictures with ISO larger than 1000,\". Once you realise these are crops of large images, and you add in sharpening, it can look as above. The question is whether your model works better without such images.\n\nThere are some other odd images though, like test/3176.jpg which looks like a payload itself.", "replies": [ { "id": 886494, "postDate": "2020-06-15T04:43:59.483Z", "content": "I guess you could get noisier images from extremely high ISO, but I don't think I've seen images this noisy before. I guess as you and as Edgar K mentioned it depends on what sort of processing is applied.
\n\nThat 3176.jpg image reminds me of the cafe wall illusion...
", "rawMarkdown": "I guess you could get noisier images from extremely high ISO, but I don't think I've seen images this noisy before. I guess as you and as Edgar K mentioned it depends on what sort of processing is applied.\n\nThat 3176.jpg image reminds me of the cafe wall illusion..." } ] }, { "id": 885472, "postDate": "2020-06-14T08:46:36.500Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 886365, "author_name": "zheng lv", "author_url": "", "post_date": "2020-06-15T01:40:25.733000", "content": "I met this problem before when I was using RMSprop, changed to Adam , then the problem gone.
", "votes": 3, "replies": [ { "id": 886489, "author_name": "datasaurus", "author_url": "", "post_date": "2020-06-15T04:37:48.493000", "content": "I think you're right. Changing some of my optimiser settings seems to solve the NaNs for now
", "votes": 0, "replies": [] } ] }, { "id": 896318, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-06-22T04:33:48.243000", "content": "Hello there,\nIt's exam period in my University, which implied quite an important workload, hence my distance.\nRobga, you are abolutely right. Our goal with ALASKA (#1 but especially this contest) is to move \"into the wilderness\" of image that you can get from random source.\nTherefore we included images that are quite blurred as well as noisy nighty shot.\nBelieve me, if you download image from FlickR (for instance) at random 95% of image are just fine but some images are really of poor quality .... :)\nThe ALASKA dataset mimic the presence of such images. There are far from being a majority but for sure you can hardly handle NaN for such pictures.\nEdgar is also right, such image come from a combinaison of nighty picture without denoising but with harsh edge sharpening....
", "votes": 4, "replies": [ { "id": 896334, "author_name": "datasaurus", "author_url": "", "post_date": "2020-06-22T05:01:19.950000", "content": "Thanks for the clarification Remi!
", "votes": 1, "replies": [] } ] }, { "id": 885510, "author_name": "robga", "author_url": "", "post_date": "2020-06-14T09:21:29.983000", "content": "It seems a deliberate choice. \"we also tried to mimic a realistic dataset with ... as well as an important ratio of 13.8% of night or indoor pictures with ISO larger than 1000,\". Once you realise these are crops of large images, and you add in sharpening, it can look as above. The question is whether your model works better without such images.
\n\nThere are some other odd images though, like test/3176.jpg which looks like a payload itself.
", "votes": 0, "replies": [ { "id": 886494, "author_name": "datasaurus", "author_url": "", "post_date": "2020-06-15T04:43:59.483000", "content": "I guess you could get noisier images from extremely high ISO, but I don't think I've seen images this noisy before. I guess as you and as Edgar K mentioned it depends on what sort of processing is applied.
\n\nThat 3176.jpg image reminds me of the cafe wall illusion...
", "votes": 0, "replies": [] } ] }, { "id": 885472, "author_name": "", "author_url": "", "post_date": "2020-06-14T08:46:36.500000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "885004": "I was facing a strange issue where occasionally my validation loss went to NaN (train loss was fine). I tracked the issue down to theses images in all 4 folders (there may be more):\n\nI'm still not sure why they were causing NaNs but my guess is that the poor quality was causing something in my model to explode\n\n\n\nIt looks like there is an underlying image in each of these but maybe a payload has been applied twice or something?\n", "886365": "I met this problem before when I was using RMSprop, changed to Adam , then the problem gone.", "896318": "Hello there,\nIt's exam period in my University, which implied quite an important workload, hence my distance.\nRobga, you are abolutely right. Our goal with ALASKA (#1 but especially this contest) is to move \"into the wilderness\" of image that you can get from random source.\nTherefore we included images that are quite blurred as well as noisy nighty shot.\nBelieve me, if you download image from FlickR (for instance) at random 95% of image are just fine but some images are really of poor quality .... :)\nThe ALASKA dataset mimic the presence of such images. There are far from being a majority but for sure you can hardly handle NaN for such pictures.\nEdgar is also right, such image come from a combinaison of nighty picture without denoising but with harsh edge sharpening....", "885510": "It seems a deliberate choice. \"we also tried to mimic a realistic dataset with ... as well as an important ratio of 13.8% of night or indoor pictures with ISO larger than 1000,\". Once you realise these are crops of large images, and you add in sharpening, it can look as above. The question is whether your model works better without such images.\n\nThere are some other odd images though, like test/3176.jpg which looks like a payload itself.", "885472": "" } }