There seems to be a several ways to handle this, none of which I can discern reliably as the 'correct' way. Here is one function to do it:
\n\ndef download(id_):\n try:\n image_dir, _, image_id = id_.partition('_')\n arr_out = np.zeros((512,512,3))\n for i, c in enumerate(['red', 'green', 'blue']):\n url = f'http://v18.proteinatlas.org/images/{image_dir}/{image_id}_{c}.jpg'\n r = requests.get(url)\n image = Image.open(BytesIO(r.content)).resize((512, 512), PIL.Image.LANCZOS).convert('L')\n arr_out[:,:,i] = np.array(image).astype('uint8')\n cv.imwrite(f'{hpa_dir}/{id_}.png', arr_out)\n except:\n print(f'{id_} broke...')\nNote that what is returned from the protein atlas site is actually a JPEG file with 3 channels. So we end up with 3 channels for each of red, green and blue. The above function is merging these 3 channels into a single grayscale channel via the .convert('L') method which then themselves go into the image as a channel. So we go from 3x3 channels into 3 for the final image. The issue is that this is producing images that have very different intensity to the training set, particularly in the final B channel.
I've tried following Alex's suggestions here where he has patiently tried to explain this but I'm none the wiser.
\n\nAny clarification greatly appreciated,
\n\nMark
\n\nP. S. I'm ignoring yellow for now.
", "messages": [ { "id": "438455", "postDate": "12/13/2018 17:52:36", "content": "There seems to be a several ways to handle this, none of which I can discern reliably as the 'correct' way. Here is one function to do it:
\n\ndef download(id_):\n try:\n image_dir, _, image_id = id_.partition('_')\n arr_out = np.zeros((512,512,3))\n for i, c in enumerate(['red', 'green', 'blue']):\n url = f'http://v18.proteinatlas.org/images/{image_dir}/{image_id}_{c}.jpg'\n r = requests.get(url)\n image = Image.open(BytesIO(r.content)).resize((512, 512), PIL.Image.LANCZOS).convert('L')\n arr_out[:,:,i] = np.array(image).astype('uint8')\n cv.imwrite(f'{hpa_dir}/{id_}.png', arr_out)\n except:\n print(f'{id_} broke...')\nNote that what is returned from the protein atlas site is actually a JPEG file with 3 channels. So we end up with 3 channels for each of red, green and blue. The above function is merging these 3 channels into a single grayscale channel via the .convert('L') method which then themselves go into the image as a channel. So we go from 3x3 channels into 3 for the final image. The issue is that this is producing images that have very different intensity to the training set, particularly in the final B channel.
I've tried following Alex's suggestions here where he has patiently tried to explain this but I'm none the wiser.
\n\nAny clarification greatly appreciated,
\n\nMark
\n\nP. S. I'm ignoring yellow for now.
", "rawMarkdown": "There seems to be a several ways to handle this, none of which I can discern reliably as the 'correct' way. Here is one function to do it:\n\n def download(id_):\n try:\n image_dir, _, image_id = id_.partition('_')\n arr_out = np.zeros((512,512,3))\n for i, c in enumerate(['red', 'green', 'blue']):\n url = f'http://v18.proteinatlas.org/images/{image_dir}/{image_id}_{c}.jpg'\n r = requests.get(url)\n image = Image.open(BytesIO(r.content)).resize((512, 512), PIL.Image.LANCZOS).convert('L')\n arr_out[:,:,i] = np.array(image).astype('uint8')\n cv.imwrite(f'{hpa_dir}/{id_}.png', arr_out)\n except:\n print(f'{id_} broke...')\n\nNote that what is returned from the protein atlas site is actually a JPEG file with 3 channels. So we end up with **3 channels for each of red, green and blue**. The above function is merging these 3 channels into a single grayscale channel via the `.convert('L')` method which then themselves go into the image as a channel. So we go from 3x3 channels into 3 for the final image. The issue is that this is producing images that have very different intensity to the training set, particularly in the final B channel.\n\nI've tried following Alex's suggestions [here][1] where he has patiently tried to explain this but I'm none the wiser. \n\nAny clarification greatly appreciated,\n\nMark\n\nP. S. I'm ignoring yellow for now.\n\n [1]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984", "votes": null }, { "id": "438470", "postDate": "12/13/2018 18:37:17", "content": "Update: I think per the attached that when we grab the 'red' image from the hpa website we should grab channel 0 from what the 3 channel JPG returns, then for the 'green' channel we grab channel 1 and for 'blue' it's channel 2. All the intensity of the images seems to lie here.
", "rawMarkdown": "**Update**: I think per the attached that when we grab the 'red' image from the hpa website we should grab channel 0 from what the 3 channel JPG returns, then for the 'green' channel we grab channel 1 and for 'blue' it's channel 2. All the intensity of the images seems to lie here.", "votes": null }, { "id": "438521", "postDate": "12/13/2018 20:44:28", "content": "I also faced with this problem. Yet I could not find any appropriate solution.\nIf I convert image to grayscale ,using cv2.imread(\"path_to_image.jpg\",0) where 0 indicades to read image in gs , or use max along channels for each pixels - > I get very low score after training BNinception proposed in the kernels compared to training without external dataset.\nMoreover If I mix original dataset with external dataset and use original dataset as validation - > I also get lower score. \nI could not find any relevant solution to this heterogeneity in the datasets. \nI would be grateful if someone will say how to fix it.
", "rawMarkdown": "I also faced with this problem. Yet I could not find any appropriate solution.\nIf I convert image to grayscale ,using cv2.imread(\"path_to_image.jpg\",0) where 0 indicades to read image in gs , or use max along channels for each pixels - > I get very low score after training BNinception proposed in the kernels compared to training without external dataset.\nMoreover If I mix original dataset with external dataset and use original dataset as validation - > I also get lower score. \nI could not find any relevant solution to this heterogeneity in the datasets. \nI would be grateful if someone will say how to fix it.", "votes": null }, { "id": "438523", "postDate": "12/13/2018 20:52:07", "content": "Perhaps try to train on the extrnal and fine tune on the train? It seems like the train is of higher quality. I am going to try this next week.
", "rawMarkdown": "Perhaps try to train on the extrnal and fine tune on the train? It seems like the train is of higher quality. I am going to try this next week.", "votes": null }, { "id": "438612", "postDate": "12/14/2018 00:00:42", "content": "@Mark
\n\nThanks for this thread and the comments on the other one. I think I understand what has been said - it makes sense.
\n\nI am comparing using the \"leak\" correspondence of jpg vs test data. It is easier comparing these very similar images.
\n\nSo far as I can see, the procedure improves greatly the image comparison and also the histogram (x axis=pixel value, y axis=pixel count in single image) comparisons. The only caveats are that the jpg image conversions are smoother (less resolved, even when resizing with \"nearest\") than the challenge images and that the jpg conversion histograms are somewhat noisy - perhaps both effects from the evil jpg compression(?). The smoothing is a worry because there is a big gain harvested by upping the resolution - do we lose that with HPA data?
\n\nThe histograms look similar enough to forgo the re-derivation of the image statistics for the lafoss fastai kernel.
\n\nI had been having no luck training the HPA and started QC'ing the conversion a couple of days ago so this is a timely thread. I'll try the training again. It'll probably fail because most things fail in this business!
", "rawMarkdown": "Mark\n\nThanks for this thread and the comments on the other one. I think I understand what has been said - it makes sense. \n\nI am comparing using the \"leak\" correspondence of jpg vs test data. It is easier comparing these very similar images.\n\nSo far as I can see, the procedure improves greatly the image comparison and also the histogram (x axis=pixel value, y axis=pixel count in single image) comparisons. The only caveats are that the jpg image conversions are smoother (less resolved, even when resizing with \"nearest\") than the challenge images and that the jpg conversion histograms are somewhat noisy - perhaps both effects from the evil jpg compression(?). The smoothing is a worry because there is a big gain harvested by upping the resolution - do we lose that with HPA data?\n\nThe histograms look similar enough to forgo the re-derivation of the image statistics for the lafoss fastai kernel.\n\nI had been having no luck training the HPA and started QC'ing the conversion a couple of days ago so this is a timely thread. I'll try the training again. It'll probably fail because most things fail in this business!", "votes": null }, { "id": "438763", "postDate": "12/14/2018 06:34:50", "content": "yes, I also face with same problem, the normalized avr. of the RGB channel on the trainset is [0.081, 0.052, 0.055], but on the external data it's [0.054, 0.051, 0.031], the R and B channel is significantly decreased, maybe its mainly because the convert('L') function uses the formula: Y = 0.299 R + 0.587 G + 0.114 B. so I will try your method, thanks for your sharing!
", "rawMarkdown": "yes, I also face with same problem, the normalized avr. of the RGB channel on the trainset is [0.081, 0.052, 0.055], but on the external data it's [0.054, 0.051, 0.031], the R and B channel is significantly decreased, maybe its mainly because the convert('L') function uses the formula: Y = 0.299 R + 0.587 G + 0.114 B. so I will try your method, thanks for your sharing!", "votes": null }, { "id": "438770", "postDate": "12/14/2018 06:42:21", "content": "Thanks for sharing!\nI tried to match the mean and variance of HPA data to the main train data by inverting this coefficients. \nIt is still training but looks promising.
", "rawMarkdown": "Thanks for sharing!\nI tried to match the mean and variance of HPA data to the main train data by inverting this coefficients. \nIt is still training but looks promising.", "votes": null }, { "id": "438789", "postDate": "12/14/2018 07:28:43", "content": "Unfortunately, theoretically, the numbers go the other way I think.
", "rawMarkdown": "Unfortunately, theoretically, the numbers go the other way I think.", "votes": null }, { "id": "438902", "postDate": "12/14/2018 11:13:19", "content": "Hi Pete,
\n\nSo you are using JPG HPA? I think some people have managed to get PNG.
\n\nI struggle to believe the 2048 pixel JPGs wouldn't help at all, though initial training runs don't seem any better. It's bizarre - we are increasing the number of some rare classes by many multiples.
", "rawMarkdown": "Hi Pete, \n\nSo you are using JPG HPA? I think some people have managed to get PNG. \n\nI struggle to believe the 2048 pixel JPGs wouldn't help at all, though initial training runs don't seem any better. It's bizarre - we are increasing the number of some rare classes by many multiples.", "votes": null }, { "id": "439175", "postDate": "12/14/2018 20:29:07", "content": "No, I am converting the JPG to PNG 512 and from there to 256 and 128 and I'm still taking care of a couple of small problems. I also need to double-check on the images that the histograms are not way different from the test data histograms and do something if they are. After that, I'll probably start by adding in just the rare classes.\nI tried the big tif images a while back but got poor results. I will revisit that if I have time with a better downsampling of the tif files.
", "rawMarkdown": "No, I am converting the JPG to PNG 512 and from there to 256 and 128 and I'm still taking care of a couple of small problems. I also need to double-check on the images that the histograms are not way different from the test data histograms and do something if they are. After that, I'll probably start by adding in just the rare classes.\nI tried the big tif images a while back but got poor results. I will revisit that if I have time with a better downsampling of the tif files.", "votes": null }, { "id": "439178", "postDate": "12/14/2018 20:40:37", "content": "Yeah, I'm doing similar - the jpg are 3 channel for each colour so I'm just extracting one as detailed above.
\n\nHere are the (raw) stats if you want them:
\n\nHPA stats (RGB):\nMean: [30, 17, 17]\nStdev: [45.5, 31.5, 45]
\n\nTrain stats (RGB):\nMean: [20.5, 13.45, 14]\nStdev: [38.22, 28.7, 39.8]
\n\nI'm normalising the HPA data separately as a consequence. I tried one run just adding data for the rare classes which improved recall for some classes but caused precision to plummet (=> some labels are unreliable?).
\n\nNext step: keep data for rare classes ~3k extra examples and then of the remaining 72k images, randomly add a sample of ~25k each epoch.
", "rawMarkdown": "Yeah, I'm doing similar - the jpg are 3 channel for each colour so I'm just extracting one as detailed above. \n\nHere are the (raw) stats if you want them:\n\nHPA stats (RGB):\nMean: [30, 17, 17]\nStdev: [45.5, 31.5, 45] \n \nTrain stats (RGB):\nMean: [20.5, 13.45, 14]\nStdev: [38.22, 28.7, 39.8] \n\nI'm normalising the HPA data separately as a consequence. I tried one run just adding data for the rare classes which improved recall for some classes but caused precision to plummet (=> some labels are unreliable?). \n\nNext step: keep data for rare classes ~3k extra examples and then of the remaining 72k images, randomly add a sample of ~25k each epoch.", "votes": null } ], "comments": [ { "id": 438470, "author_name": "maw501", "author_url": "", "post_date": "12/13/2018 18:37:17", "content": "Update: I think per the attached that when we grab the 'red' image from the hpa website we should grab channel 0 from what the 3 channel JPG returns, then for the 'green' channel we grab channel 1 and for 'blue' it's channel 2. All the intensity of the images seems to lie here.
", "votes": null, "replies": [ { "id": 438763, "author_name": "hogger", "author_url": "", "post_date": "12/14/2018 06:34:50", "content": "yes, I also face with same problem, the normalized avr. of the RGB channel on the trainset is [0.081, 0.052, 0.055], but on the external data it's [0.054, 0.051, 0.031], the R and B channel is significantly decreased, maybe its mainly because the convert('L') function uses the formula: Y = 0.299 R + 0.587 G + 0.114 B. so I will try your method, thanks for your sharing!
", "votes": null, "replies": [] }, { "id": 438770, "author_name": "spsancti", "author_url": "", "post_date": "12/14/2018 06:42:21", "content": "Thanks for sharing!\nI tried to match the mean and variance of HPA data to the main train data by inverting this coefficients. \nIt is still training but looks promising.
", "votes": null, "replies": [] }, { "id": 438789, "author_name": "petewills", "author_url": "", "post_date": "12/14/2018 07:28:43", "content": "Unfortunately, theoretically, the numbers go the other way I think.
", "votes": null, "replies": [] } ] }, { "id": 438521, "author_name": "unofloristo", "author_url": "", "post_date": "12/13/2018 20:44:28", "content": "I also faced with this problem. Yet I could not find any appropriate solution.\nIf I convert image to grayscale ,using cv2.imread(\"path_to_image.jpg\",0) where 0 indicades to read image in gs , or use max along channels for each pixels - > I get very low score after training BNinception proposed in the kernels compared to training without external dataset.\nMoreover If I mix original dataset with external dataset and use original dataset as validation - > I also get lower score. \nI could not find any relevant solution to this heterogeneity in the datasets. \nI would be grateful if someone will say how to fix it.
", "votes": null, "replies": [ { "id": 438523, "author_name": "moshel", "author_url": "", "post_date": "12/13/2018 20:52:07", "content": "Perhaps try to train on the extrnal and fine tune on the train? It seems like the train is of higher quality. I am going to try this next week.
", "votes": null, "replies": [] } ] }, { "id": 438612, "author_name": "petewills", "author_url": "", "post_date": "12/14/2018 00:00:42", "content": "@Mark
\n\nThanks for this thread and the comments on the other one. I think I understand what has been said - it makes sense.
\n\nI am comparing using the \"leak\" correspondence of jpg vs test data. It is easier comparing these very similar images.
\n\nSo far as I can see, the procedure improves greatly the image comparison and also the histogram (x axis=pixel value, y axis=pixel count in single image) comparisons. The only caveats are that the jpg image conversions are smoother (less resolved, even when resizing with \"nearest\") than the challenge images and that the jpg conversion histograms are somewhat noisy - perhaps both effects from the evil jpg compression(?). The smoothing is a worry because there is a big gain harvested by upping the resolution - do we lose that with HPA data?
\n\nThe histograms look similar enough to forgo the re-derivation of the image statistics for the lafoss fastai kernel.
\n\nI had been having no luck training the HPA and started QC'ing the conversion a couple of days ago so this is a timely thread. I'll try the training again. It'll probably fail because most things fail in this business!
", "votes": null, "replies": [ { "id": 438902, "author_name": "maw501", "author_url": "", "post_date": "12/14/2018 11:13:19", "content": "Hi Pete,
\n\nSo you are using JPG HPA? I think some people have managed to get PNG.
\n\nI struggle to believe the 2048 pixel JPGs wouldn't help at all, though initial training runs don't seem any better. It's bizarre - we are increasing the number of some rare classes by many multiples.
", "votes": null, "replies": [] }, { "id": 439175, "author_name": "petewills", "author_url": "", "post_date": "12/14/2018 20:29:07", "content": "No, I am converting the JPG to PNG 512 and from there to 256 and 128 and I'm still taking care of a couple of small problems. I also need to double-check on the images that the histograms are not way different from the test data histograms and do something if they are. After that, I'll probably start by adding in just the rare classes.\nI tried the big tif images a while back but got poor results. I will revisit that if I have time with a better downsampling of the tif files.
", "votes": null, "replies": [] }, { "id": 439178, "author_name": "maw501", "author_url": "", "post_date": "12/14/2018 20:40:37", "content": "Yeah, I'm doing similar - the jpg are 3 channel for each colour so I'm just extracting one as detailed above.
\n\nHere are the (raw) stats if you want them:
\n\nHPA stats (RGB):\nMean: [30, 17, 17]\nStdev: [45.5, 31.5, 45]
\n\nTrain stats (RGB):\nMean: [20.5, 13.45, 14]\nStdev: [38.22, 28.7, 39.8]
\n\nI'm normalising the HPA data separately as a consequence. I tried one run just adding data for the rare classes which improved recall for some classes but caused precision to plummet (=> some labels are unreliable?).
\n\nNext step: keep data for rare classes ~3k extra examples and then of the remaining 72k images, randomly add a sample of ~25k each epoch.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "438455": "There seems to be a several ways to handle this, none of which I can discern reliably as the 'correct' way. Here is one function to do it:\n\n def download(id_):\n try:\n image_dir, _, image_id = id_.partition('_')\n arr_out = np.zeros((512,512,3))\n for i, c in enumerate(['red', 'green', 'blue']):\n url = f'http://v18.proteinatlas.org/images/{image_dir}/{image_id}_{c}.jpg'\n r = requests.get(url)\n image = Image.open(BytesIO(r.content)).resize((512, 512), PIL.Image.LANCZOS).convert('L')\n arr_out[:,:,i] = np.array(image).astype('uint8')\n cv.imwrite(f'{hpa_dir}/{id_}.png', arr_out)\n except:\n print(f'{id_} broke...')\n\nNote that what is returned from the protein atlas site is actually a JPEG file with 3 channels. So we end up with **3 channels for each of red, green and blue**. The above function is merging these 3 channels into a single grayscale channel via the `.convert('L')` method which then themselves go into the image as a channel. So we go from 3x3 channels into 3 for the final image. The issue is that this is producing images that have very different intensity to the training set, particularly in the final B channel.\n\nI've tried following Alex's suggestions [here][1] where he has patiently tried to explain this but I'm none the wiser. \n\nAny clarification greatly appreciated,\n\nMark\n\nP. S. I'm ignoring yellow for now.\n\n [1]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984", "438470": "**Update**: I think per the attached that when we grab the 'red' image from the hpa website we should grab channel 0 from what the 3 channel JPG returns, then for the 'green' channel we grab channel 1 and for 'blue' it's channel 2. All the intensity of the images seems to lie here.", "438521": "I also faced with this problem. Yet I could not find any appropriate solution.\nIf I convert image to grayscale ,using cv2.imread(\"path_to_image.jpg\",0) where 0 indicades to read image in gs , or use max along channels for each pixels - > I get very low score after training BNinception proposed in the kernels compared to training without external dataset.\nMoreover If I mix original dataset with external dataset and use original dataset as validation - > I also get lower score. \nI could not find any relevant solution to this heterogeneity in the datasets. \nI would be grateful if someone will say how to fix it.", "438523": "Perhaps try to train on the extrnal and fine tune on the train? It seems like the train is of higher quality. I am going to try this next week.", "438612": "Mark\n\nThanks for this thread and the comments on the other one. I think I understand what has been said - it makes sense. \n\nI am comparing using the \"leak\" correspondence of jpg vs test data. It is easier comparing these very similar images.\n\nSo far as I can see, the procedure improves greatly the image comparison and also the histogram (x axis=pixel value, y axis=pixel count in single image) comparisons. The only caveats are that the jpg image conversions are smoother (less resolved, even when resizing with \"nearest\") than the challenge images and that the jpg conversion histograms are somewhat noisy - perhaps both effects from the evil jpg compression(?). The smoothing is a worry because there is a big gain harvested by upping the resolution - do we lose that with HPA data?\n\nThe histograms look similar enough to forgo the re-derivation of the image statistics for the lafoss fastai kernel.\n\nI had been having no luck training the HPA and started QC'ing the conversion a couple of days ago so this is a timely thread. I'll try the training again. It'll probably fail because most things fail in this business!", "438763": "yes, I also face with same problem, the normalized avr. of the RGB channel on the trainset is [0.081, 0.052, 0.055], but on the external data it's [0.054, 0.051, 0.031], the R and B channel is significantly decreased, maybe its mainly because the convert('L') function uses the formula: Y = 0.299 R + 0.587 G + 0.114 B. so I will try your method, thanks for your sharing!", "438770": "Thanks for sharing!\nI tried to match the mean and variance of HPA data to the main train data by inverting this coefficients. \nIt is still training but looks promising.", "438789": "Unfortunately, theoretically, the numbers go the other way I think.", "438902": "Hi Pete, \n\nSo you are using JPG HPA? I think some people have managed to get PNG. \n\nI struggle to believe the 2048 pixel JPGs wouldn't help at all, though initial training runs don't seem any better. It's bizarre - we are increasing the number of some rare classes by many multiples.", "439175": "No, I am converting the JPG to PNG 512 and from there to 256 and 128 and I'm still taking care of a couple of small problems. I also need to double-check on the images that the histograms are not way different from the test data histograms and do something if they are. After that, I'll probably start by adding in just the rare classes.\nI tried the big tif images a while back but got poor results. I will revisit that if I have time with a better downsampling of the tif files.", "439178": "Yeah, I'm doing similar - the jpg are 3 channel for each colour so I'm just extracting one as detailed above. \n\nHere are the (raw) stats if you want them:\n\nHPA stats (RGB):\nMean: [30, 17, 17]\nStdev: [45.5, 31.5, 45] \n \nTrain stats (RGB):\nMean: [20.5, 13.45, 14]\nStdev: [38.22, 28.7, 39.8] \n\nI'm normalising the HPA data separately as a consequence. I tried one run just adding data for the rare classes which improved recall for some classes but caused precision to plummet (=> some labels are unreliable?). \n\nNext step: keep data for rare classes ~3k extra examples and then of the remaining 72k images, randomly add a sample of ~25k each epoch." }, "source": "meta" }