![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3585046%2F1bcc8b573bf3fe940de310576091c8f6%2F1184_004.png?generation=1659015666832475&alt=media\")
Has anyone else encountered an issue where normalising the images in the inference notebook using Torchstain causes the \"Notebook Threw Exception\" error when running on the hidden test set?
\nI've tested the inference notebook with Torchstain and training images, but was unable to find an issue. The batch size was also reduced from 16 to 4 to check if GPU memory was exceeding during the hidden test set run, but that didn't help either.
", "messages": [ { "id": "1871324", "postDate": "07/26/2022 08:02:09", "content": "Has anyone else encountered an issue where normalising the images in the inference notebook using Torchstain causes the \"Notebook Threw Exception\" error when running on the hidden test set?
\nI've tested the inference notebook with Torchstain and training images, but was unable to find an issue. The batch size was also reduced from 16 to 4 to check if GPU memory was exceeding during the hidden test set run, but that didn't help either.
", "rawMarkdown": "Has anyone else encountered an issue where normalising the images in the inference notebook using Torchstain causes the \"Notebook Threw Exception\" error when running on the hidden test set?\n\nI've tested the inference notebook with Torchstain and training images, but was unable to find an issue. The batch size was also reduced from 16 to 4 to check if GPU memory was exceeding during the hidden test set run, but that didn't help either.", "votes": null }, { "id": "1873821", "postDate": "07/27/2022 23:21:15", "content": "Can you share some of your code so we can take a look?
\nThe Devastator.
", "rawMarkdown": "Can you share some of your code so we can take a look?\n\nThe Devastator.", "votes": null }, { "id": "1874295", "postDate": "07/28/2022 07:20:58", "content": "I encountered the same problem. And there is the code that generated the error when running on the hidden test set: https://www.kaggle.com/code/qiaochu02/inference-fastai-baseline-stain-with-sta-infer. Looking forward to your reply.
", "rawMarkdown": "I encountered the same problem. And there is the code that generated the error when running on the hidden test set: https://www.kaggle.com/code/qiaochu02/inference-fastai-baseline-stain-with-sta-infer. Looking forward to your reply.", "votes": null }, { "id": "1874453", "postDate": "07/28/2022 09:31:53", "content": "Out of the 3159 tiles I tried to stain normalize in numpy Spyder I got this error message on one tile:
\nFile C:\\py\\stain.py:121 in
\n Inorm, H, E = normalizeStaining(img = img,Io = 140,alpha = 1,beta = 0.25)
File C:\\py\\stain.py:51 in normalizeStaining
\n eigvals, eigvecs = np.linalg.eigh(np.cov(ODhat.T))
File <__array_function__ internals>:180 in eigh
\nFile ~\\anaconda3\\envs\\tf-gpu\\lib\\site-packages\\numpy\\linalg\\linalg.py:1458 in eigh
\n w, vt = gufunc(a, signature=signature, extobj=extobj)
File ~\\anaconda3\\envs\\tf-gpu\\lib\\site-packages\\numpy\\linalg\\linalg.py:94 in _raise_linalgerror_eigenvalues_nonconvergence
\n raise LinAlgError(\"Eigenvalues did not converge\")
Inorm, H, E = normalizeStaining(img = img,Io = 140,alpha = 1,beta = 0.25) gave error only on this tile
\nInorm, H, E = normalizeStaining(img = img,Io = 240,alpha = 1,beta = 0.25) worked ok
Depending on Io value and beta ODhat can be empty array giving error.
\n# reshape image\nimg = img.reshape((-1,3))\n\n# calculate optical density\nOD = -np.log((img.astype(float)+1)/Io)\n\n# remove transparent pixels\nODhat = OD[~np.any(OD<beta, axis=1)]\n\n# compute eigenvectors\neigvals, eigvecs = np.linalg.eigh(np.cov(ODhat.T))\nhere is the torchstain code https://github.com/EIDOSLAB/torchstain/blob/main/torchstain/normalizers/torch_macenko_normalizer.py
\nHere is the 1024X1024 tile that gave error 1184_004.png
\n
T = TF.Compose([\n TF.ToTensor(),\n TF.Lambda(lambda x: x*255)\n ])\n\ntarget = cv2.cvtColor(cv2.imread(\"../input/torch-stain-target/SAS_21883_001_107.png\"), cv2.COLOR_BGR2RGB)\ntarget = cv2.resize(target, (tile_sz, tile_sz))\nstain_normalizer = torchstain.MacenkoNormalizer(backend='torch')\nstain_normalizer.fit(T(target))\n\nTF.Compose([TF.Lambda(lambda x: stain_normalizer.normalize(I=T(x), stains=False)[0])])\nThe above code works fine during training and normalisation is applied as expected. I also tested applying Torchstain in the method @qiaochu02 used in his notebook to see if it made a change, but unfortunately the same notebook exception occurs.
\nFor me the notebook crashes 3-4 minutes after submitting to the competition which is when it starts to load the images based on the notebook runtime. It could be due to the same error @tomkkk mentioned.
", "rawMarkdown": "thedevastator I am directly passing the function through PyTorch transforms like below\n\n```\nT = TF.Compose([\n TF.ToTensor(),\n TF.Lambda(lambda x: x*255)\n ])\n\ntarget = cv2.cvtColor(cv2.imread(\"../input/torch-stain-target/SAS_21883_001_107.png\"), cv2.COLOR_BGR2RGB)\ntarget = cv2.resize(target, (tile_sz, tile_sz))\nstain_normalizer = torchstain.MacenkoNormalizer(backend='torch')\nstain_normalizer.fit(T(target))\n\nTF.Compose([TF.Lambda(lambda x: stain_normalizer.normalize(I=T(x), stains=False)[0])])\n```\n\nThe above code works fine during training and normalisation is applied as expected. I also tested applying Torchstain in the method @qiaochu02 used in his [notebook ](https://www.kaggle.com/code/qiaochu02/inference-fastai-baseline-stain-with-sta-infer)to see if it made a change, but unfortunately the same notebook exception occurs.\n\nFor me the notebook crashes 3-4 minutes after submitting to the competition which is when it starts to load the images based on the notebook runtime. It could be due to the same error @tomkkk mentioned.", "votes": null }, { "id": "1878221", "postDate": "07/31/2022 10:17:16", "content": "a possible solution is:
\n your train set = HPA + HPA stained normalised
during test, you need not have to apply stained normalisation at all.
", "rawMarkdown": "a possible solution is:\n your train set = HPA + HPA stained normalised\n\nduring test, you need not have to apply stained normalisation at all.", "votes": null }, { "id": "1878644", "postDate": "07/31/2022 14:17:09", "content": "Yes, that's what I ended up doing since there seems to be no solution yet. But I wanted to check if there was an improvement in the LB score when stain normalisation was applied.
", "rawMarkdown": "Yes, that's what I ended up doing since there seems to be no solution yet. But I wanted to check if there was an improvement in the LB score when stain normalisation was applied.", "votes": null } ], "comments": [ { "id": 1873821, "author_name": "thedevastator", "author_url": "", "post_date": "07/27/2022 23:21:15", "content": "Can you share some of your code so we can take a look?
\nThe Devastator.
", "votes": null, "replies": [ { "id": 1874295, "author_name": "qiaochu02", "author_url": "", "post_date": "07/28/2022 07:20:58", "content": "I encountered the same problem. And there is the code that generated the error when running on the hidden test set: https://www.kaggle.com/code/qiaochu02/inference-fastai-baseline-stain-with-sta-infer. Looking forward to your reply.
", "votes": null, "replies": [] }, { "id": 1874564, "author_name": "yovinyahathugoda", "author_url": "", "post_date": "07/28/2022 10:56:22", "content": "@thedevastator I am directly passing the function through PyTorch transforms like below
\nT = TF.Compose([\n TF.ToTensor(),\n TF.Lambda(lambda x: x*255)\n ])\n\ntarget = cv2.cvtColor(cv2.imread(\"../input/torch-stain-target/SAS_21883_001_107.png\"), cv2.COLOR_BGR2RGB)\ntarget = cv2.resize(target, (tile_sz, tile_sz))\nstain_normalizer = torchstain.MacenkoNormalizer(backend='torch')\nstain_normalizer.fit(T(target))\n\nTF.Compose([TF.Lambda(lambda x: stain_normalizer.normalize(I=T(x), stains=False)[0])])\nThe above code works fine during training and normalisation is applied as expected. I also tested applying Torchstain in the method @qiaochu02 used in his notebook to see if it made a change, but unfortunately the same notebook exception occurs.
\nFor me the notebook crashes 3-4 minutes after submitting to the competition which is when it starts to load the images based on the notebook runtime. It could be due to the same error @tomkkk mentioned.
", "votes": null, "replies": [] } ] }, { "id": 1874453, "author_name": "tomkkk", "author_url": "", "post_date": "07/28/2022 09:31:53", "content": "Out of the 3159 tiles I tried to stain normalize in numpy Spyder I got this error message on one tile:
\nFile C:\\py\\stain.py:121 in
\n Inorm, H, E = normalizeStaining(img = img,Io = 140,alpha = 1,beta = 0.25)
File C:\\py\\stain.py:51 in normalizeStaining
\n eigvals, eigvecs = np.linalg.eigh(np.cov(ODhat.T))
File <__array_function__ internals>:180 in eigh
\nFile ~\\anaconda3\\envs\\tf-gpu\\lib\\site-packages\\numpy\\linalg\\linalg.py:1458 in eigh
\n w, vt = gufunc(a, signature=signature, extobj=extobj)
File ~\\anaconda3\\envs\\tf-gpu\\lib\\site-packages\\numpy\\linalg\\linalg.py:94 in _raise_linalgerror_eigenvalues_nonconvergence
\n raise LinAlgError(\"Eigenvalues did not converge\")
Inorm, H, E = normalizeStaining(img = img,Io = 140,alpha = 1,beta = 0.25) gave error only on this tile
\nInorm, H, E = normalizeStaining(img = img,Io = 240,alpha = 1,beta = 0.25) worked ok
Depending on Io value and beta ODhat can be empty array giving error.
\n# reshape image\nimg = img.reshape((-1,3))\n\n# calculate optical density\nOD = -np.log((img.astype(float)+1)/Io)\n\n# remove transparent pixels\nODhat = OD[~np.any(OD<beta, axis=1)]\n\n# compute eigenvectors\neigvals, eigvecs = np.linalg.eigh(np.cov(ODhat.T))\nhere is the torchstain code https://github.com/EIDOSLAB/torchstain/blob/main/torchstain/normalizers/torch_macenko_normalizer.py
\nHere is the 1024X1024 tile that gave error 1184_004.png
\n
a possible solution is:
\n your train set = HPA + HPA stained normalised
during test, you need not have to apply stained normalisation at all.
", "votes": null, "replies": [ { "id": 1878644, "author_name": "yovinyahathugoda", "author_url": "", "post_date": "07/31/2022 14:17:09", "content": "Yes, that's what I ended up doing since there seems to be no solution yet. But I wanted to check if there was an improvement in the LB score when stain normalisation was applied.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1871324": "Has anyone else encountered an issue where normalising the images in the inference notebook using Torchstain causes the \"Notebook Threw Exception\" error when running on the hidden test set?\n\nI've tested the inference notebook with Torchstain and training images, but was unable to find an issue. The batch size was also reduced from 16 to 4 to check if GPU memory was exceeding during the hidden test set run, but that didn't help either.", "1873821": "Can you share some of your code so we can take a look?\n\nThe Devastator.", "1874295": "I encountered the same problem. And there is the code that generated the error when running on the hidden test set: https://www.kaggle.com/code/qiaochu02/inference-fastai-baseline-stain-with-sta-infer. Looking forward to your reply.", "1874453": "Out of the 3159 tiles I tried to stain normalize in numpy Spyder I got this error message on one tile:\n\n File C:\\py\\stain.py:121 in