Its my first segmentation competition and I have some hard time understanding the following code blocks + the purpose of using them.
\n1-
():\n dim=((,x.ndim))\n mean=x.mean(dim=dim,keepdim=)\n std=x.std(dim=dim,keepdim=)\n x=(x-mean)/(std+smooth)\n x[x>]=(x[x>]-)* +\n x[x<-]=(x[x<-]+)*-\n x\nWhy not just normalizing?
\nAlso, its recalled for each dataset separately. So, aren't we getting different scaling for each dataset? What is the intuition behind this? Based on my knowledge I don't think this is generalizable. Or maybe people just used it because it gave high public lb (overfitting)?
2-
\n\nTH=x.reshape(-).numpy()\nindex = -((TH) * CFG.chopping_percentile)\nTH: = np.partition(TH, index)[index]\nx[x>TH]=(TH)\n\nTH=x.reshape(-).numpy()\nindex = -((TH) * CFG.chopping_percentile)\nTH: = np.partition(TH, -index)[-index]\nx[x<TH]=(TH)\n\nx=(min_max_normalization(x.to(tc.float16)[])[]*).to(tc.uint8)\nWhat is happening here exactly? and how do we determine the chopping_percentile? hyperparam?
\nand the same code goes here in inference but with a totally different percentile value:
\nTH=[x.flatten().numpy() x output]\nTH=np.concatenate(TH)\nindex = -((TH) * CFG.th_percentile)\nTH: = np.partition(TH, index)[index]\n(TH)\nAlso, why the first code is used for the training \"images\", while the second one is used for the predicted \"masks\" (labels)?
\n3-
\nAnd the main question is:
\ndo i really need all these things :)?
\nThanks!
Its my first segmentation competition and I have some hard time understanding the following code blocks + the purpose of using them.
\n1-
():\n dim=((,x.ndim))\n mean=x.mean(dim=dim,keepdim=)\n std=x.std(dim=dim,keepdim=)\n x=(x-mean)/(std+smooth)\n x[x>]=(x[x>]-)* +\n x[x<-]=(x[x<-]+)*-\n x\nWhy not just normalizing?
\nAlso, its recalled for each dataset separately. So, aren't we getting different scaling for each dataset? What is the intuition behind this? Based on my knowledge I don't think this is generalizable. Or maybe people just used it because it gave high public lb (overfitting)?
2-
\n\nTH=x.reshape(-).numpy()\nindex = -((TH) * CFG.chopping_percentile)\nTH: = np.partition(TH, index)[index]\nx[x>TH]=(TH)\n\nTH=x.reshape(-).numpy()\nindex = -((TH) * CFG.chopping_percentile)\nTH: = np.partition(TH, -index)[-index]\nx[x<TH]=(TH)\n\nx=(min_max_normalization(x.to(tc.float16)[])[]*).to(tc.uint8)\nWhat is happening here exactly? and how do we determine the chopping_percentile? hyperparam?
\nand the same code goes here in inference but with a totally different percentile value:
\nTH=[x.flatten().numpy() x output]\nTH=np.concatenate(TH)\nindex = -((TH) * CFG.th_percentile)\nTH: = np.partition(TH, index)[index]\n(TH)\nAlso, why the first code is used for the training \"images\", while the second one is used for the predicted \"masks\" (labels)?
\n3-
\nAnd the main question is:
\ndo i really need all these things :)?
\nThanks!
I would say this is slightly related to this discussion https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/469022
\nI think you don’t need them but it could help if done well. I do think that the structure in this particular case is unnecessary. Scaling each dataset separately seems strange to me as well. I would have thought it to be better to scale together based off train and apply that scale to test. But that may also be different with the different resolutions of the train and test.
", "rawMarkdown": "I would say this is slightly related to this discussion https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/469022 \n\nI think you don’t need them but it could help if done well. I do think that the structure in this particular case is unnecessary. Scaling each dataset separately seems strange to me as well. I would have thought it to be better to scale together based off train and apply that scale to test. But that may also be different with the different resolutions of the train and test.", "votes": null }, { "id": "2611100", "postDate": "01/20/2024 14:59:40", "content": "It looks like shamanism, but it works quite effectively. I’ll show you using the first era as an example:
\nWith normalization:
\nepoch:0,loss:0.9154,score:0.2735,lr2.4942e-05: 100%|██████████| 4603/4603 [16:50<00:00, 4.56it/s] \nval-->loss:0.7888,score:0.7206: 100%|██████████| 250/250 [00:29<00:00, 8.56it/s]
Without normalization:
\nepoch:0,loss:0.9674,score:0.0396,lr5.0430e-06: 100%|██████████| 1151/1151 [04:37<00:00, 4.15it/s] \nval-->loss:0.9779,score:0.0471: 100%|██████████| 125/125 [00:08<00:00, 14.91it/s]
The result is obvious: score:0.7206 vs score:0.0471.
\nAn example to understand the work:
\nTo find the top k or bottom k elements without having to first sort the entire array.
\nimport numpy as np
\nimport torch as tc
x = np.random.randint(low=0, high=255, size=(10,10))
\nx = tc.tensor(x, dtype=tc.int32)
\nchopping_percentile = 1e-1
\ny = x.reshape(-1).numpy()
\nprint(f\"y.shape={y.shape}\")
\nk = -int(len(y) * chopping_percentile)
\nprint(f\"k={k}\")
// To find the top k:
\ntop_k = np.partition(y, -k)[-k:] // Results are not sorted
\nprint(f\"top_k={top_k}\")
\nSorted from largest to smallest
\n// np.sort(top_k)[::-1]
// To find the lower k:
\nbottom_k = np.partition(y, k)[:k] // Results are not sorted
\nprint(f\"bottom_k={bottom_k}\")
\n// np.sort(bottom_k) // Sorted from smallest to largest
TH = x.reshape(-1).numpy()
\nindex = -int(len(TH) * chopping_percentile)
\nTH:int = np.partition(TH, index)[index]
\nprint(f\"TH (High Limit)={TH}\")
\nx[x > TH] = int(TH) // Replace values exceeding threshold with threshold value
TH = x.reshape(-1).numpy()
\nindex = -int(len(TH) * chopping_percentile)
\nTH:int = np.partition(TH, -index)[-index]
\nprint(f\"TH (Low Limit)={TH}\")
\nx[x < TH] = int(TH) // Replace value less than threshold with threshold value
print(f\"x={x}\")
", "rawMarkdown": "It looks like shamanism, but it works quite effectively. I’ll show you using the first era as an example:\n\nWith normalization:\nepoch:0,loss:0.9154,score:0.2735,lr2.4942e-05: 100%|██████████| 4603/4603 [16:50<00:00, 4.56it/s] \nval-->loss:0.7888,score:0.7206: 100%|██████████| 250/250 [00:29<00:00, 8.56it/s]\n\nWithout normalization:\nepoch:0,loss:0.9674,score:0.0396,lr5.0430e-06: 100%|██████████| 1151/1151 [04:37<00:00, 4.15it/s] \nval-->loss:0.9779,score:0.0471: 100%|██████████| 125/125 [00:08<00:00, 14.91it/s]\n\nThe result is obvious: score:0.7206 vs score:0.0471.\n\nAn example to understand the work:\n\nTo find the top k or bottom k elements without having to first sort the entire array.\nimport numpy as np\nimport torch as tc\n\nx = np.random.randint(low=0, high=255, size=(10,10))\nx = tc.tensor(x, dtype=tc.int32)\nchopping_percentile = 1e-1\ny = x.reshape(-1).numpy()\nprint(f\"y.shape={y.shape}\")\nk = -int(len(y) * chopping_percentile)\nprint(f\"k={k}\")\n\n// To find the top k:\ntop_k = np.partition(y, -k)[-k:] // Results are not sorted\nprint(f\"top_k={top_k}\")\nSorted from largest to smallest\n// np.sort(top_k)[::-1] \n\n// To find the lower k:\nbottom_k = np.partition(y, k)[:k] // Results are not sorted\nprint(f\"bottom_k={bottom_k}\")\n// np.sort(bottom_k) // Sorted from smallest to largest\n\nTH = x.reshape(-1).numpy()\nindex = -int(len(TH) * chopping_percentile)\nTH:int = np.partition(TH, index)[index]\nprint(f\"TH (High Limit)={TH}\")\nx[x > TH] = int(TH) // Replace values exceeding threshold with threshold value\n\nTH = x.reshape(-1).numpy()\nindex = -int(len(TH) * chopping_percentile)\nTH:int = np.partition(TH, -index)[-index]\nprint(f\"TH (Low Limit)={TH}\")\nx[x < TH] = int(TH) // Replace value less than threshold with threshold value\n\nprint(f\"x={x}\")", "votes": null }, { "id": "2635495", "postDate": "02/04/2024 12:46:55", "content": "Similarly, in the public training notebook, there's an application of a threshold to the model input with the expression y = data['mask'] >= 127. Is it better to apply a threshold of 0.5 to the training input as well?
", "rawMarkdown": "Similarly, in the public training notebook, there's an application of a threshold to the model input with the expression y = data['mask'] >= 127. Is it better to apply a threshold of 0.5 to the training input as well?", "votes": null }, { "id": "2635520", "postDate": "02/04/2024 12:53:54", "content": "If is applied to input, input labels are 0 or 255. It doesn't matter as far is > 0
\nEDIT: Unless they are interpolating the input masks, then yes, 127 seems reasonable. I haven't good experience interpolating masks.
", "rawMarkdown": "If is applied to input, input labels are 0 or 255. It doesn't matter as far is > 0\n\nEDIT: Unless they are interpolating the input masks, then yes, 127 seems reasonable. I haven't good experience interpolating masks.", "votes": null } ], "comments": [ { "id": 2608534, "author_name": "sacuscreed", "author_url": "", "post_date": "01/18/2024 22:06:15", "content": "My approach is slighly different. But as I understand, first training TH is for clip outliers while second mask TH is for artificially binarize the final output (allowing more or less positives).
\nBut yes, general opinion is that score is mask TH overfitted.
I would say this is slightly related to this discussion https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/469022
\nI think you don’t need them but it could help if done well. I do think that the structure in this particular case is unnecessary. Scaling each dataset separately seems strange to me as well. I would have thought it to be better to scale together based off train and apply that scale to test. But that may also be different with the different resolutions of the train and test.
", "votes": null, "replies": [] }, { "id": 2611100, "author_name": "konstantinboyko", "author_url": "", "post_date": "01/20/2024 14:59:40", "content": "It looks like shamanism, but it works quite effectively. I’ll show you using the first era as an example:
\nWith normalization:
\nepoch:0,loss:0.9154,score:0.2735,lr2.4942e-05: 100%|██████████| 4603/4603 [16:50<00:00, 4.56it/s] \nval-->loss:0.7888,score:0.7206: 100%|██████████| 250/250 [00:29<00:00, 8.56it/s]
Without normalization:
\nepoch:0,loss:0.9674,score:0.0396,lr5.0430e-06: 100%|██████████| 1151/1151 [04:37<00:00, 4.15it/s] \nval-->loss:0.9779,score:0.0471: 100%|██████████| 125/125 [00:08<00:00, 14.91it/s]
The result is obvious: score:0.7206 vs score:0.0471.
\nAn example to understand the work:
\nTo find the top k or bottom k elements without having to first sort the entire array.
\nimport numpy as np
\nimport torch as tc
x = np.random.randint(low=0, high=255, size=(10,10))
\nx = tc.tensor(x, dtype=tc.int32)
\nchopping_percentile = 1e-1
\ny = x.reshape(-1).numpy()
\nprint(f\"y.shape={y.shape}\")
\nk = -int(len(y) * chopping_percentile)
\nprint(f\"k={k}\")
// To find the top k:
\ntop_k = np.partition(y, -k)[-k:] // Results are not sorted
\nprint(f\"top_k={top_k}\")
\nSorted from largest to smallest
\n// np.sort(top_k)[::-1]
// To find the lower k:
\nbottom_k = np.partition(y, k)[:k] // Results are not sorted
\nprint(f\"bottom_k={bottom_k}\")
\n// np.sort(bottom_k) // Sorted from smallest to largest
TH = x.reshape(-1).numpy()
\nindex = -int(len(TH) * chopping_percentile)
\nTH:int = np.partition(TH, index)[index]
\nprint(f\"TH (High Limit)={TH}\")
\nx[x > TH] = int(TH) // Replace values exceeding threshold with threshold value
TH = x.reshape(-1).numpy()
\nindex = -int(len(TH) * chopping_percentile)
\nTH:int = np.partition(TH, -index)[-index]
\nprint(f\"TH (Low Limit)={TH}\")
\nx[x < TH] = int(TH) // Replace value less than threshold with threshold value
print(f\"x={x}\")
", "votes": null, "replies": [] }, { "id": 2635495, "author_name": "aravind36", "author_url": "", "post_date": "02/04/2024 12:46:55", "content": "Similarly, in the public training notebook, there's an application of a threshold to the model input with the expression y = data['mask'] >= 127. Is it better to apply a threshold of 0.5 to the training input as well?
", "votes": null, "replies": [ { "id": 2635520, "author_name": "sacuscreed", "author_url": "", "post_date": "02/04/2024 12:53:54", "content": "If is applied to input, input labels are 0 or 255. It doesn't matter as far is > 0
\nEDIT: Unless they are interpolating the input masks, then yes, 127 seems reasonable. I haven't good experience interpolating masks.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2608526": "Its my first segmentation competition and I have some hard time understanding the following code blocks + the purpose of using them. \n1-\n```python\ndef norm_with_clip(x:tc.Tensor,smooth=1e-5):\n dim=list(range(1,x.ndim))\n mean=x.mean(dim=dim,keepdim=True)\n std=x.std(dim=dim,keepdim=True)\n x=(x-mean)/(std+smooth)\n x[x>5]=(x[x>5]-5)*1e-3 +5\n x[x<-3]=(x[x<-3]+3)*1e-3-3\n return x\n```\nWhy not just normalizing? \nAlso, its recalled for each dataset separately. So, aren't we getting different scaling for each dataset? What is the intuition behind this? Based on my knowledge I don't think this is generalizable. Or maybe people just used it because it gave high public lb (overfitting)?\n\n2-\n```python\n########################################################################\nTH=x.reshape(-1).numpy()\nindex = -int(len(TH) * CFG.chopping_percentile)\nTH:int = np.partition(TH, index)[index]\nx[x>TH]=int(TH)\n########################################################################\nTH=x.reshape(-1).numpy()\nindex = -int(len(TH) * CFG.chopping_percentile)\nTH:int = np.partition(TH, -index)[-index]\nx[x