I'm not familiar with segmentation nor dice metric.\nI've seen various codes to calculate dice coefficient, was confused, searched dice metric for other competitions, and found out this post in the Severstal: Steel Defect Detection competition. (The metric there is same as here.)\nLet's look at the Evaluation section in Overview tab.
\n\n\nThe Dice coefficient is defined to be 1 when both X and Y are empty. The leaderboard score is the mean of the Dice coefficients for each (Image, Label) pair in the test set.\n
The dice metric is computed by taking the mean of the Dice coefficients for each (Image, Label) pair in the test set.
\n\nI've revised the above link's code which is written for pytorch to work in numpy. Correct me if I'm wrong.
\n\n```python\ndef single_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (height, width)\n intersection = np.sum(y_true * y_pred_bin)\n if (np.sum(y_true)==0) and (np.sum(y_pred_bin)==0):\n return 1\n return (2*intersection) / (np.sum(y_true) + np.sum(y_pred_bin))
\n\ndef mean_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (n_samples, height, width, n_channels)\n batch_size = y_true.shape[0]\n channel_num = y_true.shape[-1]\n mean_dice_channel = 0.\n for i in range(batch_size):\n for j in range(channel_num):\n channel_dice = single_dice_coef(y_true[i, :, :, j], y_pred_bin[i, :, :, j])\n mean_dice_channel += channel_dice/(channel_num*batch_size)\n return mean_dice_channel\n```
\n\nHere, y_pred_bin is the rounded binary predicted mask.
Code edited with Kha Vo's suggestion
", "messages": [ { "id": 656303, "postDate": "2019-10-24T06:07:39.750Z", "content": "I'm not familiar with segmentation nor dice metric.\nI've seen various codes to calculate dice coefficient, was confused, searched dice metric for other competitions, and found out this post in the Severstal: Steel Defect Detection competition. (The metric there is same as here.)\nLet's look at the Evaluation section in Overview tab.
\n\n\nThe Dice coefficient is defined to be 1 when both X and Y are empty. The leaderboard score is the mean of the Dice coefficients for each (Image, Label) pair in the test set.\n
The dice metric is computed by taking the mean of the Dice coefficients for each (Image, Label) pair in the test set.
\n\nI've revised the above link's code which is written for pytorch to work in numpy. Correct me if I'm wrong.
\n\n```python\ndef single_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (height, width)\n intersection = np.sum(y_true * y_pred_bin)\n if (np.sum(y_true)==0) and (np.sum(y_pred_bin)==0):\n return 1\n return (2*intersection) / (np.sum(y_true) + np.sum(y_pred_bin))
\n\ndef mean_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (n_samples, height, width, n_channels)\n batch_size = y_true.shape[0]\n channel_num = y_true.shape[-1]\n mean_dice_channel = 0.\n for i in range(batch_size):\n for j in range(channel_num):\n channel_dice = single_dice_coef(y_true[i, :, :, j], y_pred_bin[i, :, :, j])\n mean_dice_channel += channel_dice/(channel_num*batch_size)\n return mean_dice_channel\n```
\n\nHere, y_pred_bin is the rounded binary predicted mask.
Code edited with Kha Vo's suggestion
", "rawMarkdown": "I'm not familiar with segmentation nor dice metric.\nI've seen various codes to calculate dice coefficient, was confused, searched dice metric for other competitions, and found out [this](https://www.kaggle.com/wh1tezzz/correct-dice-metrics-for-this-competition) post in the [Severstal: Steel Defect Detection](https://www.kaggle.com/c/severstal-steel-defect-detection) competition. (The metric there is same as here.)\nLet's look at the Evaluation section in Overview tab.\n\n```\nThe Dice coefficient is defined to be 1 when both X and Y are empty. The leaderboard score is the mean of the Dice coefficients for each (Image, Label) pair in the test set.\n```\n\nThe dice metric is computed by taking the mean of the Dice coefficients for **each (Image, Label) pair** in the test set.\n\nI've revised the above link's code which is written for pytorch to work in numpy. Correct me if I'm wrong.\n\n```python\ndef single_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (height, width)\n intersection = np.sum(y_true * y_pred_bin)\n if (np.sum(y_true)==0) and (np.sum(y_pred_bin)==0):\n return 1\n return (2*intersection) / (np.sum(y_true) + np.sum(y_pred_bin))\n\ndef mean_dice_coef(y_true, y_pred_bin):\n\t# shape of y_true and y_pred_bin: (n_samples, height, width, n_channels)\n batch_size = y_true.shape[0]\n channel_num = y_true.shape[-1]\n mean_dice_channel = 0.\n for i in range(batch_size):\n for j in range(channel_num):\n channel_dice = single_dice_coef(y_true[i, :, :, j], y_pred_bin[i, :, :, j])\n mean_dice_channel += channel_dice/(channel_num*batch_size)\n return mean_dice_channel\n```\n\nHere, `y_pred_bin` is the rounded binary predicted mask.\n\n*Code edited with Kha Vo's suggestion*", "votes": 19 }, { "id": 668770, "postDate": "2019-11-08T20:29:11.037Z", "content": "Here is another alternative for correct dice metric.
\n\nimport segmentation_models as sm\ncorrect_dice = sm.metrics.FScore(per_image=True, threshold=0.5)\nisnot is F1 score? are both same?
", "rawMarkdown": "isnot is F1 score? are both same?" } ] }, { "id": 656321, "postDate": "2019-10-24T06:37:07.293Z", "content": "Your code is still not the correct dice metric used for LB scoring, because the inclusion of smooth
I used the following correct dice metric (for a single 2D layer of pred and ground truth). For mean dice, one can simply wrap it into an outside wrapper:\n`
\n\ndef dice(img1, img2):
\n\nimg1 = np.asarray(img1).astype(np.bool)\nimg2 = np.asarray(img2).astype(np.bool)\nif img1.sum() + img2.sum() == 0: return 1\nintersection = np.logical_and(img1, img2)\nreturn 2. * intersection.sum() / (img1.sum() + img2.sum())\n`
", "rawMarkdown": "Your code is still not the correct dice metric used for LB scoring, because the inclusion of `smooth`\n\nI used the following correct dice metric (for a single 2D layer of pred and ground truth). For mean dice, one can simply wrap it into an outside wrapper:\n`\n\ndef dice(img1, img2):\n\n img1 = np.asarray(img1).astype(np.bool)\n img2 = np.asarray(img2).astype(np.bool)\n if img1.sum() + img2.sum() == 0: return 1\n intersection = np.logical_and(img1, img2)\n return 2. * intersection.sum() / (img1.sum() + img2.sum())\n\n`", "votes": 5, "replies": [ { "id": 656334, "postDate": "2019-10-24T06:46:36.830Z", "content": "Oh so we could just return 1 if ytrue and ypred both have no masks. I guess we need the smooth thing when we want to use dice coef as loss function. I'll fix the code above. Thank you.
", "rawMarkdown": "Oh so we could just return 1 if ytrue and ypred both have no masks. I guess we need the smooth thing when we want to use dice coef as loss function. I'll fix the code above. Thank you.", "votes": 1 }, { "id": 658384, "postDate": "2019-10-26T00:03:51.650Z", "content": "I think if the smooth term is small enough, 1e-9 for example, it should not be an issue.
", "rawMarkdown": "I think if the smooth term is small enough, 1e-9 for example, it should not be an issue. ", "votes": 3 } ] }, { "id": 659066, "postDate": "2019-10-27T02:44:56.293Z", "content": "EDIT: The code in the original post is more efficient for 1 fold. Below is code for the entire 5-fold OOF.
\n\nHere is what I used for Steel Comp. Note that you cannot compile this into a model. You must make all your validation predictions and then apply this to each pair of true and predicted and take an average.
\n\ndef dice_coef2(y_true, y_pred):\n y_true_f = y_true.flatten()\n y_pred_f = y_pred.flatten()\n union = np.sum(y_true_f) + np.sum(y_pred_f)\n if union==0: return 1\n intersection = np.sum(y_true_f * y_pred_f)\n return 2. * intersection / union\nAnd if you have both true and predicted rle in pandas dataframe, do this:
sub['dice'] = sub.apply(lambda x: dice_coef2( rle2mask(x['true']), \n rle2mask(x['predicted']) ),axis=1)\nprint(sub['dice'].mean())\nThank you for sharing your code, Chris. I was using the version with 'flatten' which is the same as the code you provided, but then my validation score was much lower than the public leaderboard score. I switched to the above metric that I wrote, and validation score aligned with public leaderboard score better. (but still quite lower, I don't know why)
\n\nThe metric you provided calculates dice coefficient once for all of the flattened pixels. However, according to the metric described in the Evaluation section, we need to calculate dice coefficient for each (Image, Label) pair then average them. So two versions are calculating different things.
This is the code that shows the difference.
\n\n```\nimport numpy as np\nnp.random.seed(0)\ntrue = np.random.rand(10, 5, 5, 4)>0.5\npred = np.random.rand(10, 5, 5, 4)>0.5
\n\ndef single_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (height, width)\n intersection = np.sum(y_true * y_pred_bin)\n if (np.sum(y_true)==0) and (np.sum(y_pred_bin)==0):\n return 1\n return (2*intersection) / (np.sum(y_true) + np.sum(y_pred_bin))
\n\ndef mean_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (n_samples, height, width, n_channels)\n batch_size = y_true.shape[0]\n channel_num = y_true.shape[-1]\n mean_dice_channel = 0.\n for i in range(batch_size):\n for j in range(channel_num):\n channel_dice = single_dice_coef(y_true[i, :, :, j], y_pred_bin[i, :, :, j])\n mean_dice_channel += channel_dice/(channel_num*batch_size)\n return mean_dice_channel
\n\ndef dice_coef2(y_true, y_pred):\n y_true_f = y_true.flatten()\n y_pred_f = y_pred.flatten()\n union = np.sum(y_true_f) + np.sum(y_pred_f)\n if union==0: return 1\n intersection = np.sum(y_true_f * y_pred_f)\n return 2. * intersection / union
\n\nprint(mean_dice_coef(true, pred))\nprint(dice_coef2(true, pred))
\n\n```
", "rawMarkdown": "Thank you for sharing your code, Chris. I was using the version with 'flatten' which is the same as the code you provided, but then my validation score was much lower than the public leaderboard score. I switched to the above metric that I wrote, and validation score aligned with public leaderboard score better. (but still quite lower, I don't know why)\n\nThe metric you provided calculates dice coefficient once for all of the flattened pixels. However, according to the metric described in the Evaluation section, we need to calculate dice coefficient for each `(Image, Label) pair` then average them. So two versions are calculating different things.\n\nThis is the code that shows the difference.\n\n```\nimport numpy as np\nnp.random.seed(0)\ntrue = np.random.rand(10, 5, 5, 4)>0.5\npred = np.random.rand(10, 5, 5, 4)>0.5\n\ndef single_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (height, width)\n intersection = np.sum(y_true * y_pred_bin)\n if (np.sum(y_true)==0) and (np.sum(y_pred_bin)==0):\n return 1\n return (2*intersection) / (np.sum(y_true) + np.sum(y_pred_bin))\n\ndef mean_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (n_samples, height, width, n_channels)\n batch_size = y_true.shape[0]\n channel_num = y_true.shape[-1]\n mean_dice_channel = 0.\n for i in range(batch_size):\n for j in range(channel_num):\n channel_dice = single_dice_coef(y_true[i, :, :, j], y_pred_bin[i, :, :, j])\n mean_dice_channel += channel_dice/(channel_num*batch_size)\n return mean_dice_channel\n\ndef dice_coef2(y_true, y_pred):\n y_true_f = y_true.flatten()\n y_pred_f = y_pred.flatten()\n union = np.sum(y_true_f) + np.sum(y_pred_f)\n if union==0: return 1\n intersection = np.sum(y_true_f * y_pred_f)\n return 2. * intersection / union\n\nprint(mean_dice_coef(true, pred))\nprint(dice_coef2(true, pred))\n\n# 0.4884357140842496\n# 0.499001996007984\n```", "votes": 2 }, { "id": 659315, "postDate": "2019-10-27T12:57:34.850Z", "content": "Yes your code is more efficient than mine because it can be compiled directly into a model. My code works but you need to save all your rle into a dataframe and then apply my metric to each rle pair and then average. I like your code. I'm gonna use that. Thanks for posting.
model.compile(optimizer='adam', loss=bce_jaccard_loss, metrics=[mean_dice_coef])\nhistory = model.fit_generator(train_batches, validation_data=valid_batches)\nmodel.compile(optimizer='adam', loss=bce_jaccard_loss, metrics=[dice_coef2])\nhistory = model.fit_generator(train_batches, validation_data=valid_batches)\nfor k in range(len(EPOCHS)):\n model.train_generator(train_batches, epochs=1)\n val['predicted_rle'] = mask2rle_batches( model.predict_generator(valid_batches) )\n val['dice'] = val.apply(lambda x: dice_coef2( rle2mask(x['true_rle']), \n rle2mask(x['predicted_rle']) ),axis=1)\n print(val['dice'].mean())\nhello @harangdev i am having this issue that my val dice metrics is .74 but lb is .499 i am using the metrics that you have provided and i think that you had overcome this issue so can you help
", "rawMarkdown": "hello @harangdev i am having this issue that my val dice metrics is .74 but lb is .499 i am using the metrics that you have provided and i think that you had overcome this issue so can you help", "isDeleted": true }, { "id": 666267, "postDate": "2019-11-06T00:21:35.633Z", "content": "@shekharrastogi Well, there can be many reasons. Your validation score seems too high. Maybe a bug in validation pipeline?
", "rawMarkdown": "@shekharrastogi Well, there can be many reasons. Your validation score seems too high. Maybe a bug in validation pipeline?" }, { "id": 666276, "postDate": "2019-11-06T00:32:48.143Z", "content": "@shekharrastogi Maybe you can check whether your training data mix with validating data. I had that problem few weeks ago, and the validation score also very high.
", "rawMarkdown": "@shekharrastogi Maybe you can check whether your training data mix with validating data. I had that problem few weeks ago, and the validation score also very high." } ] }, { "id": 671761, "postDate": "2019-11-13T06:03:29.053Z", "content": "We can make our Dice metric more accurate (at estimating LB) by having our Dice metric perform post processing. I published code here. (See step 2 of \"How to save time\").
", "rawMarkdown": "We can make our Dice metric more accurate (at estimating LB) by having our Dice metric perform post processing. I published code [here][1]. (See step 2 of \"How to save time\").\n\n[1]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/117021", "votes": 1, "replies": [ { "id": 671772, "postDate": "2019-11-13T06:21:21.640Z", "content": "yes, i also put kaggle lb metric in the metric meter in my training loop. it is useful. but it is important to note that:
\n\nyou will also have to monitor your back propagation loss at the same time.
", "rawMarkdown": "yes, i also put kaggle lb metric in the metric meter in my training loop. it is useful. but it is important to note that:\n\n1. kaggle lb metric is threshold dependent\n2. kaggle lb metric is non smooth\n\nyou will also have to monitor your back propagation loss at the same time.\n", "votes": 2 } ] }, { "id": 660276, "postDate": "2019-10-29T00:30:26.837Z", "content": "Can we use this calculation as dice loss? What will be the difference between using the corrected one and previous one as loss functions?
", "rawMarkdown": "Can we use this calculation as dice loss? What will be the difference between using the corrected one and previous one as loss functions?", "votes": 1, "replies": [ { "id": 660299, "postDate": "2019-10-29T01:08:32.137Z", "content": "\ndef dice_loss(y_true, y_pred, axis=(0,1,2), smooth=1):\n intersection = K.sum(y_true * y_pred, axis=axis)\n total_y_true = K.sum(y_true, axis=axis)\n total_y_pred = K.sum(y_pred, axis=axis)\n loss = (2*intersection+smooth) / (total_y_true+total_y_pred+smooth)\n if axis is None:\n return 1-loss\n return 1-K.mean(loss)\n
I'm using this as dice loss, with input shape of (n_samples, height, width, n_channels).
If you are using sm.losses.DiceLoss() from qubvel's segmentation models, #1 is the same as default mode, and #2 is the same as setting per_image=True.
By the way I think we need smooth term in the loss, because if we don't, loss will always be 1(dice coef will always be 0) if there are no masks in y_true.
But I might be wrong, so you might want to check segmentation models dice loss doc, blog post about segmentation losses and qubvel's codes.
\n\n10/31 Editted: added condition if axis is None
", "rawMarkdown": "```\ndef dice_loss(y_true, y_pred, axis=(0,1,2), smooth=1):\n intersection = K.sum(y_true * y_pred, axis=axis)\n total_y_true = K.sum(y_true, axis=axis)\n total_y_pred = K.sum(y_pred, axis=axis)\n loss = (2*intersection+smooth) / (total_y_true+total_y_pred+smooth)\n if axis is None:\n return 1-loss\n return 1-K.mean(loss)\n```\n\nI'm using this as dice loss, with input shape of `(n_samples, height, width, n_channels)`.\n\n1. If axis is (0,1,2) it calculates dice coef for each channel then averages them.\n2. If axis is (1,2) it calculates dice coef for each (image, channel) pair then averages them.\n3. If axis is None it flattens the batch then calculates dice coef once for that flattened vector.\n\nIf you are using `sm.losses.DiceLoss()` from qubvel's segmentation models, #1 is the same as default mode, and #2 is the same as setting `per_image=True`.\n\nBy the way I think we need `smooth` term in the loss, because if we don't, loss will always be 1(dice coef will always be 0) if there are no masks in `y_true`.\n\nBut I might be wrong, so you might want to check [segmentation models dice loss doc](https://segmentation-models.readthedocs.io/en/latest/api.html#segmentation_models.losses.DiceLoss), [blog post about segmentation losses](https://lars76.github.io/neural-networks/object-detection/losses-for-segmentation) and [qubvel's codes](https://github.com/qubvel/segmentation_models/tree/master/segmentation_models).\n\n*10/31 Editted: added condition if axis is None*", "votes": 6 }, { "id": 660323, "postDate": "2019-10-29T02:04:14.077Z", "content": "You need smooth because dice_loss without smooth has no gradient when it predicts a mask when there should be no mask. For example if there should not be a mask, then dice_loss = 0 and the gradient of dice_loss is also 0 without smooth. So your neural network cannot learn. That is why people use bce_dice_loss = 0.5*bce + 0.5*dice_loss. Or you can add smooth. (And smooth prevents division by zero too).
You can experiment with increasing smooth. I have not tried it, but perhaps if smooth is 100 or 1000 you will have faster convergence to empty masks (when not using additional bce).
Thanks, I use the default smooth=1 while I do see others using smooth=1e-7 or 1e-9.
", "rawMarkdown": "Thanks, I use the default smooth=1 while I do see others using smooth=1e-7 or 1e-9.", "votes": 2 }, { "id": 662033, "postDate": "2019-10-31T02:27:16.737Z", "content": "After I changed the dice_loss into this form, the metrics/loss become very unstable. Did anyone encounter the same issue? I was using per-image calculation for this loss.
", "rawMarkdown": "After I changed the dice_loss into this form, the metrics/loss become very unstable. Did anyone encounter the same issue? I was using per-image calculation for this loss." }, { "id": 662053, "postDate": "2019-10-31T03:26:36.110Z", "content": "@xiejialun What dice loss function were you using before?\nBy the way, if you use axis=(1,2) (which is per_image in qubvel's implementation), there will be more individual 'y_true's that have no mask when calculating dice coefficient, so it might be unstable. On the other hand, if you use axis=None, it is like 'flatten the batch, think of it as a single image and calculate dice coefficient once'. There would be low chance that the flattened batch would have no single mask in it, so it may be more stable.
@harangdev Thank you so much for the explanation! This helps a lot! I am using axis=(1,2) for the current training and flatten for previous training. The dice_coefficient does become unstable just like you said. Before I saw your explanation, I thought the model was overfitting. lol
", "rawMarkdown": "@harangdev Thank you so much for the explanation! This helps a lot! I am using axis=(1,2) for the current training and flatten for previous training. The dice_coefficient does become unstable just like you said. Before I saw your explanation, I thought the model was overfitting. lol", "votes": 1 } ] }, { "id": 669921, "postDate": "2019-11-10T17:24:01.253Z", "content": "one possible way to handle empty ground truth
\n\n```\ndef soft_dice_criterion(probability_mask, truth_label, truth_mask):\n batch_size,num_class,H,W = truth_mask.shape
\n\np = probability_mask.view(batch_size*num_class,-1)\nt = truth_mask.view(batch_size*num_class,-1)\n\n#non-empty\nintersection = (p * t).sum(-1)\nunion = (p).sum(-1) + (t).sum(-1)\npos_dice = 2*intersection/(union+1)\n\n#empty\n#neg_dice = 1 - (p).sum(-1)/(H*W)\nneg_dice = 1-p.max(-1)[0]\n\n\nt = truth_label.view(-1)\ndice = t*pos_dice + (1-t)*neg_dice\ndice = torch.clamp(dice,1e-7,1-1e-7)\nloss = -torch.log(dice)\nreturn loss\n```
", "rawMarkdown": "one possible way to handle empty ground truth\n\n```\ndef soft_dice_criterion(probability_mask, truth_label, truth_mask):\n batch_size,num_class,H,W = truth_mask.shape\n\n p = probability_mask.view(batch_size*num_class,-1)\n t = truth_mask.view(batch_size*num_class,-1)\n\n #non-empty\n intersection = (p * t).sum(-1)\n union = (p).sum(-1) + (t).sum(-1)\n pos_dice = 2*intersection/(union+1)\n\n #empty\n #neg_dice = 1 - (p).sum(-1)/(H*W)\n neg_dice = 1-p.max(-1)[0]\n\n\n t = truth_label.view(-1)\n dice = t*pos_dice + (1-t)*neg_dice\n dice = torch.clamp(dice,1e-7,1-1e-7)\n loss = -torch.log(dice)\n return loss\n```", "votes": 2 }, { "id": 663698, "postDate": "2019-11-02T14:57:12.457Z", "content": "I posted an example of saving OOF during K-Fold and then computing correct dice on the entire OOF predictions afterward here. During training I use single_dice_coef as a proxy for correct Kaggle dice. And then after training I use my form of mean_dice_coef to compute the correct Kaggle dice.
Here is another alternative for correct dice metric.
\n\nimport segmentation_models as sm\ncorrect_dice = sm.metrics.FScore(per_image=True, threshold=0.5)\nisnot is F1 score? are both same?
", "votes": 0, "replies": [] } ] }, { "id": 656321, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-10-24T06:37:07.293000", "content": "Your code is still not the correct dice metric used for LB scoring, because the inclusion of smooth
I used the following correct dice metric (for a single 2D layer of pred and ground truth). For mean dice, one can simply wrap it into an outside wrapper:\n`
\n\ndef dice(img1, img2):
\n\nimg1 = np.asarray(img1).astype(np.bool)\nimg2 = np.asarray(img2).astype(np.bool)\nif img1.sum() + img2.sum() == 0: return 1\nintersection = np.logical_and(img1, img2)\nreturn 2. * intersection.sum() / (img1.sum() + img2.sum())\n`
", "votes": 5, "replies": [ { "id": 656334, "author_name": "YoonSoo", "author_url": "", "post_date": "2019-10-24T06:46:36.830000", "content": "Oh so we could just return 1 if ytrue and ypred both have no masks. I guess we need the smooth thing when we want to use dice coef as loss function. I'll fix the code above. Thank you.
", "votes": 1, "replies": [] }, { "id": 658384, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-10-26T00:03:51.650000", "content": "I think if the smooth term is small enough, 1e-9 for example, it should not be an issue.
", "votes": 3, "replies": [] } ] }, { "id": 659066, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-10-27T02:44:56.293000", "content": "EDIT: The code in the original post is more efficient for 1 fold. Below is code for the entire 5-fold OOF.
\n\nHere is what I used for Steel Comp. Note that you cannot compile this into a model. You must make all your validation predictions and then apply this to each pair of true and predicted and take an average.
\n\ndef dice_coef2(y_true, y_pred):\n y_true_f = y_true.flatten()\n y_pred_f = y_pred.flatten()\n union = np.sum(y_true_f) + np.sum(y_pred_f)\n if union==0: return 1\n intersection = np.sum(y_true_f * y_pred_f)\n return 2. * intersection / union\nAnd if you have both true and predicted rle in pandas dataframe, do this:
sub['dice'] = sub.apply(lambda x: dice_coef2( rle2mask(x['true']), \n rle2mask(x['predicted']) ),axis=1)\nprint(sub['dice'].mean())\nThank you for sharing your code, Chris. I was using the version with 'flatten' which is the same as the code you provided, but then my validation score was much lower than the public leaderboard score. I switched to the above metric that I wrote, and validation score aligned with public leaderboard score better. (but still quite lower, I don't know why)
\n\nThe metric you provided calculates dice coefficient once for all of the flattened pixels. However, according to the metric described in the Evaluation section, we need to calculate dice coefficient for each (Image, Label) pair then average them. So two versions are calculating different things.
This is the code that shows the difference.
\n\n```\nimport numpy as np\nnp.random.seed(0)\ntrue = np.random.rand(10, 5, 5, 4)>0.5\npred = np.random.rand(10, 5, 5, 4)>0.5
\n\ndef single_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (height, width)\n intersection = np.sum(y_true * y_pred_bin)\n if (np.sum(y_true)==0) and (np.sum(y_pred_bin)==0):\n return 1\n return (2*intersection) / (np.sum(y_true) + np.sum(y_pred_bin))
\n\ndef mean_dice_coef(y_true, y_pred_bin):\n # shape of y_true and y_pred_bin: (n_samples, height, width, n_channels)\n batch_size = y_true.shape[0]\n channel_num = y_true.shape[-1]\n mean_dice_channel = 0.\n for i in range(batch_size):\n for j in range(channel_num):\n channel_dice = single_dice_coef(y_true[i, :, :, j], y_pred_bin[i, :, :, j])\n mean_dice_channel += channel_dice/(channel_num*batch_size)\n return mean_dice_channel
\n\ndef dice_coef2(y_true, y_pred):\n y_true_f = y_true.flatten()\n y_pred_f = y_pred.flatten()\n union = np.sum(y_true_f) + np.sum(y_pred_f)\n if union==0: return 1\n intersection = np.sum(y_true_f * y_pred_f)\n return 2. * intersection / union
\n\nprint(mean_dice_coef(true, pred))\nprint(dice_coef2(true, pred))
\n\n```
", "votes": 2, "replies": [] }, { "id": 659315, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-10-27T12:57:34.850000", "content": "Yes your code is more efficient than mine because it can be compiled directly into a model. My code works but you need to save all your rle into a dataframe and then apply my metric to each rle pair and then average. I like your code. I'm gonna use that. Thanks for posting.
model.compile(optimizer='adam', loss=bce_jaccard_loss, metrics=[mean_dice_coef])\nhistory = model.fit_generator(train_batches, validation_data=valid_batches)\nmodel.compile(optimizer='adam', loss=bce_jaccard_loss, metrics=[dice_coef2])\nhistory = model.fit_generator(train_batches, validation_data=valid_batches)\nfor k in range(len(EPOCHS)):\n model.train_generator(train_batches, epochs=1)\n val['predicted_rle'] = mask2rle_batches( model.predict_generator(valid_batches) )\n val['dice'] = val.apply(lambda x: dice_coef2( rle2mask(x['true_rle']), \n rle2mask(x['predicted_rle']) ),axis=1)\n print(val['dice'].mean())\nhello @harangdev i am having this issue that my val dice metrics is .74 but lb is .499 i am using the metrics that you have provided and i think that you had overcome this issue so can you help
", "votes": 0, "replies": [] }, { "id": 666267, "author_name": "YoonSoo", "author_url": "", "post_date": "2019-11-06T00:21:35.633000", "content": "@shekharrastogi Well, there can be many reasons. Your validation score seems too high. Maybe a bug in validation pipeline?
", "votes": 0, "replies": [] }, { "id": 666276, "author_name": "Tsai29", "author_url": "", "post_date": "2019-11-06T00:32:48.143000", "content": "@shekharrastogi Maybe you can check whether your training data mix with validating data. I had that problem few weeks ago, and the validation score also very high.
", "votes": 0, "replies": [] } ] }, { "id": 671761, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-13T06:03:29.053000", "content": "We can make our Dice metric more accurate (at estimating LB) by having our Dice metric perform post processing. I published code here. (See step 2 of \"How to save time\").
", "votes": 1, "replies": [ { "id": 671772, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-13T06:21:21.640000", "content": "yes, i also put kaggle lb metric in the metric meter in my training loop. it is useful. but it is important to note that:
\n\nyou will also have to monitor your back propagation loss at the same time.
", "votes": 2, "replies": [] } ] }, { "id": 660276, "author_name": "Yirun Zhang", "author_url": "", "post_date": "2019-10-29T00:30:26.837000", "content": "Can we use this calculation as dice loss? What will be the difference between using the corrected one and previous one as loss functions?
", "votes": 1, "replies": [ { "id": 660299, "author_name": "YoonSoo", "author_url": "", "post_date": "2019-10-29T01:08:32.137000", "content": "\ndef dice_loss(y_true, y_pred, axis=(0,1,2), smooth=1):\n intersection = K.sum(y_true * y_pred, axis=axis)\n total_y_true = K.sum(y_true, axis=axis)\n total_y_pred = K.sum(y_pred, axis=axis)\n loss = (2*intersection+smooth) / (total_y_true+total_y_pred+smooth)\n if axis is None:\n return 1-loss\n return 1-K.mean(loss)\n
I'm using this as dice loss, with input shape of (n_samples, height, width, n_channels).
If you are using sm.losses.DiceLoss() from qubvel's segmentation models, #1 is the same as default mode, and #2 is the same as setting per_image=True.
By the way I think we need smooth term in the loss, because if we don't, loss will always be 1(dice coef will always be 0) if there are no masks in y_true.
But I might be wrong, so you might want to check segmentation models dice loss doc, blog post about segmentation losses and qubvel's codes.
\n\n10/31 Editted: added condition if axis is None
", "votes": 6, "replies": [] }, { "id": 660323, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-10-29T02:04:14.077000", "content": "You need smooth because dice_loss without smooth has no gradient when it predicts a mask when there should be no mask. For example if there should not be a mask, then dice_loss = 0 and the gradient of dice_loss is also 0 without smooth. So your neural network cannot learn. That is why people use bce_dice_loss = 0.5*bce + 0.5*dice_loss. Or you can add smooth. (And smooth prevents division by zero too).
You can experiment with increasing smooth. I have not tried it, but perhaps if smooth is 100 or 1000 you will have faster convergence to empty masks (when not using additional bce).
Thanks, I use the default smooth=1 while I do see others using smooth=1e-7 or 1e-9.
", "votes": 2, "replies": [] }, { "id": 662033, "author_name": "Tsai29", "author_url": "", "post_date": "2019-10-31T02:27:16.737000", "content": "After I changed the dice_loss into this form, the metrics/loss become very unstable. Did anyone encounter the same issue? I was using per-image calculation for this loss.
", "votes": 0, "replies": [] }, { "id": 662053, "author_name": "YoonSoo", "author_url": "", "post_date": "2019-10-31T03:26:36.110000", "content": "@xiejialun What dice loss function were you using before?\nBy the way, if you use axis=(1,2) (which is per_image in qubvel's implementation), there will be more individual 'y_true's that have no mask when calculating dice coefficient, so it might be unstable. On the other hand, if you use axis=None, it is like 'flatten the batch, think of it as a single image and calculate dice coefficient once'. There would be low chance that the flattened batch would have no single mask in it, so it may be more stable.
@harangdev Thank you so much for the explanation! This helps a lot! I am using axis=(1,2) for the current training and flatten for previous training. The dice_coefficient does become unstable just like you said. Before I saw your explanation, I thought the model was overfitting. lol
", "votes": 1, "replies": [] } ] }, { "id": 669921, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-10T17:24:01.253000", "content": "one possible way to handle empty ground truth
\n\n```\ndef soft_dice_criterion(probability_mask, truth_label, truth_mask):\n batch_size,num_class,H,W = truth_mask.shape
\n\np = probability_mask.view(batch_size*num_class,-1)\nt = truth_mask.view(batch_size*num_class,-1)\n\n#non-empty\nintersection = (p * t).sum(-1)\nunion = (p).sum(-1) + (t).sum(-1)\npos_dice = 2*intersection/(union+1)\n\n#empty\n#neg_dice = 1 - (p).sum(-1)/(H*W)\nneg_dice = 1-p.max(-1)[0]\n\n\nt = truth_label.view(-1)\ndice = t*pos_dice + (1-t)*neg_dice\ndice = torch.clamp(dice,1e-7,1-1e-7)\nloss = -torch.log(dice)\nreturn loss\n```
", "votes": 2, "replies": [] }, { "id": 663698, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-02T14:57:12.457000", "content": "I posted an example of saving OOF during K-Fold and then computing correct dice on the entire OOF predictions afterward here. During training I use single_dice_coef as a proxy for correct Kaggle dice. And then after training I use my form of mean_dice_coef to compute the correct Kaggle dice.