{ "id": 128637, "title": "[pytorch] ohem loss implementation", "url": "/competitions/bengaliai-cv19/discussion/128637", "author_name": "", "post_date": "2020-02-02T01:25:58.403820500Z", "votes": 72, "comment_count": 26, "views": 0, "content": "

```python\ndef ohem_loss( n, cls_pred, cls_target ):

\n\n
ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\nsorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], n)\nif keep_num < sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

```

\n\n

update：

\n\n

```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)

\n\n
sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\nif keep_num < sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

```

", "messages": [ { "id": "734762", "postDate": "02/02/2020 01:25:58", "content": "

```python\ndef ohem_loss( n, cls_pred, cls_target ):

\n\n
ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\nsorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], n)\nif keep_num &lt; sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

```

\n\n

update：

\n\n

```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)

\n\n
sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\nif keep_num < sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

```

", "rawMarkdown": "```python\ndef ohem_loss( n, cls_pred, cls_target ):\n \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], n)\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n```\n\nupdate：\n\n```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n```", "votes": null }, { "id": "734764", "postDate": "02/02/2020 01:28:30", "content": "

update [mixup/cutmix with ohem loss] :

\n\n

```python\ndef ohem_loss( rate, cls_pred, cls_target ):

\n\n
batch_size = cls_pred.size(0) \nohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\nsorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\nif keep_num &lt; sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

def rand_bbox(size, lam):\n W = size[2]\n H = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(W * cut_rat)\n cut_h = np.int(H * cut_rat)

\n\n
# uniform\ncx = np.random.randint(W)\ncy = np.random.randint(H)\n\nbbx1 = np.clip(cx - cut_w // 2, 0, W)\nbby1 = np.clip(cy - cut_h // 2, 0, H)\nbbx2 = np.clip(cx + cut_w // 2, 0, W)\nbby2 = np.clip(cy + cut_h // 2, 0, H)\n\nreturn bbx1, bby1, bbx2, bby2\n
\n\n

def cutmix(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]

\n\n
lam = np.random.beta(alpha, alpha)\nbbx1, bby1, bbx2, bby2 = rand_bbox(data.size(), lam)\ndata[:, :, bbx1:bbx2, bby1:bby2] = data[indices, :, bbx1:bbx2, bby1:bby2]\n# adjust lambda to exactly match pixel ratio\nlam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (data.size()[-1] * data.size()[-2]))\n\ntargets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\nreturn data, targets\n
\n\n

loss

\n\n

def cutmix_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]

\n\n

def mixup(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]

\n\n
lam = np.random.beta(alpha, alpha)\ndata = data * lam + shuffled_data * (1 - lam)\ntargets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\n\nreturn data, targets\n
\n\n

def mixup_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]

\n\n

```

", "rawMarkdown": "update [mixup/cutmix with ohem loss] :\n\n```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n \n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n\n\ndef rand_bbox(size, lam):\n W = size[2]\n H = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(W * cut_rat)\n cut_h = np.int(H * cut_rat)\n\n # uniform\n cx = np.random.randint(W)\n cy = np.random.randint(H)\n\n bbx1 = np.clip(cx - cut_w // 2, 0, W)\n bby1 = np.clip(cy - cut_h // 2, 0, H)\n bbx2 = np.clip(cx + cut_w // 2, 0, W)\n bby2 = np.clip(cy + cut_h // 2, 0, H)\n\n return bbx1, bby1, bbx2, bby2\ndef cutmix(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]\n \n lam = np.random.beta(alpha, alpha)\n bbx1, bby1, bbx2, bby2 = rand_bbox(data.size(), lam)\n data[:, :, bbx1:bbx2, bby1:bby2] = data[indices, :, bbx1:bbx2, bby1:bby2]\n # adjust lambda to exactly match pixel ratio\n lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (data.size()[-1] * data.size()[-2]))\n\n targets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\n return data, targets\n# loss \ndef cutmix_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]\n\n\ndef mixup(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]\n \n lam = np.random.beta(alpha, alpha)\n data = data * lam + shuffled_data * (1 - lam)\n targets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\n\n return data, targets\n\n\ndef mixup_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]\n\n```", "votes": null }, { "id": "734778", "postDate": "02/02/2020 01:52:20", "content": "

Top K gradient for Cross Entropy (OHEM)：https://discuss.pytorch.org/t/top-k-gradient-for-cross-entropy-ohem/10020

", "rawMarkdown": "Top K gradient for Cross Entropy (OHEM)：https://discuss.pytorch.org/t/top-k-gradient-for-cross-entropy-ohem/10020", "votes": null }, { "id": "734786", "postDate": "02/02/2020 02:18:59", "content": "

Thanks! One question what is n ? a threshold ?

", "rawMarkdown": "Thanks! One question what is `n` ? a threshold ?", "votes": null }, { "id": "734788", "postDate": "02/02/2020 02:22:51", "content": "", "rawMarkdown": "", "votes": null }, { "id": "734789", "postDate": "02/02/2020 02:23:35", "content": "

yes，it's a threshold.

", "rawMarkdown": "yes，it's a threshold.", "votes": null }, { "id": "734790", "postDate": "02/02/2020 02:24:31", "content": "

thanks!

", "rawMarkdown": "thanks!", "votes": null }, { "id": "734812", "postDate": "02/02/2020 03:10:58", "content": "

update：

\n\n

```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)

\n\n
sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\nif keep_num &lt; sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

```

", "rawMarkdown": "update：\n\n```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n```", "votes": null }, { "id": "734851", "postDate": "02/02/2020 05:07:44", "content": "

loss, _ = loss.topk(k=int(ohem_percent * batchsize))

", "rawMarkdown": "`loss, _ = loss.topk(k=int(ohem_percent * batchsize))`", "votes": null }, { "id": "734854", "postDate": "02/02/2020 05:11:10", "content": "

666

", "rawMarkdown": "666", "votes": null }, { "id": "734935", "postDate": "02/02/2020 08:57:12", "content": "

Hi, thank you for your sharing!\nI want to ask should the ohem loss combing with normal cross entropy loss? In your below mixup_criterion loss function, you only give penalty on hard examples, should we combine the new mixup_criterion with original mixup_criterion(using normal CrossEntropy Loss on all examples)?

", "rawMarkdown": "Hi, thank you for your sharing!\nI want to ask should the ohem loss combing with normal cross entropy loss? In your below mixup_criterion loss function, you only give penalty on hard examples, should we combine the new mixup_criterion with original mixup_criterion(using normal CrossEntropy Loss on all examples)?", "votes": null }, { "id": "734948", "postDate": "02/02/2020 09:24:25", "content": "

Hi, what do you think about the idea of decaying rate with the number of epochs? 0.7 sounds good early on, but later on I feel smaller values could better select for the rarer classes.

", "rawMarkdown": "Hi, what do you think about the idea of decaying rate with the number of epochs? 0.7 sounds good early on, but later on I feel smaller values could better select for the rarer classes.", "votes": null }, { "id": "734988", "postDate": "02/02/2020 10:54:14", "content": "

Thanks @machinelp in mixup_criterion you forgot to take rate as parameter.

", "rawMarkdown": "Thanks @machinelp in mixup_criterion you forgot to take rate as parameter.", "votes": null }, { "id": "735015", "postDate": "02/02/2020 11:20:52", "content": "

Thanks for your reminding

", "rawMarkdown": "Thanks for your reminding", "votes": null }, { "id": "735727", "postDate": "02/03/2020 11:46:39", "content": "", "rawMarkdown": "", "votes": null }, { "id": "736023", "postDate": "02/03/2020 18:17:49", "content": "

Thanks @machinelp that's great

", "rawMarkdown": "Thanks @machinelp that's great", "votes": null }, { "id": "736226", "postDate": "02/04/2020 01:11:20", "content": "

@jamesphoward This idea looks very interesting to me. Have you implemented it? Or have you tried with different rate values?

", "rawMarkdown": "jamesphoward This idea looks very interesting to me. Have you implemented it? Or have you tried with different rate values?", "votes": null }, { "id": "736243", "postDate": "02/04/2020 01:36:17", "content": "

readable text. thank you, @machinelp

", "rawMarkdown": "readable text. thank you, @machinelp", "votes": null }, { "id": "736434", "postDate": "02/04/2020 07:15:15", "content": "

666

", "rawMarkdown": "666", "votes": null }, { "id": "736457", "postDate": "02/04/2020 08:02:53", "content": "

I haven't yet, though I'm planning on looking into it.

", "rawMarkdown": "I haven't yet, though I'm planning on looking into it.", "votes": null }, { "id": "736487", "postDate": "02/04/2020 08:54:23", "content": "

Thanks mate !

", "rawMarkdown": "Thanks mate !", "votes": null }, { "id": "737456", "postDate": "02/05/2020 11:17:02", "content": "

Thanks bro!

", "rawMarkdown": "Thanks bro!", "votes": null }, { "id": "737725", "postDate": "02/05/2020 17:36:16", "content": "

I tried it out: ratio = pow(.5,epoch/50) not sure if it helped but it didn't make my results worse

", "rawMarkdown": "I tried it out: ```ratio = pow(.5,epoch/50)``` not sure if it helped but it didn't make my results worse", "votes": null }, { "id": "737729", "postDate": "02/05/2020 17:40:16", "content": "

Dota eh? A man of culture.

", "rawMarkdown": "Dota eh? A man of culture.", "votes": null }, { "id": "754435", "postDate": "02/23/2020 14:51:39", "content": "

great job!

", "rawMarkdown": "great job!", "votes": null }, { "id": "768604", "postDate": "03/11/2020 02:31:03", "content": "

l

", "rawMarkdown": "l", "votes": null }, { "id": "768837", "postDate": "03/11/2020 08:35:25", "content": "

I have written my onw ohem-loss in keras:\n```python

\n\n
\n

def ohem_loss(ytrue, ypred):\n result = K.categorical_crossentropy(ytrue, ypred, axis=-1)\n alpha = K.variable(0.1, dtype=tf.float32)\n index = K.greater(result, alpha)\n cast = tf.cast(index, tf.float32)\n return result*cast\n ```

\n
\n\n

Is that proper implementation in keras?

", "rawMarkdown": "I have written my onw ohem-loss in keras:\n```python\n>def ohem_loss(ytrue, ypred):\n result = K.categorical_crossentropy(ytrue, ypred, axis=-1)\n alpha = K.variable(0.1, dtype=tf.float32)\n index = K.greater(result, alpha)\n cast = tf.cast(index, tf.float32)\n return result*cast\n```\n\nIs that proper implementation in keras?", "votes": null } ], "comments": [ { "id": 734764, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 01:28:30", "content": "

update [mixup/cutmix with ohem loss] :

\n\n

```python\ndef ohem_loss( rate, cls_pred, cls_target ):

\n\n
batch_size = cls_pred.size(0) \nohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\nsorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\nif keep_num &lt; sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

def rand_bbox(size, lam):\n W = size[2]\n H = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(W * cut_rat)\n cut_h = np.int(H * cut_rat)

\n\n
# uniform\ncx = np.random.randint(W)\ncy = np.random.randint(H)\n\nbbx1 = np.clip(cx - cut_w // 2, 0, W)\nbby1 = np.clip(cy - cut_h // 2, 0, H)\nbbx2 = np.clip(cx + cut_w // 2, 0, W)\nbby2 = np.clip(cy + cut_h // 2, 0, H)\n\nreturn bbx1, bby1, bbx2, bby2\n
\n\n

def cutmix(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]

\n\n
lam = np.random.beta(alpha, alpha)\nbbx1, bby1, bbx2, bby2 = rand_bbox(data.size(), lam)\ndata[:, :, bbx1:bbx2, bby1:bby2] = data[indices, :, bbx1:bbx2, bby1:bby2]\n# adjust lambda to exactly match pixel ratio\nlam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (data.size()[-1] * data.size()[-2]))\n\ntargets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\nreturn data, targets\n
\n\n

loss

\n\n

def cutmix_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]

\n\n

def mixup(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]

\n\n
lam = np.random.beta(alpha, alpha)\ndata = data * lam + shuffled_data * (1 - lam)\ntargets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\n\nreturn data, targets\n
\n\n

def mixup_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]

\n\n

```

", "votes": null, "replies": [ { "id": 734988, "author_name": "carriesmi", "author_url": "", "post_date": "02/02/2020 10:54:14", "content": "

Thanks @machinelp in mixup_criterion you forgot to take rate as parameter.

", "votes": null, "replies": [] }, { "id": 735015, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 11:20:52", "content": "

Thanks for your reminding

", "votes": null, "replies": [] } ] }, { "id": 734778, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 01:52:20", "content": "

Top K gradient for Cross Entropy (OHEM)：https://discuss.pytorch.org/t/top-k-gradient-for-cross-entropy-ohem/10020

", "votes": null, "replies": [] }, { "id": 734786, "author_name": "drhabib", "author_url": "", "post_date": "02/02/2020 02:18:59", "content": "

Thanks! One question what is n ? a threshold ?

", "votes": null, "replies": [ { "id": 734788, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 02:22:51", "content": "", "votes": null, "replies": [] }, { "id": 734789, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 02:23:35", "content": "

yes，it's a threshold.

", "votes": null, "replies": [] }, { "id": 734790, "author_name": "drhabib", "author_url": "", "post_date": "02/02/2020 02:24:31", "content": "

thanks!

", "votes": null, "replies": [] } ] }, { "id": 734812, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 03:10:58", "content": "

update：

\n\n

```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)

\n\n
sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\nkeep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\nif keep_num &lt; sorted_ohem_loss.size()[0]:\n    keep_idx_cuda = idx[:keep_num]\n    ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\ncls_loss = ohem_cls_loss.sum() / keep_num\nreturn cls_loss\n
\n\n

```

", "votes": null, "replies": [] }, { "id": 734851, "author_name": "changewow", "author_url": "", "post_date": "02/02/2020 05:07:44", "content": "

loss, _ = loss.topk(k=int(ohem_percent * batchsize))

", "votes": null, "replies": [ { "id": 734854, "author_name": "machinelp", "author_url": "", "post_date": "02/02/2020 05:11:10", "content": "

666

", "votes": null, "replies": [] }, { "id": 736434, "author_name": "syoya1997", "author_url": "", "post_date": "02/04/2020 07:15:15", "content": "

666

", "votes": null, "replies": [] } ] }, { "id": 734935, "author_name": "liang23333", "author_url": "", "post_date": "02/02/2020 08:57:12", "content": "

Hi, thank you for your sharing!\nI want to ask should the ohem loss combing with normal cross entropy loss? In your below mixup_criterion loss function, you only give penalty on hard examples, should we combine the new mixup_criterion with original mixup_criterion(using normal CrossEntropy Loss on all examples)?

", "votes": null, "replies": [] }, { "id": 734948, "author_name": "jamesphoward", "author_url": "", "post_date": "02/02/2020 09:24:25", "content": "

Hi, what do you think about the idea of decaying rate with the number of epochs? 0.7 sounds good early on, but later on I feel smaller values could better select for the rarer classes.

", "votes": null, "replies": [ { "id": 736226, "author_name": "mahtabshaan", "author_url": "", "post_date": "02/04/2020 01:11:20", "content": "

@jamesphoward This idea looks very interesting to me. Have you implemented it? Or have you tried with different rate values?

", "votes": null, "replies": [] }, { "id": 736457, "author_name": "jamesphoward", "author_url": "", "post_date": "02/04/2020 08:02:53", "content": "

I haven't yet, though I'm planning on looking into it.

", "votes": null, "replies": [] }, { "id": 737725, "author_name": "greatgamedota", "author_url": "", "post_date": "02/05/2020 17:36:16", "content": "

I tried it out: ratio = pow(.5,epoch/50) not sure if it helped but it didn't make my results worse

", "votes": null, "replies": [] }, { "id": 737729, "author_name": "jamesphoward", "author_url": "", "post_date": "02/05/2020 17:40:16", "content": "

Dota eh? A man of culture.

", "votes": null, "replies": [] } ] }, { "id": 735727, "author_name": "vladislavkr", "author_url": "", "post_date": "02/03/2020 11:46:39", "content": "", "votes": null, "replies": [] }, { "id": 736023, "author_name": "tahirmuslimscientist", "author_url": "", "post_date": "02/03/2020 18:17:49", "content": "

Thanks @machinelp that's great

", "votes": null, "replies": [] }, { "id": 736243, "author_name": "topchaebol", "author_url": "", "post_date": "02/04/2020 01:36:17", "content": "

readable text. thank you, @machinelp

", "votes": null, "replies": [] }, { "id": 736487, "author_name": "sivasakthivelan", "author_url": "", "post_date": "02/04/2020 08:54:23", "content": "

Thanks mate !

", "votes": null, "replies": [] }, { "id": 737456, "author_name": "rohitagarwal", "author_url": "", "post_date": "02/05/2020 11:17:02", "content": "

Thanks bro!

", "votes": null, "replies": [] }, { "id": 754435, "author_name": "luckysouthchou", "author_url": "", "post_date": "02/23/2020 14:51:39", "content": "

great job!

", "votes": null, "replies": [] }, { "id": 768604, "author_name": "dngh3164", "author_url": "", "post_date": "03/11/2020 02:31:03", "content": "

l

", "votes": null, "replies": [] }, { "id": 768837, "author_name": "phamquocbao", "author_url": "", "post_date": "03/11/2020 08:35:25", "content": "

I have written my onw ohem-loss in keras:\n```python

\n\n
\n

def ohem_loss(ytrue, ypred):\n result = K.categorical_crossentropy(ytrue, ypred, axis=-1)\n alpha = K.variable(0.1, dtype=tf.float32)\n index = K.greater(result, alpha)\n cast = tf.cast(index, tf.float32)\n return result*cast\n ```

\n
\n\n

Is that proper implementation in keras?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "734762": "```python\ndef ohem_loss( n, cls_pred, cls_target ):\n \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], n)\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n```\n\nupdate：\n\n```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n```", "734764": "update [mixup/cutmix with ohem loss] :\n\n```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n \n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n\n\ndef rand_bbox(size, lam):\n W = size[2]\n H = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(W * cut_rat)\n cut_h = np.int(H * cut_rat)\n\n # uniform\n cx = np.random.randint(W)\n cy = np.random.randint(H)\n\n bbx1 = np.clip(cx - cut_w // 2, 0, W)\n bby1 = np.clip(cy - cut_h // 2, 0, H)\n bbx2 = np.clip(cx + cut_w // 2, 0, W)\n bby2 = np.clip(cy + cut_h // 2, 0, H)\n\n return bbx1, bby1, bbx2, bby2\ndef cutmix(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]\n \n lam = np.random.beta(alpha, alpha)\n bbx1, bby1, bbx2, bby2 = rand_bbox(data.size(), lam)\n data[:, :, bbx1:bbx2, bby1:bby2] = data[indices, :, bbx1:bbx2, bby1:bby2]\n # adjust lambda to exactly match pixel ratio\n lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (data.size()[-1] * data.size()[-2]))\n\n targets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\n return data, targets\n# loss \ndef cutmix_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]\n\n\ndef mixup(data, targets1, targets2, targets3, alpha):\n indices = torch.randperm(data.size(0))\n shuffled_data = data[indices]\n shuffled_targets1 = targets1[indices]\n shuffled_targets2 = targets2[indices]\n shuffled_targets3 = targets3[indices]\n \n lam = np.random.beta(alpha, alpha)\n data = data * lam + shuffled_data * (1 - lam)\n targets = [targets1, shuffled_targets1, targets2, shuffled_targets2, targets3, shuffled_targets3, lam]\n\n return data, targets\n\n\ndef mixup_criterion(preds1,preds2,preds3, targets, rate=0.7):\n targets1, targets2,targets3, targets4,targets5, targets6, lam = targets[0], targets[1], targets[2], targets[3], targets[4], targets[5], targets[6]\n # criterion = nn.CrossEntropyLoss(reduction='mean')\n criterion = ohem_loss\n return [ lam * criterion(rate, preds1, targets1) + (1 - lam) * criterion(rate, preds1, targets2), lam * criterion(rate, preds2, targets3) + (1 - lam) * criterion(rate, preds2, targets4), lam * criterion(rate, preds3, targets5) + (1 - lam) * criterion(rate, preds3, targets6) ]\n\n```", "734778": "Top K gradient for Cross Entropy (OHEM)：https://discuss.pytorch.org/t/top-k-gradient-for-cross-entropy-ohem/10020", "734786": "Thanks! One question what is `n` ? a threshold ?", "734788": "", "734789": "yes，it's a threshold.", "734790": "thanks!", "734812": "update：\n\n```python\ndef ohem_loss( rate, cls_pred, cls_target ):\n batch_size = cls_pred.size(0) \n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)\n\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )\n if keep_num < sorted_ohem_loss.size()[0]:\n keep_idx_cuda = idx[:keep_num]\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]\n cls_loss = ohem_cls_loss.sum() / keep_num\n return cls_loss\n```", "734851": "`loss, _ = loss.topk(k=int(ohem_percent * batchsize))`", "734854": "666", "734935": "Hi, thank you for your sharing!\nI want to ask should the ohem loss combing with normal cross entropy loss? In your below mixup_criterion loss function, you only give penalty on hard examples, should we combine the new mixup_criterion with original mixup_criterion(using normal CrossEntropy Loss on all examples)?", "734948": "Hi, what do you think about the idea of decaying rate with the number of epochs? 0.7 sounds good early on, but later on I feel smaller values could better select for the rarer classes.", "734988": "Thanks @machinelp in mixup_criterion you forgot to take rate as parameter.", "735015": "Thanks for your reminding", "735727": "", "736023": "Thanks @machinelp that's great", "736226": "jamesphoward This idea looks very interesting to me. Have you implemented it? Or have you tried with different rate values?", "736243": "readable text. thank you, @machinelp", "736434": "666", "736457": "I haven't yet, though I'm planning on looking into it.", "736487": "Thanks mate !", "737456": "Thanks bro!", "737725": "I tried it out: ```ratio = pow(.5,epoch/50)``` not sure if it helped but it didn't make my results worse", "737729": "Dota eh? A man of culture.", "754435": "great job!", "768604": "l", "768837": "I have written my onw ohem-loss in keras:\n```python\n>def ohem_loss(ytrue, ypred):\n result = K.categorical_crossentropy(ytrue, ypred, axis=-1)\n alpha = K.variable(0.1, dtype=tf.float32)\n index = K.greater(result, alpha)\n cast = tf.cast(index, tf.float32)\n return result*cast\n```\n\nIs that proper implementation in keras?" }, "source": "meta" }