{
  "id": 265973,
  "title": "PyTorch Implementation of Class-Balanced Loss",
  "url": "/competitions/seti-breakthrough-listen/discussion/265973",
  "author_name": "Bilzard",
  "post_date": "2021-08-17T14:26:21.226000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<h2>About</h2>\n<p>To handle class imbalance, I implemented Class-Balanced Loss[1].</p>\n<h2>Implementation</h2>\n<pre><code>class ClassBalancedLoss(nn.Module):\n    def __init__(self, n_pos: int, n_neg: int, beta: float=0.9999, epsilon=1e-15):\n        super().__init__()\n        assert n_pos &gt; 0 and n_neg &gt; 0, 'sample count should be positive integer'\n\n        pos_weight = (1.0 - beta ** n_neg) / (1.0 - beta ** n_pos + epsilon)\n        alpha = 2.0 / (1.0 + pos_weight)\n\n        pos_weight = Tensor([pos_weight])\n        self.alpha = alpha\n        self.bce_with_logit = torch.nn.BCEWithLogitsLoss(pos_weight)\n\n    def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:\n        return self.alpha * self.bce_with_logit(input, target)\n</code></pre>\n<p>Note that in the original paper, the author assume the task of multi-class classification but we have to apply this loss to binary-classification problem.</p>\n<p>Since pytorch accepts BCEWithLogitsLoss with weights, we can calculate proper <code>pos_weight</code>, we got the loss we wanted.</p>\n<p>Note that since the pytorch's BCEWithLogitsLoss implementation, the loss is calculated by this equation:</p>\n<pre><code>loss = pos_weight * y * log(sigmoid(x)) + (1 - y) * log(sigmoid(1 - x))\n</code></pre>\n<p>we can to calculate <code>pos_weight = (1 - beta ** n_neg) / (1 - beta ** n_pos)</code>.</p>\n<h2>Usage</h2>\n<pre><code>criterion = ClassBalancedLoss(n_pos=4800, n_neg=43200, beta=0.9999)\n</code></pre>\n<p>If you find any mistake, please comment :)</p>\n<h2>Reference</h2>\n<p>[1]: Class-Balanced Loss Based on Effective Number of Samples, <a href=\"https://arxiv.org/abs/1901.05555v1\" target=\"_blank\">https://arxiv.org/abs/1901.05555v1</a></p>",
  "messages": [
    {
      "id": 1477549,
      "postDate": "2021-08-17T14:26:21.227Z",
      "content": "<h2>About</h2>\n<p>To handle class imbalance, I implemented Class-Balanced Loss[1].</p>\n<h2>Implementation</h2>\n<pre><code>class ClassBalancedLoss(nn.Module):\n    def __init__(self, n_pos: int, n_neg: int, beta: float=0.9999, epsilon=1e-15):\n        super().__init__()\n        assert n_pos &gt; 0 and n_neg &gt; 0, 'sample count should be positive integer'\n\n        pos_weight = (1.0 - beta ** n_neg) / (1.0 - beta ** n_pos + epsilon)\n        alpha = 2.0 / (1.0 + pos_weight)\n\n        pos_weight = Tensor([pos_weight])\n        self.alpha = alpha\n        self.bce_with_logit = torch.nn.BCEWithLogitsLoss(pos_weight)\n\n    def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:\n        return self.alpha * self.bce_with_logit(input, target)\n</code></pre>\n<p>Note that in the original paper, the author assume the task of multi-class classification but we have to apply this loss to binary-classification problem.</p>\n<p>Since pytorch accepts BCEWithLogitsLoss with weights, we can calculate proper <code>pos_weight</code>, we got the loss we wanted.</p>\n<p>Note that since the pytorch's BCEWithLogitsLoss implementation, the loss is calculated by this equation:</p>\n<pre><code>loss = pos_weight * y * log(sigmoid(x)) + (1 - y) * log(sigmoid(1 - x))\n</code></pre>\n<p>we can to calculate <code>pos_weight = (1 - beta ** n_neg) / (1 - beta ** n_pos)</code>.</p>\n<h2>Usage</h2>\n<pre><code>criterion = ClassBalancedLoss(n_pos=4800, n_neg=43200, beta=0.9999)\n</code></pre>\n<p>If you find any mistake, please comment :)</p>\n<h2>Reference</h2>\n<p>[1]: Class-Balanced Loss Based on Effective Number of Samples, <a href=\"https://arxiv.org/abs/1901.05555v1\" target=\"_blank\">https://arxiv.org/abs/1901.05555v1</a></p>",
      "rawMarkdown": "## About\n\nTo handle class imbalance, I implemented Class-Balanced Loss[1].\n\n## Implementation\n\n```python\nclass ClassBalancedLoss(nn.Module):\n    def __init__(self, n_pos: int, n_neg: int, beta: float=0.9999, epsilon=1e-15):\n        super().__init__()\n        assert n_pos > 0 and n_neg > 0, 'sample count should be positive integer'\n\n        pos_weight = (1.0 - beta ** n_neg) / (1.0 - beta ** n_pos + epsilon)\n        alpha = 2.0 / (1.0 + pos_weight)\n\n        pos_weight = Tensor([pos_weight])\n        self.alpha = alpha\n        self.bce_with_logit = torch.nn.BCEWithLogitsLoss(pos_weight)\n\n    def forward(self, input: Tensor, target: Tensor) -> Tensor:\n        return self.alpha * self.bce_with_logit(input, target)\n```\n\nNote that in the original paper, the author assume the task of multi-class classification but we have to apply this loss to binary-classification problem.\n\nSince pytorch accepts BCEWithLogitsLoss with weights, we can calculate proper `pos_weight`, we got the loss we wanted.\n\nNote that since the pytorch's BCEWithLogitsLoss implementation, the loss is calculated by this equation:\n```\nloss = pos_weight * y * log(sigmoid(x)) + (1 - y) * log(sigmoid(1 - x))\n```\nwe can to calculate `pos_weight = (1 - beta ** n_neg) / (1 - beta ** n_pos)`.\n\n## Usage\n\n```\ncriterion = ClassBalancedLoss(n_pos=4800, n_neg=43200, beta=0.9999)\n```\n\nIf you find any mistake, please comment :)\n\n## Reference\n\n[1]: Class-Balanced Loss Based on Effective Number of Samples, https://arxiv.org/abs/1901.05555v1",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1477549": "## About\n\nTo handle class imbalance, I implemented Class-Balanced Loss[1].\n\n## Implementation\n\n```python\nclass ClassBalancedLoss(nn.Module):\n    def __init__(self, n_pos: int, n_neg: int, beta: float=0.9999, epsilon=1e-15):\n        super().__init__()\n        assert n_pos > 0 and n_neg > 0, 'sample count should be positive integer'\n\n        pos_weight = (1.0 - beta ** n_neg) / (1.0 - beta ** n_pos + epsilon)\n        alpha = 2.0 / (1.0 + pos_weight)\n\n        pos_weight = Tensor([pos_weight])\n        self.alpha = alpha\n        self.bce_with_logit = torch.nn.BCEWithLogitsLoss(pos_weight)\n\n    def forward(self, input: Tensor, target: Tensor) -> Tensor:\n        return self.alpha * self.bce_with_logit(input, target)\n```\n\nNote that in the original paper, the author assume the task of multi-class classification but we have to apply this loss to binary-classification problem.\n\nSince pytorch accepts BCEWithLogitsLoss with weights, we can calculate proper `pos_weight`, we got the loss we wanted.\n\nNote that since the pytorch's BCEWithLogitsLoss implementation, the loss is calculated by this equation:\n```\nloss = pos_weight * y * log(sigmoid(x)) + (1 - y) * log(sigmoid(1 - x))\n```\nwe can to calculate `pos_weight = (1 - beta ** n_neg) / (1 - beta ** n_pos)`.\n\n## Usage\n\n```\ncriterion = ClassBalancedLoss(n_pos=4800, n_neg=43200, beta=0.9999)\n```\n\nIf you find any mistake, please comment :)\n\n## Reference\n\n[1]: Class-Balanced Loss Based on Effective Number of Samples, https://arxiv.org/abs/1901.05555v1"
  }
}