{
  "id": 481764,
  "title": "MixUp augmentation in Albumentations",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/481764",
  "author_name": "Vladimir Iglovikov",
  "post_date": "2024-03-05T01:41:01.181000",
  "votes": 40,
  "comment_count": 21,
  "views": 0,
  "content": "<p>In another <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477825\" target=\"_blank\">thread</a>, <a href=\"https://www.kaggle.com/chemdatafarmer\" target=\"_blank\">@chemdatafarmer</a> asked for MixUp augmentation.</p>\n<p>It was long on our TODO list, but it was a final push.</p>\n<p>Today we released version <a href=\"https://github.com/albumentations-team/albumentations/releases/tag/1.4.1\" target=\"_blank\">1.4.1</a>, with MixUp. The transform could be applied to <code>images</code>, <code>masks</code>, and <code>global_labels</code> (we needed some name to call an overall label for the image, and we did not like <code>class_label</code>, hence new notation)</p>\n<ul>\n<li><a href=\"https://albumentations.ai/docs/api_reference/augmentations/mixing/transforms/#albumentations.augmentations.mixing.transforms.MixUp\" target=\"_blank\">Documentation</a></li>\n<li><a href=\"https://albumentations.ai/docs/examples/example_mixup/\" target=\"_blank\">Example of use in documentation</a></li>\n</ul>\n<p>The most important - <a href=\"https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations\" target=\"_blank\"><strong>Kaggle Kernel with an example of how to apply in this competition</strong>.</a></p>\n<p><strong>P.S.</strong> If you know transforms that are helpful, but not implemented in Albumentations - feel free to ping me in DM, <a href=\"https://github.com/albumentations-team/albumentations/issues\" target=\"_blank\">create issue in the repository</a>, or write here in comments.</p>\n<p><strong>P.P.S.</strong> The effect of the transform is not clear if you are not familiar with it on the spectrogram data =&gt; here is an example of natural images.  </p>\n<p><strong>P.P.P.S.</strong> I do not know if MixUp will be useful in this competition - but good luck anyway!</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F286455%2Febfab33fa5fe5d0d8af11841d7a7feff%2FScreenshot%202024-03-04%20at%2014.52.15.png?generation=1709602735723662&amp;alt=media\"></p>\n<p><strong>UPD</strong>: 2024-03-18. In version <a href=\"https://github.com/albumentations-team/albumentations/releases/tag/1.4.2\" target=\"_blank\">1.4.2</a> we added an option to return mixing coefficient that was used by MixUp (randomly changes every time pipeline is applied). <a href=\"https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations\" target=\"_blank\">Kaggle Kernel</a> is updated to show how to return it. </p>",
  "messages": [
    {
      "id": 2681902,
      "postDate": "2024-03-05T01:41:01.180Z",
      "content": "<p>In another <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477825\" target=\"_blank\">thread</a>, <a href=\"https://www.kaggle.com/chemdatafarmer\" target=\"_blank\">@chemdatafarmer</a> asked for MixUp augmentation.</p>\n<p>It was long on our TODO list, but it was a final push.</p>\n<p>Today we released version <a href=\"https://github.com/albumentations-team/albumentations/releases/tag/1.4.1\" target=\"_blank\">1.4.1</a>, with MixUp. The transform could be applied to <code>images</code>, <code>masks</code>, and <code>global_labels</code> (we needed some name to call an overall label for the image, and we did not like <code>class_label</code>, hence new notation)</p>\n<ul>\n<li><a href=\"https://albumentations.ai/docs/api_reference/augmentations/mixing/transforms/#albumentations.augmentations.mixing.transforms.MixUp\" target=\"_blank\">Documentation</a></li>\n<li><a href=\"https://albumentations.ai/docs/examples/example_mixup/\" target=\"_blank\">Example of use in documentation</a></li>\n</ul>\n<p>The most important - <a href=\"https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations\" target=\"_blank\"><strong>Kaggle Kernel with an example of how to apply in this competition</strong>.</a></p>\n<p><strong>P.S.</strong> If you know transforms that are helpful, but not implemented in Albumentations - feel free to ping me in DM, <a href=\"https://github.com/albumentations-team/albumentations/issues\" target=\"_blank\">create issue in the repository</a>, or write here in comments.</p>\n<p><strong>P.P.S.</strong> The effect of the transform is not clear if you are not familiar with it on the spectrogram data =&gt; here is an example of natural images.  </p>\n<p><strong>P.P.P.S.</strong> I do not know if MixUp will be useful in this competition - but good luck anyway!</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F286455%2Febfab33fa5fe5d0d8af11841d7a7feff%2FScreenshot%202024-03-04%20at%2014.52.15.png?generation=1709602735723662&amp;alt=media\"></p>\n<p><strong>UPD</strong>: 2024-03-18. In version <a href=\"https://github.com/albumentations-team/albumentations/releases/tag/1.4.2\" target=\"_blank\">1.4.2</a> we added an option to return mixing coefficient that was used by MixUp (randomly changes every time pipeline is applied). <a href=\"https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations\" target=\"_blank\">Kaggle Kernel</a> is updated to show how to return it. </p>",
      "rawMarkdown": "In another [thread](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477825), @chemdatafarmer asked for MixUp augmentation.\n\nIt was long on our TODO list, but it was a final push.\n\nToday we released version [1.4.1](https://github.com/albumentations-team/albumentations/releases/tag/1.4.1), with MixUp. The transform could be applied to `images`, `masks`, and `global_labels` (we needed some name to call an overall label for the image, and we did not like `class_label`, hence new notation)\n\n* [Documentation](https://albumentations.ai/docs/api_reference/augmentations/mixing/transforms/#albumentations.augmentations.mixing.transforms.MixUp)\n* [Example of use in documentation](https://albumentations.ai/docs/examples/example_mixup/)\n\nThe most important - [**Kaggle Kernel with an example of how to apply in this competition**.](https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations)\n\n\n**P.S.** If you know transforms that are helpful, but not implemented in Albumentations - feel free to ping me in DM, [create issue in the repository](https://github.com/albumentations-team/albumentations/issues), or write here in comments.\n\n**P.P.S.** The effect of the transform is not clear if you are not familiar with it on the spectrogram data => here is an example of natural images.  \n\n**P.P.P.S.** I do not know if MixUp will be useful in this competition - but good luck anyway!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F286455%2Febfab33fa5fe5d0d8af11841d7a7feff%2FScreenshot%202024-03-04%20at%2014.52.15.png?generation=1709602735723662&alt=media)\n\n\n**UPD**: 2024-03-18. In version [1.4.2](https://github.com/albumentations-team/albumentations/releases/tag/1.4.2) we added an option to return mixing coefficient that was used by MixUp (randomly changes every time pipeline is applied). [Kaggle Kernel](https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations) is updated to show how to return it. ",
      "votes": 40
    },
    {
      "id": 2682685,
      "postDate": "2024-03-05T14:07:29.980Z",
      "content": "<p>Very helpful! And the example is hilarious haha</p>",
      "rawMarkdown": "Very helpful! And the example is hilarious haha",
      "votes": 1,
      "replies": [
        {
          "id": 2682935,
          "postDate": "2024-03-05T17:12:42.880Z",
          "content": "<p>Another good problem I am considering is what images to use to make them memorable without crossing boundaries.</p>\n<p>Spectrograms just do not have the same effect as woman + cat. </p>",
          "rawMarkdown": "Another good problem I am considering is what images to use to make them memorable without crossing boundaries.\n\nSpectrograms just do not have the same effect as woman + cat. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 2682025,
      "postDate": "2024-03-05T04:09:56.840Z",
      "content": "<p>Wow, this is awesome! Thank you so much~</p>",
      "rawMarkdown": "Wow, this is awesome! Thank you so much~",
      "votes": 1
    },
    {
      "id": 2681930,
      "postDate": "2024-03-05T02:38:37.230Z",
      "content": "<p>This is really cool! I'm very excited to give it a try :). This will make my augmentation pipeline for spectrograms really clean and easy to test/tune.</p>",
      "rawMarkdown": "This is really cool! I'm very excited to give it a try :). This will make my augmentation pipeline for spectrograms really clean and easy to test/tune.",
      "votes": 1,
      "replies": [
        {
          "id": 2682021,
          "postDate": "2024-03-05T04:06:01.423Z",
          "content": "<p>Thanks! When we created Albumentations, we did not think we would need to have an interface combining several images into one.</p>\n<p>In this sense, the existing implementations could not be as intuitive as I would like. Let me know if something is unclear.</p>",
          "rawMarkdown": "Thanks! When we created Albumentations, we did not think we would need to have an interface combining several images into one.\n\nIn this sense, the existing implementations could not be as intuitive as I would like. Let me know if something is unclear.",
          "votes": 1,
          "replies": [
            {
              "id": 2682612,
              "postDate": "2024-03-05T12:45:23.183Z",
              "content": "<p>Will do! It looks reasonably straightforward from your kaggle notebook. Work is keeping me a little busy right now, but I'll try to get to it before the weekend :)</p>",
              "rawMarkdown": "Will do! It looks reasonably straightforward from your kaggle notebook. Work is keeping me a little busy right now, but I'll try to get to it before the weekend :)",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2682070,
      "postDate": "2024-03-05T04:51:35.490Z",
      "content": "<p>I am not sure here is the right place to discuss it. <br>\n<a href=\"https://arxiv.org/pdf/1710.09412.pdf\" target=\"_blank\">In the paper of mixup</a>, mixup and its loss calculation are defined as follows.</p>\n<pre><code> (x1, y1), (x2, y2)  (loader1, loader2):\n      lam = numpy.random.beta(alpha, alpha)\n      x = Variable(lam * x1 + ( - lam) * x2)\n      y = Variable(lam * y1 + ( - lam) * y2)\n      optimizer.zero_grad()\n      loss(net(x), y).backward()\n</code></pre>\n<p>On the other hand, <a href=\"https://github.com/facebookresearch/mixup-cifar10/blob/main/train.py\" target=\"_blank\">in the official implementation of mixup</a>, mixup and its loss calculation are defined as follows.</p>\n<pre><code> ():\n       lam * criterion(pred, y_a) + ( - lam) * criterion(pred, y_b)\n\n batch_idx, (inputs, targets)  (trainloader):\n      inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, args.alpha, use_cuda)\n      outputs = net(inputs)\n      loss = mixup_criterion(criterion, outputs, targets_a, targets_b, lam)\n</code></pre>\n<p>It seems that the implementation in albumentation is based on former, but which is better?</p>",
      "rawMarkdown": "I am not sure here is the right place to discuss it. \n[In the paper of mixup](https://arxiv.org/pdf/1710.09412.pdf), mixup and its loss calculation are defined as follows.\n\n```python\nfor (x1, y1), (x2, y2) in zip(loader1, loader2):\n      lam = numpy.random.beta(alpha, alpha)\n      x = Variable(lam * x1 + (1. - lam) * x2)\n      y = Variable(lam * y1 + (1. - lam) * y2)\n      optimizer.zero_grad()\n      loss(net(x), y).backward()\n```\n\nOn the other hand, [in the official implementation of mixup](https://github.com/facebookresearch/mixup-cifar10/blob/main/train.py), mixup and its loss calculation are defined as follows.\n\n```python\ndef mixup_criterion(criterion, pred, y_a, y_b, lam):\n      return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)\n\nfor batch_idx, (inputs, targets) in enumerate(trainloader):\n      inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, args.alpha, use_cuda)\n      outputs = net(inputs)\n      loss = mixup_criterion(criterion, outputs, targets_a, targets_b, lam)\n```\n\nIt seems that the implementation in albumentation is based on former, but which is better?\n",
      "votes": 2,
      "replies": [
        {
          "id": 2682941,
          "postDate": "2024-03-05T17:17:04.540Z",
          "content": "<p>This is a great place to discuss it.</p>\n<p>Mixup as an augmentation and mixup as a loss is not directly related, although using weighted loss for examples coming of the MixUp to weight corresponding loss is a good idea. To use it, we need to update the transform to return mixing_parameter as well.</p>\n<p>In Albumentations  we use: <code>lam * x1 + (1. - lam) * x2</code> for <code>image</code>, <code>mask</code>, and <code>global_label</code></p>",
          "rawMarkdown": "This is a great place to discuss it.\n\nMixup as an augmentation and mixup as a loss is not directly related, although using weighted loss for examples coming of the MixUp to weight corresponding loss is a good idea. To use it, we need to update the transform to return mixing_parameter as well.\n\nIn Albumentations  we use: `lam * x1 + (1. - lam) * x2` for `image`, `mask`, and `global_label`",
          "votes": 1
        },
        {
          "id": 2704850,
          "postDate": "2024-03-19T00:41:46.443Z",
          "content": "<p>Just released version <a href=\"https://github.com/albumentations-team/albumentations/releases/tag/1.4.2\" target=\"_blank\">1.4.2</a>, where one can get the mixing coefficient for the MixUp transform. </p>\n<p>Updated <a href=\"https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations\" target=\"_blank\">Kaggle Kernel</a> to show the functionality. </p>",
          "rawMarkdown": "Just released version [1.4.2](https://github.com/albumentations-team/albumentations/releases/tag/1.4.2), where one can get the mixing coefficient for the MixUp transform. \n\nUpdated [Kaggle Kernel](https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations) to show the functionality. \n\n",
          "votes": 2,
          "replies": [
            {
              "id": 2704963,
              "postDate": "2024-03-19T02:58:36.733Z",
              "content": "<p><a href=\"https://www.kaggle.com/iglovikov\" target=\"_blank\">@iglovikov</a> Thank you very much for updating this great library.<br>\nTo calculate mixup as a loss, we need original global_label (y_a) and <strong>shuffled global_label (y_b)</strong>.<br>\nThis is official implementation of mixup.  </p>\n<pre><code> ():\n    \n     alpha &gt; :\n        lam = np.random.beta(alpha, alpha)\n    :\n        lam = \n    batch_size = x.size()[]\n     use_cuda:\n        index = torch.randperm(batch_size).cuda()\n    :\n        index = torch.randperm(batch_size)\n    mixed_x = lam * x + ( - lam) * x[index, :]\n    y_a, y_b = y, y[index]\n     mixed_x, y_a, y_b, lam\n</code></pre>\n<p>I am very sorry for not notifying this point.</p>",
              "rawMarkdown": "@iglovikov Thank you very much for updating this great library.\nTo calculate mixup as a loss, we need original global_label (y_a) and **shuffled global_label (y_b)**.\nThis is official implementation of mixup.  \n```\ndef mixup_data(x, y, alpha=1.0, use_cuda=True):\n    '''Returns mixed inputs, pairs of targets, and lambda'''\n    if alpha > 0:\n        lam = np.random.beta(alpha, alpha)\n    else:\n        lam = 1\n    batch_size = x.size()[0]\n    if use_cuda:\n        index = torch.randperm(batch_size).cuda()\n    else:\n        index = torch.randperm(batch_size)\n    mixed_x = lam * x + (1 - lam) * x[index, :]\n    y_a, y_b = y, y[index]\n    return mixed_x, y_a, y_b, lam\n```\n\nI am very sorry for not notifying this point.",
              "votes": 1
            },
            {
              "id": 2706665,
              "postDate": "2024-03-20T03:13:17.663Z",
              "content": "<p>You can get from the  transform:</p>\n<ol>\n<li>mixing coefficient (<code>lambda</code>)</li>\n<li>shuffled global label (<code>y_b</code>)</li>\n<li>Shuffled image (<code>mixed_x</code>)</li>\n<li>Shuffled mask </li>\n</ol>\n<pre><code>transformed = transform(image = img, global_label=global_label, mask=mask)\n(, transformed[])\n(, transformed[])\n(, transformed[])\n(, transformed[])\n</code></pre>",
              "rawMarkdown": "You can get from the  transform:\n\n1. mixing coefficient (`lambda`)\n2. shuffled global label (`y_b`)\n3. Shuffled image (`mixed_x`)\n3. Shuffled mask \n\n```python\ntransformed = transform(image = img, global_label=global_label, mask=mask)\nprint(\"Shuffled global label = \", transformed[\"global_label\"])\nprint(\"Mixing coefficient = \", transformed[\"mix_coef\"])\nprint(\"Shuffled image = \", transformed[\"image\"])\nprint(\"Shuffled mask = \", transformed[\"mask\"])\n``` ",
              "votes": 1
            },
            {
              "id": 2706711,
              "postDate": "2024-03-20T03:47:27.050Z",
              "content": "<p>Thank you for your reply, but transformed[\"global_label\"] is not shuffled global label (i.e., y_b or y[index]).<br>\nIt is mixed global label (i.e., lam * y_a + (1 - lam) * y_b or lam * y + (1 - lam) * y[index]).<br>\nCorrect me if I am wrong.</p>",
              "rawMarkdown": "Thank you for your reply, but transformed[\"global_label\"] is not shuffled global label (i.e., y_b or y[index]).\nIt is mixed global label (i.e., lam * y_a + (1 - lam) * y_b or lam * y + (1 - lam) * y[index]).\nCorrect me if I am wrong.",
              "votes": 1
            },
            {
              "id": 2709357,
              "postDate": "2024-03-21T15:37:19.720Z",
              "content": "<p>Made a mistake in the last post. </p>\n<p>Let's try again :)</p>\n<p>The important thing here is that in the code above, you work on the batch level, and mixing images within a batch looks more natural.</p>\n<p>MixUp in Albumentations is more hackish in this sense as it processes one at a time.</p>\n<p>In Albumentations:</p>\n<ol>\n<li>When we define MixUp transform, we specify a generator or list of \"reference images.\"</li>\n<li>When we apply MixUp to our target image and global label, transform mixes the target image and label with a randomly sampled pair from reference images.</li>\n</ol>\n<p>Say we picked <code>x_1</code> and <code>y_1</code> from reference images and tried to apply them to our target pair<br>\n<code>x_0</code> and <code>y_0</code>.</p>\n<p>Mixup Transform samples <code>lambda = mix_coef</code> (we do not use lambda in the code as it is a reserved word, call it mix_coef instead), computes <code>mixed_global_label</code> and <code>mixed_image</code> as:</p>\n<pre><code>new_global_label =  mix_coef * y_0 + ( - mix_coef) * y_1\nnew_image =  mix_coef * x_0 + ( - mix_coef) * x_1\n</code></pre>\n<p>and transform returns: <code>new_global_label</code>, <code>new_image</code>, <code>mix_coef</code></p>\n<p>which is enough to compute</p>\n<pre><code>mix_coef * criterion \n</code></pre>\n<p>for every transformed image.</p>\n<p>but to apply it to batch of transformed images and extra step is involved - make vector of mix_coefficients as they are different for every transformed image.</p>\n<p>=&gt; </p>\n<p>There is no shuffling in a way that standard mixup does.</p>\n<p>I guess to use Albumentations, one needs to:</p>\n<ol>\n<li>Mix with images that have the same class label, which could be done by adjusting <code>read_fn</code> and <code>reference_data</code>, which are used to initialize the transform.</li>\n<li>Mix with random images but update the loss function from</li>\n</ol>\n<pre><code> ():\n       lam * criterion(pred, y_a) + ( - lam) * criterion(pred, y_b)\n</code></pre>\n<p>to </p>\n<pre><code> ():\n       lam_vector * criterion(pred, y)\n</code></pre>\n<p>Another option is to extend MixUp to return images and labels that were used for mixing, and it will become:</p>\n<pre><code> ():\n       lam_vector * criterion(pred, y_a) + ( - lam_vector) * criterion(pred, y_b)\n</code></pre>\n<p>While writing it, I got to the point that:</p>\n<ol>\n<li>We need to return the reference_data that was used for mixing with the target image-label pair.</li>\n<li>Write a long blog post/tutorial that talks about all these things in detail, as due to the current limitations of Albumentations, it is not very natural to apply it to a batch. Although this looks like a natural direction.</li>\n</ol>\n<p>Also, Technically, we can add Mosaic, CutMix, etc., to Albumentations, but I am hesitant to do it now as it is not clear yet how to do it most naturally. In this sense, your help and attention to the details here are ultra-valuable. Thank you!</p>",
              "rawMarkdown": "Made a mistake in the last post. \n\nLet's try again :)\n\nThe important thing here is that in the code above, you work on the batch level, and mixing images within a batch looks more natural.\n\nMixUp in Albumentations is more hackish in this sense as it processes one at a time.\n\nIn Albumentations:\n\n1. When we define MixUp transform, we specify a generator or list of \"reference images.\"\n2. When we apply MixUp to our target image and global label, transform mixes the target image and label with a randomly sampled pair from reference images.\n\nSay we picked `x_1` and `y_1` from reference images and tried to apply them to our target pair\n`x_0` and `y_0`.\n\nMixup Transform samples `lambda = mix_coef` (we do not use lambda in the code as it is a reserved word, call it mix_coef instead), computes `mixed_global_label` and `mixed_image` as:\n\n```python\nnew_global_label =  mix_coef * y_0 + (1 - mix_coef) * y_1\nnew_image =  mix_coef * x_0 + (1 - mix_coef) * x_1\n```  \n\nand transform returns: `new_global_label`, `new_image`, `mix_coef`\n\nwhich is enough to compute\n\n```python\nmix_coef * criterion \n```\nfor every transformed image.\n\nbut to apply it to batch of transformed images and extra step is involved - make vector of mix_coefficients as they are different for every transformed image.\n \n=> \n\nThere is no shuffling in a way that standard mixup does.\n\nI guess to use Albumentations, one needs to:\n\n1. Mix with images that have the same class label, which could be done by adjusting `read_fn` and `reference_data`, which are used to initialize the transform.\n2. Mix with random images but update the loss function from\n\n```python\ndef mixup_criterion(criterion, pred, y_a, y_b, lam):\n      return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)\n```  \n\nto \n```python\ndef mixup_criterion(criterion, pred, y, lam_vector):\n      return lam_vector * criterion(pred, y)\n```\n\nAnother option is to extend MixUp to return images and labels that were used for mixing, and it will become:\n\n```python\ndef mixup_criterion(criterion, pred, y_a, y_b, lam_vector):\n      return lam_vector * criterion(pred, y_a) + (1 - lam_vector) * criterion(pred, y_b)\n```\n\nWhile writing it, I got to the point that:\n\n1. We need to return the reference_data that was used for mixing with the target image-label pair.\n2. Write a long blog post/tutorial that talks about all these things in detail, as due to the current limitations of Albumentations, it is not very natural to apply it to a batch. Although this looks like a natural direction.\n\nAlso, Technically, we can add Mosaic, CutMix, etc., to Albumentations, but I am hesitant to do it now as it is not clear yet how to do it most naturally. In this sense, your help and attention to the details here are ultra-valuable. Thank you!",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2721833,
      "postDate": "2024-03-29T08:44:37.190Z",
      "content": "<p>It works fine for CPU instances, but for P100 instances, I get the following error: module 'albumentations' has no attribute 'MixUp'.</p>",
      "rawMarkdown": "It works fine for CPU instances, but for P100 instances, I get the following error: module 'albumentations' has no attribute 'MixUp'."
    },
    {
      "id": 2689209,
      "postDate": "2024-03-09T18:38:26.103Z",
      "content": "<p>Hi, I'm having trouble running the example notebook using the 2 cat pics and woman. Getting errors. Can you tell me where I'm going wrong? I had downloaded the 3 pics as png but that doesn't seem to be the problem</p>\n<p>the error I get is: </p>\n<p><code>error: OpenCV(4.9.0) /io/opencv/modules/imgproc/src/color.cpp:196: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'</code></p>",
      "rawMarkdown": "Hi, I'm having trouble running the example notebook using the 2 cat pics and woman. Getting errors. Can you tell me where I'm going wrong? I had downloaded the 3 pics as png but that doesn't seem to be the problem\n\nthe error I get is: \n\n`error: OpenCV(4.9.0) /io/opencv/modules/imgproc/src/color.cpp:196: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'`\n",
      "replies": [
        {
          "id": 2706667,
          "postDate": "2024-03-20T03:15:16.513Z",
          "content": "<p>I suspect that in <code>img = load_rgb(img_path)</code> <code>img_path</code> does not point to the image. </p>",
          "rawMarkdown": "I suspect that in `img = load_rgb(img_path)` `img_path` does not point to the image. "
        }
      ]
    },
    {
      "id": 2683266,
      "postDate": "2024-03-05T21:19:14.827Z",
      "content": "<p>Super helpful - thanks!</p>\n<p>Quick question - how do I use this version on Kaggle with internet off? Do I need to add a separate notebook with the wheel file?</p>",
      "rawMarkdown": "Super helpful - thanks!\n\nQuick question - how do I use this version on Kaggle with internet off? Do I need to add a separate notebook with the wheel file?",
      "replies": [
        {
          "id": 2688505,
          "postDate": "2024-03-09T09:44:16.617Z",
          "content": "<p>You don't need to. Train online. Save and upload the model to inference notebook. And submit offline notebook.</p>",
          "rawMarkdown": "You don't need to. Train online. Save and upload the model to inference notebook. And submit offline notebook.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2721828,
      "postDate": "2024-03-29T08:40:33.290Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 2690469,
      "postDate": "2024-03-10T14:47:27.630Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 2681913,
      "postDate": "2024-03-05T01:53:02.830Z",
      "content": "<p>Great work! Thanks.</p>",
      "rawMarkdown": "Great work! Thanks.",
      "votes": 4
    }
  ],
  "comments": [
    {
      "id": 2682685,
      "author_name": "Cody_Null",
      "author_url": "",
      "post_date": "2024-03-05T14:07:29.980000",
      "content": "<p>Very helpful! And the example is hilarious haha</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2682935,
          "author_name": "Vladimir Iglovikov",
          "author_url": "",
          "post_date": "2024-03-05T17:12:42.880000",
          "content": "<p>Another good problem I am considering is what images to use to make them memorable without crossing boundaries.</p>\n<p>Spectrograms just do not have the same effect as woman + cat. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2682025,
      "author_name": "JSLim7",
      "author_url": "",
      "post_date": "2024-03-05T04:09:56.840000",
      "content": "<p>Wow, this is awesome! Thank you so much~</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2681930,
      "author_name": "chemdatafarmer",
      "author_url": "",
      "post_date": "2024-03-05T02:38:37.230000",
      "content": "<p>This is really cool! I'm very excited to give it a try :). This will make my augmentation pipeline for spectrograms really clean and easy to test/tune.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2682021,
          "author_name": "Vladimir Iglovikov",
          "author_url": "",
          "post_date": "2024-03-05T04:06:01.423000",
          "content": "<p>Thanks! When we created Albumentations, we did not think we would need to have an interface combining several images into one.</p>\n<p>In this sense, the existing implementations could not be as intuitive as I would like. Let me know if something is unclear.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2682612,
              "author_name": "chemdatafarmer",
              "author_url": "",
              "post_date": "2024-03-05T12:45:23.183000",
              "content": "<p>Will do! It looks reasonably straightforward from your kaggle notebook. Work is keeping me a little busy right now, but I'll try to get to it before the weekend :)</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2682070,
      "author_name": "tomoo inubushi",
      "author_url": "",
      "post_date": "2024-03-05T04:51:35.490000",
      "content": "<p>I am not sure here is the right place to discuss it. <br>\n<a href=\"https://arxiv.org/pdf/1710.09412.pdf\" target=\"_blank\">In the paper of mixup</a>, mixup and its loss calculation are defined as follows.</p>\n<pre><code> (x1, y1), (x2, y2)  (loader1, loader2):\n      lam = numpy.random.beta(alpha, alpha)\n      x = Variable(lam * x1 + ( - lam) * x2)\n      y = Variable(lam * y1 + ( - lam) * y2)\n      optimizer.zero_grad()\n      loss(net(x), y).backward()\n</code></pre>\n<p>On the other hand, <a href=\"https://github.com/facebookresearch/mixup-cifar10/blob/main/train.py\" target=\"_blank\">in the official implementation of mixup</a>, mixup and its loss calculation are defined as follows.</p>\n<pre><code> ():\n       lam * criterion(pred, y_a) + ( - lam) * criterion(pred, y_b)\n\n batch_idx, (inputs, targets)  (trainloader):\n      inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, args.alpha, use_cuda)\n      outputs = net(inputs)\n      loss = mixup_criterion(criterion, outputs, targets_a, targets_b, lam)\n</code></pre>\n<p>It seems that the implementation in albumentation is based on former, but which is better?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2682941,
          "author_name": "Vladimir Iglovikov",
          "author_url": "",
          "post_date": "2024-03-05T17:17:04.540000",
          "content": "<p>This is a great place to discuss it.</p>\n<p>Mixup as an augmentation and mixup as a loss is not directly related, although using weighted loss for examples coming of the MixUp to weight corresponding loss is a good idea. To use it, we need to update the transform to return mixing_parameter as well.</p>\n<p>In Albumentations  we use: <code>lam * x1 + (1. - lam) * x2</code> for <code>image</code>, <code>mask</code>, and <code>global_label</code></p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2704850,
          "author_name": "Vladimir Iglovikov",
          "author_url": "",
          "post_date": "2024-03-19T00:41:46.443000",
          "content": "<p>Just released version <a href=\"https://github.com/albumentations-team/albumentations/releases/tag/1.4.2\" target=\"_blank\">1.4.2</a>, where one can get the mixing coefficient for the MixUp transform. </p>\n<p>Updated <a href=\"https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations\" target=\"_blank\">Kaggle Kernel</a> to show the functionality. </p>",
          "votes": 2,
          "replies": [
            {
              "id": 2704963,
              "author_name": "tomoo inubushi",
              "author_url": "",
              "post_date": "2024-03-19T02:58:36.733000",
              "content": "<p><a href=\"https://www.kaggle.com/iglovikov\" target=\"_blank\">@iglovikov</a> Thank you very much for updating this great library.<br>\nTo calculate mixup as a loss, we need original global_label (y_a) and <strong>shuffled global_label (y_b)</strong>.<br>\nThis is official implementation of mixup.  </p>\n<pre><code> ():\n    \n     alpha &gt; :\n        lam = np.random.beta(alpha, alpha)\n    :\n        lam = \n    batch_size = x.size()[]\n     use_cuda:\n        index = torch.randperm(batch_size).cuda()\n    :\n        index = torch.randperm(batch_size)\n    mixed_x = lam * x + ( - lam) * x[index, :]\n    y_a, y_b = y, y[index]\n     mixed_x, y_a, y_b, lam\n</code></pre>\n<p>I am very sorry for not notifying this point.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2706665,
              "author_name": "Vladimir Iglovikov",
              "author_url": "",
              "post_date": "2024-03-20T03:13:17.663000",
              "content": "<p>You can get from the  transform:</p>\n<ol>\n<li>mixing coefficient (<code>lambda</code>)</li>\n<li>shuffled global label (<code>y_b</code>)</li>\n<li>Shuffled image (<code>mixed_x</code>)</li>\n<li>Shuffled mask </li>\n</ol>\n<pre><code>transformed = transform(image = img, global_label=global_label, mask=mask)\n(, transformed[])\n(, transformed[])\n(, transformed[])\n(, transformed[])\n</code></pre>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2706711,
              "author_name": "tomoo inubushi",
              "author_url": "",
              "post_date": "2024-03-20T03:47:27.050000",
              "content": "<p>Thank you for your reply, but transformed[\"global_label\"] is not shuffled global label (i.e., y_b or y[index]).<br>\nIt is mixed global label (i.e., lam * y_a + (1 - lam) * y_b or lam * y + (1 - lam) * y[index]).<br>\nCorrect me if I am wrong.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2709357,
              "author_name": "Vladimir Iglovikov",
              "author_url": "",
              "post_date": "2024-03-21T15:37:19.720000",
              "content": "<p>Made a mistake in the last post. </p>\n<p>Let's try again :)</p>\n<p>The important thing here is that in the code above, you work on the batch level, and mixing images within a batch looks more natural.</p>\n<p>MixUp in Albumentations is more hackish in this sense as it processes one at a time.</p>\n<p>In Albumentations:</p>\n<ol>\n<li>When we define MixUp transform, we specify a generator or list of \"reference images.\"</li>\n<li>When we apply MixUp to our target image and global label, transform mixes the target image and label with a randomly sampled pair from reference images.</li>\n</ol>\n<p>Say we picked <code>x_1</code> and <code>y_1</code> from reference images and tried to apply them to our target pair<br>\n<code>x_0</code> and <code>y_0</code>.</p>\n<p>Mixup Transform samples <code>lambda = mix_coef</code> (we do not use lambda in the code as it is a reserved word, call it mix_coef instead), computes <code>mixed_global_label</code> and <code>mixed_image</code> as:</p>\n<pre><code>new_global_label =  mix_coef * y_0 + ( - mix_coef) * y_1\nnew_image =  mix_coef * x_0 + ( - mix_coef) * x_1\n</code></pre>\n<p>and transform returns: <code>new_global_label</code>, <code>new_image</code>, <code>mix_coef</code></p>\n<p>which is enough to compute</p>\n<pre><code>mix_coef * criterion \n</code></pre>\n<p>for every transformed image.</p>\n<p>but to apply it to batch of transformed images and extra step is involved - make vector of mix_coefficients as they are different for every transformed image.</p>\n<p>=&gt; </p>\n<p>There is no shuffling in a way that standard mixup does.</p>\n<p>I guess to use Albumentations, one needs to:</p>\n<ol>\n<li>Mix with images that have the same class label, which could be done by adjusting <code>read_fn</code> and <code>reference_data</code>, which are used to initialize the transform.</li>\n<li>Mix with random images but update the loss function from</li>\n</ol>\n<pre><code> ():\n       lam * criterion(pred, y_a) + ( - lam) * criterion(pred, y_b)\n</code></pre>\n<p>to </p>\n<pre><code> ():\n       lam_vector * criterion(pred, y)\n</code></pre>\n<p>Another option is to extend MixUp to return images and labels that were used for mixing, and it will become:</p>\n<pre><code> ():\n       lam_vector * criterion(pred, y_a) + ( - lam_vector) * criterion(pred, y_b)\n</code></pre>\n<p>While writing it, I got to the point that:</p>\n<ol>\n<li>We need to return the reference_data that was used for mixing with the target image-label pair.</li>\n<li>Write a long blog post/tutorial that talks about all these things in detail, as due to the current limitations of Albumentations, it is not very natural to apply it to a batch. Although this looks like a natural direction.</li>\n</ol>\n<p>Also, Technically, we can add Mosaic, CutMix, etc., to Albumentations, but I am hesitant to do it now as it is not clear yet how to do it most naturally. In this sense, your help and attention to the details here are ultra-valuable. Thank you!</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2721833,
      "author_name": "theta",
      "author_url": "",
      "post_date": "2024-03-29T08:44:37.190000",
      "content": "<p>It works fine for CPU instances, but for P100 instances, I get the following error: module 'albumentations' has no attribute 'MixUp'.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2689209,
      "author_name": "djtrainwreckx",
      "author_url": "",
      "post_date": "2024-03-09T18:38:26.103000",
      "content": "<p>Hi, I'm having trouble running the example notebook using the 2 cat pics and woman. Getting errors. Can you tell me where I'm going wrong? I had downloaded the 3 pics as png but that doesn't seem to be the problem</p>\n<p>the error I get is: </p>\n<p><code>error: OpenCV(4.9.0) /io/opencv/modules/imgproc/src/color.cpp:196: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'</code></p>",
      "votes": 0,
      "replies": [
        {
          "id": 2706667,
          "author_name": "Vladimir Iglovikov",
          "author_url": "",
          "post_date": "2024-03-20T03:15:16.513000",
          "content": "<p>I suspect that in <code>img = load_rgb(img_path)</code> <code>img_path</code> does not point to the image. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2683266,
      "author_name": "Ry",
      "author_url": "",
      "post_date": "2024-03-05T21:19:14.827000",
      "content": "<p>Super helpful - thanks!</p>\n<p>Quick question - how do I use this version on Kaggle with internet off? Do I need to add a separate notebook with the wheel file?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2688505,
          "author_name": "Ángel Jacinto Sánchez Ruiz",
          "author_url": "",
          "post_date": "2024-03-09T09:44:16.617000",
          "content": "<p>You don't need to. Train online. Save and upload the model to inference notebook. And submit offline notebook.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2721828,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-03-29T08:40:33.290000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2690469,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-03-10T14:47:27.630000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2681913,
      "author_name": "Chris Deotte",
      "author_url": "",
      "post_date": "2024-03-05T01:53:02.830000",
      "content": "<p>Great work! Thanks.</p>",
      "votes": 4,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2681902": "In another [thread](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477825), @chemdatafarmer asked for MixUp augmentation.\n\nIt was long on our TODO list, but it was a final push.\n\nToday we released version [1.4.1](https://github.com/albumentations-team/albumentations/releases/tag/1.4.1), with MixUp. The transform could be applied to `images`, `masks`, and `global_labels` (we needed some name to call an overall label for the image, and we did not like `class_label`, hence new notation)\n\n* [Documentation](https://albumentations.ai/docs/api_reference/augmentations/mixing/transforms/#albumentations.augmentations.mixing.transforms.MixUp)\n* [Example of use in documentation](https://albumentations.ai/docs/examples/example_mixup/)\n\nThe most important - [**Kaggle Kernel with an example of how to apply in this competition**.](https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations)\n\n\n**P.S.** If you know transforms that are helpful, but not implemented in Albumentations - feel free to ping me in DM, [create issue in the repository](https://github.com/albumentations-team/albumentations/issues), or write here in comments.\n\n**P.P.S.** The effect of the transform is not clear if you are not familiar with it on the spectrogram data => here is an example of natural images.  \n\n**P.P.P.S.** I do not know if MixUp will be useful in this competition - but good luck anyway!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F286455%2Febfab33fa5fe5d0d8af11841d7a7feff%2FScreenshot%202024-03-04%20at%2014.52.15.png?generation=1709602735723662&alt=media)\n\n\n**UPD**: 2024-03-18. In version [1.4.2](https://github.com/albumentations-team/albumentations/releases/tag/1.4.2) we added an option to return mixing coefficient that was used by MixUp (randomly changes every time pipeline is applied). [Kaggle Kernel](https://www.kaggle.com/code/iglovikov/mixup-aug-in-albumentations) is updated to show how to return it. ",
    "2682685": "Very helpful! And the example is hilarious haha",
    "2682025": "Wow, this is awesome! Thank you so much~",
    "2681930": "This is really cool! I'm very excited to give it a try :). This will make my augmentation pipeline for spectrograms really clean and easy to test/tune.",
    "2682070": "I am not sure here is the right place to discuss it. \n[In the paper of mixup](https://arxiv.org/pdf/1710.09412.pdf), mixup and its loss calculation are defined as follows.\n\n```python\nfor (x1, y1), (x2, y2) in zip(loader1, loader2):\n      lam = numpy.random.beta(alpha, alpha)\n      x = Variable(lam * x1 + (1. - lam) * x2)\n      y = Variable(lam * y1 + (1. - lam) * y2)\n      optimizer.zero_grad()\n      loss(net(x), y).backward()\n```\n\nOn the other hand, [in the official implementation of mixup](https://github.com/facebookresearch/mixup-cifar10/blob/main/train.py), mixup and its loss calculation are defined as follows.\n\n```python\ndef mixup_criterion(criterion, pred, y_a, y_b, lam):\n      return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)\n\nfor batch_idx, (inputs, targets) in enumerate(trainloader):\n      inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, args.alpha, use_cuda)\n      outputs = net(inputs)\n      loss = mixup_criterion(criterion, outputs, targets_a, targets_b, lam)\n```\n\nIt seems that the implementation in albumentation is based on former, but which is better?\n",
    "2721833": "It works fine for CPU instances, but for P100 instances, I get the following error: module 'albumentations' has no attribute 'MixUp'.",
    "2689209": "Hi, I'm having trouble running the example notebook using the 2 cat pics and woman. Getting errors. Can you tell me where I'm going wrong? I had downloaded the 3 pics as png but that doesn't seem to be the problem\n\nthe error I get is: \n\n`error: OpenCV(4.9.0) /io/opencv/modules/imgproc/src/color.cpp:196: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'`\n",
    "2683266": "Super helpful - thanks!\n\nQuick question - how do I use this version on Kaggle with internet off? Do I need to add a separate notebook with the wheel file?",
    "2721828": "",
    "2690469": "",
    "2681913": "Great work! Thanks."
  }
}