I see a lot of pytorch model code that does the following to move from a batched 2D tensor of shape B x H x W to a batched image with 3 channels of shape B x 3 x H x W:
\n x = x.unsqueeze(-)\n x = torch.cat([x, x, x], dim=).permute(, , , )\nThis code does a lot of data movement. A much more efficient way to do this avoids all data movements:
\n x = x.unsqueeze()\n x = x.expand(-, , -, -)\nI see a lot of pytorch model code that does the following to move from a batched 2D tensor of shape B x H x W to a batched image with 3 channels of shape B x 3 x H x W:
\n x = x.unsqueeze(-)\n x = torch.cat([x, x, x], dim=).permute(, , , )\nThis code does a lot of data movement. A much more efficient way to do this avoids all data movements:
\n x = x.unsqueeze()\n x = x.expand(-, , -, -)\nwhich is also better than
\n = x.unsqueeze()\n = x.repeat(-, , -, -)\nas if the original dimension you want to expand is of size 1, it uses additional memory, which expand doesn't.
Indeed.
\nAnd if we want to use cat, why not stack directly on a new dimension 1?
\nx = torch.stack([x, x, x], dim=1)
Note that if your goal is to produce a tensor to feed into a computer vision ConvNet, you can sum up the channels in the network's first convolution kernel into one instead of triplicating the channel in the input tensor.
", "rawMarkdown": "Note that if your goal is to produce a tensor to feed into a computer vision ConvNet, you can sum up the channels in the network's first convolution kernel into one instead of triplicating the channel in the input tensor.", "votes": null }, { "id": "2755867", "postDate": "04/16/2024 17:44:46", "content": "That's true. I used this trick in some past competition indeed.
", "rawMarkdown": "That's true. I used this trick in some past competition indeed.", "votes": null }, { "id": "2760860", "postDate": "04/19/2024 14:33:16", "content": "Pretty sure it's the default behavior in the timm library, btw.
", "rawMarkdown": "Pretty sure it's the default behavior in the timm library, btw.", "votes": null }, { "id": "2760872", "postDate": "04/19/2024 14:41:58", "content": "what default behavior?
", "rawMarkdown": "what default behavior?", "votes": null }, { "id": "2761212", "postDate": "04/19/2024 17:46:36", "content": "In timm library, you can specify input channels. If this number is equal to one, the first conv layer is transformed so that its weights are averaged.
", "rawMarkdown": "In timm library, you can specify input channels. If this number is equal to one, the first conv layer is transformed so that its weights are averaged.", "votes": null }, { "id": "2761213", "postDate": "04/19/2024 17:49:35", "content": "Yeah, exactly. https://timm.fast.ai/models#Case-1:-When-the-number-of-input-channels-is-1
\nSo timm essentially implemented tricks that Kagglers used for years. Fantastic package.
", "rawMarkdown": "Yeah, exactly. https://timm.fast.ai/models#Case-1:-When-the-number-of-input-channels-is-1\n\nSo timm essentially implemented tricks that Kagglers used for years. Fantastic package.", "votes": null }, { "id": "2761241", "postDate": "04/19/2024 18:18:43", "content": "\n\nIf this number is equal to one,
\n
What if this number is not one?
", "rawMarkdown": "> If this number is equal to one,\n\nWhat if this number is not one?", "votes": null }, { "id": "2761254", "postDate": "04/19/2024 18:28:56", "content": "It's literally the next point in the link I sent :)
\n(Case 2)
", "rawMarkdown": "It's literally the next point in the link I sent :) \n\n(Case 2)", "votes": null }, { "id": "2761289", "postDate": "04/19/2024 18:54:13", "content": "Thanks. I should have read that do entirely.
", "rawMarkdown": "Thanks. I should have read that do entirely.", "votes": null }, { "id": "2771209", "postDate": "04/24/2024 06:36:48", "content": "Why is the method using torch.cat considered to be memory intensive?
", "rawMarkdown": "Why is the method using torch.cat considered to be memory intensive?", "votes": null }, { "id": "2771335", "postDate": "04/24/2024 07:41:44", "content": "Pretty sure it's because you create a new tensor as apposed to creating a tensor once, and then modifying it in place
", "rawMarkdown": "Pretty sure it's because you create a new tensor as apposed to creating a tensor once, and then modifying it in place", "votes": null }, { "id": "2771570", "postDate": "04/24/2024 09:58:54", "content": "Because it is. When you do cat, you create a new tensor that is 3 times the size of the original tensor. And after permute you get a view of the tensor that is not contiguous, meaning that the next operation on that tensor will be slow.
\nWith expand you don't allocate memory, and your tensor is contiguous in memory.
\nWhy allocate useless memory when you can avoid it with simple code?
\nI didn't share this trick because the wasted memory was critical. It is only about 600 kB for a 224x224 image. I shared it because it can be reused easily, and can make a difference when dealing with large images. In my previous comp it enabled me to train with larger batches, thereby running experiments faster, and getting a better final result.
", "rawMarkdown": "Because it is. When you do cat, you create a new tensor that is 3 times the size of the original tensor. And after permute you get a view of the tensor that is not contiguous, meaning that the next operation on that tensor will be slow. \n\nWith expand you don't allocate memory, and your tensor is contiguous in memory.\n\nWhy allocate useless memory when you can avoid it with simple code?\n\nI didn't share this trick because the wasted memory was critical. It is only about 600 kB for a 224x224 image. I shared it because it can be reused easily, and can make a difference when dealing with large images. In my previous comp it enabled me to train with larger batches, thereby running experiments faster, and getting a better final result.", "votes": null }, { "id": "2771572", "postDate": "04/24/2024 09:59:34", "content": "The tensor data is not modified by expand. That's why I shared.
", "rawMarkdown": "The tensor data is not modified by expand. That's why I shared.", "votes": null }, { "id": "2771635", "postDate": "04/24/2024 10:41:16", "content": "The distinction regarding \"contiguous in memory or not\" is a great one
", "rawMarkdown": "The distinction regarding \"contiguous in memory or not\" is a great one", "votes": null } ], "comments": [ { "id": 2754910, "author_name": "janmpia", "author_url": "", "post_date": "04/16/2024 09:05:48", "content": "which is also better than
\n = x.unsqueeze()\n = x.repeat(-, , -, -)\nas if the original dimension you want to expand is of size 1, it uses additional memory, which expand doesn't.
Indeed.
\nAnd if we want to use cat, why not stack directly on a new dimension 1?
\nx = torch.stack([x, x, x], dim=1)
Note that if your goal is to produce a tensor to feed into a computer vision ConvNet, you can sum up the channels in the network's first convolution kernel into one instead of triplicating the channel in the input tensor.
", "votes": null, "replies": [ { "id": 2755867, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2024 17:44:46", "content": "That's true. I used this trick in some past competition indeed.
", "votes": null, "replies": [] }, { "id": 2760860, "author_name": "ivanpan", "author_url": "", "post_date": "04/19/2024 14:33:16", "content": "Pretty sure it's the default behavior in the timm library, btw.
", "votes": null, "replies": [ { "id": 2760872, "author_name": "cpmpml", "author_url": "", "post_date": "04/19/2024 14:41:58", "content": "what default behavior?
", "votes": null, "replies": [ { "id": 2761212, "author_name": "kdmitrie", "author_url": "", "post_date": "04/19/2024 17:46:36", "content": "In timm library, you can specify input channels. If this number is equal to one, the first conv layer is transformed so that its weights are averaged.
", "votes": null, "replies": [ { "id": 2761213, "author_name": "ivanpan", "author_url": "", "post_date": "04/19/2024 17:49:35", "content": "Yeah, exactly. https://timm.fast.ai/models#Case-1:-When-the-number-of-input-channels-is-1
\nSo timm essentially implemented tricks that Kagglers used for years. Fantastic package.
", "votes": null, "replies": [ { "id": 2761241, "author_name": "cpmpml", "author_url": "", "post_date": "04/19/2024 18:18:43", "content": "\n\nIf this number is equal to one,
\n
What if this number is not one?
", "votes": null, "replies": [ { "id": 2761254, "author_name": "ivanpan", "author_url": "", "post_date": "04/19/2024 18:28:56", "content": "It's literally the next point in the link I sent :)
\n(Case 2)
", "votes": null, "replies": [ { "id": 2761289, "author_name": "cpmpml", "author_url": "", "post_date": "04/19/2024 18:54:13", "content": "Thanks. I should have read that do entirely.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2771209, "author_name": "soratakaya", "author_url": "", "post_date": "04/24/2024 06:36:48", "content": "Why is the method using torch.cat considered to be memory intensive?
", "votes": null, "replies": [ { "id": 2771335, "author_name": "ivanpan", "author_url": "", "post_date": "04/24/2024 07:41:44", "content": "Pretty sure it's because you create a new tensor as apposed to creating a tensor once, and then modifying it in place
", "votes": null, "replies": [ { "id": 2771572, "author_name": "cpmpml", "author_url": "", "post_date": "04/24/2024 09:59:34", "content": "The tensor data is not modified by expand. That's why I shared.
", "votes": null, "replies": [] } ] }, { "id": 2771570, "author_name": "cpmpml", "author_url": "", "post_date": "04/24/2024 09:58:54", "content": "Because it is. When you do cat, you create a new tensor that is 3 times the size of the original tensor. And after permute you get a view of the tensor that is not contiguous, meaning that the next operation on that tensor will be slow.
\nWith expand you don't allocate memory, and your tensor is contiguous in memory.
\nWhy allocate useless memory when you can avoid it with simple code?
\nI didn't share this trick because the wasted memory was critical. It is only about 600 kB for a 224x224 image. I shared it because it can be reused easily, and can make a difference when dealing with large images. In my previous comp it enabled me to train with larger batches, thereby running experiments faster, and getting a better final result.
", "votes": null, "replies": [ { "id": 2771635, "author_name": "ivanpan", "author_url": "", "post_date": "04/24/2024 10:41:16", "content": "The distinction regarding \"contiguous in memory or not\" is a great one
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2754439": "I see a lot of pytorch model code that does the following to move from a batched 2D tensor of shape B x H x W to a batched image with 3 channels of shape B x 3 x H x W:\n\n```python\n x = x.unsqueeze(-1)\n x = torch.cat([x, x, x], dim=3).permute(0, 3, 1, 2)\n\n```\n\nThis code does a lot of data movement. A much more efficient way to do this avoids all data movements:\n\n```python\n x = x.unsqueeze(1)\n x = x.expand(-1, 3, -1, -1)\n```", "2754910": "which is also better than \n```\nx = x.unsqueeze(1)\nx = x.repeat(-1, 3, -1, -1)\n```\n\nas if the original dimension you want to expand is of size 1, it uses additional memory, which `expand` doesn't.", "2754920": "Indeed.\n\nAnd if we want to use cat, why not stack directly on a new dimension 1?\n\n`x = torch.stack([x, x, x], dim=1)`", "2755722": "Note that if your goal is to produce a tensor to feed into a computer vision ConvNet, you can sum up the channels in the network's first convolution kernel into one instead of triplicating the channel in the input tensor.", "2755867": "That's true. I used this trick in some past competition indeed.", "2760860": "Pretty sure it's the default behavior in the timm library, btw.", "2760872": "what default behavior?", "2761212": "In timm library, you can specify input channels. If this number is equal to one, the first conv layer is transformed so that its weights are averaged.", "2761213": "Yeah, exactly. https://timm.fast.ai/models#Case-1:-When-the-number-of-input-channels-is-1\n\nSo timm essentially implemented tricks that Kagglers used for years. Fantastic package.", "2761241": "> If this number is equal to one,\n\nWhat if this number is not one?", "2761254": "It's literally the next point in the link I sent :) \n\n(Case 2)", "2761289": "Thanks. I should have read that do entirely.", "2771209": "Why is the method using torch.cat considered to be memory intensive?", "2771335": "Pretty sure it's because you create a new tensor as apposed to creating a tensor once, and then modifying it in place", "2771570": "Because it is. When you do cat, you create a new tensor that is 3 times the size of the original tensor. And after permute you get a view of the tensor that is not contiguous, meaning that the next operation on that tensor will be slow. \n\nWith expand you don't allocate memory, and your tensor is contiguous in memory.\n\nWhy allocate useless memory when you can avoid it with simple code?\n\nI didn't share this trick because the wasted memory was critical. It is only about 600 kB for a 224x224 image. I shared it because it can be reused easily, and can make a difference when dealing with large images. In my previous comp it enabled me to train with larger batches, thereby running experiments faster, and getting a better final result.", "2771572": "The tensor data is not modified by expand. That's why I shared.", "2771635": "The distinction regarding \"contiguous in memory or not\" is a great one" }, "source": "meta" }