Here is what I use to speed up training, a combination of ideas in the forum, some from Anthropic's Claude Opus based on the training scripts….
\nnp.load(path, mmap_mode='r') like in many public scripts. model = torch.compile(model, **dict(fullgraph=False, dynamic=False, mode=\"default\")). FlatVel_A scores 1.31 - subsample it with torch WeightedRandomSampler. prefetch_factor in DataLoader, not sure if this helps, but I use it.channels_last format in torch (small difference, if any)I did not try the last one. Any other ideas to make the most of the last three weeks ?
", "messages": [ { "id": 3221005, "postDate": "2025-06-10T09:07:29.520Z", "content": "Here is what I use to speed up training, a combination of ideas in the forum, some from Anthropic's Claude Opus based on the training scripts….
\nnp.load(path, mmap_mode='r') like in many public scripts. model = torch.compile(model, **dict(fullgraph=False, dynamic=False, mode=\"default\")). FlatVel_A scores 1.31 - subsample it with torch WeightedRandomSampler. prefetch_factor in DataLoader, not sure if this helps, but I use it.channels_last format in torch (small difference, if any)I did not try the last one. Any other ideas to make the most of the last three weeks ?
", "rawMarkdown": "Here is what I use to speed up training, a combination of ideas in the forum, some from Anthropic's Claude Opus based on the training scripts....\n\n- Memory map in dataloading `np.load(path, mmap_mode='r')` like in many public scripts. \n- Mixed precision training, [examples](https://docs.pytorch.org/docs/stable/notes/amp_examples.html)\n- Compile model in torch, `model = torch.compile(model, **dict(fullgraph=False, dynamic=False, mode=\"default\"))`. \n- Subsample easy families, eg., in [Bartley's CV table](https://www.kaggle.com/competitions/waveform-inversion/discussion/582785) , `FlatVel_A` scores 1.31 - subsample it with torch [WeightedRandomSampler](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler). \n- Downsize along time axis, - a bit risky. \n- Increase prefetch `prefetch_factor` in DataLoader, not sure if this helps, but I use it.\n- `channels_last` format in torch (small difference, if any)\n- FP8 training, [linky](https://pytorch.org/blog/training-using-float8-fsdp2/), gain seems to be limited for smaller models. Only works for newer GPU's - 4090, 5090, H100, etc. \n\n\nI did not try the last one. Any other ideas to make the most of the last three weeks ?", "votes": 65 }, { "id": 3221016, "postDate": "2025-06-10T09:25:55.813Z", "content": "Fused optimizer
", "rawMarkdown": "Fused optimizer", "votes": 7, "replies": [ { "id": 3221224, "postDate": "2025-06-10T15:33:36.080Z", "content": "@harshitsheoran would you mind to share what speed up Fused optimiser gave; I remember it not being huge but might try again.
", "rawMarkdown": "@harshitsheoran would you mind to share what speed up Fused optimiser gave; I remember it not being huge but might try again. ", "replies": [ { "id": 3221242, "postDate": "2025-06-10T16:06:40.017Z", "content": "@darraghdog Just going from memory, Compile+Fused gave me 30-35%, I think fused alone was 10%+
\noptimizer = optim.AdamW(model.parameters(), lr=CFG.lr, weight_decay=CFG.wd, fused=True)
Thanks, will try fused optimiser again.
\nMaybe I had it wrong how much torch.compile sped up; or else on Hopper the difference is not so big as I got on a100. This shows a100 speed ups.
", "rawMarkdown": "Thanks, will try fused optimiser again.\n\nMaybe I had it wrong how much torch.compile sped up; or else on Hopper the difference is not so big as I got on a100. [This](https://lightning.ai/blog/training-compiled-pytorch-2.0-with-pytorch-lightning/) shows a100 speed ups. " }, { "id": 3225677, "postDate": "2025-06-16T17:55:53.353Z", "content": "The Compile+Fused gave really good boost.
", "rawMarkdown": "The Compile+Fused gave really good boost.", "votes": 1 } ] } ] }, { "id": 3224767, "postDate": "2025-06-15T12:55:11.847Z", "content": "Thanks for sharing, it's friendly to newcomers!
", "rawMarkdown": "Thanks for sharing, it's friendly to newcomers!", "votes": 3, "replies": [ { "id": 3225546, "postDate": "2025-06-16T14:38:19.823Z", "content": "yes, so good for me too. thanks for author.
", "rawMarkdown": "yes, so good for me too. thanks for author.", "votes": 1 } ] }, { "id": 3223457, "postDate": "2025-06-13T10:26:12.620Z", "content": "Thanks for your kind shareing! Would you tell me the reason why not use
\ntorch.compile(mode = \"reduce overhead\")?
I played around with a few settings and got some issues with some settings and didn’t see a huge difference in the others which worked, so settled on this. I see most people use reduce overhead, I might try again. Thanks 🙏
", "rawMarkdown": "I played around with a few settings and got some issues with some settings and didn’t see a huge difference in the others which worked, so settled on this. I see most people use reduce overhead, I might try again. Thanks 🙏 " } ] }, { "id": 3221494, "postDate": "2025-06-11T04:47:40.400Z", "content": "Thanks for your kind sharing! But when I combine torch.compile and fused optimizer, codes always raise a error like \"RuntimeError: params, grads, exp_avgs, and exp_avg_sqs must have same dtype, device, and layout\" on RTX A40. Have you met this?
", "rawMarkdown": "Thanks for your kind sharing! But when I combine torch.compile and fused optimizer, codes always raise a error like \"RuntimeError: params, grads, exp_avgs, and exp_avg_sqs must have same dtype, device, and layout\" on RTX A40. Have you met this?", "votes": 2, "replies": [ { "id": 3221606, "postDate": "2025-06-11T08:37:08.770Z", "content": "Sorry, I have not seen this. I'd recommend use your choice of free [deepseek.com, chatgpt.com, etc] or paid [clause-sonnet,openai-o3, etc] LLM apis. This is my workflow for the competition 😀 ps. I test and my gains on a100 for fused optimiser were not much.
\n\n\nAnd write your question\n@i2nfinit3y I got this error as well. Did you fix it?
", "rawMarkdown": "@i2nfinit3y I got this error as well. Did you fix it?" } ] }, { "id": 3226959, "postDate": "2025-06-18T10:00:27.457Z", "content": "Thanks for sharing, great opportunity.🌟
", "rawMarkdown": "Thanks for sharing, great opportunity.🌟" }, { "id": 3224857, "postDate": "2025-06-15T15:19:32.253Z", "content": "I get this message, since I started using the caformer code:
\n/usr//lib/python3/dist-packages/torch/autograd/graph.py:: UserWarning: Grad strides do match bucket view strides. This may indicate grad was created according gradient layout contract, param's strides changed DDP was constructed. This an , may impair performance.\n\ns grad.sizes() = [, , , ], strides() = [, , , ]\n\ns bucket_view.sizes() = [, , , ], strides() = [, , , ] (Triggered internally /pytorch/torch/csrc/distributed/c10d/reducer.cpp:)\n\ns Variable._execution_engine.run_backward( \nDoes anyone know how to fix it or what's causing it?
\nI tried checking for non-contiguous tensors, but didnt find any.
\nI feel like my training slows down alot
I got the same. I did not dig into it.
", "rawMarkdown": "I got the same. I did not dig into it.", "votes": 1 }, { "id": 3225182, "postDate": "2025-06-16T04:34:30.133Z", "content": "Hey Marius,
\nThat warning seems to be related to how DistributedDataParallel (DDP) handles gradient bucketing — especially when the tensor strides mismatch between the parameter and the bucket view. Since you're using Caformer, it could be that one of the layers or reshaping operations is altering tensor memory layout in a way that DDP doesn't expect.
A few ideas to try:
\n.permute(), .transpose(), or .view() are used without following them with .contiguous(). DDP can be sensitive to non-contiguous gradients..contiguous() where it makes sense, especially just before loss computation or backpropagation.DDP, and that no parameter reshaping is done afterward.model = model.to(memory_format=torch.channels_last) early in your pipeline — Caformer might be using channels_last format internally, which can create stride inconsistencies.This warning doesn't usually cause incorrect results, but as you noted, it can hurt performance if grad bucketing isn’t efficient. If you’re seeing slower training, it might be worth running a profiler (torch.profiler) to verify where the slowdown is coming from.
Let us know if you find a clean workaround — I'm sure others using Caformer in this comp are hitting similar issues! 🔍
", "rawMarkdown": "Hey Marius,\n\nThat warning seems to be related to how `DistributedDataParallel (DDP)` handles gradient bucketing — especially when the tensor strides mismatch between the parameter and the bucket view. Since you're using Caformer, it could be that one of the layers or reshaping operations is altering tensor memory layout in a way that DDP doesn't expect.\n\nA few ideas to try:\n\n* Check if any tensor operations like `.permute()`, `.transpose()`, or `.view()` are used without following them with `.contiguous()`. DDP can be sensitive to non-contiguous gradients.\n* You can try wrapping your model parameters or outputs with `.contiguous()` where it makes sense, especially just before loss computation or backpropagation.\n* Make sure the model is initialized **before** wrapping with `DDP`, and that no parameter reshaping is done afterward.\n* Also worth trying `model = model.to(memory_format=torch.channels_last)` early in your pipeline — Caformer might be using channels\\_last format internally, which can create stride inconsistencies.\n\nThis warning doesn't usually cause incorrect results, but as you noted, it **can** hurt performance if grad bucketing isn’t efficient. If you’re seeing slower training, it might be worth running a profiler (`torch.profiler`) to verify where the slowdown is coming from.\n\nLet us know if you find a clean workaround — I'm sure others using Caformer in this comp are hitting similar issues! 🔍" } ] }, { "id": 3223478, "postDate": "2025-06-13T11:04:37.523Z", "content": "Use torch.compile, mixed precision (torch.cuda.amp), and gradient checkpointing to boost speed and reduce memory. Optimize DataLoader with more workers, pin_memory, and persistent_workers for faster data throughput.
", "rawMarkdown": "Use torch.compile, mixed precision (torch.cuda.amp), and gradient checkpointing to boost speed and reduce memory. Optimize DataLoader with more workers, pin_memory, and persistent_workers for faster data throughput.", "replies": [ { "id": 3224848, "postDate": "2025-06-15T15:04:07.663Z", "content": "Gradient checkpointing reduces memory but it doesn't speed up training.
", "rawMarkdown": "Gradient checkpointing reduces memory but it doesn't speed up training." }, { "id": 3225053, "postDate": "2025-06-15T23:17:57.933Z", "content": "It could if your vram is really low and your original batch size is small.
", "rawMarkdown": "It could if your vram is really low and your original batch size is small." } ] }, { "id": 3223318, "postDate": "2025-06-13T06:47:37.250Z", "content": "@darraghdog thank you, except FP8 training - training speed improved 18%-20% with fuse=True on rtx 5090
", "rawMarkdown": "@darraghdog thank you, except FP8 training - training speed improved 18%-20% with fuse=True on rtx 5090" }, { "id": 3221209, "postDate": "2025-06-10T15:14:18.280Z", "content": "How do you avoid NaN issues with fp16 training ?
", "rawMarkdown": "How do you avoid NaN issues with fp16 training ?", "replies": [ { "id": 3221217, "postDate": "2025-06-10T15:26:51.867Z", "content": "I get it sometimes, but not too often; feed in fp32 inputs & labels and use from torch.amp import autocast like below.
with autocast(=, =torch.bfloat16):\n output = model(batch)\nwith autocast(device_type='cuda', dtype=torch.float32):
@darraghdog Hi, what are the pros and cons of loading all the .npy files with mmap_mode='r' in __init__ versus lazy loading with mmap_mode='r' in __getitem__?
I just tried in __getitem__ .. .would need some research to see the difference.
Thanks, I get about 13% speedup with torch.compile().
", "rawMarkdown": "Thanks, I get about 13% speedup with torch.compile().", "replies": [ { "id": 3221195, "postDate": "2025-06-10T14:42:00.633Z", "content": "Same model 12hrs to 7hrs on a100 for torch.compile() like above - will double check I got that correct. Does that sound right for others ?
@darraghdog - How much training speed improved?
", "rawMarkdown": "@darraghdog - How much training speed improved? ", "replies": [ { "id": 3221196, "postDate": "2025-06-10T14:44:52.290Z", "content": "Depends on your baseline, amp / compile / subsampling … all were the big hitters for me - in the 1.5x to 2x range. Subsampling and downsizing (a type of subsampling) depends on your choices on how aggressively to do it.
", "rawMarkdown": "Depends on your baseline, amp / compile / subsampling ... all were the big hitters for me - in the 1.5x to 2x range. Subsampling and downsizing (a type of subsampling) depends on your choices on how aggressively to do it. ", "votes": 1 } ] }, { "id": 3227462, "postDate": "2025-06-19T03:02:18.767Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 3225181, "postDate": "2025-06-16T04:33:16.623Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3225683, "postDate": "2025-06-16T18:00:08.483Z", "content": "Thanks for it.
", "rawMarkdown": "Thanks for it." }, { "id": 3222491, "postDate": "2025-06-12T08:20:14.520Z", "content": "Thanks for your kind sharing…!
", "rawMarkdown": "Thanks for your kind sharing...!" } ], "comments": [ { "id": 3221016, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-06-10T09:25:55.813000", "content": "Fused optimizer
", "votes": 7, "replies": [ { "id": 3221224, "author_name": "Darragh", "author_url": "", "post_date": "2025-06-10T15:33:36.080000", "content": "@harshitsheoran would you mind to share what speed up Fused optimiser gave; I remember it not being huge but might try again.
", "votes": 0, "replies": [ { "id": 3221242, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-06-10T16:06:40.017000", "content": "@darraghdog Just going from memory, Compile+Fused gave me 30-35%, I think fused alone was 10%+
\noptimizer = optim.AdamW(model.parameters(), lr=CFG.lr, weight_decay=CFG.wd, fused=True)
Thanks, will try fused optimiser again.
\nMaybe I had it wrong how much torch.compile sped up; or else on Hopper the difference is not so big as I got on a100. This shows a100 speed ups.
", "votes": 0, "replies": [] }, { "id": 3225677, "author_name": "Gowri Shankar Penugonda", "author_url": "", "post_date": "2025-06-16T17:55:53.353000", "content": "The Compile+Fused gave really good boost.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3224767, "author_name": "Jiaming Tao", "author_url": "", "post_date": "2025-06-15T12:55:11.847000", "content": "Thanks for sharing, it's friendly to newcomers!
", "votes": 3, "replies": [ { "id": 3225546, "author_name": "Chaiho Wang", "author_url": "", "post_date": "2025-06-16T14:38:19.823000", "content": "yes, so good for me too. thanks for author.
", "votes": 1, "replies": [] } ] }, { "id": 3223457, "author_name": "Nakapen", "author_url": "", "post_date": "2025-06-13T10:26:12.620000", "content": "Thanks for your kind shareing! Would you tell me the reason why not use
\ntorch.compile(mode = \"reduce overhead\")?
I played around with a few settings and got some issues with some settings and didn’t see a huge difference in the others which worked, so settled on this. I see most people use reduce overhead, I might try again. Thanks 🙏
", "votes": 0, "replies": [] } ] }, { "id": 3221494, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-06-11T04:47:40.400000", "content": "Thanks for your kind sharing! But when I combine torch.compile and fused optimizer, codes always raise a error like \"RuntimeError: params, grads, exp_avgs, and exp_avg_sqs must have same dtype, device, and layout\" on RTX A40. Have you met this?
", "votes": 2, "replies": [ { "id": 3221606, "author_name": "Darragh", "author_url": "", "post_date": "2025-06-11T08:37:08.770000", "content": "Sorry, I have not seen this. I'd recommend use your choice of free [deepseek.com, chatgpt.com, etc] or paid [clause-sonnet,openai-o3, etc] LLM apis. This is my workflow for the competition 😀 ps. I test and my gains on a100 for fused optimiser were not much.
\n\n\nAnd write your question\n@i2nfinit3y I got this error as well. Did you fix it?
", "votes": 0, "replies": [] } ] }, { "id": 3226959, "author_name": "Khushi Yadav", "author_url": "", "post_date": "2025-06-18T10:00:27.457000", "content": "Thanks for sharing, great opportunity.🌟
", "votes": 0, "replies": [] }, { "id": 3224857, "author_name": "Marius Heuser", "author_url": "", "post_date": "2025-06-15T15:19:32.253000", "content": "I get this message, since I started using the caformer code:
\n/usr//lib/python3/dist-packages/torch/autograd/graph.py:: UserWarning: Grad strides do match bucket view strides. This may indicate grad was created according gradient layout contract, param's strides changed DDP was constructed. This an , may impair performance.\n\ns grad.sizes() = [, , , ], strides() = [, , , ]\n\ns bucket_view.sizes() = [, , , ], strides() = [, , , ] (Triggered internally /pytorch/torch/csrc/distributed/c10d/reducer.cpp:)\n\ns Variable._execution_engine.run_backward( \nDoes anyone know how to fix it or what's causing it?
\nI tried checking for non-contiguous tensors, but didnt find any.
\nI feel like my training slows down alot
I got the same. I did not dig into it.
", "votes": 1, "replies": [] }, { "id": 3225182, "author_name": "Bertnardo Mario Uskono", "author_url": "", "post_date": "2025-06-16T04:34:30.133000", "content": "Hey Marius,
\nThat warning seems to be related to how DistributedDataParallel (DDP) handles gradient bucketing — especially when the tensor strides mismatch between the parameter and the bucket view. Since you're using Caformer, it could be that one of the layers or reshaping operations is altering tensor memory layout in a way that DDP doesn't expect.
A few ideas to try:
\n.permute(), .transpose(), or .view() are used without following them with .contiguous(). DDP can be sensitive to non-contiguous gradients..contiguous() where it makes sense, especially just before loss computation or backpropagation.DDP, and that no parameter reshaping is done afterward.model = model.to(memory_format=torch.channels_last) early in your pipeline — Caformer might be using channels_last format internally, which can create stride inconsistencies.This warning doesn't usually cause incorrect results, but as you noted, it can hurt performance if grad bucketing isn’t efficient. If you’re seeing slower training, it might be worth running a profiler (torch.profiler) to verify where the slowdown is coming from.
Let us know if you find a clean workaround — I'm sure others using Caformer in this comp are hitting similar issues! 🔍
", "votes": 0, "replies": [] } ] }, { "id": 3223478, "author_name": "Abhishek Dave", "author_url": "", "post_date": "2025-06-13T11:04:37.523000", "content": "Use torch.compile, mixed precision (torch.cuda.amp), and gradient checkpointing to boost speed and reduce memory. Optimize DataLoader with more workers, pin_memory, and persistent_workers for faster data throughput.
", "votes": 0, "replies": [ { "id": 3224848, "author_name": "CPMP", "author_url": "", "post_date": "2025-06-15T15:04:07.663000", "content": "Gradient checkpointing reduces memory but it doesn't speed up training.
", "votes": 0, "replies": [] }, { "id": 3225053, "author_name": "c-number", "author_url": "", "post_date": "2025-06-15T23:17:57.933000", "content": "It could if your vram is really low and your original batch size is small.
", "votes": 0, "replies": [] } ] }, { "id": 3223318, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2025-06-13T06:47:37.250000", "content": "@darraghdog thank you, except FP8 training - training speed improved 18%-20% with fuse=True on rtx 5090
", "votes": 0, "replies": [] }, { "id": 3221209, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2025-06-10T15:14:18.280000", "content": "How do you avoid NaN issues with fp16 training ?
", "votes": 0, "replies": [ { "id": 3221217, "author_name": "Darragh", "author_url": "", "post_date": "2025-06-10T15:26:51.867000", "content": "I get it sometimes, but not too often; feed in fp32 inputs & labels and use from torch.amp import autocast like below.
with autocast(=, =torch.bfloat16):\n output = model(batch)\nwith autocast(device_type='cuda', dtype=torch.float32):
@darraghdog Hi, what are the pros and cons of loading all the .npy files with mmap_mode='r' in __init__ versus lazy loading with mmap_mode='r' in __getitem__?
I just tried in __getitem__ .. .would need some research to see the difference.
Thanks, I get about 13% speedup with torch.compile().
", "votes": 0, "replies": [ { "id": 3221195, "author_name": "Darragh", "author_url": "", "post_date": "2025-06-10T14:42:00.633000", "content": "Same model 12hrs to 7hrs on a100 for torch.compile() like above - will double check I got that correct. Does that sound right for others ?
@darraghdog - How much training speed improved?
", "votes": 0, "replies": [ { "id": 3221196, "author_name": "Darragh", "author_url": "", "post_date": "2025-06-10T14:44:52.290000", "content": "Depends on your baseline, amp / compile / subsampling … all were the big hitters for me - in the 1.5x to 2x range. Subsampling and downsizing (a type of subsampling) depends on your choices on how aggressively to do it.
", "votes": 1, "replies": [] } ] }, { "id": 3227462, "author_name": "", "author_url": "", "post_date": "2025-06-19T03:02:18.767000", "content": "", "votes": 1, "replies": [] }, { "id": 3225181, "author_name": "", "author_url": "", "post_date": "2025-06-16T04:33:16.623000", "content": "", "votes": 0, "replies": [] }, { "id": 3225683, "author_name": "Mandip Sapkota", "author_url": "", "post_date": "2025-06-16T18:00:08.483000", "content": "Thanks for it.
", "votes": 0, "replies": [] }, { "id": 3222491, "author_name": "Sarah Arshad", "author_url": "", "post_date": "2025-06-12T08:20:14.520000", "content": "Thanks for your kind sharing…!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3221005": "Here is what I use to speed up training, a combination of ideas in the forum, some from Anthropic's Claude Opus based on the training scripts....\n\n- Memory map in dataloading `np.load(path, mmap_mode='r')` like in many public scripts. \n- Mixed precision training, [examples](https://docs.pytorch.org/docs/stable/notes/amp_examples.html)\n- Compile model in torch, `model = torch.compile(model, **dict(fullgraph=False, dynamic=False, mode=\"default\"))`. \n- Subsample easy families, eg., in [Bartley's CV table](https://www.kaggle.com/competitions/waveform-inversion/discussion/582785) , `FlatVel_A` scores 1.31 - subsample it with torch [WeightedRandomSampler](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler). \n- Downsize along time axis, - a bit risky. \n- Increase prefetch `prefetch_factor` in DataLoader, not sure if this helps, but I use it.\n- `channels_last` format in torch (small difference, if any)\n- FP8 training, [linky](https://pytorch.org/blog/training-using-float8-fsdp2/), gain seems to be limited for smaller models. Only works for newer GPU's - 4090, 5090, H100, etc. \n\n\nI did not try the last one. Any other ideas to make the most of the last three weeks ?", "3221016": "Fused optimizer", "3224767": "Thanks for sharing, it's friendly to newcomers!", "3223457": "Thanks for your kind shareing! Would you tell me the reason why not use\n`torch.compile(mode = \"reduce overhead\")`?", "3221494": "Thanks for your kind sharing! But when I combine torch.compile and fused optimizer, codes always raise a error like \"RuntimeError: params, grads, exp_avgs, and exp_avg_sqs must have same dtype, device, and layout\" on RTX A40. Have you met this?", "3226959": "Thanks for sharing, great opportunity.🌟", "3224857": "I get this message, since I started using the caformer code:\n\n```\n/usr/local/lib/python3.11/dist-packages/torch/autograd/graph.py:823: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.\n\n485.7s\t55\tgrad.sizes() = [64, 128, 1, 1], strides() = [128, 1, 128, 128]\n\n485.7s\t56\tbucket_view.sizes() = [64, 128, 1, 1], strides() = [128, 1, 1, 1] (Triggered internally at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:327.)\n\n485.7s\t57\t return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass\n```\n\nDoes anyone know how to fix it or what's causing it?\nI tried checking for non-contiguous tensors, but didnt find any.\nI feel like my training slows down alot", "3223478": "Use torch.compile, mixed precision (torch.cuda.amp), and gradient checkpointing to boost speed and reduce memory. Optimize DataLoader with more workers, pin_memory, and persistent_workers for faster data throughput.", "3223318": "@darraghdog thank you, except FP8 training - training speed improved 18%-20% with fuse=True on rtx 5090", "3221209": "How do you avoid NaN issues with fp16 training ?", "3221204": "@darraghdog Hi, what are the pros and cons of loading all the .npy files with `mmap_mode='r'` in `__init__` versus lazy loading with `mmap_mode='r'` in `__getitem__`?", "3221189": "Thanks, I get about 13% speedup with torch.compile().", "3221116": "@darraghdog - How much training speed improved? ", "3227462": "", "3225181": "", "3225683": "Thanks for it.", "3222491": "Thanks for your kind sharing...!" } }