{ "id": 318854, "title": "Mixed precision with gradient accumulator", "url": "/competitions/happy-whale-and-dolphin/discussion/318854", "author_name": "", "post_date": "2022-04-14T10:08:34.356275600Z", "votes": 2, "comment_count": 2, "views": 0, "content": "

Hi everyone
\nI'm interested in how to use gradient accumulator with mixed precision
\nhere is my code snippet

\n
    scaler = GradScaler()\n    bar = tqdm(enumerate(dataloader), total=len(dataloader))\n    for step, data in bar:\n        X = data[0].to(device, dtype=torch.float)\n        y = data[1].to(device, dtype=torch.long)\n\n        optimizer.zero_grad()\n\n        with autocast():\n            pred, _ = model(X, y)\n            loss = loss_fn(pred, y)\n\n        scaler.scale(loss).backward()\n        scaler.step(optimizer)\n        scaler.update()\n\n        if scheduler is not None:\n            scheduler.step()\n
", "messages": [ { "id": "1755089", "postDate": "04/14/2022 10:08:34", "content": "

Hi everyone
\nI'm interested in how to use gradient accumulator with mixed precision
\nhere is my code snippet

\n
    scaler = GradScaler()\n    bar = tqdm(enumerate(dataloader), total=len(dataloader))\n    for step, data in bar:\n        X = data[0].to(device, dtype=torch.float)\n        y = data[1].to(device, dtype=torch.long)\n\n        optimizer.zero_grad()\n\n        with autocast():\n            pred, _ = model(X, y)\n            loss = loss_fn(pred, y)\n\n        scaler.scale(loss).backward()\n        scaler.step(optimizer)\n        scaler.update()\n\n        if scheduler is not None:\n            scheduler.step()\n
", "rawMarkdown": "Hi everyone\nI'm interested in how to use gradient accumulator with mixed precision\nhere is my code snippet\n\n```\n scaler = GradScaler()\n bar = tqdm(enumerate(dataloader), total=len(dataloader))\n for step, data in bar:\n X = data[0].to(device, dtype=torch.float)\n y = data[1].to(device, dtype=torch.long)\n \n optimizer.zero_grad()\n \n with autocast():\n pred, _ = model(X, y)\n loss = loss_fn(pred, y)\n \n scaler.scale(loss).backward()\n scaler.step(optimizer)\n scaler.update()\n \n if scheduler is not None:\n scheduler.step()\n```", "votes": null }, { "id": "1755303", "postDate": "04/14/2022 13:49:17", "content": "

Here is good example from pytorch team:

\n
scaler = GradScaler()\n\nfor epoch in epochs:\n    for i, (input, target) in enumerate(data):\n        with autocast():\n            output = model(input)\n            loss = loss_fn(output, target)\n            loss = loss / iters_to_accumulate\n\n        # Accumulates scaled gradients.\n        scaler.scale(loss).backward()\n\n        if (i + 1) % iters_to_accumulate == 0:\n            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n            scaler.step(optimizer)\n            scaler.update()\n            optimizer.zero_grad()\n
", "rawMarkdown": "[Here ](https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation) is good example from pytorch team:\n```\nscaler = GradScaler()\n\nfor epoch in epochs:\n for i, (input, target) in enumerate(data):\n with autocast():\n output = model(input)\n loss = loss_fn(output, target)\n loss = loss / iters_to_accumulate\n\n # Accumulates scaled gradients.\n scaler.scale(loss).backward()\n\n if (i + 1) % iters_to_accumulate == 0:\n # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad()\n```", "votes": null }, { "id": "1755691", "postDate": "04/14/2022 21:32:37", "content": "

thanks @kwentar 🙌

", "rawMarkdown": "thanks @kwentar 🙌", "votes": null } ], "comments": [ { "id": 1755303, "author_name": "kwentar", "author_url": "", "post_date": "04/14/2022 13:49:17", "content": "

Here is good example from pytorch team:

\n
scaler = GradScaler()\n\nfor epoch in epochs:\n    for i, (input, target) in enumerate(data):\n        with autocast():\n            output = model(input)\n            loss = loss_fn(output, target)\n            loss = loss / iters_to_accumulate\n\n        # Accumulates scaled gradients.\n        scaler.scale(loss).backward()\n\n        if (i + 1) % iters_to_accumulate == 0:\n            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n            scaler.step(optimizer)\n            scaler.update()\n            optimizer.zero_grad()\n
", "votes": null, "replies": [ { "id": 1755691, "author_name": "rafffael", "author_url": "", "post_date": "04/14/2022 21:32:37", "content": "

thanks @kwentar 🙌

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1755089": "Hi everyone\nI'm interested in how to use gradient accumulator with mixed precision\nhere is my code snippet\n\n```\n scaler = GradScaler()\n bar = tqdm(enumerate(dataloader), total=len(dataloader))\n for step, data in bar:\n X = data[0].to(device, dtype=torch.float)\n y = data[1].to(device, dtype=torch.long)\n \n optimizer.zero_grad()\n \n with autocast():\n pred, _ = model(X, y)\n loss = loss_fn(pred, y)\n \n scaler.scale(loss).backward()\n scaler.step(optimizer)\n scaler.update()\n \n if scheduler is not None:\n scheduler.step()\n```", "1755303": "[Here ](https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation) is good example from pytorch team:\n```\nscaler = GradScaler()\n\nfor epoch in epochs:\n for i, (input, target) in enumerate(data):\n with autocast():\n output = model(input)\n loss = loss_fn(output, target)\n loss = loss / iters_to_accumulate\n\n # Accumulates scaled gradients.\n scaler.scale(loss).backward()\n\n if (i + 1) % iters_to_accumulate == 0:\n # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad()\n```", "1755691": "thanks @kwentar 🙌" }, "source": "meta" }