{
  "id": 403492,
  "title": "Collecting useful resources for efficient model training. ",
  "url": "/competitions/budgeted-model-training-iccv-2023-rcv-workshop/discussion/403492",
  "author_name": "Activated Neuron",
  "post_date": "2023-04-23T12:24:41.900000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<h2>The problem statement is unique and amazing. Will add items to this list as and when I find:</h2>\n<ul>\n<li><p>1. <a href=\"https://huggingface.co/docs/transformers/perf_train_gpu_one#efficient-training-on-a-single-gpu\" target=\"_blank\">https://huggingface.co/docs/transformers/perf_train_gpu_one#efficient-training-on-a-single-gpu</a></p>\n<ul>\n<li><p>Table from the source above</p>\n<table>\n<thead>\n<tr>\n<th>Method</th>\n<th>Speed</th>\n<th>Memory</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Gradient accumulation</td>\n<td>No</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>Gradient checkpointing</td>\n<td>No</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>Mixed precision training</td>\n<td>Yes</td>\n<td>(No)</td>\n</tr>\n<tr>\n<td>Batch size</td>\n<td>Yes</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>Optimizer choice</td>\n<td>Yes</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>DataLoader</td>\n<td>Yes</td>\n<td>No</td>\n</tr>\n<tr>\n<td>DeepSpeed Zero</td>\n<td>No</td>\n<td>Yes</td>\n</tr>\n</tbody>\n</table></li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 2231550,
      "postDate": "2023-04-23T12:24:41.900Z",
      "content": "<h2>The problem statement is unique and amazing. Will add items to this list as and when I find:</h2>\n<ul>\n<li><p>1. <a href=\"https://huggingface.co/docs/transformers/perf_train_gpu_one#efficient-training-on-a-single-gpu\" target=\"_blank\">https://huggingface.co/docs/transformers/perf_train_gpu_one#efficient-training-on-a-single-gpu</a></p>\n<ul>\n<li><p>Table from the source above</p>\n<table>\n<thead>\n<tr>\n<th>Method</th>\n<th>Speed</th>\n<th>Memory</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Gradient accumulation</td>\n<td>No</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>Gradient checkpointing</td>\n<td>No</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>Mixed precision training</td>\n<td>Yes</td>\n<td>(No)</td>\n</tr>\n<tr>\n<td>Batch size</td>\n<td>Yes</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>Optimizer choice</td>\n<td>Yes</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>DataLoader</td>\n<td>Yes</td>\n<td>No</td>\n</tr>\n<tr>\n<td>DeepSpeed Zero</td>\n<td>No</td>\n<td>Yes</td>\n</tr>\n</tbody>\n</table></li></ul></li>\n</ul>",
      "rawMarkdown": "## The problem statement is unique and amazing. Will add items to this list as and when I find:\n\n- 1. https://huggingface.co/docs/transformers/perf_train_gpu_one#efficient-training-on-a-single-gpu\n       - Table from the source above\n\n       | Method\t| Speed | Memory\n       | ---- | ---- | --- |\n       |Gradient accumulation |No\t|Yes |\n       |Gradient checkpointing\t|No|\tYes|\n       |Mixed precision training\t|Yes\t|(No)|\n       |Batch size\t|Yes|\tYes|\n       |Optimizer choice\t|Yes|\tYes|\n       |DataLoader\t|Yes|\tNo|\n       |DeepSpeed Zero\t|No|\tYes|\n",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2231550": "## The problem statement is unique and amazing. Will add items to this list as and when I find:\n\n- 1. https://huggingface.co/docs/transformers/perf_train_gpu_one#efficient-training-on-a-single-gpu\n       - Table from the source above\n\n       | Method\t| Speed | Memory\n       | ---- | ---- | --- |\n       |Gradient accumulation |No\t|Yes |\n       |Gradient checkpointing\t|No|\tYes|\n       |Mixed precision training\t|Yes\t|(No)|\n       |Batch size\t|Yes|\tYes|\n       |Optimizer choice\t|Yes|\tYes|\n       |DataLoader\t|Yes|\tNo|\n       |DeepSpeed Zero\t|No|\tYes|\n"
  }
}