{
  "id": 196699,
  "title": "Regarding training loss(avg)",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/196699",
  "author_name": "",
  "post_date": "2020-11-12T09:30:46.116155Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Team,  I have a question on training <strong>loss(avg)</strong> that I am looking at, to see model performance over a number of steps. Maybe it's too late or not the right time to ask. But I wanted to know. Please help me.</p>\n<p>Loss function which I am using <strong>neg_multi_log_likelihood</strong> (reference <a href=\"https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py#L4\" target=\"_blank\">here</a> ) returns single value. It is mean(errors). Here errors = NLL of each samples in given batch.</p>\n<p>Assume that <strong>batch_size = 64</strong> </p>\n<p>at <strong>step_1</strong> ,<br>\n// model using weights_at_step_1<br>\nloss_at_step_1 = loss_of_batch_1<br>\n…<br>\nloss_avg = loss_at_step_1</p>\n<p>at <strong>step_2</strong> ,<br>\n// model using weights_at_step_2 (Note : they are new )</p>\n<p>loss_at_step_2 = loss_of_batch_2<br>\n…<br>\nloss_avg = (loss_at_step_1 + loss_at_step_2) / 2</p>\n<p>at <strong>step_3</strong> ,<br>\n…</p>\n<p>So, at training, model using different weights to calculate loss at each step. It means we are calculating the average of all these losses with different weights.  But in testing time we are using final weights to calculate the loss for each batch. </p>\n<p>so expecting training loss(avg) at LB, is that valid?</p>",
  "messages": [
    {
      "id": "1076134",
      "postDate": "11/12/2020 09:30:46",
      "content": "<p>Team,  I have a question on training <strong>loss(avg)</strong> that I am looking at, to see model performance over a number of steps. Maybe it's too late or not the right time to ask. But I wanted to know. Please help me.</p>\n<p>Loss function which I am using <strong>neg_multi_log_likelihood</strong> (reference <a href=\"https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py#L4\" target=\"_blank\">here</a> ) returns single value. It is mean(errors). Here errors = NLL of each samples in given batch.</p>\n<p>Assume that <strong>batch_size = 64</strong> </p>\n<p>at <strong>step_1</strong> ,<br>\n// model using weights_at_step_1<br>\nloss_at_step_1 = loss_of_batch_1<br>\n…<br>\nloss_avg = loss_at_step_1</p>\n<p>at <strong>step_2</strong> ,<br>\n// model using weights_at_step_2 (Note : they are new )</p>\n<p>loss_at_step_2 = loss_of_batch_2<br>\n…<br>\nloss_avg = (loss_at_step_1 + loss_at_step_2) / 2</p>\n<p>at <strong>step_3</strong> ,<br>\n…</p>\n<p>So, at training, model using different weights to calculate loss at each step. It means we are calculating the average of all these losses with different weights.  But in testing time we are using final weights to calculate the loss for each batch. </p>\n<p>so expecting training loss(avg) at LB, is that valid?</p>",
      "rawMarkdown": "Team,  I have a question on training **loss(avg)** that I am looking at, to see model performance over a number of steps. Maybe it's too late or not the right time to ask. But I wanted to know. Please help me.\n\nLoss function which I am using **neg_multi_log_likelihood** (reference [here](https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py#L4) ) returns single value. It is mean(errors). Here errors = NLL of each samples in given batch.\n\nAssume that **batch_size = 64** \n\nat **step_1** ,\n// model using weights_at_step_1\nloss_at_step_1 = loss_of_batch_1\n...\nloss_avg = loss_at_step_1\n\nat **step_2** ,\n// model using weights_at_step_2 (Note : they are new )\n\nloss_at_step_2 = loss_of_batch_2\n...\nloss_avg = (loss_at_step_1 + loss_at_step_2) / 2\n\nat **step_3** ,\n...\n\nSo, at training, model using different weights to calculate loss at each step. It means we are calculating the average of all these losses with different weights.  But in testing time we are using final weights to calculate the loss for each batch. \n\nso expecting training loss(avg) at LB, is that valid?",
      "votes": null
    },
    {
      "id": "1078396",
      "postDate": "11/14/2020 18:22:19",
      "content": "<p>No, it is not valid at the beginning of the training, since the train score highly depends on your random initialization of the weight. But at later time, when weights are stable, this effect is not that important. Though at later time, you will expect overfit (depending on your model) so you also won't expect train loss(avg) at LB.<br>\nAlso, you can try to add the last N iteration average as well.</p>",
      "rawMarkdown": "No, it is not valid at the beginning of the training, since the train score highly depends on your random initialization of the weight. But at later time, when weights are stable, this effect is not that important. Though at later time, you will expect overfit (depending on your model) so you also won't expect train loss(avg) at LB.\nAlso, you can try to add the last N iteration average as well.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1078396,
      "author_name": "louis925",
      "author_url": "",
      "post_date": "11/14/2020 18:22:19",
      "content": "<p>No, it is not valid at the beginning of the training, since the train score highly depends on your random initialization of the weight. But at later time, when weights are stable, this effect is not that important. Though at later time, you will expect overfit (depending on your model) so you also won't expect train loss(avg) at LB.<br>\nAlso, you can try to add the last N iteration average as well.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1076134": "Team,  I have a question on training **loss(avg)** that I am looking at, to see model performance over a number of steps. Maybe it's too late or not the right time to ask. But I wanted to know. Please help me.\n\nLoss function which I am using **neg_multi_log_likelihood** (reference [here](https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py#L4) ) returns single value. It is mean(errors). Here errors = NLL of each samples in given batch.\n\nAssume that **batch_size = 64** \n\nat **step_1** ,\n// model using weights_at_step_1\nloss_at_step_1 = loss_of_batch_1\n...\nloss_avg = loss_at_step_1\n\nat **step_2** ,\n// model using weights_at_step_2 (Note : they are new )\n\nloss_at_step_2 = loss_of_batch_2\n...\nloss_avg = (loss_at_step_1 + loss_at_step_2) / 2\n\nat **step_3** ,\n...\n\nSo, at training, model using different weights to calculate loss at each step. It means we are calculating the average of all these losses with different weights.  But in testing time we are using final weights to calculate the loss for each batch. \n\nso expecting training loss(avg) at LB, is that valid?",
    "1078396": "No, it is not valid at the beginning of the training, since the train score highly depends on your random initialization of the weight. But at later time, when weights are stable, this effect is not that important. Though at later time, you will expect overfit (depending on your model) so you also won't expect train loss(avg) at LB.\nAlso, you can try to add the last N iteration average as well."
  },
  "source": "meta"
}