{
  "id": 433811,
  "title": "Deterministic Behavior Error with Huggingface's wav2vec2 Training ",
  "url": "/competitions/bengaliai-speech/discussion/433811",
  "author_name": "sqrt4kaido",
  "post_date": "2023-08-23T01:50:00.077000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi!<br>\nWhen training on Kaggle, I always turn on deterministic behavior to determine whether the result is coincidental or effective.</p>\n<p>Currently, I am trying to train wav2vec2 using a model from huggingface. However, when I turn on deterministic behavior, an error occurs. If I don't make this setting, the training runs.</p>\n<p>Does anyone have a solution to this error?</p>\n<p>Deterministic Setting: </p>\n<pre><code>torch.use_deterministic_algorithms()\n</code></pre>\n<p>Error Excerpt:</p>\n<pre><code>│ /opt/conda/lib/python3/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:   │\n│  _mask_hidden_states                                                                           │\n│                                                                                                  │\n│    │   │   │   │   min_masks=self.config.mask_time_min_masks,                                │\n│    │   │   │   )                                                                             │\n│    │   │   │   mask_time_indices = torch.tensor(mask_time_indices, device=hidden_states.dev  │\n│ ❱  │   │   │   hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.d  │\n│    │   │                                                                                     │\n│    │   │    self.config.mask_feature_prob &gt;   self.training:                           │\n│    │   │   │   \n╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\nRuntimeError: linearIndex.numel()*sliceSize*nElemBefore == expandedValue.numel() INTERNAL ASSERT FAILED at \n:, please report a bug to PyTorch. number of \nflattened indices did   number of elements  the value tensor:  vs \n</code></pre>\n<p>Environment:</p>\n<pre><code>.\n.\n</code></pre>",
  "messages": [
    {
      "id": 2403979,
      "postDate": "2023-08-23T01:50:00.077Z",
      "content": "<p>Hi!<br>\nWhen training on Kaggle, I always turn on deterministic behavior to determine whether the result is coincidental or effective.</p>\n<p>Currently, I am trying to train wav2vec2 using a model from huggingface. However, when I turn on deterministic behavior, an error occurs. If I don't make this setting, the training runs.</p>\n<p>Does anyone have a solution to this error?</p>\n<p>Deterministic Setting: </p>\n<pre><code>torch.use_deterministic_algorithms()\n</code></pre>\n<p>Error Excerpt:</p>\n<pre><code>│ /opt/conda/lib/python3/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:   │\n│  _mask_hidden_states                                                                           │\n│                                                                                                  │\n│    │   │   │   │   min_masks=self.config.mask_time_min_masks,                                │\n│    │   │   │   )                                                                             │\n│    │   │   │   mask_time_indices = torch.tensor(mask_time_indices, device=hidden_states.dev  │\n│ ❱  │   │   │   hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.d  │\n│    │   │                                                                                     │\n│    │   │    self.config.mask_feature_prob &gt;   self.training:                           │\n│    │   │   │   \n╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\nRuntimeError: linearIndex.numel()*sliceSize*nElemBefore == expandedValue.numel() INTERNAL ASSERT FAILED at \n:, please report a bug to PyTorch. number of \nflattened indices did   number of elements  the value tensor:  vs \n</code></pre>\n<p>Environment:</p>\n<pre><code>.\n.\n</code></pre>",
      "rawMarkdown": "Hi!\nWhen training on Kaggle, I always turn on deterministic behavior to determine whether the result is coincidental or effective.\n\nCurrently, I am trying to train wav2vec2 using a model from huggingface. However, when I turn on deterministic behavior, an error occurs. If I don't make this setting, the training runs.\n\nDoes anyone have a solution to this error?\n\nDeterministic Setting: \n```\ntorch.use_deterministic_algorithms(True)\n```\n\nError Excerpt:\n```python\n│ /opt/conda/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:1494   │\n│ in _mask_hidden_states                                                                           │\n│                                                                                                  │\n│   1491 │   │   │   │   min_masks=self.config.mask_time_min_masks,                                │\n│   1492 │   │   │   )                                                                             │\n│   1493 │   │   │   mask_time_indices = torch.tensor(mask_time_indices, device=hidden_states.dev  │\n│ ❱ 1494 │   │   │   hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.d  │\n│   1495 │   │                                                                                     │\n│   1496 │   │   if self.config.mask_feature_prob > 0 and self.training:                           │\n│   1497 │   │   │   # generate indices & apply SpecAugment along feature axis                     │\n╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\nRuntimeError: linearIndex.numel()*sliceSize*nElemBefore == expandedValue.numel() INTERNAL ASSERT FAILED at \n\"/usr/local/src/pytorch/aten/src/ATen/native/cuda/Indexing.cu\":389, please report a bug to PyTorch. number of \nflattened indices did not match number of elements in the value tensor: 51200 vs 1024\n```\n\nEnvironment:\n```\ntorch==2.0.0\ntransformers==4.30.2\n```"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2403979": "Hi!\nWhen training on Kaggle, I always turn on deterministic behavior to determine whether the result is coincidental or effective.\n\nCurrently, I am trying to train wav2vec2 using a model from huggingface. However, when I turn on deterministic behavior, an error occurs. If I don't make this setting, the training runs.\n\nDoes anyone have a solution to this error?\n\nDeterministic Setting: \n```\ntorch.use_deterministic_algorithms(True)\n```\n\nError Excerpt:\n```python\n│ /opt/conda/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:1494   │\n│ in _mask_hidden_states                                                                           │\n│                                                                                                  │\n│   1491 │   │   │   │   min_masks=self.config.mask_time_min_masks,                                │\n│   1492 │   │   │   )                                                                             │\n│   1493 │   │   │   mask_time_indices = torch.tensor(mask_time_indices, device=hidden_states.dev  │\n│ ❱ 1494 │   │   │   hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.d  │\n│   1495 │   │                                                                                     │\n│   1496 │   │   if self.config.mask_feature_prob > 0 and self.training:                           │\n│   1497 │   │   │   # generate indices & apply SpecAugment along feature axis                     │\n╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\nRuntimeError: linearIndex.numel()*sliceSize*nElemBefore == expandedValue.numel() INTERNAL ASSERT FAILED at \n\"/usr/local/src/pytorch/aten/src/ATen/native/cuda/Indexing.cu\":389, please report a bug to PyTorch. number of \nflattened indices did not match number of elements in the value tensor: 51200 vs 1024\n```\n\nEnvironment:\n```\ntorch==2.0.0\ntransformers==4.30.2\n```"
  }
}