{
  "id": 436508,
  "title": "Word Error Rate jump to 1, Training Loss goes to 0 and output empty sequence",
  "url": "/competitions/bengaliai-speech/discussion/436508",
  "author_name": "",
  "post_date": "2023-09-02T17:07:47.560654200Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>when fine tuning the wav2vec2 based on \"ai4bharat/indicwav2vec_v1_bengali\" with data from this competition, I came across the problem that wer jump to 1, training loss goes to 0 and output empty sequence.</p>\n<p>Step    Training Loss   Validation Loss Wer<br>\n1000    2.010800             inf                       0.596092<br>\n2000    1.999700             inf                       0.596210<br>\n3000    0.000000            nan                    1.000000</p>\n<p>The wer is run on a test set that contains 1000 data from this competition and not intercept with the training data.<br>\ntransformer version is 4.31.0 and pieces of fine tuning code are shown below:</p>\n<h6>#</h6>\n<p>from transformers import Wav2Vec2ForCTC</p>\n<p>model = Wav2Vec2ForCTC.from_pretrained(<br>\n    \"ai4bharat/indicwav2vec_v1_bengali\",<br>\n    ctc_loss_reduction=\"mean\", <br>\n    pad_token_id=processor.tokenizer.pad_token_id,<br>\n)</p>\n<p>model.freeze_feature_encoder()</p>\n<p>from transformers import TrainingArguments<br>\nrepo_name = \"indicwav2vec_v1_bengali_ft\"<br>\ntraining_args = TrainingArguments(<br>\n  output_dir=repo_name,<br>\n  group_by_length=True,<br>\n  gradient_checkpointing=False,<br>\n  per_device_train_batch_size=16,<br>\n  evaluation_strategy=\"steps\",<br>\n  num_train_epochs=10,<br>\n  save_steps=20000,<br>\n  eval_steps=1000,<br>\n  logging_steps=500,<br>\n  learning_rate=8e-6,<br>\n  fp16=True,<br>\n  weight_decay=0.005,<br>\n  warmup_steps=500,<br>\n  save_total_limit=7,<br>\n  dataloader_num_workers=8, <br>\n)</p>\n<p>from transformers import Trainer</p>\n<p>trainer = Trainer(<br>\n    model=model,<br>\n    data_collator=data_collator,<br>\n    args=training_args,<br>\n    compute_metrics=compute_metrics,<br>\n    train_dataset=kgbs_25pct[\"train\"],<br>\n    eval_dataset=kgbs_25pct[\"test\"],<br>\n    tokenizer=processor.feature_extractor,<br>\n)</p>\n<h6>#</h6>\n<p>Does anyone have any clue?</p>",
  "messages": [
    {
      "id": "2420532",
      "postDate": "09/02/2023 17:07:47",
      "content": "<p>when fine tuning the wav2vec2 based on \"ai4bharat/indicwav2vec_v1_bengali\" with data from this competition, I came across the problem that wer jump to 1, training loss goes to 0 and output empty sequence.</p>\n<p>Step    Training Loss   Validation Loss Wer<br>\n1000    2.010800             inf                       0.596092<br>\n2000    1.999700             inf                       0.596210<br>\n3000    0.000000            nan                    1.000000</p>\n<p>The wer is run on a test set that contains 1000 data from this competition and not intercept with the training data.<br>\ntransformer version is 4.31.0 and pieces of fine tuning code are shown below:</p>\n<h6>#</h6>\n<p>from transformers import Wav2Vec2ForCTC</p>\n<p>model = Wav2Vec2ForCTC.from_pretrained(<br>\n    \"ai4bharat/indicwav2vec_v1_bengali\",<br>\n    ctc_loss_reduction=\"mean\", <br>\n    pad_token_id=processor.tokenizer.pad_token_id,<br>\n)</p>\n<p>model.freeze_feature_encoder()</p>\n<p>from transformers import TrainingArguments<br>\nrepo_name = \"indicwav2vec_v1_bengali_ft\"<br>\ntraining_args = TrainingArguments(<br>\n  output_dir=repo_name,<br>\n  group_by_length=True,<br>\n  gradient_checkpointing=False,<br>\n  per_device_train_batch_size=16,<br>\n  evaluation_strategy=\"steps\",<br>\n  num_train_epochs=10,<br>\n  save_steps=20000,<br>\n  eval_steps=1000,<br>\n  logging_steps=500,<br>\n  learning_rate=8e-6,<br>\n  fp16=True,<br>\n  weight_decay=0.005,<br>\n  warmup_steps=500,<br>\n  save_total_limit=7,<br>\n  dataloader_num_workers=8, <br>\n)</p>\n<p>from transformers import Trainer</p>\n<p>trainer = Trainer(<br>\n    model=model,<br>\n    data_collator=data_collator,<br>\n    args=training_args,<br>\n    compute_metrics=compute_metrics,<br>\n    train_dataset=kgbs_25pct[\"train\"],<br>\n    eval_dataset=kgbs_25pct[\"test\"],<br>\n    tokenizer=processor.feature_extractor,<br>\n)</p>\n<h6>#</h6>\n<p>Does anyone have any clue?</p>",
      "rawMarkdown": "when fine tuning the wav2vec2 based on \"ai4bharat/indicwav2vec_v1_bengali\" with data from this competition, I came across the problem that wer jump to 1, training loss goes to 0 and output empty sequence.\n\nStep\tTraining Loss\tValidation Loss\tWer\n1000\t2.010800\t         inf\t                   0.596092\n2000\t1.999700\t         inf\t                   0.596210\n3000\t0.000000\t        nan\t                   1.000000\n\nThe wer is run on a test set that contains 1000 data from this competition and not intercept with the training data.\ntransformer version is 4.31.0 and pieces of fine tuning code are shown below:\n\n############################################################\nfrom transformers import Wav2Vec2ForCTC\n\nmodel = Wav2Vec2ForCTC.from_pretrained(\n    \"ai4bharat/indicwav2vec_v1_bengali\",\n    ctc_loss_reduction=\"mean\", \n    pad_token_id=processor.tokenizer.pad_token_id,\n)\n\nmodel.freeze_feature_encoder()\n\nfrom transformers import TrainingArguments\nrepo_name = \"indicwav2vec_v1_bengali_ft\"\ntraining_args = TrainingArguments(\n  output_dir=repo_name,\n  group_by_length=True,\n  gradient_checkpointing=False,\n  per_device_train_batch_size=16,\n  evaluation_strategy=\"steps\",\n  num_train_epochs=10,\n  save_steps=20000,\n  eval_steps=1000,\n  logging_steps=500,\n  learning_rate=8e-6,\n  fp16=True,\n  weight_decay=0.005,\n  warmup_steps=500,\n  save_total_limit=7,\n  dataloader_num_workers=8, \n)\n\nfrom transformers import Trainer\n\ntrainer = Trainer(\n    model=model,\n    data_collator=data_collator,\n    args=training_args,\n    compute_metrics=compute_metrics,\n    train_dataset=kgbs_25pct[\"train\"],\n    eval_dataset=kgbs_25pct[\"test\"],\n    tokenizer=processor.feature_extractor,\n)\n\n############################################################\nDoes anyone have any clue?",
      "votes": null
    },
    {
      "id": "2421973",
      "postDate": "09/03/2023 16:16:24",
      "content": "<p>Add this params when loading model:<br>\nctc_zero_infinity=True,<br>\ndiversity_loss_weight=100</p>",
      "rawMarkdown": "Add this params when loading model:\nctc_zero_infinity=True,\ndiversity_loss_weight=100",
      "votes": null
    },
    {
      "id": "2422032",
      "postDate": "09/03/2023 17:05:51",
      "content": "<p>YOU ARE SO HELPFUL! I'm new to finetune wav2vec2 models and unfamilar to many parameters😭. ctc_zero_infinity=True recovers the Validation Loss and diversity_loss_weight=100 makes testing stuck. Apply diversity_loss_weight=1 generates the following log:<br>\n [ 1276/154850 07:46 &lt; 15:37:57, 2.73 it/s, Epoch 0.08/10]<br>\nStep    Training Loss   Validation Loss Wer<br>\n100    1.819000    1.718054    0.511892<br>\n200    1.688600    1.413108    0.515915<br>\n300    1.510200    1.242680    0.518400<br>\n400    1.423200    1.198052    0.514259<br>\n500    1.430500    1.180843    0.511892<br>\n600    1.294300    1.151243    0.506804<br>\n700    1.342300    1.143062    0.504556<br>\n800    1.281900    1.121706    0.499349<br>\n900    1.308600    1.095851    0.494853<br>\n1000    1.232200    1.083341    0.491066<br>\n1100    1.260100    1.071296    0.491776<br>\n1200    1.300100    1.074520    0.488463</p>\n<p>THANK YOU AGAIN! </p>",
      "rawMarkdown": "YOU ARE SO HELPFUL! I'm new to finetune wav2vec2 models and unfamilar to many parameters😭. ctc_zero_infinity=True recovers the Validation Loss and diversity_loss_weight=100 makes testing stuck. Apply diversity_loss_weight=1 generates the following log:\n [ 1276/154850 07:46 < 15:37:57, 2.73 it/s, Epoch 0.08/10]\nStep\tTraining Loss\tValidation Loss\tWer\n100\t1.819000\t1.718054\t0.511892\n200\t1.688600\t1.413108\t0.515915\n300\t1.510200\t1.242680\t0.518400\n400\t1.423200\t1.198052\t0.514259\n500\t1.430500\t1.180843\t0.511892\n600\t1.294300\t1.151243\t0.506804\n700\t1.342300\t1.143062\t0.504556\n800\t1.281900\t1.121706\t0.499349\n900\t1.308600\t1.095851\t0.494853\n1000\t1.232200\t1.083341\t0.491066\n1100\t1.260100\t1.071296\t0.491776\n1200\t1.300100\t1.074520\t0.488463\n\nTHANK YOU AGAIN!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2421973,
      "author_name": "hubert101",
      "author_url": "",
      "post_date": "09/03/2023 16:16:24",
      "content": "<p>Add this params when loading model:<br>\nctc_zero_infinity=True,<br>\ndiversity_loss_weight=100</p>",
      "votes": null,
      "replies": [
        {
          "id": 2422032,
          "author_name": "lvyalong9999",
          "author_url": "",
          "post_date": "09/03/2023 17:05:51",
          "content": "<p>YOU ARE SO HELPFUL! I'm new to finetune wav2vec2 models and unfamilar to many parameters😭. ctc_zero_infinity=True recovers the Validation Loss and diversity_loss_weight=100 makes testing stuck. Apply diversity_loss_weight=1 generates the following log:<br>\n [ 1276/154850 07:46 &lt; 15:37:57, 2.73 it/s, Epoch 0.08/10]<br>\nStep    Training Loss   Validation Loss Wer<br>\n100    1.819000    1.718054    0.511892<br>\n200    1.688600    1.413108    0.515915<br>\n300    1.510200    1.242680    0.518400<br>\n400    1.423200    1.198052    0.514259<br>\n500    1.430500    1.180843    0.511892<br>\n600    1.294300    1.151243    0.506804<br>\n700    1.342300    1.143062    0.504556<br>\n800    1.281900    1.121706    0.499349<br>\n900    1.308600    1.095851    0.494853<br>\n1000    1.232200    1.083341    0.491066<br>\n1100    1.260100    1.071296    0.491776<br>\n1200    1.300100    1.074520    0.488463</p>\n<p>THANK YOU AGAIN! </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2420532": "when fine tuning the wav2vec2 based on \"ai4bharat/indicwav2vec_v1_bengali\" with data from this competition, I came across the problem that wer jump to 1, training loss goes to 0 and output empty sequence.\n\nStep\tTraining Loss\tValidation Loss\tWer\n1000\t2.010800\t         inf\t                   0.596092\n2000\t1.999700\t         inf\t                   0.596210\n3000\t0.000000\t        nan\t                   1.000000\n\nThe wer is run on a test set that contains 1000 data from this competition and not intercept with the training data.\ntransformer version is 4.31.0 and pieces of fine tuning code are shown below:\n\n############################################################\nfrom transformers import Wav2Vec2ForCTC\n\nmodel = Wav2Vec2ForCTC.from_pretrained(\n    \"ai4bharat/indicwav2vec_v1_bengali\",\n    ctc_loss_reduction=\"mean\", \n    pad_token_id=processor.tokenizer.pad_token_id,\n)\n\nmodel.freeze_feature_encoder()\n\nfrom transformers import TrainingArguments\nrepo_name = \"indicwav2vec_v1_bengali_ft\"\ntraining_args = TrainingArguments(\n  output_dir=repo_name,\n  group_by_length=True,\n  gradient_checkpointing=False,\n  per_device_train_batch_size=16,\n  evaluation_strategy=\"steps\",\n  num_train_epochs=10,\n  save_steps=20000,\n  eval_steps=1000,\n  logging_steps=500,\n  learning_rate=8e-6,\n  fp16=True,\n  weight_decay=0.005,\n  warmup_steps=500,\n  save_total_limit=7,\n  dataloader_num_workers=8, \n)\n\nfrom transformers import Trainer\n\ntrainer = Trainer(\n    model=model,\n    data_collator=data_collator,\n    args=training_args,\n    compute_metrics=compute_metrics,\n    train_dataset=kgbs_25pct[\"train\"],\n    eval_dataset=kgbs_25pct[\"test\"],\n    tokenizer=processor.feature_extractor,\n)\n\n############################################################\nDoes anyone have any clue?",
    "2421973": "Add this params when loading model:\nctc_zero_infinity=True,\ndiversity_loss_weight=100",
    "2422032": "YOU ARE SO HELPFUL! I'm new to finetune wav2vec2 models and unfamilar to many parameters😭. ctc_zero_infinity=True recovers the Validation Loss and diversity_loss_weight=100 makes testing stuck. Apply diversity_loss_weight=1 generates the following log:\n [ 1276/154850 07:46 < 15:37:57, 2.73 it/s, Epoch 0.08/10]\nStep\tTraining Loss\tValidation Loss\tWer\n100\t1.819000\t1.718054\t0.511892\n200\t1.688600\t1.413108\t0.515915\n300\t1.510200\t1.242680\t0.518400\n400\t1.423200\t1.198052\t0.514259\n500\t1.430500\t1.180843\t0.511892\n600\t1.294300\t1.151243\t0.506804\n700\t1.342300\t1.143062\t0.504556\n800\t1.281900\t1.121706\t0.499349\n900\t1.308600\t1.095851\t0.494853\n1000\t1.232200\t1.083341\t0.491066\n1100\t1.260100\t1.071296\t0.491776\n1200\t1.300100\t1.074520\t0.488463\n\nTHANK YOU AGAIN!"
  },
  "source": "meta"
}