I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.
\nHere is my code for inference:
\n os\n numpy np\n tqdm.auto tqdm\n glob glob\n transformers AutoFeatureExtractor, pipeline\n pandas pd\n librosa\n IPython\n datasets load_metric\n tqdm.auto tqdm\n torch.utils.data Dataset, DataLoader\n torch\n gc\n wave\n scipy.io wavfile\n scipy.signal sps\n pyctcdecode\n\ntqdm.pandas()\n warnings\nwarnings.filterwarnings()\nos.environ[] = \n\nBATCH_SIZE = \nTEST_DIRECTORY = \npaths = glob(os.path.join(TEST_DIRECTORY,))\n :\n my_model_name = \n processor_name = \n transformers Wav2Vec2ProcessorWithLM\n\nprocessor = Wav2Vec2ProcessorWithLM.from_pretrained(CFG.processor_name)\n transformers Wav2Vec2ForCTC\n\nmodel = Wav2Vec2ForCTC.from_pretrained(CFG.my_model_name)\nmy_asrLM = pipeline(, model=model ,feature_extractor =processor.feature_extractor, tokenizer= processor.tokenizer,decoder=processor.decoder ,device=)\n ():\n ():\n self.paths = paths\n ():\n (self.paths)\n ():\n speech, sr = librosa.load(self.paths[idx], sr=processor.feature_extractor.sampling_rate) \n\n speech\ndataset = AudioDataset(paths)\ndevice = \n ():\n \n \n lengths = torch.tensor([ t.shape[] t batch ])\n \n batch = [ torch.Tensor(t) t batch ]\n batch = torch.nn.utils.rnn.pad_sequence(batch)\n \n mask = (batch != )\n batch, lengths, mask\ndataloader = DataLoader(dataset, batch_size=, shuffle=, num_workers=, collate_fn=collate_fn_padd)\n ():\n speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n\n my_LM_prediction = my_asrLM(speech)\n (my_LM_prediction[])\n my_LM_prediction[]\n ():\n \n results = []\n path audio_paths:\n pred = \n pred = infer(path)\n\n\n\n\n (pred)==:\n pred = \n results.append(pred)\n\n results\n\naudio_paths=[audio_path audio_path tqdm(paths)]\n\n\n\n\nsentences=[]\n idx tqdm((,(audio_paths),BATCH_SIZE)):\n batch_paths=audio_paths[idx:idx+BATCH_SIZE]\n sentence=batch_infer(batch_paths)\n (sentence)\n sentences+=sentence\n\n\n\n\n\nI used my own trained model in /kaggle/input/run-004-wav2vec2-15data-constant-lr2e-6-1ksteps and Yellowking's processor, then when running the core inference code
pred = infer(path)\nthe notebook met this problem:
\n---------------------------------------------------------------------------\nValueError Traceback (most recent )\n/tmp/ipykernel_25/. in \n idx in tqdm((,(audio_paths),BATCH_SIZE)):\n batch_paths=audio_paths[idx:idx+BATCH_SIZE]\n---> sentence=batch_infer(batch_paths)\n (sentence)\n sentences+=sentence\n\n/tmp/ipykernel_25/. in batch_infer(audio_paths, batch_size)\n path in audio_paths:\n pred = \n---> pred = infer(path)\n # :\n # pred = infer(path)\n\n/tmp/ipykernel_25/. in infer(audio_path)\n speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n \n----> my_LM_prediction = my_asrLM(speech)\n (my_LM_prediction[])\n my_LM_prediction[]\n\n//conda/lib/./site-packages/transformers/pipelines/automatic_speech_recognition. in __call__(self, inputs, **kwargs)\n `.(chunk[] chunk in output[])`.\n \n--> super().__call__(inputs, **kwargs)\n \n def _sanitize_parameters(self, **kwargs):\n\n//conda/lib/./site-packages/transformers/pipelines/base. in __call__(self, inputs, num_workers, batch_size, *, **kwargs)\n self.iterate(inputs, preprocess_params, forward_params, postprocess_params)\n :\n-> self.run_single(inputs, preprocess_params, forward_params, postprocess_params)\n \n def run_multi(self, inputs, preprocess_params, forward_params, postprocess_params):\n\n//conda/lib/./site-packages/transformers/pipelines/base. in run_single(self, inputs, preprocess_params, forward_params, postprocess_params)\n model_outputs = self.forward(model_inputs, **forward_params)\n all_outputs.(model_outputs)\n-> outputs = self.postprocess(all_outputs, **postprocess_params)\n outputs\n \n\n//conda/lib/./site-packages/transformers/pipelines/automatic_speech_recognition. in postprocess(self, model_outputs, decoder_kwargs, return_timestamps)\n decoder_kwargs None:\n decoder_kwargs = {}\n--> beams = self.decoder.decode_beams(, **decoder_kwargs)\n text = beams[][]\n return_timestamps:\n\n//conda/lib/./site-packages/pyctcdecode/decoder. in decode_beams(self, logits, beam_width, beam_prune_logp, token_min_logp, prune_history, hotwords, hotword_weight, lm_start_state)\n List of beams of OUTPUT_BEAM with various meta information\n \n--> self._check_logits_dimension(logits)\n # prepare hotword \n hotword_scorer = HotwordScorer.build_scorer(hotwords, weight=hotword_weight)\n\n//conda/lib/./site-packages/pyctcdecode/decoder. in _check_logits_dimension(self, logits)\n raise ValueError(\n \n--> % (logits.shape, (self._idx2vocab))\n )\n \n\nValueError: Input logits shape (, ), but vocabulary size . Need logits of shape: (time, vocabulary)\nI used Yellowking's Wav2Vec2 model as base model and trained it many steps. The validation WER seemed normal (approx. 0.4, 0.5) How could this happen? And how can I fix this? Thanks a lot!
", "messages": [ { "id": "2365239", "postDate": "07/30/2023 05:21:50", "content": "I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.
\nHere is my code for inference:
\n os\n numpy np\n tqdm.auto tqdm\n glob glob\n transformers AutoFeatureExtractor, pipeline\n pandas pd\n librosa\n IPython\n datasets load_metric\n tqdm.auto tqdm\n torch.utils.data Dataset, DataLoader\n torch\n gc\n wave\n scipy.io wavfile\n scipy.signal sps\n pyctcdecode\n\ntqdm.pandas()\n warnings\nwarnings.filterwarnings()\nos.environ[] = \n\nBATCH_SIZE = \nTEST_DIRECTORY = \npaths = glob(os.path.join(TEST_DIRECTORY,))\n :\n my_model_name = \n processor_name = \n transformers Wav2Vec2ProcessorWithLM\n\nprocessor = Wav2Vec2ProcessorWithLM.from_pretrained(CFG.processor_name)\n transformers Wav2Vec2ForCTC\n\nmodel = Wav2Vec2ForCTC.from_pretrained(CFG.my_model_name)\nmy_asrLM = pipeline(, model=model ,feature_extractor =processor.feature_extractor, tokenizer= processor.tokenizer,decoder=processor.decoder ,device=)\n ():\n ():\n self.paths = paths\n ():\n (self.paths)\n ():\n speech, sr = librosa.load(self.paths[idx], sr=processor.feature_extractor.sampling_rate) \n\n speech\ndataset = AudioDataset(paths)\ndevice = \n ():\n \n \n lengths = torch.tensor([ t.shape[] t batch ])\n \n batch = [ torch.Tensor(t) t batch ]\n batch = torch.nn.utils.rnn.pad_sequence(batch)\n \n mask = (batch != )\n batch, lengths, mask\ndataloader = DataLoader(dataset, batch_size=, shuffle=, num_workers=, collate_fn=collate_fn_padd)\n ():\n speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n\n my_LM_prediction = my_asrLM(speech)\n (my_LM_prediction[])\n my_LM_prediction[]\n ():\n \n results = []\n path audio_paths:\n pred = \n pred = infer(path)\n\n\n\n\n (pred)==:\n pred = \n results.append(pred)\n\n results\n\naudio_paths=[audio_path audio_path tqdm(paths)]\n\n\n\n\nsentences=[]\n idx tqdm((,(audio_paths),BATCH_SIZE)):\n batch_paths=audio_paths[idx:idx+BATCH_SIZE]\n sentence=batch_infer(batch_paths)\n (sentence)\n sentences+=sentence\n\n\n\n\n\nI used my own trained model in /kaggle/input/run-004-wav2vec2-15data-constant-lr2e-6-1ksteps and Yellowking's processor, then when running the core inference code
pred = infer(path)\nthe notebook met this problem:
\n---------------------------------------------------------------------------\nValueError Traceback (most recent )\n/tmp/ipykernel_25/. in \n idx in tqdm((,(audio_paths),BATCH_SIZE)):\n batch_paths=audio_paths[idx:idx+BATCH_SIZE]\n---> sentence=batch_infer(batch_paths)\n (sentence)\n sentences+=sentence\n\n/tmp/ipykernel_25/. in batch_infer(audio_paths, batch_size)\n path in audio_paths:\n pred = \n---> pred = infer(path)\n # :\n # pred = infer(path)\n\n/tmp/ipykernel_25/. in infer(audio_path)\n speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n \n----> my_LM_prediction = my_asrLM(speech)\n (my_LM_prediction[])\n my_LM_prediction[]\n\n//conda/lib/./site-packages/transformers/pipelines/automatic_speech_recognition. in __call__(self, inputs, **kwargs)\n `.(chunk[] chunk in output[])`.\n \n--> super().__call__(inputs, **kwargs)\n \n def _sanitize_parameters(self, **kwargs):\n\n//conda/lib/./site-packages/transformers/pipelines/base. in __call__(self, inputs, num_workers, batch_size, *, **kwargs)\n self.iterate(inputs, preprocess_params, forward_params, postprocess_params)\n :\n-> self.run_single(inputs, preprocess_params, forward_params, postprocess_params)\n \n def run_multi(self, inputs, preprocess_params, forward_params, postprocess_params):\n\n//conda/lib/./site-packages/transformers/pipelines/base. in run_single(self, inputs, preprocess_params, forward_params, postprocess_params)\n model_outputs = self.forward(model_inputs, **forward_params)\n all_outputs.(model_outputs)\n-> outputs = self.postprocess(all_outputs, **postprocess_params)\n outputs\n \n\n//conda/lib/./site-packages/transformers/pipelines/automatic_speech_recognition. in postprocess(self, model_outputs, decoder_kwargs, return_timestamps)\n decoder_kwargs None:\n decoder_kwargs = {}\n--> beams = self.decoder.decode_beams(, **decoder_kwargs)\n text = beams[][]\n return_timestamps:\n\n//conda/lib/./site-packages/pyctcdecode/decoder. in decode_beams(self, logits, beam_width, beam_prune_logp, token_min_logp, prune_history, hotwords, hotword_weight, lm_start_state)\n List of beams of OUTPUT_BEAM with various meta information\n \n--> self._check_logits_dimension(logits)\n # prepare hotword \n hotword_scorer = HotwordScorer.build_scorer(hotwords, weight=hotword_weight)\n\n//conda/lib/./site-packages/pyctcdecode/decoder. in _check_logits_dimension(self, logits)\n raise ValueError(\n \n--> % (logits.shape, (self._idx2vocab))\n )\n \n\nValueError: Input logits shape (, ), but vocabulary size . Need logits of shape: (time, vocabulary)\nI used Yellowking's Wav2Vec2 model as base model and trained it many steps. The validation WER seemed normal (approx. 0.4, 0.5) How could this happen? And how can I fix this? Thanks a lot!
", "rawMarkdown": "I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.\n\n#### Problem:\n\nHere is my code for inference:\n```py\nimport os\nimport numpy as np\nfrom tqdm.auto import tqdm\nfrom glob import glob\nfrom transformers import AutoFeatureExtractor, pipeline\nimport pandas as pd\nimport librosa\nimport IPython\nfrom datasets import load_metric\nfrom tqdm.auto import tqdm\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport gc\nimport wave\nfrom scipy.io import wavfile\nimport scipy.signal as sps\nimport pyctcdecode\n\ntqdm.pandas()\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nos.environ[\"WANDB_DISABLED\"] = \"true\"\n# CHANGE ACCORDINGLY\nBATCH_SIZE = 1\nTEST_DIRECTORY = '/kaggle/input/bengaliai-speech/test_mp3s'\npaths = glob(os.path.join(TEST_DIRECTORY,'*.mp3'))\nclass CFG:\n my_model_name = '/kaggle/input/run-004-wav2vec2-15data-constant-lr2e-6-1ksteps'\n processor_name = '/kaggle/input/yellowking-dlsprint-model/YellowKing_processor'\nfrom transformers import Wav2Vec2ProcessorWithLM\n\nprocessor = Wav2Vec2ProcessorWithLM.from_pretrained(CFG.processor_name)\nfrom transformers import Wav2Vec2ForCTC\n\nmodel = Wav2Vec2ForCTC.from_pretrained(CFG.my_model_name)\nmy_asrLM = pipeline(\"automatic-speech-recognition\", model=model ,feature_extractor =processor.feature_extractor, tokenizer= processor.tokenizer,decoder=processor.decoder ,device=0)\nclass AudioDataset(Dataset):\n def __init__(self, paths):\n self.paths = paths\n def __len__(self):\n return len(self.paths)\n def __getitem__(self,idx):\n speech, sr = librosa.load(self.paths[idx], sr=processor.feature_extractor.sampling_rate) \n# print(speech.shape)\n return speech\ndataset = AudioDataset(paths)\ndevice = 'cuda:0'\ndef collate_fn_padd(batch):\n '''\n Padds batch of variable length\n\n note: it converts things ToTensor manually here since the ToTensor transform\n assume it takes in images rather than arbitrary tensors.\n '''\n ## get sequence lengths\n lengths = torch.tensor([ t.shape[0] for t in batch ])\n ## padd\n batch = [ torch.Tensor(t) for t in batch ]\n batch = torch.nn.utils.rnn.pad_sequence(batch)\n ## compute mask\n mask = (batch != 0)\n return batch, lengths, mask\ndataloader = DataLoader(dataset, batch_size=1, shuffle=False, num_workers=4, collate_fn=collate_fn_padd)\ndef infer(audio_path):\n speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n\n my_LM_prediction = my_asrLM(speech)\n print(my_LM_prediction['text'])\n return my_LM_prediction['text']\ndef batch_infer(audio_paths, batch_size=BATCH_SIZE):\n '''\n infers on a batch of audio\n args:\n audio_paths : list of path to audio filesI wonder if the problem is from that I used LM in my inference. However, when training, I didn't use LM.
", "rawMarkdown": "I wonder if the problem is from that I used LM in my inference. However, when training, I didn't use LM.", "votes": null }, { "id": "2365901", "postDate": "07/30/2023 14:44:57", "content": "During training you don't need LM
", "rawMarkdown": "During training you don't need LM", "votes": null }, { "id": "2368605", "postDate": "08/01/2023 09:02:30", "content": "The problem here might be due to vocabulary mismatch. I see you used your own model with yellowking processor. What was the vocab size when you trained your model? Was it the same as yellowking's? I suggest creating your own processor and use it with your model. The LM can also give a rise to this problem. Make sure you normalize the text before building the LM as well. Should solve the problem making sure of these.
\nCurrently, the processor you used in this code has 112 characters but your model is giving the output in a 32 sized vocab. That's why the problem has risen.
", "rawMarkdown": "The problem here might be due to vocabulary mismatch. I see you used your own model with yellowking processor. What was the vocab size when you trained your model? Was it the same as yellowking's? I suggest creating your own processor and use it with your model. The LM can also give a rise to this problem. Make sure you normalize the text before building the LM as well. Should solve the problem making sure of these.\n\nCurrently, the processor you used in this code has 112 characters but your model is giving the output in a 32 sized vocab. That's why the problem has risen.", "votes": null }, { "id": "2369014", "postDate": "08/01/2023 13:50:51", "content": "I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.
", "rawMarkdown": "I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.", "votes": null } ], "comments": [ { "id": 2365371, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "07/30/2023 07:30:28", "content": "I wonder if the problem is from that I used LM in my inference. However, when training, I didn't use LM.
", "votes": null, "replies": [ { "id": 2365901, "author_name": "arunodhayan", "author_url": "", "post_date": "07/30/2023 14:44:57", "content": "During training you don't need LM
", "votes": null, "replies": [] } ] }, { "id": 2368605, "author_name": "mahfuzulkabirsourav", "author_url": "", "post_date": "08/01/2023 09:02:30", "content": "The problem here might be due to vocabulary mismatch. I see you used your own model with yellowking processor. What was the vocab size when you trained your model? Was it the same as yellowking's? I suggest creating your own processor and use it with your model. The LM can also give a rise to this problem. Make sure you normalize the text before building the LM as well. Should solve the problem making sure of these.
\nCurrently, the processor you used in this code has 112 characters but your model is giving the output in a 32 sized vocab. That's why the problem has risen.
", "votes": null, "replies": [ { "id": 2369014, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "08/01/2023 13:50:51", "content": "I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2365239": "I tried YellowKing's Model, and it reported this problem. However, I referred to the original code of Yellowking's, and used their original processor when training, thus the problem is now solved.\n\n#### Problem:\n\nHere is my code for inference:\n```py\nimport os\nimport numpy as np\nfrom tqdm.auto import tqdm\nfrom glob import glob\nfrom transformers import AutoFeatureExtractor, pipeline\nimport pandas as pd\nimport librosa\nimport IPython\nfrom datasets import load_metric\nfrom tqdm.auto import tqdm\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport gc\nimport wave\nfrom scipy.io import wavfile\nimport scipy.signal as sps\nimport pyctcdecode\n\ntqdm.pandas()\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nos.environ[\"WANDB_DISABLED\"] = \"true\"\n# CHANGE ACCORDINGLY\nBATCH_SIZE = 1\nTEST_DIRECTORY = '/kaggle/input/bengaliai-speech/test_mp3s'\npaths = glob(os.path.join(TEST_DIRECTORY,'*.mp3'))\nclass CFG:\n my_model_name = '/kaggle/input/run-004-wav2vec2-15data-constant-lr2e-6-1ksteps'\n processor_name = '/kaggle/input/yellowking-dlsprint-model/YellowKing_processor'\nfrom transformers import Wav2Vec2ProcessorWithLM\n\nprocessor = Wav2Vec2ProcessorWithLM.from_pretrained(CFG.processor_name)\nfrom transformers import Wav2Vec2ForCTC\n\nmodel = Wav2Vec2ForCTC.from_pretrained(CFG.my_model_name)\nmy_asrLM = pipeline(\"automatic-speech-recognition\", model=model ,feature_extractor =processor.feature_extractor, tokenizer= processor.tokenizer,decoder=processor.decoder ,device=0)\nclass AudioDataset(Dataset):\n def __init__(self, paths):\n self.paths = paths\n def __len__(self):\n return len(self.paths)\n def __getitem__(self,idx):\n speech, sr = librosa.load(self.paths[idx], sr=processor.feature_extractor.sampling_rate) \n# print(speech.shape)\n return speech\ndataset = AudioDataset(paths)\ndevice = 'cuda:0'\ndef collate_fn_padd(batch):\n '''\n Padds batch of variable length\n\n note: it converts things ToTensor manually here since the ToTensor transform\n assume it takes in images rather than arbitrary tensors.\n '''\n ## get sequence lengths\n lengths = torch.tensor([ t.shape[0] for t in batch ])\n ## padd\n batch = [ torch.Tensor(t) for t in batch ]\n batch = torch.nn.utils.rnn.pad_sequence(batch)\n ## compute mask\n mask = (batch != 0)\n return batch, lengths, mask\ndataloader = DataLoader(dataset, batch_size=1, shuffle=False, num_workers=4, collate_fn=collate_fn_padd)\ndef infer(audio_path):\n speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n\n my_LM_prediction = my_asrLM(speech)\n print(my_LM_prediction['text'])\n return my_LM_prediction['text']\ndef batch_infer(audio_paths, batch_size=BATCH_SIZE):\n '''\n infers on a batch of audio\n args:\n audio_paths : list of path to audio files