Step 0 - Introduction
\n##WELCOME TO SPEECH RECOGNITION CONTEST
\n\n**AUTHOR - SUJAY KAPADNIS**\n\n**DATE - 20 July 2023 - It's a nice day and sun's out - Let's go**\n\n## About Competition\n- WE are asked to recognize Bengali speech from out-of-distribution audio recordings\n- Dataset has **1,200 hours** of data from ~24,000 people from India and Bangladesh.\n- The test set contains samples from **17 different domains** that are not present in training.\n- Size of the data - **`26GB`**\n- this is a Code Competition, in which the **actual test set is hidden**. In this public version, we give some sample data in the correct format to help you author your solutions. The full test set contains about **20 hours of speech** in almost **8000 MP3** audio files. All of the files in the test set are **encoded at a sample rate of 32k, a bit rate of 48k, in one channel**.\n\n### Out-of-Source Distribution\n- Out-of means apart from \n- Source referes to the training data\n- So it simply means that the data our model will be tested on, is little different. Which is challenging because in such cases the model tends to perform poor.\n- Out-of-distribution recordings can include variations in accent, background noise, recording conditions, speaking styles, or dialects that were not sufficiently represented in the training data.\n\n### Files\n**`train/`** - The training set, comprising several thousand recordings in **MP3 format**.\n\n**`test/`** - The test set, comprising spontaneous speech recordings from **18 domains**, **17 of which are out-of-distribution with respect to the training set**. There may be domains in the private test set that are **not in the public test set**.\n\n**`examples/`** An **example recording** for **each test set domain**. You may find these example recordings **helpful for creating models robust to domain variation**. These are representative recordings and **none of them are present in the test set**.\n\n**`train.csv`** - Sentence labels for the training set.\n - `id` - A unique identifier for this instance. **Corresponds to the file {id}.mp3 in train/**.\n - `sentence` - A **plain-text transcription of the recording**. Your goal is to **predict these sentences for each recording in the test set**.\n - `split` - Whether **train or valid**. The annotations in the **valid** split have been **manually reviewed and corrected**, while the annotations in the **train** split have only been **algorithmically cleaned**. The **valid** samples will generally have **higher quality annotations** than the train samples, but are otherwise **drawn from the same distribution**.\n \n - `sample_submission.csv` - A sample submission file in the correct format. See the Evaluation page for more details.\n \n### Evaluation metric\n#### WER - Word Error Rate\n- The WER is derived from the Levenshtein distance, working at the word level instead of the phoneme level\n##### WER = (S + I + D) / N\nWhere\n- `S` is the number of word substitutions (words incorrectly recognized by the system).\n- `I` is the number of word insertions (words present in the system's output but not in the reference).\n- `D` is the number of word deletions (words present in the reference but not recognized by the system).\n- `N` is the total number of words in the reference transcription.\n\n#### Example - \n\n- **Input** - Say my name.\n- **Output** - Say your name.\n - S - There is one substitution (your instead of my).\n - I - There are no insertions.\n - D - There are no deletions.\n - N - The reference transcription has 3 words.\n - WER = (S + I + D) / N\n - WER = (1 + 0 + 0) / 3\n - WER = 1 / 3 ≈ 0.333 (approximately 33.3%)\n**Lesser the WER - the better**\n\n\n### Sample Submission\n\n>\nid,sentence\n\n0f3dac00655e,এছাড়াও নিউজিল্যান্ড এ ক্রিকেট দলের হয়েও খেলছেন তিনি।\n\na9395e01ad21,এছাড়াও নিউজিল্যান্ড এ ক্রিকেট দলের হয়েও খেলছেন তিনি।\n\nbf36ea8b718d,এছাড়াও নিউজিল্যান্ড এ ক্রিকেট দলের হয়েও খেলছেন তিনি।\n...\n\n\n### This is work in Progress, if you find this helpful kindly upvote","metadata":{"papermill":{"duration":0.010788,"end_time":"2023-05-11T09:31:29.552885","exception":false,"start_time":"2023-05-11T09:31:29.542097","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#Step 1 - LOAD YOUR DEPENDENCIES
\n\nLinks I encourage you to open in new tab\n1. https://www.kaggle.com/code/samuelcortinhas/catalogue-of-my-kaggle-notebooks(see NLP Series Notebook)","metadata":{"papermill":{"duration":0.009506,"end_time":"2023-05-11T09:31:29.572437","exception":false,"start_time":"2023-05-11T09:31:29.562931","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport librosa\nimport numpy as np\nimport IPython.display as ipd\nimport torchaudio\nimport librosa.display\nimport matplotlib.pyplot as plt\nimport random\nimport string\nimport nltk\nfrom nltk.tokenize import word_tokenize\nfrom nltk.corpus import stopwords\nfrom nltk.stem import PorterStemmer\nfrom collections import Counter\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.decomposition import LatentDirichletAllocation\nfrom gensim import models","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:21:03.697680Z","iopub.execute_input":"2023-08-20T07:21:03.698072Z","iopub.status.idle":"2023-08-20T07:21:03.705170Z","shell.execute_reply.started":"2023-08-20T07:21:03.698024Z","shell.execute_reply":"2023-08-20T07:21:03.704124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Step 2 - EDA
\n\n##Step 2.1 - Load the Data
","metadata":{}},{"cell_type":"code","source":"# Define the paths to the data directories\nBASE_DIR = '/kaggle/input/bengaliai-speech'\ntrain_data_dir = f\"{BASE_DIR}/train_mp3s/\" \ntest_data_dir = f\"{BASE_DIR}/test_mp3s/\" \ntrain_csv_path = f\"{BASE_DIR}/train.csv\" \ndomains = f\"{BASE_DIR}/examples/\" \n\npath_template = \"/kaggle/input/bengaliai-speech/train_mp3s/{}.mp3\"\n\n# Load the train.csv file using pandas\ntrain_df = pd.read_csv(train_csv_path)\n\n# Preview the first few rows of the DataFrame\ndisplay(train_df.head())\n\nDOMAINS = os.listdir(f'{BASE_DIR}/examples')","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:21:03.706827Z","iopub.execute_input":"2023-08-20T07:21:03.707686Z","iopub.status.idle":"2023-08-20T07:21:06.881346Z","shell.execute_reply.started":"2023-08-20T07:21:03.707654Z","shell.execute_reply":"2023-08-20T07:21:06.880404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # domains\n# DOMAINS = !ls /kaggle/input/bengaliai-speech/examples/\n# DOMAINS","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:21:06.883033Z","iopub.execute_input":"2023-08-20T07:21:06.883350Z","iopub.status.idle":"2023-08-20T07:21:06.888007Z","shell.execute_reply.started":"2023-08-20T07:21:06.883325Z","shell.execute_reply":"2023-08-20T07:21:06.886946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Step 2.2 - Summary
","metadata":{}},{"cell_type":"code","source":"# Load audio files and corresponding transcriptions\naudio_data = [] # List to store audio data\ntranscriptions = [] # List to store corresponding transcriptions\n\nfor idx, row in train_df.head(5).iterrows():\n audio_file_path = path_template.format(row['id'])\n\n # Load the audio file using librosa\n audio, sr = librosa.load(audio_file_path, sr=None)\n\n # Append audio data and transcription to lists\n audio_data.append(audio)\n transcriptions.append(row['sentence'])\n \naudio_data = np.array(audio_data,dtype = 'object')\ntranscriptions = np.array(transcriptions,dtype = 'object')\n\n# Check the shapes of the loaded data\nprint(\"Audio data shape:\", audio_data.shape)\nprint(\"Transcriptions shape:\", transcriptions.shape)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:21:06.889636Z","iopub.execute_input":"2023-08-20T07:21:06.889971Z","iopub.status.idle":"2023-08-20T07:21:07.015069Z","shell.execute_reply.started":"2023-08-20T07:21:06.889939Z","shell.execute_reply":"2023-08-20T07:21:07.014095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the total number of audio files in the training and test directories\ntrain_audio_files = os.listdir(train_data_dir)\ntest_audio_files = os.listdir(test_data_dir)\n\n# Get the total duration of audio data in the training set (in seconds)\ntrain_total_duration = 0\nSAMPLES_TAKEN = 10000\nfor idx, row in train_df.head(SAMPLES_TAKEN).iterrows():\n audio_file_path = path_template.format(row['id'])\n# audio_info = torchaudio.info(audio_file_path)\n# duration = audio_info.num_frames / audio_info.sample_rate\n# train_total_duration += duration\n\ntest_duration = 0\nfor test_file in test_audio_files:\n audio_file_path = os.path.join(test_data_dir, str(test_file))\n# audio_info = torchaudio.info(audio_file_path)\n# duration = audio_info.num_frames / audio_info.sample_rate\n# test_duration += duration\n \n# Get the total number of samples in the training data\ntotal_samples = train_df.shape[0]\n\n# Print the data summary\nprint(\"Data Summary:\")\nprint(f\"Total number of audio files in the training directory: {len(train_audio_files)}\")\nprint(f\"Total number of audio files in the test directory: {len(test_audio_files)}\")\n# print(f\"Total duration of audio data in the training set (in seconds): {train_total_duration:.2f}\")\n# print(f\"Average duration of audio file in training set (in seconds): {train_total_duration/SAMPLES_TAKEN:.2f}\")\n# print(f\"Total duration of audio data in the test set (in seconds): {test_duration:.2f}\")\nprint(f\"Total number of samples in the training data: {total_samples}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:23:56.842511Z","iopub.execute_input":"2023-08-20T07:23:56.842862Z","iopub.status.idle":"2023-08-20T07:23:58.955109Z","shell.execute_reply.started":"2023-08-20T07:23:56.842833Z","shell.execute_reply":"2023-08-20T07:23:58.954119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Step 2.3 - Listen
","metadata":{}},{"cell_type":"code","source":"# !pip install mutagen --quiet\n# import mutagen\n# from mutagen.mp3 import MP3\n# train_df['duration'] = train_df['id'].apply(lambda x: MP3(os.path.join(train_data_dir,(x+'.mp3'))).info.length)\n# train_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:21:43.025357Z","iopub.status.idle":"2023-08-20T07:21:43.025687Z","shell.execute_reply.started":"2023-08-20T07:21:43.025521Z","shell.execute_reply":"2023-08-20T07:21:43.025537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Hello Hello , mic check\n# Choose some random indices for checking\nrandom_indices = [0, 10, 20, 30, 40]\n\nfor idx in random_indices:\n row = train_df.iloc[idx]\n audio_file_path = path_template.format(row['id'])\n# audio = MP3(audio_file_path)\n# print(audio.info.length)\n\n # Load the audio file using librosa\n audio, sr = librosa.load(audio_file_path, sr=None)\n\n # Print the transcription and play the audio\n print(\"Transcription:\", row['sentence'])\n ipd.display(ipd.Audio(audio, rate=sr))","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:24:04.508705Z","iopub.execute_input":"2023-08-20T07:24:04.509083Z","iopub.status.idle":"2023-08-20T07:24:04.599503Z","shell.execute_reply.started":"2023-08-20T07:24:04.509033Z","shell.execute_reply":"2023-08-20T07:24:04.598707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for test_file in test_audio_files:\n audio_file_path = os.path.join(test_data_dir, str(test_file))\n # Load the audio file using librosa\n audio, sr = librosa.load(audio_file_path, sr=None)\n ipd.display(ipd.Audio(audio, rate=sr))","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:24:11.921903Z","iopub.execute_input":"2023-08-20T07:24:11.922291Z","iopub.status.idle":"2023-08-20T07:24:11.986527Z","shell.execute_reply.started":"2023-08-20T07:24:11.922260Z","shell.execute_reply":"2023-08-20T07:24:11.985517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Step 2.4 - Visualization
","metadata":{}},{"cell_type":"code","source":"domain_counts = train_df['split'].value_counts()\n\n# Plot the data distribution\nplt.figure(figsize=(10, 6))\ndomain_counts.plot(kind='bar', color='skyblue')\nplt.title(\"Data Distribution Across Domains\")\nplt.xlabel(\"Domain\")\nplt.ylabel(\"Number of Recordings\")\nplt.xticks(rotation=45, ha='right')\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:24:13.018782Z","iopub.execute_input":"2023-08-20T07:24:13.019476Z","iopub.status.idle":"2023-08-20T07:24:13.537595Z","shell.execute_reply.started":"2023-08-20T07:24:13.019442Z","shell.execute_reply":"2023-08-20T07:24:13.536642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for idx in random_indices:\n row = train_df.iloc[idx]\n audio_file_path = os.path.join(train_data_dir, f\"{row['id']}.mp3\")\n\n # Load the audio file using librosa\n audio, sr = librosa.load(audio_file_path, sr=None)\n\n # Plot the waveform\n plt.figure(figsize=(10, 4))\n librosa.display.waveshow(audio, sr=sr)\n plt.title(f\"Waveform - Audio File ID: {row['id']}\")\n plt.xlabel(\"Time (s)\")\n plt.ylabel(\"Amplitude\")\n plt.tight_layout()\n plt.show()\n\n # Plot the log Mel spectrogram\n plt.figure(figsize=(10, 4))\n mel_spec = librosa.feature.melspectrogram(y=audio, sr=sr)\n mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)\n librosa.display.specshow(mel_spec_db, sr=sr, x_axis='time', y_axis='mel')\n plt.colorbar(format='%+2.0f dB')\n plt.title(f\"Log Mel Spectrogram - Audio File ID: {row['id']}\")\n plt.xlabel(\"Time (s)\")\n plt.ylabel(\"Mel Frequency\")\n plt.tight_layout()\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:24:13.919539Z","iopub.execute_input":"2023-08-20T07:24:13.920294Z","iopub.status.idle":"2023-08-20T07:24:20.839398Z","shell.execute_reply.started":"2023-08-20T07:24:13.920256Z","shell.execute_reply":"2023-08-20T07:24:20.838446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Learn more about [Mel Spectograms](https://youtu.be/9GHCiiDLHQ4)","metadata":{}},{"cell_type":"markdown","source":"\n##Step 2.5 - Sentence Analysis
\n","metadata":{}},{"cell_type":"markdown","source":"\n###Step 2.5.1 - Text Preprocessing
","metadata":{}},{"cell_type":"code","source":"# Select the first 5 transcriptions\ntranscriptions = train_df['sentence'][:5].tolist()\n\n# Convert transcriptions to lowercase\ntranscriptions_lower = [transcription.lower() for transcription in transcriptions]\n\n# Remove punctuation\ntranslator = str.maketrans(\"\", \"\", string.punctuation)\ntranscriptions_no_punct = [transcription.translate(translator) for transcription in transcriptions_lower]\n\n# Tokenization\nnltk.download('punkt') \ntranscriptions_tokens = [word_tokenize(transcription) for transcription in transcriptions_no_punct]\n\n\nnltk.download('stopwords') \nstop_words = set(stopwords.words('bengali'))\ntranscriptions_no_stopwords = [\n [word for word in tokens if word not in stop_words]\n for tokens in transcriptions_tokens\n]\n\nnltk.download('wordnet') \nstemmer = PorterStemmer()\ntranscriptions_stemmed = [\n [stemmer.stem(word) for word in tokens]\n for tokens in transcriptions_no_stopwords\n]\n\n# Print the preprocessed transcriptions\nfor i, transcription in enumerate(transcriptions_stemmed):\n print(f\"Preprocessed transcription {i+1}: {' '.join(transcription)}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:24:20.841111Z","iopub.execute_input":"2023-08-20T07:24:20.842010Z","iopub.status.idle":"2023-08-20T07:25:20.990030Z","shell.execute_reply.started":"2023-08-20T07:24:20.841975Z","shell.execute_reply":"2023-08-20T07:25:20.988893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n###Step 2.5.2 - Tokenization
","metadata":{}},{"cell_type":"code","source":"# Select the first 100 transcriptions \ntranscriptions = train_df['sentence'].tolist()\n\n# Tokenization\nnltk.download('punkt') # Download the Punkt tokenizer\ntranscriptions_tokens = [word_tokenize(transcription) for transcription in transcriptions]\n\n# Print the tokenized transcriptions\nfor i, transcription_tokens in enumerate(transcriptions_tokens):\n if i%100000 == 0:\n print(f\"Tokenized transcription {i+1}: {transcription_tokens}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:25:20.991743Z","iopub.execute_input":"2023-08-20T07:25:20.992128Z","iopub.status.idle":"2023-08-20T07:28:23.400861Z","shell.execute_reply.started":"2023-08-20T07:25:20.992095Z","shell.execute_reply":"2023-08-20T07:28:23.399724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n###Step 2.5.3 - Vocab Analysis
","metadata":{}},{"cell_type":"code","source":"# Build vocabulary\nvocabulary = set()\nfor transcription_tokens in transcriptions_tokens:\n vocabulary.update(transcription_tokens)\n\n# print(\"Vocabulary:\")\n# print(vocabulary)\nprint(f\"Vocabulary Size: {len(vocabulary)}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:28:23.403618Z","iopub.execute_input":"2023-08-20T07:28:23.404063Z","iopub.status.idle":"2023-08-20T07:28:24.645543Z","shell.execute_reply.started":"2023-08-20T07:28:23.404013Z","shell.execute_reply":"2023-08-20T07:28:24.644483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n###Step 2.5.4 - Sentence Length analysis
","metadata":{}},{"cell_type":"code","source":"lens = train_df.sentence.apply(lambda x: len(x))\nplt.hist(lens)","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:28:24.647186Z","iopub.execute_input":"2023-08-20T07:28:24.647625Z","iopub.status.idle":"2023-08-20T07:28:25.611820Z","shell.execute_reply.started":"2023-08-20T07:28:24.647590Z","shell.execute_reply":"2023-08-20T07:28:25.610911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute descriptive statistics\nsentence_lengths = [len(tokens) for tokens in transcriptions_tokens]\nmin_length = min(sentence_lengths)\nmax_length = max(sentence_lengths)\nmean_length = sum(sentence_lengths) / len(sentence_lengths)\nmedian_length = sorted(sentence_lengths)[len(sentence_lengths) // 2]\n\n# Plot the distribution of sentence lengths\nplt.figure(figsize=(10, 6))\nplt.hist(sentence_lengths, bins=50, color='skyblue', edgecolor='black')\nplt.axvline(mean_length, color='red', linestyle='dashed', linewidth=2, label='Mean')\nplt.axvline(median_length, color='green', linestyle='dashed', linewidth=2, label='Median')\nplt.xlabel('Sentence Length')\nplt.ylabel('Frequency')\nplt.title('Distribution of Sentence Lengths')\nplt.legend()\nplt.show()\n\n# Print the descriptive statistics\nprint(\"Descriptive Statistics:\")\nprint(f\"Minimum Sentence Length: {min_length}\")\nprint(f\"Maximum Sentence Length: {max_length}\")\nprint(f\"Mean Sentence Length: {mean_length:.2f}\")\nprint(f\"Median Sentence Length: {median_length}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:28:25.614085Z","iopub.execute_input":"2023-08-20T07:28:25.614767Z","iopub.status.idle":"2023-08-20T07:28:30.377501Z","shell.execute_reply.started":"2023-08-20T07:28:25.614729Z","shell.execute_reply":"2023-08-20T07:28:30.376376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n###Step 2.5.5 - Word Frequency Analysis
","metadata":{}},{"cell_type":"code","source":"# Flatten the list of tokens\nall_tokens = [token for tokens in transcriptions_tokens for token in tokens]\n\n# Count word frequency\nword_frequency = Counter(all_tokens)\n\n# Print the word frequency analysis\nprint(\"Word Frequency:\")\nplt.plot(word_frequency.values())","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:28:30.378738Z","iopub.execute_input":"2023-08-20T07:28:30.379561Z","iopub.status.idle":"2023-08-20T07:28:32.705949Z","shell.execute_reply.started":"2023-08-20T07:28:30.379524Z","shell.execute_reply":"2023-08-20T07:28:32.704960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate the number of unique words in each sentence\nsentences = train_df['sentence'].tolist()\nunique_word_counts = [len(set(sentence.split())) for sentence in sentences]\n\n# Compute descriptive statistics\nmin_unique_words = min(unique_word_counts)\nmax_unique_words = max(unique_word_counts)\nmean_unique_words = sum(unique_word_counts) / len(unique_word_counts)\nmedian_unique_words = sorted(unique_word_counts)[len(unique_word_counts) // 2]\n\n# Plot the distribution of unique word counts\nplt.figure(figsize=(10, 6))\nplt.hist(unique_word_counts, bins=50, color='lightcoral', edgecolor='black')\nplt.axvline(mean_unique_words, color='red', linestyle='dashed', linewidth=2, label='Mean')\nplt.axvline(median_unique_words, color='green', linestyle='dashed', linewidth=2, label='Median')\nplt.xlabel('Number of Unique Words')\nplt.ylabel('Frequency')\nplt.title('Distribution of Unique Words in Transcriptions')\nplt.legend()\nplt.show()\n\n# Print the descriptive statistics\nprint(\"Descriptive Statistics:\")\nprint(f\"Minimum Number of Unique Words: {min_unique_words}\")\nprint(f\"Maximum Number of Unique Words: {max_unique_words}\")\nprint(f\"Mean Number of Unique Words: {mean_unique_words:.2f}\")\nprint(f\"Median Number of Unique Words: {median_unique_words}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:28:32.707437Z","iopub.execute_input":"2023-08-20T07:28:32.707777Z","iopub.status.idle":"2023-08-20T07:28:39.116886Z","shell.execute_reply.started":"2023-08-20T07:28:32.707745Z","shell.execute_reply":"2023-08-20T07:28:39.116016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select the first 1000 sentences \nsentences = train_df['sentence'][:10000].tolist()\n\n# Tokenization using NLTK\nnltk.download('punkt') # Download the Punkt tokenizer\nsentences_tokens = [word_tokenize(sentence) for sentence in sentences]\n\n\n# Remove Bengali stopwords\nsentences_no_stopwords = [\n [word for word in tokens if word not in stop_words]\n for tokens in sentences_tokens\n]\n\n# Convert tokenized sentences back to strings\nsentences_processed = [' '.join(tokens) for tokens in sentences_no_stopwords]\n\n# Create a CountVectorizer to convert text data to a bag-of-words representation\nvectorizer = CountVectorizer(max_features=1000)\nX = vectorizer.fit_transform(sentences_processed)\n\n# Perform LDA topic modeling\nn_topics = 10 # Number of topics to discover\nlda_model = LatentDirichletAllocation(n_components=n_topics, random_state=42)\nlda_model.fit(X)\n\n# Get the top words for each topic\nfeature_names = vectorizer.get_feature_names_out()\ntop_words_per_topic = []\nfor topic_idx, topic in enumerate(lda_model.components_):\n top_words = [feature_names[i] for i in topic.argsort()[:-4:-1]]\n top_words_per_topic.append(top_words)\n\n# Print the top words for each topic\nfor i, top_words in enumerate(top_words_per_topic):\n print(f\"Topic {i + 1}: {' '.join(top_words)}\")\n ","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:28:39.118260Z","iopub.execute_input":"2023-08-20T07:28:39.119065Z","iopub.status.idle":"2023-08-20T07:29:16.087943Z","shell.execute_reply.started":"2023-08-20T07:28:39.119014Z","shell.execute_reply":"2023-08-20T07:29:16.086914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lda_model.components_","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:29:16.092368Z","iopub.execute_input":"2023-08-20T07:29:16.092644Z","iopub.status.idle":"2023-08-20T07:29:16.106519Z","shell.execute_reply.started":"2023-08-20T07:29:16.092620Z","shell.execute_reply":"2023-08-20T07:29:16.101938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### We have transcriptions of short speeches which are just sentences, topic modelling seems like a bad idea.","metadata":{}},{"cell_type":"markdown","source":"\n###Step 3 - Domain Analysis
","metadata":{}},{"cell_type":"markdown","source":"#### About Domains distribution\n- Number of domains in training data are not known\n- Number of domains in test data = 18\n- audio samples of 17 such domains which are not present in the training data are given in examples/\n- remaining domain should be present in the training data","metadata":{}},{"cell_type":"code","source":"for idx in np.arange(5):\n audio_file_path = f'{domains}/{DOMAINS[idx]}'\n\n # Load the audio file using librosa\n audio, sr = librosa.load(audio_file_path, sr=None)\n\n # Print DOMAIN\n print(DOMAINS[idx])\n ipd.display(ipd.Audio(audio, rate=sr))","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:29:16.107840Z","iopub.execute_input":"2023-08-20T07:29:16.108405Z","iopub.status.idle":"2023-08-20T07:29:17.404789Z","shell.execute_reply.started":"2023-08-20T07:29:16.108371Z","shell.execute_reply":"2023-08-20T07:29:17.403961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the list of domains and their corresponding audio files\nDOMAINS = [\n 'Audiobook.wav', 'Parliament Session.wav', 'Bangladeshi TV Drama.wav',\n 'Poem Recital.wav', 'Bengali Advertisement.wav', 'Puthi Literature.wav',\n 'Cartoon.wav', 'Slang Profanity.mp3', 'Debate.wav', 'Stage Drama Jatra.wav',\n 'Indian TV Drama.wav', 'Talk Show Interview.wav', 'Movie.wav', 'Telemedicine.mp3',\n 'News Presentation.wav', 'Waz Islamic Sermon.wav', 'Online Class.wav'\n]\n\n# Visualize the audio files and play them\nfor idx in np.arange(5):\n audio_file_path = os.path.join(domains, DOMAINS[idx])\n\n # Load the audio file using librosa\n audio, sr = librosa.load(audio_file_path, sr=None)\n\n # Plot the waveform\n plt.figure(figsize=(10, 4))\n librosa.display.waveshow(audio, sr=sr)\n plt.title(f'Waveform - {DOMAINS[idx]}')\n plt.xlabel('Time (s)')\n plt.ylabel('Amplitude')\n plt.show()\n\n # Plot the spectrogram\n plt.figure(figsize=(10, 4))\n D = librosa.amplitude_to_db(np.abs(librosa.stft(audio)), ref=np.max)\n librosa.display.specshow(D, sr=sr, x_axis='time', y_axis='linear')\n plt.colorbar(format='%+2.0f dB')\n plt.title(f'Spectrogram - {DOMAINS[idx]}')\n plt.xlabel('Time (s)')\n plt.ylabel('Frequency (Hz)')\n plt.show()\n\n # Play the audio\n print(f\"Audio: {DOMAINS[idx]}\")\n ipd.display(ipd.Audio(audio, rate=sr))","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:29:17.406126Z","iopub.execute_input":"2023-08-20T07:29:17.407071Z","iopub.status.idle":"2023-08-20T07:29:34.583763Z","shell.execute_reply.started":"2023-08-20T07:29:17.407020Z","shell.execute_reply":"2023-08-20T07:29:34.582265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n###Step 4 - Model Building
\n1. Public Wav2Vec2 model - no FT - inference only \n - In this notebook I am using this [baseline model](https://www.kaggle.com/code/ttahara/bengali-sr-public-wav2vec2-0-w-lm-baseline) to understand the leaderboard. After that we will fine-tune or add some new models\n\n","metadata":{}},{"cell_type":"markdown","source":"\n###Step 4.1- Requirements
","metadata":{}},{"cell_type":"code","source":"# Copying packages and extracting them for installation\n!cp -r ../input/python-packages2 ./\n\n# Install jiwer (Word Error Rate calculation)\n!tar xvfz ./python-packages2/jiwer.tgz\n!pip install ./jiwer/jiwer-2.3.0-py3-none-any.whl -f ./ --no-index\n\n# Install bnunicodenormalizer (Bengali Unicode normalization)\n!tar xvfz ./python-packages2/normalizer.tgz\n!pip install ./normalizer/bnunicodenormalizer-0.0.24.tar.gz -f ./ --no-index\n\n# Install pyctcdecode (CTC decoding for Wav2Vec2)\n!tar xvfz ./python-packages2/pyctcdecode.tgz\n!pip install ./pyctcdecode/attrs-22.1.0-py2.py3-none-any.whl -f ./ --no-index --no-deps\n!pip install ./pyctcdecode/exceptiongroup-1.0.0rc9-py3-none-any.whl -f ./ --no-index --no-deps\n!pip install ./pyctcdecode/hypothesis-6.54.4-py3-none-any.whl -f ./ --no-index --no-deps\n!pip install ./pyctcdecode/numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl -f ./ --no-index --no-deps\n!pip install ./pyctcdecode/pygtrie-2.5.0.tar.gz -f ./ --no-index --no-deps\n!pip install ./pyctcdecode/sortedcontainers-2.4.0-py2.py3-none-any.whl -f ./ --no-index --no-deps\n!pip install ./pyctcdecode/pyctcdecode-0.4.0-py2.py3-none-any.whl -f ./ --no-index --no-deps\n\n# Install pypi-kenlm (KenLM language model)\n!tar xvfz ./python-packages2/pypikenlm.tgz\n!pip install ./pypikenlm/pypi-kenlm-0.1.20220713.tar.gz -f ./ --no-index --no-deps\n\n\n","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:29:34.585652Z","iopub.execute_input":"2023-08-20T07:29:34.586446Z","iopub.status.idle":"2023-08-20T07:30:46.563269Z","shell.execute_reply.started":"2023-08-20T07:29:34.586401Z","shell.execute_reply":"2023-08-20T07:30:46.561950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rm -r python-packages2 jiwer normalizer pyctcdecode pypikenlm","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:30:46.565182Z","iopub.execute_input":"2023-08-20T07:30:46.565568Z","iopub.status.idle":"2023-08-20T07:30:46.570275Z","shell.execute_reply.started":"2023-08-20T07:30:46.565538Z","shell.execute_reply":"2023-08-20T07:30:46.569339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import typing as tp # Typing module for type hints\nfrom pathlib import Path # For working with file paths\nfrom functools import partial # To create partial functions\nfrom dataclasses import dataclass, field # For creating data classes\n\nimport pandas as pd \nimport pyctcdecode # For CTC decoding\nimport numpy as np \nfrom tqdm.notebook import tqdm # For creating progress bars\n\nimport librosa # For audio processing\n\nimport pyctcdecode # For CTC decoding\nimport kenlm # For working with KenLM language model\nimport torch # PyTorch library for machine learning\nfrom transformers import Wav2Vec2Processor, Wav2Vec2ProcessorWithLM, Wav2Vec2ForCTC # For loading and processing Wav2Vec2 models\nfrom bnunicodenormalizer import Normalizer # For Bengali Unicode normalization\n\nimport cloudpickle as cpkl # For pickling objects","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:30:46.571573Z","iopub.execute_input":"2023-08-20T07:30:46.572408Z","iopub.status.idle":"2023-08-20T07:30:56.568724Z","shell.execute_reply.started":"2023-08-20T07:30:46.572376Z","shell.execute_reply":"2023-08-20T07:30:56.567748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define paths and parameters for the project\nROOT = Path.cwd().parent\nINPUT = ROOT / \"input\"\nDATA = INPUT / \"bengaliai-speech\"\nTRAIN = DATA / \"train_mp3s\"\nTEST = DATA / \"test_mp3s\"\nSAMPLING_RATE = 16_000\nMODEL_PATH = INPUT / \"bengali-sr-download-public-trained-models/indicwav2vec_v1_bengali/\"\nLM_PATH = INPUT / \"bengali-sr-download-public-trained-models/wav2vec2-xls-r-300m-bengali/language_model/\"\n","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:30:56.570223Z","iopub.execute_input":"2023-08-20T07:30:56.570587Z","iopub.status.idle":"2023-08-20T07:30:56.577771Z","shell.execute_reply.started":"2023-08-20T07:30:56.570551Z","shell.execute_reply":"2023-08-20T07:30:56.576739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load Wav2Vec2 model and processor\nmodel = Wav2Vec2ForCTC.from_pretrained(MODEL_PATH) # CTC instance\n# processor will be responsible for handling the audion data\nprocessor = Wav2Vec2Processor.from_pretrained(MODEL_PATH)","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:30:56.579188Z","iopub.execute_input":"2023-08-20T07:30:56.580103Z","iopub.status.idle":"2023-08-20T07:31:14.427919Z","shell.execute_reply.started":"2023-08-20T07:30:56.580071Z","shell.execute_reply":"2023-08-20T07:31:14.426893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# build the vocabulary and a decoder\n\n# Get the vocabulary from the model's tokenizer\nvocab_dict = processor.tokenizer.get_vocab()\nprint('lENGTH OF THE VOCABULARY: ',len(vocab_dict))\nvocab_dict\n","metadata":{"execution":{"iopub.status.busy":"2023-08-20T07:31:14.429231Z","iopub.execute_input":"2023-08-20T07:31:14.429838Z","iopub.status.idle":"2023-08-20T07:31:14.442915Z","shell.execute_reply.started":"2023-08-20T07:31:14.429802Z","shell.execute_reply":"2023-08-20T07:31:14.441835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Joiners and non joiners\nin vocabulary we can see some joiners and non joiners, lets know them: