1 | Goal ⚽️
\n\n\n \nAs it is written in the **[overview]()**, the error rate of $Google$ $Translator$ for $Bengali$ $Language$ is around $74$%, which is really huge. $Bengali.AI$, is hosting this competition to find better soltutions to the `translating system`, which will help in the communication between locals and tourists at many places.","metadata":{}},{"cell_type":"markdown","source":"#
2 | Advisory 📃
\n\n\n \n* The training data is huge, try to work on a smaller sample, to save resource and time. Once the pipeline is in the good condition, then send the whole data \n \n* Padd the sentences to the maximum length, to avoid any shape errors\n \n* The total amount of files in the `train_mp3s` were found to be $9,63,636$ which takes around $3$ Hours to load with `Librosa`. I have decided to go with $5,354$ samples which is $0.005$% of the Original Data. This took me around $15$ Seconds to load.\n* The length of all the audio files is the same which is $2,250$, denoting $2,250$, Seconds(I think so )","metadata":{}},{"cell_type":"markdown","source":"#
3 | Data 💡
\n\n\n \nLets dive into the data ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns \nimport tqdm\nimport matplotlib.pyplot as plt \nimport librosa\nimport os","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-07-18T03:53:07.159638Z","iopub.execute_input":"2023-07-18T03:53:07.159994Z","iopub.status.idle":"2023-07-18T03:53:07.164286Z","shell.execute_reply.started":"2023-07-18T03:53:07.159968Z","shell.execute_reply":"2023-07-18T03:53:07.163468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n## $3.1$ $|$ $Train$\nOur main training data is in the `train.csv` which is a `DataFrame` type object that contains $3$ columns\n* $ID$ - This is the unique ID given to each sample. In many cases ID becomes useless, but here it is the connecting point between the training point and the target data\n* $Sentences$ - This is the column that contains target data, which is in the sentence format\n* $Train/Valid$ - The last column shows, which part of the sample is training data and which is the validation data. Though we can make our own splits, but I think the host has provided this information, because they had better splits, over the distribution of the data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/bengaliai-speech/train.csv\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-18T03:18:53.216785Z","iopub.execute_input":"2023-07-18T03:18:53.217105Z","iopub.status.idle":"2023-07-18T03:18:56.026421Z","shell.execute_reply.started":"2023-07-18T03:18:53.217081Z","shell.execute_reply":"2023-07-18T03:18:56.025521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n## $3.2$ $|$ $Train_-mp3s$\n \nThis is a folder that contains our training samples. These are in audio files, and we wil thus use `Librosa` to read these files into array format","metadata":{}},{"cell_type":"code","source":"audio , _ = librosa.load('/kaggle/input/bengaliai-speech/train_mp3s/000005f3362c.mp3')\nprint(audio)\nplt.figure(figsize = (100 , 30))\nsns.lineplot(audio)","metadata":{"execution":{"iopub.status.busy":"2023-07-18T03:56:13.424389Z","iopub.execute_input":"2023-07-18T03:56:13.424750Z","iopub.status.idle":"2023-07-18T03:56:15.632086Z","shell.execute_reply.started":"2023-07-18T03:56:13.424721Z","shell.execute_reply":"2023-07-18T03:56:15.631329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\nLoading files can take a lot of time and thus I am limiting this to only the first $5354$ samples.\n","metadata":{}},{"cell_type":"code","source":"counter = 0\naudio_list = []\n\nfor x in tqdm.tqdm(os.listdir(\"/kaggle/input/bengaliai-speech/train_mp3s\") , total = 5354):\n \n audio , _ = librosa.load(\"/kaggle/input/bengaliai-speech/train_mp3s/\" + x)\n audio_list.append(audio)\n \n counter += 1\n if counter == 5354:break","metadata":{"execution":{"iopub.status.busy":"2023-07-18T03:57:20.248064Z","iopub.execute_input":"2023-07-18T03:57:20.248388Z","iopub.status.idle":"2023-07-18T03:57:55.890338Z","shell.execute_reply.started":"2023-07-18T03:57:20.248364Z","shell.execute_reply":"2023-07-18T03:57:55.889446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
4 | TO DO LIST 🗂️
\n\n\n\n* $TO$ $DO$ $1$ $:$ $TOKENIZE$ $SENTENCES$\n* $TO$ $DO$ $1$ $:$ $MAKE$ $DATALOADER$\n* $TO$ $DO$ $1$ $:$ $MAKE$ $A$ $MODEL$\n* $TO$ $DO$ $1$ $:$ $TRAIN$ $THE$ $MODEL$\n* $TO$ $DO$ $1$ $:$ $IMPORVE$ $REUSLTS$\n* $TO$ $DO$ $1$ $:$ $LESS$ $TRAINING$ $TIME$\n* $TO$ $DO$ $1$ $:$ $DANCE$","metadata":{}},{"cell_type":"markdown","source":"#
4 | TO DO LIST 🚀
\n\n\n\n**THAT IT FOR TODAY GUYS**\n\n**WE WILL GO DEEPER INTO THE DATA IN THE UPCOMING VERSIONS**\n\n**PLEASE COMMENT YOUR THOUGHTS, HIHGLY APPRICIATED**\n\n**DONT FORGET TO MAKE AN UPVOTE, IF YOU LIKED MY WORK $:)$**\n\n![](\"https://i.imgflip.com/19aadg.jpg\")
\n \n**PEACE OUT $!!!$**","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}
\n \n**PEACE OUT $!!!$**","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}