{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What is this Notebook about?","metadata":{}},{"cell_type":"markdown","source":"> This competition is a **code competition**. That means you have to submit a notebook rather than any submission.csv/prediction set. This is because the test set is hidden and they'll infer on the full test set using your NB. But the notebook will have some limitations.\n\n\n\nFrom the Code Requirements section of the competition : \n~~~\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit\" button to be active after a commit, the following conditions must be met:\n\nCPU Notebook <= 9 hours run-time\nGPU Notebook <= 9 hours run-time\nInternet access disabled\nFreely & publicly available external data is allowed, including pre-trained models\nSubmission file must be named submission.csv\n~~~\n","metadata":{}},{"cell_type":"markdown","source":"> So you have to **submit a notebook without internet connection**. This creates a problem. You may need to install external packages to run your model/evaluate your model using pip install. but pip install usually requires internet. So you won't be able to install packages using !pip install package without internet.\n\n> So what's the solution? Well, the solution is to create wheels for the required 3rd party packages and then save them as a dataset and then when you're running your code w/o internet , you can install them from these wheels.\n> \nLet's get started and dive in depth to understand.","metadata":{}},{"cell_type":"markdown","source":"In this notebook we'll create the necessary wheels . In my [other notebook](https://www.kaggle.com/code/mbmmurad/nemo-conformer-baseline-w-o-internet-inference-nb/) we'll be using these wheels to install packages and submit the code to the competition","metadata":{}},{"cell_type":"markdown","source":"# Resources","metadata":{}},{"cell_type":"markdown","source":"> * https://www.kaggle.com/code/truthr/install-python-packages-without-internet-access/notebook\n> * https://www.kaggle.com/code/narnaoot/installing-packages-without-internet-for-kaggle\n> * https://www.kaggle.com/code/samuelepino/pip-downloading-packages-to-your-local-machine/notebook?scriptVersionId=29576961","metadata":{}},{"cell_type":"markdown","source":"# Example","metadata":{}},{"cell_type":"markdown","source":"Suppose you need the ***[jiwer](https://pypi.org/project/jiwer/)*** packages in your code. Now ith internet enables you can easily install it using pip","metadata":{}},{"cell_type":"code","source":"!pip install jiwer","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:00.745078Z","iopub.execute_input":"2023-07-20T18:38:00.745467Z","iopub.status.idle":"2023-07-20T18:38:14.703466Z","shell.execute_reply.started":"2023-07-20T18:38:00.745434Z","shell.execute_reply":"2023-07-20T18:38:14.702498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from jiwer import wer\nwer(\"How are you?\",\"How r u?\")","metadata":{"execution":{"iopub.status.busy":"2023-07-20T19:24:42.881486Z","iopub.execute_input":"2023-07-20T19:24:42.882079Z","iopub.status.idle":"2023-07-20T19:24:42.976387Z","shell.execute_reply.started":"2023-07-20T19:24:42.882041Z","shell.execute_reply":"2023-07-20T19:24:42.975434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Now if we disable the internet and run the same code you'll see this will run forever and won't be able to install this package. \n\n> **Now the solution is to build .whl files and then install the packages from the whl files**. How to create wheels? well, this bit is very easy.\n> You just need to run !pip wheel jiwer and it will create a lot of required .whl files. remember to create a directory before doing it.","metadata":{}},{"cell_type":"code","source":"pwd","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:14.705569Z","iopub.execute_input":"2023-07-20T18:38:14.705968Z","iopub.status.idle":"2023-07-20T18:38:14.713730Z","shell.execute_reply.started":"2023-07-20T18:38:14.705930Z","shell.execute_reply":"2023-07-20T18:38:14.712869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%mkdir jiwer\n%cd jiwer","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:14.715089Z","iopub.execute_input":"2023-07-20T18:38:14.715437Z","iopub.status.idle":"2023-07-20T18:38:15.658905Z","shell.execute_reply.started":"2023-07-20T18:38:14.715412Z","shell.execute_reply":"2023-07-20T18:38:15.657848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip wheel jiwer\n","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:15.662541Z","iopub.execute_input":"2023-07-20T18:38:15.663120Z","iopub.status.idle":"2023-07-20T18:38:19.164058Z","shell.execute_reply.started":"2023-07-20T18:38:15.663087Z","shell.execute_reply":"2023-07-20T18:38:19.163104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now look at the files of this directory","metadata":{}},{"cell_type":"code","source":"!ls ","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:19.165619Z","iopub.execute_input":"2023-07-20T18:38:19.165937Z","iopub.status.idle":"2023-07-20T18:38:20.117201Z","shell.execute_reply.started":"2023-07-20T18:38:19.165910Z","shell.execute_reply":"2023-07-20T18:38:20.116250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here you can see the .whl files. We'll use them to install w/o internet connection.","metadata":{}},{"cell_type":"markdown","source":"# Build Packages for NeMo","metadata":{}},{"cell_type":"markdown","source":"> Now we're interested to run an inference with this [baseline model](https://huggingface.co/bengaliAI/BanglaConformer) by **Bengali.Ai (The competition organizers)**\n> \n> This model requires the [NeMo toolkit](https://github.com/NVIDIA/NeMo) and some other external packages. In this notebook we'll create the wheels for those packages and then use these in the notebook we want to submit. \n","metadata":{}},{"cell_type":"markdown","source":"> **We'll install NeMo from it's github repository.** Installing NeMo was a huge hassle.","metadata":{}},{"cell_type":"markdown","source":"**Step 1** : clone the git repo","metadata":{}},{"cell_type":"code","source":"%cd ..","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:27.474300Z","iopub.execute_input":"2023-07-20T18:38:27.474612Z","iopub.status.idle":"2023-07-20T18:38:27.480777Z","shell.execute_reply.started":"2023-07-20T18:38:27.474585Z","shell.execute_reply":"2023-07-20T18:38:27.479586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!git clone https://github.com/NVIDIA/NeMo","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:35.925758Z","iopub.execute_input":"2023-07-20T18:38:35.926184Z","iopub.status.idle":"2023-07-20T18:38:47.840757Z","shell.execute_reply.started":"2023-07-20T18:38:35.926154Z","shell.execute_reply":"2023-07-20T18:38:47.839916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Step 2:** \nNow, inside this folder there's a bash file reinstall.sh","metadata":{}},{"cell_type":"code","source":"!ls ./NeMo","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:47.843072Z","iopub.execute_input":"2023-07-20T18:38:47.843375Z","iopub.status.idle":"2023-07-20T18:38:48.786732Z","shell.execute_reply.started":"2023-07-20T18:38:47.843348Z","shell.execute_reply":"2023-07-20T18:38:48.785848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at what's inside this file","metadata":{}},{"cell_type":"code","source":"!cat ./NeMo/reinstall.sh","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:48.788572Z","iopub.execute_input":"2023-07-20T18:38:48.788874Z","iopub.status.idle":"2023-07-20T18:38:49.731117Z","shell.execute_reply.started":"2023-07-20T18:38:48.788846Z","shell.execute_reply":"2023-07-20T18:38:49.730245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The packages are getting installed by this line**","metadata":{}},{"cell_type":"markdown","source":"```\nif [[ \"$INSTALL_OPTION\" == \"dev\" ]]; then\n    ${PIP} install --editable \".[all]\"\n```","metadata":{}},{"cell_type":"markdown","source":"This pip install --editable \".[all]\" line is installing the packages. But we want to create the wheels as well. So what to do?\nWell pretty easy. We'll add \n\n```!pip wheel --editable \".[all]```\n\nunder the install line. So it'll look like that","metadata":{}},{"cell_type":"markdown","source":"```  \n${PIP} install --editable \".[all]\"\n${PIP} wheel --editable \".[all]\"\n```","metadata":{}},{"cell_type":"markdown","source":"Well I'have already changed this in this file /kaggle/input/shfile-changed/reinstall.sh. \n\nWe'll just replace this file in the NeMo directory","metadata":{}},{"cell_type":"code","source":"!rm NeMo/reinstall.sh","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:51.186655Z","iopub.execute_input":"2023-07-20T18:38:51.187039Z","iopub.status.idle":"2023-07-20T18:38:52.129514Z","shell.execute_reply.started":"2023-07-20T18:38:51.187009Z","shell.execute_reply":"2023-07-20T18:38:52.128546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!cp /kaggle/input/shfile-changed/reinstall.sh NeMo/","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:53.063616Z","iopub.execute_input":"2023-07-20T18:38:53.063978Z","iopub.status.idle":"2023-07-20T18:38:54.034664Z","shell.execute_reply.started":"2023-07-20T18:38:53.063949Z","shell.execute_reply":"2023-07-20T18:38:54.033607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls NeMo/","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:38:56.688938Z","iopub.execute_input":"2023-07-20T18:38:56.689959Z","iopub.status.idle":"2023-07-20T18:38:57.624173Z","shell.execute_reply.started":"2023-07-20T18:38:56.689924Z","shell.execute_reply":"2023-07-20T18:38:57.623227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well now we're ready. We'll need to run the bash file now.","metadata":{}},{"cell_type":"code","source":"cd NeMo","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:39:00.379024Z","iopub.execute_input":"2023-07-20T18:39:00.379425Z","iopub.status.idle":"2023-07-20T18:39:00.385430Z","shell.execute_reply.started":"2023-07-20T18:39:00.379394Z","shell.execute_reply":"2023-07-20T18:39:00.384513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture tst\n!bash reinstall.sh","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:39:04.836665Z","iopub.execute_input":"2023-07-20T18:39:04.837022Z","iopub.status.idle":"2023-07-20T18:44:11.162021Z","shell.execute_reply.started":"2023-07-20T18:39:04.836994Z","shell.execute_reply":"2023-07-20T18:44:11.159555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This'll create a lot of .whl files in the  ```/kaggle/working/NeMo``` folder","metadata":{}},{"cell_type":"code","source":"!ls ","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:44:11.166724Z","iopub.execute_input":"2023-07-20T18:44:11.167326Z","iopub.status.idle":"2023-07-20T18:44:12.251769Z","shell.execute_reply.started":"2023-07-20T18:44:11.167275Z","shell.execute_reply":"2023-07-20T18:44:12.250338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we'll commit this notebook and then save the output as a dataset and then load this dataset in our inference Notebook.\n\nBefore that let's install some other packages that we might need","metadata":{}},{"cell_type":"code","source":"!apt-get install sox libsox-fmt-mp3 -y","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:49:28.820508Z","iopub.execute_input":"2023-07-20T18:49:28.821014Z","iopub.status.idle":"2023-07-20T18:49:39.539148Z","shell.execute_reply.started":"2023-07-20T18:49:28.820976Z","shell.execute_reply":"2023-07-20T18:49:39.537708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!apt-get update && apt-get install -y libsndfile1 ffmpeg\n!pip install Cython","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:49:39.541459Z","iopub.execute_input":"2023-07-20T18:49:39.542656Z","iopub.status.idle":"2023-07-20T18:50:00.754355Z","shell.execute_reply.started":"2023-07-20T18:49:39.542612Z","shell.execute_reply":"2023-07-20T18:50:00.751378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd .. \n%mkdir ffmpeg\n%cd ffmpeg","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:52:06.588330Z","iopub.execute_input":"2023-07-20T18:52:06.588782Z","iopub.status.idle":"2023-07-20T18:52:07.717359Z","shell.execute_reply.started":"2023-07-20T18:52:06.588748Z","shell.execute_reply":"2023-07-20T18:52:07.715702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip wheel ffmpeg","metadata":{"execution":{"iopub.status.busy":"2023-07-20T18:52:26.144579Z","iopub.execute_input":"2023-07-20T18:52:26.145066Z","iopub.status.idle":"2023-07-20T18:52:31.098824Z","shell.execute_reply.started":"2023-07-20T18:52:26.145007Z","shell.execute_reply":"2023-07-20T18:52:31.096752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd .. \n%mkdir sox\n%cd sox","metadata":{"execution":{"iopub.status.busy":"2023-07-20T19:04:44.296615Z","iopub.execute_input":"2023-07-20T19:04:44.297301Z","iopub.status.idle":"2023-07-20T19:04:45.326367Z","shell.execute_reply.started":"2023-07-20T19:04:44.297257Z","shell.execute_reply":"2023-07-20T19:04:45.325412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip wheel sox","metadata":{"execution":{"iopub.status.busy":"2023-07-20T19:04:46.348289Z","iopub.execute_input":"2023-07-20T19:04:46.349153Z","iopub.status.idle":"2023-07-20T19:04:49.578640Z","shell.execute_reply.started":"2023-07-20T19:04:46.349109Z","shell.execute_reply":"2023-07-20T19:04:49.577722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We're all done. Please see my other notebook to see how to infer using these wheels.","metadata":{}}]}