I have a memory problem. Could someone helps me?
\nI want to make specrogram dataset by this code, But I'm facing error: 'Your notebook tried to allocate more memory than is available.0'.
Can I avoid this? or should I execute it in offline environment (but I don't wanna do that becouse data size of wave dataset is so big.)?
\n os\n gzip\n bz2\n pickle\n pywt\n librosa\n numpy np\n matplotlib.pyplot plt\n\n IPython.display Audio\n\n\nKAGGLE_TRAIN = \nADDED_TRAIN_1 = \nADDED_TRAIN_2 = \n\nSAVE_TRAIN = \nos.makedirs(KAGGLE_TRAIN, exist_ok=)\n\n ():\n ():\n \n self.AUDIO_DIRECTORY = AUDIO_DIRECTORY\n self.SAVE_DIRECTORY = SAVE_DIRECTORY\n self.view = view\n\n \n func_names = [method method (self) ((self, method)) method.startswith()]\n (func_names)\n os.makedirs(self.SAVE_DIRECTORY, exist_ok=)\n func_name func_names:\n func = func_name.split()[-]\n os.makedirs(self.SAVE_DIRECTORY + + func, exist_ok=)\n\n \n ():\n \n self.y, self.sr = librosa.load(audio_filepath, offset=, duration=)\n\n \n ():\n train_dict = {}\n species_list = os.listdir(self.AUDIO_DIRECTORY)\n species species_list:\n species_path = self.AUDIO_DIRECTORY + + species\n audio_file_list = os.listdir(species_path)\n audio_file audio_file_list:\n audio_filepath = species_path + + audio_file\n self.load_wave(audio_filepath) \n mfcc = function() \n train_dict[audio_file.split()[]] = mfcc \n\n\n\n \n SAVE_PATH = self.SAVE_DIRECTORY + + function.__name__.split()[-] + \n self.save_as_picke_gzip(train_dict, SAVE_PATH)\n\n (): \n gzip.(filepath, ) f:\n pickle.dump(data, f)\n\n (): \n self.view:\n (, self.y.shape)\n display(Audio(self.y, rate=self.sr))\n plt.figure(figsize=(, ))\n librosa.display.waveshow(self.y, sr=self.sr)\n plt.title()\n plt.xlabel()\n plt.ylabel()\n plt.show()\n self.y\n\n ():\n spec = librosa.amplitude_to_db(np.(librosa.stft(self.y)), ref=np.)\n\n self.view:\n (, spec.shape)\n plt.figure(figsize=(, ))\n librosa.display.specshow(spec, sr=self.sr, x_axis=, y_axis=)\n plt.colorbar(=)\n plt.title()\n plt.show()\n spec\n\n ():\n func_list = [\n\n self.func_spec,\n ]\n\n func func_list:\n self.apply_func(func)\n\n\npreprocessing_kaggle = preprocessing(KAGGLE_TRAIN, SAVE_TRAIN)\npreprocessing_added_train_1 = preprocessing(ADDED_TRAIN_1, SAVE_TRAIN)\npreprocessing_added_train_2 = preprocessing(ADDED_TRAIN_2, SAVE_TRAIN)\n\n\npreprocessing_kaggle.execute()\npreprocessing_added_train_1.execute()\npreprocessing_added_train_2.execute()\n\n\n\n\n(os.listdir(SAVE_TRAIN))\nI have a memory problem. Could someone helps me?
\nI want to make specrogram dataset by this code, But I'm facing error: 'Your notebook tried to allocate more memory than is available.0'.
Can I avoid this? or should I execute it in offline environment (but I don't wanna do that becouse data size of wave dataset is so big.)?
\n os\n gzip\n bz2\n pickle\n pywt\n librosa\n numpy np\n matplotlib.pyplot plt\n\n IPython.display Audio\n\n\nKAGGLE_TRAIN = \nADDED_TRAIN_1 = \nADDED_TRAIN_2 = \n\nSAVE_TRAIN = \nos.makedirs(KAGGLE_TRAIN, exist_ok=)\n\n ():\n ():\n \n self.AUDIO_DIRECTORY = AUDIO_DIRECTORY\n self.SAVE_DIRECTORY = SAVE_DIRECTORY\n self.view = view\n\n \n func_names = [method method (self) ((self, method)) method.startswith()]\n (func_names)\n os.makedirs(self.SAVE_DIRECTORY, exist_ok=)\n func_name func_names:\n func = func_name.split()[-]\n os.makedirs(self.SAVE_DIRECTORY + + func, exist_ok=)\n\n \n ():\n \n self.y, self.sr = librosa.load(audio_filepath, offset=, duration=)\n\n \n ():\n train_dict = {}\n species_list = os.listdir(self.AUDIO_DIRECTORY)\n species species_list:\n species_path = self.AUDIO_DIRECTORY + + species\n audio_file_list = os.listdir(species_path)\n audio_file audio_file_list:\n audio_filepath = species_path + + audio_file\n self.load_wave(audio_filepath) \n mfcc = function() \n train_dict[audio_file.split()[]] = mfcc \n\n\n\n \n SAVE_PATH = self.SAVE_DIRECTORY + + function.__name__.split()[-] + \n self.save_as_picke_gzip(train_dict, SAVE_PATH)\n\n (): \n gzip.(filepath, ) f:\n pickle.dump(data, f)\n\n (): \n self.view:\n (, self.y.shape)\n display(Audio(self.y, rate=self.sr))\n plt.figure(figsize=(, ))\n librosa.display.waveshow(self.y, sr=self.sr)\n plt.title()\n plt.xlabel()\n plt.ylabel()\n plt.show()\n self.y\n\n ():\n spec = librosa.amplitude_to_db(np.(librosa.stft(self.y)), ref=np.)\n\n self.view:\n (, spec.shape)\n plt.figure(figsize=(, ))\n librosa.display.specshow(spec, sr=self.sr, x_axis=, y_axis=)\n plt.colorbar(=)\n plt.title()\n plt.show()\n spec\n\n ():\n func_list = [\n\n self.func_spec,\n ]\n\n func func_list:\n self.apply_func(func)\n\n\npreprocessing_kaggle = preprocessing(KAGGLE_TRAIN, SAVE_TRAIN)\npreprocessing_added_train_1 = preprocessing(ADDED_TRAIN_1, SAVE_TRAIN)\npreprocessing_added_train_2 = preprocessing(ADDED_TRAIN_2, SAVE_TRAIN)\n\n\npreprocessing_kaggle.execute()\npreprocessing_added_train_1.execute()\npreprocessing_added_train_2.execute()\n\n\n\n\n(os.listdir(SAVE_TRAIN))\nmatplotlib adds a bunch of overhead - and you don't actually need it to generate spectrograms with librosa.
\nI show how to make spectrograms without matplotlib in this notebook:
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator
If you want to try to keep your current code intact - you can try the just changing matplotlib to a non-interactive \"backend\"
\nThere's an older version of the same notebook that takes this approach:
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator?scriptVersionId=170393744
Relevant code is:
\n\nmatplotlib.use()\nGetting rid of matplotlib altogether is the better answer - and probably worth the few minutes to refactor things.
\n-Rich
", "rawMarkdown": "matplotlib adds a bunch of overhead - and you don't actually need it to generate spectrograms with librosa.\n\nI show how to make spectrograms without matplotlib in this notebook:\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator\n\nIf you want to try to keep your current code intact - you can try the just changing matplotlib to a non-interactive \"backend\"\n\nThere's an older version of the same notebook that takes this approach:\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator?scriptVersionId=170393744\n\nRelevant code is:\n```python\n#this is needed to keep memory under control (prevents further generation of previews)\nmatplotlib.use('Agg')\n\n```\n\nGetting rid of matplotlib altogether is the better answer - and probably worth the few minutes to refactor things.\n\n-Rich", "votes": null }, { "id": "2774020", "postDate": "04/25/2024 03:12:04", "content": "Thank you for teaching me! I'll try to delete matplotlib, and check those notebook.
", "rawMarkdown": "Thank you for teaching me! I'll try to delete matplotlib, and check those notebook.", "votes": null } ], "comments": [ { "id": 2773469, "author_name": "richolson", "author_url": "", "post_date": "04/24/2024 18:51:35", "content": "matplotlib adds a bunch of overhead - and you don't actually need it to generate spectrograms with librosa.
\nI show how to make spectrograms without matplotlib in this notebook:
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator
If you want to try to keep your current code intact - you can try the just changing matplotlib to a non-interactive \"backend\"
\nThere's an older version of the same notebook that takes this approach:
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator?scriptVersionId=170393744
Relevant code is:
\n\nmatplotlib.use()\nGetting rid of matplotlib altogether is the better answer - and probably worth the few minutes to refactor things.
\n-Rich
", "votes": null, "replies": [ { "id": 2774020, "author_name": "moyuto", "author_url": "", "post_date": "04/25/2024 03:12:04", "content": "Thank you for teaching me! I'll try to delete matplotlib, and check those notebook.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2772139": "I have a memory problem. Could someone helps me?\nI want to make specrogram dataset by this code, But I'm facing error: 'Your notebook tried to allocate more memory than is available.0'.\n\nCan I avoid this? or should I execute it in offline environment (but I don't wanna do that becouse data size of wave dataset is so big.)?\n\n```python\nimport os\nimport gzip\nimport bz2\nimport pickle\nimport pywt\nimport librosa\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom IPython.display import Audio\n\n\nKAGGLE_TRAIN = '/kaggle/input/birdclef-2024/train_audio'\nADDED_TRAIN_1 = '/kaggle/input/birdclef2024-additional-wav-1/additional_audio-1'\nADDED_TRAIN_2 = '/kaggle/input/birdclef2024-additional-wav-2/additional_audio-2'\n\nSAVE_TRAIN = '/kaggle/working/train_image'\nos.makedirs(KAGGLE_TRAIN, exist_ok=True)\n\nclass preprocessing():\n def __init__(self, AUDIO_DIRECTORY, SAVE_DIRECTORY, view=False):\n # config\n self.AUDIO_DIRECTORY = AUDIO_DIRECTORY\n self.SAVE_DIRECTORY = SAVE_DIRECTORY\n self.view = view\n \n # make directory\n func_names = [method for method in dir(self) if callable(getattr(self, method)) and method.startswith(\"func\")]\n print(func_names)\n os.makedirs(self.SAVE_DIRECTORY, exist_ok=True)\n for func_name in func_names:\n func = func_name.split('_')[-1]\n os.makedirs(self.SAVE_DIRECTORY + '/' + func, exist_ok=True)\n \n # load data\n def load_wave(self, audio_filepath):\n # pick up first 5 seconds\n self.y, self.sr = librosa.load(audio_filepath, offset=0, duration=5)\n \n # apply and save\n def apply_func(self, function):\n train_dict = {}\n species_list = os.listdir(self.AUDIO_DIRECTORY)\n for species in species_list:\n species_path = self.AUDIO_DIRECTORY + '/' + species\n audio_file_list = os.listdir(species_path)\n for audio_file in audio_file_list:\n audio_filepath = species_path +'/' + audio_file\n self.load_wave(audio_filepath) # load audio\n mfcc = function() # apply function\n train_dict[audio_file.split(\".\")[0]] = mfcc # register to dict\n# break\n# break\n \n # set function name as filepath name\n SAVE_PATH = self.SAVE_DIRECTORY + '/' + function.__name__.split('_')[-1] + f'/train.pickle.gz'\n self.save_as_picke_gzip(train_dict, SAVE_PATH)\n \n def save_as_picke_gzip(self, data, filepath): \n with gzip.open(filepath, 'wb') as f:\n pickle.dump(data, f)\n \n def func_waveform(self): \n if self.view:\n print('waveform shape: ', self.y.shape)\n display(Audio(self.y, rate=self.sr))\n plt.figure(figsize=(10, 4))\n librosa.display.waveshow(self.y, sr=self.sr)\n plt.title('Waveform')\n plt.xlabel('Time (s)')\n plt.ylabel('Amplitude')\n plt.show()\n return self.y\n \n def func_spec(self):\n spec = librosa.amplitude_to_db(np.abs(librosa.stft(self.y)), ref=np.max)\n \n if self.view:\n print('spec shape: ', spec.shape)\n plt.figure(figsize=(10, 4))\n librosa.display.specshow(spec, sr=self.sr, x_axis='time', y_axis='log')\n plt.colorbar(format='%+2.0f dB')\n plt.title('Spectrogram')\n plt.show()\n return spec\n \n def execute(self):\n func_list = [\n# self.func_waveform,\n self.func_spec,\n ]\n\n for func in func_list:\n self.apply_func(func)\n\n# define preprocessing class\npreprocessing_kaggle = preprocessing(KAGGLE_TRAIN, SAVE_TRAIN)\npreprocessing_added_train_1 = preprocessing(ADDED_TRAIN_1, SAVE_TRAIN)\npreprocessing_added_train_2 = preprocessing(ADDED_TRAIN_2, SAVE_TRAIN)\n\n# execute preprocessing\npreprocessing_kaggle.execute()\npreprocessing_added_train_1.execute()\npreprocessing_added_train_2.execute()\n\n \n\n \nprint(os.listdir(SAVE_TRAIN))\n```", "2773469": "matplotlib adds a bunch of overhead - and you don't actually need it to generate spectrograms with librosa.\n\nI show how to make spectrograms without matplotlib in this notebook:\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator\n\nIf you want to try to keep your current code intact - you can try the just changing matplotlib to a non-interactive \"backend\"\n\nThere's an older version of the same notebook that takes this approach:\nhttps://www.kaggle.com/code/richolson/birdclef-2024-simple-mel-spectrogram-generator?scriptVersionId=170393744\n\nRelevant code is:\n```python\n#this is needed to keep memory under control (prevents further generation of previews)\nmatplotlib.use('Agg')\n\n```\n\nGetting rid of matplotlib altogether is the better answer - and probably worth the few minutes to refactor things.\n\n-Rich", "2774020": "Thank you for teaching me! I'll try to delete matplotlib, and check those notebook." }, "source": "meta" }