{
  "id": 399456,
  "title": "Long load time when working with data",
  "url": "/competitions/birdclef-2023/discussion/399456",
  "author_name": "",
  "post_date": "2023-04-04T06:44:59.359356800Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I am breaking the audio files into 5 seconds and then converting them into melSpectrograms and then saving the results to a list, but it takes a few hours and was wondering what are the best practices in these situations to reduce the load time or if there is a better way to handle the data.</p>",
  "messages": [
    {
      "id": "2208561",
      "postDate": "04/04/2023 06:44:59",
      "content": "<p>I am breaking the audio files into 5 seconds and then converting them into melSpectrograms and then saving the results to a list, but it takes a few hours and was wondering what are the best practices in these situations to reduce the load time or if there is a better way to handle the data.</p>",
      "rawMarkdown": "I am breaking the audio files into 5 seconds and then converting them into melSpectrograms and then saving the results to a list, but it takes a few hours and was wondering what are the best practices in these situations to reduce the load time or if there is a better way to handle the data.",
      "votes": null
    },
    {
      "id": "2208908",
      "postDate": "04/04/2023 11:15:24",
      "content": "<p>I don't know the best practices, but I do the following operations on the fly.</p>\n<ol>\n<li>data loading (.ogg file to numpy) : librosa</li>\n<li>waveform augmentations : numpy.ndarray operations</li>\n<li>mel spectrogram calcurations : torchaudio</li>\n<li>mel spectrogram augmentations : torch.Tensor operations</li>\n</ol>\n<p>I measured the speed of mel spectrogram calculations on kaggle notebook, the speeds were as follows.</p>\n<ul>\n<li>P100 GPU :   torchaudio &gt;&gt; tensorflow_io &gt; torchlibrosa ~ nnAudio</li>\n<li>CPU :   torchaudio &gt; librosa &gt;&gt; tensorflow_io &gt;&gt; torchlibrosa ~ nnAudio</li>\n</ul>\n<p>Note: The order will vary depending on the environment. I am not familiar with tensor flow and may not be using it well.</p>",
      "rawMarkdown": "I don't know the best practices, but I do the following operations on the fly.\n1. data loading (.ogg file to numpy) : librosa\n2. waveform augmentations : numpy.ndarray operations\n3. mel spectrogram calcurations : torchaudio\n4. mel spectrogram augmentations : torch.Tensor operations\n\nI measured the speed of mel spectrogram calculations on kaggle notebook, the speeds were as follows.\n- P100 GPU :   torchaudio >> tensorflow_io > torchlibrosa ~ nnAudio\n- CPU :   torchaudio > librosa >> tensorflow_io >> torchlibrosa ~ nnAudio\n\nNote: The order will vary depending on the environment. I am not familiar with tensor flow and may not be using it well.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2208908,
      "author_name": "shigemitsutomizawa",
      "author_url": "",
      "post_date": "04/04/2023 11:15:24",
      "content": "<p>I don't know the best practices, but I do the following operations on the fly.</p>\n<ol>\n<li>data loading (.ogg file to numpy) : librosa</li>\n<li>waveform augmentations : numpy.ndarray operations</li>\n<li>mel spectrogram calcurations : torchaudio</li>\n<li>mel spectrogram augmentations : torch.Tensor operations</li>\n</ol>\n<p>I measured the speed of mel spectrogram calculations on kaggle notebook, the speeds were as follows.</p>\n<ul>\n<li>P100 GPU :   torchaudio &gt;&gt; tensorflow_io &gt; torchlibrosa ~ nnAudio</li>\n<li>CPU :   torchaudio &gt; librosa &gt;&gt; tensorflow_io &gt;&gt; torchlibrosa ~ nnAudio</li>\n</ul>\n<p>Note: The order will vary depending on the environment. I am not familiar with tensor flow and may not be using it well.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2208561": "I am breaking the audio files into 5 seconds and then converting them into melSpectrograms and then saving the results to a list, but it takes a few hours and was wondering what are the best practices in these situations to reduce the load time or if there is a better way to handle the data.",
    "2208908": "I don't know the best practices, but I do the following operations on the fly.\n1. data loading (.ogg file to numpy) : librosa\n2. waveform augmentations : numpy.ndarray operations\n3. mel spectrogram calcurations : torchaudio\n4. mel spectrogram augmentations : torch.Tensor operations\n\nI measured the speed of mel spectrogram calculations on kaggle notebook, the speeds were as follows.\n- P100 GPU :   torchaudio >> tensorflow_io > torchlibrosa ~ nnAudio\n- CPU :   torchaudio > librosa >> tensorflow_io >> torchlibrosa ~ nnAudio\n\nNote: The order will vary depending on the environment. I am not familiar with tensor flow and may not be using it well."
  },
  "source": "meta"
}