{
  "id": 159422,
  "title": ".wav converter",
  "url": "/competitions/birdsong-recognition/discussion/159422",
  "author_name": "",
  "post_date": "2020-06-17T11:42:24.404909200Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>The following code can be used to convert the entire dataset into .wav (which is faster to load and works with more libraries, but is huuuuuuge). It takes ~1100 seconds to complete.</p>\n\n<p>```\nimport time\nimport os\nimport multiprocessing</p>\n\n<p>from pydub import AudioSegment</p>\n\n<p>def to_wav(in_dir, out_dir, num_workers=20):\n    start_time = time.time()</p>\n\n<pre><code>def bird_to_wav(names):\n    for birdname in names:\n        print(\"starting: \" +birdname)\n        for soundfile in os.listdir(in_dir + birdname):\n            path = in_dir + birdname + \"/\" + soundfile\n            sound = AudioSegment.from_file(path)\n            sound.export(out_dir + soundfile[:-3] + \"wav\", format=\"wav\")\n        print(birdname + \" finished after: %s seconds\" % (time.time() - start_time))\n    print(\"worker finished\\n\")\n\nnames = os.listdir(in_dir)\n\nnum_workers = 4\nend = len(names)\nto_each_worker = round(end / num_workers)\n\n\nfor i in range(num_workers+1):\n    args = (names[i * to_each_worker: i* to_each_worker + to_each_worker],)\n\n    p = multiprocessing.Process(target=bird_to_wav, args=(args))\n    p.start()\n\nprint(\"all jobs started\")\n</code></pre>\n\n<p>```</p>\n\n<p>The resulting \"out_dir\" will DIRECTLY (no bird-wise directories, <code>train.csv</code> contains the file names anyway) contain all audio recordings in .wav format and take up around 141 GB in size. Maybe it helps somebody.</p>\n\n<p>You'll have to install ffmpeg to make it work though.</p>",
  "messages": [
    {
      "id": "890211",
      "postDate": "06/17/2020 11:42:24",
      "content": "<p>The following code can be used to convert the entire dataset into .wav (which is faster to load and works with more libraries, but is huuuuuuge). It takes ~1100 seconds to complete.</p>\n\n<p>```\nimport time\nimport os\nimport multiprocessing</p>\n\n<p>from pydub import AudioSegment</p>\n\n<p>def to_wav(in_dir, out_dir, num_workers=20):\n    start_time = time.time()</p>\n\n<pre><code>def bird_to_wav(names):\n    for birdname in names:\n        print(\"starting: \" +birdname)\n        for soundfile in os.listdir(in_dir + birdname):\n            path = in_dir + birdname + \"/\" + soundfile\n            sound = AudioSegment.from_file(path)\n            sound.export(out_dir + soundfile[:-3] + \"wav\", format=\"wav\")\n        print(birdname + \" finished after: %s seconds\" % (time.time() - start_time))\n    print(\"worker finished\\n\")\n\nnames = os.listdir(in_dir)\n\nnum_workers = 4\nend = len(names)\nto_each_worker = round(end / num_workers)\n\n\nfor i in range(num_workers+1):\n    args = (names[i * to_each_worker: i* to_each_worker + to_each_worker],)\n\n    p = multiprocessing.Process(target=bird_to_wav, args=(args))\n    p.start()\n\nprint(\"all jobs started\")\n</code></pre>\n\n<p>```</p>\n\n<p>The resulting \"out_dir\" will DIRECTLY (no bird-wise directories, <code>train.csv</code> contains the file names anyway) contain all audio recordings in .wav format and take up around 141 GB in size. Maybe it helps somebody.</p>\n\n<p>You'll have to install ffmpeg to make it work though.</p>",
      "rawMarkdown": "The following code can be used to convert the entire dataset into .wav (which is faster to load and works with more libraries, but is huuuuuuge). It takes ~1100 seconds to complete.\n\n```\nimport time\nimport os\nimport multiprocessing\n\nfrom pydub import AudioSegment\n\ndef to_wav(in_dir, out_dir, num_workers=20):\n    start_time = time.time()\n\n    def bird_to_wav(names):\n        for birdname in names:\n            print(\"starting: \" +birdname)\n            for soundfile in os.listdir(in_dir + birdname):\n                path = in_dir + birdname + \"/\" + soundfile\n                sound = AudioSegment.from_file(path)\n                sound.export(out_dir + soundfile[:-3] + \"wav\", format=\"wav\")\n            print(birdname + \" finished after: %s seconds\" % (time.time() - start_time))\n        print(\"worker finished\\n\")\n        \n    names = os.listdir(in_dir)\n    \n    num_workers = 4\n    end = len(names)\n    to_each_worker = round(end / num_workers)\n\n\n    for i in range(num_workers+1):\n        args = (names[i * to_each_worker: i* to_each_worker + to_each_worker],)\n\n        p = multiprocessing.Process(target=bird_to_wav, args=(args))\n        p.start()\n    \n    print(\"all jobs started\")\n\n```\n\nThe resulting \"out_dir\" will DIRECTLY (no bird-wise directories, `train.csv` contains the file names anyway) contain all audio recordings in .wav format and take up around 141 GB in size. Maybe it helps somebody.\n\nYou'll have to install ffmpeg to make it work though.",
      "votes": null
    },
    {
      "id": "891489",
      "postDate": "06/18/2020 07:57:39",
      "content": "<p>If you wanted to work with WAV files for training your model, would you need to also work with WAV files for predicting on the test set (which would mean you'd have to convert and store those from the test set) ?</p>",
      "rawMarkdown": "If you wanted to work with WAV files for training your model, would you need to also work with WAV files for predicting on the test set (which would mean you'd have to convert and store those from the test set) ?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 891489,
      "author_name": "cwthompson",
      "author_url": "",
      "post_date": "06/18/2020 07:57:39",
      "content": "<p>If you wanted to work with WAV files for training your model, would you need to also work with WAV files for predicting on the test set (which would mean you'd have to convert and store those from the test set) ?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "890211": "The following code can be used to convert the entire dataset into .wav (which is faster to load and works with more libraries, but is huuuuuuge). It takes ~1100 seconds to complete.\n\n```\nimport time\nimport os\nimport multiprocessing\n\nfrom pydub import AudioSegment\n\ndef to_wav(in_dir, out_dir, num_workers=20):\n    start_time = time.time()\n\n    def bird_to_wav(names):\n        for birdname in names:\n            print(\"starting: \" +birdname)\n            for soundfile in os.listdir(in_dir + birdname):\n                path = in_dir + birdname + \"/\" + soundfile\n                sound = AudioSegment.from_file(path)\n                sound.export(out_dir + soundfile[:-3] + \"wav\", format=\"wav\")\n            print(birdname + \" finished after: %s seconds\" % (time.time() - start_time))\n        print(\"worker finished\\n\")\n        \n    names = os.listdir(in_dir)\n    \n    num_workers = 4\n    end = len(names)\n    to_each_worker = round(end / num_workers)\n\n\n    for i in range(num_workers+1):\n        args = (names[i * to_each_worker: i* to_each_worker + to_each_worker],)\n\n        p = multiprocessing.Process(target=bird_to_wav, args=(args))\n        p.start()\n    \n    print(\"all jobs started\")\n\n```\n\nThe resulting \"out_dir\" will DIRECTLY (no bird-wise directories, `train.csv` contains the file names anyway) contain all audio recordings in .wav format and take up around 141 GB in size. Maybe it helps somebody.\n\nYou'll have to install ffmpeg to make it work though.",
    "891489": "If you wanted to work with WAV files for training your model, would you need to also work with WAV files for predicting on the test set (which would mean you'd have to convert and store those from the test set) ?"
  },
  "source": "meta"
}