{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n* **Note:** Turn on your **GPU**.\n* In this kernel we will generate new bird sounds using **simple GANs**.\n\n### If you do not know **simple GANs** you can check this kernel:\n\n
\n
\n
\n
\n
Fashion Dataset GANs
\n \"...\"
\n

A kernel about generating clothes using traditional GANs

\n Go to Post\n
\n
\n
\n
\n\n\n
\n\n## Content\n* We are using [Cornell Birdcall Identification](https://www.kaggle.com/c/birdsong-recognition) dataset.\n\n* [Imports](#1)\n* [Data Analysis](#2)\n* [Audio Generation with GANs](#3)\n * [Preparing Data](#3.1)\n * [Generator](#3.2)\n * [Discriminator](#3.3)\n * [GANs](#3.4)\n * [Training](#3.5)\n * [Results](#3.6)\n* [Conclusion](#4)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# Imports ","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# data science\nimport numpy as np \nimport pandas as pd \nimport librosa\nimport matplotlib.pyplot as plt\nimport IPython\nimport IPython.display as ipd\nimport warnings\nwarnings.filterwarnings('ignore')\nimport glob\nimport time\n\n# keras (gan)\nfrom keras.layers import Dense, Dropout, Input, ReLU\nfrom keras.models import Model, Sequential\nfrom keras.optimizers import Adam\nfrom sklearn.model_selection import train_test_split\nfrom keras import models\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Data Analysis","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"_, sr = librosa.load(\"/kaggle/input/birdsong-recognition/example_test_audio/ORANGE-7-CAP_20190606_093000.pt623.mp3\")\nsr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = []\n\nfor path in glob.glob(\"../input/birdsong-recognition/train_audio/aldfly/*\"):\n file, _ = librosa.core.load(path, sr=sr)\n data.append(file)\n \ndata = np.array(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_samples = [data[0], data[2], data[10], data[20], data[42], data[50]] \n\nfor audio in audio_samples:\n IPython.display.display(ipd.Audio(data = audio, rate = sr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"length_of_audio_samples = len(audio_samples)\nfig, ax = plt.subplots(int(length_of_audio_samples/2),2, figsize = (16,length_of_audio_samples*3)) \n \n# drwing plots\ncount = 0\nfor r in range(0,int(length_of_audio_samples/2)):\n for c in range(0,2):\n sample = audio_samples[count]\n ax[r,c].plot(sample)\n count += 1\n ax[r,c].set_xlabel(\"Samples\")\n \nfig.suptitle(\"Audio Samples\", fontsize = 17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"length_of_audio_samples = len(audio_samples)\nfig, ax = plt.subplots(int(length_of_audio_samples/2),2, figsize = (16,length_of_audio_samples*3)) \n \n# drwing plots\ncount = 0\nfor r in range(0,int(length_of_audio_samples/2)):\n for c in range(0,2):\n sample = audio_samples[count]\n ax[r,c].plot(sample[:500])\n ax[r,c].plot(sample[:500], \".\", color = \"red\")\n count += 1\n ax[r,c].set_xlabel(\"Audio Samples\")\n \nfig.suptitle(\"zoomed Audio Samples\", fontsize = 17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"length_of_audio_samples = len(audio_samples)\nfig, ax = plt.subplots(int(length_of_audio_samples/2),2, figsize = (16,length_of_audio_samples*3)) \n \n# drwing plots\ncount = 0\nfor r in range(0,int(length_of_audio_samples/2)):\n for c in range(0,2):\n sample = audio_samples[count]\n fft = np.fft.fft(sample)\n\n magnitude = np.abs(fft)\n freq = np.linspace(0, sr, len(magnitude))\n\n left_freq = freq[:int(len(freq)/2)]\n left_magnitude = magnitude[:int(len(magnitude)/2)]\n\n ax[r,c].plot(left_freq, left_magnitude)\n ax[r,c].set_xlabel(\"Frequency\")\n ax[r,c].set_ylabel(\"Magnitude\") \n count += 1\n\nfig.suptitle(\"FFTs of Audio Samples \", fontsize = 17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Audio Generation with GANs","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n## Preparing Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = \"../input/birdsong-recognition/train_audio/\"\nBIRD = \"aldfly\"\nSR = 22050\n\ndef load_data_gan(path, bird, sr, length = 6):\n data = []\n\n for path in glob.glob(path + bird + \"/*\"):\n file, _ = librosa.core.load(path, sr=sr)\n \n # all files must be at the same length\n if len(file) > sr*length:\n data.append(file[:sr*length])\n \n else:\n continue\n\n return np.array(data)\n\ndef normalization(X):\n mean = X.mean(keepdims=True)\n std = X.std(keepdims=True)\n X = (X - mean) / std\n print(\"Data Normalized...\")\n return X\n\ndef rescale(X, rangeMin=-1, rangeMax=+1):\n maxi = X.max()\n mini = X.min()\n X = np.interp(X, (mini, maxi), (rangeMin, rangeMax))\n print(\"Data Rescaled...\")\n return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = load_data_gan(PATH, BIRD, SR, 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = normalization(x_train)\nx_train = rescale(x_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IPython.display.display(ipd.Audio(data = x_train[50], rate = 22050))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## Generator\n* **Our Generator's diagram:**\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_generator():\n \n generator = Sequential()\n generator.add(Dense(units = 512, input_dim = 100))\n generator.add(ReLU())\n \n generator.add(Dense(units = 512))\n generator.add(ReLU())\n \n generator.add(Dense(units = 1024))\n generator.add(ReLU())\n \n generator.add(Dense(units = 1024))\n generator.add(ReLU())\n \n generator.add(Dense(units = x_train.shape[1]))\n \n generator.compile(loss =\"binary_crossentropy\",\n optimizer = Adam(lr = 0.0001, beta_1 = 0.5))\n \n return generator\n\ng = create_generator()\ng.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## Discriminator\n* **Our Discriminator's diagram:**\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_discriminator():\n discriminator = Sequential()\n discriminator.add(Dense(units = 1024,input_dim = x_train.shape[1])) \n discriminator.add(ReLU())\n discriminator.add(Dropout(0.4))\n \n discriminator.add(Dense(units = 512)) \n discriminator.add(ReLU())\n discriminator.add(Dropout(0.4))\n \n discriminator.add(Dense(units = 256)) \n discriminator.add(ReLU())\n \n discriminator.add(Dense(units = 1, activation = \"sigmoid\"))\n \n discriminator.compile(loss = \"binary_crossentropy\",\n optimizer = Adam(lr = 0.0001, beta_1 = 0.5))\n return discriminator\n\nd = create_discriminator()\nd.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## GANs\n* **Our Simple GANs' diagram:**\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_gan(discriminator, generator):\n discriminator.trainable = False\n gan_input = Input(shape=(100,))\n x = generator(gan_input)\n gan_output = discriminator(x)\n gan = Model(inputs = gan_input, outputs = gan_output)\n gan.compile(loss = \"binary_crossentropy\", optimizer = \"adam\")\n return gan\n\ngan = create_gan(d,g)\ngan.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## Training\n* In trainig each epoch is a round between Generator and Discriminator. First we are giving a noise to our Generator. Then giving real data and generated data to Discriminator labeling real data as 1 generated as 0. After that we are turning off trainable parameter of Discriminator, generating new audios and labeling them as 1 and train GANs on these audios.\n* **Our training diagram:**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_gen_samples(epochs, samples = 3):\n samplePlot = []\n fig = plt.figure(figsize = (1, samples))\n noise = np.random.normal(0, 1, (samples,100))\n audios = g.predict(noise) \n for i, audio in enumerate(audios):\n IPython.display.display(ipd.Audio(data = audio, rate = SR))\n samplePlot.append(fig.add_subplot(1, samples, i+1))\n samplePlot[i].plot(audio.flatten(), '-', )\n plt.gcf().set_size_inches(25, 5)\n plt.subplots_adjust(wspace=0.3,hspace=0.3)\n fig.suptitle(\"{} Epochs Result\".format(epochs), fontsize = 17)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\nD_loss = []\nG_loss = []\nepochs = 40 \nbatch_size = 16\ncurrent_time = time.time()\n\nfor e in range(epochs):\n start_time = time.time()\n for _ in range(batch_size):\n # I reccomend you to look \"Training Diagram\" (at the top) \n noise = np.random.normal(0,1, [batch_size,100])\n \n generated_audio = g.predict(noise)\n \n audio_batch = x_train[np.random.randint(low = 0, high = x_train.shape[0], size = batch_size)] #get samples from real data\n \n x = np.concatenate([audio_batch, generated_audio])\n \n y_dis = np.zeros(batch_size*2) \n y_dis[:batch_size] = 1 # we labeled real audios as 1 and generated audios as 0\n \n d.trainable = True\n d_loss = d.train_on_batch(x,y_dis) # we are training discriminator (train_on_batch)\n \n noise = np.random.normal(0,1,[batch_size,100])\n \n y_gen = np.ones(batch_size) # our generator says \"these audios are real\"\n \n d.trainable = False\n \n g_loss = gan.train_on_batch(noise, y_gen) #train_on_batch\n \n D_loss.append(d_loss)\n G_loss.append(g_loss)\n \n if (e%2 == 0) or (e == epochs-1) :\n print(\"epochs: \",e)\n if e == epochs-1:\n print(\"Time since start: {}\".format(np.round(start_time - current_time)))\n print(\"Training Complete.\")\n \n # printing results\n if e%10 == 0:\n print(\"Time since start: {}\".format(np.round(start_time - current_time)))\n show_gen_samples(e)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"g.save(\"bird_sound_generator.h5\") # you can donwload this model from output part of this kernel.\nd.save(\"bird_sound_discriminator.h5\") # you can donwload this model from output part of this kernel.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## Results","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"index = 0\nD_loss2 = []\nG_loss2 = []\n\nfor i in range(1,epochs+1):\n D_loss2.append(D_loss[index])\n G_loss2.append(G_loss[index])\n index += batch_size\n\nfig, axs = plt.subplots(2, 1, figsize = (12,10))\n\naxs[0].plot(D_loss2, label = \"Discriminator Loss\")\naxs[1].plot(G_loss2, label = \"Generator Loss\", color = \"orange\")\n\naxs[0].set_xlabel(\"Epochs\")\naxs[1].set_xlabel(\"Epochs\")\n\naxs[0].set_ylabel(\"Loss\")\naxs[1].set_ylabel(\"Loss\")\n\naxs[0].set_title(\"Discriminator Loss\")\naxs[1].set_title(\"Generator Loss\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gen_samples(epochs = epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Conclusion\n* **You can download this model from Output part of the kernel.**\n\n### You can check my other kernels here: https://www.kaggle.com/mrhippo/notebooks\n\n* **My other GANs projects**:\n* https://www.kaggle.com/mrhippo/fashion-dataset-gans\n* https://www.kaggle.com/mrhippo/abstract-art-dcgans\n* https://www.kaggle.com/mrhippo/audio-generation-with-simple-gans\n\n\n* **If there is something wrong with this kernel please let me know in the comments.**\n\n\n* **References:**\n* https://github.com/rafalencar1997/Audio_Generation","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}