{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n* **Note: Turn on your GPU.**\n* In this kernel we will generate new audio using **simple GANs**. \n* Our instrument will be **Saxophone**.\n\n### If you do not know **simple GANs** you can check this kernel: \n\n
\n
\n
\n
\n
Fashion Dataset GANs
\n \"...\"
\n

A kernel about generating clothes using traditional GANs

\n Go to Post\n
\n
\n
\n
\n\n\n
\n\n\n## Content\n* We are using [Freesound General-Purpose Audio Tagging Challenge](https://www.kaggle.com/c/freesound-audio-tagging/data) dataset.\n\n\n* [Imports](#1)\n* [Preparing the Data](#2)\n* [Data Exploration](#3)\n* [Bulding GANs Model](#4)\n* [Training](#5)\n* [Results](#6)\n* [Conclusion](#7)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# Imports","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# data science and mathematical operations\nimport numpy as np\nimport pandas as pd \nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom scipy.io import wavfile\nimport librosa\nimport IPython\nimport IPython.display as ipd \n\n# deep learning\nfrom keras.layers import Dense, Dropout, Input, ReLU\nfrom keras.models import Model, Sequential\nfrom keras.optimizers import Adam\nfrom sklearn.model_selection import train_test_split\nfrom keras import models\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Preparng the Data ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Audio Config\nDURATION = 4\nSAMPLE_RATE = 16000\nAUDIO_SHAPE = SAMPLE_RATE*DURATION\n\nMFCC = 40\n\n# Paths\nDATASET_PATH = \"../input/freesound-audio-tagging/\"\n\n# Load \ndef load_train_data(label, input_length=AUDIO_SHAPE):\n train = pd.read_csv(DATASET_PATH + \"train.csv\")\n if label != None:\n train_list = train.loc[train.label == label]\n else: \n train_list = train\n cur_batch_size = len(train_list)\n train_fname_list = train_list.fname\n X = np.empty((cur_batch_size, input_length))\n for i, train_fname in enumerate(train_fname_list):\n file_path = DATASET_PATH + \"audio_train/\" + train_fname\n \n # Read and Resample the audio\n data, _ = librosa.core.load(file_path, sr=SAMPLE_RATE, res_type='kaiser_fast')\n\n # Random offset / Padding\n if len(data) > input_length:\n max_offset = len(data) - input_length\n offset = np.random.randint(max_offset)\n data = data[offset:(input_length+offset)]\n else:\n if input_length > len(data):\n max_offset = input_length - len(data)\n offset = np.random.randint(max_offset)\n else:\n offset = 0\n data = np.pad(data, (offset, input_length - len(data) - offset), \"constant\")\n X[i,] = data\n print(\"Data Loaded...\")\n return X\n\n# Stardize Data \ndef normalization(X):\n mean = X.mean(keepdims=True)\n std = X.std(keepdims=True)\n X = (X - mean) / std\n print(\"Data Normalized...\")\n return X\n\n# Rescale Data to be in range [rangeMin, rangeMax]\ndef rescale(X, rangeMin=-1, rangeMax=+1):\n maxi = X.max()\n mini = X.min()\n X = np.interp(X, (mini, maxi), (rangeMin, rangeMax))\n print(\"Data Rescaled...\")\n return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = normalization(load_train_data(\"Saxophone\", AUDIO_SHAPE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = rescale(train_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"asd","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Data Exploration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_samples = [x_train[0], x_train[2], x_train[10], x_train[20], x_train[42], x_train[100]] \n\nfor audio in audio_samples:\n IPython.display.display(ipd.Audio(data = audio, rate = SAMPLE_RATE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"length_of_audio_samples = len(audio_samples)\nfig, ax = plt.subplots(int(length_of_audio_samples/2),2, figsize = (16,length_of_audio_samples*3)) \n \n# drwing plots\ncount = 0\nfor r in range(0,int(length_of_audio_samples/2)):\n for c in range(0,2):\n sample = audio_samples[count]\n ax[r,c].plot(sample)\n count += 1\n ax[r,c].set_xlabel(\"Samples\")\n \nfig.suptitle(\"Saxophone Samples\", fontsize = 17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"length_of_audio_samples = len(audio_samples)\nfig, ax = plt.subplots(int(length_of_audio_samples/2),2, figsize = (16,length_of_audio_samples*2)) \n \n# drwing plots\ncount = 0\nfor r in range(0,int(length_of_audio_samples/2)):\n for c in range(0,2):\n sample = audio_samples[count]\n mfcc = librosa.feature.mfcc(sample, sr = SAMPLE_RATE, n_mfcc=40)\n ax[r,c].imshow(mfcc, cmap='hot', interpolation='nearest');\n count += 1\n \nfig.suptitle(\"Saxophone Samples MFCCs\", fontsize = 17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Bulding GANs Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Generator\n\n* **Our Generator's diagram:**\n\n ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_generator():\n \n generator = Sequential()\n generator.add(Dense(units = 512, input_dim = 100))\n generator.add(ReLU())\n \n generator.add(Dense(units = 512))\n generator.add(ReLU())\n \n generator.add(Dense(units = 1024))\n generator.add(ReLU())\n \n generator.add(Dense(units = 1024))\n generator.add(ReLU())\n \n generator.add(Dense(units = 64000))\n \n generator.compile(loss =\"binary_crossentropy\",\n optimizer = Adam(lr = 0.0001, beta_1 = 0.5))\n \n return generator\n\ng = create_generator()\ng.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Discriminator\n\n* **Our Discriminator's diagram:**\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_discriminator():\n discriminator = Sequential()\n discriminator.add(Dense(units = 1024,input_dim = 64000)) \n discriminator.add(ReLU())\n discriminator.add(Dropout(0.4))\n \n discriminator.add(Dense(units = 512)) \n discriminator.add(ReLU())\n discriminator.add(Dropout(0.4))\n \n discriminator.add(Dense(units = 256)) \n discriminator.add(ReLU())\n \n discriminator.add(Dense(units = 1, activation = \"sigmoid\"))\n \n discriminator.compile(loss = \"binary_crossentropy\",\n optimizer = Adam(lr = 0.0001, beta_1 = 0.5))\n return discriminator\n\nd = create_discriminator()\nd.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Simple GANs\n\n* Our Simple GANs' diagram:\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_gan(discriminator, generator):\n discriminator.trainable = False\n gan_input = Input(shape=(100,))\n x = generator(gan_input)\n gan_output = discriminator(x)\n gan = Model(inputs = gan_input, outputs = gan_output)\n gan.compile(loss = \"binary_crossentropy\", optimizer = \"adam\")\n return gan\n\ngan = create_gan(d,g)\ngan.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_gen_samples(epochs, samples = 3):\n samplePlot = []\n fig = plt.figure(figsize = (1, samples))\n noise = np.random.normal(0, 1, (samples,100))\n audios = g.predict(noise) \n for i, audio in enumerate(audios):\n IPython.display.display(ipd.Audio(data = audio, rate = SAMPLE_RATE))\n samplePlot.append(fig.add_subplot(1, samples, i+1))\n samplePlot[i].plot(audio.flatten(), '-', )\n plt.gcf().set_size_inches(25, 5)\n plt.subplots_adjust(wspace=0.3,hspace=0.3)\n fig.suptitle(\"{} Epochs Result\".format(epochs), fontsize = 17)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Training\n* In trainig each epoch is a round between Generator and Discriminator. First we are giving a noise to our Generator. Then giving real data and generated data to Discriminator labeling real data as 1 generated as 0. After that we are turning off trainable parameter of Discriminator, generating new images and labeling them as 1 and train DCGAN on these images.\n* **Our training diagram:**\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\nD_loss = []\nG_loss = []\nepochs = 40 \nbatch_size = 128\ncurrent_time = time.time()\n\nfor e in range(epochs):\n start_time = time.time()\n for _ in range(batch_size):\n # I reccomend you to look \"Training Diagram\" (at the top) \n noise = np.random.normal(0,1, [batch_size,100])\n \n generated_audio = g.predict(noise)\n \n audio_batch = x_train[np.random.randint(low = 0, high = x_train.shape[0], size = batch_size)] #get samples from real data\n \n x = np.concatenate([audio_batch, generated_audio])\n \n y_dis = np.zeros(batch_size*2) \n y_dis[:batch_size] = 1 # we labeled real audios as 1 and generated audios as 0\n \n d.trainable = True\n d_loss = d.train_on_batch(x,y_dis) # we are training discriminator (train_on_batch)\n \n noise = np.random.normal(0,1,[batch_size,100])\n \n y_gen = np.ones(batch_size) # our generator says \"these audios are real\"\n \n d.trainable = False\n \n g_loss = gan.train_on_batch(noise, y_gen) #train_on_batch\n \n D_loss.append(d_loss)\n G_loss.append(g_loss)\n \n if (e%2 == 0) or (e == epochs-1) :\n print(\"epochs: \",e)\n if e == epochs-1:\n print(\"Time since start: {}\".format(np.round(start_time - current_time)))\n print(\"Training Complete.\")\n \n # printing results\n if e%10 == 0:\n print(\"Time since start: {}\".format(np.round(start_time - current_time)))\n show_gen_samples(e)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"g.save(\"audio_generator.h5\") # you can donwload this model from output part of this kernel.\nd.save(\"audio_discriminator.h5\") # you can donwload this model from output part of this kernel.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Results","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"index = 0\nD_loss2 = []\nG_loss2 = []\n\nfor i in range(1,epochs+1):\n D_loss2.append(D_loss[index])\n G_loss2.append(G_loss[index])\n index += batch_size\n\nfig, axs = plt.subplots(2, 1, figsize = (12,10))\n\naxs[0].plot(D_loss2, label = \"Discriminator Loss\")\naxs[1].plot(G_loss2, label = \"Generator Loss\", color = \"orange\")\n\naxs[0].set_xlabel(\"Epochs\")\naxs[1].set_xlabel(\"Epochs\")\n\naxs[0].set_ylabel(\"Loss\")\naxs[1].set_ylabel(\"Loss\")\n\naxs[0].set_title(\"Discriminator Loss\")\naxs[1].set_title(\"Generator Loss\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_gen_samples(epochs = epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Denoising with Autoencoder\n* Now, our result is noisy but we can clear that noise with **autoencoders**.\n\n* If you do mot know autoencoders you can check out this notebook: https://www.kaggle.com/mrhippo/denoising-mnist-with-autoencoders","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train_one_second = []\n\nfor file in x_train:\n for x in range(0, 64):\n x_train_one_second.append(file[1000*x:1000*(x+1)])\n \nx_train_one_second = np.array(x_train_one_second)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IPython.display.display(ipd.Audio(data = x_train_one_second[0], rate = SAMPLE_RATE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fake_labels = np.random.randint(0, 10, size = (x_train_one_second.shape[0]))\n\nx_train_autoencoder, x_test_autoencoder, _, _ = train_test_split(x_train_one_second, fake_labels, test_size=0.3, random_state=42, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"noise = 0.2\n\nx_train_autoencoder_noise = x_train_autoencoder + noise*np.random.normal(-1, 1, size = x_train_autoencoder.shape)\nx_test_autoencoder_noise = x_test_autoencoder + noise*np.random.normal(-1, 1, size = x_test_autoencoder.shape)\n\nx_train_autoencoder_noise = np.clip(x_train_autoencoder_noise, -1, 1)\nx_test_autoencoder_noise = np.clip(x_test_autoencoder_noise, -1, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"noisy_samples = [x_train_autoencoder_noise[1], x_train_autoencoder_noise[3], x_train_autoencoder_noise[10]]\n\nfor noisy in noisy_samples:\n IPython.display.display(ipd.Audio(data = noisy, rate = SAMPLE_RATE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Input\ninput_img = Input(shape = (x_train_autoencoder_noise.shape[1], )) \n\n# Encoding\nencoded = Dense(128, activation = \"relu\")(input_img) \nencoded = Dense(64, activation = \"relu\")(encoded) \n\n# Bottleneck\nencoded = Dense(32, activation = \"relu\")(encoded)\n\n# Decoding\ndecoded = Dense(64, activation = \"relu\")(encoded)\ndecoded = Dense(128, activation = \"relu\")(decoded)\n\n# Output\ndecoded = Dense(x_train_autoencoder_noise.shape[1], activation = \"sigmoid\")(decoded)\n\nautoencoder = Model(input_img, decoded) # Autoencoder\n\nautoencoder.compile(loss = \"binary_crossentropy\",\n optimizer = \"rmsprop\") #rmsprop\n\nautoencoder.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist = autoencoder.fit(x_train_autoencoder_noise, x_train_autoencoder, # x_train_autoencoder_noise\n epochs = 5, \n shuffle = True, \n batch_size = 16, \n validation_data = (x_test_autoencoder_noise, x_test_autoencoder)) # x_test_autoencoder_noise","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (15, 6))\nplt.plot(hist.history[\"loss\"], label = \"Train Loss\")\nplt.plot(hist.history[\"val_loss\"], label = \"Test Loss\")\nplt.title(\"Losses\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"denoised = autoencoder.predict(x_test_autoencoder_noise)\nnoisy_samples = [denoised[1], denoised[3], denoised[10]]\n\nfor noisy in noisy_samples:\n IPython.display.display(ipd.Audio(data = noisy, rate = SAMPLE_RATE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xd = [x_test_autoencoder_noise[1], x_test_autoencoder_noise[3], x_test_autoencoder_noise[10]]\n\nfor noisy in xd:\n IPython.display.display(ipd.Audio(data = noisy, rate = SAMPLE_RATE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fft = np.fft.fft(denoised[1])\n\nmagnitude = np.abs(fft)\nfreq = np.linspace(0, 16000, len(magnitude))\n\nplt.figure(figsize = (15, 8))\nplt.plot(freq, magnitude)\nplt.title(\"Fast Fourier Transform\")\nplt.xlabel(\"Frequency\")\nplt.ylabel(\"Magnitude\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fft = np.fft.fft(x_train_autoencoder_noise[1])\n\nmagnitude = np.abs(fft)\nfreq = np.linspace(0, 16000, len(magnitude))\n\nplt.figure(figsize = (15, 8))\nplt.plot(freq, magnitude)\nplt.title(\"Fast Fourier Transform\")\nplt.xlabel(\"Frequency\")\nplt.ylabel(\"Magnitude\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (15, 6))\nplt.plot(denoised[1])\nplt.show()\n\nplt.figure(figsize = (15, 6))\nplt.plot(x_train_autoencoder_noise[1])\nplt.show()\n\nplt.figure(figsize = (15, 6))\nplt.plot(x_train_autoencoder[1])\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (15, 6))\nplt.plot(denoised[1][:500])\nplt.show()\n\nplt.figure(figsize = (15, 6))\nplt.plot(x_train_autoencoder_noise[1][:500])\nplt.show()\n\nplt.figure(figsize = (15, 6))\nplt.plot(x_train_autoencoder[1][:500])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Conclusion\n* **You can download this model from Output part of the kernel.**\n\n### You can check my other kernels here: https://www.kaggle.com/mrhippo/notebooks\n* **My other GANs projects:**\n* https://www.kaggle.com/mrhippo/fashion-dataset-gans\n* https://www.kaggle.com/mrhippo/abstract-art-dcgans\n\n\n* **If there is something wrong with this kernel please let me know in the comments.**\n\n\n* **References:**\n* https://github.com/rafalencar1997/Audio_Generation\n* https://medium.com/neuronio/audio-generation-with-gans-428bc2de5a89","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}