{"cells":[{"metadata":{},"cell_type":"markdown","source":"# musicnn example: use it as music feature extractor\n\n### musicnn pre-trained models can be used for transfer learning or as feature extractors\n\n---------------\n\nThis notebook explains how to use the `musicnn` model as a music feature extractor. `musicnn` allows you to extract features at every layer of the model. For this reason, we didactically present it so that you can understand what to expect out of each layer. To start, let's consider this music clip:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import clear_output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install git+https://github.com/jordipons/musicnn.git\nclear_output()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"file_name = '../input/audios/joram-moments_of_clarity-08-solipsism-59-88.mp3'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run these two lines of code to extract music features with our musicnn model trained with the [MagnaTagATune](https://github.com/keunwoochoi/magnatagatune-list) dataset – the `MTT_musicnn` model:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from musicnn.extractor import extractor\ntaggram, tags, features = extractor(file_name, model='MTT_musicnn', extract_features=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Out of the extractor, we get the **output** of the model (the `taggram` and its associated `tags`) and all the **intermediate representations** of it (we refer to those as `features`). The `features` are packed in a dictionary:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"list(features.keys())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These different key-features correspond to the outputs of the different layers that our musicnn model has. For this reason, it is important that you understand the basic bulding blocks of our model – that we briefly outline in the following diagram:\n\n
\n\n
\n\n>***Remark:*** *the above key-features (e.g., `timbral`, `cnn3`, or `max_pool`) are yellow-highlighed in the figures below.*\n\n-------------------\nThe **musically motivated CNN** front-end is the convolutional layer in charge of processing the log-mel spectrogram input. We employ a musically motivated CNN (outlined in the left figure), that consists of a convolutional layer with several filter shapes having different receptive fields that capture musically relevant contexts (these are depicted in the right figure):\n\n
\n\n
\n\nTimbralCNNs employ vertical filters to capture timbral traces. Temporal CNNs employ horizontal filters to capture long and short temporal dependencies in spectrograms. The `timbral` and `temporal` features might contain information associated to these low-level features.\n\n-------------------\nThe **dense layers** mid-end is in charge of extracting higher-level representations from the low-level features computed by the front-end. It has residual connections that facilitate training, and it has dense connections to allow the back-end to consider information extracted at different hierarchichal levels.\n\n
\n\n
\n\n-------------------\nThe dense **temporal-pooling** back-end part is responsible for predicting tags from the extracted features. And, importantly, it allows for variable-lengths inputs – since the temporal-pooling layers of the back-end would adapt any feature-map length to a fixed feature-map size.\n\n
\n\n
\n\n--------------------\n### How do front-end features look like?\n\nThe output of our front-end is the concatenation of `timbral` and `temporal` features:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nfrontend_features = np.concatenate([features['temporal'], features['timbral']], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's depict how these front-end features look like!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pylab as plt\nimport matplotlib.gridspec as gridspec\n\ndef depict_features(features, coordinates, title, aspect='auto', xlabel=True, fontsize=13):\n # plot features in coordinates\n ax = plt.subplot(coordinates) \n plt.imshow(features.T, interpolation=None, aspect=aspect)\n # set title\n ax.title.set_text(title + ' (' + str(features.shape[1]) + ')' )\n ax.title.set_fontsize(fontsize)\n # y-axis\n ax.get_yaxis().set_visible(False)\n # x-axis\n x_label = np.arange(0, features.shape[0], features.shape[0]//5)\n ax.set_xticks(x_label)\n ax.set_xticklabels(x_label, fontsize=fontsize)\n if xlabel:\n ax.set_xlabel('(time frames)', fontsize=fontsize)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gs = gridspec.GridSpec(1, 1) # create a figure having 1 rows and 1 cols.\ndepict_features(features=features['timbral'],\n coordinates=gs[0, 0],\n title='timbral features',\n aspect='auto')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gs = gridspec.GridSpec(1, 1) # create a figure having 1 rows and 3 cols.\ndepict_features(features=features['temporal'],\n coordinates=gs[0, 0],\n title='temporal features',\n aspect='equal')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gs = gridspec.GridSpec(1, 1) # create a figure having 1 rows and 3 cols.\ndepict_features(features=frontend_features,\n coordinates=gs[0, 0],\n title='front-end features',\n aspect='equal')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The previous figures depict the temporal evolution of the computed features (y-axis: features, x-axis: time).\n\nRemember: the `timbral` features (with 408 features) and the `temporal` features (with 153 features) are concatenated to compose the `front-end` features (with 561 feature maps). These `front-end` features are now fed to mid-end CNN layers.\n\n--------------------\n### How do mid-end CNN features look like?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"gs = gridspec.GridSpec(3, 1) # create a figure having 1 rows and 3 cols.\n\ndepict_features(features=features['cnn1'],\n coordinates=gs[0, 0],\n title='cnn1 features',\n xlabel=False)\n\ndepict_features(features=features['cnn2'],\n coordinates=gs[1, 0],\n title='cnn2 features',\n xlabel=False)\n\ndepict_features(features=features['cnn3'],\n coordinates=gs[2, 0],\n title='cnn3 features')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"----------------\n### How do output features look like?\n\nBy default, the model takes inputs of 3 seconds (with no overlap). Out of these 3 sec audio-patches, the model estimates the likelihood of the tags.\n\nProvided that the back-end contains a temporal-pooling layer (mean-pool and max-pool layers), it summarizes all the temporal content into a single value. For that reason, the output features have a temporal resolution of 3 seconds – because out of each 3 seconds input, we compute a tag and its intermediate representations. \n\nAccordingly, the temporal resolution of the last layers is of 3 seconds – while lower-level features had frame-level temporal resolution.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams[\"figure.figsize\"] = (9,6)\ngs = gridspec.GridSpec(4, 3) # create a figure having 1 rows and 3 cols.\n\ndepict_features(features=features['mean_pool'],\n coordinates=gs[:, 0],\n title='mean-pool features')\n\ndepict_features(features=features['max_pool'],\n coordinates=gs[:, 1],\n title='max-pool features')\n\ndepict_features(features=features['penultimate'],\n coordinates=gs[3, 2],\n title='penultimate-layer features')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally, let's visualize the `taggram` output:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"in_length = 3 # seconds -- by default, the model takes inputs of 3 seconds with no overlap\n\n# depict taggram\nplt.rcParams[\"figure.figsize\"] = (10,8)\nfontsize=12\nfig, ax = plt.subplots()\nax.imshow(taggram.T, interpolation=None, aspect=\"auto\")\n\n# title\nax.title.set_text('Taggram')\nax.title.set_fontsize(fontsize)\n\n# x-axis title\nax.set_xlabel('(seconds)', fontsize=fontsize)\n\n# y-axis\ny_pos = np.arange(len(tags))\nax.set_yticks(y_pos)\nax.set_yticklabels(tags, fontsize=fontsize-1)\n\n# x-axis\nx_pos = np.arange(taggram.shape[0])\nx_label = np.arange(in_length/2, in_length*taggram.shape[0], 3)\nax.set_xticks(x_pos)\nax.set_xticklabels(x_label, fontsize=fontsize)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}