{"cells":[{"metadata":{},"cell_type":"markdown","source":"

CLEF: InDepth EDA for Competition

\n\n
\n\n "},{"metadata":{},"cell_type":"markdown","source":"The [Cornell Lab of Ornithology](https://www.birds.cornell.edu/home) brings together the agility and impact of an on-the-ground nonprofit organization with world-class science and teaching as part of Cornell University’s College of Agriculture and Life Sciences. Their work spans disciplines from science to art, engineering to education. Their global community includes supporters, participants, and partners from all walks of life, united by a love of birds and nature and a commitment to help protect our planet."},{"metadata":{},"cell_type":"markdown","source":"
\n

Contents

"},{"metadata":{},"cell_type":"markdown","source":"1. [Competition Overview](#competition-overview)\n2. [Preliminaries](#preliminaries)\n3. [Load Datasets](#load-datasets)\n4. [Tabular Exploration](#tabular-exploration)\n5. [Label Exploration](#label-exploration)\n6. [Name Exploration](#name-exploration)\n7. [Rating and Type Exploration](#rating-and-type-exploration)\n8. [References](#references)"},{"metadata":{},"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation.
This notebook will be updated frequently so keep checking for furthur developments.

"},{"metadata":{},"cell_type":"markdown","source":"#### If you would have any doubts or need guidance in Data Science you can reach out to me on [LinkedIn](https://www.linkedin.com/in/ishandutta0098)"},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Competition Overview

"},{"metadata":{},"cell_type":"markdown","source":"### Description\nRecent advances in **machine listening** have improved acoustic data collection. However, it remains a challenge to generate analysis outputs with high precision and recall. The majority of data is unexamined due to a lack of effective tools for efficient and reliable extraction of the signals of interests (e.g., bird calls).\n\nThe **Cornell Lab of Ornithology** is dedicated to advancing the understanding and protection of birds and the natural world. The Lab joins with people from all walks of life to make new scientific discoveries, share insights, and galvanize conservation action. For this competition, they're collaborating with **Google Research, LifeCLEF, and Xeno-canto**.\n\nIn this competition, you’ll **automate** the **acoustic identification of birds** in soundscape recordings. You'll examine an acoustic dataset to build detectors and classifiers to extract the signals of interest (bird calls). Innovative solutions will be able to do so efficiently and reliably."},{"metadata":{},"cell_type":"markdown","source":"### Evaluation\nSubmissions will be evaluated based on their row-wise micro averaged [F1 score](https://en.wikipedia.org/wiki/F1_score).\n\nFor each row_id/time window, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window. If there are no bird calls in a time window, use the code `nocall`."},{"metadata":{},"cell_type":"markdown","source":"### Code Requirements\n\n \n\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit\" button to be active after a commit, the following conditions must be met:\n\n- **CPU Notebook** <= 9 hours run-time\n- **GPU Notebook** <= 3 hours run-time\n- **Internet** access disabled\n- Freely & publicly available **external data** is allowed, including pre-trained models\n- **Submission file** must be named submission.csv"},{"metadata":{},"cell_type":"markdown","source":"### Data Description\nYour challenge in this competition is to identify which birds are calling in long recordings, given training data generated in meaningfully different contexts. This is the exact problem facing scientists trying to automate the remote monitoring of bird populations. This competition builds on the previous one by adding soundscapes from new locations, more bird species, richer metadata about the test set recordings, and soundscapes to the train set.\n\n### Files\n**train_short_audio** - The bulk of the training data consists of short recordings of individual bird calls generously uploaded by users of xenocanto.org. These files have been downsampled to 32 kHz where applicable to match the test set audio and converted to the ogg format. The training data should have nearly all relevant files; we expect there is no benefit to looking for more on [xenocanto.org](https://www.xeno-canto.org/).\n\n**train_soundscapes** - Audio files that are quite comparable to the test set. They are all roughly ten minutes long and in the ogg format. The test set also has soundscapes from the two recording locations represented here.\n\n**test_soundscapes** - When you submit a notebook, the **test_soundscapes** directory will be populated with approximately 80 recordings to be used for scoring. These will be roughly 10 minutes long and in ogg audio format. The file names include the date the recording was taken, which can be especially useful for identifying migratory birds.\n\nThis folder also contains text files with the name and approximate coordinates of the recording location plus a csv with the set of dates the test set soundscapes were recorded.\n\n**test.csv** - Only the first three rows are available for download; the full test.csv is in the hidden test set.\n\n- `row_id`: ID code for the row.\n\n- `site`: Site ID.\n\n- `seconds`: the second ending the time window\n\n- `audio_id`: ID code for the audio file.\n\n**train_metadata.csv** - A wide range of metadata is provided for the training data. The most directly relevant fields are:\n\n- `ebird_code`: a code for the bird species. You can review detailed information about the bird codes by appending the code to `https://ebird.org/species/`, such as `https://ebird.org/species/amecro` for the American Crow.\n\n- `recodist`: the user who provided the recording.\n\n- `site`: where the recording was taken. Some bird species may have local call 'dialects,' so you may want to seek geographic diversity in your training data.\n\n- `date`: while some bird calls can be made year round, such as an alarm call, some are restricted to a specific season. You may want to seek temporal diversity in your training data.\n\n- `filename`: the name of the associated audio file.\n\n**train_soundscape_labels.csv** -\n\n- `row_id`: ID code for the row.\n\n- `site`: Site ID.\n\n- `seconds`: the second ending the time window\n\n- `audio_id`: ID code for the audio file.\n\n- `birds`: space delimited list of any bird songs present in the 5 second window. The label nocall means that no call occurred.\n\n**sample_submission.csv** - A properly formed sample submission file. Only the first three rows are public, the remainder will be provided to your notebook as part of the hidden test set.\n\n- `row_id`\n\n- `birds`: space delimited list of any bird songs present in the 5 second window. If there are no bird calls, use the label nocall."},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Preliminaries

"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport os\n\n# Data Visualization\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom matplotlib.offsetbox import AnnotationBbox, OffsetImage\n\n\nimport seaborn as sns\nsns.set_style(\"darkgrid\")\n\n# Map 1 library\nimport plotly.express as px\n\n# Map 2 libraries\nimport descartes\nimport geopandas as gpd\nfrom shapely.geometry import Point, Polygon\n\nimport cv2\nfrom wordcloud import WordCloud, STOPWORDS\n\n#Text Color\nfrom termcolor import colored\n\n# Librosa Libraries\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\n\n#Data Preprocessing\nimport sklearn\n\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV\n\n#NLP\nfrom sklearn.feature_extraction.text import CountVectorizer\n\n#WordCloud\nfrom wordcloud import WordCloud, STOPWORDS\n\n#Text Processing\nimport re\nimport nltk\nnltk.download('popular')\n\n#Language Detection\n!pip install langdetect\nimport langdetect\n\n#Sentiment\nfrom textblob import TextBlob\n\n#ner\nimport spacy\n\n#Vectorizer\nfrom sklearn import feature_extraction, manifold\n\n#Word Embedding\nimport gensim.downloader as gensim_api\n\n#Topic Modeling\nimport gensim\n\n# HTML\nfrom IPython.core.display import HTML\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Load Datasets

"},{"metadata":{},"cell_type":"markdown","source":"We begin with loading the dataset to work upon."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/birdclef-2021/train_metadata.csv\")\ntrain_labels = pd.read_csv(\"../input/birdclef-2021/train_soundscape_labels.csv\")\n\ntest = pd.read_csv('../input/birdclef-2021/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Tabular Exploration

"},{"metadata":{},"cell_type":"markdown","source":"### Dataset Head and Info"},{"metadata":{},"cell_type":"markdown","source":"Let us quickly have a brief look upon the dataset. Hope to not encounter any NULL values!"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Aha! There are no NULL values, so we are good to go."},{"metadata":{},"cell_type":"markdown","source":"### Dataset Size"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Training Dataset Shape: {colored(train.shape, 'yellow')}\")\nprint(f\"Training Dataset Labels Shape: {colored(train_labels.shape, 'yellow')}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Column-wise Unique Values"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Data: train\")\nprint(\"-----------\")\nfor col in train.columns:\n print(col + \":\" + colored(str(len(train[col].unique())), 'yellow'))\n\nprint(\"\\nData: train_labels\")\nprint(\"-----------\")\nfor col in train_labels.columns:\n print(col + \":\" + colored(str(len(train_labels[col].unique())), 'yellow'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Label Exploration

"},{"metadata":{},"cell_type":"markdown","source":"### Value Counts and DIstribution Plot"},{"metadata":{},"cell_type":"markdown","source":"A good way to understand the distribution of data in a column is by plotting a distribution plot. We write a function `plot_distribution` which will save us a lot of time."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_distribution(x, title):\n \n \"\"\"\n Function to obtain the distribution plot of given data.\n \n params: x(string) : Name of the Column for the Plot.\n title(string) : Title of the Plot\n \"\"\"\n sns.displot(x)\n\n plt.title(title, fontsize = 15)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_list = [(train.primary_label.value_counts().values, \"Primary Label: Count Distribution\")]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for count, title in label_list:\n plot_distribution(count, title)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Name Exploration

"},{"metadata":{},"cell_type":"markdown","source":"### Value Counts and Distribution Plot"},{"metadata":{"trusted":true},"cell_type":"code","source":"name_list = [(train.scientific_name.value_counts().values, \"Scientific Names: Count Distribution\"),\n (train.common_name.value_counts().values, \"Common Name: Count Distribution\"),\n ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for count, title in name_list:\n plot_distribution(count, title)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### If you would have any doubts or need guidance in Data Science you can reach out to me on [LinkedIn](https://www.linkedin.com/in/ishandutta0098)"},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Rating and Type Exploration

"},{"metadata":{},"cell_type":"markdown","source":"## Rating\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"The Ratings of songs are in range {colored(train.rating.min(), 'yellow')} upto {colored(train.rating.max(), 'yellow')}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have the range, let's see if we have some great bird singers with us!"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.countplot(train['rating'], palette=\"hls\", order = train['rating'].value_counts().index)\n\nplt.title(\"Rating of Song\", fontsize = 16)\nplt.xticks(fontsize = 13)\nplt.yticks(fontsize = 13)\nplt.ylabel(\"Frequency\", fontsize = 14)\nplt.xlabel(\"Rating\", fontsize = 14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Woww! So we have pretty decent bird singers I guess :-)"},{"metadata":{},"cell_type":"markdown","source":"## Type\n\n"},{"metadata":{},"cell_type":"markdown","source":"We can't directly plot the `type` column as multiple types are mixed together. One of the examples is shown below."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.type[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we find 2 problems, first is the **Square Brackets** i.e., **[]**, and second is multiple types placed together. We will deal with both these problems individually."},{"metadata":{},"cell_type":"markdown","source":"Firstly, to remove the square brackets, we simply use the python `replace` function which replaces a given input with desired input. So here we are removing the brackets one by one and replacing it wo no spaces (literally nothing!!)."},{"metadata":{"trusted":true},"cell_type":"code","source":"adjusted_type = train['type'].apply(lambda x: x.replace('[', ''))\nadjusted_type = adjusted_type.apply(lambda x: x.replace(']', ''))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we want to plot the types individually and guess what Python has got us covered! The function `explode` will help us with our goal."},{"metadata":{"trusted":true},"cell_type":"code","source":"adjusted_type = adjusted_type.apply(lambda x: x.split(',')).reset_index().explode(\"type\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Strip of white spaces and convert to lower chars\nadjusted_type = adjusted_type['type'].apply(lambda x: x.strip().lower()).reset_index()\nadjusted_type['type'] = adjusted_type['type'].replace('calls', 'call')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create Top 8 list with song types\ntop_8 = list(adjusted_type['type'].value_counts().head(8).reset_index()['index'])\ndata = adjusted_type[adjusted_type['type'].isin(top_8)]\n\nplt.figure(figsize=(16, 6))\nax = sns.countplot(data['type'], palette=\"hls\", order = data['type'].value_counts().index)\n\nplt.title(\"Top 8 Song Types\", fontsize=16)\nplt.ylabel(\"Frequency\", fontsize = 14)\nplt.yticks(fontsize = 13) \nplt.xticks(rotation = 45, fontsize = 13) \nplt.xlabel(\"Type\", fontsize = 14);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
\n

References

"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"- [Birdcall Recognition: EDA and Audio FE](https://www.kaggle.com/andradaolteanu/birdcall-recognition-eda-and-audio-fe)"},{"metadata":{},"cell_type":"markdown","source":"
\n

This is just the beginning!
\n
More exciting Plots and Analysis on the way!

"},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}