{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":70203,"databundleVersionId":8068726,"sourceType":"competition"}],"dockerImageVersionId":30715,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# About \n\nThe tagline of the competition is: \n\n***Bird species identification from audio, focused on under-studied species in the Western Ghats, a major biodiversity hotspot in India.***\n\n**Hence the goal of the competition is to serve as an alternative method to traditional bird surveys (which are difficult and expensive), by utilizing \"passive acoustic monitoring\" (PAM) and machine learning.**\n\n**This will aslo enable biodiversity researchers to work with a greater geographical and time scales, to study effect of interventions, and changes in the biodiversity.**\n\n\n\n    The broader goals for this Kaggle competition include:\n    \n    (1) Identify endemic bird species of the sky-islands of the Western Ghats in soundscape data.\n    (2) Detect/classify endangered bird species (species of conservation concern) featuring limited training data.\n    (3) Detect/classify nocturnal bird species which are poorly understood.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Data\n\n**TRAIN Dataset**\n\n1) train_audio/\n\n    The training data consists of short recordings of individual bird calls generously uploaded by users of xenocanto.org. These files have been downsampled to 32 kHz where applicable to match the test set audio and converted to the ogg format.\n    \n2) unlabeled_soundscapes/ \n\n    Unlabeled audio data from the same recording locations as the test soundscapes.\n    \n3) train_metadata.csv\n\n    A wide range of metadata is provided for the training data. Most notably , recording location, primary label, author and filename etc. We can use a lot of this metadata for filtering and sampling, as well as to build features.\n    \n4) eBird_Taxonomy_v2021.csv \n\n    Data on the relationships between different species. This can be used smartly to cluster data according to species.\n    \n    \n**TEST Dataset**\n\n1) test_soundscapes/\n\n    When you submit a notebook, the test_soundscapes directory will be populated with approximately 1,100 recordings to be used for scoring. They are 4 minutes long and in ogg audio format.\n    \n2) sample_submission.csv \n\n    A valid sample submission. Denotes the format in which a valid submission should be in.","metadata":{}},{"cell_type":"markdown","source":"# Challenges\n\n1) **The bird songs are often overlapping**\n\n    However, ML-based audio classification of bird species can be challenging for several reasons. For one, birds often sing over one another, especially during the “dawn chorus” when many birds are most active. Also, there aren’t clear recordings of individual birds to learn from — almost all of the available training data is recorded in noisy outdoor conditions, where other sounds from the wind, insects, and other environmental sources are often present. As a result, existing birdsong classification models struggle to identify quiet, distant and overlapping vocalizations. Additionally, some of the most common species often appear unlabeled in the background of training recordings for less common species, leading models to discount the common species. These difficult cases are very important for ecologists who want to identify endangered or invasive species using automated systems.\n    (from : https://research.google/blog/separating-birdsong-in-the-wild-for-classification/)\n    \n    From this paper, two important points to learn:\n    1) Different regions will have different recording patterns. Example, bird calls might be distant in the mountains, compared to other regions.\n    2) The author trained different classifiers based on region\n    3) The audios are split into 5 second segments and mel-spectogram of the segment is used.\n    3) Author trained a model that seperates audio signals where there is overlap","metadata":{}},{"cell_type":"markdown","source":"# Imports ","metadata":{}},{"cell_type":"code","source":"#general\nimport pandas as pd \nimport numpy as np \nimport os \nimport requests\nfrom bs4 import BeautifulSoup as bs\nimport ast \nimport pickle\n\n#geo features\nimport geopandas as gpd\nfrom shapely import geometry\nimport geopy \n\n#plotting\nimport matplotlib.pyplot as plt \nimport seaborn as sns\n\n\n#deep learning \nimport tensorflow as tf \n\n#mapping\nimport folium\nfrom folium.plugins import HeatMap,MarkerCluster\n\n#audio data\nimport librosa\nimport librosa.display as ld\n\n\n#colored print \nfrom termcolor import colored\n    \n    \nfrom IPython.display import Audio","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-05T10:04:13.915885Z","iopub.execute_input":"2024-06-05T10:04:13.916678Z","iopub.status.idle":"2024-06-05T10:04:13.923043Z","shell.execute_reply.started":"2024-06-05T10:04:13.916643Z","shell.execute_reply":"2024-06-05T10:04:13.921711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Get india boundary**","metadata":{}},{"cell_type":"code","source":"world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))\n\ncountry_outline = world[world.name == \"India\"]\n\n\ncountry_outline.plot(figsize=(5,6))\nplt.title(\"india boundary\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-05T07:24:36.882910Z","iopub.execute_input":"2024-06-05T07:24:36.883404Z","iopub.status.idle":"2024-06-05T07:24:37.205489Z","shell.execute_reply.started":"2024-06-05T07:24:36.883366Z","shell.execute_reply":"2024-06-05T07:24:37.204335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Setting random seed**","metadata":{}},{"cell_type":"code","source":"def  color_print(print_str,\n                 print_color='green'):\n    \n    '''print in given  color (default green)'''\n    print(colored(print_str,print_color))\n    \nclass config:\n    def __init__(self,\n                 seed=7):\n        self.seed = seed\n    \n    def set_seed(self):\n        '''set seed for reproduciblity'''\n\n        tf.random.set_seed(self.seed)\n        os.environ['PYTHONHASHSEED'] = str(self.seed)\n        np.random.seed(self.seed)\n        color_print(print_str=f'Setting Random Seed  to {self.seed}',\n                    print_color='yellow')\n\n        \n        \ncfg = config(seed=7)\ncfg.set_seed()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-05T07:24:41.972451Z","iopub.execute_input":"2024-06-05T07:24:41.973415Z","iopub.status.idle":"2024-06-05T07:24:41.981190Z","shell.execute_reply.started":"2024-06-05T07:24:41.973380Z","shell.execute_reply":"2024-06-05T07:24:41.980137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading MD","metadata":{}},{"cell_type":"code","source":"train_dir  =  '../input/birdclef-2024/train_audio/' # train audio\n\n\n# csv files \ntrain_meta = pd.read_csv(\"/kaggle/input/birdclef-2024/train_metadata.csv\",\n                            dtype={\"latittude\":float,\n                                   \"longitude\":float})\ntrain_meta = train_meta.rename(mapper = {\"type\":\"call_Type\"},axis =1)\ntrain_meta['filepath'] = train_dir + train_meta.filename\n\nbird_taxanomy = pd.read_csv('../input/birdclef-2024/eBird_Taxonomy_v2021.csv') \ntrain_meta.shape","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-05T07:24:43.402332Z","iopub.execute_input":"2024-06-05T07:24:43.402749Z","iopub.status.idle":"2024-06-05T07:24:43.654771Z","shell.execute_reply.started":"2024-06-05T07:24:43.402718Z","shell.execute_reply":"2024-06-05T07:24:43.653668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#convert dtype\ntrain_meta.secondary_labels = train_meta.secondary_labels.apply(ast.literal_eval)\ntrain_meta.call_Type = train_meta.call_Type.apply(ast.literal_eval)\n","metadata":{"execution":{"iopub.status.busy":"2024-06-05T07:24:48.422771Z","iopub.execute_input":"2024-06-05T07:24:48.423181Z","iopub.status.idle":"2024-06-05T07:24:49.662517Z","shell.execute_reply.started":"2024-06-05T07:24:48.423149Z","shell.execute_reply":"2024-06-05T07:24:49.661570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_meta.head(2)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T07:24:52.782837Z","iopub.execute_input":"2024-06-05T07:24:52.783334Z","iopub.status.idle":"2024-06-05T07:24:52.807032Z","shell.execute_reply.started":"2024-06-05T07:24:52.783305Z","shell.execute_reply":"2024-06-05T07:24:52.805939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fetching bird information \n\n    We can use the ebird url to fetch bird information. We will scrape the basic bird information using request and beautiful soup","metadata":{}},{"cell_type":"code","source":"\nbird_info_url = \"https://ebird.org/species/\"\n\ndef get_bird_info(html_object)->str:\n    parsed = bs(html_object,'html.parser')\n    \n    #get the relevant bird info using specific tags and its class.\n    bird_info_text = parsed.find(\"div\",\"Species-identification-text\").get_text()\n    \n    return bird_info_text\n\ndef scrape_bird_info(url:str,\n                     bird_species_name:str)->str:\n    \"\"\"get the bird basic information from Ebird site\"\"\"\n    \n    full_url = url+bird_species_name\n    resp = requests.get(full_url,timeout=(1, 3))\n    \n    if resp.status_code == 200:\n        resp_text = resp.text\n        parsed_bird_info = get_bird_info(resp_text)\n        \n        return parsed_bird_info\n    \ndef pickle_obj(obj,path):\n    with open(path,\"wb\") as f:\n        pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)\n        \n    return \n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-05T10:03:53.812489Z","iopub.execute_input":"2024-06-05T10:03:53.813473Z","iopub.status.idle":"2024-06-05T10:03:53.820937Z","shell.execute_reply.started":"2024-06-05T10:03:53.813437Z","shell.execute_reply":"2024-06-05T10:03:53.819663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# get the info for all birds\nall_unique_birds = train_meta.primary_label.unique()\n\n\nBird_info_dict = {i:scrape_bird_info(bird_info_url,i) for i in all_unique_birds} \n#save as pickle\npickle_obj(Bird_info_dict,\"/kaggle/working/bird_info_dict.pkl\")\nBird_info_dict[all_unique_birds[0]]","metadata":{"execution":{"iopub.status.busy":"2024-06-05T10:04:19.787829Z","iopub.execute_input":"2024-06-05T10:04:19.788568Z","iopub.status.idle":"2024-06-05T10:04:19.798523Z","shell.execute_reply.started":"2024-06-05T10:04:19.788513Z","shell.execute_reply":"2024-06-05T10:04:19.797516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Helper function ,For plotting spectograms**\n\n","metadata":{}},{"cell_type":"code","source":"\n    \n#helper to plot spectogram\ndef plot_spectrogram(path,birdname='',sr=None):\n    '''plot spectrogram from given path'''\n    \n    #loading audio\n    signal,sr=librosa.load(path,sr=sr)\n\n    fig,ax=plt.subplots(figsize=(16,6))\n    M = librosa.feature.melspectrogram(y=signal, sr=sr)\n    M_db = librosa.power_to_db(M, ref=np.max)\n    img = ld.specshow(M_db, y_axis='mel', x_axis='time', ax=ax)\n    plt.colorbar(img)\n    ax.set(title=f'Mel spectrogram for {birdname}')","metadata":{"execution":{"iopub.status.busy":"2024-06-05T07:25:07.053076Z","iopub.execute_input":"2024-06-05T07:25:07.053461Z","iopub.status.idle":"2024-06-05T07:25:07.060001Z","shell.execute_reply.started":"2024-06-05T07:25:07.053434Z","shell.execute_reply":"2024-06-05T07:25:07.058852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Lets take a look at some of the audios and thier spectrogram of the bird species in the western ghats","metadata":{}},{"cell_type":"code","source":"print_template = \"\"\"\nName : {species} \n\nInfo : {info}\"\"\"\n\nBird_common_name = \"Malabar Parakeet\"\nbird_species = train_meta[train_meta['common_name']==Bird_common_name]['primary_label'].to_list()[0]\nmessage = print_template.format(species = Bird_common_name,\n                                info = Bird_info_dict.get(bird_species,\"\"))\ncolor_print(message,\"yellow\")\nrow = train_meta[train_meta['common_name']==Bird_common_name].sort_values(by='rating',ascending=False).iloc[0]\nAudio(row['filepath'])\n","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:07:14.543579Z","iopub.execute_input":"2024-06-05T08:07:14.544495Z","iopub.status.idle":"2024-06-05T08:07:14.571500Z","shell.execute_reply.started":"2024-06-05T08:07:14.544454Z","shell.execute_reply":"2024-06-05T08:07:14.570598Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram(path=row['filepath'],birdname=Bird_common_name)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:06:14.273167Z","iopub.execute_input":"2024-06-05T08:06:14.273620Z","iopub.status.idle":"2024-06-05T08:06:33.109803Z","shell.execute_reply.started":"2024-06-05T08:06:14.273587Z","shell.execute_reply":"2024-06-05T08:06:33.108511Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Bird_common_name = \"Great Hornbill\"\nbird_species = train_meta[train_meta['common_name']==Bird_common_name]['primary_label'].to_list()[0]\nmessage = print_template.format(species = Bird_common_name,\n                                info = Bird_info_dict.get(bird_species,\"\"))\ncolor_print(message,\"yellow\")\nrow = train_meta[train_meta['common_name']==Bird_common_name].sort_values(by='rating',ascending=False).iloc[0]\nAudio(row['filepath'])","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:07:52.962834Z","iopub.execute_input":"2024-06-05T08:07:52.963621Z","iopub.status.idle":"2024-06-05T08:07:52.989815Z","shell.execute_reply.started":"2024-06-05T08:07:52.963585Z","shell.execute_reply":"2024-06-05T08:07:52.988830Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram(path=row['filepath'],birdname=Bird_common_name)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:08:10.328932Z","iopub.execute_input":"2024-06-05T08:08:10.329615Z","iopub.status.idle":"2024-06-05T08:08:10.932221Z","shell.execute_reply.started":"2024-06-05T08:08:10.329582Z","shell.execute_reply":"2024-06-05T08:08:10.931050Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nBird_common_name = \"Indian Peafowl\"\nbird_species = train_meta[train_meta['common_name']==Bird_common_name]['primary_label'].to_list()[0]\nmessage = print_template.format(species = Bird_common_name,\n                                info = Bird_info_dict.get(bird_species,\"\"))\ncolor_print(message,\"yellow\")\nrow = train_meta[train_meta['common_name']==Bird_common_name].sort_values(by='rating',ascending=False).iloc[0]\nAudio(row['filepath'])","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:26:58.112854Z","iopub.execute_input":"2024-06-05T08:26:58.113266Z","iopub.status.idle":"2024-06-05T08:26:58.176171Z","shell.execute_reply.started":"2024-06-05T08:26:58.113229Z","shell.execute_reply":"2024-06-05T08:26:58.174691Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram(path=row['filepath'],birdname=Bird_common_name)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:27:10.287640Z","iopub.execute_input":"2024-06-05T08:27:10.288442Z","iopub.status.idle":"2024-06-05T08:27:13.162537Z","shell.execute_reply.started":"2024-06-05T08:27:10.288406Z","shell.execute_reply":"2024-06-05T08:27:13.161312Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Bird_common_name = \"Jungle Owlet\"\nbird_species = train_meta[train_meta['common_name']==Bird_common_name]['primary_label'].to_list()[0]\nmessage = print_template.format(species = Bird_common_name,\n                                info = Bird_info_dict.get(bird_species,\"\"))\ncolor_print(message,\"yellow\")\nrow = train_meta[train_meta['common_name']==Bird_common_name].sort_values(by='rating',ascending=False).iloc[0]\nAudio(row['filepath'])","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:11:34.768119Z","iopub.execute_input":"2024-06-05T08:11:34.768978Z","iopub.status.idle":"2024-06-05T08:11:34.803813Z","shell.execute_reply.started":"2024-06-05T08:11:34.768941Z","shell.execute_reply":"2024-06-05T08:11:34.802677Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram(path=row['filepath'],birdname=Bird_common_name)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:11:46.142986Z","iopub.execute_input":"2024-06-05T08:11:46.143450Z","iopub.status.idle":"2024-06-05T08:11:47.135614Z","shell.execute_reply.started":"2024-06-05T08:11:46.143413Z","shell.execute_reply":"2024-06-05T08:11:47.134614Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nBird_common_name = \"Malabar Barbet\"\nbird_species = train_meta[train_meta['common_name']==Bird_common_name]['primary_label'].to_list()[0]\nmessage = print_template.format(species = Bird_common_name,\n                                info = Bird_info_dict.get(bird_species,\"\"))\ncolor_print(message,\"yellow\")\nrow = train_meta[train_meta['common_name']==Bird_common_name].sort_values(by='rating',ascending=False).iloc[0]\nAudio(row['filepath'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-05T08:21:06.763083Z","iopub.execute_input":"2024-06-05T08:21:06.764048Z","iopub.status.idle":"2024-06-05T08:21:06.801839Z","shell.execute_reply.started":"2024-06-05T08:21:06.764000Z","shell.execute_reply":"2024-06-05T08:21:06.800607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram(path=row['filepath'],birdname=Bird_common_name)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:21:12.928105Z","iopub.execute_input":"2024-06-05T08:21:12.928529Z","iopub.status.idle":"2024-06-05T08:21:13.963787Z","shell.execute_reply.started":"2024-06-05T08:21:12.928497Z","shell.execute_reply":"2024-06-05T08:21:13.962848Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Bird_common_name = \"Flame-throated Bulbul\"\nbird_species = train_meta[train_meta['common_name']==Bird_common_name]['primary_label'].to_list()[0]\nmessage = print_template.format(species = Bird_common_name,\n                                info = Bird_info_dict.get(bird_species,\"\"))\ncolor_print(message,\"yellow\")\nrow = train_meta[train_meta['common_name']==Bird_common_name].sort_values(by='rating',ascending=False).iloc[0]\nAudio(row['filepath'])","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:25:19.643216Z","iopub.execute_input":"2024-06-05T08:25:19.643693Z","iopub.status.idle":"2024-06-05T08:25:19.675288Z","shell.execute_reply.started":"2024-06-05T08:25:19.643661Z","shell.execute_reply":"2024-06-05T08:25:19.674318Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram(path=row['filepath'],birdname=Bird_common_name)","metadata":{"execution":{"iopub.status.busy":"2024-06-05T08:25:34.123372Z","iopub.execute_input":"2024-06-05T08:25:34.123782Z","iopub.status.idle":"2024-06-05T08:25:35.064713Z","shell.execute_reply.started":"2024-06-05T08:25:34.123749Z","shell.execute_reply":"2024-06-05T08:25:35.063616Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Connecting the taxanomy dataset with the training dataset","metadata":{}},{"cell_type":"code","source":"bird_sci_names = pd.Series(train_meta.scientific_name.unique())\n\ncolor_print(f\"Number of unique scientific species {len(bird_sci_names)}\")\n\ncolor_print(f\"Matching Species in taxanomy dataset : {bird_sci_names.isin(bird_taxanomy.SCI_NAME).sum()}\")","metadata":{"execution":{"iopub.status.busy":"2024-06-05T09:31:47.770049Z","iopub.execute_input":"2024-06-05T09:31:47.770420Z","iopub.status.idle":"2024-06-05T09:31:47.780648Z","shell.execute_reply.started":"2024-06-05T09:31:47.770393Z","shell.execute_reply":"2024-06-05T09:31:47.779560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bird_taxanomy.head()","metadata":{"execution":{"iopub.status.busy":"2024-06-05T09:17:59.960727Z","iopub.execute_input":"2024-06-05T09:17:59.961107Z","iopub.status.idle":"2024-06-05T09:17:59.975795Z","shell.execute_reply.started":"2024-06-05T09:17:59.961079Z","shell.execute_reply":"2024-06-05T09:17:59.974498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"relevant_cols = [\"SCI_NAME\",'ORDER1','FAMILY','CATEGORY']\ntrain_meta = train_meta.merge(bird_taxanomy[relevant_cols],\n                              left_on = \"scientific_name\",\n                              right_on = \"SCI_NAME\",\n                              how = 'left')\n\ntrain_meta.to_pickle(\"train_full.pkl\",protocol = -1)\n\ntrain_meta.shape","metadata":{"execution":{"iopub.status.busy":"2024-06-05T10:07:24.127626Z","iopub.execute_input":"2024-06-05T10:07:24.128434Z","iopub.status.idle":"2024-06-05T10:07:24.577089Z","shell.execute_reply.started":"2024-06-05T10:07:24.128404Z","shell.execute_reply":"2024-06-05T10:07:24.576048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"color_print(\"Number of unique groups,when grouped by taxanomical ORDER: {}\".format(train_meta['ORDER1'].nunique()))","metadata":{"execution":{"iopub.status.busy":"2024-06-05T10:08:54.425328Z","iopub.execute_input":"2024-06-05T10:08:54.426006Z","iopub.status.idle":"2024-06-05T10:08:54.432705Z","shell.execute_reply.started":"2024-06-05T10:08:54.425973Z","shell.execute_reply":"2024-06-05T10:08:54.431581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"color_print(\"Number of groups, when grouped by taxanomical family: {}\".format(train_meta['FAMILY'].nunique()))","metadata":{"execution":{"iopub.status.busy":"2024-06-05T10:08:55.244986Z","iopub.execute_input":"2024-06-05T10:08:55.245907Z","iopub.status.idle":"2024-06-05T10:08:55.253414Z","shell.execute_reply.started":"2024-06-05T10:08:55.245871Z","shell.execute_reply":"2024-06-05T10:08:55.252275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Basic EDA on distribution of species","metadata":{}},{"cell_type":"code","source":"color_print(\"Number of unique species {}\".format(train_meta['primary_label'].nunique()))","metadata":{"execution":{"iopub.status.busy":"2024-06-05T04:58:19.903974Z","iopub.execute_input":"2024-06-05T04:58:19.904426Z","iopub.status.idle":"2024-06-05T04:58:19.914332Z","shell.execute_reply.started":"2024-06-05T04:58:19.904392Z","shell.execute_reply":"2024-06-05T04:58:19.912747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Geographical distribution of species**","metadata":{}},{"cell_type":"code","source":"t_m = train_meta.dropna(axis=\"index\",inplace=False)\n# #convert to a geodataframe for plotting\nt_m = gpd.GeoDataFrame(\n    t_m, geometry=gpd.points_from_xy(t_m.longitude, t_m.latitude), crs=\"EPSG:4326\")\nm1 = folium.Map(location=(t_m.latitude.mean(),t_m.longitude.mean()),zoom_start = 4, min_zoom=3,max_zoom=7)\nh_map = HeatMap(data=t_m[['latitude','longitude']],\n                radius=4,\n                blur=2).add_to(m1)\n\nm1","metadata":{"execution":{"iopub.status.busy":"2024-06-05T09:51:21.035287Z","iopub.execute_input":"2024-06-05T09:51:21.036071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# boundaries for the Western Ghats - as per Google search (from-https://www.kaggle.com/code/ddosad/birdclef-24-data-exploration)\n#expanding on the border\nlower_latitude = 7\nupper_latitude = 24\nlower_longitude = 71\nupper_longitude = 81\ntrain_wg = t_m[(t_m['latitude'] >= lower_latitude) & \n                           (t_m['latitude'] <= upper_latitude) &\n                           (t_m['longitude'] >= lower_longitude) &\n                           (t_m['longitude'] <= upper_longitude)]\n\n\nm2 = folium.Map(location=(train_wg.latitude.mean(),train_wg.longitude.mean()),zoom_start = 6, min_zoom=4,max_zoom=7)\nh_map = HeatMap(data=train_wg[['latitude','longitude']],\n                radius=4,\n                blur=2).add_to(m2)\n\nm2","metadata":{"execution":{"iopub.status.busy":"2024-06-05T09:53:39.645647Z","iopub.execute_input":"2024-06-05T09:53:39.646621Z","iopub.status.idle":"2024-06-05T09:53:39.704094Z","shell.execute_reply.started":"2024-06-05T09:53:39.646579Z","shell.execute_reply":"2024-06-05T09:53:39.702999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"color_print(\"Number of unique species around western ghats is {}\".format(train_wg['primary_label'].nunique()))","metadata":{"execution":{"iopub.status.busy":"2024-06-05T09:52:40.775853Z","iopub.execute_input":"2024-06-05T09:52:40.776236Z","iopub.status.idle":"2024-06-05T09:52:40.783339Z","shell.execute_reply.started":"2024-06-05T09:52:40.776206Z","shell.execute_reply":"2024-06-05T09:52:40.782276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Distribution of audio quality**","metadata":{}},{"cell_type":"code","source":"#**Lets take a look at distribution of audio signal quality**\ncolor_print(train_meta['rating'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:19:38.229652Z","iopub.execute_input":"2024-06-05T05:19:38.230064Z","iopub.status.idle":"2024-06-05T05:19:38.239239Z","shell.execute_reply.started":"2024-06-05T05:19:38.230034Z","shell.execute_reply":"2024-06-05T05:19:38.238025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Let us see how many secondary labels are there**","metadata":{}},{"cell_type":"code","source":"sec_label_mask = train_meta.secondary_labels.apply(lambda x: len(x)>0)\ncolor_print(f\"Out of {train_meta.shape[0]} total records,{sec_label_mask.sum()} have secondary calls\")","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:11:40.155332Z","iopub.execute_input":"2024-06-05T05:11:40.155779Z","iopub.status.idle":"2024-06-05T05:11:40.173034Z","shell.execute_reply.started":"2024-06-05T05:11:40.155749Z","shell.execute_reply":"2024-06-05T05:11:40.171633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**What is the distribution of audio quality of secondary calls?**","metadata":{}},{"cell_type":"code","source":"color_print(train_meta.loc[sec_label_mask,'rating'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:19:24.971001Z","iopub.execute_input":"2024-06-05T05:19:24.971554Z","iopub.status.idle":"2024-06-05T05:19:24.982579Z","shell.execute_reply.started":"2024-06-05T05:19:24.971518Z","shell.execute_reply":"2024-06-05T05:19:24.981280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Lets look at the common secondary calls**","metadata":{}},{"cell_type":"code","source":"all_secondary_calls = []\nfor idx, row in train_meta.loc[sec_label_mask,:].iterrows():\n    sec_call = row['secondary_labels']\n    all_secondary_calls.extend(sec_call)\n\nall_secondary_calls = pd.Series(all_secondary_calls)\ncolor_print(f\"There are {len(all_secondary_calls)} secondary calls in the data\"\n           )\n\ncolor_print(f\"There are {all_secondary_calls.nunique()} secondary species\"\n           )","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:18:54.390415Z","iopub.execute_input":"2024-06-05T05:18:54.390838Z","iopub.status.idle":"2024-06-05T05:18:54.536951Z","shell.execute_reply.started":"2024-06-05T05:18:54.390806Z","shell.execute_reply":"2024-06-05T05:18:54.535747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**What is the type of calls**\n\n    Birds can have different type of calls, so a single species can have multiple variations of types of calls.","metadata":{}},{"cell_type":"code","source":"ct_mask = train_meta.call_Type.apply(lambda x: len(x)>1)\ncolor_print(\"how many examples have multiple type of calls in a single recording? \\n {}\".format(ct_mask.sum()))\n","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:34:37.650148Z","iopub.execute_input":"2024-06-05T05:34:37.650676Z","iopub.status.idle":"2024-06-05T05:34:37.669914Z","shell.execute_reply.started":"2024-06-05T05:34:37.650636Z","shell.execute_reply":"2024-06-05T05:34:37.668692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_type_calls = []\nfor idx, row in train_meta.iterrows():\n    call_type = row['call_Type']\n    all_type_calls.extend(call_type)\n    \n\nall_type_calls = pd.Series(all_type_calls)\ncolor_print(all_type_calls.value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:34:47.235026Z","iopub.execute_input":"2024-06-05T05:34:47.235451Z","iopub.status.idle":"2024-06-05T05:34:49.001019Z","shell.execute_reply.started":"2024-06-05T05:34:47.235420Z","shell.execute_reply":"2024-06-05T05:34:48.999652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Take a look at how is the count of species in the dataset**","metadata":{}},{"cell_type":"code","source":"#lets look at the distribution of frequencies \ntrain_meta['primary_label'].value_counts().plot(kind= \"hist\",bins =50,figsize = (14,10))\n\nplt.xlabel(\"Count of occurence in the dataset\")","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:01:13.671280Z","iopub.execute_input":"2024-06-05T05:01:13.671688Z","iopub.status.idle":"2024-06-05T05:01:14.121734Z","shell.execute_reply.started":"2024-06-05T05:01:13.671661Z","shell.execute_reply":"2024-06-05T05:01:14.120433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Looks like there are a few species which are over represented, while most of the other species have recording counts less than 50.**","metadata":{}},{"cell_type":"code","source":"#most common species in the dataset\nfig=plt.figure(figsize=(20,10))\nplt.yticks(fontsize=16)\ntop_birds=train_meta['common_name'].value_counts().sort_values(ascending=False)[:20]\nplt.title(\"Birds with most number of recordings\",font=\"Serif\", size=20)\n\ntop_birds.plot.barh(color= 'y',width = 0.75,alpha=0.75)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:01:18.570396Z","iopub.execute_input":"2024-06-05T05:01:18.570809Z","iopub.status.idle":"2024-06-05T05:01:19.095335Z","shell.execute_reply.started":"2024-06-05T05:01:18.570778Z","shell.execute_reply":"2024-06-05T05:01:19.093676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig=plt.figure(figsize=(20,10))\nplt.yticks(fontsize=16)\n\nrare_birds=train_meta['common_name'].value_counts().sort_values(ascending=True)[:20]\n\nrare_birds.sort_values(ascending=False).plot.barh(title = 'Birds with least number of recordings',color= (0.6,0.3,0.3,0.75),width = 0.75)\nplt.title(\"Birds with least number of recordings\",font=\"Serif\", size=20)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-05T05:01:22.750312Z","iopub.execute_input":"2024-06-05T05:01:22.750716Z","iopub.status.idle":"2024-06-05T05:01:23.323340Z","shell.execute_reply.started":"2024-06-05T05:01:22.750686Z","shell.execute_reply":"2024-06-05T05:01:23.321997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Resources\n\n* https://www.kaggle.com/competitions/birdclef-2023/discussion/412808\n* https://www.kaggle.com/competitions/birdclef-2024/discussion/490895\n* https://www.kaggle.com/competitions/birdclef-2023/discussion/412753\n* https://www.kaggle.com/code/tc0000/birdclef-starter-notebook\n* https://www.kaggle.com/code/ddosad/birdclef-24-data-exploration\n* https://www.kaggle.com/code/leonshangguan/faster-eb0-sed-model-inference\n* https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0\n","metadata":{}}]}