{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <h><center><b>🐧BIRDCIEF 🐤 2023🐧</b></center></h>\n\n<img src='https://img.freepik.com/free-vector/cute-nightingale-singing-cartoon-vector_96037-433.jpg?w=2000'>\n\n## **Goal of the Competition**\n\nBirds are excellent indicators of biodiversity change since they are highly mobile and have diverse habitat requirements. Changes in species assemblage and the number of birds can thus indicate the success or failure of a restoration project. However, frequently conducting traditional observer-based bird biodiversity surveys over large areas is expensive and logistically challenging. In comparison, passive acoustic monitoring (PAM) combined with new analytical tools based on machine learning allows conservationists to sample much greater spatial scales with higher temporal resolution and explore the relationship between restoration interventions and biodiversity in depth.\n\nFor this competition, you'll use your machine-learning skills to identify Eastern African bird species by sound. Specifically, you'll develop computational solutions to process continuous audio data and recognize the species by their calls. The best entries will be able to train reliable classifiers with limited training data. If successful, you'll help advance ongoing efforts to protect avian biodiversity in Africa, including those led by the Kenyan conservation organization NATURAL STATE. \n\n# <center><b>Explore Audio data - Happy learning🐧</b></center>","metadata":{}},{"cell_type":"markdown","source":"# ***Day1 - Version 4: Basic terms of Audio and plotly map***\n","metadata":{}},{"cell_type":"markdown","source":"# ***Day2 - Version 12: Identify insight of data (EDA) & Visualize geomap***\n\n### **1. Geopandas**\n### **2. Plotly**\n### **3. folium**","metadata":{}},{"cell_type":"markdown","source":"# **Day3 version 14- Audio data basic terms and preprocessing**","metadata":{}},{"cell_type":"markdown","source":"# **1. Import Necessary Library**","metadata":{}},{"cell_type":"code","source":"!pip install plotly\n!pip install geopandas","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-17T15:00:55.903308Z","iopub.execute_input":"2023-03-17T15:00:55.904108Z","iopub.status.idle":"2023-03-17T15:01:18.281566Z","shell.execute_reply.started":"2023-03-17T15:00:55.904058Z","shell.execute_reply":"2023-03-17T15:01:18.280346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\nimport os\nimport numpy as np\nimport pandas as pd\n\n#Mapping\nimport geopandas as gpd\nimport folium\nfrom folium import Marker\nfrom folium.plugins import HeatMap\nfrom folium.plugins import MarkerCluster\n\n#plotting \nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport plotly.express as px\n\n\n#Audio\n\nimport librosa\nimport librosa.display as ld\nfrom IPython.display import Audio","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:18.284376Z","iopub.execute_input":"2023-03-17T15:01:18.284805Z","iopub.status.idle":"2023-03-17T15:01:22.839844Z","shell.execute_reply.started":"2023-03-17T15:01:18.284759Z","shell.execute_reply":"2023-03-17T15:01:22.838724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /kaggle/input/birdclef-2023","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:22.841218Z","iopub.execute_input":"2023-03-17T15:01:22.841715Z","iopub.status.idle":"2023-03-17T15:01:23.803492Z","shell.execute_reply.started":"2023-03-17T15:01:22.841673Z","shell.execute_reply":"2023-03-17T15:01:23.802152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **EDA and identify insight of data**","metadata":{}},{"cell_type":"code","source":"train_meta = pd.read_csv('/kaggle/input/birdclef-2023/train_metadata.csv')\nprint(\"Identify null values\",train_meta.isnull().sum())\nprint(\"Train_meta_shape:\",train_meta.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:23.806668Z","iopub.execute_input":"2023-03-17T15:01:23.807086Z","iopub.status.idle":"2023-03-17T15:01:23.936819Z","shell.execute_reply.started":"2023-03-17T15:01:23.807041Z","shell.execute_reply":"2023-03-17T15:01:23.934703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Number of different species: {train_meta.primary_label.nunique()}')\nprint(f\"Number of species > 100 audio records: {sum(train_meta['common_name'].value_counts()<100)}\")\npercentage_low = 133/397*100\npercentage_low = \"{:.2f}\".format(percentage_low)\nprint(f\"At least {percentage_low}% of the records are infrequent\")","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:23.938637Z","iopub.execute_input":"2023-03-17T15:01:23.939022Z","iopub.status.idle":"2023-03-17T15:01:23.958489Z","shell.execute_reply.started":"2023-03-17T15:01:23.938981Z","shell.execute_reply":"2023-03-17T15:01:23.957250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Maximum and minimum of longitude and latitude**","metadata":{}},{"cell_type":"code","source":"print(f\"Minimum longitude: {train_meta['longitude'].min()}\")\nprint(f\"Maximum longitude: {train_meta['longitude'].max()}\")\nprint(f\"Minimum latitude: {train_meta['latitude'].min()}\")\nprint(f\"Maximum latitude: {train_meta['latitude'].max()}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:23.959686Z","iopub.execute_input":"2023-03-17T15:01:23.960680Z","iopub.status.idle":"2023-03-17T15:01:23.969441Z","shell.execute_reply.started":"2023-03-17T15:01:23.960639Z","shell.execute_reply":"2023-03-17T15:01:23.968032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Identify the location of birds comes using birds longitude and latitude**","metadata":{}},{"cell_type":"code","source":"\ndf = train_meta.iloc[:, [0,3,4,8,9]]\ngdf = gpd.GeoDataFrame(\n    df, geometry=gpd.points_from_xy(df.longitude, df.latitude))\ngdf_1 = gdf[(gdf.latitude<35) & (gdf.longitude>-30)]\ngdf_2 = gdf[(gdf.latitude<10) & (gdf.longitude<-20)]\ngdf_3 = gdf[(gdf.latitude>=35) & (gdf.longitude>=-40)]\ngdf_4 = gdf[(gdf.latitude>=10) & (gdf.longitude<=-40)]\nworld = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))\nax = world.plot(color='white', edgecolor='black')\n# We can now plot our ``GeoDataFrame``.\ngdf_1.plot(ax=ax, markersize=1)\ngdf_2.plot(ax=ax, markersize=1)\ngdf_3.plot(ax=ax, markersize=1)\ngdf_4.plot(ax=ax, markersize=1)\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:23.971674Z","iopub.execute_input":"2023-03-17T15:01:23.972167Z","iopub.status.idle":"2023-03-17T15:01:26.940315Z","shell.execute_reply.started":"2023-03-17T15:01:23.972125Z","shell.execute_reply":"2023-03-17T15:01:26.939130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Identify which location large amount of birds here**","metadata":{}},{"cell_type":"code","source":"total = np.sum([len(gdf_1), len(gdf_2), len(gdf_3), len(gdf_4)])\n# print(total)\n# assert total == len(train_meta.index)\nprint('The number of records for each quadrant of the earth: \\n' +\n       f'     -> {[len(gdf_1), len(gdf_3), len(gdf_2), len(gdf_4)]}\\n' +\n       f'     -> {[\"Af\", \"EU\", \"SA\", \"NA\"]}')","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:26.941513Z","iopub.execute_input":"2023-03-17T15:01:26.942873Z","iopub.status.idle":"2023-03-17T15:01:26.951207Z","shell.execute_reply.started":"2023-03-17T15:01:26.942827Z","shell.execute_reply":"2023-03-17T15:01:26.949897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Answer: Most of the bird crowds going to  AF and EU**","metadata":{}},{"cell_type":"markdown","source":"# **Explore more and verify the my result of most and least birds going in AF and EU**","metadata":{}},{"cell_type":"code","source":"#Rating \nsns.violinplot(data=train_meta, x=\"rating\")","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:26.953051Z","iopub.execute_input":"2023-03-17T15:01:26.953478Z","iopub.status.idle":"2023-03-17T15:01:27.242267Z","shell.execute_reply.started":"2023-03-17T15:01:26.953439Z","shell.execute_reply":"2023-03-17T15:01:27.241319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Rating of audio data\nplt.figure(figsize=(16,8))\nsns.countplot(x=train_meta.rating,data=train_meta)\nplt.title('Record Rating of Birds sound')","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:27.246259Z","iopub.execute_input":"2023-03-17T15:01:27.246965Z","iopub.status.idle":"2023-03-17T15:01:27.576498Z","shell.execute_reply.started":"2023-03-17T15:01:27.246914Z","shell.execute_reply":"2023-03-17T15:01:27.575325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **In the analysis 4 is top rating of birds sound record**","metadata":{}},{"cell_type":"markdown","source":"# **- Higes and least recoded 20birds in the recording**","metadata":{}},{"cell_type":"code","source":"highest_recorded=train_meta['common_name'].value_counts().sort_values(ascending=False)[:20]\nleast_recorded=train_meta['common_name'].value_counts().sort_values()[:20]\n\nplt.subplots(2,1,figsize=(16,16))\n\nplt.subplot(2,1,1)\nplt.bar(x=highest_recorded.index,height=highest_recorded.values)\nplt.xticks(rotation=45)\nplt.ylabel('Count')\nplt.title('Most Recorded birds')\n\nplt.subplot(2,1,2)\nplt.bar(x=least_recorded.index,height=least_recorded.values)\nplt.xticks(rotation=45)\nplt.ylabel('Count')\nplt.title('Least Recorded birds')\n\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:27.577970Z","iopub.execute_input":"2023-03-17T15:01:27.580480Z","iopub.status.idle":"2023-03-17T15:01:28.426262Z","shell.execute_reply.started":"2023-03-17T15:01:27.580436Z","shell.execute_reply":"2023-03-17T15:01:28.425234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Just Explore plotly in general location through birds name**","metadata":{}},{"cell_type":"code","source":"fig = px.density_mapbox(train_meta, lat='latitude',lon='longitude',hover_name='primary_label',radius=1,\n                       center=dict(lat=0,lon=180),zoom=1.5,\n                       mapbox_style=\"stamen-terrain\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:28.427884Z","iopub.execute_input":"2023-03-17T15:01:28.428557Z","iopub.status.idle":"2023-03-17T15:01:29.991119Z","shell.execute_reply.started":"2023-03-17T15:01:28.428517Z","shell.execute_reply":"2023-03-17T15:01:29.990241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Recording location of birds using folium - Indepth analysis**","metadata":{}},{"cell_type":"code","source":"\nfrom folium.plugins import HeatMap\n\nstart_loc=(train_meta['latitude'].mean(),train_meta['longitude'].mean())\n\n#map\nm_1=folium.Map(location=start_loc,\n              tiles='Open Street Map',\n              zoom_start=2,\n              min_zoom=1.5)\nm_1","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:29.992910Z","iopub.execute_input":"2023-03-17T15:01:29.993589Z","iopub.status.idle":"2023-03-17T15:01:30.036027Z","shell.execute_reply.started":"2023-03-17T15:01:29.993547Z","shell.execute_reply":"2023-03-17T15:01:30.035055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colors = [\n    'red',\n    'blue',\n    'gray',\n    'darkred',\n    'lightred',\n    'orange',\n    'beige',\n    'green',\n    'darkgreen',\n    'lightgreen',\n    'darkblue',\n    'lightblue',\n    'purple',\n    'darkpurple',\n    'pink',\n    'cadetblue',\n    'lightgray',\n    'black',\n    'red',\n    'blue']\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-17T15:01:30.038993Z","iopub.execute_input":"2023-03-17T15:01:30.039333Z","iopub.status.idle":"2023-03-17T15:01:30.046720Z","shell.execute_reply.started":"2023-03-17T15:01:30.039296Z","shell.execute_reply":"2023-03-17T15:01:30.045727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Most birds going in the location**","metadata":{}},{"cell_type":"code","source":"most_common=train_meta[train_meta['common_name'].isin(highest_recorded.index)]\n\n#adding diffrent color marker to each species:\ncolor_dict=dict(zip(highest_recorded.index,colors))\n\nm_2=folium.Map(location=start_loc,\n              tiles='Open Street Map',\n              zoom_start=2,\n              min_zoom=1.5)\n\nmc=MarkerCluster()\n\nfor idx,row in most_common.iterrows():\n    \n    location=(row['latitude'],row['longitude'])\n    bird_name=row['common_name']\n    fill_color=color_dict[bird_name]\n    \n    if not math.isnan(location[0]) and not math.isnan(location[1]):\n        mc.add_child(Marker(location,tooltip=f'<b>{bird_name}</b>',\n                    icon=folium.Icon(color=fill_color)))\n    \nm_2.add_child(mc)   ","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:30.050345Z","iopub.execute_input":"2023-03-17T15:01:30.050676Z","iopub.status.idle":"2023-03-17T15:01:42.041656Z","shell.execute_reply.started":"2023-03-17T15:01:30.050647Z","shell.execute_reply":"2023-03-17T15:01:42.038180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Least bird going location**","metadata":{}},{"cell_type":"code","source":"least_common=train_meta[train_meta['common_name'].isin(least_recorded.index)]\n\n#adding diffrent color marker to each species:\ncolor_dict=dict(zip(least_recorded.index,colors))\n\nm_2=folium.Map(location=start_loc,\n              tiles='Open Street Map',\n              zoom_start=2,\n              min_zoom=1.5)\n\nmc=MarkerCluster()\n\nfor idx,row in least_common.iterrows():\n    \n    location=(row['latitude'],row['longitude'])\n    bird_name=row['common_name']\n    fill_color=color_dict[bird_name]\n    \n    if not math.isnan(location[0]) and not math.isnan(location[1]):\n        mc.add_child(Marker(location,tooltip=f'<b>{bird_name}</b>',\n                           icon=folium.Icon(color=fill_color)))\n    \nm_2.add_child(mc)    \n","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:01:42.044204Z","iopub.execute_input":"2023-03-17T15:01:42.044753Z","iopub.status.idle":"2023-03-17T15:01:42.327783Z","shell.execute_reply.started":"2023-03-17T15:01:42.044688Z","shell.execute_reply":"2023-03-17T15:01:42.326814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Finally, Most of the bird crowds going to  AF and EU**","metadata":{}},{"cell_type":"markdown","source":"# **Day3 - Audio terms analysis**","metadata":{}},{"cell_type":"code","source":"import math\nimport numpy as np\nimport pandas as pd\nimport librosa\nimport librosa.display\nimport matplotlib.pyplot as plt\nimport IPython.display as ipd\nimport scipy as sp\nimport soundfile as sf\nimport random","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:07:07.595153Z","iopub.execute_input":"2023-03-17T15:07:07.596194Z","iopub.status.idle":"2023-03-17T15:07:07.602799Z","shell.execute_reply.started":"2023-03-17T15:07:07.596139Z","shell.execute_reply":"2023-03-17T15:07:07.601609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bird1_abhori1 = '/kaggle/input/birdclef-2023/train_audio/abhori1/XC120250.ogg'\nbird2_abethr1 = '/kaggle/input/birdclef-2023/train_audio/abethr1/XC128013.ogg'\nbird3_bawman1 = '/kaggle/input/birdclef-2023/train_audio/bawman1/XC237894.ogg'","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:09:43.354539Z","iopub.execute_input":"2023-03-17T15:09:43.354940Z","iopub.status.idle":"2023-03-17T15:09:43.360499Z","shell.execute_reply.started":"2023-03-17T15:09:43.354903Z","shell.execute_reply":"2023-03-17T15:09:43.358922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(bird1_abhori1)","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:10:16.848330Z","iopub.execute_input":"2023-03-17T15:10:16.849039Z","iopub.status.idle":"2023-03-17T15:10:16.874373Z","shell.execute_reply.started":"2023-03-17T15:10:16.848998Z","shell.execute_reply":"2023-03-17T15:10:16.873268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(bird2_abethr1)","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:11:01.215445Z","iopub.execute_input":"2023-03-17T15:11:01.216164Z","iopub.status.idle":"2023-03-17T15:11:01.269277Z","shell.execute_reply.started":"2023-03-17T15:11:01.216122Z","shell.execute_reply":"2023-03-17T15:11:01.268207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(bird3_bawman1)","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:11:17.704595Z","iopub.execute_input":"2023-03-17T15:11:17.705737Z","iopub.status.idle":"2023-03-17T15:11:17.723612Z","shell.execute_reply.started":"2023-03-17T15:11:17.705687Z","shell.execute_reply":"2023-03-17T15:11:17.722514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Load the data**","metadata":{}},{"cell_type":"code","source":"f1,sr= librosa.load(bird1_abhori1)\nf2,_ = librosa.load(bird2_abethr1)\nf3,_ = librosa.load(bird3_bawman1)\n\n#sample duration\nsample_duration = 1/sr\nprint(f\"Duration of 1 sample is: {sample_duration: .6f} seconds\")\n\n# duration of audio signal\nduration = sample_duration * len(f1)\nprint(f\"Duration of 1 sample is:{duration: .6f} seconds\")\n","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:16:59.031484Z","iopub.execute_input":"2023-03-17T15:16:59.032596Z","iopub.status.idle":"2023-03-17T15:17:01.853069Z","shell.execute_reply.started":"2023-03-17T15:16:59.032544Z","shell.execute_reply":"2023-03-17T15:17:01.851844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Visualize the loaded data**","metadata":{}},{"cell_type":"code","source":"#Visualize\nplt.figure(figsize=(15,17))\n\nplt.subplot(3,1,1)\nlibrosa.display.waveshow(f1,alpha=0.5)\nplt.ylim((-1,1))\nplt.title('bird1_abhori1')\n\nplt.subplot(3,1,2)\nlibrosa.display.waveshow(f2,alpha=0.5)\nplt.title('bird2_abethr1')\nplt.ylim((-1,1))\n\nplt.subplot(3,1,3)\nlibrosa.display.waveshow(f3,alpha=0.5)\nplt.title('bird3_bawman')\nplt.ylim((-1,1))\n\nplt.tight_layout()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-17T15:20:32.240828Z","iopub.execute_input":"2023-03-17T15:20:32.241885Z","iopub.status.idle":"2023-03-17T15:20:33.800837Z","shell.execute_reply.started":"2023-03-17T15:20:32.241809Z","shell.execute_reply":"2023-03-17T15:20:33.799659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Audio data tech terms explore**\n# **Coming soon...Explore more in audio data.......Happy learning**","metadata":{}},{"cell_type":"markdown","source":"# Reference\n\n1. https://www.kaggle.com/code/dryanfurman/birdclef-eda-and-fastai","metadata":{}}]}