{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":" # <p style=\"color:blue\"><center> Introduction</center></p>\n**We all know that birds are the creatures that are most vulnerable to risks and that they face various types of these many dangers, so we, as data scientists, must provide the best we have to help the organizations that are in the eyes of the eye specializing in helping birds and the organizations responsible for that and provide the best analyzes for them.**","metadata":{}},{"cell_type":"markdown","source":"![Birdes](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQM_tI5u29WHhcyBilT5vMfjmywWqtS-9Iscw&usqp=CAU)\n![Birdes](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcT8-TegC4ENTy0ywDXXLGWZkitqi_7s32gEmA&usqp=CAU)","metadata":{}},{"cell_type":"markdown","source":"**Recent advances in robotic listening have improved audio data collection. However, generating and retrieving analysis outputs with high accuracy remains a challenge. The majority of the data has not been examined due to the lack of effective tools to efficiently and reliably extract signals of interest (for example, bird calls).\nThe Hua Laboratory is the most popular, best and much interested laboratory, dedicated to enhancing the understanding and protection of birds and the natural world. The lab joins people from all walks of life to make new scientific discoveries, share ideas, and stimulate conservation actions. In this competition, they collaborate with Google Research, LifeCLEF, and Xeno-canto.**","metadata":{}},{"cell_type":"markdown","source":"# **Import some used Libraries**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport soundfile as sf","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# import dataset ","metadata":{}},{"cell_type":"code","source":"import matplotlib.image as mpimg\nfrom matplotlib.offsetbox import AnnotationBbox, OffsetImage\n\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport descartes\nimport geopandas as gpd\nfrom shapely.geometry import Point, Polygon\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\nfrom sklearn.model_selection import train_test_split\n\nfrom keras.utils import Sequence\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv1D, MaxPool1D, BatchNormalization\nfrom keras.optimizers import RMSprop,Adam\nfrom keras.applications import VGG19, VGG16, ResNet50\n\nimport sklearn\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_meta=pd.read_csv('../input/birdclef-2021/train_metadata.csv')\ntrain_sound=pd.read_csv('../input/birdclef-2021/train_soundscape_labels.csv')\ntest_df=pd.read_csv('../input/birdclef-2021/test.csv')\nsample_submission=pd.read_csv('../input/birdclef-2021/sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# read dataset","metadata":{}},{"cell_type":"code","source":"train_meta.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sound.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = '/kaggle/input/birdclef-2021/'\nos.listdir(path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_meta.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# showing some of information about data","metadata":{}},{"cell_type":"code","source":"train_meta.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# describe data","metadata":{}},{"cell_type":"code","source":"train_meta.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Clarify if there are empty cells in the data or not","metadata":{}},{"cell_type":"code","source":"train_meta.isnull()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Now showing the tail of data","metadata":{}},{"cell_type":"code","source":"train_meta.tail()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"row = 0\ntrain_meta.iloc[row]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# showing columns of data","metadata":{}},{"cell_type":"code","source":"train_meta.columns ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Show chart\n","metadata":{}},{"cell_type":"code","source":"species = train_meta['primary_label'].value_counts()\nfig = go.Figure(data=[go.Bar(y=species.values, x=species.index)],\n                layout=go.Layout(margin=go.layout.Margin(l=0, r=1, b=5, t=10)))\n\nfig.update_layout(title='Number of traning samples per species')\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Because we specify the time for the sound to use","metadata":{}},{"cell_type":"markdown","source":"**Here we will show you some of the audio and improve it for use**","metadata":{}},{"cell_type":"code","source":"train_meta['year'] = train_meta['date'].apply(lambda x: x.split('-')[0])\ntrain_meta['month'] = train_meta['date'].apply(lambda x: x.split('-')[1])\ntrain_meta['day_of_month'] = train_meta['date'].apply(lambda x: x.split('-')[2])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label = train_meta.loc[row, 'primary_label']\nfilename = train_meta.loc[row, 'filename']\n\n# Check if the file is in the folder\nfilename in os.listdir(path+'train_short_audio/'+label)\ndata, samplerate = sf.read(path+'train_short_audio/'+label+'/'+filename)\nprint(data[:8])\nprint(samplerate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Here we created or called this function to make and help with the sound to get the graph out","metadata":{}},{"cell_type":"code","source":"def read_ogg_file(path, file):\n    \"\"\" Read ogg audio file and return numpay array and samplerate\"\"\"\n    \n    data, samplerate = sf.read(path+file)\n    return data, samplerate\n\n\ndef plot_audio_file(data, samplerate):\n    \"\"\" Plot the audio data\"\"\"\n    \n    sr = samplerate\n    fig = plt.figure(figsize=(8, 4))\n    x = range(len(data))\n    y = data\n    plt.plot(x, y)\n    plt.plot(x, y, color='red')\n    plt.legend(loc='upper center')\n    plt.grid()\n    \n    \ndef plot_spectrogram(data, samplerate):\n    \"\"\" Plot spectrogram with mel scaling \"\"\"\n    \n    sr = samplerate\n    spectrogram = librosa.feature.melspectrogram(data, sr=sr)\n    log_spectrogram = librosa.power_to_db(spectrogram, ref=np.max)\n    librosa.display.specshow(log_spectrogram, sr=sr, x_axis='time', y_axis='mel')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_audio_file(data, samplerate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ploting spectrogram to showing result ","metadata":{}},{"cell_type":"code","source":"plot_spectrogram(data, samplerate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Do some analyzes for the data","metadata":{}},{"cell_type":"code","source":"train_sound['audio_id'].unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sound.groupby(by=['audio_id']).count()['birds'][:4]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels = []\nfor row in train_sound.index:\n    labels.extend(train_sound.loc[row, 'birds'].split(' '))\nlabels = list(set(labels))\n\nprint('Number of unique bird labels:', len(labels))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_labels_train = pd.DataFrame(index=train_sound.index, columns=labels)\nfor row in train_sound.index:\n    birds = train_sound.loc[row, 'birds'].split(' ')\n    for bird in birds:\n        df_labels_train.loc[row, bird] = 1\ndf_labels_train.fillna(0, inplace=True)\n\n# We set a dummy value for the target label in the test data because we will need for the Data Generator\ntest_df['birds'] = 'nocall'\n\ndf_labels_test = pd.DataFrame(index=test_df.index, columns=labels)\nfor row in test_df.index:\n    birds = test_df.loc[row, 'birds'].split(' ')\n    for bird in birds:\n        df_labels_test.loc[row, bird] = 1\ndf_labels_test.fillna(0, inplace=True)\ndf_labels_train.sum().sort_values(ascending=False)[:20]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Now we're going to do a simple merge between data train_sound and test_df","metadata":{}},{"cell_type":"code","source":"train_sound = pd.concat([train_sound, df_labels_train], axis=1)\ntest_df = pd.concat([test_df, df_labels_test], axis=1)\nprint(train_sound)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(test_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Now we're going to plot this matrix to see what the result is","metadata":{}},{"cell_type":"code","source":"file = os.listdir(path+'train_soundscapes')[0]\nfile\ndata, samplerate = read_ogg_file(path+'train_soundscapes/', file)\n\nsub_data = data[int(455/5)*160000:int(460/5)*160000]\nplt.figure(figsize=(14, 5))\nlibrosa.display.waveplot(sub_data, sr=samplerate)\nplt.grid()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Now we can hear some voices","metadata":{}},{"cell_type":"code","source":"import librosa\naudio_data = '../input/birdclef-2021/train_short_audio/acafly/XC109605.ogg'\nx , sr = librosa.load(audio_data)\nprint(type(x), type(sr))\nprint(x.shape, sr)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import IPython.display as ipd\nipd.Audio(audio_data)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# This is a drawing of the recorded external sound","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(14, 5))\nlibrosa.display.waveplot(x, sr=sr)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=sample_submission\ndf.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}