{"cells":[{"metadata":{},"cell_type":"markdown","source":"# BirdCLEF: Migration Patterns & Morning vs Night Birds"},{"metadata":{},"cell_type":"markdown","source":"So I have went through  introductory notebooks from @stefankahl and further explored the ideas that he presented there.\nSource: https://www.kaggle.com/stefankahl/birdclef2021-exploring-the-data\n\n* First, I decided to remove the low-rating recordings from the dataset. \n* Second, to check for the migration patterns.\n* Third, I checked the ratio of morning versus night birds. The idea would be to ensure the even distribution of both categories in training and validation sets."},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\n\nimport glob\nimport ntpath\nimport time\nfrom tqdm import tqdm\nfrom math import radians, cos, sin, asin, sqrt\nfrom collections import defaultdict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load CSV Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = '../input/birdclef-2021/'\n#load csv files\ntrain_metadata = pd.read_csv(PATH + 'train_metadata.csv')\ntrain_soundscape_labels = pd.read_csv(PATH + 'train_soundscape_labels.csv')\ntest = pd.read_csv(PATH + 'test.csv')\ntest_dates = pd.read_csv(PATH + 'test_soundscapes/test_set_recording_dates.csv')\nsample_submission = pd.read_csv(PATH + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1. Remove Low-Rating Recordings"},{"metadata":{"trusted":true},"cell_type":"code","source":"#select only the columns that I need for further work\ntrain = train_metadata[['primary_label', 'secondary_labels', 'type','date','latitude','longitude','time','rating','filename']]\nprint(f'Train set size BEFORE cleaning: {len(train_metadata)}')\n#Remove low-rating recordings and reset indexes of the dataframe\ntrain = train.loc[(train_metadata.rating < 0.5) | (train.rating >= 2.0)].reset_index().drop('index',axis=1)\nprint(f'Train set size AFTER cleaning: {len(train)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Removing ~1k examples is not a big deal. So let's stick to it."},{"metadata":{},"cell_type":"markdown","source":"### 2. Migration Patterns"},{"metadata":{},"cell_type":"markdown","source":"Prior to migration pattern analysis we should extract the week number/month number for each recording for both train and test sets."},{"metadata":{},"cell_type":"markdown","source":"#### 2.1 Extract Date-Related Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Get month or week number\nMONTH = True\n\nif MONTH:\n\n    #1. TRAIN SHORT RECORDINGS\n    #convert to datetime\n    train['month'] = pd.to_datetime(train.date.astype('str').apply(lambda x: x[5:-3]), format='%m', errors='coerce')\n    #chech how many null values were there\n    print(f'Number of null values in Date column: {len(train.loc[train.month.isnull()])}')\n    #drop null values and reset index\n    train = train.dropna().reset_index().drop('index',axis=1)\n    #extract week number from the resulting datetime\n    train['month'] = train.month.dt.month\n\n    #2. TRAIN SOUNDSCAPES\n     \n    '''\n    train_soundscape_labels does not contain date-related information but it \n    can actually be extracted from the actual filenames from the **train_soundscapes** directory:\n    ''' \n    #get the file names from the directory\n    filenames = [ntpath.basename(path).strip('.ogg') for path in glob.glob(PATH+'train_soundscapes/*.ogg')]\n\n    #map audio_id to recording date\n    id_to_date = dict()\n    for i in range(len(filenames)):\n        s = filenames[i].split('_')\n        id_to_date[int(s[0])] = s[-1]\n\n    #assign the date column\n    train_soundscape_labels['date'] = [id_to_date[idx] for idx in train_soundscape_labels.audio_id]\n    #convert the date column into datetime\n    train_soundscape_labels.date = pd.to_datetime(train_soundscape_labels.date, format='%Y%m%d')\n    #get the week number\n    train_soundscape_labels['month'] = train_soundscape_labels.date.dt.month\n\n    #3. TEST DATA\n    #convert test data's date column into datetime\n    test_dates.date = pd.to_datetime(test_dates.date.astype('str'))\n    #extract the week only\n    test_dates['month'] = test_dates.date.dt.month\n    \nelse: #WEEK\n    \n    #1. TRAIN SHORT RECORDINGS\n    #convert to datetime\n    train.date = pd.to_datetime(train.date, format='%Y-%m-%d', errors='coerce')\n    #chech how many null values were there\n    print(f'Number of null values in Date column: {len(train.loc[train.date.isnull()])}')\n    #drop null values and reset index\n    train = train.dropna().reset_index().drop('index',axis=1)\n    #extract week number from the resulting datetime\n    train['week'] = train.date.dt.week\n    \n    #2. TRAIN SOUNDSCAPES\n    #get the file names from the directory\n    filenames = [ntpath.basename(path).strip('.ogg') for path in glob.glob(PATH+'train_soundscapes/*.ogg')]\n\n    #map audio_id to recording date\n    id_to_date = dict()\n    for i in range(len(filenames)):\n        s = filenames[i].split('_')\n        id_to_date[int(s[0])] = s[-1]\n\n    #assign the date column\n    train_soundscape_labels['date'] = [id_to_date[idx] for idx in train_soundscape_labels.audio_id]\n    #convert the date column into datetime\n    train_soundscape_labels.date = pd.to_datetime(train_soundscape_labels.date, format='%Y%m%d')\n    #get the week number\n    train_soundscape_labels['week'] = train_soundscape_labels.date.dt.week\n    train_soundscape_labels.head()\n\n    #3. TEST DATA\n    #convert test data's date column into datetime\n    test_dates.date = pd.to_datetime(test_dates.date.astype('str'))\n    #extract the week only\n    test_dates['week'] = test_dates.date.dt.week\n    test_dates.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have got the week number during which every recording was made for both training and test sets. Time to do some feature engineering. \n\nHere are 2 main approaches I came up with:\n\n1. Check where is every bird vocalization observed during the given week/month and assign to it the site name which is the closest to it during this week/month. While this approach may capture the migrational patterns of mid- to long-distance migrating birds, it is most likely to fail if it is a year-round bird. For example,take a look at [House Sparrow](https://ebird.org/science/status-and-trends/houspa/abundance-map-weekly). \n2. Assign the site name to each recording based on where was the biggest amount of vocalizations of the given bird species observed during the week/month when this recording was made. I think this is the best approach.\n\nIn addition, prior to trying both approaches, one must be careful with what data is used  - it will not make sense to calculate distance between the site and recording if the recording took place somewhere in Europe or Asia while all the recording sites are located in Americas. To avoid such problems, let's work with data from America only. I know that to this point I have already removed around 5k recordings from the data, but it's just a trade-off I have to face to get more precise features.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#select the recordings which were made only in Americas (left to the 25th west meridian)\nprint(f'Trainset length BEFORE cleaning: {len(train)}')\ntrain = train.loc[train.longitude < -25.0].reset_index().drop('index',axis=1)\nprint(f'Trainset length AFTER cleaining: {len(train)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Minus another 5k recordings... Let's see how it affected the class distribution:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code adapted from https://www.kaggle.com/shahules/bird-watch-complete-eda-fe and \n# https://www.kaggle.com/stefankahl/birdclef2021-exploring-the-data\n\n# Unique eBird codes\nspecies1 = train_metadata['primary_label'].value_counts()\nspecies2 = train['primary_label'].value_counts()\n\n# Make bar chart\nfig = go.Figure(data=[go.Bar(y=species1.values, x=species1.index,name=\"Before Cleaning\"),\n                      go.Bar(y=species2.values, x=species2.index, name=\"After Cleaning\")],\n                layout=go.Layout(margin=go.layout.Margin(l=0, r=0, b=10, t=50)))\nfig.update_layout(title='Number of traning samples per species')\n\n\n# Show charts\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that our class distribution got even more imbalanced. We also lost 2 species:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#get lost species during data cleaning\nlost_species = np.setdiff1d(train_metadata.primary_label.unique(),train.primary_label.unique())\n\n#get species that have low number of records and combine with lost species\nscarce = list(train.groupby(\"primary_label\").filter(lambda x: len(x) < 25).primary_label.unique()) + list(lost_species)\n\nprint(f'Lost Species: {len(lost_species)}')\nprint(f'Scarce Species (<25 recordings): {len(scarce)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My data cleaning approach is not perfect, but I think I will still stick to it. **I would also love to get your advice on how to make it better!**\nHere are ways to possibly preserve more recordings:\n\n1. When processing date column, extract month instead of week number.\n2. Include bird recordings from all over the world. But than my migration feature engineering approaches(which you can explore in the next code cell) will not work.\n3. Get the data of scarce bird species from previous competition's data. I have tried it, but it has only limited number of classes."},{"metadata":{},"cell_type":"markdown","source":"### 2.2 Migration Feature Engineering"},{"metadata":{},"cell_type":"markdown","source":"#### Approach 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"def haversine(lon1, lat1, lon2, lat2):\n    \"\"\"\n    Calculate the great circle distance between two points \n    on the earth (specified in decimal degrees)\n    \n    Source: #https://stackoverflow.com/questions/42686300/how-to-check-if-coordinate-inside-certain-area-python\n    \"\"\"\n    # convert decimal degrees to radians \n    lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2])\n\n    # haversine formula \n    dlon = lon2 - lon1 \n    dlat = lat2 - lat1 \n    a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2\n    c = 2 * asin(sqrt(a)) \n    r = 6371 # Radius of earth in kilometers. Use 3956 for miles\n    return c * r\n    \ndef get_closest_site(df, sites):\n    '''\n        Get the site which is closest to the bird vocalization recording. Returns pandas.Series contatining site names assigned to each recording.\n        \n        Attributes:\n            df(pandas.DataFrame): df containing vocalization recordings\n            sites(list of dicts): list of dictionaries containing coordinates(lat,long) and name of the sites\n    '''\n    \n    distances = defaultdict(list)\n    #get the coordinates of all records\n    lat = df.latitude\n    long = df.longitude\n    \n    for recording_site in sites:\n        #calculate the distance between the site and the vocalization recordings with haversine function\n        for i in range(len(df)):\n\n            lat1 = recording_site['lat']\n            lon1 = recording_site['long']\n            lat2 = lat[i]\n            lon2 = long[i]\n\n            distance = haversine(lon1, lat1, lon2, lat2)\n            \n            distances[i].append(distance)\n    #get the name of the site which is the closest to the vocalization recording\n    for i in range(len(distances)):\n        idx =  distances[i].index(min(distances[i]))\n        distances[i] = sites[idx]['name']\n    \n    return pd.Series(distances.values())\n\n#Coordinates of the recording locations taken from the competition's txt data\nCOL = {'lat': 5.57, 'long': -75.85, 'name': 'COL'} # Colombia\nCOR = {'lat': 10.12, 'long': -84.51, 'name': 'COR'} # Costa Rica \nSNE = {'lat': 38.49, 'long': -119.95, 'name': 'SNE'} # Sierra Nevada\nSSW = {'lat': 42.47, 'long': -76.45, 'name': 'SSW'} # Sapsucker Woods\nsites = [COL,COR,SNE,SSW]\n\n#Assign to the train dataframe\ntrain['site'] = get_closest_site(train,sites)\n#display a sample of data\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Approach 2"},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_proximity(data, birdcode, time_period, recording_site ,radius = 250.0):\n    '''\n        Check whether the vocalization recording of the given bird is within a certain radius of the recording side during the given time period.\n        \n        Attributes:\n            data(pandas.DataFrame): training data\n            birdcode(str): eBird code\n            time_period(tuple): ('week'/'month', number)\n            recording_site(dict): dictionary with coordinates(lat,long) and name of the site\n            radius(float): radius in kilometers\n    '''\n    \n    if time_period[0] == 'week':\n        #get the data only for specific week\n        df = data.loc[data.week == time_period[1]]\n    elif time_period[0] == 'month':\n        #get the data only for specific month\n        df = data.loc[data.month == time_period[1]]\n    \n    freq = 0\n    #pick the records of the given bird\n    df = df.loc[df.primary_label == birdcode].reset_index().drop('index',axis=1)\n    lat = df.latitude\n    long = df.longitude\n    #calculate the distance between the site and the vocalization recordings with haversine function\n    for i in range(len(df)):\n        \n        lat1 = recording_site['lat']\n        lon1 = recording_site['long']\n        lat2 = lat[i]\n        lon2 = long[i]\n        \n        distance = haversine(lon1, lat1, lon2, lat2)\n        #check whether the vocalization is within the radius and update the frequency counter\n        if distance <= radius:\n            freq+=1\n        else:\n            continue\n    return freq\n\n#Coordinates of the recording locations taken from the competition's txt data\nCOL = {'lat': 5.57, 'long': -75.85, 'name': 'COL'} # Colombia\nCOR = {'lat': 10.12, 'long': -84.51, 'name': 'COR'} # Costa Rica \nSNE = {'lat': 38.49, 'long': -119.95, 'name': 'SNE'} # Sierra Nevada\nSSW = {'lat': 42.47, 'long': -76.45, 'name': 'SSW'} # Sapsucker Woods\nsites = [COL,COR,SNE,SSW]\n#pick the radius\nradius = 500.0\ntime_measure = 'month' if MONTH else 'week'\n\nfreqs = dict()\nfor t in tqdm(sorted(train[time_measure].unique())):\n    site_freqs = defaultdict(dict)\n    #loop over sites\n    for i,site in enumerate(sites):\n        #loop over bird species\n        for b in train.primary_label.unique():\n            site_freqs[sites[i]['name']][b] = check_proximity(train,b,(time_measure,t),site,radius)\n    freqs[t] = dict(pd.DataFrame(site_freqs).idxmax(axis=1))\n    \n# assign new column to the train df\ntrain['site'] = [freqs[train[time_measure][idx]][train.primary_label[idx]] for idx in range(len(train))]\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3. Morning vs Night Birds"},{"metadata":{"trusted":true},"cell_type":"code","source":"''' Check the ratio between morning/night birds'''\n\n# Convert time column into datetime and remove rows with missing values in it\ntrain['time'] = pd.to_datetime(train['time'], errors='coerce')\ntrain = train.dropna(subset=['time']).reset_index().drop('index',axis=1)\n#get only the hour\ntrain.time = train['time'].dt.hour.astype('int')\n#categorize -> morning vs night bird\ntrain.time = train.time.apply(lambda x: 0 if x<=13 and x>4 else 1)\n\ndef get_btype(bcodes):\n    \n    btype = {}\n    for b in bcodes:\n\n        v = train.loc[train.primary_label == b].time.value_counts()\n        if len(v)<2:\n            btype[b] = v.keys()[0]\n\n        elif v[0] > v[1]:\n            btype[b] = 0 # morning\n        else:\n            btype[b] = 1 # night\n    return btype\n\nbtype = get_btype(train.primary_label.unique())\ntrain['btype'] = [btype[p] for p in train.primary_label]\n#plot\nfig = px.bar(train.btype.value_counts(), title = 'Morning vs Night Birds')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is clear that the majority of the bird vocalizations were obseved during the **morning hours**. This could have happened due to the fact that people who record these vocalisations typically go out during the morning/day hours or that the birds from this dataset are really mostly singing during the mornings. Both facts might be true, but it actually does not matter. The initial idea was to ensure the even distribution of morning/night birds in training and validations sets, but as a result, it is not that important in this case."},{"metadata":{},"cell_type":"markdown","source":"## Excited to see what I will discover next!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}