\n \n
Content
\n \nIn this notebook I look to explore which birds were at the right place and in the right time to have a chance of making into a test set recording. In the [competition notebook](https://www.kaggle.com/stefankahl/birdclef2021-exploring-the-data), Stefan Kahl points that all birds were likely to be observed at the test sites, but were they there at the right time? To answer this question, each site is assigned a circular region of 200 km in radius. If any of the birds were previously recorded in the same month as the test recordings on that site, the bird could potentially be present in test recordings.","metadata":{}},{"cell_type":"code","source":"import folium\nimport warnings\nimport pandas as pd\nimport geopandas as gpd\nimport geopy.distance\nfrom shapely.geometry import Point\nimport numpy as np\nimport os\nfrom datetime import datetime, timedelta\n\nwarnings.filterwarnings(action='ignore')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, lets get the metadata","metadata":{}},{"cell_type":"code","source":"df_meta = pd.read_csv('../input/birdclef-2021/train_metadata.csv',)\ndf_meta.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will also need the description of test site parameters, such as location and date.","metadata":{}},{"cell_type":"code","source":"files = []\nfor dirname, _, filenames in os.walk('/kaggle/input/birdclef-2021/test_soundscapes'):\n files += [os.path.join(dirname, filename) for filename in filenames]\nfiles","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets draw 200 kilometer regions around each point and consider this to be a site. If the bird has ever been recorded within a circle, it could have potentially been recorded in the test run.","metadata":{}},{"cell_type":"code","source":"df_sites = pd.DataFrame()\n\nfor file in files[1:]:\n with open(file, 'r') as f:\n lines=f.readlines()\n df_sites = df_sites.append({\"name\" : lines[0].strip(), \n \"radius\" : 200, \n \"latitude\" : float(lines[-2].split()[-1]), \n \"longitude\" : float(lines[-1].split()[-1]),\n \"alias\" : file.split(\"/\")[-1].split(\"_\")[0]}, ignore_index=True)\n\ndf_sites","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"m = folium.Map(location=[21.612581945168355, -79.0603262312263], tiles=\"cartodbpositron\", zoom_start=4)\n\nfor item in df_sites.iterrows():\n folium.Circle(location=[item[1][\"latitude\"], item[1][\"longitude\"]], popup=item[1][\"alias\"], fill_color='#00CED1', radius=item[1][\"radius\"]*1000, weight=2, color=\"#000\").add_to(m)\n\nm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, lets retrieve all months, when the recording took place, in the test set. Together with location, lets check all species that were observed in the right place and in the right time.","metadata":{}},{"cell_type":"code","source":"df_dates = pd.read_csv(files[0])\ndf_dates[\"date\"] = pd.to_datetime(df_dates[\"date\"].astype(str), format=\"%Y%m%d\")\n\ndf_dates[\"month\"] = df_dates[\"date\"].apply(lambda x: x.month)\ndf_meta['month'] = df_meta['date'].apply(lambda x: x.split(\"-\")[1]).astype(int)\n\ndf_dates.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"site_params = dict([(site, []) for site in df_dates[\"site\"].unique()])\nfor row in df_dates.iterrows():\n site_params[row[1][\"site\"]].append(row[1][\"month\"])\n\nfor site in site_params:\n site_params[site] = {\"months\" : list(set(site_params[site]))}\n\nfor spatial in df_sites.values:\n site_params[spatial[0]][\"latlon\"] = (spatial[1], spatial[2])\n site_params[spatial[0]][\"R\"] = spatial[-1]\n \nsite_params","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def right_place_time(lat, lon, date):\n \"\"\"\n Calculate if an observation was made within test site parameters (coordinates and time)\n \"\"\"\n check = False\n for site, params in site_params.items():\n # Check within site\n check_site = (geopy.distance.distance(params[\"latlon\"], (lat, lon)).km < params[\"R\"]) and (date in params[\"months\"])\n check = check or (check_site > 0)\n\n return check\n\nright_place_time(42.3005, -72.5877, 0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_meta[\"right_place_time\"] = df_meta.apply(lambda r: right_place_time(r['latitude'], r['longitude'], r[\"month\"]), axis=1)\n\nprint(\"Percentage of records withing test sites at matching times of year: {:.2f}%\".format(100*len(df_meta[df_meta[\"right_place_time\"]])/len(df_meta)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Of {} species {} were observed within sites at the same time of year\".format(df_meta[\"primary_label\"].nunique(), df_meta[df_meta[\"right_place_time\"]][\"primary_label\"].nunique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Summary
\n\nWith 200 km regions around test site, 273 species have been recorded at the same time of year (same month). This narrows down a range of suspects :-) Hope you find this notebook helpful, and please consider upvoting it if you did.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}