{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\nIn this Notebook, i will be exploring the dataset through various visualization tools in hope to uncover interesting insights that could be useful at the later stage. We will look into the data and understand briefly what the data encompasses. I will also show how we can make use of a library call **Folium**, that enable us to visualize the geographical distribution of the bird species through the use of interactive maps, from various perspective.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport folium\nfrom folium.plugins import MarkerCluster\nimport json\nimport urllib\n\nsns.set_style('darkgrid')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reading Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = '../input/birdsong-recognition/'\n\ntrain_df = pd.read_csv(f'{PATH}train.csv')\ntest_df = pd.read_csv(f'{PATH}test.csv')\n\nprint(\"Train shape: \", train_df.shape, '\\t', 'Test shape: ', test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['xc_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All recordings from training set are unique","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Missing Values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Columns with missing rows\")\nprint(train_df.isnull().sum().sort_values(ascending=False).head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Univariate Analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Bird Species","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Total number of unique Bird species: \", train_df['ebird_code'].nunique())\nprint(\"Distribution of Bird species in Training set: \")\nprint(train_df.groupby(['ebird_code']).size().sort_values(ascending=False))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the distribution of the bird species in our training data span from 100 recordings to merely just 9 recordings for *redhea* species","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(18,8))\nplt.xticks(rotation=90)\nplt.title(\"Distribution of species across countries\")\nsns.countplot(data=train_df, x='country')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Where are these bird species from training set mostly retrieved from","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Top 5 countries: \\n\")\nprint(train_df.groupby(['country']).size().sort_values(ascending=False).head(5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Geographical Analysis\nI will be using the [folium](https://python-visualization.github.io/folium/) library for geographical visualization of the bird species. Folium is a python library that makes it easy to visualize geographical data with the use of interactive maps. Lets first create a world map that can help us visualize all the location of the training rows. The use of MarkerCluster in folium allows us to view a cluster that will change as we zoom in and out. A single instances will be shown as a red little dot ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"world = folium.Map(location=[27.623924, -30.471619], zoom_start=2, min_zoom=2)\nmc= MarkerCluster()\n\nfor i in range(0,len(train_df)):\n    if (train_df.iloc[i]['longitude'] != 'Not specified'):\n       mc.add_child(folium.Circle(\n          location=[float(train_df.iloc[i]['latitude']), float(train_df.iloc[i]['longitude'])],\n          radius=5000,\n          color='crimson',\n          fill=True,\n          fill_color='crimson'\n       ))\n    \nworld.add_child(mc)\nworld","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plotting different Bird Species\nWe have gathered the information earlier that the count for each bird species range from 9 ~ 100. So lets visualize these bird species individually on the map to see if they are found in similar or different region of the world. Lets choose the top and bottom 2 species: *killde, greegr, buffle, redhea*. The use of layer control allows us to the geographical distribution of each bird species individually\n  ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#base map\nbird = folium.Map(location=[27.623924, -30.471619], zoom_start=3, min_zoom=2)\n\nkillde_df = train_df.loc[train_df['ebird_code'] == 'killde']\ngreegr_df = train_df.loc[train_df['ebird_code'] == 'greegr']\nbuffle_df = train_df.loc[train_df['ebird_code'] == 'buffle']\nredhea_df = train_df.loc[train_df['ebird_code'] == 'redhea']\n\n#Create feature group\nkillde = folium.FeatureGroup(name='killde')\ngreegr = folium.FeatureGroup(name='greegr')\nbuffle = folium.FeatureGroup(name='buffle')\nredhea = folium.FeatureGroup(name='redhea')\n\ndef add_point(df, fg, color):\n    for i in range(0,len(df)):\n        if (df.iloc[i]['longitude'] != 'Not specified'):\n           fg.add_child(folium.CircleMarker(\n              location=[float(df.iloc[i]['latitude']), float(df.iloc[i]['longitude'])],\n              radius=3,\n              color=color,\n              fill=True,\n              fill_color=color\n           ))\n\n#Add each species as an overlay \nadd_point(killde_df, killde, \"red\")\nadd_point(greegr_df, greegr, \"green\")\nadd_point(buffle_df, buffle, \"blue\")\nadd_point(redhea_df, redhea, \"black\")\n\n#Add overlay to base map\nkillde.add_to(bird)\ngreegr.add_to(bird)\nbuffle.add_to(bird)\nredhea.add_to(bird)\n\n#Add layer control\nlc = folium.LayerControl(collapsed=False)\nlc.add_to(bird)\n\nbird","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plot a Choropleth map for United States\nWe have seen from our countplot earlier that USA has the most number of observations in our training data. In this section we will like to visualize the distribution of the bird species within USA itself, across the different states, using a choropleth. To create a choropleth, we will first need the boundary of all the US states, usually contained within a GEOJson format. This is so that with the boundaries specified by the json file, we can then easily mapped the state boundary on our real map in folium.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# download GEOJson for mapping to choropleth\nurl = 'https://raw.githubusercontent.com/PublicaMundi/MappingAPI/master/data/geojson/us-states.json'\nurllib.request.urlretrieve(url ,'temp.json')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets append all the states in the json file into a list, so that we can use it to match with it in our *location* column","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('temp.json') as f:\n  data = json.load(f)\n\nstate_lst = []\n\nfor i in data['features']:\n    state_lst.append(i['properties']['name'])   #get a list of states from the json","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def statein(lst, sent):      #function for state name from 'location' column\n    for i in lst:\n        if i in sent:\n            return i\n        \nus_df = train_df[train_df['country']== \"United States\"]\nus_df['state'] = us_df['location'].apply(lambda x: statein(state_lst, x)) #extract state from location column into new column\n\nusa_choro = us_df.groupby(['state']).size().reset_index()  #get the numbers of training rows for each USA state\nusa_choro.columns = ['state','count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"choro = folium.Map(location=[37.0902, -95.7129], zoom_start=4, min_zoom=2)\n\nfolium.Choropleth(\n    geo_data='temp.json',\n    name='choropleth',\n    data=usa_choro,\n    bins=9,\n    columns=['state','count'], #'state' is the col name required to match with the key from json, 'count' is the value\n    key_on='feature.properties.name', #the name property in json will be matched to the 'state'. The names must matched for choropleth to work \n    fill_opacity=0.8,\n    line_opacity=0.5,\n    fill_color='BuPu',\n    legend_name=\"Distribution of Birds Species in USA\").add_to(choro)\n\nfolium.LayerControl().add_to(choro)\n\nchoro","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see most of the training observations collected from **California** and **Arizona**. As a curious minded individual, i went to did some research to understand what is so special about the geographical characteristics in California and Arizona that made the researchers concentrating their collection efforts in these few major states. What i found out is that the regions like California and Arizona has some of the largest birds diversity in USA. And probably because of the wide diversity of species in these regions, that made data collection more attractive in these few states. <br>\n![bird_diversity](https://biodiversitymapping.org/wordpress/wp-content/uploads/2016/11/Birds_USA_total_richness_large-1024x687.jpg)\nImage credits: https://biodiversitymapping.org/wordpress/index.php/usa-birds/","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"There is a very interesting [poster](https://www.stateofthebirds.org/2016/wp-content/uploads/2016/05/SotB_16-04-26-ENGLISH-BEST.pdf) on the diversity of birds in USA which i thought could help us better understand the situation in US","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Recordings Duration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nplt.title(\"Distribution of Recordings duration\")\nsns.distplot(train_df['duration'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Duration of Recordings (in seconds): \\n\")\nprint(train_df['duration'].describe())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Apparently, there is a recording that is 2283s long! ~38min","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.loc[train_df['duration']==2283]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Date","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"with plt.style.context('seaborn-darkgrid'):\n    plt.figure(figsize=(18, 10))\n    plt.title('Date')\n    train_df['date'].value_counts().sort_index().plot()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Top 10 dates with most recordings:\\n \")\nprint(train_df['date'].value_counts().sort_values(ascending=False).head(10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Something interesting we can see from the top 10 dates with the most recordings; they are usually in the month of **May** or **June**.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Rating","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nplt.title(\"Distribution of ratings\")\nsns.countplot(data=train_df,x='rating')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Bird Seen","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.countplot(data=train_df,x='bird_seen')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Sampling Rate","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nprint(\"Minimum Sampling Rate: \", train_df['sampling_rate'].min(), '\\t', \"Maximum Sampling Rate: \", train_df['sampling_rate'].max())\ntrain_df['sampling_rate'].value_counts().sort_index().plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Please Upvote this notebook if it has helped you in any ways :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"That sums up the basic exploration of our dataset. I will continue to update and add more insightful analysis along the way, and also building up into the next section of the workflow. I hope you have enjoyed this notebook. Do ping me up if you have any feedbacks or doubts.  \nThank you for reading :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}