{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Birds!\n![](https://i.imgur.com/9lcML5X.jpg)\n\nBirds come in so many varieties of colours, shapes, sizes... and voices?\n\nHumans are quite adept at recognizing each other's voices and so are ML models now. But how good are ML models in recognizing voices of bird species in their natural habitat along with the sounds of nature? We will have an answer to it at the end of this competition."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"## importing packages\nimport bokeh\nimport os\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport IPython.display as ipd\n\nfrom bokeh.layouts import column, row\nfrom bokeh.models import ColumnDataSource, LinearAxis, Range1d\nfrom bokeh.models.tools import HoverTool\nfrom bokeh.palettes import BuGn4\nfrom bokeh.plotting import figure, output_notebook, show\nfrom bokeh.transform import cumsum\n\noutput_notebook()\n\n\n## configuring setup, constants and parameters\nPATH_TRAIN = \"../input/birdsong-recognition/train.csv\"\nPATH_TEST = \"../input/birdsong-recognition/test.csv\"\n\nPATH_TRAIN_EXTENDED = \"../input/xeno-canto-bird-recordings-extended-a-m/train_extended.csv\"\n\nPATH_AUDIO = \"../input/birdsong-recognition/train_audio\"\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Overview of data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"## reading data\ndf_train = pd.read_csv(PATH_TRAIN)\ndf_test = pd.read_csv(PATH_TEST)\n\ndf_train_extended = pd.read_csv(PATH_TRAIN_EXTENDED)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let the shape of the test data not scare you. There is more. The excerpt below is from the Data page:\n> Only the first three rows are available for download; the full test.csv is in the hidden test set.\n\nThe test data has 150 recordings but the structure and length is different from train. This is a very simple kernel showing how to properly make a submission: https://www.kaggle.com/shonenkov/sample-submission-using-custom-check\n\n## Recordings\nThe main data is the audio files of the bird recordings. Let's hear the first sample from few of the species."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_bird_map = df_train[[\"ebird_code\", \"species\"]].drop_duplicates()\n\nfor ebird_code in os.listdir(PATH_AUDIO)[:20]:\n    species = df_bird_map[df_bird_map.ebird_code == ebird_code].species.values[0]\n    audio_file = os.listdir(f\"{PATH_AUDIO}/{ebird_code}\")[0]\n    audio_path = f\"{PATH_AUDIO}/{ebird_code}/{audio_file}\"\n    ipd.display(ipd.HTML(f\"<h2>{ebird_code} ({species})</h2>\"))\n    ipd.display(ipd.Audio(audio_path))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bird (or Species)\n***ebird_code*** is the target variable that needs to be predicted. Let's look at its distribution. There are two features with the same information: species is just a prettier (and complete) version of ebird_code. Note that the values from ebird_code are used for predictions."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_bird = df_train.groupby(\"species\")[\"filename\"].count().reset_index().rename(columns = {\"filename\": \"recordings\"}).sort_values(\"recordings\")\n\nsource = ColumnDataSource(df_bird)\ntooltips = [\n    (\"Bird Species\", \"@species\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv = figure(plot_width = 650, plot_height = 3000, y_range = df_bird.species.values, tooltips = tooltips, title = \"Count of Bird Species\")\nv.hbar(\"species\", right = \"recordings\", source = source, height = 0.75, color = \"steelblue\", alpha = 0.6)\n\nv.xaxis.axis_label = \"Count\"\nv.yaxis.axis_label = \"Species\"\n\nshow(v)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are totally **264 species** and a lot of them (**134 out of 264 = 50.76%**) have exactly **100 recordings**. That is the cutoff while preparing the data as mentioned by the competition host [here](https://www.kaggle.com/c/birdsong-recognition/discussion/159970).\n\n## Extended Xeno-Canto Data\nThe extended dataset of all recordings is available in these datasets:   \nhttps://www.kaggle.com/rohanrao/xeno-canto-bird-recordings-extended-a-m   \nhttps://www.kaggle.com/rohanrao/xeno-canto-bird-recordings-extended-n-z   \n\nThe competition host has confirmed that this data can be used for training models here:   \nhttps://www.kaggle.com/c/birdsong-recognition/discussion/159970\n\nLet's look at how much more data is available for each species."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_extended.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_bird_original = df_train.groupby(\"species\")[\"filename\"].count().reset_index().rename(columns = {\"filename\": \"recordings_original\"})\ndf_bird_extended = df_train_extended.groupby(\"species\")[\"filename\"].count().reset_index().rename(columns = {\"filename\": \"recordings_extended\"})\n\ndf_bird = df_bird_original.merge(df_bird_extended, on = \"species\", how = \"left\").fillna(0)\ndf_bird[\"recordings_total\"] = df_bird.recordings_original + df_bird.recordings_extended\ndf_bird = df_bird.sort_values(\"recordings_total\").reset_index()\n\nsource = ColumnDataSource(df_bird)\ntooltips = [\n    (\"Bird Species\", \"@species\"),\n    (\"Recordings Original\", \"@recordings_original\"),\n    (\"Recordings Extended\", \"@recordings_extended\"),\n]\n\nv = figure(plot_width = 650, plot_height = 3000, y_range = df_bird.species.values, tooltips = tooltips, title = \"Count of Bird Species\")\nv.hbar_stack([\"recordings_original\", \"recordings_extended\"], y = \"species\", source = source, height = 0.75, color = [\"steelblue\", \"crimson\"], alpha = 0.6)\n\nv.xaxis.axis_label = \"Count\"\nv.yaxis.axis_label = \"Species\"\n\nshow(v)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The original train data has **21,375 recordings** and the extended train data has an **additional 23,784 recordings** for **259 out of 264** species which more than doubles the training data size.\n\nThe additional data will probably be most useful for species where the original train data has fewer observations.\n\n## Datetime\nLet's look at when these recordings were taken."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_date = df_train.groupby(\"date\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\ndf_date.date = pd.to_datetime(df_date.date, errors = \"coerce\")\ndf_date.dropna(inplace = True)\ndf_date[\"weekday\"] = df_date.date.dt.day_name()\n\nsource_1 = ColumnDataSource(df_date)\n\ntooltips_1 = [\n    (\"Date\", \"@date{%F}\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nformatters = {\n    \"@date\": \"datetime\"\n}\n\nv1 = figure(plot_width = 700, plot_height = 400, x_axis_type = \"datetime\", title = \"Date of recording\")\nv1.line(\"date\", \"recordings\", source = source_1, width = 2, color = \"orange\", alpha = 0.6)\n\nv1.add_tools(HoverTool(tooltips = tooltips_1, formatters = formatters))\n\nv1.xaxis.axis_label = \"Date\"\nv1.yaxis.axis_label = \"Recordings\"\n\n\ndf_train[\"hour\"] = pd.to_numeric(df_train.time.str.split(\":\", expand = True)[0], errors = \"coerce\")\n\ndf_hour = df_train[~df_train.hour.isna()].groupby(\"hour\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\n\nsource_2 = ColumnDataSource(df_hour)\n\ntooltips_2 = [\n    (\"Hour\", \"@hour\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv2 = figure(plot_width = 450, plot_height = 400, tooltips = tooltips_2, title = \"Hour of recording\")\nv2.vbar(\"hour\", top = \"recordings\", source = source_2, width = 0.75, color = \"maroon\", alpha = 0.6)\n\nv2.xaxis.axis_label = \"Hour of day\"\nv2.yaxis.axis_label = \"Recordings\"\n\n\ndf_weekday = df_date.groupby(\"weekday\")[\"recordings\"].sum().reset_index().sort_values(\"recordings\", ascending = False)\n\nsource_3 = ColumnDataSource(df_weekday)\n\ntooltips_3 = [\n    (\"Weekday\", \"@weekday\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv3 = figure(plot_width = 250, plot_height = 400, x_range = df_weekday.weekday.values, tooltips = tooltips_3, title = \"Weekday of recording\")\nv3.vbar(\"weekday\", top = \"recordings\", source = source_3, width = 0.75, color = \"maroon\", alpha = 0.6)\n\nv3.xaxis.axis_label = \"Day of week\"\nv3.yaxis.axis_label = \"Recordings\"\n\nv3.xaxis.major_label_orientation = np.pi / 2\n\n\nshow(column(v1, row(v2, v3)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the recordings have been taken from **2012 onwards** but there are a few dating back more than 30 years! It may or may not be data issues.\n\nMajority of the recordings have taken place during the early hours of the day **(6am - 11am)**. That is quite interesting and likely to be the best time to find the birds.\n\nThe weekday counts indicate that the recordists enjoy spending their **weekends** with the birds!\n\n## Duration\nLet's look at how long are these audio files.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_duration = df_train.groupby(\"duration\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\n\nsource = ColumnDataSource(df_duration)\n\ntooltips = [\n    (\"Duration\", \"@duration\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv = figure(plot_width = 700, plot_height = 200, tooltips = tooltips, title = \"Duration of recording\")\nv.line(\"duration\", \"recordings\", source = source, width = 2, color = \"green\", alpha = 0.6)\n\nv.xaxis.axis_label = \"Duration\"\nv.yaxis.axis_label = \"Recordings\"\n\nshow(v)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the recordings are within **2 minutes** and its intuitive since you probably don't need too much time to record the sounds of a bird.\n\n## Location\nLet's look at where these recordings were taken."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_country = df_train.groupby(\"country\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\")\n\nsource = ColumnDataSource(df_country)\n\ntooltips_1 = [\n    (\"Country\", \"@country\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv1 = figure(plot_width = 650, plot_height = 1000, y_range = df_country.country.values, tooltips = tooltips_1, title = \"Country of recording\")\nv1.hbar(\"country\", right = \"recordings\", source = source, height = 0.75, color = \"coral\", alpha = 0.6)\n\nshow(v1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_location = df_train.groupby(\"location\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False).head(20).sort_values(\"recordings\")\n\nsource = ColumnDataSource(df_location)\n\ntooltips_2 = [\n    (\"Location\", \"@location\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv2 = figure(plot_width = 650, plot_height = 400, y_range = df_location.location, tooltips = tooltips_2, title = \"Top-20 Locations of recording\")\nv2.hbar(\"location\", right = \"recordings\", source = source, height = 0.75, color = \"coral\", alpha = 0.6)\n\nshow(v2)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_train[\"elevation_clean\"] = pd.to_numeric(df_train.elevation.str.replace(\"[^0-9]\", \"\"), errors = \"coerce\")\ndf_elevation = df_train[~df_train.elevation_clean.isna()].groupby(\"elevation_clean\")[\"species\"].count().reset_index().rename(columns = {\"elevation_clean\": \"elevation\", \"species\": \"recordings\"})\n\nsource_3 = ColumnDataSource(df_elevation[df_elevation.elevation < 4700])\n\ntooltips_3 = [\n    (\"Elevation\", \"@elevation\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv3 = figure(plot_width = 650, plot_height = 300, tooltips = tooltips_3, title = \"Elevation of Recording\")\nv3.line(\"elevation\", \"recordings\", source = source_3, width = 3, color = \"lightseagreen\", alpha = 0.6)\n\nv3.xaxis.axis_label = \"Elevation (in metres)\"\nv3.yaxis.axis_label = \"Recordings\"\n\nshow(v3)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_bird_country = df_train.groupby([\"species\", \"country\"])[\"ebird_code\"].count().reset_index()\ndf_bird_country = df_bird_country.merge(df_bird, on = \"species\")\ndf_bird_country = df_bird_country.merge(df_country, on = \"country\")\ndf_bird_country.rename(columns = {\"ebird_code\": \"recordings\", \"recordings_original\": \"recordings_species\", \"recordings\": \"recordings_country\"}, inplace = True)\ndf_bird_country[\"alpha\"] = 0.2 + (0.8 * (df_bird_country.recordings - min(df_bird_country.recordings)) / (max(df_bird_country.recordings) - min(df_bird_country.recordings)))\n\ndf_bird_country = df_bird_country.sort_values([\"recordings\"], ascending = False).reset_index(drop = True)\n\nsource_4 = ColumnDataSource(df_bird_country)\n\ntooltips_4 = [\n    (\"Species\", \"@species\"),\n    (\"Country\", \"@country\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nspecies = list(df_bird_country.species.unique())\nspecies.reverse()\n\nv4 = figure(\n    plot_width = 700,\n    plot_height = 2100,\n    x_range = list(df_bird_country.country.unique()),\n    y_range = species,\n    x_axis_location = \"above\",\n    tooltips = tooltips_4,\n    title = \"Species by Country\"\n)\n\nv4.rect(\"country\", \"species\", 0.9, 0.9, source = source_4, color = \"purple\", alpha = \"alpha\", line_color = None, hover_line_color = \"black\")\n\nv4.grid.grid_line_color = None\nv4.axis.axis_line_color = None\nv4.axis.major_tick_line_color = None\nv4.xaxis.major_label_text_font_size = \"6px\"\nv4.yaxis.major_label_text_font_size = \"7px\"\nv4.axis.major_label_standoff = 0\nv4.xaxis.major_label_orientation = np.pi / 2\n\nshow(v4)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The recordings come from **95 different countries** but a majority of them **(14284 out of 21375 = 66.82%)** are from **US**.   \n**Arizona and Florida** are the popular localities within the US where a large number of recordings were taken.\n\nAbout **2248 recordings (10.52%)** have been taken at sea-level (0 metres) and rest having a long tail upto a maximum elevation of **4600 mts**.\n\n## Miscellaneous\nLet's look at some distributions of various other features in the data.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_rating = df_train.groupby(\"rating\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\n\nsource_1 = ColumnDataSource(df_rating)\n\ntooltips_1 = [\n    (\"Rating\", \"@rating{0.0}\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv1 = figure(plot_width = 350, plot_height = 300, tooltips = tooltips_1, title = \"Distribution of Rating\")\nv1.vbar(\"rating\", top = \"recordings\", source = source_1, width = 0.4, color = \"lightseagreen\", alpha = 0.6)\n\nv1.xaxis.axis_label = \"Rating\"\nv1.yaxis.axis_label = \"Recordings\"\n\n\ndf_playback = df_train.groupby(\"playback_used\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\ndf_playback[\"percentage\"] = df_playback.recordings * 100 / df_playback.recordings.sum()\ndf_playback[\"angle\"] = df_playback.recordings / df_playback.recordings.sum() * 2 * np.pi\ndf_playback[\"color\"] = [\"mediumseagreen\", \"lightseagreen\"]\n\nsource_2 = ColumnDataSource(df_playback)\n\ntooltips_2 = [\n    (\"Playback Used\", \"@playback_used\"),\n    (\"Recordings\", \"@recordings\"),\n    (\"Percentage\", \"@percentage{0}%\")\n]\n\nv2 = figure(plot_width = 350, plot_height = 300, tooltips = tooltips_2, title = \"Distribution of Playback Used\")\nv2.wedge(x = 0, y = 1, radius = 0.4, start_angle = cumsum(\"angle\", include_zero = True), end_angle = cumsum(\"angle\"), line_color = \"white\", fill_color = \"color\", legend_field = \"playback_used\", source = source_2)\n\nv2.axis.axis_label = None\nv2.axis.visible = False\nv2.grid.grid_line_color = None\n\n\ndf_pitch = df_train.groupby(\"pitch\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False)\n\nsource_3 = ColumnDataSource(df_pitch)\n\ntooltips_3 = [\n    (\"Pitch\", \"@pitch\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv3 = figure(plot_width = 350, plot_height = 300, x_range = df_pitch.pitch.values, tooltips = tooltips_3, title = \"Distribution of Pitch\")\nv3.vbar(\"pitch\", top = \"recordings\", source = source_3, width = 0.4, color = \"lightseagreen\", alpha = 0.6)\n\nv3.xaxis.axis_label = \"Pitch\"\nv3.yaxis.axis_label = \"Recordings\"\n\n\ndf_channels = df_train.groupby(\"channels\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\ndf_channels[\"percentage\"] = df_channels.recordings * 100 / df_channels.recordings.sum()\ndf_channels[\"angle\"] = df_channels.recordings / df_channels.recordings.sum() * 2 * np.pi\ndf_channels[\"color\"] = [\"mediumseagreen\", \"lightseagreen\"]\n\nsource_4 = ColumnDataSource(df_channels)\n\ntooltips_4 = [\n    (\"Channel\", \"@channels\"),\n    (\"Recordings\", \"@recordings\"),\n    (\"Percentage\", \"@percentage{0}%\")\n]\n\nv4 = figure(plot_width = 350, plot_height = 300, tooltips = tooltips_4, title = \"Distribution of Channel\")\nv4.wedge(x = 0, y = 1, radius = 0.4, start_angle = cumsum(\"angle\", include_zero = True), end_angle = cumsum(\"angle\"), line_color = \"white\", fill_color = \"color\", legend_field = \"channels\", source = source_4)\n\nv4.axis.axis_label = None\nv4.axis.visible = False\nv4.grid.grid_line_color = None\n\n\ndf_speed = df_train.groupby(\"speed\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False)\n\nsource_5 = ColumnDataSource(df_speed)\n\ntooltips_5 = [\n    (\"Speed\", \"@speed\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv5 = figure(plot_width = 350, plot_height = 300, x_range = df_speed.speed.values, tooltips = tooltips_5, title = \"Distribution of Speed\")\nv5.vbar(\"speed\", top = \"recordings\", source = source_5, width = 0.4, color = \"lightseagreen\", alpha = 0.6)\n\nv5.xaxis.axis_label = \"Speed\"\nv5.yaxis.axis_label = \"Recordings\"\n\n\ndf_bird_seen = df_train.groupby(\"bird_seen\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\ndf_bird_seen[\"percentage\"] = df_bird_seen.recordings * 100 / df_bird_seen.recordings.sum()\ndf_bird_seen[\"angle\"] = df_bird_seen.recordings / df_bird_seen.recordings.sum() * 2 * np.pi\ndf_bird_seen[\"color\"] = [\"mediumseagreen\", \"lightseagreen\"]\n\nsource_6 = ColumnDataSource(df_bird_seen)\n\ntooltips_6 = [\n    (\"Bird Seen\", \"@bird_seen\"),\n    (\"Recordings\", \"@recordings\"),\n    (\"Percentage\", \"@percentage{0}%\")\n]\n\nv6 = figure(plot_width = 350, plot_height = 300, tooltips = tooltips_6, title = \"Distribution of Bird Seen\")\nv6.wedge(x = 0, y = 1, radius = 0.4, start_angle = cumsum(\"angle\", include_zero = True), end_angle = cumsum(\"angle\"), line_color = \"white\", fill_color = \"color\", legend_field = \"bird_seen\", source = source_6)\n\nv6.axis.axis_label = None\nv6.axis.visible = False\nv6.grid.grid_line_color = None\n\n\ndf_volume = df_train.groupby(\"volume\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False)\n\nsource_7 = ColumnDataSource(df_volume)\n\ntooltips_7 = [\n    (\"Volume\", \"@volume\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv7 = figure(plot_width = 350, plot_height = 300, x_range = df_volume.volume.values, tooltips = tooltips_7, title = \"Distribution of Volume\")\nv7.vbar(\"volume\", top = \"recordings\", source = source_7, width = 0.4, color = \"lightseagreen\", alpha = 0.6)\n\nv7.xaxis.axis_label = \"Volume\"\nv7.yaxis.axis_label = \"Recordings\"\n\n\ndf_train[\"filetype\"] = \"mp3\"\ndf_train.loc[df_train.file_type != \"mp3\", \"filetype\"] = \"other\"\n\ndf_file_type = df_train.groupby(\"filetype\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"})\ndf_file_type[\"percentage\"] = df_file_type.recordings * 100 / df_file_type.recordings.sum()\ndf_file_type[\"angle\"] = df_file_type.recordings / df_file_type.recordings.sum() * 2 * np.pi\ndf_file_type[\"color\"] = [\"mediumseagreen\", \"lightseagreen\"]\n\nsource_8 = ColumnDataSource(df_file_type)\n\ntooltips_8 = [\n    (\"File Type\", \"@filetype\"),\n    (\"Recordings\", \"@recordings\"),\n    (\"Percentage\", \"@percentage{0.000}%\")\n]\n\nv8 = figure(plot_width = 350, plot_height = 300, tooltips = tooltips_8, title = \"Distribution of File Type\")\nv8.wedge(x = 0, y = 1, radius = 0.4, start_angle = cumsum(\"angle\", include_zero = True), end_angle = cumsum(\"angle\"), line_color = \"white\", fill_color = \"color\", legend_field = \"filetype\", source = source_8)\n\nv8.axis.axis_label = None\nv8.axis.visible = False\nv8.grid.grid_line_color = None\n\n\ndf_sampling_rate = df_train.groupby(\"sampling_rate\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False)\n\nsource_9 = ColumnDataSource(df_sampling_rate)\n\ntooltips_9 = [\n    (\"Sampling Rate\", \"@sampling_rate\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv9 = figure(plot_width = 350, plot_height = 300, x_range = df_sampling_rate.sampling_rate.values, tooltips = tooltips_9, title = \"Distribution of Sampling Rate\")\nv9.vbar(\"sampling_rate\", top = \"recordings\", source = source_9, width = 0.4, color = \"lightseagreen\", alpha = 0.6)\n\nv9.xaxis.axis_label = \"Sampling Rate\"\nv9.yaxis.axis_label = \"Recordings\"\n\nv9.xaxis.major_label_orientation = np.pi / 4\n\n\nlicense_map = {\n    \"Creative Commons Attribution-NonCommercial-ShareAlike 3.0\": \"CC BY-NC-SA 3.0\",\n    \"Creative Commons Attribution-NonCommercial-ShareAlike 4.0\": \"CC BY-NC-SA 4.0\",\n    \"Creative Commons Attribution-ShareAlike 3.0\": \"CC BY-SA 3.0\",\n    \"Creative Commons Attribution-ShareAlike 4.0\": \"CC BY-SA 4.0\"\n}\n\ndf_train[\"license_abbr\"] = df_train.license.map(license_map)\n\ndf_license = df_train.groupby(\"license_abbr\")[\"species\"].count().reset_index().rename(columns = {\"license_abbr\": \"license\", \"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False)\ndf_license[\"percentage\"] = df_license.recordings * 100 / df_license.recordings.sum()\ndf_license[\"angle\"] = df_license.recordings / df_license.recordings.sum() * 2 * np.pi\ndf_license[\"color\"] = BuGn4\n\nsource_10 = ColumnDataSource(df_license)\n\ntooltips_10 = [\n    (\"License\", \"@license\"),\n    (\"Recordings\", \"@recordings\"),\n    (\"Percentage\", \"@percentage{0}%\")\n]\n\nv10 = figure(plot_width = 350, plot_height = 300, tooltips = tooltips_10, title = \"Distribution of License\")\nv10.wedge(x = 0, y = 1, radius = 0.4, start_angle = cumsum(\"angle\", include_zero = True), end_angle = cumsum(\"angle\"), line_color = \"white\", fill_color = \"color\", legend_field = \"license\", source = source_10)\n\nv10.axis.axis_label = None\nv10.axis.visible = False\nv10.grid.grid_line_color = None\n\nv10.legend.label_text_font_size = \"6pt\"\n\n\nshow(column(row(v1, v2), row(v3, v4), row(v5, v6), row(v7, v8), row(v9, v10)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the recordings have a **high rating** because that was the basis on which the samples of 100 were chosen as mentioned by the competition host [here](https://www.kaggle.com/c/birdsong-recognition/discussion/159970#893042).\n\nThere are sizeable missing values **(Not Specified)** in Pitch, Speed and Volume. The channels **mono and stereo** are close to **equally distributed** while two frequencies **44100 (Hz) and 48000 (Hz)** cover **98.55%** of recordings.\n\nThere are only 8 recordings that are not in mp3 format **(6 are .wav, 1 is .aac, 1 is .mp2)** which probably can be ignored during modelling but need to be careful if they are present in test data and require special processing.\n\n## Recordist\nThis dataset would not be possible without the actual people on the field who took these recordings.\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_recordist = df_train.groupby(\"recordist\")[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False).head(20)\n\nsource_1 = ColumnDataSource(df_recordist)\n\ntooltips_1 = [\n    (\"Recordist\", \"@recordist\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv1 = figure(plot_width = 650, plot_height = 400, x_range = df_recordist.recordist.values, tooltips = tooltips_1, title = \"Top-20 Recordists\")\nv1.vbar(\"recordist\", top = \"recordings\", source = source_1, width = 0.75, color = \"olive\", alpha = 0.8)\n\nv1.xaxis.axis_label = \"Recordist\"\nv1.yaxis.axis_label = \"Recordings\"\n\nv1.xaxis.major_label_orientation = np.pi / 4\n\ndf_recordist_country = df_train.groupby([\"recordist\", \"country\"])[\"species\"].count().reset_index().rename(columns = {\"species\": \"recordings\"}).sort_values(\"recordings\", ascending = False).drop_duplicates(\"country\").head(20)\ndf_recordist_country[\"recordist_country\"] = df_recordist_country.recordist + \" (\" + df_recordist_country.country + \")\"\n\nsource_2 = ColumnDataSource(df_recordist_country)\n\ntooltips_2 = [\n    (\"Recordist\", \"@recordist\"),\n    (\"Country\", \"@country\"),\n    (\"Recordings\", \"@recordings\")\n]\n\nv2 = figure(plot_width = 650, plot_height = 400, x_range = df_recordist_country.recordist_country.values, tooltips = tooltips_2, title = \"Top-20 Recordists across Countries\")\nv2.vbar(\"recordist_country\", top = \"recordings\", source = source_2, width = 0.75, color = \"olive\", alpha = 0.8)\n\nv2.xaxis.axis_label = \"Recordist\"\nv2.yaxis.axis_label = \"Recordings\"\n\nv2.xaxis.major_label_orientation = np.pi / 4\n\n\nshow(column(v1, v2))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is [Paul Marvin's profile page](https://www.xeno-canto.org/contributor/RFTXRYBVBX) and his bio:\n>I live in San Diego, California part time and on the road when I can. Until recently, I lived in Cocoa, Florida and also visited my family a lot in San Diego.\nMostly since 2011 I've been recording bird vocalizations from all around the lower 48 states of the United States.\n\nThank You to all the **949 recordists** (not just the Top-20!) and [Cornell Lab of Ornithology’s Center for Conservation Bioacoustics (CCB)](https://www.birds.cornell.edu/ccb/) for this dataset."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}