{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Species Co-Occurrence Based on eBird Checklists\n<a id=\"intro\"></a>\n\nThis notebook is intended for support of [BirdCLEF 2022](https://www.kaggle.com/competitions/birdclef-2022). It's laughably late in the competition, but maybe it'll help someone or future students. The output of this notebook is a co-occurrence matrix of bird species at various birding hotspots in a variety of habitats. The hypothesis is that distinct habitats will have different species present: you won't find a shorebird such as a Dunlin (`dunlin`) in the upland forests where you might find (if you're lucky) an iʻiwi (`iiwi`).\n\n[![birdlcef.jpg](https://i.postimg.cc/xTpGV2qW/birdlcef.jpg)](https://postimg.cc/gnRL31VD)\n\n<a id=\"get-data\"></a>\n## Getting and using eBird data\n\nThe data in this notebook come from eBird, are owned or licensed by the Cornell Lab of Ornithology, and are shared in accordance with [eBird's terms and conditions](https://www.birds.cornell.edu/home/terms-of-use/) and the rules of Kaggle's BirdCLEF 2022. Editorial comments are my own and should not be lent any credence.\n\nTo run this notebook, you will need an eBird API key, which you can get by choosing ['Request access' from the **API Key** row here](https://ebird.org/data/download). Once you have an API key, in this notebook select the **Add-ons** menu, **Secrets** item. Create a new secret with the name `ebird_api_key` and set the value to your API key. \n\n[![Screen-Shot-2022-05-07-at-10-12-08-AM.png](https://i.postimg.cc/wTRM38wK/Screen-Shot-2022-05-07-at-10-12-08-AM.png)](https://postimg.cc/561bkGcp)\n\n<a id=\"missing-data\"></a>\n## A quick note about missing species\n\nA third of species present in the training data for BirdCLEF 2022 are not in the co-occurrence matrix. The easiest interpretation is that the birds are uncommon enough that no one has reported sightings of them, but that doesn't mean they don't occur. There was an `incter1` at South Point this Winter that every bird photographer on the Big Island mobbed, but no eBird records. Or, in the case of the Puaohi `puaioh`, at least, eBird will not share known locations, since doing so may trigger an influx of potentially-disruptive viewers. \n\nI have made some notes about the species that are missing. Pure speculation on my part. \n\n<a id=\"olelo\"></a>\n## What's with the \"apostrophes\" and \"long vowels\"? \n\nʻōlelo Hawaiʻi has relatively more vowels and diphthongs than consonants compared to English. Sometimes vowels are drawn out a little longer, which is indicated with a kahakō -- the macron over the vowel. That's pretty hard to hear, but the ʻokina (\"apostrophe\") is very easy to hear: it's a glottal stop, like when you say the word \"uh-oh\" and there's a clean break between the \"uh\" and the \"oh\". So \"Hawaiʻi\" is not \"hah-why-yee\", it's \"hah-why-ee\" (well actually, it's more like \"ha-vye-ee\" but that's kind of pretentious. Only someone who still uses \"data are\" would try to pull that off... *ahem*... Anyway, moving on...). ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"<a id=\"init\"></a>\n## Initialization","metadata":{}},{"cell_type":"code","source":"!python -m pip install ebird-api","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:32.630764Z","iopub.execute_input":"2022-05-08T02:00:32.631061Z","iopub.status.idle":"2022-05-08T02:00:46.471189Z","shell.execute_reply.started":"2022-05-08T02:00:32.631014Z","shell.execute_reply":"2022-05-08T02:00:46.47015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from enum import Enum\nfrom dataclasses import dataclass\nfrom typing import Dict\n\nfrom tqdm import tqdm \nimport time\n\nimport pandas as pd\nimport numpy as np\nimport itertools\nfrom kaggle_secrets import UserSecretsClient\n\nimport seaborn as sns # Easy-to-use heatmap plot\n\nfrom ebird.api import Client","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:46.47336Z","iopub.execute_input":"2022-05-08T02:00:46.47359Z","iopub.status.idle":"2022-05-08T02:00:47.746354Z","shell.execute_reply.started":"2022-05-08T02:00:46.473554Z","shell.execute_reply":"2022-05-08T02:00:47.745053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Species abbreviations in Bird CLEF contest are 1:1 correspondence to eBird field codes for that species. So these are all valid values in `myRecord.speciesCode)`\nspecies = 'afrsil1 akekee akepa1 akiapo akikik amewig aniani apapan arcter \\\n                      barpet bcnher belkin1 bkbplo bknsti bkwpet blkfra blknod bongul \\\n                      brant brnboo brnnod brnowl brtcur bubsan buffle bulpet burpar buwtea \\\n                      cacgoo1 calqua cangoo canvas caster1 categr chbsan chemun chukar cintea \\\n                      comgal1 commyn compea comsan comwax coopet crehon dunlin elepai ercfra eurwig \\\n                      fragul gadwal gamqua glwgul gnwtea golphe grbher3 grefri gresca gryfra gwfgoo \\\n                      hawama hawcoo hawcre hawgoo hawhaw hawpet1 hoomer houfin houspa hudgod iiwi incter1 \\\n                      jabwar japqua kalphe kauama laugul layalb lcspet leasan leater1 lessca lesyel lobdow lotjae \\\n                      madpet magpet1 mallar3 masboo mauala maupar merlin mitpar moudov norcar norhar2 normoc norpin \\\n                      norsho nutman oahama omao osprey pagplo palila parjae pecsan peflov perfal pibgre pomjae puaioh \\\n                      reccar redava redjun redpha1 refboo rempar rettro ribgul rinduc rinphe rocpig rorpar rudtur ruff \\\n                      saffin sander semplo sheowl shtsan skylar snogoo sooshe sooter1 sopsku1 sora spodov sposan \\\n                      towsol wantat1 warwhe1 wesmea wessan wetshe whfibi whiter whttro wiltur yebcar yefcan zebdov'.split()","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.747657Z","iopub.execute_input":"2022-05-08T02:00:47.747873Z","iopub.status.idle":"2022-05-08T02:00:47.754857Z","shell.execute_reply.started":"2022-05-08T02:00:47.747845Z","shell.execute_reply":"2022-05-08T02:00:47.753138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This is my internal model of likely soundscapes in Hawai'i\n\nclass Habitat(Enum):\n    UplandForest = 0,\n    LowlandForest = 1,\n    UplandDry = 2,\n    Grasslands = 3,\n    Cloudforest = 4,\n    Swamp = 5,\n    Human = 6,\n    Shoreline = 7,\n    Pelagic = 8,\n    Freshwater = 9","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.75741Z","iopub.execute_input":"2022-05-08T02:00:47.757767Z","iopub.status.idle":"2022-05-08T02:00:47.774997Z","shell.execute_reply.started":"2022-05-08T02:00:47.757738Z","shell.execute_reply":"2022-05-08T02:00:47.774186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"@dataclass\nclass Hotspot:\n    name : str\n    habitat : Habitat\n    location_code : str","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.776349Z","iopub.execute_input":"2022-05-08T02:00:47.776771Z","iopub.status.idle":"2022-05-08T02:00:47.788639Z","shell.execute_reply.started":"2022-05-08T02:00:47.776732Z","shell.execute_reply":"2022-05-08T02:00:47.787679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# These are some notable hotspots, mostly on Hawai'i Island, but Hosmer's on Maui and Alakai's on Kauai.\n# These might be helpful for characterizing a soundscape, given a signature species \n\nhotspots = [\n    Hotspot('keahole', Habitat.Shoreline, 'L2574849'),\n    Hotspot('kaloko', Habitat.Shoreline, 'L331009'),\n    Hotspot('pu`u la`au', Habitat.UplandDry, 'L285813'),\n    Hotspot('VNP Thurston', Habitat.LowlandForest, 'L837288'),\n    Hotspot('Hakalau', Habitat.UplandForest, 'L695639'),\n    Hotspot('Hosmer Grove', Habitat.UplandForest, 'L247163'),\n    Hotspot('Kaloka Mauka', Habitat.Cloudforest, 'L8599253'),\n    Hotspot('Kailua Kona', Habitat.Human, 'L577366'),\n    Hotspot('Waiki`i', Habitat.Grasslands, 'L1430539'),\n    Hotspot('Alakai Swamp Trail', Habitat.Swamp, 'L1548049'),\n    Hotspot('Honokohau Offshote', Habitat.Pelagic, 'L1662616'),\n    Hotspot('Wailoa River', Habitat.Freshwater, 'L868863')\n]","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.789991Z","iopub.execute_input":"2022-05-08T02:00:47.790299Z","iopub.status.idle":"2022-05-08T02:00:47.804933Z","shell.execute_reply.started":"2022-05-08T02:00:47.790259Z","shell.execute_reply":"2022-05-08T02:00:47.804045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#To run this notebook, you will need an eBird API key, which you can get by choosing 'Request access' from the **API Key** row at \n# https://ebird.org/data/download \n# Once you have an API key, in this notebook select the **Add-ons** menu, **Secrets** item. \n# Create a new secret with the name `ebird_api_key` and set the value to your API key. \n\nuser_secrets = UserSecretsClient()\napi_key = user_secrets.get_secret(\"ebird_api_key\")","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.806395Z","iopub.execute_input":"2022-05-08T02:00:47.806908Z","iopub.status.idle":"2022-05-08T02:00:47.961326Z","shell.execute_reply.started":"2022-05-08T02:00:47.806869Z","shell.execute_reply":"2022-05-08T02:00:47.960032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"locale = 'en'\nclient = Client(api_key, locale)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.963072Z","iopub.execute_input":"2022-05-08T02:00:47.963548Z","iopub.status.idle":"2022-05-08T02:00:47.967546Z","shell.execute_reply.started":"2022-05-08T02:00:47.963502Z","shell.execute_reply":"2022-05-08T02:00:47.966873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FETCH_FROM_EBIRD = True # Or False to re-use data from an earlier run\nif FETCH_FROM_EBIRD :\n    df = pd.DataFrame()\n    for s in tqdm(species):\n        rs = client.get_species_observations(s, 'US-HI')\n        for r in rs:\n            record_df  = pd.DataFrame([r], columns=r.keys())\n            df = pd.concat([df, record_df], axis =0).reset_index(drop=True)\n    filtered = df[df.howMany > 0]\n    filtered.to_csv(\"ebird_recs.csv\", index = False)\nelse:\n    filtered = pd.read_csv(\"../input/ebird_recs.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:00:47.968777Z","iopub.execute_input":"2022-05-08T02:00:47.969169Z","iopub.status.idle":"2022-05-08T02:01:29.098766Z","shell.execute_reply.started":"2022-05-08T02:00:47.969134Z","shell.execute_reply":"2022-05-08T02:01:29.098071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"ebird-recs\"></a>\n## Take a look at the eBird records\n\nJust a quick look at what you get from eBird... ","metadata":{}},{"cell_type":"code","source":"filtered[filtered.locId == 'L695639']","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:29.101156Z","iopub.execute_input":"2022-05-08T02:01:29.101359Z","iopub.status.idle":"2022-05-08T02:01:29.136376Z","shell.execute_reply.started":"2022-05-08T02:01:29.101331Z","shell.execute_reply":"2022-05-08T02:01:29.134996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"missing-species\"></a>\n## Missing species\n\nNot all species in training data have records in Hawaiʻi in eBird.","metadata":{"execution":{"iopub.status.busy":"2022-05-07T00:58:49.094435Z","iopub.execute_input":"2022-05-07T00:58:49.095275Z","iopub.status.idle":"2022-05-07T00:58:49.113237Z","shell.execute_reply.started":"2022-05-07T00:58:49.095238Z","shell.execute_reply":"2022-05-07T00:58:49.112241Z"}}},{"cell_type":"code","source":"found_species = filtered.speciesCode.unique()\nmissing_species = set(species) - set(found_species)\nprint(sorted(missing_species))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:29.1381Z","iopub.execute_input":"2022-05-08T02:01:29.138419Z","iopub.status.idle":"2022-05-08T02:01:29.145492Z","shell.execute_reply.started":"2022-05-08T02:01:29.138379Z","shell.execute_reply":"2022-05-08T02:01:29.144436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"missing-notes\"></a>\n## Notes on missing species\n\n- `akekee` : Akekee are critically-endangered, single habitat is above mosquito line in Kauai\n- `akikik` : Akikiki are critically-endangered, single habitat is above mosquito line in Kauai\n- `amewig` : American Wigeon. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `belkin1` : Belted Kingfisher. I've never seen a Kingfisher in Hawaiʻi. Presumably co-occurs with Wailoa River (L868863)\n- `bongul` : Bonaparte's Gull. Like all gulls but Laughing Gull uncommon-rare in Hawaiʻi.\n- `bubsan`: Buff-breasted sandpiper. Cute as anything. Shoreline habitat. \n- `buffle` : Bufflehead. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `buwtea` : Blue-winged teal. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `cangoo` : Canadian goose. I suppose the key co-occur for this would be Nene (`hawgoo`)\n- `canvas`: Canvasback. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `caster1`: Caspian tern. I've never seen one in Hawaiʻi. Pelagic. \n- `cintea` : Cinnamon teal. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863).\n- `comsan`: Common sandpiper. Shoreline habitat. \n- `crehon` : Akohekohe. Upland forest habitat.  Found only on Maui.\n- `eurwig` : European wigeon. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `fragul` : Franklin's gull. Like all gulls but Laughing Gull uncommon-rare in Hawaiʻi.\n- `gadwal` : Gadwall. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `gamqua` : Gambell's Quail. Desert bird, so Upland Dry, I suppose, possible co-occur with Palila\n- `glwgul` : Glaucous gull. Like all gulls but Laughing Gull uncommon-rare in Hawaiʻi.\n- `gnwtea` : Green-winged teal.  Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `golphe` : Golden pheasant. Wow, I didn't know we had these. Uhh... I guess anywhere you'd get Kalij Pheasant (`kalphe`)? \n- `grbher3` : Great blue heron. Never seen one in Hawaiʻi. Probably anywhere you'd get a Night Heron (`bcnher`)\n- `gresca` : Greater scaup.  Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `gwfgoo` : Greater white-fronted goose. I suppose the key co-occur for this would be Nene (`hawgoo`)\n- `hoomer` : Hooder merganser. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `hudgod` : Hudsonian godwit. Shoreline habitat. Kaloko Fishpond (L331009).\n- `incter1` : Inca tern. Rare enough so that every birder and photographer on Hawaiʻi Island mobbed the single individual at South Point this winter. I'm pretty sure this ain't gonna' be in the test-set. (But, uh, Shoreline or Pelagic habitat.)\n- `japqua` : Japanese quail. Grassland habitat, maybe Upland Dry. \n- `leasan` : Least sandpiper. Shoreline habitat. \n- `lessca` : Lesser scaup. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `lesyel` : Lesser yellowlegs. Shoreline habitat.\n- `lobdow` : Long-billed dowitcher. Shoreline habitat. \n- `madpet` : Zino's petrel. Pelagic habitat. \n- `magpet1` : Magenta petrel. Critically endangered. Found only in South Pacific. Pelagic habitat. Virtually zero chance in test set. \n- `maupar` : Maui parrotbill. Critically endangered. Upland Forest. Found only in Maui. \n- `merlin` : Merlin. Fast raptor common on mainland US. There was one not far from L285813 about 5 years ago. So Upland Dry, but I doubt it's in the test set.\n- `mitpar` : Mitred parakeet. Co-occurs with `rempar`. Maybe uncommon, but definitely not rare. Keep this one in your training set for Human habitats.\n- `norhar2` : Northern Harrier. Never seen one in Hawaii, presumably co-occurs would be anywhere you have Iʻo (`hawhaw`)\n- `norpin` : Northern pintail. Duck, so presumably co-occurs from Kaloko Fishpond (L331009) or Wailoa River (L868863)\n- `osprey` : Osprey. Never seen one in Hawaii. Pelagic, Riverine, Shoreline habitats. If I \"heard an Osprey\", I'd guess `hawhaw`. \n- `parjae` : Parasitic jaeger. Pelagic habitat. Seen 'em, never heard one. \n- `pecsan` : Pectoral sandpiper. Shoreline habitat. \n- `puaioh` : Puaiohi are critically-endangered, single habitat is above mosquito line in Kauai. (On eBird's **sensitive species** list, so will not show in checklists.)\n- `redpha1` : Red phalarope. Seasonally common in Pelagic habitat. \n- `ribgul` : Ring-billed gull. Like all gulls but Laughing Gull uncommon-rare in Hawaiʻi.\n- `ruff` : Ruff. Shoreline habitat, presumably co-occurs would be from Kaloko Fishpond (L331009). Never seen one, but I think they may only be \"uncommon\" and not all-out \"rare\". \n- `shtsan` : Sharp-tailed sandpiper.  Shoreline habitat. \n- `sopsku1`: South-polar skua. Pelagic habitat. \n- `sora` : Sora are small rails. Presumably co-occurs would be Wailoa River (L868863).\n- `sposan` : Spotted sandpiper. Shoreline habitat. \n- `towsol` : Towsend's solitaire. Never heard of it. I guess it would be Upland or Lowland Forest and maybe Human habitats (?). \n- `wessan` : Western sandpiper. Shoreline habitat.\n\n## More opinions on uncommon / rare species\n\nIʻo (`hawhaw`, \"Hawaiian Hawk\") are locally common -- I see them every day. If you get a hit on a hawk call (`\"cheer\"`), it's _almost certainly_ a `hawhaw`.\n\nThere are a lot of locally-uncommon ducks. These might be confounders or it might be the case that there's a \"duck pond\" hotspot in the test set. \n\nDitto sandpipers. I expect there are some shoreline soundscapes in the test, but I sincerely doubt they've caught some of these rarities in a natural setting.\n\nAs for the rare pelagics: No way. Unless they cut and paste a recording of a Magenta Petrel into a pelagic soundscape, it's gonna' be a bad guess. I spend a lot of time offshore and 99% of the birds you see you don't hear. If there's a sea-based soundscape in the test set, I'd guess it's either going to be just off a breeding site for tropic birds (`whttro` and `redtro`) or noddies (`blknod`, `brwnod`). \n\nPelagics in general: My guess is that the test set might well have land-based breeding and nesting calls of locally-rare pelagic birds (albatrosses, for instance, and very possibly the endangered petrel ʻuaʻu `hawpet1`). There's been a big effort to create safe breeding areas to reintroduce some of these birds to the main islands and finding and monitoring nests is a big challenge.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"cooccur\"></a>\n## Co-occurrence","metadata":{}},{"cell_type":"code","source":"co_occurs = {} # Dict<SpeciesCode> -> Dict<SpeciesCode> -> TimesPresent (not sum(HowMany))  \nlocs = filtered.groupby(filtered.locId)\nfor _, loc_records_df in locs:\n    species_here = loc_records_df.speciesCode.unique()\n    location_co_occurs = list(itertools.product(species_here, species_here))   \n    records_here_by_species = loc_records_df.groupby(loc_records_df.speciesCode)\n    for k1 in species_here:\n        if k1 not in co_occurs : \n            co_occurs[k1] = {}\n        k1_co_occurs = co_occurs[k1]\n        for k2 in species_here:\n            if k2 not in k1_co_occurs : \n                k1_co_occurs[k2] = 0\n            k1_co_occurs[k2] += 1 # Presence noted \n\n# Convert to Pandas and sort both rows and columns by field code\ndf = pd.DataFrame(co_occurs).sort_index()\ndf = df.reindex(sorted(df.columns), axis=1)\ndf.to_csv(\"ebird_co_occurrances_absolute.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:29.14686Z","iopub.execute_input":"2022-05-08T02:01:29.147109Z","iopub.status.idle":"2022-05-08T02:01:29.394723Z","shell.execute_reply.started":"2022-05-08T02:01:29.147079Z","shell.execute_reply":"2022-05-08T02:01:29.393741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"co_occur_prob_df = df.copy()\nfor species in co_occur_prob_df.columns:\n    times_seen = co_occur_prob_df[species][species] # Co-occurs w. self == count\n    co_occur_prob_df[species] = co_occur_prob_df[species] / times_seen\nco_occur_prob_df.to_csv(\"ebird_co_occurrances_probabilities.csv\")\nco_occur_prob_df","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:29.395984Z","iopub.execute_input":"2022-05-08T02:01:29.396639Z","iopub.status.idle":"2022-05-08T02:01:29.449511Z","shell.execute_reply.started":"2022-05-08T02:01:29.396563Z","shell.execute_reply":"2022-05-08T02:01:29.449094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set(rc={'figure.figsize':(30,30)})\nsns.heatmap(co_occur_prob_df, cmap=sns.color_palette(\"viridis_r\", as_cmap=True))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:16:40.095419Z","iopub.execute_input":"2022-05-08T02:16:40.095707Z","iopub.status.idle":"2022-05-08T02:16:43.52445Z","shell.execute_reply.started":"2022-05-08T02:16:40.095683Z","shell.execute_reply":"2022-05-08T02:16:43.523316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Probability = float # Range: 0.0 - 1.0\nSpecies = str # Field code\n\ndef probability_of_s2_given_s1(prob_df : pd.DataFrame, s1 : Species, s2 : species) -> Probability : \n    return prob_df[s1][s2]\n\n# Apapane are widespread upland forest birds, I'iwi are in fewer locations\ncommon_given_rare = probability_of_s2_given_s1(co_occur_prob_df, 'iiwi', 'apapan')\nrare_given_common = probability_of_s2_given_s1(co_occur_prob_df, 'apapan', 'iiwi')\n# Probs should be ~1.0 of Apapane (widespread) if I'iwi (sparser)\n(common_given_rare, rare_given_common)\n","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:33.924164Z","iopub.execute_input":"2022-05-08T02:01:33.924546Z","iopub.status.idle":"2022-05-08T02:01:33.933474Z","shell.execute_reply.started":"2022-05-08T02:01:33.924515Z","shell.execute_reply":"2022-05-08T02:01:33.932297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def co_occurrence_probabilities(prob_df : pd.DataFrame, given: Species) -> Dict[Species, Probability] : \n    return prob_df[given].dropna().to_dict()\n   \nco_occurrence_probabilities(co_occur_prob_df, 'iiwi')","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:33.934416Z","iopub.execute_input":"2022-05-08T02:01:33.934584Z","iopub.status.idle":"2022-05-08T02:01:33.953733Z","shell.execute_reply.started":"2022-05-08T02:01:33.934563Z","shell.execute_reply":"2022-05-08T02:01:33.95279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(co_occurrence_probabilities(co_occur_prob_df, 'zebdov')), len(co_occurrence_probabilities(co_occur_prob_df, 'iiwi'))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:33.95494Z","iopub.execute_input":"2022-05-08T02:01:33.955835Z","iopub.status.idle":"2022-05-08T02:01:33.978647Z","shell.execute_reply.started":"2022-05-08T02:01:33.955762Z","shell.execute_reply":"2022-05-08T02:01:33.977172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"cooccur-notes\"></a>\n## Notes\n\nThis looks pretty reasonable to me: \n\n- widespread commoners like mynahs, house finches and sparrows, doves, and white-eyes. \n- birder target species like i'iwi have reasonable co-occurs although biased towards birder interest: you don't truly have a 25% chance of a pueo (`sheowl`) given an i'iwi (`iiwi`) in any soundscape I know of. But in a couple hours / single checklist I can likely bag a (rare) `sheowl` and (rare) `iiwi`. ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"sentinels\"></a>\n## Signature species for habitats / soundscapes\n\nTODO: What birds are most characteristic of a particular habitat? \n\n* Expand `hotspots` list\n* Use `client.get_observations(locationCode)` for h in hotspots\n* Add `habitat` col to obs\n* You want the most common bird seen in `h` that is least likely to occur in `!h`\n* This might be more effort than worth, since observed != heard. You have to use domain knowledge, i.e., tropic birds are noisier than noddies. \n\n","metadata":{}},{"cell_type":"markdown","source":"<a id=\"scorebirds\"></a>\n## Only scoring birds","metadata":{}},{"cell_type":"code","source":"scored_birds = [\"akiapo\", \"aniani\", \"apapan\", \"barpet\", \"crehon\", \"elepai\", \"ercfra\", \"hawama\", \"hawcre\", \"hawgoo\", \"hawhaw\", \"hawpet1\", \"houfin\", \"iiwi\", \"jabwar\", \"maupar\", \"omao\", \"puaioh\", \"skylar\", \"warwhe1\", \"yefcan\"]\nunscored_birds = set(found_species) - set(scored_birds)\nprint(sorted(unscored_birds))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:01:55.647441Z","iopub.execute_input":"2022-05-08T02:01:55.648387Z","iopub.status.idle":"2022-05-08T02:01:55.655245Z","shell.execute_reply.started":"2022-05-08T02:01:55.648343Z","shell.execute_reply":"2022-05-08T02:01:55.654144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"They aren't even _trying_ to find palila? 😢\n\n...\n\nRows (note transpose in spec) are species that is \"given\". Color is probability of col species.  ","metadata":{}},{"cell_type":"code","source":"sns.set(rc={'figure.figsize':(30,4)})\nxdf = co_occur_prob_df.copy()\n#xdf = xdf.drop(index = unscored_birds)\nxdf = xdf.drop(columns = unscored_birds)\nsns.heatmap(xdf.T, cmap=sns.color_palette(\"viridis_r\", as_cmap=True))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:29:14.975751Z","iopub.execute_input":"2022-05-08T02:29:14.976042Z","iopub.status.idle":"2022-05-08T02:29:18.092139Z","shell.execute_reply.started":"2022-05-08T02:29:14.976Z","shell.execute_reply":"2022-05-08T02:29:18.091112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set(rc={'figure.figsize':(8,8)})\nxdf = xdf.drop(index = unscored_birds)\nsns.heatmap(xdf.T, cmap=sns.color_palette(\"viridis_r\", as_cmap=True))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T02:29:18.093739Z","iopub.execute_input":"2022-05-08T02:29:18.093954Z","iopub.status.idle":"2022-05-08T02:29:18.5928Z","shell.execute_reply.started":"2022-05-08T02:29:18.093928Z","shell.execute_reply":"2022-05-08T02:29:18.59208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}}]}