Hi everyone!!
\nCongratulations to all the teams for your hard work tackling this DIFFICULT challenge! I can't wait to find out more about what you all tried, what worked, and what didn't.
\nWe put together the following google form to learn more about the winning methods. After reading though the form submissions we will announce the 1st, 2nd, and 3rd place winners (so you have to respond to be featured as a winner!) as well as up to two honorable mentions for novelty and creativity. If you scored in the top 10 please make sure you respond to be considered!!:
\nhttps://forms.gle/srCkcHDhwQg9aJx2A
We would also like to encourage all the teams to respond to this discussion thread to tell us what you tried e.g. what worked or what didn't work. From our past experience, sharing this information has been really valuable and insightful.
\nAll the best,
\nSara, Arushi, Eli, and Vighnesh
Hi everyone!!
\nCongratulations to all the teams for your hard work tackling this DIFFICULT challenge! I can't wait to find out more about what you all tried, what worked, and what didn't.
\nWe put together the following google form to learn more about the winning methods. After reading though the form submissions we will announce the 1st, 2nd, and 3rd place winners (so you have to respond to be featured as a winner!) as well as up to two honorable mentions for novelty and creativity. If you scored in the top 10 please make sure you respond to be considered!!:
\nhttps://forms.gle/srCkcHDhwQg9aJx2A
We would also like to encourage all the teams to respond to this discussion thread to tell us what you tried e.g. what worked or what didn't work. From our past experience, sharing this information has been really valuable and insightful.
\nAll the best,
\nSara, Arushi, Eli, and Vighnesh
Reminder to everyone to fill out the form if they would like to be eligible to officially place in the competition or receive a novelty award, both to be announced at the CVPR workshop! Please reply by this Friday, June 11th for consideration.
", "rawMarkdown": "Reminder to everyone to fill out the form if they would like to be eligible to officially place in the competition or receive a novelty award, both to be announced at the CVPR workshop! Please reply by this Friday, June 11th for consideration.", "votes": 1 }, { "id": 1329058, "postDate": "2021-05-30T19:45:56.833Z", "content": "I went through the top submissions a bit today and did some more detailed analysis, because I wasn't happy with how obtuse a single metric for detection+classification+counting ended up being (at least to me!!). Take a look at the rough breakdown in this google doc if you're interested! I will probably add some more analysis in the coming weeks (look into the clips with multiple species, run multi-label classification metrics instead of hacking to single label), and I hope to release the test labels after the CVPR workshop :)
", "rawMarkdown": "I went through the top submissions a bit today and did some more detailed analysis, because I wasn't happy with how obtuse a single metric for detection+classification+counting ended up being (at least to me!!). Take a look at the rough breakdown in this [google doc](https://docs.google.com/document/d/1jxhYdeVghOGkvonc2NKUDHHFV9EBEhjslAGkHlIoYys/edit?usp=sharing) if you're interested! I will probably add some more analysis in the coming weeks (look into the clips with multiple species, run multi-label classification metrics instead of hacking to single label), and I hope to release the test labels after the CVPR workshop :)", "votes": 2, "replies": [ { "id": 1329091, "postDate": "2021-05-30T20:39:17.380Z", "content": "The MCRMSE in your document doesn’t match the ones in the private leaderbord. How come ?
", "rawMarkdown": "The MCRMSE in your document doesn’t match the ones in the private leaderbord. How come ?" }, { "id": 1329157, "postDate": "2021-05-30T22:41:20.860Z", "content": "Ah, good point! It's because I ran these metrics on the entire test set, not just the private one. Since the public and private splits were also done per-location there is most likely a distribution shift between the two that may account for the discrepancy (and really, this gets at the crux of a lot of the challenges with trying to build generalizable camera trap models - the random choice of evaluation locations can matter a lot!). I'll run this analysis on the public and private splits separately soon so we can see the differences, but congratulations nonetheless on building such a robust and awesome solution!! I can't wait to hear what you guys did :)
", "rawMarkdown": "Ah, good point! It's because I ran these metrics on the entire test set, not just the private one. Since the public and private splits were also done per-location there is most likely a distribution shift between the two that may account for the discrepancy (and really, this gets at the crux of a lot of the challenges with trying to build generalizable camera trap models - the random choice of evaluation locations can matter a lot!). I'll run this analysis on the public and private splits separately soon so we can see the differences, but congratulations nonetheless on building such a robust and awesome solution!! I can't wait to hear what you guys did :)", "votes": 3 }, { "id": 1330359, "postDate": "2021-05-31T18:00:28.797Z", "content": "Just added metrics to the document for public test and private test for UFAM and Sensing Clues!
", "rawMarkdown": "Just added metrics to the document for public test and private test for UFAM and Sensing Clues!", "votes": 2 } ] }, { "id": 1330407, "postDate": "2021-05-31T18:58:16.083Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1339956, "author_name": "sbeery", "author_url": "", "post_date": "2021-06-07T14:59:59.953000", "content": "Reminder to everyone to fill out the form if they would like to be eligible to officially place in the competition or receive a novelty award, both to be announced at the CVPR workshop! Please reply by this Friday, June 11th for consideration.
", "votes": 1, "replies": [] }, { "id": 1329058, "author_name": "sbeery", "author_url": "", "post_date": "2021-05-30T19:45:56.833000", "content": "I went through the top submissions a bit today and did some more detailed analysis, because I wasn't happy with how obtuse a single metric for detection+classification+counting ended up being (at least to me!!). Take a look at the rough breakdown in this google doc if you're interested! I will probably add some more analysis in the coming weeks (look into the clips with multiple species, run multi-label classification metrics instead of hacking to single label), and I hope to release the test labels after the CVPR workshop :)
", "votes": 2, "replies": [ { "id": 1329091, "author_name": "JohnBeuving", "author_url": "", "post_date": "2021-05-30T20:39:17.380000", "content": "The MCRMSE in your document doesn’t match the ones in the private leaderbord. How come ?
", "votes": 0, "replies": [] }, { "id": 1329157, "author_name": "sbeery", "author_url": "", "post_date": "2021-05-30T22:41:20.860000", "content": "Ah, good point! It's because I ran these metrics on the entire test set, not just the private one. Since the public and private splits were also done per-location there is most likely a distribution shift between the two that may account for the discrepancy (and really, this gets at the crux of a lot of the challenges with trying to build generalizable camera trap models - the random choice of evaluation locations can matter a lot!). I'll run this analysis on the public and private splits separately soon so we can see the differences, but congratulations nonetheless on building such a robust and awesome solution!! I can't wait to hear what you guys did :)
", "votes": 3, "replies": [] }, { "id": 1330359, "author_name": "sbeery", "author_url": "", "post_date": "2021-05-31T18:00:28.797000", "content": "Just added metrics to the document for public test and private test for UFAM and Sensing Clues!
", "votes": 2, "replies": [] } ] }, { "id": 1330407, "author_name": "", "author_url": "", "post_date": "2021-05-31T18:58:16.083000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1328908": "Hi everyone!!\n\nCongratulations to all the teams for your hard work tackling this DIFFICULT challenge! I can't wait to find out more about what you all tried, what worked, and what didn't.\n\nWe put together the following google form to learn more about the winning methods. After reading though the form submissions we will announce the 1st, 2nd, and 3rd place winners (so you have to respond to be featured as a winner!) as well as up to two honorable mentions for novelty and creativity. If you scored in the top 10 please make sure you respond to be considered!!:\n[https://forms.gle/srCkcHDhwQg9aJx2A](https://forms.gle/srCkcHDhwQg9aJx2A)\n\nWe would also like to encourage all the teams to respond to this discussion thread to tell us what you tried e.g. what worked or what didn't work. From our past experience, sharing this information has been really valuable and insightful.\n\nAll the best,\nSara, Arushi, Eli, and Vighnesh\n\n", "1339956": "Reminder to everyone to fill out the form if they would like to be eligible to officially place in the competition or receive a novelty award, both to be announced at the CVPR workshop! Please reply by this Friday, June 11th for consideration.", "1329058": "I went through the top submissions a bit today and did some more detailed analysis, because I wasn't happy with how obtuse a single metric for detection+classification+counting ended up being (at least to me!!). Take a look at the rough breakdown in this [google doc](https://docs.google.com/document/d/1jxhYdeVghOGkvonc2NKUDHHFV9EBEhjslAGkHlIoYys/edit?usp=sharing) if you're interested! I will probably add some more analysis in the coming weeks (look into the clips with multiple species, run multi-label classification metrics instead of hacking to single label), and I hope to release the test labels after the CVPR workshop :)", "1330407": "" } }