{ "id": 200757, "title": "Understanding the Evaluation Metric: Label Weighted LRAP", "url": "/competitions/rfcx-species-audio-detection/discussion/200757", "author_name": "", "post_date": "2020-12-01T18:43:32.317185100Z", "votes": 13, "comment_count": 3, "views": 0, "content": "

Before diving deep into the metric, let's understand what we need to submit, it's vector dimensions and formats.

\n

Submission Formats

\n

We have got 24 classes and some 9000 training samples in total. No, this much data is not sufficient to understand the Label weighted LRAP (label ranking average precision) metric. First, have a look at the below data chunk.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
---recording_idspecies_idsongtype_idt_minf_mint_maxf_max
000204008d21113.84003281.250014.93334125.0000
100204008d8124.49603750.000028.61875531.2500
200204008d4115.00272343.750016.85874218.7500
\n

What did you see? Well, if it didn't catch your attention, Let me explain as per my understanding. For single recording id 00204008d, we have got 3 species ids in that, i.e. 21, 8, and 4. What does that mean? The recording, which means the audio sample having this mentioned id, has audio of species 24, species 8, and species 4. Hence, our ground truth for this data sample would be:

\n
\n

[0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1].

\n
\n

But what we need to predict? We need to predict the probability of the presence of each class in this audio sample. Hence our (hypothetical) predicted vector for this sample would be:

\n
\n

[0.01, 0.01, 0.01, 0.84, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.78].

\n
\n

So, yeah, it's a multilabel classification problem.

\n

Now quoting the submission format from the competition page:

\n
recording_id,s0,...,s23\n000316da7,0.1,....,0.3\n003bc2cb2,0.0,...,0.8\n...\n
\n

And we need to do this for what, 1992 samples? Okay, that being said, let's worry about the evaluation metric.

\n

PS: I will update this thread soon. Please correct me if I am wrong anywhere, thanks :)

", "messages": [ { "id": "1098636", "postDate": "12/01/2020 18:43:32", "content": "

Before diving deep into the metric, let's understand what we need to submit, it's vector dimensions and formats.

\n

Submission Formats

\n

We have got 24 classes and some 9000 training samples in total. No, this much data is not sufficient to understand the Label weighted LRAP (label ranking average precision) metric. First, have a look at the below data chunk.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
---recording_idspecies_idsongtype_idt_minf_mint_maxf_max
000204008d21113.84003281.250014.93334125.0000
100204008d8124.49603750.000028.61875531.2500
200204008d4115.00272343.750016.85874218.7500
\n

What did you see? Well, if it didn't catch your attention, Let me explain as per my understanding. For single recording id 00204008d, we have got 3 species ids in that, i.e. 21, 8, and 4. What does that mean? The recording, which means the audio sample having this mentioned id, has audio of species 24, species 8, and species 4. Hence, our ground truth for this data sample would be:

\n
\n

[0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1].

\n
\n

But what we need to predict? We need to predict the probability of the presence of each class in this audio sample. Hence our (hypothetical) predicted vector for this sample would be:

\n
\n

[0.01, 0.01, 0.01, 0.84, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.78].

\n
\n

So, yeah, it's a multilabel classification problem.

\n

Now quoting the submission format from the competition page:

\n
recording_id,s0,...,s23\n000316da7,0.1,....,0.3\n003bc2cb2,0.0,...,0.8\n...\n
\n

And we need to do this for what, 1992 samples? Okay, that being said, let's worry about the evaluation metric.

\n

PS: I will update this thread soon. Please correct me if I am wrong anywhere, thanks :)

", "rawMarkdown": "Before diving deep into the metric, let's understand what we need to submit, it's vector dimensions and formats.\n# Submission Formats\n We have got 24 classes and some 9000 training samples in total. No, this much data is not sufficient to understand the **Label weighted LRAP (label ranking average precision)** metric. First, have a look at the below data chunk.\n\n|---|\trecording_id |\tspecies_id | songtype_id | t_min | f_min | t_max | f_max|\n|---|-------------- |------------|-------------- |-------|------ |--------|-------| \n| 0 | 00204008d | 21 | 1 | 13.8400 | 3281.2500 | 14.9333 | 4125.0000 |\n|1 | 00204008d| 8\t|1\t|24.4960\t|3750.0000 |28.6187 | 5531.2500\n|2 | 00204008d| 4 | 1| 15.0027 | 2343.7500 | 16.8587 | 4218.7500 | \n\nWhat did you see? Well, if it didn't catch your attention, Let me explain as per my understanding. For single `recording id` ***00204008d***, we have got 3 species ids in that, i.e. 21, 8, and 4. What does that mean? The recording, which means the audio sample having this mentioned id, has audio of species 24, species 8, and species 4. Hence, our ground truth for this data sample would be:\n> [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]. \n\nBut what we need to predict? We need to predict the probability of the presence of each class in this audio sample. Hence our (hypothetical) predicted vector for this sample would be:\n> [0.01, 0.01, 0.01, 0.84, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.78]. \n\nSo, yeah, it's a multilabel classification problem. \n\nNow quoting the submission format from the competition page:\n```\nrecording_id,s0,...,s23\n000316da7,0.1,....,0.3\n003bc2cb2,0.0,...,0.8\n...\n```\n\nAnd we need to do this for what, 1992 samples? Okay, that being said, let's worry about the evaluation metric.\n\nPS: I will update this thread soon. Please correct me if I am wrong anywhere, thanks :)", "votes": null }, { "id": "1106991", "postDate": "12/09/2020 09:23:14", "content": "

This seems right @mrutyunjaybiswal in my opinion too. Instead of hard thresholding of each output neuron we just have to submit the probability score.

", "rawMarkdown": "This seems right @mrutyunjaybiswal in my opinion too. Instead of hard thresholding of each output neuron we just have to submit the probability score.", "votes": null }, { "id": "1150094", "postDate": "01/12/2021 11:18:03", "content": "

That is very usefull thread. Looking forward for updates/discussions..

\n

BTW you may want to correct a typo to avoid any confusion

\n
\n

has audio of species 24, species 8, and species 4.

\n
\n

first one should be 21 (according to first line of data table)
\nsimilarly you need to correct the ground truth vector

", "rawMarkdown": "That is very usefull thread. Looking forward for updates/discussions.. \n\nBTW you may want to correct a typo to avoid any confusion\n> has audio of species 24, species 8, and species 4. \n\nfirst one should be 21 (according to first line of data table)\nsimilarly you need to correct the `ground truth` vector", "votes": null }, { "id": "1163103", "postDate": "01/21/2021 14:29:05", "content": "

Cool!

\n

Waiting for the update @mrutyunjaybiswal .

", "rawMarkdown": "Cool!\n\nWaiting for the update @mrutyunjaybiswal .", "votes": null } ], "comments": [ { "id": 1106991, "author_name": "ayuraj", "author_url": "", "post_date": "12/09/2020 09:23:14", "content": "

This seems right @mrutyunjaybiswal in my opinion too. Instead of hard thresholding of each output neuron we just have to submit the probability score.

", "votes": null, "replies": [] }, { "id": 1150094, "author_name": "imeintanis", "author_url": "", "post_date": "01/12/2021 11:18:03", "content": "

That is very usefull thread. Looking forward for updates/discussions..

\n

BTW you may want to correct a typo to avoid any confusion

\n
\n

has audio of species 24, species 8, and species 4.

\n
\n

first one should be 21 (according to first line of data table)
\nsimilarly you need to correct the ground truth vector

", "votes": null, "replies": [] }, { "id": 1163103, "author_name": "joshi98kishan", "author_url": "", "post_date": "01/21/2021 14:29:05", "content": "

Cool!

\n

Waiting for the update @mrutyunjaybiswal .

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1098636": "Before diving deep into the metric, let's understand what we need to submit, it's vector dimensions and formats.\n# Submission Formats\n We have got 24 classes and some 9000 training samples in total. No, this much data is not sufficient to understand the **Label weighted LRAP (label ranking average precision)** metric. First, have a look at the below data chunk.\n\n|---|\trecording_id |\tspecies_id | songtype_id | t_min | f_min | t_max | f_max|\n|---|-------------- |------------|-------------- |-------|------ |--------|-------| \n| 0 | 00204008d | 21 | 1 | 13.8400 | 3281.2500 | 14.9333 | 4125.0000 |\n|1 | 00204008d| 8\t|1\t|24.4960\t|3750.0000 |28.6187 | 5531.2500\n|2 | 00204008d| 4 | 1| 15.0027 | 2343.7500 | 16.8587 | 4218.7500 | \n\nWhat did you see? Well, if it didn't catch your attention, Let me explain as per my understanding. For single `recording id` ***00204008d***, we have got 3 species ids in that, i.e. 21, 8, and 4. What does that mean? The recording, which means the audio sample having this mentioned id, has audio of species 24, species 8, and species 4. Hence, our ground truth for this data sample would be:\n> [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]. \n\nBut what we need to predict? We need to predict the probability of the presence of each class in this audio sample. Hence our (hypothetical) predicted vector for this sample would be:\n> [0.01, 0.01, 0.01, 0.84, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.78]. \n\nSo, yeah, it's a multilabel classification problem. \n\nNow quoting the submission format from the competition page:\n```\nrecording_id,s0,...,s23\n000316da7,0.1,....,0.3\n003bc2cb2,0.0,...,0.8\n...\n```\n\nAnd we need to do this for what, 1992 samples? Okay, that being said, let's worry about the evaluation metric.\n\nPS: I will update this thread soon. Please correct me if I am wrong anywhere, thanks :)", "1106991": "This seems right @mrutyunjaybiswal in my opinion too. Instead of hard thresholding of each output neuron we just have to submit the probability score.", "1150094": "That is very usefull thread. Looking forward for updates/discussions.. \n\nBTW you may want to correct a typo to avoid any confusion\n> has audio of species 24, species 8, and species 4. \n\nfirst one should be 21 (according to first line of data table)\nsimilarly you need to correct the `ground truth` vector", "1163103": "Cool!\n\nWaiting for the update @mrutyunjaybiswal ." }, "source": "meta" }