![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Mar-2022/wrong.png\")
Hi kagglers π,
\nNow that we know how important it is to properly crop the images in this competition, I've been wondering about a way to make a good quality dataset of bounding boxes.
\nNumerous manual annotations are already available in public datasets, but it seems like it's still not enoughβ¦
So I've been working on a way to crowd-source a bounding box dataset. I made and hosted a little app where anybody can come and annotate images or review other people's annotations.
\nThe app is available here: http://happywhale.theoboyer.fr
\nThe source code and explanations on how the app works are available on this GitHub repo: https://github.com/TheoBoyer/Manual-bbox-annotation-tool
\nOn the home page you can find buttons to download directly either the raw annotations and review informations or the final dataset constituted of only manually reviewed samples. You can also download the dataset in a kaggle notebook with these command lines:
\n!curl http://happywhale.theoboyer.fr/data/annotations.csv > annotations.csv
!curl http://happywhale.theoboyer.fr/data/dataset.csv > dataset.csv
Do not hesitate to come and annotate some images, or review some annotations (while your models are training for example π), it will help everyone here !
\nFor more information, see the README of the repo or ask your questions on this topic !
\nHappy Kaggling everyone π
", "messages": [ { "id": "1706548", "postDate": "02/27/2022 15:05:07", "content": "Hi kagglers π,
\nNow that we know how important it is to properly crop the images in this competition, I've been wondering about a way to make a good quality dataset of bounding boxes.
\nNumerous manual annotations are already available in public datasets, but it seems like it's still not enoughβ¦
So I've been working on a way to crowd-source a bounding box dataset. I made and hosted a little app where anybody can come and annotate images or review other people's annotations.
\nThe app is available here: http://happywhale.theoboyer.fr
\nThe source code and explanations on how the app works are available on this GitHub repo: https://github.com/TheoBoyer/Manual-bbox-annotation-tool
\nOn the home page you can find buttons to download directly either the raw annotations and review informations or the final dataset constituted of only manually reviewed samples. You can also download the dataset in a kaggle notebook with these command lines:
\n!curl http://happywhale.theoboyer.fr/data/annotations.csv > annotations.csv
!curl http://happywhale.theoboyer.fr/data/dataset.csv > dataset.csv
Do not hesitate to come and annotate some images, or review some annotations (while your models are training for example π), it will help everyone here !
\nFor more information, see the README of the repo or ask your questions on this topic !
\nHappy Kaggling everyone π
", "rawMarkdown": "Hi kagglers π,\n\nNow that we know how important it is to properly crop the images in this competition, I've been wondering about a way to make a good quality dataset of bounding boxes. \nNumerous manual annotations are already available in public datasets, but it seems like it's still not enough...\n\nSo I've been working on a way to crowd-source a bounding box dataset. I made and hosted a little app where anybody can come and annotate images or review other people's annotations.\n\n### Links\n**The app is available here**: [http://happywhale.theoboyer.fr](http://happywhale.theoboyer.fr)\n\n**The source code and explanations on how the app works are available on this GitHub repo**: [https://github.com/TheoBoyer/Manual-bbox-annotation-tool](https://github.com/TheoBoyer/Manual-bbox-annotation-tool)\n\n### Download the dataset\nOn the home page you can find buttons to download directly either the raw annotations and review informations or the final dataset constituted of only manually reviewed samples. You can also download the dataset in a kaggle notebook with these command lines:\n\n`!curl http://happywhale.theoboyer.fr/data/annotations.csv > annotations.csv`\n\n`!curl http://happywhale.theoboyer.fr/data/dataset.csv > dataset.csv`\n\n### Note\n* The dataset doesn't start from scratch, I merged several public datasets to begin ! The details of sources and how I gathered manual annotations are available in the notebook: [π³&π¬ - π¨βπ¬ Merging public bounding box datasets](https://www.kaggle.com/wolfy73/merging-public-bounding-box-\n* Some automatic annotations are already entered and just need a manual review. The annotations were obtained by taking the dataset from [Happywhale: Cropped Dataset [YOLOv5] βοΈ](https://www.kaggle.com/awsaf49/happywhale-cropped-dataset-yolov5) and filtered using the methods described in [π³&π¬ - Filter YOLOv5 failure cases](https://www.kaggle.com/wolfy73/filter-yolov5-failure-cases)\n* Only the training images are in the app to comply with the competition's rules \n\nDo not hesitate to come and annotate some images, or review some annotations (while your models are training for example π), it will help everyone here !\n\nFor more information, see the README of the repo or ask your questions on this topic !\n\nHappy Kaggling everyone π", "votes": null }, { "id": "1708483", "postDate": "03/01/2022 13:39:32", "content": "We nearly reached 200 new annotations + 800 reviews in just 2 days, this is amazing !
\nA big thank you to all those who participate π
Amazing work @wolfy73. How many annotations until now ?
", "rawMarkdown": "Amazing work @wolfy73. How many annotations until now ?", "votes": null }, { "id": "1712257", "postDate": "03/04/2022 18:26:15", "content": "I made a public dataset that contains the dataset. It's updated on a daily basis using a scheduled notebook:
\nπ³&π¬ - Crowdsourced bounding boxes
The reviewed dataset contains currently 2671 annotated images !
", "rawMarkdown": "The reviewed dataset contains currently 2671 annotated images !", "votes": null }, { "id": "1712729", "postDate": "03/05/2022 08:48:58", "content": "This is a great initiative! @wolfy73
\nI am thinking if it is possible to integrate Active Learning into the whole pipeline to ramp up the marginal gain of our annotations.
\nMy raw idea is to make a notebook that is scheduled to pull our current annotations and train a model, and then rank the unlabelled images by their \"worthiness\" to be annotated. As a step further, the rank can be fed to your app to complete the loop. We keep this loop going until the performance gain diminish at a certain extent.
\nOne uncertainty is how feasible active learning can be applied on object detection, as it is typically applied on classification problems.
\nI am trying to do some research to assess its feasibility and the possible pipeline, let me know if you are interested to also work on this! I believe this is a good chance to apply active learning in real problem and assess its effectiveness!
\nFYI, here are some initial finding I have got:
\nFYI I think the link is not working
", "rawMarkdown": "FYI I think the link is not working", "votes": null }, { "id": "1712745", "postDate": "03/05/2022 09:15:12", "content": "Thank you for letting me know, it should work now !
", "rawMarkdown": "Thank you for letting me know, it should work now !", "votes": null }, { "id": "1712748", "postDate": "03/05/2022 09:19:57", "content": "I'm interested !
", "rawMarkdown": "I'm interested !", "votes": null }, { "id": "1727314", "postDate": "03/17/2022 21:22:53", "content": "[03/17/2022]
\nI support this, if we do 1 annotation per second, it will take an hour per 3.6k images.
Also it seems that the photos disappear before the bounding box does. Is this a problem? Or is that normal?
\nAlso if you could drag and drop the bounding box, it would make it faster to annotate. Auto-snapping to the edges/corners AND/OR make the corners redraggable after dropping that would be awesome. I wonder how hard this is to code in.
\nAre we bounding boxing the surface above the water? Or above and below? I've been annotating only the above areas of an animal UNLESS the camera is underwater, in which case (usually dolphins), I annotate the whole body.
\nCan there be a button to \"suggest for removal\"? ie. A photo of a clipboard, multiple shots of the same animal in the same position at the same time(potential for the AI for overemphasize certain positions/angles), photos where the primary animal isn't clear (ie pod of dolphins swimming by), bad photos of a fluke (underwater or a picture of a side fluke)
\nBounding boxes close to corners should be assumed to be at the corner, as the current implementation doesn't allow for clicking EXACTLY at the corner.
\nWe should have a FLUKE button, so that any fluke photos can be sent to a separate fluke model imo. (or maybe use the winning model from the 2018 contest). I'm not sure if the current \"top of marine animal\" detector would work for flukes.
\n[03/18/2022]
\nWhen looking through photos, I've seen:
\n-Photos of dolphin/whale fins, where the primary one isn't clear. ie. The one is front is clearer, but the one in the back is smaller.
\n-Photos of dolphins underwater in a pod
\n-Photo of dolphins from above
\n-Photos of fins so small, they're about 10 pixels high and 20 pixels wide
\n-Photos of crustaceans on the top of a whale, against a cliff that looks nearly the same, telling the difference isn't easy
\n -Photo of flukes, sides of flukes, flukes underwater, flukes so small they're about 20 pixels by 20 pixels.
\n-Baby whales next to the PRIMARY whale, obstructing the view
Looking at these photos, I would very surprised if data leakage (shapes of waves, color of water) isn't playing a MAJOR role in correct identification of individuals, so the identifying features in the images is so small, it would be impossible without other details playing a role.
\nI wouldn't be surprised if there are duplicate photos' somewhere in the dataset. I'll be checking this later today.
Cool app! Great idea!
", "rawMarkdown": "Cool app! Great idea!", "votes": null }, { "id": "1738064", "postDate": "03/29/2022 01:41:22", "content": "Maybe we should provide an example which demonstrates that people should provide an extra area around the whale/dolphin. I notice that half of the bboxs are missing some fin because i think people try to get very close and as a result remove part of the fin (like example below)
\n
Chris I have manually annotated a few thousand of these and would love tips.
\nHow much space should be given around the fin?
\nHow should whale flukes be annotated? (I've seen many so far)
\nHow should bubbles be annotated (I had a photo that I annotated that only had bubbles, as if the whale had recently dived)?
\n\"Principle marine animal\". I've seen images with a fin and a fluke in equal size and view from the camera. I usually prioritize the fin, but as flukes make up a part of the dataset, I sometimes question my choices
IMHO, I think the important thing is consistency, i.e. that everyone annotates the images the same. So we either (1) clip the tops of all the fins, or (2) we leave a little border around all the fins. Or we (3) only box the fins, Or etc etc. Without instruction, i think everyone will make different boxes and I think that will be a problem.
", "rawMarkdown": "IMHO, I think the important thing is consistency, i.e. that everyone annotates the images the same. So we either (1) clip the tops of all the fins, or (2) we leave a little border around all the fins. Or we (3) only box the fins, Or etc etc. Without instruction, i think everyone will make different boxes and I think that will be a problem.", "votes": null }, { "id": "1744183", "postDate": "04/03/2022 17:10:38", "content": "Hi, part of the annotations are coming from Yolov5 model trained on a previous whale competition. I believe this type of annotation where some parts are cut out are coming from automated annotations and note manual ones
", "rawMarkdown": "Hi, part of the annotations are coming from Yolov5 model trained on a previous whale competition. I believe this type of annotation where some parts are cut out are coming from automated annotations and note manual ones", "votes": null } ], "comments": [ { "id": 1708483, "author_name": "wolfy73", "author_url": "", "post_date": "03/01/2022 13:39:32", "content": "We nearly reached 200 new annotations + 800 reviews in just 2 days, this is amazing !
\nA big thank you to all those who participate π
Amazing work @wolfy73. How many annotations until now ?
", "votes": null, "replies": [ { "id": 1712260, "author_name": "wolfy73", "author_url": "", "post_date": "03/04/2022 18:27:39", "content": "The reviewed dataset contains currently 2671 annotated images !
", "votes": null, "replies": [] } ] }, { "id": 1712257, "author_name": "wolfy73", "author_url": "", "post_date": "03/04/2022 18:26:15", "content": "I made a public dataset that contains the dataset. It's updated on a daily basis using a scheduled notebook:
\nπ³&π¬ - Crowdsourced bounding boxes
FYI I think the link is not working
", "votes": null, "replies": [] }, { "id": 1712745, "author_name": "wolfy73", "author_url": "", "post_date": "03/05/2022 09:15:12", "content": "Thank you for letting me know, it should work now !
", "votes": null, "replies": [] } ] }, { "id": 1712729, "author_name": "alexlwh", "author_url": "", "post_date": "03/05/2022 08:48:58", "content": "This is a great initiative! @wolfy73
\nI am thinking if it is possible to integrate Active Learning into the whole pipeline to ramp up the marginal gain of our annotations.
\nMy raw idea is to make a notebook that is scheduled to pull our current annotations and train a model, and then rank the unlabelled images by their \"worthiness\" to be annotated. As a step further, the rank can be fed to your app to complete the loop. We keep this loop going until the performance gain diminish at a certain extent.
\nOne uncertainty is how feasible active learning can be applied on object detection, as it is typically applied on classification problems.
\nI am trying to do some research to assess its feasibility and the possible pipeline, let me know if you are interested to also work on this! I believe this is a good chance to apply active learning in real problem and assess its effectiveness!
\nFYI, here are some initial finding I have got:
\nI'm interested !
", "votes": null, "replies": [] } ] }, { "id": 1727314, "author_name": "dentistdad", "author_url": "", "post_date": "03/17/2022 21:22:53", "content": "[03/17/2022]
\nI support this, if we do 1 annotation per second, it will take an hour per 3.6k images.
Also it seems that the photos disappear before the bounding box does. Is this a problem? Or is that normal?
\nAlso if you could drag and drop the bounding box, it would make it faster to annotate. Auto-snapping to the edges/corners AND/OR make the corners redraggable after dropping that would be awesome. I wonder how hard this is to code in.
\nAre we bounding boxing the surface above the water? Or above and below? I've been annotating only the above areas of an animal UNLESS the camera is underwater, in which case (usually dolphins), I annotate the whole body.
\nCan there be a button to \"suggest for removal\"? ie. A photo of a clipboard, multiple shots of the same animal in the same position at the same time(potential for the AI for overemphasize certain positions/angles), photos where the primary animal isn't clear (ie pod of dolphins swimming by), bad photos of a fluke (underwater or a picture of a side fluke)
\nBounding boxes close to corners should be assumed to be at the corner, as the current implementation doesn't allow for clicking EXACTLY at the corner.
\nWe should have a FLUKE button, so that any fluke photos can be sent to a separate fluke model imo. (or maybe use the winning model from the 2018 contest). I'm not sure if the current \"top of marine animal\" detector would work for flukes.
\n[03/18/2022]
\nWhen looking through photos, I've seen:
\n-Photos of dolphin/whale fins, where the primary one isn't clear. ie. The one is front is clearer, but the one in the back is smaller.
\n-Photos of dolphins underwater in a pod
\n-Photo of dolphins from above
\n-Photos of fins so small, they're about 10 pixels high and 20 pixels wide
\n-Photos of crustaceans on the top of a whale, against a cliff that looks nearly the same, telling the difference isn't easy
\n -Photo of flukes, sides of flukes, flukes underwater, flukes so small they're about 20 pixels by 20 pixels.
\n-Baby whales next to the PRIMARY whale, obstructing the view
Looking at these photos, I would very surprised if data leakage (shapes of waves, color of water) isn't playing a MAJOR role in correct identification of individuals, so the identifying features in the images is so small, it would be impossible without other details playing a role.
\nI wouldn't be surprised if there are duplicate photos' somewhere in the dataset. I'll be checking this later today.
Cool app! Great idea!
", "votes": null, "replies": [] }, { "id": 1738064, "author_name": "cdeotte", "author_url": "", "post_date": "03/29/2022 01:41:22", "content": "Maybe we should provide an example which demonstrates that people should provide an extra area around the whale/dolphin. I notice that half of the bboxs are missing some fin because i think people try to get very close and as a result remove part of the fin (like example below)
\n
Chris I have manually annotated a few thousand of these and would love tips.
\nHow much space should be given around the fin?
\nHow should whale flukes be annotated? (I've seen many so far)
\nHow should bubbles be annotated (I had a photo that I annotated that only had bubbles, as if the whale had recently dived)?
\n\"Principle marine animal\". I've seen images with a fin and a fluke in equal size and view from the camera. I usually prioritize the fin, but as flukes make up a part of the dataset, I sometimes question my choices
IMHO, I think the important thing is consistency, i.e. that everyone annotates the images the same. So we either (1) clip the tops of all the fins, or (2) we leave a little border around all the fins. Or we (3) only box the fins, Or etc etc. Without instruction, i think everyone will make different boxes and I think that will be a problem.
", "votes": null, "replies": [] }, { "id": 1744183, "author_name": "wolfy73", "author_url": "", "post_date": "04/03/2022 17:10:38", "content": "Hi, part of the annotations are coming from Yolov5 model trained on a previous whale competition. I believe this type of annotation where some parts are cut out are coming from automated annotations and note manual ones
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1706548": "Hi kagglers π,\n\nNow that we know how important it is to properly crop the images in this competition, I've been wondering about a way to make a good quality dataset of bounding boxes. \nNumerous manual annotations are already available in public datasets, but it seems like it's still not enough...\n\nSo I've been working on a way to crowd-source a bounding box dataset. I made and hosted a little app where anybody can come and annotate images or review other people's annotations.\n\n### Links\n**The app is available here**: [http://happywhale.theoboyer.fr](http://happywhale.theoboyer.fr)\n\n**The source code and explanations on how the app works are available on this GitHub repo**: [https://github.com/TheoBoyer/Manual-bbox-annotation-tool](https://github.com/TheoBoyer/Manual-bbox-annotation-tool)\n\n### Download the dataset\nOn the home page you can find buttons to download directly either the raw annotations and review informations or the final dataset constituted of only manually reviewed samples. You can also download the dataset in a kaggle notebook with these command lines:\n\n`!curl http://happywhale.theoboyer.fr/data/annotations.csv > annotations.csv`\n\n`!curl http://happywhale.theoboyer.fr/data/dataset.csv > dataset.csv`\n\n### Note\n* The dataset doesn't start from scratch, I merged several public datasets to begin ! The details of sources and how I gathered manual annotations are available in the notebook: [π³&π¬ - π¨βπ¬ Merging public bounding box datasets](https://www.kaggle.com/wolfy73/merging-public-bounding-box-\n* Some automatic annotations are already entered and just need a manual review. The annotations were obtained by taking the dataset from [Happywhale: Cropped Dataset [YOLOv5] βοΈ](https://www.kaggle.com/awsaf49/happywhale-cropped-dataset-yolov5) and filtered using the methods described in [π³&π¬ - Filter YOLOv5 failure cases](https://www.kaggle.com/wolfy73/filter-yolov5-failure-cases)\n* Only the training images are in the app to comply with the competition's rules \n\nDo not hesitate to come and annotate some images, or review some annotations (while your models are training for example π), it will help everyone here !\n\nFor more information, see the README of the repo or ask your questions on this topic !\n\nHappy Kaggling everyone π", "1708483": "We nearly reached 200 new annotations + 800 reviews in just 2 days, this is amazing !\nA big thank you to all those who participate π", "1712241": "Amazing work @wolfy73. How many annotations until now ?", "1712257": "I made a public dataset that contains the dataset. It's updated on a daily basis using a scheduled notebook:\n[π³&π¬ - Crowdsourced bounding boxes](https://www.kaggle.com/wolfy73/wandd-crowed-sourced-bounging-boxes)", "1712260": "The reviewed dataset contains currently 2671 annotated images !", "1712729": "This is a great initiative! @wolfy73 \n\nI am thinking if it is possible to integrate Active Learning into the whole pipeline to ramp up the marginal gain of our annotations.\n\nMy raw idea is to make a notebook that is scheduled to pull our current annotations and train a model, and then rank the unlabelled images by their \"worthiness\" to be annotated. As a step further, the rank can be fed to your app to complete the loop. We keep this loop going until the performance gain diminish at a certain extent.\n\nOne uncertainty is how feasible active learning can be applied on object detection, as it is typically applied on classification problems.\n\nI am trying to do some research to assess its feasibility and the possible pipeline, let me know if you are interested to also work on this! I believe this is a good chance to apply active learning in real problem and assess its effectiveness!\n\nFYI, here are some initial finding I have got:\n1. [Active Learning for Deep Object Detection via Probabilistic Modeling](https://github.com/NVlabs/AL-MDN)\n2. [Learning Loss for Active Learning](https://github.com/superannotateai/active_learning)", "1712735": "FYI I think the link is not working", "1712745": "Thank you for letting me know, it should work now !", "1712748": "I'm interested !", "1727314": "**[03/17/2022]**\nI support this, if we do 1 annotation per second, it will take an hour per 3.6k images.\n\nAlso it seems that the photos disappear before the bounding box does. Is this a problem? Or is that normal?\n\nAlso if you could drag and drop the bounding box, it would make it faster to annotate. Auto-snapping to the edges/corners AND/OR make the corners redraggable after dropping that would be awesome. I wonder how hard this is to code in.\n\nAre we bounding boxing the surface above the water? Or above and below? I've been annotating only the above areas of an animal UNLESS the camera is underwater, in which case (usually dolphins), I annotate the whole body.\n\nCan there be a button to \"suggest for removal\"? ie. A photo of a clipboard, multiple shots of the same animal in the same position at the same time(potential for the AI for overemphasize certain positions/angles), photos where the primary animal isn't clear (ie pod of dolphins swimming by), bad photos of a fluke (underwater or a picture of a side fluke)\n\n\n\nBounding boxes close to corners should be assumed to be at the corner, as the current implementation doesn't allow for clicking EXACTLY at the corner.\n\nWe should have a FLUKE button, so that any fluke photos can be sent to a separate fluke model imo. (or maybe use the winning model from the 2018 contest). I'm not sure if the current \"top of marine animal\" detector would work for flukes.\n\n**[03/18/2022]**\n**When looking through photos, I've seen:**\n-Photos of dolphin/whale fins, where the primary one isn't clear. ie. The one is front is clearer, but the one in the back is smaller.\n-Photos of dolphins underwater in a pod\n-Photo of dolphins from above\n-Photos of fins so small, they're about 10 pixels high and 20 pixels wide\n-Photos of crustaceans on the top of a whale, against a cliff that looks nearly the same, telling the difference isn't easy\n -Photo of flukes, sides of flukes, flukes underwater, flukes so small they're about 20 pixels by 20 pixels.\n-Baby whales next to the PRIMARY whale, obstructing the view\n\n**Looking at these photos, I would very surprised if data leakage (shapes of waves, color of water) isn't playing a MAJOR role in correct identification of individuals, so the identifying features in the images is so small, it would be impossible without other details playing a role.**\nI wouldn't be surprised if there are duplicate photos' somewhere in the dataset. I'll be checking this later today.", "1736205": "Cool app! Great idea!", "1738064": "Maybe we should provide an example which demonstrates that people should provide an extra area around the whale/dolphin. I notice that half of the bboxs are missing some fin because i think people try to get very close and as a result remove part of the fin (like example below)\n\n", "1738069": "**Chris I have manually annotated a few thousand of these and would love tips.**\nHow much space should be given around the fin?\nHow should whale flukes be annotated? (I've seen many so far)\nHow should bubbles be annotated (I had a photo that I annotated that only had bubbles, as if the whale had recently dived)?\n\"Principle marine animal\". I've seen images with a fin and a fluke in equal size and view from the camera. I usually prioritize the fin, but as flukes make up a part of the dataset, I sometimes question my choices", "1738072": "IMHO, I think the important thing is consistency, i.e. that everyone annotates the images the same. So we either (1) clip the tops of all the fins, or (2) we leave a little border around all the fins. Or we (3) only box the fins, Or etc etc. Without instruction, i think everyone will make different boxes and I think that will be a problem.", "1744183": "Hi, part of the annotations are coming from Yolov5 model trained on a previous whale competition. I believe this type of annotation where some parts are cut out are coming from automated annotations and note manual ones" }, "source": "meta" }