It seems that some of the whales that are the same have different IDs. \nIt might be the problem of merging different data sources because of the human factor. This issue does not only affect the quality of the model (you are giving your labels same images but with different labels) but also brings an uncertainty during the inference as it is unclear which of the two classes should we predict.
\n\nAttaching an example of such IDs.
\n\nHere is the list of same classes with the duplicate IDs, the list is not exhaustive, though:
\n\nThe question to the admins, would it be possible to change the labels and if not, how should we deal with this issue?
", "messages": [ { "id": 442967, "postDate": "2018-12-20T19:58:31.903Z", "content": "It seems that some of the whales that are the same have different IDs. \nIt might be the problem of merging different data sources because of the human factor. This issue does not only affect the quality of the model (you are giving your labels same images but with different labels) but also brings an uncertainty during the inference as it is unclear which of the two classes should we predict.
\n\nAttaching an example of such IDs.
\n\nHere is the list of same classes with the duplicate IDs, the list is not exhaustive, though:
\n\nThe question to the admins, would it be possible to change the labels and if not, how should we deal with this issue?
", "rawMarkdown": "It seems that some of the whales that are the same have different IDs. \nIt might be the problem of merging different data sources because of the human factor. This issue does not only affect the quality of the model (you are giving your labels same images but with different labels) but also brings an uncertainty during the inference as it is unclear which of the two classes should we predict.\n\nAttaching an example of such IDs.\n\nHere is the list of same classes with the duplicate IDs, the list is not exhaustive, though:\n\n- w_c7e1b12 and w_62b631e\n- w_af1d57b and w_353d249\n- w_c543f7c and w_2d1d67a\n- w_a335fc2 and w_c99807e\n- w_f119baa and w_3f213d5\n- w_76f4d0b and w_20950a9\n- w_2f350be and w_6f22173\n- w_954fec8 and w_0ee7878\n- w_27272a5 and w_7e5b9da\n\nThe question to the admins, would it be possible to change the labels and if not, how should we deal with this issue?", "votes": 24 }, { "id": 460418, "postDate": "2019-01-23T16:54:20.580Z", "content": "Belatedly:
\n\nwc7e1b12 and w62b631e - correct, same whale\nwaf1d57b and w353d249 - correct, same whale\nwc543f7c and w2d1d67a - correct, same whale\nwa335fc2 and wc99807e - correct, same whale\nwf119baa and w3f213d5 - correct, same whale\nw76f4d0b and w20950a9 - not the same whale, as pi-null-mezon pointed out\nw2f350be and w6f22173 - correct, same whale\nw954fec8 and w0ee7878 - correct, same whale\nw27272a5 and w7e5b9da - correct, same whale
", "rawMarkdown": "Belatedly:\n\nwc7e1b12 and w62b631e - correct, same whale\nwaf1d57b and w353d249 - correct, same whale\nwc543f7c and w2d1d67a - correct, same whale\nwa335fc2 and wc99807e - correct, same whale\nwf119baa and w3f213d5 - correct, same whale\nw76f4d0b and w20950a9 - not the same whale, as pi-null-mezon pointed out\nw2f350be and w6f22173 - correct, same whale\nw954fec8 and w0ee7878 - correct, same whale\nw27272a5 and w7e5b9da - correct, same whale", "votes": 4, "replies": [ { "id": 460741, "postDate": "2019-01-24T10:44:14.060Z", "content": "Thank you for your information. The next question (or our task ?) is the way to handle this issue. Are these pairs evaluated as the same whale for scoring ?
", "rawMarkdown": "Thank you for your information. The next question (or our task ?) is the way to handle this issue. Are these pairs evaluated as the same whale for scoring ?", "votes": 1 }, { "id": 460769, "postDate": "2019-01-24T11:42:12.210Z", "content": "No need to handle the issue. Note the previous response by @Andrej Kuro:
\n\n\"I think this problem within the competition is not that big as it may seem because of two reasons:
\n\nevery competitor has the same problem so this is democratic one\nfor each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction.\"
", "rawMarkdown": "No need to handle the issue. Note the previous response by @Andrej Kuro:\n\n\"I think this problem within the competition is not that big as it may seem because of two reasons:\n\nevery competitor has the same problem so this is democratic one\nfor each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction.\"", "votes": -3 }, { "id": 460774, "postDate": "2019-01-24T11:49:31.260Z", "content": "I do not agree. See my answer to the that comment:\n\"May be it is not big, but the randomness is the problem. Say, I always predict both of them together, so I am not THAT wrong. But depending on (random) order of the predictions C1,C2 o C2,C1 I may get 1 or 0.5 point for that prediction.\"\nIt is true, that doesn`t matter for you. But it may even decide who would win and who will not.
", "rawMarkdown": "I do not agree. See my answer to the that comment:\n\"May be it is not big, but the randomness is the problem. Say, I always predict both of them together, so I am not THAT wrong. But depending on (random) order of the predictions C1,C2 o C2,C1 I may get 1 or 0.5 point for that prediction.\"\nIt is true, that doesn`t matter for you. But it may even decide who would win and who will not.", "votes": 7 }, { "id": 461788, "postDate": "2019-01-27T02:32:33.507Z", "content": "same issue: ['w_83e0076', 'w_0135f5f'] (maybe, not an expert but a similar scenario can happen for the actually confirmed mislabels)
\n\nI can't agree more with you! I just started experimenting with a siamese network and took the above two ids for toy validation. It turned out that 'w_83e0076' 's most similar image pair in top 5 was an image of 'w_0135f5f' and an image from 'w_83e0076' only appeared at top 32. I haven't done a complete inspection for each whale.
\n\nEven though everyone have the same error, it doesn't mean that it will affect everyone equally, e.g. what each of our's model learn is different based on our data, model, loss strategies. So the sensitivity of the model will be different towards these errors since they are not systematic, hence can't ideally be learned by everyone's model.
", "rawMarkdown": "**same issue:** ['w_83e0076', 'w_0135f5f'] (maybe, not an expert but a similar scenario can happen for the actually confirmed mislabels)\n\nI can't agree more with you! I just started experimenting with a siamese network and took the above two ids for toy validation. It turned out that 'w_83e0076' 's most similar image pair in top 5 was an image of 'w_0135f5f' and an image from 'w_83e0076' only appeared at top 32. I haven't done a complete inspection for each whale. \n\nEven though everyone have the same error, it doesn't mean that it will affect everyone equally, e.g. what each of our's model learn is different based on our data, model, loss strategies. So the sensitivity of the model will be different towards these errors since they are not systematic, hence can't ideally be learned by everyone's model.\n\n", "votes": 1 }, { "id": 478967, "postDate": "2019-02-26T21:16:43.143Z", "content": "@tedcheese even after https://www.kaggle.com/c/humpback-whale-identification/discussion/81885 , especially see comment when the same (ok, differently cropped, but still) image is marked as classA, class \"new\" and also present in test set do you still think that there is no need to handle the issue?
", "rawMarkdown": "@tedcheese even after https://www.kaggle.com/c/humpback-whale-identification/discussion/81885 , especially see comment when the _same_ (ok, differently cropped, but still) image is marked as classA, class \"new\" and also present in test set do you still think that there is no need to handle the issue? ", "votes": 1 } ] }, { "id": 443360, "postDate": "2018-12-21T13:51:00.027Z", "content": "It is true that there are false negatives in the dataset, whales where we failed to find an existing match, thus assigned a new ID. So, the fact that you are finding these is good news to us as it shows quality in your developing algorithm :) . But as @Andrzej Kuro posted, this doesn't affect the relative competitive result... keep at it!
", "rawMarkdown": "It is true that there are false negatives in the dataset, whales where we failed to find an existing match, thus assigned a new ID. So, the fact that you are finding these is good news to us as it shows quality in your developing algorithm :) . But as @Andrzej Kuro posted, this doesn't affect the relative competitive result... keep at it!", "votes": 2 }, { "id": 443511, "postDate": "2018-12-21T19:19:51.507Z", "content": "I am not an expert but this pair (w76f4d0b and w20950a9) looks like it is different whales
", "rawMarkdown": "I am not an expert but this pair (w76f4d0b and w20950a9) looks like it is different whales ", "votes": 1 }, { "id": 443313, "postDate": "2018-12-21T12:10:02.523Z", "content": "I think this problem within the competition is not that big as it may seem because of two reasons:\n- every competitor has the same problem so this is democratic one\n- for each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction.
", "rawMarkdown": "I think this problem within the competition is not that big as it may seem because of two reasons:\n- every competitor has the same problem so this is democratic one\n- for each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction. ", "votes": -2, "replies": [ { "id": 460759, "postDate": "2019-01-24T11:28:25.757Z", "content": "May be it is not big, but the randomness is the problem. Say, I always predict both of them together, so I am not THAT wrong. But depending on (random) order of the predictions C1,C2 o C2,C1 I may get 1 or 0.5 point for that prediction.
", "rawMarkdown": "May be it is not big, but the randomness is the problem. Say, I always predict both of them together, so I am not THAT wrong. But depending on (random) order of the predictions C1,C2 o C2,C1 I may get 1 or 0.5 point for that prediction.", "votes": 3 }, { "id": 460814, "postDate": "2019-01-24T13:19:07.950Z", "content": "If we can't distinguish C1 and C2, the expectation is 0.75. I wonder if there are any information to distinguish them or not (EXIF, image format, number of training images, and so on).
", "rawMarkdown": "If we can't distinguish C1 and C2, the expectation is 0.75. I wonder if there are any information to distinguish them or not (EXIF, image format, number of training images, and so on).", "votes": 1 }, { "id": 461790, "postDate": "2019-01-27T02:36:10.427Z", "content": "Since this error is random we can't learn it and thus can't train a robust model against these errors. So there will definitely be a luck factor depending on what each model learns. Even a lucky augmentation during training may bring the correct label closer rather than the mislabeled and vice-versa.
", "rawMarkdown": "Since this error is random we can't learn it and thus can't train a robust model against these errors. So there will definitely be a luck factor depending on what each model learns. Even a lucky augmentation during training may bring the correct label closer rather than the mislabeled and vice-versa." }, { "id": 461945, "postDate": "2019-01-27T10:15:01.053Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 472328, "postDate": "2019-02-15T17:48:53.390Z", "rawMarkdown": "", "isDeleted": true }, { "id": 443185, "postDate": "2018-12-21T07:16:02.083Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 460418, "author_name": "Ted Cheeseman", "author_url": "", "post_date": "2019-01-23T16:54:20.580000", "content": "Belatedly:
\n\nwc7e1b12 and w62b631e - correct, same whale\nwaf1d57b and w353d249 - correct, same whale\nwc543f7c and w2d1d67a - correct, same whale\nwa335fc2 and wc99807e - correct, same whale\nwf119baa and w3f213d5 - correct, same whale\nw76f4d0b and w20950a9 - not the same whale, as pi-null-mezon pointed out\nw2f350be and w6f22173 - correct, same whale\nw954fec8 and w0ee7878 - correct, same whale\nw27272a5 and w7e5b9da - correct, same whale
", "votes": 4, "replies": [ { "id": 460741, "author_name": "toshi_k", "author_url": "", "post_date": "2019-01-24T10:44:14.060000", "content": "Thank you for your information. The next question (or our task ?) is the way to handle this issue. Are these pairs evaluated as the same whale for scoring ?
", "votes": 1, "replies": [] }, { "id": 460769, "author_name": "Ted Cheeseman", "author_url": "", "post_date": "2019-01-24T11:42:12.210000", "content": "No need to handle the issue. Note the previous response by @Andrej Kuro:
\n\n\"I think this problem within the competition is not that big as it may seem because of two reasons:
\n\nevery competitor has the same problem so this is democratic one\nfor each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction.\"
", "votes": -3, "replies": [] }, { "id": 460774, "author_name": "old-ufo", "author_url": "", "post_date": "2019-01-24T11:49:31.260000", "content": "I do not agree. See my answer to the that comment:\n\"May be it is not big, but the randomness is the problem. Say, I always predict both of them together, so I am not THAT wrong. But depending on (random) order of the predictions C1,C2 o C2,C1 I may get 1 or 0.5 point for that prediction.\"\nIt is true, that doesn`t matter for you. But it may even decide who would win and who will not.
", "votes": 7, "replies": [] }, { "id": 461788, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2019-01-27T02:32:33.507000", "content": "same issue: ['w_83e0076', 'w_0135f5f'] (maybe, not an expert but a similar scenario can happen for the actually confirmed mislabels)
\n\nI can't agree more with you! I just started experimenting with a siamese network and took the above two ids for toy validation. It turned out that 'w_83e0076' 's most similar image pair in top 5 was an image of 'w_0135f5f' and an image from 'w_83e0076' only appeared at top 32. I haven't done a complete inspection for each whale.
\n\nEven though everyone have the same error, it doesn't mean that it will affect everyone equally, e.g. what each of our's model learn is different based on our data, model, loss strategies. So the sensitivity of the model will be different towards these errors since they are not systematic, hence can't ideally be learned by everyone's model.
", "votes": 1, "replies": [] }, { "id": 478967, "author_name": "old-ufo", "author_url": "", "post_date": "2019-02-26T21:16:43.143000", "content": "@tedcheese even after https://www.kaggle.com/c/humpback-whale-identification/discussion/81885 , especially see comment when the same (ok, differently cropped, but still) image is marked as classA, class \"new\" and also present in test set do you still think that there is no need to handle the issue?
", "votes": 1, "replies": [] } ] }, { "id": 443360, "author_name": "Ted Cheeseman", "author_url": "", "post_date": "2018-12-21T13:51:00.027000", "content": "It is true that there are false negatives in the dataset, whales where we failed to find an existing match, thus assigned a new ID. So, the fact that you are finding these is good news to us as it shows quality in your developing algorithm :) . But as @Andrzej Kuro posted, this doesn't affect the relative competitive result... keep at it!
", "votes": 2, "replies": [] }, { "id": 443511, "author_name": "pi-null-mezon", "author_url": "", "post_date": "2018-12-21T19:19:51.507000", "content": "I am not an expert but this pair (w76f4d0b and w20950a9) looks like it is different whales
", "votes": 1, "replies": [] }, { "id": 443313, "author_name": "Andrzej Kuro", "author_url": "", "post_date": "2018-12-21T12:10:02.523000", "content": "I think this problem within the competition is not that big as it may seem because of two reasons:\n- every competitor has the same problem so this is democratic one\n- for each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction.
", "votes": -2, "replies": [ { "id": 460759, "author_name": "old-ufo", "author_url": "", "post_date": "2019-01-24T11:28:25.757000", "content": "May be it is not big, but the randomness is the problem. Say, I always predict both of them together, so I am not THAT wrong. But depending on (random) order of the predictions C1,C2 o C2,C1 I may get 1 or 0.5 point for that prediction.
", "votes": 3, "replies": [] }, { "id": 460814, "author_name": "toshi_k", "author_url": "", "post_date": "2019-01-24T13:19:07.950000", "content": "If we can't distinguish C1 and C2, the expectation is 0.75. I wonder if there are any information to distinguish them or not (EXIF, image format, number of training images, and so on).
", "votes": 1, "replies": [] }, { "id": 461790, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2019-01-27T02:36:10.427000", "content": "Since this error is random we can't learn it and thus can't train a robust model against these errors. So there will definitely be a luck factor depending on what each model learns. Even a lucky augmentation during training may bring the correct label closer rather than the mislabeled and vice-versa.
", "votes": 0, "replies": [] }, { "id": 461945, "author_name": "", "author_url": "", "post_date": "2019-01-27T10:15:01.053000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 472328, "author_name": "", "author_url": "", "post_date": "2019-02-15T17:48:53.390000", "content": "", "votes": 0, "replies": [] }, { "id": 443185, "author_name": "", "author_url": "", "post_date": "2018-12-21T07:16:02.083000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "442967": "It seems that some of the whales that are the same have different IDs. \nIt might be the problem of merging different data sources because of the human factor. This issue does not only affect the quality of the model (you are giving your labels same images but with different labels) but also brings an uncertainty during the inference as it is unclear which of the two classes should we predict.\n\nAttaching an example of such IDs.\n\nHere is the list of same classes with the duplicate IDs, the list is not exhaustive, though:\n\n- w_c7e1b12 and w_62b631e\n- w_af1d57b and w_353d249\n- w_c543f7c and w_2d1d67a\n- w_a335fc2 and w_c99807e\n- w_f119baa and w_3f213d5\n- w_76f4d0b and w_20950a9\n- w_2f350be and w_6f22173\n- w_954fec8 and w_0ee7878\n- w_27272a5 and w_7e5b9da\n\nThe question to the admins, would it be possible to change the labels and if not, how should we deal with this issue?", "460418": "Belatedly:\n\nwc7e1b12 and w62b631e - correct, same whale\nwaf1d57b and w353d249 - correct, same whale\nwc543f7c and w2d1d67a - correct, same whale\nwa335fc2 and wc99807e - correct, same whale\nwf119baa and w3f213d5 - correct, same whale\nw76f4d0b and w20950a9 - not the same whale, as pi-null-mezon pointed out\nw2f350be and w6f22173 - correct, same whale\nw954fec8 and w0ee7878 - correct, same whale\nw27272a5 and w7e5b9da - correct, same whale", "443360": "It is true that there are false negatives in the dataset, whales where we failed to find an existing match, thus assigned a new ID. So, the fact that you are finding these is good news to us as it shows quality in your developing algorithm :) . But as @Andrzej Kuro posted, this doesn't affect the relative competitive result... keep at it!", "443511": "I am not an expert but this pair (w76f4d0b and w20950a9) looks like it is different whales ", "443313": "I think this problem within the competition is not that big as it may seem because of two reasons:\n- every competitor has the same problem so this is democratic one\n- for each picture you can predict up to five whale IDs and the paired IDs are very similar so if the model is good than it will for sure pick both of the paired IDs as its prediction. ", "472328": "", "443185": "" } }