I trained regression model to predict keypoints using Paul's keypoint dataset. https://www.kaggle.com/c/humpback-whale-identification/discussion/78699\nThank you Paul !\n
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/468930/11232/alignment.jpg\")
I'm trying piecewise alignment as below. Do you think this approach improve our score ?
I trained regression model to predict keypoints using Paul's keypoint dataset. https://www.kaggle.com/c/humpback-whale-identification/discussion/78699\nThank you Paul !\n
I'm trying piecewise alignment as below. Do you think this approach improve our score ?
I trained regression model to predict keypoints using Paul's keypoint dataset. https://www.kaggle.com/c/humpback-whale-identification/discussion/78699\nThank you Paul !\n
you can check if score improve for this image:
\n\nmost of the difficult cases are large pose angle changes or occlusion.
\n\nfor small angle, normal augmentation like rotate, stretch, shear can take care of it.
\n\nwith the object is occluded, it is important to preserved information and not to distort the non-occluded part.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/469121/11240/difficult_case.png\")
@toshi_k, once you have re-calculated by regression all your points for the left column, may I ask how you get to the right column?
", "rawMarkdown": "toshi_k, once you have re-calculated by regression all your points for the left column, may I ask how you get to the right column?", "votes": null }, { "id": "469158", "postDate": "02/10/2019 15:56:32", "content": "actually you can do this:
\n\n1) optional : perform prediction on the test as usual.
\n\n2) you can select a few poor view train images (e.g. choosing those train id with no test images identified, or those with low prediction scores)
\n\nThe rule is that you cannot hand labelled the test image or manually manipulated them. But it is legal to manually manipulated train images. Some process need not to be automatic.
", "rawMarkdown": "actually you can do this:\n\n1) optional : perform prediction on the test as usual.\n\n2) you can select a few poor view train images (e.g. choosing those train id with no test images identified, or those with low prediction scores)\n\n3. open up photoshop (maybe not photoshop but some open-source clone) and improve these selected train images (e.g. pose rectification).\n\nThe rule is that you cannot hand labelled the test image or manually manipulated them. But it is legal to manually manipulated train images. Some process need not to be automatic.", "votes": null }, { "id": "469375", "postDate": "02/11/2019 04:34:31", "content": "Set up the keypoints for destination and transform each part. You can use Piecewise Affine Transformation in scikit-image. https://scikit-image.org/docs/dev/auto_examples/transform/plot_piecewise_affine.html
Thank you !
", "rawMarkdown": "Thank you !", "votes": null }, { "id": "469807", "postDate": "02/11/2019 20:53:17", "content": "Toshi_k,
\n\nFirst, thanks for finding my dataset useful!
\n\nI think Heng is right. Because of the way the fluke can be partially occluded, just blinding mapping the bottom keypoints (6-0) may result in a very poorly stretched image in these hard cases. This is mostly my fault for the way I decided to annotate the images with partially submerged flukes, but I didn't think about it at the time.
\n\nI was thinking that each side of the fluke could be morphed separately to account for left/right side occlusion. Fluke alignment would be determined by the top 3 points for that fluke side, and the keypoint 8 (bottom center fluke or water). This would help you select the width of the image and alignment.
\n\nThe height of the image would be a fixed ratio based on the average humpback whale fluke size. This way there would be less distortion introduced.
\n\nIn Heng Style, I tried to mock it up in mspaint...
\n\n![\"https://i.imgur.com/rTMevli.png\"](\"https://i.imgur.com/rTMevli.png\")
@Paul Johnson, your work was really useful and I am really glad to have learnt something new about image alignment and affine transformation.
\n\nI was getting lost in the triangle transformations until @toshi_k gave a hint.
\n\nMy first tests show that the transformations work quite well to align the top edge of the fluke but unfortunately given the occlusion of the bottom of the fluke, the result can be quite messy. More points during labeling phase are likely to improve the results but is also a lot of work.
\n\nMy bet is that the top 10 is working on a combination of regular classification and metrics learning as Heng mentionned before.
\n\nEdit: actually, using only the top 5 keypoints give good looking results overall as it preserve the markings.
", "rawMarkdown": "Paul Johnson, your work was really useful and I am really glad to have learnt something new about image alignment and affine transformation.\n\nI was getting lost in the triangle transformations until @toshi_k gave a hint.\n\nMy first tests show that the transformations work quite well to align the top edge of the fluke but unfortunately given the occlusion of the bottom of the fluke, the result can be quite messy. More points during labeling phase are likely to improve the results but is also a lot of work. \n\nMy bet is that the top 10 is working on a combination of regular classification and metrics learning as Heng mentionned before.\n\nEdit: actually, using only the top 5 keypoints give good looking results overall as it preserve the markings.", "votes": null }, { "id": "470378", "postDate": "02/12/2019 20:54:14", "content": "Is this similar to having an STN layer before the first convolution?
", "rawMarkdown": "Is this similar to having an STN layer before the first convolution?", "votes": null }, { "id": "472182", "postDate": "02/15/2019 13:06:48", "content": "I think this performs better than STN, because STN is difficult to train. And STN only learn 1 projection, but this work provides multiple projections.
", "rawMarkdown": "I think this performs better than STN, because STN is difficult to train. And STN only learn 1 projection, but this work provides multiple projections.", "votes": null }, { "id": "472318", "postDate": "02/15/2019 17:27:12", "content": "@toshi_k I am wondering could you share some code example that how to use the alignment? thanks!
", "rawMarkdown": "toshi_k I am wondering could you share some code example that how to use the alignment? thanks!", "votes": null }, { "id": "472847", "postDate": "02/16/2019 18:50:55", "content": "If I may:
\n\nUsing the 1000 keypoints dataset from Paul Johnson, run a regression of the (top 5) keypoints using Martin Piotte bbox kernel by replacing the 2 points of the bbox by the keypoints. (the top 5 points are easier to run and don't forget to remove the NAs in the keypoints dataset).
Using piecewise affine transformation from scikit learn mentionned by toshi_k in this thread, and after choosing a nice and balanced picture of a fluke as a reference that will become the matrix dst, apply the transformation to all the other pictures as src.
Thanks!
", "rawMarkdown": "Thanks!", "votes": null } ], "comments": [ { "id": 469121, "author_name": "hengck23", "author_url": "", "post_date": "02/10/2019 14:38:33", "content": "you can check if score improve for this image:
\n\nmost of the difficult cases are large pose angle changes or occlusion.
\n\nfor small angle, normal augmentation like rotate, stretch, shear can take care of it.
\n\nwith the object is occluded, it is important to preserved information and not to distort the non-occluded part.
\n\n
Toshi_k,
\n\nFirst, thanks for finding my dataset useful!
\n\nI think Heng is right. Because of the way the fluke can be partially occluded, just blinding mapping the bottom keypoints (6-0) may result in a very poorly stretched image in these hard cases. This is mostly my fault for the way I decided to annotate the images with partially submerged flukes, but I didn't think about it at the time.
\n\nI was thinking that each side of the fluke could be morphed separately to account for left/right side occlusion. Fluke alignment would be determined by the top 3 points for that fluke side, and the keypoint 8 (bottom center fluke or water). This would help you select the width of the image and alignment.
\n\nThe height of the image would be a fixed ratio based on the average humpback whale fluke size. This way there would be less distortion introduced.
\n\nIn Heng Style, I tried to mock it up in mspaint...
\n\n
@Paul Johnson, your work was really useful and I am really glad to have learnt something new about image alignment and affine transformation.
\n\nI was getting lost in the triangle transformations until @toshi_k gave a hint.
\n\nMy first tests show that the transformations work quite well to align the top edge of the fluke but unfortunately given the occlusion of the bottom of the fluke, the result can be quite messy. More points during labeling phase are likely to improve the results but is also a lot of work.
\n\nMy bet is that the top 10 is working on a combination of regular classification and metrics learning as Heng mentionned before.
\n\nEdit: actually, using only the top 5 keypoints give good looking results overall as it preserve the markings.
", "votes": null, "replies": [] } ] }, { "id": 469138, "author_name": "chabir", "author_url": "", "post_date": "02/10/2019 15:13:42", "content": "@toshi_k, once you have re-calculated by regression all your points for the left column, may I ask how you get to the right column?
", "votes": null, "replies": [ { "id": 469375, "author_name": "toshik", "author_url": "", "post_date": "02/11/2019 04:34:31", "content": "Set up the keypoints for destination and transform each part. You can use Piecewise Affine Transformation in scikit-image. https://scikit-image.org/docs/dev/auto_examples/transform/plot_piecewise_affine.html
Thank you !
", "votes": null, "replies": [] } ] }, { "id": 469158, "author_name": "hengck23", "author_url": "", "post_date": "02/10/2019 15:56:32", "content": "actually you can do this:
\n\n1) optional : perform prediction on the test as usual.
\n\n2) you can select a few poor view train images (e.g. choosing those train id with no test images identified, or those with low prediction scores)
\n\nThe rule is that you cannot hand labelled the test image or manually manipulated them. But it is legal to manually manipulated train images. Some process need not to be automatic.
", "votes": null, "replies": [] }, { "id": 470378, "author_name": "alexanderliao", "author_url": "", "post_date": "02/12/2019 20:54:14", "content": "Is this similar to having an STN layer before the first convolution?
", "votes": null, "replies": [ { "id": 472182, "author_name": "xf1994", "author_url": "", "post_date": "02/15/2019 13:06:48", "content": "I think this performs better than STN, because STN is difficult to train. And STN only learn 1 projection, but this work provides multiple projections.
", "votes": null, "replies": [] } ] }, { "id": 472318, "author_name": "strideradu", "author_url": "", "post_date": "02/15/2019 17:27:12", "content": "@toshi_k I am wondering could you share some code example that how to use the alignment? thanks!
", "votes": null, "replies": [ { "id": 472847, "author_name": "chabir", "author_url": "", "post_date": "02/16/2019 18:50:55", "content": "If I may:
\n\nUsing the 1000 keypoints dataset from Paul Johnson, run a regression of the (top 5) keypoints using Martin Piotte bbox kernel by replacing the 2 points of the bbox by the keypoints. (the top 5 points are easier to run and don't forget to remove the NAs in the keypoints dataset).
Using piecewise affine transformation from scikit learn mentionned by toshi_k in this thread, and after choosing a nice and balanced picture of a fluke as a reference that will become the matrix dst, apply the transformation to all the other pictures as src.
Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "468930": "I'm trying piecewise alignment as below. Do you think this approach improve our score ?