![\"BlendFlip\"](\"https://i.imgur.com/q8ZAdPK.jpg\")
Thanks to the sponsors and Kaggle for organizing this important research challenge.
\nTLDR: Solution architecture includes 5 model ensemble at two different image sizes with embedding dimension reduced by PCA. Secret sauce is a newly developed augmentation method designed to deal with large occlusions while maintaining semantic cohesion of the overall scene.
\nThe core problem in this challenge is dealing with large masked occlusions in test/query images. For this my goal was to inpaint the masked region while attempting to:
\nWhen looking at hotel room images, a common pattern is many vertical and horizontal features against repeating backgrounds. For instance wall corners that run vertical, door frames that run vertical and horizontal, pictures on walls that are square, etc. This observation gave me the idea to apply image patches from the same image directly adjacent to the masked region and essentially flip them into the masked region, and then blend the flips from the vertical and horizontal direction, hence the name BlendFlip.
\n
BlendFlip algorithm:
\nBlendFlip helps address 2 1/2 of the 3 problems listed above. It helps maintain semantic cohesion with visually realistic content. It maintains continuity on 2 of the 4 sides of the masked edges. I tried GAN's, traditional CV inpainting methods, variations of CutMix, but BlendFlip significantly outperformed them all.
\n![\"Example](\"https://i.imgur.com/sl6d9IK.jpg\")
\n![\"Example](\"https://i.imgur.com/i18vrXt.jpg\")
\n![\"Example](\"https://i.imgur.com/BxLju54.jpg\")
The goal of blendflip is NOT to predict what might be in the occluded region, but rather maintain the integrity of the image embedding from the non-occluded regions.
\nThe overall solution architecture is shown in the diagram below. Images of sizes 1024x1024 (longest max side scaled to 1024, aspect ratio maintained) or 384x384 are fed into 5 models of various architecture. Each model is trained with ArcFace loss and the embedding size of each is 1536D. The embeddings are then concatenated and reduced to a total of 3072D via PCA (size determined by maintaining 99% of variance of embedding concat), and then KNN performed. Each model is trained with a 50% probability of BlendFlip on each image. BlendFlip is performed on all test images prior to inference. No post processing or re-ranking was applied.
\n![\"Solution](\"https://i.imgur.com/0vbOz3Z.jpg\")
For the CVPR paper I will include proper ablation studies and relative impact of the solution ingredients. BlendFlip accounted for ~0.03-0.04 mAP.
", "messages": [ { "id": 1807094, "postDate": "2022-05-31T17:57:21.937Z", "content": "Thanks to the sponsors and Kaggle for organizing this important research challenge.
\nTLDR: Solution architecture includes 5 model ensemble at two different image sizes with embedding dimension reduced by PCA. Secret sauce is a newly developed augmentation method designed to deal with large occlusions while maintaining semantic cohesion of the overall scene.
\nThe core problem in this challenge is dealing with large masked occlusions in test/query images. For this my goal was to inpaint the masked region while attempting to:
\nWhen looking at hotel room images, a common pattern is many vertical and horizontal features against repeating backgrounds. For instance wall corners that run vertical, door frames that run vertical and horizontal, pictures on walls that are square, etc. This observation gave me the idea to apply image patches from the same image directly adjacent to the masked region and essentially flip them into the masked region, and then blend the flips from the vertical and horizontal direction, hence the name BlendFlip.
\n
BlendFlip algorithm:
\nBlendFlip helps address 2 1/2 of the 3 problems listed above. It helps maintain semantic cohesion with visually realistic content. It maintains continuity on 2 of the 4 sides of the masked edges. I tried GAN's, traditional CV inpainting methods, variations of CutMix, but BlendFlip significantly outperformed them all.
\n
\n
\n
The goal of blendflip is NOT to predict what might be in the occluded region, but rather maintain the integrity of the image embedding from the non-occluded regions.
\nThe overall solution architecture is shown in the diagram below. Images of sizes 1024x1024 (longest max side scaled to 1024, aspect ratio maintained) or 384x384 are fed into 5 models of various architecture. Each model is trained with ArcFace loss and the embedding size of each is 1536D. The embeddings are then concatenated and reduced to a total of 3072D via PCA (size determined by maintaining 99% of variance of embedding concat), and then KNN performed. Each model is trained with a 50% probability of BlendFlip on each image. BlendFlip is performed on all test images prior to inference. No post processing or re-ranking was applied.
\n
For the CVPR paper I will include proper ablation studies and relative impact of the solution ingredients. BlendFlip accounted for ~0.03-0.04 mAP.
", "rawMarkdown": "Thanks to the sponsors and Kaggle for organizing this important research challenge.\n\n***TLDR***: Solution architecture includes 5 model ensemble at two different image sizes with embedding dimension reduced by PCA. Secret sauce is a newly developed augmentation method designed to deal with large occlusions while maintaining semantic cohesion of the overall scene.\n\n## Augmentation - Introducing BlendFlip:\nThe core problem in this challenge is dealing with large masked occlusions in test/query images. For this my goal was to inpaint the masked region while attempting to:\n- maintain semantic cohesion with the overall scene\n- maintain continuity around masked edges\n- fill in visually realistic content\n\nWhen looking at hotel room images, a common pattern is many vertical and horizontal features against repeating backgrounds. For instance wall corners that run vertical, door frames that run vertical and horizontal, pictures on walls that are square, etc. This observation gave me the idea to apply image patches from the same image directly adjacent to the masked region and essentially flip them into the masked region, and then blend the flips from the vertical and horizontal direction, hence the name BlendFlip.\n\n\n\nBlendFlip algorithm:\n1. Determine max distance between masked region edges in the vertical direction (image top to mask top, mask bottom to image bottom). Do same for horizontal. Max direction becomes the regions to be used for inpainting.\n2. Flip max vertical region into masked region. If initial flip doesn’t fill entire masked region, keep flipping in vertical region until masked region is filled, cut off at edge of masked region.\n3. Repeat step 2 for horizontal region.\n4. Take 50/50 blend of step2 and step3 (cv2.addWeighted) as final inpainted region.\n\nBlendFlip helps address 2 1/2 of the 3 problems listed above. It helps maintain semantic cohesion with visually realistic content. It maintains continuity on 2 of the 4 sides of the masked edges. I tried GAN's, traditional CV inpainting methods, variations of CutMix, but BlendFlip significantly outperformed them all.\n\n\n\n\n\nThe goal of blendflip is NOT to predict what might be in the occluded region, but rather maintain the integrity of the image embedding from the non-occluded regions.\n\n## Solution architecture\nThe overall solution architecture is shown in the diagram below. Images of sizes 1024x1024 (longest max side scaled to 1024, aspect ratio maintained) or 384x384 are fed into 5 models of various architecture. Each model is trained with ArcFace loss and the embedding size of each is 1536D. The embeddings are then concatenated and reduced to a total of 3072D via PCA (size determined by maintaining 99% of variance of embedding concat), and then KNN performed. Each model is trained with a 50% probability of BlendFlip on each image. BlendFlip is performed on all test images prior to inference. No post processing or re-ranking was applied. \n\n\n\nFor the CVPR paper I will include proper ablation studies and relative impact of the solution ingredients. BlendFlip accounted for ~0.03-0.04 mAP.", "votes": 30 }, { "id": 1807918, "postDate": "2022-06-01T12:41:47.677Z", "content": "Congrats to the 1st place, amazing results and nice solution! BlendFlip is very interesting idea, so glad to see some innovative approach for the occlusions, I believe it may become a thing how to deal with them in similar problems (I will definitely give it a try next time).
\nCan't wait to see the relative impact of the solution ingredients. Would you mind sharing the effect of PCA here? I am pretty curious.
\nI can't see in the description if you used any external data, does it mean you managed to get the results with only competition data? That would be really impressive.
", "rawMarkdown": "Congrats to the 1st place, amazing results and nice solution! BlendFlip is very interesting idea, so glad to see some innovative approach for the occlusions, I believe it may become a thing how to deal with them in similar problems (I will definitely give it a try next time).\n\nCan't wait to see the relative impact of the solution ingredients. Would you mind sharing the effect of PCA here? I am pretty curious.\n\nI can't see in the description if you used any external data, does it mean you managed to get the results with only competition data? That would be really impressive.", "votes": 1, "replies": [ { "id": 1808228, "postDate": "2022-06-01T16:14:58.990Z", "content": "\n\nWould you mind sharing the effect of PCA here?
\n
Performing PCA gives about a 0.01 mAP improvement (vs simple concat of embeddings) while reducing dimensionality at the same time.
\n\n\nI can't see in the description if you used any external data, does it mean you managed to get the results with only competition data?
\n
I used a very small amount of data from FGVC8, only from classes where <8 samples existed in FGVC9.
", "rawMarkdown": "> Would you mind sharing the effect of PCA here?\n\nPerforming PCA gives about a 0.01 mAP improvement (vs simple concat of embeddings) while reducing dimensionality at the same time.\n\n> I can't see in the description if you used any external data, does it mean you managed to get the results with only competition data?\n\nI used a very small amount of data from FGVC8, only from classes where <8 samples existed in FGVC9.", "votes": 1 } ] }, { "id": 1807330, "postDate": "2022-05-31T23:27:45.093Z", "content": "A huge Congrats David in such an important competition. Beautiful augmentation and Solution architecture.
", "rawMarkdown": "A huge Congrats David in such an important competition. Beautiful augmentation and Solution architecture.", "votes": 1 }, { "id": 1807822, "postDate": "2022-06-01T11:01:38.233Z", "content": "Hi, I am a beginner. I just partly understand your solution. Will you public your code?
", "rawMarkdown": "Hi, I am a beginner. I just partly understand your solution. Will you public your code?", "votes": 2 }, { "id": 1807108, "postDate": "2022-05-31T18:10:45.823Z", "content": "That's a really cool approach, thanks for sharing!
", "rawMarkdown": "That's a really cool approach, thanks for sharing!", "votes": 2 }, { "id": 1904823, "postDate": "2022-08-18T14:19:01.697Z", "content": "Super cool!
", "rawMarkdown": "Super cool!" }, { "id": 1808069, "postDate": "2022-06-01T14:13:14.510Z", "content": "Brilliant @tivfrvqhs5. Thanks for explaining.
", "rawMarkdown": "Brilliant @tivfrvqhs5. Thanks for explaining." }, { "id": 1811206, "postDate": "2022-06-04T12:49:26.543Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1807918, "author_name": "Michal", "author_url": "", "post_date": "2022-06-01T12:41:47.677000", "content": "Congrats to the 1st place, amazing results and nice solution! BlendFlip is very interesting idea, so glad to see some innovative approach for the occlusions, I believe it may become a thing how to deal with them in similar problems (I will definitely give it a try next time).
\nCan't wait to see the relative impact of the solution ingredients. Would you mind sharing the effect of PCA here? I am pretty curious.
\nI can't see in the description if you used any external data, does it mean you managed to get the results with only competition data? That would be really impressive.
", "votes": 1, "replies": [ { "id": 1808228, "author_name": "David Austin", "author_url": "", "post_date": "2022-06-01T16:14:58.990000", "content": "\n\nWould you mind sharing the effect of PCA here?
\n
Performing PCA gives about a 0.01 mAP improvement (vs simple concat of embeddings) while reducing dimensionality at the same time.
\n\n\nI can't see in the description if you used any external data, does it mean you managed to get the results with only competition data?
\n
I used a very small amount of data from FGVC8, only from classes where <8 samples existed in FGVC9.
", "votes": 1, "replies": [] } ] }, { "id": 1807330, "author_name": "Marília Prata", "author_url": "", "post_date": "2022-05-31T23:27:45.093000", "content": "A huge Congrats David in such an important competition. Beautiful augmentation and Solution architecture.
", "votes": 1, "replies": [] }, { "id": 1807822, "author_name": "Hien Tq", "author_url": "", "post_date": "2022-06-01T11:01:38.233000", "content": "Hi, I am a beginner. I just partly understand your solution. Will you public your code?
", "votes": 2, "replies": [] }, { "id": 1807108, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-05-31T18:10:45.823000", "content": "That's a really cool approach, thanks for sharing!
", "votes": 2, "replies": [] }, { "id": 1904823, "author_name": "bobam", "author_url": "", "post_date": "2022-08-18T14:19:01.697000", "content": "Super cool!
", "votes": 0, "replies": [] }, { "id": 1808069, "author_name": "David Roberts", "author_url": "", "post_date": "2022-06-01T14:13:14.510000", "content": "Brilliant @tivfrvqhs5. Thanks for explaining.
", "votes": 0, "replies": [] }, { "id": 1811206, "author_name": "", "author_url": "", "post_date": "2022-06-04T12:49:26.543000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1807094": "Thanks to the sponsors and Kaggle for organizing this important research challenge.\n\n***TLDR***: Solution architecture includes 5 model ensemble at two different image sizes with embedding dimension reduced by PCA. Secret sauce is a newly developed augmentation method designed to deal with large occlusions while maintaining semantic cohesion of the overall scene.\n\n## Augmentation - Introducing BlendFlip:\nThe core problem in this challenge is dealing with large masked occlusions in test/query images. For this my goal was to inpaint the masked region while attempting to:\n- maintain semantic cohesion with the overall scene\n- maintain continuity around masked edges\n- fill in visually realistic content\n\nWhen looking at hotel room images, a common pattern is many vertical and horizontal features against repeating backgrounds. For instance wall corners that run vertical, door frames that run vertical and horizontal, pictures on walls that are square, etc. This observation gave me the idea to apply image patches from the same image directly adjacent to the masked region and essentially flip them into the masked region, and then blend the flips from the vertical and horizontal direction, hence the name BlendFlip.\n\n\n\nBlendFlip algorithm:\n1. Determine max distance between masked region edges in the vertical direction (image top to mask top, mask bottom to image bottom). Do same for horizontal. Max direction becomes the regions to be used for inpainting.\n2. Flip max vertical region into masked region. If initial flip doesn’t fill entire masked region, keep flipping in vertical region until masked region is filled, cut off at edge of masked region.\n3. Repeat step 2 for horizontal region.\n4. Take 50/50 blend of step2 and step3 (cv2.addWeighted) as final inpainted region.\n\nBlendFlip helps address 2 1/2 of the 3 problems listed above. It helps maintain semantic cohesion with visually realistic content. It maintains continuity on 2 of the 4 sides of the masked edges. I tried GAN's, traditional CV inpainting methods, variations of CutMix, but BlendFlip significantly outperformed them all.\n\n\n\n\n\nThe goal of blendflip is NOT to predict what might be in the occluded region, but rather maintain the integrity of the image embedding from the non-occluded regions.\n\n## Solution architecture\nThe overall solution architecture is shown in the diagram below. Images of sizes 1024x1024 (longest max side scaled to 1024, aspect ratio maintained) or 384x384 are fed into 5 models of various architecture. Each model is trained with ArcFace loss and the embedding size of each is 1536D. The embeddings are then concatenated and reduced to a total of 3072D via PCA (size determined by maintaining 99% of variance of embedding concat), and then KNN performed. Each model is trained with a 50% probability of BlendFlip on each image. BlendFlip is performed on all test images prior to inference. No post processing or re-ranking was applied. \n\n\n\nFor the CVPR paper I will include proper ablation studies and relative impact of the solution ingredients. BlendFlip accounted for ~0.03-0.04 mAP.", "1807918": "Congrats to the 1st place, amazing results and nice solution! BlendFlip is very interesting idea, so glad to see some innovative approach for the occlusions, I believe it may become a thing how to deal with them in similar problems (I will definitely give it a try next time).\n\nCan't wait to see the relative impact of the solution ingredients. Would you mind sharing the effect of PCA here? I am pretty curious.\n\nI can't see in the description if you used any external data, does it mean you managed to get the results with only competition data? That would be really impressive.", "1807330": "A huge Congrats David in such an important competition. Beautiful augmentation and Solution architecture.", "1807822": "Hi, I am a beginner. I just partly understand your solution. Will you public your code?", "1807108": "That's a really cool approach, thanks for sharing!", "1904823": "Super cool!", "1808069": "Brilliant @tivfrvqhs5. Thanks for explaining.", "1811206": "" } }