{
  "id": 64588,
  "title": "Find overlapping images in training set",
  "url": "/competitions/airbus-ship-detection/discussion/64588",
  "author_name": "",
  "post_date": "2018-08-30T13:58:28.744672300Z",
  "votes": 7,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Can anyone suggest a way to find overlapping images in train set? I want to split train set and validation set and ensure these is no overlapping images between them. How do I detect them efficiently? I thought about using knn but the training set is too large and cannot be fitted into memory. Should I try using pretrained CNNs as feature extruder then pass results to a knn?</p>",
  "messages": [
    {
      "id": "378723",
      "postDate": "08/30/2018 13:58:28",
      "content": "<p>Can anyone suggest a way to find overlapping images in train set? I want to split train set and validation set and ensure these is no overlapping images between them. How do I detect them efficiently? I thought about using knn but the training set is too large and cannot be fitted into memory. Should I try using pretrained CNNs as feature extruder then pass results to a knn?</p>",
      "rawMarkdown": "Can anyone suggest a way to find overlapping images in train set? I want to split train set and validation set and ensure these is no overlapping images between them. How do I detect them efficiently? I thought about using knn but the training set is too large and cannot be fitted into memory. Should I try using pretrained CNNs as feature extruder then pass results to a knn?",
      "votes": null
    },
    {
      "id": "378728",
      "postDate": "08/30/2018 14:06:46",
      "content": "<p>Yes, extract features with a pretrained CNN and then find top k NN for each to compare them. That's a good general approach. In this comp, since you know now that the patches are 256px shifted version there's faster ways to do it as well.</p>",
      "rawMarkdown": "Yes, extract features with a pretrained CNN and then find top k NN for each to compare them. That's a good general approach. In this comp, since you know now that the patches are 256px shifted version there's faster ways to do it as well.",
      "votes": null
    },
    {
      "id": "378737",
      "postDate": "08/30/2018 14:11:20",
      "content": "<p>Thanks! This is a good idea.</p>",
      "rawMarkdown": "Thanks! This is a good idea.",
      "votes": null
    },
    {
      "id": "381256",
      "postDate": "09/04/2018 11:29:01",
      "content": "<p>Hi Wudi Wang. The images (at least many of them) are slices of the <em>same</em> image, not separate frames taken at different times. This fact significantly simplifies the task and allows direct comparison <code>if np.sum(crop1 != crop2) == 0: ...</code> Still, this is computationally costly, and all pixels from crop1 and crop2 might not be equal due to processing artifact. So, try to compare only small enough crops. </p>\n\n<p>There is much simpler way to find overlapping images with ships. If images contain the same ship, they overlap. At the first step, we need to find all images that contain a particular ship. For each ship you can calculate simple metrics like (count_of_pixels, sum_of_pixels, sum_of_squares_of_pixels, ...). For the same ship on all images this tuple should be the same. \nAt the second step, we need to enumerate image sets in such a way that a set contains overlapping images. For example, image1 has ship1, image2 has ship1 and ship2, image3 has ship2 and ship3, image4 has ship3 and ship4, then image1...image4 will be in the same set, although image1 and image4 are not overlapping.</p>\n\n<p>This logic will yield in sets of overlapping (however, not all pairs will be overlapping) images. Some sets will contain dozens images.</p>",
      "rawMarkdown": "Hi Wudi Wang. The images (at least many of them) are slices of the *same* image, not separate frames taken at different times. This fact significantly simplifies the task and allows direct comparison `if np.sum(crop1 != crop2) == 0: ...` Still, this is computationally costly, and all pixels from crop1 and crop2 might not be equal due to processing artifact. So, try to compare only small enough crops. \n\nThere is much simpler way to find overlapping images with ships. If images contain the same ship, they overlap. At the first step, we need to find all images that contain a particular ship. For each ship you can calculate simple metrics like (count_of_pixels, sum_of_pixels, sum_of_squares_of_pixels, ...). For the same ship on all images this tuple should be the same. \nAt the second step, we need to enumerate image sets in such a way that a set contains overlapping images. For example, image1 has ship1, image2 has ship1 and ship2, image3 has ship2 and ship3, image4 has ship3 and ship4, then image1...image4 will be in the same set, although image1 and image4 are not overlapping.\n\nThis logic will yield in sets of overlapping (however, not all pairs will be overlapping) images. Some sets will contain dozens images.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 378728,
      "author_name": "antorsae",
      "author_url": "",
      "post_date": "08/30/2018 14:06:46",
      "content": "<p>Yes, extract features with a pretrained CNN and then find top k NN for each to compare them. That's a good general approach. In this comp, since you know now that the patches are 256px shifted version there's faster ways to do it as well.</p>",
      "votes": null,
      "replies": [
        {
          "id": 378737,
          "author_name": "woodywang",
          "author_url": "",
          "post_date": "08/30/2018 14:11:20",
          "content": "<p>Thanks! This is a good idea.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 381256,
      "author_name": "buinyi",
      "author_url": "",
      "post_date": "09/04/2018 11:29:01",
      "content": "<p>Hi Wudi Wang. The images (at least many of them) are slices of the <em>same</em> image, not separate frames taken at different times. This fact significantly simplifies the task and allows direct comparison <code>if np.sum(crop1 != crop2) == 0: ...</code> Still, this is computationally costly, and all pixels from crop1 and crop2 might not be equal due to processing artifact. So, try to compare only small enough crops. </p>\n\n<p>There is much simpler way to find overlapping images with ships. If images contain the same ship, they overlap. At the first step, we need to find all images that contain a particular ship. For each ship you can calculate simple metrics like (count_of_pixels, sum_of_pixels, sum_of_squares_of_pixels, ...). For the same ship on all images this tuple should be the same. \nAt the second step, we need to enumerate image sets in such a way that a set contains overlapping images. For example, image1 has ship1, image2 has ship1 and ship2, image3 has ship2 and ship3, image4 has ship3 and ship4, then image1...image4 will be in the same set, although image1 and image4 are not overlapping.</p>\n\n<p>This logic will yield in sets of overlapping (however, not all pairs will be overlapping) images. Some sets will contain dozens images.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "378723": "Can anyone suggest a way to find overlapping images in train set? I want to split train set and validation set and ensure these is no overlapping images between them. How do I detect them efficiently? I thought about using knn but the training set is too large and cannot be fitted into memory. Should I try using pretrained CNNs as feature extruder then pass results to a knn?",
    "378728": "Yes, extract features with a pretrained CNN and then find top k NN for each to compare them. That's a good general approach. In this comp, since you know now that the patches are 256px shifted version there's faster ways to do it as well.",
    "378737": "Thanks! This is a good idea.",
    "381256": "Hi Wudi Wang. The images (at least many of them) are slices of the *same* image, not separate frames taken at different times. This fact significantly simplifies the task and allows direct comparison `if np.sum(crop1 != crop2) == 0: ...` Still, this is computationally costly, and all pixels from crop1 and crop2 might not be equal due to processing artifact. So, try to compare only small enough crops. \n\nThere is much simpler way to find overlapping images with ships. If images contain the same ship, they overlap. At the first step, we need to find all images that contain a particular ship. For each ship you can calculate simple metrics like (count_of_pixels, sum_of_pixels, sum_of_squares_of_pixels, ...). For the same ship on all images this tuple should be the same. \nAt the second step, we need to enumerate image sets in such a way that a set contains overlapping images. For example, image1 has ship1, image2 has ship1 and ship2, image3 has ship2 and ship3, image4 has ship3 and ship4, then image1...image4 will be in the same set, although image1 and image4 are not overlapping.\n\nThis logic will yield in sets of overlapping (however, not all pairs will be overlapping) images. Some sets will contain dozens images."
  },
  "source": "meta"
}