{
  "id": 324697,
  "title": "Train/test data leak?",
  "url": "/competitions/snakeclef2022/discussion/324697",
  "author_name": "",
  "post_date": "2022-05-12T18:42:34.640056200Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi, <br>\nI was looking at the competition data and it seems that there are 81 images almost identical between the train and test data. Here are some examples:</p>\n<pre><code>,from,to,distance,from_type,to_type,pair_type\n2599,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2011/Lampropeltis_abnorma/30907830.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/69365.JPG,0.999982,train,test,test-train\n3678,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2021/Hypsiglena_chlorophaea/150370061.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/104719655.jpg,0.99812,train,test,test-train\n3743,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2013/Thamnophis_radix/34351633.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/18001682.jpeg,0.997698,train,test,test-train\n3815,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874120.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328594.jpeg,0.996949,train,test,test-train\n3858,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874114.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328591.jpeg,0.996476,train,test,test-train\n3893,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2020/Emydocephalus_annulatus/69452450.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/102841459.jpg,0.996273,train,test,test-train\n3942,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874102.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328604.jpeg,0.995844,train,test,test-train\n3975,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2018/Leptotyphlops_scutifrons/98211206.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/136925661.jpeg,0.995515,train,test,test-train\n3981,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2019/Coronella_austriaca/49881068.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/49888825.jpeg,0.995484,train,test,test-train\n</code></pre>\n<p>I wonder if I am missing anything ? Could it be that the same snake is both in the train and test data?</p>",
  "messages": [
    {
      "id": "1786289",
      "postDate": "05/12/2022 18:42:34",
      "content": "<p>Hi, <br>\nI was looking at the competition data and it seems that there are 81 images almost identical between the train and test data. Here are some examples:</p>\n<pre><code>,from,to,distance,from_type,to_type,pair_type\n2599,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2011/Lampropeltis_abnorma/30907830.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/69365.JPG,0.999982,train,test,test-train\n3678,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2021/Hypsiglena_chlorophaea/150370061.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/104719655.jpg,0.99812,train,test,test-train\n3743,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2013/Thamnophis_radix/34351633.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/18001682.jpeg,0.997698,train,test,test-train\n3815,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874120.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328594.jpeg,0.996949,train,test,test-train\n3858,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874114.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328591.jpeg,0.996476,train,test,test-train\n3893,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2020/Emydocephalus_annulatus/69452450.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/102841459.jpg,0.996273,train,test,test-train\n3942,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874102.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328604.jpeg,0.995844,train,test,test-train\n3975,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2018/Leptotyphlops_scutifrons/98211206.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/136925661.jpeg,0.995515,train,test,test-train\n3981,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2019/Coronella_austriaca/49881068.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/49888825.jpeg,0.995484,train,test,test-train\n</code></pre>\n<p>I wonder if I am missing anything ? Could it be that the same snake is both in the train and test data?</p>",
      "rawMarkdown": "Hi, \nI was looking at the competition data and it seems that there are 81 images almost identical between the train and test data. Here are some examples:\n```\n,from,to,distance,from_type,to_type,pair_type\n2599,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2011/Lampropeltis_abnorma/30907830.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/69365.JPG,0.999982,train,test,test-train\n3678,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2021/Hypsiglena_chlorophaea/150370061.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/104719655.jpg,0.99812,train,test,test-train\n3743,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2013/Thamnophis_radix/34351633.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/18001682.jpeg,0.997698,train,test,test-train\n3815,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874120.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328594.jpeg,0.996949,train,test,test-train\n3858,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874114.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328591.jpeg,0.996476,train,test,test-train\n3893,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2020/Emydocephalus_annulatus/69452450.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/102841459.jpg,0.996273,train,test,test-train\n3942,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874102.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328604.jpeg,0.995844,train,test,test-train\n3975,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2018/Leptotyphlops_scutifrons/98211206.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/136925661.jpeg,0.995515,train,test,test-train\n3981,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2019/Coronella_austriaca/49881068.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/49888825.jpeg,0.995484,train,test,test-train\n```\nI wonder if I am missing anything ? Could it be that the same snake is both in the train and test data?",
      "votes": null
    },
    {
      "id": "1786365",
      "postDate": "05/12/2022 19:50:50",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/graphlab\" target=\"_blank\">@graphlab</a>,</p>\n<p>Might be possible. Technically the data are from different time and labeled as different observation. <br>\nAs the data origin is from the citizen science platform, people might submit same picture multiple times or just plagiarize it from another user.</p>\n<p>81 images should not be a big deal at all.</p>\n<p>LP</p>",
      "rawMarkdown": "Hi @graphlab,\n\nMight be possible. Technically the data are from different time and labeled as different observation. \nAs the data origin is from the citizen science platform, people might submit same picture multiple times or just plagiarize it from another user.\n\n81 images should not be a big deal at all.\n\nLP",
      "votes": null
    },
    {
      "id": "1814993",
      "postDate": "06/08/2022 14:41:56",
      "content": "<p>Please refer to my topic in discussion, I need your help, your upvote, thanks. <a href=\"https://www.kaggle.com/graphlab\" target=\"_blank\">@graphlab</a> </p>",
      "rawMarkdown": "Please refer to my topic in discussion, I need your help, your upvote, thanks. @graphlab",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1786365,
      "author_name": "picekl",
      "author_url": "",
      "post_date": "05/12/2022 19:50:50",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/graphlab\" target=\"_blank\">@graphlab</a>,</p>\n<p>Might be possible. Technically the data are from different time and labeled as different observation. <br>\nAs the data origin is from the citizen science platform, people might submit same picture multiple times or just plagiarize it from another user.</p>\n<p>81 images should not be a big deal at all.</p>\n<p>LP</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1814993,
      "author_name": "abuduori",
      "author_url": "",
      "post_date": "06/08/2022 14:41:56",
      "content": "<p>Please refer to my topic in discussion, I need your help, your upvote, thanks. <a href=\"https://www.kaggle.com/graphlab\" target=\"_blank\">@graphlab</a> </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1786289": "Hi, \nI was looking at the competition data and it seems that there are 81 images almost identical between the train and test data. Here are some examples:\n```\n,from,to,distance,from_type,to_type,pair_type\n2599,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2011/Lampropeltis_abnorma/30907830.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/69365.JPG,0.999982,train,test,test-train\n3678,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2021/Hypsiglena_chlorophaea/150370061.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/104719655.jpg,0.99812,train,test,test-train\n3743,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2013/Thamnophis_radix/34351633.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/18001682.jpeg,0.997698,train,test,test-train\n3815,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874120.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328594.jpeg,0.996949,train,test,test-train\n3858,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874114.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328591.jpeg,0.996476,train,test,test-train\n3893,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2020/Emydocephalus_annulatus/69452450.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/102841459.jpg,0.996273,train,test,test-train\n3942,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2017/Hemorrhois_hippocrepis/8874102.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/23328604.jpeg,0.995844,train,test,test-train\n3975,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2018/Leptotyphlops_scutifrons/98211206.jpg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/136925661.jpeg,0.995515,train,test,test-train\n3981,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-medium_size_train/SnakeCLEF2022-medium_size/2019/Coronella_austriaca/49881068.jpeg,/home/jupyter/data2/snakeclef2022-fgvc9/SnakeCLEF2022-test_images/SnakeCLEF2022-large_size/49888825.jpeg,0.995484,train,test,test-train\n```\nI wonder if I am missing anything ? Could it be that the same snake is both in the train and test data?",
    "1786365": "Hi @graphlab,\n\nMight be possible. Technically the data are from different time and labeled as different observation. \nAs the data origin is from the citizen science platform, people might submit same picture multiple times or just plagiarize it from another user.\n\n81 images should not be a big deal at all.\n\nLP",
    "1814993": "Please refer to my topic in discussion, I need your help, your upvote, thanks. @graphlab"
  },
  "source": "meta"
}