{
  "id": 77403,
  "title": "Idea for Siamese validation",
  "url": "/competitions/humpback-whale-identification/discussion/77403",
  "author_name": "",
  "post_date": "2019-01-12T09:18:54.194030900Z",
  "votes": 7,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi, <br>\nRegarding to siamese validation, I am working with this idea. Please let me know your comments. <br>\nI assume that most of us follows the idea of @martinpiotte at the previous playground competition. I have took a look at his solution and have an idea for validation strategy.  </p>\n\n<p>To construct training set:  </p>\n\n<ul>\n<li>All <code>new_whale</code> images are removed  </li>\n<li>All whales with a single image are removed.  </li>\n</ul>\n\n<p>Those images above are not neccessary for <code>training</code>, they can be used for <code>validation</code>. My ideas to construct validation set are: <br>\n - All <code>new_whale</code> and <code>whales with single image</code> will become <code>new_whale</code> on validation. <br>\n - The rest of training, we take random images (Ex: 10-&gt;20%) for validation if this <code>whale_id</code> has number of image greaters than X (Ex: X = 10 images). <br>\n - Merge all images above we will have a validation set. It includes <code>new_whale</code> images which model hasn't learned yet and <code>known whales</code> that model learned so far.  </p>\n\n<p>I am making my first baseline with this idea. I will report when it is completed. </p>",
  "messages": [
    {
      "id": "454818",
      "postDate": "01/12/2019 09:18:54",
      "content": "<p>Hi, <br>\nRegarding to siamese validation, I am working with this idea. Please let me know your comments. <br>\nI assume that most of us follows the idea of @martinpiotte at the previous playground competition. I have took a look at his solution and have an idea for validation strategy.  </p>\n\n<p>To construct training set:  </p>\n\n<ul>\n<li>All <code>new_whale</code> images are removed  </li>\n<li>All whales with a single image are removed.  </li>\n</ul>\n\n<p>Those images above are not neccessary for <code>training</code>, they can be used for <code>validation</code>. My ideas to construct validation set are: <br>\n - All <code>new_whale</code> and <code>whales with single image</code> will become <code>new_whale</code> on validation. <br>\n - The rest of training, we take random images (Ex: 10-&gt;20%) for validation if this <code>whale_id</code> has number of image greaters than X (Ex: X = 10 images). <br>\n - Merge all images above we will have a validation set. It includes <code>new_whale</code> images which model hasn't learned yet and <code>known whales</code> that model learned so far.  </p>\n\n<p>I am making my first baseline with this idea. I will report when it is completed. </p>",
      "rawMarkdown": "Hi,  \nRegarding to siamese validation, I am working with this idea. Please let me know your comments.  \nI assume that most of us follows the idea of @martinpiotte at the previous playground competition. I have took a look at his solution and have an idea for validation strategy.  \n\nTo construct training set:  \n\n - All `new_whale` images are removed  \n - All whales with a single image are removed.  \n\nThose images above are not neccessary for `training`, they can be used for `validation`. My ideas to construct validation set are:  \n - All `new_whale` and `whales with single image` will become `new_whale` on validation.  \n - The rest of training, we take random images (Ex: 10-&gt;20%) for validation if this `whale_id` has number of image greaters than X (Ex: X = 10 images).  \n - Merge all images above we will have a validation set. It includes `new_whale` images which model hasn't learned yet and `known whales` that model learned so far.  \n\nI am making my first baseline with this idea. I will report when it is completed.",
      "votes": null
    },
    {
      "id": "455048",
      "postDate": "01/12/2019 20:20:36",
      "content": "<p>This sounds like a good idea to me. I am interested in how your tests turn out.</p>",
      "rawMarkdown": "This sounds like a good idea to me. I am interested in how your tests turn out.",
      "votes": null
    },
    {
      "id": "459049",
      "postDate": "01/21/2019 04:32:08",
      "content": "<p>Hi @Brian, <br>\nI am sorry for the late report. Since I was busy with other projects,  I just have finished my simple baseline (backbone resnet18). <br>\nAs the local CV stratergy above, I have over a half of <code>new_whale</code>, then I could archive 0.17 CV and 0.365 LB. </p>",
      "rawMarkdown": "Hi @Brian,  \nI am sorry for the late report. Since I was busy with other projects,  I just have finished my simple baseline (backbone resnet18).   \nAs the local CV stratergy above, I have over a half of `new_whale`, then I could archive 0.17 CV and 0.365 LB.",
      "votes": null
    },
    {
      "id": "459096",
      "postDate": "01/21/2019 07:10:16",
      "content": "<p>Updated: <br>\nI trained my Siamese baseline with resnet18 as backbone. After 20 epochs, I have following observations:  </p>\n\n<ul>\n<li><p>Threshold: 0.9 <br>\nTrain: 0.88, Valid 0.17, LB: 0.365  </p></li>\n<li><p>Threshold: 0.9999 <br>\nTrain: 0.75, Valid: 0.88, LB: 0.574  </p></li>\n</ul>\n\n<p>By tunning threshold as validation set, I can archive better results.   </p>\n\n<ul>\n<li><p>High threshold will lead to have more <code>new_whale</code> in train predictions. Train ground truth does not contain <code>new_whale</code>, so score is worse.  </p></li>\n<li><p>High validation may caused by <code>new_whale</code> classes (&gt;50% is <code>new_whale</code> in validation set).  </p></li>\n<li><p>The GAP between CV and LB shows that: LB will contains more <code>rare class</code> (the class has one image).  </p></li>\n</ul>\n\n<p>So, I can say that we should pay more attention to <code>new_whale</code> and <code>rare class</code> . </p>",
      "rawMarkdown": "Updated:  \nI trained my Siamese baseline with resnet18 as backbone. After 20 epochs, I have following observations:  \n\n\n- Threshold: 0.9  \nTrain: 0.88, Valid 0.17, LB: 0.365  \n\n- Threshold: 0.9999  \nTrain: 0.75, Valid: 0.88, LB: 0.574  \n\nBy tunning threshold as validation set, I can archive better results.   \n\n\n- High threshold will lead to have more `new_whale` in train predictions. Train ground truth does not contain `new_whale`, so score is worse.  \n\n- High validation may caused by `new_whale` classes (&gt;50% is `new_whale` in validation set).  \n\n\n- The GAP between CV and LB shows that: LB will contains more `rare class` (the class has one image).  \n\nSo, I can say that we should pay more attention to `new_whale` and `rare class` .",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 455048,
      "author_name": "ldm314",
      "author_url": "",
      "post_date": "01/12/2019 20:20:36",
      "content": "<p>This sounds like a good idea to me. I am interested in how your tests turn out.</p>",
      "votes": null,
      "replies": [
        {
          "id": 459049,
          "author_name": "backaggle",
          "author_url": "",
          "post_date": "01/21/2019 04:32:08",
          "content": "<p>Hi @Brian, <br>\nI am sorry for the late report. Since I was busy with other projects,  I just have finished my simple baseline (backbone resnet18). <br>\nAs the local CV stratergy above, I have over a half of <code>new_whale</code>, then I could archive 0.17 CV and 0.365 LB. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 459096,
      "author_name": "backaggle",
      "author_url": "",
      "post_date": "01/21/2019 07:10:16",
      "content": "<p>Updated: <br>\nI trained my Siamese baseline with resnet18 as backbone. After 20 epochs, I have following observations:  </p>\n\n<ul>\n<li><p>Threshold: 0.9 <br>\nTrain: 0.88, Valid 0.17, LB: 0.365  </p></li>\n<li><p>Threshold: 0.9999 <br>\nTrain: 0.75, Valid: 0.88, LB: 0.574  </p></li>\n</ul>\n\n<p>By tunning threshold as validation set, I can archive better results.   </p>\n\n<ul>\n<li><p>High threshold will lead to have more <code>new_whale</code> in train predictions. Train ground truth does not contain <code>new_whale</code>, so score is worse.  </p></li>\n<li><p>High validation may caused by <code>new_whale</code> classes (&gt;50% is <code>new_whale</code> in validation set).  </p></li>\n<li><p>The GAP between CV and LB shows that: LB will contains more <code>rare class</code> (the class has one image).  </p></li>\n</ul>\n\n<p>So, I can say that we should pay more attention to <code>new_whale</code> and <code>rare class</code> . </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "454818": "Hi,  \nRegarding to siamese validation, I am working with this idea. Please let me know your comments.  \nI assume that most of us follows the idea of @martinpiotte at the previous playground competition. I have took a look at his solution and have an idea for validation strategy.  \n\nTo construct training set:  \n\n - All `new_whale` images are removed  \n - All whales with a single image are removed.  \n\nThose images above are not neccessary for `training`, they can be used for `validation`. My ideas to construct validation set are:  \n - All `new_whale` and `whales with single image` will become `new_whale` on validation.  \n - The rest of training, we take random images (Ex: 10-&gt;20%) for validation if this `whale_id` has number of image greaters than X (Ex: X = 10 images).  \n - Merge all images above we will have a validation set. It includes `new_whale` images which model hasn't learned yet and `known whales` that model learned so far.  \n\nI am making my first baseline with this idea. I will report when it is completed.",
    "455048": "This sounds like a good idea to me. I am interested in how your tests turn out.",
    "459049": "Hi @Brian,  \nI am sorry for the late report. Since I was busy with other projects,  I just have finished my simple baseline (backbone resnet18).   \nAs the local CV stratergy above, I have over a half of `new_whale`, then I could archive 0.17 CV and 0.365 LB.",
    "459096": "Updated:  \nI trained my Siamese baseline with resnet18 as backbone. After 20 epochs, I have following observations:  \n\n\n- Threshold: 0.9  \nTrain: 0.88, Valid 0.17, LB: 0.365  \n\n- Threshold: 0.9999  \nTrain: 0.75, Valid: 0.88, LB: 0.574  \n\nBy tunning threshold as validation set, I can archive better results.   \n\n\n- High threshold will lead to have more `new_whale` in train predictions. Train ground truth does not contain `new_whale`, so score is worse.  \n\n- High validation may caused by `new_whale` classes (&gt;50% is `new_whale` in validation set).  \n\n\n- The GAP between CV and LB shows that: LB will contains more `rare class` (the class has one image).  \n\nSo, I can say that we should pay more attention to `new_whale` and `rare class` ."
  },
  "source": "meta"
}