{
  "id": 398940,
  "title": "Intuition on why printer_id and print_id can't be used in testing",
  "url": "/competitions/early-detection-of-3d-printing-issues/discussion/398940",
  "author_name": "",
  "post_date": "2023-04-01T15:45:58.579338800Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>One rule for in using the data is you can't use either <code>printer_id</code> or <code>print_id</code> in testing.</p>\n<p>Just to give you some intuition on this rule. The fact that all pictures in the same print are labeled the same is a bug, not a \"feature\". In the real world, it's often the case that there are under-extrusion only in some parts of a print. Other parts of the same print are perfectly fine. We assigned the same label to all pictures in the same print simply because it's simply too time-consuming to label picture-by-picture.</p>\n<p>This is why if a model uses either <code>printer_id</code> or <code>print_id</code> in inference, it'll probably achieve very high F1 score. But it'll fail miserably in the real world.</p>",
  "messages": [
    {
      "id": "2205475",
      "postDate": "04/01/2023 15:45:58",
      "content": "<p>One rule for in using the data is you can't use either <code>printer_id</code> or <code>print_id</code> in testing.</p>\n<p>Just to give you some intuition on this rule. The fact that all pictures in the same print are labeled the same is a bug, not a \"feature\". In the real world, it's often the case that there are under-extrusion only in some parts of a print. Other parts of the same print are perfectly fine. We assigned the same label to all pictures in the same print simply because it's simply too time-consuming to label picture-by-picture.</p>\n<p>This is why if a model uses either <code>printer_id</code> or <code>print_id</code> in inference, it'll probably achieve very high F1 score. But it'll fail miserably in the real world.</p>",
      "rawMarkdown": "One rule for in using the data is you can't use either `printer_id` or `print_id` in testing.\n\nJust to give you some intuition on this rule. The fact that all pictures in the same print are labeled the same is a bug, not a \"feature\". In the real world, it's often the case that there are under-extrusion only in some parts of a print. Other parts of the same print are perfectly fine. We assigned the same label to all pictures in the same print simply because it's simply too time-consuming to label picture-by-picture.\n\nThis is why if a model uses either `printer_id` or `print_id` in inference, it'll probably achieve very high F1 score. But it'll fail miserably in the real world.",
      "votes": null
    },
    {
      "id": "2222071",
      "postDate": "04/14/2023 21:26:06",
      "content": "<p>One of the challenges here is that under extrusion might be present only in a fraction of the entire <code>print_id</code> image set. For example, if only 20% of the images for a specific <code>print_id</code> show signs of under extrusion, and the labelling process is such that all labels for this <code>print_id</code> are labeled as positive for having under extrusion, we may end up having a lot of mislabeled images. </p>\n<p>In my opinion, the winner should be selected on the basis of how many <code>print_id</code>s were correctly detected as having under extrusion - they should not be penalized for corrently labeling an image as \"not having under extrusion\" if the image does not show signs of under extrusion </p>",
      "rawMarkdown": "One of the challenges here is that under extrusion might be present only in a fraction of the entire `print_id` image set. For example, if only 20% of the images for a specific `print_id` show signs of under extrusion, and the labelling process is such that all labels for this `print_id` are labeled as positive for having under extrusion, we may end up having a lot of mislabeled images. \n\nIn my opinion, the winner should be selected on the basis of how many `print_id`s were correctly detected as having under extrusion - they should not be penalized for corrently labeling an image as \"not having under extrusion\" if the image does not show signs of under extrusion",
      "votes": null
    },
    {
      "id": "2222209",
      "postDate": "04/15/2023 03:49:56",
      "content": "<p>We have only a handful of <code>print_id</code>s in the test data set. Hence t doesn't make sense to base the result on <code>print_id</code>s.</p>\n<p>What you said is more about the noise in the labels. You can check this post for more details: <a href=\"https://www.kaggle.com/competitions/early-detection-of-3d-printing-issues/discussion/398725\" target=\"_blank\">https://www.kaggle.com/competitions/early-detection-of-3d-printing-issues/discussion/398725</a></p>",
      "rawMarkdown": "We have only a handful of `print_id`s in the test data set. Hence t doesn't make sense to base the result on `print_id`s.\n\nWhat you said is more about the noise in the labels. You can check this post for more details: https://www.kaggle.com/competitions/early-detection-of-3d-printing-issues/discussion/398725",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2222071,
      "author_name": "sspathak",
      "author_url": "",
      "post_date": "04/14/2023 21:26:06",
      "content": "<p>One of the challenges here is that under extrusion might be present only in a fraction of the entire <code>print_id</code> image set. For example, if only 20% of the images for a specific <code>print_id</code> show signs of under extrusion, and the labelling process is such that all labels for this <code>print_id</code> are labeled as positive for having under extrusion, we may end up having a lot of mislabeled images. </p>\n<p>In my opinion, the winner should be selected on the basis of how many <code>print_id</code>s were correctly detected as having under extrusion - they should not be penalized for corrently labeling an image as \"not having under extrusion\" if the image does not show signs of under extrusion </p>",
      "votes": null,
      "replies": [
        {
          "id": 2222209,
          "author_name": "kennethjiangobico",
          "author_url": "",
          "post_date": "04/15/2023 03:49:56",
          "content": "<p>We have only a handful of <code>print_id</code>s in the test data set. Hence t doesn't make sense to base the result on <code>print_id</code>s.</p>\n<p>What you said is more about the noise in the labels. You can check this post for more details: <a href=\"https://www.kaggle.com/competitions/early-detection-of-3d-printing-issues/discussion/398725\" target=\"_blank\">https://www.kaggle.com/competitions/early-detection-of-3d-printing-issues/discussion/398725</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2205475": "One rule for in using the data is you can't use either `printer_id` or `print_id` in testing.\n\nJust to give you some intuition on this rule. The fact that all pictures in the same print are labeled the same is a bug, not a \"feature\". In the real world, it's often the case that there are under-extrusion only in some parts of a print. Other parts of the same print are perfectly fine. We assigned the same label to all pictures in the same print simply because it's simply too time-consuming to label picture-by-picture.\n\nThis is why if a model uses either `printer_id` or `print_id` in inference, it'll probably achieve very high F1 score. But it'll fail miserably in the real world.",
    "2222071": "One of the challenges here is that under extrusion might be present only in a fraction of the entire `print_id` image set. For example, if only 20% of the images for a specific `print_id` show signs of under extrusion, and the labelling process is such that all labels for this `print_id` are labeled as positive for having under extrusion, we may end up having a lot of mislabeled images. \n\nIn my opinion, the winner should be selected on the basis of how many `print_id`s were correctly detected as having under extrusion - they should not be penalized for corrently labeling an image as \"not having under extrusion\" if the image does not show signs of under extrusion",
    "2222209": "We have only a handful of `print_id`s in the test data set. Hence t doesn't make sense to base the result on `print_id`s.\n\nWhat you said is more about the noise in the labels. You can check this post for more details: https://www.kaggle.com/competitions/early-detection-of-3d-printing-issues/discussion/398725"
  },
  "source": "meta"
}