In the data section it's specified that different particles have different levels of difficulty. In short, it's easy to predict AF, R, VLP, hard to predict BG, T, and completely impossible to predict BA.
This correlates quite well with how my models behave. So I would like to clarify: what made you think some particles are easy to predict, and some are hard, in the first place?
\nUsual reasons could include:
\nOr maybe it's something completely different.
", "messages": [ { "id": "3048791", "postDate": "11/18/2024 11:09:40", "content": "In the data section it's specified that different particles have different levels of difficulty. In short, it's easy to predict AF, R, VLP, hard to predict BG, T, and completely impossible to predict BA.
This correlates quite well with how my models behave. So I would like to clarify: what made you think some particles are easy to predict, and some are hard, in the first place?
\nUsual reasons could include:
\nOr maybe it's something completely different.
", "rawMarkdown": "In the data section it's specified that different particles have different levels of difficulty. In short, it's easy to predict `AF, R, VLP`, hard to predict `BG, T`, and completely impossible to predict `BA`. \n\nThis correlates quite well with how my models behave. So I would like to clarify: what made you think some particles are easy to predict, and some are hard, in the first place?\n\nUsual reasons could include:\n- Easy particles are more frequent that others (so more training samples)\n- Easy particles are bigger (so it's harder to miss)\n- Easy particles have simpler shape (for example, circle-shaped instead of some monstrosity for hard ones) \n- Easy particles require less temporal (z-axis) context \n\nOr maybe it's something completely different.", "votes": null }, { "id": "3048830", "postDate": "11/18/2024 11:57:38", "content": "i would like to think otherwise, lets consider the extreme:
\n1) For each partice, test = train appereance.
\nIn this case, whether big or small; densely or sparsely pouplated etc … if you have 100% accuracy in train (model is discrminative enough), your validation/test results is also 100%
2) For each particle, test != train (completely OOD, i.e.completely different)
\nobviously validation/test results accuracy is unpredictable (likely to be close to 0%), regaradless of what you have for train
so the difficulty of particles really depends on:
\n1) LABEL NOISE : whether they the particles distinctive features to differentiate themselves from each other(and background)
\n2) DOMAIN: how close the train and validation/test distribution
for (1), asking a human to label a set of samples is a good estimation of this. You can try to label some particles on crop out samples (or directly use the images) and see the performance of you, yourself. Since from the dataset paper, we already know that the ground truth annotation is produced by hybrid method (human + machine labelled), i would say the problem is not difficult if we consider \"distinctive features\" point of view; especially when given that we can have a model of infinite parameters (train on approriate/infinte number of train samples).
\nThen the diffiulty of this competition must be mainly from (2), which is usually the case of machine learning problem
", "rawMarkdown": "i would like to think otherwise, lets consider the extreme:\n\n1) For each partice, test = train appereance.\nIn this case, whether big or small; densely or sparsely pouplated etc ... if you have 100% accuracy in train (model is discrminative enough), your validation/test results is also 100%\n\n2) For each particle, test != train (completely OOD, i.e.completely different)\nobviously validation/test results accuracy is unpredictable (likely to be close to 0%), regaradless of what you have for train\n\n---\n\nso the difficulty of particles really depends on:\n1) LABEL NOISE : whether they the particles distinctive features to differentiate themselves from each other(and background)\n2) DOMAIN: how close the train and validation/test distribution\n\n\nfor (1), asking a human to label a set of samples is a good estimation of this. You can try to label some particles on crop out samples (or directly use the images) and see the performance of you, yourself. Since from the dataset paper, we already know that the ground truth annotation is produced by hybrid method (human + machine labelled), i would say the problem is not difficult if we consider \"distinctive features\" point of view; especially when given that we can have a model of infinite parameters (train on approriate/infinte number of train samples).\n\nThen the diffiulty of this competition must be mainly from (2), which is usually the case of machine learning problem", "votes": null }, { "id": "3048838", "postDate": "11/18/2024 12:05:15", "content": "Btw, since you clearly created some models for this competition, how well CV and LB correlates, in your opinion?
", "rawMarkdown": "Btw, since you clearly created some models for this competition, how well CV and LB correlates, in your opinion?", "votes": null }, { "id": "3048999", "postDate": "11/18/2024 15:36:49", "content": "I think you both have completely valid arguments in terms of what's easy or not. It is not a straightforward answer. When we started annotating the ground truth, which particles we got first defined what's less challenging. So VLP, Ribosome, ApoF were easier to do because the human eye can detect it easily, the challenge is in making sure we get most of them given the large size of the dataset. But now that we have generated labels for all of them including B-gal and THG, the definition of more challenging and less challenging can be more complicated in the context of the challenge. Which I think hengck23 described it well in that context.
", "rawMarkdown": "I think you both have completely valid arguments in terms of what's easy or not. It is not a straightforward answer. When we started annotating the ground truth, which particles we got first defined what's less challenging. So VLP, Ribosome, ApoF were easier to do because the human eye can detect it easily, the challenge is in making sure we get most of them given the large size of the dataset. But now that we have generated labels for all of them including B-gal and THG, the definition of more challenging and less challenging can be more complicated in the context of the challenge. Which I think hengck23 described it well in that context.", "votes": null }, { "id": "3049273", "postDate": "11/18/2024 22:52:23", "content": "it would be very much correlated for lb less or equal to 0.665. After that, things start to break (you probably need new observations apart from lb to judge the stability of your score)
", "rawMarkdown": "it would be very much correlated for lb less or equal to 0.665. After that, things start to break (you probably need new observations apart from lb to judge the stability of your score)", "votes": null } ], "comments": [ { "id": 3048830, "author_name": "hengck23", "author_url": "", "post_date": "11/18/2024 11:57:38", "content": "i would like to think otherwise, lets consider the extreme:
\n1) For each partice, test = train appereance.
\nIn this case, whether big or small; densely or sparsely pouplated etc … if you have 100% accuracy in train (model is discrminative enough), your validation/test results is also 100%
2) For each particle, test != train (completely OOD, i.e.completely different)
\nobviously validation/test results accuracy is unpredictable (likely to be close to 0%), regaradless of what you have for train
so the difficulty of particles really depends on:
\n1) LABEL NOISE : whether they the particles distinctive features to differentiate themselves from each other(and background)
\n2) DOMAIN: how close the train and validation/test distribution
for (1), asking a human to label a set of samples is a good estimation of this. You can try to label some particles on crop out samples (or directly use the images) and see the performance of you, yourself. Since from the dataset paper, we already know that the ground truth annotation is produced by hybrid method (human + machine labelled), i would say the problem is not difficult if we consider \"distinctive features\" point of view; especially when given that we can have a model of infinite parameters (train on approriate/infinte number of train samples).
\nThen the diffiulty of this competition must be mainly from (2), which is usually the case of machine learning problem
", "votes": null, "replies": [ { "id": 3048838, "author_name": "ivanpan", "author_url": "", "post_date": "11/18/2024 12:05:15", "content": "Btw, since you clearly created some models for this competition, how well CV and LB correlates, in your opinion?
", "votes": null, "replies": [ { "id": 3049273, "author_name": "hengck23", "author_url": "", "post_date": "11/18/2024 22:52:23", "content": "it would be very much correlated for lb less or equal to 0.665. After that, things start to break (you probably need new observations apart from lb to judge the stability of your score)
", "votes": null, "replies": [] } ] } ] }, { "id": 3048999, "author_name": "rezaparaan", "author_url": "", "post_date": "11/18/2024 15:36:49", "content": "I think you both have completely valid arguments in terms of what's easy or not. It is not a straightforward answer. When we started annotating the ground truth, which particles we got first defined what's less challenging. So VLP, Ribosome, ApoF were easier to do because the human eye can detect it easily, the challenge is in making sure we get most of them given the large size of the dataset. But now that we have generated labels for all of them including B-gal and THG, the definition of more challenging and less challenging can be more complicated in the context of the challenge. Which I think hengck23 described it well in that context.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3048791": "In the data section it's specified that different particles have different levels of difficulty. In short, it's easy to predict `AF, R, VLP`, hard to predict `BG, T`, and completely impossible to predict `BA`. \n\nThis correlates quite well with how my models behave. So I would like to clarify: what made you think some particles are easy to predict, and some are hard, in the first place?\n\nUsual reasons could include:\n- Easy particles are more frequent that others (so more training samples)\n- Easy particles are bigger (so it's harder to miss)\n- Easy particles have simpler shape (for example, circle-shaped instead of some monstrosity for hard ones) \n- Easy particles require less temporal (z-axis) context \n\nOr maybe it's something completely different.", "3048830": "i would like to think otherwise, lets consider the extreme:\n\n1) For each partice, test = train appereance.\nIn this case, whether big or small; densely or sparsely pouplated etc ... if you have 100% accuracy in train (model is discrminative enough), your validation/test results is also 100%\n\n2) For each particle, test != train (completely OOD, i.e.completely different)\nobviously validation/test results accuracy is unpredictable (likely to be close to 0%), regaradless of what you have for train\n\n---\n\nso the difficulty of particles really depends on:\n1) LABEL NOISE : whether they the particles distinctive features to differentiate themselves from each other(and background)\n2) DOMAIN: how close the train and validation/test distribution\n\n\nfor (1), asking a human to label a set of samples is a good estimation of this. You can try to label some particles on crop out samples (or directly use the images) and see the performance of you, yourself. Since from the dataset paper, we already know that the ground truth annotation is produced by hybrid method (human + machine labelled), i would say the problem is not difficult if we consider \"distinctive features\" point of view; especially when given that we can have a model of infinite parameters (train on approriate/infinte number of train samples).\n\nThen the diffiulty of this competition must be mainly from (2), which is usually the case of machine learning problem", "3048838": "Btw, since you clearly created some models for this competition, how well CV and LB correlates, in your opinion?", "3048999": "I think you both have completely valid arguments in terms of what's easy or not. It is not a straightforward answer. When we started annotating the ground truth, which particles we got first defined what's less challenging. So VLP, Ribosome, ApoF were easier to do because the human eye can detect it easily, the challenge is in making sure we get most of them given the large size of the dataset. But now that we have generated labels for all of them including B-gal and THG, the definition of more challenging and less challenging can be more complicated in the context of the challenge. Which I think hengck23 described it well in that context.", "3049273": "it would be very much correlated for lb less or equal to 0.665. After that, things start to break (you probably need new observations apart from lb to judge the stability of your score)" }, "source": "meta" }