{
  "id": 549744,
  "title": "A note on an image transformation operation that may or may not improve model performance",
  "url": "/competitions/czii-cryo-et-object-identification/discussion/549744",
  "author_name": "",
  "post_date": "2024-12-03T18:12:09.062025800Z",
  "votes": 26,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi Everyone!</p>\n<p>We wanted to bring to your attention a common cryoET technicality that may or may not make a difference to model performance. As the participants have already seen, the cryoET 3D volumes or tomograms are reconstructions of biological molecules starting with a series of tilted 2D projection images (tilt series). Since biological molecules have inherent handedness (chirality), the 3D tomograms can potentially contain information about that handedness. However, there are no existing tools that can take in tomograms as input and determine whether they have the correct biological handedness or the incorrect mirrored handedness.  A change in handedness can be introduced due to any number of steps in the processing (e.g. flipping the z axis) including further downstream processing, and subtomogram averaging as well as modification of volumes for data augmentation in model training. This is true for our phantom data and also for all the other data on the CryoET Data Portal and other public repositories.  As a result of this uncertainty, it might be worthwhile to test your models on the data as provided as well as a mirrored version of that data. We are doubtful that the handedness will make much difference to the results but we cannot be certain. And if it does it will be essential, for now, to have a model that can cope with either handedness as this is not yet well determined for most cryoET data.</p>\n<p>As a suggestion for how to find out if your model is affected by handedness, one option is to consider the following: (i) the simulated data has correct handedness, (ii) the phantom real data may have a mirrored handedness, (iii) therefore, mixing the two might affect the model performance. So if you mirror the simulated tomograms in Z to flip their handedness, and combine that with the phantom data, do you get a better leaderboard score? The mirroring can also be done more efficiently by mirroring the subtomograms, i.e. the cropped-out particles. </p>\n<p>Just as a footnote, our field has recently convened a working group to define standards for cryoET data and metadata so that we can nail down handedness (and other important geometries) going forward. We will publish the recommendations as a “white paper” early next year but it will of course take time to establish these standards, validate them, and then change the handedness of mirrored data in public data repositories so that all data has the correct biological handedness.  </p>\n<p><strong>Reference</strong><br>\n<a href=\"https://pmc.ncbi.nlm.nih.gov/articles/PMC3765063/\" target=\"_blank\">https://pmc.ncbi.nlm.nih.gov/articles/PMC3765063/</a></p>",
  "messages": [
    {
      "id": "3062621",
      "postDate": "12/03/2024 18:12:09",
      "content": "<p>Hi Everyone!</p>\n<p>We wanted to bring to your attention a common cryoET technicality that may or may not make a difference to model performance. As the participants have already seen, the cryoET 3D volumes or tomograms are reconstructions of biological molecules starting with a series of tilted 2D projection images (tilt series). Since biological molecules have inherent handedness (chirality), the 3D tomograms can potentially contain information about that handedness. However, there are no existing tools that can take in tomograms as input and determine whether they have the correct biological handedness or the incorrect mirrored handedness.  A change in handedness can be introduced due to any number of steps in the processing (e.g. flipping the z axis) including further downstream processing, and subtomogram averaging as well as modification of volumes for data augmentation in model training. This is true for our phantom data and also for all the other data on the CryoET Data Portal and other public repositories.  As a result of this uncertainty, it might be worthwhile to test your models on the data as provided as well as a mirrored version of that data. We are doubtful that the handedness will make much difference to the results but we cannot be certain. And if it does it will be essential, for now, to have a model that can cope with either handedness as this is not yet well determined for most cryoET data.</p>\n<p>As a suggestion for how to find out if your model is affected by handedness, one option is to consider the following: (i) the simulated data has correct handedness, (ii) the phantom real data may have a mirrored handedness, (iii) therefore, mixing the two might affect the model performance. So if you mirror the simulated tomograms in Z to flip their handedness, and combine that with the phantom data, do you get a better leaderboard score? The mirroring can also be done more efficiently by mirroring the subtomograms, i.e. the cropped-out particles. </p>\n<p>Just as a footnote, our field has recently convened a working group to define standards for cryoET data and metadata so that we can nail down handedness (and other important geometries) going forward. We will publish the recommendations as a “white paper” early next year but it will of course take time to establish these standards, validate them, and then change the handedness of mirrored data in public data repositories so that all data has the correct biological handedness.  </p>\n<p><strong>Reference</strong><br>\n<a href=\"https://pmc.ncbi.nlm.nih.gov/articles/PMC3765063/\" target=\"_blank\">https://pmc.ncbi.nlm.nih.gov/articles/PMC3765063/</a></p>",
      "rawMarkdown": "Hi Everyone!\n\nWe wanted to bring to your attention a common cryoET technicality that may or may not make a difference to model performance. As the participants have already seen, the cryoET 3D volumes or tomograms are reconstructions of biological molecules starting with a series of tilted 2D projection images (tilt series). Since biological molecules have inherent handedness (chirality), the 3D tomograms can potentially contain information about that handedness. However, there are no existing tools that can take in tomograms as input and determine whether they have the correct biological handedness or the incorrect mirrored handedness.  A change in handedness can be introduced due to any number of steps in the processing (e.g. flipping the z axis) including further downstream processing, and subtomogram averaging as well as modification of volumes for data augmentation in model training. This is true for our phantom data and also for all the other data on the CryoET Data Portal and other public repositories.  As a result of this uncertainty, it might be worthwhile to test your models on the data as provided as well as a mirrored version of that data. We are doubtful that the handedness will make much difference to the results but we cannot be certain. And if it does it will be essential, for now, to have a model that can cope with either handedness as this is not yet well determined for most cryoET data.\n\nAs a suggestion for how to find out if your model is affected by handedness, one option is to consider the following: (i) the simulated data has correct handedness, (ii) the phantom real data may have a mirrored handedness, (iii) therefore, mixing the two might affect the model performance. So if you mirror the simulated tomograms in Z to flip their handedness, and combine that with the phantom data, do you get a better leaderboard score? The mirroring can also be done more efficiently by mirroring the subtomograms, i.e. the cropped-out particles. \n\nJust as a footnote, our field has recently convened a working group to define standards for cryoET data and metadata so that we can nail down handedness (and other important geometries) going forward. We will publish the recommendations as a “white paper” early next year but it will of course take time to establish these standards, validate them, and then change the handedness of mirrored data in public data repositories so that all data has the correct biological handedness.  \n\n**Reference**\nhttps://pmc.ncbi.nlm.nih.gov/articles/PMC3765063/",
      "votes": null
    },
    {
      "id": "3062660",
      "postDate": "12/03/2024 19:30:56",
      "content": "<p>Interesting. I tested flipping the axes as TTA during inference and this has a large impact on <code>beta-galactosidase</code>.</p>\n<table>\n<thead>\n<tr>\n<th>Class</th>\n<th>TTA</th>\n<th>no TTA</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>apo-ferritin</td>\n<td><strong>0.8413</strong></td>\n<td>0.7297</td>\n</tr>\n<tr>\n<td>beta-galactosidase</td>\n<td>0.6445</td>\n<td><strong>0.7589</strong></td>\n</tr>\n<tr>\n<td>ribosome</td>\n<td><strong>0.7349</strong></td>\n<td>0.7294</td>\n</tr>\n<tr>\n<td>thyroglobulin</td>\n<td><strong>0.6500</strong></td>\n<td>0.5904</td>\n</tr>\n<tr>\n<td>virus-like-particle</td>\n<td><strong>0.9894</strong></td>\n<td>0.9790</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "Interesting. I tested flipping the axes as TTA during inference and this has a large impact on `beta-galactosidase`.\n\n| Class         |   TTA        | no TTA          |\n|----------------|----------------|----------------|\n| apo-ferritin     | **0.8413** | 0.7297 |\n| beta-galactosidase     | 0.6445 | **0.7589** |\n| ribosome     | **0.7349** | 0.7294 |\n| thyroglobulin      | **0.6500**  | 0.5904 |\n| virus-like-particle     | **0.9894** | 0.9790 |",
      "votes": null
    },
    {
      "id": "3062668",
      "postDate": "12/03/2024 19:48:35",
      "content": "<p>i think you need to break down the score to recall and precision,<br>\nwe should expect an increase in recall.</p>\n<hr>\n<p>the imporvement may come from the fact that:</p>\n<ul>\n<li>beta-galactosidase is the most difficult and has many fp initally.</li>\n<li>hence any augmentation (including z-flip, rotate, etc) will cause decrease of fp and hence improve  fbeta score.</li>\n</ul>",
      "rawMarkdown": "i think you need to break down the score to recall and precision,\nwe should expect an increase in recall.\n\n---\n\nthe imporvement may come from the fact that:\n- beta-galactosidase is the most difficult and has many fp initally.\n- hence any augmentation (including z-flip, rotate, etc) will cause decrease of fp and hence improve  fbeta score.",
      "votes": null
    },
    {
      "id": "3062672",
      "postDate": "12/03/2024 19:53:39",
      "content": "<p>Good point <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a>, I will look into the recall + precision.</p>\n<p>In the meantime, here are predictions for each TTA combination. I think this provides more evidence that the effect is due to \"handedness\" rather than \"difficulty\", but will keep looking.</p>\n<table>\n<thead>\n<tr>\n<th>z_flip</th>\n<th>y_flip</th>\n<th>x_flip</th>\n<th>Value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>False</td>\n<td>False</td>\n<td>False</td>\n<td>0.7589</td>\n</tr>\n<tr>\n<td>True</td>\n<td>True</td>\n<td>True</td>\n<td>0.7623</td>\n</tr>\n<tr>\n<td>True</td>\n<td>False</td>\n<td>False</td>\n<td>0.6740</td>\n</tr>\n<tr>\n<td>True</td>\n<td>False</td>\n<td>True</td>\n<td>0.6769</td>\n</tr>\n<tr>\n<td>True</td>\n<td>True</td>\n<td>False</td>\n<td>0.6830</td>\n</tr>\n<tr>\n<td>False</td>\n<td>False</td>\n<td>True</td>\n<td>0.6860</td>\n</tr>\n<tr>\n<td>False</td>\n<td>True</td>\n<td>True</td>\n<td>0.6769</td>\n</tr>\n<tr>\n<td>False</td>\n<td>True</td>\n<td>False</td>\n<td>0.6751</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "Good point @hengck23, I will look into the recall + precision.\n\nIn the meantime, here are predictions for each TTA combination. I think this provides more evidence that the effect is due to \"handedness\" rather than \"difficulty\", but will keep looking.\n\n| z_flip     | y_flip     | x_flip     | Value   |\n|-------|-------|-------|---------|\n| False | False | False | 0.7589  |\n| True  | True  | True  | 0.7623  |\n| True  | False | False | 0.6740  |\n| True  | False | True  | 0.6769  |\n| True  | True  | False | 0.6830  |\n| False | False | True  | 0.6860  |\n| False | True  | True  | 0.6769  |\n| False | True  | False | 0.6751  |",
      "votes": null
    },
    {
      "id": "3074660",
      "postDate": "12/17/2024 22:48:53",
      "content": "<p>Interesting…  I would have thought two flips (on any axis) cancel each other out…but maybe that's only for simpler molecules.</p>",
      "rawMarkdown": "Interesting...  I would have thought two flips (on any axis) cancel each other out...but maybe that's only for simpler molecules.",
      "votes": null
    },
    {
      "id": "3084490",
      "postDate": "12/30/2024 22:55:19",
      "content": "<p>Looks like very final decision is about 2D image. A single flip in xy bad, 2 if both in xy then cancel. That would explain TTT, but no FTT. Or yes, FTT could not be as good because is only a 180 rotation in xy, a smaller change than TTT. I'm not sure what to think, may be I'll give another chance to flips.</p>\n<p>EDIT: <a href=\"https://www.kaggle.com/brendanartley\" target=\"_blank\">@brendanartley</a> Thanks for share.</p>",
      "rawMarkdown": "Looks like very final decision is about 2D image. A single flip in xy bad, 2 if both in xy then cancel. That would explain TTT, but no FTT. Or yes, FTT could not be as good because is only a 180 rotation in xy, a smaller change than TTT. I'm not sure what to think, may be I'll give another chance to flips.\n\nEDIT: @brendanartley Thanks for share.",
      "votes": null
    },
    {
      "id": "3103186",
      "postDate": "01/23/2025 06:33:18",
      "content": "<p>Yeah, FTT doesn't make sense.  I would think a 180 degree rotation would be helpful.</p>",
      "rawMarkdown": "Yeah, FTT doesn't make sense.  I would think a 180 degree rotation would be helpful.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3062660,
      "author_name": "brendanartley",
      "author_url": "",
      "post_date": "12/03/2024 19:30:56",
      "content": "<p>Interesting. I tested flipping the axes as TTA during inference and this has a large impact on <code>beta-galactosidase</code>.</p>\n<table>\n<thead>\n<tr>\n<th>Class</th>\n<th>TTA</th>\n<th>no TTA</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>apo-ferritin</td>\n<td><strong>0.8413</strong></td>\n<td>0.7297</td>\n</tr>\n<tr>\n<td>beta-galactosidase</td>\n<td>0.6445</td>\n<td><strong>0.7589</strong></td>\n</tr>\n<tr>\n<td>ribosome</td>\n<td><strong>0.7349</strong></td>\n<td>0.7294</td>\n</tr>\n<tr>\n<td>thyroglobulin</td>\n<td><strong>0.6500</strong></td>\n<td>0.5904</td>\n</tr>\n<tr>\n<td>virus-like-particle</td>\n<td><strong>0.9894</strong></td>\n<td>0.9790</td>\n</tr>\n</tbody>\n</table>",
      "votes": null,
      "replies": [
        {
          "id": 3062668,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "12/03/2024 19:48:35",
          "content": "<p>i think you need to break down the score to recall and precision,<br>\nwe should expect an increase in recall.</p>\n<hr>\n<p>the imporvement may come from the fact that:</p>\n<ul>\n<li>beta-galactosidase is the most difficult and has many fp initally.</li>\n<li>hence any augmentation (including z-flip, rotate, etc) will cause decrease of fp and hence improve  fbeta score.</li>\n</ul>",
          "votes": null,
          "replies": [
            {
              "id": 3062672,
              "author_name": "brendanartley",
              "author_url": "",
              "post_date": "12/03/2024 19:53:39",
              "content": "<p>Good point <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a>, I will look into the recall + precision.</p>\n<p>In the meantime, here are predictions for each TTA combination. I think this provides more evidence that the effect is due to \"handedness\" rather than \"difficulty\", but will keep looking.</p>\n<table>\n<thead>\n<tr>\n<th>z_flip</th>\n<th>y_flip</th>\n<th>x_flip</th>\n<th>Value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>False</td>\n<td>False</td>\n<td>False</td>\n<td>0.7589</td>\n</tr>\n<tr>\n<td>True</td>\n<td>True</td>\n<td>True</td>\n<td>0.7623</td>\n</tr>\n<tr>\n<td>True</td>\n<td>False</td>\n<td>False</td>\n<td>0.6740</td>\n</tr>\n<tr>\n<td>True</td>\n<td>False</td>\n<td>True</td>\n<td>0.6769</td>\n</tr>\n<tr>\n<td>True</td>\n<td>True</td>\n<td>False</td>\n<td>0.6830</td>\n</tr>\n<tr>\n<td>False</td>\n<td>False</td>\n<td>True</td>\n<td>0.6860</td>\n</tr>\n<tr>\n<td>False</td>\n<td>True</td>\n<td>True</td>\n<td>0.6769</td>\n</tr>\n<tr>\n<td>False</td>\n<td>True</td>\n<td>False</td>\n<td>0.6751</td>\n</tr>\n</tbody>\n</table>",
              "votes": null,
              "replies": [
                {
                  "id": 3074660,
                  "author_name": "davidlist",
                  "author_url": "",
                  "post_date": "12/17/2024 22:48:53",
                  "content": "<p>Interesting…  I would have thought two flips (on any axis) cancel each other out…but maybe that's only for simpler molecules.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3084490,
                      "author_name": "sacuscreed",
                      "author_url": "",
                      "post_date": "12/30/2024 22:55:19",
                      "content": "<p>Looks like very final decision is about 2D image. A single flip in xy bad, 2 if both in xy then cancel. That would explain TTT, but no FTT. Or yes, FTT could not be as good because is only a 180 rotation in xy, a smaller change than TTT. I'm not sure what to think, may be I'll give another chance to flips.</p>\n<p>EDIT: <a href=\"https://www.kaggle.com/brendanartley\" target=\"_blank\">@brendanartley</a> Thanks for share.</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 3103186,
                          "author_name": "davidlist",
                          "author_url": "",
                          "post_date": "01/23/2025 06:33:18",
                          "content": "<p>Yeah, FTT doesn't make sense.  I would think a 180 degree rotation would be helpful.</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3062621": "Hi Everyone!\n\nWe wanted to bring to your attention a common cryoET technicality that may or may not make a difference to model performance. As the participants have already seen, the cryoET 3D volumes or tomograms are reconstructions of biological molecules starting with a series of tilted 2D projection images (tilt series). Since biological molecules have inherent handedness (chirality), the 3D tomograms can potentially contain information about that handedness. However, there are no existing tools that can take in tomograms as input and determine whether they have the correct biological handedness or the incorrect mirrored handedness.  A change in handedness can be introduced due to any number of steps in the processing (e.g. flipping the z axis) including further downstream processing, and subtomogram averaging as well as modification of volumes for data augmentation in model training. This is true for our phantom data and also for all the other data on the CryoET Data Portal and other public repositories.  As a result of this uncertainty, it might be worthwhile to test your models on the data as provided as well as a mirrored version of that data. We are doubtful that the handedness will make much difference to the results but we cannot be certain. And if it does it will be essential, for now, to have a model that can cope with either handedness as this is not yet well determined for most cryoET data.\n\nAs a suggestion for how to find out if your model is affected by handedness, one option is to consider the following: (i) the simulated data has correct handedness, (ii) the phantom real data may have a mirrored handedness, (iii) therefore, mixing the two might affect the model performance. So if you mirror the simulated tomograms in Z to flip their handedness, and combine that with the phantom data, do you get a better leaderboard score? The mirroring can also be done more efficiently by mirroring the subtomograms, i.e. the cropped-out particles. \n\nJust as a footnote, our field has recently convened a working group to define standards for cryoET data and metadata so that we can nail down handedness (and other important geometries) going forward. We will publish the recommendations as a “white paper” early next year but it will of course take time to establish these standards, validate them, and then change the handedness of mirrored data in public data repositories so that all data has the correct biological handedness.  \n\n**Reference**\nhttps://pmc.ncbi.nlm.nih.gov/articles/PMC3765063/",
    "3062660": "Interesting. I tested flipping the axes as TTA during inference and this has a large impact on `beta-galactosidase`.\n\n| Class         |   TTA        | no TTA          |\n|----------------|----------------|----------------|\n| apo-ferritin     | **0.8413** | 0.7297 |\n| beta-galactosidase     | 0.6445 | **0.7589** |\n| ribosome     | **0.7349** | 0.7294 |\n| thyroglobulin      | **0.6500**  | 0.5904 |\n| virus-like-particle     | **0.9894** | 0.9790 |",
    "3062668": "i think you need to break down the score to recall and precision,\nwe should expect an increase in recall.\n\n---\n\nthe imporvement may come from the fact that:\n- beta-galactosidase is the most difficult and has many fp initally.\n- hence any augmentation (including z-flip, rotate, etc) will cause decrease of fp and hence improve  fbeta score.",
    "3062672": "Good point @hengck23, I will look into the recall + precision.\n\nIn the meantime, here are predictions for each TTA combination. I think this provides more evidence that the effect is due to \"handedness\" rather than \"difficulty\", but will keep looking.\n\n| z_flip     | y_flip     | x_flip     | Value   |\n|-------|-------|-------|---------|\n| False | False | False | 0.7589  |\n| True  | True  | True  | 0.7623  |\n| True  | False | False | 0.6740  |\n| True  | False | True  | 0.6769  |\n| True  | True  | False | 0.6830  |\n| False | False | True  | 0.6860  |\n| False | True  | True  | 0.6769  |\n| False | True  | False | 0.6751  |",
    "3074660": "Interesting...  I would have thought two flips (on any axis) cancel each other out...but maybe that's only for simpler molecules.",
    "3084490": "Looks like very final decision is about 2D image. A single flip in xy bad, 2 if both in xy then cancel. That would explain TTT, but no FTT. Or yes, FTT could not be as good because is only a 180 rotation in xy, a smaller change than TTT. I'm not sure what to think, may be I'll give another chance to flips.\n\nEDIT: @brendanartley Thanks for share.",
    "3103186": "Yeah, FTT doesn't make sense.  I would think a 180 degree rotation would be helpful."
  },
  "source": "meta"
}