{ "id": 550481, "title": "How/what to load from 10441 data", "url": "/competitions/czii-cryo-et-object-identification/discussion/550481", "author_name": "", "post_date": "2024-12-07T21:13:11.873631700Z", "votes": 3, "comment_count": 12, "views": 0, "content": "

Since the content of the data is different from the data that we have in the competition, I don't know what could be helpfu, is it loading zar from Reconstructions folder, or Tilt Series, and please what is the difference between them

", "messages": [ { "id": "3066233", "postDate": "12/07/2024 21:13:11", "content": "

Since the content of the data is different from the data that we have in the competition, I don't know what could be helpfu, is it loading zar from Reconstructions folder, or Tilt Series, and please what is the difference between them

", "rawMarkdown": "Since the content of the data is different from the data that we have in the competition, I don't know what could be helpfu, is it loading zar from Reconstructions folder, or Tilt Series, and please what is the difference between them", "votes": null }, { "id": "3066310", "postDate": "12/08/2024 01:36:52", "content": "

This notebook should answer some of your questions. However, I should probably mention that I tried training against the simulated data and so far have obtained worse results. Very possible I'm doing something wrong though.

\n

https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization

", "rawMarkdown": "This notebook should answer some of your questions. However, I should probably mention that I tried training against the simulated data and so far have obtained worse results. Very possible I'm doing something wrong though.\n\n[https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization](https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization)", "votes": null }, { "id": "3066321", "postDate": "12/08/2024 02:31:03", "content": "

@davidlist note: https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549744

", "rawMarkdown": "davidlist note: https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549744", "votes": null }, { "id": "3066375", "postDate": "12/08/2024 05:19:45", "content": "

Thanks @kharrington! I had seen that earlier, but had figured it would be too small to worry about, but some of the comments suggest it's something to look at.

", "rawMarkdown": "Thanks @kharrington! I had seen that earlier, but had figured it would be too small to worry about, but some of the comments suggest it's something to look at.", "votes": null }, { "id": "3066517", "postDate": "12/08/2024 08:46:01", "content": "

It's my impression that the simulated data is more 'raw', and hasn't had the additional steps like denoising applied to it. Not sure if we can easily reproduce those steps ourselves.

", "rawMarkdown": "It's my impression that the simulated data is more 'raw', and hasn't had the additional steps like denoising applied to it. Not sure if we can easily reproduce those steps ourselves.", "votes": null }, { "id": "3066520", "postDate": "12/08/2024 08:50:10", "content": "

Yeah, my first guess would be they're too noisy. You almost can't see any detail: https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization.

", "rawMarkdown": "Yeah, my first guess would be they're too noisy. You almost can't see any detail: [https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization](https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization).", "votes": null }, { "id": "3066808", "postDate": "12/08/2024 13:52:16", "content": "

thank you I saw the visualization and you're right, the is no details, but i will give it shot, how about trying denoise it using the same techniques used in the existing data

", "rawMarkdown": "thank you I saw the visualization and you're right, the is no details, but i will give it shot, how about trying denoise it using the same techniques used in the existing data", "votes": null }, { "id": "3066850", "postDate": "12/08/2024 14:28:25", "content": "

Is the code for that shared anywhere?

", "rawMarkdown": "Is the code for that shared anywhere?", "votes": null }, { "id": "3066868", "postDate": "12/08/2024 14:49:04", "content": "

This is my reference https://www.biorxiv.org/content/10.1101/2024.11.04.621686v1.full#T2

", "rawMarkdown": "This is my reference https://www.biorxiv.org/content/10.1101/2024.11.04.621686v1.full#T2", "votes": null }, { "id": "3067048", "postDate": "12/08/2024 19:46:00", "content": "

I had not released my notebook for generating labels, because it didn't help, but I'll go ahead and do that now. Would be great (though obviously not required) if either of you could share anything you learn about denoising.

", "rawMarkdown": "I had not released my notebook for generating labels, because it didn't help, but I'll go ahead and do that now. Would be great (though obviously not required) if either of you could share anything you learn about denoising.", "votes": null }, { "id": "3067060", "postDate": "12/08/2024 20:05:06", "content": "

Notebook is here:
\nhttps://www.kaggle.com/code/davidlist/simulated-data-and-labels

", "rawMarkdown": "Notebook is here: \n[https://www.kaggle.com/code/davidlist/simulated-data-and-labels](https://www.kaggle.com/code/davidlist/simulated-data-and-labels)", "votes": null }, { "id": "3067125", "postDate": "12/08/2024 23:33:21", "content": "

@davidlist You'll need to visualize them with a different contrast range. If you look at the synthetic tomograms using neuroglancer, then you can see detail fairly well [link here].

", "rawMarkdown": "davidlist You'll need to visualize them with a different contrast range. If you look at the synthetic tomograms using neuroglancer, then you can see detail fairly well [[link here](https://neuroglancer-demo.appspot.com/#!%7B%22dimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22position%22:%5B317.1737060546875%2C199.67996215820312%2C100%5D%2C%22crossSectionScale%22:0.4995132310947581%2C%22projectionOrientation%22:%5B0.3826834261417389%2C0%2C0%2C0.9238795042037964%5D%2C%22projectionScale%22:693%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22zarr://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/Tomograms/100/TS_11.zarr%22%2C%22tab%22:%22rendering%22%2C%22opacity%22:0.51%2C%22shader%22:%22#uicontrol%20invlerp%20contrast%5Cn#uicontrol%20bool%20invert_contrast%20checkbox%5Cn%5Cnfloat%20get_contrast%28%29%20%7B%5Cn%20%20return%20invert_contrast%20?%201.0%20-%20contrast%28%29%20:%20contrast%28%29%3B%5Cn%7D%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20float%20outputValue%3B%5Cn%20%20outputValue%20=%20get_contrast%28%29%3B%5Cn%20%20emitGrayscale%28outputValue%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22contrast%22:%7B%22range%22:%5B-2.472108374349773%2C2.4591089384630322%5D%2C%22window%22:%5B-2.9652301056310533%2C2.9522306697443126%5D%7D%7D%2C%22name%22:%22TS_11%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/100-membrane-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#0f714d%22%2C%22name%22:%22100%20membrane%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/101-ferritin_complex-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#ff00ff%22%2C%22name%22:%22101%20ferritin%20complex%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/101-ferritin_complex-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#00ffff%22%7D%2C%22name%22:%22101%20ferritin%20complex%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/102-beta_amylase-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#ffff00%22%2C%22name%22:%22102%20Beta-amylase%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/102-beta_amylase-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#0000ff%22%7D%2C%22name%22:%22102%20Beta-amylase%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/103-beta_galactosidase-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#d80402%22%2C%22name%22:%22103%20Beta-galactosidase%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/103-beta_galactosidase-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#eebef1%22%7D%2C%22name%22:%22103%20Beta-galactosidase%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/104-cytosolic_ribosome-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#00ff00%22%2C%22name%22:%22104%20cytosolic%20ribosome%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/104-cytosolic_ribosome-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#80ff80%22%7D%2C%22name%22:%22104%20cytosolic%20ribosome%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/105-thyroglobulin-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#5e78e5%22%2C%22name%22:%22105%20Thyroglobulin%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/105-thyroglobulin-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#c88449%22%7D%2C%22name%22:%22105%20Thyroglobulin%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/106-pp7_vlp-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#6a178b%22%2C%22name%22:%22106%20virus-like%20capsid%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/106-pp7_vlp-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#f75dc4%22%7D%2C%22name%22:%22106%20virus-like%20capsid%20point%22%7D%5D%2C%22selectedLayer%22:%7B%22visible%22:true%2C%22layer%22:%22TS_11%22%7D%2C%22crossSectionBackgroundColor%22:%22#000000%22%2C%22layout%22:%224panel%22%7D)].", "votes": null }, { "id": "3067128", "postDate": "12/08/2024 23:46:49", "content": "

Yeah, when I set the range to +/-1 standard deviation, the structures become substantially more visible. I'm seeing the same behavior when I incorporate isonetcorrected data (i.e. that my score drops) so I suspect the problem is on my side.

", "rawMarkdown": "Yeah, when I set the range to +/-1 standard deviation, the structures become substantially more visible. I'm seeing the same behavior when I incorporate isonetcorrected data (i.e. that my score drops) so I suspect the problem is on my side.", "votes": null } ], "comments": [ { "id": 3066310, "author_name": "davidlist", "author_url": "", "post_date": "12/08/2024 01:36:52", "content": "

This notebook should answer some of your questions. However, I should probably mention that I tried training against the simulated data and so far have obtained worse results. Very possible I'm doing something wrong though.

\n

https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization

", "votes": null, "replies": [ { "id": 3066321, "author_name": "kharrington", "author_url": "", "post_date": "12/08/2024 02:31:03", "content": "

@davidlist note: https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549744

", "votes": null, "replies": [ { "id": 3066375, "author_name": "davidlist", "author_url": "", "post_date": "12/08/2024 05:19:45", "content": "

Thanks @kharrington! I had seen that earlier, but had figured it would be too small to worry about, but some of the comments suggest it's something to look at.

", "votes": null, "replies": [] } ] }, { "id": 3066517, "author_name": "jeroencottaar", "author_url": "", "post_date": "12/08/2024 08:46:01", "content": "

It's my impression that the simulated data is more 'raw', and hasn't had the additional steps like denoising applied to it. Not sure if we can easily reproduce those steps ourselves.

", "votes": null, "replies": [ { "id": 3066520, "author_name": "davidlist", "author_url": "", "post_date": "12/08/2024 08:50:10", "content": "

Yeah, my first guess would be they're too noisy. You almost can't see any detail: https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization.

", "votes": null, "replies": [ { "id": 3067125, "author_name": "kharrington", "author_url": "", "post_date": "12/08/2024 23:33:21", "content": "

@davidlist You'll need to visualize them with a different contrast range. If you look at the synthetic tomograms using neuroglancer, then you can see detail fairly well [link here].

", "votes": null, "replies": [ { "id": 3067128, "author_name": "davidlist", "author_url": "", "post_date": "12/08/2024 23:46:49", "content": "

Yeah, when I set the range to +/-1 standard deviation, the structures become substantially more visible. I'm seeing the same behavior when I incorporate isonetcorrected data (i.e. that my score drops) so I suspect the problem is on my side.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3066808, "author_name": "akrimi", "author_url": "", "post_date": "12/08/2024 13:52:16", "content": "

thank you I saw the visualization and you're right, the is no details, but i will give it shot, how about trying denoise it using the same techniques used in the existing data

", "votes": null, "replies": [ { "id": 3066850, "author_name": "jeroencottaar", "author_url": "", "post_date": "12/08/2024 14:28:25", "content": "

Is the code for that shared anywhere?

", "votes": null, "replies": [ { "id": 3066868, "author_name": "akrimi", "author_url": "", "post_date": "12/08/2024 14:49:04", "content": "

This is my reference https://www.biorxiv.org/content/10.1101/2024.11.04.621686v1.full#T2

", "votes": null, "replies": [] } ] }, { "id": 3067048, "author_name": "davidlist", "author_url": "", "post_date": "12/08/2024 19:46:00", "content": "

I had not released my notebook for generating labels, because it didn't help, but I'll go ahead and do that now. Would be great (though obviously not required) if either of you could share anything you learn about denoising.

", "votes": null, "replies": [] }, { "id": 3067060, "author_name": "davidlist", "author_url": "", "post_date": "12/08/2024 20:05:06", "content": "

Notebook is here:
\nhttps://www.kaggle.com/code/davidlist/simulated-data-and-labels

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3066233": "Since the content of the data is different from the data that we have in the competition, I don't know what could be helpfu, is it loading zar from Reconstructions folder, or Tilt Series, and please what is the difference between them", "3066310": "This notebook should answer some of your questions. However, I should probably mention that I tried training against the simulated data and so far have obtained worse results. Very possible I'm doing something wrong though.\n\n[https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization](https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization)", "3066321": "davidlist note: https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549744", "3066375": "Thanks @kharrington! I had seen that earlier, but had figured it would be too small to worry about, but some of the comments suggest it's something to look at.", "3066517": "It's my impression that the simulated data is more 'raw', and hasn't had the additional steps like denoising applied to it. Not sure if we can easily reproduce those steps ourselves.", "3066520": "Yeah, my first guess would be they're too noisy. You almost can't see any detail: [https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization](https://www.kaggle.com/code/davidlist/ts-0-simulated-data-visualization).", "3066808": "thank you I saw the visualization and you're right, the is no details, but i will give it shot, how about trying denoise it using the same techniques used in the existing data", "3066850": "Is the code for that shared anywhere?", "3066868": "This is my reference https://www.biorxiv.org/content/10.1101/2024.11.04.621686v1.full#T2", "3067048": "I had not released my notebook for generating labels, because it didn't help, but I'll go ahead and do that now. Would be great (though obviously not required) if either of you could share anything you learn about denoising.", "3067060": "Notebook is here: \n[https://www.kaggle.com/code/davidlist/simulated-data-and-labels](https://www.kaggle.com/code/davidlist/simulated-data-and-labels)", "3067125": "davidlist You'll need to visualize them with a different contrast range. If you look at the synthetic tomograms using neuroglancer, then you can see detail fairly well [[link here](https://neuroglancer-demo.appspot.com/#!%7B%22dimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22position%22:%5B317.1737060546875%2C199.67996215820312%2C100%5D%2C%22crossSectionScale%22:0.4995132310947581%2C%22projectionOrientation%22:%5B0.3826834261417389%2C0%2C0%2C0.9238795042037964%5D%2C%22projectionScale%22:693%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22zarr://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/Tomograms/100/TS_11.zarr%22%2C%22tab%22:%22rendering%22%2C%22opacity%22:0.51%2C%22shader%22:%22#uicontrol%20invlerp%20contrast%5Cn#uicontrol%20bool%20invert_contrast%20checkbox%5Cn%5Cnfloat%20get_contrast%28%29%20%7B%5Cn%20%20return%20invert_contrast%20?%201.0%20-%20contrast%28%29%20:%20contrast%28%29%3B%5Cn%7D%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20float%20outputValue%3B%5Cn%20%20outputValue%20=%20get_contrast%28%29%3B%5Cn%20%20emitGrayscale%28outputValue%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22contrast%22:%7B%22range%22:%5B-2.472108374349773%2C2.4591089384630322%5D%2C%22window%22:%5B-2.9652301056310533%2C2.9522306697443126%5D%7D%7D%2C%22name%22:%22TS_11%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/100-membrane-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#0f714d%22%2C%22name%22:%22100%20membrane%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/101-ferritin_complex-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#ff00ff%22%2C%22name%22:%22101%20ferritin%20complex%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/101-ferritin_complex-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#00ffff%22%7D%2C%22name%22:%22101%20ferritin%20complex%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/102-beta_amylase-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#ffff00%22%2C%22name%22:%22102%20Beta-amylase%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/102-beta_amylase-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#0000ff%22%7D%2C%22name%22:%22102%20Beta-amylase%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/103-beta_galactosidase-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#d80402%22%2C%22name%22:%22103%20Beta-galactosidase%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/103-beta_galactosidase-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#eebef1%22%7D%2C%22name%22:%22103%20Beta-galactosidase%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/104-cytosolic_ribosome-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#00ff00%22%2C%22name%22:%22104%20cytosolic%20ribosome%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/104-cytosolic_ribosome-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#80ff80%22%7D%2C%22name%22:%22104%20cytosolic%20ribosome%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/105-thyroglobulin-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#5e78e5%22%2C%22name%22:%22105%20Thyroglobulin%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/105-thyroglobulin-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#c88449%22%7D%2C%22name%22:%22105%20Thyroglobulin%20point%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/106-pp7_vlp-1.0_segmentationmask%22%2C%22transform%22:%7B%22outputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%2C%22inputDimensions%22:%7B%22x%22:%5B1e-9%2C%22m%22%5D%2C%22y%22:%5B1e-9%2C%22m%22%5D%2C%22z%22:%5B1e-9%2C%22m%22%5D%7D%7D%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22hoverHighlight%22:false%2C%22segments%22:%5B%221%22%5D%2C%22segmentDefaultColor%22:%22#6a178b%22%2C%22name%22:%22106%20virus-like%20capsid%20segmentation%22%2C%22visible%22:false%7D%2C%7B%22type%22:%22annotation%22%2C%22source%22:%22precomputed://https://files.cryoetdataportal.cziscience.com/10441/TS_11/Reconstructions/VoxelSpacing10.000/NeuroglancerPrecompute/106-pp7_vlp-1.0_orientedpoint%22%2C%22tab%22:%22rendering%22%2C%22shader%22:%22#uicontrol%20float%20pointScale%20slider%28min=0.01%2C%20max=2.0%2C%20step=0.01%29%5Cn#uicontrol%20float%20opacity%20slider%28min=0.0%2C%20max=1.0%2C%20step=0.01%29%5Cn#uicontrol%20vec3%20color%20color%5Cn%5Cnvoid%20main%28%29%20%7B%5Cn%20%20if%20%28opacity%20==%200.0%29%20discard%3B%5Cn%20%20setColor%28vec4%28color%2C%20opacity%29%29%3B%5Cn%20%20setPointMarkerSize%28pointScale%20%2A%20prop_diameter%28%29%29%3B%5Cn%20%20setPointMarkerBorderWidth%280.1%29%3B%5Cn%20%20setPointMarkerBorderColor%28vec4%280.0%2C%200.0%2C%200.0%2C%20opacity%29%29%3B%5Cn%7D%22%2C%22shaderControls%22:%7B%22pointScale%22:1%2C%22opacity%22:1%2C%22color%22:%22#f75dc4%22%7D%2C%22name%22:%22106%20virus-like%20capsid%20point%22%7D%5D%2C%22selectedLayer%22:%7B%22visible%22:true%2C%22layer%22:%22TS_11%22%7D%2C%22crossSectionBackgroundColor%22:%22#000000%22%2C%22layout%22:%224panel%22%7D)].", "3067128": "Yeah, when I set the range to +/-1 standard deviation, the structures become substantially more visible. I'm seeing the same behavior when I incorporate isonetcorrected data (i.e. that my score drops) so I suspect the problem is on my side." }, "source": "meta" }