\n
![\"TS_69_2\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8096091%2F309e3882d5f2ad83b0237683878ae5ff%2FUntitled.png?generation=1732481149169182&alt=media\")
\n
![\"TS_5_4\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8096091%2F63b916ea77d5eb7c0b1f7159e50dff49%2Funtitled%201.png?generation=1732482382298180&alt=media\")
\n
![\"TS_6_6\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8096091%2F057aa7251cbdcd5cbd68b64137a2d84c%2Funtitled%202.png?generation=1732482575046448&alt=media\")
\n
![\"TS_99_9\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8096091%2Fec5a71cc45301af906c60a5934728398%2Funtitled%203.png?generation=1732482621947948&alt=media\")
\n>
\nThanks
Hi, I saw most of you are struggling with the type of data, I have converted the competition dataset into 8/16bit PNG format. It includes Tomogram images frame by frame and extracted label (mask) images.
\nDataset Link: CZII CryoET: 630x630 PNG Dataset 8/16bit
\nNotebook Link: CZII CryoET - Data Preparation 8/16bit (PNG)
Images are captured at voxel_size=10, No changes were made to the dimensions of the images (630x630).
Tomograms and segmentations were loaded using get_tomogram & segmentation_from_picks using copick library,
tomogram = run.get_voxel_spacing(voxel_size).get_tomogram(tomo_type).numpy()\npick = run.get_picks(object_name=pickable_object.name, user_id=)\ntarget = segmentation_from_picks.from_picks(\n pick[], \n target, \n target_objects[pickable_object.name][] * ,\n target_objects[pickable_object.name][],\n voxel_spacing=voxel_size\n)\nAll the tomograms are normalized using,
\nUpdate: From the comments, I have changed min norm percentile to 1, to capture more info. For more precision to capture more detail, you can use the 16bit images.
():\n = np.percentile(data,)\n = np.percentile(data,)\n data = np.clip(data,,)\n data = (data-)/(-)\n data\nLabel and corresponding mapping with its pdb_id, particle_type & radius,
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nPreview of few tomograms with masks,
\n
\n
\n
\n
\n>
\nThanks
Ref:
\nhttps://www.kaggle.com/code/ahsuna123/3d-u-net-training-only
\nhttps://www.kaggle.com/code/kharrington/blobdetector
Hi, I saw most of you are struggling with the type of data, I have converted the competition dataset into 8/16bit PNG format. It includes Tomogram images frame by frame and extracted label (mask) images.
\nDataset Link: CZII CryoET: 630x630 PNG Dataset 8/16bit
\nNotebook Link: CZII CryoET - Data Preparation 8/16bit (PNG)
Images are captured at voxel_size=10, No changes were made to the dimensions of the images (630x630).
Tomograms and segmentations were loaded using get_tomogram & segmentation_from_picks using copick library,
tomogram = run.get_voxel_spacing(voxel_size).get_tomogram(tomo_type).numpy()\npick = run.get_picks(object_name=pickable_object.name, user_id=)\ntarget = segmentation_from_picks.from_picks(\n pick[], \n target, \n target_objects[pickable_object.name][] * ,\n target_objects[pickable_object.name][],\n voxel_spacing=voxel_size\n)\nAll the tomograms are normalized using,
\nUpdate: From the comments, I have changed min norm percentile to 1, to capture more info. For more precision to capture more detail, you can use the 16bit images.
():\n = np.percentile(data,)\n = np.percentile(data,)\n data = np.clip(data,,)\n data = (data-)/(-)\n data\nLabel and corresponding mapping with its pdb_id, particle_type & radius,
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nPreview of few tomograms with masks,
\n
\n
\n
\n
\n>
\nThanks
Ref:
\nhttps://www.kaggle.com/code/ahsuna123/3d-u-net-training-only
\nhttps://www.kaggle.com/code/kharrington/blobdetector
What is a good strategy to find the minim and max percentile to use when normalizing?
", "rawMarkdown": "What is a good strategy to find the minim and max percentile to use when normalizing?", "votes": null }, { "id": "3054651", "postDate": "11/25/2024 00:13:10", "content": "Experimentation.
\nEDIT: The reason of using percentiles as min and max scaling is to keep out outlier values, the tails of intensities distribution. As more tail you cut, more probable is you'll cut outliers, but more probable is too you'll cut valuable information. Actually you don't cut anything. Just leave those values out 0 to 1 range.
\nI think around 1-5 and 95-99 are reasonable, but you'll have to try.
", "rawMarkdown": "Experimentation.\n\nEDIT: The reason of using percentiles as min and max scaling is to keep out outlier values, the tails of intensities distribution. As more tail you cut, more probable is you'll cut outliers, but more probable is too you'll cut valuable information. Actually you don't cut anything. Just leave those values out 0 to 1 range.\n\nI think around 1-5 and 95-99 are reasonable, but you'll have to try.", "votes": null }, { "id": "3054653", "postDate": "11/25/2024 00:14:04", "content": "Hi. Thanks for share. Personally I've been tunning radius visually. Can I ask from where you took these ones?
", "rawMarkdown": "Hi. Thanks for share. Personally I've been tunning radius visually. Can I ask from where you took these ones?", "votes": null }, { "id": "3054737", "postDate": "11/25/2024 04:19:43", "content": "Hi, Thanks for the comment, the values were taken from the pinned notebook BlobDetector by competition organizers @kharrington.
", "rawMarkdown": "Hi, Thanks for the comment, the values were taken from the pinned notebook [BlobDetector](https://www.kaggle.com/code/kharrington/blobdetector) by competition organizers @kharrington.", "votes": null }, { "id": "3054894", "postDate": "11/25/2024 10:10:01", "content": "Yes that is what I was thinking. You are actually not cutting anything just squeezing all on the tails. So you need to make sure I guess the pixels being squashed are not labels as you are losing information
", "rawMarkdown": "Yes that is what I was thinking. You are actually not cutting anything just squeezing all on the tails. So you need to make sure I guess the pixels being squashed are not labels as you are losing information", "votes": null }, { "id": "3054908", "postDate": "11/25/2024 10:25:28", "content": "my suggestion is cut to 0,1 for visualiation only. for modeling you can keep the values. if u want to cut the values for modeling, the support should should come from experimental results.
\nin datascience, data speaks for itself. that is the best strategy. just need to make sure your experiment results are not bias.
", "rawMarkdown": "my suggestion is cut to 0,1 for visualiation only. for modeling you can keep the values. if u want to cut the values for modeling, the support should should come from experimental results.\n\nin datascience, data speaks for itself. that is the best strategy. just need to make sure your experiment results are not bias.", "votes": null }, { "id": "3054910", "postDate": "11/25/2024 10:27:19", "content": "one note: you can probe hidden test data to see if outliers values occur
", "rawMarkdown": "one note: you can probe hidden test data to see if outliers values occur", "votes": null }, { "id": "3055236", "postDate": "11/25/2024 16:40:47", "content": "Hi @hengck23, thanks for the comment, instead of directly trimming, maybe we can try something like this, inspired from previous SenNet + HOA Competition.
\n ():\n = np.percentile(data,)\n = np.percentile(data,)\n data[data>] = (data[data>]-)* + \n data[data<] = (data[data<]-)* + \n data = (data-)/(-+smooth)\n data\n[x_center, y_center, height, width]. Which can help in performing any Object Detection task.",
"rawMarkdown": "HI, I have also added **Bounding_Boxes** for each frame and each label `[x_center, y_center, height, width]`. Which can help in performing any Object Detection task.",
"votes": null
}
],
"comments": [
{
"id": 3054608,
"author_name": "ignasialemany",
"author_url": "",
"post_date": "11/24/2024 21:45:45",
"content": "What is a good strategy to find the minim and max percentile to use when normalizing?
", "votes": null, "replies": [ { "id": 3054651, "author_name": "sacuscreed", "author_url": "", "post_date": "11/25/2024 00:13:10", "content": "Experimentation.
\nEDIT: The reason of using percentiles as min and max scaling is to keep out outlier values, the tails of intensities distribution. As more tail you cut, more probable is you'll cut outliers, but more probable is too you'll cut valuable information. Actually you don't cut anything. Just leave those values out 0 to 1 range.
\nI think around 1-5 and 95-99 are reasonable, but you'll have to try.
", "votes": null, "replies": [ { "id": 3054894, "author_name": "ignasialemany", "author_url": "", "post_date": "11/25/2024 10:10:01", "content": "Yes that is what I was thinking. You are actually not cutting anything just squeezing all on the tails. So you need to make sure I guess the pixels being squashed are not labels as you are losing information
", "votes": null, "replies": [] }, { "id": 3054908, "author_name": "hengck23", "author_url": "", "post_date": "11/25/2024 10:25:28", "content": "my suggestion is cut to 0,1 for visualiation only. for modeling you can keep the values. if u want to cut the values for modeling, the support should should come from experimental results.
\nin datascience, data speaks for itself. that is the best strategy. just need to make sure your experiment results are not bias.
", "votes": null, "replies": [ { "id": 3054910, "author_name": "hengck23", "author_url": "", "post_date": "11/25/2024 10:27:19", "content": "one note: you can probe hidden test data to see if outliers values occur
", "votes": null, "replies": [] }, { "id": 3055236, "author_name": "gowrishankarp", "author_url": "", "post_date": "11/25/2024 16:40:47", "content": "Hi @hengck23, thanks for the comment, instead of directly trimming, maybe we can try something like this, inspired from previous SenNet + HOA Competition.
\n ():\n = np.percentile(data,)\n = np.percentile(data,)\n data[data>] = (data[data>]-)* + \n data[data<] = (data[data<]-)* + \n data = (data-)/(-+smooth)\n data\nHi. Thanks for share. Personally I've been tunning radius visually. Can I ask from where you took these ones?
", "votes": null, "replies": [ { "id": 3054737, "author_name": "gowrishankarp", "author_url": "", "post_date": "11/25/2024 04:19:43", "content": "Hi, Thanks for the comment, the values were taken from the pinned notebook BlobDetector by competition organizers @kharrington.
", "votes": null, "replies": [] } ] }, { "id": 3061809, "author_name": "gowrishankarp", "author_url": "", "post_date": "12/03/2024 02:14:07", "content": "HI, I have also added Bounding_Boxes for each frame and each label [x_center, y_center, height, width]. Which can help in performing any Object Detection task.