![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18538441%2F29191241e615e6c202092157f3203a9f%2FWechatIMG20590.jpg?generation=1739001808068077&alt=media\")
First, we thank the competition host and Kaggle staff for organizing this competition. Below, we introduce the solution of the team I Cryo Everyteim -- @sirapoabchaikunsaeng, @iamparadox, @itsuki9180, @sersasj --
\nOur final submissions consisted of four 3D U-Net model soups, trained with different model sizes, parameters, and training data to ensure strong model complementarity. All 3D U-Nets were trained using patch sizes of (128, 128, 128) but were inferred with patch sizes of (160, 384, 384), with a 25% overlap using gaussian reconstruction to handle border artifacts. Additionally, we employed geometric test-time augmentation (TTA), including flipping and transpose.
\nOur models were originally based on the host's example notebook and @fnands notebook . We utilized 3D U-Net architectures from the MONAI library, trained with patch sizes of (128, 128, 128).
\nThe models were 3 levels deep with strides of (2, 2, 1). We started with a simple model:
: \n: \n: \nThis single model with our inference strategy reached 0.744 in public leaderboard score.
\nLater, we trained more complex U-Nets with configs:
: ,\n: ,\n: ,\nThe model with above config alone achieved a score of 0.759 LB.
\n: ,\n: ,\n: ,\nWe also applied a dropout of 0.2 or 0.3 in the model.
\nThe models were pre-trained on six synthetic tomograms denoised with Gaussian denoising—specifically, the 'TS_0', 'TS_1', 'TS_10', 'TS_11', 'TS_12', and 'TS_13' tomograms.
\n@sersasj note: Gaussian denoising was applied because it visually improved the WBP tomograms particle visualization and was easy to implement. I hypothesized that better results could be achieved with a more advanced denoiser, but attempting to code a model for denoising used too much of my Kaggle quota, so I gave up.
\nPretraining not only reduced the time needed for the models to learn the particles but also increased the LB score by roughly 0.01.
\nBoth pretraining on synthetic data and fine-tuning uses the following transformations:
Compose([\n RandCropByLabelClassesd(\n keys=[, ],\n =,\n spatial_size=[128, 128, 128],\n =7, # background, all 6 classes\n =16,\n ),\n RandFlipd(keys=[, ], =0.5, =0),\n RandFlipd(keys=[, ], =0.5, =1),\n RandFlipd(keys=[, ], =0.5, =2),\n])\nWe used MONAI's DiceCELoss and optimized the models with AdamW, employing a learning rate reduction on plateau and an initial learning rate of 1e-3.
\nWe also experimented with Exponential Moving Average (EMA), which showed good results. In the last 3 days of the competition, we trained models using other tomo types: \"denoised\", \"ctfdeconvolved\", \"isonetcorrected\". For almost the entire competition, we hadn't found any increase in LB strategies other than geometric data augmentation in training (flip and rotate); nevertheless, training with other tomo types yielded good results.
We primarily relied on out-of-fold predictions from our k-fold models. Specifically, we implemented a 7-fold cross-validation approach where we trained on all tomographies except one, which was used as a validation set. This process was repeated seven times, with each fold serving as the validation set once. This helped us obtain a score that correlated well with the public leaderboard score, typically 0.01 to 0.02 points higher.
\nOne limitation we encountered was our inability to apply this validation strategy when testing ensembles of our model soups. In these cases, we had to rely on leaked data scores. While this approach didn't correlate as strongly with leaderboard scores, most of the time it allowed us to rank the performance of different ensembles, and compare TTA strategies.
During inference, we use MONAI's sliding_window_inference with 25% overlap and Gaussian reconstruction to handle border predictions more accurately; this boosted the public LB by around 0.002. One of the greatest findings was making the inference in patches as large as possible; we used a window size of (160, 384, 384), boosting our score by around 0.01 compared to a size of (128, 128, 128). (Thanks @iamparadox for finding this)
\nWe also experimented with combining different sizes of predictions, for example (160, 384, 384) + (136, 248, 248), but inference took much longer while yielding inconclusive gains. Therefore, we abandoned this approach in favor of creating more ensembles and employing more TTA.
\nFor post-processing, we improved particle detection by combining model predictions at the logits level rather than averaging probabilities directly.
\nWe then refined particle identification using watershed segmentation, which helped separate touching or overlapping particles. The watershed process created binary masks using particle-specific certainty thresholds, computed distance transforms to measure particle separation, identified distinct particles using local maxima as markers, and finally applied watershed segmentation to determine particle boundaries. We switched from using skimage to cucim and cupy to decrease the inference time.
\nWe also used specific blob threshold sizes for each particle to decrease the number of false positives in our predictions.
\nMoreover, we used flip along the X and Y axes and transpose TTA.
Ensembling was always one of the main ideas in our team. When the team was fully formed 2 weeks before the deadline, we were all around a 0.73 LB score, around 50th place on the leaderboard. At the time, I (@sersasj) had a YOLO scoring 0.682 and a U-Net scoring 0.692 that, when ensembled, reached 0.732 in LB. The rest of the team had only a 3D U-Net that reached 0.739. So it was obvious that we would ensemble everything and achieve amazing results; sadly, it didn't work.
\nNevertheless, we learned valuable lessons from this experience. My 3D U-Net was ~30 times smaller than the 0.739 scoring U-Net, but it proved quite effective when used with the team's amazing inference techniques: larger patches, blob thresholds, ensemble at logits level, and watershed processing. After implementing pretraining and additional training, it achieved a 0.744 leaderboard score.
\nThanks to our group work (especially @sirapoabchaikunsaeng and @iamparadox), we successfully trained several U-Net variants derived from the 0.744 model by modifying parameters, implementing EMA (Exponential Moving Average), incorporating more data, and applying the techniques mentioned above.
\nBy doing so, we had various U-Nets that were complementary to each other, but we couldn't use them all. To manage \"using\" all we had, @itsuki9180 presented the idea and code for a technique called model soup, which essentially averages the weights of multiple models from the same pre-trained weights.
\nWe applied this to the models from our k-fold training.
\nOur final ensemble consisted of:
8th place solution of kaggle czii competition code github\n
\nSubmission Notebook
\nSubmission Notebook Clean Version
First, we thank the competition host and Kaggle staff for organizing this competition. Below, we introduce the solution of the team I Cryo Everyteim -- @sirapoabchaikunsaeng, @iamparadox, @itsuki9180, @sersasj --
\nOur final submissions consisted of four 3D U-Net model soups, trained with different model sizes, parameters, and training data to ensure strong model complementarity. All 3D U-Nets were trained using patch sizes of (128, 128, 128) but were inferred with patch sizes of (160, 384, 384), with a 25% overlap using gaussian reconstruction to handle border artifacts. Additionally, we employed geometric test-time augmentation (TTA), including flipping and transpose.
\nOur models were originally based on the host's example notebook and @fnands notebook . We utilized 3D U-Net architectures from the MONAI library, trained with patch sizes of (128, 128, 128).
\nThe models were 3 levels deep with strides of (2, 2, 1). We started with a simple model:
: \n: \n: \nThis single model with our inference strategy reached 0.744 in public leaderboard score.
\nLater, we trained more complex U-Nets with configs:
: ,\n: ,\n: ,\nThe model with above config alone achieved a score of 0.759 LB.
\n: ,\n: ,\n: ,\nWe also applied a dropout of 0.2 or 0.3 in the model.
\nThe models were pre-trained on six synthetic tomograms denoised with Gaussian denoising—specifically, the 'TS_0', 'TS_1', 'TS_10', 'TS_11', 'TS_12', and 'TS_13' tomograms.
\n@sersasj note: Gaussian denoising was applied because it visually improved the WBP tomograms particle visualization and was easy to implement. I hypothesized that better results could be achieved with a more advanced denoiser, but attempting to code a model for denoising used too much of my Kaggle quota, so I gave up.
\nPretraining not only reduced the time needed for the models to learn the particles but also increased the LB score by roughly 0.01.
\nBoth pretraining on synthetic data and fine-tuning uses the following transformations:
Compose([\n RandCropByLabelClassesd(\n keys=[, ],\n =,\n spatial_size=[128, 128, 128],\n =7, # background, all 6 classes\n =16,\n ),\n RandFlipd(keys=[, ], =0.5, =0),\n RandFlipd(keys=[, ], =0.5, =1),\n RandFlipd(keys=[, ], =0.5, =2),\n])\nWe used MONAI's DiceCELoss and optimized the models with AdamW, employing a learning rate reduction on plateau and an initial learning rate of 1e-3.
\nWe also experimented with Exponential Moving Average (EMA), which showed good results. In the last 3 days of the competition, we trained models using other tomo types: \"denoised\", \"ctfdeconvolved\", \"isonetcorrected\". For almost the entire competition, we hadn't found any increase in LB strategies other than geometric data augmentation in training (flip and rotate); nevertheless, training with other tomo types yielded good results.
We primarily relied on out-of-fold predictions from our k-fold models. Specifically, we implemented a 7-fold cross-validation approach where we trained on all tomographies except one, which was used as a validation set. This process was repeated seven times, with each fold serving as the validation set once. This helped us obtain a score that correlated well with the public leaderboard score, typically 0.01 to 0.02 points higher.
\nOne limitation we encountered was our inability to apply this validation strategy when testing ensembles of our model soups. In these cases, we had to rely on leaked data scores. While this approach didn't correlate as strongly with leaderboard scores, most of the time it allowed us to rank the performance of different ensembles, and compare TTA strategies.
During inference, we use MONAI's sliding_window_inference with 25% overlap and Gaussian reconstruction to handle border predictions more accurately; this boosted the public LB by around 0.002. One of the greatest findings was making the inference in patches as large as possible; we used a window size of (160, 384, 384), boosting our score by around 0.01 compared to a size of (128, 128, 128). (Thanks @iamparadox for finding this)
\nWe also experimented with combining different sizes of predictions, for example (160, 384, 384) + (136, 248, 248), but inference took much longer while yielding inconclusive gains. Therefore, we abandoned this approach in favor of creating more ensembles and employing more TTA.
\nFor post-processing, we improved particle detection by combining model predictions at the logits level rather than averaging probabilities directly.
\nWe then refined particle identification using watershed segmentation, which helped separate touching or overlapping particles. The watershed process created binary masks using particle-specific certainty thresholds, computed distance transforms to measure particle separation, identified distinct particles using local maxima as markers, and finally applied watershed segmentation to determine particle boundaries. We switched from using skimage to cucim and cupy to decrease the inference time.
\nWe also used specific blob threshold sizes for each particle to decrease the number of false positives in our predictions.
\nMoreover, we used flip along the X and Y axes and transpose TTA.
Ensembling was always one of the main ideas in our team. When the team was fully formed 2 weeks before the deadline, we were all around a 0.73 LB score, around 50th place on the leaderboard. At the time, I (@sersasj) had a YOLO scoring 0.682 and a U-Net scoring 0.692 that, when ensembled, reached 0.732 in LB. The rest of the team had only a 3D U-Net that reached 0.739. So it was obvious that we would ensemble everything and achieve amazing results; sadly, it didn't work.
\nNevertheless, we learned valuable lessons from this experience. My 3D U-Net was ~30 times smaller than the 0.739 scoring U-Net, but it proved quite effective when used with the team's amazing inference techniques: larger patches, blob thresholds, ensemble at logits level, and watershed processing. After implementing pretraining and additional training, it achieved a 0.744 leaderboard score.
\nThanks to our group work (especially @sirapoabchaikunsaeng and @iamparadox), we successfully trained several U-Net variants derived from the 0.744 model by modifying parameters, implementing EMA (Exponential Moving Average), incorporating more data, and applying the techniques mentioned above.
\nBy doing so, we had various U-Nets that were complementary to each other, but we couldn't use them all. To manage \"using\" all we had, @itsuki9180 presented the idea and code for a technique called model soup, which essentially averages the weights of multiple models from the same pre-trained weights.
\nWe applied this to the models from our k-fold training.
\nOur final ensemble consisted of:
8th place solution of kaggle czii competition code github\n
\nSubmission Notebook
\nSubmission Notebook Clean Version
Congratulations to your team, and very glad to see you @iamparadox winning a gold medal.
\nI noticed a post on Xiaohongshu regarding accusations against one of your team members. Many Chinese kagglers in the discussion group echoed similar experiences. I don't know the private communications within your team regarding this matter. Here, I am posting the screenshot to remind everyone to be cautious in choosing teammates when teaming up. Of course, if the relevant accusations are proven to be untrue, I am willing to publicly apologize to you @yinhewang .
\nThis may affect the relationships among your team members, and for that I sincerely apologize.
\n
The translation:
\n\"Bro, you really managed to fool people. This person contacted me to team up on Kaggle two months ago. At that time, I was ranked relatively high on the leaderboard for a silver medal in one competition. I didn't think much about it. He said he had several L20s to run experiments. But after teaming up, he just disappeared, casting a wide net in various competitions. Then other people from different competitions approached me, asking if this person was just a freeloader doing no work. Now that CZII has ended, you really managed to get yourself a gold medal. I don't know if you disappeared in this team as well. I just know that in their team's solution, four people were mentioned, but not you. Foreigners really don’t understand just how cunning some Chinese people can be, huh?\"
I am sorry to hear about this. As a Chinese, I feel so ashamed of such inappropriate behavior. However, it is important to acknowledge that many Chinese individuals work diligently and collaborate effectively in teamworks. The key challenge is managing cunning and fool people who may ghost team members or steal your contributions. Therefore, it is essential to remain cautious and careful, and in certain situations, verification may be necessary.
", "rawMarkdown": "I am sorry to hear about this. As a Chinese, I feel so ashamed of such inappropriate behavior. However, it is important to acknowledge that many Chinese individuals work diligently and collaborate effectively in teamworks. The key challenge is managing cunning and fool people who may ghost team members or steal your contributions. Therefore, it is essential to remain cautious and careful, and in certain situations, verification may be necessary.", "votes": 3 } ] }, { "id": 3118663, "postDate": "2025-02-08T10:26:51.150Z", "content": "Hi @sweetyheehee, we are aware of @yinhewang's situation. He had no involvement in winning the gold medal. Essentially, he ghosted us from the beginning and only returned after we won.
\nWe tried to investigate, but things started getting very complicated and weird, many lies were appearing. We also reached out to other teams.
\nWe sent an email to Kaggle and were waiting for a response before going public and starting a discussion on Kaggle with all the proof we gathered.
\nI entered right before merge so don't have all info to share. But my teammates will come here later.
", "rawMarkdown": "Hi @sweetyheehee, we are aware of @yinhewang's situation. He had no involvement in winning the gold medal. Essentially, he ghosted us from the beginning and only returned after we won.\n\nWe tried to investigate, but things started getting very complicated and weird, many lies were appearing. We also reached out to other teams.\n\nWe sent an email to Kaggle and were waiting for a response before going public and starting a discussion on Kaggle with all the proof we gathered.\n\nI entered right before merge so don't have all info to share. But my teammates will come here later.\n\n\n\n", "votes": 6, "replies": [ { "id": 3118681, "postDate": "2025-02-08T10:54:41.173Z", "content": "Thanks for your feedback @sersasj ,
\nThe reason I made this post is that many Chinese kagglers are already discussing his alleged misconduct. We don't want more people to be deceived, and we also hope that he can clarify these matters. However, his account has already been deleted, which suggests that the previous accusations are true.
", "rawMarkdown": "Thanks for your feedback @sersasj , \n\nThe reason I made this post is that many Chinese kagglers are already discussing his alleged misconduct. We don't want more people to be deceived, and we also hope that he can clarify these matters. However, his account has already been deleted, which suggests that the previous accusations are true. ", "votes": 3 } ] }, { "id": 3118729, "postDate": "2025-02-08T12:24:37.533Z", "content": "As a Chinese, I feel ashamed of this behavior, but more importantly, this behavior has seriously damaged the friendly and cooperative atmosphere of the kaggle community. Therefore, I would like to warn everyone to be careful when facing the application of unknown person to join the team, especially for such low-activity accounts. It is hoped that through the joint efforts of community members, the kaggle community can be maintained in a positive and healthy direction. Finally, I would like to apologize again for this happening in the kaggle community.
\nThere is a Chinese poem that I think is a powerful criticism of such people：
\n“小人者其未得也，则忧不得； 既已得之，又恐失之。是以有终身之忧，无一日之乐也。”
Thank you very much @sweetyheehee, as @sersasj mentioned, we have started our investigations ourselves and have found a lot of evidence that points towards @yinhewang deliberately leeching off of teams.
\nFrom our side we have submitted an inquiry to kaggle, which they have forwarded to kaggle's compliance team. I'm not sure if this is correlated with @yinhewang's account being deleted or not. But as of the time I post this comment, the account is gone https://www.kaggle.com/yinhewang
\nIn addition to this, we have evidence that @yinhewang is also planning to even take parts of our prize money by submitting the winning team form. We are not sure what additional steps we can take to guarantee this doesn't happen. But we do have couple of written evidence from him that he is willing to not take any part in prize money.
", "rawMarkdown": "Thank you very much @sweetyheehee, as @sersasj mentioned, we have started our investigations ourselves and have found a lot of evidence that points towards @yinhewang deliberately leeching off of teams.\n\nFrom our side we have submitted an inquiry to kaggle, which they have forwarded to kaggle's compliance team. I'm not sure if this is correlated with @yinhewang's account being deleted or not. But as of the time I post this comment, the account is gone https://www.kaggle.com/yinhewang\n\nIn addition to this, we have evidence that @yinhewang is also planning to even take parts of our prize money by submitting the winning team form. We are not sure what additional steps we can take to guarantee this doesn't happen. But we do have couple of written evidence from him that he is willing to not take any part in prize money.", "votes": 3, "replies": [ { "id": 3118757, "postDate": "2025-02-08T13:11:34.840Z", "content": "I've made a post in the general thread to try bring more attention to this issue https://www.kaggle.com/discussions/general/561879
", "rawMarkdown": "I've made a post in the general thread to try bring more attention to this issue https://www.kaggle.com/discussions/general/561879", "votes": 4 }, { "id": 3118769, "postDate": "2025-02-08T13:29:42.593Z", "content": "Thank you, Sumo. I will forward your post to my friends as well as his former teammates.
", "rawMarkdown": "Thank you, Sumo. I will forward your post to my friends as well as his former teammates.", "votes": 2 } ] } ] }, { "id": 3117418, "postDate": "2025-02-07T01:27:27.053Z", "content": "I am super excited to team up with @sersasj , @iamparadox and @sirapoabchaikunsaeng to complete this competition! They are all wonderful mates who are very proactive in discussions and didn't hesitate to try out my crazy ideas.
\n@sersasj picked out and summarized the ideas that worked for me, but in this post I'll cover the ones that didn't.
Many of my suggestions went to waste, but I'm still happy to have contributed even a small amount. I ended up not being of much use to the team, so I would like to take this opportunity to thank the team once again.
\nIn this post, I will first state what I want to say. Most of our team's experiments (especially mine) have been failures, Maybe, as have most of the other gold medal winners. Don't be afraid of failing an experiment or not winning a medal in a competition. Don't give up just because you lost one competition. Kaggle NEVER penalize you for losing. Your competitors are also your best companions to expand your knowledge. Don't stop learning from them, and one day you will become a great Kaggler.
\nThere is a Japanese-English word, \"no side\" It is a word used to praise both sides for their good fight after the game is over, regardless of whether they are allies or opponents. This is getting long, but my final word is…
\nNO SIDE!
I had never used YOLO before, so I continued submitting in YOLO for practice. However, I did not do a team merge because I had doubts about my YOLO license and could have been expelled.
\nOne of my innovations was the calculation of Z coordinates.
\nWhen calculating the center coordinates in KDTree, only the Z coordinate is calculated with confidence as a weight. This improved the LB by about 0.005
particle_confidences = pdf['confidence'].tolist()
\nparticle_xs = pdf['x'].tolist()
\nparticle_ys = pdf['y'].tolist()
\nparticle_zs = pdf['z'].tolist()
\nparticle_zs_conf = (pdf['z']*pdf['confidence']).tolist()
\n～
\ncenters_x = np.bincount(inverse_indices, weights=particle_xs) / counts
\ncenters_y = np.bincount(inverse_indices, weights=particle_ys) / counts
\n# centers_z = np.bincount(inverse_indices, weights=particle_yz) / counts
\ncenters_z = np.bincount(inverse_indices, weights=particle_zs_conf) / conf_sums
Congrats all of you! Very nice job! Thanks for the great explanation!
", "rawMarkdown": "Congrats all of you! Very nice job! Thanks for the great explanation!", "votes": 4, "replies": [ { "id": 3117574, "postDate": "2025-02-07T04:15:37.163Z", "content": "Thanks @davidlist congrats on your gold as well!!, You posts really helped us during the competition!
", "rawMarkdown": "Thanks @davidlist congrats on your gold as well!!, You posts really helped us during the competition!", "votes": 2, "replies": [ { "id": 3117809, "postDate": "2025-02-07T09:06:46.490Z", "content": "Yeah, I may have to tone that down a little next time. Too many people ahead of me thanking me. 😀 (Kidding, of course.)
", "rawMarkdown": "Yeah, I may have to tone that down a little next time. Too many people ahead of me thanking me. 😀 (Kidding, of course.)", "votes": 2 }, { "id": 3118039, "postDate": "2025-02-07T14:05:54.540Z", "content": "People behind doing the same 😆 That's the Kaggle spirit, and it's what makes these competitions so fun!
", "rawMarkdown": "People behind doing the same 😆 That's the Kaggle spirit, and it's what makes these competitions so fun!", "votes": 1 } ] } ] }, { "id": 3119442, "postDate": "2025-02-09T09:22:03.693Z", "content": "Congratulations to your team for winning this game, and I am also sorry for yinhe wang's situation.
\nIn addition, I would like to know how you use the EMA strategy. When I tried basic EMA, it had almost no effect on improving my score. So I adjusted to update once after each round and tried using the dynamic deck method, which can achieve slight improvements in some cases. I saw that you said EMA has a good effect, so I would like to consult your usage strategy
Congratulations!
\n@iamparadox
\nThis is interesting for me.
\n\nOne of the greatest findings was making the inference in patches as large as possible; we used a window size of (160, 384, 384), boosting our score by around 0.01 compared to a size of (128, 128, 128).
\n
Do you train model with same patch size or not ?
\nIf not, I cannot understand why this improve score.
\nI would appreciate it if you could share any theoretical background.
The model was only trained on 128x128x128 patch size but it was inferred on 160x384x384 patch size. I honestly don't know a lot of theoretical background on this topic, but the idea here's that the contextual information doesn't change while we are doing the inference, in a sense that suppose, a 3x3x3 conv filter expects high value in the center pixel of the input image to activate then this doesn't change even if I increase the patch size at inference and the second reason is doing inference on larger patchsize reduces the border artifacts.
\nThe large patchsize trick will not work if we downsample the image into smaller dimension, and then infer them on higher dimension because in this case the contextual information will change, i.e it will see finer details that it has never seen before.
", "rawMarkdown": "The model was only trained on 128x128x128 patch size but it was inferred on 160x384x384 patch size. I honestly don't know a lot of theoretical background on this topic, but the idea here's that the contextual information doesn't change while we are doing the inference, in a sense that suppose, a 3x3x3 conv filter expects high value in the center pixel of the input image to activate then this doesn't change even if I increase the patch size at inference and the second reason is doing inference on larger patchsize reduces the border artifacts. \n\nThe large patchsize trick will not work if we downsample the image into smaller dimension, and then infer them on higher dimension because in this case the contextual information will change, i.e it will see finer details that it has never seen before.", "votes": 3 }, { "id": 3119101, "postDate": "2025-02-08T21:49:04.200Z", "content": "@clearwaterkzk One pretty big issue I found with the @fnands notebook was the borders of each patch were different from the rest of the model output so you could see a grid pattern in the reconstructed model output. You can see some evidence of this here: https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549615 which shows single patches. I wonder if this is the same issue. Making the patches bigger would reduce the problem. I likewise saw a big performance increase by deleting the outer 3 rows of pixels from each patch before reconstruction and averaging a small overlap. slinding_window_inference() makes this pretty straightforward.
", "rawMarkdown": "@clearwaterkzk One pretty big issue I found with the @fnands notebook was the borders of each patch were different from the rest of the model output so you could see a grid pattern in the reconstructed model output. You can see some evidence of this here: [https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549615](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549615) which shows single patches. I wonder if this is the same issue. Making the patches bigger would reduce the problem. I likewise saw a big performance increase by deleting the outer 3 rows of pixels from each patch before reconstruction and averaging a small overlap. slinding_window_inference() makes this pretty straightforward.", "votes": 1, "replies": [ { "id": 3119145, "postDate": "2025-02-08T23:27:50.210Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3119146, "postDate": "2025-02-08T23:29:04.317Z", "content": "Thanks for replying.
\nI also got the answer from itsuki9180 directly in Twitter(X) regarding this.
I found the border issue and deal with it by padding the voxel and sliding window with a lot of overlap, in which only the center part are predicted, then averaging probability like following figure.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4250230%2F88c71d74e91423a3c61ec3136ccd3c4d%2Fsliding_window.png?generation=1739056705345010&alt=media\")
\nyellow: original exp voxel size, black: patch size for inference, red: probability to use for reconstruction
It seems that this method is used in some segmentation past competition like Vesuvius Challenge, but it requires more inference time. Your method seems smarter and faster.
\nI used to have the fixed idea that CNNs must be inferred with the same size as they were trained on. However, after seeing this solution, I realized for the first time that convolution-only calculations can accept any input size. I learned a lot from this. Thank you very much.
", "rawMarkdown": "Thanks for replying.\nI also got the answer from itsuki9180 directly in Twitter(X) regarding this.\n\nI found the border issue and deal with it by padding the voxel and sliding window with a lot of overlap, in which only the center part are predicted, then averaging probability like following figure.\n\n\nyellow: original exp voxel size, black: patch size for inference, red: probability to use for reconstruction\n\n\nIt seems that this method is used in some segmentation past competition like Vesuvius Challenge, but it requires more inference time. Your method seems smarter and faster.\n\n\nI used to have the fixed idea that CNNs must be inferred with the same size as they were trained on. However, after seeing this solution, I realized for the first time that convolution-only calculations can accept any input size. I learned a lot from this. Thank you very much.", "votes": 1, "replies": [ { "id": 3119158, "postDate": "2025-02-09T00:18:31.917Z", "content": "@clearwaterkzk Just to shown the difference bigger patches made, I submitted our best submission with patches 128.
\nBest submission: public lb: 0.78022, private lb: 0.77612
\nPatches 128 submission: public lb: 0.77464, private lb: 0.76604
That's interesting, I was also under the impression that I must infer on same subvolume size I trained on, until I accidentally discovered that's not the case. However, I thought that somehow, maintaining isotropic subvolume sizes would be advantageous (I worked on 184x184x184 for most of the competition, started with 128x320x320), and I remember checking an experiment where I had my best submission perform better on the 184x184x184. I have just sent a few experiments on 128x320x320 to see if what @sersasj found stands true for my solution as well.
\nI understand that with bigger patches, there's less border artifacts and less merges to be made, but assuming a sizable overlap and gaussian weights for reconstruction, I was inclined to believe that the edges would have a poor enough weight to be negligible.
brilliant, interesting that augmentations didn't work at all. did you try mirroring?
", "rawMarkdown": "brilliant, interesting that augmentations didn't work at all. did you try mirroring?" }, { "id": 3117093, "postDate": "2025-02-06T16:19:45.010Z", "content": "@rezaparaan,
\nthank you again for the competition! we're very grateful for all the learnings we've had from this, and also all the friendships we've made (this is the first time we all team up), as well as designing the public and private LB such that there's no massive shakeup!
this is the first time our team won gold, may I ask you to outline what will happen next? As in what should we provide you with (code / model weights / reports). So that I can make sure that I don't miss anything in the process.
\nThank you!
", "rawMarkdown": "@rezaparaan,\nthank you again for the competition! we're very grateful for all the learnings we've had from this, and also all the friendships we've made (this is the first time we all team up), as well as designing the public and private LB such that there's no massive shakeup!\n\nthis is the first time our team won gold, may I ask you to outline what will happen next? As in what should we provide you with (code / model weights / reports). So that I can make sure that I don't miss anything in the process.\n\nThank you!", "replies": [ { "id": 3117094, "postDate": "2025-02-06T16:22:20.337Z", "content": "cc @kharrington, provided you're also this competition host, apologies for any mistakes in tagging anyone
", "rawMarkdown": "cc @kharrington, provided you're also this competition host, apologies for any mistakes in tagging anyone" }, { "id": 3119183, "postDate": "2025-02-09T01:28:33.623Z", "content": "Hi Sumo, thank you so much for participating. We are thrilled to see the Kaggle community in action tackling problems that are important to us. I have copy-pasted this from the Rules section:
\n\n", "rawMarkdown": "Hi Sumo, thank you so much for participating. We are thrilled to see the Kaggle community in action tackling problems that are important to us. I have copy-pasted this from the Rules section:\n>8. WINNER’S OBLIGATIONS.\na. As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:\n1. Deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation. The delivered software code should follow these documentation guidelines, must be capable of generating the winning Submission, and contain a description of resources required to build and/or run the executable code successfully. For avoidance of doubt, delivered software code should include training code, inference code, and a description of the required computational environment. To the extent that the final model’s software code includes generally commercially available software that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, then instead of delivering the code for that software to the Competition Sponsor, you must identify that software, method for procuring it, and any parameters or other information necessary to replicate the winning Submission;\n2. Grant to the Competition Sponsor the license to the winning Submission stated in the Competition Specific Rules above, and represent that you have the unrestricted right to grant that license;\n3. Sign and return all Prize acceptance documents as may be required by Competition Sponsor or Kaggle, including without limitation: (a) eligibility certifications; (b) licenses, releases and other agreements required under the Rules; and (c) U.S. tax forms (such as IRS Form W-9 if U.S. resident, IRS Form W-8BEN if foreign resident, or future equivalents).\n4. Individual Participants and Teams who create a Submission using an AMLT may win a Prize. However, for clarity, the potential winner’s Submission must still meet the requirements of these Rules, including but not limited to Section 2.5 (Winner’s License and Non-confidentiality), Section 2.8 (Winner’s Obligations), and Section 3.14 (Warranty, Indemnity, and Release).<", "votes": 2 } ] }, { "id": 3117027, "postDate": "2025-02-06T14:57:47.563Z", "content": "\n
\n- WINNER’S OBLIGATIONS.
\n
\n a. As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:- Deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation. The delivered software code should follow these documentation guidelines, must be capable of generating the winning Submission, and contain a description of resources required to build and/or run the executable code successfully. For avoidance of doubt, delivered software code should include training code, inference code, and a description of the required computational environment. To the extent that the final model’s software code includes generally commercially available software that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, then instead of delivering the code for that software to the Competition Sponsor, you must identify that software, method for procuring it, and any parameters or other information necessary to replicate the winning Submission;
\n- Grant to the Competition Sponsor the license to the winning Submission stated in the Competition Specific Rules above, and represent that you have the unrestricted right to grant that license;
\n- Sign and return all Prize acceptance documents as may be required by Competition Sponsor or Kaggle, including without limitation: (a) eligibility certifications; (b) licenses, releases and other agreements required under the Rules; and (c) U.S. tax forms (such as IRS Form W-9 if U.S. resident, IRS Form W-8BEN if foreign resident, or future equivalents).
\n- Individual Participants and Teams who create a Submission using an AMLT may win a Prize. However, for clarity, the potential winner’s Submission must still meet the requirements of these Rules, including but not limited to Section 2.5 (Winner’s License and Non-confidentiality), Section 2.8 (Winner’s Obligations), and Section 3.14 (Warranty, Indemnity, and Release).<
\n
Congratulations!!! Very insightful ensemble solutions.
\nWould you mind sharing the single model performance in the final ensemble models?
one of the single model of final ensemble was of following config:
\n\"channels\": (32, 64, 128, 256),\n\"strides\": (2, 2, 1),\n\"num_res_units\": 2,
we created model soup from all the folds and it's public LB was 0.762
", "rawMarkdown": "one of the single model of final ensemble was of following config:\n\n`\"channels\": (32, 64, 128, 256),\n\"strides\": (2, 2, 1),\n\"num_res_units\": 2,`\n\nwe created model soup from all the folds and it's public LB was 0.762\n", "votes": 1, "replies": [ { "id": 3117037, "postDate": "2025-02-06T15:10:26.230Z", "content": "thank you very much for sharing!
", "rawMarkdown": "thank you very much for sharing!" } ] } ] }, { "id": 3118080, "postDate": "2025-02-07T15:07:16.787Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3118576, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-02-08T08:14:54.210000", "content": "Congratulations to your team, and very glad to see you @iamparadox winning a gold medal.
\nI noticed a post on Xiaohongshu regarding accusations against one of your team members. Many Chinese kagglers in the discussion group echoed similar experiences. I don't know the private communications within your team regarding this matter. Here, I am posting the screenshot to remind everyone to be cautious in choosing teammates when teaming up. Of course, if the relevant accusations are proven to be untrue, I am willing to publicly apologize to you @yinhewang .
\nThis may affect the relationships among your team members, and for that I sincerely apologize.
\n
The translation:
\n\"Bro, you really managed to fool people. This person contacted me to team up on Kaggle two months ago. At that time, I was ranked relatively high on the leaderboard for a silver medal in one competition. I didn't think much about it. He said he had several L20s to run experiments. But after teaming up, he just disappeared, casting a wide net in various competitions. Then other people from different competitions approached me, asking if this person was just a freeloader doing no work. Now that CZII has ended, you really managed to get yourself a gold medal. I don't know if you disappeared in this team as well. I just know that in their team's solution, four people were mentioned, but not you. Foreigners really don’t understand just how cunning some Chinese people can be, huh?\"
I am sorry to hear about this. As a Chinese, I feel so ashamed of such inappropriate behavior. However, it is important to acknowledge that many Chinese individuals work diligently and collaborate effectively in teamworks. The key challenge is managing cunning and fool people who may ghost team members or steal your contributions. Therefore, it is essential to remain cautious and careful, and in certain situations, verification may be necessary.
", "votes": 3, "replies": [] } ] }, { "id": 3118663, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-02-08T10:26:51.150000", "content": "Hi @sweetyheehee, we are aware of @yinhewang's situation. He had no involvement in winning the gold medal. Essentially, he ghosted us from the beginning and only returned after we won.
\nWe tried to investigate, but things started getting very complicated and weird, many lies were appearing. We also reached out to other teams.
\nWe sent an email to Kaggle and were waiting for a response before going public and starting a discussion on Kaggle with all the proof we gathered.
\nI entered right before merge so don't have all info to share. But my teammates will come here later.
", "votes": 6, "replies": [ { "id": 3118681, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-02-08T10:54:41.173000", "content": "Thanks for your feedback @sersasj ,
\nThe reason I made this post is that many Chinese kagglers are already discussing his alleged misconduct. We don't want more people to be deceived, and we also hope that he can clarify these matters. However, his account has already been deleted, which suggests that the previous accusations are true.
", "votes": 3, "replies": [] } ] }, { "id": 3118729, "author_name": "StarxSky", "author_url": "", "post_date": "2025-02-08T12:24:37.533000", "content": "As a Chinese, I feel ashamed of this behavior, but more importantly, this behavior has seriously damaged the friendly and cooperative atmosphere of the kaggle community. Therefore, I would like to warn everyone to be careful when facing the application of unknown person to join the team, especially for such low-activity accounts. It is hoped that through the joint efforts of community members, the kaggle community can be maintained in a positive and healthy direction. Finally, I would like to apologize again for this happening in the kaggle community.
\nThere is a Chinese poem that I think is a powerful criticism of such people：
\n“小人者其未得也，则忧不得； 既已得之，又恐失之。是以有终身之忧，无一日之乐也。”
Thank you very much @sweetyheehee, as @sersasj mentioned, we have started our investigations ourselves and have found a lot of evidence that points towards @yinhewang deliberately leeching off of teams.
\nFrom our side we have submitted an inquiry to kaggle, which they have forwarded to kaggle's compliance team. I'm not sure if this is correlated with @yinhewang's account being deleted or not. But as of the time I post this comment, the account is gone https://www.kaggle.com/yinhewang
\nIn addition to this, we have evidence that @yinhewang is also planning to even take parts of our prize money by submitting the winning team form. We are not sure what additional steps we can take to guarantee this doesn't happen. But we do have couple of written evidence from him that he is willing to not take any part in prize money.
", "votes": 3, "replies": [ { "id": 3118757, "author_name": "Sumo", "author_url": "", "post_date": "2025-02-08T13:11:34.840000", "content": "I've made a post in the general thread to try bring more attention to this issue https://www.kaggle.com/discussions/general/561879
", "votes": 4, "replies": [] }, { "id": 3118769, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-02-08T13:29:42.593000", "content": "Thank you, Sumo. I will forward your post to my friends as well as his former teammates.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3117418, "author_name": "ITK8191", "author_url": "", "post_date": "2025-02-07T01:27:27.053000", "content": "I am super excited to team up with @sersasj , @iamparadox and @sirapoabchaikunsaeng to complete this competition! They are all wonderful mates who are very proactive in discussions and didn't hesitate to try out my crazy ideas.
\n@sersasj picked out and summarized the ideas that worked for me, but in this post I'll cover the ones that didn't.
Many of my suggestions went to waste, but I'm still happy to have contributed even a small amount. I ended up not being of much use to the team, so I would like to take this opportunity to thank the team once again.
\nIn this post, I will first state what I want to say. Most of our team's experiments (especially mine) have been failures, Maybe, as have most of the other gold medal winners. Don't be afraid of failing an experiment or not winning a medal in a competition. Don't give up just because you lost one competition. Kaggle NEVER penalize you for losing. Your competitors are also your best companions to expand your knowledge. Don't stop learning from them, and one day you will become a great Kaggler.
\nThere is a Japanese-English word, \"no side\" It is a word used to praise both sides for their good fight after the game is over, regardless of whether they are allies or opponents. This is getting long, but my final word is…
\nNO SIDE!
@ITK8191 @sersasj
\nThank you for your notebooks
I had never used YOLO before, so I continued submitting in YOLO for practice. However, I did not do a team merge because I had doubts about my YOLO license and could have been expelled.
\nOne of my innovations was the calculation of Z coordinates.
\nWhen calculating the center coordinates in KDTree, only the Z coordinate is calculated with confidence as a weight. This improved the LB by about 0.005
particle_confidences = pdf['confidence'].tolist()
\nparticle_xs = pdf['x'].tolist()
\nparticle_ys = pdf['y'].tolist()
\nparticle_zs = pdf['z'].tolist()
\nparticle_zs_conf = (pdf['z']*pdf['confidence']).tolist()
\n～
\ncenters_x = np.bincount(inverse_indices, weights=particle_xs) / counts
\ncenters_y = np.bincount(inverse_indices, weights=particle_ys) / counts
\n# centers_z = np.bincount(inverse_indices, weights=particle_yz) / counts
\ncenters_z = np.bincount(inverse_indices, weights=particle_zs_conf) / conf_sums
Congrats all of you! Very nice job! Thanks for the great explanation!
", "votes": 4, "replies": [ { "id": 3117574, "author_name": "IAmParadox", "author_url": "", "post_date": "2025-02-07T04:15:37.163000", "content": "Thanks @davidlist congrats on your gold as well!!, You posts really helped us during the competition!
", "votes": 2, "replies": [ { "id": 3117809, "author_name": "David List", "author_url": "", "post_date": "2025-02-07T09:06:46.490000", "content": "Yeah, I may have to tone that down a little next time. Too many people ahead of me thanking me. 😀 (Kidding, of course.)
", "votes": 2, "replies": [] }, { "id": 3118039, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-02-07T14:05:54.540000", "content": "People behind doing the same 😆 That's the Kaggle spirit, and it's what makes these competitions so fun!
", "votes": 1, "replies": [] } ] } ] }, { "id": 3119442, "author_name": "wpl", "author_url": "", "post_date": "2025-02-09T09:22:03.693000", "content": "Congratulations to your team for winning this game, and I am also sorry for yinhe wang's situation.
\nIn addition, I would like to know how you use the EMA strategy. When I tried basic EMA, it had almost no effect on improving my score. So I adjusted to update once after each round and tried using the dynamic deck method, which can achieve slight improvements in some cases. I saw that you said EMA has a good effect, so I would like to consult your usage strategy
Congratulations!
\n@iamparadox
\nThis is interesting for me.
\n\nOne of the greatest findings was making the inference in patches as large as possible; we used a window size of (160, 384, 384), boosting our score by around 0.01 compared to a size of (128, 128, 128).
\n
Do you train model with same patch size or not ?
\nIf not, I cannot understand why this improve score.
\nI would appreciate it if you could share any theoretical background.
The model was only trained on 128x128x128 patch size but it was inferred on 160x384x384 patch size. I honestly don't know a lot of theoretical background on this topic, but the idea here's that the contextual information doesn't change while we are doing the inference, in a sense that suppose, a 3x3x3 conv filter expects high value in the center pixel of the input image to activate then this doesn't change even if I increase the patch size at inference and the second reason is doing inference on larger patchsize reduces the border artifacts.
\nThe large patchsize trick will not work if we downsample the image into smaller dimension, and then infer them on higher dimension because in this case the contextual information will change, i.e it will see finer details that it has never seen before.
", "votes": 3, "replies": [] }, { "id": 3119101, "author_name": "David List", "author_url": "", "post_date": "2025-02-08T21:49:04.200000", "content": "@clearwaterkzk One pretty big issue I found with the @fnands notebook was the borders of each patch were different from the rest of the model output so you could see a grid pattern in the reconstructed model output. You can see some evidence of this here: https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/549615 which shows single patches. I wonder if this is the same issue. Making the patches bigger would reduce the problem. I likewise saw a big performance increase by deleting the outer 3 rows of pixels from each patch before reconstruction and averaging a small overlap. slinding_window_inference() makes this pretty straightforward.
", "votes": 1, "replies": [ { "id": 3119145, "author_name": "", "author_url": "", "post_date": "2025-02-08T23:27:50.210000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3119146, "author_name": "Aurora_blue", "author_url": "", "post_date": "2025-02-08T23:29:04.317000", "content": "Thanks for replying.
\nI also got the answer from itsuki9180 directly in Twitter(X) regarding this.
I found the border issue and deal with it by padding the voxel and sliding window with a lot of overlap, in which only the center part are predicted, then averaging probability like following figure.
\n
\nyellow: original exp voxel size, black: patch size for inference, red: probability to use for reconstruction
It seems that this method is used in some segmentation past competition like Vesuvius Challenge, but it requires more inference time. Your method seems smarter and faster.
\nI used to have the fixed idea that CNNs must be inferred with the same size as they were trained on. However, after seeing this solution, I realized for the first time that convolution-only calculations can accept any input size. I learned a lot from this. Thank you very much.
", "votes": 1, "replies": [ { "id": 3119158, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-02-09T00:18:31.917000", "content": "@clearwaterkzk Just to shown the difference bigger patches made, I submitted our best submission with patches 128.
\nBest submission: public lb: 0.78022, private lb: 0.77612
\nPatches 128 submission: public lb: 0.77464, private lb: 0.76604
That's interesting, I was also under the impression that I must infer on same subvolume size I trained on, until I accidentally discovered that's not the case. However, I thought that somehow, maintaining isotropic subvolume sizes would be advantageous (I worked on 184x184x184 for most of the competition, started with 128x320x320), and I remember checking an experiment where I had my best submission perform better on the 184x184x184. I have just sent a few experiments on 128x320x320 to see if what @sersasj found stands true for my solution as well.
\nI understand that with bigger patches, there's less border artifacts and less merges to be made, but assuming a sizable overlap and gaussian weights for reconstruction, I was inclined to believe that the edges would have a poor enough weight to be negligible.
brilliant, interesting that augmentations didn't work at all. did you try mirroring?
", "votes": 0, "replies": [] }, { "id": 3117093, "author_name": "Sumo", "author_url": "", "post_date": "2025-02-06T16:19:45.010000", "content": "@rezaparaan,
\nthank you again for the competition! we're very grateful for all the learnings we've had from this, and also all the friendships we've made (this is the first time we all team up), as well as designing the public and private LB such that there's no massive shakeup!
this is the first time our team won gold, may I ask you to outline what will happen next? As in what should we provide you with (code / model weights / reports). So that I can make sure that I don't miss anything in the process.
\nThank you!
", "votes": 0, "replies": [ { "id": 3117094, "author_name": "Sumo", "author_url": "", "post_date": "2025-02-06T16:22:20.337000", "content": "cc @kharrington, provided you're also this competition host, apologies for any mistakes in tagging anyone
", "votes": 0, "replies": [] }, { "id": 3119183, "author_name": "Reza Paraan", "author_url": "", "post_date": "2025-02-09T01:28:33.623000", "content": "Hi Sumo, thank you so much for participating. We are thrilled to see the Kaggle community in action tackling problems that are important to us. I have copy-pasted this from the Rules section:
\n\n", "votes": 2, "replies": [] } ] }, { "id": 3117027, "author_name": "Ma Edward", "author_url": "", "post_date": "2025-02-06T14:57:47.563000", "content": "\n
\n- WINNER’S OBLIGATIONS.
\n
\n a. As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:- Deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation. The delivered software code should follow these documentation guidelines, must be capable of generating the winning Submission, and contain a description of resources required to build and/or run the executable code successfully. For avoidance of doubt, delivered software code should include training code, inference code, and a description of the required computational environment. To the extent that the final model’s software code includes generally commercially available software that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, then instead of delivering the code for that software to the Competition Sponsor, you must identify that software, method for procuring it, and any parameters or other information necessary to replicate the winning Submission;
\n- Grant to the Competition Sponsor the license to the winning Submission stated in the Competition Specific Rules above, and represent that you have the unrestricted right to grant that license;
\n- Sign and return all Prize acceptance documents as may be required by Competition Sponsor or Kaggle, including without limitation: (a) eligibility certifications; (b) licenses, releases and other agreements required under the Rules; and (c) U.S. tax forms (such as IRS Form W-9 if U.S. resident, IRS Form W-8BEN if foreign resident, or future equivalents).
\n- Individual Participants and Teams who create a Submission using an AMLT may win a Prize. However, for clarity, the potential winner’s Submission must still meet the requirements of these Rules, including but not limited to Section 2.5 (Winner’s License and Non-confidentiality), Section 2.8 (Winner’s Obligations), and Section 3.14 (Warranty, Indemnity, and Release).<
\n
Congratulations!!! Very insightful ensemble solutions.
\nWould you mind sharing the single model performance in the final ensemble models?
one of the single model of final ensemble was of following config:
\n\"channels\": (32, 64, 128, 256),\n\"strides\": (2, 2, 1),\n\"num_res_units\": 2,
we created model soup from all the folds and it's public LB was 0.762
", "votes": 1, "replies": [ { "id": 3117037, "author_name": "Ma Edward", "author_url": "", "post_date": "2025-02-06T15:10:26.230000", "content": "thank you very much for sharing!
", "votes": 0, "replies": [] } ] } ] }, { "id": 3118080, "author_name": "", "author_url": "", "post_date": "2025-02-07T15:07:16.787000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3117017": "\nFirst, we thank the competition host and Kaggle staff for organizing this competition. Below, we introduce the solution of the team I Cryo Everyteim -- @sirapoabchaikunsaeng, @iamparadox, @itsuki9180, @sersasj -- \n\n## Context\n\n- Business context: [competition overview](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/overview)\n- Data context: [competition data](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/data)\n\n\n## Overview of the approach\n\nOur final submissions consisted of four 3D U-Net model soups, trained with different model sizes, parameters, and training data to ensure strong model complementarity. All 3D U-Nets were trained using patch sizes of (128, 128, 128) but were inferred with patch sizes of (160, 384, 384), with a 25% overlap using gaussian reconstruction to handle border artifacts. Additionally, we employed geometric test-time augmentation (TTA), including flipping and transpose.\n\n### Models \n\nOur models were originally based on the host's example notebook and @fnands notebook . We utilized 3D U-Net architectures from the MONAI library, trained with patch sizes of (128, 128, 128).\n\nThe models were 3 levels deep with strides of (2, 2, 1). We started with a simple model:\n\n```\n\"channels\": (32, 64, 128, 128)\n\"strides\": (2, 2, 1)\n\"num_res_units\": 1\n```\n\nThis single model with our inference strategy reached 0.744 in public leaderboard score.\n\nLater, we trained more complex U-Nets with configs:\n\n```\n\"channels\": (32, 64, 128, 256),\n\"strides\": (2, 2, 1),\n\"num_res_units\": 2,\n```\nThe model with above config alone achieved a score of 0.759 LB.\n\n```\n\"channels\": (32, 96, 256, 384),\n\"strides\": (2, 2, 1),\n\"num_res_units\": 2,\n```\n\nWe also applied a dropout of 0.2 or 0.3 in the model.\n\n### Training\n\nThe models were pre-trained on six synthetic tomograms denoised with Gaussian denoising—specifically, the 'TS_0', 'TS_1', 'TS_10', 'TS_11', 'TS_12', and 'TS_13' tomograms.\n\n*@sersasj note: Gaussian denoising was applied because it visually improved the WBP tomograms particle visualization and was easy to implement. I hypothesized that better results could be achieved with a more advanced denoiser, but attempting to code a model for denoising used too much of my Kaggle quota, so I gave up.*\n\nPretraining not only reduced the time needed for the models to learn the particles but also increased the LB score by roughly 0.01.\n\nBoth pretraining on synthetic data and fine-tuning uses the following transformations:\n```\nCompose([\n RandCropByLabelClassesd(\n keys=[\"image\", \"label\"],\n label_key=\"label\",\n spatial_size=[128, 128, 128],\n num_classes=7, # background, and all 6 classes\n num_samples=16,\n ),\n RandFlipd(keys=[\"image\", \"label\"], prob=0.5, spatial_axis=0),\n RandFlipd(keys=[\"image\", \"label\"], prob=0.5, spatial_axis=1),\n RandFlipd(keys=[\"image\", \"label\"], prob=0.5, spatial_axis=2),\n])\n```\n\nWe used MONAI's DiceCELoss and optimized the models with AdamW, employing a learning rate reduction on plateau and an initial learning rate of 1e-3.\n\nWe also experimented with Exponential Moving Average (EMA), which showed good results. In the last 3 days of the competition, we trained models using other tomo types: \"denoised\", \"ctfdeconvolved\", \"isonetcorrected\". For almost the entire competition, we hadn't found any increase in LB strategies other than geometric data augmentation in training (flip and rotate); nevertheless, training with other tomo types yielded good results.\n\n### Validation strategy\n\nWe primarily relied on out-of-fold predictions from our k-fold models. Specifically, we implemented a 7-fold cross-validation approach where we trained on all tomographies except one, which was used as a validation set. This process was repeated seven times, with each fold serving as the validation set once. This helped us obtain a score that correlated well with the public leaderboard score, typically 0.01 to 0.02 points higher.\n\nOne limitation we encountered was our inability to apply this validation strategy when testing ensembles of our model soups. In these cases, we had to rely on leaked data scores. While this approach didn't correlate as strongly with leaderboard scores, most of the time it allowed us to rank the performance of different ensembles, and compare TTA strategies. \n\n## Details of the submission\n\n### Inference\n\nDuring inference, we use MONAI's sliding_window_inference with 25% overlap and Gaussian reconstruction to handle border predictions more accurately; this boosted the public LB by around 0.002. **One of the greatest findings was making the inference in patches as large as possible; we used a window size of (160, 384, 384), boosting our score by around 0.01 compared to a size of (128, 128, 128).** (Thanks @iamparadox for finding this)\n\nWe also experimented with combining different sizes of predictions, for example (160, 384, 384) + (136, 248, 248), but inference took much longer while yielding inconclusive gains. Therefore, we abandoned this approach in favor of creating more ensembles and employing more TTA.\n\nFor post-processing, we improved particle detection by combining model predictions at the logits level rather than averaging probabilities directly.\n\nWe then refined particle identification using watershed segmentation, which helped separate touching or overlapping particles. The watershed process created binary masks using particle-specific certainty thresholds, computed distance transforms to measure particle separation, identified distinct particles using local maxima as markers, and finally applied watershed segmentation to determine particle boundaries. We switched from using skimage to cucim and cupy to decrease the inference time. \n\nWe also used specific blob threshold sizes for each particle to decrease the number of false positives in our predictions.\n\nMoreover, we used flip along the X and Y axes and transpose TTA.\n \n### Ensembles\n\nEnsembling was always one of the main ideas in our team. When the team was fully formed 2 weeks before the deadline, we were all around a 0.73 LB score, around 50th place on the leaderboard. At the time, I (@sersasj) had a YOLO scoring 0.682 and a U-Net scoring 0.692 that, when ensembled, reached 0.732 in LB. The rest of the team had only a 3D U-Net that reached 0.739. So it was obvious that we would ensemble everything and achieve amazing results; sadly, it didn't work.\n\nNevertheless, we learned valuable lessons from this experience. My 3D U-Net was ~30 times smaller than the 0.739 scoring U-Net, but it proved quite effective when used with the team's amazing inference techniques: *larger patches, blob thresholds, ensemble at logits level, and watershed processing*. After implementing pretraining and additional training, it achieved a 0.744 leaderboard score. \n\nThanks to our group work (especially @sirapoabchaikunsaeng and @iamparadox), we successfully trained several U-Net variants derived from the 0.744 model by modifying parameters, implementing EMA (Exponential Moving Average), incorporating more data, and applying the techniques mentioned above.\n\nBy doing so, we had various U-Nets that were complementary to each other, but we couldn't use them all. To manage \"using\" all we had, @itsuki9180 presented the idea and code for a technique called model soup, which essentially averages the weights of multiple models from the same pre-trained weights.\nWe applied this to the models from our k-fold training.\n\nOur final ensemble consisted of:\n1. A soup of 7 folds trained with parameters:\n - Channels: (32, 64, 128, 128)\n - Number of residual units: 1\n - Pretrained on synthetic data\n - Trained on denoised images without EMA\n\n2. A soup of 7 folds trained with parameters:\n - Channels: (32, 64, 128, 256)\n - Number of residual units: 2\n - Pretrained on synthetic data\n - Trained on all denoised, IsoNet-corrected, and CTF-deconvolved images without EMA\n\n3. A soup of 3 folds (TS_69_2, TS_86_3, TS_99_9) trained with parameters:\n - Channels: (32, 96, 256, 384)\n - Number of residual units: 2\n - Pretrained on synthetic data\n - Trained on denoised images with EMA\n\n4. A soup of 7 folds trained with parameters:\n - Channels: (32, 96, 256, 384)\n - Number of residual units: 2\n - Pretrained on synthetic data\n - Trained on all denoised, IsoNet-corrected, and CTF-deconvolved images without EMA\n\n### What didn't work:\n- Increased Augmentations\n- CutMix, MixUp\n- 2D U-Net approaches (reached 0.636 lb only)\n- Ensembling with YOLO (for final solution, works well to reach ~0.73 lb)\n- Bigger and deeper U-Nets\n- Cross entropy only loss functions\n\n\n## Sources section\n\n- [Model Soup paper](https://arxiv.org/abs/2203.05482)\n- [Host's example notebook](https://github.com/czimaginginstitute/2024_czii_mlchallenge_notebooks/blob/main/3d_unet_monai/train.ipynb)\n- [@fnands notebook](https://www.kaggle.com/code/fnands/baseline-unet-train-submit)\n\n\n## Code\n\n[8th place solution of kaggle czii competition code github\n](https://github.com/IAmPara0x/czii-8th-solution)\n[Submission Notebook](https://www.kaggle.com/code/iamparadox/czii-final-sub-reproduce)\n[Submission Notebook Clean Version](https://www.kaggle.com/code/sirapoabchaikunsaeng/czii-final-sub-reproduce)", "3118576": "Congratulations to your team, and very glad to see you @iamparadox winning a gold medal.\n\nI noticed a post on Xiaohongshu regarding accusations against one of your team members. Many Chinese kagglers in the discussion group echoed similar experiences. I don't know the private communications within your team regarding this matter. Here, I am posting the screenshot to remind everyone to be cautious in choosing teammates when teaming up. Of course, if the relevant accusations are proven to be untrue, I am willing to publicly apologize to you @yinhewang .\n\nThis may affect the relationships among your team members, and for that I sincerely apologize.\n\n", "3117418": "I am super excited to team up with @sersasj , @iamparadox and @sirapoabchaikunsaeng to complete this competition! They are all wonderful mates who are very proactive in discussions and didn't hesitate to try out my crazy ideas.\n@sersasj picked out and summarized the ideas that worked for me, but in this post I'll cover the ones that didn't.\n### What I tried that didn't work or didn't used in submission\n1. YOLO approach. I believe many competitors created their solutions based on my baseline, but our team ultimately chose a solution based primarily on 3D-UNet. Therefore, I believe that the solution for YOLO is not worthless, so I will describe it here. In this method, the following preprocessing was applied:\n・For one z, a 3-channel image (z-1, z, z+1) is used instead of a 1-channel image (z, z, z).\n・Scale the image to the range [-1, 1] and then raise the pixel values to the p-th power (approximately 1