{ "id": 575799, "title": "Yolo train differs", "url": "/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/575799", "author_name": "eikyou", "post_date": "2025-05-01T02:30:43.230000", "votes": 5, "comment_count": 21, "views": 0, "content": "

I trained YOLO11 2 times on the same data and the same parameters (don't ask me why i did this). And I got extremely different results. Both optimal F1 threshold and LB are absolutely different.
\nI do understand that training itself is kinda random and augmentations are random too. I am wondering if such a behaviour is normal or am I missing something?

", "messages": [ { "id": 3194267, "postDate": "2025-05-05T15:42:34.343Z", "content": "

FYI.Ultralytics YOLO reproducibility

\n\n
\ndef init_seeds(=0, =):\n.\n
\n\n
\n ultralytics.utils.torch_utils import init_seeds\ninit_seeds(=42, =)\n\n\nimport torch.backends.cudnn as cudnn\n(cudnn.deterministic)\n\n⇒\n
\n

P.S I have limited computing resources, so this is just an observation based on looking at the source, but I hope it will be helpful.

", "rawMarkdown": "### FYI.Ultralytics YOLO reproducibility\n* The default setting is **deterministic is false.**\n```\n# ultralytics/ultralytics/utils/torch_utils.py\ndef init_seeds(seed=0, deterministic=False):\n...\n```\n* Therefore, to ensure reproducibility, **deterministic must be set to True** in the Torch implementation.\n* According to the Ultralytics src and manual, it can be set up as follows:\n```\n# YOLO deterministic setting\nfrom ultralytics.utils.torch_utils import init_seeds\ninit_seeds(seed=42, deterministic=True)\n\n# Check torch deterministic\nimport torch.backends.cudnn as cudnn\nprint(cudnn.deterministic)\n\n⇒True\n```\n\nP.S I have limited computing resources, so this is just an observation based on looking at the source, but I hope it will be helpful.", "votes": 5, "replies": [ { "id": 3194296, "postDate": "2025-05-05T16:44:40.647Z", "content": "

who could've known that seed is not enough… thanks!

", "rawMarkdown": "who could've known that seed is not enough... thanks!", "votes": 1 } ] }, { "id": 3190706, "postDate": "2025-05-01T02:30:43.230Z", "content": "

I trained YOLO11 2 times on the same data and the same parameters (don't ask me why i did this). And I got extremely different results. Both optimal F1 threshold and LB are absolutely different.
\nI do understand that training itself is kinda random and augmentations are random too. I am wondering if such a behaviour is normal or am I missing something?

", "rawMarkdown": "I trained YOLO11 2 times on the same data and the same parameters (don't ask me why i did this). And I got extremely different results. Both optimal F1 threshold and LB are absolutely different. \nI do understand that training itself is kinda random and augmentations are random too. I am wondering if such a behaviour is normal or am I missing something?", "votes": 5 }, { "id": 3192933, "postDate": "2025-05-03T15:00:22.617Z", "content": "

With a fixed seed, I consistently obtain the same results when using the same training parameters. The training code is adapted from the Kaggle YOLO baseline.

", "rawMarkdown": "With a fixed seed, I consistently obtain the same results when using the same training parameters. The training code is adapted from the Kaggle YOLO baseline.", "votes": 3 }, { "id": 3191256, "postDate": "2025-05-01T15:42:40.890Z", "content": "

Do you have an early stopping condition? It’s possible your LR is too high or too low so one model got stuck in a local minima while the other moved closer to the global minima. Maybe try tuning your LR or using something like a decaying LR? I’m guessing one model plateaued early and hit an early stopping condition whereas the other had a better initialization and was fortunate to find a better “path”

", "rawMarkdown": "Do you have an early stopping condition? It’s possible your LR is too high or too low so one model got stuck in a local minima while the other moved closer to the global minima. Maybe try tuning your LR or using something like a decaying LR? I’m guessing one model plateaued early and hit an early stopping condition whereas the other had a better initialization and was fortunate to find a better “path”", "votes": 1, "replies": [ { "id": 3191384, "postDate": "2025-05-01T17:13:12.133Z", "content": "

I don't use eraly stopping, but LR Indeed seems to be an issue, it's quite low 1e-4, maybe 2e-4 or 3e-4 would be more more stable. Thanks for a suggestion!

", "rawMarkdown": "I don't use eraly stopping, but LR Indeed seems to be an issue, it's quite low 1e-4, maybe 2e-4 or 3e-4 would be more more stable. Thanks for a suggestion!", "replies": [ { "id": 3191481, "postDate": "2025-05-01T19:10:36.527Z", "content": "

No problem, sometimes I start it large like 1e-3 and then once it stops performing I drop it by a factor of 10. I hope this helps :D

", "rawMarkdown": "No problem, sometimes I start it large like 1e-3 and then once it stops performing I drop it by a factor of 10. I hope this helps :D" } ] } ] }, { "id": 3190744, "postDate": "2025-05-01T03:50:32.313Z", "content": "

I think the problem is about the \"optimal threshold\", there may be not so much difference in the yolo, but the LB is very sensitive to threshold..

", "rawMarkdown": "I think the problem is about the \"optimal threshold\", there may be not so much difference in the yolo, but the LB is very sensitive to threshold.. ", "votes": 1, "replies": [ { "id": 3191083, "postDate": "2025-05-01T11:15:21.090Z", "content": "

@yksinyoung I tried various thresholds and the results are much worse. First model on the threshold 0.3 shows performance 0.796, the second one on the same threshold shows 0.667, which is obviously much worse than the first model and only a bit better than the second model on optimal threshold from yolo training. ;)

", "rawMarkdown": "@yksinyoung I tried various thresholds and the results are much worse. First model on the threshold 0.3 shows performance 0.796, the second one on the same threshold shows 0.667, which is obviously much worse than the first model and only a bit better than the second model on optimal threshold from yolo training. ;)", "replies": [ { "id": 3201617, "postDate": "2025-05-14T06:39:12.213Z", "content": "

I used the setting of iou=0.5 during training. When inferencing on the same model, the iou is changed from 0.45 to 0.3, and the lb score dropped.

", "rawMarkdown": "I used the setting of iou=0.5 during training. When inferencing on the same model, the iou is changed from 0.45 to 0.3, and the lb score dropped." } ] } ] }, { "id": 3192857, "postDate": "2025-05-03T13:01:34.673Z", "content": "

Hey @eikyou, did you used seeds in your training?
\nI've trained my models with seeds and I'm going to try retraining them nd submitting to see what happens. I'll tell you later how it goes.
\nA similar thing is happening to me, trained a 5 fold cv (yolo10x) and got models scoring from 0.725 to 0.792.

", "rawMarkdown": "Hey @eikyou, did you used seeds in your training? \nI've trained my models with seeds and I'm going to try retraining them nd submitting to see what happens. I'll tell you later how it goes.\nA similar thing is happening to me, trained a 5 fold cv (yolo10x) and got models scoring from 0.725 to 0.792.", "votes": 2, "replies": [ { "id": 3192867, "postDate": "2025-05-03T13:23:57.160Z", "content": "

I didn't fix seed in YOLO itself, and I didn't use CV, i used hold out. Guess i just randomly got a good model ;) Will focus on CV scheme

", "rawMarkdown": "I didn't fix seed in YOLO itself, and I didn't use CV, i used hold out. Guess i just randomly got a good model ;) Will focus on CV scheme", "votes": 1 }, { "id": 3193557, "postDate": "2025-05-04T15:03:06.553Z", "content": "

How many epochs do you typically train your model?

", "rawMarkdown": "How many epochs do you typically train your model?", "replies": [ { "id": 3193573, "postDate": "2025-05-04T15:57:54.830Z", "content": "

@sjtuwangshuo I usually train for 50 epochs. But I noticed that my models might be overfitting, or that selecting best.pt automatically isn't as optimal as I thought. In one of my folds, I got the best.pt file at different epochs: around 28, 40, and the final one at 50. The respectives scores were 0.816, 0.752, and 0.744.

\n

I had assumed Ultralytics saves the model with the best validation loss, but now I'm not so sure (or that best validation loss = best lb). I need to take a look into that. This experiment wasn’t really planned, I just didn’t have time to train all the folds and didn't want to lose daily submissions.

", "rawMarkdown": "@sjtuwangshuo I usually train for 50 epochs. But I noticed that my models might be overfitting, or that selecting best.pt automatically isn't as optimal as I thought. In one of my folds, I got the best.pt file at different epochs: around 28, 40, and the final one at 50. The respectives scores were 0.816, 0.752, and 0.744.\n\nI had assumed Ultralytics saves the model with the best validation loss, but now I'm not so sure (or that best validation loss = best lb). I need to take a look into that. This experiment wasn’t really planned, I just didn’t have time to train all the folds and didn't want to lose daily submissions.", "votes": 1 }, { "id": 3193578, "postDate": "2025-05-04T16:19:56.293Z", "content": "

Thanks for sharing your interesting findings. In Ultralytics, the best checkpoint is usually saved based on a weighted metric: 0.1 × mAP@0.5 + 0.9 × mAP@0.5:0.95 (as referenced here).

\n

Besides, how did you decide which epoch's checkpoint to submit (e.g., 28, 40, or 50)? Does the checkpoint at epoch 28 have the lowest dfl loss?

", "rawMarkdown": "Thanks for sharing your interesting findings. In Ultralytics, the best checkpoint is usually saved based on a weighted metric: 0.1 × mAP@0.5 + 0.9 × mAP@0.5:0.95 ([as referenced here](https://github.com/ultralytics/ultralytics/issues/14137)). \n\nBesides, how did you decide which epoch's checkpoint to submit (e.g., 28, 40, or 50)? Does the checkpoint at epoch 28 have the lowest dfl loss?" }, { "id": 3193626, "postDate": "2025-05-04T17:48:24.833Z", "content": "

Thanks @sjtuwangshuo. It wasn’t a thought out decision. I just used whatever checkpoint I had ready at the time to use my daily submissions. Unfortunaly, my \"val_period\" was set to 5, so the training plot it's not that useful to find what epoch was best.

\n

The checkpoint at epoch 28 didn’t have the lowest loss overall, but the validation loss was lower than the training loss, wich is good. Also, there’s a big shift in the loss curve around epoch 40. that’s when mosaic augmentation stops.
\nMaybe if I used a later checkpoint before closing mosaic it would have a better lb than a chekpoint in epoch 28, I still have to test this.

\n

Here's the training curve below so you can see what I mean.

\n

\"training

", "rawMarkdown": "Thanks @sjtuwangshuo. It wasn’t a thought out decision. I just used whatever checkpoint I had ready at the time to use my daily submissions. Unfortunaly, my \"val_period\" was set to 5, so the training plot it's not that useful to find what epoch was best.\n\nThe checkpoint at epoch 28 didn’t have the lowest loss overall, but the validation loss was lower than the training loss, wich is good. Also, there’s a big shift in the loss curve around epoch 40. that’s when mosaic augmentation stops.\nMaybe if I used a later checkpoint before closing mosaic it would have a better lb than a chekpoint in epoch 28, I still have to test this.\n\nHere's the training curve below so you can see what I mean.\n\n![training loss curve (period=5)](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2221915%2Fe01979fd60cde14a1a364d36c99dfee9%2Fdfl_loss_curve.png?generation=1746380314405281&alt=media)", "votes": 1 }, { "id": 3193656, "postDate": "2025-05-04T18:49:59.583Z", "content": "

btw, i noticed that better models get lower optimal thresholds for F1 on validation. 0.2~ for good models and 0.4~ for much worse models. Maybe it's a coincidence, but still.
\nAlso i noticed that optimal threshold for LB is 0.3~

", "rawMarkdown": "btw, i noticed that better models get lower optimal thresholds for F1 on validation. 0.2~ for good models and 0.4~ for much worse models. Maybe it's a coincidence, but still.\nAlso i noticed that optimal threshold for LB is 0.3~", "votes": 2 }, { "id": 3195257, "postDate": "2025-05-06T19:59:53.093Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3197016, "postDate": "2025-05-07T18:11:37.047Z", "content": "

@sersasj
\nThanks for suggesting so interesting finding!
\nClose mosic may cause overfittting…
\nbtw, What did you set mosaic rate?
\n( I think yolo set mosaic as 1.0 in default setting)

", "rawMarkdown": "@sersasj \nThanks for suggesting so interesting finding!\nClose mosic may cause overfittting...\nbtw, What did you set mosaic rate?\n( I think yolo set mosaic as 1.0 in default setting)\n" }, { "id": 3197028, "postDate": "2025-05-07T18:34:20.073Z", "content": "

Thanks @npinpi! It was set to 1.0

", "rawMarkdown": "Thanks @npinpi! It was set to 1.0 " } ] } ] }, { "id": 3200223, "postDate": "2025-05-12T09:16:19.147Z", "content": "

hey, can anyone tell me from where it is considered as high LR and from where it is considered as low LR. which is the avg LR like the safest number to start with. and how initial learning rate and final learning rate works.

", "rawMarkdown": "hey, can anyone tell me from where it is considered as high LR and from where it is considered as low LR. which is the avg LR like the safest number to start with. and how initial learning rate and final learning rate works. ", "replies": [ { "id": 3200337, "postDate": "2025-05-12T12:31:23.133Z", "content": "

@mohanapavanbezawada 1e-4 or 3e-4 are usually ok to start with

", "rawMarkdown": "@mohanapavanbezawada 1e-4 or 3e-4 are usually ok to start with", "votes": 1 } ] } ], "comments": [ { "id": 3194267, "author_name": "yukiZ", "author_url": "", "post_date": "2025-05-05T15:42:34.343000", "content": "

FYI.Ultralytics YOLO reproducibility

\n\n
\ndef init_seeds(=0, =):\n.\n
\n\n
\n ultralytics.utils.torch_utils import init_seeds\ninit_seeds(=42, =)\n\n\nimport torch.backends.cudnn as cudnn\n(cudnn.deterministic)\n\n⇒\n
\n

P.S I have limited computing resources, so this is just an observation based on looking at the source, but I hope it will be helpful.

", "votes": 5, "replies": [ { "id": 3194296, "author_name": "eikyou", "author_url": "", "post_date": "2025-05-05T16:44:40.647000", "content": "

who could've known that seed is not enough… thanks!

", "votes": 1, "replies": [] } ] }, { "id": 3192933, "author_name": "atom1231", "author_url": "", "post_date": "2025-05-03T15:00:22.617000", "content": "

With a fixed seed, I consistently obtain the same results when using the same training parameters. The training code is adapted from the Kaggle YOLO baseline.

", "votes": 3, "replies": [] }, { "id": 3191256, "author_name": "Connor", "author_url": "", "post_date": "2025-05-01T15:42:40.890000", "content": "

Do you have an early stopping condition? It’s possible your LR is too high or too low so one model got stuck in a local minima while the other moved closer to the global minima. Maybe try tuning your LR or using something like a decaying LR? I’m guessing one model plateaued early and hit an early stopping condition whereas the other had a better initialization and was fortunate to find a better “path”

", "votes": 1, "replies": [ { "id": 3191384, "author_name": "eikyou", "author_url": "", "post_date": "2025-05-01T17:13:12.133000", "content": "

I don't use eraly stopping, but LR Indeed seems to be an issue, it's quite low 1e-4, maybe 2e-4 or 3e-4 would be more more stable. Thanks for a suggestion!

", "votes": 0, "replies": [ { "id": 3191481, "author_name": "Connor", "author_url": "", "post_date": "2025-05-01T19:10:36.527000", "content": "

No problem, sometimes I start it large like 1e-3 and then once it stops performing I drop it by a factor of 10. I hope this helps :D

", "votes": 0, "replies": [] } ] } ] }, { "id": 3190744, "author_name": "Yksin Young", "author_url": "", "post_date": "2025-05-01T03:50:32.313000", "content": "

I think the problem is about the \"optimal threshold\", there may be not so much difference in the yolo, but the LB is very sensitive to threshold..

", "votes": 1, "replies": [ { "id": 3191083, "author_name": "eikyou", "author_url": "", "post_date": "2025-05-01T11:15:21.090000", "content": "

@yksinyoung I tried various thresholds and the results are much worse. First model on the threshold 0.3 shows performance 0.796, the second one on the same threshold shows 0.667, which is obviously much worse than the first model and only a bit better than the second model on optimal threshold from yolo training. ;)

", "votes": 0, "replies": [ { "id": 3201617, "author_name": "Heeler-Deer", "author_url": "", "post_date": "2025-05-14T06:39:12.213000", "content": "

I used the setting of iou=0.5 during training. When inferencing on the same model, the iou is changed from 0.45 to 0.3, and the lb score dropped.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3192857, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-05-03T13:01:34.673000", "content": "

Hey @eikyou, did you used seeds in your training?
\nI've trained my models with seeds and I'm going to try retraining them nd submitting to see what happens. I'll tell you later how it goes.
\nA similar thing is happening to me, trained a 5 fold cv (yolo10x) and got models scoring from 0.725 to 0.792.

", "votes": 2, "replies": [ { "id": 3192867, "author_name": "eikyou", "author_url": "", "post_date": "2025-05-03T13:23:57.160000", "content": "

I didn't fix seed in YOLO itself, and I didn't use CV, i used hold out. Guess i just randomly got a good model ;) Will focus on CV scheme

", "votes": 1, "replies": [] }, { "id": 3193557, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-05-04T15:03:06.553000", "content": "

How many epochs do you typically train your model?

", "votes": 0, "replies": [ { "id": 3193573, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-05-04T15:57:54.830000", "content": "

@sjtuwangshuo I usually train for 50 epochs. But I noticed that my models might be overfitting, or that selecting best.pt automatically isn't as optimal as I thought. In one of my folds, I got the best.pt file at different epochs: around 28, 40, and the final one at 50. The respectives scores were 0.816, 0.752, and 0.744.

\n

I had assumed Ultralytics saves the model with the best validation loss, but now I'm not so sure (or that best validation loss = best lb). I need to take a look into that. This experiment wasn’t really planned, I just didn’t have time to train all the folds and didn't want to lose daily submissions.

", "votes": 1, "replies": [] }, { "id": 3193578, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-05-04T16:19:56.293000", "content": "

Thanks for sharing your interesting findings. In Ultralytics, the best checkpoint is usually saved based on a weighted metric: 0.1 × mAP@0.5 + 0.9 × mAP@0.5:0.95 (as referenced here).

\n

Besides, how did you decide which epoch's checkpoint to submit (e.g., 28, 40, or 50)? Does the checkpoint at epoch 28 have the lowest dfl loss?

", "votes": 0, "replies": [] }, { "id": 3193626, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-05-04T17:48:24.833000", "content": "

Thanks @sjtuwangshuo. It wasn’t a thought out decision. I just used whatever checkpoint I had ready at the time to use my daily submissions. Unfortunaly, my \"val_period\" was set to 5, so the training plot it's not that useful to find what epoch was best.

\n

The checkpoint at epoch 28 didn’t have the lowest loss overall, but the validation loss was lower than the training loss, wich is good. Also, there’s a big shift in the loss curve around epoch 40. that’s when mosaic augmentation stops.
\nMaybe if I used a later checkpoint before closing mosaic it would have a better lb than a chekpoint in epoch 28, I still have to test this.

\n

Here's the training curve below so you can see what I mean.

\n

\"training

", "votes": 1, "replies": [] }, { "id": 3193656, "author_name": "eikyou", "author_url": "", "post_date": "2025-05-04T18:49:59.583000", "content": "

btw, i noticed that better models get lower optimal thresholds for F1 on validation. 0.2~ for good models and 0.4~ for much worse models. Maybe it's a coincidence, but still.
\nAlso i noticed that optimal threshold for LB is 0.3~

", "votes": 2, "replies": [] }, { "id": 3195257, "author_name": "", "author_url": "", "post_date": "2025-05-06T19:59:53.093000", "content": "", "votes": 0, "replies": [] }, { "id": 3197016, "author_name": "shiba-inu", "author_url": "", "post_date": "2025-05-07T18:11:37.047000", "content": "

@sersasj
\nThanks for suggesting so interesting finding!
\nClose mosic may cause overfittting…
\nbtw, What did you set mosaic rate?
\n( I think yolo set mosaic as 1.0 in default setting)

", "votes": 0, "replies": [] }, { "id": 3197028, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-05-07T18:34:20.073000", "content": "

Thanks @npinpi! It was set to 1.0

", "votes": 0, "replies": [] } ] } ] }, { "id": 3200223, "author_name": "mohanapavan bezawada", "author_url": "", "post_date": "2025-05-12T09:16:19.147000", "content": "

hey, can anyone tell me from where it is considered as high LR and from where it is considered as low LR. which is the avg LR like the safest number to start with. and how initial learning rate and final learning rate works.

", "votes": 0, "replies": [ { "id": 3200337, "author_name": "eikyou", "author_url": "", "post_date": "2025-05-12T12:31:23.133000", "content": "

@mohanapavanbezawada 1e-4 or 3e-4 are usually ok to start with

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "3194267": "### FYI.Ultralytics YOLO reproducibility\n* The default setting is **deterministic is false.**\n```\n# ultralytics/ultralytics/utils/torch_utils.py\ndef init_seeds(seed=0, deterministic=False):\n...\n```\n* Therefore, to ensure reproducibility, **deterministic must be set to True** in the Torch implementation.\n* According to the Ultralytics src and manual, it can be set up as follows:\n```\n# YOLO deterministic setting\nfrom ultralytics.utils.torch_utils import init_seeds\ninit_seeds(seed=42, deterministic=True)\n\n# Check torch deterministic\nimport torch.backends.cudnn as cudnn\nprint(cudnn.deterministic)\n\n⇒True\n```\n\nP.S I have limited computing resources, so this is just an observation based on looking at the source, but I hope it will be helpful.", "3190706": "I trained YOLO11 2 times on the same data and the same parameters (don't ask me why i did this). And I got extremely different results. Both optimal F1 threshold and LB are absolutely different. \nI do understand that training itself is kinda random and augmentations are random too. I am wondering if such a behaviour is normal or am I missing something?", "3192933": "With a fixed seed, I consistently obtain the same results when using the same training parameters. The training code is adapted from the Kaggle YOLO baseline.", "3191256": "Do you have an early stopping condition? It’s possible your LR is too high or too low so one model got stuck in a local minima while the other moved closer to the global minima. Maybe try tuning your LR or using something like a decaying LR? I’m guessing one model plateaued early and hit an early stopping condition whereas the other had a better initialization and was fortunate to find a better “path”", "3190744": "I think the problem is about the \"optimal threshold\", there may be not so much difference in the yolo, but the LB is very sensitive to threshold.. ", "3192857": "Hey @eikyou, did you used seeds in your training? \nI've trained my models with seeds and I'm going to try retraining them nd submitting to see what happens. I'll tell you later how it goes.\nA similar thing is happening to me, trained a 5 fold cv (yolo10x) and got models scoring from 0.725 to 0.792.", "3200223": "hey, can anyone tell me from where it is considered as high LR and from where it is considered as low LR. which is the avg LR like the safest number to start with. and how initial learning rate and final learning rate works. " } }