\nLookahead k=10, alpha=0.5
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2Fb1e80b2c9002fd37c0176d5b7cd72c73%2FLookahead10_AnCosW10.png?generation=1598174400338373&alt=media\")
Have tried some schedulers and settings, all with various epochs, if I see a pattern a stop and run next, maybe not the best but time efficient. Below are some of the results, includes only those that have been shown to work reasonably well.
\nSo far StepLR step_size=10 alpha=0.5 or StepLR step_size=5 alpha=0.6 seems to work well, but will test more combinations.
\nI use the new released optimizer AdamP with weight_decay=5e-3.
\nhttps://clovaai.github.io/AdamP/
Model: Efficientnet_b3_ns with dropout 0.5 B64 dim 224x224 and also interpolate 224x224 in the model.
\nCosineAnnealingWarmRestarts T_0=20
\nLookahead k=10, alpha=0.5
\n
CosineAnnealingWarmRestarts T_0=10
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2Fa5f65740f9d1c286263024abce8f31f4%2FNesterov.Ann10ejLookAh.png?generation=1598172974712618&alt=media\")
StepLR step_size=20, gamma=0.5
\nNesterov
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F3d320566bb5fc8e7c3e8db3b68cd8069%2FNesterovStepLR20a5.png?generation=1598172885356940&alt=media\")
StepLR step_size=5, gamma=0.6
\nNesterov
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F008808f1717804d808aaea66be7ac13d%2FNesterovStepLR-5-a6.png?generation=1598173076202154&alt=media\")
StepLR step_size=10, gamma=0.5
\nNesterov
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F272ec41021c88ece68520c3f3f3408fd%2FStep10a5Nest.png?generation=1598174591255062&alt=media\")
Have tried some schedulers and settings, all with various epochs, if I see a pattern a stop and run next, maybe not the best but time efficient. Below are some of the results, includes only those that have been shown to work reasonably well.
\nSo far StepLR step_size=10 alpha=0.5 or StepLR step_size=5 alpha=0.6 seems to work well, but will test more combinations.
\nI use the new released optimizer AdamP with weight_decay=5e-3.
\nhttps://clovaai.github.io/AdamP/
Model: Efficientnet_b3_ns with dropout 0.5 B64 dim 224x224 and also interpolate 224x224 in the model.
\nCosineAnnealingWarmRestarts T_0=20
\nLookahead k=10, alpha=0.5
\n
CosineAnnealingWarmRestarts T_0=10
\n
StepLR step_size=20, gamma=0.5
\nNesterov
\n
StepLR step_size=5, gamma=0.6
\nNesterov
\n
StepLR step_size=10, gamma=0.5
\nNesterov
\n
Thank you, it was informative 👍
", "rawMarkdown": "Thank you, it was informative 👍", "votes": null }, { "id": "984997", "postDate": "08/25/2020 12:32:00", "content": "Hey @kirderf, what batch_size do you use?
", "rawMarkdown": "Hey @kirderf, what batch_size do you use?", "votes": null }, { "id": "985001", "postDate": "08/25/2020 12:36:02", "content": "64, I wrote B64, maybe a little cryptic
", "rawMarkdown": "64, I wrote B64, maybe a little cryptic", "votes": null }, { "id": "985307", "postDate": "08/25/2020 16:20:53", "content": "Thanks , I also tried same thing enet b2 got almost same losses. But still not able to beat baseline model. Were you able to beat it using efficient net ?. You can see me training in https://www.kaggle.com/rsinda/training-efficientnet-model . Am I doing it right ?
", "rawMarkdown": "Thanks , I also tried same thing enet b2 got almost same losses. But still not able to beat baseline model. Were you able to beat it using efficient net ?. You can see me training in https://www.kaggle.com/rsinda/training-efficientnet-model . Am I doing it right ?", "votes": null }, { "id": "985395", "postDate": "08/25/2020 17:43:57", "content": "Yes this is not easy, still testing, can't say I found a good starting point yet. I will try different sound and image quailty/dimensions, see what happens, also other models. For training one can try to use various of Image Augmentations.
\nThe time to load training data and the limit of 2h for submit is a challenge.
Use below to get a better progress view and turn off the valid progress(progress_bar=False), to much validation info in a short time.
\n while not manager.stop_trigger:\n model.train()\n progress_bar = tqdm_notebook(train_loader)\n for batch_idx, (data, target) in enumerate(progress_bar):\n with manager.run_iteration():\n data, target = data.to(device), target.to(device)\n optimizer.zero_grad()\n output = model(data)\n loss = loss_func(output, target)\n progress_bar.set_description(f'train/loss: {loss.item():.6f}')\n ppe.reporting.report({'train/loss': loss.item()})\n loss.backward()\n optimizer.step()\n scheduler.step()\nA better option:
\nAdamW
\nCosineAnnealingWarmRestarts T_0: 1 T_mult: 2 eta_min: 0.0005
\nImage + spectrogram augmentations
\n(224x547)
\nOnly tried with the smallest EfffnetB0ns so far.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F2cd5c58126e489771c6d52de01a9c96d%2Floss%20(20).png?generation=1598800446474628&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F239729f81bc33f01555aeb6dee855867%2Flr%20(19).png?generation=1598800460018029&alt=media\")
I was surprised to see you use B3 on small image size. Thanks for sharing your experiments.
", "rawMarkdown": "I was surprised to see you use B3 on small image size. Thanks for sharing your experiments.", "votes": null }, { "id": "992078", "postDate": "08/30/2020 22:18:19", "content": "used B3 as a mid range model for the testing/benchmark, but it gave a boost to the ensemble.
\nUsally keep an eye on below
# (width_coefficient, depth_coefficient, resolution, dropout_rate)\n 'efficientnet-b0': (1.0, 1.0, 224, 0.2),\n 'efficientnet-b1': (1.0, 1.1, 240, 0.2),\n 'efficientnet-b2': (1.1, 1.2, 260, 0.3),\n 'efficientnet-b3': (1.2, 1.4, 300, 0.3),\n 'efficientnet-b4': (1.4, 1.8, 380, 0.4),\n 'efficientnet-b5': (1.6, 2.2, 456, 0.4),\n 'efficientnet-b6': (1.8, 2.6, 528, 0.5),\n 'efficientnet-b7': (2.0, 3.1, 600, 0.5),\n 'efficientnet-b8': (2.2, 3.6, 672, 0.5),\n 'efficientnet-l2': (4.3, 5.3, 800, 0.5),\nThere is a lot left to discover and test in this competition. Will see if one has time to reach an OK result before the deadline.
", "rawMarkdown": "used B3 as a mid range model for the testing/benchmark, but it gave a boost to the ensemble.\nUsally keep an eye on below\n\n # (width_coefficient, depth_coefficient, resolution, dropout_rate)\n 'efficientnet-b0': (1.0, 1.0, 224, 0.2),\n 'efficientnet-b1': (1.0, 1.1, 240, 0.2),\n 'efficientnet-b2': (1.1, 1.2, 260, 0.3),\n 'efficientnet-b3': (1.2, 1.4, 300, 0.3),\n 'efficientnet-b4': (1.4, 1.8, 380, 0.4),\n 'efficientnet-b5': (1.6, 2.2, 456, 0.4),\n 'efficientnet-b6': (1.8, 2.6, 528, 0.5),\n 'efficientnet-b7': (2.0, 3.1, 600, 0.5),\n 'efficientnet-b8': (2.2, 3.6, 672, 0.5),\n 'efficientnet-l2': (4.3, 5.3, 800, 0.5),\n\nThere is a lot left to discover and test in this competition. Will see if one has time to reach an OK result before the deadline.", "votes": null }, { "id": "1003783", "postDate": "09/09/2020 08:42:22", "content": "Better late than never, I will now use this metric instead, seems like a good work environment.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2Ffffd3956672398ed985e77fdd231349b%2Floss%20(26).png?generation=1599640672197611&alt=media\")
f1_score with sample - competition metric
\nrow_wise_f1_score_micro_numpy- same as above, but will stay for a while just to be safe (?)
\nf1_score with micro - just to follow the correlation.
And I'll run a global threshold optimization in every validation step, hopefully also a local optimization per label if I succeed with the coding.
", "rawMarkdown": "Better late than never, I will now use this metric instead, seems like a good work environment.\n\n.png?generation=1599640672197611&alt=media)\n\nf1_score with sample - competition metric\nrow_wise_f1_score_micro_numpy- same as above, but will stay for a while just to be safe (?)\nf1_score with micro - just to follow the correlation.\n\nAnd I'll run a global threshold optimization in every validation step, hopefully also a local optimization per label if I succeed with the coding.", "votes": null }, { "id": "1005936", "postDate": "09/10/2020 22:27:50", "content": "EffB3Imagenet
\nAdamW
\nCosineAnnealingWarmRestarts T_0: batch update instead of epoch and with cycle of 2 epochs and changed lr with 1e-1 when converge.
\nData - upsampling and picked only sound rating >=3.
\nRandom batchcutout+mixup, dropout 0.3.
This shows training after 30 epochs.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F7fb342fc9375da89407669edc836c570%2Floss%20(31).png?generation=1599776625210733&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3924325%2F8b7298063c7387fa49483007e36a1f93%2Flr%20(28).png?generation=1599776417747942&alt=media\")
Thank you, it was informative 👍
", "votes": null, "replies": [] }, { "id": 984997, "author_name": "rsinda", "author_url": "", "post_date": "08/25/2020 12:32:00", "content": "Hey @kirderf, what batch_size do you use?
", "votes": null, "replies": [ { "id": 985001, "author_name": "kirderf", "author_url": "", "post_date": "08/25/2020 12:36:02", "content": "64, I wrote B64, maybe a little cryptic
", "votes": null, "replies": [] }, { "id": 985307, "author_name": "rsinda", "author_url": "", "post_date": "08/25/2020 16:20:53", "content": "Thanks , I also tried same thing enet b2 got almost same losses. But still not able to beat baseline model. Were you able to beat it using efficient net ?. You can see me training in https://www.kaggle.com/rsinda/training-efficientnet-model . Am I doing it right ?
", "votes": null, "replies": [] }, { "id": 985395, "author_name": "kirderf", "author_url": "", "post_date": "08/25/2020 17:43:57", "content": "Yes this is not easy, still testing, can't say I found a good starting point yet. I will try different sound and image quailty/dimensions, see what happens, also other models. For training one can try to use various of Image Augmentations.
\nThe time to load training data and the limit of 2h for submit is a challenge.
Use below to get a better progress view and turn off the valid progress(progress_bar=False), to much validation info in a short time.
\n while not manager.stop_trigger:\n model.train()\n progress_bar = tqdm_notebook(train_loader)\n for batch_idx, (data, target) in enumerate(progress_bar):\n with manager.run_iteration():\n data, target = data.to(device), target.to(device)\n optimizer.zero_grad()\n output = model(data)\n loss = loss_func(output, target)\n progress_bar.set_description(f'train/loss: {loss.item():.6f}')\n ppe.reporting.report({'train/loss': loss.item()})\n loss.backward()\n optimizer.step()\n scheduler.step()\nA better option:
\nAdamW
\nCosineAnnealingWarmRestarts T_0: 1 T_mult: 2 eta_min: 0.0005
\nImage + spectrogram augmentations
\n(224x547)
\nOnly tried with the smallest EfffnetB0ns so far.
\n
I was surprised to see you use B3 on small image size. Thanks for sharing your experiments.
", "votes": null, "replies": [] }, { "id": 992078, "author_name": "kirderf", "author_url": "", "post_date": "08/30/2020 22:18:19", "content": "used B3 as a mid range model for the testing/benchmark, but it gave a boost to the ensemble.
\nUsally keep an eye on below
# (width_coefficient, depth_coefficient, resolution, dropout_rate)\n 'efficientnet-b0': (1.0, 1.0, 224, 0.2),\n 'efficientnet-b1': (1.0, 1.1, 240, 0.2),\n 'efficientnet-b2': (1.1, 1.2, 260, 0.3),\n 'efficientnet-b3': (1.2, 1.4, 300, 0.3),\n 'efficientnet-b4': (1.4, 1.8, 380, 0.4),\n 'efficientnet-b5': (1.6, 2.2, 456, 0.4),\n 'efficientnet-b6': (1.8, 2.6, 528, 0.5),\n 'efficientnet-b7': (2.0, 3.1, 600, 0.5),\n 'efficientnet-b8': (2.2, 3.6, 672, 0.5),\n 'efficientnet-l2': (4.3, 5.3, 800, 0.5),\nThere is a lot left to discover and test in this competition. Will see if one has time to reach an OK result before the deadline.
", "votes": null, "replies": [] } ] }, { "id": 1003783, "author_name": "kirderf", "author_url": "", "post_date": "09/09/2020 08:42:22", "content": "Better late than never, I will now use this metric instead, seems like a good work environment.
\n
f1_score with sample - competition metric
\nrow_wise_f1_score_micro_numpy- same as above, but will stay for a while just to be safe (?)
\nf1_score with micro - just to follow the correlation.
And I'll run a global threshold optimization in every validation step, hopefully also a local optimization per label if I succeed with the coding.
", "votes": null, "replies": [] }, { "id": 1005936, "author_name": "kirderf", "author_url": "", "post_date": "09/10/2020 22:27:50", "content": "EffB3Imagenet
\nAdamW
\nCosineAnnealingWarmRestarts T_0: batch update instead of epoch and with cycle of 2 epochs and changed lr with 1e-1 when converge.
\nData - upsampling and picked only sound rating >=3.
\nRandom batchcutout+mixup, dropout 0.3.
This shows training after 30 epochs.
\n
\n