Which learning rates are good to train these models ? Like @appian achieved decent score by training for 7 hours(se_resnext50). Is it because of higher learning rate choice ?
\n\nI tried using differential learning rates from 1e-5 to 1e-2. But after 15 epochs, it isn't reaching a good score.
\n\nImage size: 320x320
", "messages": [ { "id": "535752", "postDate": "05/23/2019 11:52:09", "content": "Which learning rates are good to train these models ? Like @appian achieved decent score by training for 7 hours(se_resnext50). Is it because of higher learning rate choice ?
\n\nI tried using differential learning rates from 1e-5 to 1e-2. But after 15 epochs, it isn't reaching a good score.
\n\nImage size: 320x320
", "rawMarkdown": "Which learning rates are good to train these models ? Like @appian achieved decent score by training for 7 hours(se_resnext50). Is it because of higher learning rate choice ?\n\nI tried using differential learning rates from 1e-5 to 1e-2. But after 15 epochs, it isn't reaching a good score.\n\nImage size: 320x320", "votes": null }, { "id": "535911", "postDate": "05/23/2019 16:01:37", "content": "Are you using pytorch or fastai? I think it depends also on training strategies. I'll give you an example. In stage 1, train model with 224x224 image size (batch size 64) and a large learning rate until it overfits. Next stage, train it with 320x320 image size (batch size 32) and apply differential learning rates. You may also proceed to 416x416 or 384x384 with batch size 16. Do note also you have to declare Variable Input by collate function and add it to DataLoader:
\n\nexample of collate function:
\n\ndef train_collate(batch):
\n\nbatch_size = len(batch)\nimages = []\nlabels = []\nfor b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.extend(batch[b][1])\nimages = torch.stack(images, 0)\nlabels = torch.from_numpy(np.array(labels))\nreturn images, labels\ndef valid_collate(batch):
\n\nbatch_size = len(batch)\nimages = []\nlabels = []\nnames = []\nfor b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.append(batch[b][1])\n names.append(batch[b][2])\nimages = torch.stack(images, 0)\nlabels = torch.from_numpy(np.array(labels))\nreturn images, labels, names\ndataloader_train = DataLoader(train_dataset, shuffle=True, drop_last=True, batch_size=batch_size, num_workers=16, collate_fn=train_collate)\ndataloader_valid = DataLoader(valid_dataset, shuffle=False, batch_size=batch_size * 2, num_workers=8, collate_fn=valid_collate)
\n\nAlso, there are things we need to consider like network modification, proper loss function, etc. Hope this helps! :-)
", "rawMarkdown": "Are you using pytorch or fastai? I think it depends also on training strategies. I'll give you an example. In stage 1, train model with 224x224 image size (batch size 64) and a large learning rate until it overfits. Next stage, train it with 320x320 image size (batch size 32) and apply differential learning rates. You may also proceed to 416x416 or 384x384 with batch size 16. Do note also you have to declare Variable Input by collate function and add it to DataLoader:\n\nexample of collate function:\n\ndef train_collate(batch):\n\n batch_size = len(batch)\n images = []\n labels = []\n for b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.extend(batch[b][1])\n images = torch.stack(images, 0)\n labels = torch.from_numpy(np.array(labels))\n return images, labels\n\ndef valid_collate(batch):\n\n batch_size = len(batch)\n images = []\n labels = []\n names = []\n for b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.append(batch[b][1])\n names.append(batch[b][2])\n images = torch.stack(images, 0)\n labels = torch.from_numpy(np.array(labels))\n return images, labels, names\n\ndataloader_train = DataLoader(train_dataset, shuffle=True, drop_last=True, batch_size=batch_size, num_workers=16, collate_fn=train_collate)\ndataloader_valid = DataLoader(valid_dataset, shuffle=False, batch_size=batch_size * 2, num_workers=8, collate_fn=valid_collate)\n\nAlso, there are things we need to consider like network modification, proper loss function, etc. Hope this helps! :-)", "votes": null }, { "id": "535913", "postDate": "05/23/2019 16:05:05", "content": "May I ask, what batch size did you use?
", "rawMarkdown": "May I ask, what batch size did you use?", "votes": null }, { "id": "536298", "postDate": "05/24/2019 08:04:07", "content": "Thanks @projdev. I am using fastai. I haven't tried training this way yet. Let me come back with results :)
", "rawMarkdown": "Thanks @projdev. I am using fastai. I haven't tried training this way yet. Let me come back with results :)", "votes": null }, { "id": "536301", "postDate": "05/24/2019 08:07:20", "content": "I used batch size 48 for se_resnext50.
", "rawMarkdown": "I used batch size 48 for se_resnext50.", "votes": null }, { "id": "536689", "postDate": "05/25/2019 02:33:32", "content": "hi @harshthaker Are you using bce loss or focal loss in faisai? When I use bce loss, the resultis very bad, do you know how to solve it?
", "rawMarkdown": "hi @harshthaker Are you using bce loss or focal loss in faisai? When I use bce loss, the resultis very bad, do you know how to solve it?", "votes": null }, { "id": "536771", "postDate": "05/25/2019 08:21:31", "content": "Hi @guitar123 , I am using focal loss. It works just okay. Check this discussion for some other details: https://www.kaggle.com/c/imet-2019-fgvc6/discussion/88370#latest-519759
\n\nFor some participants, BCE is also doing fine.
", "rawMarkdown": "Hi @guitar123 , I am using focal loss. It works just okay. Check this discussion for some other details: https://www.kaggle.com/c/imet-2019-fgvc6/discussion/88370#latest-519759\n\nFor some participants, BCE is also doing fine.", "votes": null }, { "id": "536857", "postDate": "05/25/2019 12:55:52", "content": "Hi. If I want to train in your way, should I change the image size in collate function? Thank you.
", "rawMarkdown": "Hi. If I want to train in your way, should I change the image size in collate function? Thank you.", "votes": null } ], "comments": [ { "id": 535911, "author_name": "projdev", "author_url": "", "post_date": "05/23/2019 16:01:37", "content": "Are you using pytorch or fastai? I think it depends also on training strategies. I'll give you an example. In stage 1, train model with 224x224 image size (batch size 64) and a large learning rate until it overfits. Next stage, train it with 320x320 image size (batch size 32) and apply differential learning rates. You may also proceed to 416x416 or 384x384 with batch size 16. Do note also you have to declare Variable Input by collate function and add it to DataLoader:
\n\nexample of collate function:
\n\ndef train_collate(batch):
\n\nbatch_size = len(batch)\nimages = []\nlabels = []\nfor b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.extend(batch[b][1])\nimages = torch.stack(images, 0)\nlabels = torch.from_numpy(np.array(labels))\nreturn images, labels\ndef valid_collate(batch):
\n\nbatch_size = len(batch)\nimages = []\nlabels = []\nnames = []\nfor b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.append(batch[b][1])\n names.append(batch[b][2])\nimages = torch.stack(images, 0)\nlabels = torch.from_numpy(np.array(labels))\nreturn images, labels, names\ndataloader_train = DataLoader(train_dataset, shuffle=True, drop_last=True, batch_size=batch_size, num_workers=16, collate_fn=train_collate)\ndataloader_valid = DataLoader(valid_dataset, shuffle=False, batch_size=batch_size * 2, num_workers=8, collate_fn=valid_collate)
\n\nAlso, there are things we need to consider like network modification, proper loss function, etc. Hope this helps! :-)
", "votes": null, "replies": [ { "id": 536298, "author_name": "harshthaker", "author_url": "", "post_date": "05/24/2019 08:04:07", "content": "Thanks @projdev. I am using fastai. I haven't tried training this way yet. Let me come back with results :)
", "votes": null, "replies": [] }, { "id": 536689, "author_name": "guitar123", "author_url": "", "post_date": "05/25/2019 02:33:32", "content": "hi @harshthaker Are you using bce loss or focal loss in faisai? When I use bce loss, the resultis very bad, do you know how to solve it?
", "votes": null, "replies": [] }, { "id": 536771, "author_name": "harshthaker", "author_url": "", "post_date": "05/25/2019 08:21:31", "content": "Hi @guitar123 , I am using focal loss. It works just okay. Check this discussion for some other details: https://www.kaggle.com/c/imet-2019-fgvc6/discussion/88370#latest-519759
\n\nFor some participants, BCE is also doing fine.
", "votes": null, "replies": [] }, { "id": 536857, "author_name": "saladjay", "author_url": "", "post_date": "05/25/2019 12:55:52", "content": "Hi. If I want to train in your way, should I change the image size in collate function? Thank you.
", "votes": null, "replies": [] } ] }, { "id": 535913, "author_name": "saladjay", "author_url": "", "post_date": "05/23/2019 16:05:05", "content": "May I ask, what batch size did you use?
", "votes": null, "replies": [ { "id": 536301, "author_name": "harshthaker", "author_url": "", "post_date": "05/24/2019 08:07:20", "content": "I used batch size 48 for se_resnext50.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "535752": "Which learning rates are good to train these models ? Like @appian achieved decent score by training for 7 hours(se_resnext50). Is it because of higher learning rate choice ?\n\nI tried using differential learning rates from 1e-5 to 1e-2. But after 15 epochs, it isn't reaching a good score.\n\nImage size: 320x320", "535911": "Are you using pytorch or fastai? I think it depends also on training strategies. I'll give you an example. In stage 1, train model with 224x224 image size (batch size 64) and a large learning rate until it overfits. Next stage, train it with 320x320 image size (batch size 32) and apply differential learning rates. You may also proceed to 416x416 or 384x384 with batch size 16. Do note also you have to declare Variable Input by collate function and add it to DataLoader:\n\nexample of collate function:\n\ndef train_collate(batch):\n\n batch_size = len(batch)\n images = []\n labels = []\n for b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.extend(batch[b][1])\n images = torch.stack(images, 0)\n labels = torch.from_numpy(np.array(labels))\n return images, labels\n\ndef valid_collate(batch):\n\n batch_size = len(batch)\n images = []\n labels = []\n names = []\n for b in range(batch_size):\n if batch[b][0] is None:\n continue\n else:\n images.extend(batch[b][0])\n labels.append(batch[b][1])\n names.append(batch[b][2])\n images = torch.stack(images, 0)\n labels = torch.from_numpy(np.array(labels))\n return images, labels, names\n\ndataloader_train = DataLoader(train_dataset, shuffle=True, drop_last=True, batch_size=batch_size, num_workers=16, collate_fn=train_collate)\ndataloader_valid = DataLoader(valid_dataset, shuffle=False, batch_size=batch_size * 2, num_workers=8, collate_fn=valid_collate)\n\nAlso, there are things we need to consider like network modification, proper loss function, etc. Hope this helps! :-)", "535913": "May I ask, what batch size did you use?", "536298": "Thanks @projdev. I am using fastai. I haven't tried training this way yet. Let me come back with results :)", "536301": "I used batch size 48 for se_resnext50.", "536689": "hi @harshthaker Are you using bce loss or focal loss in faisai? When I use bce loss, the resultis very bad, do you know how to solve it?", "536771": "Hi @guitar123 , I am using focal loss. It works just okay. Check this discussion for some other details: https://www.kaggle.com/c/imet-2019-fgvc6/discussion/88370#latest-519759\n\nFor some participants, BCE is also doing fine.", "536857": "Hi. If I want to train in your way, should I change the image size in collate function? Thank you." }, "source": "meta" }