Hi,\nsince it is early in the competition let's talk about the general approach to this problem.
\n\nI will start with two obvious ones:
\n\nTrain a network from scratch or from a pretrained model. So far I tried resnet-18 since it takes about 2 hours to train a single epoch (1 image per product). Convergence is slow, but reaches 60% Acc. Bigger models will do better, but without a GPU-Workstation even pretrained networks will take weeks on my single GTX1080TI.
Use pretrained networks as feature extractors and train a classifier (e.g. Random Forests) on top of these features. I did not try this yet, but this enables even users with slow GPUs to use big networks (for example Inception-V3).
What's your approach? Do you think computing power will win this competition?
", "messages": [ { "id": "224507", "postDate": "09/26/2017 16:11:35", "content": "Hi,\nsince it is early in the competition let's talk about the general approach to this problem.
\n\nI will start with two obvious ones:
\n\nTrain a network from scratch or from a pretrained model. So far I tried resnet-18 since it takes about 2 hours to train a single epoch (1 image per product). Convergence is slow, but reaches 60% Acc. Bigger models will do better, but without a GPU-Workstation even pretrained networks will take weeks on my single GTX1080TI.
Use pretrained networks as feature extractors and train a classifier (e.g. Random Forests) on top of these features. I did not try this yet, but this enables even users with slow GPUs to use big networks (for example Inception-V3).
What's your approach? Do you think computing power will win this competition?
", "rawMarkdown": "Hi,\nsince it is early in the competition let's talk about the general approach to this problem.\n\nI will start with two obvious ones:\n\n1. Train a network from scratch or from a pretrained model. So far I tried resnet-18 since it takes about 2 hours to train a single epoch (1 image per product). Convergence is slow, but reaches 60% Acc. Bigger models will do better, but without a GPU-Workstation even pretrained networks will take weeks on my single GTX1080TI.\n\n2. Use pretrained networks as feature extractors and train a classifier (e.g. Random Forests) on top of these features. I did not try this yet, but this enables even users with slow GPUs to use big networks (for example Inception-V3).\n\nWhat's your approach? Do you think computing power will win this competition?", "votes": null }, { "id": "224545", "postDate": "09/26/2017 18:28:54", "content": "The current top leaderboard score (0.70773) is quite impressive already for an image classifier. I think state-of-the-art on ImageNet is about 80% (for top-1 accuracy)? So the current results are not too far off from that. (And to be fair, for many products you actually have 4 chances to guess right, so that makes this slightly easier. On the other hand, there are more categories and they are imbalanced, making this slightly harder than ImageNet.)
\n\nMy approach is not to compete on computing power. There will always be someone who is able or willing to buy more computing power than I am. Using the same models as everyone else also isn't very exciting to me. So instead, I'm treating this competition as a way to experiment with different training approaches, etc. \"Work smarter, not harder.\" :-D
", "rawMarkdown": "The current top leaderboard score (0.70773) is quite impressive already for an image classifier. I think state-of-the-art on ImageNet is about 80% (for top-1 accuracy)? So the current results are not too far off from that. (And to be fair, for many products you actually have 4 chances to guess right, so that makes this slightly easier. On the other hand, there are more categories and they are imbalanced, making this slightly harder than ImageNet.)\n\nMy approach is not to compete on computing power. There will always be someone who is able or willing to buy more computing power than I am. Using the same models as everyone else also isn't very exciting to me. So instead, I'm treating this competition as a way to experiment with different training approaches, etc. \"Work smarter, not harder.\" :-D", "votes": null }, { "id": "224573", "postDate": "09/26/2017 20:02:45", "content": "Thank you for your response. I am wondering whether the current leaderboard result are achieved with by training models or feature extractions since there has not been much time to train yet.
\n\nI think \"working smart\" is nice, but in the end \"working harder\" will win this, since bigger models seem to be SOTA in similiar tasks :/
", "rawMarkdown": "Thank you for your response. I am wondering whether the current leaderboard result are achieved with by training models or feature extractions since there has not been much time to train yet.\n\nI think \"working smart\" is nice, but in the end \"working harder\" will win this, since bigger models seem to be SOTA in similiar tasks :/", "votes": null }, { "id": "224975", "postDate": "09/28/2017 02:12:18", "content": "I have tried using resnet 50 for finetuning. During training, the train loss sometimes jump to high and accuracy drop to very low. I think that a modification of loss function is needed, since there are too many categories and highly imbalanced data.
\n\nBesides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).
", "rawMarkdown": "I have tried using resnet 50 for finetuning. During training, the train loss sometimes jump to high and accuracy drop to very low. I think that a modification of loss function is needed, since there are too many categories and highly imbalanced data.\n\nBesides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).", "votes": null }, { "id": "225013", "postDate": "09/28/2017 03:50:44", "content": "i met the similar problem, i also use resnet 50 for finetuning. i found that the training acc is high,but the val acc is extremely low, batch_size=1024, lr = 0.01
", "rawMarkdown": "i met the similar problem, i also use resnet 50 for finetuning. i found that the training acc is high,but the val acc is extremely low, batch_size=1024, lr = 0.01", "votes": null }, { "id": "225056", "postDate": "09/28/2017 05:52:08", "content": "\n\n\nBesides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).
\n
Can you explain what you mean by this?\nI currently have the opposite experience: Training is slow, but val-acc is like expected.
", "rawMarkdown": "> Besides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).\n\nCan you explain what you mean by this?\nI currently have the opposite experience: Training is slow, but val-acc is like expected.", "votes": null }, { "id": "225059", "postDate": "09/28/2017 05:56:46", "content": "@iFighting I'm also finetuning resnet50. I believe I was/am seeing
\n\n\n\n\ni found that the training acc is high,but the val acc is extremely low
\n
since my val split had products with only one image. Trying to test against the lb asap so I can try this hypothesis
", "rawMarkdown": "iFighting I'm also finetuning resnet50. I believe I was/am seeing \n\n> i found that the training acc is high,but the val acc is extremely low\n\nsince my val split had products with only one image. Trying to test against the lb asap so I can try this hypothesis", "votes": null }, { "id": "225422", "postDate": "09/29/2017 02:19:45", "content": "what is your loss function?
", "rawMarkdown": "what is your loss function?", "votes": null }, { "id": "225430", "postDate": "09/29/2017 03:09:35", "content": "@Tim Joseph, I extract images of one category to train the net, and the result seems ok, i can tell that my net and loss function is not too wrong. They are just not much suitable for so many categories and such data distribution.
", "rawMarkdown": "Tim Joseph, I extract images of one category to train the net, and the result seems ok, i can tell that my net and loss function is not too wrong. They are just not much suitable for so many categories and such data distribution.", "votes": null }, { "id": "225861", "postDate": "09/30/2017 06:46:41", "content": "@CSAdu my loss function is cross entropy:
\n\n` # loss ----------------------------------------
\n\ndef criterion(logits, labels):
\n\nloss = nn.CrossEntropyLoss()(logits, Variable(labels))\n\nreturn loss`\nNow I'm trying to fine-tune on pre-trained models (ResNet-50, Xception, etc.). But both train&val acc looks bad (~40%), I think it's due to the huge amount of params in the last layer (from the last pooling layer to dense layer for prediction usually have more than 10M params, i.e. 5270*2K or 3K?)
\n\nSo what about changing the dense layer to other layers (using less params) or change loss function to meet the requirement for this extreme classification problem?
", "rawMarkdown": "Now I'm trying to fine-tune on pre-trained models (ResNet-50, Xception, etc.). But both train&val acc looks bad (~40%), I think it's due to the huge amount of params in the last layer (from the last pooling layer to dense layer for prediction usually have more than 10M params, i.e. 5270*2K or 3K?)\n\nSo what about changing the dense layer to other layers (using less params) or change loss function to meet the requirement for this extreme classification problem?", "votes": null }, { "id": "226182", "postDate": "10/01/2017 09:57:23", "content": "I have fine-tuned SE-ResNet-50 (which is a variation on ResNet-50) by training it on random 160x160 crops and random horizontal flips. It gets 0.68 on the leaderboard. This is without any fancy stuff, I only added a 5270-element dense layer with softmax at the end and also set the last \"block\" of the network to be trainable (so it trains the classification layer and fine-tunes the last N layers of the network).
", "rawMarkdown": "I have fine-tuned SE-ResNet-50 (which is a variation on ResNet-50) by training it on random 160x160 crops and random horizontal flips. It gets 0.68 on the leaderboard. This is without any fancy stuff, I only added a 5270-element dense layer with softmax at the end and also set the last \"block\" of the network to be trainable (so it trains the classification layer and fine-tunes the last N layers of the network).", "votes": null }, { "id": "226253", "postDate": "10/01/2017 15:24:56", "content": "Sounds great! why you use 160*160? And how many epoch do you run to get 0.68?
", "rawMarkdown": "Sounds great! why you use 160*160? And how many epoch do you run to get 0.68?", "votes": null }, { "id": "226272", "postDate": "10/01/2017 16:50:44", "content": "The original images are 180x180 so training on random crops of 160x160 is a quick way to do data augmentation (ImageNet models are often trained on crops of 224x224 from a 256xN image, which is similar). I trained for 6 epochs.
\n\nTo get 0.68 I took 10 crops for each test image (4 corners + center, and their horizontal flips) and did a prediction for each, then took the average of those predictions. So for products with 4 images I actually did separate 40 predictions. This technique scores higher than just doing one prediction per image (which scored 0.6435 on the LB, using the exact same model).
", "rawMarkdown": "The original images are 180x180 so training on random crops of 160x160 is a quick way to do data augmentation (ImageNet models are often trained on crops of 224x224 from a 256xN image, which is similar). I trained for 6 epochs.\n\nTo get 0.68 I took 10 crops for each test image (4 corners + center, and their horizontal flips) and did a prediction for each, then took the average of those predictions. So for products with 4 images I actually did separate 40 predictions. This technique scores higher than just doing one prediction per image (which scored 0.6435 on the LB, using the exact same model).", "votes": null }, { "id": "226304", "postDate": "10/01/2017 18:43:51", "content": "Thx a lot! Ensemble model often get higher score.
", "rawMarkdown": "Thx a lot! Ensemble model often get higher score.", "votes": null }, { "id": "226358", "postDate": "10/02/2017 02:08:36", "content": "", "rawMarkdown": "", "votes": null }, { "id": "230545", "postDate": "10/12/2017 07:10:10", "content": "it is really strange that when i use 10 crops, the acc is lower than single center crop...
", "rawMarkdown": "it is really strange that when i use 10 crops, the acc is lower than single center crop...", "votes": null }, { "id": "230557", "postDate": "10/12/2017 07:56:10", "content": "it depends on how you train and what is your model structure. generally it should be better. can you provide more details?
\n\nfor example, if we make two models:
\n\nmodelA : train without augmentation
modelB: train with augmentation
depending on the dataset charateristics, it is possible that performance are
\n\nmodelA (better) > modelB
\n\nand
\n\nensemble_of_modelB (better) > ensemble_of_modelA
", "rawMarkdown": "it depends on how you train and what is your model structure. generally it should be better. can you provide more details?\n\nfor example, if we make two models:\n\n- modelA : train without augmentation\n\n- modelB: train with augmentation\n\ndepending on the dataset charateristics, it is possible that performance are\n\nmodelA (better) > modelB\n\nand\n\nensemble_of_modelB (better) > ensemble_of_modelA", "votes": null } ], "comments": [ { "id": 224545, "author_name": "humananalog", "author_url": "", "post_date": "09/26/2017 18:28:54", "content": "The current top leaderboard score (0.70773) is quite impressive already for an image classifier. I think state-of-the-art on ImageNet is about 80% (for top-1 accuracy)? So the current results are not too far off from that. (And to be fair, for many products you actually have 4 chances to guess right, so that makes this slightly easier. On the other hand, there are more categories and they are imbalanced, making this slightly harder than ImageNet.)
\n\nMy approach is not to compete on computing power. There will always be someone who is able or willing to buy more computing power than I am. Using the same models as everyone else also isn't very exciting to me. So instead, I'm treating this competition as a way to experiment with different training approaches, etc. \"Work smarter, not harder.\" :-D
", "votes": null, "replies": [ { "id": 224573, "author_name": "timjoseph", "author_url": "", "post_date": "09/26/2017 20:02:45", "content": "Thank you for your response. I am wondering whether the current leaderboard result are achieved with by training models or feature extractions since there has not been much time to train yet.
\n\nI think \"working smart\" is nice, but in the end \"working harder\" will win this, since bigger models seem to be SOTA in similiar tasks :/
", "votes": null, "replies": [] } ] }, { "id": 224975, "author_name": "jeffzhan", "author_url": "", "post_date": "09/28/2017 02:12:18", "content": "I have tried using resnet 50 for finetuning. During training, the train loss sometimes jump to high and accuracy drop to very low. I think that a modification of loss function is needed, since there are too many categories and highly imbalanced data.
\n\nBesides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).
", "votes": null, "replies": [ { "id": 225013, "author_name": "ifighting", "author_url": "", "post_date": "09/28/2017 03:50:44", "content": "i met the similar problem, i also use resnet 50 for finetuning. i found that the training acc is high,but the val acc is extremely low, batch_size=1024, lr = 0.01
", "votes": null, "replies": [] }, { "id": 225056, "author_name": "timjoseph", "author_url": "", "post_date": "09/28/2017 05:52:08", "content": "\n\n\nBesides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).
\n
Can you explain what you mean by this?\nI currently have the opposite experience: Training is slow, but val-acc is like expected.
", "votes": null, "replies": [] }, { "id": 225059, "author_name": "cpruce", "author_url": "", "post_date": "09/28/2017 05:56:46", "content": "@iFighting I'm also finetuning resnet50. I believe I was/am seeing
\n\n\n\n\ni found that the training acc is high,but the val acc is extremely low
\n
since my val split had products with only one image. Trying to test against the lb asap so I can try this hypothesis
", "votes": null, "replies": [] }, { "id": 225422, "author_name": "strideradu", "author_url": "", "post_date": "09/29/2017 02:19:45", "content": "what is your loss function?
", "votes": null, "replies": [] }, { "id": 225430, "author_name": "jeffzhan", "author_url": "", "post_date": "09/29/2017 03:09:35", "content": "@Tim Joseph, I extract images of one category to train the net, and the result seems ok, i can tell that my net and loss function is not too wrong. They are just not much suitable for so many categories and such data distribution.
", "votes": null, "replies": [] }, { "id": 225861, "author_name": "cpruce", "author_url": "", "post_date": "09/30/2017 06:46:41", "content": "@CSAdu my loss function is cross entropy:
\n\n` # loss ----------------------------------------
\n\ndef criterion(logits, labels):
\n\nloss = nn.CrossEntropyLoss()(logits, Variable(labels))\n\nreturn loss`\nNow I'm trying to fine-tune on pre-trained models (ResNet-50, Xception, etc.). But both train&val acc looks bad (~40%), I think it's due to the huge amount of params in the last layer (from the last pooling layer to dense layer for prediction usually have more than 10M params, i.e. 5270*2K or 3K?)
\n\nSo what about changing the dense layer to other layers (using less params) or change loss function to meet the requirement for this extreme classification problem?
", "votes": null, "replies": [ { "id": 226182, "author_name": "humananalog", "author_url": "", "post_date": "10/01/2017 09:57:23", "content": "I have fine-tuned SE-ResNet-50 (which is a variation on ResNet-50) by training it on random 160x160 crops and random horizontal flips. It gets 0.68 on the leaderboard. This is without any fancy stuff, I only added a 5270-element dense layer with softmax at the end and also set the last \"block\" of the network to be trainable (so it trains the classification layer and fine-tunes the last N layers of the network).
", "votes": null, "replies": [] }, { "id": 226253, "author_name": "brianlzm", "author_url": "", "post_date": "10/01/2017 15:24:56", "content": "Sounds great! why you use 160*160? And how many epoch do you run to get 0.68?
", "votes": null, "replies": [] }, { "id": 226272, "author_name": "humananalog", "author_url": "", "post_date": "10/01/2017 16:50:44", "content": "The original images are 180x180 so training on random crops of 160x160 is a quick way to do data augmentation (ImageNet models are often trained on crops of 224x224 from a 256xN image, which is similar). I trained for 6 epochs.
\n\nTo get 0.68 I took 10 crops for each test image (4 corners + center, and their horizontal flips) and did a prediction for each, then took the average of those predictions. So for products with 4 images I actually did separate 40 predictions. This technique scores higher than just doing one prediction per image (which scored 0.6435 on the LB, using the exact same model).
", "votes": null, "replies": [] }, { "id": 226304, "author_name": "brianlzm", "author_url": "", "post_date": "10/01/2017 18:43:51", "content": "Thx a lot! Ensemble model often get higher score.
", "votes": null, "replies": [] }, { "id": 230545, "author_name": "ifighting", "author_url": "", "post_date": "10/12/2017 07:10:10", "content": "it is really strange that when i use 10 crops, the acc is lower than single center crop...
", "votes": null, "replies": [] }, { "id": 230557, "author_name": "hengck23", "author_url": "", "post_date": "10/12/2017 07:56:10", "content": "it depends on how you train and what is your model structure. generally it should be better. can you provide more details?
\n\nfor example, if we make two models:
\n\nmodelA : train without augmentation
modelB: train with augmentation
depending on the dataset charateristics, it is possible that performance are
\n\nmodelA (better) > modelB
\n\nand
\n\nensemble_of_modelB (better) > ensemble_of_modelA
", "votes": null, "replies": [] } ] }, { "id": 226358, "author_name": "brianlzm", "author_url": "", "post_date": "10/02/2017 02:08:36", "content": "", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "224507": "Hi,\nsince it is early in the competition let's talk about the general approach to this problem.\n\nI will start with two obvious ones:\n\n1. Train a network from scratch or from a pretrained model. So far I tried resnet-18 since it takes about 2 hours to train a single epoch (1 image per product). Convergence is slow, but reaches 60% Acc. Bigger models will do better, but without a GPU-Workstation even pretrained networks will take weeks on my single GTX1080TI.\n\n2. Use pretrained networks as feature extractors and train a classifier (e.g. Random Forests) on top of these features. I did not try this yet, but this enables even users with slow GPUs to use big networks (for example Inception-V3).\n\nWhat's your approach? Do you think computing power will win this competition?", "224545": "The current top leaderboard score (0.70773) is quite impressive already for an image classifier. I think state-of-the-art on ImageNet is about 80% (for top-1 accuracy)? So the current results are not too far off from that. (And to be fair, for many products you actually have 4 chances to guess right, so that makes this slightly easier. On the other hand, there are more categories and they are imbalanced, making this slightly harder than ImageNet.)\n\nMy approach is not to compete on computing power. There will always be someone who is able or willing to buy more computing power than I am. Using the same models as everyone else also isn't very exciting to me. So instead, I'm treating this competition as a way to experiment with different training approaches, etc. \"Work smarter, not harder.\" :-D", "224573": "Thank you for your response. I am wondering whether the current leaderboard result are achieved with by training models or feature extractions since there has not been much time to train yet.\n\nI think \"working smart\" is nice, but in the end \"working harder\" will win this, since bigger models seem to be SOTA in similiar tasks :/", "224975": "I have tried using resnet 50 for finetuning. During training, the train loss sometimes jump to high and accuracy drop to very low. I think that a modification of loss function is needed, since there are too many categories and highly imbalanced data.\n\nBesides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).", "225013": "i met the similar problem, i also use resnet 50 for finetuning. i found that the training acc is high,but the val acc is extremely low, batch_size=1024, lr = 0.01", "225056": "> Besides, I tried to extract only one category of product to train, the network converge quickly to 100% accuracy after about 5 epochs (adam lr=1e-5).\n\nCan you explain what you mean by this?\nI currently have the opposite experience: Training is slow, but val-acc is like expected.", "225059": "iFighting I'm also finetuning resnet50. I believe I was/am seeing \n\n> i found that the training acc is high,but the val acc is extremely low\n\nsince my val split had products with only one image. Trying to test against the lb asap so I can try this hypothesis", "225422": "what is your loss function?", "225430": "Tim Joseph, I extract images of one category to train the net, and the result seems ok, i can tell that my net and loss function is not too wrong. They are just not much suitable for so many categories and such data distribution.", "225861": "CSAdu my loss function is cross entropy:\n\n \n` # loss ----------------------------------------\n\ndef criterion(logits, labels): \n\n loss = nn.CrossEntropyLoss()(logits, Variable(labels))\n\n return loss`", "226035": "Now I'm trying to fine-tune on pre-trained models (ResNet-50, Xception, etc.). But both train&val acc looks bad (~40%), I think it's due to the huge amount of params in the last layer (from the last pooling layer to dense layer for prediction usually have more than 10M params, i.e. 5270*2K or 3K?)\n\nSo what about changing the dense layer to other layers (using less params) or change loss function to meet the requirement for this extreme classification problem?", "226182": "I have fine-tuned SE-ResNet-50 (which is a variation on ResNet-50) by training it on random 160x160 crops and random horizontal flips. It gets 0.68 on the leaderboard. This is without any fancy stuff, I only added a 5270-element dense layer with softmax at the end and also set the last \"block\" of the network to be trainable (so it trains the classification layer and fine-tunes the last N layers of the network).", "226253": "Sounds great! why you use 160*160? And how many epoch do you run to get 0.68?", "226272": "The original images are 180x180 so training on random crops of 160x160 is a quick way to do data augmentation (ImageNet models are often trained on crops of 224x224 from a 256xN image, which is similar). I trained for 6 epochs.\n\nTo get 0.68 I took 10 crops for each test image (4 corners + center, and their horizontal flips) and did a prediction for each, then took the average of those predictions. So for products with 4 images I actually did separate 40 predictions. This technique scores higher than just doing one prediction per image (which scored 0.6435 on the LB, using the exact same model).", "226304": "Thx a lot! Ensemble model often get higher score.", "226358": "", "230545": "it is really strange that when i use 10 crops, the acc is lower than single center crop...", "230557": "it depends on how you train and what is your model structure. generally it should be better. can you provide more details?\n\nfor example, if we make two models:\n\n- modelA : train without augmentation\n\n- modelB: train with augmentation\n\ndepending on the dataset charateristics, it is possible that performance are\n\nmodelA (better) > modelB\n\nand\n\nensemble_of_modelB (better) > ensemble_of_modelA" }, "source": "meta" }