I noticed that the evaluation of this competition is based on MAP@5, if using siamese network, then how to score the result in our val_step?? Because as i known, the loss of siamese network is binary-crossentropy, and the last layer of network is sigmoid, and ouputs is possibility. So should we continue to use siamese network to train network or just using classification network to train mutil-label model?
", "messages": [ { "id": "445877", "postDate": "12/27/2018 06:42:26", "content": "I noticed that the evaluation of this competition is based on MAP@5, if using siamese network, then how to score the result in our val_step?? Because as i known, the loss of siamese network is binary-crossentropy, and the last layer of network is sigmoid, and ouputs is possibility. So should we continue to use siamese network to train network or just using classification network to train mutil-label model?
", "rawMarkdown": "I noticed that the evaluation of this competition is based on MAP@5, if using siamese network, then how to score the result in our val_step?? Because as i known, the loss of siamese network is binary-crossentropy, and the last layer of network is sigmoid, and ouputs is possibility. So should we continue to use siamese network to train network or just using classification network to train mutil-label model?", "votes": null }, { "id": "446044", "postDate": "12/27/2018 11:55:44", "content": "Hello Tsung-Han,
\n\nno matter what net architecture we discuss, MAP@5 is non-differentiable, therefore it cannot be used as a loss function to be optimized directly. Unfortunately, I'm not aware of smooth functions which can be used as an upper bound on MAP, so I'm going to stick to cross-entropy optimization, while just checking MAP on validation set during the training.
\n\nAfter a brief googling, this paper might be interesting to check out.
\n\nRegarding the siamese net, I'd like to bet on ideas in a similar vein, but we'll see where it gets me ¯_(ツ)_/¯
", "rawMarkdown": "Hello Tsung-Han,\n\nno matter what net architecture we discuss, MAP@5 is non-differentiable, therefore it cannot be used as a loss function to be optimized directly. Unfortunately, I'm not aware of smooth functions which can be used as an upper bound on MAP, so I'm going to stick to cross-entropy optimization, while just checking MAP on validation set during the training. \n\nAfter a brief googling, [this paper](https://arxiv.org/pdf/1607.03476.pdf) might be interesting to check out.\n\nRegarding the siamese net, I'd like to bet on ideas in a similar vein, but we'll see where it gets me ¯\\_(ツ)_/¯", "votes": null }, { "id": "446063", "postDate": "12/27/2018 12:44:49", "content": "Try this paper, they made it differential
\n\n", "rawMarkdown": "Try this paper, they made it differential\n\nhttp://openaccess.thecvf.com/content_cvpr_2018/papers/He_Local_Descriptors_Optimized_CVPR_2018_paper.pdf", "votes": null }, { "id": "447127", "postDate": "12/29/2018 07:30:00", "content": "Hello Raman, \nThanks for replying. So you still using classification model for this task with MAP to check the training step? But i got bad score by using keras to build classification model.
", "rawMarkdown": "Hello Raman, \nThanks for replying. So you still using classification model for this task with MAP to check the training step? But i got bad score by using keras to build classification model.", "votes": null }, { "id": "448204", "postDate": "12/31/2018 12:57:35", "content": "Tsung-Han,
\n\neven though I'm not using classification model, I do check MAP on validation data during training. Regarding the score, I believe our nets might have comparable performance at the moment ;) Actually, my score is worse than yours if I'm not treating new_whale's separately (0.524 LB when ignoring new_whales). Perhaps, playing with new_whale's a bit might help you to quickly improve your score.
", "rawMarkdown": "Tsung-Han,\n\neven though I'm not using classification model, I do check MAP on validation data during training. Regarding the score, I believe our nets might have comparable performance at the moment ;) Actually, my score is worse than yours if I'm not treating new_whale's separately (0.524 LB when ignoring new_whales). Perhaps, playing with new_whale's a bit might help you to quickly improve your score.", "votes": null } ], "comments": [ { "id": 446044, "author_name": "samusram", "author_url": "", "post_date": "12/27/2018 11:55:44", "content": "Hello Tsung-Han,
\n\nno matter what net architecture we discuss, MAP@5 is non-differentiable, therefore it cannot be used as a loss function to be optimized directly. Unfortunately, I'm not aware of smooth functions which can be used as an upper bound on MAP, so I'm going to stick to cross-entropy optimization, while just checking MAP on validation set during the training.
\n\nAfter a brief googling, this paper might be interesting to check out.
\n\nRegarding the siamese net, I'd like to bet on ideas in a similar vein, but we'll see where it gets me ¯_(ツ)_/¯
", "votes": null, "replies": [ { "id": 446063, "author_name": "oldufo", "author_url": "", "post_date": "12/27/2018 12:44:49", "content": "Try this paper, they made it differential
\n\n", "votes": null, "replies": [] }, { "id": 447127, "author_name": "zhlee2017", "author_url": "", "post_date": "12/29/2018 07:30:00", "content": "Hello Raman, \nThanks for replying. So you still using classification model for this task with MAP to check the training step? But i got bad score by using keras to build classification model.
", "votes": null, "replies": [] }, { "id": 448204, "author_name": "samusram", "author_url": "", "post_date": "12/31/2018 12:57:35", "content": "Tsung-Han,
\n\neven though I'm not using classification model, I do check MAP on validation data during training. Regarding the score, I believe our nets might have comparable performance at the moment ;) Actually, my score is worse than yours if I'm not treating new_whale's separately (0.524 LB when ignoring new_whales). Perhaps, playing with new_whale's a bit might help you to quickly improve your score.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "445877": "I noticed that the evaluation of this competition is based on MAP@5, if using siamese network, then how to score the result in our val_step?? Because as i known, the loss of siamese network is binary-crossentropy, and the last layer of network is sigmoid, and ouputs is possibility. So should we continue to use siamese network to train network or just using classification network to train mutil-label model?", "446044": "Hello Tsung-Han,\n\nno matter what net architecture we discuss, MAP@5 is non-differentiable, therefore it cannot be used as a loss function to be optimized directly. Unfortunately, I'm not aware of smooth functions which can be used as an upper bound on MAP, so I'm going to stick to cross-entropy optimization, while just checking MAP on validation set during the training. \n\nAfter a brief googling, [this paper](https://arxiv.org/pdf/1607.03476.pdf) might be interesting to check out.\n\nRegarding the siamese net, I'd like to bet on ideas in a similar vein, but we'll see where it gets me ¯\\_(ツ)_/¯", "446063": "Try this paper, they made it differential\n\nhttp://openaccess.thecvf.com/content_cvpr_2018/papers/He_Local_Descriptors_Optimized_CVPR_2018_paper.pdf", "447127": "Hello Raman, \nThanks for replying. So you still using classification model for this task with MAP to check the training step? But i got bad score by using keras to build classification model.", "448204": "Tsung-Han,\n\neven though I'm not using classification model, I do check MAP on validation data during training. Regarding the score, I believe our nets might have comparable performance at the moment ;) Actually, my score is worse than yours if I'm not treating new_whale's separately (0.524 LB when ignoring new_whales). Perhaps, playing with new_whale's a bit might help you to quickly improve your score." }, "source": "meta" }