Hi organizer, I wonder if there will be a baseline score provided in the leaderboard, which can help us to understand how much gain we have obtained compared with the baseline. Currently, I am confused with the big gap between valiation dataset and test dataset. In this case, it would be better if there is an official score released on test dataset. Thanks.
", "messages": [ { "id": "771345", "postDate": "03/14/2020 03:57:01", "content": "Hi organizer, I wonder if there will be a baseline score provided in the leaderboard, which can help us to understand how much gain we have obtained compared with the baseline. Currently, I am confused with the big gap between valiation dataset and test dataset. In this case, it would be better if there is an official score released on test dataset. Thanks.
", "rawMarkdown": "Hi organizer, I wonder if there will be a baseline score provided in the leaderboard, which can help us to understand how much gain we have obtained compared with the baseline. Currently, I am confused with the big gap between valiation dataset and test dataset. In this case, it would be better if there is an official score released on test dataset. Thanks.", "votes": null }, { "id": "771861", "postDate": "03/14/2020 17:52:56", "content": "Hi, a baseline of ResNet50 fine-tuned from Imagenet using only labeled data is around 46.5% acc on val (see the github page). Note that we switched to \"error rate\" on the leaderboard on 3/13.
", "rawMarkdown": "Hi, a baseline of ResNet50 fine-tuned from Imagenet using only labeled data is around 46.5% acc on val (see the [github page](https://github.com/cvl-umass/semi-inat-2020)). Note that we switched to \"error rate\" on the leaderboard on 3/13.", "votes": null }, { "id": "772096", "postDate": "03/15/2020 02:11:13", "content": "I'd like to see if anybody achieved anything close to 46.5% by fine-tuning. I have tried and got nowhere near.
", "rawMarkdown": "I'd like to see if anybody achieved anything close to 46.5% by fine-tuning. I have tried and got nowhere near.", "votes": null }, { "id": "772098", "postDate": "03/15/2020 02:15:49", "content": "Hi, have you tried using those hyper parameters on the github page?
", "rawMarkdown": "Hi, have you tried using those hyper parameters on the github page?", "votes": null }, { "id": "772100", "postDate": "03/15/2020 02:28:23", "content": "The accuracy I have obtained on the validation dataset is nearly 52%. However, when the model is emplyed in the test dataset, the error rate is relatively high according the submission result. so would the organizer release the error rate of your baseline model on the test dataset?
", "rawMarkdown": "The accuracy I have obtained on the validation dataset is nearly 52%. However, when the model is emplyed in the test dataset, the error rate is relatively high according the submission result. so would the organizer release the error rate of your baseline model on the test dataset?", "votes": null }, { "id": "772105", "postDate": "03/15/2020 02:36:41", "content": "How much is the difference between your val and test accuracy? Because I found them to be similar (within 1-2%) in my experiments.
", "rawMarkdown": "How much is the difference between your val and test accuracy? Because I found them to be similar (within 1-2%) in my experiments.", "votes": null }, { "id": "772113", "postDate": "03/15/2020 03:12:45", "content": "Hi, which hyperparameters are you referring to? I don't see such information on the page.
", "rawMarkdown": "Hi, which hyperparameters are you referring to? I don't see such information on the page.", "votes": null }, { "id": "772116", "postDate": "03/15/2020 03:17:18", "content": "Ah sorry I forgot that we removed that message. I can say that I used Adam with a batch size of 32 to train the model for 10k iterations.
", "rawMarkdown": "Ah sorry I forgot that we removed that message. I can say that I used Adam with a batch size of 32 to train the model for 10k iterations.", "votes": null }, { "id": "772198", "postDate": "03/15/2020 06:28:26", "content": "I got near 40% on validation in like 30-40 epochs without a lot of tuning.\nCurrently I had almost the same numbers for test dataset (near 0.005 accuracy which is pretty random) for both 25 accuracy on validation and 40% accuracy on validation.
", "rawMarkdown": "I got near 40% on validation in like 30-40 epochs without a lot of tuning.\nCurrently I had almost the same numbers for test dataset (near 0.005 accuracy which is pretty random) for both 25 accuracy on validation and 40% accuracy on validation.", "votes": null }, { "id": "772353", "postDate": "03/15/2020 11:48:02", "content": "Hi, as aforementioned, the accuracy of validation dataset is nearly 50% while the score I have obtained on test dataset is 0.99475. It seems that competitor DenM has similar problem. Is there something wrong when evaluating the submission file?
", "rawMarkdown": "Hi, as aforementioned, the accuracy of validation dataset is nearly 50% while the score I have obtained on test dataset is 0.99475. It seems that competitor DenM has similar problem. Is there something wrong when evaluating the submission file?", "votes": null }, { "id": "772580", "postDate": "03/15/2020 16:38:52", "content": "", "rawMarkdown": "", "votes": null }, { "id": "772687", "postDate": "03/15/2020 19:13:10", "content": "I am using ids from the test annotation. They are not the same as file names. \nLooks like something is wrong with class mapping.
", "rawMarkdown": "I am using ids from the test annotation. They are not the same as file names. \nLooks like something is wrong with class mapping.", "votes": null }, { "id": "772691", "postDate": "03/15/2020 19:17:42", "content": "Yes seems like it's wrong... can you try using the original test file names? Meanwhile I'm modifying the test annotations. Thank you!
", "rawMarkdown": "Yes seems like it's wrong... can you try using the original test file names? Meanwhile I'm modifying the test annotations. Thank you!", "votes": null }, { "id": "772697", "postDate": "03/15/2020 19:33:43", "content": "It works thanks!
", "rawMarkdown": "It works thanks!", "votes": null }, { "id": "772698", "postDate": "03/15/2020 19:34:14", "content": "Please use names from files and you will see almost the same score as in validation!
", "rawMarkdown": "Please use names from files and you will see almost the same score as in validation!", "votes": null }, { "id": "772714", "postDate": "03/15/2020 20:06:56", "content": "Thanks for letting me know. I have updated the test annotation file. Sorry for the mistake!
", "rawMarkdown": "Thanks for letting me know. I have updated the test annotation file. Sorry for the mistake!", "votes": null } ], "comments": [ { "id": 771861, "author_name": "jcfredsu", "author_url": "", "post_date": "03/14/2020 17:52:56", "content": "Hi, a baseline of ResNet50 fine-tuned from Imagenet using only labeled data is around 46.5% acc on val (see the github page). Note that we switched to \"error rate\" on the leaderboard on 3/13.
", "votes": null, "replies": [ { "id": 772096, "author_name": "ananschuett", "author_url": "", "post_date": "03/15/2020 02:11:13", "content": "I'd like to see if anybody achieved anything close to 46.5% by fine-tuning. I have tried and got nowhere near.
", "votes": null, "replies": [] }, { "id": 772098, "author_name": "jcfredsu", "author_url": "", "post_date": "03/15/2020 02:15:49", "content": "Hi, have you tried using those hyper parameters on the github page?
", "votes": null, "replies": [] }, { "id": 772113, "author_name": "ananschuett", "author_url": "", "post_date": "03/15/2020 03:12:45", "content": "Hi, which hyperparameters are you referring to? I don't see such information on the page.
", "votes": null, "replies": [] }, { "id": 772116, "author_name": "jcfredsu", "author_url": "", "post_date": "03/15/2020 03:17:18", "content": "Ah sorry I forgot that we removed that message. I can say that I used Adam with a batch size of 32 to train the model for 10k iterations.
", "votes": null, "replies": [] } ] }, { "id": 772100, "author_name": "pqzhuang", "author_url": "", "post_date": "03/15/2020 02:28:23", "content": "The accuracy I have obtained on the validation dataset is nearly 52%. However, when the model is emplyed in the test dataset, the error rate is relatively high according the submission result. so would the organizer release the error rate of your baseline model on the test dataset?
", "votes": null, "replies": [ { "id": 772105, "author_name": "jcfredsu", "author_url": "", "post_date": "03/15/2020 02:36:41", "content": "How much is the difference between your val and test accuracy? Because I found them to be similar (within 1-2%) in my experiments.
", "votes": null, "replies": [] }, { "id": 772353, "author_name": "pqzhuang", "author_url": "", "post_date": "03/15/2020 11:48:02", "content": "Hi, as aforementioned, the accuracy of validation dataset is nearly 50% while the score I have obtained on test dataset is 0.99475. It seems that competitor DenM has similar problem. Is there something wrong when evaluating the submission file?
", "votes": null, "replies": [] }, { "id": 772698, "author_name": "trrrrr", "author_url": "", "post_date": "03/15/2020 19:34:14", "content": "Please use names from files and you will see almost the same score as in validation!
", "votes": null, "replies": [] } ] }, { "id": 772198, "author_name": "trrrrr", "author_url": "", "post_date": "03/15/2020 06:28:26", "content": "I got near 40% on validation in like 30-40 epochs without a lot of tuning.\nCurrently I had almost the same numbers for test dataset (near 0.005 accuracy which is pretty random) for both 25 accuracy on validation and 40% accuracy on validation.
", "votes": null, "replies": [] }, { "id": 772580, "author_name": "jcfredsu", "author_url": "", "post_date": "03/15/2020 16:38:52", "content": "", "votes": null, "replies": [ { "id": 772687, "author_name": "trrrrr", "author_url": "", "post_date": "03/15/2020 19:13:10", "content": "I am using ids from the test annotation. They are not the same as file names. \nLooks like something is wrong with class mapping.
", "votes": null, "replies": [] }, { "id": 772691, "author_name": "jcfredsu", "author_url": "", "post_date": "03/15/2020 19:17:42", "content": "Yes seems like it's wrong... can you try using the original test file names? Meanwhile I'm modifying the test annotations. Thank you!
", "votes": null, "replies": [] }, { "id": 772697, "author_name": "trrrrr", "author_url": "", "post_date": "03/15/2020 19:33:43", "content": "It works thanks!
", "votes": null, "replies": [] }, { "id": 772714, "author_name": "jcfredsu", "author_url": "", "post_date": "03/15/2020 20:06:56", "content": "Thanks for letting me know. I have updated the test annotation file. Sorry for the mistake!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "771345": "Hi organizer, I wonder if there will be a baseline score provided in the leaderboard, which can help us to understand how much gain we have obtained compared with the baseline. Currently, I am confused with the big gap between valiation dataset and test dataset. In this case, it would be better if there is an official score released on test dataset. Thanks.", "771861": "Hi, a baseline of ResNet50 fine-tuned from Imagenet using only labeled data is around 46.5% acc on val (see the [github page](https://github.com/cvl-umass/semi-inat-2020)). Note that we switched to \"error rate\" on the leaderboard on 3/13.", "772096": "I'd like to see if anybody achieved anything close to 46.5% by fine-tuning. I have tried and got nowhere near.", "772098": "Hi, have you tried using those hyper parameters on the github page?", "772100": "The accuracy I have obtained on the validation dataset is nearly 52%. However, when the model is emplyed in the test dataset, the error rate is relatively high according the submission result. so would the organizer release the error rate of your baseline model on the test dataset?", "772105": "How much is the difference between your val and test accuracy? Because I found them to be similar (within 1-2%) in my experiments.", "772113": "Hi, which hyperparameters are you referring to? I don't see such information on the page.", "772116": "Ah sorry I forgot that we removed that message. I can say that I used Adam with a batch size of 32 to train the model for 10k iterations.", "772198": "I got near 40% on validation in like 30-40 epochs without a lot of tuning.\nCurrently I had almost the same numbers for test dataset (near 0.005 accuracy which is pretty random) for both 25 accuracy on validation and 40% accuracy on validation.", "772353": "Hi, as aforementioned, the accuracy of validation dataset is nearly 50% while the score I have obtained on test dataset is 0.99475. It seems that competitor DenM has similar problem. Is there something wrong when evaluating the submission file?", "772580": "", "772687": "I am using ids from the test annotation. They are not the same as file names. \nLooks like something is wrong with class mapping.", "772691": "Yes seems like it's wrong... can you try using the original test file names? Meanwhile I'm modifying the test annotations. Thank you!", "772697": "It works thanks!", "772698": "Please use names from files and you will see almost the same score as in validation!", "772714": "Thanks for letting me know. I have updated the test annotation file. Sorry for the mistake!" }, "source": "meta" }