I have two questions to the hosts of the competitions.
\nThe first one is why was the validation fragment rotated in the test set. I fortunately recognized it was rotated and unrotated it for predictions, but I am sure there were many people that got bad lb scores because they were unaware that it was rotated. If rotation augmentation was not used during training, scores could easily be .1-.2 lower than they should have been. I feel like data should be presented during evaluation in the same format as they were in the training set.
\nAlso, why was the public and private lb fragment a part of the same fragment? I felt like the strategy for this competition was to overfit as much as possible to the public lb because the public lb contained the closest data to the private lb. I feel like you may have gotten models that generalized better if 15% of one of the training fragments was taken and used as the public lb. This way the best models would have to generalize to unseen data.
", "messages": [ { "id": "2303133", "postDate": "06/15/2023 05:20:45", "content": "I have two questions to the hosts of the competitions.
\nThe first one is why was the validation fragment rotated in the test set. I fortunately recognized it was rotated and unrotated it for predictions, but I am sure there were many people that got bad lb scores because they were unaware that it was rotated. If rotation augmentation was not used during training, scores could easily be .1-.2 lower than they should have been. I feel like data should be presented during evaluation in the same format as they were in the training set.
\nAlso, why was the public and private lb fragment a part of the same fragment? I felt like the strategy for this competition was to overfit as much as possible to the public lb because the public lb contained the closest data to the private lb. I feel like you may have gotten models that generalized better if 15% of one of the training fragments was taken and used as the public lb. This way the best models would have to generalize to unseen data.
", "rawMarkdown": "I have two questions to the hosts of the competitions. \n\nThe first one is why was the validation fragment rotated in the test set. I fortunately recognized it was rotated and unrotated it for predictions, but I am sure there were many people that got bad lb scores because they were unaware that it was rotated. If rotation augmentation was not used during training, scores could easily be .1-.2 lower than they should have been. I feel like data should be presented during evaluation in the same format as they were in the training set.\n\nAlso, why was the public and private lb fragment a part of the same fragment? I felt like the strategy for this competition was to overfit as much as possible to the public lb because the public lb contained the closest data to the private lb. I feel like you may have gotten models that generalized better if 15% of one of the training fragments was taken and used as the public lb. This way the best models would have to generalize to unseen data.", "votes": null }, { "id": "2303326", "postDate": "06/15/2023 08:36:43", "content": "I think it is a really bad idea that they give the test samples 90° rotated. Applying the methods to the full scroll you will know the orientation of the text.
\nWe used rotations as test time augmentation because it was mentioned in one discussion but only using the 90° rotated scans would probably have further improved our result. I think chumajin reported an improvement of 0.16 between normal and 90° rotated input.
\nImho the organizers should evaluate the algorithms on the normal data and 90° rotated to make it really fair. Otherwise that is also just test set overfitting.
Our team got little worse results when adding rotation TTA during inference. (we applied rotate90's augmentation during training)
\nOur model will probably perform the same whether the fragment is rotated or not, but we can improve the generalization performance by applying augmentations.
@petersk20,
\nHow did you get the validation fragment? Was it made public after final evaluation? I think it would be worth if competion hosts make it public togather with ground truth data, after the competition is finished.
I think it is a really bad idea that they give the test samples 90° rotated. Applying the methods to the full scroll you will know the orientation of the text.
\nWe used rotations as test time augmentation because it was mentioned in one discussion but only using the 90° rotated scans would probably have further improved our result. I think chumajin reported an improvement of 0.16 between normal and 90° rotated input.
\nImho the organizers should evaluate the algorithms on the normal data and 90° rotated to make it really fair. Otherwise that is also just test set overfitting.
Our team got little worse results when adding rotation TTA during inference. (we applied rotate90's augmentation during training)
\nOur model will probably perform the same whether the fragment is rotated or not, but we can improve the generalization performance by applying augmentations.
@petersk20,
\nHow did you get the validation fragment? Was it made public after final evaluation? I think it would be worth if competion hosts make it public togather with ground truth data, after the competition is finished.