{ "id": 412513, "title": "Was the fourth fragment created correctly?", "url": "/competitions/vesuvius-challenge-ink-detection/discussion/412513", "author_name": "", "post_date": "2023-05-24T03:26:57.848506100Z", "votes": 5, "comment_count": 8, "views": 0, "content": "

Is anyone else finding that the test fragment may be a bit different than the fragments we were given to train our models with?

\n

Based on my submissions, I think the test fragment is rotated and the channels don't match up as well as the channels for the training fragments do. This might be due to a mistake in the creation of this fourth fragment, or it could just be due to differences present in the fourth fragment that aren't there for the released fragments.

\n

Regardless, I think the competition hosts should review how the fourth fragment was preprocessed for this competition and see if there were any errors present. They should also disclose if there are any known major differences present between the test fragment and the ones available to us as the moment.

\n

Let me know in the comment if you have found similar or different findings.

", "messages": [ { "id": "2271590", "postDate": "05/24/2023 03:26:57", "content": "

Is anyone else finding that the test fragment may be a bit different than the fragments we were given to train our models with?

\n

Based on my submissions, I think the test fragment is rotated and the channels don't match up as well as the channels for the training fragments do. This might be due to a mistake in the creation of this fourth fragment, or it could just be due to differences present in the fourth fragment that aren't there for the released fragments.

\n

Regardless, I think the competition hosts should review how the fourth fragment was preprocessed for this competition and see if there were any errors present. They should also disclose if there are any known major differences present between the test fragment and the ones available to us as the moment.

\n

Let me know in the comment if you have found similar or different findings.

", "rawMarkdown": "Is anyone else finding that the test fragment may be a bit different than the fragments we were given to train our models with?\n\nBased on my submissions, I think the test fragment is rotated and the channels don't match up as well as the channels for the training fragments do. This might be due to a mistake in the creation of this fourth fragment, or it could just be due to differences present in the fourth fragment that aren't there for the released fragments. \n\nRegardless, I think the competition hosts should review how the fourth fragment was preprocessed for this competition and see if there were any errors present. They should also disclose if there are any known major differences present between the test fragment and the ones available to us as the moment. \n\nLet me know in the comment if you have found similar or different findings.", "votes": null }, { "id": "2272186", "postDate": "05/24/2023 10:42:09", "content": "

there is someting very fishy.

\n

i made a bug:

\n\n

you can repeat my experiments for training a model that only predict for upright input. (just set ground truth of rotated input to be zero at training)

\n
\n

note in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?

", "rawMarkdown": "there is someting very fishy.\n\ni made a bug:\n\n- i forget to rotate90 the mask at train data augmentation (only volume is rotated). The effects is that at inference, i can only perdict for upright sub-surface and not for rotated sub-surface.\n\n- at inference, without rotate TTA, LB score is 0 to 0.2x (lower threshold). But with TTA, LB is 0.56.\n\nyou can repeat my experiments for training a model that only predict for upright input. (just set ground truth of rotated input to be zero at training)\n\n---\n\nnote in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?", "votes": null }, { "id": "2272470", "postDate": "05/24/2023 14:15:37", "content": "
\n

note in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?

\n
\n

Can you elaborate on this? I seem to be having a problem where the highest submitted score I can achieve is 0.12, while the f-beta score is over 0.45.

", "rawMarkdown": ">note in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?\n\nCan you elaborate on this? I seem to be having a problem where the highest submitted score I can achieve is 0.12, while the f-beta score is over 0.45.", "votes": null }, { "id": "2276920", "postDate": "05/27/2023 10:12:03", "content": "

I noticed that when validating on fold 1, the TTA rotation increases score only by +0.01 or +0.02, but on LB it's +0.09.
\nI look forward to seeing what's up with the test fragment.

", "rawMarkdown": "I noticed that when validating on fold 1, the TTA rotation increases score only by +0.01 or +0.02, but on LB it's +0.09. \nI look forward to seeing what's up with the test fragment.", "votes": null }, { "id": "2277592", "postDate": "05/27/2023 22:19:49", "content": "

i occurs to me that the 10% public test data could contain characters, e.g. nok. so when you rotate it, it becomes like

\n
'nok' breaks down to 'n o l <  '\n
\n
v\n_\no\nc\n
", "rawMarkdown": "i occurs to me that the 10% public test data could contain characters, e.g. nok. so when you rotate it, it becomes like\n\n```\n'nok' breaks down to 'n o l < '\n```\n```\nv\n_\no\nc\n```", "votes": null }, { "id": "2281338", "postDate": "05/30/2023 19:28:53", "content": "

We double checked the data and did not find any unexpected anomalies. Keep in mind that you should not assume any specific alignment or letter scale of the hidden fragment. Submissions should aim to detect ink and be robust to changes in rotation, translation, warping, and other such transformations, as they will occur in the real world. The ultimate use for the work here is prediction on the full scrolls, which will not be as neat as the fragments.

", "rawMarkdown": "We double checked the data and did not find any unexpected anomalies. Keep in mind that you should not assume any specific alignment or letter scale of the hidden fragment. Submissions should aim to detect ink and be robust to changes in rotation, translation, warping, and other such transformations, as they will occur in the real world. The ultimate use for the work here is prediction on the full scrolls, which will not be as neat as the fragments.", "votes": null }, { "id": "2281574", "postDate": "05/31/2023 01:55:10", "content": "

is it possible that the test data is rotated at the z-axis? if that's the case we have to do 3d augmentations

", "rawMarkdown": "is it possible that the test data is rotated at the z-axis? if that's the case we have to do 3d augmentations", "votes": null }, { "id": "2281773", "postDate": "05/31/2023 05:48:23", "content": "

No, the z-axis in the secret 4th fragment is similar to that of the public fragments.

", "rawMarkdown": "No, the z-axis in the secret 4th fragment is similar to that of the public fragments.", "votes": null }, { "id": "2285342", "postDate": "06/02/2023 16:24:23", "content": "
\n

We double checked the data and did not find any unexpected anomalies.

\n
\n

So, there are some expected anomalies, right? :)

\n

Jokes aside, is not the expected orientation of the text a useful feature even in for work in the field? From (not too many) papyri that I've looked at for the past month, it looks that the text is always aligned with the fibers that form the layers of the papyrus. One may reasonably assume that these fibers also greatly affect how the ink spreads and permeates the papyrus. Indeed, the scrolls are warped, however the virtual unwrapping, when working on the whole scroll, might be able to yield the correctly oriented unwrapped scroll.

\n

If these points hold some truth, would it be reasonable to punish a successful approach which relies on the fragment orientation?

", "rawMarkdown": "> We double checked the data and did not find any unexpected anomalies.\n\nSo, there are some *expected* anomalies, right? :) \n\nJokes aside, is not the expected orientation of the text a useful feature even in for work in the field? From (not too many) papyri that I've looked at for the past month, it looks that the text is always aligned with the fibers that form the layers of the papyrus. One may reasonably assume that these fibers also greatly affect how the ink spreads and permeates the papyrus. Indeed, the scrolls are warped, however the virtual unwrapping, when working on the whole scroll, might be able to yield the correctly oriented unwrapped scroll.\n\nIf these points hold some truth, would it be reasonable to punish a successful approach which relies on the fragment orientation?", "votes": null } ], "comments": [ { "id": 2272186, "author_name": "hengck23", "author_url": "", "post_date": "05/24/2023 10:42:09", "content": "

there is someting very fishy.

\n

i made a bug:

\n\n

you can repeat my experiments for training a model that only predict for upright input. (just set ground truth of rotated input to be zero at training)

\n
\n

note in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?

", "votes": null, "replies": [ { "id": 2272470, "author_name": "jeffborack", "author_url": "", "post_date": "05/24/2023 14:15:37", "content": "
\n

note in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?

\n
\n

Can you elaborate on this? I seem to be having a problem where the highest submitted score I can achieve is 0.12, while the f-beta score is over 0.45.

", "votes": null, "replies": [] }, { "id": 2277592, "author_name": "hengck23", "author_url": "", "post_date": "05/27/2023 22:19:49", "content": "

i occurs to me that the 10% public test data could contain characters, e.g. nok. so when you rotate it, it becomes like

\n
'nok' breaks down to 'n o l <  '\n
\n
v\n_\no\nc\n
", "votes": null, "replies": [ { "id": 2281574, "author_name": "fengqilong", "author_url": "", "post_date": "05/31/2023 01:55:10", "content": "

is it possible that the test data is rotated at the z-axis? if that's the case we have to do 3d augmentations

", "votes": null, "replies": [ { "id": 2281773, "author_name": "jpposma", "author_url": "", "post_date": "05/31/2023 05:48:23", "content": "

No, the z-axis in the secret 4th fragment is similar to that of the public fragments.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2276920, "author_name": "raki21", "author_url": "", "post_date": "05/27/2023 10:12:03", "content": "

I noticed that when validating on fold 1, the TTA rotation increases score only by +0.01 or +0.02, but on LB it's +0.09.
\nI look forward to seeing what's up with the test fragment.

", "votes": null, "replies": [] }, { "id": 2281338, "author_name": "jpposma", "author_url": "", "post_date": "05/30/2023 19:28:53", "content": "

We double checked the data and did not find any unexpected anomalies. Keep in mind that you should not assume any specific alignment or letter scale of the hidden fragment. Submissions should aim to detect ink and be robust to changes in rotation, translation, warping, and other such transformations, as they will occur in the real world. The ultimate use for the work here is prediction on the full scrolls, which will not be as neat as the fragments.

", "votes": null, "replies": [ { "id": 2285342, "author_name": "alexz0", "author_url": "", "post_date": "06/02/2023 16:24:23", "content": "
\n

We double checked the data and did not find any unexpected anomalies.

\n
\n

So, there are some expected anomalies, right? :)

\n

Jokes aside, is not the expected orientation of the text a useful feature even in for work in the field? From (not too many) papyri that I've looked at for the past month, it looks that the text is always aligned with the fibers that form the layers of the papyrus. One may reasonably assume that these fibers also greatly affect how the ink spreads and permeates the papyrus. Indeed, the scrolls are warped, however the virtual unwrapping, when working on the whole scroll, might be able to yield the correctly oriented unwrapped scroll.

\n

If these points hold some truth, would it be reasonable to punish a successful approach which relies on the fragment orientation?

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2271590": "Is anyone else finding that the test fragment may be a bit different than the fragments we were given to train our models with?\n\nBased on my submissions, I think the test fragment is rotated and the channels don't match up as well as the channels for the training fragments do. This might be due to a mistake in the creation of this fourth fragment, or it could just be due to differences present in the fourth fragment that aren't there for the released fragments. \n\nRegardless, I think the competition hosts should review how the fourth fragment was preprocessed for this competition and see if there were any errors present. They should also disclose if there are any known major differences present between the test fragment and the ones available to us as the moment. \n\nLet me know in the comment if you have found similar or different findings.", "2272186": "there is someting very fishy.\n\ni made a bug:\n\n- i forget to rotate90 the mask at train data augmentation (only volume is rotated). The effects is that at inference, i can only perdict for upright sub-surface and not for rotated sub-surface.\n\n- at inference, without rotate TTA, LB score is 0 to 0.2x (lower threshold). But with TTA, LB is 0.56.\n\nyou can repeat my experiments for training a model that only predict for upright input. (just set ground truth of rotated input to be zero at training)\n\n---\n\nnote in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?", "2272470": ">note in some kaggle competition rle encode and decode scripts uses mask.shape = (width, height) instead of mask.shape = (height,width). maybe there is problem with rle decode?\n\nCan you elaborate on this? I seem to be having a problem where the highest submitted score I can achieve is 0.12, while the f-beta score is over 0.45.", "2276920": "I noticed that when validating on fold 1, the TTA rotation increases score only by +0.01 or +0.02, but on LB it's +0.09. \nI look forward to seeing what's up with the test fragment.", "2277592": "i occurs to me that the 10% public test data could contain characters, e.g. nok. so when you rotate it, it becomes like\n\n```\n'nok' breaks down to 'n o l < '\n```\n```\nv\n_\no\nc\n```", "2281338": "We double checked the data and did not find any unexpected anomalies. Keep in mind that you should not assume any specific alignment or letter scale of the hidden fragment. Submissions should aim to detect ink and be robust to changes in rotation, translation, warping, and other such transformations, as they will occur in the real world. The ultimate use for the work here is prediction on the full scrolls, which will not be as neat as the fragments.", "2281574": "is it possible that the test data is rotated at the z-axis? if that's the case we have to do 3d augmentations", "2281773": "No, the z-axis in the secret 4th fragment is similar to that of the public fragments.", "2285342": "> We double checked the data and did not find any unexpected anomalies.\n\nSo, there are some *expected* anomalies, right? :) \n\nJokes aside, is not the expected orientation of the text a useful feature even in for work in the field? From (not too many) papyri that I've looked at for the past month, it looks that the text is always aligned with the fibers that form the layers of the papyrus. One may reasonably assume that these fibers also greatly affect how the ink spreads and permeates the papyrus. Indeed, the scrolls are warped, however the virtual unwrapping, when working on the whole scroll, might be able to yield the correctly oriented unwrapped scroll.\n\nIf these points hold some truth, would it be reasonable to punish a successful approach which relies on the fragment orientation?" }, "source": "meta" }