{ "id": 572096, "title": "Clarification on Early Sharing Prize and Temporal Cutoff Rule", "url": "/competitions/stanford-rna-3d-folding/discussion/572096", "author_name": "", "post_date": "2025-04-07T15:28:29.623633600Z", "votes": 5, "comment_count": 16, "views": 0, "content": "

@rhijudas
\n@shujun717
\n@alissahummer

\n

Hi, I’d like to follow up on this earlier post from the organizers about the Early Sharing Prize:
\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/568059

\n

In particular, the updated eligibility rule states:

\n
\n

“Submissions should only make use of information publicly available before the temporal_cutoff dates provided with test sequences.”

\n
\n

I have two questions regarding this:

\n
    \n
  1. Since test sequences and their temporal_cutoff dates are not visible to participants, it seems impossible to know in advance whether a given resource is allowed. How are participants expected to comply with this?
    2. \n
  2. When using pretrained models, the training data and its date range are rarely disclosed, and even if they are, it’s not possible to verify how the model was actually trained. How should we interpret the cutoff rule in these cases?
    3. \n
\n

Would appreciate any clarification. Thanks!

\n

Best,
\nBilzard

\n
\n

Edit:

\n

Related post:
\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687

", "messages": [ { "id": "3173149", "postDate": "04/07/2025 15:28:29", "content": "

@rhijudas
\n@shujun717
\n@alissahummer

\n

Hi, I’d like to follow up on this earlier post from the organizers about the Early Sharing Prize:
\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/568059

\n

In particular, the updated eligibility rule states:

\n
\n

“Submissions should only make use of information publicly available before the temporal_cutoff dates provided with test sequences.”

\n
\n

I have two questions regarding this:

\n
    \n
  1. Since test sequences and their temporal_cutoff dates are not visible to participants, it seems impossible to know in advance whether a given resource is allowed. How are participants expected to comply with this?
    2. \n
  2. When using pretrained models, the training data and its date range are rarely disclosed, and even if they are, it’s not possible to verify how the model was actually trained. How should we interpret the cutoff rule in these cases?
    3. \n
\n

Would appreciate any clarification. Thanks!

\n

Best,
\nBilzard

\n
\n

Edit:

\n

Related post:
\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687

", "rawMarkdown": "rhijudas\n@shujun717\n@alissahummer\n\nHi, I’d like to follow up on this earlier post from the organizers about the Early Sharing Prize:\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/568059\n\nIn particular, the updated eligibility rule states:\n\n> “Submissions should only make use of information publicly available before the temporal_cutoff dates provided with test sequences.”\n\nI have two questions regarding this:\n\n1.\t**Since test sequences and their temporal_cutoff dates are not visible to participants**, it seems impossible to know in advance whether a given resource is allowed. How are participants expected to comply with this?\n2.\t**When using pretrained models**, the training data and its date range are rarely disclosed, and even if they are, it’s not possible to verify how the model was actually trained. How should we interpret the cutoff rule in these cases?\n\nWould appreciate any clarification. Thanks!\n\nBest,\nBilzard\n\n---\n**Edit**:\n\nRelated post:\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687", "votes": null }, { "id": "3173256", "postDate": "04/07/2025 17:37:18", "content": "

Actually I have no idea but I'd say according to providen rules at least for early sharing prize only no doubt model/trainings would be eligible.

", "rawMarkdown": "Actually I have no idea but I'd say according to providen rules at least for early sharing prize only no doubt model/trainings would be eligible.", "votes": null }, { "id": "3173289", "postDate": "04/07/2025 18:18:13", "content": "

\"When using pretrained models, the training data and its date range are rarely disclosed,\"
\nmost are disclosed (espeically if they are used in CASP)

", "rawMarkdown": "\"When using pretrained models, the training data and its date range are rarely disclosed,\"\nmost are disclosed (espeically if they are used in CASP)", "votes": null }, { "id": "3173293", "postDate": "04/07/2025 18:21:44", "content": "

And if not. The weights have not a date of uploading? At the web repository I mean.

", "rawMarkdown": "And if not. The weights have not a date of uploading? At the web repository I mean.", "votes": null }, { "id": "3173400", "postDate": "04/07/2025 21:15:21", "content": "

Commenting for traction, I was also interested about this and got no response: https://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687

", "rawMarkdown": "Commenting for traction, I was also interested about this and got no response: https://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687", "votes": null }, { "id": "3173448", "postDate": "04/07/2025 23:28:43", "content": "

@hengck23

\n

For example, how do we know the release date of Protenix?

\n

https://github.com/bytedance/Protenix

\n

They seems updated weight version 2.0, but no model card seems to be released.

", "rawMarkdown": "hengck23 \n\nFor example, how do we know the release date of Protenix?\n\nhttps://github.com/bytedance/Protenix\n\nThey seems updated weight version 2.0, but no model card seems to be released.", "votes": null }, { "id": "3173451", "postDate": "04/07/2025 23:45:13", "content": "

Possibly, and I think it's most realistic trade-off. However, it might be too strict considering the time gap of data download and model upload…

", "rawMarkdown": "Possibly, and I think it's most realistic trade-off. However, it might be too strict considering the time gap of data download and model upload...", "votes": null }, { "id": "3173452", "postDate": "04/07/2025 23:47:56", "content": "

@asarvazyan
\nSorry, I missed this post. Thank you for sharing.

\n

I added this link to description.

", "rawMarkdown": "asarvazyan \nSorry, I missed this post. Thank you for sharing.\n\nI added this link to description.", "votes": null }, { "id": "3173461", "postDate": "04/08/2025 00:18:07", "content": "

It is mentioned in the paper or the webpage. I remember reading it

", "rawMarkdown": "It is mentioned in the paper or the webpage. I remember reading it", "votes": null }, { "id": "3173463", "postDate": "04/08/2025 00:28:20", "content": "

Thanks @asarvazyan and @tatamikenn for bringing up these questions relevant to the Early Sharing Prize -- we are very glad to see you have interest!

\n", "rawMarkdown": "Thanks @asarvazyan and @tatamikenn for bringing up these questions relevant to the Early Sharing Prize -- we are very glad to see you have interest!\n\n- Some of the leaderboard targets may be drawn from CASP16, whose earliest deadline was 2024-05-16. (May 16, 2024). So that is the safest cutoff to use if you want to use a single cutoff date. However, you may want to also use one or more models that are trained on the very latest available information. In that case, you could have *two* or more models, one with cutoff date of 2024-05-16 and others with cutoff date closer to when you train them in April 2025. Then you should have your Notebook select which model to use based on the most recent one that is still earlier than the `temporal_cutoff` field for each target in `test_sequences.csv`.\n\n- If your model uses MSA's, the ones provided by us as hosts in `MSA/` will be safe.\n\n- If your model searches for 3D RNA structures from the PDB as templates, please do have the notebook pay attention to those structures' release dates in the PDB and not use any PDB with date before the `temporal_cutoff`.\n\n- If your model uses LLMs trained on scientific papers and preprints, you could either find the LLM's cutoff date or check that it is responsive to prompts that ask it to only use information that existed prior to each targets' `temporal_cutoff` date.\n\n- No matter what, it would be good to include comments in your Notebook documenting how you handled `temporal_cutoff`.", "votes": null }, { "id": "3173473", "postDate": "04/08/2025 00:42:51", "content": "

@rhijudas

\n

Thank you for the detailed explanation — I understand the point.
\nRegarding pre-trained models without a declared cut-off date:
\nwould it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?

\n

Additionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?

", "rawMarkdown": "rhijudas \n\nThank you for the detailed explanation — I understand the point.\nRegarding pre-trained models without a declared cut-off date:\nwould it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?\n\nAdditionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?", "votes": null }, { "id": "3173476", "postDate": "04/08/2025 00:55:11", "content": "

@hengck23
\nI see. I’ll check it out — thanks.
\nHowever, since the repository has been actively updated after the paper was published, I’m not sure whether version 2.0.0 still uses the same cut-off date as stated in the paper.

", "rawMarkdown": "hengck23 \nI see. I’ll check it out — thanks.\nHowever, since the repository has been actively updated after the paper was published, I’m not sure whether version 2.0.0 still uses the same cut-off date as stated in the paper.", "votes": null }, { "id": "3173477", "postDate": "04/08/2025 00:57:39", "content": "

Protenix is trained with PDB cut-off 2021-09-30

", "rawMarkdown": "Protenix is trained with PDB cut-off 2021-09-30", "votes": null }, { "id": "3173478", "postDate": "04/08/2025 01:02:33", "content": "

I think it would be like the old years of kaggle when kaggle take responsibity to ensure non commercial license. I remember there is a pretrain model thread.

\n

When in doubt, can always write email to proteinx author to confirm.

\n

We can start a pretrain model thread, stating model, download url, license, cutoff date. All kaggle and host can post verified model.

\n

Most pretrain model are actively player of casp, which already required disclosure of cutoff date

", "rawMarkdown": "I think it would be like the old years of kaggle when kaggle take responsibity to ensure non commercial license. I remember there is a pretrain model thread.\n\nWhen in doubt, can always write email to proteinx author to confirm. \n\nWe can start a pretrain model thread, stating model, download url, license, cutoff date. All kaggle and host can post verified model.\n\nMost pretrain model are actively player of casp, which already required disclosure of cutoff date", "votes": null }, { "id": "3174126", "postDate": "04/08/2025 18:06:25", "content": "

Thanks for clearing this up, I was hesitant on what data to train my models on, now I can get started :)

", "rawMarkdown": "Thanks for clearing this up, I was hesitant on what data to train my models on, now I can get started :)", "votes": null }, { "id": "3174142", "postDate": "04/08/2025 18:30:05", "content": "

Good questions:

\n

would it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?
\nYes

\n

Additionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?
\nYes. But to ensure rigor, main prizes will be based only on 'future' RNA targets released in the PDB after close of notebook submissions. So all notebooks should automatically satisfy temporal_cutoff for the main prizes!

", "rawMarkdown": "Good questions:\n\n*would it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?*\nYes\n\n*Additionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?*\nYes. But to ensure rigor, main prizes will be based only on 'future' RNA targets released in the PDB after close of notebook submissions. So all notebooks should automatically satisfy `temporal_cutoff` for the main prizes!", "votes": null }, { "id": "3174255", "postDate": "04/08/2025 22:24:03", "content": "

FYI:

\n

The Protenix contributor officially stated v0.2.0's cutoff date is same as the paper (i.e. September 30, 2021).

\n

https://github.com/bytedance/Protenix/issues/103#issuecomment-2785317020

", "rawMarkdown": "**FYI:**\n\nThe Protenix contributor officially stated v0.2.0's cutoff date is same as the paper (i.e. `September 30, 2021`).\n\nhttps://github.com/bytedance/Protenix/issues/103#issuecomment-2785317020", "votes": null } ], "comments": [ { "id": 3173256, "author_name": "sacuscreed", "author_url": "", "post_date": "04/07/2025 17:37:18", "content": "

Actually I have no idea but I'd say according to providen rules at least for early sharing prize only no doubt model/trainings would be eligible.

", "votes": null, "replies": [] }, { "id": 3173289, "author_name": "hengck23", "author_url": "", "post_date": "04/07/2025 18:18:13", "content": "

\"When using pretrained models, the training data and its date range are rarely disclosed,\"
\nmost are disclosed (espeically if they are used in CASP)

", "votes": null, "replies": [ { "id": 3173293, "author_name": "sacuscreed", "author_url": "", "post_date": "04/07/2025 18:21:44", "content": "

And if not. The weights have not a date of uploading? At the web repository I mean.

", "votes": null, "replies": [ { "id": 3173451, "author_name": "tatamikenn", "author_url": "", "post_date": "04/07/2025 23:45:13", "content": "

Possibly, and I think it's most realistic trade-off. However, it might be too strict considering the time gap of data download and model upload…

", "votes": null, "replies": [ { "id": 3173461, "author_name": "hengck23", "author_url": "", "post_date": "04/08/2025 00:18:07", "content": "

It is mentioned in the paper or the webpage. I remember reading it

", "votes": null, "replies": [ { "id": 3173476, "author_name": "tatamikenn", "author_url": "", "post_date": "04/08/2025 00:55:11", "content": "

@hengck23
\nI see. I’ll check it out — thanks.
\nHowever, since the repository has been actively updated after the paper was published, I’m not sure whether version 2.0.0 still uses the same cut-off date as stated in the paper.

", "votes": null, "replies": [ { "id": 3173477, "author_name": "arunodhayan", "author_url": "", "post_date": "04/08/2025 00:57:39", "content": "

Protenix is trained with PDB cut-off 2021-09-30

", "votes": null, "replies": [] }, { "id": 3173478, "author_name": "hengck23", "author_url": "", "post_date": "04/08/2025 01:02:33", "content": "

I think it would be like the old years of kaggle when kaggle take responsibity to ensure non commercial license. I remember there is a pretrain model thread.

\n

When in doubt, can always write email to proteinx author to confirm.

\n

We can start a pretrain model thread, stating model, download url, license, cutoff date. All kaggle and host can post verified model.

\n

Most pretrain model are actively player of casp, which already required disclosure of cutoff date

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3173448, "author_name": "tatamikenn", "author_url": "", "post_date": "04/07/2025 23:28:43", "content": "

@hengck23

\n

For example, how do we know the release date of Protenix?

\n

https://github.com/bytedance/Protenix

\n

They seems updated weight version 2.0, but no model card seems to be released.

", "votes": null, "replies": [] } ] }, { "id": 3173400, "author_name": "asarvazyan", "author_url": "", "post_date": "04/07/2025 21:15:21", "content": "

Commenting for traction, I was also interested about this and got no response: https://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687

", "votes": null, "replies": [ { "id": 3173452, "author_name": "tatamikenn", "author_url": "", "post_date": "04/07/2025 23:47:56", "content": "

@asarvazyan
\nSorry, I missed this post. Thank you for sharing.

\n

I added this link to description.

", "votes": null, "replies": [] } ] }, { "id": 3173463, "author_name": "rhijudas", "author_url": "", "post_date": "04/08/2025 00:28:20", "content": "

Thanks @asarvazyan and @tatamikenn for bringing up these questions relevant to the Early Sharing Prize -- we are very glad to see you have interest!

\n", "votes": null, "replies": [ { "id": 3173473, "author_name": "tatamikenn", "author_url": "", "post_date": "04/08/2025 00:42:51", "content": "

@rhijudas

\n

Thank you for the detailed explanation — I understand the point.
\nRegarding pre-trained models without a declared cut-off date:
\nwould it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?

\n

Additionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?

", "votes": null, "replies": [ { "id": 3174142, "author_name": "rhijudas", "author_url": "", "post_date": "04/08/2025 18:30:05", "content": "

Good questions:

\n

would it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?
\nYes

\n

Additionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?
\nYes. But to ensure rigor, main prizes will be based only on 'future' RNA targets released in the PDB after close of notebook submissions. So all notebooks should automatically satisfy temporal_cutoff for the main prizes!

", "votes": null, "replies": [] } ] }, { "id": 3174126, "author_name": "asarvazyan", "author_url": "", "post_date": "04/08/2025 18:06:25", "content": "

Thanks for clearing this up, I was hesitant on what data to train my models on, now I can get started :)

", "votes": null, "replies": [] } ] }, { "id": 3174255, "author_name": "tatamikenn", "author_url": "", "post_date": "04/08/2025 22:24:03", "content": "

FYI:

\n

The Protenix contributor officially stated v0.2.0's cutoff date is same as the paper (i.e. September 30, 2021).

\n

https://github.com/bytedance/Protenix/issues/103#issuecomment-2785317020

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3173149": "rhijudas\n@shujun717\n@alissahummer\n\nHi, I’d like to follow up on this earlier post from the organizers about the Early Sharing Prize:\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/568059\n\nIn particular, the updated eligibility rule states:\n\n> “Submissions should only make use of information publicly available before the temporal_cutoff dates provided with test sequences.”\n\nI have two questions regarding this:\n\n1.\t**Since test sequences and their temporal_cutoff dates are not visible to participants**, it seems impossible to know in advance whether a given resource is allowed. How are participants expected to comply with this?\n2.\t**When using pretrained models**, the training data and its date range are rarely disclosed, and even if they are, it’s not possible to verify how the model was actually trained. How should we interpret the cutoff rule in these cases?\n\nWould appreciate any clarification. Thanks!\n\nBest,\nBilzard\n\n---\n**Edit**:\n\nRelated post:\nhttps://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687", "3173256": "Actually I have no idea but I'd say according to providen rules at least for early sharing prize only no doubt model/trainings would be eligible.", "3173289": "\"When using pretrained models, the training data and its date range are rarely disclosed,\"\nmost are disclosed (espeically if they are used in CASP)", "3173293": "And if not. The weights have not a date of uploading? At the web repository I mean.", "3173400": "Commenting for traction, I was also interested about this and got no response: https://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/569687", "3173448": "hengck23 \n\nFor example, how do we know the release date of Protenix?\n\nhttps://github.com/bytedance/Protenix\n\nThey seems updated weight version 2.0, but no model card seems to be released.", "3173451": "Possibly, and I think it's most realistic trade-off. However, it might be too strict considering the time gap of data download and model upload...", "3173452": "asarvazyan \nSorry, I missed this post. Thank you for sharing.\n\nI added this link to description.", "3173461": "It is mentioned in the paper or the webpage. I remember reading it", "3173463": "Thanks @asarvazyan and @tatamikenn for bringing up these questions relevant to the Early Sharing Prize -- we are very glad to see you have interest!\n\n- Some of the leaderboard targets may be drawn from CASP16, whose earliest deadline was 2024-05-16. (May 16, 2024). So that is the safest cutoff to use if you want to use a single cutoff date. However, you may want to also use one or more models that are trained on the very latest available information. In that case, you could have *two* or more models, one with cutoff date of 2024-05-16 and others with cutoff date closer to when you train them in April 2025. Then you should have your Notebook select which model to use based on the most recent one that is still earlier than the `temporal_cutoff` field for each target in `test_sequences.csv`.\n\n- If your model uses MSA's, the ones provided by us as hosts in `MSA/` will be safe.\n\n- If your model searches for 3D RNA structures from the PDB as templates, please do have the notebook pay attention to those structures' release dates in the PDB and not use any PDB with date before the `temporal_cutoff`.\n\n- If your model uses LLMs trained on scientific papers and preprints, you could either find the LLM's cutoff date or check that it is responsive to prompts that ask it to only use information that existed prior to each targets' `temporal_cutoff` date.\n\n- No matter what, it would be good to include comments in your Notebook documenting how you handled `temporal_cutoff`.", "3173473": "rhijudas \n\nThank you for the detailed explanation — I understand the point.\nRegarding pre-trained models without a declared cut-off date:\nwould it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?\n\nAdditionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?", "3173476": "hengck23 \nI see. I’ll check it out — thanks.\nHowever, since the repository has been actively updated after the paper was published, I’m not sure whether version 2.0.0 still uses the same cut-off date as stated in the paper.", "3173477": "Protenix is trained with PDB cut-off 2021-09-30", "3173478": "I think it would be like the old years of kaggle when kaggle take responsibity to ensure non commercial license. I remember there is a pretrain model thread.\n\nWhen in doubt, can always write email to proteinx author to confirm. \n\nWe can start a pretrain model thread, stating model, download url, license, cutoff date. All kaggle and host can post verified model.\n\nMost pretrain model are actively player of casp, which already required disclosure of cutoff date", "3174126": "Thanks for clearing this up, I was hesitant on what data to train my models on, now I can get started :)", "3174142": "Good questions:\n\n*would it be acceptable to use them if we can provide supporting evidence, such as the model’s upload date?*\nYes\n\n*Additionally, should this requirement also apply to the main competition winners (i.e., not the Early Sharing Prize)?*\nYes. But to ensure rigor, main prizes will be based only on 'future' RNA targets released in the PDB after close of notebook submissions. So all notebooks should automatically satisfy `temporal_cutoff` for the main prizes!", "3174255": "**FYI:**\n\nThe Protenix contributor officially stated v0.2.0's cutoff date is same as the paper (i.e. `September 30, 2021`).\n\nhttps://github.com/bytedance/Protenix/issues/103#issuecomment-2785317020" }, "source": "meta" }