{ "id": 344808, "title": "Some Questions about Private Evaluation (Reproducibility Specs)", "url": "/competitions/amex-default-prediction/discussion/344808", "author_name": "", "post_date": "2022-08-16T17:24:08.077841400Z", "votes": 3, "comment_count": 8, "views": 0, "content": "

With regards to private standings, I was wondering, what are the specifications for reproducing your work?

\n

I had assumed that you would need to show the host/Kaggle your end-to-end pipeline at some point but some conversations over the recent 0.8 notebook controversy made me realise I don't really understand private evaluation.

\n

For example, could someone theoretically win by just taking public submissions and ensembling them, even though they have no idea what the feature engineering/models are?

\n

Someone commented that you can ensemble your work together with some public submissions, is this true?

\n

Note: For clarity I'm talking about the private evaluation period only.
\nP.S. Sorry for the silly/obtuse question

", "messages": [ { "id": "1901492", "postDate": "08/16/2022 17:24:08", "content": "

With regards to private standings, I was wondering, what are the specifications for reproducing your work?

\n

I had assumed that you would need to show the host/Kaggle your end-to-end pipeline at some point but some conversations over the recent 0.8 notebook controversy made me realise I don't really understand private evaluation.

\n

For example, could someone theoretically win by just taking public submissions and ensembling them, even though they have no idea what the feature engineering/models are?

\n

Someone commented that you can ensemble your work together with some public submissions, is this true?

\n

Note: For clarity I'm talking about the private evaluation period only.
\nP.S. Sorry for the silly/obtuse question

", "rawMarkdown": "With regards to private standings, I was wondering, what are the specifications for reproducing your work?\n\nI had assumed that you would need to show the host/Kaggle your end-to-end pipeline at some point but some conversations over the recent 0.8 notebook controversy made me realise I don't really understand private evaluation.\n\nFor example, could someone theoretically win by just taking public submissions and ensembling them, even though they have no idea what the feature engineering/models are?\n\nSomeone commented that you can ensemble your work together with some public submissions, is this true?\n\n\nNote: For clarity I'm talking about the private evaluation period only.\nP.S. Sorry for the silly/obtuse question", "votes": null }, { "id": "1901557", "postDate": "08/16/2022 18:31:20", "content": "
\n

You acknowledge that at the end of the competition, the American Express team may request to review your model for purposes of reviewing your capabilities for the job. This license is limited for recruiting and review purposes only.

\n
\n

See below

", "rawMarkdown": "> You acknowledge that at the end of the competition, the American Express team may request to review your model for purposes of reviewing your capabilities for the job. This license is limited for recruiting and review purposes only.\n\n~~Seems only needed if you want the interview~~ See below", "votes": null }, { "id": "1901640", "postDate": "08/16/2022 19:42:17", "content": "

hey
\n@julianmukaj i don't understand why your submission with 0.8 LB not valid on the private submission

\n

i am just want to know how the private score will calaculated, And why your submsission is not valid.

", "rawMarkdown": "hey \n@julianmukaj i don't understand why your submission with 0.8 LB not valid on the private submission \n\ni am just want to know how the private score will calaculated, And why your submsission is not valid.", "votes": null }, { "id": "1901733", "postDate": "08/16/2022 21:10:35", "content": "

The submission is 51% public LB, 49% private LB. You are seeing the scores for the 51% with a 0.800 submission, but not the remaining 49%.

", "rawMarkdown": "The submission is 51% public LB, 49% private LB. You are seeing the scores for the 51% with a 0.800 submission, but not the remaining 49%.", "votes": null }, { "id": "1901781", "postDate": "08/16/2022 22:49:19", "content": "

In order to win medals Gold, Silver, Bronze (and Kaggle ranking pionts), you do not need to submit your source code. In order to win cash prizes you do need to submit reproducible code (that generates your winning private LB score). Your code is reviewed and if Kaggle and the host are satified, then you receive your cash prize. See rules section 11:

\n
\n
    \n
  1. WINNERS OBLIGATIONS.
    \n As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:
    2. \n
\n

(a) deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation.

\n
", "rawMarkdown": "In order to win medals Gold, Silver, Bronze (and Kaggle ranking pionts), you do not need to submit your source code. In order to win cash prizes you do need to submit reproducible code (that generates your winning private LB score). Your code is reviewed and if Kaggle and the host are satified, then you receive your cash prize. See rules section 11:\n\n>11. WINNERS OBLIGATIONS.\nAs a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:\n\n>(a) deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation.", "votes": null }, { "id": "1901798", "postDate": "08/16/2022 23:31:18", "content": "

That clears everything up, appreciate it!

", "rawMarkdown": "That clears everything up, appreciate it!", "votes": null }, { "id": "1901815", "postDate": "08/17/2022 00:11:26", "content": "

@cdeotte, to what exact degree do I need to reproduce solutions? I doubt that I will be able to reproduce every model training considering that many training runs happened without deterministic behavior (e.g. GPU calculations).

", "rawMarkdown": "cdeotte, to what exact degree do I need to reproduce solutions? I doubt that I will be able to reproduce every model training considering that many training runs happened without deterministic behavior (e.g. GPU calculations).", "votes": null }, { "id": "1903094", "postDate": "08/17/2022 05:16:14", "content": "

@fritzcremer Don't worry about non-determinism. Kaggle and the host understand this. If you provide code to reproduce your solution (the same that you used for your final submission even if it is non-deterministic), they accept a result that achieves close to your final LB score.

", "rawMarkdown": "fritzcremer Don't worry about non-determinism. Kaggle and the host understand this. If you provide code to reproduce your solution (the same that you used for your final submission even if it is non-deterministic), they accept a result that achieves close to your final LB score.", "votes": null }, { "id": "1903098", "postDate": "08/17/2022 05:24:05", "content": "

Here is the link that Kaggle sends to cash winners here after a competition finishes. In the beginning of section B we see that Kaggle says:

\n
\n

The core requirement is that you detail all the pieces needed by the host to reproduce your solution with the score your team achieved on the leaderboard within a reasonable margin.

\n
\n

Notice that they say \"within a reasonable margin\" to allow for non-determinism.

", "rawMarkdown": "Here is the link that Kaggle sends to cash winners [here][1] after a competition finishes. In the beginning of section B we see that Kaggle says:\n>The core requirement is that you detail all the pieces needed by the host to reproduce your solution with the score your team achieved on the leaderboard within a reasonable margin.\n\nNotice that they say \"within a reasonable margin\" to allow for non-determinism.\n\n[1]: https://www.kaggle.com/WinningModelDocumentationGuidelines", "votes": null } ], "comments": [ { "id": 1901557, "author_name": "julianmukaj", "author_url": "", "post_date": "08/16/2022 18:31:20", "content": "
\n

You acknowledge that at the end of the competition, the American Express team may request to review your model for purposes of reviewing your capabilities for the job. This license is limited for recruiting and review purposes only.

\n
\n

See below

", "votes": null, "replies": [ { "id": 1901640, "author_name": "abdellatifsassioui", "author_url": "", "post_date": "08/16/2022 19:42:17", "content": "

hey
\n@julianmukaj i don't understand why your submission with 0.8 LB not valid on the private submission

\n

i am just want to know how the private score will calaculated, And why your submsission is not valid.

", "votes": null, "replies": [] }, { "id": 1901733, "author_name": "julianmukaj", "author_url": "", "post_date": "08/16/2022 21:10:35", "content": "

The submission is 51% public LB, 49% private LB. You are seeing the scores for the 51% with a 0.800 submission, but not the remaining 49%.

", "votes": null, "replies": [] } ] }, { "id": 1901781, "author_name": "cdeotte", "author_url": "", "post_date": "08/16/2022 22:49:19", "content": "

In order to win medals Gold, Silver, Bronze (and Kaggle ranking pionts), you do not need to submit your source code. In order to win cash prizes you do need to submit reproducible code (that generates your winning private LB score). Your code is reviewed and if Kaggle and the host are satified, then you receive your cash prize. See rules section 11:

\n
\n
    \n
  1. WINNERS OBLIGATIONS.
    \n As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:
    2. \n
\n

(a) deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation.

\n
", "votes": null, "replies": [ { "id": 1901798, "author_name": "lilgaussy", "author_url": "", "post_date": "08/16/2022 23:31:18", "content": "

That clears everything up, appreciate it!

", "votes": null, "replies": [] }, { "id": 1901815, "author_name": "fritzcremer", "author_url": "", "post_date": "08/17/2022 00:11:26", "content": "

@cdeotte, to what exact degree do I need to reproduce solutions? I doubt that I will be able to reproduce every model training considering that many training runs happened without deterministic behavior (e.g. GPU calculations).

", "votes": null, "replies": [] }, { "id": 1903094, "author_name": "cdeotte", "author_url": "", "post_date": "08/17/2022 05:16:14", "content": "

@fritzcremer Don't worry about non-determinism. Kaggle and the host understand this. If you provide code to reproduce your solution (the same that you used for your final submission even if it is non-deterministic), they accept a result that achieves close to your final LB score.

", "votes": null, "replies": [] }, { "id": 1903098, "author_name": "cdeotte", "author_url": "", "post_date": "08/17/2022 05:24:05", "content": "

Here is the link that Kaggle sends to cash winners here after a competition finishes. In the beginning of section B we see that Kaggle says:

\n
\n

The core requirement is that you detail all the pieces needed by the host to reproduce your solution with the score your team achieved on the leaderboard within a reasonable margin.

\n
\n

Notice that they say \"within a reasonable margin\" to allow for non-determinism.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1901492": "With regards to private standings, I was wondering, what are the specifications for reproducing your work?\n\nI had assumed that you would need to show the host/Kaggle your end-to-end pipeline at some point but some conversations over the recent 0.8 notebook controversy made me realise I don't really understand private evaluation.\n\nFor example, could someone theoretically win by just taking public submissions and ensembling them, even though they have no idea what the feature engineering/models are?\n\nSomeone commented that you can ensemble your work together with some public submissions, is this true?\n\n\nNote: For clarity I'm talking about the private evaluation period only.\nP.S. Sorry for the silly/obtuse question", "1901557": "> You acknowledge that at the end of the competition, the American Express team may request to review your model for purposes of reviewing your capabilities for the job. This license is limited for recruiting and review purposes only.\n\n~~Seems only needed if you want the interview~~ See below", "1901640": "hey \n@julianmukaj i don't understand why your submission with 0.8 LB not valid on the private submission \n\ni am just want to know how the private score will calaculated, And why your submsission is not valid.", "1901733": "The submission is 51% public LB, 49% private LB. You are seeing the scores for the 51% with a 0.800 submission, but not the remaining 49%.", "1901781": "In order to win medals Gold, Silver, Bronze (and Kaggle ranking pionts), you do not need to submit your source code. In order to win cash prizes you do need to submit reproducible code (that generates your winning private LB score). Your code is reviewed and if Kaggle and the host are satified, then you receive your cash prize. See rules section 11:\n\n>11. WINNERS OBLIGATIONS.\nAs a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:\n\n>(a) deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation.", "1901798": "That clears everything up, appreciate it!", "1901815": "cdeotte, to what exact degree do I need to reproduce solutions? I doubt that I will be able to reproduce every model training considering that many training runs happened without deterministic behavior (e.g. GPU calculations).", "1903094": "fritzcremer Don't worry about non-determinism. Kaggle and the host understand this. If you provide code to reproduce your solution (the same that you used for your final submission even if it is non-deterministic), they accept a result that achieves close to your final LB score.", "1903098": "Here is the link that Kaggle sends to cash winners [here][1] after a competition finishes. In the beginning of section B we see that Kaggle says:\n>The core requirement is that you detail all the pieces needed by the host to reproduce your solution with the score your team achieved on the leaderboard within a reasonable margin.\n\nNotice that they say \"within a reasonable margin\" to allow for non-determinism.\n\n[1]: https://www.kaggle.com/WinningModelDocumentationGuidelines" }, "source": "meta" }