{ "id": 327926, "title": "How to identify Public and Private", "url": "/competitions/amex-default-prediction/discussion/327926", "author_name": "Ryota", "post_date": "2022-05-30T03:03:14.511000", "votes": 89, "comment_count": 8, "views": 0, "content": "

Test data includes customer history ending in April 2019 and ending in October 2019.

\n

The number of each of them
\n2019-04 → 467966 (50.6%)
\n2019-10 → 456655 (49.4%)
\nThis is same as the ratio of public to private written on the leaderboard.

\n

So I submitted prediction that a customer with a history ending in 2019-10 have zero.
\nAs a result, scores remained unchanged from the original scores.

", "messages": [ { "id": 1805272, "postDate": "2022-05-30T03:03:14.510Z", "content": "

Test data includes customer history ending in April 2019 and ending in October 2019.

\n

The number of each of them
\n2019-04 → 467966 (50.6%)
\n2019-10 → 456655 (49.4%)
\nThis is same as the ratio of public to private written on the leaderboard.

\n

So I submitted prediction that a customer with a history ending in 2019-10 have zero.
\nAs a result, scores remained unchanged from the original scores.

", "rawMarkdown": "Test data includes customer history ending in April 2019 and ending in October 2019.\n\nThe number of each of them \n2019-04 → 467966 (50.6%)\n2019-10 → 456655 (49.4%)\nThis is same as the ratio of public to private written on the leaderboard.\n\nSo I submitted prediction that a customer with a history ending in 2019-10 have zero.\nAs a result, scores remained unchanged from the original scores.", "votes": 88 }, { "id": 1805575, "postDate": "2022-05-30T10:15:56.270Z", "content": "

Thanks @ryotak12 for this information.

\n

So, all the customers with history ending in April 2019 belongs to Public Test dataset and those with history ending in October 2019 belongs to Private Test dataset.

\n

Then we need to be careful about not to overfit the public dataset too much, as the private dataset is from a different time frame.

", "rawMarkdown": "Thanks @ryotak12 for this information.\n\nSo, all the customers with history ending in `April 2019` belongs to `Public Test dataset` and those with history ending in `October 2019` belongs to `Private Test dataset`.\n\nThen we need to be careful about not to overfit the public dataset too much, as the private dataset is from a different time frame. ", "votes": 9, "replies": [ { "id": 1808048, "postDate": "2022-06-01T13:56:08.960Z", "content": "

You are right.
\nAs @lucasmorin said in this discussion, I think it is important to see how the distribution differs between public and private.

", "rawMarkdown": "You are right.\nAs @lucasmorin said in this discussion, I think it is important to see how the distribution differs between public and private.", "votes": 2 }, { "id": 1813327, "postDate": "2022-06-06T17:43:44.643Z", "content": "

Quick illustration to show the train data / public LB / private LB datasets… chart below shows the Max Date per Customer … grouped up by month… and the y-axis (bars) are customer counts.

\n

(used Tableau to create)

\n

", "rawMarkdown": "Quick illustration to show the train data / public LB / private LB datasets... chart below shows the Max Date per Customer ... grouped up by month... and the y-axis (bars) are customer counts. \n\n(used Tableau to create)\n\n", "votes": 12 } ] }, { "id": 1815309, "postDate": "2022-06-08T22:52:52.600Z", "content": "

Thanks for this information.
\nI have tried Adversarial Validation.
\nhttps://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public

\n

It seems that some more analysis is needed.

", "rawMarkdown": "Thanks for this information.\r\nI have tried Adversarial Validation.\r\nhttps://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public\r\n\r\nIt seems that some more analysis is needed.", "votes": 5 }, { "id": 1806128, "postDate": "2022-05-30T20:05:19.617Z", "content": "

I found that too and used it to try to see if data is distributed differently between Public And Private LB with UMAP/Hdbscan.

", "rawMarkdown": "I found that too and used it to try to see if data is distributed differently between Public And Private LB with UMAP/Hdbscan.", "votes": 2, "replies": [ { "id": 1808041, "postDate": "2022-06-01T13:49:30.473Z", "content": "

That's is nice idea. I will try it too

", "rawMarkdown": "That's is nice idea. I will try it too" } ] }, { "id": 1904599, "postDate": "2022-08-18T10:31:23.887Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1904654, "postDate": "2022-08-18T11:32:08.330Z", "content": "

Yes, test dataset contains private.
\nSubmissions are made by submitting files instead of notebooks in this competition.
\nTherefore, even if the data is hidden, it cannot be evaluated.

", "rawMarkdown": "Yes, test dataset contains private.\nSubmissions are made by submitting files instead of notebooks in this competition.\nTherefore, even if the data is hidden, it cannot be evaluated.", "votes": 2 } ] } ], "comments": [ { "id": 1805575, "author_name": "SRK", "author_url": "", "post_date": "2022-05-30T10:15:56.270000", "content": "

Thanks @ryotak12 for this information.

\n

So, all the customers with history ending in April 2019 belongs to Public Test dataset and those with history ending in October 2019 belongs to Private Test dataset.

\n

Then we need to be careful about not to overfit the public dataset too much, as the private dataset is from a different time frame.

", "votes": 9, "replies": [ { "id": 1808048, "author_name": "Ryota", "author_url": "", "post_date": "2022-06-01T13:56:08.960000", "content": "

You are right.
\nAs @lucasmorin said in this discussion, I think it is important to see how the distribution differs between public and private.

", "votes": 2, "replies": [] }, { "id": 1813327, "author_name": "David Dirring", "author_url": "", "post_date": "2022-06-06T17:43:44.643000", "content": "

Quick illustration to show the train data / public LB / private LB datasets… chart below shows the Max Date per Customer … grouped up by month… and the y-axis (bars) are customer counts.

\n

(used Tableau to create)

\n

", "votes": 12, "replies": [] } ] }, { "id": 1815309, "author_name": "zakopuro", "author_url": "", "post_date": "2022-06-08T22:52:52.600000", "content": "

Thanks for this information.
\nI have tried Adversarial Validation.
\nhttps://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public

\n

It seems that some more analysis is needed.

", "votes": 5, "replies": [] }, { "id": 1806128, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-05-30T20:05:19.617000", "content": "

I found that too and used it to try to see if data is distributed differently between Public And Private LB with UMAP/Hdbscan.

", "votes": 2, "replies": [ { "id": 1808041, "author_name": "Ryota", "author_url": "", "post_date": "2022-06-01T13:49:30.473000", "content": "

That's is nice idea. I will try it too

", "votes": 0, "replies": [] } ] }, { "id": 1904599, "author_name": "", "author_url": "", "post_date": "2022-08-18T10:31:23.887000", "content": "", "votes": 1, "replies": [ { "id": 1904654, "author_name": "Ryota", "author_url": "", "post_date": "2022-08-18T11:32:08.330000", "content": "

Yes, test dataset contains private.
\nSubmissions are made by submitting files instead of notebooks in this competition.
\nTherefore, even if the data is hidden, it cannot be evaluated.

", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1805272": "Test data includes customer history ending in April 2019 and ending in October 2019.\n\nThe number of each of them \n2019-04 → 467966 (50.6%)\n2019-10 → 456655 (49.4%)\nThis is same as the ratio of public to private written on the leaderboard.\n\nSo I submitted prediction that a customer with a history ending in 2019-10 have zero.\nAs a result, scores remained unchanged from the original scores.", "1805575": "Thanks @ryotak12 for this information.\n\nSo, all the customers with history ending in `April 2019` belongs to `Public Test dataset` and those with history ending in `October 2019` belongs to `Private Test dataset`.\n\nThen we need to be careful about not to overfit the public dataset too much, as the private dataset is from a different time frame. ", "1815309": "Thanks for this information.\r\nI have tried Adversarial Validation.\r\nhttps://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public\r\n\r\nIt seems that some more analysis is needed.", "1806128": "I found that too and used it to try to see if data is distributed differently between Public And Private LB with UMAP/Hdbscan.", "1904599": "" } }