{ "id": 246782, "title": "Time Stamp Leakage - Update", "url": "/competitions/seti-breakthrough-listen/discussion/246782", "author_name": "inversion", "post_date": "2021-06-17T02:17:25.300000", "votes": 111, "comment_count": 58, "views": 0, "content": "

Much appreciation to long-time Kaggler @kazanova for bringing forward the issue of time-stamp leakage. This definitely falls into the \"mistakes were made\" category.

\n

Since it was my mistake, I'll share how it happened.

\n

There are just a few of us data scientists at Kaggle launching about 50 competitions a year with many different data types over a very wide range of domains. Worrying about leakage and other failure points keeps us up at night. We absolutely value our community's time and effort and know how important it is to have fun and challenging competitions.

\n

To minimize the chance of leakage in the competitions we launch, we have a well-defined set of standard processes, which includes a final data check by a different data scientist before launch.

\n

Generally we randomize file save order in code, which is the best practice. But because of the complexity of the incoming data from the host, I opted to do a manual touch of the files after they were written. This was all fine and dandy except when I did an additional data check on the \"Download All\" dataset I found there were files missing. This led me on a multi-day debugging spree that ended up in multiple re-writes and uploads of the data before we eventually discovered it was a bug in our back-end randomly dropping a few files in the zip process.

\n

When it was all said and done, the final upload ended up not getting the time-stamp reset.

\n

We'll be putting a down-stream (i.e., the data downloaded from the Data page) check into our processes to avoid this issue in the future. And I certainly apologize for my mistake here.

\n

We're working with the host to gather a new Test dataset. It will be generated very similarly, so the work everyone has done will be preserved. We'll post again once we have timing on that reset.

", "messages": [ { "id": 1353238, "postDate": "2021-06-17T02:17:25.300Z", "content": "

Much appreciation to long-time Kaggler @kazanova for bringing forward the issue of time-stamp leakage. This definitely falls into the \"mistakes were made\" category.

\n

Since it was my mistake, I'll share how it happened.

\n

There are just a few of us data scientists at Kaggle launching about 50 competitions a year with many different data types over a very wide range of domains. Worrying about leakage and other failure points keeps us up at night. We absolutely value our community's time and effort and know how important it is to have fun and challenging competitions.

\n

To minimize the chance of leakage in the competitions we launch, we have a well-defined set of standard processes, which includes a final data check by a different data scientist before launch.

\n

Generally we randomize file save order in code, which is the best practice. But because of the complexity of the incoming data from the host, I opted to do a manual touch of the files after they were written. This was all fine and dandy except when I did an additional data check on the \"Download All\" dataset I found there were files missing. This led me on a multi-day debugging spree that ended up in multiple re-writes and uploads of the data before we eventually discovered it was a bug in our back-end randomly dropping a few files in the zip process.

\n

When it was all said and done, the final upload ended up not getting the time-stamp reset.

\n

We'll be putting a down-stream (i.e., the data downloaded from the Data page) check into our processes to avoid this issue in the future. And I certainly apologize for my mistake here.

\n

We're working with the host to gather a new Test dataset. It will be generated very similarly, so the work everyone has done will be preserved. We'll post again once we have timing on that reset.

", "rawMarkdown": "Much appreciation to long-time Kaggler @kazanova for [bringing forward the issue of time-stamp leakage](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246772). This definitely falls into the \"mistakes were made\" category.\n\nSince it was my mistake, I'll share how it happened.\n\nThere are just a few of us data scientists at Kaggle launching about 50 competitions a year with many different data types over a very wide range of domains. Worrying about leakage and other failure points keeps us up at night. We absolutely value our community's time and effort and know how important it is to have fun and challenging competitions.\n\nTo minimize the chance of leakage in the competitions we launch, we have a well-defined set of standard processes, which includes a final data check by a different data scientist before launch.\n\nGenerally we randomize file save order in code, which is the best practice. But because of the complexity of the incoming data from the host, I opted to do a manual touch of the files after they were written. This was all fine and dandy *except* when I did an additional data check on the \"Download All\" dataset I found there were files missing. This led me on a multi-day debugging spree that ended up in multiple re-writes and uploads of the data before we eventually discovered it was a bug in our back-end randomly dropping a few files in the zip process.\n\nWhen it was all said and done, the final upload ended up not getting the time-stamp reset.\n\nWe'll be putting a down-stream (i.e., the data downloaded from the Data page) check into our processes to avoid this issue in the future. And I certainly apologize for my mistake here.\n\nWe're working with the host to gather a new Test dataset. It will be generated very similarly, so the work everyone has done will be preserved. We'll post again once we have timing on that reset.\n", "votes": 111 }, { "id": 1353255, "postDate": "2021-06-17T02:37:20.107Z", "content": "

Thank you for the quick response @inversion

\n

Great to see that such process is in place and that you are going to further improve it. that is all we can ask for!

\n

There is no harm done since we will be getting a new test dataset.

\n

On the matter of the leaks . I think kaggle's stance should be a bit more clear. In my opinion kaggle should

\n

1) encourage participants to reveal the leaks (so that the competition is valuable to the host after eliminating them), making it also fairer for the participants.
\n2) not reward solutions with leaks

\n

I understand that the definition of a leak can be debatable. I think it is fine to draw a line. For example a solution that finds the hidden number generator that is used to generate the test data should not be allowed to win.

", "rawMarkdown": "Thank you for the quick response @inversion \n\nGreat to see that such process is in place and that you are going to further improve it. that is all we can ask for! \n\nThere is no harm done since we will be getting a new test dataset.\n\nOn the matter of the leaks . I think kaggle's stance should be a bit more clear. In my opinion kaggle should\n\n1) encourage participants to reveal the leaks (so that the competition is valuable to the host after eliminating them), making it also fairer for the participants.\n2) not reward solutions with leaks\n\nI understand that the definition of a leak can be debatable. I think it is fine to draw a line. For example a solution that finds the hidden number generator that is used to generate the test data should not be allowed to win. ", "votes": 42, "replies": [ { "id": 1353486, "postDate": "2021-06-17T06:15:23.080Z", "content": "

Thanks Marios for quickly finding it and sharing! I think such an obvious and very simple leak should always be shared immediately, but you are right that there should be some incentives to do so. If it would not have been for the third digit, it would have probably taken us a longer while to figure it out…

", "rawMarkdown": "Thanks Marios for quickly finding it and sharing! I think such an obvious and very simple leak should always be shared immediately, but you are right that there should be some incentives to do so. If it would not have been for the third digit, it would have probably taken us a longer while to figure it out...", "votes": 3 }, { "id": 1353716, "postDate": "2021-06-17T08:14:37.170Z", "content": "

Marios, while I agree on the intent, drawing the line between leak and no leak is very difficult. I remember we had an argument in the first Quora competition on whether question frequency was a leak (your position) or not a leak (my position). I won't restart the argument here, I mention it just as an example of how defining leak can be tricky.

", "rawMarkdown": "Marios, while I agree on the intent, drawing the line between leak and no leak is very difficult. I remember we had an argument in the first Quora competition on whether question frequency was a leak (your position) or not a leak (my position). I won't restart the argument here, I mention it just as an example of how defining leak can be tricky.", "votes": 6 }, { "id": 1355143, "postDate": "2021-06-18T05:56:47.223Z", "content": "

@inversion How about keeping a different prize dedicated to finding leakage like ethical hacking? I think this will motivate other people to find the leakage but not for solution.

", "rawMarkdown": "@inversion How about keeping a different prize dedicated to **finding leakage** like **ethical hacking**? I think this will motivate other people to find the leakage but not for solution.", "votes": 3 }, { "id": 1355438, "postDate": "2021-06-18T09:30:21.097Z", "content": "
\n

@inversion How about keeping a different prize dedicated to finding leakage like ethical hacking? I think this will motivate other people to find the leakage but not for solution.

\n
\n

Award some kaggle ranking points to people finding leakage that leads to restarts.

", "rawMarkdown": ">@inversion How about keeping a different prize dedicated to finding leakage like ethical hacking? I think this will motivate other people to find the leakage but not for solution.\n\nAward some kaggle ranking points to people finding leakage that leads to restarts. ", "votes": 3 }, { "id": 1355441, "postDate": "2021-06-18T09:31:30.913Z", "content": "

Yeah, let's have Leak GM category.

\n

I'm serious.

", "rawMarkdown": "Yeah, let's have Leak GM category.\n\nI'm serious.", "votes": 5 }, { "id": 1355838, "postDate": "2021-06-18T15:03:40.153Z", "content": "

@kazanova this doesn't necessarily have to be a kaggle ranking point it could prize money. If one can find it he'll get the prize. If there is no leak no one will get the prize and this prize could be funded by kaggle.
\nI think there is Bug bounty program in every tech-giant company why not here?

", "rawMarkdown": "@kazanova this doesn't necessarily have to be a **kaggle ranking point** it could **prize money**. If one can find it he'll get the prize. If there is no leak no one will get the prize and this prize could be funded by **kaggle**.\nI think there is **Bug bounty program** in every tech-giant company why not here? ", "votes": 3 }, { "id": 1357032, "postDate": "2021-06-19T12:21:53.337Z", "content": "

i agreed to that it. if you get a leak, you have to say to stuff. To not be unfair

", "rawMarkdown": "i agreed to that it. if you get a leak, you have to say to stuff. To not be unfair" }, { "id": 1359540, "postDate": "2021-06-21T10:55:07.337Z", "content": "

Please bring in \"Kaggle leak bounty\". Starting with new data leak playground competitions. Bug bounty has been beneficial to a lot of companies out there and their contribution in terms of security is huge. On the other side, Kaggle would benefit understanding \"leaky\" perspective from participants

", "rawMarkdown": "Please bring in **\"Kaggle leak bounty\"**. Starting with new data leak playground competitions. Bug bounty has been beneficial to a lot of companies out there and their contribution in terms of security is huge. On the other side, Kaggle would benefit understanding \"leaky\" perspective from participants" } ] }, { "id": 1354175, "postDate": "2021-06-17T12:26:05.043Z", "content": "

@inversion I've been on the top of LB for a while and got score 0.999 on LB without any usage of metadata. I am ready to share upon request my solution with you and SETI if needed to investigate if there are any leakages beside metadata leakage.

", "rawMarkdown": "@inversion I've been on the top of LB for a while and got score 0.999 on LB without any usage of metadata. I am ready to share upon request my solution with you and SETI if needed to investigate if there are any leakages beside metadata leakage.", "votes": 23, "replies": [ { "id": 1357038, "postDate": "2021-06-19T12:25:31.197Z", "content": "

wow.that very suprise for me.

", "rawMarkdown": "wow.that very suprise for me." } ] }, { "id": 1353612, "postDate": "2021-06-17T07:29:55.393Z", "content": "

@inversion , please, take a careful look at the data before releasing the new test set. I wasn’t using the timestamp nor any other metadata, and still getting 0.999 public LB. I was thinking about some kind of steganographic alien message hidden in the power spectra, but maybe there is a subtle leak, related to how the images where processed.

", "rawMarkdown": "@inversion , please, take a careful look at the *data* before releasing the new test set. I wasn’t using the timestamp nor any other metadata, and still getting 0.999 public LB. I was thinking about some kind of steganographic alien message hidden in the power spectra, but maybe there is a subtle leak, related to how the images where processed.", "votes": 15, "replies": [ { "id": 1353654, "postDate": "2021-06-17T07:43:08.363Z", "content": "

@vicensgaitan thanks for sharing, I was already expecting there to be some more subtle leak, not related to the timestamp. At least that was my first guess after seeing the 0.999 scores.

\n

If you are willing to share (ignore otherwise): are you using any external data? I was also wondering whether a leak could exist there.

", "rawMarkdown": "@vicensgaitan thanks for sharing, I was already expecting there to be some more subtle leak, not related to the timestamp. At least that was my first guess after seeing the 0.999 scores.\n\nIf you are willing to share (ignore otherwise): are you using any external data? I was also wondering whether a leak could exist there.", "votes": 4 }, { "id": 1353692, "postDate": "2021-06-17T07:58:25.313Z", "content": "

No external data. I'm using only the provided images.

", "rawMarkdown": "No external data. I'm using only the provided images.", "votes": 3 }, { "id": 1353706, "postDate": "2021-06-17T08:10:21.947Z", "content": "

Vicens, why don't you share privately with Kaggle the leak you found?

", "rawMarkdown": "Vicens, why don't you share privately with Kaggle the leak you found?", "votes": 5 }, { "id": 1353719, "postDate": "2021-06-17T08:16:10.097Z", "content": "
\n

Vicens, why don't you share privately with Kaggle the leak you found?

\n
\n

I would say, he MUST share now to avoid another reset later on.

", "rawMarkdown": ">Vicens, why don't you share privately with Kaggle the leak you found?\n\nI would say, he **MUST** share now to avoid another reset later on. ", "votes": 8 }, { "id": 1353728, "postDate": "2021-06-17T08:18:09.497Z", "content": "

I think he does not know what kind of leak it exactly is, but it seems obvious that his model / approach is picking something up. Other top contestant said similarly in the other thread. But if he has a hunch, or knows what could be an issue, it should be discussed.

", "rawMarkdown": "I think he does not know what kind of leak it exactly is, but it seems obvious that his model / approach is picking something up. Other top contestant said similarly in the other thread. But if he has a hunch, or knows what could be an issue, it should be discussed.", "votes": 8 }, { "id": 1353746, "postDate": "2021-06-17T08:22:15.217Z", "content": "

Then privately sharing his model with Kaggle could be useful.

", "rawMarkdown": "Then privately sharing his model with Kaggle could be useful.", "votes": 3 }, { "id": 1353781, "postDate": "2021-06-17T08:37:50.833Z", "content": "

I'm still triying to figure out what is actually the leak. I'll share with Kaggle what i'm doing and maybe this light a bulb.

", "rawMarkdown": "I'm still triying to figure out what is actually the leak. I'll share with Kaggle what i'm doing and maybe this light a bulb.", "votes": 16 }, { "id": 1354163, "postDate": "2021-06-17T12:14:22.357Z", "content": "

FYI @inversion I think problem is downcast truncating. For more information check this little example.

", "rawMarkdown": "FYI @inversion I think problem is downcast truncating. For more information check this [little example](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901#1354157).", "votes": 5 }, { "id": 1354297, "postDate": "2021-06-17T14:03:57Z", "content": "

At this point, I'm thinking it might be easier to just try to get actual ET signals. 😁

", "rawMarkdown": "At this point, I'm thinking it might be easier to just try to get *actual* ET signals. 😁", "votes": 34 }, { "id": 1354524, "postDate": "2021-06-17T16:24:54.697Z", "content": "

Will finally get some Area 51 secrets. :P

", "rawMarkdown": "Will finally get some Area 51 secrets. :P", "votes": 4 } ] }, { "id": 1353288, "postDate": "2021-06-17T03:10:06.160Z", "content": "

Since the test set is gonna to be renew, is it possible to make the old test set label public to all participants? As the leakage has affect the fair of the competition.

", "rawMarkdown": "Since the test set is gonna to be renew, is it possible to make the old test set label public to all participants? As the leakage has affect the fair of the competition.", "votes": 11, "replies": [ { "id": 1353296, "postDate": "2021-06-17T03:23:35.407Z", "content": "

Yes, that's the plan.

", "rawMarkdown": "Yes, that's the plan.", "votes": 11 }, { "id": 1353315, "postDate": "2021-06-17T03:43:18.053Z", "content": "

Thank you for your effort!

", "rawMarkdown": "Thank you for your effort!", "votes": 3 }, { "id": 1357034, "postDate": "2021-06-19T12:23:10.290Z", "content": "

oh. thank your effort.

", "rawMarkdown": "oh. thank your effort." } ] }, { "id": 1353500, "postDate": "2021-06-17T06:26:39.880Z", "content": "

Thank you @inversion!

\n

Probably you dont know yet, but are you able to make an estimate of when new test data will be revealed? Are we speaking of days or weeks?

\n

Also, could you make the current test labels available soon? Then, people can continue working on it.

", "rawMarkdown": "Thank you @inversion!\n\nProbably you dont know yet, but are you able to make an estimate of when new test data will be revealed? Are we speaking of days or weeks?\n\nAlso, could you make the current test labels available soon? Then, people can continue working on it. ", "votes": 5, "replies": [ { "id": 1357035, "postDate": "2021-06-19T12:23:48.490Z", "content": "

mee too. I wat to know that.

", "rawMarkdown": "mee too. I wat to know that." } ] }, { "id": 1353286, "postDate": "2021-06-17T03:05:59.647Z", "content": "
\n

Worrying about leakage and other failure points keeps us up at night.

\n
\n

Sounds like you need a hug :)
\nThank you for the hard work, and all for our fun. Big respect to the nobleman who found the leak and reveal it.

", "rawMarkdown": "> Worrying about leakage and other failure points keeps us up at night. \n\nSounds like you need a hug :) \nThank you for the hard work, and all for our fun. Big respect to the nobleman who found the leak and reveal it. ", "votes": 5, "replies": [ { "id": 1353297, "postDate": "2021-06-17T03:24:05.773Z", "content": "

Thanks! I'm constantly amazed at what an awesome community we have, willing to help and share.

", "rawMarkdown": "Thanks! I'm constantly amazed at what an awesome community we have, willing to help and share.", "votes": 8 } ] }, { "id": 1358107, "postDate": "2021-06-20T07:43:20.500Z", "content": "

will there be any update in competition timelines too due to leak?

", "rawMarkdown": "will there be any update in competition timelines too due to leak?", "votes": 6 }, { "id": 1354333, "postDate": "2021-06-17T14:38:32.047Z", "content": "

@inversion , will train/test set changes also take into account this leakage ?

", "rawMarkdown": "@inversion , will train/test set changes also take into account [this leakage](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901) ? ", "votes": 3 }, { "id": 1359165, "postDate": "2021-06-21T05:42:33.963Z", "content": "

I am wondering if the submission count will be reset as well.

", "rawMarkdown": "I am wondering if the submission count will be reset as well.", "votes": 1, "replies": [ { "id": 1359589, "postDate": "2021-06-21T11:30:23.940Z", "content": "

Why would it be?

", "rawMarkdown": "Why would it be?", "votes": 3 }, { "id": 1360157, "postDate": "2021-06-21T21:13:57.423Z", "content": "

Multiple reasons. For example, competitors who spent many submissions trying to find the leak has a disadvantage in terms of merging with other teams.

", "rawMarkdown": "Multiple reasons. For example, competitors who spent many submissions trying to find the leak has a disadvantage in terms of merging with other teams.", "votes": 1 }, { "id": 1360695, "postDate": "2021-06-22T09:37:20.857Z", "content": "

??

\n

If people search for leaks instead of searching for good models it is their choice.

\n

The leaks were disclosed the day we got an extra digit. And @inversion announced a coming reset within 24 hours of leak disclosure. People who started looking for leaks when they saw 0.999 scores can't have spent more than 5 subs searching for the leak.

", "rawMarkdown": "??\n\nIf people search for leaks instead of searching for good models it is their choice. \n\nThe leaks were disclosed the day we got an extra digit. And @inversion announced a coming reset within 24 hours of leak disclosure. People who started looking for leaks when they saw 0.999 scores can't have spent more than 5 subs searching for the leak. ", "votes": 6 }, { "id": 1360757, "postDate": "2021-06-22T10:28:26.697Z", "content": "

I agree with your opinion CPMP. 😄
\nSubmission count is one of my history that I achieve.
\nThere were memory for each numbers like a photo album.

\n

And I also believe that different people may have different opinions.
\nBy the way, you recently got a score of 0.985 without using leakage. I think it's really great.

", "rawMarkdown": "I agree with your opinion CPMP. 😄\nSubmission count is one of my history that I achieve.\nThere were memory for each numbers like a photo album.\n\nAnd I also believe that different people may have different opinions.\nBy the way, you recently got a score of 0.985 without using leakage. I think it's really great. ", "votes": 1 }, { "id": 1360877, "postDate": "2021-06-22T12:17:06.647Z", "content": "

Thanks @woosungyoon . we'll see if I still get good scores after the reset. I may just be lucky at this point.

", "rawMarkdown": "Thanks @woosungyoon . we'll see if I still get good scores after the reset. I may just be lucky at this point." }, { "id": 1361495, "postDate": "2021-06-22T21:40:43.017Z", "content": "

@cpmpml I was mainly talking about couple people who reported these leakages. They used their submission counts to contribute to our community and a better competition.

", "rawMarkdown": "@cpmpml I was mainly talking about couple people who reported these leakages. They used their submission counts to contribute to our community and a better competition.", "votes": 2 }, { "id": 1361850, "postDate": "2021-06-23T05:10:34.697Z", "content": "

They are grown up people. If they don't complain about this then it is not really an issue.

", "rawMarkdown": "They are grown up people. If they don't complain about this then it is not really an issue." } ] }, { "id": 1353439, "postDate": "2021-06-17T05:32:21.130Z", "content": "

\"We're working with the host to gather a new Test dataset\"
\nWill you please consider making this a code competition now? see Despite pros and cons it seems far better not to have access to the private Test data where possible.

\n

Thanks @inversion - you are always great at staying engaged with competitions, keeping them informed and owning up to issues.
\nThanks @kazanova for finding and explaining the leak, you deserve a reward - at least some Swag or Colab credits!

", "rawMarkdown": "\"We're working with the host to gather a new Test dataset\"\nWill you please consider making this a code competition now? [see](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/238758) Despite pros and cons it seems far better not to have access to the private Test data where possible. \n\nThanks @inversion - you are always great at staying engaged with competitions, keeping them informed and owning up to issues. \nThanks @kazanova for finding and explaining the leak, you deserve a reward - at least some Swag or Colab credits!", "votes": 1 }, { "id": 1361388, "postDate": "2021-06-22T19:14:31.637Z", "content": "

I wonder when the data will be updated.

", "rawMarkdown": "I wonder when the data will be updated.", "votes": 1, "replies": [ { "id": 1361596, "postDate": "2021-06-23T00:09:33.863Z", "content": "

I rather wait a week and be able to compete, than have new data today and find a new leak tomorrow :) they are already working very hard to solve it, preparing a new dataset is not easy.

", "rawMarkdown": "I rather wait a week and be able to compete, than have new data today and find a new leak tomorrow :) they are already working very hard to solve it, preparing a new dataset is not easy.", "votes": 2 }, { "id": 1361848, "postDate": "2021-06-23T05:09:13.010Z", "content": "

I understand that. But it would be nice to know the date.

", "rawMarkdown": "I understand that. But it would be nice to know the date.", "votes": 2 }, { "id": 1363029, "postDate": "2021-06-23T21:22:18.640Z", "content": "

I think you're totally right to ask about when we'll finally get a new dataset.

", "rawMarkdown": "I think you're totally right to ask about when we'll finally get a new dataset.", "votes": 1 }, { "id": 1364011, "postDate": "2021-06-24T14:34:23.800Z", "content": "

Sorry to interrupt but we'll be seeing an extension to the deadline too, right ?

", "rawMarkdown": "Sorry to interrupt but we'll be seeing an extension to the deadline too, right ? " }, { "id": 1364946, "postDate": "2021-06-25T10:35:40.993Z", "content": "

https://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755 yep

", "rawMarkdown": "https://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755 yep" } ] }, { "id": 1353948, "postDate": "2021-06-17T09:55:00.350Z", "content": "

@inversion Needed some clarifications
\n1) Since Test Set Labels will be made public , that along with training set would increase the size of data wouldn't it , for train and validation ?
\n2) Only Inference Part will change for this competition since there is new test set
\n3) Will there be any extension to deadline ?

", "rawMarkdown": "@inversion Needed some clarifications\n1) Since Test Set Labels will be made public , that along with training set would increase the size of data wouldn't it , for train and validation ?\n2) Only Inference Part will change for this competition since there is new test set \n3) Will there be any extension to deadline ? ", "votes": 1 }, { "id": 1353526, "postDate": "2021-06-17T06:47:42.780Z", "content": "

Thanks @inversion for saving this competition and time that we invested in this comp.

", "rawMarkdown": "Thanks @inversion for saving this competition and time that we invested in this comp.", "votes": 1 }, { "id": 1353359, "postDate": "2021-06-17T04:26:17.090Z", "content": "

Thanks for sharing more details @inversion. Any estimate on the new deadline? 🤔

", "rawMarkdown": "Thanks for sharing more details @inversion. Any estimate on the new deadline? 🤔 ", "votes": 1 }, { "id": 1353266, "postDate": "2021-06-17T02:47:41.153Z", "content": "

This leakage already gives people who used that new training samples, I think just replace the test set is not enough.

", "rawMarkdown": "This leakage already gives people who used that new training samples, I think just replace the test set is not enough.", "votes": 1, "replies": [ { "id": 1353295, "postDate": "2021-06-17T03:23:15.070Z", "content": "

Agreed. We'd also bundle the old Test into the new Train.

", "rawMarkdown": "Agreed. We'd also bundle the old Test into the new Train. ", "votes": 5 }, { "id": 1353298, "postDate": "2021-06-17T03:26:24.870Z", "content": "

Glad to hear that. Thanks for the efforts.

", "rawMarkdown": "Glad to hear that. Thanks for the efforts." } ] }, { "id": 1353257, "postDate": "2021-06-17T02:38:33.640Z", "content": "

Thanks for the prompt update @inversion does this mean the LB also will be reset at this point in time.

", "rawMarkdown": "Thanks for the prompt update @inversion does this mean the LB also will be reset at this point in time.", "votes": 2, "replies": [ { "id": 1353263, "postDate": "2021-06-17T02:41:41.247Z", "content": "

Yes, we will invalidate all submissions with the current Test data.

", "rawMarkdown": "Yes, we will invalidate all submissions with the current Test data.", "votes": 7 }, { "id": 1353264, "postDate": "2021-06-17T02:44:04.880Z", "content": "

Thanks for the confirmation.

", "rawMarkdown": "Thanks for the confirmation.", "votes": 1 } ] }, { "id": 1480400, "postDate": "2021-08-19T02:26:47.440Z", "content": "

Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:

\n

One account per participant
\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\nNo private sharing outside teams
\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.
\nbut our results were cancelled. Is there a staff member who can explain?

", "rawMarkdown": "Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:\n\nOne account per participant\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.\nNo private sharing outside teams\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.\nbut our results were cancelled. Is there a staff member who can explain?" }, { "id": 1364472, "postDate": "2021-06-25T03:54:55.310Z", "content": "

Good news! (Partial) update is coming:
\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755

", "rawMarkdown": "Good news! (Partial) update is coming:\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755" } ], "comments": [ { "id": 1353255, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2021-06-17T02:37:20.107000", "content": "

Thank you for the quick response @inversion

\n

Great to see that such process is in place and that you are going to further improve it. that is all we can ask for!

\n

There is no harm done since we will be getting a new test dataset.

\n

On the matter of the leaks . I think kaggle's stance should be a bit more clear. In my opinion kaggle should

\n

1) encourage participants to reveal the leaks (so that the competition is valuable to the host after eliminating them), making it also fairer for the participants.
\n2) not reward solutions with leaks

\n

I understand that the definition of a leak can be debatable. I think it is fine to draw a line. For example a solution that finds the hidden number generator that is used to generate the test data should not be allowed to win.

", "votes": 42, "replies": [ { "id": 1353486, "author_name": "Psi", "author_url": "", "post_date": "2021-06-17T06:15:23.080000", "content": "

Thanks Marios for quickly finding it and sharing! I think such an obvious and very simple leak should always be shared immediately, but you are right that there should be some incentives to do so. If it would not have been for the third digit, it would have probably taken us a longer while to figure it out…

", "votes": 3, "replies": [] }, { "id": 1353716, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-17T08:14:37.170000", "content": "

Marios, while I agree on the intent, drawing the line between leak and no leak is very difficult. I remember we had an argument in the first Quora competition on whether question frequency was a leak (your position) or not a leak (my position). I won't restart the argument here, I mention it just as an example of how defining leak can be tricky.

", "votes": 6, "replies": [] }, { "id": 1355143, "author_name": "Awsaf", "author_url": "", "post_date": "2021-06-18T05:56:47.223000", "content": "

@inversion How about keeping a different prize dedicated to finding leakage like ethical hacking? I think this will motivate other people to find the leakage but not for solution.

", "votes": 3, "replies": [] }, { "id": 1355438, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2021-06-18T09:30:21.097000", "content": "
\n

@inversion How about keeping a different prize dedicated to finding leakage like ethical hacking? I think this will motivate other people to find the leakage but not for solution.

\n
\n

Award some kaggle ranking points to people finding leakage that leads to restarts.

", "votes": 3, "replies": [] }, { "id": 1355441, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-18T09:31:30.913000", "content": "

Yeah, let's have Leak GM category.

\n

I'm serious.

", "votes": 5, "replies": [] }, { "id": 1355838, "author_name": "Awsaf", "author_url": "", "post_date": "2021-06-18T15:03:40.153000", "content": "

@kazanova this doesn't necessarily have to be a kaggle ranking point it could prize money. If one can find it he'll get the prize. If there is no leak no one will get the prize and this prize could be funded by kaggle.
\nI think there is Bug bounty program in every tech-giant company why not here?

", "votes": 3, "replies": [] }, { "id": 1357032, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-19T12:21:53.337000", "content": "

i agreed to that it. if you get a leak, you have to say to stuff. To not be unfair

", "votes": 0, "replies": [] }, { "id": 1359540, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2021-06-21T10:55:07.337000", "content": "

Please bring in \"Kaggle leak bounty\". Starting with new data leak playground competitions. Bug bounty has been beneficial to a lot of companies out there and their contribution in terms of security is huge. On the other side, Kaggle would benefit understanding \"leaky\" perspective from participants

", "votes": 0, "replies": [] } ] }, { "id": 1354175, "author_name": "Oleg Panichev", "author_url": "", "post_date": "2021-06-17T12:26:05.043000", "content": "

@inversion I've been on the top of LB for a while and got score 0.999 on LB without any usage of metadata. I am ready to share upon request my solution with you and SETI if needed to investigate if there are any leakages beside metadata leakage.

", "votes": 23, "replies": [ { "id": 1357038, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-19T12:25:31.197000", "content": "

wow.that very suprise for me.

", "votes": 0, "replies": [] } ] }, { "id": 1353612, "author_name": "Vicens Gaitan", "author_url": "", "post_date": "2021-06-17T07:29:55.393000", "content": "

@inversion , please, take a careful look at the data before releasing the new test set. I wasn’t using the timestamp nor any other metadata, and still getting 0.999 public LB. I was thinking about some kind of steganographic alien message hidden in the power spectra, but maybe there is a subtle leak, related to how the images where processed.

", "votes": 15, "replies": [ { "id": 1353654, "author_name": "Psi", "author_url": "", "post_date": "2021-06-17T07:43:08.363000", "content": "

@vicensgaitan thanks for sharing, I was already expecting there to be some more subtle leak, not related to the timestamp. At least that was my first guess after seeing the 0.999 scores.

\n

If you are willing to share (ignore otherwise): are you using any external data? I was also wondering whether a leak could exist there.

", "votes": 4, "replies": [] }, { "id": 1353692, "author_name": "Vicens Gaitan", "author_url": "", "post_date": "2021-06-17T07:58:25.313000", "content": "

No external data. I'm using only the provided images.

", "votes": 3, "replies": [] }, { "id": 1353706, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-17T08:10:21.947000", "content": "

Vicens, why don't you share privately with Kaggle the leak you found?

", "votes": 5, "replies": [] }, { "id": 1353719, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2021-06-17T08:16:10.097000", "content": "
\n

Vicens, why don't you share privately with Kaggle the leak you found?

\n
\n

I would say, he MUST share now to avoid another reset later on.

", "votes": 8, "replies": [] }, { "id": 1353728, "author_name": "Psi", "author_url": "", "post_date": "2021-06-17T08:18:09.497000", "content": "

I think he does not know what kind of leak it exactly is, but it seems obvious that his model / approach is picking something up. Other top contestant said similarly in the other thread. But if he has a hunch, or knows what could be an issue, it should be discussed.

", "votes": 8, "replies": [] }, { "id": 1353746, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-17T08:22:15.217000", "content": "

Then privately sharing his model with Kaggle could be useful.

", "votes": 3, "replies": [] }, { "id": 1353781, "author_name": "Vicens Gaitan", "author_url": "", "post_date": "2021-06-17T08:37:50.833000", "content": "

I'm still triying to figure out what is actually the leak. I'll share with Kaggle what i'm doing and maybe this light a bulb.

", "votes": 16, "replies": [] }, { "id": 1354163, "author_name": "Sergey Bryansky", "author_url": "", "post_date": "2021-06-17T12:14:22.357000", "content": "

FYI @inversion I think problem is downcast truncating. For more information check this little example.

", "votes": 5, "replies": [] }, { "id": 1354297, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T14:03:57", "content": "

At this point, I'm thinking it might be easier to just try to get actual ET signals. 😁

", "votes": 34, "replies": [] }, { "id": 1354524, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-06-17T16:24:54.697000", "content": "

Will finally get some Area 51 secrets. :P

", "votes": 4, "replies": [] } ] }, { "id": 1353288, "author_name": "PengBo", "author_url": "", "post_date": "2021-06-17T03:10:06.160000", "content": "

Since the test set is gonna to be renew, is it possible to make the old test set label public to all participants? As the leakage has affect the fair of the competition.

", "votes": 11, "replies": [ { "id": 1353296, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T03:23:35.407000", "content": "

Yes, that's the plan.

", "votes": 11, "replies": [] }, { "id": 1353315, "author_name": "PengBo", "author_url": "", "post_date": "2021-06-17T03:43:18.053000", "content": "

Thank you for your effort!

", "votes": 3, "replies": [] }, { "id": 1357034, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-19T12:23:10.290000", "content": "

oh. thank your effort.

", "votes": 0, "replies": [] } ] }, { "id": 1353500, "author_name": "Psi", "author_url": "", "post_date": "2021-06-17T06:26:39.880000", "content": "

Thank you @inversion!

\n

Probably you dont know yet, but are you able to make an estimate of when new test data will be revealed? Are we speaking of days or weeks?

\n

Also, could you make the current test labels available soon? Then, people can continue working on it.

", "votes": 5, "replies": [ { "id": 1357035, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-19T12:23:48.490000", "content": "

mee too. I wat to know that.

", "votes": 0, "replies": [] } ] }, { "id": 1353286, "author_name": "yukiya", "author_url": "", "post_date": "2021-06-17T03:05:59.647000", "content": "
\n

Worrying about leakage and other failure points keeps us up at night.

\n
\n

Sounds like you need a hug :)
\nThank you for the hard work, and all for our fun. Big respect to the nobleman who found the leak and reveal it.

", "votes": 5, "replies": [ { "id": 1353297, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T03:24:05.773000", "content": "

Thanks! I'm constantly amazed at what an awesome community we have, willing to help and share.

", "votes": 8, "replies": [] } ] }, { "id": 1358107, "author_name": "sajwankit", "author_url": "", "post_date": "2021-06-20T07:43:20.500000", "content": "

will there be any update in competition timelines too due to leak?

", "votes": 6, "replies": [] }, { "id": 1354333, "author_name": "Volodymyr", "author_url": "", "post_date": "2021-06-17T14:38:32.047000", "content": "

@inversion , will train/test set changes also take into account this leakage ?

", "votes": 3, "replies": [] }, { "id": 1359165, "author_name": "sin", "author_url": "", "post_date": "2021-06-21T05:42:33.963000", "content": "

I am wondering if the submission count will be reset as well.

", "votes": 1, "replies": [ { "id": 1359589, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-21T11:30:23.940000", "content": "

Why would it be?

", "votes": 3, "replies": [] }, { "id": 1360157, "author_name": "sin", "author_url": "", "post_date": "2021-06-21T21:13:57.423000", "content": "

Multiple reasons. For example, competitors who spent many submissions trying to find the leak has a disadvantage in terms of merging with other teams.

", "votes": 1, "replies": [] }, { "id": 1360695, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-22T09:37:20.857000", "content": "

??

\n

If people search for leaks instead of searching for good models it is their choice.

\n

The leaks were disclosed the day we got an extra digit. And @inversion announced a coming reset within 24 hours of leak disclosure. People who started looking for leaks when they saw 0.999 scores can't have spent more than 5 subs searching for the leak.

", "votes": 6, "replies": [] }, { "id": 1360757, "author_name": "WOOSUNG YOON", "author_url": "", "post_date": "2021-06-22T10:28:26.697000", "content": "

I agree with your opinion CPMP. 😄
\nSubmission count is one of my history that I achieve.
\nThere were memory for each numbers like a photo album.

\n

And I also believe that different people may have different opinions.
\nBy the way, you recently got a score of 0.985 without using leakage. I think it's really great.

", "votes": 1, "replies": [] }, { "id": 1360877, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-22T12:17:06.647000", "content": "

Thanks @woosungyoon . we'll see if I still get good scores after the reset. I may just be lucky at this point.

", "votes": 0, "replies": [] }, { "id": 1361495, "author_name": "sin", "author_url": "", "post_date": "2021-06-22T21:40:43.017000", "content": "

@cpmpml I was mainly talking about couple people who reported these leakages. They used their submission counts to contribute to our community and a better competition.

", "votes": 2, "replies": [] }, { "id": 1361850, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-23T05:10:34.697000", "content": "

They are grown up people. If they don't complain about this then it is not really an issue.

", "votes": 0, "replies": [] } ] }, { "id": 1353439, "author_name": "something4kag", "author_url": "", "post_date": "2021-06-17T05:32:21.130000", "content": "

\"We're working with the host to gather a new Test dataset\"
\nWill you please consider making this a code competition now? see Despite pros and cons it seems far better not to have access to the private Test data where possible.

\n

Thanks @inversion - you are always great at staying engaged with competitions, keeping them informed and owning up to issues.
\nThanks @kazanova for finding and explaining the leak, you deserve a reward - at least some Swag or Colab credits!

", "votes": 1, "replies": [] }, { "id": 1361388, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-06-22T19:14:31.637000", "content": "

I wonder when the data will be updated.

", "votes": 1, "replies": [ { "id": 1361596, "author_name": "Nanashi", "author_url": "", "post_date": "2021-06-23T00:09:33.863000", "content": "

I rather wait a week and be able to compete, than have new data today and find a new leak tomorrow :) they are already working very hard to solve it, preparing a new dataset is not easy.

", "votes": 2, "replies": [] }, { "id": 1361848, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-06-23T05:09:13.010000", "content": "

I understand that. But it would be nice to know the date.

", "votes": 2, "replies": [] }, { "id": 1363029, "author_name": "kkiller", "author_url": "", "post_date": "2021-06-23T21:22:18.640000", "content": "

I think you're totally right to ask about when we'll finally get a new dataset.

", "votes": 1, "replies": [] }, { "id": 1364011, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-06-24T14:34:23.800000", "content": "

Sorry to interrupt but we'll be seeing an extension to the deadline too, right ?

", "votes": 0, "replies": [] }, { "id": 1364946, "author_name": "Nanashi", "author_url": "", "post_date": "2021-06-25T10:35:40.993000", "content": "

https://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755 yep

", "votes": 0, "replies": [] } ] }, { "id": 1353948, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-06-17T09:55:00.350000", "content": "

@inversion Needed some clarifications
\n1) Since Test Set Labels will be made public , that along with training set would increase the size of data wouldn't it , for train and validation ?
\n2) Only Inference Part will change for this competition since there is new test set
\n3) Will there be any extension to deadline ?

", "votes": 1, "replies": [] }, { "id": 1353526, "author_name": "Salman", "author_url": "", "post_date": "2021-06-17T06:47:42.780000", "content": "

Thanks @inversion for saving this competition and time that we invested in this comp.

", "votes": 1, "replies": [] }, { "id": 1353359, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-06-17T04:26:17.090000", "content": "

Thanks for sharing more details @inversion. Any estimate on the new deadline? 🤔

", "votes": 1, "replies": [] }, { "id": 1353266, "author_name": "Chenglu", "author_url": "", "post_date": "2021-06-17T02:47:41.153000", "content": "

This leakage already gives people who used that new training samples, I think just replace the test set is not enough.

", "votes": 1, "replies": [ { "id": 1353295, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T03:23:15.070000", "content": "

Agreed. We'd also bundle the old Test into the new Train.

", "votes": 5, "replies": [] }, { "id": 1353298, "author_name": "Chenglu", "author_url": "", "post_date": "2021-06-17T03:26:24.870000", "content": "

Glad to hear that. Thanks for the efforts.

", "votes": 0, "replies": [] } ] }, { "id": 1353257, "author_name": "Vishy", "author_url": "", "post_date": "2021-06-17T02:38:33.640000", "content": "

Thanks for the prompt update @inversion does this mean the LB also will be reset at this point in time.

", "votes": 2, "replies": [ { "id": 1353263, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T02:41:41.247000", "content": "

Yes, we will invalidate all submissions with the current Test data.

", "votes": 7, "replies": [] }, { "id": 1353264, "author_name": "Vishy", "author_url": "", "post_date": "2021-06-17T02:44:04.880000", "content": "

Thanks for the confirmation.

", "votes": 1, "replies": [] } ] }, { "id": 1480400, "author_name": "Ctrl_CV", "author_url": "", "post_date": "2021-08-19T02:26:47.440000", "content": "

Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:

\n

One account per participant
\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\nNo private sharing outside teams
\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.
\nbut our results were cancelled. Is there a staff member who can explain?

", "votes": 0, "replies": [] }, { "id": 1364472, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-25T03:54:55.310000", "content": "

Good news! (Partial) update is coming:
\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1353238": "Much appreciation to long-time Kaggler @kazanova for [bringing forward the issue of time-stamp leakage](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246772). This definitely falls into the \"mistakes were made\" category.\n\nSince it was my mistake, I'll share how it happened.\n\nThere are just a few of us data scientists at Kaggle launching about 50 competitions a year with many different data types over a very wide range of domains. Worrying about leakage and other failure points keeps us up at night. We absolutely value our community's time and effort and know how important it is to have fun and challenging competitions.\n\nTo minimize the chance of leakage in the competitions we launch, we have a well-defined set of standard processes, which includes a final data check by a different data scientist before launch.\n\nGenerally we randomize file save order in code, which is the best practice. But because of the complexity of the incoming data from the host, I opted to do a manual touch of the files after they were written. This was all fine and dandy *except* when I did an additional data check on the \"Download All\" dataset I found there were files missing. This led me on a multi-day debugging spree that ended up in multiple re-writes and uploads of the data before we eventually discovered it was a bug in our back-end randomly dropping a few files in the zip process.\n\nWhen it was all said and done, the final upload ended up not getting the time-stamp reset.\n\nWe'll be putting a down-stream (i.e., the data downloaded from the Data page) check into our processes to avoid this issue in the future. And I certainly apologize for my mistake here.\n\nWe're working with the host to gather a new Test dataset. It will be generated very similarly, so the work everyone has done will be preserved. We'll post again once we have timing on that reset.\n", "1353255": "Thank you for the quick response @inversion \n\nGreat to see that such process is in place and that you are going to further improve it. that is all we can ask for! \n\nThere is no harm done since we will be getting a new test dataset.\n\nOn the matter of the leaks . I think kaggle's stance should be a bit more clear. In my opinion kaggle should\n\n1) encourage participants to reveal the leaks (so that the competition is valuable to the host after eliminating them), making it also fairer for the participants.\n2) not reward solutions with leaks\n\nI understand that the definition of a leak can be debatable. I think it is fine to draw a line. For example a solution that finds the hidden number generator that is used to generate the test data should not be allowed to win. ", "1354175": "@inversion I've been on the top of LB for a while and got score 0.999 on LB without any usage of metadata. I am ready to share upon request my solution with you and SETI if needed to investigate if there are any leakages beside metadata leakage.", "1353612": "@inversion , please, take a careful look at the *data* before releasing the new test set. I wasn’t using the timestamp nor any other metadata, and still getting 0.999 public LB. I was thinking about some kind of steganographic alien message hidden in the power spectra, but maybe there is a subtle leak, related to how the images where processed.", "1353288": "Since the test set is gonna to be renew, is it possible to make the old test set label public to all participants? As the leakage has affect the fair of the competition.", "1353500": "Thank you @inversion!\n\nProbably you dont know yet, but are you able to make an estimate of when new test data will be revealed? Are we speaking of days or weeks?\n\nAlso, could you make the current test labels available soon? Then, people can continue working on it. ", "1353286": "> Worrying about leakage and other failure points keeps us up at night. \n\nSounds like you need a hug :) \nThank you for the hard work, and all for our fun. Big respect to the nobleman who found the leak and reveal it. ", "1358107": "will there be any update in competition timelines too due to leak?", "1354333": "@inversion , will train/test set changes also take into account [this leakage](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901) ? ", "1359165": "I am wondering if the submission count will be reset as well.", "1353439": "\"We're working with the host to gather a new Test dataset\"\nWill you please consider making this a code competition now? [see](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/238758) Despite pros and cons it seems far better not to have access to the private Test data where possible. \n\nThanks @inversion - you are always great at staying engaged with competitions, keeping them informed and owning up to issues. \nThanks @kazanova for finding and explaining the leak, you deserve a reward - at least some Swag or Colab credits!", "1361388": "I wonder when the data will be updated.", "1353948": "@inversion Needed some clarifications\n1) Since Test Set Labels will be made public , that along with training set would increase the size of data wouldn't it , for train and validation ?\n2) Only Inference Part will change for this competition since there is new test set \n3) Will there be any extension to deadline ? ", "1353526": "Thanks @inversion for saving this competition and time that we invested in this comp.", "1353359": "Thanks for sharing more details @inversion. Any estimate on the new deadline? 🤔 ", "1353266": "This leakage already gives people who used that new training samples, I think just replace the test set is not enough.", "1353257": "Thanks for the prompt update @inversion does this mean the LB also will be reset at this point in time.", "1480400": "Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:\n\nOne account per participant\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.\nNo private sharing outside teams\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.\nbut our results were cancelled. Is there a staff member who can explain?", "1364472": "Good news! (Partial) update is coming:\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/248755" } }