{ "id": 248755, "title": "(Partial) Competition Update", "url": "/competitions/seti-breakthrough-listen/discussion/248755", "author_name": "inversion", "post_date": "2021-06-24T19:38:18.762000", "votes": 68, "comment_count": 45, "views": 0, "content": "

Hi everyone -

\n

Here's an update of where things stand on the competition reset:

\n\n

An additional bit of detail on the time stamp leakage failure point (which I did a post-mortem for):

\n

Kaggle actually has a triple-redundant system for time stamp leakage, where all points failed: I previously mentioned our best practices of random file saves and secondary checks, but our system should have been tolerant to any human failure points, since our backend actually re-creates time stamps for our competition files on the Data page after upload. In this instance, because our zipping process was erroring out, the \"Download All\" zip file was manually put into position, rather than being automatically created from the uploaded competition files. (Which, by the way, is the reason the timestamp leakage didn't show up when the data in the Notebooks environment, because it was processed normally, and only when using the downloaded zip file from the competition page, which wasn't.)

\n

It's a good lesson on the importances of sticking to best practices. :-)

", "messages": [ { "id": 1364280, "postDate": "2021-06-24T19:38:18.763Z", "content": "

Hi everyone -

\n

Here's an update of where things stand on the competition reset:

\n\n

An additional bit of detail on the time stamp leakage failure point (which I did a post-mortem for):

\n

Kaggle actually has a triple-redundant system for time stamp leakage, where all points failed: I previously mentioned our best practices of random file saves and secondary checks, but our system should have been tolerant to any human failure points, since our backend actually re-creates time stamps for our competition files on the Data page after upload. In this instance, because our zipping process was erroring out, the \"Download All\" zip file was manually put into position, rather than being automatically created from the uploaded competition files. (Which, by the way, is the reason the timestamp leakage didn't show up when the data in the Notebooks environment, because it was processed normally, and only when using the downloaded zip file from the competition page, which wasn't.)

\n

It's a good lesson on the importances of sticking to best practices. :-)

", "rawMarkdown": "Hi everyone -\n\nHere's an update of where things stand on the competition reset:\n- A new test set has been generated, taking into account the fantastic feedback we've received from the community. (Thank you for all of the insights!)\n- We're in the process of scrutinizing the data, and are targeting a relaunch mid-next week (but may need more time if we find any issues along the way).\n- The new training set will include the current training _and_ test set (with the test set labels)\n- We'll adjust the competition timeline at that time to reflect the time lost with the reset\n\nAn additional bit of detail on the time stamp leakage failure point (which I did a post-mortem for):\n\nKaggle actually has a triple-redundant system for time stamp leakage, where all points failed: I [previously mentioned](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246782) our best practices of random file saves and secondary checks, but our system *should* have been tolerant to any human failure points, since our backend actually re-creates time stamps for our competition files on the Data page after upload. In this instance, because our zipping process was erroring out, the \"Download All\" zip file was manually put into position, rather than being automatically created from the uploaded competition files. (Which, by the way, is the reason the timestamp leakage *didn't* show up when the data in the Notebooks environment, because it was processed normally, and only when using the downloaded zip file from the competition page, which wasn't.) \n\nIt's a good lesson on the importances of sticking to best practices. :-)", "votes": 67 }, { "id": 1373457, "postDate": "2021-07-02T13:57:15.993Z", "content": "

Were we currently stand: The data generation has taken a bit longer after it was determined the training set would also need to be re-generated, but it is nearly complete. We're going into a US long weekend, and I don't expect significant work over the next 4 days, but I'll pick things back up next Wednesday. With processing and checking, I'd imagine things will be ready for relaunch early the week of July 12.

", "rawMarkdown": "Were we currently stand: The data generation has taken a bit longer after it was determined the training set would also need to be re-generated, but it is nearly complete. We're going into a US long weekend, and I don't expect significant work over the next 4 days, but I'll pick things back up next Wednesday. With processing and checking, I'd imagine things will be ready for relaunch early the week of July 12.", "votes": 35, "replies": [ { "id": 1386038, "postDate": "2021-07-13T07:45:46.483Z", "content": "

@inversion Any updates ?

", "rawMarkdown": "@inversion Any updates ?", "votes": 1 }, { "id": 1386062, "postDate": "2021-07-13T08:13:07.793Z", "content": "

early the week of July 12 , which means July 13-July 15 is possible

", "rawMarkdown": "**early the week of July 12** , which means July 13-July 15 is possible", "votes": 3 } ] }, { "id": 1371593, "postDate": "2021-07-01T05:50:18.417Z", "content": "

Posting as a top-level comment for visibility:

\n

Just investigated whether renormalizing the current training set is able to remove the statistics leak, and unfortunately the truncation from casting into FP16 caused minute differences in the mean and standard deviations between positive and negative samples that we weren't able to remove.

\n

I will be discussing with @inversion to decide on the best way to mitigate this. We will most likely be generating a new training set without leaks.

\n

Average of Means from On and Off (blue is needles, orange is new haystack)

\n
Average Means from On and Off
\n\n

Average of STDs from On and Off (blue is needles, orange is new haystack)

\n
Average of STDs from On and Off
\n", "rawMarkdown": "Posting as a top-level comment for visibility:\n\nJust investigated whether renormalizing the current training set is able to remove the statistics leak, and unfortunately the truncation from casting into FP16 caused minute differences in the mean and standard deviations between positive and negative samples that we weren't able to remove.\n\nI will be discussing with @inversion to decide on the best way to mitigate this. We will most likely be generating a new training set without leaks.\n\nAverage of Means from On and Off (blue is needles, orange is new haystack)\n
Average Means from On and Off
\n\nAverage of STDs from On and Off (blue is needles, orange is new haystack)\n
Average of STDs from On and Off
", "votes": 18, "replies": [ { "id": 1371944, "postDate": "2021-07-01T10:16:54.833Z", "content": "

Thanks for the update @yuhongc - if you would create a new training set, would you create the exact same current train / test data? Otherwise I think people will still try to re-use current data to make the data larger, and then I am wondering whether it could also be an option for people to try to remove the inherent bias themselves. Of course as long as the test data has no bias.
\nCurious what others think.

", "rawMarkdown": "Thanks for the update @yuhongc - if you would create a new training set, would you create the exact same current train / test data? Otherwise I think people will still try to re-use current data to make the data larger, and then I am wondering whether it could also be an option for people to try to remove the inherent bias themselves. Of course as long as the test data has no bias.\nCurious what others think.", "votes": 7 }, { "id": 1372004, "postDate": "2021-07-01T10:55:43.763Z", "content": "

Does this mean that it will take more time for the dataset to be updated ? If so, Do you have an estimate on when it will go live.

", "rawMarkdown": "Does this mean that it will take more time for the dataset to be updated ? If so, Do you have an estimate on when it will go live." }, { "id": 1372041, "postDate": "2021-07-01T11:26:58.937Z", "content": "

Thanks for sharing the info with us!

\n

A small statistical leak in the training data is not as critical as in the test data.
\nOf course, it needs to be verified, but if it is difficult to prepare new training data, it may be enough to re-normalize the current training data even if there are still small leaks.
\nAs @philippsinger pointed out, even if you prepare completely new training data, many participants will try to use the old training data along with it, and in that case, participants will eventually have to deal with this small leak.

", "rawMarkdown": "Thanks for sharing the info with us!\n\nA small statistical leak in the training data is not as critical as in the test data.\nOf course, it needs to be verified, but if it is difficult to prepare new training data, it may be enough to re-normalize the current training data even if there are still small leaks.\nAs @philippsinger pointed out, even if you prepare completely new training data, many participants will try to use the old training data along with it, and in that case, participants will eventually have to deal with this small leak.", "votes": 1 }, { "id": 1372060, "postDate": "2021-07-01T11:51:39.547Z", "content": "

Indeed, so I think the issue could only be solved if you re-create the exact same current train/test data without leak.

", "rawMarkdown": "Indeed, so I think the issue could only be solved if you re-create the exact same current train/test data without leak.", "votes": 7 }, { "id": 1373353, "postDate": "2021-07-02T12:32:51.197Z", "content": "

Thank you, @yuhongc! Do you think generating the entire data (train and test) as uint16 could help?

", "rawMarkdown": "Thank you, @yuhongc! Do you think generating the entire data (train and test) as uint16 could help?", "votes": 1 }, { "id": 1373401, "postDate": "2021-07-02T13:17:29.127Z", "content": "
\n

Indeed, so I think the issue could only be solved if you re-create the exact same current train/test data without leak.

\n
\n

I agree.

\n

However, this assumes that any randomness used for generating signals can be reproduced. I hope all random seeds were saved…

", "rawMarkdown": "> Indeed, so I think the issue could only be solved if you re-create the exact same current train/test data without leak.\n\nI agree.\n\nHowever, this assumes that any randomness used for generating signals can be reproduced. I hope all random seeds were saved...", "votes": 3 }, { "id": 1374969, "postDate": "2021-07-03T18:35:46.200Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ] }, { "id": 1365400, "postDate": "2021-06-25T17:17:51.893Z", "content": "

Can it be a code competition with the new data?

", "rawMarkdown": "Can it be a code competition with the new data?", "votes": 11, "replies": [ { "id": 1370229, "postDate": "2021-06-30T03:05:15.877Z", "content": "

@inversion Please Take this into consideration

", "rawMarkdown": "@inversion Please Take this into consideration ", "votes": 1 }, { "id": 1370483, "postDate": "2021-06-30T07:35:39.150Z", "content": "

I am really glad that more and more people see the advantages of code competitions :) I always try to argue for it.

", "rawMarkdown": "I am really glad that more and more people see the advantages of code competitions :) I always try to argue for it.", "votes": 8 }, { "id": 1370836, "postDate": "2021-06-30T12:48:07.573Z", "content": "

I'll be a full supporter of code competitions when we have GPU more recent than 5 years old P100.

", "rawMarkdown": "I'll be a full supporter of code competitions when we have GPU more recent than 5 years old P100.", "votes": 1 }, { "id": 1370838, "postDate": "2021-06-30T12:49:12.073Z", "content": "

Main pro for code competition is to make it harder to use test data distribution to overfit models to test data.

", "rawMarkdown": "Main pro for code competition is to make it harder to use test data distribution to overfit models to test data." }, { "id": 1370850, "postDate": "2021-06-30T12:57:19.890Z", "content": "
\n

I'll be a full supporter of code competitions when we have GPU more recent than 5 years old P100.

\n
\n

It will be difficult for them to afford anything of a substantial upgrade to the P100. Maybe use a T4 cause it`s better in Mixed_float16 ?

", "rawMarkdown": "> I'll be a full supporter of code competitions when we have GPU more recent than 5 years old P100.\n\n\n It will be difficult for them to afford anything of a substantial upgrade to the P100. Maybe use a T4 cause it`s better in Mixed_float16 ?" }, { "id": 1370911, "postDate": "2021-06-30T13:37:45.320Z", "content": "

For inference P100 is more than enough, not sure why you would need something better.

", "rawMarkdown": "For inference P100 is more than enough, not sure why you would need something better.", "votes": 3 }, { "id": 1371050, "postDate": "2021-06-30T15:53:05.347Z", "content": "

Small GPU memory can be an issue in some cases.

", "rawMarkdown": "Small GPU memory can be an issue in some cases. " }, { "id": 1371054, "postDate": "2021-06-30T15:54:33.797Z", "content": "

I dont know of any model not fitting into P100 memory, apart from some ultra heavy transformer nlp models that are basically not used on kaggle anyways.

", "rawMarkdown": "I dont know of any model not fitting into P100 memory, apart from some ultra heavy transformer nlp models that are basically not used on kaggle anyways. ", "votes": 3 }, { "id": 1371104, "postDate": "2021-06-30T16:52:16.440Z", "content": "

Not all models are NNs. I got stuck when using RAPIDS once.

", "rawMarkdown": "Not all models are NNs. I got stuck when using RAPIDS once." }, { "id": 1371884, "postDate": "2021-07-01T09:39:22.257Z", "content": "

@cpmpml no offense but kaggle code competition do replicate real life condition where you do not always have access to the Best hardware for deployment.

", "rawMarkdown": "@cpmpml no offense but kaggle code competition do replicate real life condition where you do not always have access to the Best hardware for deployment.", "votes": 4 } ] }, { "id": 1364351, "postDate": "2021-06-24T22:03:12.797Z", "content": "

Good news, thanks. Will you also provide instructions on how to remove the distribution leak from existing training an d test data? Indeed, if we train on data with the leak and predict on new test without leak then results may be irrelevant.

\n

I suggested some way, but there maybe other, better ways.

", "rawMarkdown": "Good news, thanks. Will you also provide instructions on how to remove the distribution leak from existing training an d test data? Indeed, if we train on data with the leak and predict on new test without leak then results may be irrelevant.\n\nI suggested [some way](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/248194), but there maybe other, better ways.", "votes": 4, "replies": [ { "id": 1364358, "postDate": "2021-06-24T22:09:50.863Z", "content": "

If possible, it'd be great if instead of just the original (zip) data, when the new training set is provided (current train + current test), they are re-processed according to best practices. That way individual contestants don't need to do any further processing.

", "rawMarkdown": "If possible, it'd be great if instead of just the original (zip) data, when the new training set is provided (current train + current test), they are re-processed according to best practices. That way individual contestants don't need to do any further processing." }, { "id": 1364374, "postDate": "2021-06-24T22:33:58.717Z", "content": "

Fully fixing the distribution leak in the current dataset may be difficult because the data was already in float16 pre-injection. We really didn't see the truncation issue coming 😔.

\n

We may consider making a new training set with the truncation issue fixed, but some internal discussion is needed before I can say for sure.

", "rawMarkdown": "Fully fixing the distribution leak in the current dataset may be difficult because the data was already in float16 pre-injection. We really didn't see the truncation issue coming 😔.\n\nWe may consider making a new training set with the truncation issue fixed, but some internal discussion is needed before I can say for sure.", "votes": 10 }, { "id": 1364709, "postDate": "2021-06-25T06:49:16.003Z", "content": "

But you will fix it in new test set, right @yuhongc ?

", "rawMarkdown": "But you will fix it in new test set, right @yuhongc ?", "votes": 5 }, { "id": 1364965, "postDate": "2021-06-25T10:55:53.927Z", "content": "

@yuhongc Fixing only in new test isn't enough as it introduces a distribution shift between new train and new test.

\n

I get that creating a new train set is a lot of work, which is why I am asking if the prepossessing I proposed was enough to remove this distribution shift.

\n

And thanks for confirming my guess that your performed standardization in FP16.

", "rawMarkdown": "@yuhongc Fixing only in new test isn't enough as it introduces a distribution shift between new train and new test.\n\nI get that creating a new train set is a lot of work, which is why I am asking if the prepossessing I proposed was enough to remove this distribution shift.\n\nAnd thanks for confirming my guess that your performed standardization in FP16.", "votes": 2 }, { "id": 1364971, "postDate": "2021-06-25T11:03:31.737Z", "content": "

Fixing only new test should be enough, if you standardize properly yourself in training / validation.

", "rawMarkdown": "Fixing only new test should be enough, if you standardize properly yourself in training / validation.", "votes": 2 }, { "id": 1364993, "postDate": "2021-06-25T11:26:07.033Z", "content": "
\n

if you standardize properly yourself

\n
\n

This is what I proposed, and this is what I am asking confirmation for.

\n

Given some files were standardized in FP32 and others (snippets with message) in FP16 in current training set, there maybe distribution differences that cannot be removed.

", "rawMarkdown": "> if you standardize properly yourself\n\nThis is what I proposed, and this is what I am asking confirmation for.\n\nGiven some files were standardized in FP32 and others (snippets with message) in FP16 in current training set, there maybe distribution differences that cannot be removed.", "votes": 5 }, { "id": 1365465, "postDate": "2021-06-25T18:55:32.440Z", "content": "

@cpmpml Really appreciate your comments and support. I will investigate whether the standardization you did or some sort of other preprocessing is enough to fix the distribution difference from quantization.

\n

If it's indeed irreversible, I think we'll release a replacement training set, (while keeping the old data available out of fairness), but again can't say for sure without internal discussion.

", "rawMarkdown": "@cpmpml Really appreciate your comments and support. I will investigate whether the standardization you did or some sort of other preprocessing is enough to fix the distribution difference from quantization.\n\nIf it's indeed irreversible, I think we'll release a replacement training set, (while keeping the old data available out of fairness), but again can't say for sure without internal discussion.", "votes": 4 }, { "id": 1365473, "postDate": "2021-06-25T19:03:09.737Z", "content": "

@philippsinger We believe it's fixed in the new test. We'll be verifying this over the next few days. The worry with the current train+test right now is that casting back and forth between FP16 and FP32 (mea culpa) may have introduced tiny irreversible differences in the statistics because of the difference in accuracy between the two data formats.

", "rawMarkdown": "@philippsinger We believe it's fixed in the new test. We'll be verifying this over the next few days. The worry with the current train+test right now is that casting back and forth between FP16 and FP32 (mea culpa) may have introduced tiny irreversible differences in the statistics because of the difference in accuracy between the two data formats.", "votes": 6 }, { "id": 1365856, "postDate": "2021-06-26T07:41:56.873Z", "content": "

@yuhongc thanks you. Having a responsive host means a lot for competitors.

", "rawMarkdown": "@yuhongc thanks you. Having a responsive host means a lot for competitors.", "votes": 2 } ] }, { "id": 1381061, "postDate": "2021-07-08T15:30:59.720Z", "content": "

hello @inversion,
\ncould you please provide an update on the datasets? also will the timeline for submission be extened?

", "rawMarkdown": "hello @inversion, \ncould you please provide an update on the datasets? also will the timeline for submission be extened? ", "votes": 1 }, { "id": 1364437, "postDate": "2021-06-25T02:05:44.763Z", "content": "

Thank you for updating information. I really appreciate your hard work.

\n

I've explored several methods while the competition was suspended.
\nI'm looking forward to trying them out on new datasets😃

", "rawMarkdown": "Thank you for updating information. I really appreciate your hard work.\n\nI've explored several methods while the competition was suspended.\nI'm looking forward to trying them out on new datasets😃", "votes": 2 }, { "id": 1370452, "postDate": "2021-06-30T07:16:51.023Z", "content": "

Can it be a code competition with the new data?

", "rawMarkdown": "Can it be a code competition with the new data?", "votes": -1 }, { "id": 1371177, "postDate": "2021-06-30T18:19:51.503Z", "content": "

Hi, can it be a code competition with a new dataset

", "rawMarkdown": "Hi, can it be a code competition with a new dataset\n", "votes": -1 }, { "id": 1480398, "postDate": "2021-08-19T02:26:35.560Z", "content": "

Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:

\n

One account per participant
\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\nNo private sharing outside teams
\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.
\nbut our results were cancelled. Is there a staff member who can explain?

", "rawMarkdown": "Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:\n\nOne account per participant\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.\nNo private sharing outside teams\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.\nbut our results were cancelled. Is there a staff member who can explain?\n" }, { "id": 1371216, "postDate": "2021-06-30T18:58:45.170Z", "content": "

@inversion , One doubt since the competition is going to restart does it means that the public Leaderboard as well all submissions made till now by all participants for this Competition will reset ?

", "rawMarkdown": "@inversion , One doubt since the competition is going to restart does it means that the public Leaderboard as well all submissions made till now by all participants for this Competition will reset ?" }, { "id": 1364559, "postDate": "2021-06-25T04:51:20.143Z", "content": "

@inversion Could you also please check the How do you zip the files. As your adding More files to the train data it will become impossible to use this dataset with Colab Pro. So a well zipped dataset will surely help

", "rawMarkdown": "@inversion Could you also please check the How do you zip the files. As your adding More files to the train data it will become impossible to use this dataset with Colab Pro. So a well zipped dataset will surely help", "replies": [ { "id": 1365015, "postDate": "2021-06-25T11:46:31.497Z", "content": "

use GCS directly (if you use tensorflow)
\nyou can access data even if you use colab w/o pro

\n

but one thing clear is training time will be increase almost 1.7

", "rawMarkdown": "use GCS directly (if you use tensorflow)\nyou can access data even if you use colab w/o pro\n\nbut one thing clear is training time will be increase almost 1.7", "votes": 4 }, { "id": 1365366, "postDate": "2021-06-25T16:43:44.987Z", "content": "

@assign first of all you have pay a lot for GCS. And my training times are like already one hour and increase by 1.7 will mean i have a nearly 2 hour training Time which honestly a lot

", "rawMarkdown": "@assign first of all you have pay a lot for GCS. And my training times are like already one hour and increase by 1.7 will mean i have a nearly 2 hour training Time which honestly a lot " }, { "id": 1365631, "postDate": "2021-06-26T00:59:29.067Z", "content": "

I means

\n
    \n
  1. make it as kaggle dataset
    2. \n
  2. get gcs bucket id
    3. \n
  3. read by tensorflow record format
    4. \n
\n

1 preprocess dataset as tfrec format

\n

2 Kaggle kernel after add the dataset

\n
from kaggle_datasets import KaggleDatasets\nKaggleDatasets().get_gcs_path(<target dataset name>)# without ../input/\n
\n

example of it

\n

example of output:
\n'gs://kds-0752b66b41214b9aeeb3741bbd6b1dcf6c443007693b57cc9f4e2ea5'

\n

3 Colab

\n
GCS_PATH = <output of upper code>\nfiles = list(np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))) # depend on your file name\nds = tf.data.TFRecordDataset(files, num_parallel_reads = len(files))\nds = ds.map(<your parsing function>)\nds = ds.batch(512)\nfor I in ds: ~~~\n
\n

you need to preprocess dataset as serialized tensor,
\nand design suitable parsing function.
\nI recommend to read tf.io documents

\n

I don't know whether this is google's design for using gcs freely.
\nBut, we can use gcs without cost

\n

well, with this approach, using TPU much easier and we can expect speed up(1 epoch: 17min -> 1min)

\n

I'll delete it if it violate kaggle community guidelines

", "rawMarkdown": "I means\n1. make it as kaggle dataset \n2. get gcs bucket id\n3. read by tensorflow record format\n\n\n\n1 preprocess dataset as tfrec format\n\n2 Kaggle kernel after add the dataset\n```\nfrom kaggle_datasets import KaggleDatasets\nKaggleDatasets().get_gcs_path()# without ../input/\n```\n[example of it](https://www.kaggle.com/assign/get-data-info)\n\nexample of output:\n'gs://kds-0752b66b41214b9aeeb3741bbd6b1dcf6c443007693b57cc9f4e2ea5'\n\n\n3 Colab\n```\nGCS_PATH = \nfiles = list(np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))) # depend on your file name\nds = tf.data.TFRecordDataset(files, num_parallel_reads = len(files))\nds = ds.map()\nds = ds.batch(512)\nfor I in ds: ~~~\n```\nyou need to preprocess dataset as serialized tensor, \nand design suitable parsing function.\nI recommend to read tf.io documents\n\nI don't know whether this is google's design for using gcs freely. \nBut, we can use gcs without cost\n\nwell, with this approach, using TPU much easier and we can expect speed up(1 epoch: 17min -> 1min)\n\n\nI'll delete it if it violate kaggle community guidelines", "votes": 2 } ] }, { "id": 1370957, "postDate": "2021-06-30T14:12:53.690Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1370699, "postDate": "2021-06-30T10:44:01.613Z", "content": "

Thank you for the update 😊

", "rawMarkdown": "Thank you for the update 😊" }, { "id": 1364380, "postDate": "2021-06-24T22:52:59.537Z", "content": "

Thank you for the update 😊

", "rawMarkdown": "Thank you for the update 😊", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1373457, "author_name": "inversion", "author_url": "", "post_date": "2021-07-02T13:57:15.993000", "content": "

Were we currently stand: The data generation has taken a bit longer after it was determined the training set would also need to be re-generated, but it is nearly complete. We're going into a US long weekend, and I don't expect significant work over the next 4 days, but I'll pick things back up next Wednesday. With processing and checking, I'd imagine things will be ready for relaunch early the week of July 12.

", "votes": 35, "replies": [ { "id": 1386038, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-13T07:45:46.483000", "content": "

@inversion Any updates ?

", "votes": 1, "replies": [] }, { "id": 1386062, "author_name": "zhangeng", "author_url": "", "post_date": "2021-07-13T08:13:07.793000", "content": "

early the week of July 12 , which means July 13-July 15 is possible

", "votes": 3, "replies": [] } ] }, { "id": 1371593, "author_name": "Yuhong Chen", "author_url": "", "post_date": "2021-07-01T05:50:18.417000", "content": "

Posting as a top-level comment for visibility:

\n

Just investigated whether renormalizing the current training set is able to remove the statistics leak, and unfortunately the truncation from casting into FP16 caused minute differences in the mean and standard deviations between positive and negative samples that we weren't able to remove.

\n

I will be discussing with @inversion to decide on the best way to mitigate this. We will most likely be generating a new training set without leaks.

\n

Average of Means from On and Off (blue is needles, orange is new haystack)

\n
Average Means from On and Off
\n\n

Average of STDs from On and Off (blue is needles, orange is new haystack)

\n
Average of STDs from On and Off
\n", "votes": 18, "replies": [ { "id": 1371944, "author_name": "Psi", "author_url": "", "post_date": "2021-07-01T10:16:54.833000", "content": "

Thanks for the update @yuhongc - if you would create a new training set, would you create the exact same current train / test data? Otherwise I think people will still try to re-use current data to make the data larger, and then I am wondering whether it could also be an option for people to try to remove the inherent bias themselves. Of course as long as the test data has no bias.
\nCurious what others think.

", "votes": 7, "replies": [] }, { "id": 1372004, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-01T10:55:43.763000", "content": "

Does this mean that it will take more time for the dataset to be updated ? If so, Do you have an estimate on when it will go live.

", "votes": 0, "replies": [] }, { "id": 1372041, "author_name": "nyanp", "author_url": "", "post_date": "2021-07-01T11:26:58.937000", "content": "

Thanks for sharing the info with us!

\n

A small statistical leak in the training data is not as critical as in the test data.
\nOf course, it needs to be verified, but if it is difficult to prepare new training data, it may be enough to re-normalize the current training data even if there are still small leaks.
\nAs @philippsinger pointed out, even if you prepare completely new training data, many participants will try to use the old training data along with it, and in that case, participants will eventually have to deal with this small leak.

", "votes": 1, "replies": [] }, { "id": 1372060, "author_name": "Psi", "author_url": "", "post_date": "2021-07-01T11:51:39.547000", "content": "

Indeed, so I think the issue could only be solved if you re-create the exact same current train/test data without leak.

", "votes": 7, "replies": [] }, { "id": 1373353, "author_name": "FelipeKitamura, MD, PhD", "author_url": "", "post_date": "2021-07-02T12:32:51.197000", "content": "

Thank you, @yuhongc! Do you think generating the entire data (train and test) as uint16 could help?

", "votes": 1, "replies": [] }, { "id": 1373401, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-02T13:17:29.127000", "content": "
\n

Indeed, so I think the issue could only be solved if you re-create the exact same current train/test data without leak.

\n
\n

I agree.

\n

However, this assumes that any randomness used for generating signals can be reproduced. I hope all random seeds were saved…

", "votes": 3, "replies": [] }, { "id": 1374969, "author_name": "", "author_url": "", "post_date": "2021-07-03T18:35:46.200000", "content": "", "votes": -1, "replies": [] } ] }, { "id": 1365400, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-06-25T17:17:51.893000", "content": "

Can it be a code competition with the new data?

", "votes": 11, "replies": [ { "id": 1370229, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-06-30T03:05:15.877000", "content": "

@inversion Please Take this into consideration

", "votes": 1, "replies": [] }, { "id": 1370483, "author_name": "Psi", "author_url": "", "post_date": "2021-06-30T07:35:39.150000", "content": "

I am really glad that more and more people see the advantages of code competitions :) I always try to argue for it.

", "votes": 8, "replies": [] }, { "id": 1370836, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-30T12:48:07.573000", "content": "

I'll be a full supporter of code competitions when we have GPU more recent than 5 years old P100.

", "votes": 1, "replies": [] }, { "id": 1370838, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-30T12:49:12.073000", "content": "

Main pro for code competition is to make it harder to use test data distribution to overfit models to test data.

", "votes": 0, "replies": [] }, { "id": 1370850, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-06-30T12:57:19.890000", "content": "
\n

I'll be a full supporter of code competitions when we have GPU more recent than 5 years old P100.

\n
\n

It will be difficult for them to afford anything of a substantial upgrade to the P100. Maybe use a T4 cause it`s better in Mixed_float16 ?

", "votes": 0, "replies": [] }, { "id": 1370911, "author_name": "Psi", "author_url": "", "post_date": "2021-06-30T13:37:45.320000", "content": "

For inference P100 is more than enough, not sure why you would need something better.

", "votes": 3, "replies": [] }, { "id": 1371050, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-30T15:53:05.347000", "content": "

Small GPU memory can be an issue in some cases.

", "votes": 0, "replies": [] }, { "id": 1371054, "author_name": "Psi", "author_url": "", "post_date": "2021-06-30T15:54:33.797000", "content": "

I dont know of any model not fitting into P100 memory, apart from some ultra heavy transformer nlp models that are basically not used on kaggle anyways.

", "votes": 3, "replies": [] }, { "id": 1371104, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-30T16:52:16.440000", "content": "

Not all models are NNs. I got stuck when using RAPIDS once.

", "votes": 0, "replies": [] }, { "id": 1371884, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-01T09:39:22.257000", "content": "

@cpmpml no offense but kaggle code competition do replicate real life condition where you do not always have access to the Best hardware for deployment.

", "votes": 4, "replies": [] } ] }, { "id": 1364351, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-24T22:03:12.797000", "content": "

Good news, thanks. Will you also provide instructions on how to remove the distribution leak from existing training an d test data? Indeed, if we train on data with the leak and predict on new test without leak then results may be irrelevant.

\n

I suggested some way, but there maybe other, better ways.

", "votes": 4, "replies": [ { "id": 1364358, "author_name": "عثمان", "author_url": "", "post_date": "2021-06-24T22:09:50.863000", "content": "

If possible, it'd be great if instead of just the original (zip) data, when the new training set is provided (current train + current test), they are re-processed according to best practices. That way individual contestants don't need to do any further processing.

", "votes": 0, "replies": [] }, { "id": 1364374, "author_name": "Yuhong Chen", "author_url": "", "post_date": "2021-06-24T22:33:58.717000", "content": "

Fully fixing the distribution leak in the current dataset may be difficult because the data was already in float16 pre-injection. We really didn't see the truncation issue coming 😔.

\n

We may consider making a new training set with the truncation issue fixed, but some internal discussion is needed before I can say for sure.

", "votes": 10, "replies": [] }, { "id": 1364709, "author_name": "Psi", "author_url": "", "post_date": "2021-06-25T06:49:16.003000", "content": "

But you will fix it in new test set, right @yuhongc ?

", "votes": 5, "replies": [] }, { "id": 1364965, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-25T10:55:53.927000", "content": "

@yuhongc Fixing only in new test isn't enough as it introduces a distribution shift between new train and new test.

\n

I get that creating a new train set is a lot of work, which is why I am asking if the prepossessing I proposed was enough to remove this distribution shift.

\n

And thanks for confirming my guess that your performed standardization in FP16.

", "votes": 2, "replies": [] }, { "id": 1364971, "author_name": "Psi", "author_url": "", "post_date": "2021-06-25T11:03:31.737000", "content": "

Fixing only new test should be enough, if you standardize properly yourself in training / validation.

", "votes": 2, "replies": [] }, { "id": 1364993, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-25T11:26:07.033000", "content": "
\n

if you standardize properly yourself

\n
\n

This is what I proposed, and this is what I am asking confirmation for.

\n

Given some files were standardized in FP32 and others (snippets with message) in FP16 in current training set, there maybe distribution differences that cannot be removed.

", "votes": 5, "replies": [] }, { "id": 1365465, "author_name": "Yuhong Chen", "author_url": "", "post_date": "2021-06-25T18:55:32.440000", "content": "

@cpmpml Really appreciate your comments and support. I will investigate whether the standardization you did or some sort of other preprocessing is enough to fix the distribution difference from quantization.

\n

If it's indeed irreversible, I think we'll release a replacement training set, (while keeping the old data available out of fairness), but again can't say for sure without internal discussion.

", "votes": 4, "replies": [] }, { "id": 1365473, "author_name": "Yuhong Chen", "author_url": "", "post_date": "2021-06-25T19:03:09.737000", "content": "

@philippsinger We believe it's fixed in the new test. We'll be verifying this over the next few days. The worry with the current train+test right now is that casting back and forth between FP16 and FP32 (mea culpa) may have introduced tiny irreversible differences in the statistics because of the difference in accuracy between the two data formats.

", "votes": 6, "replies": [] }, { "id": 1365856, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-26T07:41:56.873000", "content": "

@yuhongc thanks you. Having a responsive host means a lot for competitors.

", "votes": 2, "replies": [] } ] }, { "id": 1381061, "author_name": "VikramanK", "author_url": "", "post_date": "2021-07-08T15:30:59.720000", "content": "

hello @inversion,
\ncould you please provide an update on the datasets? also will the timeline for submission be extened?

", "votes": 1, "replies": [] }, { "id": 1364437, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-25T02:05:44.763000", "content": "

Thank you for updating information. I really appreciate your hard work.

\n

I've explored several methods while the competition was suspended.
\nI'm looking forward to trying them out on new datasets😃

", "votes": 2, "replies": [] }, { "id": 1370452, "author_name": "Gia Huy Nguyễn", "author_url": "", "post_date": "2021-06-30T07:16:51.023000", "content": "

Can it be a code competition with the new data?

", "votes": -1, "replies": [] }, { "id": 1371177, "author_name": "shiv", "author_url": "", "post_date": "2021-06-30T18:19:51.503000", "content": "

Hi, can it be a code competition with a new dataset

", "votes": -1, "replies": [] }, { "id": 1480398, "author_name": "Ctrl_CV", "author_url": "", "post_date": "2021-08-19T02:26:35.560000", "content": "

Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:

\n

One account per participant
\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\nNo private sharing outside teams
\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.
\nbut our results were cancelled. Is there a staff member who can explain?

", "votes": 0, "replies": [] }, { "id": 1371216, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-06-30T18:58:45.170000", "content": "

@inversion , One doubt since the competition is going to restart does it means that the public Leaderboard as well all submissions made till now by all participants for this Competition will reset ?

", "votes": 0, "replies": [] }, { "id": 1364559, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-06-25T04:51:20.143000", "content": "

@inversion Could you also please check the How do you zip the files. As your adding More files to the train data it will become impossible to use this dataset with Colab Pro. So a well zipped dataset will surely help

", "votes": 0, "replies": [ { "id": 1365015, "author_name": "assign", "author_url": "", "post_date": "2021-06-25T11:46:31.497000", "content": "

use GCS directly (if you use tensorflow)
\nyou can access data even if you use colab w/o pro

\n

but one thing clear is training time will be increase almost 1.7

", "votes": 4, "replies": [] }, { "id": 1365366, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-06-25T16:43:44.987000", "content": "

@assign first of all you have pay a lot for GCS. And my training times are like already one hour and increase by 1.7 will mean i have a nearly 2 hour training Time which honestly a lot

", "votes": 0, "replies": [] }, { "id": 1365631, "author_name": "assign", "author_url": "", "post_date": "2021-06-26T00:59:29.067000", "content": "

I means

\n
    \n
  1. make it as kaggle dataset
    2. \n
  2. get gcs bucket id
    3. \n
  3. read by tensorflow record format
    4. \n
\n

1 preprocess dataset as tfrec format

\n

2 Kaggle kernel after add the dataset

\n
from kaggle_datasets import KaggleDatasets\nKaggleDatasets().get_gcs_path(<target dataset name>)# without ../input/\n
\n

example of it

\n

example of output:
\n'gs://kds-0752b66b41214b9aeeb3741bbd6b1dcf6c443007693b57cc9f4e2ea5'

\n

3 Colab

\n
GCS_PATH = <output of upper code>\nfiles = list(np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))) # depend on your file name\nds = tf.data.TFRecordDataset(files, num_parallel_reads = len(files))\nds = ds.map(<your parsing function>)\nds = ds.batch(512)\nfor I in ds: ~~~\n
\n

you need to preprocess dataset as serialized tensor,
\nand design suitable parsing function.
\nI recommend to read tf.io documents

\n

I don't know whether this is google's design for using gcs freely.
\nBut, we can use gcs without cost

\n

well, with this approach, using TPU much easier and we can expect speed up(1 epoch: 17min -> 1min)

\n

I'll delete it if it violate kaggle community guidelines

", "votes": 2, "replies": [] } ] }, { "id": 1370957, "author_name": "", "author_url": "", "post_date": "2021-06-30T14:12:53.690000", "content": "", "votes": 0, "replies": [] }, { "id": 1370699, "author_name": "Mateusz Ostrowski", "author_url": "", "post_date": "2021-06-30T10:44:01.613000", "content": "

Thank you for the update 😊

", "votes": 0, "replies": [] }, { "id": 1364380, "author_name": "", "author_url": "", "post_date": "2021-06-24T22:52:59.537000", "content": "

Thank you for the update 😊

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1364280": "Hi everyone -\n\nHere's an update of where things stand on the competition reset:\n- A new test set has been generated, taking into account the fantastic feedback we've received from the community. (Thank you for all of the insights!)\n- We're in the process of scrutinizing the data, and are targeting a relaunch mid-next week (but may need more time if we find any issues along the way).\n- The new training set will include the current training _and_ test set (with the test set labels)\n- We'll adjust the competition timeline at that time to reflect the time lost with the reset\n\nAn additional bit of detail on the time stamp leakage failure point (which I did a post-mortem for):\n\nKaggle actually has a triple-redundant system for time stamp leakage, where all points failed: I [previously mentioned](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246782) our best practices of random file saves and secondary checks, but our system *should* have been tolerant to any human failure points, since our backend actually re-creates time stamps for our competition files on the Data page after upload. In this instance, because our zipping process was erroring out, the \"Download All\" zip file was manually put into position, rather than being automatically created from the uploaded competition files. (Which, by the way, is the reason the timestamp leakage *didn't* show up when the data in the Notebooks environment, because it was processed normally, and only when using the downloaded zip file from the competition page, which wasn't.) \n\nIt's a good lesson on the importances of sticking to best practices. :-)", "1373457": "Were we currently stand: The data generation has taken a bit longer after it was determined the training set would also need to be re-generated, but it is nearly complete. We're going into a US long weekend, and I don't expect significant work over the next 4 days, but I'll pick things back up next Wednesday. With processing and checking, I'd imagine things will be ready for relaunch early the week of July 12.", "1371593": "Posting as a top-level comment for visibility:\n\nJust investigated whether renormalizing the current training set is able to remove the statistics leak, and unfortunately the truncation from casting into FP16 caused minute differences in the mean and standard deviations between positive and negative samples that we weren't able to remove.\n\nI will be discussing with @inversion to decide on the best way to mitigate this. We will most likely be generating a new training set without leaks.\n\nAverage of Means from On and Off (blue is needles, orange is new haystack)\n
Average Means from On and Off
\n\nAverage of STDs from On and Off (blue is needles, orange is new haystack)\n
Average of STDs from On and Off
", "1365400": "Can it be a code competition with the new data?", "1364351": "Good news, thanks. Will you also provide instructions on how to remove the distribution leak from existing training an d test data? Indeed, if we train on data with the leak and predict on new test without leak then results may be irrelevant.\n\nI suggested [some way](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/248194), but there maybe other, better ways.", "1381061": "hello @inversion, \ncould you please provide an update on the datasets? also will the timeline for submission be extened? ", "1364437": "Thank you for updating information. I really appreciate your hard work.\n\nI've explored several methods while the competition was suspended.\nI'm looking forward to trying them out on new datasets😃", "1370452": "Can it be a code competition with the new data?", "1371177": "Hi, can it be a code competition with a new dataset\n", "1480398": "Why was our team cancelled? After communicating with my teammates, I found that we did not violate two principles:\n\nOne account per participant\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.\nNo private sharing outside teams\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.\nbut our results were cancelled. Is there a staff member who can explain?\n", "1371216": "@inversion , One doubt since the competition is going to restart does it means that the public Leaderboard as well all submissions made till now by all participants for this Competition will reset ?", "1364559": "@inversion Could you also please check the How do you zip the files. As your adding More files to the train data it will become impossible to use this dataset with Colab Pro. So a well zipped dataset will surely help", "1370957": "", "1370699": "Thank you for the update 😊", "1364380": "Thank you for the update 😊" } }