In this notebook, I trained the model only with off-taget images (i.e., img[[1,3,5]]) to check these images do not have any information about needles.
\nInterestingly, I got the oof score 0.610164 and LB score 0.53. It means CV score is better than chance level, and shows some over-fitting. Do you know the reason why? My hypothesis is that our CV strategy (e.g., train-val split) is not appropriate, but I am not sure.
\n| fold | \nmetric | \n
|---|---|
| 0 | \n0.602645 | \n
| 1 | \n0.613821 | \n
| 2 | \n0.613558 | \n
| 3 | \n0.612594 | \n
| 4 | \n0.612873 | \n
| oof | \n0.610164 | \n
| LB | \n0.53 | \n
In this notebook, I trained the model only with off-taget images (i.e., img[[1,3,5]]) to check these images do not have any information about needles.
\nInterestingly, I got the oof score 0.610164 and LB score 0.53. It means CV score is better than chance level, and shows some over-fitting. Do you know the reason why? My hypothesis is that our CV strategy (e.g., train-val split) is not appropriate, but I am not sure.
\n| fold | \nmetric | \n
|---|---|
| 0 | \n0.602645 | \n
| 1 | \n0.613821 | \n
| 2 | \n0.613558 | \n
| 3 | \n0.612594 | \n
| 4 | \n0.612873 | \n
| oof | \n0.610164 | \n
| LB | \n0.53 | \n
because mean and variance are different for images with target =1 from images with target 0.
", "rawMarkdown": "because mean and variance are different for images with target =1 from images with target 0.", "votes": 1, "replies": [ { "id": 1357040, "postDate": "2021-06-19T12:25:43.910Z", "content": "please can you help clarify a bit, are you saying that mean and variance of off-target images with 0 label are different from off-target images with 1 label?
", "rawMarkdown": "please can you help clarify a bit, are you saying that mean and variance of off-target images with 0 label are different from off-target images with 1 label?", "votes": 1 }, { "id": 1357093, "postDate": "2021-06-19T13:26:05.403Z", "content": "@ankitsajwan have you checked this and this?
\nThe differences of mean and variance for off-target images are not clear compared with those for on-target images, but I could see they have some predictive power.
@ankitsajwan yes, see https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901#1354331
", "rawMarkdown": "@ankitsajwan yes, see https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901#1354331", "votes": 1 } ] }, { "id": 1356383, "postDate": "2021-06-19T01:46:10.040Z", "content": "Deep learning models work with the principle of \"garbage in garbage out\". Your experiment is really interesting however, given the unexplainable nature of the model we cannot be certain if there's a necessary signal in the off-target spectrograms.
\nCNNs are feature extractors and given a supervised target, it will find something in the image to map it to the target. There's a paper that actually investigates this -
\nThe authors randomly shuffled the ground truth labels for the training data, thus an image of a cat might have a label elephant. They trained the model with this mislabeled dataset and to surprise the training accuracy reached, say, 90%. If you train the same model with correct labels, the training accuracy is also close to 90%. However, as expected the model trained with bad labels generalized POORly on the test dataset.
\nTL;DR, deep learning models can extract signals in order to map them to a target label, but it necessarily doesn't mean that we have relevant signal in the image.
\nI might be wrong but I will conduct the same experiment on my experimental setup. :)
", "rawMarkdown": "Deep learning models work with the principle of \"garbage in garbage out\". Your experiment is really interesting however, given the unexplainable nature of the model we cannot be certain if there's a necessary signal in the off-target spectrograms. \n\nCNNs are feature extractors and given a supervised target, it will find something in the image to map it to the target. There's a paper that actually investigates this - \n\nThe authors randomly shuffled the ground truth labels for the training data, thus an image of a cat might have a label elephant. They trained the model with this mislabeled dataset and to surprise the training accuracy reached, say, 90%. If you train the same model with correct labels, the training accuracy is also close to 90%. However, as expected the model trained with bad labels generalized POORly on the test dataset.\n\nTL;DR, deep learning models can extract signals in order to map them to a target label, but it necessarily doesn't mean that we have relevant signal in the image. \n\nI might be wrong but I will conduct the same experiment on my experimental setup. :)", "votes": 1, "replies": [ { "id": 1357077, "postDate": "2021-06-19T13:06:03.430Z", "content": "Thank you for your comment.
\nYes, the input was a garbage, but the output was not just a garbage. It was an interesting garbage.
\nI could reach to the statistical anomaly of the dataset on my way to digging into this strange result. I did not know why this statistical anomaly happened, and could not notice it was caused by a leakage. I am feeling sorry for it.
One thing I am still wondering is why the LB score was not as good as 0.61, because this statistical leakage gives me higher LB score. I will find the answer when the new dataset is released.
", "rawMarkdown": "Thank you for your comment.\nYes, the input was a garbage, but the output was not just a garbage. It was an *interesting* garbage. \nI could reach to the [statistical anomaly of the dataset](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901) on my way to digging into this strange result. I did not know why this statistical anomaly happened, and could not notice it was caused by a leakage. I am feeling sorry for it.\n\nOne thing I am still wondering is why the LB score was not as good as 0.61, because this statistical leakage gives me higher LB score. I will find the answer when the new dataset is released.", "votes": 1 }, { "id": 1357095, "postDate": "2021-06-19T13:31:29.760Z", "content": "I agree it's an interesting garbage.
", "rawMarkdown": "I agree it's an interesting garbage." } ] }, { "id": 1351287, "postDate": "2021-06-16T07:29:08.130Z", "content": "(just personal idea)
\nThank you for your reply.
\nIt is reasonable that aliens should live in a specific area of the universe.
\nI believe this strange property is related to how they 'measure' the data.
because mean and variance are different for images with target =1 from images with target 0.
", "votes": 1, "replies": [ { "id": 1357040, "author_name": "sajwankit", "author_url": "", "post_date": "2021-06-19T12:25:43.910000", "content": "please can you help clarify a bit, are you saying that mean and variance of off-target images with 0 label are different from off-target images with 1 label?
", "votes": 1, "replies": [] }, { "id": 1357093, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-06-19T13:26:05.403000", "content": "@ankitsajwan have you checked this and this?
\nThe differences of mean and variance for off-target images are not clear compared with those for on-target images, but I could see they have some predictive power.
@ankitsajwan yes, see https://www.kaggle.com/c/seti-breakthrough-listen/discussion/246901#1354331
", "votes": 1, "replies": [] } ] }, { "id": 1356383, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-06-19T01:46:10.040000", "content": "Deep learning models work with the principle of \"garbage in garbage out\". Your experiment is really interesting however, given the unexplainable nature of the model we cannot be certain if there's a necessary signal in the off-target spectrograms.
\nCNNs are feature extractors and given a supervised target, it will find something in the image to map it to the target. There's a paper that actually investigates this -
\nThe authors randomly shuffled the ground truth labels for the training data, thus an image of a cat might have a label elephant. They trained the model with this mislabeled dataset and to surprise the training accuracy reached, say, 90%. If you train the same model with correct labels, the training accuracy is also close to 90%. However, as expected the model trained with bad labels generalized POORly on the test dataset.
\nTL;DR, deep learning models can extract signals in order to map them to a target label, but it necessarily doesn't mean that we have relevant signal in the image.
\nI might be wrong but I will conduct the same experiment on my experimental setup. :)
", "votes": 1, "replies": [ { "id": 1357077, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-06-19T13:06:03.430000", "content": "Thank you for your comment.
\nYes, the input was a garbage, but the output was not just a garbage. It was an interesting garbage.
\nI could reach to the statistical anomaly of the dataset on my way to digging into this strange result. I did not know why this statistical anomaly happened, and could not notice it was caused by a leakage. I am feeling sorry for it.
One thing I am still wondering is why the LB score was not as good as 0.61, because this statistical leakage gives me higher LB score. I will find the answer when the new dataset is released.
", "votes": 1, "replies": [] }, { "id": 1357095, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-06-19T13:31:29.760000", "content": "I agree it's an interesting garbage.
", "votes": 0, "replies": [] } ] }, { "id": 1351287, "author_name": "assign", "author_url": "", "post_date": "2021-06-16T07:29:08.130000", "content": "(just personal idea)
\nThank you for your reply.
\nIt is reasonable that aliens should live in a specific area of the universe.
\nI believe this strange property is related to how they 'measure' the data.