It is amazing to see deep learning based approaches have achieved very good results. Before exploring the DL side, I would love to try and push forward the limit of (manual) feature engineering, specifically the signal processing based feature engineering techniques (compute peak statistics after wavelet de-noising, signal entropy, ...).
\n\nIn many previous competitions, I am more of a consumer than producer. As also promised to myself, I will share more of my thoughts and solutions in the future competitions. In this competition, I would like to share my starter code written in Pytorch, which is compatible to both signal processing workflow and deep learning workflow. Please find the code here: https://github.com/randxie/Kaggle-VSB-Baseline
\n\nKey Features are:
\n\nThoughts:
\n\nThe biggest challenge I have is to design a stable validation scheme. Currently, I used group KFold to make sure signals from same group of power lines are in the same CV folds. However, the validation score is still way off from the LB scores. I suspect there are some feature distribution shift or label distribution shift. Please also share your validation journey.
\n\nFinally, happy new year and enjoy the competition!!
", "messages": [ { "id": 448348, "postDate": "2018-12-31T20:48:36.610Z", "content": "It is amazing to see deep learning based approaches have achieved very good results. Before exploring the DL side, I would love to try and push forward the limit of (manual) feature engineering, specifically the signal processing based feature engineering techniques (compute peak statistics after wavelet de-noising, signal entropy, ...).
\n\nIn many previous competitions, I am more of a consumer than producer. As also promised to myself, I will share more of my thoughts and solutions in the future competitions. In this competition, I would like to share my starter code written in Pytorch, which is compatible to both signal processing workflow and deep learning workflow. Please find the code here: https://github.com/randxie/Kaggle-VSB-Baseline
\n\nKey Features are:
\n\nThoughts:
\n\nThe biggest challenge I have is to design a stable validation scheme. Currently, I used group KFold to make sure signals from same group of power lines are in the same CV folds. However, the validation score is still way off from the LB scores. I suspect there are some feature distribution shift or label distribution shift. Please also share your validation journey.
\n\nFinally, happy new year and enjoy the competition!!
", "rawMarkdown": "It is amazing to see deep learning based approaches have achieved very good results. Before exploring the DL side, I would love to try and push forward the limit of (manual) feature engineering, specifically the signal processing based feature engineering techniques (compute peak statistics after wavelet de-noising, signal entropy, ...).\n\nIn many previous competitions, I am more of a consumer than producer. As also promised to myself, I will share more of my thoughts and solutions in the future competitions. In this competition, I would like to share my starter code written in Pytorch, which is compatible to both signal processing workflow and deep learning workflow. Please find the code here: https://github.com/randxie/Kaggle-VSB-Baseline\n\nKey Features are:\n\n1. Use Pytorch dataloader to do feature extraction, which leverages the build-in multiprocessing capability. It also makes an easy extension to deep learning workflow.\n2. Add new feature extractor only requires inheriting the AbstractExtractor then write your **process_fn** and **extract_fn**. Before computation, feature extractor will use the first signal to infer feature dimension so that you don't need to pre-specify that. The pipeline will also automatically cache the computed features for you.\n3. Use json to configure parameters in the pipeline. It helps you to document your experiments.\n4. Use random seed to control data augmentation so that results are reproducible.\n\nThoughts:\n\nThe biggest challenge I have is to design a stable validation scheme. Currently, I used group KFold to make sure signals from same group of power lines are in the same CV folds. However, the validation score is still way off from the LB scores. I suspect there are some feature distribution shift or label distribution shift. Please also share your validation journey.\n\nFinally, happy new year and enjoy the competition!! \n", "votes": 30 }, { "id": 448421, "postDate": "2019-01-01T05:22:06.740Z", "content": "\n\n\nThe biggest challenge I have is to design a stable validation scheme.
\n
That's the biggest challenge in every Kaggle competition. Figure that one out, and you are 90% on your way to Gold. :)
", "rawMarkdown": ">The biggest challenge I have is to design a stable validation scheme.\n\nThat's the biggest challenge in _every_ Kaggle competition. Figure that one out, and you are 90% on your way to Gold. :)", "votes": 3, "replies": [ { "id": 448763, "postDate": "2019-01-02T03:19:21.133Z", "content": "That's true. In this competition, the unbalance label makes it more challenging to build a good validation scheme.
", "rawMarkdown": "That's true. In this competition, the unbalance label makes it more challenging to build a good validation scheme. ", "votes": 2 } ] }, { "id": 469044, "postDate": "2019-02-10T11:23:17.407Z", "content": "\n\n\nHowever, the validation score is still way off from the LB scores.
\n
I saw the following code snippet in src/common.py
\n\nSAMPLING_FREQ = 80000/0.02 # 80,000 data points taken over 20 ms
However, according to the data page, \"Each signal contains 800,000 measurements of a power line's voltage, taken over 20 milliseconds.\"\nI am just wondering if you can check this, just in case!
", "rawMarkdown": ">However, the validation score is still way off from the LB scores.\n\nI saw the following code snippet in src/common.py\n\n`SAMPLING_FREQ = 80000/0.02 # 80,000 data points taken over 20 ms`\n\nHowever, according to the data page, \"Each signal contains 800,000 measurements of a power line's voltage, taken over 20 milliseconds.\"\nI am just wondering if you can check this, just in case!", "votes": 1, "replies": [ { "id": 471955, "postDate": "2019-02-15T06:23:05.030Z", "content": "This may be caused by unbalanced distribution of the train & test dataset. I see the auc between train & test is above 0.9. You can refer to :https://www.kaggle.com/its7171/train-vs-test-analisys.
", "rawMarkdown": "This may be caused by unbalanced distribution of the train & test dataset. I see the auc between train & test is above 0.9. You can refer to :https://www.kaggle.com/its7171/train-vs-test-analisys." }, { "id": 472673, "postDate": "2019-02-16T12:56:36.107Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 464755, "postDate": "2019-02-01T12:39:19.130Z", "content": "Great Code and documentation as always ! Thanks for sharing :)
", "rawMarkdown": "Great Code and documentation as always ! Thanks for sharing :)", "votes": 1 }, { "id": 471954, "postDate": "2019-02-15T06:22:44.603Z", "rawMarkdown": "", "isDeleted": true }, { "id": 449654, "postDate": "2019-01-03T14:14:56.103Z", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": 1 } ], "comments": [ { "id": 448421, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2019-01-01T05:22:06.740000", "content": "\n\n\nThe biggest challenge I have is to design a stable validation scheme.
\n
That's the biggest challenge in every Kaggle competition. Figure that one out, and you are 90% on your way to Gold. :)
", "votes": 3, "replies": [ { "id": 448763, "author_name": "Rand Xie", "author_url": "", "post_date": "2019-01-02T03:19:21.133000", "content": "That's true. In this competition, the unbalance label makes it more challenging to build a good validation scheme.
", "votes": 2, "replies": [] } ] }, { "id": 469044, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2019-02-10T11:23:17.407000", "content": "\n\n\nHowever, the validation score is still way off from the LB scores.
\n
I saw the following code snippet in src/common.py
\n\nSAMPLING_FREQ = 80000/0.02 # 80,000 data points taken over 20 ms
However, according to the data page, \"Each signal contains 800,000 measurements of a power line's voltage, taken over 20 milliseconds.\"\nI am just wondering if you can check this, just in case!
", "votes": 1, "replies": [ { "id": 471955, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2019-02-15T06:23:05.030000", "content": "This may be caused by unbalanced distribution of the train & test dataset. I see the auc between train & test is above 0.9. You can refer to :https://www.kaggle.com/its7171/train-vs-test-analisys.
", "votes": 0, "replies": [] }, { "id": 472673, "author_name": "", "author_url": "", "post_date": "2019-02-16T12:56:36.107000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 464755, "author_name": "olivier", "author_url": "", "post_date": "2019-02-01T12:39:19.130000", "content": "Great Code and documentation as always ! Thanks for sharing :)
", "votes": 1, "replies": [] }, { "id": 471954, "author_name": "", "author_url": "", "post_date": "2019-02-15T06:22:44.603000", "content": "", "votes": 0, "replies": [] }, { "id": 449654, "author_name": "xie", "author_url": "", "post_date": "2019-01-03T14:14:56.103000", "content": "Thanks!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "448348": "It is amazing to see deep learning based approaches have achieved very good results. Before exploring the DL side, I would love to try and push forward the limit of (manual) feature engineering, specifically the signal processing based feature engineering techniques (compute peak statistics after wavelet de-noising, signal entropy, ...).\n\nIn many previous competitions, I am more of a consumer than producer. As also promised to myself, I will share more of my thoughts and solutions in the future competitions. In this competition, I would like to share my starter code written in Pytorch, which is compatible to both signal processing workflow and deep learning workflow. Please find the code here: https://github.com/randxie/Kaggle-VSB-Baseline\n\nKey Features are:\n\n1. Use Pytorch dataloader to do feature extraction, which leverages the build-in multiprocessing capability. It also makes an easy extension to deep learning workflow.\n2. Add new feature extractor only requires inheriting the AbstractExtractor then write your **process_fn** and **extract_fn**. Before computation, feature extractor will use the first signal to infer feature dimension so that you don't need to pre-specify that. The pipeline will also automatically cache the computed features for you.\n3. Use json to configure parameters in the pipeline. It helps you to document your experiments.\n4. Use random seed to control data augmentation so that results are reproducible.\n\nThoughts:\n\nThe biggest challenge I have is to design a stable validation scheme. Currently, I used group KFold to make sure signals from same group of power lines are in the same CV folds. However, the validation score is still way off from the LB scores. I suspect there are some feature distribution shift or label distribution shift. Please also share your validation journey.\n\nFinally, happy new year and enjoy the competition!! \n", "448421": ">The biggest challenge I have is to design a stable validation scheme.\n\nThat's the biggest challenge in _every_ Kaggle competition. Figure that one out, and you are 90% on your way to Gold. :)", "469044": ">However, the validation score is still way off from the LB scores.\n\nI saw the following code snippet in src/common.py\n\n`SAMPLING_FREQ = 80000/0.02 # 80,000 data points taken over 20 ms`\n\nHowever, according to the data page, \"Each signal contains 800,000 measurements of a power line's voltage, taken over 20 milliseconds.\"\nI am just wondering if you can check this, just in case!", "464755": "Great Code and documentation as always ! Thanks for sharing :)", "471954": "", "449654": "Thanks!" } }