![\"Image\"](\"https://i.imgur.com/LLVIFge.png\")
\nLarger versionJust wondering if anyone else is seeing a consistent difference in performance of a model on the train set compared to when submitting results.
\n\nI have split off 1/8 of the training samples (split on the id_measurement column) as a validation set and I am confident that I have not made any errors in my code that means this set is seen during training. I consistently score around 0.6 --> 0.7 with this validation set but my best score on submission is very much weaker (c. 0.4)
I understand that I have only a small local validation set and there are lots of avenues for me still to explore but it seemed like a large gulf in performance so I thought I'd ask if anyone else had noticed a similar pattern?
\n\nDo we have any information from the organisers as to how the data was gathered? Is it simply random chance that a given measurement ends up in the test set instead of the train set?
", "messages": [ { "id": "458683", "postDate": "01/20/2019 09:15:10", "content": "Just wondering if anyone else is seeing a consistent difference in performance of a model on the train set compared to when submitting results.
\n\nI have split off 1/8 of the training samples (split on the id_measurement column) as a validation set and I am confident that I have not made any errors in my code that means this set is seen during training. I consistently score around 0.6 --> 0.7 with this validation set but my best score on submission is very much weaker (c. 0.4)
I understand that I have only a small local validation set and there are lots of avenues for me still to explore but it seemed like a large gulf in performance so I thought I'd ask if anyone else had noticed a similar pattern?
\n\nDo we have any information from the organisers as to how the data was gathered? Is it simply random chance that a given measurement ends up in the test set instead of the train set?
", "rawMarkdown": "Just wondering if anyone else is seeing a consistent difference in performance of a model on the train set compared to when submitting results.\n\nI have split off 1/8 of the training samples (split on the `id_measurement` column) as a validation set and I am confident that I have not made any errors in my code that means this set is seen during training. I consistently score around 0.6 --> 0.7 with this validation set but my best score on submission is very much weaker (c. 0.4)\n\nI understand that I have only a small local validation set and there are lots of avenues for me still to explore but it seemed like a large gulf in performance so I thought I'd ask if anyone else had noticed a similar pattern?\n\nDo we have any information from the organisers as to how the data was gathered? Is it simply random chance that a given measurement ends up in the test set instead of the train set?", "votes": null }, { "id": "459105", "postDate": "01/21/2019 07:31:47", "content": "There is a similar thread:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/76984
", "rawMarkdown": "There is a similar thread:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/76984", "votes": null }, { "id": "459416", "postDate": "01/21/2019 17:24:25", "content": "I'm presently right there with you.
\n\nMy approach to tackling this problem is to take the raw signal, perform some signal processing, and then extract what I'm perceiving to be useful features into a data frame that can be used to train a Random Forest machine learning model.
\n\nI've been running Monte Carlo trials varying the random_state of my train_test_split of my extracted signal features and my model locally is producing Matthews Correlation Coefficient scores consistently within 0.87 - 0.91.
My final step is to use the extracted features from the entire training set to train the model and then produce predictions on the test set and create a file in the submission format. My first submission netted a 0.34 on the leader board.
\n\nI understand this is a difficult problem to solve given the heavy imbalance of good and fault labeled data but I thought a Random Forest model would offer some resilience to overfitting to the dominant class.
\n\nI too am at a loss on how to improve further with a Random Forest classifier since I already believe I'm reaching the high end locally. From what I'm seeing int he discussion forums, it seems most of the successful teams on the leader board are using k-Fold Cross Validation or Recurrent Neural Nets like LSTM.
", "rawMarkdown": "I'm presently right there with you. \n\nMy approach to tackling this problem is to take the raw signal, perform some signal processing, and then extract what I'm perceiving to be useful features into a data frame that can be used to train a Random Forest machine learning model.\n\nI've been running Monte Carlo trials varying the random_state of my `train_test_split` of my extracted signal features and my model locally is producing Matthews Correlation Coefficient scores consistently within 0.87 - 0.91.\n\nMy final step is to use the extracted features from the entire training set to train the model and then produce predictions on the test set and create a file in the submission format. My first submission netted a 0.34 on the leader board.\n\nI understand this is a difficult problem to solve given the heavy imbalance of good and fault labeled data but I thought a Random Forest model would offer some resilience to overfitting to the dominant class.\n\nI too am at a loss on how to improve further with a Random Forest classifier since I already believe I'm reaching the high end locally. From what I'm seeing int he discussion forums, it seems most of the successful teams on the leader board are using k-Fold Cross Validation or Recurrent Neural Nets like LSTM.", "votes": null }, { "id": "459450", "postDate": "01/21/2019 18:58:50", "content": "\n\n\nMatthews Correlation Coefficient scores consistently within 0.87 - 0.91.
\n
This is suspicious! My CV is 0.72 and 0.63 LB (using LSTM).
\n\nBefore that I tried LGBM with simple features, both CV and LB was low, but the gap was wider.
", "rawMarkdown": "> Matthews Correlation Coefficient scores consistently within 0.87 - 0.91.\n\nThis is suspicious! My CV is 0.72 and 0.63 LB (using LSTM). \n\nBefore that I tried LGBM with simple features, both CV and LB was low, but the gap was wider.", "votes": null }, { "id": "459696", "postDate": "01/22/2019 07:32:32", "content": "Just make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case. Best Lightgbm model for me has CV = 0.697872, LB = 0.555. \nHope, You are not using id_measurement as a feature also.
", "rawMarkdown": "Just make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case. Best Lightgbm model for me has CV = 0.697872, LB = 0.555. \nHope, You are not using id_measurement as a feature also.", "votes": null }, { "id": "461594", "postDate": "01/26/2019 13:12:34", "content": "\n\n\nHope, You are not using
\nid_measurementas a feature also.
You had me scared that I had made a mistake there for a minute because that would have made perfect sense to get such high MCC scores with that present. I double checked that I am not inadvertently passing this feature to my model for training and test.
\n\nMy current approach was to essentially follow the outline laid out in the academic paper published by Tomas Vantuch. I'm doing my own signal processing to de-noise, some basic pulse cancellation, and extract features that I can be used to fit the model. Namely many of the features he found useful in the paper: entropy, peak characteristics, etc are the same ones I'm using. My latest theory is that some of these characteristics between signals correlate just as well as id_measurement and help allow for an unusually high MCC.
\n\n\nJust make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case.
\n
I'm going to spend some time this weekend following up on your suggestion to re-evaluate the distribution of features in each set and re-run models with features that have similar distributions in both sets. In all likelihood I'll probably have to circle back to my signal processing code and make some tweaks in data prep. I attached a pair plot of my current feature distribution (along with \"fault\") if you were curious where they stood at the moment.
\n\nThanks for the feedback and suggestions!
\n\n", "rawMarkdown": "> Hope, You are not using `id_measurement` as a feature also.\n\nYou had me scared that I had made a mistake there for a minute because that would have made perfect sense to get such high MCC scores with that present. I double checked that I am not inadvertently passing this feature to my model for training and test.\n\nMy current approach was to essentially follow the outline laid out in the academic paper published by Tomas Vantuch. I'm doing my own signal processing to de-noise, some basic pulse cancellation, and extract features that I can be used to fit the model. Namely many of the features he found useful in the paper: entropy, peak characteristics, etc are the same ones I'm using. My latest theory is that some of these characteristics between signals correlate just as well as `id_measurement` and help allow for an unusually high MCC.\n\n> Just make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case.\n\nI'm going to spend some time this weekend following up on your suggestion to re-evaluate the distribution of features in each set and re-run models with features that have similar distributions in both sets. In all likelihood I'll probably have to circle back to my signal processing code and make some tweaks in data prep. I attached a pair plot of my current feature distribution (along with \"fault\") if you were curious where they stood at the moment.\n\nThanks for the feedback and suggestions!\n\n \n[Larger version](https://i.imgur.com/LLVIFge.png)", "votes": null }, { "id": "461683", "postDate": "01/26/2019 18:08:37", "content": "Great! Let me know how the boosting is working for you. I wasn’t able to get good score from boosting, hence moved to lstm, but those models are unstable so far, fixing seed can reproduce the results but changing the seed can lead to a lot of variation.
", "rawMarkdown": "Great! Let me know how the boosting is working for you. I wasn’t able to get good score from boosting, hence moved to lstm, but those models are unstable so far, fixing seed can reproduce the results but changing the seed can lead to a lot of variation.", "votes": null }, { "id": "462120", "postDate": "01/27/2019 18:09:15", "content": "A bit of a mixed bag. The \"good\" news is that now my models are scoring between 0.26-0.43 locally which is consistent with my leader board scoring around 0.34. The bad news is that I'm also starting to get the sense that this is the high end of possible performance for Random Forest. I also tried k-Nearest Neighbors and SVM with worse performance.
\n\nI'm going to follow your lead and start exploring use of some models I haven't really used before like the LTSM. Looking forward to learning something new!
", "rawMarkdown": "A bit of a mixed bag. The \"good\" news is that now my models are scoring between 0.26-0.43 locally which is consistent with my leader board scoring around 0.34. The bad news is that I'm also starting to get the sense that this is the high end of possible performance for Random Forest. I also tried k-Nearest Neighbors and SVM with worse performance.\n\nI'm going to follow your lead and start exploring use of some models I haven't really used before like the LTSM. Looking forward to learning something new!", "votes": null }, { "id": "464523", "postDate": "02/01/2019 03:07:26", "content": "Hi Jeffrey,
\n\nDid you try using those features in a LSTM ? I tried using similar features but same thing I got very different results as well between validation set and LB.
", "rawMarkdown": "Hi Jeffrey,\n\nDid you try using those features in a LSTM ? I tried using similar features but same thing I got very different results as well between validation set and LB.", "votes": null }, { "id": "464524", "postDate": "02/01/2019 03:09:12", "content": "Hi @harshit92, were you able to fix the seed to make it fully reproductible with Keras ? It seems that the GPU computation is not deterministic. I always have very different results between runs.
", "rawMarkdown": "Hi @harshit92, were you able to fix the seed to make it fully reproductible with Keras ? It seems that the GPU computation is not deterministic. I always have very different results between runs.", "votes": null }, { "id": "464529", "postDate": "02/01/2019 03:16:13", "content": "@Antoine Reveillion, I have only checked on my laptop, i will check on kaggle GPU with same code and will let you know.
", "rawMarkdown": "Antoine Reveillion, I have only checked on my laptop, i will check on kaggle GPU with same code and will let you know.", "votes": null }, { "id": "464829", "postDate": "02/01/2019 15:35:22", "content": "That's probably why then ! Thanks !
", "rawMarkdown": "That's probably why then ! Thanks !", "votes": null }, { "id": "464855", "postDate": "02/01/2019 16:46:05", "content": "@Antoine, I tried it on kaggle platform and getting differences in the results for same seed and everything else. Tried it three times and cv was 0.684,0.699 and 0.7097.
", "rawMarkdown": "Antoine, I tried it on kaggle platform and getting differences in the results for same seed and everything else. Tried it three times and cv was 0.684,0.699 and 0.7097.", "votes": null }, { "id": "464962", "postDate": "02/01/2019 22:33:36", "content": "Yes that exactly what I have, and the LB is moving the same way as well...
", "rawMarkdown": "Yes that exactly what I have, and the LB is moving the same way as well...", "votes": null } ], "comments": [ { "id": 459105, "author_name": "sergeyzlobin", "author_url": "", "post_date": "01/21/2019 07:31:47", "content": "There is a similar thread:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/76984
", "votes": null, "replies": [] }, { "id": 459416, "author_name": "jeffreyegan", "author_url": "", "post_date": "01/21/2019 17:24:25", "content": "I'm presently right there with you.
\n\nMy approach to tackling this problem is to take the raw signal, perform some signal processing, and then extract what I'm perceiving to be useful features into a data frame that can be used to train a Random Forest machine learning model.
\n\nI've been running Monte Carlo trials varying the random_state of my train_test_split of my extracted signal features and my model locally is producing Matthews Correlation Coefficient scores consistently within 0.87 - 0.91.
My final step is to use the extracted features from the entire training set to train the model and then produce predictions on the test set and create a file in the submission format. My first submission netted a 0.34 on the leader board.
\n\nI understand this is a difficult problem to solve given the heavy imbalance of good and fault labeled data but I thought a Random Forest model would offer some resilience to overfitting to the dominant class.
\n\nI too am at a loss on how to improve further with a Random Forest classifier since I already believe I'm reaching the high end locally. From what I'm seeing int he discussion forums, it seems most of the successful teams on the leader board are using k-Fold Cross Validation or Recurrent Neural Nets like LSTM.
", "votes": null, "replies": [ { "id": 459450, "author_name": "sergeyzlobin", "author_url": "", "post_date": "01/21/2019 18:58:50", "content": "\n\n\nMatthews Correlation Coefficient scores consistently within 0.87 - 0.91.
\n
This is suspicious! My CV is 0.72 and 0.63 LB (using LSTM).
\n\nBefore that I tried LGBM with simple features, both CV and LB was low, but the gap was wider.
", "votes": null, "replies": [] }, { "id": 459696, "author_name": "harshit92", "author_url": "", "post_date": "01/22/2019 07:32:32", "content": "Just make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case. Best Lightgbm model for me has CV = 0.697872, LB = 0.555. \nHope, You are not using id_measurement as a feature also.
", "votes": null, "replies": [] }, { "id": 461594, "author_name": "jeffreyegan", "author_url": "", "post_date": "01/26/2019 13:12:34", "content": "\n\n\nHope, You are not using
\nid_measurementas a feature also.
You had me scared that I had made a mistake there for a minute because that would have made perfect sense to get such high MCC scores with that present. I double checked that I am not inadvertently passing this feature to my model for training and test.
\n\nMy current approach was to essentially follow the outline laid out in the academic paper published by Tomas Vantuch. I'm doing my own signal processing to de-noise, some basic pulse cancellation, and extract features that I can be used to fit the model. Namely many of the features he found useful in the paper: entropy, peak characteristics, etc are the same ones I'm using. My latest theory is that some of these characteristics between signals correlate just as well as id_measurement and help allow for an unusually high MCC.
\n\n\nJust make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case.
\n
I'm going to spend some time this weekend following up on your suggestion to re-evaluate the distribution of features in each set and re-run models with features that have similar distributions in both sets. In all likelihood I'll probably have to circle back to my signal processing code and make some tweaks in data prep. I attached a pair plot of my current feature distribution (along with \"fault\") if you were curious where they stood at the moment.
\n\nThanks for the feedback and suggestions!
\n\n", "votes": null, "replies": [] }, { "id": 461683, "author_name": "harshit92", "author_url": "", "post_date": "01/26/2019 18:08:37", "content": "Great! Let me know how the boosting is working for you. I wasn’t able to get good score from boosting, hence moved to lstm, but those models are unstable so far, fixing seed can reproduce the results but changing the seed can lead to a lot of variation.
", "votes": null, "replies": [] }, { "id": 462120, "author_name": "jeffreyegan", "author_url": "", "post_date": "01/27/2019 18:09:15", "content": "A bit of a mixed bag. The \"good\" news is that now my models are scoring between 0.26-0.43 locally which is consistent with my leader board scoring around 0.34. The bad news is that I'm also starting to get the sense that this is the high end of possible performance for Random Forest. I also tried k-Nearest Neighbors and SVM with worse performance.
\n\nI'm going to follow your lead and start exploring use of some models I haven't really used before like the LTSM. Looking forward to learning something new!
", "votes": null, "replies": [] }, { "id": 464523, "author_name": "areveillon", "author_url": "", "post_date": "02/01/2019 03:07:26", "content": "Hi Jeffrey,
\n\nDid you try using those features in a LSTM ? I tried using similar features but same thing I got very different results as well between validation set and LB.
", "votes": null, "replies": [] }, { "id": 464524, "author_name": "areveillon", "author_url": "", "post_date": "02/01/2019 03:09:12", "content": "Hi @harshit92, were you able to fix the seed to make it fully reproductible with Keras ? It seems that the GPU computation is not deterministic. I always have very different results between runs.
", "votes": null, "replies": [] }, { "id": 464529, "author_name": "harshit92", "author_url": "", "post_date": "02/01/2019 03:16:13", "content": "@Antoine Reveillion, I have only checked on my laptop, i will check on kaggle GPU with same code and will let you know.
", "votes": null, "replies": [] }, { "id": 464829, "author_name": "areveillon", "author_url": "", "post_date": "02/01/2019 15:35:22", "content": "That's probably why then ! Thanks !
", "votes": null, "replies": [] }, { "id": 464855, "author_name": "harshit92", "author_url": "", "post_date": "02/01/2019 16:46:05", "content": "@Antoine, I tried it on kaggle platform and getting differences in the results for same seed and everything else. Tried it three times and cv was 0.684,0.699 and 0.7097.
", "votes": null, "replies": [] }, { "id": 464962, "author_name": "areveillon", "author_url": "", "post_date": "02/01/2019 22:33:36", "content": "Yes that exactly what I have, and the LB is moving the same way as well...
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "458683": "Just wondering if anyone else is seeing a consistent difference in performance of a model on the train set compared to when submitting results.\n\nI have split off 1/8 of the training samples (split on the `id_measurement` column) as a validation set and I am confident that I have not made any errors in my code that means this set is seen during training. I consistently score around 0.6 --> 0.7 with this validation set but my best score on submission is very much weaker (c. 0.4)\n\nI understand that I have only a small local validation set and there are lots of avenues for me still to explore but it seemed like a large gulf in performance so I thought I'd ask if anyone else had noticed a similar pattern?\n\nDo we have any information from the organisers as to how the data was gathered? Is it simply random chance that a given measurement ends up in the test set instead of the train set?", "459105": "There is a similar thread:\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/76984", "459416": "I'm presently right there with you. \n\nMy approach to tackling this problem is to take the raw signal, perform some signal processing, and then extract what I'm perceiving to be useful features into a data frame that can be used to train a Random Forest machine learning model.\n\nI've been running Monte Carlo trials varying the random_state of my `train_test_split` of my extracted signal features and my model locally is producing Matthews Correlation Coefficient scores consistently within 0.87 - 0.91.\n\nMy final step is to use the extracted features from the entire training set to train the model and then produce predictions on the test set and create a file in the submission format. My first submission netted a 0.34 on the leader board.\n\nI understand this is a difficult problem to solve given the heavy imbalance of good and fault labeled data but I thought a Random Forest model would offer some resilience to overfitting to the dominant class.\n\nI too am at a loss on how to improve further with a Random Forest classifier since I already believe I'm reaching the high end locally. From what I'm seeing int he discussion forums, it seems most of the successful teams on the leader board are using k-Fold Cross Validation or Recurrent Neural Nets like LSTM.", "459450": "> Matthews Correlation Coefficient scores consistently within 0.87 - 0.91.\n\nThis is suspicious! My CV is 0.72 and 0.63 LB (using LSTM). \n\nBefore that I tried LGBM with simple features, both CV and LB was low, but the gap was wider.", "459696": "Just make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case. Best Lightgbm model for me has CV = 0.697872, LB = 0.555. \nHope, You are not using id_measurement as a feature also.", "461594": "> Hope, You are not using `id_measurement` as a feature also.\n\nYou had me scared that I had made a mistake there for a minute because that would have made perfect sense to get such high MCC scores with that present. I double checked that I am not inadvertently passing this feature to my model for training and test.\n\nMy current approach was to essentially follow the outline laid out in the academic paper published by Tomas Vantuch. I'm doing my own signal processing to de-noise, some basic pulse cancellation, and extract features that I can be used to fit the model. Namely many of the features he found useful in the paper: entropy, peak characteristics, etc are the same ones I'm using. My latest theory is that some of these characteristics between signals correlate just as well as `id_measurement` and help allow for an unusually high MCC.\n\n> Just make sure you use features which have pretty much the same distributions in train and test set. I Initially used Lightgbm and had that problem, though difference wasn't that high in my case.\n\nI'm going to spend some time this weekend following up on your suggestion to re-evaluate the distribution of features in each set and re-run models with features that have similar distributions in both sets. In all likelihood I'll probably have to circle back to my signal processing code and make some tweaks in data prep. I attached a pair plot of my current feature distribution (along with \"fault\") if you were curious where they stood at the moment.\n\nThanks for the feedback and suggestions!\n\n \n[Larger version](https://i.imgur.com/LLVIFge.png)", "461683": "Great! Let me know how the boosting is working for you. I wasn’t able to get good score from boosting, hence moved to lstm, but those models are unstable so far, fixing seed can reproduce the results but changing the seed can lead to a lot of variation.", "462120": "A bit of a mixed bag. The \"good\" news is that now my models are scoring between 0.26-0.43 locally which is consistent with my leader board scoring around 0.34. The bad news is that I'm also starting to get the sense that this is the high end of possible performance for Random Forest. I also tried k-Nearest Neighbors and SVM with worse performance.\n\nI'm going to follow your lead and start exploring use of some models I haven't really used before like the LTSM. Looking forward to learning something new!", "464523": "Hi Jeffrey,\n\nDid you try using those features in a LSTM ? I tried using similar features but same thing I got very different results as well between validation set and LB.", "464524": "Hi @harshit92, were you able to fix the seed to make it fully reproductible with Keras ? It seems that the GPU computation is not deterministic. I always have very different results between runs.", "464529": "Antoine Reveillion, I have only checked on my laptop, i will check on kaggle GPU with same code and will let you know.", "464829": "That's probably why then ! Thanks !", "464855": "Antoine, I tried it on kaggle platform and getting differences in the results for same seed and everything else. Tried it three times and cv was 0.684,0.699 and 0.7097.", "464962": "Yes that exactly what I have, and the LB is moving the same way as well..." }, "source": "meta" }