{ "id": 75771, "title": "Problem description", "url": "/competitions/vsb-power-line-fault-detection/discussion/75771", "author_name": "Tomas Vantuch", "post_date": "2018-12-26T11:02:45.985000", "votes": 85, "comment_count": 42, "views": 0, "content": "

Dear All,

\n\n

I would like to welcome you here in this competition, my name is Tomas Vantuch and I'm a data scientist from ENET at VSB-TU Ostrava responsible for this party :) In a few following lines, I would like to give you some hints about the data and what was helpful for us during our research.

\n\n

As you may notice, the signal data comes from the real environment, not a lab, and they contain a lot of background noise. These signals are measured by our patented device with lower sampling rate (cost efficiency purpose) therefore I do not recommend to use any other publicly available dataset containing partial discharge patterns (PD patterns). - also we deployed the metering devices on more than 20 different locations. This implies that the spectrum of noise and quality of PD's are so different from each other, that the correct and robust classification is a still ongoing problem (the main motivation of this competition). The comparison and broadening of our view is also considered as beneficial and necessary in our research.

\n\n

PD pattern is therefore the main star tonight and there is a lot of literature about this phenomenon. I would recommend to read some papers, during my dissertation I tried a lot of different feature extraction models, but those based on fundamentals worked the best. In general the imbalanced dataset is very natural because PD pattern implies some degradation or damage of the observed system which is happening (fortunately) less often than the states when the system is operating correctly.

\n\n

In our case, the measurements on the medium voltage overhead lines, PD pattern may look like this (pd_pattern.png). But because of a lot of various noise interference (overhead lines work as a huge antena grabbing all signals around), a lot of interpolated patterns may look similar (see samples.png).

\n\n

To use any kind of wavelet transformation is very reasonable, butterworth filter was helpful for me to suppress the sine shape, DWT to obtain its close approximation - sometimes it is disrupted, and denoising with feature extractions are the alchemy of this competition.

\n\n

I wish you a lot of fun and interesting knowledge obtained in this competition. I'm looking forward to see your approaches.

\n\n

BW,

\n\n

Tomas

", "messages": [ { "id": 445388, "postDate": "2018-12-26T11:02:45.987Z", "content": "

Dear All,

\n\n

I would like to welcome you here in this competition, my name is Tomas Vantuch and I'm a data scientist from ENET at VSB-TU Ostrava responsible for this party :) In a few following lines, I would like to give you some hints about the data and what was helpful for us during our research.

\n\n

As you may notice, the signal data comes from the real environment, not a lab, and they contain a lot of background noise. These signals are measured by our patented device with lower sampling rate (cost efficiency purpose) therefore I do not recommend to use any other publicly available dataset containing partial discharge patterns (PD patterns). - also we deployed the metering devices on more than 20 different locations. This implies that the spectrum of noise and quality of PD's are so different from each other, that the correct and robust classification is a still ongoing problem (the main motivation of this competition). The comparison and broadening of our view is also considered as beneficial and necessary in our research.

\n\n

PD pattern is therefore the main star tonight and there is a lot of literature about this phenomenon. I would recommend to read some papers, during my dissertation I tried a lot of different feature extraction models, but those based on fundamentals worked the best. In general the imbalanced dataset is very natural because PD pattern implies some degradation or damage of the observed system which is happening (fortunately) less often than the states when the system is operating correctly.

\n\n

In our case, the measurements on the medium voltage overhead lines, PD pattern may look like this (pd_pattern.png). But because of a lot of various noise interference (overhead lines work as a huge antena grabbing all signals around), a lot of interpolated patterns may look similar (see samples.png).

\n\n

To use any kind of wavelet transformation is very reasonable, butterworth filter was helpful for me to suppress the sine shape, DWT to obtain its close approximation - sometimes it is disrupted, and denoising with feature extractions are the alchemy of this competition.

\n\n

I wish you a lot of fun and interesting knowledge obtained in this competition. I'm looking forward to see your approaches.

\n\n

BW,

\n\n

Tomas

", "rawMarkdown": "Dear All,\n\nI would like to welcome you here in this competition, my name is Tomas Vantuch and I'm a data scientist from ENET at VSB-TU Ostrava responsible for this party :) In a few following lines, I would like to give you some hints about the data and what was helpful for us during our research.\n\nAs you may notice, the signal data comes from the real environment, not a lab, and they contain a lot of background noise. These signals are measured by our patented device with lower sampling rate (cost efficiency purpose) therefore I do not recommend to use any other publicly available dataset containing partial discharge patterns (PD patterns). - also we deployed the metering devices on more than 20 different locations. This implies that the spectrum of noise and quality of PD's are so different from each other, that the correct and robust classification is a still ongoing problem (the main motivation of this competition). The comparison and broadening of our view is also considered as beneficial and necessary in our research.\n\nPD pattern is therefore the main star tonight and there is a lot of literature about this phenomenon. I would recommend to read some papers, during my dissertation I tried a lot of different feature extraction models, but those based on fundamentals worked the best. In general the imbalanced dataset is very natural because PD pattern implies some degradation or damage of the observed system which is happening (fortunately) less often than the states when the system is operating correctly.\n\nIn our case, the measurements on the medium voltage overhead lines, PD pattern may look like this (pd_pattern.png). But because of a lot of various noise interference (overhead lines work as a huge antena grabbing all signals around), a lot of interpolated patterns may look similar (see samples.png).\n\nTo use any kind of wavelet transformation is very reasonable, butterworth filter was helpful for me to suppress the sine shape, DWT to obtain its close approximation - sometimes it is disrupted, and denoising with feature extractions are the alchemy of this competition.\n\nI wish you a lot of fun and interesting knowledge obtained in this competition. I'm looking forward to see your approaches.\n\nBW,\n\nTomas ", "votes": 85 }, { "id": 446402, "postDate": "2018-12-28T02:26:51.457Z", "content": "

Can you share this paper with us?

\n\n
\n

A Complex Classification Approach of Partial Discharges from Covered Conductors in Real Environment\n https://ieeexplore.ieee.org/document/7909221

\n
", "rawMarkdown": "Can you share this paper with us? \n> A Complex Classification Approach of Partial Discharges from Covered Conductors in Real Environment\nhttps://ieeexplore.ieee.org/document/7909221\n", "votes": 6, "replies": [ { "id": 446666, "postDate": "2018-12-28T13:04:54.400Z", "content": "

If the host is OK with it, I can share as a link to my google drive.

", "rawMarkdown": "If the host is OK with it, I can share as a link to my google drive. ", "votes": 3 }, { "id": 447236, "postDate": "2018-12-29T12:37:15.587Z", "content": "

here is the preprint version\nhttps://www.dropbox.com/s/2ltuvpw1b1ms2uu/A%20Complex%20Classification%20Approach%20of%20Partial%20Discharges%20from%20Covered%20Conductors%20in%20Real%20Environment%20%28preprint%29.pdf?dl=0

", "rawMarkdown": "here is the preprint version\nhttps://www.dropbox.com/s/2ltuvpw1b1ms2uu/A%20Complex%20Classification%20Approach%20of%20Partial%20Discharges%20from%20Covered%20Conductors%20in%20Real%20Environment%20%28preprint%29.pdf?dl=0", "votes": 18 }, { "id": 447242, "postDate": "2018-12-29T12:41:41.950Z", "content": "

Thank you!

", "rawMarkdown": "Thank you!", "votes": 1 }, { "id": 454990, "postDate": "2019-01-12T17:46:03.773Z", "content": "

... small trick for all future papers... https://en.wikipedia.org/wiki/Sci-Hub ... ;)

", "rawMarkdown": "... small trick for all future papers... https://en.wikipedia.org/wiki/Sci-Hub ... ;)", "votes": 3 } ] }, { "id": 456372, "postDate": "2019-01-15T17:25:48.057Z", "content": "

@tvantuch in the Feature Extraction section of your thesis, you describe the process of false peaks removal. How do you define a peak from the denoised signal? Does it need to be above a certain amplitude threshold?

", "rawMarkdown": "@tvantuch in the Feature Extraction section of your thesis, you describe the process of false peaks removal. How do you define a peak from the denoised signal? Does it need to be above a certain amplitude threshold?", "votes": 3, "replies": [ { "id": 456609, "postDate": "2019-01-16T07:19:34.043Z", "content": "

Exactly, but the threshold is one of the alchemy parts...you can adjust it as a static constant or variable based on a defined conditions - some low level thresholds will give you too much false peaks while too high threshold will remove the relevant information from the signal...

", "rawMarkdown": "Exactly, but the threshold is one of the alchemy parts...you can adjust it as a static constant or variable based on a defined conditions - some low level thresholds will give you too much false peaks while too high threshold will remove the relevant information from the signal...", "votes": 4 } ] }, { "id": 445441, "postDate": "2018-12-26T13:11:47.650Z", "content": "

Hi Tomas,

\n\n

You said also we deployed the metering devices on more than 20 different locations. Is there any relation between locations of measurements and how they are split in train / test / public LB / private LB? Thanks.

", "rawMarkdown": "Hi Tomas,\n\nYou said `also we deployed the metering devices on more than 20 different locations`. Is there any relation between locations of measurements and how they are split in train / test / public LB / private LB? Thanks.", "votes": 3, "replies": [ { "id": 445456, "postDate": "2018-12-26T13:50:44.313Z", "content": "

Hello,

\n\n

thank you for good question. That statement was supposed to describe the size of the problem but nothing more, so do not bother about that. Signals are properly mixed to possess similar distributions across train/test/etc.

", "rawMarkdown": "Hello,\n\nthank you for good question. That statement was supposed to describe the size of the problem but nothing more, so do not bother about that. Signals are properly mixed to possess similar distributions across train/test/etc.", "votes": 5 } ] }, { "id": 462482, "postDate": "2019-01-28T11:04:40.990Z", "content": "

@tvantuch in the feature generation part of your thesis you describe the generation of 28 attributes. 24 are the max/min/mean 8*(3 parts) derived from the extracted peaks. But what are the other 4? The standard deviation of peaks positions, width and amplitude for each of the 3 parts and for the complete signal would sum up to 12.\nThank you

", "rawMarkdown": "@tvantuch in the feature generation part of your thesis you describe the generation of 28 attributes. 24 are the max/min/mean 8*(3 parts) derived from the extracted peaks. But what are the other 4? The standard deviation of peaks positions, width and amplitude for each of the 3 parts and for the complete signal would sum up to 12.\nThank you\n", "votes": 4 }, { "id": 459537, "postDate": "2019-01-22T00:03:31.297Z", "content": "

@tvantuch In a public kernel I was trying to teach the method of auto-associative neural networks (auto-encoders) to remove common features and leave information carrying residuals (similar to wavelet decomposition). I noticed, besides the obvious first harmonic, there is a recurring seventh harmonic (power line frequency * 7). I do not believe it is an artifact of the rectangular wave sampling window, since I am not performing any convolutions (and hence multiplications in the frequency domain). I thought it might be some reflection due an impedance difference in the patented measurement device, but I'm only guessing. Any ideas?

\n\n

You can see what I'm talking about here: https://www.kaggle.com/pnussbaum/vsb-power-using-autoencoding-v09

", "rawMarkdown": "@tvantuch In a public kernel I was trying to teach the method of auto-associative neural networks (auto-encoders) to remove common features and leave information carrying residuals (similar to wavelet decomposition). I noticed, besides the obvious first harmonic, there is a recurring seventh harmonic (power line frequency * 7). I do not believe it is an artifact of the rectangular wave sampling window, since I am not performing any convolutions (and hence multiplications in the frequency domain). I thought it might be some reflection due an impedance difference in the patented measurement device, but I'm only guessing. Any ideas?\n\nYou can see what I'm talking about here: https://www.kaggle.com/pnussbaum/vsb-power-using-autoencoding-v09 ", "votes": 4, "replies": [ { "id": 459757, "postDate": "2019-01-22T10:03:52.690Z", "content": "

thank you very much for your input, I will resend it directly to our hardware specialist.

", "rawMarkdown": "thank you very much for your input, I will resend it directly to our hardware specialist.", "votes": 1 }, { "id": 1249919, "postDate": "2021-03-23T16:16:12.473Z", "content": "

Hi Tomas; I was following up on that \"seventh harmonic\" item above. Were you able to forward that on to the hardware specialist, and if so, can you forward my name and contact info to them for me? I wanted to follow up on what was found, and possibly co-author a short paper on the process of finding data issues through auto-encoders. My name is Paul Nussbaum, and he can reach out to me via my university email address pnussbaum@ecpi.edu. Thank you!

", "rawMarkdown": "Hi Tomas; I was following up on that \"seventh harmonic\" item above. Were you able to forward that on to the hardware specialist, and if so, can you forward my name and contact info to them for me? I wanted to follow up on what was found, and possibly co-author a short paper on the process of finding data issues through auto-encoders. My name is Paul Nussbaum, and he can reach out to me via my university email address pnussbaum@ecpi.edu. Thank you!" } ] }, { "id": 450452, "postDate": "2019-01-05T01:14:01.177Z", "content": "

@tvantuch How did you label these target? By human eye or something like threshold stuff?

", "rawMarkdown": "@tvantuch How did you label these target? By human eye or something like threshold stuff?", "votes": 4, "replies": [ { "id": 450752, "postDate": "2019-01-05T17:05:12.217Z", "content": "

As a follow-on question, are you certain that all of the non-targets are in fact non-PD? Is it possible that some of my false positives are really undetected cases of PD?

", "rawMarkdown": "As a follow-on question, are you certain that all of the non-targets are in fact non-PD? Is it possible that some of my false positives are really undetected cases of PD?", "votes": 4 }, { "id": 450823, "postDate": "2019-01-05T21:09:42.103Z", "content": "

Thank you both, your questions are really good. They reveal how difficult the detection is. Due to physical deployment in a real forested and hardly-accessible terrain, it is pretty difficult to estimate the exact time of fault occurrence. When an interconnection (just for example) between two phase-lines happens, PD patter is propagated as set of very small pulses - reason is that the heat is slowly burning the insulation - when it is burned, the interconnection if fully established and PD pattern is much much stronger. After this short phase, the tree branch is completely burned or (if branch is very heavy) one of the lines may be torn - therefore signal progress is again changed.

\n\n

All of the signals were labeled by an human expert, but as I said, it is very difficult to estimate when the fault starts - small PD pattern may be hidden in the signal's noise etc. - so you know, in all cases, there is a plenty of space for mistakes.

\n\n

Don't worry, you are already doing really good and we are very curious how the proper custom-feature-evaluation will affect feature extraction and training capabilities (I saw this in several discussion streams and I believe in that).

", "rawMarkdown": "Thank you both, your questions are really good. They reveal how difficult the detection is. Due to physical deployment in a real forested and hardly-accessible terrain, it is pretty difficult to estimate the exact time of fault occurrence. When an interconnection (just for example) between two phase-lines happens, PD patter is propagated as set of very small pulses - reason is that the heat is slowly burning the insulation - when it is burned, the interconnection if fully established and PD pattern is much much stronger. After this short phase, the tree branch is completely burned or (if branch is very heavy) one of the lines may be torn - therefore signal progress is again changed. \n\nAll of the signals were labeled by an human expert, but as I said, it is very difficult to estimate when the fault starts - small PD pattern may be hidden in the signal's noise etc. - so you know, in all cases, there is a plenty of space for mistakes.\n\nDon't worry, you are already doing really good and we are very curious how the proper custom-feature-evaluation will affect feature extraction and training capabilities (I saw this in several discussion streams and I believe in that).", "votes": 8 }, { "id": 452006, "postDate": "2019-01-08T02:52:58.840Z", "content": "

Thank you for answering.

\n\n

But I do want to know how did that human expert label these targets? That human expert labeled targets with own eye by plotting 3-phase? If so, we are predicting how does that human expert feel if he/she see the 3-phase?

", "rawMarkdown": "Thank you for answering. \n\nBut I do want to know how did that human expert label these targets? That human expert labeled targets with own eye by plotting 3-phase? If so, we are predicting how does that human expert feel if he/she see the 3-phase?", "votes": 4 }, { "id": 453460, "postDate": "2019-01-10T08:20:58.720Z", "content": "

You are correct, the plotting of all three phases is one of the key techniques but that's not all for sure. We are using our own classification algorithms and also every detected fault (as well as false-hit) was checked and maintained physically.

", "rawMarkdown": "You are correct, the plotting of all three phases is one of the key techniques but that's not all for sure. We are using our own classification algorithms and also every detected fault (as well as false-hit) was checked and maintained physically.", "votes": 6 }, { "id": 454079, "postDate": "2019-01-11T05:21:20.310Z", "content": "

Fair enough! Thank you!

", "rawMarkdown": "Fair enough! Thank you!", "votes": 1 } ] }, { "id": 460201, "postDate": "2019-01-23T07:24:06.300Z", "content": "

One of possible exlpanation of generation 7th harmonic is this work:\nhttps://www.industry.usa.siemens.com/drives/us/en/electric-drives/ac-drives/Documents/DRV-WP-drive_harmonics_in_power_systems.pdf page 21\nThis may mean harmonic is generated during the measument. Is it is true there is another question- why there is no 5th harmonic.

", "rawMarkdown": "One of possible exlpanation of generation 7th harmonic is this work:\nhttps://www.industry.usa.siemens.com/drives/us/en/electric-drives/ac-drives/Documents/DRV-WP-drive_harmonics_in_power_systems.pdf page 21\nThis may mean harmonic is generated during the measument. Is it is true there is another question- why there is no 5th harmonic.", "votes": 1, "replies": [ { "id": 460364, "postDate": "2019-01-23T14:50:44.007Z", "content": "

True - clipping or semiconductor rectification can cause harmonics, but I didn't see either in the signal captured (at least not obvious) and these are power companies who have already adjusted for impedance matching with their cabling and infrastructure. I'm suspecting the data collection device itself is introducing this, but that's just a wild guess. I haven't seen the device.

", "rawMarkdown": "True - clipping or semiconductor rectification can cause harmonics, but I didn't see either in the signal captured (at least not obvious) and these are power companies who have already adjusted for impedance matching with their cabling and infrastructure. I'm suspecting the data collection device itself is introducing this, but that's just a wild guess. I haven't seen the device." } ] }, { "id": 445789, "postDate": "2018-12-27T04:13:22.967Z", "content": "

Are those figures from your dissertation? I am trying to understand the terminology better. What is RPI?

", "rawMarkdown": "Are those figures from your dissertation? I am trying to understand the terminology better. What is RPI? ", "votes": 1, "replies": [ { "id": 446018, "postDate": "2018-12-27T10:59:57.813Z", "content": "

You can find the thesis with lots of interesting stuff here http://dspace.vsb.cz/bitstream/handle/10084/133114/VAN431_FEI_P1807_1801V001_2018.pdf

", "rawMarkdown": "You can find the thesis with lots of interesting stuff here http://dspace.vsb.cz/bitstream/handle/10084/133114/VAN431_FEI_P1807_1801V001_2018.pdf", "votes": 12 }, { "id": 446025, "postDate": "2018-12-27T11:13:17.350Z", "content": "

Thank you for sharing.\nI could find the terminologies RPI and DSI in the thesis as

\n\n

RPI : Random pulses interference (lightning, switching operations, corona)\nDSI : Discrete spectral interference (radio emissions)

", "rawMarkdown": "Thank you for sharing.\nI could find the terminologies RPI and DSI in the thesis as\n\nRPI : Random pulses interference (lightning, switching operations, corona)\nDSI : Discrete spectral interference (radio emissions)", "votes": 4 } ] }, { "id": 445445, "postDate": "2018-12-26T13:24:35.720Z", "content": "

Thank you for the hints!\nI have a question. What are the RPI and DSI in the pd_pattern.png ?

", "rawMarkdown": "Thank you for the hints!\nI have a question. What are the RPI and DSI in the pd_pattern.png ?", "votes": 2, "replies": [ { "id": 446021, "postDate": "2018-12-27T11:04:49.707Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1629353, "postDate": "2021-12-26T03:29:38.690Z", "content": "

Discrete Spectral Interferences (DSI) is a particular type of noise that affects PD measurement. RPI is the Random Pulse Interference which can be caused by lightning, switching operations or by electric discharges.

", "rawMarkdown": "Discrete Spectral Interferences (DSI) is a particular type of noise that affects PD measurement. RPI is the Random Pulse Interference which can be caused by lightning, switching operations or by electric discharges. " } ] }, { "id": 476782, "postDate": "2019-02-23T06:19:35.030Z", "content": "

@tvantuch \nIn order for all of us to understand the problem better, is it possible for you to explain strange phenomenon observed in

\n\n

https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/77600

\n\n

Or is it against the rule somehow?

", "rawMarkdown": "@tvantuch \nIn order for all of us to understand the problem better, is it possible for you to explain strange phenomenon observed in\n\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/77600 \n\nOr is it against the rule somehow?" }, { "id": 460638, "postDate": "2019-01-24T05:40:58.090Z", "content": "

@tvantuch Dear Tomas, in the paper you shared, apparently choosing relevant areas of data was quite helpful to improve prediction. In 400,000 samples, 1st quarter was ignored and the rest was used as relevant area, if my understanding is correct. It is unclear why 1st quarter is dumped. Can we apply similar technique to choose relevant area? Thanks!

", "rawMarkdown": "@tvantuch Dear Tomas, in the paper you shared, apparently choosing relevant areas of data was quite helpful to improve prediction. In 400,000 samples, 1st quarter was ignored and the rest was used as relevant area, if my understanding is correct. It is unclear why 1st quarter is dumped. Can we apply similar technique to choose relevant area? Thanks! ", "replies": [ { "id": 460739, "postDate": "2019-01-24T10:33:33.367Z", "content": "

Hi, this issue about relevant areas has been mentioned in discussion already - basically, if you want to detect on what phase the fault occurred than you have to focus on relevant areas. If PD pattern is visible on the rising side of sine curve, than it is highly likely that the fault is on the observed phase - if PD pattern is visible on other parts of the signal - than we are observing just noise propagation from fault phase (probably) - there are various physical conditions that may vary this statements but in general this is good to know.

", "rawMarkdown": "Hi, this issue about relevant areas has been mentioned in discussion already - basically, if you want to detect on what phase the fault occurred than you have to focus on relevant areas. If PD pattern is visible on the rising side of sine curve, than it is highly likely that the fault is on the observed phase - if PD pattern is visible on other parts of the signal - than we are observing just noise propagation from fault phase (probably) - there are various physical conditions that may vary this statements but in general this is good to know.", "votes": 7 } ] }, { "id": 459183, "postDate": "2019-01-21T11:08:00.637Z", "content": "

@tvantuch What's the difference between train and test? Different location, datetime or something like that? My adversarial validation's AUC is already 0.999.

", "rawMarkdown": "@tvantuch What's the difference between train and test? Different location, datetime or something like that? My adversarial validation's AUC is already 0.999.", "replies": [ { "id": 459204, "postDate": "2019-01-21T11:38:11.163Z", "content": "

This question I can easily answer as I don't know :) I published here, into Kaggle, a dataset having all signals and data scientist from Kaggle's team interviewed me and separated the data the way to reflect all fundamental issues while to make it most suitable for the competition.

\n\n

As you would understand, I don't have any clue how it is separated and I did no investigation to know that.

", "rawMarkdown": "This question I can easily answer as I don't know :) I published here, into Kaggle, a dataset having all signals and data scientist from Kaggle's team interviewed me and separated the data the way to reflect all fundamental issues while to make it most suitable for the competition.\n\nAs you would understand, I don't have any clue how it is separated and I did no investigation to know that.", "votes": 2 } ] }, { "id": 458275, "postDate": "2019-01-19T08:36:09.593Z", "content": "

I could not open files of .parquet type by apache viewer. Can anyone tell me another way to open it. Please help!!

", "rawMarkdown": "I could not open files of .parquet type by apache viewer. Can anyone tell me another way to open it. Please help!!" }, { "id": 455727, "postDate": "2019-01-14T12:52:03.383Z", "content": "

Hi Tomas. I was looking for your thesis and could not find so far. Is it publically available? Could you share the link? I feel it will be good for diving into domain knowledge :)

", "rawMarkdown": "Hi Tomas. I was looking for your thesis and could not find so far. Is it publically available? Could you share the link? I feel it will be good for diving into domain knowledge :)", "replies": [ { "id": 455733, "postDate": "2019-01-14T13:01:49.510Z", "content": "

Here you go @Blonde: http://dspace.vsb.cz/bitstream/handle/10084/133114/VAN431_FEI_P1807_1801V001_2018.pdf

", "rawMarkdown": "Here you go @Blonde: http://dspace.vsb.cz/bitstream/handle/10084/133114/VAN431_FEI_P1807_1801V001_2018.pdf", "votes": 2 } ] }, { "id": 455664, "postDate": "2019-01-14T10:26:26.770Z", "content": "

I tried to using pywt to get wavelet features, got an array(40000,), is this correct?

", "rawMarkdown": "I tried to using pywt to get wavelet features, got an array(40000,), is this correct?" }, { "id": 445788, "postDate": "2018-12-27T04:09:19.923Z", "content": "

Tomas, could you disclose the regions where your power lines are located? We can remove the local radio stations.

", "rawMarkdown": "Tomas, could you disclose the regions where your power lines are located? We can remove the local radio stations. " }, { "id": 445412, "postDate": "2018-12-26T12:02:06.970Z", "content": "

Thanks for the great information. I would add these stuff as features.

", "rawMarkdown": "Thanks for the great information. I would add these stuff as features." }, { "id": 497098, "postDate": "2019-03-23T00:31:49.273Z", "rawMarkdown": "", "isDeleted": true }, { "id": 465943, "postDate": "2019-02-04T11:16:59.103Z", "rawMarkdown": "", "isDeleted": true }, { "id": 457331, "postDate": "2019-01-17T08:47:28.410Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 541517, "postDate": "2019-06-02T15:35:45.493Z", "content": "

Thank you!

", "rawMarkdown": "Thank you!" } ], "comments": [ { "id": 446402, "author_name": "ONODERA", "author_url": "", "post_date": "2018-12-28T02:26:51.457000", "content": "

Can you share this paper with us?

\n\n
\n

A Complex Classification Approach of Partial Discharges from Covered Conductors in Real Environment\n https://ieeexplore.ieee.org/document/7909221

\n
", "votes": 6, "replies": [ { "id": 446666, "author_name": "JayNP", "author_url": "", "post_date": "2018-12-28T13:04:54.400000", "content": "

If the host is OK with it, I can share as a link to my google drive.

", "votes": 3, "replies": [] }, { "id": 447236, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2018-12-29T12:37:15.587000", "content": "

here is the preprint version\nhttps://www.dropbox.com/s/2ltuvpw1b1ms2uu/A%20Complex%20Classification%20Approach%20of%20Partial%20Discharges%20from%20Covered%20Conductors%20in%20Real%20Environment%20%28preprint%29.pdf?dl=0

", "votes": 18, "replies": [] }, { "id": 447242, "author_name": "ONODERA", "author_url": "", "post_date": "2018-12-29T12:41:41.950000", "content": "

Thank you!

", "votes": 1, "replies": [] }, { "id": 454990, "author_name": "dr stoop", "author_url": "", "post_date": "2019-01-12T17:46:03.773000", "content": "

... small trick for all future papers... https://en.wikipedia.org/wiki/Sci-Hub ... ;)

", "votes": 3, "replies": [] } ] }, { "id": 456372, "author_name": "Fernando Ramacciotti", "author_url": "", "post_date": "2019-01-15T17:25:48.057000", "content": "

@tvantuch in the Feature Extraction section of your thesis, you describe the process of false peaks removal. How do you define a peak from the denoised signal? Does it need to be above a certain amplitude threshold?

", "votes": 3, "replies": [ { "id": 456609, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2019-01-16T07:19:34.043000", "content": "

Exactly, but the threshold is one of the alchemy parts...you can adjust it as a static constant or variable based on a defined conditions - some low level thresholds will give you too much false peaks while too high threshold will remove the relevant information from the signal...

", "votes": 4, "replies": [] } ] }, { "id": 445441, "author_name": "Sangxia", "author_url": "", "post_date": "2018-12-26T13:11:47.650000", "content": "

Hi Tomas,

\n\n

You said also we deployed the metering devices on more than 20 different locations. Is there any relation between locations of measurements and how they are split in train / test / public LB / private LB? Thanks.

", "votes": 3, "replies": [ { "id": 445456, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2018-12-26T13:50:44.313000", "content": "

Hello,

\n\n

thank you for good question. That statement was supposed to describe the size of the problem but nothing more, so do not bother about that. Signals are properly mixed to possess similar distributions across train/test/etc.

", "votes": 5, "replies": [] } ] }, { "id": 462482, "author_name": "Manuel Carranza García", "author_url": "", "post_date": "2019-01-28T11:04:40.990000", "content": "

@tvantuch in the feature generation part of your thesis you describe the generation of 28 attributes. 24 are the max/min/mean 8*(3 parts) derived from the extracted peaks. But what are the other 4? The standard deviation of peaks positions, width and amplitude for each of the 3 parts and for the complete signal would sum up to 12.\nThank you

", "votes": 4, "replies": [] }, { "id": 459537, "author_name": "Paul Nussbaum, PhD", "author_url": "", "post_date": "2019-01-22T00:03:31.297000", "content": "

@tvantuch In a public kernel I was trying to teach the method of auto-associative neural networks (auto-encoders) to remove common features and leave information carrying residuals (similar to wavelet decomposition). I noticed, besides the obvious first harmonic, there is a recurring seventh harmonic (power line frequency * 7). I do not believe it is an artifact of the rectangular wave sampling window, since I am not performing any convolutions (and hence multiplications in the frequency domain). I thought it might be some reflection due an impedance difference in the patented measurement device, but I'm only guessing. Any ideas?

\n\n

You can see what I'm talking about here: https://www.kaggle.com/pnussbaum/vsb-power-using-autoencoding-v09

", "votes": 4, "replies": [ { "id": 459757, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2019-01-22T10:03:52.690000", "content": "

thank you very much for your input, I will resend it directly to our hardware specialist.

", "votes": 1, "replies": [] }, { "id": 1249919, "author_name": "Paul Nussbaum, PhD", "author_url": "", "post_date": "2021-03-23T16:16:12.473000", "content": "

Hi Tomas; I was following up on that \"seventh harmonic\" item above. Were you able to forward that on to the hardware specialist, and if so, can you forward my name and contact info to them for me? I wanted to follow up on what was found, and possibly co-author a short paper on the process of finding data issues through auto-encoders. My name is Paul Nussbaum, and he can reach out to me via my university email address pnussbaum@ecpi.edu. Thank you!

", "votes": 0, "replies": [] } ] }, { "id": 450452, "author_name": "ONODERA", "author_url": "", "post_date": "2019-01-05T01:14:01.177000", "content": "

@tvantuch How did you label these target? By human eye or something like threshold stuff?

", "votes": 4, "replies": [ { "id": 450752, "author_name": "Lazarp", "author_url": "", "post_date": "2019-01-05T17:05:12.217000", "content": "

As a follow-on question, are you certain that all of the non-targets are in fact non-PD? Is it possible that some of my false positives are really undetected cases of PD?

", "votes": 4, "replies": [] }, { "id": 450823, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2019-01-05T21:09:42.103000", "content": "

Thank you both, your questions are really good. They reveal how difficult the detection is. Due to physical deployment in a real forested and hardly-accessible terrain, it is pretty difficult to estimate the exact time of fault occurrence. When an interconnection (just for example) between two phase-lines happens, PD patter is propagated as set of very small pulses - reason is that the heat is slowly burning the insulation - when it is burned, the interconnection if fully established and PD pattern is much much stronger. After this short phase, the tree branch is completely burned or (if branch is very heavy) one of the lines may be torn - therefore signal progress is again changed.

\n\n

All of the signals were labeled by an human expert, but as I said, it is very difficult to estimate when the fault starts - small PD pattern may be hidden in the signal's noise etc. - so you know, in all cases, there is a plenty of space for mistakes.

\n\n

Don't worry, you are already doing really good and we are very curious how the proper custom-feature-evaluation will affect feature extraction and training capabilities (I saw this in several discussion streams and I believe in that).

", "votes": 8, "replies": [] }, { "id": 452006, "author_name": "ONODERA", "author_url": "", "post_date": "2019-01-08T02:52:58.840000", "content": "

Thank you for answering.

\n\n

But I do want to know how did that human expert label these targets? That human expert labeled targets with own eye by plotting 3-phase? If so, we are predicting how does that human expert feel if he/she see the 3-phase?

", "votes": 4, "replies": [] }, { "id": 453460, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2019-01-10T08:20:58.720000", "content": "

You are correct, the plotting of all three phases is one of the key techniques but that's not all for sure. We are using our own classification algorithms and also every detected fault (as well as false-hit) was checked and maintained physically.

", "votes": 6, "replies": [] }, { "id": 454079, "author_name": "ONODERA", "author_url": "", "post_date": "2019-01-11T05:21:20.310000", "content": "

Fair enough! Thank you!

", "votes": 1, "replies": [] } ] }, { "id": 460201, "author_name": "Kacper", "author_url": "", "post_date": "2019-01-23T07:24:06.300000", "content": "

One of possible exlpanation of generation 7th harmonic is this work:\nhttps://www.industry.usa.siemens.com/drives/us/en/electric-drives/ac-drives/Documents/DRV-WP-drive_harmonics_in_power_systems.pdf page 21\nThis may mean harmonic is generated during the measument. Is it is true there is another question- why there is no 5th harmonic.

", "votes": 1, "replies": [ { "id": 460364, "author_name": "Paul Nussbaum, PhD", "author_url": "", "post_date": "2019-01-23T14:50:44.007000", "content": "

True - clipping or semiconductor rectification can cause harmonics, but I didn't see either in the signal captured (at least not obvious) and these are power companies who have already adjusted for impedance matching with their cabling and infrastructure. I'm suspecting the data collection device itself is introducing this, but that's just a wild guess. I haven't seen the device.

", "votes": 0, "replies": [] } ] }, { "id": 445789, "author_name": "JayNP", "author_url": "", "post_date": "2018-12-27T04:13:22.967000", "content": "

Are those figures from your dissertation? I am trying to understand the terminology better. What is RPI?

", "votes": 1, "replies": [ { "id": 446018, "author_name": "Sangxia", "author_url": "", "post_date": "2018-12-27T10:59:57.813000", "content": "

You can find the thesis with lots of interesting stuff here http://dspace.vsb.cz/bitstream/handle/10084/133114/VAN431_FEI_P1807_1801V001_2018.pdf

", "votes": 12, "replies": [] }, { "id": 446025, "author_name": "Tom", "author_url": "", "post_date": "2018-12-27T11:13:17.350000", "content": "

Thank you for sharing.\nI could find the terminologies RPI and DSI in the thesis as

\n\n

RPI : Random pulses interference (lightning, switching operations, corona)\nDSI : Discrete spectral interference (radio emissions)

", "votes": 4, "replies": [] } ] }, { "id": 445445, "author_name": "Tom", "author_url": "", "post_date": "2018-12-26T13:24:35.720000", "content": "

Thank you for the hints!\nI have a question. What are the RPI and DSI in the pd_pattern.png ?

", "votes": 2, "replies": [ { "id": 446021, "author_name": "", "author_url": "", "post_date": "2018-12-27T11:04:49.707000", "content": "", "votes": 0, "replies": [] }, { "id": 1629353, "author_name": "Gulshan Savanth Dommu", "author_url": "", "post_date": "2021-12-26T03:29:38.690000", "content": "

Discrete Spectral Interferences (DSI) is a particular type of noise that affects PD measurement. RPI is the Random Pulse Interference which can be caused by lightning, switching operations or by electric discharges.

", "votes": 0, "replies": [] } ] }, { "id": 476782, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-02-23T06:19:35.030000", "content": "

@tvantuch \nIn order for all of us to understand the problem better, is it possible for you to explain strange phenomenon observed in

\n\n

https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/77600

\n\n

Or is it against the rule somehow?

", "votes": 0, "replies": [] }, { "id": 460638, "author_name": "Domain Knowledge", "author_url": "", "post_date": "2019-01-24T05:40:58.090000", "content": "

@tvantuch Dear Tomas, in the paper you shared, apparently choosing relevant areas of data was quite helpful to improve prediction. In 400,000 samples, 1st quarter was ignored and the rest was used as relevant area, if my understanding is correct. It is unclear why 1st quarter is dumped. Can we apply similar technique to choose relevant area? Thanks!

", "votes": 0, "replies": [ { "id": 460739, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2019-01-24T10:33:33.367000", "content": "

Hi, this issue about relevant areas has been mentioned in discussion already - basically, if you want to detect on what phase the fault occurred than you have to focus on relevant areas. If PD pattern is visible on the rising side of sine curve, than it is highly likely that the fault is on the observed phase - if PD pattern is visible on other parts of the signal - than we are observing just noise propagation from fault phase (probably) - there are various physical conditions that may vary this statements but in general this is good to know.

", "votes": 7, "replies": [] } ] }, { "id": 459183, "author_name": "ONODERA", "author_url": "", "post_date": "2019-01-21T11:08:00.637000", "content": "

@tvantuch What's the difference between train and test? Different location, datetime or something like that? My adversarial validation's AUC is already 0.999.

", "votes": 0, "replies": [ { "id": 459204, "author_name": "Tomas Vantuch", "author_url": "", "post_date": "2019-01-21T11:38:11.163000", "content": "

This question I can easily answer as I don't know :) I published here, into Kaggle, a dataset having all signals and data scientist from Kaggle's team interviewed me and separated the data the way to reflect all fundamental issues while to make it most suitable for the competition.

\n\n

As you would understand, I don't have any clue how it is separated and I did no investigation to know that.

", "votes": 2, "replies": [] } ] }, { "id": 458275, "author_name": "Harshit Patidar", "author_url": "", "post_date": "2019-01-19T08:36:09.593000", "content": "

I could not open files of .parquet type by apache viewer. Can anyone tell me another way to open it. Please help!!

", "votes": 0, "replies": [] }, { "id": 455727, "author_name": "Blonde", "author_url": "", "post_date": "2019-01-14T12:52:03.383000", "content": "

Hi Tomas. I was looking for your thesis and could not find so far. Is it publically available? Could you share the link? I feel it will be good for diving into domain knowledge :)

", "votes": 0, "replies": [ { "id": 455733, "author_name": "Max Halford", "author_url": "", "post_date": "2019-01-14T13:01:49.510000", "content": "

Here you go @Blonde: http://dspace.vsb.cz/bitstream/handle/10084/133114/VAN431_FEI_P1807_1801V001_2018.pdf

", "votes": 2, "replies": [] } ] }, { "id": 455664, "author_name": "simon", "author_url": "", "post_date": "2019-01-14T10:26:26.770000", "content": "

I tried to using pywt to get wavelet features, got an array(40000,), is this correct?

", "votes": 0, "replies": [] }, { "id": 445788, "author_name": "JayNP", "author_url": "", "post_date": "2018-12-27T04:09:19.923000", "content": "

Tomas, could you disclose the regions where your power lines are located? We can remove the local radio stations.

", "votes": 0, "replies": [] }, { "id": 445412, "author_name": "ONODERA", "author_url": "", "post_date": "2018-12-26T12:02:06.970000", "content": "

Thanks for the great information. I would add these stuff as features.

", "votes": 0, "replies": [] }, { "id": 497098, "author_name": "", "author_url": "", "post_date": "2019-03-23T00:31:49.273000", "content": "", "votes": 0, "replies": [] }, { "id": 465943, "author_name": "", "author_url": "", "post_date": "2019-02-04T11:16:59.103000", "content": "", "votes": 0, "replies": [] }, { "id": 457331, "author_name": "", "author_url": "", "post_date": "2019-01-17T08:47:28.410000", "content": "", "votes": 3, "replies": [] }, { "id": 541517, "author_name": "BNake", "author_url": "", "post_date": "2019-06-02T15:35:45.493000", "content": "

Thank you!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "445388": "Dear All,\n\nI would like to welcome you here in this competition, my name is Tomas Vantuch and I'm a data scientist from ENET at VSB-TU Ostrava responsible for this party :) In a few following lines, I would like to give you some hints about the data and what was helpful for us during our research.\n\nAs you may notice, the signal data comes from the real environment, not a lab, and they contain a lot of background noise. These signals are measured by our patented device with lower sampling rate (cost efficiency purpose) therefore I do not recommend to use any other publicly available dataset containing partial discharge patterns (PD patterns). - also we deployed the metering devices on more than 20 different locations. This implies that the spectrum of noise and quality of PD's are so different from each other, that the correct and robust classification is a still ongoing problem (the main motivation of this competition). The comparison and broadening of our view is also considered as beneficial and necessary in our research.\n\nPD pattern is therefore the main star tonight and there is a lot of literature about this phenomenon. I would recommend to read some papers, during my dissertation I tried a lot of different feature extraction models, but those based on fundamentals worked the best. In general the imbalanced dataset is very natural because PD pattern implies some degradation or damage of the observed system which is happening (fortunately) less often than the states when the system is operating correctly.\n\nIn our case, the measurements on the medium voltage overhead lines, PD pattern may look like this (pd_pattern.png). But because of a lot of various noise interference (overhead lines work as a huge antena grabbing all signals around), a lot of interpolated patterns may look similar (see samples.png).\n\nTo use any kind of wavelet transformation is very reasonable, butterworth filter was helpful for me to suppress the sine shape, DWT to obtain its close approximation - sometimes it is disrupted, and denoising with feature extractions are the alchemy of this competition.\n\nI wish you a lot of fun and interesting knowledge obtained in this competition. I'm looking forward to see your approaches.\n\nBW,\n\nTomas ", "446402": "Can you share this paper with us? \n> A Complex Classification Approach of Partial Discharges from Covered Conductors in Real Environment\nhttps://ieeexplore.ieee.org/document/7909221\n", "456372": "@tvantuch in the Feature Extraction section of your thesis, you describe the process of false peaks removal. How do you define a peak from the denoised signal? Does it need to be above a certain amplitude threshold?", "445441": "Hi Tomas,\n\nYou said `also we deployed the metering devices on more than 20 different locations`. Is there any relation between locations of measurements and how they are split in train / test / public LB / private LB? Thanks.", "462482": "@tvantuch in the feature generation part of your thesis you describe the generation of 28 attributes. 24 are the max/min/mean 8*(3 parts) derived from the extracted peaks. But what are the other 4? The standard deviation of peaks positions, width and amplitude for each of the 3 parts and for the complete signal would sum up to 12.\nThank you\n", "459537": "@tvantuch In a public kernel I was trying to teach the method of auto-associative neural networks (auto-encoders) to remove common features and leave information carrying residuals (similar to wavelet decomposition). I noticed, besides the obvious first harmonic, there is a recurring seventh harmonic (power line frequency * 7). I do not believe it is an artifact of the rectangular wave sampling window, since I am not performing any convolutions (and hence multiplications in the frequency domain). I thought it might be some reflection due an impedance difference in the patented measurement device, but I'm only guessing. Any ideas?\n\nYou can see what I'm talking about here: https://www.kaggle.com/pnussbaum/vsb-power-using-autoencoding-v09 ", "450452": "@tvantuch How did you label these target? By human eye or something like threshold stuff?", "460201": "One of possible exlpanation of generation 7th harmonic is this work:\nhttps://www.industry.usa.siemens.com/drives/us/en/electric-drives/ac-drives/Documents/DRV-WP-drive_harmonics_in_power_systems.pdf page 21\nThis may mean harmonic is generated during the measument. Is it is true there is another question- why there is no 5th harmonic.", "445789": "Are those figures from your dissertation? I am trying to understand the terminology better. What is RPI? ", "445445": "Thank you for the hints!\nI have a question. What are the RPI and DSI in the pd_pattern.png ?", "476782": "@tvantuch \nIn order for all of us to understand the problem better, is it possible for you to explain strange phenomenon observed in\n\nhttps://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/77600 \n\nOr is it against the rule somehow?", "460638": "@tvantuch Dear Tomas, in the paper you shared, apparently choosing relevant areas of data was quite helpful to improve prediction. In 400,000 samples, 1st quarter was ignored and the rest was used as relevant area, if my understanding is correct. It is unclear why 1st quarter is dumped. Can we apply similar technique to choose relevant area? Thanks! ", "459183": "@tvantuch What's the difference between train and test? Different location, datetime or something like that? My adversarial validation's AUC is already 0.999.", "458275": "I could not open files of .parquet type by apache viewer. Can anyone tell me another way to open it. Please help!!", "455727": "Hi Tomas. I was looking for your thesis and could not find so far. Is it publically available? Could you share the link? I feel it will be good for diving into domain knowledge :)", "455664": "I tried to using pywt to get wavelet features, got an array(40000,), is this correct?", "445788": "Tomas, could you disclose the regions where your power lines are located? We can remove the local radio stations. ", "445412": "Thanks for the great information. I would add these stuff as features.", "497098": "", "465943": "", "457331": "", "541517": "Thank you!" } }