After inspecting data, here is my understanding of how sensor works:\n1. It makes measurements in bursts.\n2. Each burst has 4096 measurements (separated by around 1 nanosecond).\n3. Measurement bursts happen at a frequency around 1KHz (1000 bursts per second).
\n\nConcerns:\n- Sometimes burst contains 4095 measurements. How to reproduce:\n\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.unique(np.diff(np.where(np.diff(s) < -1e-4)[0]), return_counts=True)\n\n- Above artefact happens at regular interval (once per 1280 bursts). How to reproduce:\n\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.diff(np.where(np.diff(np.where(np.diff(s) < -1e-4)[0]) == 4095))\n\n- We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions.
Request to admins/organisers\n- Clarify artefact reasons.\n- Supply delay between measurements in test data.
", "messages": [ { "id": 497368, "postDate": "2019-03-23T12:58:06.460Z", "content": "After inspecting data, here is my understanding of how sensor works:\n1. It makes measurements in bursts.\n2. Each burst has 4096 measurements (separated by around 1 nanosecond).\n3. Measurement bursts happen at a frequency around 1KHz (1000 bursts per second).
\n\nConcerns:\n- Sometimes burst contains 4095 measurements. How to reproduce:\n\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.unique(np.diff(np.where(np.diff(s) < -1e-4)[0]), return_counts=True)\n\n- Above artefact happens at regular interval (once per 1280 bursts). How to reproduce:\n\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.diff(np.where(np.diff(np.where(np.diff(s) < -1e-4)[0]) == 4095))\n\n- We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions.
Request to admins/organisers\n- Clarify artefact reasons.\n- Supply delay between measurements in test data.
", "rawMarkdown": "After inspecting data, here is my understanding of how sensor works:\n1. It makes measurements in bursts.\n2. Each burst has 4096 measurements (separated by around 1 nanosecond).\n3. Measurement bursts happen at a frequency around 1KHz (1000 bursts per second).\n\n**Concerns**:\n- Sometimes burst contains 4095 measurements. How to reproduce:\n```\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.unique(np.diff(np.where(np.diff(s) < -1e-4)[0]), return_counts=True)\n```\n- Above artefact happens at regular interval (once per 1280 bursts). How to reproduce:\n```\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.diff(np.where(np.diff(np.where(np.diff(s) < -1e-4)[0]) == 4095))\n```\n- We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions.\n\n**Request to admins/organisers**\n- Clarify artefact reasons.\n- Supply delay between measurements in test data.\n\n\n ", "votes": 24 }, { "id": 499490, "postDate": "2019-03-24T21:04:21.720Z", "content": "To add to this, I spent a day trying to use frequency analysis to identify the 'skips' in the time_to_failure readings. My conclusions were:
\n\nMaybe somebody more skilled than me can counter my findings, but this is what I've got so far.
", "rawMarkdown": "To add to this, I spent a day trying to use frequency analysis to identify the 'skips' in the time_to_failure readings. My conclusions were:\n\n* The signal frequency profile is all but identical both with the current TTF column, and with a manufactured column with regular intervals of 1.1e-9s\n* It was not possible to identify the rows where the 'skips' happen by frequency analysis alone\n\nMaybe somebody more skilled than me can counter my findings, but this is what I've got so far.", "votes": 3 }, { "id": 497561, "postDate": "2019-03-23T16:57:11.417Z", "content": "I agree and I follow
", "rawMarkdown": "I agree and I follow", "votes": 1 }, { "id": 1586863, "postDate": "2021-11-18T09:51:13.517Z", "content": "In simple terms, data quality indicates how trustworthy a set of data is and whether or not it is suitable for use in decision-making by a user. This attribute is frequently graded on a scale of one to ten.
\nBut, in practical terms, what is data quality?
\nData quality refers to how relevant data is for a certain purpose, as well as its completeness, correctness, timeliness (i.e., is it up to date? ), consistency, validity, and uniqueness.
\nData quality analysts are in charge of doing data quality evaluations, which entail evaluating and interpreting each quality data measure. The analyst then calculates an aggregate score for the data's overall quality and assigns a percentage grade to the company based on how accurate the data is.
\nTo put it another way, data quality refers to the quality of the data and how valuable it is for the task at hand. However, the phrase also refers to the activities of planning, implementing, and regulating the necessary quality management procedures and methodologies to ensure that the data is actionable and valuable to the data consumers.
Dimensions of Data Quality
\nData quality is divided into six basics, or core, characteristics. These are the parameters that analysts use to assess the data's feasibility and utility to those who require it.
Accuracy
\nThe data must reflect real-world items and occurrences and must correspond to true, real-world scenarios. To check the measure of correctness, analysts should employ verifiable sources, which are defined by how closely the data match the verified correct information sources.
Completeness
\nCompleteness assesses the data's capacity to correctly offer all of the mandatory values.
Consistency
\nThe uniformity of data as it flows between applications and networks, as well as as it comes from numerous sources, is referred to as data consistency. Consistency also implies that the same datasets kept in several locations should be identical and should not contradict. It's important to remember that even if data is consistent, it might still be incorrect.
Timeliness
\nData that is easily available whenever it is needed is referred to as timely data. This dimension also includes keeping data current; data should be updated in real-time to ensure that it is always available.
Uniqueness
\nThe term \"uniqueness\" refers to the absence of duplications or redundant information across all datasets. There are no duplicate records in the dataset.
Validity
\nData must be collected in accordance with the business rules and parameters established by the company. All dataset values should be within the proper range, and the data should follow the correct, accepted forms.
\"We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions\" - Andrii
\n\nTotally Agree. Why can't we have access to the full data collection regime as would the researcher setting up the experiment?
", "rawMarkdown": "\"We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions\" - Andrii\n\nTotally Agree. Why can't we have access to the full data collection regime as would the researcher setting up the experiment?" }, { "id": 519959, "postDate": "2019-04-19T21:28:43.323Z", "content": "The \"additional info\" thread, in the main comment, discusses the numbers. I posted a possible interpretation:
\n\n\n\n\n\n
\n- Within the bin, the sample rate really is 4 MHz giving delta t of 0.25 microseconds. This means that the times are mis-labelled within the bins.
\n- The 4096 samples give, for an entire bin, a delta t of 4096*0.25 microseconds = 1024 microseconds. This is compared with a result of 4.5 microseconds if the single sample delta t is 1.1 nanoseconds.
\n- The next bin appears to start at a time increment of 996/1095 microseconds from its predecessor, which is in the ballpark if 1. above is correct, where we expect 1024. If 1. is incorrect, it is way out of the ballpark.
\n- The 996/1095 microsecond \"gap\" is introduced only because of the mis-labelling. In reality, it varies between (-23.5/75.5) microseconds.
\n
\n This explanation does involve real gaps, but these gaps are now consistent with sample rate as quoted. The variations would be part of the inaccuracies built into the sensors.
The boundaries are not known but the size of the gaps are and correspond to roughly 50 microseconds or 200 samples. I think the differences would show up as a heavyside-like function in the Fourier domain with magnitude dependent on the data at the boundary. Low frequency stuff with a peak at zero frequency that is proportional to the size of the discontinuity. Perhaps looking for the peak may help localize it.
", "rawMarkdown": "The \"additional info\" thread, in the main comment, discusses the numbers. I posted a possible interpretation:\n> \n1. Within the bin, the sample rate really is 4 MHz giving delta t of 0.25 microseconds. This means that the times are mis-labelled within the bins.\n2. The 4096 samples give, for an entire bin, a delta t of 4096*0.25 microseconds = 1024 microseconds. This is compared with a result of 4.5 microseconds if the single sample delta t is 1.1 nanoseconds.\n3. The next bin appears to start at a time increment of 996/1095 microseconds from its predecessor, which is in the ballpark if 1. above is correct, where we expect 1024. If 1. is incorrect, it is way out of the ballpark.\n4. The 996/1095 microsecond \"gap\" is introduced only because of the mis-labelling. In reality, it varies between (-23.5/75.5) microseconds. \nThis explanation does involve real gaps, but these gaps are now consistent with sample rate as quoted. The variations would be part of the inaccuracies built into the sensors.\n\nThe boundaries are not known but the size of the gaps are and correspond to roughly 50 microseconds or 200 samples. I think the differences would show up as a heavyside-like function in the Fourier domain with magnitude dependent on the data at the boundary. Low frequency stuff with a peak at zero frequency that is proportional to the size of the discontinuity. Perhaps looking for the peak may help localize it.", "replies": [ { "id": 519974, "postDate": "2019-04-19T22:15:38.813Z", "content": "Are there even such sensors that can record at 1GHz?
", "rawMarkdown": "Are there even such sensors that can record at 1GHz?" }, { "id": 519998, "postDate": "2019-04-19T23:40:25.413Z", "content": "The spectrum does have data at high frequency. Nyquist is 2 MHz and I think there is data up to around half that. I don’t know about the sensor specs, though - it’s certainly a fair question.
", "rawMarkdown": "The spectrum does have data at high frequency. Nyquist is 2 MHz and I think there is data up to around half that. I don’t know about the sensor specs, though - it’s certainly a fair question." } ] }, { "id": 519754, "postDate": "2019-04-19T15:26:46.930Z", "content": "I fully agree with you, the 'burst' boundaries could have been provided.
", "rawMarkdown": "I fully agree with you, the 'burst' boundaries could have been provided." }, { "id": 504890, "postDate": "2019-04-01T09:49:06.970Z", "content": "Is there a commonly-accepted way of mitigating this issue?
", "rawMarkdown": "Is there a commonly-accepted way of mitigating this issue?", "replies": [ { "id": 504989, "postDate": "2019-04-01T12:32:01.170Z", "content": "\"It's just variance!\"
", "rawMarkdown": "\"It's just variance!\"" } ] }, { "id": 519927, "postDate": "2019-04-19T20:33:15.843Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 499490, "author_name": "RNA", "author_url": "", "post_date": "2019-03-24T21:04:21.720000", "content": "To add to this, I spent a day trying to use frequency analysis to identify the 'skips' in the time_to_failure readings. My conclusions were:
\n\nMaybe somebody more skilled than me can counter my findings, but this is what I've got so far.
", "votes": 3, "replies": [] }, { "id": 497561, "author_name": "bluetrain", "author_url": "", "post_date": "2019-03-23T16:57:11.417000", "content": "I agree and I follow
", "votes": 1, "replies": [] }, { "id": 1586863, "author_name": "Datalearning", "author_url": "", "post_date": "2021-11-18T09:51:13.517000", "content": "In simple terms, data quality indicates how trustworthy a set of data is and whether or not it is suitable for use in decision-making by a user. This attribute is frequently graded on a scale of one to ten.
\nBut, in practical terms, what is data quality?
\nData quality refers to how relevant data is for a certain purpose, as well as its completeness, correctness, timeliness (i.e., is it up to date? ), consistency, validity, and uniqueness.
\nData quality analysts are in charge of doing data quality evaluations, which entail evaluating and interpreting each quality data measure. The analyst then calculates an aggregate score for the data's overall quality and assigns a percentage grade to the company based on how accurate the data is.
\nTo put it another way, data quality refers to the quality of the data and how valuable it is for the task at hand. However, the phrase also refers to the activities of planning, implementing, and regulating the necessary quality management procedures and methodologies to ensure that the data is actionable and valuable to the data consumers.
Dimensions of Data Quality
\nData quality is divided into six basics, or core, characteristics. These are the parameters that analysts use to assess the data's feasibility and utility to those who require it.
Accuracy
\nThe data must reflect real-world items and occurrences and must correspond to true, real-world scenarios. To check the measure of correctness, analysts should employ verifiable sources, which are defined by how closely the data match the verified correct information sources.
Completeness
\nCompleteness assesses the data's capacity to correctly offer all of the mandatory values.
Consistency
\nThe uniformity of data as it flows between applications and networks, as well as as it comes from numerous sources, is referred to as data consistency. Consistency also implies that the same datasets kept in several locations should be identical and should not contradict. It's important to remember that even if data is consistent, it might still be incorrect.
Timeliness
\nData that is easily available whenever it is needed is referred to as timely data. This dimension also includes keeping data current; data should be updated in real-time to ensure that it is always available.
Uniqueness
\nThe term \"uniqueness\" refers to the absence of duplications or redundant information across all datasets. There are no duplicate records in the dataset.
Validity
\nData must be collected in accordance with the business rules and parameters established by the company. All dataset values should be within the proper range, and the data should follow the correct, accepted forms.
\"We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions\" - Andrii
\n\nTotally Agree. Why can't we have access to the full data collection regime as would the researcher setting up the experiment?
", "votes": 0, "replies": [] }, { "id": 519959, "author_name": "pete", "author_url": "", "post_date": "2019-04-19T21:28:43.323000", "content": "The \"additional info\" thread, in the main comment, discusses the numbers. I posted a possible interpretation:
\n\n\n\n\n\n
\n- Within the bin, the sample rate really is 4 MHz giving delta t of 0.25 microseconds. This means that the times are mis-labelled within the bins.
\n- The 4096 samples give, for an entire bin, a delta t of 4096*0.25 microseconds = 1024 microseconds. This is compared with a result of 4.5 microseconds if the single sample delta t is 1.1 nanoseconds.
\n- The next bin appears to start at a time increment of 996/1095 microseconds from its predecessor, which is in the ballpark if 1. above is correct, where we expect 1024. If 1. is incorrect, it is way out of the ballpark.
\n- The 996/1095 microsecond \"gap\" is introduced only because of the mis-labelling. In reality, it varies between (-23.5/75.5) microseconds.
\n
\n This explanation does involve real gaps, but these gaps are now consistent with sample rate as quoted. The variations would be part of the inaccuracies built into the sensors.
The boundaries are not known but the size of the gaps are and correspond to roughly 50 microseconds or 200 samples. I think the differences would show up as a heavyside-like function in the Fourier domain with magnitude dependent on the data at the boundary. Low frequency stuff with a peak at zero frequency that is proportional to the size of the discontinuity. Perhaps looking for the peak may help localize it.
", "votes": 0, "replies": [ { "id": 519974, "author_name": "redstr", "author_url": "", "post_date": "2019-04-19T22:15:38.813000", "content": "Are there even such sensors that can record at 1GHz?
", "votes": 0, "replies": [] }, { "id": 519998, "author_name": "pete", "author_url": "", "post_date": "2019-04-19T23:40:25.413000", "content": "The spectrum does have data at high frequency. Nyquist is 2 MHz and I think there is data up to around half that. I don’t know about the sensor specs, though - it’s certainly a fair question.
", "votes": 0, "replies": [] } ] }, { "id": 519754, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T15:26:46.930000", "content": "I fully agree with you, the 'burst' boundaries could have been provided.
", "votes": 0, "replies": [] }, { "id": 504890, "author_name": "Jack Harrison", "author_url": "", "post_date": "2019-04-01T09:49:06.970000", "content": "Is there a commonly-accepted way of mitigating this issue?
", "votes": 0, "replies": [ { "id": 504989, "author_name": "RNA", "author_url": "", "post_date": "2019-04-01T12:32:01.170000", "content": "\"It's just variance!\"
", "votes": 0, "replies": [] } ] }, { "id": 519927, "author_name": "", "author_url": "", "post_date": "2019-04-19T20:33:15.843000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "497368": "After inspecting data, here is my understanding of how sensor works:\n1. It makes measurements in bursts.\n2. Each burst has 4096 measurements (separated by around 1 nanosecond).\n3. Measurement bursts happen at a frequency around 1KHz (1000 bursts per second).\n\n**Concerns**:\n- Sometimes burst contains 4095 measurements. How to reproduce:\n```\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.unique(np.diff(np.where(np.diff(s) < -1e-4)[0]), return_counts=True)\n```\n- Above artefact happens at regular interval (once per 1280 bursts). How to reproduce:\n```\ns = train_df.iloc[245829585:307838917].time_to_failure.values\nnp.diff(np.where(np.diff(np.where(np.diff(s) < -1e-4)[0]) == 4095))\n```\n- We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions.\n\n**Request to admins/organisers**\n- Clarify artefact reasons.\n- Supply delay between measurements in test data.\n\n\n ", "499490": "To add to this, I spent a day trying to use frequency analysis to identify the 'skips' in the time_to_failure readings. My conclusions were:\n\n* The signal frequency profile is all but identical both with the current TTF column, and with a manufactured column with regular intervals of 1.1e-9s\n* It was not possible to identify the rows where the 'skips' happen by frequency analysis alone\n\nMaybe somebody more skilled than me can counter my findings, but this is what I've got so far.", "497561": "I agree and I follow", "1586863": "In simple terms, data quality indicates how trustworthy a set of data is and whether or not it is suitable for use in decision-making by a user. This attribute is frequently graded on a scale of one to ten.\n\n**But, in practical terms, what is data quality?**\n\n[Data quality](https://www.learnbay.co/data-science-course/) refers to how relevant data is for a certain purpose, as well as its completeness, correctness, timeliness (i.e., is it up to date? ), consistency, validity, and uniqueness.\nData quality analysts are in charge of doing data quality evaluations, which entail evaluating and interpreting each quality data measure. The analyst then calculates an aggregate score for the data's overall quality and assigns a percentage grade to the company based on how accurate the data is.\nTo put it another way, [data quality refers to the quality of the data ](https://www.learnbay.co/data-science-course/)and how valuable it is for the task at hand. However, the phrase also refers to the activities of planning, implementing, and regulating the necessary quality management procedures and methodologies to ensure that the data is actionable and valuable to the data consumers.\n\n**Dimensions of Data Quality**\nData quality is divided into six basics, or core, characteristics. These are the parameters that analysts use to assess the data's feasibility and utility to those who require it.\n\n**Accuracy**\nThe data must reflect real-world items and occurrences and must correspond to true, real-world scenarios. To check the measure of correctness, analysts should employ verifiable sources, which are defined by how closely the data match the verified correct information sources.\n\n**Completeness**\nCompleteness assesses the data's capacity to correctly offer all of the mandatory values.\n\n**Consistency**\nThe uniformity of data as it flows between applications and networks, as well as as it comes from numerous sources, is referred to as data consistency. Consistency also implies that the same datasets kept in several locations should be identical and should not contradict. It's important to remember that even if data is consistent, it might still be incorrect.\n\n**Timeliness**\nData that is easily available whenever it is needed is referred to as timely data. This dimension also includes keeping data current; data should be updated in real-time to ensure that it is always available.\n\n**Uniqueness**\nThe term \"uniqueness\" refers to the absence of duplications or redundant information across all datasets. There are no duplicate records in the dataset.\n\n**Validity**\nData must be collected in accordance with the business rules and parameters established by the company. All dataset values should be within the proper range, and the data should follow the correct, accepted forms.\n\n", "519977": "\"We don't know alignment of the test data. As such we cannot easily identify burst boundaries. Such data would be present while doing real-world prediction, so I don't see a reason to hide it. Absence of this data restricts participants in applying advanced signal processing techniques, thus limiting the quality of solutions\" - Andrii\n\nTotally Agree. Why can't we have access to the full data collection regime as would the researcher setting up the experiment?", "519959": "The \"additional info\" thread, in the main comment, discusses the numbers. I posted a possible interpretation:\n> \n1. Within the bin, the sample rate really is 4 MHz giving delta t of 0.25 microseconds. This means that the times are mis-labelled within the bins.\n2. The 4096 samples give, for an entire bin, a delta t of 4096*0.25 microseconds = 1024 microseconds. This is compared with a result of 4.5 microseconds if the single sample delta t is 1.1 nanoseconds.\n3. The next bin appears to start at a time increment of 996/1095 microseconds from its predecessor, which is in the ballpark if 1. above is correct, where we expect 1024. If 1. is incorrect, it is way out of the ballpark.\n4. The 996/1095 microsecond \"gap\" is introduced only because of the mis-labelling. In reality, it varies between (-23.5/75.5) microseconds. \nThis explanation does involve real gaps, but these gaps are now consistent with sample rate as quoted. The variations would be part of the inaccuracies built into the sensors.\n\nThe boundaries are not known but the size of the gaps are and correspond to roughly 50 microseconds or 200 samples. I think the differences would show up as a heavyside-like function in the Fourier domain with magnitude dependent on the data at the boundary. Low frequency stuff with a peak at zero frequency that is proportional to the size of the discontinuity. Perhaps looking for the peak may help localize it.", "519754": "I fully agree with you, the 'burst' boundaries could have been provided.", "504890": "Is there a commonly-accepted way of mitigating this issue?", "519927": "" } }