{
  "id": 92564,
  "title": "Do you subtract the mean of signals before feature generation or not? why? ",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/92564",
  "author_name": "",
  "post_date": "2019-05-18T00:36:59.439230400Z",
  "votes": 4,
  "comment_count": 12,
  "views": 0,
  "content": "<p>We all know by now that the mean of train and test sets are different (~ 4.2 and 4.5, respectively). Would you subtract the mean of signals before feature generation or not? why? </p>\n\n<p>On one hand, it affects the features mostly percentile-based ones; hence, makes it harder for a model that's been trained on a dataset with a higher mean to predict a test set with a lower mean value. On the other hand, the difference in means is not artificially added and having it there might actually help a model somehow. Specially that train and test sets are part of the same experiment.</p>\n\n<p>I appreciate if you share your experience. Did you try subtracting means? did it help? do you expect it to help? </p>",
  "messages": [
    {
      "id": "532880",
      "postDate": "05/18/2019 00:36:59",
      "content": "<p>We all know by now that the mean of train and test sets are different (~ 4.2 and 4.5, respectively). Would you subtract the mean of signals before feature generation or not? why? </p>\n\n<p>On one hand, it affects the features mostly percentile-based ones; hence, makes it harder for a model that's been trained on a dataset with a higher mean to predict a test set with a lower mean value. On the other hand, the difference in means is not artificially added and having it there might actually help a model somehow. Specially that train and test sets are part of the same experiment.</p>\n\n<p>I appreciate if you share your experience. Did you try subtracting means? did it help? do you expect it to help? </p>",
      "rawMarkdown": "We all know by now that the mean of train and test sets are different (~ 4.2 and 4.5, respectively). Would you subtract the mean of signals before feature generation or not? why? \n\nOn one hand, it affects the features mostly percentile-based ones; hence, makes it harder for a model that's been trained on a dataset with a higher mean to predict a test set with a lower mean value. On the other hand, the difference in means is not artificially added and having it there might actually help a model somehow. Specially that train and test sets are part of the same experiment.\n\nI appreciate if you share your experience. Did you try subtracting means? did it help? do you expect it to help?",
      "votes": null
    },
    {
      "id": "532884",
      "postDate": "05/18/2019 00:59:10",
      "content": "<p>I just saw that it's been discussed in the discussions that the difference in means might be artifacts of recording instruments. It's not artificial or due to calibration, etc. as mentioned in my discussion; refer to this figure:\n<a href=\"https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/92367#532883\">https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/92367#532883</a>\n<img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/532883/13232/Untitled.jpg\" alt=\"\"></p>",
      "rawMarkdown": "I just saw that it's been discussed in the discussions that the difference in means might be artifacts of recording instruments. It's not artificial or due to calibration, etc. as mentioned in my discussion; refer to this figure:\nhttps://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/92367#532883\n![](https://storage.googleapis.com/kaggle-forum-message-attachments/532883/13232/Untitled.jpg)",
      "votes": null
    },
    {
      "id": "532906",
      "postDate": "05/18/2019 02:29:45",
      "content": "<p>this figure shows no acoustic data, it's shear stress. \nAnd yes, i'd definetly substract the mean in the acoustic data.</p>",
      "rawMarkdown": "this figure shows no acoustic data, it's shear stress. \nAnd yes, i'd definetly substract the mean in the acoustic data.",
      "votes": null
    },
    {
      "id": "532930",
      "postDate": "05/18/2019 04:13:01",
      "content": "<p><a href=\"/ilu000\">@ilu000</a> i know this is shear stress. The acoustic data is induced by this shear stress so I think since shear stresses reduce over time, we can expect acoustic data reduce over time as well leading to a lower mean value. </p>\n\n<p>The acoustic signal should be zero, when there is no shear stress acting on the device. I wish competition's sponsors could give us more information on this. But as of now, I cannot think of any other explanation for the lower mean value on test set. </p>\n\n<p>Did you find it better to subtract the mean? did it improve your scores?</p>",
      "rawMarkdown": "ilu000 i know this is shear stress. The acoustic data is induced by this shear stress so I think since shear stresses reduce over time, we can expect acoustic data reduce over time as well leading to a lower mean value. \n\nThe acoustic signal should be zero, when there is no shear stress acting on the device. I wish competition's sponsors could give us more information on this. But as of now, I cannot think of any other explanation for the lower mean value on test set. \n\n\nDid you find it better to subtract the mean? did it improve your scores?",
      "votes": null
    },
    {
      "id": "533003",
      "postDate": "05/18/2019 08:09:27",
      "content": "<p>I guess acoustic data are collected by sensors, not induced by shear stress. Shear stress is used to identify quakes, and thus time to failure. </p>",
      "rawMarkdown": "I guess acoustic data are collected by sensors, not induced by shear stress. Shear stress is used to identify quakes, and thus time to failure.",
      "votes": null
    },
    {
      "id": "533007",
      "postDate": "05/18/2019 08:18:05",
      "content": "<p>&gt; we can expect acoustic data reduce over time as well leading to a lower mean value. </p>\n\n<p>You are confusing signal amplitude and signal mean.  Amplitude is related to perceived intensity, and is roughly the difference between max and min.  If your hypothesis is right (and could well be AFAIK) then we would see amplitude decrease.</p>\n\n<p>Mean of  acoustic signal is 0.  What we observe in acoustic data (non null mean) is a bias created by an uncalibrated sensor.</p>\n\n<p>Note also that calibration would not remove the difference between train and test acoustic mean.  It would move them both close to 0.</p>",
      "rawMarkdown": "&gt; we can expect acoustic data reduce over time as well leading to a lower mean value. \n\nYou are confusing signal amplitude and signal mean.  Amplitude is related to perceived intensity, and is roughly the difference between max and min.  If your hypothesis is right (and could well be AFAIK) then we would see amplitude decrease.\n\nMean of  acoustic signal is 0.  What we observe in acoustic data (non null mean) is a bias created by an uncalibrated sensor.\n\nNote also that calibration would not remove the difference between train and test acoustic mean.  It would move them both close to 0.",
      "votes": null
    },
    {
      "id": "533011",
      "postDate": "05/18/2019 08:28:14",
      "content": "<p>The mean of acoustic signal is usually 0 at the source of vibration, but when the sensor moves away from the source, the mean seems to be meaningless. Maybe the mean of acoustic signal here represents a constant displacement or pressure on the sensor.</p>",
      "rawMarkdown": "The mean of acoustic signal is usually 0 at the source of vibration, but when the sensor moves away from the source, the mean seems to be meaningless. Maybe the mean of acoustic signal here represents a constant displacement or pressure on the sensor.",
      "votes": null
    },
    {
      "id": "533021",
      "postDate": "05/18/2019 09:16:47",
      "content": "<p>The mean is most likely artificial, but that doesn't mean you should remove it. Many kaggle competitions, as far as I know, were won by exploiting leaks. So I would suggest you evaluate the performance of your model and make a decision accordingly.\nI do think that removing artificial signals should theoretically enhance the performance of any model. However, this should have been done globally (before splitting train and test), or quake-wise by the organizers.</p>",
      "rawMarkdown": "The mean is most likely artificial, but that doesn't mean you should remove it. Many kaggle competitions, as far as I know, were won by exploiting leaks. So I would suggest you evaluate the performance of your model and make a decision accordingly.\nI do think that removing artificial signals should theoretically enhance the performance of any model. However, this should have been done globally (before splitting train and test), or quake-wise by the organizers.",
      "votes": null
    },
    {
      "id": "533146",
      "postDate": "05/18/2019 14:33:11",
      "content": "<p>I'm more convinced now that the mean should be zero. but if the man is artificial, how can we explain the difference? if it was due to calibration, I would expect it to be consistent within the whole dataset because they all come from one part of the same experiment (about 200 sec?) </p>\n\n<blockquote>\n  <p>this should have been done globally (before splitting train and test), or quake-wise by the organizers.</p>\n</blockquote>\n\n<p>I wish they were more responsive and we could hear from them on this. </p>",
      "rawMarkdown": "I'm more convinced now that the mean should be zero. but if the man is artificial, how can we explain the difference? if it was due to calibration, I would expect it to be consistent within the whole dataset because they all come from one part of the same experiment (about 200 sec?) \n\n&gt; this should have been done globally (before splitting train and test), or quake-wise by the organizers.\n\nI wish they were more responsive and we could hear from them on this.",
      "votes": null
    },
    {
      "id": "533147",
      "postDate": "05/18/2019 14:40:28",
      "content": "<p>Your explanations are reasonable. However, it is still not clear to me that if we assume it's a calibration issue, why is it not consistent throughout the whole dataset (train and test). They are all part of the same experiment with a fairly short duration (about 200 sec). Also, looking at the mean over the train set, I can tell, despite oscillations, the mean is going down over time in a systematic fashion. How can we explain this with your hypotheses? \n<img src=\"https://www.kaggleusercontent.com/kf/14309733/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..NjjAOBMxJal72S1ILuOABQ.8kvCviTCIYj0Xr3fmd2pIjg8HEmqyLNWgteDNdo1OhWS2BwEkxSVJtHxgrULieF8hubydl1iC0Z7rXVV_MqbFYd1YU8c70pF8QHr55G_jXXN4laCim7b6RTH3AyFye0vI1kQiOK6efEe5dIEoG0uSolv7VNYFNEgHXiDYPFoPD8.QB6UdUI1hjlUzGEGIHvdhA/__results___files/__results___34_0.png\" alt=\"\"></p>",
      "rawMarkdown": "Your explanations are reasonable. However, it is still not clear to me that if we assume it's a calibration issue, why is it not consistent throughout the whole dataset (train and test). They are all part of the same experiment with a fairly short duration (about 200 sec). Also, looking at the mean over the train set, I can tell, despite oscillations, the mean is going down over time in a systematic fashion. How can we explain this with your hypotheses? \n![](https://www.kaggleusercontent.com/kf/14309733/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..NjjAOBMxJal72S1ILuOABQ.8kvCviTCIYj0Xr3fmd2pIjg8HEmqyLNWgteDNdo1OhWS2BwEkxSVJtHxgrULieF8hubydl1iC0Z7rXVV_MqbFYd1YU8c70pF8QHr55G_jXXN4laCim7b6RTH3AyFye0vI1kQiOK6efEe5dIEoG0uSolv7VNYFNEgHXiDYPFoPD8.QB6UdUI1hjlUzGEGIHvdhA/__results___files/__results___34_0.png)",
      "votes": null
    },
    {
      "id": "533163",
      "postDate": "05/18/2019 15:07:28",
      "content": "<p>There is adrift of mean, sure.  But that's not what calibration is about. Calibration is about having 0 mean overall.  It they had subtracted 4 from acoustic data, then it would have been calibrated.</p>",
      "rawMarkdown": "There is adrift of mean, sure.  But that's not what calibration is about. Calibration is about having 0 mean overall.  It they had subtracted 4 from acoustic data, then it would have been calibrated.",
      "votes": null
    },
    {
      "id": "533169",
      "postDate": "05/18/2019 15:28:49",
      "content": "<blockquote>\n  <p>There is adrift of mean, sure.</p>\n</blockquote>\n\n<p>why?</p>",
      "rawMarkdown": "&gt; There is adrift of mean, sure.\n\nwhy?",
      "votes": null
    },
    {
      "id": "533280",
      "postDate": "05/18/2019 21:40:31",
      "content": "<p>How would I know?</p>",
      "rawMarkdown": "How would I know?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 532884,
      "author_name": "mhviraf",
      "author_url": "",
      "post_date": "05/18/2019 00:59:10",
      "content": "<p>I just saw that it's been discussed in the discussions that the difference in means might be artifacts of recording instruments. It's not artificial or due to calibration, etc. as mentioned in my discussion; refer to this figure:\n<a href=\"https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/92367#532883\">https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/92367#532883</a>\n<img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/532883/13232/Untitled.jpg\" alt=\"\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 532906,
          "author_name": "ilu000",
          "author_url": "",
          "post_date": "05/18/2019 02:29:45",
          "content": "<p>this figure shows no acoustic data, it's shear stress. \nAnd yes, i'd definetly substract the mean in the acoustic data.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 532930,
          "author_name": "mhviraf",
          "author_url": "",
          "post_date": "05/18/2019 04:13:01",
          "content": "<p><a href=\"/ilu000\">@ilu000</a> i know this is shear stress. The acoustic data is induced by this shear stress so I think since shear stresses reduce over time, we can expect acoustic data reduce over time as well leading to a lower mean value. </p>\n\n<p>The acoustic signal should be zero, when there is no shear stress acting on the device. I wish competition's sponsors could give us more information on this. But as of now, I cannot think of any other explanation for the lower mean value on test set. </p>\n\n<p>Did you find it better to subtract the mean? did it improve your scores?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533003,
          "author_name": "lucaskg",
          "author_url": "",
          "post_date": "05/18/2019 08:09:27",
          "content": "<p>I guess acoustic data are collected by sensors, not induced by shear stress. Shear stress is used to identify quakes, and thus time to failure. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533007,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/18/2019 08:18:05",
          "content": "<p>&gt; we can expect acoustic data reduce over time as well leading to a lower mean value. </p>\n\n<p>You are confusing signal amplitude and signal mean.  Amplitude is related to perceived intensity, and is roughly the difference between max and min.  If your hypothesis is right (and could well be AFAIK) then we would see amplitude decrease.</p>\n\n<p>Mean of  acoustic signal is 0.  What we observe in acoustic data (non null mean) is a bias created by an uncalibrated sensor.</p>\n\n<p>Note also that calibration would not remove the difference between train and test acoustic mean.  It would move them both close to 0.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533011,
          "author_name": "lucaskg",
          "author_url": "",
          "post_date": "05/18/2019 08:28:14",
          "content": "<p>The mean of acoustic signal is usually 0 at the source of vibration, but when the sensor moves away from the source, the mean seems to be meaningless. Maybe the mean of acoustic signal here represents a constant displacement or pressure on the sensor.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533147,
          "author_name": "mhviraf",
          "author_url": "",
          "post_date": "05/18/2019 14:40:28",
          "content": "<p>Your explanations are reasonable. However, it is still not clear to me that if we assume it's a calibration issue, why is it not consistent throughout the whole dataset (train and test). They are all part of the same experiment with a fairly short duration (about 200 sec). Also, looking at the mean over the train set, I can tell, despite oscillations, the mean is going down over time in a systematic fashion. How can we explain this with your hypotheses? \n<img src=\"https://www.kaggleusercontent.com/kf/14309733/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..NjjAOBMxJal72S1ILuOABQ.8kvCviTCIYj0Xr3fmd2pIjg8HEmqyLNWgteDNdo1OhWS2BwEkxSVJtHxgrULieF8hubydl1iC0Z7rXVV_MqbFYd1YU8c70pF8QHr55G_jXXN4laCim7b6RTH3AyFye0vI1kQiOK6efEe5dIEoG0uSolv7VNYFNEgHXiDYPFoPD8.QB6UdUI1hjlUzGEGIHvdhA/__results___files/__results___34_0.png\" alt=\"\"></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533163,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/18/2019 15:07:28",
          "content": "<p>There is adrift of mean, sure.  But that's not what calibration is about. Calibration is about having 0 mean overall.  It they had subtracted 4 from acoustic data, then it would have been calibrated.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533169,
          "author_name": "mhviraf",
          "author_url": "",
          "post_date": "05/18/2019 15:28:49",
          "content": "<blockquote>\n  <p>There is adrift of mean, sure.</p>\n</blockquote>\n\n<p>why?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 533280,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/18/2019 21:40:31",
          "content": "<p>How would I know?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 533021,
      "author_name": "amjad85",
      "author_url": "",
      "post_date": "05/18/2019 09:16:47",
      "content": "<p>The mean is most likely artificial, but that doesn't mean you should remove it. Many kaggle competitions, as far as I know, were won by exploiting leaks. So I would suggest you evaluate the performance of your model and make a decision accordingly.\nI do think that removing artificial signals should theoretically enhance the performance of any model. However, this should have been done globally (before splitting train and test), or quake-wise by the organizers.</p>",
      "votes": null,
      "replies": [
        {
          "id": 533146,
          "author_name": "mhviraf",
          "author_url": "",
          "post_date": "05/18/2019 14:33:11",
          "content": "<p>I'm more convinced now that the mean should be zero. but if the man is artificial, how can we explain the difference? if it was due to calibration, I would expect it to be consistent within the whole dataset because they all come from one part of the same experiment (about 200 sec?) </p>\n\n<blockquote>\n  <p>this should have been done globally (before splitting train and test), or quake-wise by the organizers.</p>\n</blockquote>\n\n<p>I wish they were more responsive and we could hear from them on this. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "532880": "We all know by now that the mean of train and test sets are different (~ 4.2 and 4.5, respectively). Would you subtract the mean of signals before feature generation or not? why? \n\nOn one hand, it affects the features mostly percentile-based ones; hence, makes it harder for a model that's been trained on a dataset with a higher mean to predict a test set with a lower mean value. On the other hand, the difference in means is not artificially added and having it there might actually help a model somehow. Specially that train and test sets are part of the same experiment.\n\nI appreciate if you share your experience. Did you try subtracting means? did it help? do you expect it to help?",
    "532884": "I just saw that it's been discussed in the discussions that the difference in means might be artifacts of recording instruments. It's not artificial or due to calibration, etc. as mentioned in my discussion; refer to this figure:\nhttps://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/92367#532883\n![](https://storage.googleapis.com/kaggle-forum-message-attachments/532883/13232/Untitled.jpg)",
    "532906": "this figure shows no acoustic data, it's shear stress. \nAnd yes, i'd definetly substract the mean in the acoustic data.",
    "532930": "ilu000 i know this is shear stress. The acoustic data is induced by this shear stress so I think since shear stresses reduce over time, we can expect acoustic data reduce over time as well leading to a lower mean value. \n\nThe acoustic signal should be zero, when there is no shear stress acting on the device. I wish competition's sponsors could give us more information on this. But as of now, I cannot think of any other explanation for the lower mean value on test set. \n\n\nDid you find it better to subtract the mean? did it improve your scores?",
    "533003": "I guess acoustic data are collected by sensors, not induced by shear stress. Shear stress is used to identify quakes, and thus time to failure.",
    "533007": "&gt; we can expect acoustic data reduce over time as well leading to a lower mean value. \n\nYou are confusing signal amplitude and signal mean.  Amplitude is related to perceived intensity, and is roughly the difference between max and min.  If your hypothesis is right (and could well be AFAIK) then we would see amplitude decrease.\n\nMean of  acoustic signal is 0.  What we observe in acoustic data (non null mean) is a bias created by an uncalibrated sensor.\n\nNote also that calibration would not remove the difference between train and test acoustic mean.  It would move them both close to 0.",
    "533011": "The mean of acoustic signal is usually 0 at the source of vibration, but when the sensor moves away from the source, the mean seems to be meaningless. Maybe the mean of acoustic signal here represents a constant displacement or pressure on the sensor.",
    "533021": "The mean is most likely artificial, but that doesn't mean you should remove it. Many kaggle competitions, as far as I know, were won by exploiting leaks. So I would suggest you evaluate the performance of your model and make a decision accordingly.\nI do think that removing artificial signals should theoretically enhance the performance of any model. However, this should have been done globally (before splitting train and test), or quake-wise by the organizers.",
    "533146": "I'm more convinced now that the mean should be zero. but if the man is artificial, how can we explain the difference? if it was due to calibration, I would expect it to be consistent within the whole dataset because they all come from one part of the same experiment (about 200 sec?) \n\n&gt; this should have been done globally (before splitting train and test), or quake-wise by the organizers.\n\nI wish they were more responsive and we could hear from them on this.",
    "533147": "Your explanations are reasonable. However, it is still not clear to me that if we assume it's a calibration issue, why is it not consistent throughout the whole dataset (train and test). They are all part of the same experiment with a fairly short duration (about 200 sec). Also, looking at the mean over the train set, I can tell, despite oscillations, the mean is going down over time in a systematic fashion. How can we explain this with your hypotheses? \n![](https://www.kaggleusercontent.com/kf/14309733/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..NjjAOBMxJal72S1ILuOABQ.8kvCviTCIYj0Xr3fmd2pIjg8HEmqyLNWgteDNdo1OhWS2BwEkxSVJtHxgrULieF8hubydl1iC0Z7rXVV_MqbFYd1YU8c70pF8QHr55G_jXXN4laCim7b6RTH3AyFye0vI1kQiOK6efEe5dIEoG0uSolv7VNYFNEgHXiDYPFoPD8.QB6UdUI1hjlUzGEGIHvdhA/__results___files/__results___34_0.png)",
    "533163": "There is adrift of mean, sure.  But that's not what calibration is about. Calibration is about having 0 mean overall.  It they had subtracted 4 from acoustic data, then it would have been calibrated.",
    "533169": "&gt; There is adrift of mean, sure.\n\nwhy?",
    "533280": "How would I know?"
  },
  "source": "meta"
}