Good (part of the day you have now),
\n\nto normalise flux per object per band I try the following variants:
\n\n```train = train_df
\n\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.min())/(x.max()-x.min()))
\n\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.mean())/x.mean())
\n\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.median())/x.median())```
\n\nBut then I need to normalise flux_err the same way. And I just do not understand how to make it within few lines, without writing a long code, that will \"remember\" flux min,max and mean per object per band...
\n\nWhich variant do you think works better? I'll check them all if I have time and manage to make it all work, but maybe you can share your experience -- it will save time? Or maybe as the data are already somehow normalized it's better do not touch them, it's good enough?
\n\nThank you for your opinions
", "messages": [ { "id": "417776", "postDate": "11/08/2018 19:10:35", "content": "Good (part of the day you have now),
\n\nto normalise flux per object per band I try the following variants:
\n\n```train = train_df
\n\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.min())/(x.max()-x.min()))
\n\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.mean())/x.mean())
\n\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.median())/x.median())```
\n\nBut then I need to normalise flux_err the same way. And I just do not understand how to make it within few lines, without writing a long code, that will \"remember\" flux min,max and mean per object per band...
\n\nWhich variant do you think works better? I'll check them all if I have time and manage to make it all work, but maybe you can share your experience -- it will save time? Or maybe as the data are already somehow normalized it's better do not touch them, it's good enough?
\n\nThank you for your opinions
", "rawMarkdown": "Good (part of the day you have now),\n\nto normalise flux per object per band I try the following variants:\n\n```train = train_df\n# 1\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.min())/(x.max()-x.min()))\n#2\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.mean())/x.mean())\n#3\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.median())/x.median())```\n\n\nBut then I need to normalise flux_err the same way. And I just do not understand how to make it within few lines, without writing a long code, that will \"remember\" flux min,max and mean per object per band...\n\nWhich variant do you think works better? I'll check them all if I have time and manage to make it all work, but maybe you can share your experience -- it will save time? Or maybe as the data are already somehow normalized it's better do not touch them, it's good enough?\n\nThank you for your opinions", "votes": null }, { "id": "417783", "postDate": "11/08/2018 19:22:50", "content": "One way to achieve what you need is to create additional features to represent the flux per passband. For instance:
\n\n for pb in range(6):\n filter_p = (df.passband == pb) \n flux_pb = 'flux_%d' % pb\n df[flux_pb] = np.NaN\n df.loc[filter_p, flux_pb] = df.loc[filter_p, 'flux']\nthen you can use these 6 additional features as you see fit.
", "rawMarkdown": "One way to achieve what you need is to create additional features to represent the flux per passband. For instance:\n\n for pb in range(6):\n filter_p = (df.passband == pb) \n flux_pb = 'flux_%d' % pb\n df[flux_pb] = np.NaN\n df.loc[filter_p, flux_pb] = df.loc[filter_p, 'flux']\n\nthen you can use these 6 additional features as you see fit.", "votes": null }, { "id": "417793", "postDate": "11/08/2018 19:38:38", "content": "Thank you. It gives more flexibility further, indeed
", "rawMarkdown": "Thank you. It gives more flexibility further, indeed", "votes": null }, { "id": "417814", "postDate": "11/08/2018 20:16:26", "content": "> But then I need to normalise fluxerr the same way.
\n\nWhatever you divide flux by, divide fluxerr by the same thing. So either run the fluxerr transform first, or cache the divisor of your transform function then divide fluxerr by that groupwise.
", "rawMarkdown": "> But then I need to normalise fluxerr the same way.\n\nWhatever you divide flux by, divide fluxerr by the same thing. So either run the fluxerr transform first, or cache the divisor of your transform function then divide fluxerr by that groupwise.", "votes": null }, { "id": "418089", "postDate": "11/09/2018 09:13:40", "content": "I fail to see why you cannot transform flux through, say, a sigmoid and flux_err through, say, tanh. Both transforms are even bijective...
", "rawMarkdown": "I fail to see why you cannot transform flux through, say, a sigmoid and flux_err through, say, tanh. Both transforms are even bijective...", "votes": null }, { "id": "418279", "postDate": "11/09/2018 15:29:05", "content": "I just started on this a week ago. And from the initial prospective I see that some features use both flux and fluxerr (it may change in future once i manage to calculate and add better features). Therefore, I want to normalize them the same way. Solution to normalize first fluxerr is so obvious that I don't understand how I did not see that...
", "rawMarkdown": "I just started on this a week ago. And from the initial prospective I see that some features use both flux and fluxerr (it may change in future once i manage to calculate and add better features). Therefore, I want to normalize them the same way. Solution to normalize first fluxerr is so obvious that I don't understand how I did not see that...", "votes": null }, { "id": "418295", "postDate": "11/09/2018 15:53:42", "content": "Should be something like this:
\n\n\nflux_mean = train.groupby(['object_id', 'passband'])['flux'].transform('mean')
\nflux_std = train.groupby(['object_id', 'passband'])['flux'].transform('std')
\nflux_normalised = (train['flux'] - flux_mean) / flux_std\nerr_normalised = train['flux_err'] / flux_std\n
I have not tried all the normalisations yet, but I am personally leaning towards either per-band standard normalisation (with mean and std) or per-object standard normalisation. min-max is too sensitive to outliers (or lack of). Per-band generally works better for periodic objects where all bands tend to move in the same direction, but might not work very well for transient objects. Per-object preserves the relative difference between bands, so might be more useful for objects like supernovae where different bands can behave differently.
\n\nI'd also add the values you used in the transformation (mean and std for example) as new features, because you always put back what you discard from the data (well, unless they are proven not useful).
", "rawMarkdown": "Should be something like this:\n \n```\nflux_mean = train.groupby(['object_id', 'passband'])['flux'].transform('mean') \nflux_std = train.groupby(['object_id', 'passband'])['flux'].transform('std') \nflux_normalised = (train['flux'] - flux_mean) / flux_std\nerr_normalised = train['flux_err'] / flux_std\n```\n\nI have not tried all the normalisations yet, but I am personally leaning towards either per-band standard normalisation (with mean and std) or per-object standard normalisation. min-max is too sensitive to outliers (or lack of). Per-band generally works better for periodic objects where all bands tend to move in the same direction, but might not work very well for transient objects. Per-object preserves the relative difference between bands, so might be more useful for objects like supernovae where different bands can behave differently.\n\nI'd also add the values you used in the transformation (mean and std for example) as new features, because you always put back what you discard from the data (well, unless they are proven not useful).", "votes": null }, { "id": "418679", "postDate": "11/10/2018 12:08:17", "content": "Min-max doe not look good, indeed... I think I'll continuer adding features without normalization and then come back to it when i have a good features set. I think it makes sense to compare few norm options, but it's not critical for many features.
", "rawMarkdown": "Min-max doe not look good, indeed... I think I'll continuer adding features without normalization and then come back to it when i have a good features set. I think it makes sense to compare few norm options, but it's not critical for many features.", "votes": null }, { "id": "419857", "postDate": "11/12/2018 16:36:30", "content": "I tried the method 3 per-band a couple of days ago because I thought logically the 'median' should be able to represent the real 'background' flux, instead of the flux on a certain night in a certain year. I performed the same feature engineering on those new 'median-subtracted' flux features. However, both CV and LB dropped, which seems weird to me, actually.
", "rawMarkdown": "I tried the method 3 per-band a couple of days ago because I thought logically the 'median' should be able to represent the real 'background' flux, instead of the flux on a certain night in a certain year. I performed the same feature engineering on those new 'median-subtracted' flux features. However, both CV and LB dropped, which seems weird to me, actually.", "votes": null }, { "id": "419863", "postDate": "11/12/2018 16:51:03", "content": "@Ifcpeng17 thank you for sharing. 1. variant is not good as well. It look like norm will not help to improve here
", "rawMarkdown": "Ifcpeng17 thank you for sharing. 1. variant is not good as well. It look like norm will not help to improve here", "votes": null }, { "id": "422291", "postDate": "11/16/2018 02:51:23", "content": "If you work at Harvard Smithsonian Center for Astrophysics, then you z-normalize (Option 2) according to pg 19 of the link below AND you discard samples with two sigma measurement error.
\n\nI think there are some features (like period, time above mean, zero crossings) that are easier to analyze normalized and other features (like max amplitude) where normalization within the example destroys information.
\n\nPersonally, I'm using a normalized light set of curves for engineering some features while using raw parameters for others. Some of the libraries like AstroPy's LS algo will normalize while they extract curve parameters behind the scenes.
\n\n", "rawMarkdown": "If you work at Harvard Smithsonian Center for Astrophysics, then you z-normalize (Option 2) according to pg 19 of the link below AND you discard samples with two sigma measurement error.\n\nI think there are some features (like period, time above mean, zero crossings) that are easier to analyze normalized and other features (like max amplitude) where normalization within the example destroys information. \n\nPersonally, I'm using a normalized light set of curves for engineering some features while using raw parameters for others. Some of the libraries like AstroPy's LS algo will normalize while they extract curve parameters behind the scenes.\n\nhttps://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=2ahUKEwiJyb6L8tfeAhVCC6wKHRODBhUQFjACegQIBxAC&url=https%3A%2F%2Fhea-www.harvard.edu%2FAstroStat%2Fetc%2Fhead2011_pprotopapas_20110907.pptx&usg=AOvVaw3O7kLBqjAS3FHMxbl9X8Mj", "votes": null }, { "id": "422681", "postDate": "11/16/2018 16:12:11", "content": "thank you! very useful info. PS you have a lovely daughter :)
", "rawMarkdown": "thank you! very useful info. PS you have a lovely daughter :)", "votes": null }, { "id": "422701", "postDate": "11/16/2018 16:43:47", "content": "Why do you want to normalize in the first place? I am asking because it depends on the kind of model you want to train. Gbms like xgboost and lightgbm are not requiring normalized data. NNs and general linear models need normalized data.
", "rawMarkdown": "Why do you want to normalize in the first place? I am asking because it depends on the kind of model you want to train. Gbms like xgboost and lightgbm are not requiring normalized data. NNs and general linear models need normalized data.", "votes": null }, { "id": "422905", "postDate": "11/17/2018 02:36:48", "content": "I am new to ML/DL. I've read some papers and astronomers write they normalize... I am experimentalist, my approach is: try and see :). so I tried and I see exactly what you wrote :)
", "rawMarkdown": "I am new to ML/DL. I've read some papers and astronomers write they normalize... I am experimentalist, my approach is: try and see :). so I tried and I see exactly what you wrote :)", "votes": null } ], "comments": [ { "id": 417783, "author_name": "cpmpml", "author_url": "", "post_date": "11/08/2018 19:22:50", "content": "One way to achieve what you need is to create additional features to represent the flux per passband. For instance:
\n\n for pb in range(6):\n filter_p = (df.passband == pb) \n flux_pb = 'flux_%d' % pb\n df[flux_pb] = np.NaN\n df.loc[filter_p, flux_pb] = df.loc[filter_p, 'flux']\nthen you can use these 6 additional features as you see fit.
", "votes": null, "replies": [ { "id": 417793, "author_name": "blondinka", "author_url": "", "post_date": "11/08/2018 19:38:38", "content": "Thank you. It gives more flexibility further, indeed
", "votes": null, "replies": [] } ] }, { "id": 417814, "author_name": "authman", "author_url": "", "post_date": "11/08/2018 20:16:26", "content": "> But then I need to normalise fluxerr the same way.
\n\nWhatever you divide flux by, divide fluxerr by the same thing. So either run the fluxerr transform first, or cache the divisor of your transform function then divide fluxerr by that groupwise.
", "votes": null, "replies": [] }, { "id": 418089, "author_name": "glimmung", "author_url": "", "post_date": "11/09/2018 09:13:40", "content": "I fail to see why you cannot transform flux through, say, a sigmoid and flux_err through, say, tanh. Both transforms are even bijective...
", "votes": null, "replies": [ { "id": 418279, "author_name": "blondinka", "author_url": "", "post_date": "11/09/2018 15:29:05", "content": "I just started on this a week ago. And from the initial prospective I see that some features use both flux and fluxerr (it may change in future once i manage to calculate and add better features). Therefore, I want to normalize them the same way. Solution to normalize first fluxerr is so obvious that I don't understand how I did not see that...
", "votes": null, "replies": [] } ] }, { "id": 418295, "author_name": "mithrillion", "author_url": "", "post_date": "11/09/2018 15:53:42", "content": "Should be something like this:
\n\n\nflux_mean = train.groupby(['object_id', 'passband'])['flux'].transform('mean')
\nflux_std = train.groupby(['object_id', 'passband'])['flux'].transform('std')
\nflux_normalised = (train['flux'] - flux_mean) / flux_std\nerr_normalised = train['flux_err'] / flux_std\n
I have not tried all the normalisations yet, but I am personally leaning towards either per-band standard normalisation (with mean and std) or per-object standard normalisation. min-max is too sensitive to outliers (or lack of). Per-band generally works better for periodic objects where all bands tend to move in the same direction, but might not work very well for transient objects. Per-object preserves the relative difference between bands, so might be more useful for objects like supernovae where different bands can behave differently.
\n\nI'd also add the values you used in the transformation (mean and std for example) as new features, because you always put back what you discard from the data (well, unless they are proven not useful).
", "votes": null, "replies": [ { "id": 418679, "author_name": "blondinka", "author_url": "", "post_date": "11/10/2018 12:08:17", "content": "Min-max doe not look good, indeed... I think I'll continuer adding features without normalization and then come back to it when i have a good features set. I think it makes sense to compare few norm options, but it's not critical for many features.
", "votes": null, "replies": [] } ] }, { "id": 419857, "author_name": "lfcpeng17", "author_url": "", "post_date": "11/12/2018 16:36:30", "content": "I tried the method 3 per-band a couple of days ago because I thought logically the 'median' should be able to represent the real 'background' flux, instead of the flux on a certain night in a certain year. I performed the same feature engineering on those new 'median-subtracted' flux features. However, both CV and LB dropped, which seems weird to me, actually.
", "votes": null, "replies": [] }, { "id": 419863, "author_name": "blondinka", "author_url": "", "post_date": "11/12/2018 16:51:03", "content": "@Ifcpeng17 thank you for sharing. 1. variant is not good as well. It look like norm will not help to improve here
", "votes": null, "replies": [] }, { "id": 422291, "author_name": "mikeholcomb", "author_url": "", "post_date": "11/16/2018 02:51:23", "content": "If you work at Harvard Smithsonian Center for Astrophysics, then you z-normalize (Option 2) according to pg 19 of the link below AND you discard samples with two sigma measurement error.
\n\nI think there are some features (like period, time above mean, zero crossings) that are easier to analyze normalized and other features (like max amplitude) where normalization within the example destroys information.
\n\nPersonally, I'm using a normalized light set of curves for engineering some features while using raw parameters for others. Some of the libraries like AstroPy's LS algo will normalize while they extract curve parameters behind the scenes.
\n\n", "votes": null, "replies": [ { "id": 422681, "author_name": "blondinka", "author_url": "", "post_date": "11/16/2018 16:12:11", "content": "thank you! very useful info. PS you have a lovely daughter :)
", "votes": null, "replies": [] } ] }, { "id": 422701, "author_name": "cpmpml", "author_url": "", "post_date": "11/16/2018 16:43:47", "content": "Why do you want to normalize in the first place? I am asking because it depends on the kind of model you want to train. Gbms like xgboost and lightgbm are not requiring normalized data. NNs and general linear models need normalized data.
", "votes": null, "replies": [ { "id": 422905, "author_name": "blondinka", "author_url": "", "post_date": "11/17/2018 02:36:48", "content": "I am new to ML/DL. I've read some papers and astronomers write they normalize... I am experimentalist, my approach is: try and see :). so I tried and I see exactly what you wrote :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "417776": "Good (part of the day you have now),\n\nto normalise flux per object per band I try the following variants:\n\n```train = train_df\n# 1\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.min())/(x.max()-x.min()))\n#2\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.mean())/x.mean())\n#3\ntrain_df['flux'] = train.groupby(['object_id', 'passband'])['flux'].transform(lambda x: (x-x.median())/x.median())```\n\n\nBut then I need to normalise flux_err the same way. And I just do not understand how to make it within few lines, without writing a long code, that will \"remember\" flux min,max and mean per object per band...\n\nWhich variant do you think works better? I'll check them all if I have time and manage to make it all work, but maybe you can share your experience -- it will save time? Or maybe as the data are already somehow normalized it's better do not touch them, it's good enough?\n\nThank you for your opinions", "417783": "One way to achieve what you need is to create additional features to represent the flux per passband. For instance:\n\n for pb in range(6):\n filter_p = (df.passband == pb) \n flux_pb = 'flux_%d' % pb\n df[flux_pb] = np.NaN\n df.loc[filter_p, flux_pb] = df.loc[filter_p, 'flux']\n\nthen you can use these 6 additional features as you see fit.", "417793": "Thank you. It gives more flexibility further, indeed", "417814": "> But then I need to normalise fluxerr the same way.\n\nWhatever you divide flux by, divide fluxerr by the same thing. So either run the fluxerr transform first, or cache the divisor of your transform function then divide fluxerr by that groupwise.", "418089": "I fail to see why you cannot transform flux through, say, a sigmoid and flux_err through, say, tanh. Both transforms are even bijective...", "418279": "I just started on this a week ago. And from the initial prospective I see that some features use both flux and fluxerr (it may change in future once i manage to calculate and add better features). Therefore, I want to normalize them the same way. Solution to normalize first fluxerr is so obvious that I don't understand how I did not see that...", "418295": "Should be something like this:\n \n```\nflux_mean = train.groupby(['object_id', 'passband'])['flux'].transform('mean') \nflux_std = train.groupby(['object_id', 'passband'])['flux'].transform('std') \nflux_normalised = (train['flux'] - flux_mean) / flux_std\nerr_normalised = train['flux_err'] / flux_std\n```\n\nI have not tried all the normalisations yet, but I am personally leaning towards either per-band standard normalisation (with mean and std) or per-object standard normalisation. min-max is too sensitive to outliers (or lack of). Per-band generally works better for periodic objects where all bands tend to move in the same direction, but might not work very well for transient objects. Per-object preserves the relative difference between bands, so might be more useful for objects like supernovae where different bands can behave differently.\n\nI'd also add the values you used in the transformation (mean and std for example) as new features, because you always put back what you discard from the data (well, unless they are proven not useful).", "418679": "Min-max doe not look good, indeed... I think I'll continuer adding features without normalization and then come back to it when i have a good features set. I think it makes sense to compare few norm options, but it's not critical for many features.", "419857": "I tried the method 3 per-band a couple of days ago because I thought logically the 'median' should be able to represent the real 'background' flux, instead of the flux on a certain night in a certain year. I performed the same feature engineering on those new 'median-subtracted' flux features. However, both CV and LB dropped, which seems weird to me, actually.", "419863": "Ifcpeng17 thank you for sharing. 1. variant is not good as well. It look like norm will not help to improve here", "422291": "If you work at Harvard Smithsonian Center for Astrophysics, then you z-normalize (Option 2) according to pg 19 of the link below AND you discard samples with two sigma measurement error.\n\nI think there are some features (like period, time above mean, zero crossings) that are easier to analyze normalized and other features (like max amplitude) where normalization within the example destroys information. \n\nPersonally, I'm using a normalized light set of curves for engineering some features while using raw parameters for others. Some of the libraries like AstroPy's LS algo will normalize while they extract curve parameters behind the scenes.\n\nhttps://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=2ahUKEwiJyb6L8tfeAhVCC6wKHRODBhUQFjACegQIBxAC&url=https%3A%2F%2Fhea-www.harvard.edu%2FAstroStat%2Fetc%2Fhead2011_pprotopapas_20110907.pptx&usg=AOvVaw3O7kLBqjAS3FHMxbl9X8Mj", "422681": "thank you! very useful info. PS you have a lovely daughter :)", "422701": "Why do you want to normalize in the first place? I am asking because it depends on the kind of model you want to train. Gbms like xgboost and lightgbm are not requiring normalized data. NNs and general linear models need normalized data.", "422905": "I am new to ML/DL. I've read some papers and astronomers write they normalize... I am experimentalist, my approach is: try and see :). so I tried and I see exactly what you wrote :)" }, "source": "meta" }