{
  "id": 71871,
  "title": "Distance Modulus",
  "url": "/competitions/PLAsTiCC-2018/discussion/71871",
  "author_name": "CPMP",
  "post_date": "2018-11-17T16:04:29.339000",
  "votes": 27,
  "comment_count": 22,
  "views": 0,
  "content": "<p>The distmod feature in the dataset is what astronomers call the distance modulus.  I find this wikipedia article useful to astronomy noobs like me: <a href=\"https://en.wikipedia.org/wiki/Distance_modulus\">https://en.wikipedia.org/wiki/Distance_modulus</a></p>",
  "messages": [
    {
      "id": 423698,
      "postDate": "2018-11-18T22:19:00.117Z",
      "content": "<p>The distance modulus is a pretty important concept, so I thought that I would do a little writeup about how it applies to this competition.</p>\n\n<p>The distance modulus is a measure of how much fainter a distant object appears to be compared to what it would look like if it were close by. Nearby, this follows the \"one over r-squared\" law where if you move the object twice as far away it appears to be four times fainter . At distances on the scale of the universe, the expansion of the universe modifies the one over r-squared law. The distance modulus captures those modifications.</p>\n\n<p>When you measure the brightness of an object with a camera, you record what we call the \"flux\" which is proportional to how many photons the telescope sees from a distant object in a given time. The brightness of astronomical objects varies over huge scales, so we typically work in \"magnitudes\" which is a log-transform of the flux. The \"magnitude\" is calculated as -2.5*log10(measured flux) with a zeropoint that will depend on the telescope and passband.</p>\n\n<p>Now if you subtract the distance modulus from the measured magnitude, you will get the absolute magnitude of the object in question. The absolute magnitude is a measure of how bright the object would appear to be if you observed it at a fixed distance away. This can be used to tell objects apart. For example, at their brightest, Type Ia supernovae, have an absolute magnitude of around -19 mag with a scatter of ~0.5 mag. If you observe something that peaks at -22 mag, that probably isn't a Type Ia supernova. There is a list of peak absolute magnitudes for different supernovae here: <a href=\"https://en.wikipedia.org/wiki/Supernova\">https://en.wikipedia.org/wiki/Supernova</a>. Note that we don't really know the zeropoint of the magnitudes that we are given, but all that really matters is magnitude differences for classification.</p>\n\n<p>Finally, the distance moduli that are provided in this competition come directly from the photo-zs. These photo-zs are noisy (compare to the spec-zs which are the \"truth\"), so the distance moduli are noisy too. You can calculate a distance modulus for a given redshift using the astropy python package assuming the value of cosmological parameters in our universe. If you look at the provided data, it appears that the calculation that we were given assumed a Hubble Constant of 70 km/s/Mpc, a flat universe, a matter density (Ωm) of 0.3 and a CMB temperature of 2.725. The following code will calculate the distance modulus column from the photo-z column in the data:</p>\n\n<pre><code>from astropy.cosmology import FlatLambdaCDM\ncosmo = FlatLambdaCDM(H0=70, Om0=0.3, Tcmb0=2.725)\ndistance_modulus = cosmo.distmod(redshift)\n</code></pre>",
      "rawMarkdown": "The distance modulus is a pretty important concept, so I thought that I would do a little writeup about how it applies to this competition.\n\nThe distance modulus is a measure of how much fainter a distant object appears to be compared to what it would look like if it were close by. Nearby, this follows the \"one over r-squared\" law where if you move the object twice as far away it appears to be four times fainter . At distances on the scale of the universe, the expansion of the universe modifies the one over r-squared law. The distance modulus captures those modifications.\n\nWhen you measure the brightness of an object with a camera, you record what we call the \"flux\" which is proportional to how many photons the telescope sees from a distant object in a given time. The brightness of astronomical objects varies over huge scales, so we typically work in \"magnitudes\" which is a log-transform of the flux. The \"magnitude\" is calculated as -2.5*log10(measured flux) with a zeropoint that will depend on the telescope and passband.\n\nNow if you subtract the distance modulus from the measured magnitude, you will get the absolute magnitude of the object in question. The absolute magnitude is a measure of how bright the object would appear to be if you observed it at a fixed distance away. This can be used to tell objects apart. For example, at their brightest, Type Ia supernovae, have an absolute magnitude of around -19 mag with a scatter of ~0.5 mag. If you observe something that peaks at -22 mag, that probably isn't a Type Ia supernova. There is a list of peak absolute magnitudes for different supernovae here: https://en.wikipedia.org/wiki/Supernova. Note that we don't really know the zeropoint of the magnitudes that we are given, but all that really matters is magnitude differences for classification.\n\nFinally, the distance moduli that are provided in this competition come directly from the photo-zs. These photo-zs are noisy (compare to the spec-zs which are the \"truth\"), so the distance moduli are noisy too. You can calculate a distance modulus for a given redshift using the astropy python package assuming the value of cosmological parameters in our universe. If you look at the provided data, it appears that the calculation that we were given assumed a Hubble Constant of 70 km/s/Mpc, a flat universe, a matter density (Ωm) of 0.3 and a CMB temperature of 2.725. The following code will calculate the distance modulus column from the photo-z column in the data:\n\n    from astropy.cosmology import FlatLambdaCDM\n    cosmo = FlatLambdaCDM(H0=70, Om0=0.3, Tcmb0=2.725)\n    distance_modulus = cosmo.distmod(redshift)",
      "votes": 39,
      "replies": [
        {
          "id": 423708,
          "postDate": "2018-11-18T22:47:03.753Z",
          "content": "<p><a href=\"/kyleboone\">@kyleboone</a>\nThank you Kyle, </p>\n\n<p>as you are in a good mood for sharing valuable information, maybe you would advance on the negative flux, sometimes even the baseline is negative: \n1. do you shift it all up to have a zero baseline\n2. do you delete detected negative flux ? \nThe preprocessing varies in papers...</p>",
          "rawMarkdown": "@kyleboone\nThank you Kyle, \n\nas you are in a good mood for sharing valuable information, maybe you would advance on the negative flux, sometimes even the baseline is negative: \n1. do you shift it all up to have a zero baseline\n2. do you delete detected negative flux ? \nThe preprocessing varies in papers...",
          "votes": 6
        },
        {
          "id": 423796,
          "postDate": "2018-11-19T03:57:35.553Z",
          "content": "<p>The negative flux is a bit tricky. For something like a supernova, you have a lot of light where there was nothing before so it makes sense to have a zero baseline. For the objects with negative flux, I think that they are something like an eclipsing binary (see eg: <a href=\"https://imagine.gsfc.nasa.gov/educators/hera_college/binary-model.html\">https://imagine.gsfc.nasa.gov/educators/hera_college/binary-model.html</a> ). What is going on there is that you have two stars orbiting each other, and when one goes in front of the other it appears to be dimmer because the star in front is blocking out some of the light from the one behind. To convert to magnitudes, you need to know the baseline amount of light at the stars' location which isn't provided to us in this competition.</p>\n\n<p>The negative fluxes do contain a lot of information, and unfortunately there isn't really much of an option other than using their values directly. If you want to restrict the range of values, you could use something like a two-sided log transformation (see eg: <a href=\"https://matplotlib.org/api/_as_gen/matplotlib.colors.SymLogNorm.html\">https://matplotlib.org/api/_as_gen/matplotlib.colors.SymLogNorm.html</a> ) which preserves the magnitude scale for large positive values and keeps the negative fluxes bounded.</p>",
          "rawMarkdown": "The negative flux is a bit tricky. For something like a supernova, you have a lot of light where there was nothing before so it makes sense to have a zero baseline. For the objects with negative flux, I think that they are something like an eclipsing binary (see eg: https://imagine.gsfc.nasa.gov/educators/hera_college/binary-model.html ). What is going on there is that you have two stars orbiting each other, and when one goes in front of the other it appears to be dimmer because the star in front is blocking out some of the light from the one behind. To convert to magnitudes, you need to know the baseline amount of light at the stars' location which isn't provided to us in this competition.\n\nThe negative fluxes do contain a lot of information, and unfortunately there isn't really much of an option other than using their values directly. If you want to restrict the range of values, you could use something like a two-sided log transformation (see eg: https://matplotlib.org/api/_as_gen/matplotlib.colors.SymLogNorm.html ) which preserves the magnitude scale for large positive values and keeps the negative fluxes bounded.",
          "votes": 9
        },
        {
          "id": 423870,
          "postDate": "2018-11-19T07:02:53.457Z",
          "content": "<p>Kyle, a little thing: your links are broken because they include the trailing punctuation, a closing bracket in the above comment, a dot in another one.  Make sure you type a white space after every url you enter in this forum ;)</p>\n\n<p>This will make your very informative posts even better!</p>",
          "rawMarkdown": "Kyle, a little thing: your links are broken because they include the trailing punctuation, a closing bracket in the above comment, a dot in another one.  Make sure you type a white space after every url you enter in this forum ;)\n\nThis will make your very informative posts even better!"
        },
        {
          "id": 423909,
          "postDate": "2018-11-19T08:52:52.977Z",
          "content": "<p>Thank you !</p>",
          "rawMarkdown": "Thank you !"
        },
        {
          "id": 424125,
          "postDate": "2018-11-19T16:03:37.613Z",
          "content": "<p>Thanks, I fixed the links. They should work now.</p>",
          "rawMarkdown": "Thanks, I fixed the links. They should work now.",
          "votes": 1
        },
        {
          "id": 424328,
          "postDate": "2018-11-19T22:47:19.483Z",
          "content": "<p>Thank you Kyle, </p>\n\n<p>one other thing with ambiguity is weather to consider detected flux. Some consider only detected, while some not, does it depend on the kind of object we are looking at, or on the kind of features we want to extract ?</p>",
          "rawMarkdown": "Thank you Kyle, \n\none other thing with ambiguity is weather to consider detected flux. Some consider only detected, while some not, does it depend on the kind of object we are looking at, or on the kind of features we want to extract ?",
          "votes": 1
        },
        {
          "id": 424385,
          "postDate": "2018-11-20T02:58:12.267Z",
          "content": "<p>Why not consider both?</p>",
          "rawMarkdown": "Why not consider both?"
        },
        {
          "id": 424392,
          "postDate": "2018-11-20T03:13:30.757Z",
          "content": "<p>The detected flag seems to just be something like a flag that the total signal-to-noise of all observations that night is &gt; 5. It is useful for visualization purposes, but I don't think that it provides any information that you can't get from flux and fluxerr.</p>\n\n<p>With that in mind, you probably want to use more of the data than just the detected points. The lack of flux at a given time can give you a lot of information. For example, you might only have observations of the decline of a supernova lightcurve, but observations with zero flux a few weeks before can be used to constrain the time of maximum light.</p>",
          "rawMarkdown": "The detected flag seems to just be something like a flag that the total signal-to-noise of all observations that night is &gt; 5. It is useful for visualization purposes, but I don't think that it provides any information that you can't get from flux and fluxerr.\n\nWith that in mind, you probably want to use more of the data than just the detected points. The lack of flux at a given time can give you a lot of information. For example, you might only have observations of the decline of a supernova lightcurve, but observations with zero flux a few weeks before can be used to constrain the time of maximum light.",
          "votes": 4
        },
        {
          "id": 424580,
          "postDate": "2018-11-20T11:02:52.197Z",
          "content": "<p>Thank you</p>",
          "rawMarkdown": "Thank you"
        },
        {
          "id": 424597,
          "postDate": "2018-11-20T11:47:55.003Z",
          "content": "<p>@Blonde</p>\n\n<p>Everything is simulated here...</p>\n\n<p>We have to create our model classification.</p>\n\n<p>Read the data note, page 5 : \"<em>The question we address in this challenge is: how well can we classify astronomical transients and variables from a simulated light curve data set designed to mimic the data\nfrom LSST?</em>\"</p>\n\n<p>Hope this helps.</p>",
          "rawMarkdown": "@Blonde\n\nEverything is simulated here...\n\nWe have to create our model classification.\n\nRead the data note, page 5 : \"*The question we address in this challenge is: how well can we classify astronomical transients and variables from a simulated light curve data set designed to mimic the data\nfrom LSST?*\"\n\nHope this helps."
        },
        {
          "id": 424725,
          "postDate": "2018-11-20T15:21:45.137Z",
          "content": "<p>I know everything is simulated :)... it's not what I meant, but as I did not formulate it clear enough I deleted </p>",
          "rawMarkdown": "I know everything is simulated :)... it's not what I meant, but as I did not formulate it clear enough I deleted "
        },
        {
          "id": 424784,
          "postDate": "2018-11-20T16:45:04.183Z",
          "content": "<p>@Blonde,  you asked a fair question: does the flux_err contain useful information.  The best way to know is to try with it and without using it IMHO ;)</p>\n\n<p>One possibility that it contains info comes from flux correction.  given filters don't have the same throughput, flux have been corrected for us according to organizers (I haven't the link handy).  Therefore fux_err for filters with lower throughput is larger.  Is that useful info?  I don't know.  </p>",
          "rawMarkdown": "@Blonde,  you asked a fair question: does the flux_err contain useful information.  The best way to know is to try with it and without using it IMHO ;)\n\nOne possibility that it contains info comes from flux correction.  given filters don't have the same throughput, flux have been corrected for us according to organizers (I haven't the link handy).  Therefore fux_err for filters with lower throughput is larger.  Is that useful info?  I don't know.  ",
          "votes": 1
        },
        {
          "id": 424834,
          "postDate": "2018-11-20T18:25:56.970Z",
          "content": "<p><a href=\"/cpmpml\">@cpmpml</a> \nYes, I already got the answer in the data note...  BTW when you refer I am <a href=\"/blondinka\">@blondinka</a> (Blonde is just a display name)</p>",
          "rawMarkdown": "@cpmpml \nYes, I already got the answer in the data note...  BTW when you refer I am @blondinka (Blonde is just a display name)\n"
        }
      ]
    },
    {
      "id": 423148,
      "postDate": "2018-11-17T16:04:29.340Z",
      "content": "<p>The distmod feature in the dataset is what astronomers call the distance modulus.  I find this wikipedia article useful to astronomy noobs like me: <a href=\"https://en.wikipedia.org/wiki/Distance_modulus\">https://en.wikipedia.org/wiki/Distance_modulus</a></p>",
      "rawMarkdown": "The distmod feature in the dataset is what astronomers call the distance modulus.  I find this wikipedia article useful to astronomy noobs like me: https://en.wikipedia.org/wiki/Distance_modulus",
      "votes": 27
    },
    {
      "id": 423264,
      "postDate": "2018-11-17T20:14:10.527Z",
      "content": "<p>it is 0.999 correlated with log(hostgal_photoz)</p>",
      "rawMarkdown": "it is 0.999 correlated with log(hostgal_photoz)",
      "votes": 6
    },
    {
      "id": 423832,
      "postDate": "2018-11-19T05:50:01.673Z",
      "content": "<p>Hi, I assume the redshift and distance modulus changes over time for an object, but in the data, these values are at metadata level. So they are not varying for an object over time  or assumed to be constant ?</p>",
      "rawMarkdown": "Hi, I assume the redshift and distance modulus changes over time for an object, but in the data, these values are at metadata level. So they are not varying for an object over time  or assumed to be constant ?",
      "votes": 3,
      "replies": [
        {
          "id": 423840,
          "postDate": "2018-11-19T06:05:08.690Z",
          "content": "<p>The redshift/distance modulus doesn't change. The extragalactic objects can be billions of light years away from us, and any motion that they have is negligible.</p>\n\n<p>On the other hand, the brightness of the object does change. When we talk about the \"absolute magnitude\" or \"apparent magnitude\" of things like supernovae, we typically mean the magnitude when the supernova is at its brightest.</p>",
          "rawMarkdown": "The redshift/distance modulus doesn't change. The extragalactic objects can be billions of light years away from us, and any motion that they have is negligible.\n\nOn the other hand, the brightness of the object does change. When we talk about the \"absolute magnitude\" or \"apparent magnitude\" of things like supernovae, we typically mean the magnitude when the supernova is at its brightest.",
          "votes": 3
        },
        {
          "id": 423851,
          "postDate": "2018-11-19T06:24:57.883Z",
          "content": "<p>Kyle, thanks for your time, so the impact of motion of extra galactic objects is negligible for classification task.</p>",
          "rawMarkdown": "Kyle, thanks for your time, so the impact of motion of extra galactic objects is negligible for classification task."
        }
      ]
    },
    {
      "id": 424130,
      "postDate": "2018-11-19T16:06:45.517Z",
      "content": "<p>good</p>",
      "rawMarkdown": "good\n",
      "votes": 1
    },
    {
      "id": 423640,
      "postDate": "2018-11-18T19:18:03.320Z",
      "content": "<p>so the astronomer on LB really motivated you to go deeper into domain knowledge :-)))</p>",
      "rawMarkdown": "so the astronomer on LB really motivated you to go deeper into domain knowledge :-)))",
      "replies": [
        {
          "id": 423651,
          "postDate": "2018-11-18T19:42:36.700Z",
          "content": "<p>I didn't wait for him ;) Whatever the domain, if you want effective machine learning, the more you understand the context, the better.</p>",
          "rawMarkdown": "I didn't wait for him ;) Whatever the domain, if you want effective machine learning, the more you understand the context, the better.",
          "votes": 4
        }
      ]
    },
    {
      "id": 423182,
      "postDate": "2018-11-17T17:20:10.713Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 423698,
      "author_name": "Kyle Boone",
      "author_url": "",
      "post_date": "2018-11-18T22:19:00.117000",
      "content": "<p>The distance modulus is a pretty important concept, so I thought that I would do a little writeup about how it applies to this competition.</p>\n\n<p>The distance modulus is a measure of how much fainter a distant object appears to be compared to what it would look like if it were close by. Nearby, this follows the \"one over r-squared\" law where if you move the object twice as far away it appears to be four times fainter . At distances on the scale of the universe, the expansion of the universe modifies the one over r-squared law. The distance modulus captures those modifications.</p>\n\n<p>When you measure the brightness of an object with a camera, you record what we call the \"flux\" which is proportional to how many photons the telescope sees from a distant object in a given time. The brightness of astronomical objects varies over huge scales, so we typically work in \"magnitudes\" which is a log-transform of the flux. The \"magnitude\" is calculated as -2.5*log10(measured flux) with a zeropoint that will depend on the telescope and passband.</p>\n\n<p>Now if you subtract the distance modulus from the measured magnitude, you will get the absolute magnitude of the object in question. The absolute magnitude is a measure of how bright the object would appear to be if you observed it at a fixed distance away. This can be used to tell objects apart. For example, at their brightest, Type Ia supernovae, have an absolute magnitude of around -19 mag with a scatter of ~0.5 mag. If you observe something that peaks at -22 mag, that probably isn't a Type Ia supernova. There is a list of peak absolute magnitudes for different supernovae here: <a href=\"https://en.wikipedia.org/wiki/Supernova\">https://en.wikipedia.org/wiki/Supernova</a>. Note that we don't really know the zeropoint of the magnitudes that we are given, but all that really matters is magnitude differences for classification.</p>\n\n<p>Finally, the distance moduli that are provided in this competition come directly from the photo-zs. These photo-zs are noisy (compare to the spec-zs which are the \"truth\"), so the distance moduli are noisy too. You can calculate a distance modulus for a given redshift using the astropy python package assuming the value of cosmological parameters in our universe. If you look at the provided data, it appears that the calculation that we were given assumed a Hubble Constant of 70 km/s/Mpc, a flat universe, a matter density (Ωm) of 0.3 and a CMB temperature of 2.725. The following code will calculate the distance modulus column from the photo-z column in the data:</p>\n\n<pre><code>from astropy.cosmology import FlatLambdaCDM\ncosmo = FlatLambdaCDM(H0=70, Om0=0.3, Tcmb0=2.725)\ndistance_modulus = cosmo.distmod(redshift)\n</code></pre>",
      "votes": 39,
      "replies": [
        {
          "id": 423708,
          "author_name": "Blonde",
          "author_url": "",
          "post_date": "2018-11-18T22:47:03.753000",
          "content": "<p><a href=\"/kyleboone\">@kyleboone</a>\nThank you Kyle, </p>\n\n<p>as you are in a good mood for sharing valuable information, maybe you would advance on the negative flux, sometimes even the baseline is negative: \n1. do you shift it all up to have a zero baseline\n2. do you delete detected negative flux ? \nThe preprocessing varies in papers...</p>",
          "votes": 6,
          "replies": []
        },
        {
          "id": 423796,
          "author_name": "Kyle Boone",
          "author_url": "",
          "post_date": "2018-11-19T03:57:35.553000",
          "content": "<p>The negative flux is a bit tricky. For something like a supernova, you have a lot of light where there was nothing before so it makes sense to have a zero baseline. For the objects with negative flux, I think that they are something like an eclipsing binary (see eg: <a href=\"https://imagine.gsfc.nasa.gov/educators/hera_college/binary-model.html\">https://imagine.gsfc.nasa.gov/educators/hera_college/binary-model.html</a> ). What is going on there is that you have two stars orbiting each other, and when one goes in front of the other it appears to be dimmer because the star in front is blocking out some of the light from the one behind. To convert to magnitudes, you need to know the baseline amount of light at the stars' location which isn't provided to us in this competition.</p>\n\n<p>The negative fluxes do contain a lot of information, and unfortunately there isn't really much of an option other than using their values directly. If you want to restrict the range of values, you could use something like a two-sided log transformation (see eg: <a href=\"https://matplotlib.org/api/_as_gen/matplotlib.colors.SymLogNorm.html\">https://matplotlib.org/api/_as_gen/matplotlib.colors.SymLogNorm.html</a> ) which preserves the magnitude scale for large positive values and keeps the negative fluxes bounded.</p>",
          "votes": 9,
          "replies": []
        },
        {
          "id": 423870,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2018-11-19T07:02:53.457000",
          "content": "<p>Kyle, a little thing: your links are broken because they include the trailing punctuation, a closing bracket in the above comment, a dot in another one.  Make sure you type a white space after every url you enter in this forum ;)</p>\n\n<p>This will make your very informative posts even better!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 423909,
          "author_name": "Blonde",
          "author_url": "",
          "post_date": "2018-11-19T08:52:52.977000",
          "content": "<p>Thank you !</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 424125,
          "author_name": "Kyle Boone",
          "author_url": "",
          "post_date": "2018-11-19T16:03:37.613000",
          "content": "<p>Thanks, I fixed the links. They should work now.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 424328,
          "author_name": "Blonde",
          "author_url": "",
          "post_date": "2018-11-19T22:47:19.483000",
          "content": "<p>Thank you Kyle, </p>\n\n<p>one other thing with ambiguity is weather to consider detected flux. Some consider only detected, while some not, does it depend on the kind of object we are looking at, or on the kind of features we want to extract ?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 424385,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2018-11-20T02:58:12.267000",
          "content": "<p>Why not consider both?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 424392,
          "author_name": "Kyle Boone",
          "author_url": "",
          "post_date": "2018-11-20T03:13:30.757000",
          "content": "<p>The detected flag seems to just be something like a flag that the total signal-to-noise of all observations that night is &gt; 5. It is useful for visualization purposes, but I don't think that it provides any information that you can't get from flux and fluxerr.</p>\n\n<p>With that in mind, you probably want to use more of the data than just the detected points. The lack of flux at a given time can give you a lot of information. For example, you might only have observations of the decline of a supernova lightcurve, but observations with zero flux a few weeks before can be used to constrain the time of maximum light.</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 424580,
          "author_name": "Blonde",
          "author_url": "",
          "post_date": "2018-11-20T11:02:52.197000",
          "content": "<p>Thank you</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 424597,
          "author_name": "mezoganet",
          "author_url": "",
          "post_date": "2018-11-20T11:47:55.003000",
          "content": "<p>@Blonde</p>\n\n<p>Everything is simulated here...</p>\n\n<p>We have to create our model classification.</p>\n\n<p>Read the data note, page 5 : \"<em>The question we address in this challenge is: how well can we classify astronomical transients and variables from a simulated light curve data set designed to mimic the data\nfrom LSST?</em>\"</p>\n\n<p>Hope this helps.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 424725,
          "author_name": "Blonde",
          "author_url": "",
          "post_date": "2018-11-20T15:21:45.137000",
          "content": "<p>I know everything is simulated :)... it's not what I meant, but as I did not formulate it clear enough I deleted </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 424784,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2018-11-20T16:45:04.183000",
          "content": "<p>@Blonde,  you asked a fair question: does the flux_err contain useful information.  The best way to know is to try with it and without using it IMHO ;)</p>\n\n<p>One possibility that it contains info comes from flux correction.  given filters don't have the same throughput, flux have been corrected for us according to organizers (I haven't the link handy).  Therefore fux_err for filters with lower throughput is larger.  Is that useful info?  I don't know.  </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 424834,
          "author_name": "Blonde",
          "author_url": "",
          "post_date": "2018-11-20T18:25:56.970000",
          "content": "<p><a href=\"/cpmpml\">@cpmpml</a> \nYes, I already got the answer in the data note...  BTW when you refer I am <a href=\"/blondinka\">@blondinka</a> (Blonde is just a display name)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 423264,
      "author_name": "Ahmet Erdem",
      "author_url": "",
      "post_date": "2018-11-17T20:14:10.527000",
      "content": "<p>it is 0.999 correlated with log(hostgal_photoz)</p>",
      "votes": 6,
      "replies": []
    },
    {
      "id": 423832,
      "author_name": "Chinta",
      "author_url": "",
      "post_date": "2018-11-19T05:50:01.673000",
      "content": "<p>Hi, I assume the redshift and distance modulus changes over time for an object, but in the data, these values are at metadata level. So they are not varying for an object over time  or assumed to be constant ?</p>",
      "votes": 3,
      "replies": [
        {
          "id": 423840,
          "author_name": "Kyle Boone",
          "author_url": "",
          "post_date": "2018-11-19T06:05:08.690000",
          "content": "<p>The redshift/distance modulus doesn't change. The extragalactic objects can be billions of light years away from us, and any motion that they have is negligible.</p>\n\n<p>On the other hand, the brightness of the object does change. When we talk about the \"absolute magnitude\" or \"apparent magnitude\" of things like supernovae, we typically mean the magnitude when the supernova is at its brightest.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 423851,
          "author_name": "Chinta",
          "author_url": "",
          "post_date": "2018-11-19T06:24:57.883000",
          "content": "<p>Kyle, thanks for your time, so the impact of motion of extra galactic objects is negligible for classification task.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 424130,
      "author_name": "Yorca",
      "author_url": "",
      "post_date": "2018-11-19T16:06:45.517000",
      "content": "<p>good</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 423640,
      "author_name": "Blonde",
      "author_url": "",
      "post_date": "2018-11-18T19:18:03.320000",
      "content": "<p>so the astronomer on LB really motivated you to go deeper into domain knowledge :-)))</p>",
      "votes": 0,
      "replies": [
        {
          "id": 423651,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2018-11-18T19:42:36.700000",
          "content": "<p>I didn't wait for him ;) Whatever the domain, if you want effective machine learning, the more you understand the context, the better.</p>",
          "votes": 4,
          "replies": []
        }
      ]
    },
    {
      "id": 423182,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-11-17T17:20:10.713000",
      "content": "",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "423698": "The distance modulus is a pretty important concept, so I thought that I would do a little writeup about how it applies to this competition.\n\nThe distance modulus is a measure of how much fainter a distant object appears to be compared to what it would look like if it were close by. Nearby, this follows the \"one over r-squared\" law where if you move the object twice as far away it appears to be four times fainter . At distances on the scale of the universe, the expansion of the universe modifies the one over r-squared law. The distance modulus captures those modifications.\n\nWhen you measure the brightness of an object with a camera, you record what we call the \"flux\" which is proportional to how many photons the telescope sees from a distant object in a given time. The brightness of astronomical objects varies over huge scales, so we typically work in \"magnitudes\" which is a log-transform of the flux. The \"magnitude\" is calculated as -2.5*log10(measured flux) with a zeropoint that will depend on the telescope and passband.\n\nNow if you subtract the distance modulus from the measured magnitude, you will get the absolute magnitude of the object in question. The absolute magnitude is a measure of how bright the object would appear to be if you observed it at a fixed distance away. This can be used to tell objects apart. For example, at their brightest, Type Ia supernovae, have an absolute magnitude of around -19 mag with a scatter of ~0.5 mag. If you observe something that peaks at -22 mag, that probably isn't a Type Ia supernova. There is a list of peak absolute magnitudes for different supernovae here: https://en.wikipedia.org/wiki/Supernova. Note that we don't really know the zeropoint of the magnitudes that we are given, but all that really matters is magnitude differences for classification.\n\nFinally, the distance moduli that are provided in this competition come directly from the photo-zs. These photo-zs are noisy (compare to the spec-zs which are the \"truth\"), so the distance moduli are noisy too. You can calculate a distance modulus for a given redshift using the astropy python package assuming the value of cosmological parameters in our universe. If you look at the provided data, it appears that the calculation that we were given assumed a Hubble Constant of 70 km/s/Mpc, a flat universe, a matter density (Ωm) of 0.3 and a CMB temperature of 2.725. The following code will calculate the distance modulus column from the photo-z column in the data:\n\n    from astropy.cosmology import FlatLambdaCDM\n    cosmo = FlatLambdaCDM(H0=70, Om0=0.3, Tcmb0=2.725)\n    distance_modulus = cosmo.distmod(redshift)",
    "423148": "The distmod feature in the dataset is what astronomers call the distance modulus.  I find this wikipedia article useful to astronomy noobs like me: https://en.wikipedia.org/wiki/Distance_modulus",
    "423264": "it is 0.999 correlated with log(hostgal_photoz)",
    "423832": "Hi, I assume the redshift and distance modulus changes over time for an object, but in the data, these values are at metadata level. So they are not varying for an object over time  or assumed to be constant ?",
    "424130": "good\n",
    "423640": "so the astronomer on LB really motivated you to go deeper into domain knowledge :-)))",
    "423182": ""
  }
}