I can see a big gap between TOP3 and the rest, so some tips for people who do not know what to do during weekend;)
\n\n1) flux is a function of distance (flux ~ 1/distance^2 is quite a good approximation)\nSo, if train and test differ regarding distance/redshift, it may be a good idea to create a feature beeing fluxes \"visible\" from the same distance
\n\n2) Green light emited by objects may be measured by 'g' passband detector if redshift is small or by 'r', 'i'. etc. if redshift is big enough.\nIf you want to determine the \"real\" color of the object, i,e. the relation of fluxes for 'g' and 'r' passbands for conditions same for all objects, it may be a good idea to create features for almost the same redshift, i.e. to change 'r' to 'g', 'i' to 'r', etc. (and to decrease redshift by delta redshift) for redshifts exceeding base redshift by properly calculated delta redshift.
\n\n3) Time for fast moving objects flows slower, so however some objects emit the same amount of photons per second, the flux measured for objects of time 2 times slower is two times lower, because every second we catch photons emitted during \"object's\" 0.5 second.\nThis effect is probably compensated if you use color features, but it may be important if you compare fluxes of the same passband.\nI'm not a relativistic physicist, so I'm not quite sure, if I'm right. Even if I'm right, I do not know, if the data simulator used by the organizers respects that phenomenon.
\n\n4) LSST is located at Earth, not at the orbit. Light is partially absorbed/scattered by the atmosphere. The problem is that extinction is not constant, It depends not only on the concentration of aerosols etc. in the atmosphere, but also on the location of the observatory above sea level and the angle of observation (may be calculated from the observatory GPS location, time of observation and astronomical position of the observed object). Search for \"air mass\" if the above is not clear enough. \nThe GPS location of the observatory is not given by the organizers, so maybe this effect is not taken into consideration during simulations.
\n\nGood luck.
", "messages": [ { "id": 438922, "postDate": "2018-12-14T12:23:56.733Z", "content": "I can see a big gap between TOP3 and the rest, so some tips for people who do not know what to do during weekend;)
\n\n1) flux is a function of distance (flux ~ 1/distance^2 is quite a good approximation)\nSo, if train and test differ regarding distance/redshift, it may be a good idea to create a feature beeing fluxes \"visible\" from the same distance
\n\n2) Green light emited by objects may be measured by 'g' passband detector if redshift is small or by 'r', 'i'. etc. if redshift is big enough.\nIf you want to determine the \"real\" color of the object, i,e. the relation of fluxes for 'g' and 'r' passbands for conditions same for all objects, it may be a good idea to create features for almost the same redshift, i.e. to change 'r' to 'g', 'i' to 'r', etc. (and to decrease redshift by delta redshift) for redshifts exceeding base redshift by properly calculated delta redshift.
\n\n3) Time for fast moving objects flows slower, so however some objects emit the same amount of photons per second, the flux measured for objects of time 2 times slower is two times lower, because every second we catch photons emitted during \"object's\" 0.5 second.\nThis effect is probably compensated if you use color features, but it may be important if you compare fluxes of the same passband.\nI'm not a relativistic physicist, so I'm not quite sure, if I'm right. Even if I'm right, I do not know, if the data simulator used by the organizers respects that phenomenon.
\n\n4) LSST is located at Earth, not at the orbit. Light is partially absorbed/scattered by the atmosphere. The problem is that extinction is not constant, It depends not only on the concentration of aerosols etc. in the atmosphere, but also on the location of the observatory above sea level and the angle of observation (may be calculated from the observatory GPS location, time of observation and astronomical position of the observed object). Search for \"air mass\" if the above is not clear enough. \nThe GPS location of the observatory is not given by the organizers, so maybe this effect is not taken into consideration during simulations.
\n\nGood luck.
", "rawMarkdown": "I can see a big gap between TOP3 and the rest, so some tips for people who do not know what to do during weekend;)\n\n1) flux is a function of distance (flux ~ 1/distance^2 is quite a good approximation)\nSo, if train and test differ regarding distance/redshift, it may be a good idea to create a feature beeing fluxes \"visible\" from the same distance\n\n2) Green light emited by objects may be measured by 'g' passband detector if redshift is small or by 'r', 'i'. etc. if redshift is big enough.\nIf you want to determine the \"real\" color of the object, i,e. the relation of fluxes for 'g' and 'r' passbands for conditions same for all objects, it may be a good idea to create features for almost the same redshift, i.e. to change 'r' to 'g', 'i' to 'r', etc. (and to decrease redshift by delta redshift) for redshifts exceeding base redshift by properly calculated delta redshift.\n\n3) Time for fast moving objects flows slower, so however some objects emit the same amount of photons per second, the flux measured for objects of time 2 times slower is two times lower, because every second we catch photons emitted during \"object's\" 0.5 second.\nThis effect is probably compensated if you use color features, but it may be important if you compare fluxes of the same passband.\nI'm not a relativistic physicist, so I'm not quite sure, if I'm right. Even if I'm right, I do not know, if the data simulator used by the organizers respects that phenomenon.\n\n4) LSST is located at Earth, not at the orbit. Light is partially absorbed/scattered by the atmosphere. The problem is that extinction is not constant, It depends not only on the concentration of aerosols etc. in the atmosphere, but also on the location of the observatory above sea level and the angle of observation (may be calculated from the observatory GPS location, time of observation and astronomical position of the observed object). Search for \"air mass\" if the above is not clear enough. \nThe GPS location of the observatory is not given by the organizers, so maybe this effect is not taken into consideration during simulations.\n\nGood luck.", "votes": 31 }, { "id": 438936, "postDate": "2018-12-14T12:48:59.747Z", "content": "I think there is a thin line between sharing high scoring kernels in the last week and sharing good ideas in the last week. If you have waited until the last 3 days, you could also wait for 3 more days to share your ideas because maybe not everyone has time to try your ideas in this weekend. I believe you have good intent but if one of your ideas turns out to be something significant, it will be unfair to people who came up with these ideas themselves.
", "rawMarkdown": "I think there is a thin line between sharing high scoring kernels in the last week and sharing good ideas in the last week. If you have waited until the last 3 days, you could also wait for 3 more days to share your ideas because maybe not everyone has time to try your ideas in this weekend. I believe you have good intent but if one of your ideas turns out to be something significant, it will be unfair to people who came up with these ideas themselves.", "votes": 8 }, { "id": 438930, "postDate": "2018-12-14T12:38:24.847Z", "content": "For the rest, why are you sharing this during last week? It is like sharing good kernels during last week IMHO. See https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74375
", "rawMarkdown": "For the rest, why are you sharing this during last week? It is like sharing good kernels during last week IMHO. See https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74375", "votes": 7, "replies": [ { "id": 438938, "postDate": "2018-12-14T12:49:14.047Z", "content": "@CPMP, you are really nice, but my LB position proves that the above has nothing to do with good or high-scoring.
", "rawMarkdown": "@CPMP, you are really nice, but my LB position proves that the above has nothing to do with good or high-scoring.", "votes": 3 }, { "id": 438976, "postDate": "2018-12-14T13:54:54.913Z", "content": "I am not commenting on your score, I am commenting on you sharing useful info at the last minute. This is similar to sharing kernels, and I am not the only one saying it.
", "rawMarkdown": "I am not commenting on your score, I am commenting on you sharing useful info at the last minute. This is similar to sharing kernels, and I am not the only one saying it.", "votes": -3 }, { "id": 438983, "postDate": "2018-12-14T14:07:25.443Z", "content": "In the end it seems it's a lot down to the fact that kernels and discussions can get you shiny medals and colors under your avatar... I'm all in favor of removing these gimmick awards, I think it's giving people a little dopamine rush that gives them the incentive to share stuff carelessly.
", "rawMarkdown": "In the end it seems it's a lot down to the fact that kernels and discussions can get you shiny medals and colors under your avatar... I'm all in favor of removing these gimmick awards, I think it's giving people a little dopamine rush that gives them the incentive to share stuff carelessly.", "votes": 1 }, { "id": 439002, "postDate": "2018-12-14T14:57:36.713Z", "content": "No, it is about sharing stuff too late to make sure everybody has a fair chance to use what is shared. This can be way worse than what we are discussing here. First Grzegorz did not share a ready to use, high value, submission file. Second there are 3.5 days left still.
\n\nThe worst I saw was a kernel share that was worth a silver medal few hours before end in Talking data competition. I shared a link to the resulting discussion somewhere else in this forum. I was not impacted at all, being high enough on the LB, but I felt for all whose efforts were ruined when hundreds of people submitted what was shared when themselves had no sub left.
\n\nI'd also like to say I am not worried for our ranking here. Except for Ahmet I doubt anyone can dispute our rank, and those who can also know what Grzegorz shared. I just sincerely think it is unfair for the competitors in general.
\n\nAnyway, this is just my opinion, I am not linked to Kaggle staff by any mean, and one can safely ignore my opinion.
", "rawMarkdown": "No, it is about sharing stuff too late to make sure everybody has a fair chance to use what is shared. This can be way worse than what we are discussing here. First Grzegorz did not share a ready to use, high value, submission file. Second there are 3.5 days left still.\n\nThe worst I saw was a kernel share that was worth a silver medal few hours before end in Talking data competition. I shared a link to the resulting discussion somewhere else in this forum. I was not impacted at all, being high enough on the LB, but I felt for all whose efforts were ruined when hundreds of people submitted what was shared when themselves had no sub left.\n\nI'd also like to say I am not worried for our ranking here. Except for Ahmet I doubt anyone can dispute our rank, and those who can also know what Grzegorz shared. I just sincerely think it is unfair for the competitors in general.\n\nAnyway, this is just my opinion, I am not linked to Kaggle staff by any mean, and one can safely ignore my opinion. \n\n", "votes": 7 }, { "id": 439045, "postDate": "2018-12-14T16:33:06.350Z", "content": "On the one hand, I am pretty sure there was no bad intent in this case, grzegorz generously shared a very useful feature very early in the competition. On the other hand, personally I might be affected negatively as I devoted a lot of time to get where I am and these last few days I don't have a lot of time to devote to this competition and implement last minute useful ideas.
\n\nSo, primarily, @sionek thank you for sharing and offering to the community, but also though very interesting, IMHO it's a bit late.
\n\nPS: No one complained when the leader gave large hints equally late, so I guess everyone is biased to criticize what is hurting them personally.
", "rawMarkdown": "On the one hand, I am pretty sure there was no bad intent in this case, grzegorz generously shared a very useful feature very early in the competition. On the other hand, personally I might be affected negatively as I devoted a lot of time to get where I am and these last few days I don't have a lot of time to devote to this competition and implement last minute useful ideas. \n\nSo, primarily, @sionek thank you for sharing and offering to the community, but also though very interesting, IMHO it's a bit late. \n\nPS: No one complained when the leader gave large hints equally late, so I guess everyone is biased to criticize what is hurting them personally.", "votes": 5 }, { "id": 439118, "postDate": "2018-12-14T19:02:17.573Z", "content": "Sure, I appreciate Grzegorz sharing. It is just a timing issue.
\n\n\n\n\nNo one complained when the leader gave large hints equally late, so I guess everyone is biased to criticize what is hurting them personally.
\n
The hint was way less precise in that case. But yes, there could be an issue too.
", "rawMarkdown": "Sure, I appreciate Grzegorz sharing. It is just a timing issue.\n\n> No one complained when the leader gave large hints equally late, so I guess everyone is biased to criticize what is hurting them personally.\n\nThe hint was way less precise in that case. But yes, there could be an issue too." }, { "id": 439192, "postDate": "2018-12-14T21:39:28.367Z", "content": "All of this info is in the data note, so I don't think that there is anything new here.
", "rawMarkdown": "All of this info is in the data note, so I don't think that there is anything *new* here.", "votes": 14 }, { "id": 439258, "postDate": "2018-12-15T02:56:00.127Z", "content": "I agree with Kyle.
", "rawMarkdown": "I agree with Kyle.", "votes": 1 }, { "id": 439310, "postDate": "2018-12-15T06:16:17.423Z", "content": "Kyle,
\n\n> All of this info is in the data note, so I don't think that there is anything new here.
\n\nThese are specific instructions on what to do to improve a machine learning approach. They are NOT in the data note. I just reread it to check. What is in the data note is astronomy facts that could be used as input to items 2-4 from Grzegorz. Item 1 is totally absent from it.
\n\nThe fact that you could immediately translate your astronomer knowledge, or the part of it described in the data note, into useful feature engineering does not mean that people without any astronomy background would be able to do it as well.
\n\nSome of the reactions posted here coming from people with pretty good score already, show they haven't thought of what Grzegorz disclose. And it is not because they don't know how to read the data_note.
\n\nThis sharing is therefore significant. It is not the sharing that is an issue. It is its timing.
", "rawMarkdown": "Kyle,\n\n> All of this info is in the data note, so I don't think that there is anything new here.\n\nThese are specific instructions on what to do to improve a machine learning approach. They are NOT in the data note. I just reread it to check. What is in the data note is astronomy facts that could be used as input to items 2-4 from Grzegorz. Item 1 is totally absent from it. \n\nThe fact that you could immediately translate your astronomer knowledge, or the part of it described in the data note, into useful feature engineering does not mean that people without any astronomy background would be able to do it as well. \n\nSome of the reactions posted here coming from people with pretty good score already, show they haven't thought of what Grzegorz disclose. And it is not because they don't know how to read the data_note.\n\nThis sharing is therefore significant. It is not the sharing that is an issue. It is its timing.", "votes": 4 }, { "id": 439334, "postDate": "2018-12-15T07:11:41.117Z", "content": "I reread the original post and I see what you mean. I guess we'll have to wait and see what happens this weekend...
", "rawMarkdown": "I reread the original post and I see what you mean. I guess we'll have to wait and see what happens this weekend...", "votes": 3 }, { "id": 439362, "postDate": "2018-12-15T08:41:47.673Z", "content": "Thanks.
", "rawMarkdown": "Thanks.", "votes": 1 }, { "id": 439366, "postDate": "2018-12-15T08:50:49.787Z", "content": "I'll see if I beat my own record of down votes for a single post ;) Let me try with this one...
\n\nI hope people get I am not against sharing. I've proved it more than enough by sharing quite a lot here. I wish my down voters shared as much as I do.
", "rawMarkdown": "I'll see if I beat my own record of down votes for a single post ;) Let me try with this one...\n\nI hope people get I am not against sharing. I've proved it more than enough by sharing quite a lot here. I wish my down voters shared as much as I do.", "votes": 3 }, { "id": 439497, "postDate": "2018-12-15T16:22:04.623Z", "content": "Who thinks you are against sharing? you are bestfitting at discussion :)
", "rawMarkdown": "Who thinks you are against sharing? you are bestfitting at discussion :)", "votes": 4 }, { "id": 439562, "postDate": "2018-12-15T19:49:37.487Z", "content": "> Who thinks you are against sharing? you are bestfitting at discussion :)
\n\nThanks.
\n\nI don't know if people think that or not, but I see I get down votes that are interesting. For instance getting a down vote because I thank Kyle ;)
\n\nIt's been a long time I stopped trying to make sense of how people think and behave. I guess I'm not a very good mind reader. I just try to state what I think honestly, and reiterate if I think it is misunderstood. This is why I state again that I am in favor of sharing, except during last week.
\n\nThen people can ignore my opinion, and can down vote it if they are too lazy or too coward to write why they disagree ;)
", "rawMarkdown": "> Who thinks you are against sharing? you are bestfitting at discussion :)\n\nThanks.\n\nI don't know if people think that or not, but I see I get down votes that are interesting. For instance getting a down vote because I thank Kyle ;)\n\nIt's been a long time I stopped trying to make sense of how people think and behave. I guess I'm not a very good mind reader. I just try to state what I think honestly, and reiterate if I think it is misunderstood. This is why I state again that I am in favor of sharing, except during last week.\n\nThen people can ignore my opinion, and can down vote it if they are too lazy or too coward to write why they disagree ;)", "votes": 7 }, { "id": 439629, "postDate": "2018-12-16T00:44:15.990Z", "content": "yea i learned a lot from you as well. i think people just have to respect if you have different view on this matter
", "rawMarkdown": "yea i learned a lot from you as well. i think people just have to respect if you have different view on this matter", "votes": 1 } ] }, { "id": 439008, "postDate": "2018-12-14T15:37:17.817Z", "content": "I guess, #4 should not be a problem. If I'm correct, what we have here is flux calibrated data.
", "rawMarkdown": "I guess, #4 should not be a problem. If I'm correct, what we have here is flux calibrated data.", "votes": 1 }, { "id": 438926, "postDate": "2018-12-14T12:35:25.730Z", "content": "I think 4 is taken into account in the flux we are getting.
", "rawMarkdown": "I think 4 is taken into account in the flux we are getting. " }, { "id": 439937, "postDate": "2018-12-16T17:26:31.113Z", "content": "I tried 2 by using linear interpolation and 3 by adjusting mjd, but had no luck.
\n\nI my opinion, this information and the correct name for star classes should have been given from the start. I know it was left out \"not to give astronomers an advantage\", but the effect has been the opposite.
", "rawMarkdown": "I tried 2 by using linear interpolation and 3 by adjusting mjd, but had no luck.\n\nI my opinion, this information and the correct name for star classes should have been given from the start. I know it was left out \"not to give astronomers an advantage\", but the effect has been the opposite.", "replies": [ { "id": 439953, "postDate": "2018-12-16T18:01:38.283Z", "content": "I am an astronomer and I still have no idea what most of the classes are. Sure, I could go figure them out if I wanted, but knowing the true labels doesn't help you classify at all. The hard things to classify are the various types of supernovae that all behave very similarly.
", "rawMarkdown": "I am an astronomer and I still have no idea what most of the classes are. Sure, I could go figure them out if I wanted, but knowing the true labels doesn't help you classify at all. The hard things to classify are the various types of supernovae that all behave very similarly.", "votes": 1 }, { "id": 439955, "postDate": "2018-12-16T18:03:01.267Z", "content": "Rather than class names which would be useless to me, I wish we had the reference flux that is subtracted from the flux to yield what we get. I am not sure why this was not provided to us.
", "rawMarkdown": "Rather than class names which would be useless to me, I wish we had the reference flux that is subtracted from the flux to yield what we get. I am not sure why this was not provided to us." }, { "id": 439957, "postDate": "2018-12-16T18:04:46.117Z", "content": "\n\n\nThe hard things to classify are the various types of supernovae that all behave very similarly.
\n
I feel somewhat relieved that someone who is doing a PhD on supernovae finds the ones in this dataset hard to separate :)
", "rawMarkdown": "> The hard things to classify are the various types of supernovae that all behave very similarly.\n\nI feel somewhat relieved that someone who is doing a PhD on supernovae finds the ones in this dataset hard to separate :)", "votes": 2 }, { "id": 439982, "postDate": "2018-12-16T19:27:50.827Z", "content": "I see, Kyle. No harm intended. As for Sionkowskis 1. point I used log(flux) - log(distmod) wouldn't that give the same effect?
", "rawMarkdown": "I see, Kyle. No harm intended. As for Sionkowskis 1. point I used log(flux) - log(distmod) wouldn't that give the same effect? " }, { "id": 441296, "postDate": "2018-12-18T14:04:09.190Z", "content": "> I used log(flux) - log(distmod)
\n\ndistmod is already a log transformed value, you should not use log on it; The right formula is
\n\n-2.5 log10(flux) - distmod.
", "rawMarkdown": "> I used log(flux) - log(distmod)\n\ndistmod is already a log transformed value, you should not use log on it; The right formula is \n\n-2.5 log10(flux) - distmod." } ] }, { "id": 439872, "postDate": "2018-12-16T15:01:35.560Z", "content": "Number 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.
\n\nI tried using #2... \nHere's my understanding of it:
\n\nWe have lambda(emit)= lambda(observed)/(1+z)\nThe mid points of each observed passband are (in nm?):\nU: 365\nG: 475\nR: 658\nI: 806\nZ: 900\nY: 1020
\n\nSo if z=0.3, for the Z passband, you would get an observed mid point of 900, but the real emitted wavelength of 900/(1+0.3) = 692. Which is closer to the R passband.
\n\nHow to use this information, I don't really know.
\n\nI tried using the following (very crude) approximation, which didn't help with my model:
\n\nz< 0.2 : no change
\n\n0.2 \n\n
0.35\n\n
z>1 subtract 3 from passband
\n\nThen remove data where passband<0
", "rawMarkdown": "Number 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.\n\nI tried using #2... \nHere's my understanding of it:\n\nWe have lambda(emit)= lambda(observed)/(1+z)\nThe mid points of each observed passband are (in nm?):\nU: 365\nG: 475\nR: 658\nI: 806\nZ: 900\nY: 1020\n\nSo if z=0.3, for the Z passband, you would get an observed mid point of 900, but the real emitted wavelength of 900/(1+0.3) = 692. Which is closer to the R passband. \n\nHow to use this information, I don't really know.\n\nI tried using the following (very crude) approximation, which didn't help with my model:\n\nz< 0.2 : no change\n\n0.2 ", "replies": [ { "id": 439873, "postDate": "2018-12-16T15:05:05.093Z", "content": "Would be nice to know how others dealt with that issue, probably we will know soon enough!
", "rawMarkdown": "Would be nice to know how others dealt with that issue, probably we will know soon enough!" }, { "id": 439877, "postDate": "2018-12-16T15:13:47.667Z", "content": "im just wondering, did you create another feature for #1? or just modified the flux directly?\nI also did the color shifting but it did not help my model as well. in 24 hours we will get our questions answered
", "rawMarkdown": "im just wondering, did you create another feature for #1? or just modified the flux directly?\nI also did the color shifting but it did not help my model as well. in 24 hours we will get our questions answered" }, { "id": 439905, "postDate": "2018-12-16T16:11:57.290Z", "content": "\n\n\nNumber 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.
\n
I hinted at it indeed, but I did not describe it explicitly. Glad you found it, but not everybody did. Until this post.
", "rawMarkdown": "> Number 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.\n\nI hinted at it indeed, but I did not describe it explicitly. Glad you found it, but not everybody did. Until this post.", "votes": 2 }, { "id": 439954, "postDate": "2018-12-16T18:02:22.030Z", "content": "DylonLL : I'll share what I did at the end of the competition
", "rawMarkdown": "DylonLL : I'll share what I did at the end of the competition\n" }, { "id": 439991, "postDate": "2018-12-16T20:20:51.803Z", "content": "sure. in a few hours ill get answers to much questions. i wonder what i did wrong with fluxes. i used #1 a while ago as well but i am not sure as to why it did not give my local CV boost.
", "rawMarkdown": "sure. in a few hours ill get answers to much questions. i wonder what i did wrong with fluxes. i used #1 a while ago as well but i am not sure as to why it did not give my local CV boost." } ] }, { "id": 439344, "postDate": "2018-12-15T08:05:04.783Z", "content": "time is fair for everybody . if not , nothing is fair at all
", "rawMarkdown": "time is fair for everybody . if not , nothing is fair at all", "replies": [ { "id": 439368, "postDate": "2018-12-15T08:58:00.190Z", "content": "No it is not fair. Not everybody planned to, or even can, spend this week end on the competition. For an example see https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74679#439045
", "rawMarkdown": "No it is not fair. Not everybody planned to, or even can, spend this week end on the competition. For an example see https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74679#439045", "votes": -1 }, { "id": 439417, "postDate": "2018-12-15T12:18:56.087Z", "content": "first your and Grzegorz and other's sharings is admiring. we are learning and enjoying kaggling, thank you for that!
\n\nsecond I get the quiet period, and I upvote it , but I think there's a main difference between kernals and ideas because kernals let you gain without work and think, discussion make you work and think.
\n\nthird I think it's in the competition time. and time is fair, because the rule is clear.
\n\nIf someone only have few time everyday on the competition, and he love it , but don't have enough time to view all the discussions, is it also unfair?
\n\nif someone doesn't have last week or even last month to spend on the competition, then is it also unfair for him?
\n\nif share ideas is a good thing, I think it' s good all the time, after all , it' s some ideas activate people not kernals let people gain without work
\n\nif it's not, discussion made last week, or last month still unfair to some people maybe. It always affects someone who does't have the time, computation, background, language, or even they come up with this by themselves already. But I think no one sharing has a bad intent, we all want to contribute
\n\ndont know if I misunderstood something or make my meaning clear , if true, please remind me~
", "rawMarkdown": "first your and Grzegorz and other's sharings is admiring. we are learning and enjoying kaggling, thank you for that! \n\nsecond I get the quiet period, and I upvote it , but I think there's a main difference between kernals and ideas because kernals let you gain without work and think, discussion make you work and think. \n\nthird I think it's in the competition time. and time is fair, because the rule is clear.\n\nIf someone only have few time everyday on the competition, and he love it , but don't have enough time to view all the discussions, is it also unfair?\n\nif someone doesn't have last week or even last month to spend on the competition, then is it also unfair for him? \n\nif share ideas is a good thing, I think it' s good all the time, after all , it' s some ideas activate people not kernals let people gain without work\n\nif it's not, discussion made last week, or last month still unfair to some people maybe. It always affects someone who does't have the time, computation, background, language, or even they come up with this by themselves already. But I think no one sharing has a bad intent, we all want to contribute\n\ndont know if I misunderstood something or make my meaning clear , if true, please remind me~", "votes": 2 }, { "id": 439420, "postDate": "2018-12-15T12:27:47.357Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 439223, "postDate": "2018-12-14T23:35:09.590Z", "content": "I realized I didn't read the data note carefully. Hope it is not too late.
", "rawMarkdown": "I realized I didn't read the data note carefully. Hope it is not too late." }, { "id": 439207, "postDate": "2018-12-14T22:22:16.823Z", "content": "Thank you, it looks like I'm going to have a busy weekend! :)
", "rawMarkdown": "Thank you, it looks like I'm going to have a busy weekend! :)" }, { "id": 440224, "postDate": "2018-12-17T08:48:47.823Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 438970, "postDate": "2018-12-14T13:28:42.500Z", "content": "Thanks, I'll try them.
", "rawMarkdown": "Thanks, I'll try them." }, { "id": 438944, "postDate": "2018-12-14T13:00:02.690Z", "content": "thanks. i will try all of these
", "rawMarkdown": "thanks. i will try all of these" } ], "comments": [ { "id": 438936, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-14T12:48:59.747000", "content": "I think there is a thin line between sharing high scoring kernels in the last week and sharing good ideas in the last week. If you have waited until the last 3 days, you could also wait for 3 more days to share your ideas because maybe not everyone has time to try your ideas in this weekend. I believe you have good intent but if one of your ideas turns out to be something significant, it will be unfair to people who came up with these ideas themselves.
", "votes": 8, "replies": [] }, { "id": 438930, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-14T12:38:24.847000", "content": "For the rest, why are you sharing this during last week? It is like sharing good kernels during last week IMHO. See https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74375
", "votes": 7, "replies": [ { "id": 438938, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2018-12-14T12:49:14.047000", "content": "@CPMP, you are really nice, but my LB position proves that the above has nothing to do with good or high-scoring.
", "votes": 3, "replies": [] }, { "id": 438976, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-14T13:54:54.913000", "content": "I am not commenting on your score, I am commenting on you sharing useful info at the last minute. This is similar to sharing kernels, and I am not the only one saying it.
", "votes": -3, "replies": [] }, { "id": 438983, "author_name": "Max Halford", "author_url": "", "post_date": "2018-12-14T14:07:25.443000", "content": "In the end it seems it's a lot down to the fact that kernels and discussions can get you shiny medals and colors under your avatar... I'm all in favor of removing these gimmick awards, I think it's giving people a little dopamine rush that gives them the incentive to share stuff carelessly.
", "votes": 1, "replies": [] }, { "id": 439002, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-14T14:57:36.713000", "content": "No, it is about sharing stuff too late to make sure everybody has a fair chance to use what is shared. This can be way worse than what we are discussing here. First Grzegorz did not share a ready to use, high value, submission file. Second there are 3.5 days left still.
\n\nThe worst I saw was a kernel share that was worth a silver medal few hours before end in Talking data competition. I shared a link to the resulting discussion somewhere else in this forum. I was not impacted at all, being high enough on the LB, but I felt for all whose efforts were ruined when hundreds of people submitted what was shared when themselves had no sub left.
\n\nI'd also like to say I am not worried for our ranking here. Except for Ahmet I doubt anyone can dispute our rank, and those who can also know what Grzegorz shared. I just sincerely think it is unfair for the competitors in general.
\n\nAnyway, this is just my opinion, I am not linked to Kaggle staff by any mean, and one can safely ignore my opinion.
", "votes": 7, "replies": [] }, { "id": 439045, "author_name": "iprapas", "author_url": "", "post_date": "2018-12-14T16:33:06.350000", "content": "On the one hand, I am pretty sure there was no bad intent in this case, grzegorz generously shared a very useful feature very early in the competition. On the other hand, personally I might be affected negatively as I devoted a lot of time to get where I am and these last few days I don't have a lot of time to devote to this competition and implement last minute useful ideas.
\n\nSo, primarily, @sionek thank you for sharing and offering to the community, but also though very interesting, IMHO it's a bit late.
\n\nPS: No one complained when the leader gave large hints equally late, so I guess everyone is biased to criticize what is hurting them personally.
", "votes": 5, "replies": [] }, { "id": 439118, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-14T19:02:17.573000", "content": "Sure, I appreciate Grzegorz sharing. It is just a timing issue.
\n\n\n\n\nNo one complained when the leader gave large hints equally late, so I guess everyone is biased to criticize what is hurting them personally.
\n
The hint was way less precise in that case. But yes, there could be an issue too.
", "votes": 0, "replies": [] }, { "id": 439192, "author_name": "Kyle Boone", "author_url": "", "post_date": "2018-12-14T21:39:28.367000", "content": "All of this info is in the data note, so I don't think that there is anything new here.
", "votes": 14, "replies": [] }, { "id": 439258, "author_name": "mamas", "author_url": "", "post_date": "2018-12-15T02:56:00.127000", "content": "I agree with Kyle.
", "votes": 1, "replies": [] }, { "id": 439310, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-15T06:16:17.423000", "content": "Kyle,
\n\n> All of this info is in the data note, so I don't think that there is anything new here.
\n\nThese are specific instructions on what to do to improve a machine learning approach. They are NOT in the data note. I just reread it to check. What is in the data note is astronomy facts that could be used as input to items 2-4 from Grzegorz. Item 1 is totally absent from it.
\n\nThe fact that you could immediately translate your astronomer knowledge, or the part of it described in the data note, into useful feature engineering does not mean that people without any astronomy background would be able to do it as well.
\n\nSome of the reactions posted here coming from people with pretty good score already, show they haven't thought of what Grzegorz disclose. And it is not because they don't know how to read the data_note.
\n\nThis sharing is therefore significant. It is not the sharing that is an issue. It is its timing.
", "votes": 4, "replies": [] }, { "id": 439334, "author_name": "Kyle Boone", "author_url": "", "post_date": "2018-12-15T07:11:41.117000", "content": "I reread the original post and I see what you mean. I guess we'll have to wait and see what happens this weekend...
", "votes": 3, "replies": [] }, { "id": 439362, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-15T08:41:47.673000", "content": "Thanks.
", "votes": 1, "replies": [] }, { "id": 439366, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-15T08:50:49.787000", "content": "I'll see if I beat my own record of down votes for a single post ;) Let me try with this one...
\n\nI hope people get I am not against sharing. I've proved it more than enough by sharing quite a lot here. I wish my down voters shared as much as I do.
", "votes": 3, "replies": [] }, { "id": 439497, "author_name": "mamas", "author_url": "", "post_date": "2018-12-15T16:22:04.623000", "content": "Who thinks you are against sharing? you are bestfitting at discussion :)
", "votes": 4, "replies": [] }, { "id": 439562, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-15T19:49:37.487000", "content": "> Who thinks you are against sharing? you are bestfitting at discussion :)
\n\nThanks.
\n\nI don't know if people think that or not, but I see I get down votes that are interesting. For instance getting a down vote because I thank Kyle ;)
\n\nIt's been a long time I stopped trying to make sense of how people think and behave. I guess I'm not a very good mind reader. I just try to state what I think honestly, and reiterate if I think it is misunderstood. This is why I state again that I am in favor of sharing, except during last week.
\n\nThen people can ignore my opinion, and can down vote it if they are too lazy or too coward to write why they disagree ;)
", "votes": 7, "replies": [] }, { "id": 439629, "author_name": "dylonLL", "author_url": "", "post_date": "2018-12-16T00:44:15.990000", "content": "yea i learned a lot from you as well. i think people just have to respect if you have different view on this matter
", "votes": 1, "replies": [] } ] }, { "id": 439008, "author_name": "Vig", "author_url": "", "post_date": "2018-12-14T15:37:17.817000", "content": "I guess, #4 should not be a problem. If I'm correct, what we have here is flux calibrated data.
", "votes": 1, "replies": [] }, { "id": 438926, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-14T12:35:25.730000", "content": "I think 4 is taken into account in the flux we are getting.
", "votes": 0, "replies": [] }, { "id": 439937, "author_name": "PeterSorensen", "author_url": "", "post_date": "2018-12-16T17:26:31.113000", "content": "I tried 2 by using linear interpolation and 3 by adjusting mjd, but had no luck.
\n\nI my opinion, this information and the correct name for star classes should have been given from the start. I know it was left out \"not to give astronomers an advantage\", but the effect has been the opposite.
", "votes": 0, "replies": [ { "id": 439953, "author_name": "Kyle Boone", "author_url": "", "post_date": "2018-12-16T18:01:38.283000", "content": "I am an astronomer and I still have no idea what most of the classes are. Sure, I could go figure them out if I wanted, but knowing the true labels doesn't help you classify at all. The hard things to classify are the various types of supernovae that all behave very similarly.
", "votes": 1, "replies": [] }, { "id": 439955, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-16T18:03:01.267000", "content": "Rather than class names which would be useless to me, I wish we had the reference flux that is subtracted from the flux to yield what we get. I am not sure why this was not provided to us.
", "votes": 0, "replies": [] }, { "id": 439957, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-16T18:04:46.117000", "content": "\n\n\nThe hard things to classify are the various types of supernovae that all behave very similarly.
\n
I feel somewhat relieved that someone who is doing a PhD on supernovae finds the ones in this dataset hard to separate :)
", "votes": 2, "replies": [] }, { "id": 439982, "author_name": "PeterSorensen", "author_url": "", "post_date": "2018-12-16T19:27:50.827000", "content": "I see, Kyle. No harm intended. As for Sionkowskis 1. point I used log(flux) - log(distmod) wouldn't that give the same effect?
", "votes": 0, "replies": [] }, { "id": 441296, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-18T14:04:09.190000", "content": "> I used log(flux) - log(distmod)
\n\ndistmod is already a log transformed value, you should not use log on it; The right formula is
\n\n-2.5 log10(flux) - distmod.
", "votes": 0, "replies": [] } ] }, { "id": 439872, "author_name": "S D", "author_url": "", "post_date": "2018-12-16T15:01:35.560000", "content": "Number 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.
\n\nI tried using #2... \nHere's my understanding of it:
\n\nWe have lambda(emit)= lambda(observed)/(1+z)\nThe mid points of each observed passband are (in nm?):\nU: 365\nG: 475\nR: 658\nI: 806\nZ: 900\nY: 1020
\n\nSo if z=0.3, for the Z passband, you would get an observed mid point of 900, but the real emitted wavelength of 900/(1+0.3) = 692. Which is closer to the R passband.
\n\nHow to use this information, I don't really know.
\n\nI tried using the following (very crude) approximation, which didn't help with my model:
\n\nz< 0.2 : no change
\n\n0.2 \n\n
0.35\n\n
z>1 subtract 3 from passband
\n\nThen remove data where passband<0
", "votes": 0, "replies": [ { "id": 439873, "author_name": "S D", "author_url": "", "post_date": "2018-12-16T15:05:05.093000", "content": "Would be nice to know how others dealt with that issue, probably we will know soon enough!
", "votes": 0, "replies": [] }, { "id": 439877, "author_name": "dylonLL", "author_url": "", "post_date": "2018-12-16T15:13:47.667000", "content": "im just wondering, did you create another feature for #1? or just modified the flux directly?\nI also did the color shifting but it did not help my model as well. in 24 hours we will get our questions answered
", "votes": 0, "replies": [] }, { "id": 439905, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-16T16:11:57.290000", "content": "\n\n\nNumber 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.
\n
I hinted at it indeed, but I did not describe it explicitly. Glad you found it, but not everybody did. Until this post.
", "votes": 2, "replies": [] }, { "id": 439954, "author_name": "S D", "author_url": "", "post_date": "2018-12-16T18:02:22.030000", "content": "DylonLL : I'll share what I did at the end of the competition
", "votes": 0, "replies": [] }, { "id": 439991, "author_name": "dylonLL", "author_url": "", "post_date": "2018-12-16T20:20:51.803000", "content": "sure. in a few hours ill get answers to much questions. i wonder what i did wrong with fluxes. i used #1 a while ago as well but i am not sure as to why it did not give my local CV boost.
", "votes": 0, "replies": [] } ] }, { "id": 439344, "author_name": "yangDDD", "author_url": "", "post_date": "2018-12-15T08:05:04.783000", "content": "time is fair for everybody . if not , nothing is fair at all
", "votes": 0, "replies": [ { "id": 439368, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-15T08:58:00.190000", "content": "No it is not fair. Not everybody planned to, or even can, spend this week end on the competition. For an example see https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74679#439045
", "votes": -1, "replies": [] }, { "id": 439417, "author_name": "yangDDD", "author_url": "", "post_date": "2018-12-15T12:18:56.087000", "content": "first your and Grzegorz and other's sharings is admiring. we are learning and enjoying kaggling, thank you for that!
\n\nsecond I get the quiet period, and I upvote it , but I think there's a main difference between kernals and ideas because kernals let you gain without work and think, discussion make you work and think.
\n\nthird I think it's in the competition time. and time is fair, because the rule is clear.
\n\nIf someone only have few time everyday on the competition, and he love it , but don't have enough time to view all the discussions, is it also unfair?
\n\nif someone doesn't have last week or even last month to spend on the competition, then is it also unfair for him?
\n\nif share ideas is a good thing, I think it' s good all the time, after all , it' s some ideas activate people not kernals let people gain without work
\n\nif it's not, discussion made last week, or last month still unfair to some people maybe. It always affects someone who does't have the time, computation, background, language, or even they come up with this by themselves already. But I think no one sharing has a bad intent, we all want to contribute
\n\ndont know if I misunderstood something or make my meaning clear , if true, please remind me~
", "votes": 2, "replies": [] }, { "id": 439420, "author_name": "", "author_url": "", "post_date": "2018-12-15T12:27:47.357000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 439223, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2018-12-14T23:35:09.590000", "content": "I realized I didn't read the data note carefully. Hope it is not too late.
", "votes": 0, "replies": [] }, { "id": 439207, "author_name": "S D", "author_url": "", "post_date": "2018-12-14T22:22:16.823000", "content": "Thank you, it looks like I'm going to have a busy weekend! :)
", "votes": 0, "replies": [] }, { "id": 440224, "author_name": "", "author_url": "", "post_date": "2018-12-17T08:48:47.823000", "content": "", "votes": -1, "replies": [] }, { "id": 438970, "author_name": "mamas", "author_url": "", "post_date": "2018-12-14T13:28:42.500000", "content": "Thanks, I'll try them.
", "votes": 0, "replies": [] }, { "id": 438944, "author_name": "dylonLL", "author_url": "", "post_date": "2018-12-14T13:00:02.690000", "content": "thanks. i will try all of these
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "438922": "I can see a big gap between TOP3 and the rest, so some tips for people who do not know what to do during weekend;)\n\n1) flux is a function of distance (flux ~ 1/distance^2 is quite a good approximation)\nSo, if train and test differ regarding distance/redshift, it may be a good idea to create a feature beeing fluxes \"visible\" from the same distance\n\n2) Green light emited by objects may be measured by 'g' passband detector if redshift is small or by 'r', 'i'. etc. if redshift is big enough.\nIf you want to determine the \"real\" color of the object, i,e. the relation of fluxes for 'g' and 'r' passbands for conditions same for all objects, it may be a good idea to create features for almost the same redshift, i.e. to change 'r' to 'g', 'i' to 'r', etc. (and to decrease redshift by delta redshift) for redshifts exceeding base redshift by properly calculated delta redshift.\n\n3) Time for fast moving objects flows slower, so however some objects emit the same amount of photons per second, the flux measured for objects of time 2 times slower is two times lower, because every second we catch photons emitted during \"object's\" 0.5 second.\nThis effect is probably compensated if you use color features, but it may be important if you compare fluxes of the same passband.\nI'm not a relativistic physicist, so I'm not quite sure, if I'm right. Even if I'm right, I do not know, if the data simulator used by the organizers respects that phenomenon.\n\n4) LSST is located at Earth, not at the orbit. Light is partially absorbed/scattered by the atmosphere. The problem is that extinction is not constant, It depends not only on the concentration of aerosols etc. in the atmosphere, but also on the location of the observatory above sea level and the angle of observation (may be calculated from the observatory GPS location, time of observation and astronomical position of the observed object). Search for \"air mass\" if the above is not clear enough. \nThe GPS location of the observatory is not given by the organizers, so maybe this effect is not taken into consideration during simulations.\n\nGood luck.", "438936": "I think there is a thin line between sharing high scoring kernels in the last week and sharing good ideas in the last week. If you have waited until the last 3 days, you could also wait for 3 more days to share your ideas because maybe not everyone has time to try your ideas in this weekend. I believe you have good intent but if one of your ideas turns out to be something significant, it will be unfair to people who came up with these ideas themselves.", "438930": "For the rest, why are you sharing this during last week? It is like sharing good kernels during last week IMHO. See https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74375", "439008": "I guess, #4 should not be a problem. If I'm correct, what we have here is flux calibrated data.", "438926": "I think 4 is taken into account in the flux we are getting. ", "439937": "I tried 2 by using linear interpolation and 3 by adjusting mjd, but had no luck.\n\nI my opinion, this information and the correct name for star classes should have been given from the start. I know it was left out \"not to give astronomers an advantage\", but the effect has been the opposite.", "439872": "Number 1) was already out there, in another discussion that involved Grzegorz and CPMP. I used it and it helped.\n\nI tried using #2... \nHere's my understanding of it:\n\nWe have lambda(emit)= lambda(observed)/(1+z)\nThe mid points of each observed passband are (in nm?):\nU: 365\nG: 475\nR: 658\nI: 806\nZ: 900\nY: 1020\n\nSo if z=0.3, for the Z passband, you would get an observed mid point of 900, but the real emitted wavelength of 900/(1+0.3) = 692. Which is closer to the R passband. \n\nHow to use this information, I don't really know.\n\nI tried using the following (very crude) approximation, which didn't help with my model:\n\nz< 0.2 : no change\n\n0.2 ", "439344": "time is fair for everybody . if not , nothing is fair at all", "439223": "I realized I didn't read the data note carefully. Hope it is not too late.", "439207": "Thank you, it looks like I'm going to have a busy weekend! :)", "440224": "", "438970": "Thanks, I'll try them.", "438944": "thanks. i will try all of these" } }