{"cells":[{"metadata":{"nbpresent":{"id":"aaffd2c3-54c1-4f88-840b-02111a6237a8"},"slideshow":{"slide_type":"slide"},"_uuid":"034e73ae1a13904dd04a7a2b099695023f8f27c0"},"cell_type":"markdown","source":"# **The PLAsTiCC Astronomy \"Starter Kit\"**\n### -Gautham Narayan, 20180831\n\nThis kernel was developed for PLAsTiCC on Kaggle. The original version of this kernel is available as a Jupyter Notebook on <a href=\"https://github.com/LSSTDESC/plasticc-kit\">LSST DESC GitHub</a>. \n\n***"},{"metadata":{"nbpresent":{"id":"c06c5e08-417f-4806-84ca-55c1fc26b576"},"slideshow":{"slide_type":"skip"},"trusted":true,"_uuid":"c9ad93c9583b438947705ff7dbf89006c6eb5eeb"},"cell_type":"code","source":"# You can edit the font size here to make rendered text more comfortable to read\n# It was built on a 13\" retina screen with 18px\nfrom IPython.core.display import display, HTML\ndisplay(HTML(\"<style>.rendered_html { font-size: 18px; }</style>\"))\n\n# we'll also use this package to read tables\n# it's generally useful for astrophysics work, including this challenge\n# so we'd suggest installing it, even if you elect to work with pandas\nfrom astropy.table import Table","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"1ca8bd63e9ebb47bccb32d2bbb336f0a03da3ec0"},"cell_type":"markdown","source":"This Kaggle Kernel provides an introduction to some of the astrophysics that might be relevant for the \"Photometric LSST Astronomical Time-series Classification Challenge\" (PLAsTiCC) on Kaggle. It's intended to provide some domain knowledge to data scientists taking part in the challenge, and may even prove useful to some of the astronomers competing. \n\nWhile there's no practical way to cover all of the background astrophysics in a single Kernel, there should be enough relevant material to tackle the challenge. You can also safely ignore it and dive right in and do things your own way. Astrophysics is interesting stuff though, and if you are curious and want to know more about something, or how to do something that isn't covered here then just ask! You'll find us on the Kaggle forum for this challenge! \n\n***\n\n## Contents:\n- [1. LSST, PLAsTiCC and You](#section1)\n    - [1a. The Large Synoptic Survey Telescope](#section1a)\n    - [1b. Light Curves: Time-series of Brightness of Astronomical Sources](#section1b)\n    - [1c. How Astronomers have Studied Astronomical Sources Thus Far](#section1c)\n    - [1d. The PLAsTiCC Challenge: Classifying Astronomical Sources with Light Curves](#section1d)\n- [2. A Primer on the Data - Light Curves](#section2)\n    - [2a. Time](#section2a)\n    - [2b. Passband](#section2b)\n    - [2c. Flux and Flux Error](#section2c)\n- [3. Exploring the Light Curves](#section3)\n- [4. Primer cont'd - Metadata Provided with Light Curves](#section4)\n    - [4a. Coordinates on the Sky](#section4a)\n    - [4b. Galactic Coordinates](#section4b)\n    - [4c. Redshift](#section4c)\n    - [4d. A view of our cosmic neighborhood](#section4d)\n    - [4e. Cosmology](#section4e)\n    - [4f. Dust](#section4f)\n\n***"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"3364ec9caec8ed8abd1793695ab11e5082dba277"},"cell_type":"markdown","source":"## 1. LSST, PLAsTiCC and You\n<a id=\"section1\"></a>"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"7554690f013b260fa4af4c01e0c3bd80168d55d6"},"cell_type":"markdown","source":"### 1a. The Large Synoptic Survey Telescope\n<a id=\"section1a\"></a>\n\nPLAsTiCC is intended to simulate sources that vary with time in the night-sky as seen during the first three years of operation of the <a href=\"http://www.lsst.org\">Large Synoptic Survey Telescope (LSST)</a>. \n\nThe LSST (illustrated below) is a telescope with an 8.4 meter primary mirror diameter being built high up in Atacama desert of Chile, on a mountain called Cerro Pachon.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e65509b7387dc838a128933d3fc849f21f2ca713"},"cell_type":"code","source":"\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport numpy as np\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\n\ndef displayimage(image):\n    img = np.array(Image.open(image))\n    plt.figure(figsize=(10,10))\n    plt.axis('off')\n    _ = plt.imshow(img)\ndisplayimage('../input/plasticc-astronomy-starter-kit-media/LSST_night.jpg')\n","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"8a34db75f3ed68115aa5e5521bb0da604785c8e7"},"cell_type":"markdown","source":"#### _Figure 1: The LSST Telescope, Credit: LSST/NSF/AURA/Todd Mason Productions Inc_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"d6bb6f4e61c09b6021785aa8b3ab81d38e79ebad"},"cell_type":"markdown","source":"The telescope is going to have a 3.2 gigapixel camera that's about the size of an adult human to take images of the night sky. Those pixel numbers may not seem all that meaningful without a reference, so here is the LSST field-of-view  (FoV) compared against the full moon and a similar class telescope:\n\n"},{"metadata":{"trusted":true,"_uuid":"bedb32994a095d351235f5c4d095d7fa23775cae","_kg_hide-input":true},"cell_type":"code","source":"displayimage('../input/plasticc-astronomy-starter-kit-media/LSST_FoV.jpg')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"f039a88a69b59fd2d200317a9bceee0be4e3b948"},"cell_type":"markdown","source":"#### _Figure 2: The LSST Field of View compared to a camera on a similar telescope and the Moon. Credit: Zelko Ivezic, LSST_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"65c6e0112d7c0f756cb808bf296fefc1697a6c05"},"cell_type":"markdown","source":"This large FoV is what makes LSST so revolutionary. If you like photography, LSST has a 6.5 *meter* effective aperture and a focal ratio of f/# of 1.23. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"2d24d0ca74daa6395d06b7bb67521b3f40695b19"},"cell_type":"markdown","source":"This means that LSST can make a map of the entire southern sky, resolving details that are smaller than the width of a human hair held at arm's length... every 4 nights. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"1d72988ad1b2d65993676ca2e50aeaef68aefe9a"},"cell_type":"markdown","source":"That's a data rate of 20--40 terabytes, or about 1 US Library of Congress every night. It is, for want of a better word, ginormous."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"32c5c32bdce84e04ead7083ac1e127567d182d16"},"cell_type":"markdown","source":"The difference is even more stark when compared against telescopes that are both smaller and have smaller FoVs - like most everything astronomers have access to now. The Hubble Space Telescope only has a 2.4 meter mirror, and here's it's FoV... relative to one of those full moons:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f6890b8c187f230b887057baf6a4b300c6be89ae"},"cell_type":"code","source":"displayimage('../input/plasticc-astronomy-starter-kit-media/HST_field_of_view.jpg')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"ff297c2cb34d1b584e3a6ed475cce10260639f5f"},"cell_type":"markdown","source":"#### _Figure 3: The Hubble Space Telescope's field of view compared to the Moon_"},{"metadata":{"nbpresent":{"id":"afdbac60-a049-4077-b22a-79d255306198"},"slideshow":{"slide_type":"slide"},"_uuid":"7808360a8b55ba98a448e1f531b0109394e4aefe"},"cell_type":"markdown","source":"Hubble isn't just looking at a smaller piece of sky, it is also collecting light from this region much more slowly than LSST. \n\nConsequently, each and every week LSST will find more sources that vary with time than Hubble has ever seen... in its entire 28+ year life. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"766a79141bcc9a4c2d54e8f312f3a30a274c6660"},"cell_type":"markdown","source":"### 1b. Light Curves: Time-series of Brightness of Astronomical Sources\n<a id=\"section1b\"></a>\n\nScanning the entire sky repeatedly means we can identify astrophysical sources in the night sky that change in brightness over time, and we can create a plot of the the brightness vs time. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"1a589ad5d0b63f4a835ee957b85afc08f1ce4404"},"cell_type":"markdown","source":"The *time-series* of these change in brightness. We call these time-series, \"light curves.\" "},{"metadata":{"nbpresent":{"id":"3a9a5622-cd6e-492f-9ef0-a7518d16f756"},"slideshow":{"slide_type":"fragment"},"_uuid":"bb0e6b18973defdd47d24ec29092aa7c188f1a33"},"cell_type":"markdown","source":"Here's an illustration of one of these source - supernova 1998bu, imaged with the 1.2m telescope at the Fred Lawrence Whipple Observatory. "},{"metadata":{"ExecuteTime":{"end_time":"2018-09-06T18:26:48.487010Z","start_time":"2018-09-06T18:26:48.482835Z"},"nbpresent":{"id":"ffeae5bf-12ea-4581-bfcb-c88b01b57f60"},"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"f9eac1d37693861493ec416d6d3ea9fd2aef58b1","_kg_hide-input":true},"cell_type":"code","source":"%%HTML\n<div align=\"middle\">\n<video width=\"60%\" controls>\n      <source src=\"https://storage.googleapis.com/kaggle-datasets/52432/98042/SN98bu_LC.mp4?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1538266058&Signature=SgHGTPeH4qfe%2BvxF08OY0mBaQX5GxbSOUe2X2ibg%2FxWhGnJNaQ7JT57lFaAOpjN9HAEC%2Fy1o4OCzD3AdR%2F%2FLRXeSfY6BMZdsXclppAzS2Uf%2BXyhIYFXUhGe%2F1gc5Gch5myz%2FOre1h%2F5Fgh7AhK6dKAVMHhLYREWTWEyV8MJkhXtsy3a%2B%2BjwerdcdOiGtuQe3JzjHudBsw%2FjOCtLxYsA4SMNtI6F6750cgqAvC7OgNX6vt4tCpCQW6LxyZSn64sFD0ulZJF9IWWX9tqyMdVgcdb2CEYRke91kpXivdGAvjQrZDr1OkJXSYq6QcXY3JMGrx9egN0iMe3pIJ35Z8DPEsg%3D%3D\" type=\"video/mp4\">\n</video>\n</div>","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"c2b16093-fffa-4db6-9ecb-a216d2e99b0d"},"slideshow":{"slide_type":"-"},"_uuid":"8ddcb03cba48faf0075d56f9a4135c3e2d56c9d3"},"cell_type":"markdown","source":"#### _Figure 4: The light curve of supernova 1998bu. Credit: Peter Challis, Harvard/CfA_ \n\nThe left panel shows you how the supernova event looked, while the right panel shows a plot of the brightness of the supernova, measured using a process called \"photometry,\" used to determine the light curve."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"6c1d41aa5014ada8b545b21840be03d7be9e072f"},"cell_type":"markdown","source":"LSST presents us with a great opportunity here because it will find an order of magnitude (or more) of these kinds of events than astronomers have ever observed. \n\nThe same event rate also presents a new challenge - supernovae aren't the only sorts of variable and transient events in the night sky. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"dfccec2e65bb0838b42fc18a47170b192d03bd72"},"cell_type":"markdown","source":"Here's an (incomplete!) tree diagram of the various kinds of variable sources in the night sky:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a8bcb6896f4833fcebe72279d8869b9bbe836859"},"cell_type":"code","source":"displayimage('../input/plasticc-astronomy-starter-kit-media/variability_tree.jpg')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"895f34ae8e69904cdd3d65ef77623440ef7cc55f"},"cell_type":"markdown","source":"#### _Figure 5: The hierarchy of variable and transient sources. Credit: Laurent Eyer and Nami Mowlawi, Université de Genève_"},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"7d0a66ead87c39c7dfaab87ee97c1c4c9adaaefb"},"cell_type":"markdown","source":"Some sources are intrinsically variable, while others are extrinsic - they vary, not because they the source itself has some fundamental physical property that causes them to vary, but rather because of extra attributes - e.g. rotation, or forming part of a binary system that just happens to eclipse along our line of sight from Earth. Some sources vary periodically, while others show random or stochastic behavior. Yet others, like supernovae, typically explode once and fade away. \n\nThe sources in the training set are from a mixture of these different astrophysical classes, including:\n\n- Secular, Pulsating and Eruptive Variable Stars\n- Tidal Disruption Events\n- Kilonovae\n- Supernovae of different types\n- Active Galactic Nucleii \n- Microlensing Events\n- Eclipsing Binaries"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"65713ca248c0d68823615b12a70e2863b2ead1c7"},"cell_type":"markdown","source":"The diversity of these sources is important. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"10680495ca751888fef0164fec08cf88e2906ee2"},"cell_type":"markdown","source":"LSST data will be useful for many different scientific studies by different groups - from studying stars in our Galaxy to understand how solar systems like ours formed, probing nuclear physics with brief burps before stars explode, how galaxies themselves formed and the role massive stars play in shaping their chemistry, to measuring how much matter there is in our Universe - and everything in between. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"777dba828d122d0ac7d1807f26e1064dab66b8e4"},"cell_type":"markdown","source":"#### _It'll likely answers questions we don't even know to ask yet._"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"9905ebce4f4f9dc8cf3be6dd6f966328ca1d2df6"},"cell_type":"markdown","source":"There's a lot that we can get out of understanding the time-domain of our Universe."},{"metadata":{"nbpresent":{"id":"5b3f47c0-1126-47e8-9fbb-1d06b1396819"},"slideshow":{"slide_type":"skip"},"_uuid":"eb014f6754ba112633fc76a82050ae22efaea77d"},"cell_type":"markdown","source":"That said, we're being deliberately vague about exactly what sorts of astrophysical sources you'll find in the data, and even what `target` number in the data corresponds to what astrophysical class.\n\nWe'll give you a few external resources for these events in a companion Kernel, if you are determined to augment the training set. \n\nNote that there's been exactly one kilonova event ever seen (we hope this will be out of date by the time LSST turns on though!), so it's not really possible to provide multiple examples in some cases!"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"51c2cc3b58dbdf9c0c16d71d1ae9c5665b1e1051"},"cell_type":"markdown","source":"### 1c. How Astronomers have Studied Astronomical Sources Thus Far\n<a id=\"section1c\"></a>\n\nTo determine the class of any particular astrophysical source, we currently need lots of observational data. \n\nAt present, we classify the sources not using just their light curves, but using more detailed follow-up with other telescopes. We use \"spectrographs\" - instruments that, much like a prism, disperse a beam of light, spreading different colors over position - to study astrophysical sources.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f93f717447e6c3f87fcaf47da7cf1d05c2c2c67e"},"cell_type":"code","source":"displayimage('../input/plasticc-astronomy-starter-kit-media/prism.jpg')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"47adf93d8b216afaa19be4075e02564c0da5fd8e"},"cell_type":"markdown","source":"#### _Figure 6: A prism dispersing light, creating a spectrum, open access jpeg._"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"c5441a93cbc1ebf65e467fb2b0774f16dd33e486"},"cell_type":"markdown","source":"Dispersing the light like this is called \"spectroscopy\", and allows us to look for characteristic atomic signatures from astrophysical sources. "},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"60ec318956c9200edebf0bd768c3f22059f231ac"},"cell_type":"markdown","source":"#### _Figure 7: Spectra of different kinds of supernovae, with their atomic signatures labeled, Credit: Daniel Kasen (UC Berkeley)._"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0abf68c8f986a66edada8aa678567ffe71a458ee"},"cell_type":"code","source":"displayimage('../input/plasticc-astronomy-starter-kit-media/SN_Spectra.jpg')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"4348669c6c59876c9edf858770e51a36bc6782f0"},"cell_type":"markdown","source":"This is the most reliable method for determining what a point of light in the night sky actually is, and there's a ton of information you can get from it. \n\nYou can learn more about it <a href=\"https://ed.ted.com/lessons/how-do-we-study-the-stars-yuan-sen-ting\">here</a>. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"85e1745d2d189726c7a5d2d01148fd538f332e45"},"cell_type":"markdown","source":"Unfortunately, spectroscopy takes a tremendous amount of telescope time and there aren't enough telescopes in the world _combined_ to keep up with the LSST event rate. "},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"56a22f0c9bb028d65232a0f9bb0c775e74d8d2ab"},"cell_type":"markdown","source":"Additionally, if you remember the video of the supernova, you saw it getting brighter initially and then getting fainter. The light from the supernova explosion changes with time and therefore so do the spectra. We try to obtain a spectrum close to when the object is brightest, but often we can't schedule a telescope in advance exactly then, and even when we do the weather might not cooperate. If you can't see the stars on a cloudy night, then it's probably not good conditions to try and get a spectrum."},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"aadcae9bfbca5df2855560bc9b764841370f3509"},"cell_type":"markdown","source":"Other fields have similar issues - if we wanted to uniquely identify a human being for instance, we could test a DNA sample, but that'd be completely impractical given the number of people we encounter. Largely, we use facial recognition. There's less detailed information available, but you can still use it to identify people you see, and categorize them into family, friends, co-workers, strangers etc."},{"metadata":{"nbpresent":{"id":"cd99255f-d6d0-466b-af79-c9b8b11f842d"},"slideshow":{"slide_type":"slide"},"_uuid":"6bd54c133b43cee65debc94e640fce37ee239030"},"cell_type":"markdown","source":"We need to invent new techniques to classify astrophysical sources without using the detailed \"DNA sample\" that you'd get from spectroscopy, but using the less detailed information in their light curves - their \"photometry.\"\n\n**This is where PLAsTiCC and you come in.**"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"9d8ef6f504765d769a71fa957d1c2120bfeab5d0"},"cell_type":"markdown","source":"### 1d. The PLAsTiCC Challenge: Classifying Astronomical Sources with Light Curves\n<a id=\"section1d\"></a>\nPLAsTiCC is based on a simulation of the light curves that LSST can expect to collect (it doesn't exist yet, so we can't give you real LSST data!). These light curves have realistic time-sampling and noise properties, as well as realistic astrophysical effects for a variety of the sources in the variability tree. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"1bd3f6e8b9cf9317295759c6651b4cd1dc382310"},"cell_type":"markdown","source":"These sources, even closely related classes, can have light curves that look quite different from each other. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"f2816ba85deebcdd6b9e7cf78ee7eaddd24021cf"},"cell_type":"markdown","source":"You saw that different kinds of supernovae have different spectra. \nThis also means that they'll have different light curves. \n\nDifferent rise and fall times, different absolute energies, a different structure of bumps and wiggles. \n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3cbe674598b5b338f9456872434530c13b69e48b"},"cell_type":"code","source":"displayimage('../input/plasticc-astronomy-starter-kit-media/smartt_supernovae_diversity.jpg')","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"8c1389d8-4921-45d9-b51e-02aaa8e650a5"},"slideshow":{"slide_type":"-"},"_uuid":"1c2fe4ea015fa4475529860b8f34cbecfb0d5db1"},"cell_type":"markdown","source":"#### _Figure 8: The diversity of Supernovae Light Curves, Credit: Stephen J. Smartt, Queen's University, Belfast, https://www.nature.com/articles/nature11643_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"d96e2c6248a475a9048e2e03d4e244ce3e09149b"},"cell_type":"markdown","source":"The light curves of these explosive transients can look radically different from the light curves of periodic variables. Take for instance the eclipsing binary system <a href=\"http://simbad.u-strasbg.fr/simbad/sim-id?protocol=html&Ident=Y+Psc&NbIdent=1&Radius=2&Radius.unit=arcmin&submit=submit+id\">Y Piscium</a>. This is an eclipsing binary system (a name which the animation below should help explain) with a light curve that looks like this:\n\n<center><img src=\"https://storage.googleapis.com/kaggle-datasets/52432/98042/Y_Psc_Animated_LC.gif?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1538262235&Signature=kNFpXU3UFisXxfv4RppCSkLhC2xBgrIuFzVgdI7%2BbVMOUggcui9WFXF7jfFFRb36N5Iv3m9W%2B5SG6ryImaPx0phmNWfTYfjH7mHW8DPtrKjVouE4NwX%2BeLXwX2LLB%2Fr%2F5LG8bzsVY%2BJDeZR4Tfh30ToGu0aOGIlJxthIlOzzPqvUleCTow34vwcNg%2BolVRSCYKMvwtvd0iXDtmzMvOSWxJs3ynDJxbZ7E3AYvU4pF5sz16%2FonQ8BY1q%2BCnNWSOYbX%2BXU72CITygLqn6d9NOa%2FJnBlTFYlXtCuWgnnSYkS7wZyXHAPS7hdoQ9jHYX9JN3wREEf6Fb6Y85mbV8U%2B%2B%2BYw%3D%3D\"></center>"},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"b1bf4f5aec25045860d4079a18be971383bd3d4d"},"cell_type":"markdown","source":"#### _Figure 9: The light curves of an eclipsing binary system, Credit: AstroLab, Durham University, https://community.dur.ac.uk/physics.astrolab/eclipsing.html_"},{"metadata":{"nbpresent":{"id":"4448a0da-7353-4796-b2b8-02cd627211f1"},"slideshow":{"slide_type":"skip"},"_uuid":"080c2cd767f4ebe60b474a4e2cd298ac25fdf396"},"cell_type":"markdown","source":"The exact details of each eclipsing binary system determine the properties of the light curve - how bright each star is, their relative sizes, how fast they orbit each other - but all of these events have the same general light curve shape. And of course, depending on the period, the time scale of the variability can be anything from hours to years!"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"c7ee77e5decdaa3d327f5a42e4d4c9cdf179eccd"},"cell_type":"markdown","source":"Other variable stars have very different light curve shapes from eclipsing binaries. Indeed, variable stars were the first example of astronomers using light curve shapes for classification. \n\nHere's a hand-drawn illustration of some of the different known types of variable stars from <a href=\"https://archive.org/details/popularsciencemo69newy\">Popular Science Monthly in 1906</a> (page 179)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a495a0352496dda4bee921b1bf621c6d8fe08695"},"cell_type":"code","source":"def displayimageconverted(image):\n    img = np.array(Image.open(image).convert(\"RGBA\"))\n    plt.figure(figsize=(10,10))\n    plt.axis('off')\n    _ = plt.imshow(img)\ndisplayimageconverted('../input/plasticc-astronomy-starter-kit-media/atlas_of_variable_stars.png')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"2628f0a58b1835be90a476b43debcf2c79174c47"},"cell_type":"markdown","source":"#### _Figure 10: The Atlas of Variable Stars, Credit: Popular Science Monthly, from 1906_"},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"a38803a861202e59bdc92270d560df35a199ee6b"},"cell_type":"markdown","source":"The variables here (from left to right, top to bottom) are a Mira, an eclipsing binary and \"RR Lyrae\" of type AB and an RR Lyrae of type C. Eclipsing binaries are still called Algol-type stars some times, but note that they look really different from the bottom two panels, and if you look at the x-axis on the horizontal panel, you'll notice that the characteristic time-scale there is much different from the rest of them!"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"e725d3a6a566aa6cad5a53f8301f2bb8801e2c64"},"cell_type":"markdown","source":"And you have more information than just the one time-series. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"faa7214c4d5478b92c01d2fcbd8a4f7a9628dc8e"},"cell_type":"markdown","source":"If you watched that video of the supernova video carefully, you'll have noticed that there's actually two traces of the brightness - one that covers the optical, and one that covers the infrared. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"a169c7bdf606fbc1d39c991038da489ab25f1be6"},"cell_type":"markdown","source":"This is because our telescope detectors are sensitive to wavelengths of light well beyond the range our eyes can see. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"32678eb0b8cc9f5a9244305c1336b71591e70912"},"cell_type":"markdown","source":"We use optical filters - elements that only permit a certain range of wavelengths to pass through - to measure the brightness in different \"passbands.\" "},{"metadata":{"nbpresent":{"id":"7d3965c8-5ae2-442a-bf90-7de613a76694"},"slideshow":{"slide_type":"fragment"},"_uuid":"cd91bc9f7a1a72a55d6f87bf6bdbc9e8b548470a"},"cell_type":"markdown","source":"Here's the filter exchanger for the LSST camera (adult human for scale):"},{"metadata":{"ExecuteTime":{"end_time":"2018-09-06T18:26:22.711468Z","start_time":"2018-09-06T18:26:22.700884Z"},"nbpresent":{"id":"63b94464-92dd-40c0-bcb6-5625b28e0b1e"},"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"b5e69559700358d1aad6e1d0f125f60766d4f8d6","_kg_hide-input":true},"cell_type":"code","source":"import io\nimport base64\n\nvideo = io.open('../input/plasticc-astronomy-starter-kit-media/LSST_filter_change.mp4', 'rb').read()\nencoded = base64.b64encode(video)\nHTML(data='''\n<video width = 90% controls>\n   <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n</video>'''.format(encoded.decode('ascii')))\n\n","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"6073a488-ee3b-4fcb-bae4-77710fdf4ae2"},"slideshow":{"slide_type":"-"},"_uuid":"6adaa89979f23cd50de2a62e512d5d5c230ed77a"},"cell_type":"markdown","source":"#### _Figure 11: LSST Filter Exchanger, Credit: LSST/NSF/AURA/Todd Mason Productions Inc_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"b6a51d7e8324d7d0ac362190822a524114a828c9"},"cell_type":"markdown","source":"This is a bit like taking a low-resolution spectrum - the light is still being split up and we're getting information about different colors, but it's not as detailed. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"6c569e630bc5c0f2bed19507b252c0217c2b0f43"},"cell_type":"markdown","source":"With a spectrum, you can distinguish absorption and emission lines of atomic lines that are a few \"Angstroms\" across (the symbol for it is an A with the circle on top) - very approximately 10,000 times smaller than the width of an average strange of human hair. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"4ae9162621e13af2ec4666fab3fc36136edd4741"},"cell_type":"markdown","source":"The advantage of LSST and its giant camera is that it can take an image of several 100,000 astrophysical sources through a single filter at the same time. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"6d72794b2e5473b444a0bb115b36ab40baae78ae"},"cell_type":"markdown","source":"Each filter is a summing the total amount of light in one small region of the spectrum, so you can't measure narrow atomic lines anymore, but you can get many more light curves in the time it takes to get one spectrum. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"ef22d2e255d65d05300b00cbb32854cbfb6b38fa"},"cell_type":"markdown","source":"Even the biggest spectrographs with optical fibers dedicated to each sources can only manage a few thousand sources at most..."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"094be05ead33ff349a3a7272aa2ca822eb828270"},"cell_type":"markdown","source":"... in addition to taking much longer to collect enough photons to have a statistically significant measurement."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"9a210e6acf61270b5768aab8420bc8036d7fb438"},"cell_type":"markdown","source":"How much light gets through each filter depends not only on the properties of the filter, but really the transmission of everything between the camera and the source - the atmosphere, dust in our galaxy, dust in the galaxy of the source... \n\nWe refer to the total transmission of each filter as the \"passband\" (and often use the terms interchangeably). "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"e06549e8608636c77c20aa206d67ec3855a91638"},"cell_type":"markdown","source":"Each object has observations in every passband. The passbands that LSST will use are called *u, g, r, i, z*, and *y*. In other words, each object has six time-series you can use for classification. \n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"fc3140d911eca2803c91b1c5643ac7af4aba8539"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/LSST_passbands.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"4f753bdf-1b34-4c5c-afa6-dd3ddc9d160e"},"slideshow":{"slide_type":"-"},"_uuid":"2c55a62745149a345ffbe3d94526a14c7ab1ade9"},"cell_type":"markdown","source":"#### _Figure 12: The throughput of the LSST passbands with wavelength. Your eyes are sensitive to just the *g and r* bands_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"f50a27fc4a481f15abc2f27cf7cbd876a69ed138"},"cell_type":"markdown","source":"The training sample is what the entire astronomical community can expect to have before LSST turns on, except where astronomers have a mishmash of observations from different telescopes (something you'll have to deal with if you decide and try to augment it from the literature), you get a nice homogeneous training set. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"2b4b688827527bf4a1323b8cdfbc9a0c8358e3c3"},"cell_type":"markdown","source":"The test set is a version what we expect to have after 3 years of LSST operations. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"f2485fe683a508b0266bc2c9a49bb9c44a623c2e"},"cell_type":"markdown","source":"It's much, much bigger than the training set. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"421d14c527add30a7716677a17908648d77445f1"},"cell_type":"markdown","source":"You'll have to design your algorithm with speed as a consideration. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"051bf42d5037af3d612778ad1e022ee4e52c4f6d"},"cell_type":"markdown","source":"Also, the training set is not perfectly representative - we can usually only classify the brightest objects we find with spectroscopy. To do well in PLAsTiCC though, you'll have to classify everything"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"e7904fcfc9f6135870f25fb1e4352f03d7b91cd9"},"cell_type":"markdown","source":"You can even expect sources in the test set that aren't in the training set - we expect to find new classes of objects with LSST that we've never seen before, and we've simulated light curves based on theoretical predictions of how they ought to look!"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"4483f0ff2f08878ab11d07eb798dc79bc8690f3a"},"cell_type":"markdown","source":"**Your mission, should you choose to accept it, is to classify the things that go bump in the night.**"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"f2a56ae315f6a884c9b5d840a09da0c449905db4"},"cell_type":"markdown","source":"You can win prizes, eternal glory, and the satisfaction of knowing that you made a real impact and a valuable contribution to the biggest astrophysical experiment to ever be attempted!"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"58bcf67fb6dcd6079bd81e24e8830501013f8c33"},"cell_type":"markdown","source":"There's not much time too - LSST is coming up fast. Here's a construction picture from August 2018, and it is looking a lot like the render at the top of this page! \n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5735abd7ea1fb33e600f4d075b72645d8a627b4a"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/LSST_construction.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"b52cae10-93c2-4c86-a92b-025730dbf030"},"slideshow":{"slide_type":"-"},"_uuid":"a2d727910b5805c2a96a70829aa1febad4241f54"},"cell_type":"markdown","source":"#### _Figure 13: The LSST telescope under construction on Cerro Pachon in Chile, Credit: Gianluca Lombardi, Gemini Observatory_\n\n***"},{"metadata":{"nbpresent":{"id":"99ea822c-dbfc-4fa7-a106-aebfce425acb"},"slideshow":{"slide_type":"slide"},"_uuid":"6ba12471314c40dfcb2ec989ec40f61125e49b1c"},"cell_type":"markdown","source":"## 2. A Primer on the Dataset - Light Curves\n<a id=\"section2\"></a>"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"36e948977e2438dcdd408d11c24b6df4362f7dec"},"cell_type":"markdown","source":"We've included some example light curve data along with this Kernel to illustrate some of the astrophysical effects you'll encounter in the full data set. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"32e90dadad75629a50ecadfbf5cadda282d18216"},"cell_type":"markdown","source":"This example data is simpler than the real data - it has only four of the six passbands, and different properties from the full sample, so that the signal is more obvious, and only a few classes. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"e7633d2cc1489fc22e22fc06fe21b117b5b70cf8"},"cell_type":"markdown","source":"These simplifications will make it easier to work with and highlight some of the issues you'll encounter will the full dataset."},{"metadata":{"nbpresent":{"id":"b144fac1-5b01-441f-b89a-8b52e19b713e"},"slideshow":{"slide_type":"skip"},"_uuid":"7253597fe3c5216aba9d1384fb391ece4e608693"},"cell_type":"markdown","source":"The other reason why this is using example data and not the training data supplied with the challenge is that we've identified the target classes in the challenge with random integers. This should help reduce astronomer's advantage of domain knowledge - astronomers will have to identify which class corresponds to which integers before they can take advantage of their understanding of the properties of different classes. Most astronomers only specialize in a few kinds of objects though, so even their domain knowledge isn't complete. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"51ea6e6f2b6bed56b4a1d8fa3b159405124d6c33"},"cell_type":"markdown","source":"Lets begin by looking at the format of the light curve and explaining what the columns are (we'll get to the header in Section 3):"},{"metadata":{"nbpresent":{"id":"8d65ff80-af34-4ddc-afba-b9874ac23e02"},"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"ad6903343cbd21bf203fe6e596116773a67fdd41"},"cell_type":"code","source":"obj1 = Table.read('../input/plasticc-astronomy-starter-kit-media/fake010.csv', format='csv')\nobj1","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"7f5be4a6a2a05b0d8b4ffe478518ca1924f2423e"},"cell_type":"markdown","source":"The first of the columns is just the `object_id`. In the real data, you'll get many objects in the same CSV, and you'll need some way of getting all the observations for one object. `object_id` gives you that index."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"7c04ab93ae9ff347a32cf7a77f230e4e70d04c8e"},"cell_type":"markdown","source":"### 2a. Time\n<a id=\"section2a\"></a>\n\nThe second column, labeled `mjd`, is the time that LSST took an image. More on why it is called this later. First, you should get a sense for what the pattern of LSST observations is."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"283268c9c0b6b9a90e2e39db26bed12da3fbe836"},"cell_type":"markdown","source":"On any given night, LSST will begin by taking a picture of one patch for 15--30 seconds long (the results from PLAsTiCC might help decide the exposure time!)."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"2f9cd172c15c64befeb22e0cfb7b0b5150970554"},"cell_type":"markdown","source":"This is followed by a gap of 30 minutes -- 2 hours (the results from PLAsTiCC might help decide this too!) where the telescope looks at other patches of sky, before coming back to the original patch of sky."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"52491f5b78d4d5e3d80ba6388012892a14261322"},"cell_type":"markdown","source":"It will then take another 15--30 second picture. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"c9673494f9978d2c641f0edb2dd6aac74c4ad051"},"cell_type":"markdown","source":"The data from these two images on the same night are combined to remove spurious glitches like cosmic rays, asteroids, airplane contrails etc."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"0a792f7183c7251f2773134b7852f51b8cb39f64"},"cell_type":"markdown","source":"LSST will then come back to the same patch of sky, the next night, but it will take the picture in a different passband. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"653210e2717821d8d9e27449665fe1adbac71922"},"cell_type":"markdown","source":"LSST will repeat this cycle, returning to the original filter on the original patch of sky 1--2 weeks later. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"9d6fce7018b0be211d68732e797b76dacf04ca07"},"cell_type":"markdown","source":"This pattern of how a telescope observes the night sky is called the \"cadence\" and it has a large effect on how well the light curve of an astrophysical source is sampled. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"9762c04465bf943f7e718b752fcb95383b04dcf9"},"cell_type":"markdown","source":"Because the characteristic times that we conduct our observations on is a day (we can't observe when the sun is up after all!), we measure time in units of decimal day. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"cfeb231d756f0dfedfe312731bb1810432dc91d6"},"cell_type":"markdown","source":"The specific time format we will use in PLAsTiCC is called the Modified Julian Date (MJD) - simply the number of decimal days from midnight on November 17, 1858. "},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"d08a350fdb1b688947b3844a279b59b6f3f482d3"},"cell_type":"markdown","source":"(Historical parenthetical: The original Julian Date which began counting up from zero on noon on Jan 1, 4713 BC. When the Soviet Sputnik launched, the Smithsonian Astrophysical Observatory decided to make this zero reference date a little more current. The SAO needed a way to keep track of Sputnik's orbits but the reference for the Julian date was so far in the past that the numbers were too large for their VAX/VMS system to handle. So they did the simple thing and made the reference date of the VAX/VMS system the same as the reference date for their new time system - creatively named the \"Modified Julian date\" - and consequently, most of astronomy. This date is conveniently 2400000.5 - the extra 0.5 because the VAX reference starts at midnight, while the Julian Date starts from noon.)"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"a10fce2a45cf2d2f2102093032ee84c238813cbe"},"cell_type":"markdown","source":"If you want another time format like UNIX time (which counts decimal seconds from a reference date of Jan 1st, 1970), then it's easy to convert between the two. \n\nThe UNIX epoch in MJD is 40587 (or JD 2440587.5), so you'd use:\n\n$$\\textrm{UNIX time} = (\\textrm{MJD} - 40587) \\times 86400$$\n\nwhere 86400 is the number of seconds in 24 hours."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"899c13e37d442a47d6ca1d6822efcf7f7b80d9f7"},"cell_type":"markdown","source":"### 2b. Passband\n<a id=\"section2b\"></a>\n\nThe next column, labeled `passband` tells you the passband that the observation at time `mjd` was taken. "},{"metadata":{"nbpresent":{"id":"df1626e3-77e6-441c-a3a2-017b2f8c43ee"},"slideshow":{"slide_type":"fragment"},"_uuid":"07917b5d2155cb399e0f003bc099b0a1288cf155"},"cell_type":"markdown","source":"We've mapped the passband names to integers, but the mapping is straightforward:\n\n$$ \\{ u \\rightarrow 0,~g \\rightarrow 1,~r \\rightarrow 2,~i \\rightarrow 3,~z \\rightarrow 4,~y \\rightarrow 5 \\} $$"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"b4fffbb1e9d6be7da27d8acd46ef64f3319b616f"},"cell_type":"markdown","source":"You probably guessed as much looking at Figure 10, but $u$ covers the ultraviolet, $g$ covers what your eye perceives as blue/green, $r$ covers red, $i$ covers the infrared. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"6455a7882aa8224f9efb8c9053cdd01263be05bc"},"cell_type":"markdown","source":"I'm not even going to pretend that $z$ and $Y$ have good reasons to be named what they are. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"b7ff6c70cff9be0fa4b9bdd47cb24045cd3e4cab"},"cell_type":"markdown","source":"Unlike other multi-channel time-series data, you do not get all the other channels or passbands at once. There's only one filter in front of the camera at any time."},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"f19b69b5da28d09d9e3f87e727d15982a2329add"},"cell_type":"markdown","source":"(Historical parenthetical: See <a href=\"http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?2008JAVSO..36..110M&amp;data_type=PDF_HIGH&amp;whole_paper=YES&amp;type=PRINTER&amp;filetype=.pdf\">Milone and Young, 2008</a>, Sec. 5)"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"6b003ec05027cba7fae0baad32c3c4a7fb1573df"},"cell_type":"markdown","source":"### 2c. Flux and Flux Error\n<a id=\"section2c\"></a>\n\nThe last two columns in this file are named `flux` contains the actual photometry - the measurements of brightness of this source as a function of time. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"252028217b114d33a65541846d1224c0a17844ae"},"cell_type":"markdown","source":"The 68% confidence interval of the measurement is in the `flux_err` column. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"f40904207dd60c12b62abb9d8bcecd39fcdb2f1c"},"cell_type":"markdown","source":"It's worth noting that while astrophysical sources have to emit enough light to be brighter than the background sky (more or less the definition of what a source is), the flux can be positive or negative."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"87e0599ca9786e1b3c48d095228de4223b305968"},"cell_type":"markdown","source":"This is because of how we actually detect these sources against the background sky. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"146ffd5cfa78960068a687b4f4c8846eaf1c9e01"},"cell_type":"markdown","source":"LSST will have a reference image for each position on the sky - called a \"template\". "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"7066ff0b25540a91cbc3a77a4a9ec96e1e2eeeb0"},"cell_type":"markdown","source":"Each image it takes of a particular region is differenced with the template. "},{"metadata":{"_uuid":"4ae51e04a73d8f80e8fe77a24e60c8f39fe6d4f9"},"cell_type":"markdown","source":"Each image then includes a contribution from the source and the background. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"b414ba0129a645ec525552ac7485b07dbb09870c"},"cell_type":"markdown","source":"Objects for which the total flux from source and background is higher than that of the template have positive flux, and when the total flux is lower than the template flux, they have have negative flux."},{"metadata":{"_uuid":"4c1c871ee335eeb9882f756c81a411cde65ee328"},"cell_type":"markdown","source":"As the source flux changes with time, some sources can have some flux entries as positive and others that are negative."},{"metadata":{"_uuid":"e339cba02e0e4a77aaed934f91a22b4886fcdf44"},"cell_type":"markdown","source":"For some objects, the contribution from the source will fade over time, leaving just the background - which is about the same level as the flux in the template, so some flux entries will be consistent with zero."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"1b525e851160e2a52414ef5a6b678d7cbe53289f"},"cell_type":"markdown","source":"Here's how that looks:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"21ac37d42e372f837aa4818f40aaa2e2f6b42db3"},"cell_type":"code","source":"displayimageconverted(\"../input/plasticc-astronomy-starter-kit-media/2007-X-025_R.PS.png\")\ndisplayimageconverted(\"../input/plasticc-astronomy-starter-kit-media/2007-X-025_I.PS.png\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"f21030cd6dca7b2b4dc9a8dd9864127ef330b3c6"},"cell_type":"markdown","source":"#### _Figure 14: Difference Imaging \"Postage Stamps\" (Reference, Image, Difference)_"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"201be228ee10dd29a5932037d036c203a09b8bdc"},"cell_type":"markdown","source":"The template image in the left panels is from 2004, while the \"current\" images are from 2007. The third panel has the difference between the two - and you can see there's some light left over near the fuzzy galaxy. The two rows correspond to roughly the $r$ filter and the $i$ filter. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"db1bd86be21453a4b5a0fa23805cef8826cb638d"},"cell_type":"markdown","source":"You can see the properties of the two sets of images are different. There's a much higher background in the bottom row thanks to a bright source off to the top right. "},{"metadata":{"_uuid":"6dadd5dcf576c520def37b15d12a165dc6e05553"},"cell_type":"markdown","source":"This is also one of the reasons why LSST is being built on top of a mountain high in the Atacama desert of Chile - it's far away from bright city lights and the sky background is lower, allowing us to detect fainter sources. "},{"metadata":{"_uuid":"42b1019e237a27733fd8844c0da3519f23b80098"},"cell_type":"markdown","source":"This particular source was a supernova, but the same technique can be used to identify all manner of variable and transient sources in astronomical images."},{"metadata":{"nbpresent":{"id":"95f13cf1-5956-4b5d-a40e-dae6c52f8ee6"},"slideshow":{"slide_type":"fragment"},"_uuid":"8b51064b41a2af9f299c69547964c78b90a1711b"},"cell_type":"markdown","source":"Because of this difference imaging techniques, objects have to have changed significantly with respect to the template to be detected by LSST. Whether or not they were detected with respect to the template is reported as `detected_bool`."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"d4fd2f4b57e4bae75d5cd064c50f8d9f42dedc62"},"cell_type":"markdown","source":"## 3. Exploring the Light Curves\n<a id=\"section3\"></a>\n\nTo get a better sense of what the `flux` and `flux_err` look like, let's take a look at some of their light curves."},{"metadata":{"nbpresent":{"id":"1fe8a6e9-5eba-4ab1-a2ca-7888f3275dde"},"slideshow":{"slide_type":"skip"},"_uuid":"297ce64b583e8769d2fae4abe2c2fb2a5bcad57a"},"cell_type":"markdown","source":"We'll begin by defining a class to handle this example, with a few illustrative features you can use for classification:"},{"metadata":{"nbpresent":{"id":"9d5ae1ff-bcc0-42a8-b366-38703a122397"},"slideshow":{"slide_type":"skip"},"trusted":true,"_uuid":"e842ed69bc5e82ff57d5ba10efea457efcb4a01a"},"cell_type":"code","source":"import os\nimport numpy as np\nimport scipy.stats as spstat\nimport matplotlib.pyplot as plt\nfrom collections import OrderedDict\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"233e0dfe-c579-42b2-ab56-3b1e83492c6b"},"slideshow":{"slide_type":"skip"},"trusted":true,"_uuid":"29ba2f55916b27d0641185d60893005b87a516ec"},"cell_type":"code","source":"class LightCurve(object):\n    '''Light curve object for PLAsTiCC formatted data'''\n    \n    _passbands = OrderedDict([(0,'C4'),\\\n                              (1,'C2'),\\\n                              (2,'C3'),\\\n                              (3,'C1'),\\\n                              (4,'k'),\\\n                              (5,'C5')])\n    \n    _pbnames = ['u','g','r','i','z','y']\n    \n    def __init__(self, filename):\n        '''Read in light curve data'''\n\n        self.DFlc     = Table.read(filename, format='ascii.csv')\n        self.filename = filename.replace('.csv','')\n        self._finalize()\n     \n    # this is some simple code to demonstrate how to calculate features on these multiband light curves\n    # we're not suggesting using these features specifically\n    # there also might be additional pre-processing you do before computing anything\n    # it's purely for illustration\n    def _finalize(self):\n        '''Store individual passband fluxes as object attributes'''\n        # in this example, we'll use the weighted mean to normalize the features\n        weighted_mean = lambda flux, dflux: np.sum(flux*(flux/dflux)**2)/np.sum((flux/dflux)**2)\n        \n        # define some functions to compute simple descriptive statistics\n        normalized_flux_std = lambda flux, wMeanFlux: np.std(flux/wMeanFlux, ddof = 1)\n        normalized_amplitude = lambda flux, wMeanFlux: (np.max(flux) - np.min(flux))/wMeanFlux\n        normalized_MAD = lambda flux, wMeanFlux: np.median(np.abs((flux - np.median(flux))/wMeanFlux))\n        beyond_1std = lambda flux, wMeanFlux: sum(np.abs(flux - wMeanFlux) > np.std(flux, ddof = 1))/len(flux)\n        \n        for pb in self._passbands:\n            ind = self.DFlc['passband'] == pb\n            pbname = self._pbnames[pb]\n            \n            if len(self.DFlc[ind]) == 0:\n                setattr(self, f'{pbname}Std', np.nan)\n                setattr(self, f'{pbname}Amp', np.nan)\n                setattr(self, f'{pbname}MAD', np.nan)\n                setattr(self, f'{pbname}Beyond', np.nan)\n                setattr(self, f'{pbname}Skew', np.nan)\n                continue\n            \n            f  = self.DFlc['flux'][ind]\n            df = self.DFlc['flux_err'][ind]\n            m  = weighted_mean(f, df)\n            \n            # we'll save the measurements in each passband to simplify access.\n            setattr(self, f'{pbname}Flux', f)\n            setattr(self, f'{pbname}FluxUnc', df)\n            setattr(self, f'{pbname}Mean', m)\n            \n            # compute the features\n            std = normalized_flux_std(f, df)\n            amp = normalized_amplitude(f, m)\n            mad = normalized_MAD(f, m)\n            beyond = beyond_1std(f, m)\n            skew = spstat.skew(f) \n            \n            # and save the features\n            setattr(self, f'{pbname}Std', std)\n            setattr(self, f'{pbname}Amp', amp)\n            setattr(self, f'{pbname}MAD', mad)\n            setattr(self, f'{pbname}Beyond', beyond)\n            setattr(self, f'{pbname}Skew', skew)\n        \n        # we can also construct features between passbands\n        pbs = list(self._passbands.keys())\n        for i, lpb in enumerate(pbs[0:-1]):\n            rpb = pbs[i+1]\n            \n            lpbname = self._pbnames[lpb]\n            rpbname = self._pbnames[rpb]\n            \n            colname = '{}Minus{}'.format(lpbname, rpbname.upper())\n            lMean = getattr(self, f'{lpbname}Mean', np.nan)\n            rMean = getattr(self, f'{rpbname}Mean', np.nan)\n            col = -2.5*np.log10(lMean/rMean) if lMean> 0 and rMean > 0 else -999\n            setattr(self, colname, col)\n    \n    # this is a simple routine to visualize a light curve\n    # it can plot vs the MJD array of the light curve\n    # or vs an optional `phase` array that you pass \n    def plot_multicolor_lc(self, phase=None):\n        '''Plot the multiband light curve'''\n\n        fig, ax = plt.subplots(figsize=(8,6))\n\n        \n        if phase is None:\n            phase = []\n        if len(phase) != len(self.DFlc):\n            phase = self.DFlc['mjd']\n            xlabel = 'MJD'\n        else:\n            xlabel = 'Phase'\n            \n        for i, pb in enumerate(self._passbands):\n            pbname = self._pbnames[pb]\n            ind = self.DFlc['passband'] == pb\n            if len(self.DFlc[ind]) == 0:\n                continue\n            ax.errorbar(phase[ind], \n                     self.DFlc['flux'][ind],\n                     self.DFlc['flux_err'][ind],\n                     fmt = 'o', color = self._passbands[pb], label = f'{pbname}')\n        ax.legend(ncol = 4, frameon = True)\n        ax.set_xlabel(f'{xlabel}', fontsize='large')\n        ax.set_ylabel('Flux', fontsize='large')\n        fig.suptitle(self.filename, fontsize='x-large')\n        fig.tight_layout(rect=[0, 0, 1, 0.97])\n    \n    def get_features(self):\n        '''Return all the features for this object'''\n        variables = ['Std', 'Amp', 'MAD', 'Beyond', 'Skew']\n        feats = []\n        for i, pb in enumerate(self._passbands):\n            pbname = self._pbnames[pb]\n            feats += [getattr(self, f'{pbname}{x}', np.nan) for x in variables]\n        return feats","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"73d35180085f3c50cc6a4872db8045a5d7e816f2"},"cell_type":"markdown","source":"Some of the light curves in the training set should already look familiar - this is a type Ia supernova - like SN1998bu in the animation. "},{"metadata":{"nbpresent":{"id":"84797026-4473-47dd-ac31-082c0759dca5"},"slideshow":{"slide_type":"fragment"},"trusted":true,"_uuid":"f1e6e556a281bfdf031befdd280f99a970d17595"},"cell_type":"code","source":"lc = LightCurve('../input/plasticc-astronomy-starter-kit-media/fake010.csv')\nlc.plot_multicolor_lc()","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"33e7e365824ece90775ad65aa95f2c523ec65e31"},"cell_type":"markdown","source":"#### *Figure 15: The light curve of FAKE010 - a supernova, Credit: Richard Kessler (University of Chicago), <a href=\"http://adsabs.harvard.edu/abs/2010PASP..122.1415K\">SNPhotCC</a>*"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"a3ee1ba41ecb448fc2dd23a29fdadfe97d70ae09"},"cell_type":"markdown","source":"You might look at something like this light curve, and try to estimate something like a full-width at half maximum to determine a time-scale. "},{"metadata":{"nbpresent":{"id":"d593a449-c884-4790-a4eb-cc89872764d6"},"slideshow":{"slide_type":"slide"},"_uuid":"7e6548682b93e7014e73f1476e0e29f587779f59"},"cell_type":"markdown","source":"That wouldn't be a bad idea for type Ia supernovae, but of course not everything in the dataset is a supernova..."},{"metadata":{"nbpresent":{"id":"12fe61b3-b5f8-4cc2-8f58-eb930306f950"},"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"798d37b57d78c03c6d9cab1154619efa51e0ab06"},"cell_type":"code","source":"lc = LightCurve('../input/plasticc-astronomy-starter-kit-media/fake030.csv')\nlc.plot_multicolor_lc()","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"12baf7f8a26cb9a3977ae0d65f91fcecd0a5e248"},"cell_type":"markdown","source":"#### _Figure 16: The ligt curve of FAKE030 - looks a bit more messy. Credit: Abhijit Saha (NOAO)_\n\nThis object for instance is something very different - it's a variable star. It's got a characteristic time scale, but you wouldn't know it from a simple light curve plot."},{"metadata":{"nbpresent":{"id":"3ff5aa2b-f582-44ab-9552-54c6f9c512ba"},"slideshow":{"slide_type":"slide"},"_uuid":"c6a6543c2adf80f64cf4f3fc71ab2f2f5726ac85"},"cell_type":"markdown","source":"We can use a Fourier technique known as the Lomb-Scargle Periodogram to estimate the characteristic scale, at least given some search range."},{"metadata":{"nbpresent":{"id":"38cc8bd3-7ed0-4fb0-8234-63b9e147ca31"},"slideshow":{"slide_type":"skip"},"trusted":true,"_uuid":"83839944ea309437679fda48fec1b603f73c9e11"},"cell_type":"code","source":"from gatspy.periodic import LombScargleMultiband","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"dee7fd9a-43d4-4233-bfa7-f27cad3a6756"},"slideshow":{"slide_type":"skip"},"trusted":true,"_uuid":"bebd273634013d6d63e6bdb9e3bb982d30471a39"},"cell_type":"code","source":"model = LombScargleMultiband(fit_period=True)\n\n# we'll window the search range by setting minimums and maximums here\n# but in general, the search range you want to evaluate will depend on the data\n# and you will not be able to window like this unless you know something about\n# the class of the object a priori\nt_min = max(np.median(np.diff(sorted(lc.DFlc['mjd']))), 0.1)\nt_max = min(10., (lc.DFlc['mjd'].max() - lc.DFlc['mjd'].min())/2.)\n\nmodel.optimizer.set(period_range=(t_min, t_max), first_pass_coverage=5)\nmodel.fit(lc.DFlc['mjd'], lc.DFlc['flux'], dy=lc.DFlc['flux_err'], filts=lc.DFlc['passband'])\nperiod = model.best_period\nprint(f'{lc.filename} has a period of {period} days')","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"02147e35819e50305154e6b15865a582fe9bc70f"},"cell_type":"markdown","source":"Takes a while eh... you could narrow the search range a bit, if you knew something about the astrophysical class of the object of interest... which is what you want to determine."},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"2bbf384fbccb9aacc52d9b6bf0cb57513fa19111"},"cell_type":"markdown","source":"Still it's a fairly reliable way to extract a period from an irregularly sampled recurring time-series. "},{"metadata":{"nbpresent":{"id":"e3aa63d2-7e78-484a-9538-aa46909fc1a6"},"slideshow":{"slide_type":"slide"},"_uuid":"162896d4f0ead46a00619f40a5df7b130c63636a"},"cell_type":"markdown","source":"Folding the observations to this period takes that noisy time-series and shows you some of the structure in the data. \n\nIf you look at the light curve below and compare it to the hand drawn diagram of variable stars from 1906, bearing in mind that it's period is a bit less than 12 hours, you should be able to classify it pretty easily."},{"metadata":{"nbpresent":{"id":"241a12f0-8c15-42e6-93c0-9c5a1e25dcaa"},"slideshow":{"slide_type":"-"},"trusted":true,"_uuid":"e5f35e83e8908100b4fab66536eb5b358acfae49"},"cell_type":"code","source":"phase = (lc.DFlc['mjd'] /period) % 1\nlc.plot_multicolor_lc(phase=phase)","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"3c72675ee70582cb813e50418cb74e6f443770b9"},"cell_type":"markdown","source":"#### _Figure 17: The \"folded\" light curve of FAKE030, after determining a period using the Lomb-Scargle algorithm_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"f803beaf516fca2229654aed125b0d067a85e4c6"},"cell_type":"markdown","source":"This folded curve is suitable for feature extraction, but we don't a priori know what objects have periodic structure and which ones don't. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"14e1558464886687d28e87608b037e18f54bce52"},"cell_type":"markdown","source":"Also, some classes of objects exhibit stochastic variability - they don't repeat periodically, but they do change more on some time scales than others. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"7f050f880526fb809cd73c9090383ce6fe2282f6"},"cell_type":"markdown","source":"Active galactic nucleii - massive black holes surrounded by hot dust and gas at the centers of galaxies - exhibit this kind of behavior."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"5c7dab213007f3d4811c6904535d17c80aead307"},"cell_type":"markdown","source":"Still, you might look at the RR Lyrae and supernova light curve and conclude that extracting information about shape and the relative strengths of the signal in different passbands is a promising way to encode the information on each object... "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"67c41bf3e6ffab9a6a7a77f277f7650c918aec57"},"cell_type":"markdown","source":"...and would be useful for classification. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"804171cc5098ea169d58a6c8f4ab5a58912c8bf0"},"cell_type":"markdown","source":"And you'd be right! "},{"metadata":{"nbpresent":{"id":"c9695847-347a-4def-a778-9e34fb236a6e"},"slideshow":{"slide_type":"slide"},"_uuid":"c0c1a83618a3ddb67ae624fe79d2bbc3640cbd37"},"cell_type":"markdown","source":"But there's another astrophysical effect that affects the determination of time scales and colors, and this one has nothing to do with the nature of the object - what it is - but rather, *where* it is."},{"metadata":{"nbpresent":{"id":"ad0703e9-0909-488d-a584-4c9de29c3d9a"},"slideshow":{"slide_type":"slide"},"_uuid":"033c1dd5a1ee8afbfa7bd67d3e15fbc74ecedca7"},"cell_type":"markdown","source":"## 4. Primer cont'd - Metadata Provided with Light Curves\n<a id=\"section4\"></a>"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"deb1ed1711fd69c183cc31cda0012562f340b092"},"cell_type":"markdown","source":"Real astrophysical data is from sources on the sky, which means that we normally have contextual information in addition to the light curve to help with classification - if a source is near a galaxy or if it is isolated for instance."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"bdfd97b4d689963adc5df5db206d1abe600d9500"},"cell_type":"markdown","source":"With PLAsTiCC, since we're simulating the time series, we don't have the full breadth of this contextual information. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"c8fe09441b70dc48f122f847ed1690e51a5003b5"},"cell_type":"markdown","source":"We can't really simulate it from previous surveys, because none of them achieve the same sort of signal-to-noise that LSST will - one of the big motivations for building LSST in the first place.   "},{"metadata":{"nbpresent":{"id":"303d20e7-e53b-4f0e-8c12-75c8b7a5eee1"},"slideshow":{"slide_type":"slide"},"_uuid":"2693e940ebed4ee607e3c5a705f56f396b64d232"},"cell_type":"markdown","source":"While we can't give you all the contextual information, we do provide some metadata - a few essential elements that you _may_ want to use to augment the features extracted directly from the light curves (this has pluses and minuses). \n\nLet's start by taking a look at the metadata table (a stub of which is provided with the starter kit)"},{"metadata":{"nbpresent":{"id":"f1550ad8-9777-4ab1-8ad6-7551434a4e5b"},"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"aa3899263eaa7645767d4bd9154251e12d7c1fff"},"cell_type":"code","source":"header = Table.read('../input/plasticc-astronomy-starter-kit-media/plasticc_training_set_metadata_stub.csv', format='csv')\nheader","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"3dbbfc53-6fda-4ffb-b90a-b09bc4ddb92d"},"slideshow":{"slide_type":"-"},"_uuid":"cca693c77411444067809473fc32cc19f5933780"},"cell_type":"markdown","source":"That's a fair number of fields. `object_id` is a simple integer ID number that uniquely identifies each object, and corresponds to the object's ID in the light curve files, but the rest might not be familiar."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"49c5e3ab7f9390d3b9d18ffc85323b2b350cf5b9"},"cell_type":"markdown","source":"### 4a. Coordinates on the Sky\n<a id=\"section4a\"></a>\n\nThe next few columns of metadata are related to an object's position in our three spatial dimensions."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"32cb2922cbd6122aa3544a478e3405e189ac9c1e"},"cell_type":"markdown","source":"We use a spherical coordinate system on the sky, with the positions of stars indexed by a two-tuple that describes where they were on the 1st of January in the year 2000."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"c22386d37c8831d81d94699c3080a4909202651d"},"cell_type":"markdown","source":"These two angles are called the \"Right Ascension\" (R.A.) and the \"Declination\" (often just Dec.) corresponding to longitude and latitude on the Earth. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"b9c5bbd97f606318ee5ca2e3c024fdb9ed777fe0"},"cell_type":"markdown","source":"These coordinates are supplied in the `ra` and `decl` columns of the metadata table."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"5d89cecfe60bcb204bffa57a509ae7e5929c7e56"},"cell_type":"markdown","source":"Wiki has a good illustration of this coordinate system:\n<center><img src=\"https://storage.googleapis.com/kaggle-datasets/52432/98042/Celestial_Coordinates.gif?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1538263082&Signature=cjgIOaVBICGMHskVyXJz%2FgwKcrpDyurWP2yWIi6i4CdmhjDjLk2QnrOO%2FJhhAGH5wI6g7HFo3YCUd2hrvRDc4ZMBHSXSS1n6hU%2FNgSfsRa4RQw7%2FPMylQPIQLCegx395nx8%2F%2FbSzXxTmgmekzWMJWI4GEAYzz2rymJPMa93li%2B8Vaswl4tco7o03ar0dpprbDF3Mwxb8cA%2FfqnuSw6agO4b5FXMYcx8t53UQ%2BbuhC1XDEe%2BwARif4tLg1NIHilvxTIxaWj9S8CFWIrlS%2F%2Botc1lxXlLbhKTY07UqW3JsfW6s7RA0ZOgrDgP1j%2BCyE2g8otS1Ct4F1DxeF7zCxHhjgw%3D%3D\"></center>"},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"a7b0cb9b514b2f1ce1556b3f05962a28bfab981f"},"cell_type":"markdown","source":"#### *Figure 18: Equatorial coordinates, Credit: https://en.wikipedia.org/wiki/Right_ascension*"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"015cad911a2e645ee38c13af5920c6ad1db0d0a9"},"cell_type":"markdown","source":"If we unroll the sky into a flat <a href=\"https://en.wikipedia.org/wiki/Aitoff_projection\">Aitoff projection</a>, we can get a view of the entire sky:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"720c42bda8ced8a68aa2875844e5d9741c548dc9"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/allsky_equatorial.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"f254b9011f527b340a79ea15f00910f5533d3dd3"},"cell_type":"markdown","source":"#### _Figure 19: The sky in equatorial coordinates with a galactic coordinate grid overlaid_\n\nSplitting the sky into two hemispheres by extending the Earth's equator outwards, the Milky Way forms a graceful arc.\n\nLines of constant Right Ascension and Declination are in grey, while the green lines indicate a coordinate transform, where the plane of the galaxy defines the equatorial plane, instead of the Earth's equator - a galactic coordinate system."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"567fcd542d3b5160db097ba61eef980267969dd6"},"cell_type":"markdown","source":"LSST, based on Chile, views the southern sky. Comparing how the galaxy looks in this projection should help make sense of the otherwise weird looking pattern of observations that LSST plans to obtain:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"67e0667a1fd26ad19b5df827fc4b3ad179badeb5"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/skymap_minion1016.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"14ebbacf5b7ff803a363e611997f25442a02bf44"},"cell_type":"markdown","source":"#### _Figure 20: The LSST \"minion 1016\" exposure map in celestial coordinates. Credit: LSST Corporation_"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"c7555b62c3a9b87efb2a6185e3ce29e39e9d6296"},"cell_type":"markdown","source":"PLAsTiCC simulates the \"Wide-Fast-Deep\" Survey - the main LSST program, and the \"Deep Drilling Fields\" - a few small regions where LSST will get more observations for specialized scientific studies. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"bf7b8d874372625eab065bcf5db224e3c84e8efb"},"cell_type":"markdown","source":"Which objects are in the WFD vs the DDF are specified by the `ddf_bool` column in the header (1 if in DDF, 0 otherwise). "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"25141e67effa1e021fa33eff779390d12304b35f"},"cell_type":"markdown","source":"Note that you can also get this information from the `object_id` column (`object_id` < 1,000,000 are in DDF)."},{"metadata":{"nbpresent":{"id":"d3f1260b-b72c-4fa6-9bc0-20e0b852e673"},"slideshow":{"slide_type":"skip"},"_uuid":"9e6135fc7bb8cc4f79a7295c907daea370dbb540"},"cell_type":"markdown","source":"The DDFs will have more observations, but cover a much smaller chunk of sky. Consequently, the light curves will have better sampling, and potentially be found farther away in our Universe, but there's fewer objects. You may wish to treat the DDF and WFD objects separately and build different sorts of classifiers for them. This is entirely up to you, and the two sets of objects are weighted exactly the same, irrespective of which field they originated from."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"de0f02f7c16efde14c86e882c3170df5e8371b44"},"cell_type":"markdown","source":"### 4b. Galactic Coordinates\n<a id=\"section4b\"></a>\n\nRA and Dec are \"equatorial\" coordinates - a coordinate system where the Earth's equator is extended outwards into infinity and splits space into an imaginary north and south hemisphere."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"5f0a0a2de88208910c3f3c13951cfe2a671f7634"},"cell_type":"markdown","source":"This coordinate system makes sense if you are building a telescope on Earth in the southern hemisphere and have to decide how you want to observe the sky."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"ae721a463b568246cd35260fa3a42f693e0bf3ef"},"cell_type":"markdown","source":"But, if we're talking about objects in the distant Universe, it makes sense to not make the Earth the center of it."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"33f3ac65718c7fe05af0e0065eb8075f6cbee176"},"cell_type":"markdown","source":"We can rotate the sky so that the plane of our galaxy defines the equator, like so:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c8bdf5e274000f305c6ee7272b4d9f5b94891ba3"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/allsky_galactic.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"9b10bceed2e317559d69eae5779b5cca1059afc3"},"cell_type":"markdown","source":"#### _Figure 21: The sky in galactic coordinates with a equatorial coordinate grid overlaid_\n\nwhere I'm leaving the celestial coordinate grid (in yellow - notice how they no longer correspond to the North and South relative to the plane of our Galaxy)."},{"metadata":{"nbpresent":{"id":"580fe614-1d41-41ff-861c-311e5a39f014"},"slideshow":{"slide_type":"fragment"},"_uuid":"76ac358a1eeabeedbc91f9953d589eac15b4799b"},"cell_type":"markdown","source":"If you prefer to work with galactic coordinates instead of equatorial coordinates (`ra` and `decl`), these are provided in the header in the columns `gall` and `galb`."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"f296db93f27c7b59540c2623dd940f3e95fbd07c"},"cell_type":"markdown","source":"### 4c. Redshift\n<a id=\"section4c\"></a>\n\nThe position of a source on the sky is a tuple of two numbers (in either equatorial or galactic coordinates), but we live in a Universe with (at least) 3 spatial dimensions. The third number we use to locate objects in 3D space isn't even a distance from the origin - it's a speed! We call it the \"redshift.\"\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"442a16ae7c1dad14b65a83968121b1f6fac2db96"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/redshift.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"76cbe08e160d4e561509ddb1d0a85763dd60d3fd"},"cell_type":"markdown","source":"#### _Figure 22: \"Redshift\" is a measure of how fast a distant source (with more distant on the top) is moving relative to us along the line of sight, and is determined from how much the wavelength (or alternately, the frequency) of the absorption and emission lines of atoms change, relative to what we see here on Earth (bottom panel)_"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"74de9b0b9e6a0c8f96f304ccfa2d0456970d611a"},"cell_type":"markdown","source":"Redshift is so named because if you take the light from distance astrophysical sources and pass it through a prism, you see the same atomic fingerprint of elements and molecules you find here on Earth... just shifted."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"a0fe904181f5429833e0874b5c55ab5d90e6e975"},"cell_type":"markdown","source":"If the source is moving towards us, things are shifted to the blue (or higher frequencies), while if the source is moving away from us, things are shifted to the red (or lower frequencies). "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"f3d27219293fb650d1939aa39cac568625dce3a2"},"cell_type":"markdown","source":"This isn't all that different from the Doppler shift of the sound of an ambulance or police siren moving across you - it'll increase in frequency as the vehicle approaches you, and decrease in frequency as it moves away. "},{"metadata":{"nbpresent":{"id":"f2a31eb0-95f9-49fd-8d2c-2de39fe21204"},"slideshow":{"slide_type":"fragment"},"_uuid":"ca98fac4fbf12b6a2df58207d1163b7af0dea5d7"},"cell_type":"markdown","source":"You can convert the change in frequency or wavelength into a speed using:"},{"metadata":{"nbpresent":{"id":"43e3daf6-c170-4d31-9bca-e146ce7557d4"},"slideshow":{"slide_type":"-"},"_uuid":"95216962f3fb8d186e74883bcc9ac09e235980a7"},"cell_type":"markdown","source":"$$z = \\frac{\\lambda_{\\text{Source}} - \\lambda_{\\text{Earth}}}{\\lambda_{\\text{Earth}}} = \\frac{\\Delta \\lambda}{\\lambda}  \\sim \\frac{v_r}{c}$$"},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"d08476962152687c7a6fd9b4481650e9a5137336"},"cell_type":"markdown","source":"where $z$ is the redshift, $\\lambda_{\\text{Earth}}$ is the original wavelength we measure on Earth, and $\\Delta \\lambda$ is the change in wavelength, $v_r$ is the recession velocity and $c$ is the speed of light."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"88c71d5aa233598186f185e07053ccceaf79b302"},"cell_type":"markdown","source":"## _<center> AN IMPORTANT NOTE </center>_\n\nFor historic reasons, redshift is denoted with the letter $z$. This is also the same character as the z passband in LSST (and other surveys), but the two aren't related."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"1a3cf13a17003a535aeedac46570d4677dfee493"},"cell_type":"markdown","source":"There's a problem though - to measure this accurately we need _spectroscopy_. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"0a4917aa3f8234eee22037eaebdb99550b62f9c0"},"cell_type":"markdown","source":"The same spectroscopy that is expensive and hard to get to classify transients, which is the raison d'être for PLAsTiCC in the first place..."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"3079e76b664db8f10e84ed01ee16b86120a71a43"},"cell_type":"markdown","source":"But just as with this challenge, there's a solution that involves getting images through broad passbands and extracting ratios of brightness in different passbands to get an estimator of the redshift. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"29ecfaa07047acd5cc8089b94a30d0b8b8ad6567"},"cell_type":"markdown","source":"Therefore, in the header, you get two redshifts! `hostgal_specz` is the spectroscopic redshift of the host-galaxy of an event, while `hostgal_photoz` is the photometric redshift of the host-galaxy. "},{"metadata":{"nbpresent":{"id":"e511216f-a8ad-4ff6-a9da-956499318fe7"},"slideshow":{"slide_type":"skip"},"_uuid":"fd57ed84e87945d2fc019e7f6bef3daabe9e514c"},"cell_type":"markdown","source":"## _<center> ANOTHER IMPORTANT NOTE </center>_\n\nThe training set has spectroscopic redshifts for all sources. You'll notice quickly that they don't always agree with the photometric redshift. The photometric redshift is sometimes wrong. Consequently, we also give you an estimate of the 68% uncertainty on the photometric redshift (which itself might be inaccurate...) - `hostgal_photoz_err`. Be warned though that the actual photometric redshift distribution is asymmetric, and the redshift is not likely to be less than zero, so you might want to treat it as just a weight. Why is the redshift unlikely to be less than 0... we'll get to that in a bit.\n\nThe test set does not have spectroscopic redshifts for all sources. There's a few because we simulate occasionally getting lucky and finding a new source in a galaxy which already had spectroscopy, but the vast majority of them will be `NaN`. You could potentially try to use the training set, together with the subset of the test data that does have both spectroscopic and photometric redshifts to make a better redshift predictor if you like, or you could ignore it altogether.\n\nThat said, redshift is a great feature! We know how to separate extragalactic and galactic sources pretty well, so we simulate this ability by setting the redshifts of galactic sources to exactly `0`. That tells you they are within the galaxy and reduces the size of the possible models they might be, but doesn't identify which is which - that's still your challenge! Most machine learning techniques will pick this feature of redshift up easily, but you might use it to build separate classifiers for extragalactic and galactic sources if you like."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"bc93ff52d0619b2210d01ab9aec78f0fa41e9145"},"cell_type":"markdown","source":"### 4d. A view of our cosmic neighborhood\n<a id=\"section4d\"></a>\n\nYou might already have noticed the fuzzy blob in the Southern hemisphere, below and to the right of center of our galaxy in galactic coordinates."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"31ebebd9145650513433948532c0207036859aea"},"cell_type":"markdown","source":"You can get a clearer view of it from ESA's GAIA satellite without a coordinate grid on top. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"b155c746ba933ba545d25d8a6fce77d0bd9ce824"},"cell_type":"markdown","source":""},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4960a27eda3382c0a791cb80eceb3d72f2be3fd9"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/Gaia_milky_way.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"5ea4106db7388ceb4674997d6df08fcec31bd965"},"cell_type":"markdown","source":"#### _Figure 23: The GAIA view of our Galaxy, Credit: ESA/GAIA_\n\nThis fuzzy blob below and to the right of center is called the Large Magellanic cloud - a companion galaxy to our Milky Way. \n\nThe other fuzzy blob near it is called the Small Magellanic Cloud. You can see it with your own eyes from a dark site in the southern hemisphere. "},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"7bf0f3b4c921f6b3f8d2b859dc352e659b87a066"},"cell_type":"markdown","source":"(You might even be able to spot some galaxies in the northern hemisphere, but remember the sky in galactic coordinates doesn't look like the sky you'd see from Earth)"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"22bf102728dd08307a7d58cc21c603a5d229a5d1"},"cell_type":"markdown","source":"The redshift of the Large Magellanic Cloud is a whopping 0.000875 - i.e. the wavelength of light hasn't exactly changed much. \n\nThat's about 163 miles/second (or 262 km/s) - fast, but less than a thousandth of the speed of light."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"92953af08458f00ecc5366ebbaf104a1a39b1383"},"cell_type":"markdown","source":"If we color the galaxies in the nearby Universe that we can see from Earth by redshift with a small telescope, there are many sources that we can see with higher redshifts. \n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7b68303d2b3e1720c7aa7adf8fb450fb78fbabb0"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/large_scale_structure.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"e53e5dfcfc102e745f0d4a338fb2057e59efa6d6"},"cell_type":"markdown","source":"#### _Figure 24: The low redshift Universe, Credit: Tom Jarrett, IPAC/Caltech_"},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"6a8f0c502e67c055ad14ba88e97bff9ee4a2e73f"},"cell_type":"markdown","source":"Our spectrographs are sensitive enough for it to have been possible to measure these redshifts even a century ago. Many of the redshifts of these nearby galaxies was first measured by Vesto Slipher back in 1917. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"874a45d320d56739562f3328180a5a4e2aedc1ad"},"cell_type":"markdown","source":"### 4e. Cosmology\n<a id=\"section4e\"></a>\n\nIn fact, you might notice that all of the redshifts are positive - almost everything is moving *away* from us, "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"69896a2e1d963297ad1c94dca119d21f89d9a070"},"cell_type":"markdown","source":"In 1927, Georges Lemaître, proposed that this meant the Universe was expanding. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"92b73b8cc7499dfefd37032a53c0cdd706f016d0"},"cell_type":"markdown","source":"The first widely known measurement of this expansion rate came two years later, from Edwin Hubble who showed definitely that the further galaxies were from our own, the faster they were moving away."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"c8dfdfa6bddd0a5c0e4113efe49fd0cf61a9908e"},"cell_type":"markdown","source":"Hubble did this by measuring the distances to these galaxies using a special class of variable stars called Cepheids (see that figure from Popular Science Monthly again for an example) that have a relation between their period and how intrinsically bright they are. \n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"256ee294863b64d913439fca7cff577a519eaaa6"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/hubble_cepheid.jpeg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"56812af89a9ef211489e5b8e1d94b71d733eab5d"},"cell_type":"markdown","source":"#### _Figure 25. Hubble's Glass Plate - notice the \"Var!\" in the corner. Hubble was using visual inspection for classification, Credit: Carnegie Observatories_"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"ad31d1d22661d0a25a23b18b37a20b96962b0fc4"},"cell_type":"markdown","source":"Measure the period  of a Cepheid and you figure out how bright they are supposed to be. Measure how bright they appear, and compare the two, and you've got yourself a measurement of the distance."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"69541069f9cbaff542d6435c18e56d10042707b4"},"cell_type":"markdown","source":"And the data at the time suggested that this increase was linear.\n\n$$ v_r = H_0 \\cdot D \\sim c\\cdot z $$\n\nwhere $D$ is the \"luminosity\" distance to the object and the dot ($\\cdot$) operator is a shorthand for the multiplication ($\\times$) operator. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"c8c908ee3e1862da67bf4d4395b09ae8b0d1b796"},"cell_type":"markdown","source":"This is Hubble's plot of his measurements:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0dd11b299e18acea073d8116a0438717253c6bed"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/hubble_law.jpeg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"cc6a407a47bf40b411b17e2dd659b901bb73221c"},"cell_type":"markdown","source":"#### _Figure 26. Hubble's Hubble Diagram showing the Universe expanding, Credit: Edwin Hubble_\n\nWe call this the \"Hubble Law\" and this kind of plot a \"Hubble Diagram\" (well Hubble didn't - he called it \"Figure 1.\")"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"cc54feb57b0fbf470effdcbf461fd48f0741f059"},"cell_type":"markdown","source":"This linear relation between redshift and distance is approximate (hence the $\\sim$ sign)."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"4296fb8ab1f90d5061d836eaef569c87ee15b1ab"},"cell_type":"markdown","source":"As we look to farther horizons, to determine the expansion history of the Universe takes the equations of Einstein's General Theory of Relativity. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"08eb104ac0f5eb636ae0a7274a8884cc56523d80"},"cell_type":"markdown","source":"Measuring this expansion history takes bright sources that we can determine the distance to, like those type Ia supernovae like SN1998bu. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"348a2a1a95d76d807e0ae54f4647d3db2a7c156e"},"cell_type":"markdown","source":"When scientists used type Ia supernovae to do this for the first time in 1998, they discovered something remarkable - that the Universe wasn't just expanding, but the expansion was accelerating.\n\n#### *Something is _pushing_ distant galaxies away from us at an ever increasing rate!*"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"736c7f55a341061f8fbb694ce7043eb3abf10f8f"},"cell_type":"markdown","source":"This is one of the best current map of that expansion using type Ia supernovae:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"bfe309fdba006083eb389ff79c8003a24d331f2d"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/JLA_HubbleDiagram.png\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"319223fec40a446f7eda857c83b3d19ce903ab36"},"cell_type":"markdown","source":"#### _Figure 27. The JLA Hubble Diagram, Credit: Marc Betoule (Paris Center for Cosmological Physics)_\n\nHubble's data covers such a small range that it doesn't even show up on this plot. We've done a lot in 80 years!"},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"993327e7b130c2f9a68769d01ae79b18a5c8e4e5"},"cell_type":"markdown","source":"The teams that discovered this won the Nobel prize for their work in 2011. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"db45ea97373d462c8e4fd497fecb153ed677aaa0"},"cell_type":"markdown","source":"So we've got to identify such type Ia supernovae among all of the ones that LSST will find... hopefully things are beginning to fit together, and this is giving you a sense for the big picture!"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"bff3d5f4a95e38945826b5204ce8d58348bd8b07"},"cell_type":"markdown","source":"You might prefer to have the y-axis - called the distance modulus - related to the base 10 logarithm of the luminosity distance, rather than the redshift provided in `hostgal_photoz`. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"8230008f3c5199a39cfb1f921ce78ac2d8e0b6a4"},"cell_type":"markdown","source":"For your convenience, this is provided in the header as a column named `distmod`. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"0527b1b3ba90ff0bf40adbcd030aca1310bdac49"},"cell_type":"markdown","source":"For galactic objects with `hostgal_photoz`$ = 0 $, the `distmod` is reported as `NaN` (the distance would be 0, and taking the logarithm of 0 is a bad idea)."},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"bcf7ff32465ac8ec3aae59dbc5ff0007725405a9"},"cell_type":"markdown","source":"We don't provide another column for distance modulus that was derived from `hostgal_specz` but there's a lookup table included with the dataset if you need to compute this (you probably don't). This is a purely computed quantity and is provided for convenience. It's not a measured distance to each source - actually measuring distances is complicated. You can watch a video about how we do it <a href=\"https://ed.ted.com/lessons/how-do-we-measure-distances-in-space-yuan-sen-ting\">here</a>. \n\nBecause the speed of light is fixed, and the distance to sources is increasing, you can also compute a how long it took for light from a source at some redshift took to reach up. This is also in the lookup table. You also don't need this, but it is cool to get a sense for the history of our Universe. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"63bf41b88aac441d63a45daf78288e18491afa1e"},"cell_type":"markdown","source":"General relativity in the expanding Universe gives us a time machine. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"045bbb6ed3c0130411ad416fd36564ae3cdc2473"},"cell_type":"markdown","source":"As we look farther our, we look further back in time. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"d3d4b3d80fa63038d0133f940effe11cc13cda29"},"cell_type":"markdown","source":"Here's a visual timeline of the history of our Universe:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ee0bd24b08036bbeb81cacfe5e961638d845aded"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/UniverseTimeline.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"bb327c025664eeb8a80ce2f12c954218aa5ae07b"},"cell_type":"markdown","source":"#### _Figure 28. A Timeline of our Universe, Credit: BBC Two, Stargazing Live, Public access NASA images_"},{"metadata":{"nbpresent":{"id":"cfeed02f-c105-4d0e-91ea-2fc31e2c0502"},"slideshow":{"slide_type":"skip"},"_uuid":"97e2150924f446557bb7c7c4546fdbfb1918087d"},"cell_type":"markdown","source":"Now looking at Figure 25 and the equation above it, you might notice that sources can have a redshift of greater than one i.e. appear to be moving faster than the speed of light! And you might have heard that it isn't possible to go faster than the speed of light. This is how the cosmological redshift differs from the Doppler effect. It's not the physical motion of the sources that is causing the change in frequency, but rather the expansion of the Universe itself - that the space between us and the source is stretching! Your PLAsTiCC dataset will include objects up to a redshift of 3. \n\nIf everything seems to be moving away with respect to us on Earth, you might also be tempted to think we're the center of the expansion of the Universe. We aren't. We could move to any other point in the Universe and we'd see the same behavior. This is a counterintuitive consequence of Einstein's General Theory of Relativity, and living in a Universe that is homogeneous (same from all positions) and isotropic (same in all directions). If you find this stuff fascinating, you might want to read <a href=\"https://www.forbes.com/sites/startswithabang/2018/04/17/the-most-important-equation-in-the-universe/#8869ba60da5c\">this.</a>"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"e2876c471031a2e784b92597a18f94c04c676bf0"},"cell_type":"markdown","source":"### 4f. Dust\n<a id=\"section4f\"></a>\n\nDepending on where you are looking on the sky, there's a lot of structure in our Milky Way - lanes of gas and dust obscure the view of stars and galaxies behind it."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"22bb673d7b20464b52afc0d187e73ae22fda8311"},"cell_type":"markdown","source":"This becomes even more apparent if we look with a different instrument and in a different passband, more sensitive to only the dust. "},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"0f587f9996a61df052e47122afef89cf4e11b1dd"},"cell_type":"markdown","source":"This view from ESA's Planck satellite really drives that home:\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"aa2d6410c9fd7de0aad882913229a968c2140ae1"},"cell_type":"code","source":"displayimage(\"../input/plasticc-astronomy-starter-kit-media/dust_map.jpeg\")","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"-"},"_uuid":"72928ff814b37a48d4f0abe493743a4eafc68251"},"cell_type":"markdown","source":"#### _Figure 29: A dust map of the Milky Way from Planck, Credit: ESA / NASA / JPL-Caltech_"},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"0f6256efcaa026eff90cfa21e606562e3c588e8a"},"cell_type":"markdown","source":"Of course, the dust isn't a single layer, and has 3D structure. \n\nAstronomers can map this structure using careful observations of lots of stars."},{"metadata":{"nbpresent":{"id":"428d720b-4e26-4f3c-b49a-4fbeef165c0b"},"slideshow":{"slide_type":"slide"},"_uuid":"a5d9143b8e43812c3450cfed02ed4375beea4b3d"},"cell_type":"markdown","source":"This 3D map was made with the Pan-STARRS telescope which has a 1.2 gigapixel camera and a similar field-of-view to LSST, but is a much smaller telescope."},{"metadata":{"nbpresent":{"id":"3bc10b72-7d54-4d07-860c-ab7410ca7070"},"slideshow":{"slide_type":"-"},"trusted":true,"_uuid":"c07e8b54c6e9a40c0fd1565d2b6f5a593fca70ab","_kg_hide-input":true},"cell_type":"code","source":"video = io.open('../input/plasticc-astronomy-starter-kit-media/MilkyWayDust.mp4', 'rb').read()\nencoded = base64.b64encode(video)\nHTML(data='''\n<video width = 60% controls>\n   <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n</video>'''.format(encoded.decode('ascii')))","execution_count":null,"outputs":[]},{"metadata":{"nbpresent":{"id":"f4ef7cfa-3744-4329-b2f4-bf804ea36f73"},"slideshow":{"slide_type":"-"},"_uuid":"1b5663fc4f0b8487747ffc82e6df00482e49d6cf"},"cell_type":"markdown","source":"#### _Figure 30: A Fly Through of our Galaxy's Dust, Credit: Gregory Green (Stanford/KIPAC)_"},{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"abbec65a83f0ef5f1ad21e6e6d5f0e882f22954c"},"cell_type":"markdown","source":"Dust, like increasing luminosity distance (and therefore redshift) in our Universe, makes things fainter and redder. We're all used to this effect - we see it with our Sun and Moon everyday. Our atmosphere scatters blue light more than the red light, making the sun and the moon appear more red than if we could see them from space. As the sun gets near the horizon and the light from it has to pass through more of the Earth's atmosphere to reach us, it appears even redder than when it is high in the sky. \n\nWhile the light curves we've provided are corrected for the dimming and reddening - called the extinction - along the line of sight in our Galaxy, this correction isn't perfect, because we've not got perfect knowledge of how much dust there is along any line of sight. You'll notice the map in the video is higher resolution towards the center of the galaxy where there are more stars, and lower resolution in the opposite direction for instance.\n\nAdditionally, extragalactic sources can also suffer from extinction _in their own host galaxies._ Galactic sources are nearer us and brighter, and because they are _in_ our Galaxy, they are likely to be in the plane. All said then, you should expect to find that Galactic sources have a different sort of distribution of dust along the line of sight to them than extragalactic sources."},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"08cb87505e826443f605e1ea8ac86e1f61998bcc"},"cell_type":"markdown","source":"The correction we applied to the light curves can be described by a single number for our Milky Way."},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"6a00f7812d37808b07309bf50ec46dca070789e7"},"cell_type":"markdown","source":"This is called $\\text{MW}~E(B-V)$ (MW = Milky Way, E = Extinction between the Blue and Visual, or B-V) and reported in the header in the column named `mwebv`. "},{"metadata":{"slideshow":{"slide_type":"fragment"},"_uuid":"e3976a75a97e2a992bce6ca9e0612bca783c54b1"},"cell_type":"markdown","source":"You will not need to correct the light curves anymore, but you might be able to use this piece of metadata to distinguish some populations from others using clustering techniques for instance.\n\n***"},{"metadata":{"slideshow":{"slide_type":"slide"},"_uuid":"5764ba6918b920d8bc6d566aeac905d20c85d642"},"cell_type":"markdown","source":"That's it! Phew! It's the minimal bits of the astrophysics you may need to make sense of the data and start to tackle this challenge. \n\nThis Kernel does not need the actual challenge data.\n\nIn a companion Kernel, `plasticc_classification_demo`, we'll build a toy classifier on the challenge data, and introduce some of the useful packages astronomers use to tackle this problem.\n\n***"},{"metadata":{"nbpresent":{"id":"4f1e8284-d413-43fa-a649-c1c9fb3927b8"},"slideshow":{"slide_type":"fragment"},"_uuid":"36f6b962455a0234b0855002596714d350ba9d9b"},"cell_type":"markdown","source":"If you have more questions, you can find the PLAsTiCC dev team on the Kaggle forums, and we'll do our best to answer your questions if we can. We hope you have fun and we look forward to your submissions - remember, your work will influence the entire LSST project. \n\nCome help us explore the Universe!"},{"metadata":{"nbpresent":{"id":"4b2c29b8-d895-4495-8301-23bf16f0ff7f"},"slideshow":{"slide_type":"fragment"},"_uuid":"97976cba0a810ee6dab16c0d3c8b6330909894d6"},"cell_type":"markdown","source":"/cheesy"}],"metadata":{"anaconda-cloud":{},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"livereveal":{"scroll":true},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":true,"toc_position":{},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":1}