{"cells":[{"metadata":{"_uuid":"ec8b9c783481d8924c50adccadc613333dc37d31"},"cell_type":"markdown","source":"*Warning : This is an experimental kernel. There are wrong information.*\n\n*Note: https://www.kaggle.com/danilodiogo/the-astronomical-complete-eda-plasticc-dataset convert to python code*"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd #Analysis \nimport matplotlib.pyplot as plt #Visulization\nimport seaborn as sns #Visulization\nimport numpy as np #Analysis \nfrom scipy.stats import norm #Analysis \nfrom sklearn.preprocessing import StandardScaler #Analysis \nfrom scipy import stats #Analysis \nimport warnings \nwarnings.filterwarnings('ignore')\n%matplotlib inline\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport plotly.figure_factory as ff\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_absolute_error\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63fd35e62401d200466ba77ebfb1cdc60c0c327c"},"cell_type":"markdown","source":"- `[training/test]_set_metadata`: Information about objects in the that doesn't change over time, like the coordinates of the object.\n- `[training/test]_set`: Time series of observations of the objects. Maps to the metadata via object_id."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/training_set.csv\")\ntrain_meta = pd.read_csv(\"../input/training_set_metadata.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"921cb18b777c044a22c30efe8a2a705ecbbe0cac"},"cell_type":"markdown","source":"### Metadata File Column Descriptions"},{"metadata":{"trusted":true,"_uuid":"6eb2f9119093193e416466a409fba862da59db10"},"cell_type":"code","source":"train_meta.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cef32f07890f09523807c0b6319841abf8d303b7"},"cell_type":"markdown","source":"- `object_id`: unique object identifier. Int32\n- `ra`: right ascension, sky coordinate: co-longitude in degrees. Float32\n- `decl`: declination, sky coordinate: co-latitude in degrees. Float32\n- `gal_l`: galactic longitude in degrees. Float32\n- `gal_b`: galactic latitude in degrees. Float32\n- `ddf`: A flag to identify the object as coming from the DDF survey area (with value DDF = 1 for the DDF, DDF = 0 for the WFD survey). Note that while the - DDF fields are contained within the full WFD survey area, the DDF fluxes have significantly smaller uncertainties. Boolean\n- `hostgal_specz`: the spectroscopic redshift of the source. This is an extremely accurate measure of redshift, available for the training set and a small fraction of the test set. Float32 (적색편이)\n- `hostgal_photoz`: The photometric redshift of the host galaxy of the astronomical source. While this is meant to be a proxy for hostgal_specz, there can be large differences between the two and should be regarded as a far less accurate version of hostgal_specz. Float32\n- `hostgal_photoz_err`: The uncertainty on the hostgal_photoz based on LSST survey projections. Float32\n- `distmod`: The distance to the source calculated from hostgal_photoz and using general relativity. Float32\n- `mwebv`: MW E(B-V). this ‘extinction’ of light is a property of the Milky Way (MW) dust along the line of sight to the astronomical source, and is thus a function of the sky coordinates of the source ra, decl. This is used to determine a passband dependent dimming and redenning of light from astronomical sources as described in subsection 2.1, and based on the Schlafly et al. (2011) and Schlegel et al. (1998) dust models. Float32\n- `target`: The class of the astronomical source. This is provided in the training data. Correctly determining the target (correctly assigning classification probabilities to the objects) is the ‘goal’ of the classification challenge for the test data. Note that there is one class in the test set that does not occur in the training set: class_99 serves as an \"other\" class for objects that don't belong in any of the 14 classes in the training set. Int8"},{"metadata":{"_uuid":"56fcaf1af13f3d5770c558aa13cf49358d85d241"},"cell_type":"markdown","source":"![](https://upload.wikimedia.org/wikipedia/commons/9/98/Ra_and_dec_on_celestial_sphere.png)\n\n**Right ascension (abbreviated RA; symbol α)** is the angular distance measured only eastward along the celestial equator from the Sun at the March equinox to the (hour circle of the) point above the earth in question. When paired with **declination**, these astronomical coordinates specify the direction of a point on the celestial sphere in the equatorial coordinate system."},{"metadata":{"_uuid":"0315ef654dacba87455e72daae3557777c54d6fb"},"cell_type":"markdown","source":"![](http://upload.wikimedia.org/wikipedia/commons/2/2a/Galactic_coordinates.JPG)"},{"metadata":{"_uuid":"50c82bb933012106bf0cd6cdba3b8335ae7d74dd"},"cell_type":"markdown","source":"**Galactic longitude (l)** is measured with primary direction from the Sun to the center of the galaxy in the galactic plane, while the **galactic latitude (b)** measures the angle of the object above the galactic plane."},{"metadata":{"_uuid":"25274e714fa5eca025f50772e47194d4dcc9c283"},"cell_type":"markdown","source":"**redshift** : when light or other electromagnetic radiation from an object is increased in wavelength, or shifted to the red end of the spectrum. In general, whether or not the radiation is within the visible spectrum, \"redder\" means an increase in wavelength – equivalent to a lower frequency and a lower photon energy, in accordance with, respectively, the wave and quantum theories of light.\n![](https://upload.wikimedia.org/wikipedia/commons/thumb/0/0a/High-redshift_galaxy_candidates_in_the_Hubble_Ultra_Deep_Field_2012.jpg/800px-High-redshift_galaxy_candidates_in_the_Hubble_Ultra_Deep_Field_2012.jpg)\n\nResource : https://en.wikipedia.org/wiki/Redshift\n\n**spectroscopic redshift** vs **photometric redshift** : \n\nResource : https://physics.stackexchange.com/questions/169734/photometric-surveys-vs-spectroscopic-surveys\n\n**LSST** : \nhttps://www.lsst.org/scientists/survey-design\n"},{"metadata":{"trusted":true,"_uuid":"4ec83cfc95e39d8b0355b0729f250512a2fd4c8a"},"cell_type":"code","source":"print(\"unique values of object_id:\",len(train_meta['object_id'].unique()),\"\\nshape of train_meta:\",train_meta.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a5aebc5ffdc04505058d22a47a27364418c5a3c"},"cell_type":"code","source":"#missing data\ntotal = train_meta.isnull().sum().sort_values(ascending=False)\npercent = (train_meta.isnull().sum()/train_meta.isnull().count()).sort_values(ascending=False)\nmissing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)\n\n#histogram\n#missing_data = missing_data.head(20)\npercent_data = percent.head(20)\npercent_data.plot(kind=\"bar\", figsize = (10,8), fontsize = 15)\nplt.xlabel(\"Columns\", fontsize = 20)\nplt.ylabel(\"Percent of Missing Value (%)\", fontsize = 20)\n#plt.title(\"Total Missing Value (%)\", fontsize = 20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8ecc35038d973a00447efff0cf96e53bd62be6d"},"cell_type":"markdown","source":"- `distmod`: The distance to the source calculated from hostgal_photoz and using general relativity. Float32\n- `hostgal_photoz`: The photometric redshift of the host galaxy of the astronomical source. While this is meant to be a proxy for hostgal_specz, there can be large differences between the two and should be regarded as a far less accurate version of hostgal_specz. Float32"},{"metadata":{"trusted":true,"_uuid":"9594a5442a52ca85342cb579cbd45c5a8a8226f4"},"cell_type":"code","source":"train_meta[train_meta['distmod'].isnull()].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"165e2acf64d3690bf7d4fabc17608e0aa8df8049"},"cell_type":"markdown","source":"- hostgal_specz, hostgal_photoz, hostgal_photoz_err have no value. it is mean that the object has no redshift value. so don't calculate distmod."},{"metadata":{"_uuid":"81d325072afa8f79373f2a69e8c2349fe5ef02ae"},"cell_type":"markdown","source":"Comment in the https://www.kaggle.com/danilodiogo/the-astronomical-complete-eda-plasticc-dataset , Danilo da Silva makes the variable `Milky Way` because the dataset introduction file says \"the given redshift for objects in our own Milky Way galaxy is given as zero\"."},{"metadata":{"trusted":true,"_uuid":"722b67114b88476c0610bd8b4f44cea9931ede39"},"cell_type":"code","source":"pd.isna(train_meta['distmod'].iloc[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d99d02951cfd19d3ab425aece0cea760da92e255"},"cell_type":"code","source":"train_meta['Milky.Way'] = train_meta[\"distmod\"].apply(lambda x: 1 if pd.isnull(x) == True else 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"366df4fc139a669cff438fca6929dfd333b9fa74"},"cell_type":"code","source":"train_meta.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"94a705d9fc2a71f75370994ee1b2bcc792843646"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\ntrain_meta['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e683db78067dc31c2a6cbf62849e2e4f779dd070"},"cell_type":"markdown","source":"Target value is uneven. it seems class imbalance. I wonder if I should use sampling here."},{"metadata":{"trusted":true,"_uuid":"822d9c001b8154426279c446a1b9b23eb50f03e0"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, hue = 'target',height=8)\ng.map(plt.scatter, 'gal_l','gal_b')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"48fabe1a0c1025e367d688738c56872ef65e91e0"},"cell_type":"markdown","source":"Whale...?"},{"metadata":{"trusted":true,"_uuid":"3723c3860689143ca0151e581b406ca1772f9093"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, col = 'target',col_wrap=4)\ng.map(plt.scatter, 'gal_l','gal_b')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"95bb289f5d88637fd0a05e1cea01e4a41b322bdc"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, hue = 'target',height=8)\ng.map(plt.scatter, 'ra','decl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1d53ac5c1a6925a97709a37e6e8d94c05ebf833"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, col = 'target',col_wrap=4)\ng.map(plt.scatter, 'ra','decl')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f286bcb2814b9e72901016758e045e89989243d"},"cell_type":"markdown","source":"![](https://steemitimages.com/DQmQR7j1yGHYhP1sheeGprjyYEiYG7dSfft1z1jJbHpsy6v/9.PNG)"},{"metadata":{"trusted":true,"_uuid":"9a18dd447d4e3a3851b6348c80cac0cb488eab6e"},"cell_type":"code","source":"np.sin(349.046051 * np.pi / 180)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0edd66541c65506d9e4ecc8f003f72320f67580"},"cell_type":"code","source":"train_meta['skycoordinate_x'] = train_meta[\"distmod\"] * train_meta[\"decl\"].apply(lambda x: np.sin(x * np.pi/180)) * train_meta[\"ra\"].apply(lambda x: np.cos(x * np.pi/180))\ntrain_meta['skycoordinate_y'] = train_meta[\"distmod\"] * train_meta[\"decl\"].apply(lambda x: np.sin(x * np.pi/180)) * train_meta[\"ra\"].apply(lambda x: np.sin(x * np.pi/180))\ntrain_meta['skycoordinate_z'] = train_meta[\"distmod\"] * train_meta[\"decl\"].apply(lambda x: np.cos(x * np.pi/180))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2f2df153ec67f3f489d9ad9fbc52ead80906a95f"},"cell_type":"code","source":"#https://matplotlib.org/examples/mplot3d/scatter3d_demo.html\nfrom mpl_toolkits.mplot3d import Axes3D\n\nfig = plt.figure(figsize=(10,8))\nax = Axes3D(fig)\n\n#ax.scatter(swasp_mag1,swasp_per1,swasp_age1,edgecolor='none',c='r',marker='o',s=35,label='SWASP')\n#ax.scatter(hyd_mag1,hyd_per1,hyd_age1,edgecolor='none',c='y',marker='o',s=35,label='Hyades')\n#ax.scatter(pld_mag1,pld_per1,pld_age1,edgecolor='none',c='b',marker='o',s=35,label='Pleiades')\n\nax.scatter(train_meta[train_meta['target']==90]['skycoordinate_x'], train_meta[train_meta['target']==90]['skycoordinate_y'], train_meta[train_meta['target']==90]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==42]['skycoordinate_x'], train_meta[train_meta['target']==42]['skycoordinate_y'], train_meta[train_meta['target']==42]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==65]['skycoordinate_x'], train_meta[train_meta['target']==65]['skycoordinate_y'], train_meta[train_meta['target']==65]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==16]['skycoordinate_x'], train_meta[train_meta['target']==16]['skycoordinate_y'], train_meta[train_meta['target']==16]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==15]['skycoordinate_x'], train_meta[train_meta['target']==15]['skycoordinate_y'], train_meta[train_meta['target']==15]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==62]['skycoordinate_x'], train_meta[train_meta['target']==62]['skycoordinate_y'], train_meta[train_meta['target']==62]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==88]['skycoordinate_x'], train_meta[train_meta['target']==88]['skycoordinate_y'], train_meta[train_meta['target']==88]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==92]['skycoordinate_x'], train_meta[train_meta['target']==92]['skycoordinate_y'], train_meta[train_meta['target']==92]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==67]['skycoordinate_x'], train_meta[train_meta['target']==67]['skycoordinate_y'], train_meta[train_meta['target']==67]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==52]['skycoordinate_x'], train_meta[train_meta['target']==52]['skycoordinate_y'], train_meta[train_meta['target']==52]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==95]['skycoordinate_x'], train_meta[train_meta['target']==95]['skycoordinate_y'], train_meta[train_meta['target']==95]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==6]['skycoordinate_x'], train_meta[train_meta['target']==6]['skycoordinate_y'], train_meta[train_meta['target']==6]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==64]['skycoordinate_x'], train_meta[train_meta['target']==64]['skycoordinate_y'], train_meta[train_meta['target']==64]['skycoordinate_z'])\nax.scatter(train_meta[train_meta['target']==53]['skycoordinate_x'], train_meta[train_meta['target']==53]['skycoordinate_y'], train_meta[train_meta['target']==53]['skycoordinate_z'])\n\nplt.title(\"sky coordinate\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"93f74211d937c2dc901575e835d1a0ac8f0626fc"},"cell_type":"code","source":"train_meta['galatic_x'] = train_meta[\"distmod\"] * train_meta[\"gal_b\"].apply(lambda x: np.sin(x * np.pi/180)) * train_meta[\"gal_l\"].apply(lambda x: np.cos(x * np.pi/180))\ntrain_meta['galatic_y'] = train_meta[\"distmod\"] * train_meta[\"gal_b\"].apply(lambda x: np.sin(x * np.pi/180)) * train_meta[\"gal_l\"].apply(lambda x: np.sin(x * np.pi/180))\ntrain_meta['galatic_z'] = train_meta[\"distmod\"] * train_meta[\"gal_b\"].apply(lambda x: np.cos(x * np.pi/180))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a450a0aa30d2a56b9fdbe8b1bc34629d081c78ac"},"cell_type":"code","source":"#https://matplotlib.org/examples/mplot3d/scatter3d_demo.html\n\nfig = plt.figure(figsize=(10,8))\nax = Axes3D(fig)\n\n#ax.scatter(swasp_mag1,swasp_per1,swasp_age1,edgecolor='none',c='r',marker='o',s=35,label='SWASP')\n#ax.scatter(hyd_mag1,hyd_per1,hyd_age1,edgecolor='none',c='y',marker='o',s=35,label='Hyades')\n#ax.scatter(pld_mag1,pld_per1,pld_age1,edgecolor='none',c='b',marker='o',s=35,label='Pleiades')\n\nax.scatter(train_meta[train_meta['target']==90]['galatic_x'], train_meta[train_meta['target']==90]['galatic_y'], train_meta[train_meta['target']==90]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==42]['galatic_x'], train_meta[train_meta['target']==42]['galatic_y'], train_meta[train_meta['target']==42]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==65]['galatic_x'], train_meta[train_meta['target']==65]['galatic_y'], train_meta[train_meta['target']==65]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==16]['galatic_x'], train_meta[train_meta['target']==16]['galatic_y'], train_meta[train_meta['target']==16]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==15]['galatic_x'], train_meta[train_meta['target']==15]['galatic_y'], train_meta[train_meta['target']==15]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==62]['galatic_x'], train_meta[train_meta['target']==62]['galatic_y'], train_meta[train_meta['target']==62]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==88]['galatic_x'], train_meta[train_meta['target']==88]['galatic_y'], train_meta[train_meta['target']==88]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==92]['galatic_x'], train_meta[train_meta['target']==92]['galatic_y'], train_meta[train_meta['target']==92]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==67]['galatic_x'], train_meta[train_meta['target']==67]['galatic_y'], train_meta[train_meta['target']==67]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==52]['galatic_x'], train_meta[train_meta['target']==52]['galatic_y'], train_meta[train_meta['target']==52]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==95]['galatic_x'], train_meta[train_meta['target']==95]['galatic_y'], train_meta[train_meta['target']==95]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==6]['galatic_x'], train_meta[train_meta['target']==6]['galatic_y'], train_meta[train_meta['target']==6]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==64]['galatic_x'], train_meta[train_meta['target']==64]['galatic_y'], train_meta[train_meta['target']==64]['galatic_z'])\nax.scatter(train_meta[train_meta['target']==53]['galatic_x'], train_meta[train_meta['target']==53]['galatic_y'], train_meta[train_meta['target']==53]['galatic_z'])\n\nplt.title(\"galatic coordinate\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf7234f9ea6bd0e2a4209cd5d61b865349902be9"},"cell_type":"code","source":"train_meta['ddf'].plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1bc153f18b95f83980f332f7885c5f30eb4c9a38"},"cell_type":"markdown","source":"hmm...?"},{"metadata":{"trusted":true,"_uuid":"5cd86fdfb89a058c956e659a29435583126b3a3c"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\ntrain_meta['ddf'].value_counts().sort_values(ascending=False).plot.bar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"351af9b226c8acac96a0678e2fccf106451bd4cf"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\ntrain_meta[train_meta['ddf']==1]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title(\"ddf is 1 of target\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5be86c7573c2b2e3e5ad87c811fc5b8a73791089"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\ntrain_meta[train_meta['ddf']==0]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title(\"ddf is 0 of target\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fdb4e1c1c55953f2d43c6a2aa502fc3000cad7e"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, hue = 'ddf',height=8)\ng.map(plt.scatter, 'gal_l','gal_b')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b4df9628b2d472dae7d137c25c6cf5117cb801d"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, hue = 'ddf',height=8)\ng.map(plt.scatter, 'ra','decl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"392257340aeb4eabd4fa0d2c7df3ff7d4739a30b"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\nfig = sns.boxplot(x='target', y=\"hostgal_specz\", data=train_meta)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"edd174685983efe7c24d2976590a16af84815fa6"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\nfig = sns.boxplot(x='target', y=\"hostgal_photoz\", data=train_meta)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54bb83432402d1e5c2aac6a063146cc1aab07092"},"cell_type":"code","source":"fig, (axis1,axis2) = plt.subplots(1,2,figsize=(14,12),sharey=True)\nsns.boxplot(x='target', y=\"hostgal_specz\", data=train_meta,ax=axis1)\naxis1.set_title('Spectroscopic Redshift')\nsns.boxplot(x='target', y=\"hostgal_photoz\", data=train_meta,ax=axis2)\naxis2.set_title('Photometric Redshift')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5dc53d858211302e71acae470a81902b6c373c9b"},"cell_type":"markdown","source":"Photometric seems to more outliers"},{"metadata":{"trusted":true,"_uuid":"dd17461ee7ecd2c8aa7df2786b6ceec50d8694ba"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\nfig = sns.boxplot(x='target', y=\"distmod\", data=train_meta)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be9847b74c4203bc389cc58ef49635a15ec466f8"},"cell_type":"markdown","source":"target 6, 16, 53, 65, 92 has no distmod "},{"metadata":{"trusted":true,"_uuid":"0f5a8e52f177c06d77b7b3247100d4f9cf0bac61"},"cell_type":"code","source":"train_meta[train_meta['distmod'].isnull()]['target'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e884e0f92177321e2602d602e1c9effb0b735e6"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, hue = 'Milky.Way',height=8)\ng.map(plt.scatter, 'gal_l','gal_b')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7bc9c7a03412f94e826cde388a91247ab96603a5"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, hue = 'Milky.Way',height=8)\ng.map(plt.scatter, 'ra','decl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d8a48bc1ce9086efdf6efbee42767d353a54fd65"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, col = 'Milky.Way',col_wrap=2)\ng.map(plt.scatter, 'gal_l','gal_b')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"89354d3528bd9a92ff8009aaded24ceefba6be3c"},"cell_type":"code","source":"g = sns.FacetGrid(train_meta, col = 'Milky.Way',col_wrap=2)\ng.map(plt.scatter, 'ra','decl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e498ac44e3f839621a5f491c1dc88c8d9d16df1","_kg_hide-input":true},"cell_type":"code","source":"print(\"Milky.Way is 0 of Target:\",train_meta[train_meta['Milky.Way']==0]['target'].unique(),\"\\nMilky.Way is 1 of Target:\",train_meta[train_meta['Milky.Way']==1]['target'].unique())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3e819d9ce42d0247298087479b1f3fc3ca1aabe"},"cell_type":"markdown","source":"### Train File Column Descriptions"},{"metadata":{"trusted":true,"_uuid":"51b0ecea2113e77245911e4b4e7adafbc2cb6bab"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49ff8e430fbebf62af8aa10b20413165b5bab145"},"cell_type":"markdown","source":"- object id: Same key as in the metadata table abov.\n- mjd: the time in Modified Julian Date (MJD) of the observation.\n- passband: The specific LSST passband integer, such that u, g, r, i, z, y = 0, 1, 2, 3, 4, 5 in which it was viewed.\n- flux: the measured flux (brightness) in the passband of observation as listed in the passband column. The flux is corrected for MWEBV.\n- flux err: the uncertainty on the measurement of the flux listed above.\ndetected: If detected= 1, the object’s brightness is significantly different at the 3σ level relative to the reference template."},{"metadata":{"trusted":true,"_uuid":"1ccba7c59d84a5a9fbde683e39a5cab8c1291b6e"},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47a014bbd6a5a41896c7458aa80292a350f49bcc"},"cell_type":"code","source":"train['object_id'] = train['object_id'].astype(np.object)\ntrain_meta['object_id'] = train_meta['object_id'].astype(np.object)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46c850ec6dbd35179a5157550aa296f3094be9e1"},"cell_type":"code","source":"train['object_id'].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"102f329381f14c5dc225db4ff5c60dda310173f7"},"cell_type":"code","source":"train_all = train_meta.merge(train, how='left', on='object_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"89ff0a3bdb69b55ba127d87abfcceca8c8a628ae"},"cell_type":"code","source":"train_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b356c4d92606c0b31b43ac30fc38209fe1fbded5"},"cell_type":"code","source":"del train, train_meta\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cef2a70b0f63fc8f3aad4481f8538afcf63745f"},"cell_type":"code","source":"train_all['mjd'].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"60280e79104bbb9eb0b77952148eaf62434a9952"},"cell_type":"code","source":"train_all['date'] = pd.to_datetime((train_all['mjd']-40587)*86400,unit='s')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9684d3fb5e5cc310b94fb7e8919d239a9b823717"},"cell_type":"code","source":"train_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a26852595ba669a7bd4a270c20842e84e7e96439"},"cell_type":"code","source":"train_all['target'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70c3e711b276fd14843302dbb9795a24a547b581"},"cell_type":"code","source":"import plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport plotly.figure_factory as ff\nimport random\n\n#https://www.kaggle.com/ashishpatel26/bird-eye-view-of-two-sigma-xgb\ndef generate_color():\n    color = '#{:02x}{:02x}{:02x}'.format(*map(lambda x: random.randint(0, 255), range(3)))\n    return color\nfor i in [6, 15, 16, 42, 52, 53, 62, 64, 65, 67, 88, 90, 92, 95]:\n    \n    data = []\n    for j in [0,1,2,3,4,5]:\n        j = go.Scatter(\n                x = train_all[(train_all['target']==i)]['date'].dt.strftime(date_format='%Y-%m-%d').values,\n                y = train_all[(train_all['target']==i) & (train_all['passband']==j)]['flux'].values,\n                line = dict(color = generate_color()),opacity = 0.8\n            )\n\n        data.append(j)\n                    \n    layout = dict(title = \"Target {}\".format(i),\n                          xaxis = dict(title = 'date'),\n                          yaxis = dict(title = 'flux'),\n                          )\n    py.iplot(dict(data=data, layout=layout), filename='basic-line')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a402cd1a43a1d9d69fa9ab49ed37898a4adaa00"},"cell_type":"markdown","source":"If you click the trace on the upper right, you can see more better."},{"metadata":{"trusted":true,"_uuid":"9058192d552a03dfe361c3deb78700a63f0f2345","_kg_hide-input":true},"cell_type":"code","source":"fig, (axes) = plt.subplots(2,3,figsize=(18,12),sharey=True)\nsns.boxplot(x='target', y=\"flux_err\", data=train_all[train_all['passband']==0],ax=axes[0,0])\naxes[0,0].set_title('0')\nsns.boxplot(x='target', y=\"flux_err\", data=train_all[train_all['passband']==1],ax=axes[0,1])\naxes[0,1].set_title('1')\nsns.boxplot(x='target', y=\"flux_err\", data=train_all[train_all['passband']==2],ax=axes[0,2])\naxes[0,2].set_title('2')\nsns.boxplot(x='target', y=\"flux_err\", data=train_all[train_all['passband']==3],ax=axes[1,0])\naxes[1,0].set_title('3')\nsns.boxplot(x='target', y=\"flux_err\", data=train_all[train_all['passband']==4],ax=axes[1,1])\naxes[1,1].set_title('4')\nsns.boxplot(x='target', y=\"flux_err\", data=train_all[train_all['passband']==5],ax=axes[1,2])\naxes[1,2].set_title('5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"163633702d073943b38ea5cc9aa7c0b87dff42ec"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,8))\ntrain_all['detected'].value_counts().sort_values(ascending=False).plot.bar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d32b415ac1e52c92348c82d2b6c736930be3b40"},"cell_type":"code","source":"fig, (axes) = plt.subplots(2,2,figsize=(18,12))\nax1 = plt.subplot(2,2,1)\ntrain_all[(train_all['detected']==0) & (train_all['ddf']==0)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 0 & ddf : 0')\nax1 = plt.subplot(2,2,2)\ntrain_all[(train_all['detected']==0) & (train_all['ddf']==1)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 0 & ddf : 1')\nax1 = plt.subplot(2,2,3)\ntrain_all[(train_all['detected']==1) & (train_all['ddf']==0)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 1 & ddf : 0')\nax1 = plt.subplot(2,2,4)\ntrain_all[(train_all['detected']==1) & (train_all['ddf']==1)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 1 & ddf : 1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6818e5944bbb2d533e6b7f7811bcc4adb64f75ed"},"cell_type":"code","source":"fig, (axes) = plt.subplots(2,2,figsize=(18,12))\nax1 = plt.subplot(2,2,1)\ntrain_all[(train_all['detected']==0) & (train_all['Milky.Way']==0)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 0 & Milky.Way : 0')\nax1 = plt.subplot(2,2,2)\ntrain_all[(train_all['detected']==0) & (train_all['Milky.Way']==1)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 0 & Milky.Way : 1')\nax1 = plt.subplot(2,2,3)\ntrain_all[(train_all['detected']==1) & (train_all['Milky.Way']==0)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 1 & Milky.Way : 0')\nax1 = plt.subplot(2,2,4)\ntrain_all[(train_all['detected']==1) & (train_all['Milky.Way']==1)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('detected : 1 & Milky.Way : 1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f763b356c943d8568b53c0e1feaef5ac450be3ef"},"cell_type":"code","source":"fig, (axes) = plt.subplots(2,2,figsize=(18,12))\nax1 = plt.subplot(2,2,1)\ntrain_all[(train_all['ddf']==0) & (train_all['Milky.Way']==0)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('ddf : 0 & Milky.Way : 0')\nax1 = plt.subplot(2,2,2)\ntrain_all[(train_all['ddf']==0) & (train_all['Milky.Way']==1)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('ddf : 0 & Milky.Way : 1')\nax1 = plt.subplot(2,2,3)\ntrain_all[(train_all['ddf']==1) & (train_all['Milky.Way']==0)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('ddf : 1 & Milky.Way : 0')\nax1 = plt.subplot(2,2,4)\ntrain_all[(train_all['ddf']==1) & (train_all['Milky.Way']==1)]['target'].value_counts().sort_values(ascending=False).plot.bar()\nplt.title('ddf : 1 & Milky.Way : 1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17f1f9ab66a42f2300cf0b52e6ec7d7fe52f2e7e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}