{"cells":[{"metadata":{"_kg_hide-input":true,"_uuid":"f303093677675698aa09f094fd60203270058572","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport math\nimport plotly.graph_objs as go\nimport matplotlib.pyplot as plt \nimport plotly.offline as py\nfrom plotly import tools\nimport plotly.figure_factory as ff\nimport seaborn as sns\nimport warnings\npy.init_notebook_mode(connected=True)\nwarnings.simplefilter('ignore', FutureWarning)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"951624d5ae5165883e5f5a4b17e1ff5b2a3e28b8","trusted":true},"cell_type":"code","source":"training_set = pd.read_csv('../input/training_set.csv')\n# training_set = pd.read_csv('training_set.csv')\nmeta_training_set = pd.read_csv(\"../input/training_set_metadata.csv\")\n# meta_training_set = pd.read_csv(\"training_set_metadata.csv\")\ntest_set_meta = pd.read_csv(\"../input/test_set_metadata.csv\")\n# test_set_meta = pd.read_csv(\"test_set_metadata.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"c502d6a3b2b5e355e3ebf83ef1b00d6760bfa9ea","trusted":true},"cell_type":"code","source":"full_meta_data = pd.concat([meta_training_set,test_set_meta],sort=True)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd801cdd3c7968bc0a2ae6de91bf6f9b50bb5c73"},"cell_type":"markdown","source":"## Meta_training data"},{"metadata":{"_kg_hide-input":true,"_uuid":"4257ee045ff6a28f7cdafd35157ef8859959516e","trusted":true},"cell_type":"code","source":"print(\"sample of meta_training data\")\nmeta_training_set.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"1cc30c2ca5ff20ffe146796221e027fdcd2ba8b8","trusted":true},"cell_type":"code","source":"print(\"training_set_metadata info\")\nmeta_training_set.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e9d57c4098707f2afe0b29bc317aa56cb038692"},"cell_type":"markdown","source":" The columns in training_set metadata  represents:\n\n- **object_id:** unique object identifier.\n- **ra:** right ascension, sky coordinate: co-longitude in degrees. \n- **decl:** declination, sky coordinate: co-latitude in degrees. \n- **gal_l:** galactic longitude in degrees. \n- **gal_b:** galactic latitude in degrees. \n- **ddf:** A flag to identify the object as coming from the DDF survey area (with value DDF = 1 for the DDF, DDF = 0 for the WFD survey).Boolean\n- **hostgal_specz:** the spectroscopic redshift of the source. This is an extremely accurate measure of redshift, available for the training set and a small fraction of the test set. \n- **hostgal_photoz:** The photometric redshift of the host galaxy of the astronomical source. While this is meant to be a proxy for hostgal_specz, there can be large differences between the two and should be regarded as a far less accurate version of hostgal_specz. \n- **hostgal_photoz_err:** The uncertainty on the hostgal_photoz based on LSST survey projections. \n- **distmod:** The distance to the source calculated from hostgal_photoz by using General Relativity, and assumed values of the dark\nenergy and dark matter content of the Universe .\n- **MWEBV:** MW E(B-V). this ‘extinction’ of light is a property of the Milky Way (MW) dust along the line of sight to the astronomical source, and is thus a function of the sky coordinates of the source ra, decl. This is used to determine a passband dependent dimming and redenning of light from astronomical sources as described in subsection 2.1, and based on the Schlafly et al. (2011) and Schlegel et al. (1998) dust models. \n- **target:** The class of the astronomical source. This is provided in the training data. Correctly determining the target (correctly assigning classification probabilities to the objects) is the ‘goal’ of the classification challenge for the test data. Note that there is one class in the test set that does not occur in the training set: class_99 serves as an \"other\" class for objects that don't belong in any of the 14 classes in the training set. \n"},{"metadata":{"_uuid":"feee6fcdb580585c743347d65ab119ed60fb1946"},"cell_type":"markdown","source":"## Training data"},{"metadata":{"trusted":true,"_uuid":"56335944c4d13550a57bf8f3f8e57a633cc58639","_kg_hide-input":true},"cell_type":"code","source":"print (\"sample of training data\")\ntraining_set.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bab530dc05d8478008a6a62a309f1faf46939197","_kg_hide-input":true},"cell_type":"code","source":"print(\"training_set info\")\ntraining_set.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"232beceb5f01c96bdfaa37b9bc00fd928897a041"},"cell_type":"markdown","source":" The columns in training_set represents:\n- **object id:** Same key as in the metadata given as int32 numbers.\n- **mjd:** the time in Modified Julian Date (MJD) of the observation.  The MJD can\nbe converted to Unix epoch time with the formula unix time = (MJD40587)86400.\nThe units are days, and the numbers are given as float64 numbers.\n- **passband:** The specific LSST passband integer, such that u, g, r, i, z, y = 0, 1, 2, 3, 4, 5\nin which it was viewed. These are given as int8 numbers.\n- **flux:** the measured flux (brightness) in the passband of observation as listed in the\npassband column.\n- **flux err:** the uncertainty on the measurement of the flux listed above. \n- **detected:** If detected= 1, the object’s brightness is significantly different at the 3σ\nlevel relative to the reference template. This is given as a Boolean flag."},{"metadata":{"_uuid":"f244fa928f89bddf96fb7eb55516db6d3fbd2773"},"cell_type":"markdown","source":"## NAN count"},{"metadata":{"trusted":true,"_uuid":"c4efd80d9a2354e0b857995b2c40b899d2a94dd4","_kg_hide-input":true},"cell_type":"code","source":"fig,ax = plt.subplots(1,2,figsize=(12,5))\nax[0].barh(meta_training_set.drop(['target'], axis=1).isnull().sum().index,meta_training_set.drop(['target'], axis=1).isnull().sum().values)\nax[0].set_xlabel(\"NAN count\")\nax[0].set_title(\"NAN count per feature\\n(traing_meta data)\")\nax[1].barh(test_set_meta.isnull().sum().index,test_set_meta.isnull().sum().values)\nax[1].set_xlabel(\"NAN count\")\nax[1].set_title(\"NAN count per feature\\n(test_meata data)\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e07d4e63992cb7118ac107a3f8852fed72e457d"},"cell_type":"markdown","source":"## Classes analysis\n\nThe classes are the classification of objects accotrding to its light curves. The challenge is to analyze each set of light curves and determine a probability that each object belongs to each of these classes. So, the classes represent the target in the challange."},{"metadata":{"_kg_hide-input":true,"_uuid":"f72c1969374936f6a92625d998365894801ed6e7","trusted":true},"cell_type":"code","source":"targets_classes = meta_training_set.target.unique()\nprint (\"There are {} unique classes.\".format(len(targets_classes)))\nfor i in range(len(targets_classes)):\n    print(\"class_{}\".format(targets_classes[i]))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"79c83f1e4c02ee92e72e781a1e53b74c1a101f2a","trusted":true},"cell_type":"code","source":"objects_per_target = pd.DataFrame(meta_training_set.groupby(\"target\", as_index = False)[\"object_id\"].count()).sort_values(by=\"object_id\",ascending=False)\nobjects_per_target['target_list']=list(map(lambda x: \"class_{}\".format(x),objects_per_target.target))\nobjects_per_target = objects_per_target.rename(columns = {\"object_id\": \"objects_count\"})\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"994793feed2383ad7c817bee1794dde3542f5489","trusted":true},"cell_type":"code","source":"pie_plot= go.Pie(values = objects_per_target['objects_count'],\n                 labels = objects_per_target['target_list'],\n                 hoverinfo=\"label+percent\",\n                 hole= .3)\n\nlayout = go.Layout(title = \"Classes distribution \")\n\nfig = go.Figure(data=[pie_plot], layout=layout)\n\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"4eab44bfd2b0cf91287dcb842d55aa9e1ebe5ac9","trusted":true},"cell_type":"code","source":"bar_trace  = go.Bar(x= objects_per_target.target_list,\n                 y= objects_per_target.objects_count,\n                 marker=dict(color='#f0000a',\n                             line=dict(color='rgb(8,48,107)',width=1.5,)),\n                 name = \"objects count\",\n                 opacity=0.7,\n                 hoverinfo=\"name + y\")\n\nlayout = go.Layout(title='Count of objects per class',\n                   xaxis=dict(tickangle=-45),\n                   yaxis = dict(title = \" Number of objects\"))\n\nfig = go.Figure(data=[bar_trace], layout=layout)\n\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"297af1e592455acc3051ed9f264d9002b68d3df4"},"cell_type":"markdown","source":"### Classes distrbution in the space"},{"metadata":{"_kg_hide-input":true,"_uuid":"73b7f8679b1705f62135dfc91f95de8bd37d1bc7","scrolled":false,"trusted":false},"cell_type":"code","source":"colors = ['blue','gray','red','green','pink',\n          'steelblue','yellow','magenta','brown',\n          'orange','tan','seagreen','mintcream',\n          'yellowgreen','chocolate','rosybrown',\n          'dodgerblue','heather']\nfor i in range (0,len(objects_per_target.target)):\n    class_ = meta_training_set[meta_training_set.target == objects_per_target.target.values[i]]\n    trace=go.Scatter(\n        x=class_['gal_l'],\n        y=class_['gal_b'],\n        mode = 'markers',\n        marker=dict(color=colors[i]),\n        text= \"Longitude = {} °\".format(class_['gal_l'].values[i])+\"<br>\"+ \"Latitude = {} °\".format(class_['gal_b'].values[i]),\n        hoverinfo=\"text\",\n        connectgaps=True,\n        name = objects_per_target.target_list.values[i],\n        textfont=dict(family='Arial', size=12),\n    )\n    layout = go.Layout(\n        title = objects_per_target.target_list.values[i]+\" distrbution in space\",\n       xaxis=dict(\n                showline=True,\n                showgrid=True,\n                showticklabels=True,\n                linecolor='rgb(150, 150, 150)',\n                linewidth=2,\n                gridcolor='rgb(90, 90, 90)',\n                ticks='outside',\n                tickcolor='rgb(80, 80, 80)',\n                tickwidth=2,\n                ticklen=5,\n                tickfont=dict(\n                family='Arial',\n                size=13,\n                color='rgb(180, 180, 180)',\n            ),\n        ),\n        yaxis=dict(\n                showgrid=True,\n                zeroline=True,\n                showline=False,\n                gridcolor='rgb(80, 80, 80)',\n                showticklabels=True,\n                tickcolor='rgb(150, 150, 150)',\n                tickwidth=2,\n                ticklen=5,\n                tickfont=dict(\n                family='Arial',\n                size=13,\n                color='rgb(180, 180, 180)')\n        ),\n       font=dict(family='Arial', size=12,\n                color='rgb(180, 180, 180)'),\n                showlegend=True, \n                width = 600,\n                height = 300,\n                paper_bgcolor='rgba(0, 0, 0,.9)',\n                plot_bgcolor='rgba(0, 0, 0,0)')\n    \n    fig = go.Figure(data=[trace], layout= layout)\n    py.iplot(fig)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"0467d421d325bdd9cc7c5160d2719f133be6b264","scrolled":false,"trusted":false},"cell_type":"code","source":"traces = []\nfor i in range (len(targets_classes)):\n    class_ = meta_training_set[meta_training_set.target == targets_classes[i]]\n    traces.append(go.Scatter(\n        x=class_['gal_l'],\n        y=class_['gal_b'],\n        mode = 'markers',\n        marker=dict(color=colors[i]),\n        text= \"class_{}\".format(targets_classes[i])+\"<br>\"+\"Longitude = {} °\".format(class_['gal_l'].values[i])+\"<br>\"+ \"Latitude = {} °\".format(class_['gal_b'].values[i]),\n        hoverinfo=\"text\",\n        connectgaps=True,\n        name = \"class_{}\".format(targets_classes[i]),\n        textfont=dict(family='Arial', size=12),\n    ))\nlayout = go.Layout(\n    title = \"Classes distrbution in space\",\n   xaxis=dict(\n            title = \"Galactical Longitude (°)\",\n            showline=True,\n            showgrid=True,\n            showticklabels=True,\n            linecolor='rgb(150, 150, 150)',\n            linewidth=2,\n            gridcolor='rgb(90, 90, 90)',\n            ticks='outside',\n            tickcolor='rgb(80, 80, 80)',\n            tickwidth=2,\n            ticklen=5,\n            tickfont=dict(\n            color='rgb(180, 180, 180)',\n        ),\n    ),\n    yaxis=dict(\n            title = \"Galactical Latitude (°)\",\n            showgrid=True,\n            zeroline=True,\n            showline=False,\n            gridcolor='rgb(80, 80, 80)',\n            showticklabels=True,\n            tickcolor='rgb(150, 150, 150)',\n            tickwidth=2,\n            ticklen=5,\n            tickfont=dict(\n            color='rgb(180, 180, 180)')\n    ),\n   font=dict(family='Arial', size=12,\n            color='rgb(200, 200, 200)'),\n            showlegend=True, \n            width = 750,\n            height = 550,\n            paper_bgcolor='rgba(0, 0, 0,.9)',\n            plot_bgcolor='rgba(0, 0, 0,0)'\n)\nfig = go.Figure(data=traces, layout= layout)\npy.iplot(fig)\n\nfor trace in traces :\n    trace.marker.opacity = 0.1\nlayout.title = \"Classes distrbution in space (low opacity) \"\nfig = go.Figure(data=traces, layout= layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4e904c2cd477b9d36dde0f0e39eddab1ec0121d5"},"cell_type":"markdown","source":"The highlighted spots in the low-opacity figure show the most congested areas in the sky. this is insbired by this  [kernel](https://www.kaggle.com/hrmello/dataset-overview-exploration-and-comments)"},{"metadata":{"_uuid":"65d46e11f656086c12d155f1dd49488e630ea926"},"cell_type":"markdown","source":"### Correlation between classes and features"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"acef293e47d90897e8c6987775da8633aa09ae9d"},"cell_type":"code","source":"# the galactic feature determines whether the object inside the milky way galactic==1 or outside milky way galactic==0\nmeta_training_set['galactic'] = list(map(lambda x: 1 if x==0 else 0,meta_training_set['hostgal_photoz']))\ndef is_class (class_number):\n    meta_training_set['_{}'.format(class_number)]=list(map(lambda x: 1 if x==class_number else 0,meta_training_set.target))\nfor class_num in objects_per_target.target:\n    is_class(class_num)\ncorr_meta = meta_training_set.drop(['object_id','target'],axis=1).corr()\nplt.subplots(figsize=(12,10))\nsns.heatmap(corr_meta[11:].drop(corr_meta.columns[11:],axis=1),annot=True)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a88bb9dc42c3d613dd01d9e11a9ed9f2e305f04c"},"cell_type":"markdown","source":" ### Compute classes weights\n \nAccording to sklearn documentation,  class weights will be given by  **n_samples / (n_classes * np.bincount(y))** . \n\n***y*** is array of original class labels per sample."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6d717eb6979140f0ef7a2688ea13104e816ba5c6"},"cell_type":"code","source":"from sklearn.utils import class_weight\nclass_weights = class_weight.compute_class_weight('balanced',\n                                                 objects_per_target.target,\n                                                 meta_training_set['target'])\nprint(\"Classes weights:\"+\"\\n\\n\",list(zip(objects_per_target.target,class_weights)))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7463c947ccf3b1d2bed120ce191b1a0126e5a6e7"},"cell_type":"code","source":"bar_trace  = go.Bar(x= objects_per_target.target_list,\n                 y= class_weights,\n                 marker=dict(color='#f0000a',\n                             line=dict(color='rgb(8,48,107)',width=1.5,)),\n                 name = \"class weight\",\n                 opacity=0.7,\n                 hoverinfo=\"name + y\")\n\nlayout = go.Layout(title='Classes weights',\n                   xaxis=dict(tickangle=-45),\n                   yaxis = dict(title = \" class weight \"))\n\nfig = go.Figure(data=[bar_trace], layout=layout)\n\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31c2b29159a4707f1089c2949bebbdca0a8d4056"},"cell_type":"markdown","source":"## Distmod and redshift"},{"metadata":{"_kg_hide-input":true,"_uuid":"8f1c8b2e00b0210cc64bb562c86513cec668144c","scrolled":true,"trusted":true},"cell_type":"code","source":"group_labels = ['distmod distribution']\ndist_plot = ff.create_distplot([meta_training_set['distmod'].dropna().values],\n                               group_labels ,\n                               bin_size=.3,\n                               colors = ['rgba(5,20,100,.8)'])\ndist_plot['layout'].update(title=\"distmod distribution plot\",\n                           width = 700,\n                           height = 450)\n\npy.iplot(dist_plot)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d6ca9bb77012649b60b46c50a314d0e139474a3"},"cell_type":"markdown","source":"## Galactic vs Extragalactic \n\nAs described in the data note, The given redshift for objects in our own Milky Way galaxy is given as zero. According to this, we can classify the objects to Galactic (meaning that the object inside Milky Way galaxy). Extragalactic (meaning that the object outside Milky Way galaxy)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"471d2da206aeb61fcffd81f10fda93ea6af76592"},"cell_type":"code","source":"full_meta_data['galactic'] = list(map(lambda x: 1 if x==0 else 0,full_meta_data['hostgal_photoz']))\ntest_set_meta['galactic'] = list(map(lambda x: 1 if x==0 else 0,test_set_meta['hostgal_photoz']))\n\npie_plot_test= go.Pie(values = test_set_meta[\"galactic\"].value_counts(),\n                 labels = [\"Extragalactic\",\"Galactic\"],\n                 hoverinfo=\"label+percent\",\n                 domain = dict(x=[0, .5],\n                               y=[0.5,1]),\n                 hole = 0.4\n                    )\n\npie_plot_train= go.Pie(values = meta_training_set[\"galactic\"].value_counts(),\n                 labels = [\"Extragalactic\",\"Galactic\"],\n                 hoverinfo=\"label+percent\",\n                 domain = dict(x=[0.5, 1],\n                               y=[0.5,1]),\n                 hole = 0.4)\n\npie_plot_all= go.Pie(values = full_meta_data[\"galactic\"].value_counts(),\n                 labels = [\"Extragalactic\",\"Galactic\"],\n                 hoverinfo=\"label+percent\",\n                 domain = dict(x=[0, 1],\n                               y=[0,0.5]),\n                 hole = 0.4)\n\nlayout = go.Layout(title = \"Galactic vs Extragalactic\",\n                  annotations = [dict(text = \"test\",\n                                      font = dict(size=15),\n                                      x=0.225,\n                                      y=0.775,\n                                      showarrow= False),\n                                 dict(text = \"train\",\n                                      font = dict(size=15),\n                                      x=0.775,\n                                      y=0.775,\n                                     showarrow= False),\n                                 dict(text = \"all\",\n                                      font = dict(size=15),\n                                      x=0.50,\n                                      y=0.225,\n                                      showarrow= False)]\n                                    )\n\nfig = go.Figure(data=[pie_plot_test,pie_plot_train,pie_plot_all], layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b467c595e9c1ee14982c55203930299ccbe3f627"},"cell_type":"markdown","source":"The test pie for test dataset , the train pie for the train dataset and the all for the combination of two datasets. "},{"metadata":{"trusted":true,"_uuid":"f6f6f9a76aaf19b7ac3ac7ffd73be796b382d664","_kg_hide-input":true},"cell_type":"code","source":"galactic_classes = meta_training_set.groupby(['galactic']).get_group(1)['target'].value_counts()\nextragalactic_classes = meta_training_set.groupby(['galactic']).get_group(0)['target'].value_counts()\ngalactic_classes_list = list(map(lambda x: \"class_{}\".format(x),galactic_classes.index))\nextragalactic_classes_list = list(map(lambda x: \"class_{}\".format(x),extragalactic_classes.index))\n\nbar_trace_1  = go.Bar(x=galactic_classes_list,\n                 y= galactic_classes.values,\n                 marker=dict(color='#f0000a',\n                             line=dict(color='rgb(8,48,107)',width=1.5,)),\n                 name = \"galactic\",\n                 opacity=0.7,\n                 hoverinfo=\"name + y\")\nbar_trace_2  = go.Bar(x= extragalactic_classes_list,\n                 y= extragalactic_classes.values,\n                 marker=dict(color='#fff00a',\n                             line=dict(color='rgb(8,48,107)',width=1.5,)),\n                 name = \"extragalactic\",\n                 opacity=0.7,\n                 hoverinfo=\"name + y\")\n\nlayout = go.Layout(title='Galactic vs Extragalactic per class',\n                   xaxis=dict(tickangle=-45),\n                   yaxis = dict(title = \" Number of objects\"))\n\nfig = go.Figure(data=[bar_trace_1,bar_trace_2], layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a311ade24ec00a7ed86d9cba71ad7a7bb8e33ccc"},"cell_type":"markdown","source":"It is appeared that, There is no overlap between galactic and extragalactic classes. the galactic classes are [65, 16, 92, 6, 53] and the extragalactic classes are [90, 42, 15, 62, 88, 67, 52, 95, 64]."},{"metadata":{"_uuid":"13f02669deebaaa70f24bf48e456bd4a522fca89"},"cell_type":"markdown","source":"## MWEBV\nAs described in the data note, MWEBV  is an astronomical measure of how much redder an object appears compared to a Milky Way without dust. Larger MWEBV values correspond to more Milky Way dust along the line of sight to the objects, making the objects appearing redder ( move toward red). The high frequencies wavas like the Ultra-violet are distributed by small particles like the dust but low frequencies waves like red and infra-red dose not affected a lot by dust."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"32dad221b40a3b12dfe4146f139582291fa02226"},"cell_type":"code","source":"group_labels = ['MWEBV distribution']\ndist_plot = ff.create_distplot([meta_training_set['mwebv'].dropna().values],\n                               group_labels ,\n                               bin_size=.1,\n                               colors = ['rgba(5,20,100,.8)'])\ndist_plot['layout'].update(title=\"MWEBV distribution plot\",\n                           width = 700,\n                           height = 450)\n\npy.iplot(dist_plot)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8dc9e9382b7adfb736ae87d2131e0b29d9d40dcd"},"cell_type":"markdown","source":"\n"},{"metadata":{"_uuid":"47ac76ec48d2ef557e8557d11f4a6c349c8dee17"},"cell_type":"markdown","source":" ### Deep Drilling Fields (DDF) survey \n \n Deep Drilling Fields (DDF) are small patches of the sky that will be sampled often to achieve great depth\n(i.e. to be able to measure the flux from fainter objects). Objects in these DDF patches\nwill have light-curve points that are extremely well determined and therefore have small\nerrors in flux. The Wide-Fast-Deep (WFD) survey covers a larger part of the sky (almost\n400 times the area of the deep fields) that will be observed less frequently (and so light-\ncurve points will have larger uncertainties), but will discover many more objects over the\nlarger area."},{"metadata":{"_kg_hide-input":true,"_uuid":"b0a440fa9c2089e436292a28e8a9e0f7b5a00b1f","trusted":true},"cell_type":"code","source":"\npie_plot_test= go.Pie(values = test_set_meta[\"ddf\"].value_counts(),\n                 labels = [\"Outside DDF\",\"Inside DDF\"],\n                 hoverinfo=\"label+percent\",\n                 domain = dict(x=[0, .5],\n                               y=[0.5,1]),\n                 hole = 0.4\n                    )\n\npie_plot_train= go.Pie(values = meta_training_set[\"ddf\"].value_counts(),\n                 labels = [\"Outside DDF\",\"Inside DDF\"],\n                 hoverinfo=\"label+percent\",\n                 domain = dict(x=[0.5, 1],\n                               y=[0.5,1]),\n                 hole = 0.4)\n\npie_plot_all= go.Pie(values = full_meta_data[\"ddf\"].value_counts(),\n                 labels = [\"Outside DDF\",\"Inside DDF\"],\n                 hoverinfo=\"label+percent\",\n                 domain = dict(x=[0, 1],\n                               y=[0,0.5]),\n                 hole = 0.4)\n\nlayout = go.Layout(title = \"Object distribution according to DDF survey\",\n                  annotations = [dict(text = \"test\",\n                                      font = dict(size=15),\n                                      x=0.225,\n                                      y=0.775,\n                                      showarrow= False),\n                                 dict(text = \"train\",\n                                      font = dict(size=15),\n                                      x=0.775,\n                                      y=0.775,\n                                     showarrow= False),\n                                 dict(text = \"all\",\n                                      font = dict(size=15),\n                                      x=0.50,\n                                      y=0.225,\n                                      showarrow= False)]\n                                    )\n\nfig = go.Figure(data=[pie_plot_test,pie_plot_train,pie_plot_all], layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"5805ec81e089103616ad0a82527bfdacb69ac784","trusted":true},"cell_type":"code","source":"inside_ddf_target = meta_training_set.groupby(['ddf']).get_group(1)['target'].value_counts()\ninside_ddf_target_list = list(map(lambda x: \"class_{}\".format(x),inside_ddf_target.index))\nbar_trace  = go.Bar(x= inside_ddf_target_list,\n                 y= inside_ddf_target.values,\n                 marker=dict(color='#f0000a',\n                             line=dict(color='rgb(8,48,107)',width=1.5,)),\n                 name = \"objects count\",\n                 opacity=0.7,\n                 hoverinfo=\"name + y\")\n\nlayout = go.Layout(title='Count of objects per class (inside DDF survey area)',\n                   xaxis=dict(tickangle=-45),\n                   yaxis = dict(title = \" Number of objects\"))\n\nfig = go.Figure(data=[bar_trace], layout=layout)\n\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d21c1eda02d236a0d6d738bd3e0d9e4d2297073d"},"cell_type":"markdown","source":"## Light curves visualization \n\nThere are six passbands denoted u, g, r, i, z, y (0,1,2,3,4,5 in the dataset) that select light within different wavelength ranges: wavelengths between 300 and 400 nanometers for the u band, between 400 and 600 nm for the g passband, between 500 and 700 nm for the r band, between 650 and 850 for the i band, between 800 and 950 nm for the z band, and between 950 and 1050 nm for the y band."},{"metadata":{"trusted":true,"_uuid":"48f9bde504b37e6fd8038fdd8f7eadc410fb6b24","_kg_hide-input":true},"cell_type":"code","source":"training_passband_0 = training_set[training_set['passband'] == 0]\ntraining_passband_1 = training_set[training_set['passband'] == 1]\ntraining_passband_2 = training_set[training_set['passband'] == 2]\ntraining_passband_3 = training_set[training_set['passband'] == 3]\ntraining_passband_4 = training_set[training_set['passband'] == 4]\ntraining_passband_5 = training_set[training_set['passband'] == 5]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d9ea2320e2260f3aeb0e75accb69b29bfcb7de06"},"cell_type":"code","source":"def plot_class_time_series (class_,training_passband):\n    f, ax = plt.subplots(6,figsize=(8, 12))\n    f.suptitle('class_{}'.format(class_))\n    class_len = len(meta_training_set[meta_training_set['target']== class_])\n    for i in range (0,6):\n                       object_id = meta_training_set[meta_training_set['target'] == class_]['object_id'].values[i+np.random.randint(class_len-i)]\n                       ax[i].scatter(training_passband[training_passband['object_id'] == object_id]['mjd'],\n                                    training_passband[training_passband['object_id'] == object_id]['flux'])\n                       ax[i].plot(training_passband[training_passband['object_id'] == object_id]['mjd'],\n                                    training_passband[training_passband['object_id'] == object_id]['flux'])\n                       ax[i].set_xlabel('')\n                       ax[5].set_xlabel('mjd time')\n    f.tight_layout()\n    f.subplots_adjust(top=.95)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ecb1bdf11626e4e70bdafada762753193b22c859"},"cell_type":"code","source":"print (\"sample of light curves of u passband (passband_0) \")\nfor i in range(0,5):\n    plot_class_time_series(targets_classes[i],training_passband_0)","execution_count":null,"outputs":[]}],"metadata":{"celltoolbar":"Raw Cell Format","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}