{"cells":[{"metadata":{"_uuid":"d7cd508612d2159b3209765df20e32dcaa8e8b3b"},"cell_type":"markdown","source":"# PLAsTiCC Data Visualization Tools with Periodograms"},{"metadata":{"_uuid":"7313b5fe7713ebc7596d7d491dd1a52ec1440d1a"},"cell_type":"markdown","source":"## Intro\nThe Lomb-Scargle periodogram (named for Lomb (1976) and Scargle (1982)) is a classic method for finding periodicity in irregularly-sampled data. It is in many ways analogous to the more familiar Fourier Power Spectral Density (PSD) often used for detecting periodicity in regularly-sampled data.\n\nIf we want to classify the sound of two similar objects that emit a certain frequency, let's say C, we would find some problems if the data sources are not perfectly aligned. \n\n!['Signal vs Frequency'](https://www.kaggle.com/nanmaran/plasticc/downloads/signal%20vs%20freq.png)\n\nIf we want to get an analysis of the nature of the 2 sounds from the pressure vs time points, at first the 2 datasets seem very different. Where the first one reaches a minimum at t=10, the other one reaches a maximum. But if we observe the periodogram, we can appreciate the similitud between the 2 signals. This is one of the signal interpretation logistics we will use to visualize astronomical sources that vary with time.\n\nThe same way a musical note sounds higher or lower depending on the relative movement of the source and the receptor because of the Doppler effect, astronomical signals will be affected by their distance to the Earth, the farther they are, the faster they are moving away from Earth. So a redshift correction may have to be made to classify the read frequencies of each source.\n\n"},{"metadata":{"_uuid":"02a74190ebe75d6f8318852b5cf8c1821ac5cc82"},"cell_type":"markdown","source":"## Load libraries and data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\nfrom sklearn import linear_model\nfrom sklearn.metrics import classification_report\nfrom sklearn.preprocessing import StandardScaler\n\nimport scipy.signal as signal\n\nimport timeit as t\nimport os\nprint(os.listdir(\"../input\"))\nimport time\nfrom tqdm import tqdm, tqdm_notebook","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Loading the data\ndf = pd.read_csv('../input/PLAsTiCC-2018/training_set.csv')\ndf.name = 'Training Set'\ndf_meta = pd.read_csv('../input/PLAsTiCC-2018/training_set_metadata.csv')\ndf_meta.name = 'Training Metadata Set'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"758c1733f6c5d51a956411c2cea3a4edcb3c4d87"},"cell_type":"markdown","source":"## Draw random charts\nThis tool was made for having an overview of the different types of astronomical sources. The arguments are:\n*  signalBool: 0 for raw data and 1 for Lomb-Scargle periodogram\n*  f: array of frequencies for Lomb-Scargle analysis"},{"metadata":{"trusted":true,"_uuid":"cb5911b3b24fea40477e795314a7530814725f5a"},"cell_type":"code","source":"def drawRandomCharts(signalBool = 0, f = np.linspace(1, 100, 100)):\n    plt.figure(figsize=(20,20))\n    for i,target in enumerate(df_meta['target'].unique()):\n        rnd = np.random.randint(0, 20)\n        object_id = df_meta[df_meta['target'] == target].iloc[rnd].object_id\n        plt.subplot(4, 4, i+1)\n        for passband in df['passband'].unique():\n            passbandData = df[(df['object_id'] == object_id) & (df['passband'] == passband)]\n            plt.title('Target: '+ str(df_meta.loc[df_meta['object_id']==object_id, 'target'].iloc[0])\n                     + '\\n Object: ' + str(int(object_id)))\n            if (signalBool):\n                plt.plot(signal.lombscargle(passbandData['mjd'], passbandData['flux'], f, normalize=True))\n            else:\n                plt.plot('mjd', 'flux', data=passbandData )\n                plt.scatter('mjd', 'flux', data=passbandData )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"155894bb25cce287448080dcd696cf50855f10a1"},"cell_type":"code","source":"drawRandomCharts(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"973230f27ca4d1c5866abd447354feee7ce0a6f6"},"cell_type":"code","source":"drawRandomCharts(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a8d50b67f51d14e0ecf21069a7e00560cb6e542f"},"cell_type":"code","source":"drawRandomCharts(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"daab144b943c24f45275c8800fd1617d535cc1e0"},"cell_type":"code","source":"drawRandomCharts(1, np.linspace(1,10,1000))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43e18a0081bcf9b18cbd2c446b748a81ec9a9262"},"cell_type":"markdown","source":"## Draw passbands charts\nThis tool was made for having deeper visualization of an astronomical object, it divides it's values between the different passbands. The arguments are:\n* object_id\n*  signalBool: 0 for raw data and 1 for Lomb-Scargle periodogram\n*  f: array of frequencies for Lomb-Scargle analysis"},{"metadata":{"trusted":true,"_uuid":"b6735aca28a97ed3974279dac2749abe7b631538"},"cell_type":"code","source":"def drawPassbands(object_id, signalBool = 0, f = np.linspace(1, 100, 100)):\n    plt.figure(figsize=(18,20))\n    for i,passband in enumerate(df['passband'].unique()):\n        passbandData = df[(df['object_id'] == object_id) & (df['passband'] == passband)].copy()\n        passbandData['markerSize'] = (passbandData['detected'] + 0.15) * 20\n        plt.subplot(6, 1, i+1)\n        plt.title('Target: '+ str(df_meta.loc[df_meta['object_id']==object_id, 'target'].iloc[0])\n                 + '\\n Object: ' + str(int(object_id)))\n        if (signalBool):\n            plt.suptitle('\\n\\nLomb-Scargle frequencies', fontsize=18)\n            plt.plot(signal.lombscargle(passbandData['mjd'], passbandData['flux'], f, normalize=True)) \n        else:\n            plt.suptitle('\\n\\nRaw signals', fontsize=18)\n            plt.plot('mjd', 'flux', data=passbandData, alpha=0.3 )\n            plt.scatter('mjd', 'flux', 'markerSize' , data=passbandData )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8004faf6f828664c113f3839c1bf6f01ecfb43fb"},"cell_type":"code","source":"drawPassbands(713,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e1f97221df46575f18e8a1c00cd5dd3e06323dd"},"cell_type":"code","source":"drawPassbands(713,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"666440f1fa3e99f2a1b043aa89eb9f4333233eea"},"cell_type":"markdown","source":"## Astronomical object inspector\nThis tool was made for creating a report of a certain astronomical object, including the raw data and the corresponding periodogram. The arguments are:\n* object_id\n*  f: array of frequencies for Lomb-Scargle analysis"},{"metadata":{"trusted":true,"_uuid":"e189df2ea7353e7aa89ad7285a172d6cd51d6d6e"},"cell_type":"code","source":"def starInspector(object_id, f = np.linspace(1, 100, 200)):\n    plt.figure(figsize=(18,20), num=1)\n    plt.suptitle('Target: '+ str(df_meta.loc[df_meta['object_id']==object_id, 'target'].iloc[0])\n                 + '\\n Object: ' + str(int(object_id))\n                 + '\\n ddf = ' + str(df_meta.loc[df_meta['object_id']==object_id, 'ddf'].iloc[0])\n                 + '\\n specz = ' + str(df_meta.loc[df_meta['object_id']==object_id, 'hostgal_specz'].iloc[0])\n                 + '\\n photoz = ' + str(df_meta.loc[df_meta['object_id']==object_id, 'hostgal_photoz'].iloc[0])\n                 + '\\n distance = ' + str(df_meta.loc[df_meta['object_id']==object_id, 'distmod'].iloc[0])\n                 + '\\n mwebv = ' + str(df_meta.loc[df_meta['object_id']==object_id, 'mwebv'].iloc[0])\n                 + '\\n freq = ' + str(f.min()) + ' - ' + str(f.max()) + ' --- ' + str(f.shape)\n                 , fontsize=16)\n    initTime = t.timeit()\n    for i,passband in enumerate(df['passband'].unique()):\n        passbandData = df[(df['object_id'] == object_id) & (df['passband'] == passband)].copy()\n        passbandData['markerSize'] = (passbandData['detected'] + 0.15) * 20\n        plt.subplot(6, 2, 2*(i+1)-1)\n        plt.title('Passband: ' + str(i) + ' (' + str(passbandData.shape[0]) + ') ')\n        plt.plot('mjd', 'flux', data=passbandData, alpha=0.3 )\n        plt.scatter('mjd', 'flux', 'markerSize' , data=passbandData )\n\n        plt.subplot(6, 2, 2*(i+1))\n        plt.plot(signal.lombscargle(passbandData['mjd'], passbandData['flux'], f, normalize=True))    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e1bae1fea9da4425ffa63e5f8d6aad98ee6924ae"},"cell_type":"code","source":"starInspector(713)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9725270344a84d185718489c23d226c68a29a7de"},"cell_type":"code","source":"starInspector(713, np.linspace(0.1,10,1000))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35461814b8484a9d4988defaa0377533b4c1d35d"},"cell_type":"markdown","source":"Iterating the above function to every object in a target class could give us a good visualization of the behaviour of objects of a certain class."},{"metadata":{"trusted":true,"_uuid":"a0688a52394df289eb4e09d1b68689e44c5294f8"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}