{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"Light curves are irregularly spaced on the time axis.  They all exhibit some pretty large gaps. \n\nLet's look at these.\n\nFirst, let's load some data and packages"},{"metadata":{"trusted":true,"_uuid":"8de3d20e3763519d230bb8c3d557dc3045ae77cc"},"cell_type":"code","source":"from matplotlib import pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33402de6c5b7ca05fca98e840ccc02a5d7bf2e55"},"cell_type":"code","source":"train = pd.read_csv('../input/training_set.csv')\ntrain_meta = pd.read_csv('../input/training_set_metadata.csv')\ntrain = train.merge(train_meta[['object_id', 'ddf', 'ra', 'decl', 'target']], \n                    how='left', on='object_id')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bf5b82c9211abcf3749e37d216bcab0a5c6214e"},"cell_type":"markdown","source":"Al light curves have gaps at regularly spaced intervals, see for instance this curve:"},{"metadata":{"trusted":true,"_uuid":"d08c5e3bdc70a96758d28739c5de6b107c73975d"},"cell_type":"code","source":"object_id = 105869076\nprint('object_id', object_id)\nfig, ax = plt.subplots(1, 1, figsize=(15, 5))\ndf = train[train.object_id == object_id]\nax.scatter(df.mjd, df.flux, c=df.passband, cmap='rainbow', marker='+')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8e36fa94833af1853b684fedfdbddb9b6fb1548"},"cell_type":"markdown","source":"We see 3 gaps, repating every year.  These are due to the relative position of the source, earth and the sun: the source must be visible during the night from the location of the telescope on earth. \n\nLet's analyse this further."},{"metadata":{"_uuid":"37315524e370da34174448e4a4c25a0375a857b2"},"cell_type":"markdown","source":"First, we reuse the time analysis we did in https://www.kaggle.com/cpmpml/some-peculiarity# : when is the night in term of mjd?"},{"metadata":{"trusted":true,"_uuid":"2681746e3d7e5c25d0fad8af0f88b3b84021b6dd"},"cell_type":"code","source":"train['mjd_frac'] = np.modf(train.mjd)[0]\n\n_ = plt.hist(train.mjd_frac, bins=100)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"318a452e967e43c0c0c198dcf618e96f6672b712"},"cell_type":"markdown","source":"Seems the night starts around 0.95 and ends around 0.45. Let's shift time to have night in a consecutive integer part."},{"metadata":{"trusted":true,"_uuid":"8006ebf08087f4bd79fe8446d3c19eb20584d250"},"cell_type":"code","source":"train['mjd_frac'] = np.modf(train.mjd + 0.3)[0]\n\n_ = plt.hist(train.mjd_frac, bins=100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"edbf5345d5df6d4a321317bb3376d6aa12e2d99b"},"cell_type":"markdown","source":"Now the night starts at around 0.25 and ends around 0.75. We can define the night index accordingly."},{"metadata":{"trusted":true,"_uuid":"2f9e9aab020fb0cb90a613ff4cca383f315e7be1"},"cell_type":"code","source":"train['mjd_night'] = np.modf(train.mjd + 0.3)[1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73939d95d1f93c2fdf13a2958e9de4f6a41b8ec9"},"cell_type":"markdown","source":"Let's look for the largest of these gaps, and compute its middle date within a year by taking its fraction modulo 365.  We do it for each object_id."},{"metadata":{"trusted":true,"_uuid":"d7c75dd6f583b443d16f754f016aac291684585b"},"cell_type":"code","source":"def middle_gap(s):\n    s = s.values\n    s_prev = np.roll(s, 1)\n    s_delta = s - s_prev\n    s_delta_max = np.argmax(s_delta)\n    s_middle_gap = (s[s_delta_max] + s_prev[s_delta_max]) / 2\n    s_middle_gap = np.modf(s_middle_gap / 365)[0] * 365\n    return s_middle_gap\n    \ntrain['middle_gap'] = train.groupby('object_id'\n                                   ).mjd_night.transform(middle_gap)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"873510ded7c4776b09b240428a4f5261eeeb5fef"},"cell_type":"markdown","source":"We can see how the middle gap relates to the location of the source in earth coordinates."},{"metadata":{"trusted":true,"_uuid":"ad6fc2cd0e6ce9771dd8666729c86181edbcdd5e"},"cell_type":"code","source":"fix, ax = plt.subplots(1, 1, figsize=(12, 12))\nplt.title('Middle gap')\nplt.ylabel('Decl')\nplt.xlabel('Ra')\nax.scatter(train.ra, train.decl, \n           cmap='rainbow', c=train.middle_gap, marker='+')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4539a9929ef75ee36122e031a475c85c9ed13597"},"cell_type":"markdown","source":"It seems that the  middle gap is highly correlated with the ra coordinate."},{"metadata":{"trusted":true,"_uuid":"68b0711525296c85851a0e229fc59a0152651a35"},"cell_type":"code","source":"fix, ax = plt.subplots(1, 1, figsize=(12, 12))\nplt.ylabel('Middle gap')\nplt.xlabel('Ra')\nax.scatter(train.ra, train.middle_gap, marker='+')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1b0de61ee1e452e2254c4e5d9f26f11090b9833"},"cell_type":"markdown","source":"We see that the middle gap and ra are proportional to each other, modulo 365 for the middle gap, and modulo 360 for ra."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}