{"cells":[{"metadata":{"_uuid":"d873444b96f33b95042a08ba3af8c351d7f49f8e"},"cell_type":"markdown","source":"The kernel contains some basic visualizations for the DMDT Images that are described in the kernel - [CNN based Classification of Light Curves](https://www.kaggle.com/pankajb64/cnn-based-classification-of-light-curves/)\n\nThe DMDT Images are included here as an additional dataset.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nimport pickle\nimport multiprocessing\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport dask.dataframe as dd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tnrange, tqdm_notebook\nfrom collections import OrderedDict\nfrom sklearn.model_selection import StratifiedShuffleSplit\nfrom datetime import datetime, timedelta\nfrom matplotlib import gridspec\n\nprint(os.listdir(\"../input/\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc02a3aeee3310a0dfd71103a53b877c0bcc674a"},"cell_type":"markdown","source":"We'll set warning to ignore, since matplotlib generates a few warnings about incompatible axes in our case when displaying plots."},{"metadata":{"trusted":true,"_uuid":"71779c3c754cca02c6453a845fde2cc6492e24f6"},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"62efc070ac105325037c096a33f2515eb81ed9a7"},"cell_type":"markdown","source":"Read it the data frames for the meta-data and the time series data for objects in the training set."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/PLAsTiCC-2018/training_set.csv')\ndf_meta = pd.read_csv('../input/PLAsTiCC-2018/training_set_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e26047f2d044185ba404ceb7791b716c8b1dd7ec"},"cell_type":"markdown","source":"Lets convert the Modified Julian Date to a Pandas datetime object so it's easier to read on the plots. To do that, we first convert the MJD to a unix timestamp and then parse it as a datetime."},{"metadata":{"trusted":true,"_uuid":"636cf7733ebc0f5c5a41538fc5f0a2c0049c93a5"},"cell_type":"code","source":"df['unix_time'] = (df['mjd'] - 40587)*86400\ndf['datetime'] = pd.to_datetime(df['unix_time'], unit='s')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a7f4012e54755a031a5922643ead8ef5dc45e53"},"cell_type":"markdown","source":"The code to load the dmdt images, same as in the parent kernel."},{"metadata":{"trusted":true,"_uuid":"aa94730c521a5720acbacc915cee403a650a5f9b"},"cell_type":"code","source":"objects = df_meta['object_id'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4daba693b71275168139c64129161020af524068"},"cell_type":"code","source":"def load_dmdt_images(objects, base_dir='train'):\n    dmdt_img_dict = OrderedDict()\n    for obj in objects:\n        key = '{}/{}_dmdt.pkl'.format(base_dir, obj)\n        if os.path.isfile(key):\n            with(open(key, 'rb')) as f:\n                dmdt_img_dict[obj] = pickle.load(f)\n    return dmdt_img_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc09d541edac44fed1932929f49eb5342433b0d0"},"cell_type":"code","source":"dmdt_img_dict = load_dmdt_images(objects, '../input/plasticc_dmdt_images/train/train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b593127ac5e2b84596e84d69e1b2979e326dbb2e"},"cell_type":"code","source":"classes = np.sort(df_meta['target'].drop_duplicates().values)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ddb1d9a51300bc3ceb83c12cb1eefd23fa2bb7ac"},"cell_type":"markdown","source":"Lets look at the distribution of objects across classes."},{"metadata":{"trusted":true,"_uuid":"8147740313e5db41e05df8c5837ca1847a53d40d"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,6))\nax = sns.countplot(df_meta['target'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"192792f5a2ce0cbc607d0db1def5e32d6aa30a6d"},"cell_type":"markdown","source":"Its uneven, so there is a class imbalance. Lets take a sample object per class and look at its Time-series light curve and its DMDT Image."},{"metadata":{"trusted":true,"_uuid":"daf79529486a83bd0846a82d029e09f4edf5350d"},"cell_type":"code","source":"samples = df_meta.groupby('target')['object_id', 'target'].head(1).values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e18e0a7f4d398790f5d2abf5f9187d9429412ff"},"cell_type":"markdown","source":"The code below generates the plots for each object in the sample. The two plots are shown side by side to make it easy to associate characteristics.\n\nNote - You'll see 7 legend entries in the time-series plot, this is because I'm doing a group by passband and generating a plot for each group, and pandas calls apply twice on the first group (they do it for code optimization, and presently there is no way around it), so 7 different scatter plots are generted. I couldn't find a way to remove the duplicate label from the legend, but if you know how to, let me know!"},{"metadata":{"trusted":true,"_uuid":"f90e5ee5a3152b46e91190e72f79be2955c01515"},"cell_type":"code","source":"def gen_plots(df, samples):\n    for sample in samples:\n        fig = plt.figure(figsize=(21,9))\n        cbar_ax = fig.add_axes([.91, .3, .03, .4])\n        outer_grid = gridspec.GridSpec(1, 2)\n        object_id = sample[0]\n        label = sample[1]\n        df_obj = df.loc[df.object_id == object_id]\n        gen_flux_plots(df_obj, object_id, label, outer_grid[0], fig)\n        viz_dmdt(object_id, label, outer_grid[1], fig, cbar_ax)\n        fig.suptitle(\"Time-series Light Curve and DMDT Images for all 6 passband for object ID - {} of class {}\".format(object_id, label), fontsize=16)\n        #rect=[0, 0, 0.91, 0.95]\n        fig.tight_layout(rect=[0, 0, 0.91, 0.95])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"081f4da6c8b28df8de3f7304ec2105a29c6b2ecd"},"cell_type":"code","source":"def gen_flux_plot(df, ax, labels):\n    passband = df['passband'].drop_duplicates().values[0]\n    label = labels[passband]\n    sns.scatterplot(ax=ax, x=df['datetime'], y=df['flux'], label=label)\n    ax.set_xlim(df.iloc[0]['datetime'] - timedelta(days=20), df.iloc[-1]['datetime'] + timedelta(days=20))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6bf294009168feaf747d9696407f126dfc0a25f1"},"cell_type":"code","source":"def gen_flux_plots(df, object_id, label, outer_grid, fig):\n    ax = fig.add_subplot(outer_grid)\n    labels = ['u', 'g', 'r', 'i', 'z', 'Y']\n    sps = df.groupby('passband').apply(lambda x : gen_flux_plot(x, ax, labels))\n    ax.legend()\n    fig.add_subplot(ax)\n    #fig.suptitle('Time-series Light Curve for all 6 passbands for object - {} of class {}'.format(object_id, label), fontsize=16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7e2534f603b2b9dd672ac3ec9063a05f9348884"},"cell_type":"code","source":"def viz_dmdt(object_id, label, outer_grid, fig, cbar_ax):\n    dmdt_img = dmdt_img_dict[object_id]\n    inner_grid = gridspec.GridSpecFromSubplotSpec(2, 3, subplot_spec=outer_grid)\n    shared_ax = None\n    for i in range(6): #num passband\n        i_idx = 0 if i < 3 else 1\n        j_idx = i%3\n        gs = inner_grid[i_idx, j_idx]\n        ax = fig.add_subplot(gs) if shared_ax is None else fig.add_subplot(gs, sharex=shared_ax, sharey=shared_ax)\n        sns.heatmap(ax=ax, data=dmdt_img[:,:,i], cmap=\"hot\", cbar=(i==0), cbar_ax=None if i else cbar_ax)\n    #fig.suptitle(\"DMDT Images for all 6 passband for object ID - {} of class {}\".format(object_id, label), fontsize=16)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"_kg_hide-input":false,"trusted":true,"_uuid":"de39f087ad3bc6acd265e37720020c9b1ca6f248"},"cell_type":"code","source":"gen_plots(df, samples)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a35ffd8760c01b12d0cbde3d6272d795b82e3125"},"cell_type":"markdown","source":"- Load the model, with the custom loss function.\n- Get individual losses for each of the sample in the training set (since we don't really know which was train and which was val originally) - maybe do this in the original kernel post training ?\n- Sort by loss values, descending.\n- For the top k losses, generate flux plots and dmdt images.\n    - Also do this for the lowest k, to know what was it that made them easy to qualify ?\n- Maybe visualize the inner layers of the CNN ? See https://github.com/raghakot/keras-vis and https://www.codeastar.com/visualize-convolutional-neural-network/"},{"metadata":{"trusted":true,"_uuid":"78c6fde3b6b40fea773cc2c1a5a205eb3eda4950"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}