{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport matplotlib as mpl\nfrom cycler import cycler\nimport scipy.io\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n   for filename in filenames:\n       print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Some words on this notebook\n\nUnder the pretext of 'analysing' a bit the EEG files, I created this notebook which in reality is more a playground for me to get familiar with the data. For this prupose I wanted to investigate how to extract the information from the .mat files, how to plot the data of the EEG files, etc.\nI also produced some artificial data and tried to find patterns that could be used to make a prediction.\nI also wanted to see if it was possible to use the image of an espectogram for the same purposes.\nAll this without understanding really anything about EEG.\n\nAfter a bit of research I realized it could have been interesting to know what kind of montage was used to produce these EEG values, so that we could train an algorithm to identify which electrodes (and the value differences between them) are more likely to yield a good prediction. However, after looking at the EEG plot (below) it seems that the EEGs were produced using one common reference for all electrode measurements.\n\n\nSo, I don't want to spend too much time on getting educated on the vast world of neuro science interpretations. Since I wasn't able to -very- easily find something visually useful, I will stop this notebook and instead I will go directly into trying to train an RNN algorithm for predictions.","metadata":{}},{"cell_type":"code","source":"#Visualizing an example:\ninterictal_tst = '/kaggle/input/seizure-prediction/Patient_1/Patient_1/Patient_1_interictal_segment_0001.mat'\ninterictal_data = scipy.io.loadmat(interictal_tst)          # loads the file as a dictionary\nfor item in interictal_data.items():\n    print(item)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"interictal_data.get('interictal_segment_1')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check for data inconsistencies, like for example, that there are as much data samples needed as data seconds and that these two correspond to the data samples per seconds.","metadata":{}},{"cell_type":"code","source":"search_key = \"_segment_\"       # search key string\n\nfor dirname, _, filenames in os.walk('/kaggle/input/seizure-prediction/'):\n    for filename in filenames:\n        if filename != \"sampleSubmission.csv\":\n            try:\n                segment_dict = scipy.io.loadmat(os.path.join(dirname, filename))        # loads the .mat file as a dictionary\n                segment = dict(filter(lambda item: search_key in item[0], segment_dict.items()))  # retrieves only the item which contain the data of interes\n                segment_list_of_values = list(segment.values())\n                total_num_dataSamples = np.array(segment_list_of_values[0][0][0][0].shape[1], dtype=np.float)\n                datarows_duration_secs = np.array(segment_list_of_values[0][0][0][1], dtype=np.float)\n                datasamples_perSecond = np.array(segment_list_of_values[0][0][0][2], dtype=np.float)\n\n                if total_num_dataSamples != np.around((datarows_duration_secs * datasamples_perSecond)):\n                    print(\"Data inconsistency in file: %s\" % os.path.join(dirname, filename))\n            except:\n                e = sys.exc_info()[0]\n                print(\"Exception %s caught on file %s \" % (e, os.path.join(dirname, filename)))\n                break\nprint(\"Check done.\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Does all the files of the same patient or dog (i.e. Patient_1 or Patient_2 or Dog_1, etc.) have the same number of electrodes?\nAnd do they all have the same names?","metadata":{}},{"cell_type":"code","source":"search_key = \"_segment_\"       # search key string\nsubdirs = [\n    '/kaggle/input/seizure-prediction/Patient_1/Patient_1',\n    '/kaggle/input/seizure-prediction/Patient_2/Patient_2',\n    '/kaggle/input/seizure-prediction/Dog_1/Dog_1',\n    '/kaggle/input/seizure-prediction/Dog_2/Dog_2',\n    '/kaggle/input/seizure-prediction/Dog_3/Dog_3',\n    '/kaggle/input/seizure-prediction/Dog_4/Dog_4',\n    '/kaggle/input/seizure-prediction/Dog_5/Dog_5',\n]\n\nfor dirname in subdirs:\n    for dirname, _, filenames in os.walk(dirname):\n        # auxiliar variables needed to monitor both the\n        # number of electrodes and electrode names between\n        # the last file analyzed and the current file analyzed\n        numElectrodes_lastFile = 0\n        firstComparisonDone = False\n        lastFileName = \"\"\n        electrodeNames_lastFile = []\n        for filename in filenames:\n            matFile = scipy.io.loadmat(os.path.join(dirname, filename))\n            segment = dict(filter(lambda item: search_key in item[0], matFile.items()))  # retrieves only the item which contain the data of interes\n            segment_list_values = list(segment.values())\n            # number of electrodes on the currently analyzed file\n            numElectrodes_currentFile = segment_list_values[0][0][0][3][0].shape[0]\n            # next two lines of code are relevant only if during the\n            # first cycle of this loop\n            numElectrodes_lastFile = numElectrodes_currentFile if True is not firstComparisonDone else numElectrodes_lastFile\n            firstComparisonDone = True if True != firstComparisonDone else firstComparisonDone\n            # check if number of electrodes between the current file and the last\n            # file analyzed are equal\n            if numElectrodes_currentFile != numElectrodes_lastFile:\n                print(\"Discrepancy in number of electrodes between files:\")\n                print(\"     Last file    %s with %d electrodes\" % (lastFileName, numElectrodes_lastFile))\n                print(\"     Current file %s with %d electrodes\" % (filename, numElectrodes_currentFile))\n            numElectrodes_lastFile = numElectrodes_currentFile\n            # collect the name of the electrodes on the current file being analyzed\n            electrode_names = [name[0] for name in segment_list_values[0][0][0][3][0]]\n            if len(electrodeNames_lastFile) > 0:\n                # check if the electrode names of the last file analyzed and \n                # the current file analyzed are equal\n                commonNames = set(electrodeNames_lastFile) & set(electrode_names)\n                if len(commonNames) != len(electrode_names):\n                    print(\"Different electrode names on files: \")\n                    print(\"     %s (last faile name)\" % lastFileName)\n                    print(\"         Electrode names: \", electrodeNames_lastFile)\n                    print(\"     %s \" % filename)\n                    print(\"         Electrode names: \", electrode_names)\n            electrodeNames_lastFile = electrode_names\n            lastFileName = filename\nprint(\"Check done\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here's an example of how an eeg looks like.","metadata":{}},{"cell_type":"code","source":"# Code in this cell was partially taken from:\n# https://matplotlib.org/3.3.1/gallery/specialty_plots/mri_with_eeg.html\n\nimport matplotlib.pyplot as plt\nfrom matplotlib.collections import LineCollection\nimport numpy as np\n\nfile_to_inspect = scipy.io.loadmat('/kaggle/input/seizure-prediction/Patient_1/Patient_1/Patient_1_interictal_segment_0001.mat')\n# retrieves only the item which contain the data of interese\nsearch_key = '_segment_'\nsegment = dict(filter(lambda item: search_key in item[0], file_to_inspect.items()))\nsegment = list(segment.values())\ndata = segment[0][0][0][0]\nnum_electrodes = data.shape[0]             # rows (i.e. electrodes) in the data matrix\nn_samples = data.shape[1]                  # number of samples on each row (i.e. electrode's samples)\nelectrode_names = segment[0][0][0][3][0]   # name or labels of the electrodes\nt = 10 * np.arange(n_samples) / n_samples\n\n# create the 'figure'\nfig = plt.figure(\"EEG samples\",figsize=(30,15))\n\nticklocs = []\nax = fig.add_subplot(1, 1, 1)\nax.set_xlim(0, 10)\nax.set_xticks(np.arange(10))\ndata_min = data.min()\ndata_max = data.max()\ndr = (data_max - data_min) * 0.7   #crowd it a bit\ny0 = data_min\ny1 = (num_electrodes -1) * dr + data_max\nax.set_ylim(y0, y1)\n\nsegs = []\nfor electrode in range(num_electrodes):\n    segs.append(np.column_stack((t, data[electrode, :])))\n    ticklocs.append(electrode * dr)\n    \noffsets = np.zeros((num_electrodes, 2), dtype=float)\noffsets[:, 1] = ticklocs\n\nlines = LineCollection(segs, offsets=offsets, transOffset=None)\nax.add_collection(lines)\n\n# Set the yticks to use axes coordinates on the y axis\nax.set_yticks(ticklocs)\nax.set_yticklabels(electrode_names)\n\nax.set_xlabel('Time (minutes)')\n\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Figure out if the average maximum and average minimum levels of each one of the subjects (for both the interictal and preictal measurements) are similar (or near form each other).\nIn the following cells it will be done:\n * Create a .csv file for each subject and its ictal and preictal measurements. Such .csv files will contain the average of each channel on each one of the 10 seconds sub-segments for both the ictal and precital .mat files\n * Plot all the averages to be able to make a visual comparison","metadata":{}},{"cell_type":"code","source":"search_key = \"_segment_\"       # search key string\nsubdirs = [\n    '/kaggle/input/seizure-prediction/Patient_1/Patient_1',\n    '/kaggle/input/seizure-prediction/Patient_2/Patient_2',\n    '/kaggle/input/seizure-prediction/Dog_1/Dog_1',\n    '/kaggle/input/seizure-prediction/Dog_2/Dog_2',\n    '/kaggle/input/seizure-prediction/Dog_3/Dog_3',\n    '/kaggle/input/seizure-prediction/Dog_4/Dog_4',\n    '/kaggle/input/seizure-prediction/Dog_5/Dog_5',\n]\nfor dirname in subdirs:\n    for dirname, _, filenames in os.walk(dirname):\n        electrodeNamesInterictal_pos = {}\n        electrodeNamesInterictal_neg = {}\n        electrodeNamesPreictal_pos = {}\n        electrodeNamesPreictal_neg = {}\n        # electrodesAllValues = {} --> ArHe\n        print(\"In directory: \", os.path.basename(os.path.normpath(dirname)))\n        # create a list of all the files in the current directory\n        # and sort them alphabetically\n        files = []\n        for f in filenames:\n            files.append(f)\n        files = sorted(files)\n        for file in files:\n            matFile = scipy.io.loadmat(os.path.join(dirname, file))\n            # retrieves only the item which contain the data of interes\n            segment = dict(filter(lambda item: search_key in item[0], matFile.items()))\n            segment_list_values = list(segment.values())\n            # number of electrodes on the currently analyzed file\n            for idx, electrode in enumerate(segment_list_values[0][0][0][3][0]):\n                maxValue = segment_list_values[0][0][0][0][idx].max()\n                minValue = segment_list_values[0][0][0][0][idx].min()\n                if '_interictal_' in file:\n                    if electrode[0] not in electrodeNamesInterictal_pos.keys():\n                        electrodeNamesInterictal_pos[electrode[0]] = [maxValue]\n                    else:\n                        #electrodeNamesInterictal_pos[electrode[0]] = electrodeNamesInterictal_pos[electrode[0]] + (maxValue,)\n                        electrodeNamesInterictal_pos[electrode[0]].append(maxValue)\n                    if electrode[0] not in electrodeNamesInterictal_neg.keys():\n                        electrodeNamesInterictal_neg[electrode[0]] = [minValue]\n                    else:\n                        #electrodeNamesInterictal_neg[electrode[0]] = electrodeNamesInterictal_neg[electrode[0]] + (minValue,)\n                        electrodeNamesInterictal_neg[electrode[0]].append(minValue)\n                    ## I need also to store the values of the electrodes in sequencial order...\n                    #if electrode[0] not in electrodesAllValues.keys():\n                    #    electrodesAllValues[electrode[0]] = np.array(segment_list_values[0][0][0][0][idx])\n                    #else:\n                    #    electrodesAllValues[electrode[0]] = np.concatenate(\n                    #        (electrodesAllValues[electrode[0]], segment_list_values[0][0][0][0][idx]), axis=None\n                    #    ) --> ArHe\n                elif '_preictal_' in file:\n                    if electrode[0] not in electrodeNamesPreictal_pos.keys():\n                        electrodeNamesPreictal_pos[electrode[0]] = [maxValue]\n                    else:\n                        #electrodeNamesPreictal_pos[electrode[0]] = electrodeNamesPreictal_pos[electrode[0]] + (maxValue,)\n                        electrodeNamesPreictal_pos[electrode[0]].append(maxValue)\n                    if electrode[0] not in electrodeNamesPreictal_neg.keys():\n                        electrodeNamesPreictal_neg[electrode[0]] = [minValue]\n                    else:\n                        #electrodeNamesPreictal_neg[electrode[0]] = electrodeNamesPreictal_neg[electrode[0]] + (minValue,)\n                        electrodeNamesPreictal_neg[electrode[0]].append(minValue)\n                    ## I need also to store the values of the electrodes in sequencial order...\n                    #if electrode[0] not in electrodesAllValues.keys():\n                    #    electrodesAllValues[electrode[0]] = np.array(segment_list_values[0][0][0][0][idx])\n                    #else:\n                    #    electrodesAllValues[electrode[0]] = np.concatenate(\n                    #        (electrodesAllValues[electrode[0]], segment_list_values[0][0][0][0][idx]), axis=None\n                    #    ) --> ArHe\n        \n        # for now path is just the name of the folder holding the files\n        basePath = os.path.basename(os.path.normpath(dirname))\n        # writing interictal file\n        path = basePath + \"_interictal.csv\"\n        electrodeNamesInterictal_dict = {'max': electrodeNamesInterictal_pos, 'min': electrodeNamesInterictal_neg}\n        df = pd.DataFrame(electrodeNamesInterictal_dict)\n        df.to_csv(path)\n        print(\"File %s written.\" % path)\n        # writing preictal file\n        path = basePath + \"_preictal.csv\"\n        electrodeNamesPreictal_dict = {'max': electrodeNamesPreictal_pos, 'min': electrodeNamesPreictal_neg}\n        df = pd.DataFrame(electrodeNamesPreictal_dict)\n        df.to_csv(path)\n        print(\"File %s written.\" % path)\n        ## writing all electrode values\n        #path = basePath + \"_allElectrodeValues.csv\"\n        #df = pd.DataFrame(electrodesAllValues)\n        #df.to_csv(path)\n        #print(\"File %s written\" % path)\nprint(\"Done\")","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plotMaxMinAveragesDF(dataframe, title):\n    df = pd.read_csv(dataframe, converters={'max': eval, 'min':eval}, index_col=0)\n    fig = plt.figure(figsize=(30,15))\n    ax = fig.add_subplot(1,1,1)\n    mpl.rcParams['axes.prop_cycle'] = cycler(color=[\n        '#1f77b4', '#1f77b4',\n        '#ff7f0e', '#ff7f0e',\n        '#2ca02c', '#2ca02c',\n        '#d62728', '#d62728',\n        '#9467bd', '#9467bd',\n        '#8c564b', '#8c564b',\n        '#e377c2', '#e377c2',\n        '#7f7f7f', '#7f7f7f',\n        '#bcbd22', '#bcbd22',\n        '#17becf', '#17becf',\n        '#aaaaaa', '#aaaaaa',\n        '#123456', '#132456',\n        '#cdcdcd', '#cdcdcd',\n        '#ff00ff', '#ff00ff',\n        '#000000', '#000000',\n        '#174038', '#174038',\n        '#362819', '#362819',\n        '#4B0082', '#4B0082',\n        '#76EE00', '#76EE00',\n        '#8B7500', '#8B7500',\n        '#FF4500', '#FF4500',\n        '#FF83FA', '#FF83FA',\n        '#FFE600', '#FFE600',\n        '#CC4E5C', '#CC4E5C',])\n    legend = []\n    for electrode in df.T.columns:\n        if electrode not in legend:\n            legend.append(electrode)\n            legend.append('')\n        for avrArray in df.T[electrode]:\n            ax.plot(avrArray, 'o:')\n    plt.legend(legend)\n    plt.title(title)\n    plt.show()\n\naveragesFiles = []\nfor dirname, _, filenames in os.walk('./'):\n    for f in filenames:\n        averagesFiles.append(f)\n        averagesFiles.sort()\nfor f in averagesFiles:\n    if '_notebook_' not in f:\n        plotMaxMinAveragesDF(os.path.join(dirname, f), f)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observations on the plots:\n * There seems to exist some sort of symmetry between the positive and negative averages curves.\n * The values of both positive and negative curves are different between dogs for all the interictal and preictal curves as well as the values for the human patients. That is, it doen't seem to be a relation between the positive and negative values of the different dog patients nor of the different human patients.\n * On the human patients, the values of the curves during the preictal measurements get very close to zero forming an almost flat line pattern.\n * On the dogs, the values of the curves during the preictal measurments seem to follow some sort of alignment pattern which doesn't seem to exist on the interictal measurments. This sharpens the symmetry pattern on the preictal curves.\n\n In general it is not clear if there is really information on these graphs that could be used as clear indications of whether and when a seasure is going to occur other the fact that the values of the electrodes (all of them) seem to stay close around the same values during a preictal phase.\n\n Maybe an alternative for visual inspection could be to produce a spectogram of the patients EEGs.","metadata":{}},{"cell_type":"markdown","source":"How does an espectogram of an eeg electrode looks like? Could we use it to make a prediction?","metadata":{}},{"cell_type":"code","source":"def plotSpectograms(file, electrode=0):\n    search_key = \"_segment_\"       # search key string\n    matFile = scipy.io.loadmat(os.path.join(file))\n    segment = dict(filter(lambda item: search_key in item[0], matFile.items()))\n    segment_list_values = list(segment.values())\n\n    x = np.array(segment_list_values[0][0][0][0][electrode], dtype=np.float)\n    secs = np.array(segment_list_values[0][0][0][1], dtype=np.float)\n    Fs = np.array(segment_list_values[0][0][0][2], dtype=np.float)[0][0]\n    NFFT = 1024  # the length of the windowing segments\n    dt = 1/Fs\n    t = np.arange(0.0, 600.0, dt)\n\n    fig, (ax1, ax2) = plt.subplots(nrows=2, figsize=(15,10))\n    ax1.plot(t, x)\n    Pxx, freqs, bins, im = ax2.specgram(x, NFFT=NFFT, Fs=Fs, noverlap=100, sides='twosided')\n    # The `specgram` method returns 4 objects. They are:\n    # - Pxx: the periodogram\n    # - freqs: the frequency vector\n    # - bins: the centers of the time bins\n    # - im: the .image.AxesImage instance representing the data in the plot\n    plt.show()\n    \nfile = [\n    '/kaggle/input/seizure-prediction/Patient_1/Patient_1/Patient_1_interictal_segment_0001.mat',\n    '/kaggle/input/seizure-prediction/Patient_1/Patient_1/Patient_1_preictal_segment_0001.mat',\n    '/kaggle/input/seizure-prediction/Dog_3/Dog_3/Dog_3_interictal_segment_0005.mat',\n    '/kaggle/input/seizure-prediction/Dog_3/Dog_3/Dog_3_preictal_segment_0005.mat'\n]\nfor f in file:\n    plotSpectograms(f, electrode=5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}