{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"}],"dockerImageVersionId":30648,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":22321.864322,"end_time":"2024-01-28T12:56:49.320132","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-28T06:44:47.45581","version":"2.4.0"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"020b844a5f1c4a998793589345d58187":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"387ef6c55cf34ee8b06cef960ab5709e":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"4551e56c177e48c7b32c50244181acce":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_020b844a5f1c4a998793589345d58187","placeholder":"​","style":"IPY_MODEL_a76c83a1114641e9a0278861a509523d","value":" 21.4M/21.4M [00:00&lt;00:00, 39.7MB/s]"}},"54c978d8e366477fa1f2e4de7d812eaa":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"585b9ad521bc42f88613a33772d14b09":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_54c978d8e366477fa1f2e4de7d812eaa","placeholder":"​","style":"IPY_MODEL_387ef6c55cf34ee8b06cef960ab5709e","value":"model.safetensors: 100%"}},"5c5e6c736e884230a714ede1aac584e8":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"5d247d0364d444f4bda51aea5d8e9103":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_585b9ad521bc42f88613a33772d14b09","IPY_MODEL_f84cec30df65438192b9c608c0d1d8b4","IPY_MODEL_4551e56c177e48c7b32c50244181acce"],"layout":"IPY_MODEL_5c5e6c736e884230a714ede1aac584e8"}},"a76c83a1114641e9a0278861a509523d":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"f34a838e5b6b4dcc82ea12ad3977b520":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"f84cec30df65438192b9c608c0d1d8b4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_fe2e5aa594d24e889081a556aebdec4f","max":21355344,"min":0,"orientation":"horizontal","style":"IPY_MODEL_f34a838e5b6b4dcc82ea12ad3977b520","value":21355344}},"fe2e5aa594d24e889081a556aebdec4f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}}},"version_major":2,"version_minor":0}}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n## Acknowledgements\nThe original base of this notebook was copied from @andreasbis. We thank them for supplying a useful baseline to expand upon. Please take a look at their work: https://www.kaggle.com/code/andreasbis/hms-train-efficientnetb1.","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import gc\nimport os\nimport random\nimport warnings\nfrom IPython.display import display\n\nimport numpy as np\nimport pandas as pd\n\nimport timm\nimport torch\nimport torch.nn as nn  \nimport torch.optim as optim\nimport torch.nn.functional as F\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nimport torchvision.transforms as transforms\nimport matplotlib.pyplot as plt\n\nfrom scipy import signal\n\nwarnings.filterwarnings('ignore', category=Warning)\ngc.collect()","metadata":{"papermill":{"duration":5.810106,"end_time":"2024-01-28T06:44:56.777717","exception":false,"start_time":"2024-01-28T06:44:50.967611","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setup","metadata":{"papermill":{"duration":0.003167,"end_time":"2024-01-28T06:44:56.784246","exception":false,"start_time":"2024-01-28T06:44:56.781079","status":"completed"},"tags":[]}},{"cell_type":"code","source":"labels = ['seizure', 'lpd', 'gpd', 'lrda', 'grda', 'other']\n\nclass Config:\n    seed = 3131 \n    image_transform = transforms.Resize((512,512))  \n    batch_size = 16\n    num_epochs = 9\n    num_folds = 5\n\ndef set_seed(seed):\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    \n    torch.manual_seed(seed)\n    np.random.seed(seed)\n    random.seed(seed)\n\ndef kl_loss(p, q):\n    epsilon = 10 ** (-15)\n    \n    p = torch.clamp(p, epsilon, 1 - epsilon)\n    log_p = torch.log(p)\n    log_q = nn.functional.log_softmax(q, dim=1)\n    \n    kl_divergence_per_point = p * (log_p - log_q)\n    kl_divergence_per_label = torch.sum(kl_divergence_per_point, dim=1)\n    \n    return torch.mean(kl_divergence_per_label)\n\nset_seed(Config.seed)\ngc.collect()","metadata":{"papermill":{"duration":0.139048,"end_time":"2024-01-28T06:44:56.926359","exception":false,"start_time":"2024-01-28T06:44:56.787311","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loading","metadata":{"papermill":{"duration":0.003288,"end_time":"2024-01-28T06:44:56.933246","exception":false,"start_time":"2024-01-28T06:44:56.929958","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Code changed to grouping by eeg-id. \n\ntrain_df = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\")\ntrain_features = pd.DataFrame()\n\nfor label in labels:\n    train_grouped_by_eeg_id = train_df[f'{label}_vote'].groupby(train_df['eeg_id']).sum()\n\n    label_vote_sum = pd.DataFrame()\n    label_vote_sum[\"eeg_id\"] = train_grouped_by_eeg_id.index\n    label_vote_sum[f\"{label}_vote_sum\"] = train_grouped_by_eeg_id.values\n\n    if label == labels[0]:\n        train_features = label_vote_sum\n    else:\n        train_features = train_features.merge(label_vote_sum, on='eeg_id', how='left')\n\n# Add a column to sum all votes\ntrain_features['total_vote'] = 0\nfor label in labels:\n    train_features['total_vote'] += train_features[f'{label}_vote_sum']\n\n# Calculate and store the normalized vote for each label\nfor label in labels:\n    train_features[f'{label}_vote'] = train_features[f'{label}_vote_sum'] / train_features['total_vote']\n\n# Select relevant columns for the training features\nchoose_cols = ['eeg_id']\nfor label in labels:\n    choose_cols += [f'{label}_vote']\ntrain_features = train_features[choose_cols]\n\n# Add a column with the path to the spectrogram files\ntrain_features['path'] = train_features['eeg_id'].apply(lambda x: \"/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/\" + str(x) + \".parquet\")\n\n# Reclaim memory no longer in use.\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating spectrograms\nSpectrograms are created based on the 'Magic Formula to Convert EEG to Spectrograms'. (https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760)","metadata":{}},{"cell_type":"code","source":"def create_spectrogram(data):\n    \"\"\"This function will create the spectrograms based on the EEG data with the 'magic formula'.\"\"\"\n    nperseg = 150  # Length of each segment\n    noverlap = 128  # Overlap between segments\n    NFFT = max(256, 2 ** int(np.ceil(np.log2(nperseg))))\n\n    # LL Spec = ( spec(Fp1 - F7) + spec(F7 - T3) + spec(T3 - T5) + spec(T5 - O1) )/4\n    freqs, t,spectrum_LL1 = signal.spectrogram(data['Fp1']-data['F7'],nfft=NFFT,noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_LL2 = signal.spectrogram(data['F7']-data['T3'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_LL3 = signal.spectrogram(data['T3']-data['T5'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_LL4 = signal.spectrogram(data['T5']-data['O1'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n\n    LL = (spectrum_LL1+ spectrum_LL2 +spectrum_LL3 + spectrum_LL4)/4\n\n    # LP Spec = ( spec(Fp1 - F3) + spec(F3 - C3) + spec(C3 - P3) + spec(P3 - O1) )/4\n    freqs, t,spectrum_LP1 = signal.spectrogram(data['Fp1']-data['F3'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_LP2 = signal.spectrogram(data['F3']-data['C3'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_LP3 = signal.spectrogram(data['C3']-data['P3'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_LP4 = signal.spectrogram(data['P3']-data['O1'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n\n    LP = (spectrum_LP1+ spectrum_LP2 +spectrum_LP3 + spectrum_LP4)/4\n\n    # RP Spec = ( spec(Fp2 - F4) + spec(F4 - C4) + spec(C4 - P4) + spec(P4 - O2) )/4\n    freqs, t,spectrum_RP1 = signal.spectrogram(data['Fp2']-data['F4'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_RP2 = signal.spectrogram(data['F4']-data['C4'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_RP3 = signal.spectrogram(data['C4']-data['P4'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_RP4 = signal.spectrogram(data['P4']-data['O2'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n\n    RP = (spectrum_RP1+ spectrum_RP2 +spectrum_RP3 + spectrum_RP4)/4\n\n\n    # RL Spec = ( spec(Fp2 - F8) + spec(F8 - T4) + spec(T4 - T6) + spec(T6 - O2) )/4\n    freqs, t,spectrum_RL1 = signal.spectrogram(data['Fp2']-data['F8'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_RL2 = signal.spectrogram(data['F8']-data['T4'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_RL3 = signal.spectrogram(data['T4']-data['T6'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    freqs, t,spectrum_RL4 = signal.spectrogram(data['T6']-data['O2'],nfft=NFFT, noverlap = noverlap,nperseg=nperseg)\n    RL = (spectrum_RL1+ spectrum_RL2 +spectrum_RL3 + spectrum_RL4)/4\n    spectogram = np.concatenate((LL, LP,RP,RL), axis=0)\n    return spectogram","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{"papermill":{"duration":0.003848,"end_time":"2024-01-28T06:44:57.41335","exception":false,"start_time":"2024-01-28T06:44:57.409502","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def datasetwide_mean(paths):\n    \"\"\"This function will calculate the mean of the entire dataset.\"\"\"\n    data_means = []\n    total_values = 0\n    # Iterate over each path in the provided paths\n    for path in paths:\n        # Read data from parquet file\n        data = pd.read_parquet(path[0])\n        data = create_spectrogram(data)\n        \n        # Fill missing values with the specified constant\n        mask = np.isnan(data)\n        data[mask] = -1\n        \n        # Clip values and apply logarithmic transformation\n        data = np.clip(data, np.exp(-6), np.exp(10))\n        data = np.log(data)\n        \n        # Calculate sum and amount of values of the data\n        data_sum = data.sum(axis=(0, 1))\n        rows,columns = data.shape\n        total_values += rows*columns\n        \n    return data_sum/total_values\n\ndef datasetwide_std(paths, mean):\n    \"\"\"This function will calculate the standard deviation of the entire dataset.\"\"\"\n    data_stds = []\n    # Iterate over each path in the provided paths\n    sum_std = 0\n    total_values = 0\n    for path in paths:\n        # Read data from parquet file\n        data = pd.read_parquet(path[0])\n        data = create_spectrogram(data)\n        \n        # Fill missing values with the specified constant\n        mask = np.isnan(data)\n        data[mask] = -1\n\n        # Clip values and apply logarithmic transformation\n        data = np.clip(data, np.exp(-6), np.exp(10))\n        data = np.log(data)\n        \n        # Calculate values needed for std\n        sum_std+= np.sum((data-mean)**2)\n        rows,columns = data.shape\n        total_values += rows*columns\n    \n    return np.sqrt(sum_std/(total_values-1))\n\n# Calculate the mean and std of dataset. \ndata_mean = datasetwide_mean(train_features[['path']].values)\ndata_std = datasetwide_std(train_features[['path']].values,mean)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_batch_datasetwidenorm(paths,data_mean,data_std, batch_size=Config.batch_size):\n    \"\"\"This function will get the batch and preprocess it.\"\"\"\n    # Set a small epsilon to avoid division by zero\n    eps = 1e-6\n\n    # Initialize a list to store batch data\n    batch_data = []\n\n    # Iterate over each path in the provided paths\n    for path in paths:\n        # Read data from parquet file\n        data = pd.read_parquet(path[0])\n        data = create_spectrogram(data)\n        \n        # Fill missing values with the specified constant\n        mask = np.isnan(data)\n        data[mask] = -1\n        \n        # Clip values and apply logarithmic transformation\n        data = np.clip(data, np.exp(-6), np.exp(10))\n        data = np.log(data)\n        \n        # Normalize the data\n        data = (data - data_mean) / (data_std + eps)\n\n        # Convert data to a PyTorch tensor and apply transformations\n        data_tensor = torch.unsqueeze(torch.Tensor(data), dim=0)\n        data = Config.image_transform(data_tensor)\n\n        # Append the processed data to the batch_data list\n        batch_data.append(data)\n\n    # Stack all the batch data into a single tensor\n    batch_data = torch.stack(batch_data)\n\n    # Return the batch data\n    return batch_data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training","metadata":{"papermill":{"duration":0.003763,"end_time":"2024-01-28T06:44:57.440662","exception":false,"start_time":"2024-01-28T06:44:57.436899","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def train(lr = 0.001 ,data_mean=0,data_std=1):\n    # Cross-validation loop\n    train_losses_folds = []\n    test_losses_folds = []\n    for fold in range(Config.num_folds):\n        # Split data into train and test sets for this fold\n        test_idx = total_idx[fold * len(total_idx) // Config.num_folds:(fold + 1) * len(total_idx) // Config.num_folds]\n        \n        #If folds is put to 1, the function will still work. \n        if Config.num_folds==1:\n            start = np.random.choice(len(total_idx)- int(np.round(0.2*len(total_idx)))-1)\n            end = start + int(np.round(0.2*len(total_idx)))\n            test_idx = total_idx[start:end]\n            \n        train_idx = np.array([idx for idx in total_idx if idx not in test_idx])\n\n        # Initialize EfficientNet-B1 model with pretrained weights\n        model = timm.create_model('efficientnet_b1', pretrained=True, num_classes=6, in_chans=1)\n        model.to(device)\n        \n        optimizer = optim.AdamW(model.parameters(), lr=lr, betas=(0.5, 0.999), weight_decay=0.01)\n        scheduler = CosineAnnealingLR(optimizer, T_max=Config.num_epochs)\n\n        best_test_loss = float('inf')\n        train_losses = []\n        test_losses = []\n\n        print(f\"Starting training for fold {fold + 1}\")\n\n        # Training loop\n        for epoch in range(Config.num_epochs):\n            model.train()\n            train_loss = []\n            random_num = np.arange(len(train_idx))\n            np.random.shuffle(random_num)\n            train_idx = train_idx[random_num]\n            # Iterate over batches in the training set\n            for idx in range(0, len(train_idx), Config.batch_size):\n                optimizer.zero_grad()\n                train_idx1 = train_idx[idx:idx + Config.batch_size]\n                train_X1_path = train_features[['path']].iloc[train_idx1].values\n                \n                # Normalize data with given mean and std\n                train_X1 = get_batch_datasetwidenorm(train_X1_path,data_mean,data_std, batch_size=Config.batch_size)\n                \n                train_y1 = train_features[['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']].iloc[train_idx1].values\n                train_y1 = torch.Tensor(train_y1)\n                train_pred = model(train_X1.to(device))\n                loss = kl_loss(train_y1.to(device), train_pred)\n                loss.backward()\n                optimizer.step()\n                train_loss.append(loss.item())\n\n            epoch_train_loss = np.mean(train_loss)\n            train_losses.append(epoch_train_loss)\n            print(f\"Epoch {epoch + 1}: Train Loss = {epoch_train_loss:.2f}\")\n            \n            scheduler.step()\n\n            # Evaluation loop\n            model.eval()\n            test_loss = []\n            with torch.no_grad():\n                for idx in range(0, len(test_idx), Config.batch_size):\n                    test_idx1 = test_idx[idx:idx + Config.batch_size]\n                    test_X1_path = train_features[['path']].iloc[test_idx1].values\n                    \n                    # Normalize data with given mean and std. \n                    test_X1= get_batch_datasetwidenorm(test_X1_path,data_mean,data_std, batch_size=Config.batch_size)\n                    \n                    test_y1 = train_features[['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']].iloc[test_idx1].values\n                    test_y1 = torch.Tensor(test_y1)\n\n                    test_pred = model(test_X1.to(device))\n                    loss = kl_loss(test_y1.to(device), test_pred)\n                    test_loss.append(loss.item())\n\n            epoch_test_loss = np.mean(test_loss)\n                \n            test_losses.append(epoch_test_loss)\n            print(f\"Epoch {epoch + 1}: Test Loss = {epoch_test_loss:.2f}\")\n\n            # Save the model if it has the best test loss so far\n            if epoch_test_loss < best_test_loss:\n                best_test_loss = epoch_test_loss\n                torch.save(model.state_dict(), f\"efficientnet_b1_fold{fold}.pth\")\n\n            gc.collect()\n\n        print(f\"Fold {fold + 1} Best Test Loss: {best_test_loss:.2f}\")\n        train_losses_folds.append(train_losses)\n        test_losses_folds.append(test_losses)\n        \n    return train_losses_folds, test_losses_folds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_loss_graph(train_losses_folds, test_losses_folds):\n    \"\"\"This function will plot the loss curves of all the different folds.\"\"\"\n    for i in range(len(train_losses_folds)):\n        train_losses = train_losses_folds[i]\n        test_losses = test_losses_folds[i]\n        plt.plot(train_losses, label='Train Loss')\n        plt.plot(test_losses, label='Test Loss')\n        plt.xlabel('Epoch')\n        plt.ylabel('Loss')\n        plt.legend()\n        plt.savefig(f'fold_{i}.png')\n        plt.close()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Determine device availability\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint(f\"Using device: {device}\")\n\n# Assuming train_features is defined and contains the training features and labels\ntotal_idx = np.arange(len(train_features))\nnp.random.shuffle(total_idx)\n\n# The model will be trained\ntrain_losses_folds, test_losses_folds = train(lr = 0.001,data_mean=data_mean,data_std=data_std)\nplot_loss_graph(train_losses_folds, test_losses_folds)\ngc.collect()","metadata":{"papermill":{"duration":22310.196541,"end_time":"2024-01-28T12:56:47.64122","exception":false,"start_time":"2024-01-28T06:44:57.444679","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]}]}