{
  "id": 471447,
  "title": "Author SPaRCNet Code insights + Extra dataset (25 eeg sequences) + Unlabelled data (not clean)",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/471447",
  "author_name": "",
  "post_date": "2024-01-28T11:24:24.109188Z",
  "votes": 25,
  "comment_count": 4,
  "views": 0,
  "content": "<h1>From the <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471439\" target=\"_blank\">shared discussion</a></h1>\n<h1>Preprocess</h1>\n<pre><code> mne.  filter_data, notch_filter\nX2 = X[[,,,, ,,,, ,,,, ,,,]] - X[[,,,, ,,,, ,,,, ,,,]]\nX2 = notch_filter(X2, , , n_jobs=-, verbose=)\nX2 = filter_data(X2, , , , n_jobs=-, verbose=) \n==&gt;  to  kHz\n==&gt; Frequencies to notch   Hz =  ?? (  idea about  used )\n==&gt; After Difference, applied notch  frequency filters ??\n</code></pre>\n<h1>Loss Function : <strong>WeightedKLDivWithLogitsLoss</strong> (</h1>\n<pre><code> (nn.KLDivLoss):\n     ():\n        (WeightedKLDivWithLogitsLoss, self).__init__(size_average=, reduce=, reduction=)\n        self.register_buffer(, weight)\n\n     ():\n        \n        \n        batch_size = .size()\n        log_prob = F.log_softmax(, )\n        element_loss = (WeightedKLDivWithLogitsLoss, self).forward(log_prob, target)\n\n        sample_loss = torch.(element_loss, dim=)\n        sample_weight = torch.(target * self.weight, dim=)\n\n        weighted_loss = sample_loss*sample_weight\n        \n        avg_loss = torch.(weighted_loss) / batch_size\n\n         avg_loss  \n</code></pre>\n<h1>Optimiser</h1>\n<pre><code>optimizer = optim.Adam(model_cnn.parameters(), lr=*, betas = (,),eps = *, weight_decay=*)\n==&gt; lr = *\n==&gt; eps = *\n==&gt; betas = (, )\n</code></pre>\n<h1>batch size</h1>\n<pre><code>batch_size = \n</code></pre>\n<h1>Model</h1>\n<pre><code> (nn.Sequential):\n     ():\n        (_DenseLayer, self).__init__()\n         batch_norm:\n            self.add_module(, nn.BatchNorm1d(num_input_features)),\n        \n        self.add_module(, nn.ELU()),\n        self.add_module(, nn.Conv1d(num_input_features, bn_size * growth_rate, kernel_size=, stride=, bias=conv_bias)),\n         batch_norm:\n            self.add_module(, nn.BatchNorm1d(bn_size * growth_rate)),\n        \n        self.add_module(, nn.ELU()),\n        self.add_module(, nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=, stride=, padding=, bias=conv_bias)),\n        \n        self.drop_rate = drop_rate\n\n     ():\n        \n        \n        new_features = (_DenseLayer, self).forward(x)\n        \n        \n         self.drop_rate &gt; :\n            new_features = F.dropout(new_features, p=self.drop_rate, training=self.training)\n         torch.cat([x, new_features], )\n\n\n (nn.Sequential):\n     ():\n        (_DenseBlock, self).__init__()\n         i  (num_layers):\n            layer = _DenseLayer(num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate, conv_bias, batch_norm)\n            self.add_module( % (i + ), layer)\n\n\n (nn.Sequential):\n     ():\n        (_Transition, self).__init__()\n         batch_norm:\n            self.add_module(, nn.BatchNorm1d(num_input_features))\n        \n        self.add_module(, nn.ELU())\n        self.add_module(, nn.Conv1d(num_input_features, num_output_features, kernel_size=, stride=, bias=conv_bias))\n        self.add_module(, nn.AvgPool1d(kernel_size=, stride=))\n\n\n (nn.Module):\n     ():\n\n        (DenseNetEnconder, self).__init__()\n\n        \n        first_conv = OrderedDict([(, nn.Conv1d(in_channels, num_init_features, kernel_size=, stride=, padding=, bias=conv_bias))])\n        \n        \n\n        \n        \n        \n        \n\n         batch_norm:\n            first_conv[] = nn.BatchNorm1d(num_init_features)\n        \n        first_conv[] = nn.ELU()\n        first_conv[] = nn.MaxPool1d(kernel_size=, stride=, padding=)\n\n        self.densenet = nn.Sequential(first_conv)\n\n        num_features = num_init_features\n         i, num_layers  (block_config):\n            block = _DenseBlock(num_layers=num_layers, num_input_features=num_features,\n                                bn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate, conv_bias=conv_bias, batch_norm=batch_norm)\n            self.densenet.add_module( % (i + ), block)\n            num_features = num_features + num_layers * growth_rate\n             i != (block_config) - :\n                trans = _Transition(num_input_features=num_features, num_output_features=num_features // , conv_bias=conv_bias, batch_norm=batch_norm)\n                self.densenet.add_module( % (i + ), trans)\n                num_features = num_features // \n\n        \n         batch_norm:\n            self.densenet.add_module(.((block_config) + ), nn.BatchNorm1d(num_features))\n        \n\n        self.densenet.add_module(.((block_config) + ), nn.ReLU())\n        self.densenet.add_module(.((block_config) + ), nn.AvgPool1d(kernel_size=, stride=))  \n\n        self.num_features = num_features\n\n        \n         m  self.modules():\n             (m, nn.Conv1d):\n                nn.init.kaiming_normal_(m.weight.data)\n             (m, nn.BatchNorm1d):\n                m.weight.data.fill_()\n                m.bias.data.zero_()\n             (m, nn.Linear):\n                m.bias.data.zero_()\n\n     ():\n        features = self.densenet(x)\n        \n        \n         features.view(features.size(), -)\n\n\n (nn.Module):\n    \n    \n     ():\n\n        (DenseNetClassifier, self).__init__()\n\n        self.features = DenseNetEnconder(growth_rate=growth_rate, block_config=block_config, in_channels=in_channels,\n                                         num_init_features=num_init_features, bn_size=bn_size, drop_rate=drop_rate,\n                                         conv_bias=conv_bias, batch_norm=batch_norm)\n\n        \n        self.classifier = nn.Sequential(\n            nn.Dropout(p=drop_fc),\n            nn.Linear(self.features.num_features, num_classes)\n        )\n\n        \n         m  self.modules():\n             (m, nn.Conv1d):\n                nn.init.kaiming_normal_(m.weight.data)\n             (m, nn.BatchNorm1d):\n                m.weight.data.fill_()\n                m.bias.data.zero_()\n             (m, nn.Linear):\n                m.bias.data.zero_()\n\n     ():\n        features = self.features(x)\n        out = self.classifier(features)\n         out, features\n</code></pre>\n<h1><a href=\"https://www.kaggle.com/code/seshurajup/sparcnet-git?scriptVersionId=160730131\" target=\"_blank\">Notebook - SPaRCNet Git</a></h1>\n<h1><a href=\"https://www.kaggle.com/datasets/seshurajup/sparcnet-dataset/data\" target=\"_blank\">Extra Train Dataset &amp; Model - SPaRCNet Dataset</a></h1>",
  "messages": [
    {
      "id": "2623797",
      "postDate": "01/28/2024 11:24:24",
      "content": "<h1>From the <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471439\" target=\"_blank\">shared discussion</a></h1>\n<h1>Preprocess</h1>\n<pre><code> mne.  filter_data, notch_filter\nX2 = X[[,,,, ,,,, ,,,, ,,,]] - X[[,,,, ,,,, ,,,, ,,,]]\nX2 = notch_filter(X2, , , n_jobs=-, verbose=)\nX2 = filter_data(X2, , , , n_jobs=-, verbose=) \n==&gt;  to  kHz\n==&gt; Frequencies to notch   Hz =  ?? (  idea about  used )\n==&gt; After Difference, applied notch  frequency filters ??\n</code></pre>\n<h1>Loss Function : <strong>WeightedKLDivWithLogitsLoss</strong> (</h1>\n<pre><code> (nn.KLDivLoss):\n     ():\n        (WeightedKLDivWithLogitsLoss, self).__init__(size_average=, reduce=, reduction=)\n        self.register_buffer(, weight)\n\n     ():\n        \n        \n        batch_size = .size()\n        log_prob = F.log_softmax(, )\n        element_loss = (WeightedKLDivWithLogitsLoss, self).forward(log_prob, target)\n\n        sample_loss = torch.(element_loss, dim=)\n        sample_weight = torch.(target * self.weight, dim=)\n\n        weighted_loss = sample_loss*sample_weight\n        \n        avg_loss = torch.(weighted_loss) / batch_size\n\n         avg_loss  \n</code></pre>\n<h1>Optimiser</h1>\n<pre><code>optimizer = optim.Adam(model_cnn.parameters(), lr=*, betas = (,),eps = *, weight_decay=*)\n==&gt; lr = *\n==&gt; eps = *\n==&gt; betas = (, )\n</code></pre>\n<h1>batch size</h1>\n<pre><code>batch_size = \n</code></pre>\n<h1>Model</h1>\n<pre><code> (nn.Sequential):\n     ():\n        (_DenseLayer, self).__init__()\n         batch_norm:\n            self.add_module(, nn.BatchNorm1d(num_input_features)),\n        \n        self.add_module(, nn.ELU()),\n        self.add_module(, nn.Conv1d(num_input_features, bn_size * growth_rate, kernel_size=, stride=, bias=conv_bias)),\n         batch_norm:\n            self.add_module(, nn.BatchNorm1d(bn_size * growth_rate)),\n        \n        self.add_module(, nn.ELU()),\n        self.add_module(, nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=, stride=, padding=, bias=conv_bias)),\n        \n        self.drop_rate = drop_rate\n\n     ():\n        \n        \n        new_features = (_DenseLayer, self).forward(x)\n        \n        \n         self.drop_rate &gt; :\n            new_features = F.dropout(new_features, p=self.drop_rate, training=self.training)\n         torch.cat([x, new_features], )\n\n\n (nn.Sequential):\n     ():\n        (_DenseBlock, self).__init__()\n         i  (num_layers):\n            layer = _DenseLayer(num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate, conv_bias, batch_norm)\n            self.add_module( % (i + ), layer)\n\n\n (nn.Sequential):\n     ():\n        (_Transition, self).__init__()\n         batch_norm:\n            self.add_module(, nn.BatchNorm1d(num_input_features))\n        \n        self.add_module(, nn.ELU())\n        self.add_module(, nn.Conv1d(num_input_features, num_output_features, kernel_size=, stride=, bias=conv_bias))\n        self.add_module(, nn.AvgPool1d(kernel_size=, stride=))\n\n\n (nn.Module):\n     ():\n\n        (DenseNetEnconder, self).__init__()\n\n        \n        first_conv = OrderedDict([(, nn.Conv1d(in_channels, num_init_features, kernel_size=, stride=, padding=, bias=conv_bias))])\n        \n        \n\n        \n        \n        \n        \n\n         batch_norm:\n            first_conv[] = nn.BatchNorm1d(num_init_features)\n        \n        first_conv[] = nn.ELU()\n        first_conv[] = nn.MaxPool1d(kernel_size=, stride=, padding=)\n\n        self.densenet = nn.Sequential(first_conv)\n\n        num_features = num_init_features\n         i, num_layers  (block_config):\n            block = _DenseBlock(num_layers=num_layers, num_input_features=num_features,\n                                bn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate, conv_bias=conv_bias, batch_norm=batch_norm)\n            self.densenet.add_module( % (i + ), block)\n            num_features = num_features + num_layers * growth_rate\n             i != (block_config) - :\n                trans = _Transition(num_input_features=num_features, num_output_features=num_features // , conv_bias=conv_bias, batch_norm=batch_norm)\n                self.densenet.add_module( % (i + ), trans)\n                num_features = num_features // \n\n        \n         batch_norm:\n            self.densenet.add_module(.((block_config) + ), nn.BatchNorm1d(num_features))\n        \n\n        self.densenet.add_module(.((block_config) + ), nn.ReLU())\n        self.densenet.add_module(.((block_config) + ), nn.AvgPool1d(kernel_size=, stride=))  \n\n        self.num_features = num_features\n\n        \n         m  self.modules():\n             (m, nn.Conv1d):\n                nn.init.kaiming_normal_(m.weight.data)\n             (m, nn.BatchNorm1d):\n                m.weight.data.fill_()\n                m.bias.data.zero_()\n             (m, nn.Linear):\n                m.bias.data.zero_()\n\n     ():\n        features = self.densenet(x)\n        \n        \n         features.view(features.size(), -)\n\n\n (nn.Module):\n    \n    \n     ():\n\n        (DenseNetClassifier, self).__init__()\n\n        self.features = DenseNetEnconder(growth_rate=growth_rate, block_config=block_config, in_channels=in_channels,\n                                         num_init_features=num_init_features, bn_size=bn_size, drop_rate=drop_rate,\n                                         conv_bias=conv_bias, batch_norm=batch_norm)\n\n        \n        self.classifier = nn.Sequential(\n            nn.Dropout(p=drop_fc),\n            nn.Linear(self.features.num_features, num_classes)\n        )\n\n        \n         m  self.modules():\n             (m, nn.Conv1d):\n                nn.init.kaiming_normal_(m.weight.data)\n             (m, nn.BatchNorm1d):\n                m.weight.data.fill_()\n                m.bias.data.zero_()\n             (m, nn.Linear):\n                m.bias.data.zero_()\n\n     ():\n        features = self.features(x)\n        out = self.classifier(features)\n         out, features\n</code></pre>\n<h1><a href=\"https://www.kaggle.com/code/seshurajup/sparcnet-git?scriptVersionId=160730131\" target=\"_blank\">Notebook - SPaRCNet Git</a></h1>\n<h1><a href=\"https://www.kaggle.com/datasets/seshurajup/sparcnet-dataset/data\" target=\"_blank\">Extra Train Dataset &amp; Model - SPaRCNet Dataset</a></h1>",
      "rawMarkdown": "# From the [shared discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471439)\n\n# Preprocess \n```python\nfrom mne.filter import filter_data, notch_filter\nX2 = X[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]] - X[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\nX2 = notch_filter(X2, 200, 60, n_jobs=-1, verbose='ERROR')\nX2 = filter_data(X2, 200, 0.5, 40, n_jobs=-1, verbose='ERROR') \n==> 0.5 to 44 kHz\n==> Frequencies to notch filter in Hz = 60 ?? ( any idea about 60 used )\n==> After Difference, applied notch and frequency filters ??\n```\n\n# Loss Function : **WeightedKLDivWithLogitsLoss** ( \n```python\nclass WeightedKLDivWithLogitsLoss(nn.KLDivLoss):\n\tdef __init__(self, weight):\n\t\tsuper(WeightedKLDivWithLogitsLoss, self).__init__(size_average=None, reduce=None, reduction='none')\n\t\tself.register_buffer('weight', weight)\n\n\tdef forward(self, input, target):\n\t\t# TODO: For KLDivLoss: input should 'log-probability' and target should be 'probability'\n\t\t# TODO: input for this method is logits, and target is probabilities\n\t\tbatch_size = input.size(0)\n\t\tlog_prob = F.log_softmax(input, 1)\n\t\telement_loss = super(WeightedKLDivWithLogitsLoss, self).forward(log_prob, target)\n\n\t\tsample_loss = torch.sum(element_loss, dim=1)\n\t\tsample_weight = torch.sum(target * self.weight, dim=1)\n\n\t\tweighted_loss = sample_loss*sample_weight\n\t\t# Average over mini-batch, not element-wise\n\t\tavg_loss = torch.sum(weighted_loss) / batch_size\n\n\t\treturn avg_loss  \n```\n# Optimiser\n```python\noptimizer = optim.Adam(model_cnn.parameters(), lr=6.25*1e-5, betas = (0.9,0.999),eps = 1.0*1e-8, weight_decay=1.0*1e-3)\n==> lr = 6.25*1e-5\n==> eps = 1.0*1e-8\n==> betas = (0.9, 0.999)\n```\n\n# batch size\n```python\nbatch_size = 32\n```\n\n# Model\n```python\nclass _DenseLayer(nn.Sequential):\n\tdef __init__(self, num_input_features, growth_rate, bn_size, drop_rate, conv_bias, batch_norm):\n\t\tsuper(_DenseLayer, self).__init__()\n\t\tif batch_norm:\n\t\t\tself.add_module('norm1', nn.BatchNorm1d(num_input_features)),\n\t\t# self.add_module('relu1', nn.ReLU()),\n\t\tself.add_module('elu1', nn.ELU()),\n\t\tself.add_module('conv1', nn.Conv1d(num_input_features, bn_size * growth_rate, kernel_size=1, stride=1, bias=conv_bias)),\n\t\tif batch_norm:\n\t\t\tself.add_module('norm2', nn.BatchNorm1d(bn_size * growth_rate)),\n\t\t# self.add_module('relu2', nn.ReLU()),\n\t\tself.add_module('elu2', nn.ELU()),\n\t\tself.add_module('conv2', nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=3, stride=1, padding=1, bias=conv_bias)),\n\t\t# self.add_module('conv2', nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=7, stride=1, padding=3, bias=conv_bias)),\n\t\tself.drop_rate = drop_rate\n\n\tdef forward(self, x):\n\t\t# print(\"Dense Layer Input: \")\n\t\t# print(x.size())\n\t\tnew_features = super(_DenseLayer, self).forward(x)\n\t\t# print(\"Dense Layer Output:\")\n\t\t# print(new_features.size())\n\t\tif self.drop_rate > 0:\n\t\t\tnew_features = F.dropout(new_features, p=self.drop_rate, training=self.training)\n\t\treturn torch.cat([x, new_features], 1)\n\n\nclass _DenseBlock(nn.Sequential):\n\tdef __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate, conv_bias, batch_norm):\n\t\tsuper(_DenseBlock, self).__init__()\n\t\tfor i in range(num_layers):\n\t\t\tlayer = _DenseLayer(num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate, conv_bias, batch_norm)\n\t\t\tself.add_module('denselayer%d' % (i + 1), layer)\n\n\nclass _Transition(nn.Sequential):\n\tdef __init__(self, num_input_features, num_output_features, conv_bias, batch_norm):\n\t\tsuper(_Transition, self).__init__()\n\t\tif batch_norm:\n\t\t\tself.add_module('norm', nn.BatchNorm1d(num_input_features))\n\t\t# self.add_module('relu', nn.ReLU())\n\t\tself.add_module('elu', nn.ELU())\n\t\tself.add_module('conv', nn.Conv1d(num_input_features, num_output_features, kernel_size=1, stride=1, bias=conv_bias))\n\t\tself.add_module('pool', nn.AvgPool1d(kernel_size=2, stride=2))\n\n\nclass DenseNetEnconder(nn.Module):\n\tdef __init__(self, growth_rate=32, block_config=(4, 4, 4, 4, 4, 4, 4),  #block_config=(6, 12, 24, 48, 24, 20, 16),  #block_config=(6, 12, 24, 16),\n\t\t\t\t in_channels=16, num_init_features=64, bn_size=4, drop_rate=0.2, conv_bias=True, batch_norm=False):\n\n\t\tsuper(DenseNetEnconder, self).__init__()\n\n\t\t# First convolution\n\t\tfirst_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, kernel_size=7, stride=2, padding=3, bias=conv_bias))])\n\t\t# first_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, groups=in_channels, kernel_size=7, stride=2, padding=3, bias=conv_bias))])\n\t\t# first_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, kernel_size=15, stride=2, padding=7, bias=conv_bias))])\n\n\t\t# first_conv = OrderedDict([\n\t\t# \t('conv0-depth', nn.Conv1d(in_channels, 32, groups=in_channels, kernel_size=7, stride=2, padding=3, bias=conv_bias)),\n\t\t# \t('conv0-point', nn.Conv1d(32, num_init_features, kernel_size=1, stride=1, bias=conv_bias)),\n\t\t# ])\n\n\t\tif batch_norm:\n\t\t\tfirst_conv['norm0'] = nn.BatchNorm1d(num_init_features)\n\t\t# first_conv['relu0'] = nn.ReLU()\n\t\tfirst_conv['elu0'] = nn.ELU()\n\t\tfirst_conv['pool0'] = nn.MaxPool1d(kernel_size=3, stride=2, padding=1)\n\n\t\tself.densenet = nn.Sequential(first_conv)\n\n\t\tnum_features = num_init_features\n\t\tfor i, num_layers in enumerate(block_config):\n\t\t\tblock = _DenseBlock(num_layers=num_layers, num_input_features=num_features,\n\t\t\t\t\t\t\t\tbn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate, conv_bias=conv_bias, batch_norm=batch_norm)\n\t\t\tself.densenet.add_module('denseblock%d' % (i + 1), block)\n\t\t\tnum_features = num_features + num_layers * growth_rate\n\t\t\tif i != len(block_config) - 1:\n\t\t\t\ttrans = _Transition(num_input_features=num_features, num_output_features=num_features // 2, conv_bias=conv_bias, batch_norm=batch_norm)\n\t\t\t\tself.densenet.add_module('transition%d' % (i + 1), trans)\n\t\t\t\tnum_features = num_features // 2\n\n\t\t# Final batch norm\n\t\tif batch_norm:\n\t\t\tself.densenet.add_module('norm{}'.format(len(block_config) + 1), nn.BatchNorm1d(num_features))\n\t\t# self.features.add_module('norm5', BatchReNorm1d(num_features))\n\n\t\tself.densenet.add_module('relu{}'.format(len(block_config) + 1), nn.ReLU())\n\t\tself.densenet.add_module('pool{}'.format(len(block_config) + 1), nn.AvgPool1d(kernel_size=7, stride=3))  # stride originally 1\n\n\t\tself.num_features = num_features\n\n\t\t# Official init from torch repo.\n\t\tfor m in self.modules():\n\t\t\tif isinstance(m, nn.Conv1d):\n\t\t\t\tnn.init.kaiming_normal_(m.weight.data)\n\t\t\telif isinstance(m, nn.BatchNorm1d):\n\t\t\t\tm.weight.data.fill_(1)\n\t\t\t\tm.bias.data.zero_()\n\t\t\telif isinstance(m, nn.Linear):\n\t\t\t\tm.bias.data.zero_()\n\n\tdef forward(self, x):\n\t\tfeatures = self.densenet(x)\n\t\t# print(\"Final Output\")\n\t\t# print(features.size())\n\t\treturn features.view(features.size(0), -1)\n\n\nclass DenseNetClassifier(nn.Module):\n\t# def __init__(self, growth_rate=16, block_config=(3, 6, 12, 8),  #block_config=(6, 12, 24, 48, 24, 20, 16),  #block_config=(6, 12, 24, 16),\n\t# \t\t\t in_channels=16, num_init_features=32, bn_size=2, drop_rate=0, conv_bias=False, drop_fc=0.5, num_classes=6):\n\tdef __init__(self, growth_rate=32, block_config=(4, 4, 4, 4, 4, 4, 4),\n\t\t\t\t in_channels=16, num_init_features=64, bn_size=4, drop_rate=0.2, conv_bias=True, batch_norm=False, drop_fc=0.5, num_classes=6):\n\n\t\tsuper(DenseNetClassifier, self).__init__()\n\n\t\tself.features = DenseNetEnconder(growth_rate=growth_rate, block_config=block_config, in_channels=in_channels,\n\t\t\t\t\t\t\t\t\t\t num_init_features=num_init_features, bn_size=bn_size, drop_rate=drop_rate,\n\t\t\t\t\t\t\t\t\t\t conv_bias=conv_bias, batch_norm=batch_norm)\n\n\t\t# Linear layer\n\t\tself.classifier = nn.Sequential(\n\t\t\tnn.Dropout(p=drop_fc),\n\t\t\tnn.Linear(self.features.num_features, num_classes)\n\t\t)\n\n\t\t# Official init from torch repo.\n\t\tfor m in self.modules():\n\t\t\tif isinstance(m, nn.Conv1d):\n\t\t\t\tnn.init.kaiming_normal_(m.weight.data)\n\t\t\telif isinstance(m, nn.BatchNorm1d):\n\t\t\t\tm.weight.data.fill_(1)\n\t\t\t\tm.bias.data.zero_()\n\t\t\telif isinstance(m, nn.Linear):\n\t\t\t\tm.bias.data.zero_()\n\n\tdef forward(self, x):\n\t\tfeatures = self.features(x)\n\t\tout = self.classifier(features)\n\t\treturn out, features\n```\n\n# [Notebook - SPaRCNet Git](https://www.kaggle.com/code/seshurajup/sparcnet-git?scriptVersionId=160730131) \n# [Extra Train Dataset & Model - SPaRCNet Dataset](https://www.kaggle.com/datasets/seshurajup/sparcnet-dataset/data)",
      "votes": null
    },
    {
      "id": "2623938",
      "postDate": "01/28/2024 13:35:18",
      "content": "<p><a href=\"https://www.kaggle.com/seshurajup\" target=\"_blank\">@seshurajup</a> wow! you are quick. Upvoted!</p>",
      "rawMarkdown": "seshurajup wow! you are quick. Upvoted!",
      "votes": null
    },
    {
      "id": "2623980",
      "postDate": "01/28/2024 14:05:14",
      "content": "<p>Some study material for the future 😋</p>\n<p>upvoted</p>",
      "rawMarkdown": "Some study material for the future 😋\n\nupvoted",
      "votes": null
    },
    {
      "id": "2624003",
      "postDate": "01/28/2024 14:26:48",
      "content": "<p><a href=\"https://www.kaggle.com/renyiwei\" target=\"_blank\">@renyiwei</a> i like this paper (exact match with competition data), so informative and now trying to train the code. </p>",
      "rawMarkdown": "renyiwei i like this paper (exact match with competition data), so informative and now trying to train the code.",
      "votes": null
    },
    {
      "id": "2624745",
      "postDate": "01/29/2024 00:15:11",
      "content": "<p>Regarding:</p>\n<pre><code>X2 = notch_filter(X2, 200, 60, =-1, =)\nX2 = filter_data(X2, 200, 0.5, 40, =-1, =) \n</code></pre>\n<p>The notch filter is at 60 Hz which is the AC \"Utility frequency\"; wiki says: \"in large parts of the world this is 50 Hz, although in the Americas and parts of Asia it is typically 60 Hz.\" This signal can be relatively strong, even after differencing the channels.<br>\nThe next filter selects the band of frequencies from 0.5 to 40 Hz; the 0.5 highpass gets rid of slow drifts in the waveform. </p>",
      "rawMarkdown": "Regarding:\n```\nX2 = notch_filter(X2, 200, 60, n_jobs=-1, verbose='ERROR')\nX2 = filter_data(X2, 200, 0.5, 40, n_jobs=-1, verbose='ERROR') \n```\nThe notch filter is at 60 Hz which is the AC \"Utility frequency\"; wiki says: \"in large parts of the world this is 50 Hz, although in the Americas and parts of Asia it is typically 60 Hz.\" This signal can be relatively strong, even after differencing the channels.\nThe next filter selects the band of frequencies from 0.5 to 40 Hz; the 0.5 highpass gets rid of slow drifts in the waveform.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2623938,
      "author_name": "renyiwei",
      "author_url": "",
      "post_date": "01/28/2024 13:35:18",
      "content": "<p><a href=\"https://www.kaggle.com/seshurajup\" target=\"_blank\">@seshurajup</a> wow! you are quick. Upvoted!</p>",
      "votes": null,
      "replies": [
        {
          "id": 2624003,
          "author_name": "seshurajup",
          "author_url": "",
          "post_date": "01/28/2024 14:26:48",
          "content": "<p><a href=\"https://www.kaggle.com/renyiwei\" target=\"_blank\">@renyiwei</a> i like this paper (exact match with competition data), so informative and now trying to train the code. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2623980,
      "author_name": "gabrielfreddi",
      "author_url": "",
      "post_date": "01/28/2024 14:05:14",
      "content": "<p>Some study material for the future 😋</p>\n<p>upvoted</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2624745,
      "author_name": "dan3dewey",
      "author_url": "",
      "post_date": "01/29/2024 00:15:11",
      "content": "<p>Regarding:</p>\n<pre><code>X2 = notch_filter(X2, 200, 60, =-1, =)\nX2 = filter_data(X2, 200, 0.5, 40, =-1, =) \n</code></pre>\n<p>The notch filter is at 60 Hz which is the AC \"Utility frequency\"; wiki says: \"in large parts of the world this is 50 Hz, although in the Americas and parts of Asia it is typically 60 Hz.\" This signal can be relatively strong, even after differencing the channels.<br>\nThe next filter selects the band of frequencies from 0.5 to 40 Hz; the 0.5 highpass gets rid of slow drifts in the waveform. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2623797": "# From the [shared discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471439)\n\n# Preprocess \n```python\nfrom mne.filter import filter_data, notch_filter\nX2 = X[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]] - X[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\nX2 = notch_filter(X2, 200, 60, n_jobs=-1, verbose='ERROR')\nX2 = filter_data(X2, 200, 0.5, 40, n_jobs=-1, verbose='ERROR') \n==> 0.5 to 44 kHz\n==> Frequencies to notch filter in Hz = 60 ?? ( any idea about 60 used )\n==> After Difference, applied notch and frequency filters ??\n```\n\n# Loss Function : **WeightedKLDivWithLogitsLoss** ( \n```python\nclass WeightedKLDivWithLogitsLoss(nn.KLDivLoss):\n\tdef __init__(self, weight):\n\t\tsuper(WeightedKLDivWithLogitsLoss, self).__init__(size_average=None, reduce=None, reduction='none')\n\t\tself.register_buffer('weight', weight)\n\n\tdef forward(self, input, target):\n\t\t# TODO: For KLDivLoss: input should 'log-probability' and target should be 'probability'\n\t\t# TODO: input for this method is logits, and target is probabilities\n\t\tbatch_size = input.size(0)\n\t\tlog_prob = F.log_softmax(input, 1)\n\t\telement_loss = super(WeightedKLDivWithLogitsLoss, self).forward(log_prob, target)\n\n\t\tsample_loss = torch.sum(element_loss, dim=1)\n\t\tsample_weight = torch.sum(target * self.weight, dim=1)\n\n\t\tweighted_loss = sample_loss*sample_weight\n\t\t# Average over mini-batch, not element-wise\n\t\tavg_loss = torch.sum(weighted_loss) / batch_size\n\n\t\treturn avg_loss  \n```\n# Optimiser\n```python\noptimizer = optim.Adam(model_cnn.parameters(), lr=6.25*1e-5, betas = (0.9,0.999),eps = 1.0*1e-8, weight_decay=1.0*1e-3)\n==> lr = 6.25*1e-5\n==> eps = 1.0*1e-8\n==> betas = (0.9, 0.999)\n```\n\n# batch size\n```python\nbatch_size = 32\n```\n\n# Model\n```python\nclass _DenseLayer(nn.Sequential):\n\tdef __init__(self, num_input_features, growth_rate, bn_size, drop_rate, conv_bias, batch_norm):\n\t\tsuper(_DenseLayer, self).__init__()\n\t\tif batch_norm:\n\t\t\tself.add_module('norm1', nn.BatchNorm1d(num_input_features)),\n\t\t# self.add_module('relu1', nn.ReLU()),\n\t\tself.add_module('elu1', nn.ELU()),\n\t\tself.add_module('conv1', nn.Conv1d(num_input_features, bn_size * growth_rate, kernel_size=1, stride=1, bias=conv_bias)),\n\t\tif batch_norm:\n\t\t\tself.add_module('norm2', nn.BatchNorm1d(bn_size * growth_rate)),\n\t\t# self.add_module('relu2', nn.ReLU()),\n\t\tself.add_module('elu2', nn.ELU()),\n\t\tself.add_module('conv2', nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=3, stride=1, padding=1, bias=conv_bias)),\n\t\t# self.add_module('conv2', nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=7, stride=1, padding=3, bias=conv_bias)),\n\t\tself.drop_rate = drop_rate\n\n\tdef forward(self, x):\n\t\t# print(\"Dense Layer Input: \")\n\t\t# print(x.size())\n\t\tnew_features = super(_DenseLayer, self).forward(x)\n\t\t# print(\"Dense Layer Output:\")\n\t\t# print(new_features.size())\n\t\tif self.drop_rate > 0:\n\t\t\tnew_features = F.dropout(new_features, p=self.drop_rate, training=self.training)\n\t\treturn torch.cat([x, new_features], 1)\n\n\nclass _DenseBlock(nn.Sequential):\n\tdef __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate, conv_bias, batch_norm):\n\t\tsuper(_DenseBlock, self).__init__()\n\t\tfor i in range(num_layers):\n\t\t\tlayer = _DenseLayer(num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate, conv_bias, batch_norm)\n\t\t\tself.add_module('denselayer%d' % (i + 1), layer)\n\n\nclass _Transition(nn.Sequential):\n\tdef __init__(self, num_input_features, num_output_features, conv_bias, batch_norm):\n\t\tsuper(_Transition, self).__init__()\n\t\tif batch_norm:\n\t\t\tself.add_module('norm', nn.BatchNorm1d(num_input_features))\n\t\t# self.add_module('relu', nn.ReLU())\n\t\tself.add_module('elu', nn.ELU())\n\t\tself.add_module('conv', nn.Conv1d(num_input_features, num_output_features, kernel_size=1, stride=1, bias=conv_bias))\n\t\tself.add_module('pool', nn.AvgPool1d(kernel_size=2, stride=2))\n\n\nclass DenseNetEnconder(nn.Module):\n\tdef __init__(self, growth_rate=32, block_config=(4, 4, 4, 4, 4, 4, 4),  #block_config=(6, 12, 24, 48, 24, 20, 16),  #block_config=(6, 12, 24, 16),\n\t\t\t\t in_channels=16, num_init_features=64, bn_size=4, drop_rate=0.2, conv_bias=True, batch_norm=False):\n\n\t\tsuper(DenseNetEnconder, self).__init__()\n\n\t\t# First convolution\n\t\tfirst_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, kernel_size=7, stride=2, padding=3, bias=conv_bias))])\n\t\t# first_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, groups=in_channels, kernel_size=7, stride=2, padding=3, bias=conv_bias))])\n\t\t# first_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, kernel_size=15, stride=2, padding=7, bias=conv_bias))])\n\n\t\t# first_conv = OrderedDict([\n\t\t# \t('conv0-depth', nn.Conv1d(in_channels, 32, groups=in_channels, kernel_size=7, stride=2, padding=3, bias=conv_bias)),\n\t\t# \t('conv0-point', nn.Conv1d(32, num_init_features, kernel_size=1, stride=1, bias=conv_bias)),\n\t\t# ])\n\n\t\tif batch_norm:\n\t\t\tfirst_conv['norm0'] = nn.BatchNorm1d(num_init_features)\n\t\t# first_conv['relu0'] = nn.ReLU()\n\t\tfirst_conv['elu0'] = nn.ELU()\n\t\tfirst_conv['pool0'] = nn.MaxPool1d(kernel_size=3, stride=2, padding=1)\n\n\t\tself.densenet = nn.Sequential(first_conv)\n\n\t\tnum_features = num_init_features\n\t\tfor i, num_layers in enumerate(block_config):\n\t\t\tblock = _DenseBlock(num_layers=num_layers, num_input_features=num_features,\n\t\t\t\t\t\t\t\tbn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate, conv_bias=conv_bias, batch_norm=batch_norm)\n\t\t\tself.densenet.add_module('denseblock%d' % (i + 1), block)\n\t\t\tnum_features = num_features + num_layers * growth_rate\n\t\t\tif i != len(block_config) - 1:\n\t\t\t\ttrans = _Transition(num_input_features=num_features, num_output_features=num_features // 2, conv_bias=conv_bias, batch_norm=batch_norm)\n\t\t\t\tself.densenet.add_module('transition%d' % (i + 1), trans)\n\t\t\t\tnum_features = num_features // 2\n\n\t\t# Final batch norm\n\t\tif batch_norm:\n\t\t\tself.densenet.add_module('norm{}'.format(len(block_config) + 1), nn.BatchNorm1d(num_features))\n\t\t# self.features.add_module('norm5', BatchReNorm1d(num_features))\n\n\t\tself.densenet.add_module('relu{}'.format(len(block_config) + 1), nn.ReLU())\n\t\tself.densenet.add_module('pool{}'.format(len(block_config) + 1), nn.AvgPool1d(kernel_size=7, stride=3))  # stride originally 1\n\n\t\tself.num_features = num_features\n\n\t\t# Official init from torch repo.\n\t\tfor m in self.modules():\n\t\t\tif isinstance(m, nn.Conv1d):\n\t\t\t\tnn.init.kaiming_normal_(m.weight.data)\n\t\t\telif isinstance(m, nn.BatchNorm1d):\n\t\t\t\tm.weight.data.fill_(1)\n\t\t\t\tm.bias.data.zero_()\n\t\t\telif isinstance(m, nn.Linear):\n\t\t\t\tm.bias.data.zero_()\n\n\tdef forward(self, x):\n\t\tfeatures = self.densenet(x)\n\t\t# print(\"Final Output\")\n\t\t# print(features.size())\n\t\treturn features.view(features.size(0), -1)\n\n\nclass DenseNetClassifier(nn.Module):\n\t# def __init__(self, growth_rate=16, block_config=(3, 6, 12, 8),  #block_config=(6, 12, 24, 48, 24, 20, 16),  #block_config=(6, 12, 24, 16),\n\t# \t\t\t in_channels=16, num_init_features=32, bn_size=2, drop_rate=0, conv_bias=False, drop_fc=0.5, num_classes=6):\n\tdef __init__(self, growth_rate=32, block_config=(4, 4, 4, 4, 4, 4, 4),\n\t\t\t\t in_channels=16, num_init_features=64, bn_size=4, drop_rate=0.2, conv_bias=True, batch_norm=False, drop_fc=0.5, num_classes=6):\n\n\t\tsuper(DenseNetClassifier, self).__init__()\n\n\t\tself.features = DenseNetEnconder(growth_rate=growth_rate, block_config=block_config, in_channels=in_channels,\n\t\t\t\t\t\t\t\t\t\t num_init_features=num_init_features, bn_size=bn_size, drop_rate=drop_rate,\n\t\t\t\t\t\t\t\t\t\t conv_bias=conv_bias, batch_norm=batch_norm)\n\n\t\t# Linear layer\n\t\tself.classifier = nn.Sequential(\n\t\t\tnn.Dropout(p=drop_fc),\n\t\t\tnn.Linear(self.features.num_features, num_classes)\n\t\t)\n\n\t\t# Official init from torch repo.\n\t\tfor m in self.modules():\n\t\t\tif isinstance(m, nn.Conv1d):\n\t\t\t\tnn.init.kaiming_normal_(m.weight.data)\n\t\t\telif isinstance(m, nn.BatchNorm1d):\n\t\t\t\tm.weight.data.fill_(1)\n\t\t\t\tm.bias.data.zero_()\n\t\t\telif isinstance(m, nn.Linear):\n\t\t\t\tm.bias.data.zero_()\n\n\tdef forward(self, x):\n\t\tfeatures = self.features(x)\n\t\tout = self.classifier(features)\n\t\treturn out, features\n```\n\n# [Notebook - SPaRCNet Git](https://www.kaggle.com/code/seshurajup/sparcnet-git?scriptVersionId=160730131) \n# [Extra Train Dataset & Model - SPaRCNet Dataset](https://www.kaggle.com/datasets/seshurajup/sparcnet-dataset/data)",
    "2623938": "seshurajup wow! you are quick. Upvoted!",
    "2623980": "Some study material for the future 😋\n\nupvoted",
    "2624003": "renyiwei i like this paper (exact match with competition data), so informative and now trying to train the code.",
    "2624745": "Regarding:\n```\nX2 = notch_filter(X2, 200, 60, n_jobs=-1, verbose='ERROR')\nX2 = filter_data(X2, 200, 0.5, 40, n_jobs=-1, verbose='ERROR') \n```\nThe notch filter is at 60 Hz which is the AC \"Utility frequency\"; wiki says: \"in large parts of the world this is 50 Hz, although in the Americas and parts of Asia it is typically 60 Hz.\" This signal can be relatively strong, even after differencing the channels.\nThe next filter selects the band of frequencies from 0.5 to 40 Hz; the 0.5 highpass gets rid of slow drifts in the waveform."
  },
  "source": "meta"
}