First of all, I would like to thank my teammates( @naoism, @hidehisaarai1213, @yasufuminakama, @sinpcw ) for competing with me. I would also like to thank kaggle and EGO for organizing an interesting competition.
\nI explain my approach. My teammates will add their own approaches in the comments!
I joined this competition after the SETI competition was over. This task is very similar to the SETI competition task. This task is very similar to the SETI competition task, so I reused most of the SETI code.
\nx = x1 + x2\ny = y1 + y2 -(y1*y2)\nUsing pseudo-labels showed some improvement, but not a lot.
\nUsing other team members models, and finally stacking 137 models with NN or XGB, there was a significant score increase. Other models include 1dcnn, swin transformer and efficientnet b5-8.
\nTeam members will explain these details later.
First of all, I would like to thank my teammates( @naoism, @hidehisaarai1213, @yasufuminakama, @sinpcw ) for competing with me. I would also like to thank kaggle and EGO for organizing an interesting competition.
\nI explain my approach. My teammates will add their own approaches in the comments!
I joined this competition after the SETI competition was over. This task is very similar to the SETI competition task. This task is very similar to the SETI competition task, so I reused most of the SETI code.
\nx = x1 + x2\ny = y1 + y2 -(y1*y2)\nUsing pseudo-labels showed some improvement, but not a lot.
\nUsing other team members models, and finally stacking 137 models with NN or XGB, there was a significant score increase. Other models include 1dcnn, swin transformer and efficientnet b5-8.
\nTeam members will explain these details later.
@naoism, I'm glad that you eventually made it! after we missed gold in the previous competition. Big congratulations to u and your team.
", "rawMarkdown": "naoism, I'm glad that you eventually made it! after we missed gold in the previous competition. Big congratulations to u and your team.", "votes": null }, { "id": "1529051", "postDate": "09/30/2021 05:48:04", "content": "congratulation！ 137 models!!!!
", "rawMarkdown": "congratulation！ 137 models!!!!", "votes": null }, { "id": "1529060", "postDate": "09/30/2021 05:59:04", "content": "Nice to see that it required you 137 models to beat us 🤣
\nGood job for the gold 💪
Hi @hirune924 , thanks for sharing the solution , looks like a very elegant one
\nx = x1 + x2\ny = y1 + y2 -(y1*y2)
Can you report the score increase due to this , we also tried something on similar terms but didn't get a lot out of it
", "rawMarkdown": "Hi @hirune924 , thanks for sharing the solution , looks like a very elegant one\n\n`x = x1 + x2\ny = y1 + y2 -(y1*y2)`\n\nCan you report the score increase due to this , we also tried something on similar terms but didn't get a lot out of it", "votes": null }, { "id": "1529095", "postDate": "09/30/2021 06:41:08", "content": "I don't know the exact difference in scores from regular mixup because I used that trick from the beginning, but using mixup prevented overfitting during long training.
\nAlso, considering the possibility that this trick was negatively affecting the training, I reduced the frequency of its use from 50% to 15%, and the cv score decreased by about 0.0002.
First of all, I would like to thank my teammates for competing with me.
\nAnd I would like to thank kaggle & host for the interesting competition and to all the participants for giving us a lot of ideas.
Here is my approach.
\nMEAN_ALL = [6.901084819126393e-26, 5.117726793550612e-26, -1.3831247922595496e-26]\nSTD_ALL = [7.420282939705547e-21, 7.41993950281645e-21, 1.838329278493792e-21]\n\ndef z_score_normalization_wave(_wave, site):\n x_mean = MEAN_ALL[site]\n x_std = STD_ALL[site]\n wave = (_wave - x_mean) / (x_std)\n return wave\nNo gain compared to dividing the all wave form by 4.6152116213830774e-20 (max(abs) of the entire train and test data).
\ndef mixup(image, label, max_noise=0.5, probability=1.0):\n if np.random.uniform(0, 1) <= probability:\n bs, c, h, w = image.size()\n indices = torch.randperm(bs)\n shuffled_image = image[indices]\n shuffled_label = label[indices]\n lam = np.random.uniform(0, 1) * (1.0 - shuffled_label) * max_noise # don't mix shuffled_label=1\n lam = torch.flatten(lam)\n image = (image.reshape(-1, bs) * (1 - lam)).reshape(bs, c, h, w) + (shuffled_image.reshape(-1, bs) * lam).reshape(bs, c, h, w)\n return image, label\nw_diff = int(torch.randint(low=0, high=5, size=(1,)))\nif torch.rand(1) < 0.5:\n x = x[:,:,self.h_div_start:self.h_div_end,self.w_div_start+w_diff:self.w_div_end+w_diff]\nelse:\n x = x[:,:,self.h_div_start:self.h_div_end,self.w_div_start-w_diff:self.w_div_end-w_diff]\nRe-training on soft pseudo-labelled test dataset improved score.
\nI implemented stacking models as I did in OpenVaccine competition.
\nNN stacking model performed better than GBDT stacking models.
Thank you very much. I was finally able to achieve my goal of becoming a Master. I'm really happy.
\nAlso, as a team member of the previous competition, I am happy that you finally become a GM. Congratulations!
Thank you to all the participants for their hard work.
\nThank you to the organizers, hosts and teammates.
Up to a week before the competition, we were working on 2d CNN, but in the final stage, I was looking for improvements to 1d CNN.
\nIt is almost the same preprocessing as @hirune924 but differs in a few details.
\nI applied 1dGridDistortion (only in the frequency direction) as a unique element. It has slightly improved the CV.
\nIn training the model, we used only nuttall window among the CQTs. This is because nuttall was the best window for CV.
\nI tried deleting data that was prone to miss-prediction, but not work.
Improvements were made based on the (published notebook).
\nI used a strategy that of a noisy student, creating pseudo-labels in my own model and learning them again.
\nDue to deadlines, I was not able to try enough of them, but the following ones worked well.
At first, I would like to express my thanks to all the organizers for hosting this interesting competition and my teammates.
\nAnd thanks to all other competitors for sharing valuable information in forums and notebooks.
I was finally able to achieve my goal of becoming a Master :)
\nMy solution is very similar to @yasufuminakama's one.
\nSo I won't explain each method in detail.
def z_score_normalization_wave(_wave, site):\n MEAN_ALL = [-7.687203989766856e-31, 1.5238509999790067e-30, -1.7078072503529966e-30]\n STD_ALL = [1.618816429314542e-22, 1.6188170892087954e-22, 2.621679130415917e-22]\n x_mean = MEAN_ALL[site]\n x_std = STD_ALL[site]\n wave = (_wave-x_mean) / (x_std)\n return wave\n\ndef load_wave(path):\n waves = np.load(path)\n new_waves = []\n window = signal.tukey(4096)\n for i in [0,1,2]:\n _wave = pycbc.filter.resample.highpass_fir(pycbc.types.TimeSeries(waves[i], epoch=0, delta_t=1.0/2048), frequency=32, order=100)\n _wave = pycbc.filter.resample.notch_fir(_wave, f1=30, f2=80, order=10, beta=5)\n _wave = np.array(_wave)\n _wave = _wave * window\n wave = z_score_normalization_wave(_wave, site=i)\n new_waves.append(wave)\n return np.array(new_waves)\nI and @yasufuminakama and @naoism have formed a team from comparatively early stage of the competition, so our approaches were quite similar each other, so you may find some duplicates between what I've done and what others have done.
\nHere's the list that I've done and those worked.
\nclass CWT(nn.Module):\n def __init__(\n self,\n wavelet_width,\n fs,\n lower_freq,\n upper_freq,\n n_scales,\n size_factor=1.0,\n border_crop=0,\n stride=1\n ):\n super().__init__()\n\n self.initial_wavelet_width = wavelet_width\n self.fs = fs\n self.lower_freq = lower_freq\n self.upper_freq = upper_freq\n self.size_factor = size_factor\n self.n_scales = n_scales\n self.wavelet_width = wavelet_width\n self.border_crop = border_crop\n self.stride = stride\n wavelet_bank_real, wavelet_bank_imag = self._build_wavelet_kernel()\n self.wavelet_bank_real = nn.Parameter(wavelet_bank_real, requires_grad=False)\n self.wavelet_bank_imag = nn.Parameter(wavelet_bank_imag, requires_grad=False)\n\n self.kernel_size = self.wavelet_bank_real.size(3)\n\n def _build_wavelet_kernel(self):\n s_0 = 1 / self.upper_freq\n s_n = 1 / self.lower_freq\n\n base = np.power(s_n / s_0, 1 / (self.n_scales - 1))\n scales = s_0 * np.power(base, np.arange(self.n_scales))\n\n frequencies = 1 / scales\n truncation_size = scales.max() * np.sqrt(4.5 * self.initial_wavelet_width) * self.fs\n one_side = int(self.size_factor * truncation_size)\n kernel_size = 2 * one_side + 1\n\n k_array = np.arange(kernel_size, dtype=np.float32) - one_side\n t_array = k_array / self.fs\n\n wavelet_bank_real = []\n wavelet_bank_imag = []\n\n for scale in scales:\n norm_constant = np.sqrt(np.pi * self.wavelet_width) * scale * self.fs / 2.0\n scaled_t = t_array / scale\n exp_term = np.exp(-(scaled_t ** 2) / self.wavelet_width)\n kernel_base = exp_term / norm_constant\n kernel_real = kernel_base * np.cos(2 * np.pi * scaled_t)\n kernel_imag = kernel_base * np.sin(2 * np.pi * scaled_t)\n wavelet_bank_real.append(kernel_real)\n wavelet_bank_imag.append(kernel_imag)\n\n wavelet_bank_real = np.stack(wavelet_bank_real, axis=0)\n wavelet_bank_imag = np.stack(wavelet_bank_imag, axis=0)\n\n wavelet_bank_real = torch.from_numpy(wavelet_bank_real).unsqueeze(1).unsqueeze(2)\n wavelet_bank_imag = torch.from_numpy(wavelet_bank_imag).unsqueeze(1).unsqueeze(2)\n return wavelet_bank_real, wavelet_bank_imag\n\n def forward(self, x):\n border_crop = self.border_crop // self.stride\n start = border_crop\n end = (-border_crop) if border_crop > 0 else None\n\n # x [n_batch, n_channels, time_len]\n out_reals = []\n out_imags = []\n\n in_width = x.size(2)\n out_width = int(np.ceil(in_width / self.stride))\n pad_along_width = np.max((out_width - 1) * self.stride + self.kernel_size - in_width, 0)\n padding = pad_along_width // 2 + 1\n\n for i in range(3):\n # [n_batch, 1, 1, time_len]\n x_ = x[:, i, :].unsqueeze(1).unsqueeze(2)\n out_real = nn.functional.conv2d(x_, self.wavelet_bank_real, stride=(1, self.stride), padding=(0, padding))\n out_imag = nn.functional.conv2d(x_, self.wavelet_bank_imag, stride=(1, self.stride), padding=(0, padding))\n out_real = out_real.transpose(2, 1)\n out_imag = out_imag.transpose(2, 1)\n out_reals.append(out_real)\n out_imags.append(out_imag)\n\n out_real = torch.cat(out_reals, axis=1)\n out_imag = torch.cat(out_imags, axis=1)\n\n out_real = out_real[:, :, :, start:end]\n out_imag = out_imag[:, :, :, start:end]\n\n scalograms = torch.sqrt(out_real ** 2 + out_imag ** 2)\n return scalograms\nHey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
@naoism, I'm glad that you eventually made it! after we missed gold in the previous competition. Big congratulations to u and your team.
", "votes": null, "replies": [ { "id": 1529317, "author_name": "naoism", "author_url": "", "post_date": "09/30/2021 09:41:22", "content": "Thank you very much. I was finally able to achieve my goal of becoming a Master. I'm really happy.
\nAlso, as a team member of the previous competition, I am happy that you finally become a GM. Congratulations!
congratulation！ 137 models!!!!
", "votes": null, "replies": [] }, { "id": 1529060, "author_name": "callmeb", "author_url": "", "post_date": "09/30/2021 05:59:04", "content": "Nice to see that it required you 137 models to beat us 🤣
\nGood job for the gold 💪
Hi @hirune924 , thanks for sharing the solution , looks like a very elegant one
\nx = x1 + x2\ny = y1 + y2 -(y1*y2)
Can you report the score increase due to this , we also tried something on similar terms but didn't get a lot out of it
", "votes": null, "replies": [ { "id": 1529095, "author_name": "hirune924", "author_url": "", "post_date": "09/30/2021 06:41:08", "content": "I don't know the exact difference in scores from regular mixup because I used that trick from the beginning, but using mixup prevented overfitting during long training.
\nAlso, considering the possibility that this trick was negatively affecting the training, I reduced the frequency of its use from 50% to 15%, and the cv score decreased by about 0.0002.
First of all, I would like to thank my teammates for competing with me.
\nAnd I would like to thank kaggle & host for the interesting competition and to all the participants for giving us a lot of ideas.
Here is my approach.
\nMEAN_ALL = [6.901084819126393e-26, 5.117726793550612e-26, -1.3831247922595496e-26]\nSTD_ALL = [7.420282939705547e-21, 7.41993950281645e-21, 1.838329278493792e-21]\n\ndef z_score_normalization_wave(_wave, site):\n x_mean = MEAN_ALL[site]\n x_std = STD_ALL[site]\n wave = (_wave - x_mean) / (x_std)\n return wave\nNo gain compared to dividing the all wave form by 4.6152116213830774e-20 (max(abs) of the entire train and test data).
\ndef mixup(image, label, max_noise=0.5, probability=1.0):\n if np.random.uniform(0, 1) <= probability:\n bs, c, h, w = image.size()\n indices = torch.randperm(bs)\n shuffled_image = image[indices]\n shuffled_label = label[indices]\n lam = np.random.uniform(0, 1) * (1.0 - shuffled_label) * max_noise # don't mix shuffled_label=1\n lam = torch.flatten(lam)\n image = (image.reshape(-1, bs) * (1 - lam)).reshape(bs, c, h, w) + (shuffled_image.reshape(-1, bs) * lam).reshape(bs, c, h, w)\n return image, label\nw_diff = int(torch.randint(low=0, high=5, size=(1,)))\nif torch.rand(1) < 0.5:\n x = x[:,:,self.h_div_start:self.h_div_end,self.w_div_start+w_diff:self.w_div_end+w_diff]\nelse:\n x = x[:,:,self.h_div_start:self.h_div_end,self.w_div_start-w_diff:self.w_div_end-w_diff]\nRe-training on soft pseudo-labelled test dataset improved score.
\nI implemented stacking models as I did in OpenVaccine competition.
\nNN stacking model performed better than GBDT stacking models.
Thank you to all the participants for their hard work.
\nThank you to the organizers, hosts and teammates.
Up to a week before the competition, we were working on 2d CNN, but in the final stage, I was looking for improvements to 1d CNN.
\nIt is almost the same preprocessing as @hirune924 but differs in a few details.
\nI applied 1dGridDistortion (only in the frequency direction) as a unique element. It has slightly improved the CV.
\nIn training the model, we used only nuttall window among the CQTs. This is because nuttall was the best window for CV.
\nI tried deleting data that was prone to miss-prediction, but not work.
Improvements were made based on the (published notebook).
\nI used a strategy that of a noisy student, creating pseudo-labels in my own model and learning them again.
\nDue to deadlines, I was not able to try enough of them, but the following ones worked well.
At first, I would like to express my thanks to all the organizers for hosting this interesting competition and my teammates.
\nAnd thanks to all other competitors for sharing valuable information in forums and notebooks.
I was finally able to achieve my goal of becoming a Master :)
\nMy solution is very similar to @yasufuminakama's one.
\nSo I won't explain each method in detail.
def z_score_normalization_wave(_wave, site):\n MEAN_ALL = [-7.687203989766856e-31, 1.5238509999790067e-30, -1.7078072503529966e-30]\n STD_ALL = [1.618816429314542e-22, 1.6188170892087954e-22, 2.621679130415917e-22]\n x_mean = MEAN_ALL[site]\n x_std = STD_ALL[site]\n wave = (_wave-x_mean) / (x_std)\n return wave\n\ndef load_wave(path):\n waves = np.load(path)\n new_waves = []\n window = signal.tukey(4096)\n for i in [0,1,2]:\n _wave = pycbc.filter.resample.highpass_fir(pycbc.types.TimeSeries(waves[i], epoch=0, delta_t=1.0/2048), frequency=32, order=100)\n _wave = pycbc.filter.resample.notch_fir(_wave, f1=30, f2=80, order=10, beta=5)\n _wave = np.array(_wave)\n _wave = _wave * window\n wave = z_score_normalization_wave(_wave, site=i)\n new_waves.append(wave)\n return np.array(new_waves)\nI and @yasufuminakama and @naoism have formed a team from comparatively early stage of the competition, so our approaches were quite similar each other, so you may find some duplicates between what I've done and what others have done.
\nHere's the list that I've done and those worked.
\nclass CWT(nn.Module):\n def __init__(\n self,\n wavelet_width,\n fs,\n lower_freq,\n upper_freq,\n n_scales,\n size_factor=1.0,\n border_crop=0,\n stride=1\n ):\n super().__init__()\n\n self.initial_wavelet_width = wavelet_width\n self.fs = fs\n self.lower_freq = lower_freq\n self.upper_freq = upper_freq\n self.size_factor = size_factor\n self.n_scales = n_scales\n self.wavelet_width = wavelet_width\n self.border_crop = border_crop\n self.stride = stride\n wavelet_bank_real, wavelet_bank_imag = self._build_wavelet_kernel()\n self.wavelet_bank_real = nn.Parameter(wavelet_bank_real, requires_grad=False)\n self.wavelet_bank_imag = nn.Parameter(wavelet_bank_imag, requires_grad=False)\n\n self.kernel_size = self.wavelet_bank_real.size(3)\n\n def _build_wavelet_kernel(self):\n s_0 = 1 / self.upper_freq\n s_n = 1 / self.lower_freq\n\n base = np.power(s_n / s_0, 1 / (self.n_scales - 1))\n scales = s_0 * np.power(base, np.arange(self.n_scales))\n\n frequencies = 1 / scales\n truncation_size = scales.max() * np.sqrt(4.5 * self.initial_wavelet_width) * self.fs\n one_side = int(self.size_factor * truncation_size)\n kernel_size = 2 * one_side + 1\n\n k_array = np.arange(kernel_size, dtype=np.float32) - one_side\n t_array = k_array / self.fs\n\n wavelet_bank_real = []\n wavelet_bank_imag = []\n\n for scale in scales:\n norm_constant = np.sqrt(np.pi * self.wavelet_width) * scale * self.fs / 2.0\n scaled_t = t_array / scale\n exp_term = np.exp(-(scaled_t ** 2) / self.wavelet_width)\n kernel_base = exp_term / norm_constant\n kernel_real = kernel_base * np.cos(2 * np.pi * scaled_t)\n kernel_imag = kernel_base * np.sin(2 * np.pi * scaled_t)\n wavelet_bank_real.append(kernel_real)\n wavelet_bank_imag.append(kernel_imag)\n\n wavelet_bank_real = np.stack(wavelet_bank_real, axis=0)\n wavelet_bank_imag = np.stack(wavelet_bank_imag, axis=0)\n\n wavelet_bank_real = torch.from_numpy(wavelet_bank_real).unsqueeze(1).unsqueeze(2)\n wavelet_bank_imag = torch.from_numpy(wavelet_bank_imag).unsqueeze(1).unsqueeze(2)\n return wavelet_bank_real, wavelet_bank_imag\n\n def forward(self, x):\n border_crop = self.border_crop // self.stride\n start = border_crop\n end = (-border_crop) if border_crop > 0 else None\n\n # x [n_batch, n_channels, time_len]\n out_reals = []\n out_imags = []\n\n in_width = x.size(2)\n out_width = int(np.ceil(in_width / self.stride))\n pad_along_width = np.max((out_width - 1) * self.stride + self.kernel_size - in_width, 0)\n padding = pad_along_width // 2 + 1\n\n for i in range(3):\n # [n_batch, 1, 1, time_len]\n x_ = x[:, i, :].unsqueeze(1).unsqueeze(2)\n out_real = nn.functional.conv2d(x_, self.wavelet_bank_real, stride=(1, self.stride), padding=(0, padding))\n out_imag = nn.functional.conv2d(x_, self.wavelet_bank_imag, stride=(1, self.stride), padding=(0, padding))\n out_real = out_real.transpose(2, 1)\n out_imag = out_imag.transpose(2, 1)\n out_reals.append(out_real)\n out_imags.append(out_imag)\n\n out_real = torch.cat(out_reals, axis=1)\n out_imag = torch.cat(out_imags, axis=1)\n\n out_real = out_real[:, :, :, start:end]\n out_imag = out_imag[:, :, :, start:end]\n\n scalograms = torch.sqrt(out_real ** 2 + out_imag ** 2)\n return scalograms\nHey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris