It was my first Kaggle competition on which I spent a lot of time. I'm honor to be part of it, big thanks to hosts, Kaggle and all participants.
\nI started with @kaerunantoka public notebook with change mel spectrogram hop length and input shape [224, 512] and add class weights to a submission.
\nWeights array is 500 divided to amount of each birds species, and clamp it to max value 10. Then I multiply model output to weights array. 0.77 Public LB.
I tried this approach with secondary labels with 0.3, 0.4 and 0.5 labels. And with ensembling I'v got 0.79 on LB
For waveform:
\nCompose([OneOf(\n [AddGaussianNoise(min_amplitude=0.001, max_amplitude=0.011, p=1),\n NoiseInjection(p=1, max_noise_level=0.04)], p=0.4), \n PitchShift(min_semitones=-4, max_semitones=4, p=0.1),\n Shift(min_fraction=-0.5, max_fraction=0.5, p=0.1),\n Gain(min_gain_in_db=-12, max_gain_in_db=12, p=0.2), \n Normalize(p=1.) ])\nFor spectrogram:
\ntorchaudio.transforms.FrequencyMasking(24)\ntorchaudio.transforms.TimeMasking(96)\nSED with tf_efficientnet_b0_ns backbone
\nI validate models on 7 folds CV with f1-macro metric. Thresholds for f1: [0.5, 0.1, 0.15, 0.20, 0.25, 0.30, 0.35, 0.4, 0.45, 0.5]. Best checkpoints chosen by 0.3 and 0.5 thresholds.
\nSame
\nSame
\nSED with tf_efficientnet_b0_ns backbone
\nwaveform, sr = ta.load(filename)\nwaveform = crop_or_pad(waveform, sr=SR, mode=self.mode)\nwaveform = waveform - waveform.mean()\nwaveform = torch.tensor(self.wave_transforms(samples=waveform[0].numpy(), sample_rate=SR)).unsqueeze(0)\nfbank = ta.compliance.kaldi.fbank(waveform, htk_compat=True, sample_frequency=SR, use_energy=False, window_type='hanning', num_mel_bins=self.melbins, dither=0.0, frame_shift=9.7)\nfbank = (fbank - self.norm_mean) / (self.norm_std * 2)\nMean and std calculated for all train dataset.
\nSED with tf_efficientnet_b0_ns backbone
\nSame
\n full_med = np.median(full_events, axis=1) \n full_mean = np.mean(full_events, axis=1)\n full_events = np.mean(np.array([full_med, full_mean]), axis=0) \n logits_max = full_events.max(0)\n for jk in range(full_events.shape[1]):\n if logits_max[jk] > threshold * 2.5:\n full_events[:, jk] += threshold * 0.5\nWhen there were 2 days left until the end of the competition, I thought about using a pseudo labeling. And separate my data by 0.8 threshold to remove all noisy records. Only ~15.000 left from ~40.000. Then I trained a new model on this data. Because there were not enough time I trained model on all train data and validate it on some random part and random part of only scored birds.
\nAll training and preprocessing parameters was the same with previous.
It was last day of competition and only 5 submissions. So the first 3 I lost because i forgot to make model.eval() :)
\n4th attempt I used with single new model and get 0.79 LB.
\nAnd the last one was mean between my best previous attempt and this pseudo labeling new model.
And it gave me 0.79 private, when my previous best private was 0.78
", "messages": [ { "id": 1800868, "postDate": "2022-05-25T09:28:36.957Z", "content": "It was my first Kaggle competition on which I spent a lot of time. I'm honor to be part of it, big thanks to hosts, Kaggle and all participants.
\nI started with @kaerunantoka public notebook with change mel spectrogram hop length and input shape [224, 512] and add class weights to a submission.
\nWeights array is 500 divided to amount of each birds species, and clamp it to max value 10. Then I multiply model output to weights array. 0.77 Public LB.
I tried this approach with secondary labels with 0.3, 0.4 and 0.5 labels. And with ensembling I'v got 0.79 on LB
For waveform:
\nCompose([OneOf(\n [AddGaussianNoise(min_amplitude=0.001, max_amplitude=0.011, p=1),\n NoiseInjection(p=1, max_noise_level=0.04)], p=0.4), \n PitchShift(min_semitones=-4, max_semitones=4, p=0.1),\n Shift(min_fraction=-0.5, max_fraction=0.5, p=0.1),\n Gain(min_gain_in_db=-12, max_gain_in_db=12, p=0.2), \n Normalize(p=1.) ])\nFor spectrogram:
\ntorchaudio.transforms.FrequencyMasking(24)\ntorchaudio.transforms.TimeMasking(96)\nSED with tf_efficientnet_b0_ns backbone
\nI validate models on 7 folds CV with f1-macro metric. Thresholds for f1: [0.5, 0.1, 0.15, 0.20, 0.25, 0.30, 0.35, 0.4, 0.45, 0.5]. Best checkpoints chosen by 0.3 and 0.5 thresholds.
\nSame
\nSame
\nSED with tf_efficientnet_b0_ns backbone
\nwaveform, sr = ta.load(filename)\nwaveform = crop_or_pad(waveform, sr=SR, mode=self.mode)\nwaveform = waveform - waveform.mean()\nwaveform = torch.tensor(self.wave_transforms(samples=waveform[0].numpy(), sample_rate=SR)).unsqueeze(0)\nfbank = ta.compliance.kaldi.fbank(waveform, htk_compat=True, sample_frequency=SR, use_energy=False, window_type='hanning', num_mel_bins=self.melbins, dither=0.0, frame_shift=9.7)\nfbank = (fbank - self.norm_mean) / (self.norm_std * 2)\nMean and std calculated for all train dataset.
\nSED with tf_efficientnet_b0_ns backbone
\nSame
\n full_med = np.median(full_events, axis=1) \n full_mean = np.mean(full_events, axis=1)\n full_events = np.mean(np.array([full_med, full_mean]), axis=0) \n logits_max = full_events.max(0)\n for jk in range(full_events.shape[1]):\n if logits_max[jk] > threshold * 2.5:\n full_events[:, jk] += threshold * 0.5\nWhen there were 2 days left until the end of the competition, I thought about using a pseudo labeling. And separate my data by 0.8 threshold to remove all noisy records. Only ~15.000 left from ~40.000. Then I trained a new model on this data. Because there were not enough time I trained model on all train data and validate it on some random part and random part of only scored birds.
\nAll training and preprocessing parameters was the same with previous.
It was last day of competition and only 5 submissions. So the first 3 I lost because i forgot to make model.eval() :)
\n4th attempt I used with single new model and get 0.79 LB.
\nAnd the last one was mean between my best previous attempt and this pseudo labeling new model.
And it gave me 0.79 private, when my previous best private was 0.78
", "rawMarkdown": "It was my first Kaggle competition on which I spent a lot of time. I'm honor to be part of it, big thanks to hosts, Kaggle and all participants.\n\n## First step\n\n1. I started with @kaerunantoka public [notebook](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0) with change mel spectrogram hop length and input shape [224, 512] and add **class weights** to a submission.\nWeights array is 500 divided to amount of each birds species, and clamp it to max value 10. Then I multiply model output to weights array. **0.77** Public LB.\n\n2. I tried this approach with secondary labels with 0.3, 0.4 and 0.5 labels. And with ensembling I'v got **0.79** on LB\n\n##### Augmentations\nFor waveform:\n```\nCompose([OneOf(\n [AddGaussianNoise(min_amplitude=0.001, max_amplitude=0.011, p=1),\n NoiseInjection(p=1, max_noise_level=0.04)], p=0.4), \n PitchShift(min_semitones=-4, max_semitones=4, p=0.1),\n Shift(min_fraction=-0.5, max_fraction=0.5, p=0.1),\n Gain(min_gain_in_db=-12, max_gain_in_db=12, p=0.2), \n Normalize(p=1.) ])\n```\nFor spectrogram:\n ```\ntorchaudio.transforms.FrequencyMasking(24)\ntorchaudio.transforms.TimeMasking(96)\n```\n##### Model \nSED with tf_efficientnet_b0_ns backbone\n\n##### Training\n* Epochs = 30\n* LR = 0.001\n* weight_decay = 0.0001\n* dropout = 0.4\n* Loss: Focal BCE Loss\n* Optimizer: Adam (betas=(0.95, 0.999))\n* Scheduler: CosineAnnealingLR\n\n##### Validation\nI validate models on 7 folds CV with f1-macro metric. Thresholds for f1: [0.5, 0.1, 0.15, 0.20, 0.25, 0.30, 0.35, 0.4, 0.45, 0.5]. Best checkpoints chosen by 0.3 and 0.5 thresholds.\n\n##### What didn't work on this step\n1. Noise reduction\n2. Oversampling rare birds\n3. Trim long silence with noise reduction\n4. Weighted loss\n5. 3 channels input, where 1st channel is power_to_db, 2nd melspectrogram, 3rd normalized melspectrogram\n\n\n## Second step\n1. I manually trim all segments without bird's sounds from the scored audios, except skylar and houfin, on them I processed only 70 records per class. \n2. Splited data to 15 seconds chunks for audios with length less than 1 minute and 30 seconds chunks for more than 1 minute. Got ~40.000 records.\n3. Used weights arrays with 0.25-0.75 power to reduce their impact to inference.\n4. Training on this data with previous pipeline and ensemble with previous models gives me **0.81** on LB\n\n##### Data preprocessing\nSame\n\n##### Augmentations\nSame\n\n##### Model \nSED with tf_efficientnet_b0_ns backbone\n\n##### Training\n* Epochs = 60\n* LR = 0.0013\n* weight_decay = 0.0001\n* dropout = 0.4\n* Loss: Focal BCE Loss\n* Optimizer: Adam (betas=(0.95, 0.999))\n* Scheduler: CosineAnnealingLR\n\n##### What didn't work on this step\n1. PaSST model\n2. AST model\n3. Linear head models\n4. PaSST preprocessing\n\n## Third step\n1. I used similar with AST preprocessing.\n2. Every epoch from all data randomly choose up to 300 records for every class.\n3. Train on random 5 seconds crop, validate on first 5 seconds.\n4. Secondary labels 0.4\n5. Class weights array clamped with max value 8. And used with power 0.6. \n6. Used mixup 0.4 for first 15 epochs, mixup 0.07 for 16-22 epochs, and no mixup for others.\n7. Ensembling of this approach with previous gives me **0.82** LB\n\n##### Data preprocessing\n```\nwaveform, sr = ta.load(filename)\nwaveform = crop_or_pad(waveform, sr=SR, mode=self.mode)\nwaveform = waveform - waveform.mean()\nwaveform = torch.tensor(self.wave_transforms(samples=waveform[0].numpy(), sample_rate=SR)).unsqueeze(0)\nfbank = ta.compliance.kaldi.fbank(waveform, htk_compat=True, sample_frequency=SR, use_energy=False, window_type='hanning', num_mel_bins=self.melbins, dither=0.0, frame_shift=9.7)\nfbank = (fbank - self.norm_mean) / (self.norm_std * 2)\n```\nMean and std calculated for all train dataset.\n\n##### Model \nSED with tf_efficientnet_b0_ns backbone\n\n##### Augmentations\nSame\n\n##### Training\n* Epochs = 85\n* mixup 0.4, when epochs < 15\n* LR = 0.0008\n* weight_decay = 0.0001\n* dropout = 0.4\n* Loss: Focal BCE Loss\n* Optimizer: Adam (betas=(0.95, 0.999))\n* Scheduler: CosineAnnealingLR\n\n##### Postprocessing\n* Mean-median averaging of predictions\n```\n full_med = np.median(full_events, axis=1) \n full_mean = np.mean(full_events, axis=1)\n full_events = np.mean(np.array([full_med, full_mean]), axis=0) \n```\n* Max adder.\n```\n logits_max = full_events.max(0)\n for jk in range(full_events.shape[1]):\n if logits_max[jk] > threshold * 2.5:\n full_events[:, jk] += threshold * 0.5\n```\n\n## Story of gold medal\nWhen there were 2 days left until the end of the competition, I thought about using a pseudo labeling. And separate my data by 0.8 threshold to remove all noisy records. Only ~15.000 left from ~40.000. Then I trained a new model on this data. Because there were not enough time I trained model on all train data and validate it on some random part and random part of only scored birds. \nAll training and preprocessing parameters was the same with previous.\n\nIt was last day of competition and only 5 submissions. So the first 3 I lost because i forgot to make model.eval() :)\n4th attempt I used with single new model and get **0.79** LB.\nAnd the last one was mean between my best previous attempt and this pseudo labeling new model.\n\nAnd it gave me **0.79** private, when my previous best private was **0.78** ", "votes": 32 }, { "id": 1825327, "postDate": "2022-06-19T08:51:06.513Z", "content": "@kolyaforrat Thanks for sharing great work. Keep it up.
\nBookmarked and upvoted πππ
", "rawMarkdown": "@kolyaforrat Thanks for sharing great work. Keep it up.\n\nBookmarked and upvoted πππ", "votes": 1 }, { "id": 1802983, "postDate": "2022-05-27T10:58:49.313Z", "content": "Congrats! on solo gold
", "rawMarkdown": "Congrats! on solo gold", "votes": 1 }, { "id": 1842005, "postDate": "2022-07-03T15:54:53.840Z", "content": "@kolyaforrat Congrats! Thanks for sharing great work.
", "rawMarkdown": "@kolyaforrat Congrats! Thanks for sharing great work." }, { "id": 1825270, "postDate": "2022-06-19T07:08:40.677Z", "content": "great work π
\njust one quastion
\nas i know we use the class weights in our loss function to focus on the classes with small number of samples during training but here you just multiply the predictions with the class weights can you explain it to me theoretically
Thanks!
\nWeighted loss didn't work for me here. This multiplies it's something like tuning thresholds which other participants does, but I didn't tune thresholds to lower values and just made predictions higher.
\nAnd with this method I didn't tune threshold manually, it always linearly depends of the number of training samples for each class.
", "rawMarkdown": "Thanks!\n\nWeighted loss didn't work for me here. This multiplies it's something like tuning thresholds which other participants does, but I didn't tune thresholds to lower values and just made predictions higher.\n\nAnd with this method I didn't tune threshold manually, it always linearly depends of the number of training samples for each class.", "votes": 1 }, { "id": 1825516, "postDate": "2022-06-19T12:23:38.523Z", "content": "Thank you π
", "rawMarkdown": "Thank you π" }, { "id": 1841456, "postDate": "2022-07-03T06:07:51.480Z", "content": "Hello @kolyaforrat , sorry for the inconvenience. I tried to do what you mentioned and as shown in the following code, but I did not see any improvement in the validation results. Also, I think that a threshold value should be extracted at the end as well. If it is true, did I rely on the validation set to find it .Thanks againπ
\n#params.target_columns contains 152 labels\nweights_dict = {i:int(0) for i in params.target_columns}\nfor targs in train.new_target:\n targs = targs.split(' ')\n for targ in targs:\n if targ != '':\n weights_dict[targ] += 1\nfor targ in params.target_columns:\n if targ != '':\n weights_dict[targ] = 500 / weights_dict[targ]\n if weights_dict[targ] > 10:\n weights_dict[targ] = 10\nweights_list = np.array([weights_dict[x] for x in weights_dict])\n\nparams.mode = \"valid\"\nwith torch.no_grad():\n all_preds = np.array([])\n all_targs = np.array([])\n for n,(xb,yb) in enumerate(tqdm(learn.dls.valid)):\n preds_dict = learn.model(xb)\n yb = yb.detach().cpu().numpy()\n preds = preds_dict['clipwise_output'].detach().cpu().numpy()\n preds = preds * weights_list #<---\n if n == 0:\n all_preds = preds\n all_targs = yb\n else:\n all_preds = np.vstack((all_preds,preds))\n all_targs = np.vstack((all_targs,yb))\n\nthreshs = np.linspace(0.0,2,1000)\nbest_score = 0\nfor thresh in tqdm(list(threshs)):\n score = metrics.f1_score(all_targs,all_preds > thresh, average=\"micro\") \n print(\"score\",score)\n print(\"thresh\",thresh)\n if score > best_score:\n best_score = score\nprint(best_score )\n@kolyaforrat Thanks for sharing great work. Keep it up.
\nBookmarked and upvoted πππ
", "votes": 1, "replies": [] }, { "id": 1802983, "author_name": "Neo", "author_url": "", "post_date": "2022-05-27T10:58:49.313000", "content": "Congrats! on solo gold
", "votes": 1, "replies": [] }, { "id": 1842005, "author_name": "Vipin Kumar", "author_url": "", "post_date": "2022-07-03T15:54:53.840000", "content": "@kolyaforrat Congrats! Thanks for sharing great work.
", "votes": 0, "replies": [] }, { "id": 1825270, "author_name": "RIYADH HAITHAM", "author_url": "", "post_date": "2022-06-19T07:08:40.677000", "content": "great work π
\njust one quastion
\nas i know we use the class weights in our loss function to focus on the classes with small number of samples during training but here you just multiply the predictions with the class weights can you explain it to me theoretically
Thanks!
\nWeighted loss didn't work for me here. This multiplies it's something like tuning thresholds which other participants does, but I didn't tune thresholds to lower values and just made predictions higher.
\nAnd with this method I didn't tune threshold manually, it always linearly depends of the number of training samples for each class.
", "votes": 1, "replies": [] }, { "id": 1825516, "author_name": "RIYADH HAITHAM", "author_url": "", "post_date": "2022-06-19T12:23:38.523000", "content": "Thank you π
", "votes": 0, "replies": [] }, { "id": 1841456, "author_name": "RIYADH HAITHAM", "author_url": "", "post_date": "2022-07-03T06:07:51.480000", "content": "Hello @kolyaforrat , sorry for the inconvenience. I tried to do what you mentioned and as shown in the following code, but I did not see any improvement in the validation results. Also, I think that a threshold value should be extracted at the end as well. If it is true, did I rely on the validation set to find it .Thanks againπ
\n#params.target_columns contains 152 labels\nweights_dict = {i:int(0) for i in params.target_columns}\nfor targs in train.new_target:\n targs = targs.split(' ')\n for targ in targs:\n if targ != '':\n weights_dict[targ] += 1\nfor targ in params.target_columns:\n if targ != '':\n weights_dict[targ] = 500 / weights_dict[targ]\n if weights_dict[targ] > 10:\n weights_dict[targ] = 10\nweights_list = np.array([weights_dict[x] for x in weights_dict])\n\nparams.mode = \"valid\"\nwith torch.no_grad():\n all_preds = np.array([])\n all_targs = np.array([])\n for n,(xb,yb) in enumerate(tqdm(learn.dls.valid)):\n preds_dict = learn.model(xb)\n yb = yb.detach().cpu().numpy()\n preds = preds_dict['clipwise_output'].detach().cpu().numpy()\n preds = preds * weights_list #<---\n if n == 0:\n all_preds = preds\n all_targs = yb\n else:\n all_preds = np.vstack((all_preds,preds))\n all_targs = np.vstack((all_targs,yb))\n\nthreshs = np.linspace(0.0,2,1000)\nbest_score = 0\nfor thresh in tqdm(list(threshs)):\n score = metrics.f1_score(all_targs,all_preds > thresh, average=\"micro\") \n print(\"score\",score)\n print(\"thresh\",thresh)\n if score > best_score:\n best_score = score\nprint(best_score )\n