Thanks to everybody who organized this competition. Congratulations to all who received medals.
\nSeems that results of this competition aren't make a lot of sense because the host solution is the best.
Few words about my trying, hope you find it interesting or even useful.
\nMetrics
\nI start with macro Fb-score which seems are suitable for case of noisy labels. I chose quite big beta - 16 to boost recall.
\nAfter reading some comments of hosts about the metric I switched to:
\nM = (1/N) * (1/21) * ∑∑( (TP/(TP+FN) + TN/(TN+FP))/2 )
For any amount of training classes metric was calculated just over scored:
\ny_true = tf.gather(y_true, indices=self.mask, axis=1)
\ny_pred = tf.gather(y_pred, indices=self.mask, axis=1)
External data
\n2 recent recordings from Hawaii which can be found on Xeno-Canto.
\nff1010bird records without birdcalls to mix with.
Input
\nAs input pre-computed mel-spectrogram (mainly) or PCEN (in few cases) were used. Both calculated by Librosa with default parameters, nmels=128. So for 5s shape is 128x313.
\nFor mel-spectrogram primitive denoising strategy was used to reduce stationary noise over whole file. On inference same denoising approach was applied but on 5s chunk level instead whole file.
In case of mel-spec, after augmentations and before mixup it was log-scaled and normed from 0 to 1.
\nIn case of PCEN after tiny augmentations and before mixup it was normed from 0 to 1.
Idea to beat noisy labels a little bit (was implemented for few training attempts):
\nDrop time bins with low std (can be done for PCEN or log-melspectrogram):
std_over_f = pcen.std(axis=0)
\npcen_f = pcen[:, std_over_f > std_over_f.mean()]
Some mask smoothing can be applied before resampling by scipy.ndimage.convolve.
\nApproaches
\n2 different approaches were implemented:
ff1010bird dataset was also tried as initial input as nocall class in 1st approach and as absence of all classes in the 2nd instead of use it for mixing.
\nMainly the efficientnets (B0, B2, B3) were used with noisy-student weights, also seresnet101 was trained for mel-spec multiclass classifier.
\nInstead of simple global pooling in most attempts AutoPool over one axis followed by mean pooling was used.
Augmentations
\nThresholds
\nTrained models were evaluated over whole files from validation set and best thresholds were estimated. Obtained threshold for different approaches were averaged and tuned by knowledge about amount of data per bird in dataset.
\nMin value of 0.25 was used for 7 rare classes and just two species (skylar and houfin) have thresholds more than 0.5.
Ensemble
\nAt first average predictions are estimated per each approach (mel-spec multiclass, pcen multiclass, mel-spec mixin multilabel) and then final averaging was done with coefficients selected from LB results.
Post-processing
\nFor reduce FP and find out nocall parts few checks were implemented:
Thanks to everybody who organized this competition. Congratulations to all who received medals.
\nSeems that results of this competition aren't make a lot of sense because the host solution is the best.
Few words about my trying, hope you find it interesting or even useful.
\nMetrics
\nI start with macro Fb-score which seems are suitable for case of noisy labels. I chose quite big beta - 16 to boost recall.
\nAfter reading some comments of hosts about the metric I switched to:
\nM = (1/N) * (1/21) * ∑∑( (TP/(TP+FN) + TN/(TN+FP))/2 )
For any amount of training classes metric was calculated just over scored:
\ny_true = tf.gather(y_true, indices=self.mask, axis=1)
\ny_pred = tf.gather(y_pred, indices=self.mask, axis=1)
External data
\n2 recent recordings from Hawaii which can be found on Xeno-Canto.
\nff1010bird records without birdcalls to mix with.
Input
\nAs input pre-computed mel-spectrogram (mainly) or PCEN (in few cases) were used. Both calculated by Librosa with default parameters, nmels=128. So for 5s shape is 128x313.
\nFor mel-spectrogram primitive denoising strategy was used to reduce stationary noise over whole file. On inference same denoising approach was applied but on 5s chunk level instead whole file.
In case of mel-spec, after augmentations and before mixup it was log-scaled and normed from 0 to 1.
\nIn case of PCEN after tiny augmentations and before mixup it was normed from 0 to 1.
Idea to beat noisy labels a little bit (was implemented for few training attempts):
\nDrop time bins with low std (can be done for PCEN or log-melspectrogram):
std_over_f = pcen.std(axis=0)
\npcen_f = pcen[:, std_over_f > std_over_f.mean()]
Some mask smoothing can be applied before resampling by scipy.ndimage.convolve.
\nApproaches
\n2 different approaches were implemented:
ff1010bird dataset was also tried as initial input as nocall class in 1st approach and as absence of all classes in the 2nd instead of use it for mixing.
\nMainly the efficientnets (B0, B2, B3) were used with noisy-student weights, also seresnet101 was trained for mel-spec multiclass classifier.
\nInstead of simple global pooling in most attempts AutoPool over one axis followed by mean pooling was used.
Augmentations
\nThresholds
\nTrained models were evaluated over whole files from validation set and best thresholds were estimated. Obtained threshold for different approaches were averaged and tuned by knowledge about amount of data per bird in dataset.
\nMin value of 0.25 was used for 7 rare classes and just two species (skylar and houfin) have thresholds more than 0.5.
Ensemble
\nAt first average predictions are estimated per each approach (mel-spec multiclass, pcen multiclass, mel-spec mixin multilabel) and then final averaging was done with coefficients selected from LB results.
Post-processing
\nFor reduce FP and find out nocall parts few checks were implemented: