![\"\"](\"https://i.ibb.co/txpjSWk/Selection-041.png\")
\n
![\"\"](\"https://i.ibb.co/C9fQh7P/Selection-042.png\")
\n
![\"\"](\"https://i.ibb.co/wymW9fB/Selection-040.png\")
\n
![\"\"](\"https://i.ibb.co/9hhCfrw/Selection-039.png\")
[summary]
\nDesign a sliding window conv classifier net. Trained with tp and fp annotations (no pesudo label) on resnet34, this architechiture achieved LB 0.937/0.939 (public/private) in 5 fold model. A single fold model gives 0.881/0.888 (public/private). The first max pooling resnet34 is removed to make the stride of the backbone 16 (instead of 32)
Final submission is LB 0.945/0.943 (public rank 15/private rank 19) is an ensemble of this network with different CNN backbone.
I am grateful for Kaggle and the host Rainforest Connection (RFCx) for organizing the competition.
\nAs a Z by HP & NVidia global datascience ambassador under https://datascience.hp.com/us/en.html.HP & Nvidia has kindly provided a Z8 datascience workstation for my use in this competition. Without this powerful workstation, it would not be possible for me to develop my idea from scratch within 13 days.
\nBecause I can finish experiments at a very great speed (z8 has ax quadro rtx 8000, NVlink 96GB), I gain a lot of insights into model training and model design when the experimental feedback is almost instantaneous. I want to share these insights with the kaggle community.
\nHence I started another thread to guide and supervise kagglers who wants to improve their training skills. This is targeted to bring them from sliver to gold levels. You can refer to:
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220379
\nhttps://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/217238
data preprocessing
\naugmentation
\nmodel and loss
\n
\n
\n
\n
daily progress:
\n![\"\"](\"https://i.ibb.co/7kGcfpj/Selection-050.png\")
\nI enter the competition are reading CPMP post, saying that he has 0.930 LB in the first submission. WOW! Reading between the lines, it means:
I was able to train 5-fold efficientB0 and resnet50 in the last day. Because of the large GPU card of the HP workstation I am using, I can train large models with large batch size. When I compared training the same model with smaller batch size on my old machines, I find that the results are different and inferior, even if I use gradient accumulation.
\nI strongly feel that we have reached a new era. I think this is also why the latest RTX3090 has larger GPU memory than the previous cards. The future is the transformers …. meaning more GPU memory. That's is how fast deep learning are moving!
", "messages": [ { "id": "1207607", "postDate": "02/18/2021 00:13:24", "content": "[summary]
\nDesign a sliding window conv classifier net. Trained with tp and fp annotations (no pesudo label) on resnet34, this architechiture achieved LB 0.937/0.939 (public/private) in 5 fold model. A single fold model gives 0.881/0.888 (public/private). The first max pooling resnet34 is removed to make the stride of the backbone 16 (instead of 32)
Final submission is LB 0.945/0.943 (public rank 15/private rank 19) is an ensemble of this network with different CNN backbone.
I am grateful for Kaggle and the host Rainforest Connection (RFCx) for organizing the competition.
\nAs a Z by HP & NVidia global datascience ambassador under https://datascience.hp.com/us/en.html.HP & Nvidia has kindly provided a Z8 datascience workstation for my use in this competition. Without this powerful workstation, it would not be possible for me to develop my idea from scratch within 13 days.
\nBecause I can finish experiments at a very great speed (z8 has ax quadro rtx 8000, NVlink 96GB), I gain a lot of insights into model training and model design when the experimental feedback is almost instantaneous. I want to share these insights with the kaggle community.
\nHence I started another thread to guide and supervise kagglers who wants to improve their training skills. This is targeted to bring them from sliver to gold levels. You can refer to:
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220379
\nhttps://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/217238
data preprocessing
\naugmentation
\nmodel and loss
\n
\n
\n
\n
daily progress:
\n
\nI enter the competition are reading CPMP post, saying that he has 0.930 LB in the first submission. WOW! Reading between the lines, it means:
I was able to train 5-fold efficientB0 and resnet50 in the last day. Because of the large GPU card of the HP workstation I am using, I can train large models with large batch size. When I compared training the same model with smaller batch size on my old machines, I find that the results are different and inferior, even if I use gradient accumulation.
\nI strongly feel that we have reached a new era. I think this is also why the latest RTX3090 has larger GPU memory than the previous cards. The future is the transformers …. meaning more GPU memory. That's is how fast deep learning are moving!
", "rawMarkdown": "[summary]\n\n- Design a sliding window conv classifier net. Trained with tp and fp annotations (no pesudo label) on resnet34, this architechiture achieved LB 0.937/0.939 (public/private) in 5 fold model. A single fold model gives 0.881/0.888 (public/private). The first max pooling resnet34 is removed to make the stride of the backbone 16 (instead of 32)\n\n- Final submission is LB 0.945/0.943 (public rank 15/private rank 19) is an ensemble of this network with different CNN backbone.\n\n---\n\nI am grateful for Kaggle and the host Rainforest Connection (RFCx) for organizing the competition. \n\nAs a Z by HP & NVidia global datascience ambassador under https://datascience.hp.com/us/en.html.HP & Nvidia has kindly provided a Z8 datascience workstation for my use in this competition. Without this powerful workstation, it would not be possible for me to develop my idea from scratch within 13 days. \n\nBecause I can finish experiments at a very great speed (z8 has ax quadro rtx 8000, NVlink 96GB), I gain a lot of insights into model training and model design when the experimental feedback is almost instantaneous. I want to share these insights with the kaggle community. \n\nHence I started another thread to guide and supervise kagglers who wants to improve their training skills. This is targeted to bring them from sliver to gold levels. You can refer to:\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220379\nhttps://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/217238\n\n---\ndata preprocessing\n- log mel spectrogram with modified PCEN denoising (per- channel energy normalization). A 10sec clip has melspec of size 128x938\nn_fft = 2048\nwin_length = 2048\nhop_length = 512 \nnum_freq = 128\n\n\n---\naugmentation\n- I haven't tried much augmentation yet. For TP annotation, random shift by 0.02 sec. For FP annotation, random shift width of the annotation and then random flip, mixup, cut and paste of the original annotation and its shift version.\n- Augmentation is done in the temporal time domain because of my code. (this is not the best solution)\n- It can be observed that if I increased the FP augmentation, the bce loss of the validation FP decreases and the public LB score improved.\n- heavy dropout in model\n\n---\nmodel and loss\n- please see the below images\n- during training, I monitor the separate log loss of validation TP and FP. I also use the LRAP assumpting one label (i.e. top-1 correctness). This is to decide when to early stop or decrease learning rate. These are not the best approach.\n\n\n\n\n\n\n\ndaily progress:\n\nI enter the competition are reading CPMP post, saying that he has 0.930 LB in the first submission. WOW! Reading between the lines, it means:\n1. the public kernels are doing it \"wrong\". It could be the problem setting, the network architecture, the data, or some magic features. there is something very fundamental that can do better than the public kernel.\n2. 0.930 is quite high score (in the gold region). With 2 more week to the deadline, I decided to give it a try. if I solved this unobvious puzzle, i may end up in the gold region as well.\n3. the first step is to go through the public kernel (some PANN SED) and the baseline image classifier. I want to see what most people are doing (and could be improved)\n4. after working for a week, I realize that there are two major \"potential flaws\":\n - treating it as a multi-instance learning SED. Well, MIL could be a solution, but the problem is that we don't have bag label (clip label). Most MIL required bag level label, but lack instance level label (segment label).Here we have the opposite.\n - not using FP in train. Most public kernel use only TP as train only. \n5. hence I start to design my own network architecture. The first step is an efficient way to do crop classification based on mean annotation box. Hence I design the slide window network.\n6. The next step is to use TP+FP in training\n7. The last step is to use pesudo labels, but I don't have time to complete this. But I do have some initial experimental results on this. Top-1 (max over time) pesudo labels is about 95% accurate for LB score of 0.94. This is good enough for distillation.\n8. Because pesudo labeling requires many probing to prevent error propagation, I could not do it because I don't have sufficient slots in the last days. Worst still, there is no train data at clip level at all. This makes local testing impossible.\n\nI was able to train 5-fold efficientB0 and resnet50 in the last day. Because of the large GPU card of the HP workstation I am using, I can train large models with large batch size. When I compared training the same model with smaller batch size on my old machines, I find that the results are different and inferior, even if I use gradient accumulation.\n\nI strongly feel that we have reached a new era. I think this is also why the latest RTX3090 has larger GPU memory than the previous cards. The future is the transformers .... meaning more GPU memory. That's is how fast deep learning are moving!", "votes": null }, { "id": "1207639", "postDate": "02/18/2021 00:35:19", "content": "anyone can suggest a good free image hosting website. The one I am using may delete the images after some time.
", "rawMarkdown": "anyone can suggest a good free image hosting website. The one I am using may delete the images after some time.", "votes": null }, { "id": "1207642", "postDate": "02/18/2021 00:37:12", "content": "I think github gist is good
", "rawMarkdown": "I think github gist is good", "votes": null }, { "id": "1207646", "postDate": "02/18/2021 00:38:15", "content": "Devil is in detail but it seems we have a similar method. I welcome comments about what we did differently. And congrats on the good result in few days.
", "rawMarkdown": "Devil is in detail but it seems we have a similar method. I welcome comments about what we did differently. And congrats on the good result in few days.", "votes": null }, { "id": "1207754", "postDate": "02/18/2021 02:52:44", "content": "Congratulations for strong solo finish👍 Looking forward to hear about 0.881(1fold)->0.937(5fold) boost! I was able to get 0.92x with single fold single model but my score worsened when doing 5fold ensembling..
\nEdit) Never mind, I checked and my other fold lb scores were bad, that was the reason
I've been using imgur
", "rawMarkdown": "I've been using imgur", "votes": null }, { "id": "1207870", "postDate": "02/18/2021 04:21:02", "content": "@hengck23 Congratulations :D, can you share ur make_freq_mask function ?
", "rawMarkdown": "hengck23 Congratulations :D, can you share ur make_freq_mask function ?", "votes": null }, { "id": "1208355", "postDate": "02/18/2021 09:00:01", "content": "@CPMP Thanks for the comment, Your first 0.930 submission inspires me to enter the competition and race against time. Thanks a lot and congrats on your solo gold!
\nhere are what I think are the same and differences of our approaches:
\nthe same: high level objectives
\nthe differences: implementation details
\nyou crop the sepctrograms but I didn't. I have a larger receptive field (could be more signal or more noise). I use sliding roi window instead. There is no resizing of the roi to fixed size like 224x224. hence I preserved the same scale across all classes(with may or may not be a good thing??)
most CNN do average pooling at the last layer before classification. Here I use a conv kernel to do aggregation instead. In this way I make the CNN variant to the frequency information.
Apart from these, I think the rest are the same. I can see that the use of TP and FP and loss function are the same. There will be some differences in augmentation, etc.
", "rawMarkdown": "CPMP Thanks for the comment, Your first 0.930 submission inspires me to enter the competition and race against time. Thanks a lot and congrats on your solo gold!\n\nhere are what I think are the same and differences of our approaches:\n\nthe same: high level objectives\n - to use limited frequency fmin and fmax per class\n - basically, train a classifier on the \"crop spectrograms\"\n - make the CNN **variant** to the frequency information\n\nthe differences: implementation details\n - you crop the sepctrograms but I didn't. I have a larger receptive field (could be more signal or more noise). I use sliding roi window instead. There is no resizing of the roi to fixed size like 224x224. hence I preserved the same scale across all classes(with may or may not be a good thing??)\n\n - most CNN do average pooling at the last layer before classification. Here I use a conv kernel to do aggregation instead. In this way I make the CNN **variant** to the frequency information.\n\nApart from these, I think the rest are the same. I can see that the use of TP and FP and loss function are the same. There will be some differences in augmentation, etc.", "votes": null }, { "id": "1208501", "postDate": "02/18/2021 09:57:21", "content": "Congrats @hengck23 on your solo achievement. Waiting eagerly to see full writeup. I can learn a lot here!
", "rawMarkdown": "Congrats @hengck23 on your solo achievement. Waiting eagerly to see full writeup. I can learn a lot here!", "votes": null }, { "id": "1209446", "postDate": "02/18/2021 22:45:48", "content": "Thanks for adding great explanations. Our methods are very close, but your implementation is probably more efficient as you avoid my overlapping crops along the time dimension for a given f_min, f_max pair. It works becaus eyou use max pooling where I used mean pooling.
\nAnd thank you for mentioning my first sub as motivation. Unless mistaken you're the only one acknowledging it helped. Little fix: I got 0.931 at first sub, not 0.930 ;)
\nGood work in few days, congrats!
", "rawMarkdown": "Thanks for adding great explanations. Our methods are very close, but your implementation is probably more efficient as you avoid my overlapping crops along the time dimension for a given f_min, f_max pair. It works becaus eyou use max pooling where I used mean pooling.\n\nAnd thank you for mentioning my first sub as motivation. Unless mistaken you're the only one acknowledging it helped. Little fix: I got 0.931 at first sub, not 0.930 ;)\n\nGood work in few days, congrats!", "votes": null } ], "comments": [ { "id": 1207639, "author_name": "hengck23", "author_url": "", "post_date": "02/18/2021 00:35:19", "content": "anyone can suggest a good free image hosting website. The one I am using may delete the images after some time.
", "votes": null, "replies": [ { "id": 1207642, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "02/18/2021 00:37:12", "content": "I think github gist is good
", "votes": null, "replies": [] }, { "id": 1207868, "author_name": "ryches", "author_url": "", "post_date": "02/18/2021 04:17:59", "content": "I've been using imgur
", "votes": null, "replies": [] } ] }, { "id": 1207646, "author_name": "cpmpml", "author_url": "", "post_date": "02/18/2021 00:38:15", "content": "Devil is in detail but it seems we have a similar method. I welcome comments about what we did differently. And congrats on the good result in few days.
", "votes": null, "replies": [ { "id": 1208355, "author_name": "hengck23", "author_url": "", "post_date": "02/18/2021 09:00:01", "content": "@CPMP Thanks for the comment, Your first 0.930 submission inspires me to enter the competition and race against time. Thanks a lot and congrats on your solo gold!
\nhere are what I think are the same and differences of our approaches:
\nthe same: high level objectives
\nthe differences: implementation details
\nyou crop the sepctrograms but I didn't. I have a larger receptive field (could be more signal or more noise). I use sliding roi window instead. There is no resizing of the roi to fixed size like 224x224. hence I preserved the same scale across all classes(with may or may not be a good thing??)
most CNN do average pooling at the last layer before classification. Here I use a conv kernel to do aggregation instead. In this way I make the CNN variant to the frequency information.
Apart from these, I think the rest are the same. I can see that the use of TP and FP and loss function are the same. There will be some differences in augmentation, etc.
", "votes": null, "replies": [] } ] }, { "id": 1207754, "author_name": "harangdev", "author_url": "", "post_date": "02/18/2021 02:52:44", "content": "Congratulations for strong solo finish👍 Looking forward to hear about 0.881(1fold)->0.937(5fold) boost! I was able to get 0.92x with single fold single model but my score worsened when doing 5fold ensembling..
\nEdit) Never mind, I checked and my other fold lb scores were bad, that was the reason
@hengck23 Congratulations :D, can you share ur make_freq_mask function ?
", "votes": null, "replies": [] }, { "id": 1208501, "author_name": "rajkumarl", "author_url": "", "post_date": "02/18/2021 09:57:21", "content": "Congrats @hengck23 on your solo achievement. Waiting eagerly to see full writeup. I can learn a lot here!
", "votes": null, "replies": [] }, { "id": 1209446, "author_name": "cpmpml", "author_url": "", "post_date": "02/18/2021 22:45:48", "content": "Thanks for adding great explanations. Our methods are very close, but your implementation is probably more efficient as you avoid my overlapping crops along the time dimension for a given f_min, f_max pair. It works becaus eyou use max pooling where I used mean pooling.
\nAnd thank you for mentioning my first sub as motivation. Unless mistaken you're the only one acknowledging it helped. Little fix: I got 0.931 at first sub, not 0.930 ;)
\nGood work in few days, congrats!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1207607": "[summary]\n\n- Design a sliding window conv classifier net. Trained with tp and fp annotations (no pesudo label) on resnet34, this architechiture achieved LB 0.937/0.939 (public/private) in 5 fold model. A single fold model gives 0.881/0.888 (public/private). The first max pooling resnet34 is removed to make the stride of the backbone 16 (instead of 32)\n\n- Final submission is LB 0.945/0.943 (public rank 15/private rank 19) is an ensemble of this network with different CNN backbone.\n\n---\n\nI am grateful for Kaggle and the host Rainforest Connection (RFCx) for organizing the competition. \n\nAs a Z by HP & NVidia global datascience ambassador under https://datascience.hp.com/us/en.html.HP & Nvidia has kindly provided a Z8 datascience workstation for my use in this competition. Without this powerful workstation, it would not be possible for me to develop my idea from scratch within 13 days. \n\nBecause I can finish experiments at a very great speed (z8 has ax quadro rtx 8000, NVlink 96GB), I gain a lot of insights into model training and model design when the experimental feedback is almost instantaneous. I want to share these insights with the kaggle community. \n\nHence I started another thread to guide and supervise kagglers who wants to improve their training skills. This is targeted to bring them from sliver to gold levels. You can refer to:\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220379\nhttps://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/217238\n\n---\ndata preprocessing\n- log mel spectrogram with modified PCEN denoising (per- channel energy normalization). A 10sec clip has melspec of size 128x938\nn_fft = 2048\nwin_length = 2048\nhop_length = 512 \nnum_freq = 128\n\n\n---\naugmentation\n- I haven't tried much augmentation yet. For TP annotation, random shift by 0.02 sec. For FP annotation, random shift width of the annotation and then random flip, mixup, cut and paste of the original annotation and its shift version.\n- Augmentation is done in the temporal time domain because of my code. (this is not the best solution)\n- It can be observed that if I increased the FP augmentation, the bce loss of the validation FP decreases and the public LB score improved.\n- heavy dropout in model\n\n---\nmodel and loss\n- please see the below images\n- during training, I monitor the separate log loss of validation TP and FP. I also use the LRAP assumpting one label (i.e. top-1 correctness). This is to decide when to early stop or decrease learning rate. These are not the best approach.\n\n\n\n\n\n\n\ndaily progress:\n\nI enter the competition are reading CPMP post, saying that he has 0.930 LB in the first submission. WOW! Reading between the lines, it means:\n1. the public kernels are doing it \"wrong\". It could be the problem setting, the network architecture, the data, or some magic features. there is something very fundamental that can do better than the public kernel.\n2. 0.930 is quite high score (in the gold region). With 2 more week to the deadline, I decided to give it a try. if I solved this unobvious puzzle, i may end up in the gold region as well.\n3. the first step is to go through the public kernel (some PANN SED) and the baseline image classifier. I want to see what most people are doing (and could be improved)\n4. after working for a week, I realize that there are two major \"potential flaws\":\n - treating it as a multi-instance learning SED. Well, MIL could be a solution, but the problem is that we don't have bag label (clip label). Most MIL required bag level label, but lack instance level label (segment label).Here we have the opposite.\n - not using FP in train. Most public kernel use only TP as train only. \n5. hence I start to design my own network architecture. The first step is an efficient way to do crop classification based on mean annotation box. Hence I design the slide window network.\n6. The next step is to use TP+FP in training\n7. The last step is to use pesudo labels, but I don't have time to complete this. But I do have some initial experimental results on this. Top-1 (max over time) pesudo labels is about 95% accurate for LB score of 0.94. This is good enough for distillation.\n8. Because pesudo labeling requires many probing to prevent error propagation, I could not do it because I don't have sufficient slots in the last days. Worst still, there is no train data at clip level at all. This makes local testing impossible.\n\nI was able to train 5-fold efficientB0 and resnet50 in the last day. Because of the large GPU card of the HP workstation I am using, I can train large models with large batch size. When I compared training the same model with smaller batch size on my old machines, I find that the results are different and inferior, even if I use gradient accumulation.\n\nI strongly feel that we have reached a new era. I think this is also why the latest RTX3090 has larger GPU memory than the previous cards. The future is the transformers .... meaning more GPU memory. That's is how fast deep learning are moving!", "1207639": "anyone can suggest a good free image hosting website. The one I am using may delete the images after some time.", "1207642": "I think github gist is good", "1207646": "Devil is in detail but it seems we have a similar method. I welcome comments about what we did differently. And congrats on the good result in few days.", "1207754": "Congratulations for strong solo finish👍 Looking forward to hear about 0.881(1fold)->0.937(5fold) boost! I was able to get 0.92x with single fold single model but my score worsened when doing 5fold ensembling..\nEdit) Never mind, I checked and my other fold lb scores were bad, that was the reason", "1207868": "I've been using imgur", "1207870": "hengck23 Congratulations :D, can you share ur make_freq_mask function ?", "1208355": "CPMP Thanks for the comment, Your first 0.930 submission inspires me to enter the competition and race against time. Thanks a lot and congrats on your solo gold!\n\nhere are what I think are the same and differences of our approaches:\n\nthe same: high level objectives\n - to use limited frequency fmin and fmax per class\n - basically, train a classifier on the \"crop spectrograms\"\n - make the CNN **variant** to the frequency information\n\nthe differences: implementation details\n - you crop the sepctrograms but I didn't. I have a larger receptive field (could be more signal or more noise). I use sliding roi window instead. There is no resizing of the roi to fixed size like 224x224. hence I preserved the same scale across all classes(with may or may not be a good thing??)\n\n - most CNN do average pooling at the last layer before classification. Here I use a conv kernel to do aggregation instead. In this way I make the CNN **variant** to the frequency information.\n\nApart from these, I think the rest are the same. I can see that the use of TP and FP and loss function are the same. There will be some differences in augmentation, etc.", "1208501": "Congrats @hengck23 on your solo achievement. Waiting eagerly to see full writeup. I can learn a lot here!", "1209446": "Thanks for adding great explanations. Our methods are very close, but your implementation is probably more efficient as you avoid my overlapping crops along the time dimension for a given f_min, f_max pair. It works becaus eyou use max pooling where I used mean pooling.\n\nAnd thank you for mentioning my first sub as motivation. Unless mistaken you're the only one acknowledging it helped. Little fix: I got 0.931 at first sub, not 0.930 ;)\n\nGood work in few days, congrats!" }, "source": "meta" }