![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc983825b38881f09b12733c260f0e650%2FSelection_153.png?generation=1721740500859348&alt=media\")
https://github.com/hengck23/solution-leash-BELKA
\nWe use only character tokenization. we use CNN embedding (kernel size=3, stride=1) to learned combination of characters. We try other tokenizers like BPE, sentence piece, atom/smiles based, etc. But all perform worse than the simplest character-based tokenization.
\n
The final solution is an ensemble of 3 net architectures: cnn1d, transformer, mamba (SSM). We treat input as a sequence and the task as a 3-class ('BRD4', 'HSA', 'sEH') multi-label problem. We train with large batch sizes: cnn1d=5000, transformer=2500, mamba=2000. For cnn1d, performance is very sensitive to BN for large batch sizes. We think it is because of:
\nWe use high eps=5e-3 and low momentum=0.2 for cnn1d net.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0a518b669428d59245b0f0cb51bde9ed%2FSelection_161.png?generation=1721740806722371&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F1cd8da1f05892e2921dabea4c956529c%2FSelection_167.png?generation=1721777492055677&alt=media\")
With 98 million training molecules, it takes quite a lot of time to train each neural net. We do not have sufficient time to train different nets for each fold. Instead, we used different folds for different nets to improve ensemble diversity.
\nThis makes it difficult to compare performance for different nets. After the competition, we make some late submissions. Here are the results. It can be seen that the transformer is the most robust net.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F74090653a5768b187232ff9615694b92%2FSelection_169.png?generation=1721908106311645&alt=media\")
Next, we compare some heatmaps generated by gradCAM:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F8a3fe8f4b64f8d5da65d8df136d7488e%2FSelection_190.png?generation=1721960668364141&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F084217c2f874635eacd44d98e3a21b1c%2FSelection_189.png?generation=1721960706372788&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7f9feba17c936bec539bb9249c05c14f%2FSelection_187.png?generation=1721960724767966&alt=media\")
As expected(?) activation of cnn1d is quite local. Transformer has much global activation.
\nhttps://github.com/hengck23/solution-leash-BELKA
\nWe use only character tokenization. we use CNN embedding (kernel size=3, stride=1) to learned combination of characters. We try other tokenizers like BPE, sentence piece, atom/smiles based, etc. But all perform worse than the simplest character-based tokenization.
\n
The final solution is an ensemble of 3 net architectures: cnn1d, transformer, mamba (SSM). We treat input as a sequence and the task as a 3-class ('BRD4', 'HSA', 'sEH') multi-label problem. We train with large batch sizes: cnn1d=5000, transformer=2500, mamba=2000. For cnn1d, performance is very sensitive to BN for large batch sizes. We think it is because of:
\nWe use high eps=5e-3 and low momentum=0.2 for cnn1d net.
\n
With 98 million training molecules, it takes quite a lot of time to train each neural net. We do not have sufficient time to train different nets for each fold. Instead, we used different folds for different nets to improve ensemble diversity.
\nThis makes it difficult to compare performance for different nets. After the competition, we make some late submissions. Here are the results. It can be seen that the transformer is the most robust net.
\n
Next, we compare some heatmaps generated by gradCAM:
\n
As expected(?) activation of cnn1d is quite local. Transformer has much global activation.
\nThanks for sharing and congratulations! Very interesting to see the model interpretation using gradcam and xsmiles. Would you mind sharing the code for that?
", "rawMarkdown": "Thanks for sharing and congratulations! Very interesting to see the model interpretation using gradcam and xsmiles. Would you mind sharing the code for that?", "votes": null }, { "id": "2939224", "postDate": "07/28/2024 21:05:54", "content": "please see
\nhttps://www.kaggle.com/code/hengck23/example-to-use-gradcam-and-xsmiles
awesome !
", "rawMarkdown": "awesome !", "votes": null } ], "comments": [ { "id": 2939090, "author_name": "lililycai", "author_url": "", "post_date": "07/28/2024 18:21:41", "content": "Thanks for sharing and congratulations! Very interesting to see the model interpretation using gradcam and xsmiles. Would you mind sharing the code for that?
", "votes": null, "replies": [ { "id": 2939224, "author_name": "hengck23", "author_url": "", "post_date": "07/28/2024 21:05:54", "content": "please see
\nhttps://www.kaggle.com/code/hengck23/example-to-use-gradcam-and-xsmiles
awesome !
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2933090": "#GitHub code:\nhttps://github.com/hengck23/solution-leash-BELKA \n\n---\n\n#Brief description of the method:\n##1. Tokenization\nWe use only character tokenization. we use CNN embedding (kernel size=3, stride=1) to learned combination of characters. We try other tokenizers like BPE, sentence piece, atom/smiles based, etc. But all perform worse than the simplest character-based tokenization.\n\n\n\n##2. Network architecture\nThe final solution is an ensemble of 3 net architectures: cnn1d, transformer, mamba (SSM). We treat input as a sequence and the task as a 3-class ('BRD4', 'HSA', 'sEH') multi-label problem. We train with large batch sizes: cnn1d=5000, transformer=2500, mamba=2000. For cnn1d, performance is very sensitive to BN for large batch sizes. We think it is because of:\n - in-distribution and out-distribution samples have different feature values.\n - class is imbalance (positive class is less than 1%). positive and negative samples also have different feature values.\n\nWe use high eps=5e-3 and low momentum=0.2 for cnn1d net.\n\n\n\n\n\n\n---\n#Key observation:\n\nWith 98 million training molecules, it takes quite a lot of time to train each neural net. We do not have sufficient time to train different nets for each fold. Instead, we used different folds for different nets to improve ensemble diversity.\n\n- 7 hr for cnn1d (one fold)\n- 28 hr for transformer (one fold) \n- 36 hr for mamba (one fold) \n\nThis makes it difficult to compare performance for different nets. After the competition, we make some late submissions. Here are the results. It can be seen that the transformer is the most robust net.\n\n\n\nNext, we compare some heatmaps generated by gradCAM:\n\n\n\n\n\n\nAs expected(?) activation of cnn1d is quite local. Transformer has much global activation.\n\n---\n\n##Acknowledgement\n##\"We extend our thanks to HP for providing the Z8 Fury-G5 Data Science Workstation, which empowered our deep learning experiments. The high computational power and large GPU memory enabled us to design our models swiftly.\"", "2939090": "Thanks for sharing and congratulations! Very interesting to see the model interpretation using gradcam and xsmiles. Would you mind sharing the code for that?", "2939224": "please see\nhttps://www.kaggle.com/code/hengck23/example-to-use-gradcam-and-xsmiles", "2939301": "awesome !" }, "source": "meta" }