Business context: https://www.kaggle.com/competitions/leash-BELKA/overview
\nData context: https://www.kaggle.com/competitions/leash-BELKA/data
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F312334%2Fac5a96eb60939a87e40bfdc3a0bfd59c%2Fleash-bio-solution.png?generation=1720838439586161&alt=media\")
\nA summary of the solution: ligand-based, SMILES-only, multi-target transformer model trained independently on shared and non-shared building blocks splits. A special hit identification data splitting technique, and k-fold training and validation are used for non-shared blocks. The models are independently averaged for the shared and non-shared parts of the test set.
This competition hinges on the fact that similar molecules tend to have similar properties - this is the main assumption in lead optimization of initially identified hits. However, the correct evaluation strategy is rarely applied, which is why Polaris, a collective effort to establish unbiased benchmarks for drug discovery, was launched recently. Moreover, it's hard to say what is correct for a specific task. Yet, it was shown recently by my colleague Simon Steshin that a splitting strategy by 0.4 Tanimoto similarity should be considered for novel hits identification (Lo-Hi: Practical ML Drug Discovery Benchmark). Perhaps we can apply a similar approach to build a more generalizable model in other tasks, particularly in this competition where 66% of the private score is dedicated to the out-of-distribution molecules.
\nThe molecules provided by the organizers consist of three building blocks, so we may consider splitting by the similarity of building blocks. Let’s examine the distribution of test building blocks to training building blocks:
\n![\"Distribution](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F312334%2F4348e00ecff143d7bcc71fa29175851a%2Ftest2train_bb_tanimoto_dists.png?generation=1720837870293702&alt=media\")
There are many building blocks shared between train and test sets (Max Tanimoto is 1.0), and we obviously need to avoid the same occurrence in our non-shared train/valid split. We see that BB1 doesn't intersect with BB2, and BB2 significantly intersects with BB3. Besides the clear duplicates, there are many similar building blocks, which, as per the above assumption, will introduce a strong bias into our evaluation.
\nSo, let's eliminate this bias. We will use the Hi splitting algorithm from the Lo-Hi benchmark which solves a Balanced Vertex Minimum k-Cut problem to construct such training and validation sets that the closest molecules between the two sets will be at least the cutoff Tanimoto distance apart (code). To decide what threshold to use, let's look at the training set's Tanimoto similarity distributions per block position, considering only the same positions (we will deal with BB2-BB3 cross-positions later):
\n![\"Distribution](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F312334%2F47e417d61abf31653b9885b3c3044426%2Ftrain2train_bb_tanimoto_dists.png?generation=1720837978207046&alt=media\")
It's an open question of what threshold to set, and most likely, using multiple ones is the best approach. I chose the thresholds close to the mean of the per-position similarities (don't ask me why): 0.7 for BB1<->BB2 and 0.4 for (BB2+BB3<->BB2+BB3). Yes, I merged BB2 with BB3, as they are known to intersect each other. Ok, this split is the hardest one. Training models on it will quickly show that nothing is working (in other words, structure similarity drives a lot of correlation with activity).
\n![\"HIBB](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F312334%2F938d225e60ef54b163b8be07b2a5b688%2Fleash-bio-solution-HIBB-split.png?generation=1720838534349396&alt=media\")
Pay attention to the exclusivity of building blocks - if a block appears in one set, it cannot appear in another set in any possible combination.
\nAlthough the Hi split is useful, it might be over-pessimistic for the competition's problem, and it removes a lot of data due to the exclusive assignment of building blocks to the train/valid sets. So, I decided to create a simpler split close to what other participants did when replicating the host's split but different in a few aspects:
\nany positive molecules (the ones that bind to at least one target) and the same amount of all negative molecules (no binding to all targets) randomly sampled from the corresponding fold's building blocksany positive set and an amount of all negative compounds to approximately match the imbalance of the original training dataset, i.e., any positive rate is ~1.5%Overall, the folds look like this (randomization not taken into account for simplicity):
\n| Fold | \nBB1 | \nBB2&BB3 | \nBB3\\BB2 | \ntrain size | \ntrain pos rate | \nvalid size | \nvalid pos rate | \n
|---|---|---|---|---|---|---|---|
| 1 | \n0-50 | \n0-101 | \n0-5 | \n1 875 494 | \n50% | \n356 620 | \n1.67% | \n
| 2 | \n51-101 | \n102-203 | \n6-11 | \n1 777 354 | \n50% | \n406 629 | \n1.62% | \n
| 3 | \n102-152 | \n204-305 | \n12-17 | \n1 430 522 | \n50% | \n363 503 | \n2.96% | \n
| 4 | \n153-203 | \n306-407 | \n18-23 | \n1 924 372 | \n50% | \n311 943 | \n1.47% | \n
| 5 | \n204-254 | \n408-509 | \n24-29 | \n1 903 992 | \n50% | \n313 838 | \n1.47% | \n
As with the HIBB split, the validation set is of similar size per fold, but we now have much more training data:
\n![\"BB](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F312334%2F3d2b650b86bb046c919b1cf20ec2418e%2Fleash-bio-solution-BB-split.png?generation=1720838175494292&alt=media\")
The strategy for the building blocks shared between train and test sets is clear—we just need to overfit to the known building blocks while covering as much data as possible. So, here, I used all 98M molecules, of which I randomly selected 1% for validation and 99% for training.
\nTo put it simply, the training strategies I used are the following:
\nI tried A LOT of models and performed extensive hyperparameters tuning (especially on the non-shared BBs) for the following models:
\nSimAttn model) could learn from the closest ranked list of molecules, and it did, but I wasn't able to reach high enough AP with itFrom the above, the best performing models are the fine-tuned MolFormer (45M parameters) and a custom RoBERTa (~8M parameters) as described in my company's recent paper. The customization is mainly related to the 500-sized BPE SMILES tokenizer, 15% of masked tokens in 30% of cases, and SMILES re-enumeration in 50% of cases.
\nMy take on generalizability to unseen chemical subspaces:
\nI thank Copilot and Continue.dev for being my coding teammates all the time. I also acknowledge my colleague Simon for his public work on molecule benchmarking. I thank other participants for their solutions and discussions; I didn't find time to contribute to the discussions, but I was aligned with many of them. Huge thanks to Leash Bio and Kaggle for organizing the competition. And I'm sorry for the teams shuffled on the private leaderboard. I had a solution for 0.299 ten days before the end of the competition, so I expect the same to happen to many other teams. I believe they all tried hard to crack this unsolved but intriguing small molecules generalizability problem.
", "messages": [ { "id": "2919585", "postDate": "07/13/2024 02:46:10", "content": "Business context: https://www.kaggle.com/competitions/leash-BELKA/overview
\nData context: https://www.kaggle.com/competitions/leash-BELKA/data
\nA summary of the solution: ligand-based, SMILES-only, multi-target transformer model trained independently on shared and non-shared building blocks splits. A special hit identification data splitting technique, and k-fold training and validation are used for non-shared blocks. The models are independently averaged for the shared and non-shared parts of the test set.
This competition hinges on the fact that similar molecules tend to have similar properties - this is the main assumption in lead optimization of initially identified hits. However, the correct evaluation strategy is rarely applied, which is why Polaris, a collective effort to establish unbiased benchmarks for drug discovery, was launched recently. Moreover, it's hard to say what is correct for a specific task. Yet, it was shown recently by my colleague Simon Steshin that a splitting strategy by 0.4 Tanimoto similarity should be considered for novel hits identification (Lo-Hi: Practical ML Drug Discovery Benchmark). Perhaps we can apply a similar approach to build a more generalizable model in other tasks, particularly in this competition where 66% of the private score is dedicated to the out-of-distribution molecules.
\nThe molecules provided by the organizers consist of three building blocks, so we may consider splitting by the similarity of building blocks. Let’s examine the distribution of test building blocks to training building blocks:
\n
There are many building blocks shared between train and test sets (Max Tanimoto is 1.0), and we obviously need to avoid the same occurrence in our non-shared train/valid split. We see that BB1 doesn't intersect with BB2, and BB2 significantly intersects with BB3. Besides the clear duplicates, there are many similar building blocks, which, as per the above assumption, will introduce a strong bias into our evaluation.
\nSo, let's eliminate this bias. We will use the Hi splitting algorithm from the Lo-Hi benchmark which solves a Balanced Vertex Minimum k-Cut problem to construct such training and validation sets that the closest molecules between the two sets will be at least the cutoff Tanimoto distance apart (code). To decide what threshold to use, let's look at the training set's Tanimoto similarity distributions per block position, considering only the same positions (we will deal with BB2-BB3 cross-positions later):
\n
It's an open question of what threshold to set, and most likely, using multiple ones is the best approach. I chose the thresholds close to the mean of the per-position similarities (don't ask me why): 0.7 for BB1<->BB2 and 0.4 for (BB2+BB3<->BB2+BB3). Yes, I merged BB2 with BB3, as they are known to intersect each other. Ok, this split is the hardest one. Training models on it will quickly show that nothing is working (in other words, structure similarity drives a lot of correlation with activity).
\n
Pay attention to the exclusivity of building blocks - if a block appears in one set, it cannot appear in another set in any possible combination.
\nAlthough the Hi split is useful, it might be over-pessimistic for the competition's problem, and it removes a lot of data due to the exclusive assignment of building blocks to the train/valid sets. So, I decided to create a simpler split close to what other participants did when replicating the host's split but different in a few aspects:
\nany positive molecules (the ones that bind to at least one target) and the same amount of all negative molecules (no binding to all targets) randomly sampled from the corresponding fold's building blocksany positive set and an amount of all negative compounds to approximately match the imbalance of the original training dataset, i.e., any positive rate is ~1.5%Overall, the folds look like this (randomization not taken into account for simplicity):
\n| Fold | \nBB1 | \nBB2&BB3 | \nBB3\\BB2 | \ntrain size | \ntrain pos rate | \nvalid size | \nvalid pos rate | \n
|---|---|---|---|---|---|---|---|
| 1 | \n0-50 | \n0-101 | \n0-5 | \n1 875 494 | \n50% | \n356 620 | \n1.67% | \n
| 2 | \n51-101 | \n102-203 | \n6-11 | \n1 777 354 | \n50% | \n406 629 | \n1.62% | \n
| 3 | \n102-152 | \n204-305 | \n12-17 | \n1 430 522 | \n50% | \n363 503 | \n2.96% | \n
| 4 | \n153-203 | \n306-407 | \n18-23 | \n1 924 372 | \n50% | \n311 943 | \n1.47% | \n
| 5 | \n204-254 | \n408-509 | \n24-29 | \n1 903 992 | \n50% | \n313 838 | \n1.47% | \n
As with the HIBB split, the validation set is of similar size per fold, but we now have much more training data:
\n
The strategy for the building blocks shared between train and test sets is clear—we just need to overfit to the known building blocks while covering as much data as possible. So, here, I used all 98M molecules, of which I randomly selected 1% for validation and 99% for training.
\nTo put it simply, the training strategies I used are the following:
\nI tried A LOT of models and performed extensive hyperparameters tuning (especially on the non-shared BBs) for the following models:
\nSimAttn model) could learn from the closest ranked list of molecules, and it did, but I wasn't able to reach high enough AP with itFrom the above, the best performing models are the fine-tuned MolFormer (45M parameters) and a custom RoBERTa (~8M parameters) as described in my company's recent paper. The customization is mainly related to the 500-sized BPE SMILES tokenizer, 15% of masked tokens in 30% of cases, and SMILES re-enumeration in 50% of cases.
\nMy take on generalizability to unseen chemical subspaces:
\nI thank Copilot and Continue.dev for being my coding teammates all the time. I also acknowledge my colleague Simon for his public work on molecule benchmarking. I thank other participants for their solutions and discussions; I didn't find time to contribute to the discussions, but I was aligned with many of them. Huge thanks to Leash Bio and Kaggle for organizing the competition. And I'm sorry for the teams shuffled on the private leaderboard. I had a solution for 0.299 ten days before the end of the competition, so I expect the same to happen to many other teams. I believe they all tried hard to crack this unsolved but intriguing small molecules generalizability problem.
", "rawMarkdown": "# Context\nBusiness context: https://www.kaggle.com/competitions/leash-BELKA/overview \nData context: https://www.kaggle.com/competitions/leash-BELKA/data\n\n# Summary\n\n\nA summary of the solution: ligand-based, SMILES-only, multi-target transformer model trained independently on shared and non-shared building blocks splits. A special hit identification data splitting technique, and k-fold training and validation are used for non-shared blocks. The models are independently averaged for the shared and non-shared parts of the test set.\n\n# Data splits\nThis competition hinges on the fact that similar molecules tend to have similar properties - this is the main assumption in lead optimization of initially identified hits. However, the correct evaluation strategy is rarely applied, which is why [Polaris](https://polarishub.io/), a collective effort to establish unbiased benchmarks for drug discovery, was launched recently. Moreover, it's hard to say what is correct for a specific task. Yet, it was shown recently by my colleague Simon Steshin that a splitting strategy by 0.4 Tanimoto similarity should be considered for novel hits identification ([Lo-Hi: Practical ML Drug Discovery Benchmark](https://arxiv.org/abs/2310.06399)). Perhaps we can apply a similar approach to build a more generalizable model in other tasks, particularly in this competition where 66% of the private score is dedicated to the out-of-distribution molecules.\n\n## Non-shared BBs\nThe molecules provided by the organizers consist of three building blocks, so we may consider splitting by the similarity of building blocks. Let’s examine the distribution of test building blocks to training building blocks:\n\n\n\nThere are many building blocks shared between train and test sets (Max Tanimoto is 1.0), and we obviously need to avoid the same occurrence in our non-shared train/valid split. We see that BB1 doesn't intersect with BB2, and BB2 significantly intersects with BB3. Besides the clear duplicates, there are many similar building blocks, which, as per the above assumption, will introduce a strong bias into our evaluation.\n\n### Hit identification split by building blocks (HIBB)\nSo, let's eliminate this bias. We will use the Hi splitting algorithm from the Lo-Hi benchmark which solves a Balanced Vertex Minimum k-Cut problem to construct such training and validation sets that the closest molecules between the two sets will be at least the cutoff Tanimoto distance apart ([code](https://github.com/SteshinSS/lohi_splitter)). To decide what threshold to use, let's look at the training set's Tanimoto similarity distributions per block position, considering only the same positions (we will deal with BB2-BB3 cross-positions later):\n\n\n\nIt's an open question of what threshold to set, and most likely, using multiple ones is the best approach. I chose the thresholds close to the mean of the per-position similarities (don't ask me why): 0.7 for BB1<->BB2 and 0.4 for (BB2+BB3<->BB2+BB3). Yes, I merged BB2 with BB3, as they are known to intersect each other. Ok, this split is the hardest one. Training models on it will quickly show that nothing is working (in other words, structure similarity drives a lot of correlation with activity).\n\n\n\nPay attention to the exclusivity of building blocks - if a block appears in one set, it cannot appear in another set in any possible combination.\n\n### Weighted building blocks splits (BB)\nAlthough the Hi split is useful, it might be over-pessimistic for the competition's problem, and it removes a lot of data due to the exclusive assignment of building blocks to the train/valid sets. So, I decided to create a simpler split close to what other participants did when replicating the host's split but different in a few aspects:\n- The blocks are again exclusive to the train/valid sets\n- There's an increased amount of blocks at each position by 3 times compared to the BB split from the great [notebook](https://www.kaggle.com/code/thedrcat/belka-split-cv-like-the-host/) of @thedrcat and work of both @roberthatch and @hengck23 [here](https://www.kaggle.com/competitions/leash-BELKA/discussion/496576)\n- There are rolling 5 folds where, for each fold, different blocks are selected at each position\n- Training data consists of a mix of `any positive` molecules (the ones that bind to at least one target) and the same amount of `all negative` molecules (no binding to all targets) randomly sampled from the corresponding fold's building blocks\n- Validation data consists of a mix of what's left from the `any positive` set and an amount of `all negative` compounds to approximately match the imbalance of the original training dataset, i.e., `any positive` rate is ~1.5%\n\nOverall, the folds look like this (randomization not taken into account for simplicity):\n| Fold | BB1 | BB2&BB3 | BB3\\BB2 | train size | train pos rate | valid size | valid pos rate |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| 1 | 0-50 | 0-101 | 0-5 | 1 875 494 | 50% | 356 620 | 1.67%\n| 2 | 51-101 | 102-203 | 6-11 | 1 777 354 | 50% | 406 629 | 1.62%\n| 3 | 102-152 | 204-305 | 12-17 | 1 430 522 | 50% | 363 503 | 2.96%\n| 4 | 153-203 | 306-407 | 18-23 | 1 924 372 | 50% | 311 943 | 1.47%\n| 5 | 204-254 | 408-509 | 24-29 | 1 903 992 | 50% | 313 838 | 1.47%\n\nAs with the HIBB split, the validation set is of similar size per fold, but we now have much more training data:\n\n\n\n## Shared BBs\nThe strategy for the building blocks shared between train and test sets is clear—we just need to overfit to the known building blocks while covering as much data as possible. So, here, I used all 98M molecules, of which I randomly selected 1% for validation and 99% for training.\n\n# Models and training\nTo put it simply, the training strategies I used are the following:\n- **shared BBs**: overfitting while tracking performance on validation, taking the last checkpoint\n- **non-shared BBs**: taking the best checkpoint on the validation set (the same strategy for both HIBB and BB splits)\n\nI tried A LOT of models and performed extensive hyperparameters tuning (especially on the non-shared BBs) for the following models:\n- [MolFormer](https://huggingface.co/ibm/MoLFormer-XL-both-10pct), [RoBERTa ZINC](https://huggingface.co/entropy/roberta_zinc_480m), [ChemBERTa](https://huggingface.co/DeepChem/ChemBERTa-77M-MTR): all showed similar quality with MolFormer and a custom RoBERTa performing better in an ensemble\n- GPS++: SOTA full-attention GNN with atoms and bonds features (impl. adopted from [Graphium](https://github.com/datamol-io/graphium)). For atoms, the following features are used: atomic-number, group, period, total-valence, degree, formal-charge, radical-electron, hybridization, chirality, implicit-valence, num_h_atoms, aromatic, in-ring, electronegativity. For bonds: bond-type-onehot, stereo, in-ring\n- MPNN++: GNN with the same atoms and bonds features (impl. adopted from [Graphium](https://github.com/datamol-io/graphium)), see performance in the [scaling GNNs paper](https://arxiv.org/abs/2404.11568). In my case, MPNN++ also performed substantially better than GPS++, potentially due to not attending everything to everything, thus having a stronger inductive bias\n- Tanimoto similarity scores of building blocks to the top-50 closest molecules put together in a multi-head attention model where queries/keys are the similarities and values are the ground truth values. It was supposed that this kind of a model (I call it the `SimAttn` model) could learn from the closest ranked list of molecules, and it did, but I wasn't able to reach high enough AP with it\n- XGBoost on RDKit 210 descriptors normalized with standard scaler and ECFP4\n- MLP on RDKit + ECFP4 + similarity features\n- Meta MLP model on RDKit + ECFP4 + similarity feats + best performing MolFormer: slightly improved performance on BB split, but not enough to convince me to submit\n\nFrom the above, the best performing models are the fine-tuned MolFormer (45M parameters) and a custom RoBERTa (~8M parameters) as described in [my company's recent paper](https://arxiv.org/abs/2406.14572). The customization is mainly related to the 500-sized BPE SMILES tokenizer, 15% of masked tokens in 30% of cases, and SMILES re-enumeration in 50% of cases.\n\n# Conclusion\nMy take on generalizability to unseen chemical subspaces:\n1. This isn't a solved problem, not only for this competition but everywhere in the public domain\n2. We can address the issue with extensive benchmarking\n3. SMILES-based transformer models perform slightly better than the SOTA GNN models\n4. Scaling number of parameters **improves** the quality on out-of-distribution molecules\n5. Scaling SMILES-based transformers is way easier than GNNs due to the absence of pre-processing, so scaling to billion-size models and datasets to improve generalizability should be considered in the follow-up research\n\n# Acknowledgements\nI thank Copilot and Continue.dev for being my coding teammates all the time. I also acknowledge my colleague Simon for his public work on molecule benchmarking. I thank other participants for their solutions and discussions; I didn't find time to contribute to the discussions, but I was aligned with many of them. Huge thanks to Leash Bio and Kaggle for organizing the competition. And I'm sorry for the teams shuffled on the private leaderboard. I had a solution for 0.299 ten days before the end of the competition, so I expect the same to happen to many other teams. I believe they all tried hard to crack this unsolved but intriguing small molecules generalizability problem.", "votes": null }, { "id": "2919777", "postDate": "07/13/2024 07:39:46", "content": "Congratulations on the gold, it's a very interesting splitting approach, thanks for sharing!
\nAnd what was the difference in the results with random splitting by BBs (e.g. holding a subset of BBs for validation) vs. splitting by similarity??
\nHow much did the splitting you used improve the LB score for the non-shared part (private and public)?
", "rawMarkdown": "Congratulations on the gold, it's a very interesting splitting approach, thanks for sharing!\n\nAnd what was the difference in the results with random splitting by BBs (e.g. holding a subset of BBs for validation) vs. splitting by similarity?? \n\nHow much did the splitting you used improve the LB score for the non-shared part (private and public)?", "votes": null }, { "id": "2920275", "postDate": "07/13/2024 14:50:33", "content": "This solution highlights an innovative approach using a SMILES-based transformer model for the competition.. The strategy of training independently on shared and non-shared building block splits, alongside the use of a special hit identification data splitting technique (HIBB), appears robust for handling out-of-distribution molecules effectively…The extensive model exploration and hyperparameter tuning efforts are commendable, showcasing a thorough investigation into optimizing model performance across different molecular subspaces.. Overall.. this solution sets a high standard for future research in small molecule drug discovery…!!!!
", "rawMarkdown": "This solution highlights an innovative approach using a SMILES-based transformer model for the competition.. The strategy of training independently on shared and non-shared building block splits, alongside the use of a special hit identification data splitting technique (HIBB), appears robust for handling out-of-distribution molecules effectively...The extensive model exploration and hyperparameter tuning efforts are commendable, showcasing a thorough investigation into optimizing model performance across different molecular subspaces.. Overall.. this solution sets a high standard for future research in small molecule drug discovery...!!!!", "votes": null }, { "id": "2923450", "postDate": "07/15/2024 22:53:00", "content": "Hey @antoninadolgorukova ! Thanks for the great question.
\nI had to run some additional submissions to have a clear answer. Look at these results - they perfectly show that the HIBB split dominates for the non-shared BBs while it's the worst split for the shared BBs (due to underfitting and early stopping). And vice versa, random split with its overfitting on all data performs the worst on non-shared BBs but is best on shared BBs. And the BB split is somewhere between both scenarios.
| Model | \nSplit type | \nTrain data | \nPrivate LB (non-shared) | \nPublic LB (non-shared) | \nPrivate LB (shared) | \nPublic LB (shared) | \n
|---|---|---|---|---|---|---|
| MolFormer (10 epochs, class weights = neg frac) | \nrandom | \n99% of all 98M dataset (random) | \n0.04688 | \n0.05891 | \n0.19826 | \n0.30954 | \n
| MolFormer x 5 fold ensemble: avg (each best on valid) | \nBB split | \n~1.8M pos/neg balance (see solution) | \n0.07169 | \n0.09743 | \n0.13066 | \n0.20767 | \n
| MolFormer (best on valid per target) | \nHIBB split | \n~362k (pos rate ~35%) | \n0.08100 | \n0.10530 | \n0.05856 | \n0.08926 | \n
Congratulations on the gold, it's a very interesting splitting approach, thanks for sharing!
\nAnd what was the difference in the results with random splitting by BBs (e.g. holding a subset of BBs for validation) vs. splitting by similarity??
\nHow much did the splitting you used improve the LB score for the non-shared part (private and public)?
", "votes": null, "replies": [ { "id": 2923450, "author_name": "vladvin", "author_url": "", "post_date": "07/15/2024 22:53:00", "content": "Hey @antoninadolgorukova ! Thanks for the great question.
\nI had to run some additional submissions to have a clear answer. Look at these results - they perfectly show that the HIBB split dominates for the non-shared BBs while it's the worst split for the shared BBs (due to underfitting and early stopping). And vice versa, random split with its overfitting on all data performs the worst on non-shared BBs but is best on shared BBs. And the BB split is somewhere between both scenarios.
| Model | \nSplit type | \nTrain data | \nPrivate LB (non-shared) | \nPublic LB (non-shared) | \nPrivate LB (shared) | \nPublic LB (shared) | \n
|---|---|---|---|---|---|---|
| MolFormer (10 epochs, class weights = neg frac) | \nrandom | \n99% of all 98M dataset (random) | \n0.04688 | \n0.05891 | \n0.19826 | \n0.30954 | \n
| MolFormer x 5 fold ensemble: avg (each best on valid) | \nBB split | \n~1.8M pos/neg balance (see solution) | \n0.07169 | \n0.09743 | \n0.13066 | \n0.20767 | \n
| MolFormer (best on valid per target) | \nHIBB split | \n~362k (pos rate ~35%) | \n0.08100 | \n0.10530 | \n0.05856 | \n0.08926 | \n
This solution highlights an innovative approach using a SMILES-based transformer model for the competition.. The strategy of training independently on shared and non-shared building block splits, alongside the use of a special hit identification data splitting technique (HIBB), appears robust for handling out-of-distribution molecules effectively…The extensive model exploration and hyperparameter tuning efforts are commendable, showcasing a thorough investigation into optimizing model performance across different molecular subspaces.. Overall.. this solution sets a high standard for future research in small molecule drug discovery…!!!!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2919585": "# Context\nBusiness context: https://www.kaggle.com/competitions/leash-BELKA/overview \nData context: https://www.kaggle.com/competitions/leash-BELKA/data\n\n# Summary\n\n\nA summary of the solution: ligand-based, SMILES-only, multi-target transformer model trained independently on shared and non-shared building blocks splits. A special hit identification data splitting technique, and k-fold training and validation are used for non-shared blocks. The models are independently averaged for the shared and non-shared parts of the test set.\n\n# Data splits\nThis competition hinges on the fact that similar molecules tend to have similar properties - this is the main assumption in lead optimization of initially identified hits. However, the correct evaluation strategy is rarely applied, which is why [Polaris](https://polarishub.io/), a collective effort to establish unbiased benchmarks for drug discovery, was launched recently. Moreover, it's hard to say what is correct for a specific task. Yet, it was shown recently by my colleague Simon Steshin that a splitting strategy by 0.4 Tanimoto similarity should be considered for novel hits identification ([Lo-Hi: Practical ML Drug Discovery Benchmark](https://arxiv.org/abs/2310.06399)). Perhaps we can apply a similar approach to build a more generalizable model in other tasks, particularly in this competition where 66% of the private score is dedicated to the out-of-distribution molecules.\n\n## Non-shared BBs\nThe molecules provided by the organizers consist of three building blocks, so we may consider splitting by the similarity of building blocks. Let’s examine the distribution of test building blocks to training building blocks:\n\n\n\nThere are many building blocks shared between train and test sets (Max Tanimoto is 1.0), and we obviously need to avoid the same occurrence in our non-shared train/valid split. We see that BB1 doesn't intersect with BB2, and BB2 significantly intersects with BB3. Besides the clear duplicates, there are many similar building blocks, which, as per the above assumption, will introduce a strong bias into our evaluation.\n\n### Hit identification split by building blocks (HIBB)\nSo, let's eliminate this bias. We will use the Hi splitting algorithm from the Lo-Hi benchmark which solves a Balanced Vertex Minimum k-Cut problem to construct such training and validation sets that the closest molecules between the two sets will be at least the cutoff Tanimoto distance apart ([code](https://github.com/SteshinSS/lohi_splitter)). To decide what threshold to use, let's look at the training set's Tanimoto similarity distributions per block position, considering only the same positions (we will deal with BB2-BB3 cross-positions later):\n\n\n\nIt's an open question of what threshold to set, and most likely, using multiple ones is the best approach. I chose the thresholds close to the mean of the per-position similarities (don't ask me why): 0.7 for BB1<->BB2 and 0.4 for (BB2+BB3<->BB2+BB3). Yes, I merged BB2 with BB3, as they are known to intersect each other. Ok, this split is the hardest one. Training models on it will quickly show that nothing is working (in other words, structure similarity drives a lot of correlation with activity).\n\n\n\nPay attention to the exclusivity of building blocks - if a block appears in one set, it cannot appear in another set in any possible combination.\n\n### Weighted building blocks splits (BB)\nAlthough the Hi split is useful, it might be over-pessimistic for the competition's problem, and it removes a lot of data due to the exclusive assignment of building blocks to the train/valid sets. So, I decided to create a simpler split close to what other participants did when replicating the host's split but different in a few aspects:\n- The blocks are again exclusive to the train/valid sets\n- There's an increased amount of blocks at each position by 3 times compared to the BB split from the great [notebook](https://www.kaggle.com/code/thedrcat/belka-split-cv-like-the-host/) of @thedrcat and work of both @roberthatch and @hengck23 [here](https://www.kaggle.com/competitions/leash-BELKA/discussion/496576)\n- There are rolling 5 folds where, for each fold, different blocks are selected at each position\n- Training data consists of a mix of `any positive` molecules (the ones that bind to at least one target) and the same amount of `all negative` molecules (no binding to all targets) randomly sampled from the corresponding fold's building blocks\n- Validation data consists of a mix of what's left from the `any positive` set and an amount of `all negative` compounds to approximately match the imbalance of the original training dataset, i.e., `any positive` rate is ~1.5%\n\nOverall, the folds look like this (randomization not taken into account for simplicity):\n| Fold | BB1 | BB2&BB3 | BB3\\BB2 | train size | train pos rate | valid size | valid pos rate |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| 1 | 0-50 | 0-101 | 0-5 | 1 875 494 | 50% | 356 620 | 1.67%\n| 2 | 51-101 | 102-203 | 6-11 | 1 777 354 | 50% | 406 629 | 1.62%\n| 3 | 102-152 | 204-305 | 12-17 | 1 430 522 | 50% | 363 503 | 2.96%\n| 4 | 153-203 | 306-407 | 18-23 | 1 924 372 | 50% | 311 943 | 1.47%\n| 5 | 204-254 | 408-509 | 24-29 | 1 903 992 | 50% | 313 838 | 1.47%\n\nAs with the HIBB split, the validation set is of similar size per fold, but we now have much more training data:\n\n\n\n## Shared BBs\nThe strategy for the building blocks shared between train and test sets is clear—we just need to overfit to the known building blocks while covering as much data as possible. So, here, I used all 98M molecules, of which I randomly selected 1% for validation and 99% for training.\n\n# Models and training\nTo put it simply, the training strategies I used are the following:\n- **shared BBs**: overfitting while tracking performance on validation, taking the last checkpoint\n- **non-shared BBs**: taking the best checkpoint on the validation set (the same strategy for both HIBB and BB splits)\n\nI tried A LOT of models and performed extensive hyperparameters tuning (especially on the non-shared BBs) for the following models:\n- [MolFormer](https://huggingface.co/ibm/MoLFormer-XL-both-10pct), [RoBERTa ZINC](https://huggingface.co/entropy/roberta_zinc_480m), [ChemBERTa](https://huggingface.co/DeepChem/ChemBERTa-77M-MTR): all showed similar quality with MolFormer and a custom RoBERTa performing better in an ensemble\n- GPS++: SOTA full-attention GNN with atoms and bonds features (impl. adopted from [Graphium](https://github.com/datamol-io/graphium)). For atoms, the following features are used: atomic-number, group, period, total-valence, degree, formal-charge, radical-electron, hybridization, chirality, implicit-valence, num_h_atoms, aromatic, in-ring, electronegativity. For bonds: bond-type-onehot, stereo, in-ring\n- MPNN++: GNN with the same atoms and bonds features (impl. adopted from [Graphium](https://github.com/datamol-io/graphium)), see performance in the [scaling GNNs paper](https://arxiv.org/abs/2404.11568). In my case, MPNN++ also performed substantially better than GPS++, potentially due to not attending everything to everything, thus having a stronger inductive bias\n- Tanimoto similarity scores of building blocks to the top-50 closest molecules put together in a multi-head attention model where queries/keys are the similarities and values are the ground truth values. It was supposed that this kind of a model (I call it the `SimAttn` model) could learn from the closest ranked list of molecules, and it did, but I wasn't able to reach high enough AP with it\n- XGBoost on RDKit 210 descriptors normalized with standard scaler and ECFP4\n- MLP on RDKit + ECFP4 + similarity features\n- Meta MLP model on RDKit + ECFP4 + similarity feats + best performing MolFormer: slightly improved performance on BB split, but not enough to convince me to submit\n\nFrom the above, the best performing models are the fine-tuned MolFormer (45M parameters) and a custom RoBERTa (~8M parameters) as described in [my company's recent paper](https://arxiv.org/abs/2406.14572). The customization is mainly related to the 500-sized BPE SMILES tokenizer, 15% of masked tokens in 30% of cases, and SMILES re-enumeration in 50% of cases.\n\n# Conclusion\nMy take on generalizability to unseen chemical subspaces:\n1. This isn't a solved problem, not only for this competition but everywhere in the public domain\n2. We can address the issue with extensive benchmarking\n3. SMILES-based transformer models perform slightly better than the SOTA GNN models\n4. Scaling number of parameters **improves** the quality on out-of-distribution molecules\n5. Scaling SMILES-based transformers is way easier than GNNs due to the absence of pre-processing, so scaling to billion-size models and datasets to improve generalizability should be considered in the follow-up research\n\n# Acknowledgements\nI thank Copilot and Continue.dev for being my coding teammates all the time. I also acknowledge my colleague Simon for his public work on molecule benchmarking. I thank other participants for their solutions and discussions; I didn't find time to contribute to the discussions, but I was aligned with many of them. Huge thanks to Leash Bio and Kaggle for organizing the competition. And I'm sorry for the teams shuffled on the private leaderboard. I had a solution for 0.299 ten days before the end of the competition, so I expect the same to happen to many other teams. I believe they all tried hard to crack this unsolved but intriguing small molecules generalizability problem.", "2919777": "Congratulations on the gold, it's a very interesting splitting approach, thanks for sharing!\n\nAnd what was the difference in the results with random splitting by BBs (e.g. holding a subset of BBs for validation) vs. splitting by similarity?? \n\nHow much did the splitting you used improve the LB score for the non-shared part (private and public)?", "2920275": "This solution highlights an innovative approach using a SMILES-based transformer model for the competition.. The strategy of training independently on shared and non-shared building block splits, alongside the use of a special hit identification data splitting technique (HIBB), appears robust for handling out-of-distribution molecules effectively...The extensive model exploration and hyperparameter tuning efforts are commendable, showcasing a thorough investigation into optimizing model performance across different molecular subspaces.. Overall.. this solution sets a high standard for future research in small molecule drug discovery...!!!!", "2923450": "Hey @antoninadolgorukova ! Thanks for the great question.\nI had to run some additional submissions to have a clear answer. Look at these results - they perfectly show that the HIBB split dominates for the non-shared BBs while it's the worst split for the shared BBs (due to underfitting and early stopping). And vice versa, random split with its overfitting on all data performs the worst on non-shared BBs but is best on shared BBs. And the BB split is somewhere between both scenarios.\n\n| Model | Split type | Train data | Private LB (non-shared) | Public LB (non-shared) | Private LB (shared) | Public LB (shared) |\n| --- | --- | --- | --- | --- |\n| MolFormer (10 epochs, class weights = neg frac) | random | 99% of all 98M dataset (random) | 0.04688 | 0.05891 | **0.19826** | **0.30954** |\n| MolFormer x 5 fold ensemble: avg (each best on valid) | BB split | ~1.8M pos/neg balance (see solution) | 0.07169 | 0.09743 | 0.13066 | 0.20767 |\n| MolFormer (best on valid per target) | HIBB split | ~362k (pos rate ~35%) | **0.08100** | **0.10530** | 0.05856 | 0.08926 |" }, "source": "meta" }