{ "id": 518993, "title": "11th place solution - SSL Pretraining, Multi-models, Multi-representations and Luck", "url": "/competitions/leash-BELKA/writeups/ng-11th-place-solution-ssl-pretraining-multi-model", "author_name": "", "post_date": "2024-07-10T06:59:09.573Z", "votes": 33, "comment_count": 3, "views": 0, "content": "

First, I would like to thank Kaggle and competition's host for another interesting challenge, which bring me headaches for a long time :D\nThank you to all participants, especially who actively discuss and provide many good insights/code in the forum. As usual on Kaggle, I have a great time learning new things.

\n

First impression

\n\n

How to Cross-Validate?

\n

As mentioned above, I don't know.

\n

My split strategy for each fold:

\n
1. [11k samples] Hold out a number of building blocks for validation: 17 BB1 + 36 BB2 with positive-fraction-aware balance on each fold.\n2. [78k samples] Leave 20% molecules with most regular scaffolds (> 6116 mols/scaffold) for training, then do a Scaffold Split on the remaining 80% molecules\n3. [103k samples] Stratified Random split on the remaining\n
\n

Total: 11k non-shared + 181k share, simulate the LB\nThis strategy create a little \"harder\" than pure random split on the shared part.

\n

The hard part lie in 11k non-shared. Some observation on non-shared CV:

\n\n

In conclusion, more and more experiments messed up my mind. The variation is too large and seem to be random for me. I gave up and focus on SSL pretraining, improve the diversity by using multiple models and multiple input representations, and allmost all models were blindly train on all competition data with no cross-validation, since I did not trust a single CV split alone and train on multiple CV splits seem to be too time-intensive

\n

SSL Pretraining

\n

2 pretraining tasks:

\n\n

Joint MTR + MLM pretraining + SMILES Enumeration has success as mentioned in MolBERT, which inspired me to give it a try

\n

Pretraining using all data (both train + test), with larger sampling weights to test dataset to equalize the frequency of each building blocks and be \"more familiar\" with new domain test dataset.

\n

I trained 3 models to be finetuned on competition task:

\n\n

Modeling

\n

All models simultaneously predict 3 targets. Some models are not converged in the last day, and training must be terminated to finish in time:

\n\n

Some modeling results used in final ensemble. I think analyze share/nonshared/new library scores is needed for further analysis, and I will update that results later.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
File NamePrivate LeaderboardPublic Leaderboard
final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808_all0.3010.399
final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412_all0.2900.373
ensemble_v20.2780.448
ensemble_v10.2770.435
ensemble_v00.2740.438
final_smileschar-pregen_cnn1d-8-64_all0.2700.389
final_smileschar-pregen_cnn1d-3-512_all0.2570.386
final_selfies-pregen_cnn1d-3-512_all0.2530.367
final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898_all0.2510.407
final_selfies-pregen_cnn1d-6-128_all0.2440.365
final_ais-pregen_cnn1d-6-128_all0.2440.395
final_sage_all0.2420.433
final_deepsmiles-pregen_cnn1d-6-128_all0.2420.388
final_pretrain-gin_all0.2400.385
topo_mlp_all0.2360.377
final_smileschar-pregen_cnn1d-6-128_all0.2350.392
final_ais_squeeze_all0.2350.388
final_gcn_all0.2330.396
final_mamba_smileschar_all0.2330.394
squeezeformer_0.1033_0.6376_all0.2300.410
catboost_12chunks_all0.2240.385
ecfp6_mlp_all0.2180.359
mhfp_mlp_all0.1930.307
\n

Best single model was a Roberta with join MLM + MTR pretraining, scored 0.301 on Private Leaderboard and 0.399 on Public Leaderboard.

\n

Simple weighted ensemble with heuristic weights scores PB=27.8 and LB=44.8\nIndeed there are some single model with higher PB scores, but ensemble reduce variance and give more stable/trustworthy results. This result is luckily enough for a gold medal.

\n

Code

\n\n

Thanks for your attention !

", "messages": [ { "id": "2913113", "postDate": "07/09/2024 09:02:53", "content": "

First, I would like to thank Kaggle and competition's host for another interesting challenge, which bring me headaches for a long time :D\nThank you to all participants, especially who actively discuss and provide many good insights/code in the forum. As usual on Kaggle, I have a great time learning new things.

\n

First impression

\n\n

How to Cross-Validate?

\n

As mentioned above, I don't know.

\n

My split strategy for each fold:

\n
1. [11k samples] Hold out a number of building blocks for validation: 17 BB1 + 36 BB2 with positive-fraction-aware balance on each fold.\n2. [78k samples] Leave 20% molecules with most regular scaffolds (> 6116 mols/scaffold) for training, then do a Scaffold Split on the remaining 80% molecules\n3. [103k samples] Stratified Random split on the remaining\n
\n

Total: 11k non-shared + 181k share, simulate the LB\nThis strategy create a little \"harder\" than pure random split on the shared part.

\n

The hard part lie in 11k non-shared. Some observation on non-shared CV:

\n\n

In conclusion, more and more experiments messed up my mind. The variation is too large and seem to be random for me. I gave up and focus on SSL pretraining, improve the diversity by using multiple models and multiple input representations, and allmost all models were blindly train on all competition data with no cross-validation, since I did not trust a single CV split alone and train on multiple CV splits seem to be too time-intensive

\n

SSL Pretraining

\n

2 pretraining tasks:

\n\n

Joint MTR + MLM pretraining + SMILES Enumeration has success as mentioned in MolBERT, which inspired me to give it a try

\n

Pretraining using all data (both train + test), with larger sampling weights to test dataset to equalize the frequency of each building blocks and be \"more familiar\" with new domain test dataset.

\n

I trained 3 models to be finetuned on competition task:

\n\n

Modeling

\n

All models simultaneously predict 3 targets. Some models are not converged in the last day, and training must be terminated to finish in time:

\n\n

Some modeling results used in final ensemble. I think analyze share/nonshared/new library scores is needed for further analysis, and I will update that results later.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
File NamePrivate LeaderboardPublic Leaderboard
final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808_all0.3010.399
final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412_all0.2900.373
ensemble_v20.2780.448
ensemble_v10.2770.435
ensemble_v00.2740.438
final_smileschar-pregen_cnn1d-8-64_all0.2700.389
final_smileschar-pregen_cnn1d-3-512_all0.2570.386
final_selfies-pregen_cnn1d-3-512_all0.2530.367
final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898_all0.2510.407
final_selfies-pregen_cnn1d-6-128_all0.2440.365
final_ais-pregen_cnn1d-6-128_all0.2440.395
final_sage_all0.2420.433
final_deepsmiles-pregen_cnn1d-6-128_all0.2420.388
final_pretrain-gin_all0.2400.385
topo_mlp_all0.2360.377
final_smileschar-pregen_cnn1d-6-128_all0.2350.392
final_ais_squeeze_all0.2350.388
final_gcn_all0.2330.396
final_mamba_smileschar_all0.2330.394
squeezeformer_0.1033_0.6376_all0.2300.410
catboost_12chunks_all0.2240.385
ecfp6_mlp_all0.2180.359
mhfp_mlp_all0.1930.307
\n

Best single model was a Roberta with join MLM + MTR pretraining, scored 0.301 on Private Leaderboard and 0.399 on Public Leaderboard.

\n

Simple weighted ensemble with heuristic weights scores PB=27.8 and LB=44.8\nIndeed there are some single model with higher PB scores, but ensemble reduce variance and give more stable/trustworthy results. This result is luckily enough for a gold medal.

\n

Code

\n\n

Thanks for your attention !

", "rawMarkdown": "First, I would like to thank Kaggle and competition's host for another interesting challenge, which bring me headaches for a long time :D\nThank you to all participants, especially who actively discuss and provide many good insights/code in the forum. As usual on Kaggle, I have a great time learning new things.\n\n## First impression\n- I trained an Embedding + MLP models on building block id only, e.g [BB1, BB2, BB3] = [100, 200, 300]. Yes, it just see the ID. This model scores shared_AP=65.5 on StratifiedKFold(n_splits=20), which indeed surpass some of my SMILES+CNN1d without looking for what each building block looks like. This strongly indicate that model could memorize target binding and overfiting is nearby.\n- Many attempts but actually I failed to setup a trustworthy CV scheme. In such situation, I decided to be blind at all and try to focus the fancy word \"diversity\": Multiple models, multiple input representations and SSL pretraining. I think that strategy indeed survived me in this shuffle competition.\n\n\n## How to Cross-Validate?\nAs mentioned above, I don't know.\n\nMy split strategy for each fold:\n\n 1. [11k samples] Hold out a number of building blocks for validation: 17 BB1 + 36 BB2 with positive-fraction-aware balance on each fold.\n 2. [78k samples] Leave 20% molecules with most regular scaffolds (> 6116 mols/scaffold) for training, then do a Scaffold Split on the remaining 80% molecules\n 3. [103k samples] Stratified Random split on the remaining\nTotal: 11k non-shared + 181k share, simulate the LB\nThis strategy create a little \"harder\" than pure random split on the shared part.\n\nThe hard part lie in 11k non-shared. Some observation on non-shared CV:\n- CV scores vary largely between different folds. I find it hard to identify a trend/correlation, or which is work/not work\n- Early stopping should help, but it also hard to answer WHEN? Usually best score is found in first 2 epochs, but also could be much longer\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fa266b63c8b17261dc0768365c16b09ab%2Fnonshare-cv.png?generation=1720515545119485&alt=media)\n *Each column in form: `score (best_epoch)`. All results belong to a single split, with nonshare positive rate `[0.571, 0.638, 0.714] %`*\n\n- Suitable hyperparams also vary largely between folds. It looks like best setting with best score in fold A could give worst score on fold B :(\n- For one particular fold: score also change significantly due to just small change in hyperparams like random seed, validate interval, .etc. (13.5 -> 32.4, what?)\n- Overfit a model on LB-nonshared-similar building blocks improve nonshared LB score. `Building block ECFP6 -> Tanimoto similarity -> Linear Assignment Matching` to select most similar/disimilar 170 BB1 + 360 BB2/3, then overfit a 1D-CNN on this subset. LB score for similar=8.8, disimilar=3.9. This indicate that we can improve LB by focusing more on these similar samples. I later select most similar 17 BB1 + 36 BB2/3 as validation set and try to improve CV on this split, but improved CV led to decrease in non-shared LB, which confused me\n\nIn conclusion, more and more experiments messed up my mind. The variation is too large and seem to be random for me. I gave up and focus on **SSL pretraining**, improve the diversity by using **multiple models** and **multiple input representations**, and allmost all models were **blindly train on all competition data with no cross-validation**, since I did not trust a single CV split alone and train on multiple CV splits seem to be too time-intensive\n\n\n## SSL Pretraining\n2 pretraining tasks:\n- MLM (Masked Language Modeling) with standard setting: 15% masked tokens, 80%-10%-10% replaced with `[MASK]`/random tokens/keep unchanged.\n- MTR (Multi-Task Regression): regress 189 pre-calculated RDKIT Descriptors. These target values are min-max normalized in contrast to CDF Transform in literature.\n\n`Joint MTR + MLM pretraining` + `SMILES Enumeration` has success as mentioned in [MolBERT](https://github.com/BenevolentAI/MolBERT), which inspired me to give it a try\n\nPretraining using all data (both train + test), with larger sampling weights to test dataset to equalize the frequency of each building blocks and be \"more familiar\" with new domain test dataset.\n\nI trained 3 models to be finetuned on competition task:\n- Squeezeformer + `MTR`\n- Squeezeformer + `Joint MTR + MLM`\n- Roberta + `Joint MTR + MLM`\n \n\n## Modeling\n\nAll models simultaneously predict 3 targets. Some models are not converged in the last day, and training must be terminated to finish in time:\n\n- **Molecule Fingerprint + MLP**\n + Fingerprints: ECFP6, Topological Torsion, MHFP\n + MLP with hidden channels [1024, 1024], dropout = 0.3\n + Tried KAN but did not outperform MLP\n + [Independent Feature Matching (IFM)](https://openreview.net/forum?id=NLFqlDeuzt) gave no boost\n + Feature Selection based on SHAP also gave no boost. For a long time I had believe Feature Selection is the key to reduce share-nonshare gap and overfitting caused by noisy signal, but had no success with it.\n\n- **String-based 1D-CNN**\n + Input representations: SMILES, Atom-In-Smiles (AIS), SELFIES, DeepSMILES\n + Model: Improved from the [awesome public notebook](https://www.kaggle.com/code/ahmedelfazouan/belka-1dcnn-starter-with-all-data), just add MaskedBatchNorm, MaskedAttentionPooling and scale to larger size (depth = 6, dim=128)\n + All input representations give equally good result on shared part. SELFIES give slightly worse results on non-shared and converge slower than other input representations\n + Deeper -> better generalisation. `(depth=8, dim=64)` is `>` `(depth=3, dim=512)` on non-shared, but `<` on shared. `(depth=3, dim=512)` also better remember the training set (higher train AP) while archive similar share AP. This need more experiments to confirm.\n\n- **String-based Hybrid CNN-Transformer**\n + Input representation: SMILES, Atom-In-Smiles\n + Model: Squeezeformer, depth = 6, dim=64/96/128\n + No positional encoding, CNN do the job. ROPE decrease CV score\n + Pretrain: `No pretrain` or `MTR` or `MLM+MTR`\n\n- **String-based Transformer**\n + Input representation: SMILES\n + Model: Roberta, depth=6, dim=256, Absolute PE\n + Pretrain: `MLM+MTR`\n\n- **String-based MAMBA**\n + Input representation: SMILES\n + Model: Mamba, depth=8, dim=128\n- **GNNs**\n + GIN with depth=5, dim=300, finetune from public pretrained [Mole-BERT](https://github.com/junxia97/Mole-BERT) (pretrained on ZINC)\n + GCN/GraphSAGE with depth=5, dim=128\n- **Catboost**\n + Ensemble of 12 folds, covering all train dataset. Each fold contain all positive samples and 5.0 times of negative samples.\n + Max_depth=10, lr=0.2, iterations=4000, bootstrap_type='No'\n\nSome modeling results used in final ensemble. I think analyze share/nonshared/new library scores is needed for further analysis, and I will update that results later.\n\n\n| File Name | Private Leaderboard | Public Leaderboard |\n| --- | --- | --- |\n| `final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808_all` | **0.301** | 0.399 |\n| `final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412_all` | 0.290 | 0.373 |\n| `ensemble_v2` | ***0.278*** | ***0.448*** |\n| `ensemble_v1` | 0.277 | 0.435 |\n| `ensemble_v0` | 0.274 | 0.438 |\n| `final_smileschar-pregen_cnn1d-8-64_all` | 0.270 | 0.389 |\n| `final_smileschar-pregen_cnn1d-3-512_all` | 0.257 | 0.386 |\n| `final_selfies-pregen_cnn1d-3-512_all` | 0.253 | 0.367 |\n| `final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898_all` | 0.251 | 0.407 |\n| `final_selfies-pregen_cnn1d-6-128_all` | 0.244 | 0.365 |\n| `final_ais-pregen_cnn1d-6-128_all` | 0.244 | 0.395 |\n| `final_sage_all` | 0.242 | **0.433** |\n| `final_deepsmiles-pregen_cnn1d-6-128_all` | 0.242 | 0.388 |\n| `final_pretrain-gin_all` | 0.240 | 0.385 |\n| `topo_mlp_all` | 0.236 | 0.377 |\n| `final_smileschar-pregen_cnn1d-6-128_all` | 0.235 | 0.392 |\n| `final_ais_squeeze_all` | 0.235 | 0.388 |\n| `final_gcn_all` | 0.233 | 0.396 |\n| `final_mamba_smileschar_all` | 0.233 | 0.394 |\n| `squeezeformer_0.1033_0.6376_all` | 0.230 | 0.410 |\n| `catboost_12chunks_all` | 0.224 | 0.385 |\n| `ecfp6_mlp_all` | 0.218 | 0.359 |\n| `mhfp_mlp_all` | 0.193 | 0.307 |\n\n\n\nBest single model was a **Roberta with join MLM + MTR pretraining**, scored **0.301** on Private Leaderboard and 0.399 on Public Leaderboard.\n\nSimple weighted ensemble with heuristic weights scores **PB=27.8 and LB=44.8**\nIndeed there are some single model with higher PB scores, but ensemble reduce variance and give more stable/trustworthy results. This result is luckily enough for a gold medal.\n\n## Code\n- Training code: https://github.com/dangnh0611/kaggle_leash_belka\n\nThanks for your attention !", "votes": null }, { "id": "2913666", "postDate": "07/09/2024 15:42:32", "content": "

Great to see so many models! Congrats!

", "rawMarkdown": "Great to see so many models! Congrats!", "votes": null }, { "id": "2914572", "postDate": "07/10/2024 03:42:26", "content": "

Thank you for sharing! I'm really impressed by the SSL pretraining approach! I'm looking forward to seeing your code :)

\n

Also, if it's okay with you, can I ask which models you used for the ensemble? Did you only use the finetuned three models, or were there more?

", "rawMarkdown": "Thank you for sharing! I'm really impressed by the SSL pretraining approach! I'm looking forward to seeing your code :)\n\nAlso, if it's okay with you, can I ask which models you used for the ensemble? Did you only use the finetuned three models, or were there more?", "votes": null }, { "id": "2914788", "postDate": "07/10/2024 06:54:05", "content": "

Thank you! The code is released. There are more than three models in PB=27.8 ensemble, with different weights for share/non-share groups:

\n
SHARE = [\n    \n    \n    \n    \n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    \n    \n    \n    (, ),\n    \n    \n    (, ),\n    (, ),\n    \n    \n    \n]\n\nNONSHARE = [\n    \n    \n    \n    \n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, )\n]\n
", "rawMarkdown": "Thank you! The code is [released](https://github.com/dangnh0611/kaggle_leash_belka). There are more than three models in PB=27.8 ensemble, with different weights for share/non-share groups:\n \n```python\nSHARE = [\n # ('catboost_12chunks', 0.0),\n # ('ecfp6_mlp', 0.0),\n # ('topo_mlp', 0.0),\n # ('mhfp_mlp', 0.0),\n ('final_smileschar-pregen_cnn1d-6-128', 1),\n ('final_ais-pregen_cnn1d-6-128', 1),\n ('final_selfies-pregen_cnn1d-6-128', 1),\n ('final_deepsmiles-pregen_cnn1d-6-128', 1),\n # ('final_smileschar-pregen_cnn1d-8-64', 0.0),\n # ('final_smileschar-pregen_cnn1d-3-512', 0.0),\n # ('final_selfies-pregen_cnn1d-3-512', 0.0),\n ('final_ais_squeeze', 1.0),\n # ('final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412', 0.0),\n # ('final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898', 0.0),\n ('final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808', 1.0),\n ('final_mamba_smileschar', 1.0),\n # ('final_pretrain-gin', 0.0),\n # ('final_sage', 0.0),\n # ('final_gcn', 0.0)\n]\n\nNONSHARE = [\n # ('catboost_12chunks', 0.0),\n # ('ecfp6_mlp', 0.0),\n # ('topo_mlp', 0.0),\n # ('mhfp_mlp', 0.0),\n ('final_smileschar-pregen_cnn1d-6-128', 0.5),\n ('final_ais-pregen_cnn1d-6-128', 0.5),\n ('final_selfies-pregen_cnn1d-6-128', 0.5),\n ('final_deepsmiles-pregen_cnn1d-6-128', 0.5),\n ('final_smileschar-pregen_cnn1d-8-64', 0.2),\n ('final_smileschar-pregen_cnn1d-3-512', 0.2),\n ('final_selfies-pregen_cnn1d-3-512', 0.2),\n ('squeeze_first_version', 3.0),\n ('final_ais_squeeze', 1.0),\n ('final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412', 0.8),\n ('final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898', 0.8),\n ('final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808', 1.0),\n ('final_mamba_smileschar', 1.0),\n ('final_pretrain-gin', 0.5),\n ('final_sage', 0.1),\n ('final_gcn', 0.1)\n]\n```", "votes": null } ], "comments": [ { "id": 2913666, "author_name": "lililycai", "author_url": "", "post_date": "07/09/2024 15:42:32", "content": "

Great to see so many models! Congrats!

", "votes": null, "replies": [] }, { "id": 2914572, "author_name": "kyu999", "author_url": "", "post_date": "07/10/2024 03:42:26", "content": "

Thank you for sharing! I'm really impressed by the SSL pretraining approach! I'm looking forward to seeing your code :)

\n

Also, if it's okay with you, can I ask which models you used for the ensemble? Did you only use the finetuned three models, or were there more?

", "votes": null, "replies": [ { "id": 2914788, "author_name": "dangnh0611", "author_url": "", "post_date": "07/10/2024 06:54:05", "content": "

Thank you! The code is released. There are more than three models in PB=27.8 ensemble, with different weights for share/non-share groups:

\n
SHARE = [\n    \n    \n    \n    \n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    \n    \n    \n    (, ),\n    \n    \n    (, ),\n    (, ),\n    \n    \n    \n]\n\nNONSHARE = [\n    \n    \n    \n    \n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, ),\n    (, )\n]\n
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2913113": "First, I would like to thank Kaggle and competition's host for another interesting challenge, which bring me headaches for a long time :D\nThank you to all participants, especially who actively discuss and provide many good insights/code in the forum. As usual on Kaggle, I have a great time learning new things.\n\n## First impression\n- I trained an Embedding + MLP models on building block id only, e.g [BB1, BB2, BB3] = [100, 200, 300]. Yes, it just see the ID. This model scores shared_AP=65.5 on StratifiedKFold(n_splits=20), which indeed surpass some of my SMILES+CNN1d without looking for what each building block looks like. This strongly indicate that model could memorize target binding and overfiting is nearby.\n- Many attempts but actually I failed to setup a trustworthy CV scheme. In such situation, I decided to be blind at all and try to focus the fancy word \"diversity\": Multiple models, multiple input representations and SSL pretraining. I think that strategy indeed survived me in this shuffle competition.\n\n\n## How to Cross-Validate?\nAs mentioned above, I don't know.\n\nMy split strategy for each fold:\n\n 1. [11k samples] Hold out a number of building blocks for validation: 17 BB1 + 36 BB2 with positive-fraction-aware balance on each fold.\n 2. [78k samples] Leave 20% molecules with most regular scaffolds (> 6116 mols/scaffold) for training, then do a Scaffold Split on the remaining 80% molecules\n 3. [103k samples] Stratified Random split on the remaining\nTotal: 11k non-shared + 181k share, simulate the LB\nThis strategy create a little \"harder\" than pure random split on the shared part.\n\nThe hard part lie in 11k non-shared. Some observation on non-shared CV:\n- CV scores vary largely between different folds. I find it hard to identify a trend/correlation, or which is work/not work\n- Early stopping should help, but it also hard to answer WHEN? Usually best score is found in first 2 epochs, but also could be much longer\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fa266b63c8b17261dc0768365c16b09ab%2Fnonshare-cv.png?generation=1720515545119485&alt=media)\n *Each column in form: `score (best_epoch)`. All results belong to a single split, with nonshare positive rate `[0.571, 0.638, 0.714] %`*\n\n- Suitable hyperparams also vary largely between folds. It looks like best setting with best score in fold A could give worst score on fold B :(\n- For one particular fold: score also change significantly due to just small change in hyperparams like random seed, validate interval, .etc. (13.5 -> 32.4, what?)\n- Overfit a model on LB-nonshared-similar building blocks improve nonshared LB score. `Building block ECFP6 -> Tanimoto similarity -> Linear Assignment Matching` to select most similar/disimilar 170 BB1 + 360 BB2/3, then overfit a 1D-CNN on this subset. LB score for similar=8.8, disimilar=3.9. This indicate that we can improve LB by focusing more on these similar samples. I later select most similar 17 BB1 + 36 BB2/3 as validation set and try to improve CV on this split, but improved CV led to decrease in non-shared LB, which confused me\n\nIn conclusion, more and more experiments messed up my mind. The variation is too large and seem to be random for me. I gave up and focus on **SSL pretraining**, improve the diversity by using **multiple models** and **multiple input representations**, and allmost all models were **blindly train on all competition data with no cross-validation**, since I did not trust a single CV split alone and train on multiple CV splits seem to be too time-intensive\n\n\n## SSL Pretraining\n2 pretraining tasks:\n- MLM (Masked Language Modeling) with standard setting: 15% masked tokens, 80%-10%-10% replaced with `[MASK]`/random tokens/keep unchanged.\n- MTR (Multi-Task Regression): regress 189 pre-calculated RDKIT Descriptors. These target values are min-max normalized in contrast to CDF Transform in literature.\n\n`Joint MTR + MLM pretraining` + `SMILES Enumeration` has success as mentioned in [MolBERT](https://github.com/BenevolentAI/MolBERT), which inspired me to give it a try\n\nPretraining using all data (both train + test), with larger sampling weights to test dataset to equalize the frequency of each building blocks and be \"more familiar\" with new domain test dataset.\n\nI trained 3 models to be finetuned on competition task:\n- Squeezeformer + `MTR`\n- Squeezeformer + `Joint MTR + MLM`\n- Roberta + `Joint MTR + MLM`\n \n\n## Modeling\n\nAll models simultaneously predict 3 targets. Some models are not converged in the last day, and training must be terminated to finish in time:\n\n- **Molecule Fingerprint + MLP**\n + Fingerprints: ECFP6, Topological Torsion, MHFP\n + MLP with hidden channels [1024, 1024], dropout = 0.3\n + Tried KAN but did not outperform MLP\n + [Independent Feature Matching (IFM)](https://openreview.net/forum?id=NLFqlDeuzt) gave no boost\n + Feature Selection based on SHAP also gave no boost. For a long time I had believe Feature Selection is the key to reduce share-nonshare gap and overfitting caused by noisy signal, but had no success with it.\n\n- **String-based 1D-CNN**\n + Input representations: SMILES, Atom-In-Smiles (AIS), SELFIES, DeepSMILES\n + Model: Improved from the [awesome public notebook](https://www.kaggle.com/code/ahmedelfazouan/belka-1dcnn-starter-with-all-data), just add MaskedBatchNorm, MaskedAttentionPooling and scale to larger size (depth = 6, dim=128)\n + All input representations give equally good result on shared part. SELFIES give slightly worse results on non-shared and converge slower than other input representations\n + Deeper -> better generalisation. `(depth=8, dim=64)` is `>` `(depth=3, dim=512)` on non-shared, but `<` on shared. `(depth=3, dim=512)` also better remember the training set (higher train AP) while archive similar share AP. This need more experiments to confirm.\n\n- **String-based Hybrid CNN-Transformer**\n + Input representation: SMILES, Atom-In-Smiles\n + Model: Squeezeformer, depth = 6, dim=64/96/128\n + No positional encoding, CNN do the job. ROPE decrease CV score\n + Pretrain: `No pretrain` or `MTR` or `MLM+MTR`\n\n- **String-based Transformer**\n + Input representation: SMILES\n + Model: Roberta, depth=6, dim=256, Absolute PE\n + Pretrain: `MLM+MTR`\n\n- **String-based MAMBA**\n + Input representation: SMILES\n + Model: Mamba, depth=8, dim=128\n- **GNNs**\n + GIN with depth=5, dim=300, finetune from public pretrained [Mole-BERT](https://github.com/junxia97/Mole-BERT) (pretrained on ZINC)\n + GCN/GraphSAGE with depth=5, dim=128\n- **Catboost**\n + Ensemble of 12 folds, covering all train dataset. Each fold contain all positive samples and 5.0 times of negative samples.\n + Max_depth=10, lr=0.2, iterations=4000, bootstrap_type='No'\n\nSome modeling results used in final ensemble. I think analyze share/nonshared/new library scores is needed for further analysis, and I will update that results later.\n\n\n| File Name | Private Leaderboard | Public Leaderboard |\n| --- | --- | --- |\n| `final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808_all` | **0.301** | 0.399 |\n| `final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412_all` | 0.290 | 0.373 |\n| `ensemble_v2` | ***0.278*** | ***0.448*** |\n| `ensemble_v1` | 0.277 | 0.435 |\n| `ensemble_v0` | 0.274 | 0.438 |\n| `final_smileschar-pregen_cnn1d-8-64_all` | 0.270 | 0.389 |\n| `final_smileschar-pregen_cnn1d-3-512_all` | 0.257 | 0.386 |\n| `final_selfies-pregen_cnn1d-3-512_all` | 0.253 | 0.367 |\n| `final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898_all` | 0.251 | 0.407 |\n| `final_selfies-pregen_cnn1d-6-128_all` | 0.244 | 0.365 |\n| `final_ais-pregen_cnn1d-6-128_all` | 0.244 | 0.395 |\n| `final_sage_all` | 0.242 | **0.433** |\n| `final_deepsmiles-pregen_cnn1d-6-128_all` | 0.242 | 0.388 |\n| `final_pretrain-gin_all` | 0.240 | 0.385 |\n| `topo_mlp_all` | 0.236 | 0.377 |\n| `final_smileschar-pregen_cnn1d-6-128_all` | 0.235 | 0.392 |\n| `final_ais_squeeze_all` | 0.235 | 0.388 |\n| `final_gcn_all` | 0.233 | 0.396 |\n| `final_mamba_smileschar_all` | 0.233 | 0.394 |\n| `squeezeformer_0.1033_0.6376_all` | 0.230 | 0.410 |\n| `catboost_12chunks_all` | 0.224 | 0.385 |\n| `ecfp6_mlp_all` | 0.218 | 0.359 |\n| `mhfp_mlp_all` | 0.193 | 0.307 |\n\n\n\nBest single model was a **Roberta with join MLM + MTR pretraining**, scored **0.301** on Private Leaderboard and 0.399 on Public Leaderboard.\n\nSimple weighted ensemble with heuristic weights scores **PB=27.8 and LB=44.8**\nIndeed there are some single model with higher PB scores, but ensemble reduce variance and give more stable/trustworthy results. This result is luckily enough for a gold medal.\n\n## Code\n- Training code: https://github.com/dangnh0611/kaggle_leash_belka\n\nThanks for your attention !", "2913666": "Great to see so many models! Congrats!", "2914572": "Thank you for sharing! I'm really impressed by the SSL pretraining approach! I'm looking forward to seeing your code :)\n\nAlso, if it's okay with you, can I ask which models you used for the ensemble? Did you only use the finetuned three models, or were there more?", "2914788": "Thank you! The code is [released](https://github.com/dangnh0611/kaggle_leash_belka). There are more than three models in PB=27.8 ensemble, with different weights for share/non-share groups:\n \n```python\nSHARE = [\n # ('catboost_12chunks', 0.0),\n # ('ecfp6_mlp', 0.0),\n # ('topo_mlp', 0.0),\n # ('mhfp_mlp', 0.0),\n ('final_smileschar-pregen_cnn1d-6-128', 1),\n ('final_ais-pregen_cnn1d-6-128', 1),\n ('final_selfies-pregen_cnn1d-6-128', 1),\n ('final_deepsmiles-pregen_cnn1d-6-128', 1),\n # ('final_smileschar-pregen_cnn1d-8-64', 0.0),\n # ('final_smileschar-pregen_cnn1d-3-512', 0.0),\n # ('final_selfies-pregen_cnn1d-3-512', 0.0),\n ('final_ais_squeeze', 1.0),\n # ('final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412', 0.0),\n # ('final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898', 0.0),\n ('final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808', 1.0),\n ('final_mamba_smileschar', 1.0),\n # ('final_pretrain-gin', 0.0),\n # ('final_sage', 0.0),\n # ('final_gcn', 0.0)\n]\n\nNONSHARE = [\n # ('catboost_12chunks', 0.0),\n # ('ecfp6_mlp', 0.0),\n # ('topo_mlp', 0.0),\n # ('mhfp_mlp', 0.0),\n ('final_smileschar-pregen_cnn1d-6-128', 0.5),\n ('final_ais-pregen_cnn1d-6-128', 0.5),\n ('final_selfies-pregen_cnn1d-6-128', 0.5),\n ('final_deepsmiles-pregen_cnn1d-6-128', 0.5),\n ('final_smileschar-pregen_cnn1d-8-64', 0.2),\n ('final_smileschar-pregen_cnn1d-3-512', 0.2),\n ('final_selfies-pregen_cnn1d-3-512', 0.2),\n ('squeeze_first_version', 3.0),\n ('final_ais_squeeze', 1.0),\n ('final_squeeze-mtr_chartokenize-02_ep11_0.033802_0.651412', 0.8),\n ('final_squeeze-mtr-mlm_chartokenize-02_ep11_0.11199_0.648898', 0.8),\n ('final_roberta-mtr-mlm_chartokenize-all_ep11.2_na_0.668808', 1.0),\n ('final_mamba_smileschar', 1.0),\n ('final_pretrain-gin', 0.5),\n ('final_sage', 0.1),\n ('final_gcn', 0.1)\n]\n```" }, "source": "meta" }