![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1328185%2Fa57a0fd2ce86c30dd3b0345958eb2c4f%2FScreenshot%202024-04-30%20at%2009.46.49.png?generation=1714463680621641&alt=media\")
Please share your CV/LB results here.
\nI'm using my CV split, here are most recent results:
\n| Model | \nCV-random | \nCV-noshare | \nLB | \n
|---|---|---|---|
| Molformer | \n0.64 | \n0.008 | \n0.564 | \n
| Molformer | \n0.62 | \n0.017 | \n0.554 | \n
From other discussions and my initial results it's likely that public LB seems correlated with random split, but private results will most likely depend on how well you can model the no-share / test distribution.
\nBelow more details by protein target (if you're not using W&B for tracking, you're missing out):
\n
Please share your CV/LB results here.
\nI'm using my CV split, here are most recent results:
\n| Model | \nCV-random | \nCV-noshare | \nLB | \n
|---|---|---|---|
| Molformer | \n0.64 | \n0.008 | \n0.564 | \n
| Molformer | \n0.62 | \n0.017 | \n0.554 | \n
From other discussions and my initial results it's likely that public LB seems correlated with random split, but private results will most likely depend on how well you can model the no-share / test distribution.
\nBelow more details by protein target (if you're not using W&B for tracking, you're missing out):
\n
My CV LB split is very split dependent. The more samples I add to my experiments the lower my CV (using scaffold splitting) but higher LB. I'd put some examples here, but I am still experimenting.
", "rawMarkdown": "My CV LB split is very split dependent. The more samples I add to my experiments the lower my CV (using scaffold splitting) but higher LB. I'd put some examples here, but I am still experimenting.", "votes": null }, { "id": "2785685", "postDate": "05/01/2024 01:45:03", "content": "the best split is probably random, see also paper.
\ni suggest 2 separate splits and experiments
my strategy:
\n[1] cv score versus num of share blocks
[2] cv score for share blocks
\npaper on cv split, datasize, etc:
\nUncovering Neural Scaling Laws in Molecular Representation Learning
\nhttps://arxiv.org/abs/2309.15123
\"I'm using my CV split, here are most recent results:\"
\nboth molformer and chemberta should be about to give you lb 0.575. (CV 0.670), i am using random splits
\nI think this is the limit of SMILES string transformer?
\n(i think some kaggler can get up to lb0.605(CV 0.700))?
molformer = 12 layers
\nchemberta = 3 layers
different pretrain chemberta 5,10,77M MLM,MTR has slight different results
\nNote there a 2 different molformer in papers (one is based on SMILES string , another is a GNN)
", "rawMarkdown": "\"I'm using my CV split, here are most recent results:\"\n\nboth molformer and chemberta should be about to give you lb 0.575. (CV 0.670), i am using random splits\nI think this is the limit of SMILES string transformer?\n(i think some kaggler can get up to lb0.605(CV 0.700))?\n\nmolformer = 12 layers\nchemberta = 3 layers\n\ndifferent pretrain chemberta 5,10,77M MLM,MTR has slight different results\n\nNote there a 2 different molformer in papers (one is based on SMILES string , another is a GNN)", "votes": null }, { "id": "2786617", "postDate": "05/01/2024 11:27:41", "content": "I agree there will likely need to be ways to handle compounds close to the train set and those that aren't. I have 64gb of RAM at home and it seems that even with np.packbits I can't train models with half of the train rows. So, now I'm focused on finding a subsampling strategy. I'm learning a lot about how to manage large datasets in this competition.
", "rawMarkdown": "I agree there will likely need to be ways to handle compounds close to the train set and those that aren't. I have 64gb of RAM at home and it seems that even with np.packbits I can't train models with half of the train rows. So, now I'm focused on finding a subsampling strategy. I'm learning a lot about how to manage large datasets in this competition.", "votes": null }, { "id": "2786912", "postDate": "05/01/2024 13:47:12", "content": "actually i am think that for those kagglers that shown good performance, espeically those with good domain knowledge, google should get them better compute power, e.g. discount price at google cloud for better gpu, etc
\nor kaggler ourselves can setup some fund.
\nif the solution is good, then we can all learn something valuable, rather than trial and error like a headless fly
CV 0.69, LB 0.585
\nmodel : 1dcnn trained on all data
\nI made this model public : https://www.kaggle.com/ahmedelfazouan/belka-1dcnn-starter-with-all-data
This is pretty cool @ahmedelfazouan ! Looking forward to reading through it.
", "rawMarkdown": "This is pretty cool @ahmedelfazouan ! Looking forward to reading through it.", "votes": null }, { "id": "2787630", "postDate": "05/01/2024 21:22:27", "content": "I've considered renting more compute power, but I do think exploring a subsampling strategy is a good learning experience for me.
", "rawMarkdown": "I've considered renting more compute power, but I do think exploring a subsampling strategy is a good learning experience for me.", "votes": null }, { "id": "2788308", "postDate": "05/02/2024 07:17:13", "content": "CV: 0.597, 0.600, 0.608, 0.613
\nLB: 0.575, 0.586, 0.589, 0.585
Model: small nn on top of frozen molformer
\nVal Split: positive ratio 1% (random, len 1e5)
your cv looks low.
\ni am not sure if it is the effect of different split or frozen (i think frozen is more likely)
the good news is that CV and LB gap is very small, which means that you actually score very well in nonshare b block (you can confirm by local experiment or separate submission)
\nand i think if you starts unfreeze, you will start to have different results.
\nyou can try ensemble of freeze and frozen models, or using freeze and frozen models for share and unshare blocks
i suddenly have an idea.
\nhow about LORA like model?
\nuse some layers to modify molformer intermediate or/and final layer output?
I think there are still some lower hanging fruits that can be picked. These models were trained (with 4090*1) in around 5-7 hours with a subset of (neg) train, only iterating over the train set about twice.
\nBefore I set the fixed positive ratio val, a different random split/sample had cv around 0.62, so yeah this split is lower.
\nI don't think I'll unfreeze molformer just yet as that adds a massive amount of compute cost and will destroy my batch size.
", "rawMarkdown": "I think there are still some lower hanging fruits that can be picked. These models were trained (with 4090*1) in around 5-7 hours with a subset of (neg) train, only iterating over the train set about twice.\n\nBefore I set the fixed positive ratio val, a different random split/sample had cv around 0.62, so yeah this split is lower.\n\nI don't think I'll unfreeze molformer just yet as that adds a massive amount of compute cost and will destroy my batch size.", "votes": null }, { "id": "2788934", "postDate": "05/02/2024 13:37:36", "content": "Hmmm will have to look into loras, definitely seems like a possible path forward.
", "rawMarkdown": "Hmmm will have to look into loras, definitely seems like a possible path forward.", "votes": null }, { "id": "2790293", "postDate": "05/03/2024 05:40:06", "content": "i load my data, and sample 1024 * 1024 molecules, with X% positive samples, and save it to a parquet file. I do this 100 times. For training i load one file at a time to save ram. You could also have a separate file for just postives , and multiple files for negative if you want to change +ve/-ve split on the fly.
", "rawMarkdown": "i load my data, and sample 1024 * 1024 molecules, with X% positive samples, and save it to a parquet file. I do this 100 times. For training i load one file at a time to save ram. You could also have a separate file for just postives , and multiple files for negative if you want to change +ve/-ve split on the fly.", "votes": null }, { "id": "2790563", "postDate": "05/03/2024 08:31:35", "content": "molformer works slightly better for me.
\nWith Molformer i get .69 CV, .583 lb.
\nWith chemberta i get .65 CV, .56 lb.
i tried using Selformer, ChemGPT (selfies models) but got much worse scores.
", "rawMarkdown": "molformer works slightly better for me.\nWith Molformer i get .69 CV, .583 lb.\nWith chemberta i get .65 CV, .56 lb. \n\ni tried using Selformer, ChemGPT (selfies models) but got much worse scores.", "votes": null }, { "id": "2790810", "postDate": "05/03/2024 10:51:02", "content": "Using a variation on @ahmedelfazouan notebook, I get the following CV-LB scores.
\n0.72 - 0.61
\n0.71 - 0.6
\n0.65 - 0.57
I've only tried a few simple things here, but I am really surprised how well a 1D-CNN handles the in-distribution samples in such a simple yet elegant way. I really learned something from this notebook.
", "rawMarkdown": "Using a variation on @ahmedelfazouan notebook, I get the following CV-LB scores.\n\n0.72 - 0.61\n0.71 - 0.6\n0.65 - 0.57\n\nI've only tried a few simple things here, but I am really surprised how well a 1D-CNN handles the in-distribution samples in such a simple yet elegant way. I really learned something from this notebook.", "votes": null }, { "id": "2790830", "postDate": "05/03/2024 10:58:04", "content": "Nice work @chemdatafarmer !
", "rawMarkdown": "Nice work @chemdatafarmer !", "votes": null }, { "id": "2790833", "postDate": "05/03/2024 10:58:56", "content": "It's very much thanks to learning from you, so thank you!
", "rawMarkdown": "It's very much thanks to learning from you, so thank you!", "votes": null }, { "id": "2791018", "postDate": "05/03/2024 12:44:31", "content": "![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fabe5677ac9f32a81dfa7bdc35868e0db%2FSelection_061.png?generation=1714740269070323&alt=media\")
some observation:
\n@hengck23 my best model uses batch norm, though I haven't systematically studied the impact of taking it out
", "rawMarkdown": "hengck23 my best model uses batch norm, though I haven't systematically studied the impact of taking it out", "votes": null }, { "id": "2796173", "postDate": "05/06/2024 06:23:41", "content": "\"best model uses batch norm, \"
\ni think the reason for my poorer performance could be this
\nhttps://medium.com/deeplearningmadeeasy/ghost-batchnorm-explained-e0fa9d651e03
\" It seems neural networks tends to do worse for unseen data when being trained on large batch sizes.\"
\nif you are using 8xTPU, you don't see this problem because data is split into 8 chunks and bn is applied independently to each chunk (i.e natural ghost batchnorm)
\nit is a problem for me because i am using single gpu.
\nNow i change bn to ghost bn and it seems to work better
\nfor simple 3 layer cnn1d, it doesn't matter if you use batch norm or not. But if you use deep layer, or shalow/deep layer for rnn, trasnformer, then it is important for normalisation because of the use in exp family function in activation (e.g. tahn,sigmoid in rnn or exp in softmax in attnetion)
", "rawMarkdown": "\"best model uses batch norm, \"\n\ni think the reason for my poorer performance could be this\nhttps://medium.com/deeplearningmadeeasy/ghost-batchnorm-explained-e0fa9d651e03\n\n\" It seems neural networks tends to do worse for unseen data when being trained on large batch sizes.\"\n\nif you are using 8xTPU, you don't see this problem because data is split into 8 chunks and bn is applied independently to each chunk (i.e natural ghost batchnorm)\n\nit is a problem for me because i am using single gpu.\n\nNow i change bn to ghost bn and it seems to work better\n\n---\n\nfor simple 3 layer cnn1d, it doesn't matter if you use batch norm or not. But if you use deep layer, or shalow/deep layer for rnn, trasnformer, then it is important for normalisation because of the use in exp family function in activation (e.g. tahn,sigmoid in rnn or exp in softmax in attnetion)", "votes": null }, { "id": "2796700", "postDate": "05/06/2024 11:31:23", "content": "Ah, yes, I didn't appreciate the differences between GPU and TPU here. Indeed, my batch norms were added when I tried to make the network deeper.
", "rawMarkdown": "Ah, yes, I didn't appreciate the differences between GPU and TPU here. Indeed, my batch norms were added when I tried to make the network deeper.", "votes": null }, { "id": "2799055", "postDate": "05/07/2024 15:17:44", "content": "| CV-random | \nCV-noshare | \n
|---|---|
| 0.664 | \n0.022 | \n
I still have not submitted it because I am currently toying around (or rather, fighting) with RDKit for molecule representation within a 3d space, because I think that it would be quite informative to have
", "rawMarkdown": "| CV-random | CV-noshare |\n| --- | --- |\n| 0.664 | 0.022 |\n\n\nI still have not submitted it because I am currently toying around (or rather, fighting) with RDKit for molecule representation within a 3d space, because I think that it would be quite informative to have", "votes": null }, { "id": "2804645", "postDate": "05/10/2024 06:21:09", "content": "set all sharing blocks to zero at submission.csv.
\nI get nonshare sore = lb0.055.
\n(actually 0.055 include score of both share(truth distrubution) and nonshare binds(prediction ap)).
if anyone has nonshare score, please put it here.
", "rawMarkdown": "set all sharing blocks to zero at submission.csv.\nI get nonshare sore = lb0.055.\n(actually 0.055 include score of both share(truth distrubution) and nonshare binds(prediction ap)).\n\n\nif anyone has nonshare score, please put it here.", "votes": null }, { "id": "2805029", "postDate": "05/10/2024 10:20:33", "content": "
\nThe new result is quite puzzling, not sure what happened here.
| LB | \nLB (mask noshare) | \nLB (mask share) | \n
|---|---|---|
| 0.629 | \n0.596 | \n0.053 | \n
| 0.663 | \n0.596 | \n0.058 | \n
thanks. your results are probably correct.
\nif we have perfect nonshare prediction, we can get nonshare lb=0.2+
check the following
\nhttps://www.kaggle.com/code/hengck23/how-to-probe
\nhttps://www.kaggle.com/code/junkoda/unknowns-in-public-test-set/notebook
just to double check, i attached \"submit_sharing.zip\" to mask share and nonshare in submit. you may check if it is correct.
", "rawMarkdown": "just to double check, i attached \"submit_sharing.zip\" to mask share and nonshare in submit. you may check if it is correct.", "votes": null }, { "id": "2806324", "postDate": "05/11/2024 03:28:37", "content": "Thank you, I will check it this night.
", "rawMarkdown": "Thank you, I will check it this night.", "votes": null } ], "comments": [ { "id": 2785587, "author_name": "chemdatafarmer", "author_url": "", "post_date": "04/30/2024 22:56:03", "content": "My CV LB split is very split dependent. The more samples I add to my experiments the lower my CV (using scaffold splitting) but higher LB. I'd put some examples here, but I am still experimenting.
", "votes": null, "replies": [ { "id": 2785685, "author_name": "hengck23", "author_url": "", "post_date": "05/01/2024 01:45:03", "content": "the best split is probably random, see also paper.
\ni suggest 2 separate splits and experiments
my strategy:
\n[1] cv score versus num of share blocks
[2] cv score for share blocks
\npaper on cv split, datasize, etc:
\nUncovering Neural Scaling Laws in Molecular Representation Learning
\nhttps://arxiv.org/abs/2309.15123
I agree there will likely need to be ways to handle compounds close to the train set and those that aren't. I have 64gb of RAM at home and it seems that even with np.packbits I can't train models with half of the train rows. So, now I'm focused on finding a subsampling strategy. I'm learning a lot about how to manage large datasets in this competition.
", "votes": null, "replies": [ { "id": 2786912, "author_name": "hengck23", "author_url": "", "post_date": "05/01/2024 13:47:12", "content": "actually i am think that for those kagglers that shown good performance, espeically those with good domain knowledge, google should get them better compute power, e.g. discount price at google cloud for better gpu, etc
\nor kaggler ourselves can setup some fund.
\nif the solution is good, then we can all learn something valuable, rather than trial and error like a headless fly
I've considered renting more compute power, but I do think exploring a subsampling strategy is a good learning experience for me.
", "votes": null, "replies": [ { "id": 2790293, "author_name": "amishm", "author_url": "", "post_date": "05/03/2024 05:40:06", "content": "i load my data, and sample 1024 * 1024 molecules, with X% positive samples, and save it to a parquet file. I do this 100 times. For training i load one file at a time to save ram. You could also have a separate file for just postives , and multiple files for negative if you want to change +ve/-ve split on the fly.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2785690, "author_name": "hengck23", "author_url": "", "post_date": "05/01/2024 01:50:00", "content": "\"I'm using my CV split, here are most recent results:\"
\nboth molformer and chemberta should be about to give you lb 0.575. (CV 0.670), i am using random splits
\nI think this is the limit of SMILES string transformer?
\n(i think some kaggler can get up to lb0.605(CV 0.700))?
molformer = 12 layers
\nchemberta = 3 layers
different pretrain chemberta 5,10,77M MLM,MTR has slight different results
\nNote there a 2 different molformer in papers (one is based on SMILES string , another is a GNN)
", "votes": null, "replies": [ { "id": 2790563, "author_name": "amishm", "author_url": "", "post_date": "05/03/2024 08:31:35", "content": "molformer works slightly better for me.
\nWith Molformer i get .69 CV, .583 lb.
\nWith chemberta i get .65 CV, .56 lb.
i tried using Selformer, ChemGPT (selfies models) but got much worse scores.
", "votes": null, "replies": [] } ] }, { "id": 2787584, "author_name": "ahmedelfazouan", "author_url": "", "post_date": "05/01/2024 20:36:32", "content": "CV 0.69, LB 0.585
\nmodel : 1dcnn trained on all data
\nI made this model public : https://www.kaggle.com/ahmedelfazouan/belka-1dcnn-starter-with-all-data
This is pretty cool @ahmedelfazouan ! Looking forward to reading through it.
", "votes": null, "replies": [ { "id": 2790810, "author_name": "chemdatafarmer", "author_url": "", "post_date": "05/03/2024 10:51:02", "content": "Using a variation on @ahmedelfazouan notebook, I get the following CV-LB scores.
\n0.72 - 0.61
\n0.71 - 0.6
\n0.65 - 0.57
I've only tried a few simple things here, but I am really surprised how well a 1D-CNN handles the in-distribution samples in such a simple yet elegant way. I really learned something from this notebook.
", "votes": null, "replies": [ { "id": 2790830, "author_name": "ahmedelfazouan", "author_url": "", "post_date": "05/03/2024 10:58:04", "content": "Nice work @chemdatafarmer !
", "votes": null, "replies": [ { "id": 2790833, "author_name": "chemdatafarmer", "author_url": "", "post_date": "05/03/2024 10:58:56", "content": "It's very much thanks to learning from you, so thank you!
", "votes": null, "replies": [ { "id": 2791018, "author_name": "hengck23", "author_url": "", "post_date": "05/03/2024 12:44:31", "content": "
some observation:
\n@hengck23 my best model uses batch norm, though I haven't systematically studied the impact of taking it out
", "votes": null, "replies": [ { "id": 2796173, "author_name": "hengck23", "author_url": "", "post_date": "05/06/2024 06:23:41", "content": "\"best model uses batch norm, \"
\ni think the reason for my poorer performance could be this
\nhttps://medium.com/deeplearningmadeeasy/ghost-batchnorm-explained-e0fa9d651e03
\" It seems neural networks tends to do worse for unseen data when being trained on large batch sizes.\"
\nif you are using 8xTPU, you don't see this problem because data is split into 8 chunks and bn is applied independently to each chunk (i.e natural ghost batchnorm)
\nit is a problem for me because i am using single gpu.
\nNow i change bn to ghost bn and it seems to work better
\nfor simple 3 layer cnn1d, it doesn't matter if you use batch norm or not. But if you use deep layer, or shalow/deep layer for rnn, trasnformer, then it is important for normalisation because of the use in exp family function in activation (e.g. tahn,sigmoid in rnn or exp in softmax in attnetion)
", "votes": null, "replies": [ { "id": 2796700, "author_name": "chemdatafarmer", "author_url": "", "post_date": "05/06/2024 11:31:23", "content": "Ah, yes, I didn't appreciate the differences between GPU and TPU here. Indeed, my batch norms were added when I tried to make the network deeper.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2788308, "author_name": "sroger", "author_url": "", "post_date": "05/02/2024 07:17:13", "content": "CV: 0.597, 0.600, 0.608, 0.613
\nLB: 0.575, 0.586, 0.589, 0.585
Model: small nn on top of frozen molformer
\nVal Split: positive ratio 1% (random, len 1e5)
your cv looks low.
\ni am not sure if it is the effect of different split or frozen (i think frozen is more likely)
the good news is that CV and LB gap is very small, which means that you actually score very well in nonshare b block (you can confirm by local experiment or separate submission)
\nand i think if you starts unfreeze, you will start to have different results.
\nyou can try ensemble of freeze and frozen models, or using freeze and frozen models for share and unshare blocks
i suddenly have an idea.
\nhow about LORA like model?
\nuse some layers to modify molformer intermediate or/and final layer output?
Hmmm will have to look into loras, definitely seems like a possible path forward.
", "votes": null, "replies": [] } ] }, { "id": 2788682, "author_name": "sroger", "author_url": "", "post_date": "05/02/2024 10:54:42", "content": "I think there are still some lower hanging fruits that can be picked. These models were trained (with 4090*1) in around 5-7 hours with a subset of (neg) train, only iterating over the train set about twice.
\nBefore I set the fixed positive ratio val, a different random split/sample had cv around 0.62, so yeah this split is lower.
\nI don't think I'll unfreeze molformer just yet as that adds a massive amount of compute cost and will destroy my batch size.
", "votes": null, "replies": [] } ] } ] }, { "id": 2799055, "author_name": "giorgiomicaletto", "author_url": "", "post_date": "05/07/2024 15:17:44", "content": "| CV-random | \nCV-noshare | \n
|---|---|
| 0.664 | \n0.022 | \n
I still have not submitted it because I am currently toying around (or rather, fighting) with RDKit for molecule representation within a 3d space, because I think that it would be quite informative to have
", "votes": null, "replies": [] }, { "id": 2804645, "author_name": "hengck23", "author_url": "", "post_date": "05/10/2024 06:21:09", "content": "set all sharing blocks to zero at submission.csv.
\nI get nonshare sore = lb0.055.
\n(actually 0.055 include score of both share(truth distrubution) and nonshare binds(prediction ap)).
if anyone has nonshare score, please put it here.
", "votes": null, "replies": [ { "id": 2805029, "author_name": "w5833946", "author_url": "", "post_date": "05/10/2024 10:20:33", "content": "
\nThe new result is quite puzzling, not sure what happened here.
| LB | \nLB (mask noshare) | \nLB (mask share) | \n
|---|---|---|
| 0.629 | \n0.596 | \n0.053 | \n
| 0.663 | \n0.596 | \n0.058 | \n
thanks. your results are probably correct.
\nif we have perfect nonshare prediction, we can get nonshare lb=0.2+
check the following
\nhttps://www.kaggle.com/code/hengck23/how-to-probe
\nhttps://www.kaggle.com/code/junkoda/unknowns-in-public-test-set/notebook
just to double check, i attached \"submit_sharing.zip\" to mask share and nonshare in submit. you may check if it is correct.
", "votes": null, "replies": [ { "id": 2806324, "author_name": "w5833946", "author_url": "", "post_date": "05/11/2024 03:28:37", "content": "Thank you, I will check it this night.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2784340": "Please share your CV/LB results here. \n\nI'm using my [CV split](https://www.kaggle.com/datasets/thedrcat/belka-cv-split), here are most recent results:\n\n| Model | CV-random | CV-noshare | LB |\n| --- | --- | --- | --- |\n| Molformer | 0.64 | 0.008 | 0.564 |\n| Molformer | 0.62 | 0.017 | 0.554 |\n\n\nFrom other discussions and my initial results it's likely that public LB seems correlated with random split, but private results will most likely depend on how well you can model the no-share / test distribution. \n\nBelow more details by protein target (if you're not using [W&B](https://wandb.ai) for tracking, you're missing out):\n\n", "2785587": "My CV LB split is very split dependent. The more samples I add to my experiments the lower my CV (using scaffold splitting) but higher LB. I'd put some examples here, but I am still experimenting.", "2785685": "the best split is probably random, see also paper.\ni suggest 2 separate splits and experiments\n\nmy strategy:\n[1] cv score versus num of share blocks\n- train samples = share blocks\n- valid samples with 0,1,2,3 share blocks\nmake a plot of cv score versus num of share blocks\nmake tsnet map or knn results for the blocks\ntry some external dataset\nmy conclusion here so far is that for non share blocks, results are really bad (need to solve later)\n\n[2] cv score for share blocks\n- best results is probably using random split and all samples\n\n\n\npaper on cv split, datasize, etc:\nUncovering Neural Scaling Laws in Molecular Representation Learning\nhttps://arxiv.org/abs/2309.15123", "2785690": "\"I'm using my CV split, here are most recent results:\"\n\nboth molformer and chemberta should be about to give you lb 0.575. (CV 0.670), i am using random splits\nI think this is the limit of SMILES string transformer?\n(i think some kaggler can get up to lb0.605(CV 0.700))?\n\nmolformer = 12 layers\nchemberta = 3 layers\n\ndifferent pretrain chemberta 5,10,77M MLM,MTR has slight different results\n\nNote there a 2 different molformer in papers (one is based on SMILES string , another is a GNN)", "2786617": "I agree there will likely need to be ways to handle compounds close to the train set and those that aren't. I have 64gb of RAM at home and it seems that even with np.packbits I can't train models with half of the train rows. So, now I'm focused on finding a subsampling strategy. I'm learning a lot about how to manage large datasets in this competition.", "2786912": "actually i am think that for those kagglers that shown good performance, espeically those with good domain knowledge, google should get them better compute power, e.g. discount price at google cloud for better gpu, etc\n\nor kaggler ourselves can setup some fund.\nif the solution is good, then we can all learn something valuable, rather than trial and error like a headless fly", "2787584": "CV 0.69, LB 0.585\nmodel : 1dcnn trained on all data\nI made this model public : https://www.kaggle.com/ahmedelfazouan/belka-1dcnn-starter-with-all-data", "2787594": "This is pretty cool @ahmedelfazouan ! Looking forward to reading through it.", "2787630": "I've considered renting more compute power, but I do think exploring a subsampling strategy is a good learning experience for me.", "2788308": "CV: 0.597, 0.600, 0.608, 0.613\nLB: 0.575, 0.586, 0.589, 0.585\n\nModel: small nn on top of frozen molformer\nVal Split: positive ratio 1% (random, len 1e5)", "2788485": "your cv looks low.\ni am not sure if it is the effect of different split or frozen (i think frozen is more likely)\n\nthe good news is that CV and LB gap is very small, which means that you actually score very well in nonshare b block (you can confirm by local experiment or separate submission)\n\nand i think if you starts unfreeze, you will start to have different results.\nyou can try ensemble of freeze and frozen models, or using freeze and frozen models for share and unshare blocks", "2788579": "i suddenly have an idea.\nhow about LORA like model?\nuse some layers to modify molformer intermediate or/and final layer output?", "2788682": "I think there are still some lower hanging fruits that can be picked. These models were trained (with 4090*1) in around 5-7 hours with a subset of (neg) train, only iterating over the train set about twice.\n\nBefore I set the fixed positive ratio val, a different random split/sample had cv around 0.62, so yeah this split is lower.\n\nI don't think I'll unfreeze molformer just yet as that adds a massive amount of compute cost and will destroy my batch size.", "2788934": "Hmmm will have to look into loras, definitely seems like a possible path forward.", "2790293": "i load my data, and sample 1024 * 1024 molecules, with X% positive samples, and save it to a parquet file. I do this 100 times. For training i load one file at a time to save ram. You could also have a separate file for just postives , and multiple files for negative if you want to change +ve/-ve split on the fly.", "2790563": "molformer works slightly better for me.\nWith Molformer i get .69 CV, .583 lb.\nWith chemberta i get .65 CV, .56 lb. \n\ni tried using Selformer, ChemGPT (selfies models) but got much worse scores.", "2790810": "Using a variation on @ahmedelfazouan notebook, I get the following CV-LB scores.\n\n0.72 - 0.61\n0.71 - 0.6\n0.65 - 0.57\n\nI've only tried a few simple things here, but I am really surprised how well a 1D-CNN handles the in-distribution samples in such a simple yet elegant way. I really learned something from this notebook.", "2790830": "Nice work @chemdatafarmer !", "2790833": "It's very much thanks to learning from you, so thank you!", "2791018": "\n\n\nsome observation:\n1. i train on my local machines, batch size = 5k to 10k, single gpu in pytorch.\nsurprising, adding batch norm don't work as well. (meaning some bias in data, or 10k is not representative\" enough)", "2791747": "hengck23 my best model uses batch norm, though I haven't systematically studied the impact of taking it out", "2796173": "\"best model uses batch norm, \"\n\ni think the reason for my poorer performance could be this\nhttps://medium.com/deeplearningmadeeasy/ghost-batchnorm-explained-e0fa9d651e03\n\n\" It seems neural networks tends to do worse for unseen data when being trained on large batch sizes.\"\n\nif you are using 8xTPU, you don't see this problem because data is split into 8 chunks and bn is applied independently to each chunk (i.e natural ghost batchnorm)\n\nit is a problem for me because i am using single gpu.\n\nNow i change bn to ghost bn and it seems to work better\n\n---\n\nfor simple 3 layer cnn1d, it doesn't matter if you use batch norm or not. But if you use deep layer, or shalow/deep layer for rnn, trasnformer, then it is important for normalisation because of the use in exp family function in activation (e.g. tahn,sigmoid in rnn or exp in softmax in attnetion)", "2796700": "Ah, yes, I didn't appreciate the differences between GPU and TPU here. Indeed, my batch norms were added when I tried to make the network deeper.", "2799055": "| CV-random | CV-noshare |\n| --- | --- |\n| 0.664 | 0.022 |\n\n\nI still have not submitted it because I am currently toying around (or rather, fighting) with RDKit for molecule representation within a 3d space, because I think that it would be quite informative to have", "2804645": "set all sharing blocks to zero at submission.csv.\nI get nonshare sore = lb0.055.\n(actually 0.055 include score of both share(truth distrubution) and nonshare binds(prediction ap)).\n\n\nif anyone has nonshare score, please put it here.", "2805029": "~~My best submission is currently overwritten by a small sample experiment, so I checked one of my past submission.~~\nThe new result is quite puzzling, not sure what happened here.\n| LB | LB (mask noshare) | LB (mask share)|\n| --- | --- | ---|\n| 0.629| 0.596 | 0.053 |\n| 0.663 | 0.596 | 0.058 |", "2806318": "w5833946 \n\nthanks. your results are probably correct.\nif we have perfect nonshare prediction, we can get nonshare lb=0.2+\n\ncheck the following\nhttps://www.kaggle.com/code/hengck23/how-to-probe\nhttps://www.kaggle.com/code/junkoda/unknowns-in-public-test-set/notebook", "2806321": "just to double check, i attached \"submit_sharing.zip\" to mask share and nonshare in submit. you may check if it is correct.", "2806324": "Thank you, I will check it this night." }, "source": "meta" }