I created a CV split that intends to mirror the test set split created by the host.
\nSpecifically, I create CV split based on:
\nThis is based on discussion shared by host here.
\nDATASET WITH CV SPLIT: https://www.kaggle.com/datasets/thedrcat/belka-cv-split
\nCODE: https://www.kaggle.com/code/thedrcat/belka-split-cv-like-the-host/
I'm learning chemistry on the competition, so if I made any mistakes - apologies and would really appreciate feedback!
\nEDIT: V2 updated based on the discussion below - removed Murcko scaffold, corrected BB split.
", "messages": [ { "id": "2773779", "postDate": "04/24/2024 22:12:55", "content": "I created a CV split that intends to mirror the test set split created by the host.
\nSpecifically, I create CV split based on:
\nThis is based on discussion shared by host here.
\nDATASET WITH CV SPLIT: https://www.kaggle.com/datasets/thedrcat/belka-cv-split
\nCODE: https://www.kaggle.com/code/thedrcat/belka-split-cv-like-the-host/
I'm learning chemistry on the competition, so if I made any mistakes - apologies and would really appreciate feedback!
\nEDIT: V2 updated based on the discussion below - removed Murcko scaffold, corrected BB split.
", "rawMarkdown": "I created a CV split that intends to mirror the test set split created by the host.\n\nSpecifically, I create CV split based on:\n- Scaffold split\n- Building block split\n- Random split\n\nThis is based on discussion shared by host [here](https://www.kaggle.com/competitions/leash-BELKA/discussion/491362#2737103).\n\nDATASET WITH CV SPLIT: https://www.kaggle.com/datasets/thedrcat/belka-cv-split\nCODE: https://www.kaggle.com/code/thedrcat/belka-split-cv-like-the-host/\n\nI'm learning chemistry on the competition, so if I made any mistakes - apologies and would really appreciate feedback!\n\nEDIT: V2 updated based on the discussion below - removed Murcko scaffold, corrected BB split.", "votes": null }, { "id": "2773798", "postDate": "04/24/2024 22:57:22", "content": "A word of caution- remember to exclude all BBs in BB split from train for it to mirror the no share BB split in test i.e. the group df.buildingblock1_smiles.isin(bbs1) | df.buildingblock2_smiles.isin(bbs2) | df.buildingblock3_smiles.isin(bbs2)
\n(| Instead &)
To be more clear, what we need is like this:
\ndf[] = \ndf[].iloc[df.buildingblock1_smiles.isin(bbs1) & df.buildingblock2_smiles.isin(bbs2) & df.buildingblock3_smiles.isin(bbs2)] = \ndf[].iloc[df.buildingblock1_smiles.isin(bbs1) | df.buildingblock2_smiles.isin(bbs2) | df.buildingblock3_smiles.isin(bbs2)] = \nThis is great. Couple comments:
\nAlso, I'm not sure exactly what you're doing in the MurckoScaffold section, but I'm highly doubtful that there's anything in train that's at all like the non-triazines in (private) test.
\nInstead, I think holding out an entire external dataset would be the best we can do to test on unseen data? Keep in mind, though, that only sEH is represented in that dataset, and is much easier to bind than the other two.
", "rawMarkdown": "Also, I'm not sure exactly what you're doing in the MurckoScaffold section, but I'm highly doubtful that there's anything in train that's at all like the non-triazines in (private) test. \n\nInstead, I think holding out an [entire external dataset](https://www.kaggle.com/code/chemdatafarmer/additional-seh-data) would be the best we can do to test on unseen data? Keep in mind, though, that only sEH is represented in that dataset, and is much easier to bind than the other two.", "votes": null }, { "id": "2773895", "postDate": "04/25/2024 00:25:49", "content": "Another reasonable option is to hold out a LOT more BBs. Triazine core or not, there may not be a ton of difference between unseen \"everything BUT triazine core\" and unseen \"everything\".
", "rawMarkdown": "Another reasonable option is to hold out a LOT more BBs. Triazine core or not, there may not be a ton of difference between unseen \"everything BUT triazine core\" and unseen \"everything\".", "votes": null }, { "id": "2773914", "postDate": "04/25/2024 00:53:56", "content": "i haven't tried this yet , but i think:
\nif we get better results for share blocks (local CV), the \"share blocks LB\" should also get better, but the \"nonshare blocks LB\" may get worst.
\nto verify, submit separately for test share and nonshare blocks.
in fact maybe two set of models, one for share and another for nonshare blocks prediction.
\nThen there is actually two set of data split
Hi, thanks for the notebook and datasets!
\nRegarding the scaffold split, just to mention that as was previously shared here, all molecules in the training set contain a triazine core. I haven't checked it myself, but I would expect that the same core was used to react with the different building blocks to synthesize the final molecules (one could check that by comparing the 3 building blocks and the final molecule, taking into account some differences due to the reactions used bind the BBs and core together). I.e. the MurckoScaffold is incorporating parts of the building blocks in the detected scaffolds. So I am not sure if this is better than splitting the train dataset based on building blocks similarity.
", "rawMarkdown": "Hi, thanks for the notebook and datasets!\n\nRegarding the scaffold split, just to mention that as was previously shared [here](https://www.kaggle.com/competitions/leash-BELKA/discussion/493294), all molecules in the training set contain a triazine core. I haven't checked it myself, but I would expect that the same core was used to react with the different building blocks to synthesize the final molecules (one could check that by comparing the 3 building blocks and the final molecule, taking into account some differences due to the reactions used bind the BBs and core together). I.e. the MurckoScaffold is incorporating parts of the building blocks in the detected scaffolds. So I am not sure if this is better than splitting the train dataset based on building blocks similarity.", "votes": null }, { "id": "2775732", "postDate": "04/25/2024 19:02:18", "content": "Do you mean all BBs in BB split and used in the train set should be excluded from the test set?
", "rawMarkdown": "Do you mean all BBs in BB split and used in the train set should be excluded from the test set?", "votes": null }, { "id": "2775805", "postDate": "04/25/2024 20:08:36", "content": "Thanks @shlomoron & @roberthatch, super helpful! Yep, I think the && group should be in the noshare split, but the II group maybe we don't need to exclude, just add to the random split…. We have quite a lot, but still don't like wasting data 😅
Yeah, that makes sense. One option is a bit of both. You set it all aside as test data, but when you predict you look at all predictions AND look at the subset predictions (excluding the partial overlap cases) to see your score against a distribution more closely matching LB.
", "rawMarkdown": "Yeah, that makes sense. One option is a bit of both. You set it all aside as test data, but when you predict you look at all predictions AND look at the subset predictions (excluding the partial overlap cases) to see your score against a distribution more closely matching LB.", "votes": null }, { "id": "2775822", "postDate": "04/25/2024 20:26:56", "content": "Also, I don't recall if you do this as CV? If so, at the least you have to modify so that some data is held out of train for more than one fold. For instance If you reserve 1-30 for fold 1, and 300-330 for fold 2, then at least you can't allow the intersection of those BBs to be used in train data except when training folds 3,4,5.
", "rawMarkdown": "Also, I don't recall if you do this as CV? If so, at the least you have to modify so that some data is held out of train for more than one fold. For instance If you reserve 1-30 for fold 1, and 300-330 for fold 2, then at least you can't allow the intersection of those BBs to be used in train data except when training folds 3,4,5.", "votes": null }, { "id": "2775904", "postDate": "04/25/2024 21:45:53", "content": "Yep, mainly for CV - for now I only want to train single fold and find correlation between CV/LB (and ideally in a way that will transfer to private). I'll figure out how to use held out data for training once I have some good results.
", "rawMarkdown": "Yep, mainly for CV - for now I only want to train single fold and find correlation between CV/LB (and ideally in a way that will transfer to private). I'll figure out how to use held out data for training once I have some good results.", "votes": null } ], "comments": [ { "id": 2773798, "author_name": "shlomoron", "author_url": "", "post_date": "04/24/2024 22:57:22", "content": "A word of caution- remember to exclude all BBs in BB split from train for it to mirror the no share BB split in test i.e. the group df.buildingblock1_smiles.isin(bbs1) | df.buildingblock2_smiles.isin(bbs2) | df.buildingblock3_smiles.isin(bbs2)
\n(| Instead &)
To be more clear, what we need is like this:
\ndf[] = \ndf[].iloc[df.buildingblock1_smiles.isin(bbs1) & df.buildingblock2_smiles.isin(bbs2) & df.buildingblock3_smiles.isin(bbs2)] = \ndf[].iloc[df.buildingblock1_smiles.isin(bbs1) | df.buildingblock2_smiles.isin(bbs2) | df.buildingblock3_smiles.isin(bbs2)] = \nDo you mean all BBs in BB split and used in the train set should be excluded from the test set?
", "votes": null, "replies": [] }, { "id": 2775805, "author_name": "thedrcat", "author_url": "", "post_date": "04/25/2024 20:08:36", "content": "Thanks @shlomoron & @roberthatch, super helpful! Yep, I think the && group should be in the noshare split, but the II group maybe we don't need to exclude, just add to the random split…. We have quite a lot, but still don't like wasting data 😅
Yeah, that makes sense. One option is a bit of both. You set it all aside as test data, but when you predict you look at all predictions AND look at the subset predictions (excluding the partial overlap cases) to see your score against a distribution more closely matching LB.
", "votes": null, "replies": [] }, { "id": 2775822, "author_name": "roberthatch", "author_url": "", "post_date": "04/25/2024 20:26:56", "content": "Also, I don't recall if you do this as CV? If so, at the least you have to modify so that some data is held out of train for more than one fold. For instance If you reserve 1-30 for fold 1, and 300-330 for fold 2, then at least you can't allow the intersection of those BBs to be used in train data except when training folds 3,4,5.
", "votes": null, "replies": [ { "id": 2775904, "author_name": "thedrcat", "author_url": "", "post_date": "04/25/2024 21:45:53", "content": "Yep, mainly for CV - for now I only want to train single fold and find correlation between CV/LB (and ideally in a way that will transfer to private). I'll figure out how to use held out data for training once I have some good results.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2773879, "author_name": "roberthatch", "author_url": "", "post_date": "04/25/2024 00:17:26", "content": "This is great. Couple comments:
\nAlso, I'm not sure exactly what you're doing in the MurckoScaffold section, but I'm highly doubtful that there's anything in train that's at all like the non-triazines in (private) test.
\nInstead, I think holding out an entire external dataset would be the best we can do to test on unseen data? Keep in mind, though, that only sEH is represented in that dataset, and is much easier to bind than the other two.
", "votes": null, "replies": [ { "id": 2773895, "author_name": "roberthatch", "author_url": "", "post_date": "04/25/2024 00:25:49", "content": "Another reasonable option is to hold out a LOT more BBs. Triazine core or not, there may not be a ton of difference between unseen \"everything BUT triazine core\" and unseen \"everything\".
", "votes": null, "replies": [] } ] } ] }, { "id": 2773914, "author_name": "hengck23", "author_url": "", "post_date": "04/25/2024 00:53:56", "content": "i haven't tried this yet , but i think:
\nif we get better results for share blocks (local CV), the \"share blocks LB\" should also get better, but the \"nonshare blocks LB\" may get worst.
\nto verify, submit separately for test share and nonshare blocks.
in fact maybe two set of models, one for share and another for nonshare blocks prediction.
\nThen there is actually two set of data split
Hi, thanks for the notebook and datasets!
\nRegarding the scaffold split, just to mention that as was previously shared here, all molecules in the training set contain a triazine core. I haven't checked it myself, but I would expect that the same core was used to react with the different building blocks to synthesize the final molecules (one could check that by comparing the 3 building blocks and the final molecule, taking into account some differences due to the reactions used bind the BBs and core together). I.e. the MurckoScaffold is incorporating parts of the building blocks in the detected scaffolds. So I am not sure if this is better than splitting the train dataset based on building blocks similarity.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2773779": "I created a CV split that intends to mirror the test set split created by the host.\n\nSpecifically, I create CV split based on:\n- Scaffold split\n- Building block split\n- Random split\n\nThis is based on discussion shared by host [here](https://www.kaggle.com/competitions/leash-BELKA/discussion/491362#2737103).\n\nDATASET WITH CV SPLIT: https://www.kaggle.com/datasets/thedrcat/belka-cv-split\nCODE: https://www.kaggle.com/code/thedrcat/belka-split-cv-like-the-host/\n\nI'm learning chemistry on the competition, so if I made any mistakes - apologies and would really appreciate feedback!\n\nEDIT: V2 updated based on the discussion below - removed Murcko scaffold, corrected BB split.", "2773798": "A word of caution- remember to exclude all BBs in BB split from train for it to mirror the no share BB split in test i.e. the group df.buildingblock1_smiles.isin(bbs1) | df.buildingblock2_smiles.isin(bbs2) | df.buildingblock3_smiles.isin(bbs2)\n(| Instead &)", "2773870": "To be more clear, what we need is like this:\n```python\ndf['bb_group'] = 'train'\ndf['bb_group'].iloc[df.buildingblock1_smiles.isin(bbs1) & df.buildingblock2_smiles.isin(bbs2) & df.buildingblock3_smiles.isin(bbs2)] = 'test'\ndf['bb_group'].iloc[df.buildingblock1_smiles.isin(bbs1) | df.buildingblock2_smiles.isin(bbs2) | df.buildingblock3_smiles.isin(bbs2)] = 'exclude! This is neither train nor test!'\n```", "2773879": "This is great. Couple comments:\n\n* You might consider using exactly the number of BBs in the [test distribution](https://www.kaggle.com/competitions/leash-BELKA/discussion/496576). 17BB1, 36BB2. (You might want to take a couple from BB3 that aren't in BB2, there's 181 of those in train).\n* In the end, I will want two different test CV distributions. One for comparing with public LB, one for comparing with private LB. Those are going to be 2 different distributions, and mostly I'll care more about the private LB distribution. BUT, the public LB one is for running sanity checks and making sure that CV and LB stay in sync during feature engineering etc.", "2773889": "Also, I'm not sure exactly what you're doing in the MurckoScaffold section, but I'm highly doubtful that there's anything in train that's at all like the non-triazines in (private) test. \n\nInstead, I think holding out an [entire external dataset](https://www.kaggle.com/code/chemdatafarmer/additional-seh-data) would be the best we can do to test on unseen data? Keep in mind, though, that only sEH is represented in that dataset, and is much easier to bind than the other two.", "2773895": "Another reasonable option is to hold out a LOT more BBs. Triazine core or not, there may not be a ton of difference between unseen \"everything BUT triazine core\" and unseen \"everything\".", "2773914": "i haven't tried this yet , but i think:\nif we get better results for share blocks (local CV), the \"share blocks LB\" should also get better, but the \"nonshare blocks LB\" may get worst.\nto verify, submit separately for test share and nonshare blocks.\n\nin fact maybe two set of models, one for share and another for nonshare blocks prediction.\nThen there is actually two set of data split", "2774472": "Hi, thanks for the notebook and datasets!\n\nRegarding the scaffold split, just to mention that as was previously shared [here](https://www.kaggle.com/competitions/leash-BELKA/discussion/493294), all molecules in the training set contain a triazine core. I haven't checked it myself, but I would expect that the same core was used to react with the different building blocks to synthesize the final molecules (one could check that by comparing the 3 building blocks and the final molecule, taking into account some differences due to the reactions used bind the BBs and core together). I.e. the MurckoScaffold is incorporating parts of the building blocks in the detected scaffolds. So I am not sure if this is better than splitting the train dataset based on building blocks similarity.", "2775732": "Do you mean all BBs in BB split and used in the train set should be excluded from the test set?", "2775805": "Thanks @shlomoron & @roberthatch, super helpful! Yep, I think the `&&` group should be in the `noshare` split, but the `II` group maybe we don't need to exclude, just add to the random split.... We have quite a lot, but still don't like wasting data 😅", "2775816": "Yeah, that makes sense. One option is a bit of both. You set it all aside as test data, but when you predict you look at all predictions AND look at the subset predictions (excluding the partial overlap cases) to see your score against a distribution more closely matching LB.", "2775822": "Also, I don't recall if you do this as CV? If so, at the least you have to modify so that some data is held out of train for more than one fold. For instance If you reserve 1-30 for fold 1, and 300-330 for fold 2, then at least you can't allow the intersection of those BBs to be used in train data except when training folds 3,4,5.", "2775904": "Yep, mainly for CV - for now I only want to train single fold and find correlation between CV/LB (and ideally in a way that will transfer to private). I'll figure out how to use held out data for training once I have some good results." }, "source": "meta" }