\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. Due to the overlapping nature of DEL chemistry, the test-train splits necessarily shrink the amount of data available during the competition (e.g., for a given building block in the test set, all molecules containing that building block must be removed from the training and validation sets ). These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements)
\n
sklearn.model_selection StratifiedGroupKFold\ntrain[] = train[]++train[]++train[]++train[]\ntrain[] = train[]++train[].apply( x: (x))\n\nsgkf = StratifiedGroupKFold(n_splits=, random_state=)\n\n train_index, valid_index sgkf.split(train, train.target, train.buildingblocks):\n --- \n\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
is this right CV strategy as per the requirements?
\n", "messages": [ { "id": 2737517, "postDate": "2024-04-05T19:38:26.507Z", "content": "\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. Due to the overlapping nature of DEL chemistry, the test-train splits necessarily shrink the amount of data available during the competition (e.g., for a given building block in the test set, all molecules containing that building block must be removed from the training and validation sets ). These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements)
\n
sklearn.model_selection StratifiedGroupKFold\ntrain[] = train[]++train[]++train[]++train[]\ntrain[] = train[]++train[].apply( x: (x))\n\nsgkf = StratifiedGroupKFold(n_splits=, random_state=)\n\n train_index, valid_index sgkf.split(train, train.target, train.buildingblocks):\n --- \n\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
is this right CV strategy as per the requirements?
\n", "rawMarkdown": "> All data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. Due to the overlapping nature of DEL chemistry, the test-train splits necessarily shrink the amount of data available during the competition (e.g., **for a given building block in the test set, all molecules containing that building block must be removed from the training and validation sets** ). These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements)\n\n```python\nfrom sklearn.model_selection import StratifiedGroupKFold\ntrain['buildingblocks'] = train['buildingblock1_smiles']+\"-\"+train['buildingblock2_smiles']+\"-\"+train['buildingblock3_smiles']+\"-\"+train[\"protein_name\"]\ntrain['target'] = train[\"protein_name\"]+\"-\"+train[\"binds\"].apply(lambda x: str(x))\n\nsgkf = StratifiedGroupKFold(n_splits=5, random_state=42)\n\nfor train_index, valid_index in sgkf.split(train, train.target, train.buildingblocks):\n --- \n```\n\n---\n\n> All data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. **To test generalizability, the test set contains building blocks that are not in the training set.** These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).\n\n- **Need to split molecule between folds so it will match with private dataset**\n\n---\n\n**is this right CV strategy as per the requirements?**\n\n~~i miss understood initially it is a code competition, after it understood not~~", "votes": 16 }, { "id": 2745246, "postDate": "2024-04-10T13:51:11.107Z", "content": "My split:
\ndf = df(\n (\n ~((pl() < ) &\n (pl() < ) &\n (pl() < ))\n )()(pl.Int8)\n)\nI'm using this shrinked dataset
\nUpd:
\nsince blocks are not unique per-column, this split is not correct, in order to get right split you should first re-enumerate all unique blocks in all three columns and then do this split again
I think splitting on all 3 building blocks concatenated together is (basically) the same as splitting on the molecule.
\nHere are a few resources to help think about ways to split your data:
\nhttps://deepchem.readthedocs.io/en/latest/api_reference/splitters.html
\nhttps://tdcommons.ai/functions/data_split/
\nhttps://lifesci.dgl.ai/api/utils.splitters.html
@andrewdblevins as i understand molecule is together of 3 building blocks, could you share any resource how it combined since it is not just string concatenations and is the order of 3 building blocks sequence also random?.
\nit's not a simple operation. You have to simulate all the chemical reactions.
\nLater I will post the nitty gritty details on all the chemical reactions that go into constructing the DELs. But that knowledge shouldn't be necessary for this competition.
", "rawMarkdown": "it's not a simple operation. You have to simulate all the chemical reactions. \n\nLater I will post the nitty gritty details on all the chemical reactions that go into constructing the DELs. But that knowledge shouldn't be necessary for this competition.", "votes": 3 }, { "id": 2737662, "postDate": "2024-04-05T21:28:17.440Z", "content": "Ok @andrewdblevins thank you.
", "rawMarkdown": "Ok @andrewdblevins thank you.", "votes": 1 }, { "id": 2739776, "postDate": "2024-04-07T09:19:47.783Z", "content": "@andrewdblevins as molecule have chemical combined of 3 building blocks
\n- binds=0/1 is based on molecule and protein name - is this correct ?
\n- are building blocks helpful? - any reference mater to understand building blocks and molecule relationship?
| Molecules/building blocks | \nTrain (total samples) | \nTest (total samples) | \nCommon with Test | \nCommon with Train | \n
|---|---|---|---|---|
| Molecules | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks concat | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks combinations | \n98,415,610 | \n878,022 | \n98,415,068 => 99% | \n508,983 => 58% | \n
Building Blocks were included because much of the previous work predicting DEL data depends on them.
\nSee this paper by google/xchem and this one out of anagenex
Thanks for the clarity, I gone through this paper anagenex and shared Discussion - Papers: DNA-encoded chemical library (DEL), will go through other paper too.
", "rawMarkdown": "Thanks for the clarity, I gone through this paper [anagenex](https://arxiv.org/pdf/2205.08020.pdf) and shared [Discussion - Papers: DNA-encoded chemical library (DEL)](https://www.kaggle.com/competitions/leash-BELKA/discussion/491401), will go through other paper too." } ] }, { "id": 2740984, "postDate": "2024-04-08T04:31:00.837Z", "content": "\nYour comment on 3 building blocks suggests that no molecule was created with different initial 3 blocks. Am I reading into your comment correctly, or can molecules end up being the same but with a different creation path in train or test?
", "rawMarkdown": "[Andrew](https://www.kaggle.com/andrewdblevins)\n\nYour comment on 3 building blocks suggests that no molecule was created with different initial 3 blocks. Am I reading into your comment correctly, or can molecules end up being the same but with a different creation path in train or test?\n", "votes": 1, "replies": [ { "id": 2741040, "postDate": "2024-04-08T05:07:47.217Z", "content": "I believe there is none of that in this set of molecules, but in general, it's possible to create the same molecule with different sets of 3 building blocks.
", "rawMarkdown": "I believe there is none of that in this set of molecules, but in general, it's possible to create the same molecule with different sets of 3 building blocks.", "votes": 1 } ] } ] }, { "id": 2739986, "postDate": "2024-04-07T12:48:24.707Z", "content": "\nI have a question about how the public and private leaderboard data was split. Is the split random? Or is the public leaderboard correspond to a validation set, and private leaderboard to a test set which have purposely different distributions of building blocks?
\nThanks for being active and giving references and links they have been useful in getting started with this competition.
", "rawMarkdown": "@andrewdblevins \n\nI have a question about how the public and private leaderboard data was split. Is the split random? Or is the public leaderboard correspond to a validation set, and private leaderboard to a test set which have purposely different distributions of building blocks?\n\nThanks for being active and giving references and links they have been useful in getting started with this competition. ", "votes": 3, "replies": [ { "id": 2740129, "postDate": "2024-04-07T15:44:42.823Z", "content": "For now, I am not going to say anything about the composition of the private test set other than both test sets were both chosen to test generalizability to new molecules.
\nIf these models are successful, you should be able to start evaluating material from chemistry catalogs like wuxi or enamine, order 100 top hits to be tested, and expect a good percentage of those compounds to come back as active. Since chemical space is so big, it is very unlikely that the catalogs will have any molecules in common with your training set, so generalization is crucial.
", "rawMarkdown": "For now, I am not going to say anything about the composition of the private test set other than both test sets were both chosen to test generalizability to new molecules.\n\nIf these models are successful, you should be able to start evaluating material from chemistry catalogs like wuxi or enamine, order 100 top hits to be tested, and expect a good percentage of those compounds to come back as active. Since chemical space is so big, it is very unlikely that the catalogs will have any molecules in common with your training set, so generalization is crucial.", "votes": 8, "replies": [ { "id": 2740206, "postDate": "2024-04-07T17:06:31.387Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 2740195, "postDate": "2024-04-07T16:58:28.147Z", "content": "Thanks for info. I think I am getting a handle on validation set up. Definitely a little complicated.
", "rawMarkdown": "Thanks for info. I think I am getting a handle on validation set up. Definitely a little complicated. ", "votes": 2 } ] }, { "id": 2745178, "postDate": "2024-04-10T12:50:52.640Z", "content": "So far I am using scaffold split of individual BBs, this gives you fold 0, 1, 2, etc for BB1, fold 0, 1, 2, etc for BB2 and same for BB3.
\nIn that case you can make a val fold where all BBs don't overlap with train BBs, and structurally different
@kvigly55 as scaffold on BBs will make each fold will have the binds/no binds distributions and # of samples
", "rawMarkdown": "@kvigly55 as scaffold on BBs will make each fold will have the binds/no binds distributions and # of samples" } ] } ], "comments": [ { "id": 2745246, "author_name": "slime", "author_url": "", "post_date": "2024-04-10T13:51:11.107000", "content": "My split:
\ndf = df(\n (\n ~((pl() < ) &\n (pl() < ) &\n (pl() < ))\n )()(pl.Int8)\n)\nI'm using this shrinked dataset
\nUpd:
\nsince blocks are not unique per-column, this split is not correct, in order to get right split you should first re-enumerate all unique blocks in all three columns and then do this split again
I think splitting on all 3 building blocks concatenated together is (basically) the same as splitting on the molecule.
\nHere are a few resources to help think about ways to split your data:
\nhttps://deepchem.readthedocs.io/en/latest/api_reference/splitters.html
\nhttps://tdcommons.ai/functions/data_split/
\nhttps://lifesci.dgl.ai/api/utils.splitters.html
@andrewdblevins as i understand molecule is together of 3 building blocks, could you share any resource how it combined since it is not just string concatenations and is the order of 3 building blocks sequence also random?.
\nit's not a simple operation. You have to simulate all the chemical reactions.
\nLater I will post the nitty gritty details on all the chemical reactions that go into constructing the DELs. But that knowledge shouldn't be necessary for this competition.
", "votes": 3, "replies": [] }, { "id": 2737662, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-04-05T21:28:17.440000", "content": "Ok @andrewdblevins thank you.
", "votes": 1, "replies": [] }, { "id": 2739776, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-04-07T09:19:47.783000", "content": "@andrewdblevins as molecule have chemical combined of 3 building blocks
\n- binds=0/1 is based on molecule and protein name - is this correct ?
\n- are building blocks helpful? - any reference mater to understand building blocks and molecule relationship?
| Molecules/building blocks | \nTrain (total samples) | \nTest (total samples) | \nCommon with Test | \nCommon with Train | \n
|---|---|---|---|---|
| Molecules | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks concat | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks combinations | \n98,415,610 | \n878,022 | \n98,415,068 => 99% | \n508,983 => 58% | \n
Building Blocks were included because much of the previous work predicting DEL data depends on them.
\nSee this paper by google/xchem and this one out of anagenex
Thanks for the clarity, I gone through this paper anagenex and shared Discussion - Papers: DNA-encoded chemical library (DEL), will go through other paper too.
", "votes": 0, "replies": [] } ] }, { "id": 2740984, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2024-04-08T04:31:00.837000", "content": "\nYour comment on 3 building blocks suggests that no molecule was created with different initial 3 blocks. Am I reading into your comment correctly, or can molecules end up being the same but with a different creation path in train or test?
", "votes": 1, "replies": [ { "id": 2741040, "author_name": "Andrew D. Blevins", "author_url": "", "post_date": "2024-04-08T05:07:47.217000", "content": "I believe there is none of that in this set of molecules, but in general, it's possible to create the same molecule with different sets of 3 building blocks.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2739986, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2024-04-07T12:48:24.707000", "content": "\nI have a question about how the public and private leaderboard data was split. Is the split random? Or is the public leaderboard correspond to a validation set, and private leaderboard to a test set which have purposely different distributions of building blocks?
\nThanks for being active and giving references and links they have been useful in getting started with this competition.
", "votes": 3, "replies": [ { "id": 2740129, "author_name": "Andrew D. Blevins", "author_url": "", "post_date": "2024-04-07T15:44:42.823000", "content": "For now, I am not going to say anything about the composition of the private test set other than both test sets were both chosen to test generalizability to new molecules.
\nIf these models are successful, you should be able to start evaluating material from chemistry catalogs like wuxi or enamine, order 100 top hits to be tested, and expect a good percentage of those compounds to come back as active. Since chemical space is so big, it is very unlikely that the catalogs will have any molecules in common with your training set, so generalization is crucial.
", "votes": 8, "replies": [ { "id": 2740206, "author_name": "", "author_url": "", "post_date": "2024-04-07T17:06:31.387000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2740195, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2024-04-07T16:58:28.147000", "content": "Thanks for info. I think I am getting a handle on validation set up. Definitely a little complicated.
", "votes": 2, "replies": [] } ] }, { "id": 2745178, "author_name": "kvigly", "author_url": "", "post_date": "2024-04-10T12:50:52.640000", "content": "So far I am using scaffold split of individual BBs, this gives you fold 0, 1, 2, etc for BB1, fold 0, 1, 2, etc for BB2 and same for BB3.
\nIn that case you can make a val fold where all BBs don't overlap with train BBs, and structurally different
@kvigly55 as scaffold on BBs will make each fold will have the binds/no binds distributions and # of samples
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2737517": "> All data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. Due to the overlapping nature of DEL chemistry, the test-train splits necessarily shrink the amount of data available during the competition (e.g., **for a given building block in the test set, all molecules containing that building block must be removed from the training and validation sets** ). These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements)\n\n```python\nfrom sklearn.model_selection import StratifiedGroupKFold\ntrain['buildingblocks'] = train['buildingblock1_smiles']+\"-\"+train['buildingblock2_smiles']+\"-\"+train['buildingblock3_smiles']+\"-\"+train[\"protein_name\"]\ntrain['target'] = train[\"protein_name\"]+\"-\"+train[\"binds\"].apply(lambda x: str(x))\n\nsgkf = StratifiedGroupKFold(n_splits=5, random_state=42)\n\nfor train_index, valid_index in sgkf.split(train, train.target, train.buildingblocks):\n --- \n```\n\n---\n\n> All data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. **To test generalizability, the test set contains building blocks that are not in the training set.** These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).\n\n- **Need to split molecule between folds so it will match with private dataset**\n\n---\n\n**is this right CV strategy as per the requirements?**\n\n~~i miss understood initially it is a code competition, after it understood not~~", "2745246": "My split:\n```\ndf = df.with_columns(\n (\n ~((pl.col('buildingblock1_smiles') < 250) &\n (pl.col('buildingblock2_smiles') < 600) &\n (pl.col('buildingblock3_smiles') < 800))\n ).alias('fold').cast(pl.Int8)\n)\n```\nI'm using this [shrinked dataset](https://www.kaggle.com/datasets/shlomoron/belka-shrunken-train-set)\n\nUpd:\nsince blocks are not unique per-column, this split is not correct, in order to get right split you should first re-enumerate all unique blocks in all three columns and then do this split again", "2737565": "I think splitting on all 3 building blocks concatenated together is (basically) the same as splitting on the molecule.\n\nHere are a few resources to help think about ways to split your data:\nhttps://deepchem.readthedocs.io/en/latest/api_reference/splitters.html\nhttps://tdcommons.ai/functions/data_split/\nhttps://lifesci.dgl.ai/api/utils.splitters.html", "2739986": "@andrewdblevins \n\nI have a question about how the public and private leaderboard data was split. Is the split random? Or is the public leaderboard correspond to a validation set, and private leaderboard to a test set which have purposely different distributions of building blocks?\n\nThanks for being active and giving references and links they have been useful in getting started with this competition. ", "2745178": "So far I am using scaffold split of individual BBs, this gives you fold 0, 1, 2, etc for BB1, fold 0, 1, 2, etc for BB2 and same for BB3. \nIn that case you can make a val fold where all BBs don't overlap with train BBs, and structurally different " } }