\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein
\n
| protein_name | \ntrain counts | \ntest count | \n
|---|---|---|
| HSA | \n98,415,610 | \n557,895 | \n
| BRD4 | \n98,415,610 | \n558,859 | \n
| sEH | \n98,415,610 | \n558,142 | \n
So, validation examples per protein is 200k so 600k samples ( total test samples = 878,022 )
\n| Public/Private | \nSamples | \n
|---|---|
| Public 35% | \n307,308 | \n
| Private 65% | \n570,714 | \n
| Building Blocks | \nTrain | \nTest | \nCommon | \nTest New | \nTrain Extra | \n
|---|---|---|---|---|---|
| buildingblock1_smiles | \n271 | \n340 | \n271 | \n69 | \n0 | \n
| buildingblock2_smiles | \n693 | \n1,139 | \n693 | \n446 | \n0 | \n
| buildingblock3_smiles | \n871 | \n1,388 | \n870 | \n517 | \n1 | \n
| Molecules/building blocks | \nTrain (total samples) | \nTest (total samples) | \nCommon with Train | \nCommon with Test | \n
|---|---|---|---|---|
| Molecules | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks concat | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks combinations | \n98,415,610 | \n878,022 | \n98,415,068 => 99% | \n508,983 => 58% | \n
| - no common molecule smiels between train and test datasets. | \n\n | \n | \n | \n |
| - 58% of the building blocks of test set are in train's building blocks | \n\n | \n | \n | \n |
\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
Answers - Discussion - deleted topic
\n\n\nThis is a good catch and something we weren't super clear on. The test set is made from the combination of several different splitting strategies. That statement was only meant to describe one of the strategies: the bb-split. for this split we hold out certain building blocks. But the overall test set also includes molecules from a scaffold split and a random split, hence the overlap. by @andrewdblevins
\n\n
\n- bb-split => new molecules
\n- scaffold split => based on scaffold of molecules diversity ( nice article about scaffold split -> https://www.oloren.ai/blog/scaff-split )
\n- random split => --
\n
Answers - Discussion - CV Strategy
\n\n\nI think splitting on all 3 building blocks concatenated together is (basically) the same as splitting on the molecule.
\n
\n Here are a few resources to help think about ways to split your data:
\n https://deepchem.readthedocs.io/en/latest/api_reference/splitters.html
\n https://tdcommons.ai/functions/data_split/
\n https://lifesci.dgl.ai/api/utils.splitters.html
\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein
\n
| protein_name | \ntrain counts | \ntest count | \n
|---|---|---|
| HSA | \n98,415,610 | \n557,895 | \n
| BRD4 | \n98,415,610 | \n558,859 | \n
| sEH | \n98,415,610 | \n558,142 | \n
So, validation examples per protein is 200k so 600k samples ( total test samples = 878,022 )
\n| Public/Private | \nSamples | \n
|---|---|
| Public 35% | \n307,308 | \n
| Private 65% | \n570,714 | \n
| Building Blocks | \nTrain | \nTest | \nCommon | \nTest New | \nTrain Extra | \n
|---|---|---|---|---|---|
| buildingblock1_smiles | \n271 | \n340 | \n271 | \n69 | \n0 | \n
| buildingblock2_smiles | \n693 | \n1,139 | \n693 | \n446 | \n0 | \n
| buildingblock3_smiles | \n871 | \n1,388 | \n870 | \n517 | \n1 | \n
| Molecules/building blocks | \nTrain (total samples) | \nTest (total samples) | \nCommon with Train | \nCommon with Test | \n
|---|---|---|---|---|
| Molecules | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks concat | \n98,415,610 | \n878,022 | \n0 | \n0 | \n
| 3 building blocks combinations | \n98,415,610 | \n878,022 | \n98,415,068 => 99% | \n508,983 => 58% | \n
| - no common molecule smiels between train and test datasets. | \n\n | \n | \n | \n |
| - 58% of the building blocks of test set are in train's building blocks | \n\n | \n | \n | \n |
\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
Answers - Discussion - deleted topic
\n\n\nThis is a good catch and something we weren't super clear on. The test set is made from the combination of several different splitting strategies. That statement was only meant to describe one of the strategies: the bb-split. for this split we hold out certain building blocks. But the overall test set also includes molecules from a scaffold split and a random split, hence the overlap. by @andrewdblevins
\n\n
\n- bb-split => new molecules
\n- scaffold split => based on scaffold of molecules diversity ( nice article about scaffold split -> https://www.oloren.ai/blog/scaff-split )
\n- random split => --
\n
Answers - Discussion - CV Strategy
\n\n\nI think splitting on all 3 building blocks concatenated together is (basically) the same as splitting on the molecule.
\n
\n Here are a few resources to help think about ways to split your data:
\n https://deepchem.readthedocs.io/en/latest/api_reference/splitters.html
\n https://tdcommons.ai/functions/data_split/
\n https://lifesci.dgl.ai/api/utils.splitters.html
\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
So we have both the priveta and public test in advance right?
\nSo we can predict every test case without time and resource constraint?
\nVery unusual… Why did they do this?
\nAlso, how do we have 870K test samples? Isn't it 200k*3 = 600k?
\nWhat does 'validation set' refer to? There's no validation.csv and only test.csv and train.csv.
\n\nSo we have both the priveta and public test in advance right?
\n
Yes, no hidden test set
\n\n\nSo we can predict every test case without time and resource constraint?
\n
Yes
\n\n\nVery unusual… Why did they do this?
\n
Because it is not code competition as test set is large maybe
\n\n\nAlso, how do we have 870K test samples? Isn't it 200k*3 = 600k? What does 'validation set' refer to?
\n
There's no validation.csv and only test.csv and train.csv.
\n\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n
So we have both the priveta and public test in advance right?
\nSo we can predict every test case without time and resource constraint?
\nVery unusual… Why did they do this?
\nAlso, how do we have 870K test samples? Isn't it 200k*3 = 600k?
\nWhat does 'validation set' refer to? There's no validation.csv and only test.csv and train.csv.
\n\nSo we have both the priveta and public test in advance right?
\n
Yes, no hidden test set
\n\n\nSo we can predict every test case without time and resource constraint?
\n
Yes
\n\n\nVery unusual… Why did they do this?
\n
Because it is not code competition as test set is large maybe
\n\n\nAlso, how do we have 870K test samples? Isn't it 200k*3 = 600k? What does 'validation set' refer to?
\n
There's no validation.csv and only test.csv and train.csv.
\n\n\nAll data were generated in-house at Leash Biosciences. We are providing roughly 98M training examples per protein, 200K validation examples per protein, and 360K test molecules per protein. To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders; we used 3 rounds of selection in triplicate to identify binders experimentally. Following the competition, Leash will make all the data available for future use (3 targets * 3 rounds of selection * 3 replicates * 133M molecules, or 3.6B measurements).
\n