\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F97f5a323b292a958c07110e61dac6764%2FSelection_103.png?generation=1715456726680517&alt=media\")
In short: my score boost from 0.639 to 0.668 (before metric update) comes from training model for noshare part with smaller weight for negative samples.
\nBased on @junkoda's amazing work-https://www.kaggle.com/code/junkoda/unknowns-in-public-test-set, it seems LB consists of several groups of data with different positive rate. I want to known if such difference comes from different filtering rules related to molecule property only or it is filtered directly based on target value? (minor structure changes based past positive experiment should also belongs to this case)
\nIf it is the latter case there is no way we can predict the unseen experiment design/filtering rules only based on molecule and this shouldn’t be what host really cares. Given specific molecule the binding should be deterministic but unless we have perfect model the negative rate matters when compare predicted probability.
\nTo more concentrate on the problem itself, I think it is better to compute MAP within each design group and then average the scores or provide us the exact positive rate of each groups such that we can do calibration without gambling.
\nHope for host reply. @andrewdblevins
\nEdit: Maybe we can do calibration based on predicted average probability but this involves unnecessary complexity.
In short: my score boost from 0.639 to 0.668 (before metric update) comes from training model for noshare part with smaller weight for negative samples.
\nBased on @junkoda's amazing work-https://www.kaggle.com/code/junkoda/unknowns-in-public-test-set, it seems LB consists of several groups of data with different positive rate. I want to known if such difference comes from different filtering rules related to molecule property only or it is filtered directly based on target value? (minor structure changes based past positive experiment should also belongs to this case)
\nIf it is the latter case there is no way we can predict the unseen experiment design/filtering rules only based on molecule and this shouldn’t be what host really cares. Given specific molecule the binding should be deterministic but unless we have perfect model the negative rate matters when compare predicted probability.
\nTo more concentrate on the problem itself, I think it is better to compute MAP within each design group and then average the scores or provide us the exact positive rate of each groups such that we can do calibration without gambling.
\nHope for host reply. @andrewdblevins
\nEdit: Maybe we can do calibration based on predicted average probability but this involves unnecessary complexity.
instead of retraining noshare part with smaller weight for negative samples, i implement in another way. post process the results to give boost to non-share prediction
\n(mathematically, loss weight in bce loss is power in probability, aka temperature scaling)
\n nonshare_submit_df = submit_df()\n nonshare_submit_df **= \n nonshare_submit_df(ensemble_file(, ), index=False)\n (nonshare_submit_df())\nlb increase from 0.620 to 0.643.
\nso it is not only calibration for different protein, it can also be calibration for different group (share vs non share)
", "rawMarkdown": "instead of retraining noshare part with smaller weight for negative samples, i implement in another way. post process the results to give boost to non-share prediction\n\n\n(mathematically, loss weight in bce loss is power in probability, aka temperature scaling)\n\n```\n\t\tnonshare_submit_df = submit_df.copy()\n\t\tnonshare_submit_df.loc[sharing_df.is_share == 0, 'binds'] **= 0.75\n\t\tnonshare_submit_df.to_csv(ensemble_file.replace('.csv', '.nonshare-boost0.75.csv'), index=False)\n\t\tprint(nonshare_submit_df.sum())\n```\n\nlb increase from 0.620 to 0.643.\n\n---\n\nso it is not only calibration for different protein, it can also be calibration for different group (share vs non share)", "votes": 10, "replies": [ { "id": 2809133, "postDate": "2024-05-12T14:46:07.303Z", "content": "So this is the mysterious 'pow(p,n)'? 🤣
\nThis competition is too much tricks and micro-management…the opposite of what I thought it would be.
actually calibration is part of model building in bioinformatics.
\n(they need this to do hypothesis tests)
but it would be incredibly difficult here because of different domain for share and nonshare molecules. But it isn't impossible to do … now non-ML methods like MD and docking would become important (and with active probing) to provide guidance for ML calibration.
\nonce we have full excess to the test data, anything is possible
", "rawMarkdown": "actually calibration is part of model building in bioinformatics.\n(they need this to do hypothesis tests)\n\nbut it would be incredibly difficult here because of different domain for share and nonshare molecules. But it isn't impossible to do ... now non-ML methods like MD and docking would become important (and with active probing) to provide guidance for ML calibration.\n\nonce we have full excess to the test data, anything is possible", "votes": 3 }, { "id": 2809198, "postDate": "2024-05-12T15:23:15.010Z", "content": "I thought it would be like Ribonanza, where we had a massive train set that generalized to test, and we could just build the best model.
\nBut it starts to look more like Novozymes…
I've been thinking of it as half and half for awhile now. Half Novozymes has interesting implications, this thread is touching on a problem I've been thinking about for weeks, actually. If half your predictions are great, and half are an unknown level of terrible… How do you blend? Is docking a subset a way of determining how bad your second model is (and thus calibrating)? Is there some trustworthy way to quantify the comparative predictive power? Do you just guess?
\nI see it as three areas. The \"Ribonanza\", the \"Novozymes\", but also the blending/docking/calibrating area of focus.
\nBut the better the final models, the less the last area matters!
", "rawMarkdown": "I've been thinking of it as half and half for awhile now. Half Novozymes has interesting implications, this thread is touching on a problem I've been thinking about for weeks, actually. If half your predictions are great, and half are an unknown level of terrible... How do you blend? Is docking a subset a way of determining how bad your second model is (and thus calibrating)? Is there some trustworthy way to quantify the comparative predictive power? Do you just guess?\n\nI see it as three areas. The \"Ribonanza\", the \"Novozymes\", but also the blending/docking/calibrating area of focus. \n\nBut the better the final models, the less the last area matters!", "votes": 1 }, { "id": 2809463, "postDate": "2024-05-12T19:06:15.803Z", "content": "The problem is…[Accurate (Ribonanza]]+[Random (Novozymes)] = Random hahaha
\nI mean…yeah you will have a slightly better chance to finish at top if you are top in the first half. But it will still be only a (small as far as we can tell now) chance…sigh. I hate random competitions. It's good I have the LEAP competition; now THAT is a great one.
Oh it's very problematic, yeah. I just enjoy trying to overcome that sort of thing. Lol
", "rawMarkdown": "Oh it's very problematic, yeah. I just enjoy trying to overcome that sort of thing. Lol", "votes": 1 }, { "id": 2809893, "postDate": "2024-05-13T03:44:14.470Z", "content": "Might have to give LEAP a shot, quite unfortunate that this seems to be a \"random\" competition as I was looking for something closer to Ribonanza too.
", "rawMarkdown": "Might have to give LEAP a shot, quite unfortunate that this seems to be a \"random\" competition as I was looking for something closer to Ribonanza too.", "votes": 2 }, { "id": 2809908, "postDate": "2024-05-13T04:03:22.080Z", "content": "LEAP is hard core DL. You just need to build a really good model and be able to train it on large data. It's like Ribonanza on steroids in that aspect. The difference- no bio, no external data/special features like BPP/CapR etc. Also no edges data. So it's even more simple/pure. IDK why it's so unpopular…well outside the fact that the total size of available data is ~490 GB hahaha
", "rawMarkdown": "LEAP is hard core DL. You just need to build a really good model and be able to train it on large data. It's like Ribonanza on steroids in that aspect. The difference- no bio, no external data/special features like BPP/CapR etc. Also no edges data. So it's even more simple/pure. IDK why it's so unpopular...well outside the fact that the total size of available data is ~490 GB hahaha", "votes": 1 }, { "id": 2809924, "postDate": "2024-05-13T04:15:15.743Z", "content": "i don't mind nonshare block. But the distribution of share and nonshare should be similar in public and private test split. This would make our life easier.
\n\"Ideas for calibration: Do 10, 20, 30+ folds of CV with this same problem.\" …
\nThere is alternative solution. Someone have to standardize the split and probing. all kagglers can disclose some non-share statistics and submission score. I think we can find some pattern.
NOTE: the current way to mask out share (set to zero) to probe nonshare is wrong.
\nThis is because if you submit (share=0, nonshare) and (share=0, monotonic function(share)), they would have the same results.
AP is a ranking problem. you basically need to submit share = some good prediction (e.g from public notebook) as a reference, and see how nonshare rank relative to this reference.
", "rawMarkdown": "i don't mind nonshare block. But the distribution of share and nonshare should be similar in public and private test split. This would make our life easier.\n\n---\n\n\"Ideas for calibration: Do 10, 20, 30+ folds of CV with this same problem.\" ...\nThere is alternative solution. Someone have to standardize the split and probing. all kagglers can disclose some non-share statistics and submission score. I think we can find some pattern.\n\nNOTE: the current way to mask out share (set to zero) to probe nonshare is wrong.\nThis is because if you submit (share=0, nonshare) and (share=0, monotonic function(share)), they would have the same results. \n\nAP is a ranking problem. you basically need to submit share = some good prediction (e.g from public notebook) as a reference, and see how nonshare rank relative to this reference. \n\n", "votes": 1 }, { "id": 2810024, "postDate": "2024-05-13T05:29:16.713Z", "content": "There's a lot of ways to do it. But I would caution against focusing mainly on public LB. Or even a (single) standard split. We have a large training dataset, and can look at what generally works on a variety of target non share blocks and what doesn't, instead of just over fitting public LB.
", "rawMarkdown": "There's a lot of ways to do it. But I would caution against focusing mainly on public LB. Or even a (single) standard split. We have a large training dataset, and can look at what generally works on a variety of target non share blocks and what doesn't, instead of just over fitting public LB. " }, { "id": 2810618, "postDate": "2024-05-13T11:01:05.623Z", "content": "i just realise one thing interesting
\nthere about unalbelled 880k test. my training is batch size = 5k
\nthis means 176 batches.
I did a \"thought experiment\". it takes 5 weeks to submit all batches, of which 35% (public) will return with batch-level label (pos fraction) .
\nthen actually we don't need to submit and probe at all.
\nif the batch size is large enough, we definitely have some positive samples.
\nso actually we do have some batch-level label for the unlabelled dataset.
so if we are training with large batch size,
\nwe have sample loss for labelled and batch level loss for unlabelled
\n(something like multiple instant ranking … concept of bag loss)
i haven't did experiments, but i can imaging, this forces the unlabelled score to stretch out during training.
\nnow if we imagine some parametric distribution, e.g. poisson as target, batch loss can be just kl fitting.
currently, my resources can go up to batch-size = 10k . so we can train and calibrate at the same time
", "rawMarkdown": "i just realise one thing interesting\nthere about unalbelled 880k test. my training is batch size = 5k\nthis means 176 batches. \n\nI did a \"thought experiment\". it takes 5 weeks to submit all batches, of which 35% (public) will return with batch-level label (pos fraction) .\n\n---\n\nthen actually we don't need to submit and probe at all.\nif the batch size is large enough, we definitely have some positive samples.\nso actually we **do have some batch-level label for the unlabelled dataset**.\n\nso if we are training with **large batch size**,\n1. all train label pos must rank better than all train labelled neg\n2. some train unlabelled must rank better than all train labelled neg\n\nwe have sample loss for labelled and batch level loss for unlabelled\n(something like multiple instant ranking ... concept of bag loss)\n\n---\ni haven't did experiments, but i can imaging, this forces the unlabelled score to stretch out during training.\nnow if we imagine some parametric distribution, e.g. poisson as target, batch loss can be just kl fitting.\n\ncurrently, my resources can go up to batch-size = 10k . so we can train and calibrate at the same time", "votes": 1 } ] } ] }, { "id": 2807698, "postDate": "2024-05-11T19:45:35Z", "content": "updated my diagram
\n
so we can \"cheat\" by \"stretching the prediction score\" in post processing in the second case.
", "rawMarkdown": "updated my diagram\n\n\nso we can \"cheat\" by \"stretching the prediction score\" in post processing in the second case.", "votes": 7 }, { "id": 2809236, "postDate": "2024-05-12T16:05:37.370Z", "content": "I'm not too happy with the scoring mechanics for this competition. I think it would've been better to score groups and then average those scores. And/or score top 1% or in some way emphasize the importance of getting as many positives into your high confidence region as possible, rather than trying to detangle which of hundreds of thousands of low probability are 'less low'.
\nHOWEVER, we don't know whether this calibration issue is mainly a solvable problem or not. Mostly people didn't realize the issue existed so haven't even tried. Also the private LB will frankly have much bigger problems then merely calibration, heh.
\nIdeas for calibration: Do 10, 20, 30+ folds of CV with this same problem. Do we get similar clumps of (lower and) higher numbers of positive samples? Can we train a model to have any confidence in distinguishing GROUP positive rate? Even if not, can we directly calibrate a large amount of many folds of CV \"non-share\" vs \"share\" predictions and get appropriately less confident predictions on nonshare, and thus the same LB scoring bump? Or does it not work, and the LB bump is only from knowing there's more positive samples in that group?
\nIt's discouraging when you can get a huge LB increase with \"tricks\", but let's not give up hope it can be handled by a smart model :)
", "rawMarkdown": "I'm not too happy with the scoring mechanics for this competition. I think it would've been better to score groups and then average those scores. And/or score top 1% or in some way emphasize the importance of getting as many positives into your high confidence region as possible, rather than trying to detangle which of hundreds of thousands of low probability are 'less low'. \n\nHOWEVER, we don't know whether this calibration issue is mainly a solvable problem or not. Mostly people didn't realize the issue existed so haven't even tried. Also the private LB will frankly have much bigger problems then merely calibration, heh. \n\nIdeas for calibration: Do 10, 20, 30+ folds of CV with this same problem. Do we get similar clumps of (lower and) higher numbers of positive samples? Can we train a model to have any confidence in distinguishing GROUP positive rate? Even if not, can we *directly* calibrate a large amount of many folds of CV \"non-share\" vs \"share\" predictions and get appropriately less confident predictions on nonshare, and thus the same LB scoring bump? Or does it not work, and the LB bump is only from knowing there's more positive samples in that group?\n\nIt's discouraging when you can get a huge LB increase with \"tricks\", but let's not give up hope it can be handled by a smart model :)", "votes": 5, "replies": [ { "id": 2809326, "postDate": "2024-05-12T17:11:14.847Z", "content": "I completely agree. In a realistic drug discovery setting, what we really care about is to maximize the number of true actives in the top 1%. After all, due to cost constraints, only less than a hundred of these compounds would be routinely synthesized off-DNA for pharmacological assays.
\nFurthermore, the winning model will mostly likely be used to screen virtual libraries. Having a model that can rank the best molecules in the top1% would indeed be the most useful, rather than a model that can distinguish a weak binder from a very weak binder.
", "rawMarkdown": "I completely agree. In a realistic drug discovery setting, what we really care about is to maximize the number of true actives in the top 1%. After all, due to cost constraints, only less than a hundred of these compounds would be routinely synthesized off-DNA for pharmacological assays. \n\nFurthermore, the winning model will mostly likely be used to screen virtual libraries. Having a model that can rank the best molecules in the top1% would indeed be the most useful, rather than a model that can distinguish a weak binder from a very weak binder. ", "votes": 8 } ] }, { "id": 2806722, "postDate": "2024-05-11T08:35:09.187Z", "content": "to put into picture, the issue is:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F67f9f78321b75c65228ebbe841d635d1%2FSelection_093.png?generation=1715416504126885&alt=media\")
I think it is a bit different from difficulty. Even all proteins has same difficulty, score calibration will also boost the score by give higher probability to those proteins with more positive molecule candidates. By the way your example are all correctly ordered so score in picture can't represent meaningful probability, if your score is 0.9 then there should be 10% blue dot and 90% red dot nearby.
", "rawMarkdown": "I think it is a bit different from difficulty. Even all proteins has same difficulty, score calibration will also boost the score by give higher probability to those proteins with more positive molecule candidates. By the way your example are all correctly ordered so score in picture can't represent meaningful probability, if your score is 0.9 then there should be 10% blue dot and 90% red dot nearby.", "replies": [ { "id": 2806893, "postDate": "2024-05-11T11:28:49.700Z", "content": "\"so score in picture can't represent meaningful probability, if your score is 0.9 then there should be 10% blue dot and 90% red dot nearby.\"
\nyou can imagine after training a model, you take 10 test samples and make the plot.
\n\"10% blue dot and 90% red dot nearby.\" . this is for train samples.
\nit really depends on how good are your model.
\nI did experiments on dummy data. we can simulate difficulty by adding label noise.
\ni measure ap(each and average micro) on the train data as train learning iteration proceeds.
\nyou can monitor how the ap changes (and also the predict prob distributions) as the model get stronger and become perfect (since it is train data)
I also think that it is better to use the average score of MAPs calculated separately for each target protein as the competition metric. These proteins have somewhat different properties, and thus complex calibration might sometimes be required to optimize the metric currently used. Is there any benefit to calculating these together in this sort of virtual screening?
", "rawMarkdown": "I also think that it is better to use the average score of MAPs calculated separately for each target protein as the competition metric. These proteins have somewhat different properties, and thus complex calibration might sometimes be required to optimize the metric currently used. Is there any benefit to calculating these together in this sort of virtual screening?", "votes": 1 }, { "id": 2810887, "postDate": "2024-05-13T13:46:25.127Z", "content": "So if the unknown ratio is larger in private lb, there could be very large shake up?
", "rawMarkdown": "So if the unknown ratio is larger in private lb, there could be very large shake up?", "votes": 1, "replies": [ { "id": 2810895, "postDate": "2024-05-13T13:51:23.550Z", "content": "Exactly, what I'm skeptical of as well.
", "rawMarkdown": "Exactly, what I'm skeptical of as well. ", "replies": [ { "id": 2810912, "postDate": "2024-05-13T14:00:44.173Z", "content": "shake up is what i call \"kaggle version of short selling\" 卖空 做空
", "rawMarkdown": "shake up is what i call \"kaggle version of short selling\" 卖空 做空", "votes": 2 }, { "id": 2810951, "postDate": "2024-05-13T14:32:51.937Z", "content": "Hilarious analogy! On point thou! 😀
", "rawMarkdown": "Hilarious analogy! On point thou! 😀" } ] }, { "id": 2810933, "postDate": "2024-05-13T14:14:49.047Z", "content": "Yes if it is this case. But based on competition data description- \"To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders\", maybe public noshare part is just the result of an accident. If test positive rate is very different I don't think host will release such misleading information. Personally I will try to collect more hints before make decision.
", "rawMarkdown": "Yes if it is this case. But based on competition data description- \"To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: **roughly 0.5% of examples are classified as binders**\", maybe public noshare part is just the result of an accident. If test positive rate is very different I don't think host will release such misleading information. Personally I will try to collect more hints before make decision.\n", "votes": 1, "replies": [ { "id": 2811070, "postDate": "2024-05-13T15:41:57.983Z", "content": "It would be better asking host directly for clarification upon it.
", "rawMarkdown": "It would be better asking host directly for clarification upon it." } ] } ] }, { "id": 2808802, "postDate": "2024-05-12T12:05:23.933Z", "content": "I have a very silly question to ask but I'll ask it anyway, what are you referring to no-share here? The building blocks that are not in the training set? Or I am getting it all wrong?
", "rawMarkdown": "I have a very silly question to ask but I'll ask it anyway, what are you referring to **no-share** here? The building blocks that are not in the training set? Or I am getting it all wrong? ", "votes": 1, "replies": [ { "id": 2808806, "postDate": "2024-05-12T12:07:47.657Z", "content": "Yes,some test samples with BB not seen in training set.
", "rawMarkdown": "Yes,some test samples with BB not seen in training set.", "votes": 2, "replies": [ { "id": 2808817, "postDate": "2024-05-12T12:12:32.057Z", "content": "Thank you so much for clarifying! And kudos to you for currently being at 1st! :)🥳
", "rawMarkdown": "Thank you so much for clarifying! And kudos to you for currently being at 1st! :)🥳" } ] } ] }, { "id": 2816314, "postDate": "2024-05-16T08:49:52.930Z", "content": "Would also prefer to use average score
", "rawMarkdown": "Would also prefer to use average score", "votes": -1 }, { "id": 2811712, "postDate": "2024-05-13T20:28:37.463Z", "content": "Let me check my understanding, I've been doing research on noshare in silo for quite a while. Tuning the predictions to share part is what seem to happen here (downplaying the noshare predictions) and it works well for public LB. But this will be a mistake to submit as a final prediction as \"true\" private LB is actually 34% noshare and the good generalization there will be important. I assume that competition authors included noshare part in the public test set, but not randomly sampled from noshare part, but mostly sampled positive signals (reason for this is unknown). If 5% of noshare in public has conversion ratio of 4%, when when it will be expanded to private test set - it will get diluted (lets assume only negative samples for no share) to about 0.5%. Which means that average binding ratio is still 0.5% for (private + public) sets. WDYT?
", "rawMarkdown": "Let me check my understanding, I've been doing research on noshare in silo for quite a while. Tuning the predictions to share part is what seem to happen here (downplaying the noshare predictions) and it works well for public LB. But this will be a mistake to submit as a final prediction as \"true\" private LB is actually 34% noshare and the good generalization there will be important. I assume that competition authors included noshare part in the public test set, but not randomly sampled from noshare part, but mostly sampled positive signals (reason for this is unknown). If 5% of noshare in public has conversion ratio of 4%, when when it will be expanded to private test set - it will get diluted (lets assume only negative samples for no share) to about 0.5%. Which means that average binding ratio is still 0.5% for (private + public) sets. WDYT?", "replies": [ { "id": 2811727, "postDate": "2024-05-13T20:55:40.523Z", "content": "Couple quick comments:
\ni have a question. 0.5% of 890k test molecules is 4450.
\nThis is not a large number. would it be easy to verify these 4450 cases using more time consuming methods like MD or docking software?
instead of retraining noshare part with smaller weight for negative samples, i implement in another way. post process the results to give boost to non-share prediction
\n(mathematically, loss weight in bce loss is power in probability, aka temperature scaling)
\n nonshare_submit_df = submit_df()\n nonshare_submit_df **= \n nonshare_submit_df(ensemble_file(, ), index=False)\n (nonshare_submit_df())\nlb increase from 0.620 to 0.643.
\nso it is not only calibration for different protein, it can also be calibration for different group (share vs non share)
", "votes": 10, "replies": [ { "id": 2809133, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-12T14:46:07.303000", "content": "So this is the mysterious 'pow(p,n)'? 🤣
\nThis competition is too much tricks and micro-management…the opposite of what I thought it would be.
actually calibration is part of model building in bioinformatics.
\n(they need this to do hypothesis tests)
but it would be incredibly difficult here because of different domain for share and nonshare molecules. But it isn't impossible to do … now non-ML methods like MD and docking would become important (and with active probing) to provide guidance for ML calibration.
\nonce we have full excess to the test data, anything is possible
", "votes": 3, "replies": [] }, { "id": 2809198, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-12T15:23:15.010000", "content": "I thought it would be like Ribonanza, where we had a massive train set that generalized to test, and we could just build the best model.
\nBut it starts to look more like Novozymes…
I've been thinking of it as half and half for awhile now. Half Novozymes has interesting implications, this thread is touching on a problem I've been thinking about for weeks, actually. If half your predictions are great, and half are an unknown level of terrible… How do you blend? Is docking a subset a way of determining how bad your second model is (and thus calibrating)? Is there some trustworthy way to quantify the comparative predictive power? Do you just guess?
\nI see it as three areas. The \"Ribonanza\", the \"Novozymes\", but also the blending/docking/calibrating area of focus.
\nBut the better the final models, the less the last area matters!
", "votes": 1, "replies": [] }, { "id": 2809463, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-12T19:06:15.803000", "content": "The problem is…[Accurate (Ribonanza]]+[Random (Novozymes)] = Random hahaha
\nI mean…yeah you will have a slightly better chance to finish at top if you are top in the first half. But it will still be only a (small as far as we can tell now) chance…sigh. I hate random competitions. It's good I have the LEAP competition; now THAT is a great one.
Oh it's very problematic, yeah. I just enjoy trying to overcome that sort of thing. Lol
", "votes": 1, "replies": [] }, { "id": 2809893, "author_name": "sroger", "author_url": "", "post_date": "2024-05-13T03:44:14.470000", "content": "Might have to give LEAP a shot, quite unfortunate that this seems to be a \"random\" competition as I was looking for something closer to Ribonanza too.
", "votes": 2, "replies": [] }, { "id": 2809908, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-13T04:03:22.080000", "content": "LEAP is hard core DL. You just need to build a really good model and be able to train it on large data. It's like Ribonanza on steroids in that aspect. The difference- no bio, no external data/special features like BPP/CapR etc. Also no edges data. So it's even more simple/pure. IDK why it's so unpopular…well outside the fact that the total size of available data is ~490 GB hahaha
", "votes": 1, "replies": [] }, { "id": 2809924, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-13T04:15:15.743000", "content": "i don't mind nonshare block. But the distribution of share and nonshare should be similar in public and private test split. This would make our life easier.
\n\"Ideas for calibration: Do 10, 20, 30+ folds of CV with this same problem.\" …
\nThere is alternative solution. Someone have to standardize the split and probing. all kagglers can disclose some non-share statistics and submission score. I think we can find some pattern.
NOTE: the current way to mask out share (set to zero) to probe nonshare is wrong.
\nThis is because if you submit (share=0, nonshare) and (share=0, monotonic function(share)), they would have the same results.
AP is a ranking problem. you basically need to submit share = some good prediction (e.g from public notebook) as a reference, and see how nonshare rank relative to this reference.
", "votes": 1, "replies": [] }, { "id": 2810024, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-05-13T05:29:16.713000", "content": "There's a lot of ways to do it. But I would caution against focusing mainly on public LB. Or even a (single) standard split. We have a large training dataset, and can look at what generally works on a variety of target non share blocks and what doesn't, instead of just over fitting public LB.
", "votes": 0, "replies": [] }, { "id": 2810618, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-13T11:01:05.623000", "content": "i just realise one thing interesting
\nthere about unalbelled 880k test. my training is batch size = 5k
\nthis means 176 batches.
I did a \"thought experiment\". it takes 5 weeks to submit all batches, of which 35% (public) will return with batch-level label (pos fraction) .
\nthen actually we don't need to submit and probe at all.
\nif the batch size is large enough, we definitely have some positive samples.
\nso actually we do have some batch-level label for the unlabelled dataset.
so if we are training with large batch size,
\nwe have sample loss for labelled and batch level loss for unlabelled
\n(something like multiple instant ranking … concept of bag loss)
i haven't did experiments, but i can imaging, this forces the unlabelled score to stretch out during training.
\nnow if we imagine some parametric distribution, e.g. poisson as target, batch loss can be just kl fitting.
currently, my resources can go up to batch-size = 10k . so we can train and calibrate at the same time
", "votes": 1, "replies": [] } ] } ] }, { "id": 2807698, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-11T19:45:35", "content": "updated my diagram
\n
so we can \"cheat\" by \"stretching the prediction score\" in post processing in the second case.
", "votes": 7, "replies": [] }, { "id": 2809236, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-05-12T16:05:37.370000", "content": "I'm not too happy with the scoring mechanics for this competition. I think it would've been better to score groups and then average those scores. And/or score top 1% or in some way emphasize the importance of getting as many positives into your high confidence region as possible, rather than trying to detangle which of hundreds of thousands of low probability are 'less low'.
\nHOWEVER, we don't know whether this calibration issue is mainly a solvable problem or not. Mostly people didn't realize the issue existed so haven't even tried. Also the private LB will frankly have much bigger problems then merely calibration, heh.
\nIdeas for calibration: Do 10, 20, 30+ folds of CV with this same problem. Do we get similar clumps of (lower and) higher numbers of positive samples? Can we train a model to have any confidence in distinguishing GROUP positive rate? Even if not, can we directly calibrate a large amount of many folds of CV \"non-share\" vs \"share\" predictions and get appropriately less confident predictions on nonshare, and thus the same LB scoring bump? Or does it not work, and the LB bump is only from knowing there's more positive samples in that group?
\nIt's discouraging when you can get a huge LB increase with \"tricks\", but let's not give up hope it can be handled by a smart model :)
", "votes": 5, "replies": [ { "id": 2809326, "author_name": "Rced_AI", "author_url": "", "post_date": "2024-05-12T17:11:14.847000", "content": "I completely agree. In a realistic drug discovery setting, what we really care about is to maximize the number of true actives in the top 1%. After all, due to cost constraints, only less than a hundred of these compounds would be routinely synthesized off-DNA for pharmacological assays.
\nFurthermore, the winning model will mostly likely be used to screen virtual libraries. Having a model that can rank the best molecules in the top1% would indeed be the most useful, rather than a model that can distinguish a weak binder from a very weak binder.
", "votes": 8, "replies": [] } ] }, { "id": 2806722, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-11T08:35:09.187000", "content": "to put into picture, the issue is:
\n
I think it is a bit different from difficulty. Even all proteins has same difficulty, score calibration will also boost the score by give higher probability to those proteins with more positive molecule candidates. By the way your example are all correctly ordered so score in picture can't represent meaningful probability, if your score is 0.9 then there should be 10% blue dot and 90% red dot nearby.
", "votes": 0, "replies": [ { "id": 2806893, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-11T11:28:49.700000", "content": "\"so score in picture can't represent meaningful probability, if your score is 0.9 then there should be 10% blue dot and 90% red dot nearby.\"
\nyou can imagine after training a model, you take 10 test samples and make the plot.
\n\"10% blue dot and 90% red dot nearby.\" . this is for train samples.
\nit really depends on how good are your model.
\nI did experiments on dummy data. we can simulate difficulty by adding label noise.
\ni measure ap(each and average micro) on the train data as train learning iteration proceeds.
\nyou can monitor how the ap changes (and also the predict prob distributions) as the model get stronger and become perfect (since it is train data)
I also think that it is better to use the average score of MAPs calculated separately for each target protein as the competition metric. These proteins have somewhat different properties, and thus complex calibration might sometimes be required to optimize the metric currently used. Is there any benefit to calculating these together in this sort of virtual screening?
", "votes": 1, "replies": [] }, { "id": 2810887, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-05-13T13:46:25.127000", "content": "So if the unknown ratio is larger in private lb, there could be very large shake up?
", "votes": 1, "replies": [ { "id": 2810895, "author_name": "AC", "author_url": "", "post_date": "2024-05-13T13:51:23.550000", "content": "Exactly, what I'm skeptical of as well.
", "votes": 0, "replies": [ { "id": 2810912, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-13T14:00:44.173000", "content": "shake up is what i call \"kaggle version of short selling\" 卖空 做空
", "votes": 2, "replies": [] }, { "id": 2810951, "author_name": "AC", "author_url": "", "post_date": "2024-05-13T14:32:51.937000", "content": "Hilarious analogy! On point thou! 😀
", "votes": 0, "replies": [] } ] }, { "id": 2810933, "author_name": "Ruby", "author_url": "", "post_date": "2024-05-13T14:14:49.047000", "content": "Yes if it is this case. But based on competition data description- \"To test generalizability, the test set contains building blocks that are not in the training set. These datasets are very imbalanced: roughly 0.5% of examples are classified as binders\", maybe public noshare part is just the result of an accident. If test positive rate is very different I don't think host will release such misleading information. Personally I will try to collect more hints before make decision.
", "votes": 1, "replies": [ { "id": 2811070, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-05-13T15:41:57.983000", "content": "It would be better asking host directly for clarification upon it.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2808802, "author_name": "AC", "author_url": "", "post_date": "2024-05-12T12:05:23.933000", "content": "I have a very silly question to ask but I'll ask it anyway, what are you referring to no-share here? The building blocks that are not in the training set? Or I am getting it all wrong?
", "votes": 1, "replies": [ { "id": 2808806, "author_name": "Ruby", "author_url": "", "post_date": "2024-05-12T12:07:47.657000", "content": "Yes,some test samples with BB not seen in training set.
", "votes": 2, "replies": [ { "id": 2808817, "author_name": "AC", "author_url": "", "post_date": "2024-05-12T12:12:32.057000", "content": "Thank you so much for clarifying! And kudos to you for currently being at 1st! :)🥳
", "votes": 0, "replies": [] } ] } ] }, { "id": 2816314, "author_name": "Natarajan Vijaikumar", "author_url": "", "post_date": "2024-05-16T08:49:52.930000", "content": "Would also prefer to use average score
", "votes": -1, "replies": [] }, { "id": 2811712, "author_name": "yamu_duck", "author_url": "", "post_date": "2024-05-13T20:28:37.463000", "content": "Let me check my understanding, I've been doing research on noshare in silo for quite a while. Tuning the predictions to share part is what seem to happen here (downplaying the noshare predictions) and it works well for public LB. But this will be a mistake to submit as a final prediction as \"true\" private LB is actually 34% noshare and the good generalization there will be important. I assume that competition authors included noshare part in the public test set, but not randomly sampled from noshare part, but mostly sampled positive signals (reason for this is unknown). If 5% of noshare in public has conversion ratio of 4%, when when it will be expanded to private test set - it will get diluted (lets assume only negative samples for no share) to about 0.5%. Which means that average binding ratio is still 0.5% for (private + public) sets. WDYT?
", "votes": 0, "replies": [ { "id": 2811727, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-05-13T20:55:40.523000", "content": "Couple quick comments:
\ni have a question. 0.5% of 890k test molecules is 4450.
\nThis is not a large number. would it be easy to verify these 4450 cases using more time consuming methods like MD or docking software?