![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F18c0f3c299addf7c9ae36692d2545c19%2FSelection_080.png?generation=1715154492163145&alt=media\")
UPDATED!! the scoring metric has been updated. so some strategy and discussion may not be valid anymore !!!
\n
i made a third submission:
\n =model2_submit_df.copy()\nreplaced.loc[nonshare,]=model1_submit_df.loc[nonshare,]\n\n\nlbscore = .\nimplications:
\nUPDATED!! the scoring metric has been updated. so some strategy and discussion may not be valid anymore !!!
\n
i made a third submission:
\n =model2_submit_df.copy()\nreplaced.loc[nonshare,]=model1_submit_df.loc[nonshare,]\n\n\nlbscore = .\nimplications:
\nmaybe very good news?
\numap of ecfp fingerprint of building blocks
\nblue : share bb
\nred : nonshare bb
\ngrey: external bb (external data from my other post)
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fab4bcf5987371c81571a1b677ee11ee4%2FSelection_082.png?generation=1715219732483573&alt=media\")
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fad5c52f7a81c1c28a1e7f5ed93b90d87%2FSelection_083.png?generation=1715219744081586&alt=media\")
The external data is the sEH data you posted about weeks(?) ago?
", "rawMarkdown": "The external data is the sEH data you posted about weeks(?) ago? ", "replies": [ { "id": 2802419, "postDate": "2024-05-09T03:32:53.933Z", "content": "yes. if it gives binding affinity as readout values (not 0,1 label).
\nif you have ideas how to use that, please let me know.
right now i can only think of ranking loss for external data
\nupdate: i figure it out. there is overlap between blue and gray that can used to transform targets of different dataset
", "rawMarkdown": "yes. if it gives binding affinity as readout values (not 0,1 label).\nif you have ideas how to use that, please let me know.\n\nright now i can only think of ranking loss for external data\n\nupdate: i figure it out. there is overlap between blue and gray that can used to transform targets of different dataset" }, { "id": 2802425, "postDate": "2024-05-09T03:35:23.703Z", "content": "\"The 133M small molecule library used here, AMA014, was provided by AlphaMa\"
\nit is probably easy to find compounds similar to AMA014
", "rawMarkdown": "\"The 133M small molecule library used here, AMA014, was provided by AlphaMa\"\n\nit is probably easy to find compounds similar to AMA014" }, { "id": 2803330, "postDate": "2024-05-09T12:33:12.993Z", "content": "You can convert to binary with all non-zero label = 1. That's actually pretty equivalent to the train data of this competition, the external data is just giving more info that we don't have access to with the competition hosted data.
", "rawMarkdown": "You can convert to binary with all non-zero label = 1. That's actually pretty equivalent to the train data of this competition, the external data is just giving more info that we don't have access to with the competition hosted data. " } ] } ] }, { "id": 2812021, "postDate": "2024-05-14T04:20:55.247Z", "content": "i have new results and prove that my observations are correct:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0e4305dd56d5c61c6561cbccb5ad9617%2FSelection_112.png?generation=1715660177707043&alt=media\")
if you perform worse than other kagglers who you using the model architecture and same CV, it could be because of the nonshare part. hence those top in the leaderboard has better nonshare block score then you (either via training or artificial adjusting). I think top 20 kagglers have now about same share lb score about 0.595.
\ni think the top peforming cv of share is about 0.73 (one model)
", "rawMarkdown": "i have new results and prove that my observations are correct:\n\n\n\n- if you overfit share blocks, nonshare blocks can get worse\n- CV/LB of share block correlates very strong.\n\nif you perform worse than other kagglers who you using the model architecture and same CV, it could be because of the nonshare part. hence those top in the leaderboard has better nonshare block score then you (either via training or artificial adjusting). I think top 20 kagglers have now about same share lb score about 0.595.\n\ni think the top peforming cv of share is about 0.73 (one model)", "votes": 1 }, { "id": 2800527, "postDate": "2024-05-08T08:08:06.530Z", "content": "while it is difficult to find external data that has pos labels, i think it is easier to find those with neg labels. can anyone advised?
", "rawMarkdown": "while it is difficult to find external data that has pos labels, i think it is easier to find those with neg labels. can anyone advised?", "votes": 1 }, { "id": 2816163, "postDate": "2024-05-16T07:33:53.090Z", "content": "![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F481be0a0f5cf01848f5f1f9506254401%2FSelection_126.png?generation=1715844792449133&alt=media\")
YES!!! training with external dataset improves nonshare LB, but by a little
", "rawMarkdown": "\n\nYES!!! training with external dataset improves nonshare LB, but by a little", "votes": 2 }, { "id": 2803679, "postDate": "2024-05-09T15:36:23.593Z", "content": "at first i though it is the question if we can generalised, i am wrong.
\nit is whether we can detect at all !!!
\n(or maybe external data protein segment is different, or experiment setting, etc … let's wait for the lb score)
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F13b1bde7ec5435bcb59b8356a8012df8%2FSelection_092.png?generation=1715309251814154&alt=media\")
i have to change my strategy
", "rawMarkdown": "at first i though it is the question if we can generalised, i am wrong.\nit is whether we can detect at all !!!\n(or maybe external data protein segment is different, or experiment setting, etc ... let's wait for the lb score)\n \n\n\ni have to change my strategy", "votes": 2, "replies": [ { "id": 2803782, "postDate": "2024-05-09T16:05:22.747Z", "content": "That's your score matrix for train on one dataset, predict on both? Neither dataset is able to predict the other hardly at all, it appears? That was just with xgb on ECFP?
\nYikes that's low though
", "rawMarkdown": "That's your score matrix for train on one dataset, predict on both? Neither dataset is able to predict the other hardly at all, it appears? That was just with xgb on ECFP?\n\nYikes that's low though", "votes": 1, "replies": [ { "id": 2803789, "postDate": "2024-05-09T16:05:55.900Z", "content": "I wish it surprised me more, lol.
", "rawMarkdown": "I wish it surprised me more, lol. ", "votes": 1 }, { "id": 2803801, "postDate": "2024-05-09T16:08:53.423Z", "content": "\" Neither dataset is able to predict the other hardly at all, it appears? \"
\nThat is true.
the lb score confirms it.
\nif that is true then private score will be ….
Also even if your score had been high, this could've been true about the other two proteins which may not have external data. Each protein may behave differently in terms of generalization, is what I would expect.
\nThe state of the art seems more about getting the best hits within your top 10% or 1% of guesses. But that can be achieved and still get a bad MAP, I believe. End of competition reveal will be interesting, for sure!
", "rawMarkdown": "Also even if your score had been high, this could've been true about the other two proteins which may not have external data. Each protein may behave differently in terms of generalization, is what I would expect. \n\nThe state of the art seems more about getting the best hits within your top 10% or 1% of guesses. But that can be achieved and still get a bad MAP, I believe. End of competition reveal will be interesting, for sure!", "votes": 1 }, { "id": 2803811, "postDate": "2024-05-09T16:18:21.357Z", "content": "there are some catch:
\nHello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can you help? Connect on rajivkjs@gmail.com
", "rawMarkdown": "Hello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can you help? Connect on rajivkjs@gmail.com" }, { "id": 2801196, "postDate": "2024-05-08T14:36:34.490Z", "content": "By sharing blocks, you mean building blocks that are in common between the training set and the test set? And nonsharing means building blocks that are unique to the test set?
", "rawMarkdown": "By sharing blocks, you mean building blocks that are in common between the training set and the test set? And nonsharing means building blocks that are unique to the test set? ", "replies": [ { "id": 2801228, "postDate": "2024-05-08T14:50:15.843Z", "content": "yes, you are correct
", "rawMarkdown": "yes, you are correct", "votes": 1 } ] }, { "id": 2800488, "postDate": "2024-05-08T07:56:51.167Z", "content": "@hengck23 yes
", "rawMarkdown": "@hengck23 yes" }, { "id": 2816887, "postDate": "2024-05-16T15:23:11.633Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2814672, "postDate": "2024-05-15T13:36:42.667Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 2802312, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-09T01:56:57.627000", "content": "maybe very good news?
\numap of ecfp fingerprint of building blocks
\nblue : share bb
\nred : nonshare bb
\ngrey: external bb (external data from my other post)
The external data is the sEH data you posted about weeks(?) ago?
", "votes": 0, "replies": [ { "id": 2802419, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-09T03:32:53.933000", "content": "yes. if it gives binding affinity as readout values (not 0,1 label).
\nif you have ideas how to use that, please let me know.
right now i can only think of ranking loss for external data
\nupdate: i figure it out. there is overlap between blue and gray that can used to transform targets of different dataset
", "votes": 0, "replies": [] }, { "id": 2802425, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-09T03:35:23.703000", "content": "\"The 133M small molecule library used here, AMA014, was provided by AlphaMa\"
\nit is probably easy to find compounds similar to AMA014
", "votes": 0, "replies": [] }, { "id": 2803330, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-05-09T12:33:12.993000", "content": "You can convert to binary with all non-zero label = 1. That's actually pretty equivalent to the train data of this competition, the external data is just giving more info that we don't have access to with the competition hosted data.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2812021, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-14T04:20:55.247000", "content": "i have new results and prove that my observations are correct:
\n
if you perform worse than other kagglers who you using the model architecture and same CV, it could be because of the nonshare part. hence those top in the leaderboard has better nonshare block score then you (either via training or artificial adjusting). I think top 20 kagglers have now about same share lb score about 0.595.
\ni think the top peforming cv of share is about 0.73 (one model)
", "votes": 1, "replies": [] }, { "id": 2800527, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-08T08:08:06.530000", "content": "while it is difficult to find external data that has pos labels, i think it is easier to find those with neg labels. can anyone advised?
", "votes": 1, "replies": [] }, { "id": 2816163, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-16T07:33:53.090000", "content": "
YES!!! training with external dataset improves nonshare LB, but by a little
", "votes": 2, "replies": [] }, { "id": 2803679, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-09T15:36:23.593000", "content": "at first i though it is the question if we can generalised, i am wrong.
\nit is whether we can detect at all !!!
\n(or maybe external data protein segment is different, or experiment setting, etc … let's wait for the lb score)
i have to change my strategy
", "votes": 2, "replies": [ { "id": 2803782, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-05-09T16:05:22.747000", "content": "That's your score matrix for train on one dataset, predict on both? Neither dataset is able to predict the other hardly at all, it appears? That was just with xgb on ECFP?
\nYikes that's low though
", "votes": 1, "replies": [ { "id": 2803789, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-05-09T16:05:55.900000", "content": "I wish it surprised me more, lol.
", "votes": 1, "replies": [] }, { "id": 2803801, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-09T16:08:53.423000", "content": "\" Neither dataset is able to predict the other hardly at all, it appears? \"
\nThat is true.
the lb score confirms it.
\nif that is true then private score will be ….
Also even if your score had been high, this could've been true about the other two proteins which may not have external data. Each protein may behave differently in terms of generalization, is what I would expect.
\nThe state of the art seems more about getting the best hits within your top 10% or 1% of guesses. But that can be achieved and still get a bad MAP, I believe. End of competition reveal will be interesting, for sure!
", "votes": 1, "replies": [] }, { "id": 2803811, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-09T16:18:21.357000", "content": "there are some catch:
\nHello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can you help? Connect on rajivkjs@gmail.com
", "votes": 0, "replies": [] }, { "id": 2801196, "author_name": "KirkDCO", "author_url": "", "post_date": "2024-05-08T14:36:34.490000", "content": "By sharing blocks, you mean building blocks that are in common between the training set and the test set? And nonsharing means building blocks that are unique to the test set?
", "votes": 0, "replies": [ { "id": 2801228, "author_name": "hengck23", "author_url": "", "post_date": "2024-05-08T14:50:15.843000", "content": "yes, you are correct
", "votes": 1, "replies": [] } ] }, { "id": 2800488, "author_name": "Shubham Kale", "author_url": "", "post_date": "2024-05-08T07:56:51.167000", "content": "@hengck23 yes
", "votes": 0, "replies": [] }, { "id": 2816887, "author_name": "", "author_url": "", "post_date": "2024-05-16T15:23:11.633000", "content": "", "votes": 0, "replies": [] }, { "id": 2814672, "author_name": "", "author_url": "", "post_date": "2024-05-15T13:36:42.667000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2800473": "**UPDATED!! the scoring metric has been updated. so some strategy and discussion may not be valid anymore !!!**\n\n\n\ni made a third submission:\n```\nreplaced =model2_submit_df.copy()\nreplaced.loc[nonshare,'binds']=model1_submit_df.loc[nonshare,'binds']\n# this is actually not very correct because of different calibration of the two models\n\nlbscore = 0.612\n```\n\nimplications:\n- nonsharing block indeed contribute to public LB score\n- be care in validation and interpretation of local and public scores", "2802312": "maybe very good news?\n\numap of ecfp fingerprint of building blocks\nblue : share bb\nred : nonshare bb\ngrey: external bb (external data from my other post)\n\n\n\n\n", "2812021": "i have new results and prove that my observations are correct:\n\n\n\n- if you overfit share blocks, nonshare blocks can get worse\n- CV/LB of share block correlates very strong.\n\nif you perform worse than other kagglers who you using the model architecture and same CV, it could be because of the nonshare part. hence those top in the leaderboard has better nonshare block score then you (either via training or artificial adjusting). I think top 20 kagglers have now about same share lb score about 0.595.\n\ni think the top peforming cv of share is about 0.73 (one model)", "2800527": "while it is difficult to find external data that has pos labels, i think it is easier to find those with neg labels. can anyone advised?", "2816163": "\n\nYES!!! training with external dataset improves nonshare LB, but by a little", "2803679": "at first i though it is the question if we can generalised, i am wrong.\nit is whether we can detect at all !!!\n(or maybe external data protein segment is different, or experiment setting, etc ... let's wait for the lb score)\n \n\n\ni have to change my strategy", "2812070": "Hello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can you help? Connect on rajivkjs@gmail.com", "2801196": "By sharing blocks, you mean building blocks that are in common between the training set and the test set? And nonsharing means building blocks that are unique to the test set? ", "2800488": "@hengck23 yes", "2816887": "", "2814672": "" } }