![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F4c4e9dab30849fcc8e874e0fceea3a58%2FSelection_129.png?generation=1715910346727473&alt=media\")
\ni forget to snapshot the results and LB before the metric change to do comparison and analysis.
\nThis is lesson learned.
i will need to redo many of the experiments.
\nthe update has made the previous results obsolete.
this will update as i go along …. please wait.
\nthe change in metric and update confirm the following ….
\n(1) external data help:
\nsEH submission only: external : 0.032, kaggle only: 0.026
(2) models can overfit on training on share. (i note that some models improve on share, while nonshare get significantly worse). sometimes weak model generalized better (share =0.6, nonshare = 0.1) compared to strong ones (share=0.8, nonshare=0.01) this is because you don't have nonshare in training set and high negative class means learning is classifying unknown as negative.
\n(3) the ranking of xgboost, cnn1d, chermberta, molformer, gnn has changed! This is a surprised for me. This methods that they perform differently for different new subsets.
\n(4) because of (2)(3), you can get better LB if you submit different models (e.g. train with share data but different level of regularization) for share and nonshare.
\nbut the kaggle objective is to get generalized model (i.e. one model to take care of all). I need to take of how to achieve this later (e.g. distillation/fusion to one model, mixture of experts, etc)
\n
\ni forget to snapshot the results and LB before the metric change to do comparison and analysis.
\nThis is lesson learned.
i will need to redo many of the experiments.
\nthe update has made the previous results obsolete.
this will update as i go along …. please wait.
\nthe change in metric and update confirm the following ….
\n(1) external data help:
\nsEH submission only: external : 0.032, kaggle only: 0.026
(2) models can overfit on training on share. (i note that some models improve on share, while nonshare get significantly worse). sometimes weak model generalized better (share =0.6, nonshare = 0.1) compared to strong ones (share=0.8, nonshare=0.01) this is because you don't have nonshare in training set and high negative class means learning is classifying unknown as negative.
\n(3) the ranking of xgboost, cnn1d, chermberta, molformer, gnn has changed! This is a surprised for me. This methods that they perform differently for different new subsets.
\n(4) because of (2)(3), you can get better LB if you submit different models (e.g. train with share data but different level of regularization) for share and nonshare.
\nbut the kaggle objective is to get generalized model (i.e. one model to take care of all). I need to take of how to achieve this later (e.g. distillation/fusion to one model, mixture of experts, etc)
\n
\ni forget to snapshot the results and LB before the metric change to do comparison and analysis.
\nThis is lesson learned.
we are back to where we start!
\nconsider the \"nonshare molecule\". let's split by building blocks bb1.
\nNow if the nonshare bb1 is similar to the train share bb1, we would have the same predict distribution and good results.
\nif they are different, nonshare bb has very low scores … but the ranking \"may be preserved\" /
confused? check the code below:
\n test cases as \np=npz your model\nt=npz\nnonshare_idx=npz row the reduced train csv file\nBRD4_non = (t, )\nHSA_non = (t, )\nsEH_non = (t, )\navg_non = (BRD4_non + HSA_non + sEH_non) / \n\n\n\n\n\n\n results:\navg_non : \nBRD4_non : \nHSA_non : \nsEH_non : \n each block\n\nreduce_df = pd()\nnonshare_df = reduce_df\nnonshare_df=\nnonshare_df=\nnonshare_df=\n\ngb = nonshare_df()\nBRD4_bb = \nHSA_bb = \nsEH_bb = \n bb,df gb:\n (df)\n = df]\n t = df]\n BRD4 = (t, )\n HSA = (t, )\n sEH = (t, )\n BRD4_bb(BRD4)\n HSA_bb(HSA)\n sEH_bb(sEH)\n ((df), bb,BRD4,HSA,sEH)\nzz=\n\n\n)\n)\n)\n)\n\n\n results:\navg: \n\n\n\nput it simply:
\na note about nonshare triazine molecules in public/private test set:
\n \n \n\n \n \n \n \n\n \n \n \n \n\n\n\n \n\n \n \n \n \n\n \n \n \n \nwe are back to where we start!
\nconsider the \"nonshare molecule\". let's split by building blocks bb1.
\nNow if the nonshare bb1 is similar to the train share bb1, we would have the same predict distribution and good results.
\nif they are different, nonshare bb has very low scores … but the ranking \"may be preserved\" /
confused? check the code below:
\n test cases as \np=npz your model\nt=npz\nnonshare_idx=npz row the reduced train csv file\nBRD4_non = (t, )\nHSA_non = (t, )\nsEH_non = (t, )\navg_non = (BRD4_non + HSA_non + sEH_non) / \n\n\n\n\n\n\n results:\navg_non : \nBRD4_non : \nHSA_non : \nsEH_non : \n each block\n\nreduce_df = pd()\nnonshare_df = reduce_df\nnonshare_df=\nnonshare_df=\nnonshare_df=\n\ngb = nonshare_df()\nBRD4_bb = \nHSA_bb = \nsEH_bb = \n bb,df gb:\n (df)\n = df]\n t = df]\n BRD4 = (t, )\n HSA = (t, )\n sEH = (t, )\n BRD4_bb(BRD4)\n HSA_bb(HSA)\n sEH_bb(sEH)\n ((df), bb,BRD4,HSA,sEH)\nzz=\n\n\n)\n)\n)\n)\n\n\n results:\navg: \n\n\n\nput it simply:
\na note about nonshare triazine molecules in public/private test set:
\n \n \n\n \n \n \n \n\n \n \n \n \n\n\n\n \n\n \n \n \n \n\n \n \n \n \n