{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":67356,"databundleVersionId":8006601,"sourceType":"competition"},{"sourceId":8275617,"sourceType":"datasetVersion","datasetId":4914065},{"sourceId":8355401,"sourceType":"datasetVersion","datasetId":4964802},{"sourceId":8373479,"sourceType":"datasetVersion","datasetId":4977435},{"sourceId":8375131,"sourceType":"datasetVersion","datasetId":4979606}],"dockerImageVersionId":30514,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Design The tokens?\n\nIn ahmedelfazouan [datasets](https://www.kaggle.com/datasets/ahmedelfazouan/belka-enc-dataset), he has already completed the encoding. But my question is, how do you design the token to be chemical and how do you code it faster.\n\n**Here is the dataset after re-encode:**\nhttps://www.kaggle.com/datasets/horikitasaku/belka-chemical-encoding/data\n\n\n# ！I'm not a chemistry major, so please let me know if there are any mistakes！","metadata":{}},{"cell_type":"markdown","source":"# First, let's take a look at some SMILES symbols in our data.\n\n## Uppercase non-aromatic Atoms:\n    These are atoms typically found in organic molecules and represent the core elements. Each uppercase letter corresponds to a specific element\n    \n    B, C, N, O, S, F, Cl, Br, I, H\n## Downcase aromatic Atoms:\n     These are aromatic atoms, typically representing heteroatoms in aromatic rings\n\n    s, o, c, n, i\n    \n## Bonds:\n    \n### Single, Double, triple bonds and non-bond: \n       These symbols represent the types of bonds between atoms. Where . Does not appear in molecule_smiles.\n        \n        -\n        \n        =\n        \n        #\n        \n        .\n    \n### Stereochemistry(Single bonds adjacent to double bonds): \n       This symbol indicates the stereochemistry of single bonds adjacent to double bonds\n        \n        /\n## Branching:\n    These symbols denote branching in the molecular structure\n\n    (\n    \n    )\n## Specified Atomic Mass:\n    [\n    \n    ]\n## Configuration around tetrahedral centers:\n    \n    These symbols represent the configuration around tetrahedral centers\n    @ is Clockwise configuration and @@ is Counterclockwise configuration\n    \n    @\n    \n    @@\n## Charge:\n\n    -\n    \n    +\n## Amount:\n\n    These symbols represent the amount of atoms or groups present\n    \n    1~9","metadata":{}},{"cell_type":"markdown","source":"## That said, there are some of them that we can't encode with simple iterations (Or say, difficult.). For example, Ci, we can't iterate over C and i to encode. This is patently false.\n## If you use regular expressions, the speed will be very slow. ↓\n![](https://pbs.twimg.com/media/GNEy0R5bYAAr9LA?format=png&name=900x900)","metadata":{}},{"cell_type":"markdown","source":"# So we choose","metadata":{}},{"cell_type":"markdown","source":"## The ```tokenizer``` in transformer, because it's written in RUST, is very fast.In this case we can also add multithreading, Time will be very short.(I didn't do it in kaggle, but it's easy to do it in joblib.)\n## And at the same time we can do batch processing↓\n\n![](https://pbs.twimg.com/media/GNIH_dBbIAIvU2-?format=png&name=900x900)\n\n## with joblib (local)↓ Time cut to less than 10 minutes❗❗\n![](https://pbs.twimg.com/media/GNMQNL0agAAP6Ca?format=png&name=900x900)","metadata":{}},{"cell_type":"code","source":"!pip install fastparquet -q","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-05-10T09:14:32.369600Z","iopub.execute_input":"2024-05-10T09:14:32.370722Z","iopub.status.idle":"2024-05-10T09:14:45.069714Z","shell.execute_reply.started":"2024-05-10T09:14:32.370688Z","shell.execute_reply":"2024-05-10T09:14:45.068785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nimport os\nimport pickle\nimport random\nimport joblib\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-05-10T11:45:13.158495Z","iopub.execute_input":"2024-05-10T11:45:13.158860Z","iopub.status.idle":"2024-05-10T11:45:13.184941Z","shell.execute_reply.started":"2024-05-10T11:45:13.158834Z","shell.execute_reply":"2024-05-10T11:45:13.184070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n    predict = True\n    PREPROCESS = False\n    EPOCHS = 18\n    BATCH_SIZE = 4096\n    LR = 1e-3\n    WD = 0.05\n\n    NBR_FOLDS = 15\n    SELECTED_FOLDS = [0]\n\n    SEED = 2024","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:13.302812Z","iopub.execute_input":"2024-05-10T11:45:13.303510Z","iopub.status.idle":"2024-05-10T11:45:13.308323Z","shell.execute_reply.started":"2024-05-10T11:45:13.303478Z","shell.execute_reply":"2024-05-10T11:45:13.307323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\ndef set_seeds(seed):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    tf.random.set_seed(seed)\n    np.random.seed(seed)\n\nset_seeds(seed=CFG.SEED)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-05-10T11:45:14.458285Z","iopub.execute_input":"2024-05-10T11:45:14.458949Z","iopub.status.idle":"2024-05-10T11:45:22.592304Z","shell.execute_reply.started":"2024-05-10T11:45:14.458915Z","shell.execute_reply":"2024-05-10T11:45:22.591473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=\"local\") # \"local\" for 1VM TPU\n    strategy = tf.distribute.TPUStrategy(tpu)\n    print(\"Running on TPU\")\n    print(\"REPLICAS: \", strategy.num_replicas_in_sync)\n    all_data = True\nexcept tf.errors.NotFoundError:\n    all_data = False\n\n    strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n    print(\"Not on TPU\")","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:22.594059Z","iopub.execute_input":"2024-05-10T11:45:22.594675Z","iopub.status.idle":"2024-05-10T11:45:25.028169Z","shell.execute_reply.started":"2024-05-10T11:45:22.594648Z","shell.execute_reply":"2024-05-10T11:45:25.027199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"code","source":"!pip install tokenizers\n!pip install transformers","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:25.029338Z","iopub.execute_input":"2024-05-10T11:45:25.029694Z","iopub.status.idle":"2024-05-10T11:45:47.785315Z","shell.execute_reply.started":"2024-05-10T11:45:25.029667Z","shell.execute_reply":"2024-05-10T11:45:47.784080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create a tokenizer\n\nWe use Tokenizer and PreTrainedTokenizerFast,\n\nFirst create the dictionary enc.","metadata":{}},{"cell_type":"code","source":"from tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\nfrom tokenizers.models import WordLevel,BPE\n\nfrom tokenizers.pre_tokenizers import Whitespace,Split,ByteLevel\n\nfrom tokenizers.normalizers import Lowercase, NFKC\n\nenc = { '[PAD]':0,\n        'Br':1, 'C':2, 'N':3, 'O':4, 'H':5, 'S':6, 'F':7, 'Cl':8, 'B':9, 'I':10, \n        's':11,'o':12, 'c':13, 'n':14, 'i':15, # is Atomic\n        '.':16 ,'=':17 ,'#':18, # bond\n        '/':19, # direction\n        '-':20, '+': 21, # charge\n        '[':22,']':23, # Atomic mass\n        '(':24,')':25, # Branches\n        '@@':26, '@':27, # tetrahedron\n        '1':28,'2':29,'3':30,'4':31,'5':32,'6':33,'7':34,'8':35,'9':36\n      }","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:47.788123Z","iopub.execute_input":"2024-05-10T11:45:47.788472Z","iopub.status.idle":"2024-05-10T11:45:48.135904Z","shell.execute_reply.started":"2024-05-10T11:45:47.788442Z","shell.execute_reply":"2024-05-10T11:45:48.135034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Then create a Tokenizer with BPE\nPay attention to ```merges```. In the case that both @ and @@ exist, we should find @@ first, then @\n\nthen converted to PreTrainedTokenizerFast\n\nCreate special tokens at the same time\n\nAlso, we should set [Dy] to sep_token","metadata":{}},{"cell_type":"code","source":"tokenizer = Tokenizer(BPE(vocab=enc, unk_token=\"[UNK]\",merges=[('@','@')]))\n\ntokenizer_fast = PreTrainedTokenizerFast(tokenizer_object=tokenizer)\ntokenizer_fast.add_special_tokens({'pad_token': '[PAD]','sep_token':'[Dy]'})\ntokenizer_fast.add_special_tokens({'additional_special_tokens':['Br','Cl']})\n\nlen(set(enc.values()))","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.137115Z","iopub.execute_input":"2024-05-10T11:45:48.137425Z","iopub.status.idle":"2024-05-10T11:45:48.163452Z","shell.execute_reply.started":"2024-05-10T11:45:48.137398Z","shell.execute_reply":"2024-05-10T11:45:48.162460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenizer_fast.all_special_ids","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.166302Z","iopub.execute_input":"2024-05-10T11:45:48.166661Z","iopub.status.idle":"2024-05-10T11:45:48.179939Z","shell.execute_reply.started":"2024-05-10T11:45:48.166627Z","shell.execute_reply":"2024-05-10T11:45:48.179117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Then implement batch spliting.","metadata":{}},{"cell_type":"code","source":"# def compile_model():\n#     with strategy.scope():\n#         model = tf.keras.Sequential([\n#             tf.keras.layers.Dense(64, input_shape=(64,))\n#         ])\n#         model.compile(metrics=['accuracy'])\n#         return model\n# model = compile_model()\n# dummy_input = tf.random.uniform((1, 64))\n# output = model(dummy_input)\n# print(\"TPU shape:\", output.shape)\n","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.181187Z","iopub.execute_input":"2024-05-10T11:45:48.181532Z","iopub.status.idle":"2024-05-10T11:45:48.190890Z","shell.execute_reply.started":"2024-05-10T11:45:48.181500Z","shell.execute_reply":"2024-05-10T11:45:48.190065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from joblib import Parallel, delayed\n\ndef encode_smiles_batch(smiles_chunk, max_length=142, batch_size=32):\n    encoded_batch = tokenizer_fast.batch_encode_plus(\n        smiles_chunk,\n        max_length=max_length,\n        padding='max_length',\n        return_tensors='np',\n        truncation=True\n    )['input_ids']\n\n    return encoded_batch.astype(np.int8)\n\ndef encode_smiles_parallel(smiles, max_length=142, batch_size=4096*8):\n    smiles_chunks = [smiles[i:i + batch_size] for i in range(0, len(smiles), batch_size)]\n    encoded_chunks = Parallel(n_jobs=-1)(delayed(encode_smiles_batch)(chunk, max_length, batch_size) for chunk in tqdm(smiles_chunks))\n    \n    encoded_smiles = np.concatenate(encoded_chunks, axis=0)\n    \n    return encoded_smiles\n","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.192187Z","iopub.execute_input":"2024-05-10T11:45:48.193046Z","iopub.status.idle":"2024-05-10T11:45:48.203121Z","shell.execute_reply.started":"2024-05-10T11:45:48.192990Z","shell.execute_reply":"2024-05-10T11:45:48.202223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TEST","metadata":{}},{"cell_type":"code","source":"tokenizer_fast.batch_encode_plus(['BrBC#CCOc1ccc(CNc2nc(Nc3cc(-c4ccc(Cl)cc4)sc3C(=O)OC)nc(N[C@@H](CC#C)CC(=O-)N[Dy])n2)cc1'],max_length=142,padding='max_length')\n","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.204159Z","iopub.execute_input":"2024-05-10T11:45:48.204465Z","iopub.status.idle":"2024-05-10T11:45:48.221640Z","shell.execute_reply.started":"2024-05-10T11:45:48.204440Z","shell.execute_reply":"2024-05-10T11:45:48.220800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## We can see that Br, Cl and @@ have been successfully separated","metadata":{}},{"cell_type":"code","source":"tokenizer_fast.tokenize('BrBC#CCOc1ccc(CNc2nc(Nc3cc(-c4ccc(Cl)cc4)sc3C(=O)OC)nc(N[C@@H](CC#C)CC(=O-)N[Dy])n2)cc1')\n","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.224054Z","iopub.execute_input":"2024-05-10T11:45:48.224358Z","iopub.status.idle":"2024-05-10T11:45:48.232468Z","shell.execute_reply.started":"2024-05-10T11:45:48.224326Z","shell.execute_reply":"2024-05-10T11:45:48.231617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TEST on testset","metadata":{}},{"cell_type":"markdown","source":"only 5 minutes","metadata":{}},{"cell_type":"code","source":"# test_raw = pd.read_parquet('/kaggle/input/leash-BELKA/test.parquet')\n# smiles = test_raw['molecule_smiles'].values\n\n# smiles_enc = encode_smiles_batch(smiles.tolist(),max_length=142,batch_size = 4096*4)\n\n# smiles_enc = np.stack(smiles_enc)\n\n# test = pd.DataFrame(smiles_enc, columns = [f'm_enc{i}' for i in range(142)])\n# test.to_parquet('test_enc.parquet')\n# 100%|██████████| 103/103 [03:49<00:00,  2.23s/it]","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.233747Z","iopub.execute_input":"2024-05-10T11:45:48.234042Z","iopub.status.idle":"2024-05-10T11:45:48.241817Z","shell.execute_reply.started":"2024-05-10T11:45:48.234001Z","shell.execute_reply":"2024-05-10T11:45:48.241065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocess","metadata":{}},{"cell_type":"markdown","source":"Since some libraries, such as tokenizers, initialize some global states when used,  these states can cause problems after the process forks. So we'll explicitly set the environment variable ```TOKENIZERS_PARALLELISM=false``` to disable parallel processing for tokenizers. This will ensure that parallel processing is  no longer used after the process forks.","metadata":{}},{"cell_type":"code","source":"import os\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.243053Z","iopub.execute_input":"2024-05-10T11:45:48.243329Z","iopub.status.idle":"2024-05-10T11:45:48.256359Z","shell.execute_reply.started":"2024-05-10T11:45:48.243306Z","shell.execute_reply":"2024-05-10T11:45:48.255404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CFG.PREPROCESS:\n    if all_data:\n        train_raw = pd.read_parquet('/kaggle/input/leash-BELKA/train.parquet')\n        smiles = train_raw[train_raw['protein_name']=='BRD4']['molecule_smiles'].values\n        assert (smiles!=train_raw[train_raw['protein_name']=='HSA']['molecule_smiles'].values).sum() == 0\n        assert (smiles!=train_raw[train_raw['protein_name']=='sEH']['molecule_smiles'].values).sum() == 0\n\n    else:\n        train_raw = pd.read_csv('/kaggle/input/leash-BELKA/train.csv',nrows = 1000)\n        smiles = train_raw[train_raw['protein_name']=='BRD4']['molecule_smiles'].values[:333]\n        HSA = train_raw[train_raw['protein_name']=='HSA'].reset_index(drop=True)\n        sEH = train_raw[train_raw['protein_name']=='sEH'].reset_index(drop=True)\n        assert (smiles!=HSA['molecule_smiles'].values).sum() == 0\n        assert (smiles!=sEH['molecule_smiles'].values).sum() == 0\n\n    smiles_enc = encode_smiles_parallel(smiles.tolist(), max_length=142, batch_size=4096*8)\n    del smiles\n    gc.collect()\n    smiles_enc = np.stack(smiles_enc)\n\n    \n    train = pd.DataFrame(smiles_enc, columns = [f'm_enc{i}' for i in range(142)])\n    \n    train['bind1'] = train_raw[train_raw['protein_name']=='BRD4']['binds'].values\n    train['bind2'] = train_raw[train_raw['protein_name']=='HSA']['binds'].values\n    train['bind3'] = train_raw[train_raw['protein_name']=='sEH']['binds'].values\n    train.to_parquet('train_enc.parquet')\n\n    test_raw = pd.read_parquet('/kaggle/input/leash-BELKA/test.parquet')\n    smiles = test_raw['molecule_smiles'].values\n\n    smiles_enc = encode_smiles_batch(smiles.tolist(),max_length=142,batch_size = 4096*8)\n    \n    smiles_enc = np.stack(smiles_enc)\n    \n    test = pd.DataFrame(smiles_enc, columns = [f'm_enc{i}' for i in range(142)])\n    test.to_parquet('test_enc.parquet')\n\nelse:\n    if not CFG.predict:\n        train = pd.read_parquet('/kaggle/input/belka-chemical-encoding/train_enc.parquet')\n    test = pd.read_parquet('/kaggle/input/belka-chemical-encoding/test_enc.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:48.257455Z","iopub.execute_input":"2024-05-10T11:45:48.257759Z","iopub.status.idle":"2024-05-10T11:45:49.870831Z","shell.execute_reply.started":"2024-05-10T11:45:48.257735Z","shell.execute_reply":"2024-05-10T11:45:49.870063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('done')","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:53.905600Z","iopub.execute_input":"2024-05-10T11:45:53.906341Z","iopub.status.idle":"2024-05-10T11:45:53.910776Z","shell.execute_reply.started":"2024-05-10T11:45:53.906309Z","shell.execute_reply":"2024-05-10T11:45:53.909835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\ndef my_model():\n    with strategy.scope():\n        INP_LEN = 142\n        NUM_FILTERS = 32\n        hidden_dim = 128\n        tokens_num = 37\n\n        inputs = tf.keras.layers.Input(shape=(INP_LEN,), dtype='int32')\n        x = tf.keras.layers.Embedding(input_dim=tokens_num, output_dim=hidden_dim, input_length=INP_LEN, mask_zero = True)(inputs)\n        x = tf.keras.layers.Conv1D(filters=NUM_FILTERS, kernel_size=3,  activation='relu', padding='valid',  strides=1)(x)\n        x = tf.keras.layers.Conv1D(filters=NUM_FILTERS*2, kernel_size=3,  activation='relu', padding='valid',  strides=1)(x)\n        x = tf.keras.layers.Conv1D(filters=NUM_FILTERS*3, kernel_size=3,  activation='relu', padding='valid',  strides=1)(x)\n        x = tf.keras.layers.GlobalMaxPooling1D()(x)\n\n        x = tf.keras.layers.Dense(1024, activation='relu')(x)\n        x = tf.keras.layers.Dropout(0.1)(x)\n        x = tf.keras.layers.Dense(1024, activation='relu')(x)\n        x = tf.keras.layers.Dropout(0.1)(x)\n        x = tf.keras.layers.Dense(512, activation='relu')(x)\n        x = tf.keras.layers.Dropout(0.1)(x)\n\n        outputs = tf.keras.layers.Dense(3, activation='sigmoid')(x)\n\n        model = tf.keras.models.Model(inputs = inputs, outputs = outputs)\n        optimizer = tf.keras.optimizers.AdamW(learning_rate=CFG.LR, weight_decay=CFG.WD)\n        loss = 'binary_crossentropy'\n        weighted_metrics = [tf.keras.metrics.AUC(curve='PR', name = 'avg_precision')]\n        model.compile(\n        loss=loss,\n        optimizer=optimizer,\n        weighted_metrics=weighted_metrics,\n        )\n        return model\nmodel = my_model()\n\nmodel.summary()\ntf.keras.backend.clear_session()","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:45:54.709389Z","iopub.execute_input":"2024-05-10T11:45:54.709743Z","iopub.status.idle":"2024-05-10T11:45:55.333292Z","shell.execute_reply.started":"2024-05-10T11:45:54.709715Z","shell.execute_reply":"2024-05-10T11:45:55.332094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train & Inference","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import average_precision_score as APS\nFEATURES = [f'm_enc{i}' for i in range(142)]\nTARGETS = ['bind1', 'bind2', 'bind3']\nskf = StratifiedKFold(n_splits = CFG.NBR_FOLDS, shuffle = True, random_state = 42)\n\nall_preds = []\nif not CFG.predict:\n    for fold,(train_idx, valid_idx) in enumerate(skf.split(train, train[TARGETS].sum(1))):\n\n        if fold not in CFG.SELECTED_FOLDS:\n            continue;\n\n        X_train = train.loc[train_idx, FEATURES]\n        y_train = train.loc[train_idx, TARGETS]\n        X_val = train.loc[valid_idx, FEATURES]\n        y_val = train.loc[valid_idx, TARGETS]\n\n        es = tf.keras.callbacks.EarlyStopping(patience=5, monitor=\"val_loss\", mode='min', verbose=1)\n        checkpoint = tf.keras.callbacks.ModelCheckpoint(monitor='val_loss', filepath=f\"model-{fold}.tf\",\n                                                            save_best_only=True, save_weights_only=True,\n                                                        mode='min')\n        reduce_lr_loss = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.05, patience=4, verbose=1)\n        model = my_model()\n        history = model.fit(\n                X_train, y_train,\n                validation_data=(X_val, y_val),\n                epochs=CFG.EPOCHS,\n                callbacks=[checkpoint, reduce_lr_loss, es],\n                batch_size=CFG.BATCH_SIZE,\n                verbose=1,\n            )\n        model.load_weights(f\"model-{fold}.tf\")\n        model.load_weights(f\"/kaggle/input/re-encode-model/model-{fold}.tf\")\n\n        oof = model.predict(X_val, batch_size = CFG.BATCH_SIZE)\n        print('fold :', fold, 'CV score =', APS(y_val, oof, average = 'micro'))\n\n        preds = model.predict(test, batch_size = CFG.BATCH_SIZE)\n        all_preds.append(preds)\nif CFG.predict:\n    model.load_weights(f\"/kaggle/input/re-encode-model/model-0.tf\")\n    preds = model.predict(test, batch_size = CFG.BATCH_SIZE)\n    all_preds.append(preds)\npreds = np.mean(all_preds, 0)","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:49:16.911790Z","iopub.execute_input":"2024-05-10T11:49:16.912173Z","iopub.status.idle":"2024-05-10T11:49:28.892372Z","shell.execute_reply.started":"2024-05-10T11:49:16.912145Z","shell.execute_reply":"2024-05-10T11:49:28.891436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"tst = pd.read_parquet('/kaggle/input/leash-BELKA/test.parquet')\ntst['binds'] = 0\ntst.loc[tst['protein_name']=='BRD4', 'binds'] = preds[(tst['protein_name']=='BRD4').values, 0]\ntst.loc[tst['protein_name']=='HSA', 'binds'] = preds[(tst['protein_name']=='HSA').values, 1]\ntst.loc[tst['protein_name']=='sEH', 'binds'] = preds[(tst['protein_name']=='sEH').values, 2]\ntst[['id', 'binds']].to_csv('submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2024-05-10T11:49:38.006201Z","iopub.execute_input":"2024-05-10T11:49:38.006935Z","iopub.status.idle":"2024-05-10T11:49:47.305521Z","shell.execute_reply.started":"2024-05-10T11:49:38.006903Z","shell.execute_reply":"2024-05-10T11:49:47.304494Z"},"trusted":true},"execution_count":null,"outputs":[]}]}