{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59094,"databundleVersionId":7010844,"sourceType":"competition"},{"sourceId":7712331,"sourceType":"datasetVersion","datasetId":4441094}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What is about ? \n\nWe will try to rewrite top2 solution in Kaggle notebook format and better understand it.\n\nWriteup: https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/458738 \n\nGithub: https://github.com/Eliorkalfon/single_cell_pb\n\n\n#### Solution idea\n\nIt is a bit like standard \"FT-transformer\" (Feature Tokenizer Transformer for tabular data): first features go into linear layer (kind of trainable tokenizer)  then followed by transformer encoder block.  (See https://paperswithcode.com/method/ft-transformer )\n\nWhat is great - that the key code of the model is quite compact - see writeup and many thanks to the author.\n\nBut there are many details which are important, we plan to analyse them here.\n\n\nSee section \"detailed desciption\" below for further information. \n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T08:44:22.593921Z","iopub.execute_input":"2024-03-06T08:44:22.594263Z","iopub.status.idle":"2024-03-06T08:44:22.599869Z","shell.execute_reply.started":"2024-03-06T08:44:22.594235Z","shell.execute_reply":"2024-03-06T08:44:22.598831Z"}}},{"cell_type":"markdown","source":"# Key params for the notebook","metadata":{}},{"cell_type":"code","source":"verbose = 1000\n\nimport torch\nif torch.cuda.is_available():\n    device_default = 'cuda'\nelse:\n    device_default = 'cpu'\n    \nclass Cfg:\n    def __init__(self):\n        # Access specific values from the config\n        self.n_components_list = [18211] #  config.get('n_components_list', [])\n        self.d_models_list = [128] #  config.get('d_models_list', [])  # embedding dimensions for the transformer models\n        self.batch_size = 32 # config.get('batch_size', 32)\n        self.sampling_strategy = 'random' # Original default was \"k-means\", but we change it to \"non-k-means\"\n            #'k-means'# config.get('sampling_strategy', 'random')\n        self.data_file = '/kaggle/input/open-problems-single-cell-perturbations/de_train.parquet' #  config.get('data_file', '')\n        self.id_map_file = '/kaggle/input/open-problems-single-cell-perturbations/id_map.csv' #  config.get('id_map_file', '')\n        self.validation_percentage = 0.1 #  config.get('validation_percentage', 0.2)\n        self.device = device_default #  'cpu' #  config.get('device', 'cuda')\n        self.seed = None # config.get('seed', None)\n        self.num_epochs = 20000 # 2000#  20000 # config.get('num_epochs', 20000)\n        self.early_stopping = 5000 # config.get('early_stopping', 5000)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:12.992682Z","iopub.execute_input":"2024-03-08T07:59:12.993093Z","iopub.status.idle":"2024-03-08T07:59:16.962547Z","shell.execute_reply.started":"2024-03-08T07:59:12.993059Z","shell.execute_reply":"2024-03-08T07:59:16.961326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Detailed description ","metadata":{}},{"cell_type":"markdown","source":"\n#### Technicalities code: calls pipeline\n\n    In train.py:\n    main->train_[non]_k_means_strategy -> [ train_transformer_k_means_learning ] \n        -> train_func -> train_epoch \n        \n    train_[non]_k_means_strategy -> [ train_transformer_k_means_learning ]     \n        \n#### Technicalities code: key functions one by one \n    \n- main: \n\nfeature preparation by prepare_augmented_data \n    \n- train_[non]_k_means_strategy \n\n    fake loop over n_components - never(?) used, and embed dimensions - only 128\n    \n    Call train_func - key (\"meat\") function with loop over epochs\n    \n    Save models\n\n- train_func  (key (\"meat\") function - loop over epochs\n\n    model = CustomTransformer_v3\n    \n    dataset = TensorDataset(torch.tensor(X_train, dtype=torch.float32).to(device), torch.tensor(Y_reduced, dtype=torch.float32).to(device)) \n    \n    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n\n    val_dataloader = DataLoader    \n\n    optimizer = Lion(model.parameters(), lr=lr, weight_decay=1e-4)\n    \n    scheduler = lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode=\"min\", factor=0.9999, patience=500,\n                                               verbose=True)    \n \n    criterion = nn.HuberLoss()\n\n\n    Call train_epoch \n        \n- train_epoch        \n\n    def train_epoch(model, dataloader, optimizer, criterion, device='cuda'):\n        model.train()\n        total_loss = 0.0\n        for inputs, targets in dataloader:\n            optimizer.zero_grad()\n            inputs, targets = inputs.to(device), targets.to(device)\n            predictions = model(inputs)\n            loss = criterion(predictions, targets)\n            loss.backward()\n            clip_grad_norm_(model.parameters(), max_norm=1.0)\n            optimizer.step()\n            total_loss += loss.item()\n        return total_loss / len(dataloader)\n\n\n\n    \n\n        \n#### Technicalities organization of the code - more details\n\n\n    train.py - Starter (train models)\n        Function \"main\" - line 235 (last function in train.py (as usual))\n            Parsing config\n            prepare_augmented_data - prepare features - concatentation of both onehot+TE\n                (variable naming - misleading - it says only onehot, but it is both)\n            Call training: \n                train_k_means_strategy or train_non_k_means_strategy -depends on config \n        Function train_non_k_means_strategy - line 203\n            Fake loops over n_components,  d_model (embed_size) \n                        fake because the actual lists contain only single element\n                Call reduce_labels - in default mode it does NOTHING\n                    It can be used as an opttion to TSVD reduce targets, predict reduced,\n                        and tsvd.inverse back. Standard scheme.  \n                    It is \"5-line\" function from utils\n                    label_reducer = TruncatedSVD(n_components=n_components, n_iter=10)\n                    scaler = StandardScaler()\n                    Y_reduced = label_reducer.fit_transform(scaler.fit_transform(Y))\n                    It returns 3 things - trained tsvd,scaler and Y_reduced\n                Call train_func\n                \n        Function train_func - line 58 - loop over training epochs\n            Call train_epoch\n        Function train_epoch - line 12 \n            Training loop over batches \n            \n\n        Alternatively for train_k_means_strategy (with k-means)\n            there is one step more: \n                train_k_means_strategy->train_transformer_k_means_learning\n        Detailed:\n        Function train_k_means_strategy - line  178\n            Fake loops over n_components,  d_model (embed_size) \n                        fake because the actual lists contain only single element\n                Call train_transformer_k_means_learning \n                Save models\n        Function train_transformer_k_means_learning - line 133\n            Call train_func\n        Function train_func - line 58 - loop over training epochs\n            Call train_epoch\n        Function train_epoch - line 12 \n            Training loop over batches \n        \n        \n    utils.py - different technical functions like load configs etc\n\n\n#### Technicalities: Some remarks around code \n\n\n- Features are onehot-concat+target, but var naming is \"onehot\" - misleading\n\nIn \"main\" line 267 - key feature preparation , and naming goes down through all the code:\n\none_hot_encode_features, targets, one_hot_test = prepare_augmented_data(data_file=data_file,\n                                                                            id_map_file=id_map_file)\n\n\nPrepare features - concatentation of both onehot+TE\n    (variable naming - misleading - it says only onehot, but it is both)\n\n\n- TSVD reduction is NOT used in default config, \n\nalthough seems to be supported by code:  change n_components_list to smaller values and should work\n\n- There is loop to compute models with different params - but actually default config have single-values lists\n\n    Function train_non_k_means_strategy\n    \n    for n_components in n_components_list:\n            for d_model in d_models_list:\n        \n","metadata":{}},{"cell_type":"markdown","source":"# Install/Imports","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport time\nt0start = time.time() \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        if '.git' not in os.path.join(dirname, filename):\n            print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n\nfrom sklearn.cluster import KMeans\nimport copy\nfrom torch.nn.utils import clip_grad_norm_\nfrom tqdm import tqdm\nimport pickle\nimport argparse\nimport os","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-03-08T07:59:16.964701Z","iopub.execute_input":"2024-03-08T07:59:16.965233Z","iopub.status.idle":"2024-03-08T07:59:21.262194Z","shell.execute_reply.started":"2024-03-08T07:59:16.965192Z","shell.execute_reply":"2024-03-08T07:59:21.260804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lion optimizer","metadata":{}},{"cell_type":"code","source":"!pip install lion-pytorch","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:21.263670Z","iopub.execute_input":"2024-03-08T07:59:21.264279Z","iopub.status.idle":"2024-03-08T07:59:38.378483Z","shell.execute_reply.started":"2024-03-08T07:59:21.264244Z","shell.execute_reply":"2024-03-08T07:59:38.377131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## \"utils.py\" - import from Kaggle dataset\n\nCode from author's github is placed in Kaggle dataset.\n\nSo we can import that module","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/followup-open-problems-single-cell-perturbations/top2solution_single_cell_pb/')\n\nfrom utils import *\nfrom models import CustomTransformer_v3  # Can be changed to other models in models.py\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:38.382901Z","iopub.execute_input":"2024-03-08T07:59:38.383333Z","iopub.status.idle":"2024-03-08T07:59:38.452151Z","shell.execute_reply.started":"2024-03-08T07:59:38.383295Z","shell.execute_reply":"2024-03-08T07:59:38.451018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n\nModel from models.py  - CustomTransformer_v3\n\n\nIt is a bit like standard \"FT-transformer\" (Feature Tokenizer Transformer for tabular data): first features go into linear layer (kind of trainable tokenizer)  then followed by transformer encoder block.  (See https://paperswithcode.com/method/ft-transformer )\n\nWhat is great - that the key code of the model is quite compact - see writeup and many thanks to the author.\n\n\nWriteup: https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/458738 \n\nGithub: https://github.com/Eliorkalfon/single_cell_pb\n\n\n\n","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim\n\nclass CustomTransformer_v3(nn.Module):  # mean + std\n    def __init__(self, num_features, num_labels, d_model=128, num_heads=8, num_layers=6, dropout=0.3):\n        super(CustomTransformer_v3, self).__init__()\n        self.num_target_encodings = 18211 * 4\n        self.num_sparse_features = num_features - self.num_target_encodings\n\n        self.sparse_feature_embedding = nn.Linear(self.num_sparse_features, d_model)\n        self.target_encoding_embedding = nn.Linear(self.num_target_encodings, d_model)\n        self.norm = nn.LayerNorm(d_model)\n\n        self.concatenation_layer = nn.Linear(2 * d_model, d_model)\n        self.transformer = nn.TransformerEncoder(\n            nn.TransformerEncoderLayer(d_model=d_model, nhead=num_heads, dropout=dropout, activation=nn.GELU(),\n                                       batch_first=True),\n            num_layers=num_layers\n        )\n        self.fc = nn.Linear(d_model, num_labels)\n\n    def forward(self, x):\n        sparse_features = x[:, :self.num_sparse_features]\n        target_encodings = x[:, self.num_sparse_features:]\n\n        sparse_features = self.sparse_feature_embedding(sparse_features)\n        target_encodings = self.target_encoding_embedding(target_encodings)\n\n        combined_features = torch.cat((sparse_features, target_encodings), dim=1)\n        combined_features = self.concatenation_layer(combined_features)\n        combined_features = self.norm(combined_features)\n\n        x = self.transformer(combined_features)\n        x = self.norm(x)\n\n        x = self.fc(x)\n        return x","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:38.453647Z","iopub.execute_input":"2024-03-08T07:59:38.454124Z","iopub.status.idle":"2024-03-08T07:59:38.467167Z","shell.execute_reply.started":"2024-03-08T07:59:38.454092Z","shell.execute_reply":"2024-03-08T07:59:38.465790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configs\n","metadata":{}},{"cell_type":"markdown","source":"## Look on original","metadata":{}},{"cell_type":"code","source":"!head /kaggle/input/followup-open-problems-single-cell-perturbations/top2solution_single_cell_pb/config_train.yaml","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:38.469076Z","iopub.execute_input":"2024-03-08T07:59:38.469428Z","iopub.status.idle":"2024-03-08T07:59:39.631124Z","shell.execute_reply.started":"2024-03-08T07:59:38.469397Z","shell.execute_reply":"2024-03-08T07:59:39.629497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Own config dictionary\n\nBetter to work with config directly defined in notebook, rather than loading files \n\nSo write down config here\n\ndict_cfg_train - corresponds to config_train.yaml","metadata":{}},{"cell_type":"code","source":"# Lifehack:  copy text from txt-file,  add \"self.\" and so we get Python class,  after we convert it to dictionary   \n# class Cfg:\n#     def __init__(self):\n#         # Access specific values from the config\n#         self.n_components_list = [18211] #  config.get('n_components_list', [])\n#         self.d_models_list = [128] #  config.get('d_models_list', [])  # embedding dimensions for the transformer models\n#         self.batch_size = 32 # config.get('batch_size', 32)\n#         self.sampling_strategy = 'k-means'# config.get('sampling_strategy', 'random')\n#         self.data_file = '/kaggle/input/open-problems-single-cell-perturbations/de_train.parquet' #  config.get('data_file', '')\n#         self.id_map_file = '/kaggle/input/open-problems-single-cell-perturbations/id_map.csv' #  config.get('id_map_file', '')\n#         self.validation_percentage = 0.1 #  config.get('validation_percentage', 0.2)\n#         self.device = 'cpu' #  config.get('device', 'cuda')\n#         self.seed = None # config.get('seed', None)\n#         self.num_epochs = 20000 # config.get('num_epochs', 20000)\n#         self.early_stopping = 5000 # config.get('early_stopping', 5000)\n    \ncfg = Cfg()\ndict_cfg = {}\nfor k in dir(cfg):\n    if '__' not in k:\n        dict_cfg[k] = getattr(cfg, k ) \ndict_cfg_train = dict_cfg\ndict_cfg_train","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:39.633842Z","iopub.execute_input":"2024-03-08T07:59:39.635031Z","iopub.status.idle":"2024-03-08T07:59:39.649817Z","shell.execute_reply.started":"2024-03-08T07:59:39.634967Z","shell.execute_reply":"2024-03-08T07:59:39.648413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Main ( train.py)","metadata":{"execution":{"iopub.status.busy":"2024-03-06T13:08:24.072815Z","iopub.execute_input":"2024-03-06T13:08:24.073306Z","iopub.status.idle":"2024-03-06T13:08:24.080629Z","shell.execute_reply.started":"2024-03-06T13:08:24.073263Z","shell.execute_reply":"2024-03-06T13:08:24.079501Z"}}},{"cell_type":"markdown","source":"## Original code from train.py","metadata":{}},{"cell_type":"code","source":"# def main():\n#     # Set up command-line argument parser\n#     parser = argparse.ArgumentParser(description=\"Your script description here.\")\n#     parser.add_argument('--config', type=str, help=\"Path to the YAML config file.\", default='config_train.yaml')\n#     args = parser.parse_args()\n\n#     # Check if the config file is provided\n#     if not args.config:\n#         print(\"Please provide a config file using --config.\")\n#         return\n\n#     # Load and print configurations\n#     config_file = args.config\n#     config = load_and_print_config(config_file)\n\n#     # Access specific values from the config\n#     n_components_list = config.get('n_components_list', [])\n#     d_models_list = config.get('d_models_list', [])  # embedding dimensions for the transformer models\n#     batch_size = config.get('batch_size', 32)\n#     sampling_strategy = config.get('sampling_strategy', 'random')\n#     data_file = config.get('data_file', '')\n#     id_map_file = config.get('id_map_file', '')\n#     validation_percentage = config.get('validation_percentage', 0.2)\n#     device = config.get('device', 'cuda')\n#     seed = config.get('seed', None)\n#     num_epochs = config.get('num_epochs', 20000)\n#     early_stopping = config.get('early_stopping', 5000)\n\n#     # Validate the sampling strategy\n#     validate_sampling_strategy(sampling_strategy)\n\n#     # Prepare augmented data\n#     one_hot_encode_features, targets, one_hot_test = prepare_augmented_data(data_file=data_file,\n#                                                                             id_map_file=id_map_file)\n\n#     # one_hot_encode_features, targets, one_hot_test = prepare_augmented_data_mean_only(data_file=data_file,\n#     #                                                                         id_map_file=id_map_file)\n#     if sampling_strategy == 'k-means':\n#         train_k_means_strategy(n_components_list, d_models_list, one_hot_encode_features, targets, num_epochs,\n#                                early_stopping, batch_size, device)\n#     else:\n#         train_non_k_means_strategy(n_components_list, d_models_list, one_hot_encode_features, targets, num_epochs,\n#                                    early_stopping, batch_size, device, seed, validation_percentage)\n\n\n# if __name__ == \"__main__\":\n#     main()","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:39.651842Z","iopub.execute_input":"2024-03-08T07:59:39.652536Z","iopub.status.idle":"2024-03-08T07:59:39.663349Z","shell.execute_reply.started":"2024-03-08T07:59:39.652493Z","shell.execute_reply":"2024-03-08T07:59:39.662123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  Setting the config params","metadata":{}},{"cell_type":"code","source":"config = dict_cfg_train\nprint(config)\n\n# Access specific values from the config\nn_components_list = config.get('n_components_list', [])\nd_models_list = config.get('d_models_list', [])  # embedding dimensions for the transformer models\nbatch_size = config.get('batch_size', 32)\nsampling_strategy = config.get('sampling_strategy', 'random')\ndata_file = config.get('data_file', '')\nid_map_file = config.get('id_map_file', '')\nvalidation_percentage = config.get('validation_percentage', 0.2)\ndevice = config.get('device', 'cuda')\nseed = config.get('seed', None)\nnum_epochs = config.get('num_epochs', 20000)\nearly_stopping = config.get('early_stopping', 5000)\n\ndef validate_sampling_strategy(sampling_strategy):\n    allowed_strategies = ['k-means', 'random']\n    if sampling_strategy not in allowed_strategies:\n        raise ValueError(f\"Invalid sampling strategy. Choose from: {', '.join(allowed_strategies)}\")\n\n\n# Validate the sampling strategy\nvalidate_sampling_strategy(sampling_strategy)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:39.665076Z","iopub.execute_input":"2024-03-08T07:59:39.665544Z","iopub.status.idle":"2024-03-08T07:59:39.679984Z","shell.execute_reply.started":"2024-03-08T07:59:39.665502Z","shell.execute_reply":"2024-03-08T07:59:39.678943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare the data prepare_augmented_data\n\nnames of variables - misleading - it is not only onehot , but concat: one-hot + target encoding","metadata":{}},{"cell_type":"code","source":"%%time\n# Prepare augmented data\none_hot_encode_features, targets, one_hot_test = prepare_augmented_data(data_file='/kaggle/input/open-problems-single-cell-perturbations/de_train.parquet',\n                                                                        id_map_file= '/kaggle/input/open-problems-single-cell-perturbations/id_map.csv')\n\n\n# one_hot_encode_features.shape # (614, 72996)\n# targets.shape # (614, 18211)\n# one_hot_test.shape # (255, 72996)\n\n# print( one_hot_encode_features[:3,:3] )\n# print( one_hot_encode_features[:3,-3:] )\n# [[0. 0. 1.]\n#  [0. 0. 0.]\n#  [0. 0. 0.]]\n# [[0.34976903 0.46707103 0.51265854]\n#  [0.34976903 0.46707103 0.51265854]\n#  [0.34976903 0.46707103 0.51265854]]\n\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T07:59:39.685111Z","iopub.execute_input":"2024-03-08T07:59:39.685990Z","iopub.status.idle":"2024-03-08T08:00:48.379035Z","shell.execute_reply.started":"2024-03-08T07:59:39.685952Z","shell.execute_reply":"2024-03-08T08:00:48.377811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if verbose >= 1000:\n    print(one_hot_encode_features.shape, targets.shape, one_hot_test.shape)\n ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# train_non_k_means_strategy - function from train.py","metadata":{}},{"cell_type":"markdown","source":"## Original code","metadata":{}},{"cell_type":"code","source":"# def train_non_k_means_strategy(n_components_list, d_models_list, one_hot_encode_features, targets, num_epochs,\n#                                early_stopping, batch_size, device, seed, validation_percentage):\n#     # Split the data for non-k_means sampling strategy\n#     X_train, X_val, y_train, y_val = split_data(one_hot_encode_features, targets, test_size=validation_percentage,\n#                                                 shuffle=True, random_state=seed)\n\n#     # Training loop for non-k_means sampling strategy\n#     for n_components in n_components_list:\n#         for d_model in d_models_list:\n#             label_reducer, scaler, Y_reduced = reduce_labels(y_train, n_components)\n#             transformer_model = train_func(X_train, y_train, X_val, y_val,\n#                                            n_components,\n#                                            num_epochs=num_epochs,\n#                                            early_stopping=early_stopping,\n#                                            batch_size=batch_size,\n#                                            d_model=d_model,\n#                                            label_reducer=label_reducer,\n#                                            scaler=scaler,\n#                                            device=device)\n\n#             # Save the trained models\n#             os.makedirs('trained_models_k-means', exist_ok=True)\n#             with open(f'trained_models_k-means/label_reducer_{n_components}_{d_model}.pkl', 'wb') as file:\n#                 pickle.dump(label_reducer, file)\n\n#             with open(f'trained_models_k-means/scaler_{n_components}_{d_model}.pkl', 'wb') as file:\n#                 pickle.dump(scaler, file)\n\n#             torch.save(transformer_model.state_dict(),\n#                        f'trained_models_k-means/transformer_model_{n_components}_{d_model}.pt')\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:48.380541Z","iopub.execute_input":"2024-03-08T08:00:48.380932Z","iopub.status.idle":"2024-03-08T08:00:48.388034Z","shell.execute_reply.started":"2024-03-08T08:00:48.380900Z","shell.execute_reply":"2024-03-08T08:00:48.386921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(n_components_list)\nprint(d_models_list)","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:48.389705Z","iopub.execute_input":"2024-03-08T08:00:48.390329Z","iopub.status.idle":"2024-03-08T08:00:48.407174Z","shell.execute_reply.started":"2024-03-08T08:00:48.390294Z","shell.execute_reply":"2024-03-08T08:00:48.405818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## \"train_k_means_strategy\" in notebook format","metadata":{}},{"cell_type":"code","source":"try:\n    verbose\nexcept:        \n    verbose = 1000","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:48.408907Z","iopub.execute_input":"2024-03-08T08:00:48.409267Z","iopub.status.idle":"2024-03-08T08:00:48.416777Z","shell.execute_reply.started":"2024-03-08T08:00:48.409235Z","shell.execute_reply":"2024-03-08T08:00:48.415539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train test split","metadata":{}},{"cell_type":"code","source":"# Split the data for non-k_means sampling strategy\nX_train, X_val, y_train, y_val = split_data(one_hot_encode_features, targets, test_size=validation_percentage,\n                                            shuffle=True, random_state=seed)\n\nif verbose >= 1000:\n    print(X_train.shape, X_val.shape, y_train.shape, y_val.shape)\n    ","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:48.418671Z","iopub.execute_input":"2024-03-08T08:00:48.419414Z","iopub.status.idle":"2024-03-08T08:00:49.284108Z","shell.execute_reply.started":"2024-03-08T08:00:48.419379Z","shell.execute_reply":"2024-03-08T08:00:49.282839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Code uses both notations:\n\nY_train = y_train\nY_val =  y_val","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:49.285455Z","iopub.execute_input":"2024-03-08T08:00:49.285862Z","iopub.status.idle":"2024-03-08T08:00:49.291298Z","shell.execute_reply.started":"2024-03-08T08:00:49.285826Z","shell.execute_reply":"2024-03-08T08:00:49.289888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Skip fake loop, skip fake tsvd-reducer and scaler for targets","metadata":{}},{"cell_type":"code","source":"# Training loop for non-k_means sampling strategy\n# for n_components in n_components_list:\n#     for d_model in d_models_list:\n\nn_components = n_components_list[0]\nd_model = d_models_list[0]\n\nlabel_reducer, scaler, Y_reduced = reduce_labels(y_train, n_components)\n\nif verbose >= 1000:\n    print(label_reducer, scaler, Y_reduced.shape)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:49.293318Z","iopub.execute_input":"2024-03-08T08:00:49.294258Z","iopub.status.idle":"2024-03-08T08:00:49.303433Z","shell.execute_reply.started":"2024-03-08T08:00:49.294213Z","shell.execute_reply":"2024-03-08T08:00:49.302199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# train_func - key-\"meat\" function - epochs loop","metadata":{}},{"cell_type":"markdown","source":"## Original code","metadata":{}},{"cell_type":"code","source":"# def train_func(X_train, Y_reduced, X_val, Y_val, n_components, num_epochs, batch_size, label_reducer, scaler,\n#                d_model=128, early_stopping=5000, device='cuda', ):\n#     best_mrrmse = float('inf')\n#     best_model = None\n#     best_val_loss = float('inf')\n#     best_epoch = 0\n#     # model = CustomTransformer(num_features=X_train.shape[1], num_labels=n_components, d_model=d_model).to(device)\n#     model = CustomTransformer_v3(num_features=X_train.shape[1], num_labels=n_components, d_model=d_model).to(device)\n#     # model = CustomDeeperModel(X_train.shape[1], d_model, n_components).to(device)\n\n#     dataset = TensorDataset(torch.tensor(X_train, dtype=torch.float32).to(device),\n#                             torch.tensor(Y_reduced, dtype=torch.float32).to(device))\n#     dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n#     val_dataloader = DataLoader(TensorDataset(torch.tensor(X_val, dtype=torch.float32).to(device),\n#                                               torch.tensor(\n#                                                   Y_val,\n#                                                   dtype=torch.float32).to(device)),\n#                                 batch_size=batch_size, shuffle=False)\n#     if n_components < 18211:\n#         lr = 1e-3\n\n#     else:\n#         lr = 1e-5\n#     # optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=1e-4)\n#     optimizer = Lion(model.parameters(), lr=lr, weight_decay=1e-4)\n#     # scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=10, eta_min=1e-7, verbose=False)\n#     scheduler = lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode=\"min\", factor=0.9999, patience=500,\n#                                                verbose=True)\n#     criterion = nn.HuberLoss()\n#     # criterion = nn.L1Loss()\n#     # criterion = CustomLoss()\n#     # criterion = nn.MSELoss()\n#     model.train()\n#     counter = 0\n#     pbar = tqdm(range(num_epochs), position=0, leave=True)\n#     for epoch in range(num_epochs):\n#         _ = train_epoch(model, dataloader, optimizer, criterion)\n\n#         if counter >= early_stopping:\n#             break\n#         if scaler:\n#             val_loss, val_targets_stacked, val_predictions_stacked = validate(model, val_dataloader, criterion,\n#                                                                               label_reducer, scaler)\n#             # Calculate MRRMSE for the entire validation set\n#             val_mrrmse = calculate_mrrmse_np(\n#                 val_targets_stacked.cpu().detach().numpy(),\n#                 scaler.inverse_transform((label_reducer.inverse_transform(\n#                     val_predictions_stacked.cpu().detach().numpy()))))\n#         else:\n#             val_loss, val_targets_stacked, val_predictions_stacked = validate(model, val_dataloader, criterion)\n#             val_mrrmse = calculate_mrrmse_np(val_targets_stacked.cpu().detach().numpy(),\n\n#                                              val_predictions_stacked.cpu().detach().numpy())\n\n#         if val_mrrmse < best_mrrmse:\n#             best_mrrmse = val_mrrmse\n#             # best_model = copy.deepcopy(model)\n\n#         if val_loss < best_val_loss:\n#             best_val_loss = val_loss\n#             best_model = copy.deepcopy(model)\n#             counter = 0\n#             best_epoch = epoch\n#         else:\n#             counter += 1\n\n#         pbar.set_description(\n#             f\"Validation best MRRMSE: {best_mrrmse:.4f} Validation best loss:\"\n#             f\" {best_val_loss:.4f} Last epoch: {best_epoch}\")\n#         pbar.update(1)\n#         # scheduler.step()  # for cosine anealing\n#         scheduler.step(val_loss)\n#     return label_reducer, scaler, best_model\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:49.305322Z","iopub.execute_input":"2024-03-08T08:00:49.305958Z","iopub.status.idle":"2024-03-08T08:00:49.317158Z","shell.execute_reply.started":"2024-03-08T08:00:49.305916Z","shell.execute_reply":"2024-03-08T08:00:49.315840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## \"train_func\" - notebook format\n","metadata":{}},{"cell_type":"markdown","source":"### Init: Model, datasets, optimizer, loss, params ","metadata":{"execution":{"iopub.status.busy":"2024-03-07T12:49:23.490356Z","iopub.execute_input":"2024-03-07T12:49:23.490882Z","iopub.status.idle":"2024-03-07T12:49:23.497475Z","shell.execute_reply.started":"2024-03-07T12:49:23.490824Z","shell.execute_reply":"2024-03-07T12:49:23.496049Z"}}},{"cell_type":"code","source":"\nbest_mrrmse = float('inf')\nbest_model = None\nbest_val_loss = float('inf')\nbest_epoch = 0\n\n# model = CustomTransformer(num_features=X_train.shape[1], num_labels=n_components, d_model=d_model).to(device)\nmodel = CustomTransformer_v3(num_features=X_train.shape[1], num_labels=n_components, d_model=d_model).to(device)\n# model = CustomDeeperModel(X_train.shape[1], d_model, n_components).to(device)\n\ndataset = TensorDataset(torch.tensor(X_train, dtype=torch.float32).to(device),\n                        torch.tensor(Y_reduced, dtype=torch.float32).to(device))\ndataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\nval_dataloader = DataLoader(TensorDataset(torch.tensor(X_val, dtype=torch.float32).to(device),\n                                          torch.tensor(\n                                              Y_val,\n                                              dtype=torch.float32).to(device)),\n                            batch_size=batch_size, shuffle=False)\nif n_components < 18211:\n    lr = 1e-3\n\nelse:\n    lr = 1e-5\n    \n    \n# optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=1e-4)\noptimizer = Lion(model.parameters(), lr=lr, weight_decay=1e-4)\n# scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=10, eta_min=1e-7, verbose=False)\nscheduler = lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode=\"min\", factor=0.9999, patience=500,\n                                           verbose=True)\ncriterion = nn.HuberLoss()\n# criterion = nn.L1Loss()\n# criterion = CustomLoss()\n# criterion = nn.MSELoss()\nmodel.train()\ncounter = 0\n    ","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:49.318701Z","iopub.execute_input":"2024-03-08T08:00:49.319503Z","iopub.status.idle":"2024-03-08T08:00:52.828665Z","shell.execute_reply.started":"2024-03-08T08:00:49.319456Z","shell.execute_reply":"2024-03-08T08:00:52.827415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if verbose >= 1000:\n    print(model)\n    print(dataset )\n    print(dataloader )\n    print(val_dataloader)\n    \nif verbose >= 1000:\n    print(lr)\n    \n    \nif verbose >= 1000:\n    print(num_epochs)    ","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:52.830673Z","iopub.execute_input":"2024-03-08T08:00:52.831835Z","iopub.status.idle":"2024-03-08T08:00:52.839207Z","shell.execute_reply.started":"2024-03-08T08:00:52.831787Z","shell.execute_reply":"2024-03-08T08:00:52.838402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Auxiliary functions for epoch loop","metadata":{}},{"cell_type":"code","source":"def train_epoch(model, dataloader, optimizer, criterion, device= device_default ):# 'cuda'):\n    model.train()\n    total_loss = 0.0\n    for inputs, targets in dataloader:\n        optimizer.zero_grad()\n        inputs, targets = inputs.to(device), targets.to(device)\n        predictions = model(inputs)\n        loss = criterion(predictions, targets)\n        loss.backward()\n        clip_grad_norm_(model.parameters(), max_norm=1.0)\n        optimizer.step()\n        total_loss += loss.item()\n    return total_loss / len(dataloader)\n\ndef validate(model, val_dataloader, criterion, label_reducer=None, scaler=None, device= device_default): # 'cuda'):\n    model.eval()\n    val_loss = 0.0\n    val_predictions_list = []\n    val_targets_list = []\n    with torch.no_grad():\n        for val_inputs, val_targets in val_dataloader:\n            val_targets_list.append(val_targets.clone().cpu())\n            val_inputs, val_targets = val_inputs.to(device), val_targets.to(device)\n            val_predictions = model(val_inputs)\n            if label_reducer:\n                val_targets = torch.tensor(\n                    label_reducer.transform(scaler.transform(val_targets.clone().cpu().detach().numpy())),\n                    dtype=torch.float32).to(device)\n            val_loss += criterion(val_predictions, val_targets).item()\n            val_predictions_list.append(val_predictions.cpu())\n\n    val_loss /= len(val_dataloader)\n\n    val_predictions_stacked = torch.cat(val_predictions_list, dim=0)\n    val_targets_stacked = torch.cat(val_targets_list, dim=0)\n\n    return val_loss, val_targets_stacked, val_predictions_stacked","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:52.840677Z","iopub.execute_input":"2024-03-08T08:00:52.841328Z","iopub.status.idle":"2024-03-08T08:00:52.854105Z","shell.execute_reply.started":"2024-03-08T08:00:52.841296Z","shell.execute_reply":"2024-03-08T08:00:52.852925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loop epochs\n\n20 epochs - 41 sec on CPU","metadata":{}},{"cell_type":"code","source":"%%time\n\npbar = tqdm(range(num_epochs), position=0, leave=True)\n\nfor epoch in range(num_epochs):\n    _ = train_epoch(model, dataloader, optimizer, criterion)\n\n    if counter >= early_stopping:\n        break\n    if scaler:\n        val_loss, val_targets_stacked, val_predictions_stacked = validate(model, val_dataloader, criterion,\n                                                                          label_reducer, scaler)\n        # Calculate MRRMSE for the entire validation set\n        val_mrrmse = calculate_mrrmse_np(\n            val_targets_stacked.cpu().detach().numpy(),\n            scaler.inverse_transform((label_reducer.inverse_transform(\n                val_predictions_stacked.cpu().detach().numpy()))))\n    else:\n        val_loss, val_targets_stacked, val_predictions_stacked = validate(model, val_dataloader, criterion)\n        val_mrrmse = calculate_mrrmse_np(val_targets_stacked.cpu().detach().numpy(),\n\n                                         val_predictions_stacked.cpu().detach().numpy())\n\n    if val_mrrmse < best_mrrmse:\n        best_mrrmse = val_mrrmse\n        # best_model = copy.deepcopy(model)\n\n    if val_loss < best_val_loss:\n        best_val_loss = val_loss\n        best_model = copy.deepcopy(model)\n        counter = 0\n        best_epoch = epoch\n    else:\n        counter += 1\n\n    pbar.set_description(\n        f\"Validation best MRRMSE: {best_mrrmse:.4f} Validation best loss:\"\n        f\" {best_val_loss:.4f} Last epoch: {best_epoch}\")\n    pbar.update(1)\n    # scheduler.step()  # for cosine anealing\n    scheduler.step(val_loss)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:00:52.855724Z","iopub.execute_input":"2024-03-08T08:00:52.856392Z","iopub.status.idle":"2024-03-08T08:02:05.224405Z","shell.execute_reply.started":"2024-03-08T08:00:52.856358Z","shell.execute_reply":"2024-03-08T08:02:05.223216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer_model = model","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:02:05.226247Z","iopub.execute_input":"2024-03-08T08:02:05.226634Z","iopub.status.idle":"2024-03-08T08:02:05.232527Z","shell.execute_reply.started":"2024-03-08T08:02:05.226601Z","shell.execute_reply":"2024-03-08T08:02:05.231130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save models\n\n(Done in train_non_k_means_strategy function )\n\n\nOriginal code naming - directory always \"k_means\" - a bit misleading since in theory there can be two way with k-means and without\n","metadata":{}},{"cell_type":"code","source":"%%time\n# Save the trained models\nos.makedirs('trained_models_k-means', exist_ok=True)\nwith open(f'trained_models_k-means/label_reducer_{n_components}_{d_model}.pkl', 'wb') as file:\n    pickle.dump(label_reducer, file)\n\nwith open(f'trained_models_k-means/scaler_{n_components}_{d_model}.pkl', 'wb') as file:\n    pickle.dump(scaler, file)\n\ntorch.save(transformer_model.state_dict(),\n           f'trained_models_k-means/transformer_model_{n_components}_{d_model}.pt')","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:02:05.234576Z","iopub.execute_input":"2024-03-08T08:02:05.234995Z","iopub.status.idle":"2024-03-08T08:02:05.344272Z","shell.execute_reply.started":"2024-03-08T08:02:05.234963Z","shell.execute_reply":"2024-03-08T08:02:05.343015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction - \"Main\"\n\nFile \"predict.py\"","metadata":{}},{"cell_type":"code","source":"!head /kaggle/input/followup-open-problems-single-cell-perturbations/top2solution_single_cell_pb/config_test.yaml","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:14:06.537648Z","iopub.execute_input":"2024-03-08T08:14:06.539138Z","iopub.status.idle":"2024-03-08T08:14:07.689916Z","shell.execute_reply.started":"2024-03-08T08:14:06.539090Z","shell.execute_reply":"2024-03-08T08:14:07.688331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Original code","metadata":{}},{"cell_type":"code","source":"# def main():\n#     # Set up command-line argument parser\n#     parser = argparse.ArgumentParser(description=\"Your script description here.\")\n#     parser.add_argument('--config', type=str, help=\"Path to the YAML config file.\", default='config_train.yaml')\n#     args = parser.parse_args()\n\n#     # Check if the config file is provided\n#     if not args.config:\n#         print(\"Please provide a config file using --config.\")\n#         return\n\n#     # Load and print configurations\n#     config_file = args.config\n#     config = load_and_print_config(config_file)\n\n#     # Access specific values from the config\n#     n_components_list = config.get('n_components_list', [])\n#     d_models_list = config.get('d_models_list', [])  # embedding dimensions for the transformer models\n#     batch_size = config.get('batch_size', 32)\n#     data_file = config.get('data_file', '')\n#     id_map_file = config.get('id_map_file', '')\n#     device = config.get('device', 'cuda')\n#     models_dir = config.get('dir', 'model_1_mean_std_only')\n#     # Prepare augmented data\n#     if 'std' in models_dir:\n#         one_hot_encode_features, targets, one_hot_test = prepare_augmented_data(data_file=data_file,\n#                                                                                 id_map_file=id_map_file)\n#     else:\n#         one_hot_encode_features, targets, one_hot_test = prepare_augmented_data_mean_only(data_file=data_file,\n#                                                                                           id_map_file=id_map_file)\n#     unseen_data = torch.tensor(one_hot_test, dtype=torch.float32).to(device)  # Replace X_unseen with your new data\n#     transformer_models = {}\n#     for n_components in n_components_list:\n#         for d_model in d_models_list:\n#             label_reducer, scaler, transformer_model = load_transformer_model(n_components,\n#                                                                               input_features=\n#                                                                               one_hot_encode_features.shape[\n#                                                                                   1],\n#                                                                               d_model=d_model,\n#                                                                               models_foler=f'{models_dir}',\n#                                                                               device=device)\n#             transformer_model.eval()\n#             transformer_models[f'{n_components},{d_model}'] = (\n#                 copy.deepcopy(label_reducer), copy.deepcopy(scaler), copy.deepcopy(transformer_model))\n#     predict_test(unseen_data, transformer_models, n_components_list, d_models_list, batch_size, device=device)\n\n\n# if __name__ == \"__main__\":\n#     main()\n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Config\n\nMostly use same as in train - so we do not need to redefine variables","metadata":{}},{"cell_type":"code","source":"models_dir = '/kaggle/working/trained_models_k-means/'","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:39:21.750124Z","iopub.execute_input":"2024-03-08T08:39:21.750699Z","iopub.status.idle":"2024-03-08T08:39:21.757913Z","shell.execute_reply.started":"2024-03-08T08:39:21.750652Z","shell.execute_reply":"2024-03-08T08:39:21.756595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:39:45.781815Z","iopub.execute_input":"2024-03-08T08:39:45.782301Z","iopub.status.idle":"2024-03-08T08:39:46.925793Z","shell.execute_reply.started":"2024-03-08T08:39:45.782264Z","shell.execute_reply":"2024-03-08T08:39:46.924521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  Prepare features - same as for train  (From \"main-predict\")\n\nThat is the same function as in train. But just use the output \"one_hot_test\" , not the ones used in train: one_hot_encode_features, targets\n\nThe alternative fucntion with \"_mean_only\" - seems not used\n\n","metadata":{}},{"cell_type":"code","source":"%%time\n\none_hot_encode_features, targets, one_hot_test = prepare_augmented_data(data_file=data_file,\n                                                                                id_map_file=id_map_file)\n#     else:\n#         one_hot_encode_features, targets, one_hot_test = prepare_augmented_data_mean_only(data_file=data_file,\n#                                                                                           id_map_file=id_map_file)\n\nunseen_data = torch.tensor(one_hot_test, dtype=torch.float32).to(device)  # Replace X_unseen with your new data\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:26:33.315295Z","iopub.execute_input":"2024-03-08T08:26:33.315818Z","iopub.status.idle":"2024-03-08T08:27:32.887331Z","shell.execute_reply.started":"2024-03-08T08:26:33.315782Z","shell.execute_reply":"2024-03-08T08:27:32.886192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if verbose >= 1000:\n    print(one_hot_encode_features.shape, targets.shape, one_hot_test.shape)\n    print( unseen_data.shape )","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:28:02.361899Z","iopub.execute_input":"2024-03-08T08:28:02.362347Z","iopub.status.idle":"2024-03-08T08:28:02.368415Z","shell.execute_reply.started":"2024-03-08T08:28:02.362317Z","shell.execute_reply":"2024-03-08T08:28:02.367364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  Load model  (from \"main-predict\")\n\n\nSome bug in loader function - model always loaded from \"trained_models\" folder, variable name \"models_foler\" is not used for model.\nSo it cannot find the model. Corrected by direct loading.\n\n\n","metadata":{}},{"cell_type":"code","source":"%%time\nn_components = n_components_list[0]\nd_model = d_models_list[0]\n\n# label_reducer, scaler, transformer_model = load_transformer_model(n_components,\n#                                                                               input_features=\n#                                                                               one_hot_encode_features.shape[1],\n#                                                                               d_model=d_model,\n#                                                                               models_foler=f'{models_dir}',\n#                                                                               device=device)\n\ninput_features = one_hot_encode_features.shape[1]\nmodels_foler=f'{models_dir}'\n\n\n# def load_transformer_model(n_components, input_features, d_model, models_foler='trained_models', device='cuda'):\n\nif 1: # Corrected code for load_transformer_model from utils.py\n    # transformer_model = CustomTransformer(num_features=input_features, num_labels=n_components, d_model=d_model).to(\n    #     device)\n    transformer_model = CustomTransformer_v3(num_features=input_features, num_labels=n_components, d_model=d_model).to(\n        device)\n    # transformer_model = CustomDeeperModel(input_features, d_model, n_components).to(device)\n    transformer_model.load_state_dict(torch.load(f'{models_foler}/transformer_model_{n_components}_{d_model}.pt'))\n    transformer_model.eval()\n    if n_components == 18211:\n#         return None, None, transformer_model\n        pass\n    label_reducer = pickle.load(open(f'{models_foler}/label_reducer_{n_components}_{d_model}.pkl', 'rb'))\n    scaler = pickle.load(open(f'{models_foler}/scaler_{n_components}_{d_model}.pkl', 'rb'))\n#     return label_reducer, scaler, transformer_model\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:47:18.184839Z","iopub.execute_input":"2024-03-08T08:47:18.185806Z","iopub.status.idle":"2024-03-08T08:47:18.743435Z","shell.execute_reply.started":"2024-03-08T08:47:18.185725Z","shell.execute_reply":"2024-03-08T08:47:18.742032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer_models = {}\n\ntransformer_model.eval()\ntransformer_models[f'{n_components},{d_model}'] = (\n    copy.deepcopy(label_reducer), copy.deepcopy(scaler), copy.deepcopy(transformer_model))\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:54:16.341216Z","iopub.execute_input":"2024-03-08T08:54:16.341692Z","iopub.status.idle":"2024-03-08T08:54:16.377052Z","shell.execute_reply.started":"2024-03-08T08:54:16.341658Z","shell.execute_reply":"2024-03-08T08:54:16.376046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# predict_test - from (\"predict.py\")\n\n\nFunction \"predict_test\" contained many small bugs.\n\n- Inference was batch-wise, but results not merged.\n\n- Load sample.csv was from incorrect path\n\n- there was no squeeze for empty dimension for output array before conversion to data-frame, so conversion was not working\n\n\n","metadata":{}},{"cell_type":"code","source":"!cp /kaggle/input/open-problems-single-cell-perturbations/sample_submission.csv sample_submission.csv\n!ls","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"@torch.no_grad()\ndef predict_test(data, models, n_components_list, d_list, batch_size, device='cuda'):\n    num_samples = len(data)\n\n    for i, n_components in enumerate(n_components_list):\n        for j, d_model in enumerate(d_list):\n            combined_outputs = []\n            label_reducer, scaler, transformer_model = models[f'{n_components},{d_model}']\n            transformer_model.eval()\n#             for i in range(0, num_samples, batch_size):\n#                 batch_unseen_data = data[i:i + batch_size]\n#                 transformed_data = transformer_model(batch_unseen_data)\n                \n            transformed_data = transformer_model(data)\n            if scaler:\n                transformed_data = torch.tensor(scaler.inverse_transform(\n                    label_reducer.inverse_transform(transformed_data.cpu().detach().numpy()))).to(device)\n            combined_outputs.append(transformed_data)\n\n            # Stack the combined outputs\n            combined_outputs = torch.stack(combined_outputs, dim=0)\n            \n            combined_outputs = torch.squeeze( combined_outputs )\n            \n            sample_submission = pd.read_csv(\n                f\"sample_submission.csv\")\n            sample_columns = sample_submission.columns\n            sample_columns = sample_columns[1:]\n            print(combined_outputs.cpu().detach().numpy().shape)\n            submission_df = pd.DataFrame(combined_outputs.cpu().detach().numpy(), columns=sample_columns)\n            submission_df.insert(0, 'id', range(255))\n            \n            submission_df.to_csv(f\"result_{n_components}_{d_model}.csv\", index=False)\n            \n            display( submission_df.head(10) )\n            display( submission_df.tail(10) )\n            print(submission_df.shape)\n    return","metadata":{"execution":{"iopub.status.busy":"2024-03-08T09:14:22.157553Z","iopub.execute_input":"2024-03-08T09:14:22.158650Z","iopub.status.idle":"2024-03-08T09:14:22.173063Z","shell.execute_reply.started":"2024-03-08T09:14:22.158594Z","shell.execute_reply":"2024-03-08T09:14:22.171580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\npredict_test(unseen_data, transformer_models, n_components_list, d_models_list, batch_size, device=device)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-08T09:14:22.620593Z","iopub.execute_input":"2024-03-08T09:14:22.621123Z","iopub.status.idle":"2024-03-08T09:14:35.898526Z","shell.execute_reply.started":"2024-03-08T09:14:22.621083Z","shell.execute_reply":"2024-03-08T09:14:35.897413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !head /kaggle/working/sample_submission.csv","metadata":{"execution":{"iopub.status.busy":"2024-03-08T09:23:32.302848Z","iopub.execute_input":"2024-03-08T09:23:32.304075Z","iopub.status.idle":"2024-03-08T09:23:32.309589Z","shell.execute_reply.started":"2024-03-08T09:23:32.304026Z","shell.execute_reply":"2024-03-08T09:23:32.308044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final timing","metadata":{}},{"cell_type":"code","source":"print('%.1f seconds passed total '%(time.time()-t0start) )\nprint('%.1f minutes passed total '%( (time.time()-t0start)/60)  )\nprint('%.2f hours passed total '%( (time.time()-t0start)/3600)  )","metadata":{"execution":{"iopub.status.busy":"2024-03-08T08:02:05.345790Z","iopub.execute_input":"2024-03-08T08:02:05.346904Z","iopub.status.idle":"2024-03-08T08:02:05.354451Z","shell.execute_reply.started":"2024-03-08T08:02:05.346861Z","shell.execute_reply":"2024-03-08T08:02:05.353028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}