{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59094,"databundleVersionId":6541963,"sourceType":"competition"},{"sourceId":6567471,"sourceType":"datasetVersion","datasetId":3755626},{"sourceId":6783194,"sourceType":"datasetVersion","datasetId":3841755}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What is about ?\n\nHere is a componet of U900 team (#13) solution for the \"Open Problems – Single-Cell Perturbations\" challenge.\n\nSee main writeup here: https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/460858\n\nHere are the models developped at the early stage of the challenge. They did not entered the final solution, but had given some exprience.\n\n\nNotebook is somewhat long (sorry), but hope of some use. Resulting predictions (oof+submits, LB scores) stored here https://www.kaggle.com/datasets/alexandervc/open-problems-single-cell-perturbations-submitsetc. Summarized with LB/CV scores here: https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing and discussed here: https://docs.google.com/presentation/d/1wiz0Wmt4D54pqMMsIOyJHuQYMZ3hTBZQQnjbLzwoGYY/edit?usp=sharing. Top models can achieve around LB 0.60* and blended with some publics achieve 0.589 (top public - at the momemnt), see: https://www.kaggle.com/code/liudacheldieva/1-op2-eda-linearsvr-regressor-nn-kishan and reproduced here. \n\nSome extracts and simplifications are in the other notebooks: custom CV-schemes: https://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes, category encoders, chembert, Morgan fingerprints, molecular descriptors: https://www.kaggle.com/code/alexandervc/op2-category-encoders-chembert-fingerpints-moldes, target category encoders: https://www.kaggle.com/alexandervc/op2-target-encoders, \"Gentle\" param tuner: https://www.kaggle.com/alexandervc/op2-gentle-param-tuner . \n\n\nThe notebook provides pipelines/wrappers to do the following:\n\n#### Highlights: \n\n 1 Separate models for each target (dim-reduced) are supported\n 2 Tuning separate params for each of these models (param-by-param hand-made optimizer)\n 3 One param_config rules all: model, feature engineering, dimensional reduction \n Moreover \n 4 Support of different CV-schemes AmbrosM, MT, Random and easy to customize new ones\n 5 Optimizer may work with improving several CV scores simultaneously\n 5 Various category encoders or arbitrary feature sets - easy to add more \n 6 Dimensional reduction or other targets preprocessings: tsvd, pca, ica, etc. - easy to add more\n 7 Service functions which in one line run CV, submit, prepares oof, saves all results, etc \n\n#### One config to ruler them all \nThe idea of the notebook - EVERYTHING related to modeling is described by configuration - dictionary: \"main_config_model_feature_etc\" and all functions accept that configuration as input. The main points - run various models with different features: onehot, target encoded, embeddings, whatever and dimensional reduction schemes (tsvd,pca,ica,...) - just by changing few lines in the configuration - NOT CHANGING THE CODE. \n\nWe have only 2 features ( categorical), but 18211 targets, thus dozens category encoders applied to some of 18211 targets and their derivates (pca,ica...) can provide UNLIMETED NUMBER of engineered FEATURES. To check at least part of these approaches in consistent way - seems worth to pack everything into config, othewise code will become unmanagable.\n\n#### Results stored in the dataset - welcome to use ready submits \n\nSome results are stored in the dataset https://www.kaggle.com/datasets/alexandervc/open-problems-single-cell-perturbations-submitsetc - each subfolder contains one modeling results - oof-predicitons, submits, configuration information etc. These results are generated by function:\ndo_modeling_submit_prepare_save_etc(main_config_model_feature_etc, subdirectory_path_postfix = 'datetime', verbose = 0 )\n\n\n\n#### Some outcomes\nSome outcomes (see the next section for details). \n\n 1 Start with simple baseline by MT LB0.615 - onehot(compound only), Ridge0.1, tsvd30, make some improvemens\n 2 Analysis of the CV schemes - at the moment we see descrepancies CV-LB for all schemes: AmbrosM,MT, Random. \n At the moment MT scheme is considered preferable. But analysis to be continued. \n \n \n#### Key functions\n\n\n go_modeling_separate_model_for_each_target( main_config_model_feature_etc, ...) - main core function to train models and compute preditions \n do_modeling_submit_prepare_save_etc(main_config_model_feature_etc,... ) - launches modeling, computes oof with respect to several CV-schemes, computes submits (direct and ensemled with \"priors\"), saves submits, oof, configs to disk. \n\n#### Thanks to \n\n(please upvote): \n\nCV schemes (see discussion https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/444494 ):\n\nAmbrosM: https://www.kaggle.com/code/ambrosm/scp-quickstart\n\nMT: https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy\n\nTrick to add \"priors\" (aggregates over the cell type and compound - which typically improves scores by 0.01 ): ( df_submit = (1-w_compound - w_cell_type)*(df_submit) + w_compound * df_submit_aggr_compound + w_cell_type * df_submit_aggr_cell_type )\n\nLIUDA CHELDIEVA: https://www.kaggle.com/code/liudacheldieva/streamlined-baseline-approach\n\nZXMKCD : https://www.kaggle.com/code/zmcxjt/streamlined-baseline-approach\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## Outcomes:\n\n#### Optimization for TWO CV scores simultaneusly - i.e. accept new param value if both scores improved\n\n Conclusion: Fail. We get improvement of both CV, but LB goes down. \n We were trying to improve LB0.616 model - \"TE-for-all=tsvd-compounds\", tsvd25, Ridge.\n We strongly improve both CV, but LB goes down to 0.638. \n Other less agrresive optimization with more restricted Alpha and Smoothing searches give 0.619, 0.620\n \n\n LB0.619 69 opt2CV 8h40m alph, sm, 1round [0.995313 2.580669] [0.9953 2.5807], 'alpha': [1e4,1e5,1e3], 'smoothing0': [1e1,1e2,0]\n 68 opt2CV 12h cancel \n 67 opt2CV 12h cancel\n 66 opt2CV 12h cancel\n 65 opt2CV 6h52 alpha smooth0-4 [0.997736 2.608154] [0.9977 2.6082] 'alpha': [1e4,1e5], 'smoothing0': [1e1,1e2] \n LB0.638 64 opt2CV 9h34 alpha smooth0-4 [0.989163 2.476522] [0.9892 2.4765] 'smoothing4': [0,1e1,1e2], 'alpha': [1e3,1e4,1e5]}\n LB0.620 63 opt2CV 7h44 smooth0-4 [0.997175 2.573051] [0.9972 2.5731] 'smoothing4': [0,1e1,1e2] alpha = 2e4\n 62 opt2CV 4h53 smooth0-2 [0.997718 2.593763] [0.9977 2.5938] 'smoothing2': [0,1e1,1e2]\n 61 opt2CV 1h51 smooth0 [0.998429 2.623597] [0.9984 2.6236],'smoothing0': [0,1e1,1e2] alpha =2e4\n 60 opt2CV 2h 'alpha': [1e3,1e4,1e5] [1.007734 2.605643] [1.0077 2.6056] \n\n\n#### V1-20 Around original AmrosM & MT model: Ridge,tsvd, onehot only drug\n \n Up to now: Extensive optimization improves CV a lot, but does not correspond to LB.\n Still some small bonuses on LB.\n \n Let us start with small good things:\n orignal MT LB0.615 CV2.670 (tsvd30, Ridge alpha = 0.1(fit_intercept=False), onehot only drug) \n Uplift: LB0.613 CV2.66 (change alpha -> 1, fit_itercept = True)\n Uplift: LB0.604: blend original MT with priors \n Strange: LB0.605 - same blend with better 0.613 solution gives worse result \n\n Results of the param tuning - much better CV than baselines, but first submit is not promising - so we have not submitted others. \n MT baseline: LB 0.615 Overall 2.670 svd n_comps=30 ridge alpha=0.1\n best (mrrmse) score 2.559229 2.5592 tsvd25 V3 MT scheme 1h\n best (mrrmse) score 2.5572 2.5572 tsvd30 V1 MT scheme 1.5h LB0.621(simple) LB0.619(20random) \n best (mrrmse) score 2.556453 2.5565 tsvd35 V2 MT scheme 1.5h \n best (mrrmse) score 2.555291 2.5553 tsvd50 V4 MT scheme 2.5h\n best (mrrmse) score 2.553397 2.5534 tsvd100 v5 MT scheme 6h5m = best\n\n AmbrosM baseline: LB 0.629 Overall 0.965 svd n_components=100 ridge alpha=5\n best (mrrmse) score 0.963743 0.9637 tsvd25 AmbrosM scheme V14 47min\n best (mrrmse) score 0.962005 0.962 tsvd30 AmbrosM scheme V15 1h18min\n best (mrrmse) score 0.961337 0.9613 tsvd35 AmbrosM scheme V16 2h18\n best (mrrmse) score 0.960724 0.9607 tsvd50 AmbrosM scheme V17 3h21m\n best (mrrmse) score 0.959723 0.9597 tsvd100 AmbrosM scheme V18 more than 8h42min - BEST \n \n \n\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport time\nt0start = time.time() \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n if '/kaggle/input/open-problems-single-cell-perturbations-submitsetc' in dirname: continue \n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-24T09:55:30.941729Z","iopub.execute_input":"2023-10-24T09:55:30.942160Z","iopub.status.idle":"2023-10-24T09:55:32.728631Z","shell.execute_reply.started":"2023-10-24T09:55:30.942128Z","shell.execute_reply":"2023-10-24T09:55:32.727651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data","metadata":{}},{"cell_type":"code","source":"%%time\nfn = '/kaggle/input/open-problems-single-cell-perturbations/de_train.parquet'\ndf_de_train = pd.read_parquet(fn)# , index_col = 0)\nprint(df_de_train.shape)\ndf_de_train\n\n# %%time\nfn = '/kaggle/input/open-problems-single-cell-perturbations/id_map.csv'\ndf_id_map = pd.read_csv(fn)\nprint(df_id_map.shape)\ndisplay(df_id_map)\nfn = '/kaggle/input/open-problems-single-cell-perturbations/sample_submission.csv'\ndf_sample_submit = pd.read_csv(fn, index_col = 0)\nprint(df_sample_submit.shape)\ndisplay( df_sample_submit )","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:32.730938Z","iopub.execute_input":"2023-10-24T09:55:32.731463Z","iopub.status.idle":"2023-10-24T09:55:40.607386Z","shell.execute_reply.started":"2023-10-24T09:55:32.731429Z","shell.execute_reply":"2023-10-24T09:55:40.606324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## \"Prior\" submissions for optional further blend. Averages over cell type and compound ","metadata":{}},{"cell_type":"code","source":"%%time \n#group by drug and take the mean\ndf_tmp = df_de_train.iloc[:, [1] + list(range(5, df_de_train.shape[1]))] # Take only numeric columns and \"sm_name\"\ndf_aggr = df_tmp.groupby('sm_name').mean().reset_index()\nprint(df_aggr.shape)\ndf_submit_aggr_compound = pd.merge( df_id_map, df_aggr, on='sm_name', how = 'left' ).sort_values('id').drop(columns = ['cell_type', 'sm_name']).set_index('id')\nprint(df_submit_aggr_compound.shape)\ndisplay(df_submit_aggr_compound.head(3))\n\nprint( )\n\ndf_tmp = df_de_train.iloc[:, [0] + list(range(5, df_de_train.shape[1]))] # Take only numeric columns and \"cell_type\"\ndf_aggr = df_tmp.groupby('cell_type').mean().reset_index()\nprint(df_aggr.shape)\ndf_submit_aggr_cell_type = pd.merge( df_id_map, df_aggr, on='cell_type', how = 'left' ).sort_values('id').drop(columns = ['cell_type', 'sm_name']).set_index('id')\nprint(df_submit_aggr_cell_type.shape)\ndisplay(df_submit_aggr_cell_type.head(3))","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:40.608700Z","iopub.execute_input":"2023-10-24T09:55:40.609047Z","iopub.status.idle":"2023-10-24T09:55:41.307938Z","shell.execute_reply.started":"2023-10-24T09:55:40.609001Z","shell.execute_reply":"2023-10-24T09:55:41.306776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Core Functions: CV, Modeling, Feature Engineering (Encoding), Prepare Submits, etc ","metadata":{}},{"cell_type":"markdown","source":"# Main modeling function","metadata":{}},{"cell_type":"code","source":"def go_modeling_separate_model_for_each_target( main_config_model_feature_etc, CV_scheme = 'AmbrosM', verbose = 1):\n '''\n Main core function.\n Computes oof, scores, submit predictions, etc.\n For each target (or pca of targets) its own model can be used (different from other targets).\n Configuration of models, features, dimensional reduction is described in main_config_model_feature_etc \n '''\n \n Y = df_de_train.values[:,5:].astype(float) \n Y_oof_pred = np.zeros_like( Y ) # np.zeros( (614, 18211) ) #\n Y_submit_pred = np.zeros( (255, 18211) ) # prediction for submission part will be stored here \n i_submit_blend = 0\n \n list_fold_ids = get_list_fold_ids( CV_scheme, flag_add_full_data=False)\n reducer, str_reducer_id = get_reducer( main_config_model_feature_etc , verbose = 0) \n if verbose >= 100: print(str_reducer_id, reducer) \n t0_all_folds = time.time(); mrrmse_list_train = []; r2_list_train = []\n for i_fold,fold_id in enumerate(list_fold_ids):\n\n mask_tr,mask_va = get_fold_data(fold_id, CV_scheme = CV_scheme )\n\n Y_train = Y[mask_tr,:]\n Y_valid = Y[mask_va,:]\n Y_proc_train = reducer.fit_transform(Y_train) # Y_proc_train - \"processed\" e.g. dimensionally reduced Y \n Y_proc_valid = reducer.transform(Y_valid)\n Y_proc_full = reducer.transform(Y)\n\n Y_proc_valid_pred = np.zeros( Y_proc_valid.shape ) # Predictions \n Y_proc_train_pred = np.zeros( Y_proc_train.shape )\n Y_proc_submit_pred = np.zeros( (Y_submit_pred.shape[0], Y_proc_train.shape[1] ) )\n for i_target in range(Y_proc_train.shape[1] ):\n\n X_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y_proc_full, verbose = 0 )\n #X_train = X[mask_tr,:]\n #X_valid = X[mask_va,:]\n\n model, str_model_id = get_model_for_i_th_target(i_target, main_config_model_feature_etc)\n if verbose >= 1000: print(i_target, str_model_id, model)\n\n model.fit(X_train, Y_proc_train[:,i_target] )\n Y_proc_valid_pred[:, i_target ] = model.predict(X_valid )\n Y_proc_train_pred[:, i_target ] = model.predict(X_train )\n Y_proc_submit_pred[:, i_target ] = model.predict(X_submit )\n\n Y_valid_pred = reducer.inverse_transform( Y_proc_valid_pred )\n Y_train_pred = reducer.inverse_transform( Y_proc_train_pred )\n Y_submit_pred = (Y_submit_pred * i_submit_blend + reducer.inverse_transform( Y_proc_submit_pred ) )/ ( i_submit_blend +1 )\n i_submit_blend += 1\n\n Y_oof_pred[mask_va,:] = Y_valid_pred\n\n mrrmse = np.sqrt(np.square(Y_valid - Y_valid_pred).mean(axis=1)).mean(); r2 = r2_score( Y_valid , Y_valid_pred ) \n if verbose >= 100: print(f\"# Fold: {fold_id}, Valid: mse: {mrrmse:5.3f} r2: {r2:5.3f} \") \n\n mrrmse = np.sqrt(np.square(Y_train - Y_train_pred).mean(axis=1)).mean(); r2 = r2_score( Y_train , Y_train_pred )\n mrrmse_list_train.append(mrrmse); r2_list_train.append(r2)\n if verbose >= 1000: print(f\"# Fold: {fold_id}, Train: mse: {mrrmse:5.3f} r2: {r2:5.3f} \") \n\n\n mrrmse_list, r2_list = get_cv_scores( Y, Y_oof_pred , CV_scheme = CV_scheme, verbose = 0 )\n if verbose >= 10:\n print('by folds mrrmse:',np.round(mrrmse_list,6),'r2:', np.round(r2_list,6) )\n if verbose >= 1:\n print('Valid mrrmse:', np.round(np.mean( mrrmse_list ),6) , 'r2:', np.round(np.mean(r2_list ),6), \n 'Train:', np.round(np.mean( mrrmse_list_train ),6), np.round(np.mean( r2_list_train ),6) )\n\n if verbose >= 10:\n print(f'{time.time()-t0_all_folds:.1f} secs passed')\n \n dict_optional_res = {}; dict_optional_res['mrrmse_list_train'] = mrrmse_list_train; dict_optional_res['r2_list_train'] = r2_list_train\n return Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res\n \nmain_config_model_feature_etc = {'model':'Ridge', 'features_mode':'target_enc', 'list_features_in' : ['sm_name'], 'reducer':'tsvd','n_components':2 } \n# Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, CV_scheme = 'AmbrosM', verbose = 1) \n \n \n# main_config_model_feature_etc = {'model':'Ridge'} \n# Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, CV_scheme = 'AmbrosM', verbose = 1) \n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:41.309445Z","iopub.execute_input":"2023-10-24T09:55:41.309864Z","iopub.status.idle":"2023-10-24T09:55:41.332527Z","shell.execute_reply.started":"2023-10-24T09:55:41.309833Z","shell.execute_reply":"2023-10-24T09:55:41.331594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature engineering, encoding","metadata":{}},{"cell_type":"code","source":"# import numpy as np\n# a = np.array([[1,2],[3,4]])\n# a = np.concatenate( [a, np.array([10,20]).reshape(-1,1) ],axis = 1)\n# a","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:41.335701Z","iopub.execute_input":"2023-10-24T09:55:41.336102Z","iopub.status.idle":"2023-10-24T09:55:41.349475Z","shell.execute_reply.started":"2023-10-24T09:55:41.336062Z","shell.execute_reply":"2023-10-24T09:55:41.348405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nimport time \nimport warnings\nimport numbers\n\nfrom sklearn.preprocessing import OrdinalEncoder\nfrom sklearn.preprocessing import OneHotEncoder\n# from sklearn.preprocessing import TargetEncoder\nimport category_encoders as ce\n\nX_fulltrain_chembert_cls = np.load( '/kaggle/input/chemberta-v2-77-mtr/pad_false/train_ChemBERTa_v2_77MTR_cls_pad_False.npy')\nX_submit_chembert_cls = np.load('/kaggle/input/chemberta-v2-77-mtr/pad_false/test_ChemBERTa_v2_77MTR_cls_pad_False.npy')\nX_fulltrain_chembert_mean = np.load('/kaggle/input/chemberta-v2-77-mtr/pad_false/train_ChemBERTa_v2_77MTR_mean_pad_False.npy')\nX_submit_chembert_mean = np.load('/kaggle/input/chemberta-v2-77-mtr/pad_false/test_ChemBERTa_v2_77MTR_mean_pad_False.npy')\nprint('chemBerta shapes:', X_fulltrain_chembert_cls.shape, X_submit_chembert_cls.shape, X_fulltrain_chembert_mean.shape, X_submit_chembert_mean.shape ) \n\n\ndef get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va , Y = None , verbose = 0 ):\n '''\n Returns train, valid, submit features (probably constructed by some feature engineering).\n \n '''\n\n warnings.filterwarnings(\"ignore\")\n \n # X_full_input - input features\n # X_submit_input - input features\n X_full_input = df_de_train[['cell_type','sm_name']]\n X_submit_input = df_id_map[['cell_type','sm_name']] \n list_default_features_to_process = ['cell_type','sm_name']\n \n if verbose >= 10_000:\n if hasattr(Y,'shape'):\n print('get_features starting', 'i_target', i_target, 'Y.shape', Y.shape )\n else:\n print('get_features starting', 'i_target', i_target, 'type(Y)', type(Y) )\n print('first 100 chars main_config_model_feature_etc ', str(main_config_model_feature_etc)[:100])\n \n main_config_model_feature_etc_loc = main_config_model_feature_etc\n if ( 'i_target_cfg' in main_config_model_feature_etc.keys() ) and ( len(main_config_model_feature_etc['i_target_cfg']) > i_target ):\n main_config_model_feature_etc_loc = main_config_model_feature_etc['i_target_cfg'][i_target]\n \n list_features_to_process = main_config_model_feature_etc_loc.get( 'list_features_in' , list_default_features_to_process) \n if 'features_mode' in main_config_model_feature_etc_loc.keys():\n features_mode = main_config_model_feature_etc_loc['features_mode']\n if verbose >= 10_000:\n print('features_mode', features_mode )\n \n if features_mode.lower() == 'ordinal':\n enc = OrdinalEncoder()\n X_full = enc.fit_transform( X_full_input[ list_features_to_process ] )\n X_submit = enc.transform( X_submit_input[ list_features_to_process ] )\n elif features_mode.lower() == 'onehot':\n enc = OneHotEncoder(handle_unknown='ignore')\n X_full = enc.fit_transform( X_full_input[ list_features_to_process ] ).toarray()\n X_submit = enc.transform( X_submit_input[ list_features_to_process ] ).toarray()\n elif features_mode.lower() == 'chembert_cls':\n X_full = X_fulltrain_chembert_cls# = np.load( '/kaggle/input/chemberta-v2-77-mtr/pad_false/train_ChemBERTa_v2_77MTR_cls_pad_False.npy')\n X_submit = X_submit_chembert_cls# = np.load('/kaggle/input/chemberta-v2-77-mtr/pad_false/test_ChemBERTa_v2_77MTR_cls_pad_False.npy')\n elif features_mode.lower() == 'chembert_mean':\n X_full = X_fulltrain_chembert_mean# = np.load('/kaggle/input/chemberta-v2-77-mtr/pad_false/train_ChemBERTa_v2_77MTR_mean_pad_False.npy')\n X_submit = X_submit_chembert_mean# = np.load('/kaggle/input/chemberta-v2-77-mtr/pad_false/test_ChemBERTa_v2_77MTR_mean_pad_False.npy')\n \n \n elif features_mode in ['HelmertEncoder', 'BackwardDifferenceEncoder' ]:\n enc = getattr(ce,features_mode)(handle_unknown = 0)\n for i_col, col in enumerate(list_features_to_process):\n v = X_full_input[mask_tr][col]\n enc.fit(v )\n if i_col == 0:\n X_full = enc.transform( X_full_input[col] ).values\n X_submit = enc.transform( X_submit_input[col] ).values\n else:\n X_full = np.concatenate( (X_full, enc.transform( X_full_input[col] ).values ) , axis = 1 )\n X_submit = np.concatenate( (X_submit, enc.transform( X_submit_input[col] ).values ) , axis = 1 )\n elif features_mode.lower() == 'CountEncoder'.lower():\n n1 = len( list_features_to_process )\n X_full = np.zeros( (X_full_input.shape[0], n1) )\n X_submit = np.zeros( (X_submit_input.shape[0], n1) )\n for i_col, col in enumerate(list_features_to_process):\n v = X_full_input[mask_tr][col]\n enc = ce.CountEncoder(handle_unknown = 0)\n enc.fit(v )\n X_full[:, i_col ] = enc.transform( X_full_input[col] ).values.ravel()\n X_submit[: , i_col ] = enc.transform( X_submit_input[col] ).values.ravel()\n elif features_mode.lower() == 'target_enc_i_th_target':\n # Target encoder - use only single target component - i_target (input param)\n # smoothing can be float - applies to all transformers, or a vector - than applies same value to all \n # or can be a vector - than we take components of the vector for smoothing each \n \n smoothing_default = main_config_model_feature_etc_loc.get( 'smoothing_default' , 10) # 10 Defailt smoothing for ce.TargetEncoder\n if 'smoothing' in main_config_model_feature_etc_loc.keys():\n smoothing = main_config_model_feature_etc_loc.get( 'smoothing' , 10)\n else:\n # Another way to pass params in form of many scalar: smoothing0, smoothing1, ... - used in some optimizers here \n tmp_dict = {}; tmp_max_ix = -1; \n for k in range(500):\n if 'smoothing'+str(k) in main_config_model_feature_etc_loc.keys():\n tmp_dict[k] = main_config_model_feature_etc_loc['smoothing'+str(k)]\n tmp_max_ix = k\n if len(tmp_dict) > 0:\n smoothing = smoothing_default*np.ones( tmp_max_ix + 1 ) # \n for k in tmp_dict: smoothing[k] = main_config_model_feature_etc_loc['smoothing'+str(k)]\n else:\n smoothing = smoothing_default\n #print(smoothing)\n n1 = len( list_features_to_process )\n X_full = np.zeros( (X_full_input.shape[0], n1) )\n X_submit = np.zeros( (X_submit_input.shape[0], n1) )\n for i_col, col in enumerate(list_features_to_process):\n v = X_full_input[mask_tr][col]\n j_target_loc = i_target\n smoothing_loc = smoothing # 10 Defailt smoothing for ce.TargetEncoder\n if isinstance(smoothing, list) or isinstance(smoothing,np.ndarray) or isinstance(smoothing,tuple ) : \n smoothing_loc = 10\n if ( len( smoothing) > i_col ):\n smoothing_loc = smoothing[ i_col ]\n enc = ce.TargetEncoder(smoothing=smoothing_loc)\n y_tr = Y[mask_tr][:,j_target_loc]\n enc.fit(v, y_tr )\n X_full[:, i_col ] = enc.transform( X_full_input[col] ).values.ravel()\n X_submit[: , i_col ] = enc.transform( X_submit_input[col] ).values.ravel()\n \n elif features_mode.lower() == 'target_enc':\n # Target encoder - all input targets , not only single target component - i_target (input param)\n # smoothing can be float - applies to all transformers, or a vector - than applies same value to all \n # or can be a vector - than we take components of the vector for smoothing each \n \n smoothing_default = main_config_model_feature_etc_loc.get( 'smoothing_default' , 10) # 10 Defailt smoothing for ce.TargetEncoder\n if 'smoothing' in main_config_model_feature_etc_loc.keys():\n smoothing = main_config_model_feature_etc_loc.get( 'smoothing' , 10)\n else:\n # Another way to pass params in form of many scalar: smoothing0, smoothing1, ... - used in some optimizers here \n tmp_dict = {}; tmp_max_ix = -1; \n for k in range(500):\n if 'smoothing'+str(k) in main_config_model_feature_etc_loc.keys():\n tmp_dict[k] = main_config_model_feature_etc_loc['smoothing'+str(k)]\n tmp_max_ix = k\n if len(tmp_dict) > 0:\n smoothing = smoothing_default*np.ones( tmp_max_ix + 1 ) # \n for k in tmp_dict: smoothing[k] = main_config_model_feature_etc_loc['smoothing'+str(k)]\n else:\n smoothing = smoothing_default\n if verbose >= 10_000:\n print('smoothing', smoothing)\n \n n1 = len( list_features_to_process ) * Y.shape[1]\n X_full = np.zeros( (X_full_input.shape[0], n1) )\n X_submit = np.zeros( (X_submit_input.shape[0], n1) )\n for i_col, col in enumerate(list_features_to_process):\n v = X_full_input[mask_tr][col]\n for j_target_loc in range(Y.shape[1]): \n smoothing_loc = smoothing # 10 Defailt smoothing for ce.TargetEncoder\n if isinstance(smoothing, list) or isinstance(smoothing,np.ndarray) or isinstance(smoothing,tuple ) : \n smoothing_loc = 10\n if ( len( smoothing) > i_col*Y.shape[1] + j_target_loc ):\n smoothing_loc = smoothing[ i_col*Y.shape[1] + j_target_loc ]\n \n enc = ce.TargetEncoder(smoothing=smoothing_loc)\n y_tr = Y[mask_tr][:,j_target_loc]\n enc.fit(v, y_tr )\n X_full[:, i_col*Y.shape[1] + j_target_loc ] = enc.transform( X_full_input[col] ).values.ravel()\n X_submit[: , i_col*Y.shape[1] + j_target_loc ] = enc.transform( X_submit_input[col] ).values.ravel()\n elif features_mode in ['LeaveOneOutEncoder', 'JamesSteinEncoder', 'CatBoostEncoder','QuantileEncoder' ]: # 'GLMMEncoder' - got some erros - analyse later \n '''\n Note: option to add noise to train part - we offer in subsequent code - not relying on \"ce\" package - see below. \n '''\n \n dict_encoder_list_tuneable_prms = {}\n dict_encoder_list_tuneable_prms['LeaveOneOutEncoder'] = []\n dict_encoder_list_tuneable_prms['JamesSteinEncoder'] = []\n dict_encoder_list_tuneable_prms['CatBoostEncoder'] = ['a'] # smoothing # big values converts to constant (opposite direction to \"smoothing in TE\")\n dict_encoder_list_tuneable_prms['GLMMEncoder'] = []\n dict_encoder_list_tuneable_prms['QuantileEncoder'] = ['quantile', 'm'] # Tuneable params: two tunable parameter m (default 1) and quantile (default - 0.5)\n #Higher value of m results into stronger shrinking. M is non-negative. 0 for no smoothing.\n #dict_encoder_list_tuneable_prms[''] = []\n\n list_tuneable_prms = dict_encoder_list_tuneable_prms[ features_mode ]\n \n prm_enc_loc = {}\n if 'targets_to_encode' in main_config_model_feature_etc.keys():\n if main_config_model_feature_etc['targets_to_encode'] == 'i_th_target':\n list_targets_indexes = [ i_target ]\n else:\n list_targets_indexes = list(range(Y.shape[1]))\n else:\n list_targets_indexes = list(range(Y.shape[1]))\n \n ix_cnt_total = 0\n for i_col, col in enumerate(list_features_to_process):\n v = X_full_input[mask_tr][col]\n for j_target_loc in list_targets_indexes:\n y_tr = Y[mask_tr][:,j_target_loc]\n prm_enc_loc = get_feature_encoder_prms( main_config_model_feature_etc, list_tuneable_prms , ix_cnt_total, y_tr, i_target, i_col, col, j_target_loc )\n if verbose >= 100_000:\n print(prm_enc_loc, i_col, col, j_target_loc, ix_cnt_total )\n enc = getattr(ce,features_mode)(**prm_enc_loc) # handle_unknown = 0)\n if verbose >= 100_000:\n print(enc)\n enc.fit(v, y_tr )\n if ix_cnt_total == 0:\n X_full = enc.transform( X_full_input[col] ).values.ravel().reshape(-1,1)\n X_submit = enc.transform( X_submit_input[col] ).values.ravel().reshape(-1,1) \n else:\n v_tmp = enc.transform( X_full_input[col] ).values.ravel().reshape(-1,1)\n X_full = np.concatenate( (X_full, v_tmp ), axis = 1 )\n v_tmp = enc.transform( X_submit_input[col] ).values.ravel().reshape(-1,1) \n X_submit = np.concatenate( (X_submit, v_tmp ), axis = 1 ) \n ix_cnt_total += 1\n pass\n# sigma in [None,0.01,0.05,0.1,0.2,0.3,0.4, 0.5,0.6,0.7,0.8,0.9, 1,100]:\n# n1 = len( list_features_to_process )\n# X_full = np.zeros( (X_full_input.shape[0], n1) )\n# X_submit = np.zeros( (X_submit_input.shape[0], n1) )\n# for i_col, col in enumerate(list_features_to_process):\n# v = X_full_input[mask_tr][col]\n# enc = ce.LeaveOneOutEncoder(handle_unknown = 0)\n# enc.fit(v )\n# X_full[:, i_col ] = enc.transform( X_full_input[col] ).values.ravel()\n# X_submit[: , i_col ] = enc.transform( X_submit_input[col] ).values.ravel()\n \n X_train = X_full[ mask_tr,: ]\n X_valid= X_full[ mask_va,: ]\n # Optional noising the train part : \n for ix in range(X_train.shape[1]):\n prm_enc_loc = get_feature_encoder_prms( main_config_model_feature_etc_loc, ['sigma'],ix)\n if verbose >= 100_000:\n print(ix, prm_enc_loc )\n if ('sigma' in prm_enc_loc.keys()) and ( prm_enc_loc['sigma'] is not None ) and ( isinstance(prm_enc_loc['sigma'], numbers.Number) ):\n X_train[:,ix] += np.random.randn( len(X_train) ) * float(prm_enc_loc['sigma'])\n\n \n warnings.filterwarnings(\"default\")\n\n if verbose >= 1:\n print('X_train.shape', X_train.shape, 'X_valid.shape', X_valid.shape, ' X_submit.shaep', X_submit.shape, 'i_target', i_target, 'get_features end' )\n dict_optional_res = {}\n return X_train, X_valid, X_submit, X_full, dict_optional_res \n\ndef get_feature_encoder_prms( main_config_model_feature_etc_loc, list_prms_to_set,ix_cnt_total, y_tr=np.array([]), i_target=0, i_col=0, col='', j_target_loc = 0 ):\n '''\n Sets params for feature encoder, extracting them from main config: main_config_model_feature_etc, taking into account other input params\n For example LeaveOneOutEncoder have a param \"sigma\", to set it :\n assign: list_prms_to_set = ['sigma'] \n Then function look for ['sigma_LeaveOneOutEncoder'] or ['sigma'] or ['sigmaNNN'] keys in main_config_model_feature_etc and get params from that\n and if found will assign prm_enc_loc['sigma'] = that value found in main config\n in case not found - we assign nothing - so encoder will use default value \n '''\n prm_enc_loc = {}\n features_mode = main_config_model_feature_etc_loc.get('features_mode','')\n #if features_mode in ['LeaveOneOutEncoder'] :\n # #prm_enc_loc['handle_unknown'] = np.mean(y_tr) # it is by default\n\n for prm_key in list_prms_to_set:\n # priority for key 'sigma_LeaveOneOutEncoder'\n if prm_key + '_' + str(features_mode) in main_config_model_feature_etc_loc.keys():\n prm_loc = main_config_model_feature_etc_loc[prm_key + '_' + str(features_mode) ]\n elif prm_key in main_config_model_feature_etc_loc.keys():\n prm_loc = main_config_model_feature_etc_loc[prm_key]\n elif prm_key + str( ix_cnt_total ) in main_config_model_feature_etc_loc.keys():\n prm_loc = main_config_model_feature_etc_loc[prm_key + str( ix_cnt_total )]\n else: \n prm_loc = None\n if isinstance(prm_loc, list) or isinstance(prm_loc,np.ndarray) or isinstance(prm_loc,tuple ) : \n if len(prm_loc) > ix_cnt_total:\n prm_enc_loc[prm_key] = prm_loc[ ix_cnt_total ]\n elif prm_loc is not None :\n prm_enc_loc[prm_key] = prm_loc\n else:\n pass # Cannot find information on param in main config - so it will not be assigned and encoder will use default value \n\n return prm_enc_loc\n\n\n#X_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va )\n\ni_target = 0; mask_tr = mask_va = np.ones( len(df_de_train) ).astype(bool);\nY = np.ones( (len(df_de_train), 2 ) ); Y[:,1] = np.arange( len(df_de_train) )\n\nmain_config_model_feature_etc = {'features_mode':'chembert_mean' }\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\n\nmain_config_model_feature_etc = {'features_mode':'CountEncoder', 'list_features_in':['cell_type'] }\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:3])\n\nmain_config_model_feature_etc = {'features_mode':'BackwardDifferenceEncoder', 'list_features_in':['cell_type', 'sm_name'] }\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:3])\n\nmain_config_model_feature_etc = {'features_mode':'HelmertEncoder', 'list_features_in':['cell_type', 'sm_name'] }\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:3])\n\nmain_config_model_feature_etc = {'features_mode':'LeaveOneOutEncoder', 'list_features_in':['cell_type', 'sm_name'] }\nmain_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\nmain_config_model_feature_etc['sigma'] = 0.06\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:3])\n\n\nmain_config_model_feature_etc = {'features_mode':'QuantileEncoder', 'list_features_in':['cell_type', 'sm_name'] }\n# main_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\nmain_config_model_feature_etc['quantile'] = 0.4\nmain_config_model_feature_etc['m'] = 0.1\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:3])\n# print(np.unique(X_train))\n\n# main_config_model_feature_etc = {'features_mode':'GLMMEncoder', 'list_features_in':['cell_type', 'sm_name'] }\n# # main_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\n# X_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 10_000 )\n# print(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\n# print(X_train[:3,:3])\n#\n# Got error: \n# File /opt/conda/lib/python3.10/site-packages/statsmodels/genmod/bayes_mixed_glm.py:1035, in BinomialBayesMixedGLM.__init__(self, endog, exog, exog_vc, ident, vcp_p, fe_p, fep_names, vcp_names, vc_names)\n# 1033 if not np.all(np.unique(endog) == np.r_[0, 1]):\n# 1034 msg = \"endog values must be 0 and 1, and not all identical\"\n# -> 1035 raise ValueError(msg)\n# ValueError: endog values must be 0 and 1, and not all identical\n \n\nmain_config_model_feature_etc = {'features_mode':'JamesSteinEncoder', 'list_features_in':['cell_type', 'sm_name'] }\n# main_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:3,:3])\n\nmain_config_model_feature_etc = {'features_mode':'CatBoostEncoder', 'list_features_in':['cell_type', 'sm_name'] }\n# main_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\nmain_config_model_feature_etc['a'] = 1e3\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:3,:3])\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:41.351340Z","iopub.execute_input":"2023-10-24T09:55:41.351702Z","iopub.status.idle":"2023-10-24T09:55:42.412468Z","shell.execute_reply.started":"2023-10-24T09:55:41.351671Z","shell.execute_reply":"2023-10-24T09:55:42.411274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"main_config_model_feature_etc = {'features_mode':'QuantileEncoder', 'sigma':10, 'list_features_in':['cell_type', 'sm_name'] }\n# main_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\nmain_config_model_feature_etc['quantile'] = 0.4\nmain_config_model_feature_etc['m'] = 0.1\n\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 10_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:5])\n# print(np.unique(X_train))\nprint()\n\n########################################################################################\n\n\nmain_config_model_feature_etc = {'features_mode':'QuantileEncoder', 'sigma0':100, 'sigma1':100, 'list_features_in':['cell_type', 'sm_name'] }\n# main_config_model_feature_etc['targets_to_encode'] = 'i_th_target'\nmain_config_model_feature_etc['quantile'] = 0.4\nmain_config_model_feature_etc['m'] = 0.1\n\nX_train, X_valid, X_submit, X_full, dict_optional_res = get_features(main_config_model_feature_etc, i_target, mask_tr, mask_va, Y , verbose = 100_000 )\nprint(X_train.shape,X_valid.shape, X_submit.shape, X_full.shape, dict_optional_res )\nprint(X_train[:2,:5])\n# print(np.unique(X_train))\nprint()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:42.414002Z","iopub.execute_input":"2023-10-24T09:55:42.414363Z","iopub.status.idle":"2023-10-24T09:55:42.696498Z","shell.execute_reply.started":"2023-10-24T09:55:42.414332Z","shell.execute_reply":"2023-10-24T09:55:42.695381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV schemes","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom sklearn.metrics import r2_score\nfrom sklearn.model_selection import KFold\n\ndef get_list_fold_ids( CV_scheme, flag_add_full_data=False, dict_optional_params = {}, verbose = 0 ):\n if CV_scheme == 'AmbrosM':\n list_fold_ids = ['NK cells', 'T cells CD4+', 'T cells CD8+', 'T regulatory cells']\n elif CV_scheme == 'MT':\n list_fold_ids = [0,1,2]\n elif CV_scheme == 'Full': # All data in one fold - it is mainly for submit preparations\n list_fold_ids = ['Full'] \n elif 'random' in CV_scheme.lower() : # Random folds \n n_splits = int( CV_scheme.split('_')[1] )\n random_state = int( CV_scheme.split('_')[2] )\n list_fold_ids = list(range( n_splits )) \n \n if (flag_add_full_data) and ( CV_scheme != 'Full' ):\n list_fold_ids += ['full']\n \n if verbose >= 100:\n print( list_fold_ids )\n return list_fold_ids\n\n# For MT CV scheme:\nfold_to_compounds = {0: ['Alvocidib', 'Belinostat', 'Foretinib', 'LDN 193189', 'Linagliptin', 'O-Demethylated Adapalene'],\n 1: ['Dabrafenib', 'Dactolisib', 'Idelalisib', 'MLN 2238', 'Palbociclib', 'Porcn Inhibitor III'],\n 2: ['CHIR-99021', 'Crizotinib', 'Oprozomib (ONX 0912)', 'Penfluridol', 'R428']}\n\n# For AmbrosM scheme: \ntrain_sm_names = ['Idelalisib', 'Crizotinib', 'Linagliptin', 'Palbociclib',\n 'Dabrafenib', 'Alvocidib', 'LDN 193189', 'R428',\n 'Porcn Inhibitor III', 'Belinostat', 'Foretinib', 'MLN 2238',\n 'Penfluridol', 'Dactolisib', 'O-Demethylated Adapalene',\n 'Oprozomib (ONX 0912)', 'CHIR-99021']\n# ['NK cells', 'T cells CD4+', 'T cells CD8+', 'T regulatory cells']\n# val_cell_type = 'NK cells'\n# mask_va = (df_de_train.cell_type == val_cell_type) & ~df_de_train.sm_name.isin(train_sm_names)\n# mask_tr = ~mask_va # 485 or 487 training rows\n# print('fold:', mask_va.sum() , mask_tr.sum(), mask_va.sum()+ mask_tr.sum() , )\n\ndef get_fold_data( fold_id , CV_scheme='AmbrosM', verbose = 0):\n if verbose >= 100:\n print(fold_id, CV_scheme, verbose)\n if CV_scheme == 'AmbrosM':\n # fold_id: ['NK cells', 'T cells CD4+', 'T cells CD8+', 'T regulatory cells']\n mask_va = (df_de_train.cell_type == fold_id) & ~df_de_train.sm_name.isin(train_sm_names)\n mask_tr = ~mask_va # 485 or 487 training rows\n elif CV_scheme == 'MT':\n mask_va = df_de_train['cell_type'].isin(['Myeloid cells', 'B cells']) & df_de_train['sm_name'].isin(fold_to_compounds[fold_id])\n mask_tr = ~mask_va \n elif CV_scheme == 'Full': # All data in one fold - it is mainly for submit preparations\n mask_tr = pd.Series(index = df_de_train.index, data = True)\n mask_va = pd.Series(index = df_de_train.index, data = True)\n elif 'random' in CV_scheme.lower() : # Random folds . Like: 'Random_5_42' \n l = (CV_scheme.split('_'))\n n_splits = int( l[1] )\n random_state = int( l[2] )\n if verbose >= 1000:\n print( n_splits,random_state, type(n_splits) )\n kf = KFold(n_splits=n_splits, random_state = 42, shuffle = True )\n df_IX = pd.DataFrame(); df_IX['IX'] = range(len(df_de_train))\n mask_tr = df_de_train.index.isin( list( kf.split(df_IX) )[fold_id][0] )\n mask_va = df_de_train.index.isin( list( kf.split(df_IX) )[fold_id][1] )\n\n \n if verbose >= 100:\n print('mask_tr.sum()', mask_tr.sum() , 'mask_va.sum()' , mask_va.sum() )\n \n return mask_tr, mask_va\n\ndef show_folds_info(CV_scheme, verbose = 0 ):\n list_fold_ids = get_list_fold_ids( CV_scheme, flag_add_full_data=False) \n print('CV_scheme:',CV_scheme, 'n_splits:', len(list_fold_ids ))\n for i_fold,fold_id in enumerate(list_fold_ids):\n mask_tr,mask_va = get_fold_data(fold_id, CV_scheme = CV_scheme )\n print(f\"# Fold: {fold_id}, train size: {mask_tr.sum()}, valid size: {mask_va.sum()} \") \n\ndef get_cv_scores( Y_true, Y_oof_pred , CV_scheme, verbose = 0 ):\n if verbose >= 1000:\n print('CV_scheme',CV_scheme, 'get_cv_scores' )\n list_fold_ids = get_list_fold_ids( CV_scheme, flag_add_full_data=False) \n mrrmse_list = []; r2_list = []\n for i_fold,fold_id in enumerate(list_fold_ids):\n mask_tr,mask_va = get_fold_data(fold_id, CV_scheme = CV_scheme )\n if verbose >= 1000:\n print(f\"# Fold {fold_id}: train size {mask_tr.sum()} valid size: {mask_va.sum()} \") \n \n mrrmse = np.sqrt(np.square(Y_true[mask_va,:] - Y_oof_pred[mask_va,:]).mean(axis=1)).mean()\n r2 = r2_score( Y_true[mask_va,:] , Y_oof_pred[mask_va,:] ) \n mrrmse_list.append(mrrmse) \n r2_list.append(r2) \n if verbose >= 10:\n print(f\"# Fold {fold_id}: mse: {mrrmse:5.3f} r2: {r2:5.3f} \") \n\n return mrrmse_list, r2_list\n\n\nfor CV_scheme in ['AmbrosM','MT', 'Random_5_42','Full']:\n show_folds_info(CV_scheme, verbose = 0 )\n print()\n\n\nY = df_de_train.values[:,5:].astype(float) # Stange but causes problems without that .astype(float) from 09 Oct 2023, np.sqrt line above causes problems\n# mrrmse_list, r2_list = get_cv_scores( Y, Y , CV_scheme = 'AmbrosM', verbose = 0 ) \n# print('mrrmse_list, r2_list', mrrmse_list, r2_list ) \n# mask_tr,mask_va = get_fold_data('NK cells')\n# print(mask_tr.sum() , mask_va.sum() )\n# mrrmse_list, r2_list = get_cv_scores( Y, Y , CV_scheme = 'Full', verbose = 0 ) \n# print('mrrmse_list, r2_list', mrrmse_list, r2_list ) \nmrrmse_list, r2_list = get_cv_scores( Y, Y , CV_scheme = 'Random_5_42', verbose = 0 ) \nprint('mrrmse_list, r2_list', mrrmse_list, r2_list ) \n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:42.698235Z","iopub.execute_input":"2023-10-24T09:55:42.698562Z","iopub.status.idle":"2023-10-24T09:55:45.187510Z","shell.execute_reply.started":"2023-10-24T09:55:42.698533Z","shell.execute_reply":"2023-10-24T09:55:45.186351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Get model, reducer","metadata":{}},{"cell_type":"code","source":"%%time\nfrom sklearn.metrics import r2_score\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn.decomposition import FastICA\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn.linear_model import Ridge\nfrom sklearn.svm import LinearSVR\nfrom sklearn.svm import SVR\nfrom sklearn.kernel_ridge import KernelRidge\n\nimport lightgbm as lgb\nimport catboost\nfrom catboost import CatBoostRegressor, Pool\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import ExtraTreesRegressor\n\n\nfrom sklearn.multioutput import MultiOutputRegressor\n\n\n\ndef get_model(main_config_model_feature_etc , verbose = 0):\n \n str_model_id = str( main_config_model_feature_etc['model'] )\n if main_config_model_feature_etc['model'] == 'Ridge':\n alpha = main_config_model_feature_etc.get('alpha',1)\n fit_intercept = main_config_model_feature_etc.get('fit_intercept',True)\n positive=main_config_model_feature_etc.get('positive',False)\n model = Ridge( alpha , fit_intercept = fit_intercept, positive = positive,)\n str_model_id = 'Ridge'+str(alpha)\n elif main_config_model_feature_etc['model'] == 'LSVR':\n C = main_config_model_feature_etc.get('C',1)# default=1.0 Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive.\n max_iter = main_config_model_feature_etc.get('max_iter',1000) # sklearn default default=1000 \n epsilon = main_config_model_feature_etc.get('epsilon',0.1) # Epsilon parameter in the epsilon-insensitive loss function. Note that the value of this parameter depends on the scale of the target variable y. If unsure, set epsilon=0. \n # 0.1 is used: https://www.kaggle.com/code/mehrankazeminia/1-op2-eda-linearsvr-regressorchain?scriptVersionId=145592242&cellId=88\n model = LinearSVR(max_iter= max_iter, epsilon= epsilon, C=C)\n elif main_config_model_feature_etc['model'] == 'KRR':\n # main_config_model_feature_etc = {'model': 'KRR', 'alpha':1, 'kernel': 'linear', 'gamma': None, 'degree': 3, 'coef0': 1, 'reducer': 'tsvd', 'n_components': n_components, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}\n alpha = main_config_model_feature_etc.get('alpha',1)\n kernel = main_config_model_feature_etc.get('kernel','linear' ) # {‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’} or callable, default=’rbf’\n gamma = main_config_model_feature_etc.get('gamma',None ) # Gamma parameter for the RBF, laplacian, polynomial, exponential chi2 and sigmoid kernels. Interpretation of the default value is left to the kernel; see the documentation for sklearn.metrics.pairwise. Ignored by other kernels.\n coef0 = main_config_model_feature_etc.get('coef0',1.0 ) # Independent term in kernel function. It is only significant in ‘poly’ and ‘sigmoid’.\n degree = main_config_model_feature_etc.get('degree',3 ) ##Degree of the polynomial kernel function (‘poly’). Must be non-negative. Ignored by all other kernels.\n model = KernelRidge(alpha=alpha, kernel = kernel, gamma = gamma, coef0=coef0, degree = degree)\n \n elif main_config_model_feature_etc['model'] == 'SVR':\n # main_config_model_feature_etc = {'model':'SVR', 'kernel':'rbf','C':1 }\n kernel = main_config_model_feature_etc.get('kernel','rbf' ) # {‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’} or callable, default=’rbf’\n C = main_config_model_feature_etc.get('C',1)# default=1.0 Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive.\n epsilon = main_config_model_feature_etc.get('epsilon',0.1) # Epsilon parameter in the epsilon-insensitive loss function. Note that the value of this parameter depends on the scale of the target variable y. If unsure, set epsilon=0. \n gamma = main_config_model_feature_etc.get('gamma','scale' ) # Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’. \n coef0 = main_config_model_feature_etc.get('coef0',0.0 ) # Independent term in kernel function. It is only significant in ‘poly’ and ‘sigmoid’.\n tol = main_config_model_feature_etc.get('tol',0.001 ) #\n degree = main_config_model_feature_etc.get('degree',3 ) ##Degree of the polynomial kernel function (‘poly’). Must be non-negative. Ignored by all other kernels.\n max_iter = main_config_model_feature_etc.get('max_iter',-1) # sklearn default default=1000 \n shrinking = main_config_model_feature_etc.get('shrinking', True) # Whether to use the shrinking heuristic. See the User Guide. \n model = SVR(kernel = kernel, max_iter= max_iter, epsilon= epsilon, C=C, gamma = gamma, coef0=coef0, degree = degree, tol = tol, shrinking=shrinking)\n \n\n elif main_config_model_feature_etc['model'] == 'CATB':\n params_loc = main_config_model_feature_etc.get('CATB_params',{} )\n params_loc['loss_function'] = main_config_model_feature_etc.get('loss_function', 'RMSE' )\n categorical_features = main_config_model_feature_etc.get('categorical_features',[])\n for prm_name in ['iterations', 'depth','learning_rate','subsample', 'colsample_bylevel', 'min_data_in_leaf' ]: # 'l2_leaf_reg' - no effect ? \n if prm_name in main_config_model_feature_etc.keys(): params_loc[prm_name] = main_config_model_feature_etc[prm_name]\n model = CatBoostRegressor(cat_features=categorical_features, verbose = 0, **params_loc ) # Categorical features \n \n # https://forecastegy.com/posts/catboost-hyperparameter-tuning-guide-with-optuna/#subsample-subsample\n # def objective(trial):\n # params = {\n # \"iterations\": 1000,\n # \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-3, 0.1, log=True),\n # \"depth\": trial.suggest_int(\"depth\", 1, 10),\n # \"subsample\": trial.suggest_float(\"subsample\", 0.05, 1.0),\n # \"colsample_bylevel\": trial.suggest_float(\"colsample_bylevel\", 0.05, 1.0),\n # \"min_data_in_leaf\": trial.suggest_int(\"min_data_in_leaf\", 1, 100),\n # }\n\n # model = cb.CatBoostRegressor(**params, silent=True)\n # model.fit(X_train, y_train)\n # predictions = model.predict(X_val)\n # rmse = mean_squared_error(y_val, predictions, squared=False)\n # return rmse\n\n \n elif main_config_model_feature_etc['model'] == 'RFR':\n #n_estimators=100,*, criterion='squared_error', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=1.0, max_leaf_nodes=None, min_impurity_decrease=0.0, bootstrap=True, oob_score=False, n_jobs=None, random_state=None, verbose=0, warm_start=False, ccp_alpha=0.0, max_samples=None\n params_loc = main_config_model_feature_etc.get('RFR_params',{} )\n for prm_name in ['criterion', 'n_estimators', 'max_depth', 'min_samples_split', 'min_samples_split', 'min_samples_leaf', 'min_weight_fraction_leaf', 'max_features', 'max_leaf_nodes','min_impurity_decrease','random_state','ccp_alpha','max_samples' ]:\n if prm_name in main_config_model_feature_etc.keys(): params_loc[prm_name] = main_config_model_feature_etc[prm_name]\n model = RandomForestRegressor( **params_loc ) \n \n elif main_config_model_feature_etc['model'] == 'ETR':\n # n_estimators=100, *, criterion='squared_error', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=1.0, max_leaf_nodes=None, min_impurity_decrease=0.0, bootstrap=False, oob_score=False, n_jobs=None, random_state=None, verbose=0, warm_start=False, ccp_alpha=0.0, max_samples=None\n params_loc = main_config_model_feature_etc.get('ETR_params',{} )\n for prm_name in ['criterion', 'n_estimators', 'max_depth', 'min_samples_split', 'min_samples_split', 'min_samples_leaf', 'min_weight_fraction_leaf', 'max_features', 'max_leaf_nodes','min_impurity_decrease','random_state','ccp_alpha','max_samples' ]:\n if prm_name in main_config_model_feature_etc.keys(): params_loc[prm_name] = main_config_model_feature_etc[prm_name]\n model = ExtraTreesRegressor( **params_loc ) \n \n\n elif main_config_model_feature_etc['model'] == 'LGB':\n params_loc = main_config_model_feature_etc.get('LGB_params',{} )\n for prm_name in ['n_estimators', 'max_depth','learning_rate', 'colsample_bytree', 'subsample', 'random_state', 'reg_alpha', 'reg_lambda', 'num_leaves','min_child_samples' ]:\n if prm_name in main_config_model_feature_etc.keys(): params_loc[prm_name] = main_config_model_feature_etc[prm_name]\n model = lgb.LGBMRegressor( **params_loc ) \n elif main_config_model_feature_etc['model'] == 'LGBcv1_036':\n # Params for LGB just on two categorical features as it is found by optimization in https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning#Optuna+LightGBM\n # But LB score is terrible - \n params_best1_cv1_036 = {'random_state': 0, 'n_estimators': 20, 'reg_alpha': 6.764079452929363, 'reg_lambda': 0.41900776876588564, \n 'colsample_bytree': 0.3, 'subsample': 0.7, 'max_depth': 1, 'learning_rate': 0.08456104070184789, 'num_leaves': 682, 'min_child_samples': 102}\n model = lgb.LGBMRegressor( **params_best1_cv1_036 ) \n \n if verbose >= 100:\n print( str_model_id )\n print( model )\n print( main_config_model_feature_etc )\n \n return model, str_model_id\n\nmodel, str_model_id = get_model({'model':'Ridge'}, verbose = 100)\nprint(model, str_model_id ); print()\nmodel, str_model_id = get_model({'model':'CATB'}, verbose = 100)\nprint(model, str_model_id ); print()\n\n\ndef get_model_for_i_th_target(i_target, main_config_model_feature_etc, verbose = 0):\n str_model_id = ''\n if 'i_target_cfg' in main_config_model_feature_etc.keys():\n main_config_model_feature_etc_loc = main_config_model_feature_etc['i_target_cfg'][i_target]\n model, str_model_id = get_model(main_config_model_feature_etc_loc , verbose = 0)\n else:\n model, str_model_id = get_model(main_config_model_feature_etc , verbose = 0)\n\n if verbose >= 100:\n print( str_model_id )\n print( model )\n print( main_config_model_feature_etc )\n \n return model, str_model_id\n\n\ndef get_reducer( main_config_model_feature_etc , verbose = 0): \n if 'reducer' not in main_config_model_feature_etc.keys():\n n_components = 25\n reducer = TruncatedSVD(n_components=n_components, n_iter=7, random_state=42)\n str_reducer_id = 'tsvd25'\n return reducer, str_reducer_id \n \n str_reducer_id = str( main_config_model_feature_etc['reducer'] )\n if main_config_model_feature_etc['reducer']=='tsvd':\n n_components = main_config_model_feature_etc.get('n_components',25)\n reducer = TruncatedSVD(n_components=n_components, n_iter=7, random_state=42)\n str_reducer_id = 'tsvd'+str( n_components )\n elif main_config_model_feature_etc['reducer']=='ica':\n n_components = main_config_model_feature_etc.get('n_components',25)\n reducer = FastICA(n_components=n_components, random_state=0, whiten='unit-variance')\n str_reducer_id = 'ica'+str( n_components )\n elif main_config_model_feature_etc['reducer']=='pca':\n n_components = main_config_model_feature_etc.get('n_components',25)\n reducer = FastICA(n_components=n_components, random_state=0, whiten='unit-variance')\n str_reducer_id = 'pca'+str( n_components )\n \n if verbose >= 100:\n print( str_reducer_id )\n print( reducer )\n print( main_config_model_feature_etc )\n\n return reducer, str_reducer_id\n\nget_reducer( {'reducer': 'tsvd'} , verbose = 100)","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:45.189017Z","iopub.execute_input":"2023-10-24T09:55:45.189386Z","iopub.status.idle":"2023-10-24T09:55:47.488393Z","shell.execute_reply.started":"2023-10-24T09:55:45.189356Z","shell.execute_reply":"2023-10-24T09:55:47.487098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# get_brief_string_info_on_config","metadata":{}},{"cell_type":"code","source":"def get_brief_string_info_on_config(main_config_model_feature_etc):\n str_inf_cfg = ''\n if 'reducer' in main_config_model_feature_etc.keys():\n str_inf_cfg += main_config_model_feature_etc['reducer']\n n_components = main_config_model_feature_etc.get('n_components', 25)\n str_inf_cfg += str(n_components)\n for key_loc in ['model', 'features_mode' ]:\n if key_loc in main_config_model_feature_etc.keys():\n str_inf_cfg += '_'+main_config_model_feature_etc[key_loc]\n else:\n if ('i_target_cfg' in main_config_model_feature_etc.keys()) and ( 0 in main_config_model_feature_etc['i_target_cfg'].keys() ) :\n main_config_model_feature_etc_tmp = main_config_model_feature_etc['i_target_cfg'][0]\n if key_loc in main_config_model_feature_etc_tmp.keys():\n str_inf_cfg += '_'+main_config_model_feature_etc_tmp[key_loc]\n key_loc = 'list_features_in'\n if key_loc in main_config_model_feature_etc.keys():\n if len( main_config_model_feature_etc[key_loc]) == 2: \n str_inf_cfg += '_'+'BothEncoded'\n else:\n str_inf_cfg += '_'+ str( main_config_model_feature_etc[key_loc][0] ) + 'Encoded'\n else:\n if ('i_target_cfg' in main_config_model_feature_etc.keys()) and ( 0 in main_config_model_feature_etc['i_target_cfg'].keys() ) :\n main_config_model_feature_etc_tmp = main_config_model_feature_etc['i_target_cfg'][0]\n if key_loc in main_config_model_feature_etc_tmp.keys():\n if len( main_config_model_feature_etc_tmp[key_loc]) == 2: \n str_inf_cfg += '_'+'BothEncoded'\n else:\n str_inf_cfg += '_'+ str( main_config_model_feature_etc_tmp[key_loc][0] ) + 'Encoded'\n\n return str_inf_cfg\n\ncfg1 = {'reducer': 'tsvd', 'n_components': 2, 'i_target_cfg': {0: {'model': 'Ridge', 'alpha': 10.0, 'features_mode': 'target_enc_i_th_target',\n 'list_features_in': ['cell_type', 'sm_name'], 'smoothing': [1, 100.0]} } }\nstr_inf_cfg = get_brief_string_info_on_config(cfg1)\nprint(str_inf_cfg)\n\ncfg2 = {'reducer': 'tsvd', 'n_components': 2, 'i_target_cfg': {0: {'model': 'Ridge', 'alpha': 10.0, 'features_mode': 'target_enc_i_th_target',\n 'list_features_in': ['cell_type'], 'smoothing': [1, 100.0]} } }\nstr_inf_cfg = get_brief_string_info_on_config(cfg2)\nprint(str_inf_cfg)\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.489907Z","iopub.execute_input":"2023-10-24T09:55:47.490568Z","iopub.status.idle":"2023-10-24T09:55:47.505743Z","shell.execute_reply.started":"2023-10-24T09:55:47.490532Z","shell.execute_reply":"2023-10-24T09:55:47.504854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# do_modeling_submit_prepare_save_etc\n\n\nService function. Create final output for models with good params. \nFor given config - to launch modeling, several submission preparation schemes\nSave oof, submissions, config information etc \nThe folder will be created. All files will be stored there. \n \nSelected results generated by that function are stored in the dataset https://www.kaggle.com/datasets/alexandervc/open-problems-single-cell-perturbations-submitsetc - each subfolder contains one modeling results - oof-predicitons, submits, configuration information etc.","metadata":{}},{"cell_type":"code","source":"%%time\nimport time\nimport pickle\nimport json\nimport datetime\nimport os \ndef do_modeling_submit_prepare_save_etc(main_config_model_feature_etc, subdirectory_path_postfix = 'datetime', flag_save = True, verbose = 0 ):\n '''\n Service function. Create final output for models with good params. \n For given config - to launch modeling, several submission preparation schemes\n Save oof, submissions, config information etc \n The folder will be created. All files will be stored there. \n '''\n t0 = time.time()\n str_config_inf = get_brief_string_info_on_config(main_config_model_feature_etc)\n print(str_config_inf)\n \n if subdirectory_path_postfix == 'datetime':\n current_datetime = datetime.datetime.now()\n subdirectory_path_postfix = str(current_datetime)[5:16].replace(' ','-').replace(':','-') #creates: month-day-hour-minute\n subdirectory_path = str_config_inf + '_' + subdirectory_path_postfix \n print('subdirectory_path', subdirectory_path )\n \n ### ------------ save config itself ----------------------------------------------### \n if flag_save:\n if not os.path.exists(subdirectory_path):\n os.makedirs(subdirectory_path) # Create the subdirectory\n fn = os.path.join(subdirectory_path,'config.json')\n with open(fn, \"w\") as json_file:\n json.dump(main_config_model_feature_etc, json_file)\n fn = os.path.join(subdirectory_path,'config.pkl')\n with open(fn, \"wb\") as pickle_file:\n pickle.dump(main_config_model_feature_etc, pickle_file)\n\n ### ------------ save OOF predictions ----------------------------------------------### \n print()\n for CV_scheme in ['AmbrosM','MT']:\n print('CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme , verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n cv_score = str(np.round(np.mean(mrrmse_list),4)).replace('.','')\n fn = 'oof_'+CV_scheme + '_CV' + cv_score + '.npy'; fn = os.path.join(subdirectory_path,fn)\n if flag_save:\n print(fn, ' file save')\n np.save(fn , Y_oof_pred )\n print()\n \n ### ------------ save Submissions predictions ----------------------------------------------### \n for CV_scheme in ['Random_20_42', 'Full', ]:\n CV_scheme_inf = CV_scheme\n if CV_scheme == 'Full': CV_scheme_inf = 'simple' \n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme , verbose = 1) \n \n print('CV_scheme', CV_scheme )\n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n \n if 'Random' in CV_scheme:\n cv_score = str(np.round(np.mean(mrrmse_list),4)).replace('.','')\n fn = 'oof_'+CV_scheme + '_CV' + cv_score + '.npy'; fn = os.path.join(subdirectory_path,fn)\n if flag_save:\n print(fn, ' file save')\n np.save(fn , Y_oof_pred )\n print()\n \n df_submit = pd.DataFrame(Y_submit_pred, columns = df_de_train.columns[5:])\n df_submit.index.name = 'id'\n fn = 'submission_' + CV_scheme_inf+ '.csv'; fn = os.path.join(subdirectory_path,fn)\n display(df_submit.head(3))\n if flag_save:\n print(fn, ' file save')\n df_submit.to_csv(fn) \n print()\n ### ------------ save \"blended with priors (trick by ZXMKCD and Cheldieva )\" ----------------------------------------------### \n w_cell_type = 0.1\n w_compound = 0.45\n df_submit = (1-w_compound - w_cell_type)*(df_submit) + w_compound * df_submit_aggr_compound + w_cell_type * df_submit_aggr_cell_type\n fn = 'submission_' + CV_scheme_inf+'_blend_with_priors_'+str(w_compound)+'_'+str( w_cell_type )+'.csv'; fn = os.path.join(subdirectory_path,fn)\n if flag_save:\n print(fn, ' file save')\n df_submit.to_csv(fn) \n display(df_submit.head(3))\n print()\n \n print('%.1f seconds passed '%(time.time()-t0) )\n\n \nmain_config_model_feature_etc = {'model':'Ridge', 'alpha':0.1, 'fit_intercept' : False, 'features_mode':'onehot', 'list_features_in' : ['sm_name'], 'reducer':'tsvd','n_components':30, } \nif 0:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'reproduceMT', flag_save = False ) \n \n# Output:\n\n# tsvd30_Ridge_onehot_sm_nameEncoded\n# subdirectory_path tsvd30_Ridge_onehot_sm_nameEncoded_reproduceMT# CV_scheme AmbrosM\n# Valid mrrmse: 1.022552 r2: -0.318785 Train: 1.107081 0.3464\n# Valid foldwise mrrmse: [1.0746 0.9762 0.9436 1.0958] r2: [ 0.0085 -0.1043 -0.7352 -0.4441]\n# Train foldwise mrrmse: [1.0856 1.1215 1.129 1.0923] r2: [0.3392 0.3333 0.3576 0.3556]\n# CV_scheme MT\n# Valid mrrmse: 2.669793 r2: -0.600294 Train: 1.047872 0.325514\n# Valid foldwise mrrmse: [2.8869 2.9826 2.1398] r2: [-0.7283 -0.1729 -0.8997]\n# Train foldwise mrrmse: [1.0446 1.039 1.06 ] r2: [0.3243 0.3273 0.325 ]\n# Valid mrrmse: 1.266385 r2: -0.85118 Train: 1.06972 0.334287\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9912 1.068 0.9569 1.1219 1.4207 1.2548 1.6323 1.931 1.242 1.0372\n# 0.9413 1.3421 2.4325 1.4284 1.0891 1.1773 1.0735 1.0442 0.7957 1.3476] r2: [ 0.1146 -5.4907 -0.1872 -0.6942 -0.27 -0.0861 -2.1344 -0.202 -0.4483\n# -2.7082 -0.5409 -1.7129 -0.7778 -0.215 -0.6324 -0.3098 -0.1439 -0.0685\n# -0.0963 -0.4196]\n# Train foldwise mrrmse: [1.0847 1.0773 1.0849 1.0784 1.061 1.0772 1.0483 1.0367 1.067 1.0793\n# 1.0879 1.0644 1.0234 1.0596 1.0787 1.0729 1.0795 1.0763 1.0924 1.0648] r2: [0.3264 0.3389 0.3275 0.3318 0.334 0.3292 0.3664 0.3399 0.3359 0.338\n# 0.3315 0.3452 0.3279 0.329 0.3314 0.3315 0.3306 0.3318 0.3296 0.3291]\n# Valid mrrmse: 1.074921 r2: 0.327538 Train: 1.074921 0.327538\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0749] r2: [0.3275]\n# Train foldwise mrrmse: [1.0749] r2: [0.3275]\n# 91.1 seconds passed \n# CPU times: user 3min 2s, sys: 1min 44s, total: 4min 46s\n# Wall time: 1min 31s\n \n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.507018Z","iopub.execute_input":"2023-10-24T09:55:47.507546Z","iopub.status.idle":"2023-10-24T09:55:47.544433Z","shell.execute_reply.started":"2023-10-24T09:55:47.507509Z","shell.execute_reply":"2023-10-24T09:55:47.543332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models gallery and stories about them \n\nHere we collect some selected models mostly submitted to LB and some with kind of comments - CV-scores, what will happen if params will be changed, etc ... ","metadata":{}},{"cell_type":"markdown","source":"## Some first examples - to show how to use \"go_modeling_separate_model_for_each_target\" function","metadata":{}},{"cell_type":"code","source":"%%time\n\nif 0:\n CV_scheme = 'AmbrosM'; main_config_model_feature_etc = {'model':'Ridge', 'features_mode':'onehot', 'reducer':'tsvd','n_components':25} \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme , verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n print()\n CV_scheme = 'MT'; main_config_model_feature_etc = {'model':'Ridge', 'features_mode':'onehot', 'reducer':'tsvd','n_components':25 } \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme, verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n\n print()\n CV_scheme = 'AmbrosM'; main_config_model_feature_etc = {'model':'Ridge', 'features_mode':'target_enc', 'list_features_in' : ['sm_name'], 'reducer':'tsvd','n_components':2 } \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme, verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n print() \n \n# tsvd25_Ridge_onehot CV_scheme AmbrosM\n# Valid mrrmse: 1.509382 r2: -0.493002 Train: 1.146376 0.365074\n# Valid foldwise mrrmse: [1.9662 1.4974 1.202 1.3719] r2: [-0.2536 -0.3358 -0.8697 -0.513 ]\n# Train foldwise mrrmse: [1.1206 1.1557 1.1714 1.1378] r2: [0.3622 0.3553 0.3731 0.3698]\n\n# tsvd25_Ridge_onehot CV_scheme MT\n# Valid mrrmse: 2.891229 r2: -0.46452 Train: 1.071881 0.341264\n# Valid foldwise mrrmse: [3.12 3.2001 2.3536] r2: [-0.4056 -0.2746 -0.7134]\n# Train foldwise mrrmse: [1.0702 1.0611 1.0843] r2: [0.3384 0.3429 0.3424]\n\n# tsvd2_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 1.004335 r2: -0.127058 Train: 1.170478 0.268769\n# Valid foldwise mrrmse: [1.091 1.0198 0.8815 1.0251] r2: [ 0.0668 -0.0455 -0.3804 -0.1491]\n# Train foldwise mrrmse: [1.1445 1.1636 1.2041 1.1697] r2: [0.2617 0.2699 0.2706 0.2729]\n\n# CPU times: user 44.4 s, sys: 22.5 s, total: 1min 6s\n# Wall time: 23 s \n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.545763Z","iopub.execute_input":"2023-10-24T09:55:47.546216Z","iopub.status.idle":"2023-10-24T09:55:47.571513Z","shell.execute_reply.started":"2023-10-24T09:55:47.546185Z","shell.execute_reply":"2023-10-24T09:55:47.570302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ridge+OneHot+tsvd - AmbrosM and MT like models\n\nAmbrosM and MT propsed their CV-schemes and explored these schemes on simple examples of the models: onehot (compound only), Ridge, tsvd for targets.\nMore precisely model is: onehot (compound only), reduce target to N components by tsvd, predict these N-components by Ridge, make tsvd.inverse_transform.\n\nSee: CV schemes (see discussion https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/444494 ):\nAmbrosM: https://www.kaggle.com/code/ambrosm/scp-quickstart\nMT: https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy\n\nThe models differ by N - for tsvd, alpha - Ridge, fit_intercept (True,False) - Ridge. \n\n\nLB 0.629 - original AmbrosM model: alpha = 5, fit_intercept = False, N-tsvd = 100\n\nLB 0.615 - original MT model: alpha = 0.1, fit_intercept = False, N-tsvd = 30 \n\nLB 0.613 - update MT model: alpha = 1 (was 0.1), fit_intercept = True (was False), N-tsvd = 30 (the same as MT) - found by improving CV-MT.\n\n#### CV-MT better CV-AmbrosM (in that case)\n\nOne of the conclusions from here that in that case MT scheme better corresponds to LB, than CV-AmbrosM, because:\n\nThe point is parameter alpha change from 1 to 5 changes is the main reason to change from 0.615 to 0.629, the choice is 5 is optimal for CV-AmbrosM, the choice 0.1 by CV-MT, and 0.1 is better for LB (0.615 vs 0.629) -> thus CV-MT is better CV-AmbrosM (in that case). In general both schemes only partly correspond to LB. \n\n#### Both CV (and also random CV) fail choosing N=2 tsvd - bad on LB and common sense.\n\nN=2 for tsvd is the best for these CV schemes but it gives poor result on LB and not common sense reasonable. \nThus we cannot trust full these CV schemes. \n\n#### LB 0.605 - just blend with \"priors\" \n\nSmall blend with simple constant like solutions improves score - trick by ZXMKCD and Cheldieva - see end of the function \"do_modeling_submit_prepare_save_etc\". (That function automatically generates such files also)\n\n\nSubmission made in version 20 here: https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=145940386\n\n","metadata":{}},{"cell_type":"markdown","source":"### LB 0.613 - bit updated MT model, alpha = 1 (instead of 1), fit_intercept = True (instead of False)\n\n","metadata":{}},{"cell_type":"code","source":"# Alpha =1, is better for fit_intercept = True, instead of 0.1 is better for fit_intercept = False (MT notebook)\n# (LB 0.613 vs LB0.615 (oringal MT)\n \nmain_config_model_feature_etc = {'model':'Ridge', 'alpha':0.1, 'fit_intercept' : True, 'features_mode':'onehot', 'list_features_in' : ['sm_name'], \n 'reducer':'tsvd','n_components':30, } \nif 0:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'MT_fitinterceptTrue', flag_save = False )\n\nmain_config_model_feature_etc = {'model':'Ridge', 'alpha':1, 'fit_intercept' : True, 'features_mode':'onehot', 'list_features_in' : ['sm_name'], \n 'reducer':'tsvd','n_components':30, } \nif 0:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'MT_fitinterceptTrue', flag_save = False )\n \n \n# tsvd30_Ridge_onehot_sm_nameEncoded\n# subdirectory_path tsvd30_Ridge_onehot_sm_nameEncoded_MT_fitinterceptTrue\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 1.023384 r2: -0.322444 Train: 1.107282 0.346441\n# Valid foldwise mrrmse: [1.0741 0.9777 0.9476 1.0942] r2: [ 0.0082 -0.1101 -0.7447 -0.4432]\n# Train foldwise mrrmse: [1.0858 1.1217 1.1293 1.0923] r2: [0.3392 0.3333 0.3576 0.3556]\n\n# CV_scheme MT\n# Valid mrrmse: 2.668008 r2: -0.601851 Train: 1.048056 0.325545\n# Valid foldwise mrrmse: [2.8864 2.9806 2.137 ] r2: [-0.7315 -0.1727 -0.9014]\n# Train foldwise mrrmse: [1.0448 1.0392 1.0602] r2: [0.3243 0.3273 0.325 ]\n\n# Valid mrrmse: 1.266651 r2: -0.853667 Train: 1.069911 0.334317\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9906 1.0684 0.958 1.1235 1.4224 1.2552 1.6335 1.9323 1.2412 1.0371\n# 0.9426 1.3437 2.4317 1.4278 1.0918 1.1763 1.0731 1.0418 0.7952 1.3468] r2: [ 0.1156 -5.5087 -0.1908 -0.6994 -0.2739 -0.0894 -2.1358 -0.2029 -0.447\n# -2.7144 -0.5431 -1.7188 -0.7759 -0.2148 -0.6418 -0.3105 -0.1433 -0.065\n# -0.0935 -0.42 ]\n# Train foldwise mrrmse: [1.0849 1.0775 1.0851 1.0787 1.0612 1.0775 1.0484 1.0368 1.0672 1.0795\n# 1.088 1.0646 1.0236 1.0598 1.0788 1.073 1.0796 1.0765 1.0925 1.0649] r2: [0.3264 0.3389 0.3276 0.3318 0.3341 0.3292 0.3665 0.34 0.3359 0.3381\n# 0.3316 0.3452 0.3279 0.3291 0.3314 0.3315 0.3306 0.3319 0.3296 0.3291]\n \n# Valid mrrmse: 1.075112 r2: 0.327568 Train: 1.075112 0.327568\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0751] r2: [0.3276]\n# Train foldwise mrrmse: [1.0751] r2: [0.3276]\n \n#--------------------------------------------------------------------------------------------------------\n\n# tsvd30_Ridge_onehot_sm_nameEncoded\n# subdirectory_path tsvd30_Ridge_onehot_sm_nameEncoded_MT_fitinterceptTrueAlpha1\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.997847 r2: -0.164359 Train: 1.118492 0.333585\n# Valid foldwise mrrmse: [1.0666 0.977 0.909 1.0387] r2: [ 0.0488 -0.0034 -0.473 -0.2299]\n# Train foldwise mrrmse: [1.0955 1.1301 1.1434 1.1049] r2: [0.3273 0.3213 0.3439 0.3418]\n\n# CV_scheme MT\n# Valid mrrmse: 2.660319 r2: -0.32441 Train: 1.053426 0.314451\n# Valid foldwise mrrmse: [2.7933 3.0215 2.1662] r2: [-0.3347 -0.1219 -0.5166]\n# Train foldwise mrrmse: [1.051 1.0443 1.0649] r2: [0.3115 0.317 0.3148]\n\n# Valid mrrmse: 1.229049 r2: -0.517222 Train: 1.076346 0.323271\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [1.0023 1.0092 0.9599 1.0858 1.3413 1.1612 1.6164 1.8912 1.2111 1.0012\n# 0.9015 1.2891 2.3412 1.4234 1.0526 1.1077 1.0478 1.011 0.7854 1.3417] r2: [ 0.1214 -3.507 -0.0387 -0.416 -0.0466 0.1268 -1.5044 -0.0614 -0.2376\n# -2.01 -0.3064 -1.29 -0.427 -0.0507 -0.3032 -0.0573 -0.0342 -0.0144\n# -0.0444 -0.2434]\n# Train foldwise mrrmse: [1.0894 1.0847 1.0907 1.0847 1.0695 1.0841 1.054 1.0443 1.0745 1.0864\n# 1.0942 1.0722 1.0295 1.0656 1.0856 1.0807 1.0852 1.0834 1.0981 1.07 ] r2: [0.3164 0.3275 0.317 0.3209 0.3223 0.3174 0.3541 0.3271 0.325 0.3275\n# 0.3213 0.3341 0.3155 0.3183 0.321 0.3204 0.3202 0.3217 0.3196 0.3181]\n# Valid mrrmse: 1.080212 r2: 0.317753 Train: 1.080212 0.317753\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0802] r2: [0.3178]\n# Train foldwise mrrmse: [1.0802] r2: [0.3178]\n# 65.7 seconds passed \n# CPU times: user 2min 14s, sys: 1min 14s, total: 3min 28s\n# Wall time: 1min 5s\n\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.573165Z","iopub.execute_input":"2023-10-24T09:55:47.573670Z","iopub.status.idle":"2023-10-24T09:55:47.593761Z","shell.execute_reply.started":"2023-10-24T09:55:47.573625Z","shell.execute_reply":"2023-10-24T09:55:47.592485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reconsider examples from AmbrosM and MT notebooks:\n\nAmbrosM: https://www.kaggle.com/code/ambrosm/scp-quickstart\n\nMT: https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy\n\nAmbrosM and Random CV schemes highlight wrong parameters (with respect to LB) - in contrast to MT at that example - confirmed. \n\nI.e. Ridge alpha = 5 is worse on LB that Ridge 1 or 0.1 - MT scheme - works fine in that case.\n\n\nMoreover we can improve LB a bit (0.615->0.613) by setting fit_intersept = False and alpha = 1 - as suggested by MT scheme. But for AmbrosM scheme fit_intercept = True give significantly worse results than fit_intercept = False. \n\n\n\n\n","metadata":{}},{"cell_type":"code","source":"%%time\nif 0:\n for fit_intercept in [False, True]:\n print('fit_intercept', fit_intercept)\n for n_components in [30,100]:\n print('n_components', n_components )\n for CV_scheme in [ 'AmbrosM', 'MT', 'Random_10_42']:\n print('CV_scheme', CV_scheme )\n for alpha in [0.1, 1, 5,10]:# , 10]:\n main_config_model_feature_etc = {'model':'Ridge', 'alpha':alpha, 'fit_intercept':fit_intercept, 'features_mode':'onehot', 'list_features_in' : ['sm_name'], 'reducer':'tsvd','n_components': n_components } \n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, CV_scheme = CV_scheme, verbose = 0) \n print('mrrmse:', np.round(np.mean(mrrmse_list) ,4), 'alpha', alpha, get_brief_string_info_on_config(main_config_model_feature_etc), \n 'CV_scheme', CV_scheme, np.round(mrrmse_list ,4) )\n # print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n # print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n # print() \n\n print()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.601475Z","iopub.execute_input":"2023-10-24T09:55:47.601974Z","iopub.status.idle":"2023-10-24T09:55:47.615775Z","shell.execute_reply.started":"2023-10-24T09:55:47.601939Z","shell.execute_reply":"2023-10-24T09:55:47.614563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n fit_intercept False\n n_components 30\n CV_scheme AmbrosM\n mrrmse: 1.0226 alpha 0.1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0746 0.9762 0.9436 1.0958]\n mrrmse: 0.9878 alpha 1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0688 0.9617 0.8729 1.0479]\n mrrmse: 0.9662 alpha 5 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0916 0.9767 0.7899 1.0067]\n mrrmse: 0.9737 alpha 10 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.1117 0.9955 0.7791 1.0087]\n\n CV_scheme MT\n mrrmse: 2.6698 alpha 0.1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.8869 2.9826 2.1398]\n mrrmse: 2.677 alpha 1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.8038 3.0378 2.1893]\n mrrmse: 2.8556 alpha 5 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [3.0373 3.1753 2.3542]\n mrrmse: 2.9778 alpha 10 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [3.2455 3.2465 2.4415]\n\n CV_scheme Random_10_42\n mrrmse: 1.2843 alpha 0.1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0441 1.053 1.4345 1.7499 1.1518 1.149 1.9788 1.1702 0.9977 1.1142]\n mrrmse: 1.2329 alpha 1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0079 1.0185 1.2785 1.7184 1.1163 1.0887 1.9048 1.1093 0.9803 1.1066]\n mrrmse: 1.2092 alpha 5 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [0.9628 1.0179 1.1898 1.7333 1.0764 1.0157 1.9089 1.0682 0.9912 1.1281]\n mrrmse: 1.2232 alpha 10 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [0.9547 1.0448 1.2048 1.7658 1.0724 1.0031 1.935 1.089 1.0101 1.1521]\n\n n_components 100\n CV_scheme AmbrosM\n mrrmse: 1.0345 alpha 0.1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0825 0.9738 0.9636 1.1182]\n mrrmse: 0.9928 alpha 1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0706 0.9564 0.8839 1.0604]\n mrrmse: 0.9647 alpha 5 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0887 0.9711 0.7906 1.0083]\n mrrmse: 0.9718 alpha 10 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.1091 0.9914 0.7782 1.0084]\n\n CV_scheme MT\n mrrmse: 2.6749 alpha 0.1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.8958 2.9805 2.1483]\n mrrmse: 2.6783 alpha 1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.8091 3.0338 2.1919]\n mrrmse: 2.8533 alpha 5 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [3.038 3.1707 2.3513]\n mrrmse: 2.9756 alpha 10 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [3.2454 3.2429 2.4385]\n\n CV_scheme Random_10_42\n mrrmse: 1.3003 alpha 0.1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0585 1.0664 1.4583 1.7684 1.1689 1.1558 2.0019 1.1846 1.011 1.1288]\n mrrmse: 1.2403 alpha 1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0147 1.0238 1.2905 1.7288 1.1247 1.0901 1.9156 1.1163 0.986 1.1131]\n mrrmse: 1.2088 alpha 5 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [0.9628 1.016 1.1905 1.7347 1.077 1.013 1.9088 1.0677 0.9903 1.1273]\n mrrmse: 1.2219 alpha 10 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [0.9538 1.0425 1.2041 1.7658 1.0719 1.0007 1.9335 1.0876 1.0087 1.1506]\n\n\n\n fit_intercept True\n n_components 30\n CV_scheme AmbrosM\n mrrmse: 1.0234 alpha 0.1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0741 0.9777 0.9476 1.0942]\n mrrmse: 0.9978 alpha 1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0666 0.977 0.909 1.0387]\n mrrmse: 1.009 alpha 5 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0993 1.0347 0.9033 0.9986]\n mrrmse: 1.0349 alpha 10 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.1295 1.0772 0.9271 1.0058]\n\n CV_scheme MT\n mrrmse: 2.668 alpha 0.1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.8864 2.9806 2.137 ]\n mrrmse: 2.6603 alpha 1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.7933 3.0215 2.1662]\n mrrmse: 2.8064 alpha 5 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.9905 3.1312 2.2976]\n mrrmse: 2.9168 alpha 10 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme MT [3.1841 3.191 2.3753]\n\n CV_scheme Random_10_42\n mrrmse: 1.2846 alpha 0.1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.043 1.0541 1.4365 1.7512 1.152 1.1507 1.9789 1.1705 0.9951 1.114 ]\n mrrmse: 1.2386 alpha 1 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0102 1.0321 1.2953 1.7294 1.1198 1.1041 1.9038 1.1166 0.9665 1.1077]\n mrrmse: 1.2354 alpha 5 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0034 1.069 1.2376 1.7609 1.0975 1.0622 1.9076 1.0958 0.9765 1.1435]\n mrrmse: 1.2592 alpha 10 tsvd30_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0177 1.1114 1.2639 1.7988 1.1036 1.0614 1.9353 1.1225 1.0008 1.1763]\n\n n_components 100\n CV_scheme AmbrosM\n mrrmse: 1.0354 alpha 0.1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0821 0.9753 0.9676 1.1167]\n mrrmse: 1.0033 alpha 1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0691 0.972 0.9202 1.0518]\n mrrmse: 1.0083 alpha 5 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.0975 1.0299 0.9047 1.0011]\n mrrmse: 1.0339 alpha 10 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme AmbrosM [1.1279 1.0738 0.9272 1.0067]\n\n CV_scheme MT\n mrrmse: 2.6731 alpha 0.1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.8953 2.9785 2.1455]\n mrrmse: 2.6619 alpha 1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.7988 3.0177 2.1691]\n mrrmse: 2.8046 alpha 5 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [2.9916 3.1271 2.2951]\n mrrmse: 2.9151 alpha 10 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme MT [3.1846 3.1879 2.3727]\n\n CV_scheme Random_10_42\n mrrmse: 1.3006 alpha 0.1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0574 1.0676 1.4604 1.7697 1.1691 1.1574 2.002 1.185 1.0086 1.1286]\n mrrmse: 1.2459 alpha 1 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.017 1.0375 1.3072 1.7395 1.1283 1.1053 1.9145 1.1235 0.9723 1.114 ]\n mrrmse: 1.2347 alpha 5 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0028 1.067 1.238 1.7618 1.0981 1.0592 1.9073 1.0951 0.9755 1.1422]\n mrrmse: 1.2576 alpha 10 tsvd100_Ridge_onehot_sm_nameEncoded CV_scheme Random_10_42 [1.0161 1.109 1.2628 1.7981 1.1031 1.0585 1.9337 1.1209 0.9991 1.1742]\n\n CPU times: user 35min 29s, sys: 19min 14s, total: 54min 43s\n Wall time: 17min 18s","metadata":{}},{"cell_type":"markdown","source":"### Change n_components \n\n\nFor MT CV scheme n_components = 2 gives outstandingly good result 2.6072 . But LB 0.625 so probably the lesson - should not trust strange CV results. \nSee submit from version 23 https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=146026031\n\n n_components 1 Valid mrrmse: 2.6871 r2: 0.0023 time: 4.4\n n_components 2 Valid mrrmse: 2.6072 r2: -0.1568 time: 9.0 B\n n_components 3 Valid mrrmse: 2.6697 r2: -0.5257 time: 13.7\n","metadata":{}},{"cell_type":"code","source":"%%time\nif 0 :\n import time\n t00 = time.time()\n CV_scheme = 'Random_10_42' # 'AmbrosM'\n l = []\n for n_components in range(1,50):\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components, 'model':'Ridge', 'alpha':0.1, 'fit_intercept':True, \n 'features_mode':'onehot', 'list_features_in' : ['sm_name'] }\n\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc,CV_scheme = CV_scheme, verbose = 0) \n\n print('n_components',n_components, 'Valid mrrmse:', np.round(np.mean( mrrmse_list ),4) , 'r2:', np.round(np.mean(r2_list ),4), 'time:', np.round(time.time() - t00,1) ) \n l.append(np.round(np.mean( mrrmse_list ),4))\n\n print(l)\n plt.figure(figsize = (20,3))\n plt.plot(range(1,50), l)\n plt.xlabel('n_components',fontsize = 20)\n plt.grid()\n plt.title(CV_scheme + ' CV_scheme scores. Ridge onehot compound only', fontsize = 20 )\n plt.show()\n pd.Series(data = l , index = range(1,len(l)+1) ).sort_values().head(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.616794Z","iopub.execute_input":"2023-10-24T09:55:47.617197Z","iopub.status.idle":"2023-10-24T09:55:47.636739Z","shell.execute_reply.started":"2023-10-24T09:55:47.617168Z","shell.execute_reply":"2023-10-24T09:55:47.635834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# MT scheme saved results: \nCV_scheme = 'MT'\nl = [2.6871, 2.6072, 2.6697, 2.6585, 2.6799, 2.6745, 2.6756, 2.6921, 2.6907, 2.6917, 2.6906, 2.6901, 2.6879, 2.6884, 2.6882, 2.6843, 2.6831, 2.6792, 2.6795, 2.6801, 2.6796, 2.6743, 2.6721, 2.6716, 2.6696, 2.6697, 2.6699, 2.6688, 2.6681, 2.668, 2.6679, 2.6678, 2.6687, 2.6689, 2.6688, 2.6683, 2.6681, 2.6683, 2.6688, 2.6688, 2.6688, 2.6688, 2.6687, 2.6688, 2.6691, 2.6689, 2.6695, 2.6695, 2.67]\nprint('Best:')\ndisplay ( pd.Series(data = l , index = range(1,len(l)+1) ).sort_values().head(1) )\nplt.figure(figsize = (20,3))\nplt.plot(range(1,len(l)+1), l)\nplt.xlabel('n_components',fontsize = 20)\nplt.grid()\nplt.title(CV_scheme + ' CV_scheme scores. Ridge0.1 onehot compound only', fontsize = 20 )\nplt.show()\n#print(l)\n\nl = [1.0141, 1.0124, 1.0129, 1.0132, 1.0186, 1.0212, 1.0213, 1.023, 1.0221, 1.0239, 1.0237, 1.0237, 1.0245, 1.0246, 1.0251, 1.0264, 1.027, 1.0271, 1.0268, 1.0263, 1.0254, 1.0252, 1.0247, 1.0237, 1.0235, 1.0235, 1.0235, 1.0233, 1.0232, 1.0234, 1.0234, 1.0233, 1.0236, 1.0238, 1.0241, 1.0242, 1.0246, 1.0249, 1.0251, 1.0251, 1.0254, 1.0255, 1.0255, 1.0256, 1.0258, 1.026, 1.0262, 1.0264, 1.0265]\nCV_scheme = 'AmbrosM'\n# print(l)\nprint('Best:')\ndisplay( pd.Series(data = l , index = range(1,len(l)+1) ).sort_values().head(1) )\nplt.figure(figsize = (20,3))\nplt.plot(range(1,50), l)\nplt.xlabel('n_components',fontsize = 20)\nplt.grid()\nplt.title(CV_scheme + ' CV_scheme scores. Ridge0.1 onehot compound only', fontsize = 20 )\nplt.show()\n\n\n\nCV_scheme = 'Random_10_42'\nl = [1.2829, 1.2699, 1.2703, 1.2723, 1.2784, 1.2782, 1.2803, 1.2825, 1.2823, 1.2829, 1.283, 1.2837, 1.2853, 1.2853, 1.2864, 1.2863, 1.2857, 1.2857, 1.2851, 1.2847, 1.2842, 1.2843, 1.2842, 1.284, 1.2841, 1.2845, 1.2845, 1.2844, 1.2845, 1.2846, 1.2846, 1.2849, 1.2851, 1.2855, 1.2859, 1.2863, 1.2864, 1.2866, 1.2867, 1.287, 1.2872, 1.2874, 1.2876, 1.2878, 1.2879, 1.2883, 1.2886, 1.2888, 1.2891]\nplt.figure(figsize = (20,3))\nplt.plot(range(1,len(l)+1), l)\nplt.xlabel('n_components',fontsize = 20)\nplt.grid()\nplt.title(CV_scheme + ' CV_scheme scores. Ridge0.1 onehot compound only', fontsize = 20 )\nplt.show()\nprint('Best:')\ndisplay ( pd.Series(data = l , index = range(1,len(l)+1) ).sort_values().head(1) )\n#print(l)\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:47.638092Z","iopub.execute_input":"2023-10-24T09:55:47.638711Z","iopub.status.idle":"2023-10-24T09:55:48.641427Z","shell.execute_reply.started":"2023-10-24T09:55:47.638665Z","shell.execute_reply":"2023-10-24T09:55:48.640280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Change Alpha","metadata":{}},{"cell_type":"code","source":"%%time\nimport time\nt00 = time.time()\n\nif 0: \n l = []\n n_components = 32\n for alpha in [1,1e2,1e3,1e4,1e5,1e6]:\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components, 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n 'features_mode':'target_enc_i_th_target', 'list_features_in' : ['cell_type', 'sm_name'] }\n\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc,CV_scheme = CV_scheme, verbose = 0) \n print('alpha', alpha, 'n_components',n_components, 'Valid mrrmse:', np.round(np.mean( mrrmse_list ),4) , 'r2:', np.round(np.mean(r2_list ),4), 'time:', np.round(time.time() - t00,1) ) \n l.append(np.round(np.mean( mrrmse_list ),4))\n\n print(l) \n\n# Outcome: \n# for alpha in [1,1e2,1e3,1e4,1e5,1e6]:\n# main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components, 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n# 'features_mode':'target_enc_i_th_target', 'list_features_in' : ['cell_type', 'sm_name'] }\n# alpha 1 n_components 32 Valid mrrmse: 1.2741 r2: -0.2509 time: 44.5\n# alpha 100.0 n_components 32 Valid mrrmse: 1.2712 r2: -0.2441 time: 87.7\n# alpha 1000.0 n_components 32 Valid mrrmse: 1.2603 r2: -0.2147 time: 130.4\n# alpha 10000.0 n_components 32 Valid mrrmse: 1.2482 r2: -0.1569 time: 174.8\n# alpha 100000.0 n_components 32 Valid mrrmse: 1.2387 r2: -0.0563 time: 217.4\n# alpha 1000000.0 n_components 32 Valid mrrmse: 1.2618 r2: 0.0323 time: 259.4\n# [1.2741, 1.2712, 1.2603, 1.2482, 1.2387, 1.2618]\n# CPU times: user 7min 47s, sys: 4min 8s, total: 11min 55s\n# Wall time: 4min 19s\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:48.642881Z","iopub.execute_input":"2023-10-24T09:55:48.643247Z","iopub.status.idle":"2023-10-24T09:55:48.654666Z","shell.execute_reply.started":"2023-10-24T09:55:48.643216Z","shell.execute_reply":"2023-10-24T09:55:48.653383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Target Encoding only i-th tsvd, Ridge, params optimized , LB0.610... , LB0.605 tsvd, \n\nModel (it is quite simple): predict each i-th tsvd component based only on TWO(!) features - target encoding for ONLY THAT i-th component for cell type and drug. \nUse Ridge or other model for each i-th tsvd (or pca/ica/.... ). \n\nSo params: n_components * ( smoothing for cell type, smoothing for drug, model params) \n\nLB 0.610 (V89) - tsvd50, Ridge , LB 0.611 (V91) - tsvd35, Ridge, LB 0.612 (V25) - tsvd 25 , LB 0.619 (V83) - LSVR, tsvd25, etc... - params optimized by searches. Initial model LB0.612 was found optimized AmbrosM scheme. \n\n\nParams found by \"naive\" optimization with AmbrosM scheme, while code for that naive optimization was not polished needed to change notebook for every step - at the notebook https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145060453\nReconsidered (resubmitted) here in version V25: https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=146036640\n\n\nFail improvement idea - try to reduce Alpha-Ridge - hope it will get better CV , since AmbrosM scheme typically overestimates regularization required for best LB. But ideal fails - we clip alpha-Ridge by 1e5 - but got LB 0.615 insteat of 0.612\n\nFor fixed like 25 - searches unable to find better params improving TWO CV-schemes AmbrosM and MT. \n","metadata":{}},{"cell_type":"markdown","source":"#### Plot params: alpha, smooth","metadata":{}},{"cell_type":"code","source":"dict_best_smooth_celltype_for_component = {0: 0.0, 1: 1.0, 2: 1.0, 3: 1000000000000000.0, 4: 10.0, 5: 1000.0, 6: 1000000000000000.0, 7: 100.0, 8: 1000000000000000.0, 9: 100.0, 10: 10.0, 11: 100.0, 12: 0.0, 13: 1000.0, 14: 10000000000000.0, 15: 100.0, 16: 100.0, 17: 10.0, 18: 1.0, 19: 10000000000000.0, 20: 100.0, 21: 10.0, 22: 0.0, 23: 1.0, 24: 0.0}\n# Version 51: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145015609\ndict_best_smooth_compound_for_component = {0: 1000000000000000.0, 1: 1000000000000000.0, 2: 1000000000000000.0, 3: 1000000000000000.0, 4: 10.0, 5: 1000000000000000.0, 6: 10000000000000.0, 7: 1000000000000000.0, 8: 1000000000000000.0, 9: 100.0, 10: 100.0, 11: 100.0, 12: 100.0, 13: 100.0, 14: 1000000000000000.0, 15: 1000.0, 16: 1000000000000000.0, 17: 100.0, 18: 100.0, 19: 10000000000000.0, 20: 1000000000000000.0, 21: 10000000000000.0, 22: 1000000000000000.0, 23: 100.0, 24: 1000000000000000.0}\n# Version 52: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145015691\ndict_best_alpha_for_component = {0: 1000000.0, 1: 100000.0, 2: 100000.0, 3: 1000000.0, 4: 100000.0, 5: 1000000.0, 6: 1000000.0, 7: 1000000.0, 8: 100000.0, 9: 100000.0, 10: 10000.0, 11: 10000.0, 12: 100000.0, 13: 10000.0, 14: 100000.0, 15: 100000.0, 16: 100000.0, 17: 10000.0, 18: 10000.0, 19: 10000.0, 20: 10000.0, 21: 10000.0, 22: 10000.0, 23: 1000.0, 24: 10000.0}\n# From versions 47 of the notebook:\n# https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144999017\n\nlist_alpha_LB0612_v47 = [ t for t in dict_best_alpha_for_component.values() ]\nprint(list_alpha_LB0612_v47 )\nplt.figure(figsize = (15,4))\nplt.plot(np.log10(list_alpha_LB0612_v47 )) \nplt.title('Log10 of Alphas list_alpha_LB0612_v47', fontsize = 20)\nplt.xlabel('tsvd component index', fontsize = 20)\nplt.grid()\nplt.show()\n\nlist_smooth_celltype_LB0612_v47_v25here = [ t for t in dict_best_smooth_celltype_for_component.values() ]\nprint(list_smooth_celltype_LB0612_v47_v25here )\nplt.figure(figsize = (15,4))\nplt.plot(np.log10(list_smooth_celltype_LB0612_v47_v25here )) \nplt.title('Log10 of list_smooth_celltype_LB0612_v47_v25here', fontsize = 20)\nplt.xlabel('tsvd component index', fontsize = 20)\nplt.grid()\nplt.show()\n\nlist_smooth_compound_LB0612_v47_v25here = [ t for t in dict_best_smooth_compound_for_component.values() ]\nprint(list_smooth_compound_LB0612_v47_v25here )\nplt.figure(figsize = (15,4))\nplt.plot(np.log10(list_smooth_compound_LB0612_v47_v25here )) \nplt.title('Log10 of list_smooth_compound_LB0612_v47_v25here', fontsize = 20)\nplt.xlabel('tsvd component index', fontsize = 20)\nplt.grid()\nplt.show()\n\n\nprint()\nn_components = 25\nfor i_target in range( n_components ):\n alpha = dict_best_alpha_for_component[i_target]\n smoothing = [ np.clip(dict_best_smooth_celltype_for_component[i_target],0,1e4), np.clip(dict_best_smooth_compound_for_component[i_target],0,1e4) ]\n print(i_target, 'alpha', alpha, 'smoothing', smoothing )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:48.656445Z","iopub.execute_input":"2023-10-24T09:55:48.657378Z","iopub.status.idle":"2023-10-24T09:55:49.674668Z","shell.execute_reply.started":"2023-10-24T09:55:48.657335Z","shell.execute_reply":"2023-10-24T09:55:49.673394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Prepate main_config_model_feature_etc for such configuration and run do_modeling_submit_prepare_save_etc","metadata":{}},{"cell_type":"code","source":"%%time\n# CV_scheme AmbrosM n_components 25 Valid mrrmse: 0.9966 r2: -0.1163 time: 4144.4\n# CV_scheme MT n_components 25 Valid mrrmse: 2.6151 r2: -0.2525 time: 4154.4\n \nimport time\nt00 = time.time()\n\nif 0:\n n_components = 25\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components} # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg']={}\n\n for i_target in range( n_components ):\n alpha = dict_best_alpha_for_component[i_target]\n smoothing = [ np.clip(dict_best_smooth_celltype_for_component[i_target],0,1e4), np.clip(dict_best_smooth_compound_for_component[i_target],0,1e4) ]\n print(i_target, 'alpha', alpha, 'smoothing', smoothing )\n main_config_model_feature_etc_tmp = { 'model':'Ridge', 'alpha':alpha, 'features_mode':'target_enc_i_th_target', 'smoothing': smoothing,\n 'list_features_in' : ['cell_type', 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy()\n\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'target_enc_i_th_targetOptmized1' , flag_save = False) \n\n# Report:\n\n# CV_scheme AmbrosM n_components 25 Valid mrrmse: 0.9966 r2: -0.1161 time: 16.8\n# CV_scheme MT n_components 25 Valid mrrmse: 2.6151 r2: -0.2522 time: 29.8\n# tsvd25_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd25_Ridge_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOptmized1\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.996637 r2: -0.116146 Train: 1.133717 0.309676\n# Valid foldwise mrrmse: [1.0755 1.0021 0.8912 1.0178] r2: [ 0.0748 -0.0087 -0.3743 -0.1564]\n# Train foldwise mrrmse: [1.1063 1.1369 1.167 1.1247] r2: [0.3072 0.3024 0.3138 0.3153]\n\n# CV_scheme MT\n# Valid mrrmse: 2.615145 r2: -0.252213 Train: 1.066718 0.291127\n# Valid foldwise mrrmse: [2.6779 3.0432 2.1244] r2: [-0.2436 -0.0998 -0.4133]\n# Train foldwise mrrmse: [1.064 1.059 1.0771] r2: [0.2855 0.2938 0.2941]\n\n# Valid mrrmse: 1.091596 r2: 0.296875 Train: 1.091596 0.296875\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:49.676211Z","iopub.execute_input":"2023-10-24T09:55:49.676620Z","iopub.status.idle":"2023-10-24T09:55:49.690309Z","shell.execute_reply.started":"2023-10-24T09:55:49.676579Z","shell.execute_reply":"2023-10-24T09:55:49.688960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### LB 0.615 (Idea Fail) Let us clip alpha by 1e5 hope to uplift from 0.612, but fail\n\n Idea (fail): since AmbrosM seems to require stronger regularization than actually is good for LB / MT-scheme,\n we try to reduce Ridge-Alpha - and hope to get better LB score. But we fail, score goes a bit down to 0.615 instead of 0.612\n \n Submitted in version 70 here https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=146486445\n \n \n That LB0.615 model:\n -----------------------------------------------------------\n CV_scheme AmbrosM\n Valid mrrmse: 1.009924 r2: -0.188889 Train: 1.122814 0.332774\n Valid foldwise mrrmse: [1.0799 1.0155 0.9143 1.03 ] r2: [ 0.0791 -0.0905 -0.526 -0.2181]\n Train foldwise mrrmse: [1.097 1.1284 1.1483 1.1176] r2: [0.3284 0.322 0.3394 0.3413]\n tsvd25_Ridge_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOpt_clipAlpha1e5/oof_AmbrosM_CV10099.npy file save\n\n CV_scheme MT\n Valid mrrmse: 2.589438 r2: -0.328872 Train: 1.061879 0.314701\n Valid foldwise mrrmse: [2.7457 2.9751 2.0476] r2: [-0.4563 -0.0764 -0.454 ]\n Train foldwise mrrmse: [1.0587 1.0533 1.0736] r2: [0.3099 0.317 0.3172]\n\n Original LB0.612 model:\n -----------------------------------------------------------\n CV_scheme AmbrosM\n Valid mrrmse: 0.996637 r2: -0.116146 Train: 1.133717 0.309676\n Valid foldwise mrrmse: [1.0755 1.0021 0.8912 1.0178] r2: [ 0.0748 -0.0087 -0.3743 -0.1564]\n Train foldwise mrrmse: [1.1063 1.1369 1.167 1.1247] r2: [0.3072 0.3024 0.3138 0.3153]\n\n CV_scheme MT\n Valid mrrmse: 2.615145 r2: -0.252213 Train: 1.066718 0.291127\n Valid foldwise mrrmse: [2.6779 3.0432 2.1244] r2: [-0.2436 -0.0998 -0.4133]\n Train foldwise mrrmse: [1.064 1.059 1.0771] r2: [0.2855 0.2938 0.2941]\n\n","metadata":{}},{"cell_type":"code","source":"%%time\nimport time\nt00 = time.time()\n\n\nif 0:\n n_components = 25\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components} # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg']={}\n for i_target in range( n_components ):\n alpha = np.clip(dict_best_alpha_for_component[i_target],0,1e5)\n smoothing = [ np.clip(dict_best_smooth_celltype_for_component[i_target],0,1e4), np.clip(dict_best_smooth_compound_for_component[i_target],0,1e4) ]\n #print(i_target, 'alpha', alpha, 'smoothing', smoothing )\n main_config_model_feature_etc_tmp = { 'model':'LSVR', 'alpha':alpha, 'features_mode':'target_enc_i_th_target', 'smoothing': smoothing,\n 'list_features_in' : ['cell_type', 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy()\n\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'target_enc_i_th_targetOpt_clipAlpha1e5' , flag_save = True) \n\n\n# tsvd25_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd25_Ridge_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOpt_clipAlpha1e5\n# CV_scheme AmbrosM\n# Valid mrrmse: 1.009924 r2: -0.188889 Train: 1.122814 0.332774\n# Valid foldwise mrrmse: [1.0799 1.0155 0.9143 1.03 ] r2: [ 0.0791 -0.0905 -0.526 -0.2181]\n# Train foldwise mrrmse: [1.097 1.1284 1.1483 1.1176] r2: [0.3284 0.322 0.3394 0.3413]\n# CV_scheme MT\n# Valid mrrmse: 2.589438 r2: -0.328872 Train: 1.061879 0.314701\n# Valid foldwise mrrmse: [2.7457 2.9751 2.0476] r2: [-0.4563 -0.0764 -0.454 ]\n# Train foldwise mrrmse: [1.0587 1.0533 1.0736] r2: [0.3099 0.317 0.3172]\n# Valid mrrmse: 1.236955 r2: -0.68193 Train: 1.083304 0.323622\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9715 1.099 0.9464 1.1319 1.4335 1.237 1.5427 1.8731 1.1855 1.0157\n# 0.9305 1.2909 2.3448 1.3904 1.0753 1.1033 1.0421 1.0142 0.7965 1.3148] r2: [ 2.2380e-01 -5.6808e+00 7.6000e-02 -5.4540e-01 -2.1450e-01 -1.1880e-01\n# -1.5887e+00 -5.3600e-02 -8.8600e-02 -2.4556e+00 -4.5120e-01 -1.4225e+00\n# -5.1580e-01 7.4500e-02 -4.6550e-01 -1.5490e-01 -8.4000e-03 2.5600e-02\n# 5.1000e-03 -2.7910e-01]\n# Train foldwise mrrmse: [1.0967 1.0906 1.0978 1.0908 1.0768 1.0933 1.0619 1.0487 1.0803 1.0934\n# 1.101 1.0774 1.0325 1.0719 1.0937 1.0884 1.0919 1.0939 1.1048 1.0804] r2: [0.3178 0.3329 0.3175 0.3226 0.3249 0.3208 0.3541 0.3238 0.3258 0.325\n# 0.3226 0.3349 0.3132 0.3186 0.3232 0.3168 0.3206 0.3215 0.3204 0.3154]\n# Valid mrrmse: 1.086687 r2: 0.31904 Train: 1.086687 0.31904\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0867] r2: [0.319]\n# Train foldwise mrrmse: [1.0867] r2: [0.319]\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:49.692023Z","iopub.execute_input":"2023-10-24T09:55:49.692397Z","iopub.status.idle":"2023-10-24T09:55:49.714273Z","shell.execute_reply.started":"2023-10-24T09:55:49.692368Z","shell.execute_reply":"2023-10-24T09:55:49.713192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### LB0.610 tsvd50 (Ridge, TE-i-th only) - V89\n\n # CV scores: [0.9943 2.6077][0.994318 2.607723] \n \n # V89 https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=146554313\n \n # Search was done with params: dict_prms_lists = { 'smoothing0': [0,1e4, 1e1,1e2] , 'smoothing1': [1e4, 1e1,1e2,0], 'alpha': [1e6,1e5,1e4],}\n ","metadata":{}},{"cell_type":"code","source":"%%time\n# CV scores: [0.9943 2.6077] [0.994318 2.607723] \nmain_config_model_feature_etc = {'reducer': 'tsvd', 'n_components': 50, 'i_target_cfg': {0: {'model': 'Ridge', 'alpha': 1000000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 1: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 1.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 2: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 1.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 3: {'model': 'Ridge', 'alpha': 1000000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 4: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 0, 'list_features_in': ['cell_type', 'sm_name']}, 5: {'model': 'Ridge', 'alpha': 1000000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 1000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 6: {'model': 'Ridge', 'alpha': 1000000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 7: {'model': 'Ridge', 'alpha': 1000000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 8: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 9: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 10: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 11: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 12: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 13: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 1000.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 14: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 15: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 1000.0, 'list_features_in': ['cell_type', 'sm_name']}, 16: {'model': 'Ridge', 'alpha': 100000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 17: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 18: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 1.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 19: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 20: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 21: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 22: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 23: {'model': 'Ridge', 'alpha': 1000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 1.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 24: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 25: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 26: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 27: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 28: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 29: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 30: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 31: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 32: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 100.0, 'list_features_in': ['cell_type', 'sm_name']}, 33: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10000.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 34: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 35: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 36: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 37: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 10.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 38: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 39: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 40: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 41: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 42: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 43: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 44: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 45: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 46: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 47: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 48: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}, 49: {'model': 'Ridge', 'alpha': 10000.0, 'features_mode': 'target_enc_i_th_target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}}}\n\nfor prm_name in ['smoothing0', 'smoothing1', 'alpha']: # dict_prms_lists.keys():\n list_ix = []\n list_val = []\n for i_target in range(500):\n if 'i_target_cfg' in main_config_model_feature_etc.keys():\n if i_target in main_config_model_feature_etc['i_target_cfg'].keys():\n list_ix.append(i_target); list_val.append( main_config_model_feature_etc['i_target_cfg'][i_target][prm_name] )\n plt.figure(figsize = (15,4))\n if (prm_name in ['alpha','C']) or ( 'smoothing' in prm_name):\n plt.plot(list_ix,np.log10( list_val ) , '*-')\n plt.title(' log10 ' + prm_name ,fontsize = 20 )\n else:\n plt.plot(list_ix, ( list_val ) , '*-')\n plt.title(prm_name ,fontsize = 20 )\n plt.xlabel('target index',fontsize = 20)\n plt.grid()\n plt.show()\n \nif 0: \n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'LB0610' , \n flag_save = False) \n# Report from function above:\n\n# tsvd50_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd50_Ridge_target_enc_i_th_target_BothEncoded_LB0610\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.994318 r2: -0.111647 Train: 1.122945 0.316951\n# Valid foldwise mrrmse: [1.0724 0.9979 0.8901 1.0168] r2: [ 0.0805 -0.0038 -0.3738 -0.1495]\n# Train foldwise mrrmse: [1.0972 1.1291 1.1552 1.1103] r2: [0.3133 0.3076 0.3215 0.3254]\n# CV_scheme MT\n# Valid mrrmse: 2.607723 r2: -0.224573 Train: 1.056371 0.299031\n# Valid foldwise mrrmse: [2.6726 3.0356 2.115 ] r2: [-0.2255 -0.0873 -0.3609]\n# Train foldwise mrrmse: [1.0544 1.0485 1.0662] r2: [0.2934 0.3016 0.3021]\n# Valid mrrmse: 1.211682 r2: -0.451886 Train: 1.078872 0.308166\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9771 1.0147 0.9355 1.0797 1.3689 1.188 1.5611 1.8657 1.1693 0.9828\n# 0.8938 1.2439 2.3511 1.3906 1.0203 1.0695 1.0339 0.9877 0.776 1.3241] r2: [ 2.0360e-01 -3.7058e+00 1.4850e-01 -3.2900e-01 -7.3400e-02 2.6400e-02\n# -1.5357e+00 -2.3000e-03 2.5700e-02 -1.7921e+00 -2.9440e-01 -1.1174e+00\n# -4.3880e-01 1.2940e-01 -1.4720e-01 1.0100e-02 5.7000e-03 4.3100e-02\n# 3.5600e-02 -2.2980e-01]\n# Train foldwise mrrmse: [1.0905 1.0822 1.0931 1.086 1.069 1.0864 1.0607 1.0488 1.0764 1.0897\n# 1.0959 1.0743 1.0338 1.0674 1.0894 1.0842 1.0866 1.0882 1.099 1.0757] \n# Valid mrrmse: 1.080813 r2: 0.304811 Train: 1.080813 0.304811\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0808] r2: [0.3048]\n# Train foldwise mrrmse: [1.0808] r2: [0.3048]","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:49.715619Z","iopub.execute_input":"2023-10-24T09:55:49.716189Z","iopub.status.idle":"2023-10-24T09:55:50.645221Z","shell.execute_reply.started":"2023-10-24T09:55:49.716157Z","shell.execute_reply":"2023-10-24T09:55:50.644080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Extending to tsvd100 ","metadata":{}},{"cell_type":"code","source":"%%time\n# Create config for tsvd100 just interpolating the seen constant region for tsvd50:\nn_components = 100\nmain_config_model_feature_etc_tsvd100 = main_config_model_feature_etc.copy()\nmain_config_model_feature_etc_tsvd100['n_components'] = n_components\nfor i_target in range(50, n_components ):\n alpha = 1e4\n smoothing0, smoothing1 = 0,10000\n smoothing0, smoothing1 = 1e2,1e4\n smoothing = [1e2,1e4]\n main_config_model_feature_etc_tmp = { 'model':'Ridge', 'alpha':alpha, 'features_mode':'target_enc_i_th_target',\n 'smoothing0': smoothing[0],'smoothing1': smoothing[1],\n 'list_features_in' : ['cell_type', 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc_tsvd100['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy() \n \nif 0:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc_tsvd100 , subdirectory_path_postfix = 'extContPrm4tsvd50' , \n flag_save = True) \n \n\n## Version 108 by smoothing0, smoothing1 = 1e2,1e4 - as planned\n# 0.993831 , 2.606942\n\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.993831 r2: -0.111379 Train: 1.118765 0.320006\n# Valid foldwise mrrmse: [1.0717 0.997 0.8899 1.0167] r2: [ 0.0809 -0.003 -0.3741 -0.1493]\n# Train foldwise mrrmse: [1.0933 1.1259 1.1505 1.1053] r2: [0.3163 0.3102 0.3247 0.3289]\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_AmbrosM_CV09938.npy file save\n# CV_scheme MT\n# Valid mrrmse: 2.606942 r2: -0.224347 Train: 1.053236 0.301576\n# Valid foldwise mrrmse: [2.6725 3.0348 2.1135] r2: [-0.2262 -0.0871 -0.3597]\n# Train foldwise mrrmse: [1.0513 1.0454 1.063 ] r2: [0.296 0.3041 0.3046]\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_MT_CV26069.npy file save\n# Valid mrrmse: 1.211007 r2: -0.451304 Train: 1.075489 0.31086\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9761 1.0139 0.9348 1.0786 1.3682 1.1871 1.5605 1.8653 1.1686 0.9825\n# 0.8931 1.2433 2.3508 1.39 1.0192 1.0689 1.0332 0.9873 0.7753 1.3234] r2: [ 2.0550e-01 -3.7051e+00 1.4880e-01 -3.2900e-01 -7.3300e-02 2.6500e-02\n# -1.5346e+00 -2.3000e-03 2.6300e-02 -1.7918e+00 -2.9350e-01 -1.1169e+00\n# -4.3810e-01 1.3000e-01 -1.4620e-01 1.0500e-02 6.6000e-03 4.3400e-02\n# 3.6500e-02 -2.2950e-01]\n# Train foldwise mrrmse: [1.0869 1.079 1.0898 1.0829 1.0655 1.0831 1.0572 1.0456 1.0729 1.0862\n# 1.0924 1.0711 1.0302 1.0639 1.0861 1.0809 1.083 1.0849 1.0954 1.0726] r2: [0.3056 0.3236 0.3042 0.3097 0.3128 0.3082 0.3372 0.3086 0.3128 0.314\n# 0.3101 0.3258 0.2978 0.3033 0.3103 0.3054 0.3087 0.3092 0.3086 0.3012]\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_Random_20_42_CV1211.npy file save\n# Valid mrrmse: 1.077403 r2: 0.307538 Train: 1.077403 0.307538\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0774] r2: [0.3075]\n# Train foldwise mrrmse: [1.0774] r2: [0.3075]\n \n\n## ---------------------------------------------------------------------------------------------\n\n\n## Version 107 - extended incrorrectly by smoothing0, smoothing1 = 0,10000\n# 0.993963 , 2.607182\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.993963 r2: -0.111587 Train: 1.118783 0.319993\n# Valid foldwise mrrmse: [1.0719 0.9973 0.8899 1.0167] r2: [ 0.0807 -0.0035 -0.3741 -0.1494]\n# Train foldwise mrrmse: [1.0933 1.1259 1.1505 1.1053] r2: [0.3162 0.3102 0.3247 0.3289]\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_AmbrosM_CV0994.npy file save\n\n# CV_scheme MT\n# Valid mrrmse: 2.607182 r2: -0.224514 Train: 1.053246 0.301572\n# Valid foldwise mrrmse: [2.6726 3.0351 2.1139] r2: [-0.2263 -0.0872 -0.36 ]\n# Train foldwise mrrmse: [1.0514 1.0454 1.063 ] r2: [0.296 0.3041 0.3045]\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_MT_CV26072.npy file save\n\n# Valid mrrmse: 1.211001 r2: -0.45128 Train: 1.075481 0.310873\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.976 1.0139 0.9348 1.0787 1.3682 1.1871 1.5605 1.8653 1.1685 0.9825\n# 0.8931 1.2433 2.3508 1.39 1.0193 1.0689 1.0332 0.9873 0.7752 1.3234] r2: [ 2.0570e-01 -3.7051e+00 1.4870e-01 -3.2900e-01 -7.3300e-02 2.6500e-02\n# -1.5345e+00 -2.3000e-03 2.6400e-02 -1.7918e+00 -2.9350e-01 -1.1169e+00\n# -4.3810e-01 1.3000e-01 -1.4620e-01 1.0600e-02 6.7000e-03 4.3400e-02\n# 3.6600e-02 -2.2950e-01]\n# Train foldwise mrrmse: [1.0868 1.079 1.0898 1.0829 1.0655 1.0831 1.0572 1.0456 1.0729 1.0862\n# 1.0923 1.0711 1.0303 1.0639 1.0861 1.0809 1.083 1.0849 1.0954 1.0726] r2: [0.3057 0.3236 0.3042 0.3097 0.3128 0.3082 0.3372 0.3086 0.3128 0.314\n# 0.3102 0.3258 0.2978 0.3033 0.3103 0.3054 0.3087 0.3092 0.3087 0.3012]\n# tsvd100_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_Random_20_42_CV1211.npy file save\n\n# Valid mrrmse: 1.077388 r2: 0.307561 Train: 1.077388 0.307561\n# CV_scheme Full\n# Valid foldwise mrrmse: [1.0774] r2: [0.3076]\n# Train foldwise mrrmse: [1.0774] r2: [0.3076]\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.647224Z","iopub.execute_input":"2023-10-24T09:55:50.647671Z","iopub.status.idle":"2023-10-24T09:55:50.663075Z","shell.execute_reply.started":"2023-10-24T09:55:50.647631Z","shell.execute_reply":"2023-10-24T09:55:50.661699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(str(main_config_model_feature_etc_tsvd100)[:100] )\nprint(str(main_config_model_feature_etc_tsvd100)[-100:] )","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.664805Z","iopub.execute_input":"2023-10-24T09:55:50.665345Z","iopub.status.idle":"2023-10-24T09:55:50.681406Z","shell.execute_reply.started":"2023-10-24T09:55:50.665295Z","shell.execute_reply":"2023-10-24T09:55:50.679998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Extending to tsvd200,tsvd300 - potential gain is less than 0.001 comparing to tsvd100","metadata":{}},{"cell_type":"code","source":"%%time\n# Create config for tsvd100 just interpolating the seen constant region for tsvd50:\nn_components = 200\nmain_config_model_feature_etc_tsvd200 = main_config_model_feature_etc_tsvd100.copy()\nmain_config_model_feature_etc_tsvd200['n_components'] = n_components\nfor i_target in range(100, n_components ):\n smoothing = [1e2,1e4]\n main_config_model_feature_etc_tmp = { 'model':'Ridge', 'alpha':alpha, 'features_mode':'target_enc_i_th_target',\n 'smoothing0': smoothing[0],'smoothing1': smoothing[1],\n 'list_features_in' : ['cell_type', 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc_tsvd200['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy() \n \nprint(str(main_config_model_feature_etc_tsvd200)[:100] )\nprint(str(main_config_model_feature_etc_tsvd200)[-100:] )\n\nif 0:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc_tsvd200 , subdirectory_path_postfix = 'extContPrm4tsvd200' , \n flag_save = True) \n \n# {'reducer': 'tsvd', 'n_components': 200, 'i_target_cfg': {0: {'model': 'Ridge', 'alpha': 1000000.0, \n# target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}}}\n# tsvd200_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd200_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.993386 r2: -0.110786 Train: 1.116937 0.321293\n# Valid foldwise mrrmse: [1.0713 0.9966 0.8894 1.0163] r2: [ 0.0813 -0.0026 -0.3731 -0.1487]\n# Train foldwise mrrmse: [1.0918 1.1244 1.1485 1.1031] r2: [0.3174 0.3113 0.326 0.3305]\n# tsvd200_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_AmbrosM_CV09934.npy file save\n\n# CV_scheme MT\n# Valid mrrmse: 2.606661 r2: -0.224087 Train: 1.051765 0.302751\n# Valid foldwise mrrmse: [2.6723 3.0346 2.1131] r2: [-0.2262 -0.0869 -0.3591]\n# Train foldwise mrrmse: [1.0499 1.0439 1.0615] r2: [0.2973 0.3053 0.3057]\n# tsvd200_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_MT_CV26067.npy file save\n\n# Valid mrrmse: 1.210668 r2: -0.450951 Train: 1.073905 0.312068\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9759 1.0136 0.9345 1.0781 1.3679 1.1867 1.5602 1.8651 1.1683 0.9822\n# 0.8927 1.243 2.3506 1.3897 1.0188 1.0687 1.0328 0.9871 0.7749 1.3228] r2: [ 2.0590e-01 -3.7048e+00 1.4920e-01 -3.2870e-01 -7.2900e-02 2.6900e-02\n# -1.5343e+00 -2.3000e-03 2.6700e-02 -1.7915e+00 -2.9290e-01 -1.1166e+00\n# -4.3800e-01 1.3040e-01 -1.4560e-01 1.0700e-02 7.1000e-03 4.3800e-02\n# 3.7100e-02 -2.2940e-01]\n# Train foldwise mrrmse: [1.0853 1.0775 1.0882 1.0813 1.064 1.0815 1.0556 1.0441 1.0713 1.0846\n# 1.0907 1.0696 1.0287 1.0623 1.0845 1.0794 1.0814 1.0834 1.0938 1.071 ] r2: [0.3068 0.3248 0.3055 0.3109 0.314 0.3094 0.3385 0.3099 0.3139 0.3152\n# 0.3113 0.327 0.299 0.3045 0.3115 0.3066 0.3099 0.3104 0.3098 0.3025]\n# tsvd200_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_Random_20_42_CV12107.npy file save\n \n \nn_components = 300\nmain_config_model_feature_etc_tsvd300 = main_config_model_feature_etc_tsvd100.copy()\nmain_config_model_feature_etc_tsvd300['n_components'] = n_components\nfor i_target in range(100, n_components ):\n smoothing = [1e2,1e4]\n main_config_model_feature_etc_tmp = { 'model':'Ridge', 'alpha':alpha, 'features_mode':'target_enc_i_th_target',\n 'smoothing0': smoothing[0],'smoothing1': smoothing[1],\n 'list_features_in' : ['cell_type', 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc_tsvd300['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy() \n \nprint(str(main_config_model_feature_etc_tsvd300)[:100] )\nprint(str(main_config_model_feature_etc_tsvd300)[-100:] )\n\nif 0:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc_tsvd300 , subdirectory_path_postfix = 'extContPrm4tsvd300' , \n flag_save = True) \n \n# {'reducer': 'tsvd', 'n_components': 200, 'i_target_cfg': {0: {'model': 'Ridge', 'alpha': 1000000.0, \n# target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}}}\n# {'reducer': 'tsvd', 'n_components': 300, 'i_target_cfg': {0: {'model': 'Ridge', 'alpha': 1000000.0, \n# target', 'smoothing0': 100.0, 'smoothing1': 10000.0, 'list_features_in': ['cell_type', 'sm_name']}}}\n# tsvd300_Ridge_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd300_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.993351 r2: -0.11075 Train: 1.116095 0.321885\n# Valid foldwise mrrmse: [1.0713 0.9965 0.8894 1.0163] r2: [ 0.0814 -0.0025 -0.3731 -0.1487]\n# Train foldwise mrrmse: [1.091 1.1236 1.1477 1.1022] r2: [0.3179 0.3119 0.3266 0.3311]\n# tsvd300_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_AmbrosM_CV09934.npy file save\n\n# CV_scheme MT\n# Valid mrrmse: 2.606639 r2: -0.224095 Train: 1.051277 0.303145\n# Valid foldwise mrrmse: [2.6723 3.0346 2.1131] r2: [-0.2262 -0.0869 -0.3592]\n# Train foldwise mrrmse: [1.0494 1.0434 1.061 ] r2: [0.2977 0.3056 0.3061]\n# tsvd300_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_MT_CV26066.npy file save\n\n# Valid mrrmse: 1.210643 r2: -0.450934 Train: 1.073364 0.312484\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [0.9758 1.0136 0.9344 1.0781 1.3678 1.1867 1.5602 1.865 1.1683 0.9822\n# 0.8926 1.2429 2.3506 1.3897 1.0187 1.0686 1.0328 0.987 0.7749 1.3228] r2: [ 2.0590e-01 -3.7048e+00 1.4920e-01 -3.2870e-01 -7.2800e-02 2.6900e-02\n# -1.5343e+00 -2.2000e-03 2.6700e-02 -1.7915e+00 -2.9290e-01 -1.1165e+00\n# -4.3800e-01 1.3040e-01 -1.4550e-01 1.0700e-02 7.1000e-03 4.3900e-02\n# 3.7100e-02 -2.2940e-01]\n# Train foldwise mrrmse: [1.0847 1.0769 1.0876 1.0807 1.0634 1.081 1.0551 1.0435 1.0708 1.0841\n# 1.0902 1.0691 1.0282 1.0618 1.0839 1.0788 1.0809 1.0829 1.0933 1.0705] r2: [0.3072 0.3252 0.3059 0.3113 0.3144 0.3098 0.339 0.3103 0.3143 0.3156\n# 0.3117 0.3274 0.2994 0.3049 0.3119 0.307 0.3103 0.3108 0.3102 0.3029]\n# tsvd300_Ridge_target_enc_i_th_target_BothEncoded_extContPrm4tsvd50/oof_Random_20_42_CV12106.npy file save\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.682973Z","iopub.execute_input":"2023-10-24T09:55:50.683844Z","iopub.status.idle":"2023-10-24T09:55:50.708012Z","shell.execute_reply.started":"2023-10-24T09:55:50.683809Z","shell.execute_reply":"2023-10-24T09:55:50.706634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## LB0.620 LSVR tsvd, Target Encoding only i-th tsvd\n\nExaplored several C (inverse transform to regularization strength)\n\nIt seems C = 1/100 the most reasonable. Though AmbrosM is better for 1/1000. Submit gives LB 0.620 - submit in version 74 here. \n\nWe use TE for both cell type and compound by only i-th tsvd. So to predict i-th tsvd we use just 2 features made from it - the same approach used for Ridge above. We take smoothing params from the best (for the moment) Ridge model with LB0.612. \n\nBelow we considered several choices for \"C\" it seemed 1/100 is good to start as balance between AmbrosM and MT CV's scores. \n\nIn versions 75, 76 we launched search for C and C,smoothings to improve BOTH CV-scores.\n","metadata":{}},{"cell_type":"code","source":"%%time\nimport time\nt00 = time.time()\n\ndict_best_smooth_celltype_for_component = {0: 0.0, 1: 1.0, 2: 1.0, 3: 1000000000000000.0, 4: 10.0, 5: 1000.0, 6: 1000000000000000.0, 7: 100.0, 8: 1000000000000000.0, 9: 100.0, 10: 10.0, 11: 100.0, 12: 0.0, 13: 1000.0, 14: 10000000000000.0, 15: 100.0, 16: 100.0, 17: 10.0, 18: 1.0, 19: 10000000000000.0, 20: 100.0, 21: 10.0, 22: 0.0, 23: 1.0, 24: 0.0}\n# Version 51: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145015609\ndict_best_smooth_compound_for_component = {0: 1000000000000000.0, 1: 1000000000000000.0, 2: 1000000000000000.0, 3: 1000000000000000.0, 4: 10.0, 5: 1000000000000000.0, 6: 10000000000000.0, 7: 1000000000000000.0, 8: 1000000000000000.0, 9: 100.0, 10: 100.0, 11: 100.0, 12: 100.0, 13: 100.0, 14: 1000000000000000.0, 15: 1000.0, 16: 1000000000000000.0, 17: 100.0, 18: 100.0, 19: 10000000000000.0, 20: 1000000000000000.0, 21: 10000000000000.0, 22: 1000000000000000.0, 23: 100.0, 24: 1000000000000000.0}\n# Version 52: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145015691\ndict_best_alpha_for_component = {0: 1000000.0, 1: 100000.0, 2: 100000.0, 3: 1000000.0, 4: 100000.0, 5: 1000000.0, 6: 1000000.0, 7: 1000000.0, 8: 100000.0, 9: 100000.0, 10: 10000.0, 11: 10000.0, 12: 100000.0, 13: 10000.0, 14: 100000.0, 15: 100000.0, 16: 100000.0, 17: 10000.0, 18: 10000.0, 19: 10000.0, 20: 10000.0, 21: 10000.0, 22: 10000.0, 23: 1000.0, 24: 10000.0}\n# From versions 47 of the notebook:\n# https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144999017\n\n\nif 0:\n n_components = 25\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components} # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg']={}\n for i_target in range( n_components ):\n C = 1/100# 1e4 / np.clip(dict_best_alpha_for_component[i_target],0,1e10)\n smoothing = [ np.clip(dict_best_smooth_celltype_for_component[i_target],0,1e4), np.clip(dict_best_smooth_compound_for_component[i_target],0,1e4) ]\n #print(i_target, 'alpha', alpha, 'smoothing', smoothing )\n main_config_model_feature_etc_tmp = { 'model':'LSVR', 'C':C, 'features_mode':'target_enc_i_th_target', 'smoothing': smoothing,\n 'list_features_in' : ['cell_type', 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy()\n\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'target_enc_i_th_targetOpt_LVSRc1d100' , flag_save = True) \n if 0:\n for CV_scheme in ['AmbrosM','MT']:\n print('CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme , verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n cv_score = str(np.round(np.mean(mrrmse_list),4)).replace('.','')\n #print()\n\n# C = 1/100\n# tsvd25_LSVR_target_enc_i_th_target_BothEncoded\n# subdirectory_path tsvd25_LSVR_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOpt_LVSRc1d100\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.975566 r2: -0.060393 Train: 1.181659 0.230809\n# Valid foldwise mrrmse: [1.0767 0.9746 0.8456 1.0055] r2: [ 0.0462 0.1097 -0.2727 -0.1248]\n# Train foldwise mrrmse: [1.1696 1.1871 1.1985 1.1715] r2: [0.2013 0.2239 0.259 0.2391]\n# tsvd25_LSVR_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOpt_LVSRc1d100/oof_AmbrosM_CV09756.npy file save\n\n# CV_scheme MT\n# Valid mrrmse: 2.837578 r2: -0.058018 Train: 1.112733 0.189772\n# Valid foldwise mrrmse: [3.0566 3.1481 2.308 ] r2: [ 0.0579 -0.0978 -0.1341]\n# Train foldwise mrrmse: [1.1065 1.1063 1.1254] r2: [0.1866 0.1919 0.1908]\n# tsvd25_LSVR_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOpt_LVSRc1d100/oof_MT_CV28376.npy file save\n\n# Valid mrrmse: 1.214168 r2: -0.100559 Train: 1.137613 0.206346\n# CV_scheme Random_20_42\n# Valid foldwise mrrmse: [1.028 0.8965 1.0088 1.0534 1.2967 1.1261 1.6129 1.921 1.1854 0.9504\n# 0.8377 1.1712 2.354 1.5103 0.9678 1.1095 1.0625 1.0089 0.7834 1.399 ] r2: [ 0.0672 -0.7024 0.1644 0.0319 0.0457 0.1593 -0.5536 0.0443 -0.2374\n# -0.6091 -0.0626 -0.401 -0.207 -0.1166 0.1497 0.1487 0.0507 -0.0039\n# -0.0205 0.0411]\n# Train foldwise mrrmse: [1.1523 1.1474 1.1545 1.1449 1.1295 1.148 1.1079 1.0991 1.1414 1.1493\n# 1.157 1.1369 1.0735 1.1238 1.1507 1.1455 1.1476 1.1447 1.1623 1.1362] r2: [0.1933 0.2209 0.1872 0.2057 0.2123 0.1872 0.2478 0.2174 0.2046 0.214\n# 0.2067 0.2134 0.2212 0.2027 0.2013 0.1947 0.1984 0.2119 0.1988 0.1877]\n# tsvd25_LSVR_target_enc_i_th_target_BothEncoded_target_enc_i_th_targetOpt_LVSRc1d100/oof_Random_20_42_CV12142.npy file save\n\n\n# C = 1/100\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.976042 r2: -0.061257 Train: 1.181187 0.231554\n# Valid foldwise mrrmse: [1.0761 0.9758 0.8468 1.0055] r2: [ 0.0476 0.1081 -0.2757 -0.125 ]\n# Train foldwise mrrmse: [1.1688 1.1865 1.1979 1.1716] r2: [0.2027 0.2249 0.2598 0.2388]\n# CV_scheme MT\n# Valid mrrmse: 2.835835 r2: -0.057504 Train: 1.112649 0.189855\n# Valid foldwise mrrmse: [3.0482 3.1508 2.3085] r2: [ 0.0604 -0.0999 -0.1329]\n# Train foldwise mrrmse: [1.1054 1.107 1.1255] r2: [0.1891 0.1902 0.1902]\n# CPU times: user 48.5 s, sys: 25 s, total: 1min 13s\n# Wall time: 27.8 s\n \n# C = 1/1000\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.968187 r2: -0.004964 Train: 1.204663 0.194331\n# Valid foldwise mrrmse: [1.0838 0.9886 0.8144 0.9859] r2: [ 0.045 0.1009 -0.1241 -0.0417]\n# Train foldwise mrrmse: [1.189 1.2051 1.2241 1.2004] r2: [0.1708 0.1893 0.2218 0.1954]\n# CV_scheme MT\n# Valid mrrmse: 2.862461 r2: -0.01144 Train: 1.129013 0.163856\n# Valid foldwise mrrmse: [3.1007 3.1722 2.3144] r2: [ 0.105 -0.0842 -0.0552]\n# Train foldwise mrrmse: [1.1232 1.1223 1.1416] r2: [0.1585 0.1672 0.1659]\n# CPU times: user 39 s, sys: 22 s, total: 1min 1s\n# Wall time: 22.3 s\n \n# C = 1/10000\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.974775 r2: -0.00039 Train: 1.2635 0.094522\n# Valid foldwise mrrmse: [1.1093 1.0115 0.7915 0.9868] r2: [ 0.0056 0.0624 -0.0294 -0.0401]\n# Train foldwise mrrmse: [1.24 1.2511 1.3047 1.2582] r2: [0.0799 0.0995 0.0944 0.1044]\n# CV_scheme MT\n# Valid mrrmse: 2.994605 r2: -0.052136 Train: 1.169889 0.087041\n# Valid foldwise mrrmse: [3.3214 3.2525 2.41 ] r2: [ 0.0318 -0.1137 -0.0745]\n# Train foldwise mrrmse: [1.162 1.1643 1.1834] r2: [0.0837 0.0892 0.0882]\n# CPU times: user 40.6 s, sys: 23 s, total: 1min 3s\n# Wall time: 23.3 s\n \n\n# C = 1/10\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.978399 r2: -0.092786 Train: 1.152397 0.272324\n# Valid foldwise mrrmse: [1.0683 0.964 0.8578 1.0236] r2: [ 0.0492 0.1164 -0.3304 -0.2064]\n# Train foldwise mrrmse: [1.1329 1.1654 1.1719 1.1395] r2: [0.2577 0.2539 0.2972 0.2805]\n# CV_scheme MT\n# Valid mrrmse: 2.776903 r2: -0.150548 Train: 1.083186 0.242422\n# Valid foldwise mrrmse: [2.9195 3.1166 2.2946] r2: [ 0.0057 -0.1111 -0.3462]\n# Train foldwise mrrmse: [1.0826 1.0738 1.0931] r2: [0.2292 0.2513 0.2468]\n# CPU times: user 46.7 s, sys: 24 s, total: 1min 10s\n# Wall time: 26.7 s\n\n# C = 1e5/ list_alpha \n# CV_scheme AmbrosM\n# Valid mrrmse: 0.981241 r2: -0.114639 Train: 1.148085 0.278721\n# Valid foldwise mrrmse: [1.0706 0.9589 0.861 1.0345] r2: [ 0.0388 0.1136 -0.3453 -0.2657]\n# Train foldwise mrrmse: [1.1287 1.164 1.167 1.1326] r2: [0.264 0.2574 0.3035 0.2898]\n# CV_scheme MT\n# Valid mrrmse: 2.767487 r2: -0.232101 Train: 1.079887 0.248517\n# Valid foldwise mrrmse: [2.879 3.1172 2.3063] r2: [-0.1404 -0.1325 -0.4235]\n# Train foldwise mrrmse: [1.0769 1.0712 1.0916] r2: [0.2405 0.2561 0.249 ]\n# CPU times: user 50.1 s, sys: 26 s, total: 1min 16s\n# Wall time: 29.1 s\n\n \n# C = 1\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.987434 r2: -0.135983 Train: 1.116957 0.319224\n# Valid foldwise mrrmse: [1.0642 0.9614 0.8811 1.0431] r2: [ 0.0625 0.0987 -0.422 -0.2831]\n# Train foldwise mrrmse: [1.0892 1.1329 1.1442 1.1015] r2: [0.3216 0.2933 0.3293 0.3326]\n# CV_scheme MT\n# Valid mrrmse: 2.665948 r2: -0.190679 Train: 1.054618 0.293654\n# Valid foldwise mrrmse: [2.7743 3.0515 2.172 ] r2: [-0.1455 -0.0923 -0.3343]\n# Train foldwise mrrmse: [1.055 1.0446 1.0642] r2: [0.2803 0.3017 0.299 ]\n# CPU times: user 48.5 s, sys: 25.2 s, total: 1min 13s\n# Wall time: 27.9 s\n \n\n# 1e4/ list_alpha\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.978057 r2: -0.086044 Train: 1.163197 0.260618\n# Valid foldwise mrrmse: [1.0717 0.9632 0.8563 1.0209] r2: [ 0.0394 0.1171 -0.3161 -0.1846]\n# Train foldwise mrrmse: [1.1483 1.1743 1.1761 1.154 ] r2: [0.2386 0.2471 0.294 0.2628]\n# CV_scheme MT\n# Valid mrrmse: 2.821095 r2: -0.167148 Train: 1.096864 0.218909\n# Valid foldwise mrrmse: [2.9714 3.1476 2.3443] r2: [-0.0025 -0.1282 -0.3707]\n# Train foldwise mrrmse: [1.0939 1.0893 1.1074] r2: [0.2102 0.224 0.2225]\n# CPU times: user 48.5 s, sys: 25.5 s, total: 1min 13s\n# Wall time: 27.9 s\n \n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.709671Z","iopub.execute_input":"2023-10-24T09:55:50.710336Z","iopub.status.idle":"2023-10-24T09:55:50.744279Z","shell.execute_reply.started":"2023-10-24T09:55:50.710267Z","shell.execute_reply":"2023-10-24T09:55:50.742954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Params found in v180: 0.9684 2.649 180 tsvd30 vary: extend around 0.01 #2h41m\nmain_config_model_feature_etc_opt_tsvd30 = {'reducer': 'tsvd', 'n_components': 30, 'i_target_cfg': {0: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 1: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 2: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 3: {'model': 'SVR', 'kernel': 'linear', 'C': 0.1, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 4: {'model': 'SVR', 'kernel': 'linear', 'C': 0.001, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 5: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 6: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 7: {'model': 'SVR', 'kernel': 'linear', 'C': 0.005, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 8: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 9: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 10: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 11: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 12: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 13: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 14: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 15: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 16: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 17: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 18: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 19: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 20: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 21: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 22: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 23: {'model': 'SVR', 'kernel': 'linear', 'C': 0.1, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 24: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 25: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 26: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 27: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 28: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 29: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}}}\nl = []\nfor i_target in range( 30 ):\n C = main_config_model_feature_etc_opt_tsvd30['i_target_cfg'][i_target]['C']\n l.append(C)\nprint(l) \nprint(main_config_model_feature_etc_opt_tsvd30['i_target_cfg'][0] )\nlist_C_SVR_fromV180 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.02, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02]\nimport numpy as np\nnp.unique(list_C_SVR_fromV180)","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.747984Z","iopub.execute_input":"2023-10-24T09:55:50.748396Z","iopub.status.idle":"2023-10-24T09:55:50.795063Z","shell.execute_reply.started":"2023-10-24T09:55:50.748364Z","shell.execute_reply":"2023-10-24T09:55:50.793984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"main_config_model_feature_etc_opt_tsvd50 ={'reducer': 'tsvd', 'n_components': 50, 'i_target_cfg': {0: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 1: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 2: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 3: {'model': 'SVR', 'kernel': 'linear', 'C': 0.1, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 4: {'model': 'SVR', 'kernel': 'linear', 'C': 0.001, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 5: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 6: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 7: {'model': 'SVR', 'kernel': 'linear', 'C': 0.005, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 8: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 9: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 10: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 11: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 12: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 13: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 14: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 15: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 16: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 17: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 18: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 19: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 20: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 21: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 22: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 23: {'model': 'SVR', 'kernel': 'linear', 'C': 0.1, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 24: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 25: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 26: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 27: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 28: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 29: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 30: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 31: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 32: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 33: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 34: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 35: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 36: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 37: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 38: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 39: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 40: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 41: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 42: {'model': 'SVR', 'kernel': 'linear', 'C': 0.05, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 43: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 44: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 45: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 46: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 47: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 48: {'model': 'SVR', 'kernel': 'linear', 'C': 0.02, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}, 49: {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}}}\nl = []\nfor i_target in range( 50 ):\n C = main_config_model_feature_etc_opt_tsvd50['i_target_cfg'][i_target]['C']\n l.append(C)\nprint(l) \nprint(main_config_model_feature_etc_opt_tsvd50['i_target_cfg'][0] )\nlist_C_SVR_fromV181 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.05, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.05, 0.02, 0.01, 0.01, 0.02, 0.02, 0.02, 0.01, 0.02, 0.05, 0.01, 0.05, 0.05, 0.02, 0.02, 0.01, 0.01, 0.02, 0.02, 0.01]\nimport numpy as np\nprint( np.unique(list_C_SVR_fromV181) )\nlist_C_SVR_fromV180[:10] == list_C_SVR_fromV181[:10],list_C_SVR_fromV180[:20] == list_C_SVR_fromV181[:20]","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.796821Z","iopub.execute_input":"2023-10-24T09:55:50.797179Z","iopub.status.idle":"2023-10-24T09:55:50.867372Z","shell.execute_reply.started":"2023-10-24T09:55:50.797122Z","shell.execute_reply":"2023-10-24T09:55:50.866221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TE ALL tsvd25 compounds. LB0.617, 0.605 reproduce in new framework\n\n\nHere we use as feautures TE for ALL 25 tstvd components. All of them are used to predict all tsvd25 components. That is in contrast to the previous model where we used TE of only i-th tsvd to predict itself\n\nReproduce V75 in new framework. Original https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145388832\n\nLB0.617 - just model LB0.605 - blend with priors.\n\nSaved in version V43 here: https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=146148456\n\n","metadata":{}},{"cell_type":"code","source":"%%time\n# Take Alpha and Smoothings from optimization of i-th-target only \n#dict_best_alpha_for_component = {0: 1000000.0, 1: 100000.0, 2: 100000.0, 3: 1000000.0, 4: 100000.0, 5: 1000000.0, 6: 1000000.0, 7: 1000000.0, 8: 100000.0, 9: 100000.0, 10: 10000.0, 11: 10000.0, 12: 100000.0, 13: 10000.0, 14: 100000.0, 15: 100000.0, 16: 100000.0, 17: 10000.0, 18: 10000.0, 19: 10000.0, 20: 10000.0, 21: 10000.0, 22: 10000.0, 23: 1000.0, 24: 10000.0}\nalpha_opt1 = [1000000.0, 100000.0, 100000.0, 1000000.0, 100000.0, 1000000.0, 1000000.0, 1000000.0, 100000.0, 100000.0, 10000.0, 10000.0, 100000.0, 10000.0, 100000.0, 100000.0, 100000.0, 10000.0, 10000.0, 10000.0, 10000.0, 10000.0, 10000.0, 1000.0, 10000.0]\n# From versions 47 of the notebook:\n# https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144999017\nplt.figure(figsize = (15,4))\nplt.plot(np.log10(alpha_opt1 )) \nplt.title('Log10 of Alphas LB0.617 TE for ALL tsvd25', fontsize = 20)\nplt.xlabel('tsvd component index', fontsize = 20)\nplt.grid()\nplt.show()\n\n#dict_best_smooth_celltype_for_component = {0: 0.0, 1: 1.0, 2: 1.0, 3: 1000000000000000.0, 4: 10.0, 5: 1000.0, 6: 1000000000000000.0, 7: 100.0, 8: 1000000000000000.0, 9: 100.0, 10: 10.0, 11: 100.0, 12: 0.0, 13: 1000.0, 14: 10000000000000.0, 15: 100.0, 16: 100.0, 17: 10.0, 18: 1.0, 19: 10000000000000.0, 20: 100.0, 21: 10.0, 22: 0.0, 23: 1.0, 24: 0.0}\n# smoothings_compound_opt1 = [np.clip(t,0,1e4) for t in dict_best_smooth_celltype_for_component.values() ]\nsmoothings_compound_opt1 = [10000.0, 10000.0, 10000.0, 10000.0, 10.0, 10000.0, 10000.0, 10000.0, 10000.0, 100.0, 100.0, 100.0, 100.0, 100.0, 10000.0, 1000.0, 10000.0, 100.0, 100.0, 10000.0, 10000.0, 10000.0, 10000.0, 100.0, 10000.0]\n# smoothings_compound_opt1 = list( dict_best_smooth_compound_for_component.values() )\n# Version 51: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145015609\nplt.figure(figsize = (15,4))\nplt.plot(np.log10(smoothings_compound_opt1 )) \nplt.title('Log10 of Smoothing LB0.617 TE for ALL tsvd25. (Small - not used)', fontsize = 20)\nplt.xlabel('tsvd component index', fontsize = 20)\nplt.grid()\nplt.show()\n\nprint(len( smoothings_compound_opt1 ), len( alpha_opt1 ))\n\nif 0:\n #print('TE(target encoding) both cell type and compound, but TE i-th target - i.e. the only target which we predict is used for creating TWO TE features')\n n_components = len( smoothings_compound_opt1 )\n\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components}; main_config_model_feature_etc['i_target_cfg']={}\n for i_target in range( n_components ):\n alpha = alpha_opt1[i_target]\n smoothing = list(dict_best_smooth_compound_for_component.values()) # smoothings_compound_opt1\n smoothing = smoothings_compound_opt1\n main_config_model_feature_etc_tmp = { 'model':'Ridge', 'alpha':alpha, 'features_mode':'target_enc', # 'target_enc', \n 'smoothing': smoothing,\n 'list_features_in' : [ 'sm_name'] } # 'model':'Ridge', 'alpha':alpha, 'fit_intercept':True, \n main_config_model_feature_etc['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy()\n\n # print(main_config_model_feature_etc) \n print('alpha0', main_config_model_feature_etc['i_target_cfg'][0]['alpha'] )\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'target_enc_ALL_Targs_Opt1oldV75' , flag_save = True) \n\n# -------------------------------------------------------------------------------------------------------\n# Report:\n\n# tsvd25_Ridge_target_enc_sm_nameEncoded\n# subdirectory_path tsvd25_Ridge_target_enc_sm_nameEncoded_target_enc_ALL_Targs_Opt1oldV75\n\n# CV_scheme AmbrosM\n# Valid mrrmse: 0.993146 r2: -0.118609 Train: 1.140908 0.306897\n# Valid foldwise mrrmse: [1.0667 0.986 0.8901 1.0298] r2: [ 0.0756 0.0116 -0.3851 -0.1764]\n# Train foldwise mrrmse: [1.1169 1.1457 1.1714 1.1297] r2: [0.3023 0.2997 0.3121 0.3136]\n# tsvd25_Ridge_target_enc_sm_nameEncoded_target_enc_ALL_Targs_Opt1oldV75/oof_AmbrosM_CV09931.npy file save\n\n# CV_scheme MT\n# Valid mrrmse: 2.636502 r2: -0.288637 Train: 1.073903 0.28558\n# Valid foldwise mrrmse: [2.7731 3.0265 2.1099] r2: [-0.4073 -0.0825 -0.3762]\n# Train foldwise mrrmse: [1.0697 1.066 1.086 ] r2: [0.2821 0.2886 0.286 ]\n# tsvd25_Ridge_target_enc_sm_nameEncoded_target_enc_ALL_Targs_Opt1oldV75/oof_MT_CV26365.npy file save\n\n# Valid mrrmse: 1.100088 r2: 0.290073 Train: 1.100088 0.290073\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:50.869147Z","iopub.execute_input":"2023-10-24T09:55:50.870330Z","iopub.status.idle":"2023-10-24T09:55:51.421502Z","shell.execute_reply.started":"2023-10-24T09:55:50.870282Z","shell.execute_reply":"2023-10-24T09:55:51.420116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Towards improving TE-all svd models","metadata":{}},{"cell_type":"markdown","source":"### Alpha = 2e4 , seems better 1e4,5e4 for smoohting = 10 for MT scheme \n\nThat simple model is LB0.616 (V41 with tsvd30 , Alpha = 1e4, smoothing = 10 ) \n\nBut difference is quite small","metadata":{}},{"cell_type":"code","source":"%%time\n\nfor alpha in []:# [2e4]:\n print('alpha', alpha)\n main_config_model_feature_etc = {'model':'Ridge', 'alpha':alpha, 'fit_intercept' : True, \n 'features_mode':'target_enc', 'list_features_in' : ['sm_name'], 'reducer':'tsvd','n_components':30, \n 'smoothing':10, # 'smoothing0':100, 'smoothing1':1000, 'smoothing2':10000, \n } \n\n CV_scheme = 'AmbrosM'; \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme , verbose = 1) \n\n CV_scheme = 'MT'#'AmbrosM'; \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme , verbose = 1) \n\n \n\n# alpha: 20000.0\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 0.997311 r2: -0.121184 Train: 1.140404 0.310006\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme MT\n# Valid mrrmse: 2.632053 r2: -0.250641 Train: 1.07362 0.291756\n# CPU times: user 2min 31s, sys: 1min 6s, total: 3min 38s\n# Wall time: 1min 47s\n \n# main_config_model_feature_etc = {'model':'Ridge', 'alpha':1e4, 'fit_intercept' : True, \n# 'features_mode':'target_enc', 'list_features_in' : ['sm_name'], 'reducer':'tsvd','n_components':30, \n# 'smoothing':10 } # 'smoothing0':100, 'smoothing1':1000, 'smoothing2':10000, # CV_scheme = 'AmbrosM'; \n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 1.000191 r2: -0.152321 Train: 1.130333 0.320543\n# CV_scheme = 'MT'#'AmbrosM'; \n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme MT\n# Valid mrrmse: 2.64004 r2: -0.344846 Train: 1.065289 0.302294 \n\n# 100000.0\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 0.99415 r2: -0.072428 Train: 1.168886 0.278436\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme MT\n# Valid mrrmse: 2.657113 r2: -0.058743 Train: 1.096688 0.258614\n# CPU times: user 2min 29s, sys: 1min 8s, total: 3min 37s\n# Wall time: 1min 45s\n \n# 50000.0\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 0.994853 r2: -0.09114 Train: 1.155033 0.294238\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme MT\n# Valid mrrmse: 2.634557 r2: -0.130596 Train: 1.08564 0.275217\n# CPU times: user 2min 28s, sys: 1min 8s, total: 3min 36s\n# Wall time: 1min 45s\n \n# 1000.0\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 1.014796 r2: -0.275861 Train: 1.110769 0.342031\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme MT\n# Valid mrrmse: 2.660268 r2: -0.551028 Train: 1.049729 0.322237\n# 5000.0\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme AmbrosM\n# Valid mrrmse: 1.004132 r2: -0.190514 Train: 1.121753 0.32955\n# tsvd30_Ridge_target_enc_sm_nameEncoded CV_scheme MT\n# Valid mrrmse: 2.648631 r2: -0.427012 Train: 1.058199 0.310988\n# CPU times: user 4min 58s, sys: 2min 15s, total: 7min 14s\n# Wall time: 3min 32s\n \n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.423489Z","iopub.execute_input":"2023-10-24T09:55:51.424204Z","iopub.status.idle":"2023-10-24T09:55:51.439949Z","shell.execute_reply.started":"2023-10-24T09:55:51.424157Z","shell.execute_reply":"2023-10-24T09:55:51.438717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Change smoothing0 for that model \n \n smoothing0 0 Valid mrrmse: 2.7834 r2: -0.2588 time: 47.6 count 1 best: 2.7834\n smoothing0 1 Valid mrrmse: 2.7834 r2: -0.2588 time: 95.5 count 2 best: 2.7834\n smoothing0 10.0 Valid mrrmse: 2.6321 r2: -0.2506 time: 143.8 count 3 best: 2.6321\n smoothing0 100.0 Valid mrrmse: 2.6256 r2: -0.2676 time: 191.6 count 4 best: 2.6256\n smoothing0 1000.0 Valid mrrmse: 2.6251 r2: -0.2683 time: 239.7 count 5 best: 2.6251\n smoothing0 10000.0 Valid mrrmse: 2.6251 r2: -0.2684 time: 287.4 count 6 best: 2.6251\n Finished loop for param smoothing0 best value: 10000.0 best score 2.625056 Round 0 i_target 0 time: 287.4 count 6\n smoothing0 0 Valid mrrmse: 2.6228 r2: -0.2593 time: 334.9 count 7 best: 2.6228\n smoothing0 1 Valid mrrmse: 2.6228 r2: -0.2593 time: 381.7 count 8 best: 2.6228\n smoothing0 10.0 Valid mrrmse: 2.6251 r2: -0.2684 time: 428.9 count 9 best: 2.6228\n smoothing0 100.0 Valid mrrmse: 2.6257 r2: -0.2701 time: 476.2 count 10 best: 2.6228\n smoothing0 1000.0 Valid mrrmse: 2.6257 r2: -0.2702 time: 524.0 count 11 best: 2.6228\n smoothing0 10000.0 Valid mrrmse: 2.6257 r2: -0.2702 time: 571.3 count 12 best: 2.6228\n Finished loop for param smoothing0 best value: 1 best score 2.622753 Round 0 i_target 1 time: 571.3 count 12\n smoothing0 0 Valid mrrmse: 2.6192 r2: -0.2495 time: 618.5 count 13 best: 2.6192\n smoothing0 1 Valid mrrmse: 2.6192 r2: -0.2495 time: 665.8 count 14 best: 2.6192\n smoothing0 10.0 Valid mrrmse: 2.6228 r2: -0.2593 time: 712.8 count 15 best: 2.6192\n smoothing0 100.0 Valid mrrmse: 2.6236 r2: -0.2631 time: 759.9 count 16 best: 2.6192\n smoothing0 1000.0 Valid mrrmse: 2.6237 r2: -0.2633 time: 806.7 count 17 best: 2.6192\n smoothing0 10000.0 Valid mrrmse: 2.6237 r2: -0.2634 time: 853.6 count 18 best: 2.6192\n Finished loop for param smoothing0 best value: 1 best score 2.619206 Round 0 i_target 2 time: 853.6 count 18\n smoothing0 0 Valid mrrmse: 2.6208 r2: -0.2644 time: 900.3 count 19 best: 2.6192\n smoothing0 1 Valid mrrmse: 2.6208 r2: -0.2644 time: 946.7 count 20 best: 2.6192\n smoothing0 10.0 Valid mrrmse: 2.6192 r2: -0.2495 time: 993.2 count 21 best: 2.6192\n smoothing0 100.0 Valid mrrmse: 2.6189 r2: -0.2481 time: 1039.3 count 22 best: 2.6189\n smoothing0 1000.0 Valid mrrmse: 2.6189 r2: -0.2481 time: 1085.2 count 23 best: 2.6189\n smoothing0 10000.0 Valid mrrmse: 2.6189 r2: -0.248 time: 1131.5 count 24 best: 2.6189\n Finished loop for param smoothing0 best value: 10000.0 best score 2.618862 Round 0 i_target 3 time: 1131.5 count 24\n smoothing0 0 Valid mrrmse: 2.6166 r2: -0.2459 time: 1178.4 count 25 best: 2.6166\n smoothing0 1 Valid mrrmse: 2.6166 r2: -0.2459 time: 1224.9 count 26 best: 2.6166\n smoothing0 10.0 Valid mrrmse: 2.6189 r2: -0.248 time: 1271.2 count 27 best: 2.6166\n smoothing0 100.0 Valid mrrmse: 2.621 r2: -0.2503 time: 1317.5 count 28 best: 2.6166\n smoothing0 1000.0 Valid mrrmse: 2.6211 r2: -0.2504 time: 1364.7 count 29 best: 2.6166\n smoothing0 10000.0 Valid mrrmse: 2.6211 r2: -0.2504 time: 1412.3 count 30 best: 2.6166\n Finished loop for param smoothing0 best value: 1 best score 2.616595 Round 0 i_target 4 time: 1412.3 count 30\n smoothing0 0 Valid mrrmse: 2.6174 r2: -0.2473 time: 1459.6 count 31 best: 2.6166","metadata":{}},{"cell_type":"markdown","source":"## SVR & TE-ith component\n\nNote: kernel \"poly\" seems to work forever - something wrong - may be we need some params like gamma, or shrinking are bad for that kernel\n\nmain_config_model_feature_etc = {'model':'SVR', 'kernel':'poly','C':0.1 ,'epsilon': 10,'gamma':2,'shrinking':False, 'reducer':'tsvd','n_components':25, 'features_mode':'target_enc_i_th_target', 'list_features_in' : [ 'sm_name'] } # 'cell_type',\n\n","metadata":{}},{"cell_type":"code","source":"%%time\nmain_config_model_feature_etc = {'model':'SVR', 'kernel':'linear','C':0.1 ,'epsilon': 10,'gamma':2,'shrinking':False, 'reducer':'tsvd','n_components':25,\n 'features_mode':'target_enc_i_th_target', 'list_features_in' : [ 'sm_name'] } # 'cell_type',\n\n# if 0:\n# do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'SVR', flag_save = False )\n\n\nif 0:\n CV_scheme = 'AmbrosM'; \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc,\n CV_scheme = CV_scheme , verbose = 1) \n \n# 'model':'SVR', 'kernel':'linear','C':0.1 ,'epsilon': 10,'gamma':2,'shrinking':False, \n# CV_scheme = 'MT';Valid mrrmse: 2.677495 r2: -0.114331 Train: 1.066806 0.29076 \n# CV_scheme = 'AmbrosM'; Valid mrrmse: 0.98835 r2: -0.110644 Train: 1.128265 0.318724\n\nif 0:\n for kernel in ['linear', 'sigmoid', 'rbf', ]: # 'poly',\n print('kernel',kernel)\n for C in [0.01,0.1,1,10, 100,1000,10000]:#0.01,0.1,1,10]:\n print('C', C)\n for CV_scheme in ['AmbrosM', 'MT'] : \n #print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n main_config_model_feature_etc = {'model':'SVR', 'kernel':kernel,'C':C ,'epsilon': 10,'gamma':2,'shrinking':False, 'reducer':'tsvd','n_components':25,\n 'features_mode':'target_enc_i_th_target', 'list_features_in' : [ 'sm_name'] } # 'cell_type',\n\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc,\n CV_scheme = CV_scheme , verbose = 1) \n\n# 'kernel':'linear'\n# C 0.01\n# Valid mrrmse: 0.983939 r2: -0.050357 Train: 1.153575 0.287145\n# Valid mrrmse: 2.736589 r2: 0.013788 Train: 1.090046 0.253241\n# C 0.1\n# Valid mrrmse: 0.98835 r2: -0.110644 Train: 1.128265 0.318724\n# Valid mrrmse: 2.677495 r2: -0.114331 Train: 1.066806 0.29076\n# C 1\n# Valid mrrmse: 0.99368 r2: -0.145498 Train: 1.12346 0.32596\n# Valid mrrmse: 2.675536 r2: -0.194728 Train: 1.062862 0.296483\n# C 10\n# Valid mrrmse: 0.995011 r2: -0.151705 Train: 1.122937 0.326743\n# Valid mrrmse: 2.67625 r2: -0.199888 Train: 1.062622 0.296927\n# CPU times: user 2min 36s, sys: 1min 27s, total: 4min 3s\n# Wall time: 1min 29s\n\n# kernel sigmoid - seems quite bad results \n# C 0.01\n# Valid mrrmse: 1.002086 r2: -0.060452 Train: 1.33895 -0.023515\n# Valid mrrmse: 3.203481 r2: -0.209517 Train: 1.231531 -0.020728\n# C 0.1\n# Valid mrrmse: 1.002398 r2: -0.061648 Train: 1.336988 -0.021339\n# Valid mrrmse: 3.203355 r2: -0.210237 Train: 1.229494 -0.018376\n# C 1\n# Valid mrrmse: 1.144822 r2: -0.335724 Train: 1.494153 -0.117303\n# Valid mrrmse: 3.48948 r2: -0.83366 Train: 1.658785 -0.366145\n# C 10\n# Valid mrrmse: 5.60502 r2: -36.939928 Train: 6.258345 -10.892807\n# Valid mrrmse: 12.267385 r2: -70.489477 Train: 10.192137 -32.965574\n# C 100\n# Valid mrrmse: 54.704232 r2: -3724.750141 Train: 59.638227 -1073.723676\n# Valid mrrmse: 117.00033 r2: -7010.532666 Train: 100.130421 -3239.207934\n# C 1000\n# Valid mrrmse: 546.61865 r2: -373976.11056 Train: 595.449354 -107093.272865\n# Valid mrrmse: 1169.28488 r2: -693003.281736 Train: 1000.579304 -323573.856231\n# C 10000\n# Valid mrrmse: 5466.018331 r2: -37393899.655325 Train: 5953.906941 -10704920.104564\n# Valid mrrmse: 11692.535439 r2: -69292050.016762 Train: 10005.154912 -32351253.810354\n \n \n# kernel rbf\n# C 0.01\n# Valid mrrmse: 1.001893 r2: -0.060127 Train: 1.338691 -0.023431\n# Valid mrrmse: 3.202978 r2: -0.208852 Train: 1.231231 -0.020575\n# C 0.1\n# Valid mrrmse: 0.999177 r2: -0.05652 Train: 1.333245 -0.020005\n# Valid mrrmse: 3.199424 r2: -0.205869 Train: 1.225617 -0.01673\n# C 1\n# Valid mrrmse: 0.99056 r2: -0.044384 Train: 1.309842 -0.001858\n# Valid mrrmse: 3.175429 r2: -0.18757 Train: 1.201564 0.003356\n# C 10\n# Valid mrrmse: 0.982573 r2: -0.030393 Train: 1.261408 0.055767\n# Valid mrrmse: 3.110276 r2: -0.148172 Train: 1.159657 0.056447\n# C 100\n# Valid mrrmse: 0.991463 r2: -0.088346 Train: 1.182907 0.187596\n# Valid mrrmse: 2.967973 r2: -0.151289 Train: 1.10923 0.156046\n# C 1000\n# Valid mrrmse: 1.026384 r2: -0.429782 Train: 1.123656 0.279301\n# Valid mrrmse: 2.743499 r2: -0.123901 Train: 1.07219 0.235735\n# C 10000\n# Valid mrrmse: 1.028693 r2: -0.433324 Train: 1.125317 0.278156\n# Valid mrrmse: 2.765051 r2: -0.160152 Train: 1.073087 0.234549\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.441794Z","iopub.execute_input":"2023-10-24T09:55:51.442208Z","iopub.status.idle":"2023-10-24T09:55:51.466360Z","shell.execute_reply.started":"2023-10-24T09:55:51.442175Z","shell.execute_reply":"2023-10-24T09:55:51.464981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SVR & onehot (drug)\n\nV131 SVR experiments - not promising, played with params (same-params-for-all-targets) - got results similar to LinearSVR on MT-CV - worse than Ridge.\nWe already played with LinearSVR quite a lot and it is seems worse than Ridge.\n","metadata":{}},{"cell_type":"code","source":"%%time\nCV_scheme = 'MT'# 'AmbrosM'; \nmain_config_model_feature_etc = {'model':'SVR', 'kernel':'linear','C':1000 ,'epsilon': 10,'gamma':2,'shrinking':False, 'reducer':'tsvd','n_components':25,'features_mode':'onehot', 'list_features_in' : ['sm_name'] }\n\n# if 0:\n# do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'SVR', flag_save = False )\n\n\nprint(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\nif 0:\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc,\n CV_scheme = CV_scheme , verbose = 1) \n\n# kernel = 'rbf'\n# C=1: Valid mrrmse: 3.191269 r2: -0.198852 Train: 1.215616 -0.014202\n# C=10: Valid mrrmse: 3.150516 r2: -0.168673 Train: 1.179825 0.025534\n# C=100: Valid mrrmse: 3.037169 r2: -0.168323 Train: 1.127846 0.121573\n# C=1000: Valid mrrmse: 2.793585 r2: -0.122368 Train: 1.085916 0.208127\n# C=10_000 Valid mrrmse: 2.820658 r2: -0.186872 Train: 1.084133 0.209033\n# Fix C = 1000\n# epsilon = 1: Valid mrrmse: 2.792721 r2: -0.120532 Train: 1.084276 0.209686\n# epsilon = 2: Valid mrrmse: 2.791353 r2: -0.118282 Train: 1.083059 0.211111\n# epsilon = 5: Valid mrrmse: 2.78866 r2: -0.113379 Train: 1.082751 0.214278\n# epsilon = 10: Valid mrrmse: 2.787714 r2: -0.109064 Train: 1.090909 0.217053\n# epsilon = 20: Valid mrrmse: 2.795828 r2: -0.123426 Train: 1.116957 0.217645\n# epsilon = 100: Valid mrrmse: 3.21497 r2: -0.77155 Train: 1.805399 -0.088656\n# Fix C = 1000, epsilon = 10\n# gamma = 1e-4 Valid mrrmse: 3.201718 r2: -0.207786 Train: 1.22992 -0.019238\n# gamma = 1 Valid mrrmse: 2.787825 r2: -0.108964 Train: 1.090932 0.217017\n# gamma = 2 Valid mrrmse: 2.785153 r2: -0.118757 Train: 1.089762 0.218484\n# gamma = 1e4 Valid mrrmse: 2.785524 r2: -0.120936 Train: 1.089638 0.218566\n# gammm = 1e6 Valid mrrmse: 2.785524 r2: -0.120936 Train: 1.089638 0.218566\n# Fix Fix C = 1000, epsilon = 10, gamma = 2\n# shrinking':False Valid mrrmse: 2.785153 r2: -0.118757 Train: 1.089762 0.218484\n\n# 'kernel':'sigmoid'\n# Valid mrrmse: 2.784997 r2: -0.11682 Train: 1.089897 0.218367\n# 'kernel':'poly'\n# Valid mrrmse: 2.814076 r2: -0.172368 Train: 1.089106 0.217822\n# precomputed - does not work - input should be square \n# 'linear' : Valid mrrmse: 2.785524 r2: -0.120936 Train: 1.089638 0.218566\n\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.468722Z","iopub.execute_input":"2023-10-24T09:55:51.469200Z","iopub.status.idle":"2023-10-24T09:55:51.485547Z","shell.execute_reply.started":"2023-10-24T09:55:51.469155Z","shell.execute_reply":"2023-10-24T09:55:51.484103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ChemBert features examples\n\nChemBERT embeddings for compounds created by ALEKSEY TREPETSKY: https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/441550 (please upvote)\n\nDataset https://www.kaggle.com/datasets/alekseytrepetsky/chemberta-v2-77-mtr \n\n### Seems for ChemBert Mean alpha = 1 - optimial for ALL CVs (!) - saved in version 299 here\n\n### Seems for ChemBert CLS alpha = 0.1 - optimial for ALL CVs (!) - saved in version 300 here\n\n\nPreliminary results with Ridge shows that local scores are similar to those of one-hot - that should correspond to LB0.615. \nOptimal alpha around 0.1 or 1. \n\nLook on version 292,293,294,295,296, 297, 299 (most detaile alpha steps) here:\ne.g. https://www.kaggle.com/code/alexandervc/op2-advanced-modeling-tuning-featengineering-etc?scriptVersionId=147114732\n\nPS\n\nFor some EDA on these embeddings take a look on: https://www.kaggle.com/code/alexandervc/look-on-chembert-data\n","metadata":{}},{"cell_type":"code","source":"%%time\n\nflag_chembert_analysis = False # True \n\nif flag_chembert_analysis:\n n_components = 50\n \n # CV_scheme = 'AmbrosM'; \n\n # To compare:\n #0.615\t\t1.0234\t2.6680\tRidge\tonehot drug\ttsvd30\t19\tMT with fit_intercept = True - CV is better but LB is the same \n\n df_stat_loc = pd.DataFrame(); IX_loc = 0;\n import warnings\n warnings.filterwarnings(\"ignore\") \n for alpha in [0.001,0.002,0.005,0.008, 0.01,0.02,0.05,0.08, 0.1,0.2,0.5,0.8, 1,2,5,8,10,20,50,100]:\n print(alpha)# = 100\n for CV_scheme in ['AmbrosM', 'MT','Random_5_42']:\n for features_mode in ['chembert_cls', 'chembert_mean']:\n print(features_mode, CV_scheme, 'alpha:', alpha )\n main_config_model_feature_etc = {'model':'Ridge', 'alpha':alpha, 'features_mode':features_mode, 'reducer':'tsvd','n_components':n_components} \n print(get_brief_string_info_on_config(main_config_model_feature_etc), 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n\n CV_scheme = CV_scheme , verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n print()\n\n df_stat_loc.loc[str(alpha), CV_scheme+' '+ features_mode + ' mrrmse' ] = np.round(np.mean(mrrmse_list) ,4)\n df_stat_loc.loc[str(alpha), CV_scheme+' '+ features_mode + ' r2' ] = np.round(np.mean(r2_list) ,4)\n warnings.filterwarnings(\"default\")\n display(df_stat_loc)\n df_stat_loc.to_csv('df_report_chembert_Ridge_tsvd'+str(n_components)+'_scores.csv')","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.487457Z","iopub.execute_input":"2023-10-24T09:55:51.488204Z","iopub.status.idle":"2023-10-24T09:55:51.508399Z","shell.execute_reply.started":"2023-10-24T09:55:51.488161Z","shell.execute_reply":"2023-10-24T09:55:51.507480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if flag_chembert_analysis:\n\n plt.figure(figsize = (20,6));\n plt.suptitle('MRRMSE scores tsvd'+str(n_components)+'_Ridge_chembert', fontsize = 20 )\n for ii, l in [ (1,[0,2]), (2,[4,6]), [3,(8,10)]]:\n plt.subplot(1,3,ii)\n for k in l:\n v = df_stat_loc.iloc[:,k]\n plt.plot(np.log10([float(t) for t in v.index]) , v.values,'*-', label = df_stat_loc.columns[k])\n plt.xlabel('log10 alpha')\n st = str( df_stat_loc.columns[k].split(' ')[0]) + ' ' + str( df_stat_loc.columns[k].split(' ')[2])\n plt.title(st )\n plt.grid()\n plt.legend()\n plt.show()\n\n plt.figure(figsize = (20,6));\n plt.suptitle('R2-scores tsvd'+str(n_components)+'_Ridge_chembert' , fontsize = 20 )\n for ii, l in [ (1,[1,3]), (2,[5,7]), [3,(9,11)]]:\n plt.subplot(1,3,ii)\n for k in l:\n v = df_stat_loc.iloc[:,k]\n plt.plot(np.log10([float(t) for t in v.index]) , v.values,'*-', label = df_stat_loc.columns[k])\n plt.xlabel('log10 alpha')\n st = str( df_stat_loc.columns[k].split(' ')[0]) + ' ' + str( df_stat_loc.columns[k].split(' ')[2])\n plt.title(st )\n plt.grid()\n plt.legend()\n plt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.509536Z","iopub.execute_input":"2023-10-24T09:55:51.510487Z","iopub.status.idle":"2023-10-24T09:55:51.530956Z","shell.execute_reply.started":"2023-10-24T09:55:51.510452Z","shell.execute_reply":"2023-10-24T09:55:51.529331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if flag_chembert_analysis:\n features_mode='chembert_cls'\n main_config_model_feature_etc = {'model':'Ridge', 'alpha':0.1, 'features_mode':features_mode, 'reducer':'tsvd','n_components':n_components} \n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc , subdirectory_path_postfix = 'ChemBertCLSRidge01' ) ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.532791Z","iopub.execute_input":"2023-10-24T09:55:51.534027Z","iopub.status.idle":"2023-10-24T09:55:51.548852Z","shell.execute_reply.started":"2023-10-24T09:55:51.533975Z","shell.execute_reply":"2023-10-24T09:55:51.547620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n\nif flag_chembert_analysis:\n n_components = 50\n\n str_model_inf = 'KRR-lin'\n # CV_scheme = 'AmbrosM'; \n\n # To compare:\n #0.615\t\t1.0234\t2.6680\tRidge\tonehot drug\ttsvd30\t19\tMT with fit_intercept = True - CV is better but LB is the same \n\n df_stat_loc = pd.DataFrame(); IX_loc = 0;\n import warnings\n warnings.filterwarnings(\"ignore\") \n for alpha in [0.001,0.002,0.005,0.008, 0.01,0.02,0.05,0.08, 0.1,0.2,0.5,0.8, 1,2,5,8,10,20,50,100]:\n print(alpha)# = 100\n for CV_scheme in ['AmbrosM', 'MT','Random_5_42']:\n for features_mode in ['chembert_cls', 'chembert_mean']:\n print(features_mode, CV_scheme, 'alpha:', alpha )\n main_config_model_feature_etc = {'model':'KRR', 'kernel':'linear', 'alpha':alpha, 'features_mode':features_mode, 'reducer':'tsvd','n_components':n_components} \n str_inf = get_brief_string_info_on_config(main_config_model_feature_etc)\n print(str_inf, 'CV_scheme', CV_scheme )\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n\n CV_scheme = CV_scheme , verbose = 1) \n print('Valid foldwise mrrmse:', np.round(mrrmse_list ,4), 'r2:', np.round(r2_list,4) )\n print('Train foldwise mrrmse:', np.round(dict_optional_res['mrrmse_list_train'],4), 'r2:', np.round(dict_optional_res['r2_list_train'],4) )\n print()\n\n df_stat_loc.loc[str(alpha), CV_scheme+' '+ features_mode + ' mrrmse' ] = np.round(np.mean(mrrmse_list) ,4)\n df_stat_loc.loc[str(alpha), CV_scheme+' '+ features_mode + ' r2' ] = np.round(np.mean(r2_list) ,4)\n warnings.filterwarnings(\"default\")\n display(df_stat_loc)\n df_stat_loc.to_csv('df_report_chembert_'+str_model_inf+'_tsvd'+str(n_components)+'_scores.csv')\n \n \nif flag_chembert_analysis:\n\n plt.figure(figsize = (20,6));\n plt.suptitle('MRRMSE scores tsvd'+str(n_components)+'_'+str_model_inf+'_chembert', fontsize = 20 )\n for ii, l in [ (1,[0,2]), (2,[4,6]), [3,(8,10)]]:\n plt.subplot(1,3,ii)\n for k in l:\n v = df_stat_loc.iloc[:,k]\n plt.plot(np.log10([float(t) for t in v.index]) , v.values,'*-', label = df_stat_loc.columns[k])\n plt.xlabel('log10 alpha')\n st = str( df_stat_loc.columns[k].split(' ')[0]) + ' ' + str( df_stat_loc.columns[k].split(' ')[2])\n plt.title(st )\n plt.grid()\n plt.legend()\n plt.show()\n\n plt.figure(figsize = (20,6));\n plt.suptitle('R2-scores tsvd'+str(n_components)+'_'+str_model_inf+'_chembert' , fontsize = 20 )\n for ii, l in [ (1,[1,3]), (2,[5,7]), [3,(9,11)]]:\n plt.subplot(1,3,ii)\n for k in l:\n v = df_stat_loc.iloc[:,k]\n plt.plot(np.log10([float(t) for t in v.index]) , v.values,'*-', label = df_stat_loc.columns[k])\n plt.xlabel('log10 alpha')\n st = str( df_stat_loc.columns[k].split(' ')[0]) + ' ' + str( df_stat_loc.columns[k].split(' ')[2])\n plt.title(st )\n plt.grid()\n plt.legend()\n plt.show()\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.551173Z","iopub.execute_input":"2023-10-24T09:55:51.551555Z","iopub.status.idle":"2023-10-24T09:55:51.581501Z","shell.execute_reply.started":"2023-10-24T09:55:51.551523Z","shell.execute_reply":"2023-10-24T09:55:51.580099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Comments on Category Encoders","metadata":{}},{"cell_type":"markdown","source":"## Target Encoder","metadata":{}},{"cell_type":"code","source":"%%time\nimport category_encoders as ce\nprint('smoothing 0 - sends everything to constant , difference with 1 is very small')\nprint('smoothing 10_000 or np.inf almost the same - no \"prior\" effect ')\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nfor smoothing in [0,0.5,1,5, 1e4, np.inf]:\n enc = ce.TargetEncoder(smoothing = smoothing)\n d['smoothing'+str(smoothing)] = enc.fit_transform(v, v_targets) \ndisplay( d.head(10).round(3) )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.583022Z","iopub.execute_input":"2023-10-24T09:55:51.583484Z","iopub.status.idle":"2023-10-24T09:55:51.698917Z","shell.execute_reply.started":"2023-10-24T09:55:51.583445Z","shell.execute_reply":"2023-10-24T09:55:51.698010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## QuantileEncoder","metadata":{}},{"cell_type":"code","source":"import category_encoders as ce\n\nprint(''' Info on QuantileEncoder Category Encoder. \n\nTuneable params: two tunable parameter m (default 1) and quantile (default - 0.5)\n\nHigher value of m results into stronger shrinking. M is non-negative. 0 for no smoothing.\n\nThis a statistically modified version of target MEstimate encoder where selected features are replaced by the statistical quantile instead of the mean. Replacing with the median is a particular case where self.quantile = 0.5. In comparison to MEstimateEncoder it has two tunable parameter m and quantile\n''')\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nfor quantile in [ 0.5, 0.1,0.9]:\n for m in [0.0, 1,10, 100,1000,10000,]:\n enc = ce.QuantileEncoder(quantile = quantile , m =m )# smoothing = smoothing)\n d['QuantileEncoder quantile' +str(quantile) + ' m(smoothing)'+str(m)] = enc.fit_transform(v, v_targets) \ndisplay( d.head(10).round(3) )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:51.700465Z","iopub.execute_input":"2023-10-24T09:55:51.700804Z","iopub.status.idle":"2023-10-24T09:55:52.031270Z","shell.execute_reply.started":"2023-10-24T09:55:51.700775Z","shell.execute_reply":"2023-10-24T09:55:52.030094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## GLMMEncoder","metadata":{}},{"cell_type":"code","source":"# %%time\n\n# import category_encoders as ce\n\n# print(''' Info on GLMMEncoder Category Encoder. \n\n# Tuneable params:\n\n# randomized: bool,\n# adds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched).\n\n# sigma: float (only used if randomized = True )\n# standard deviation (spread or “width”) of the normal distribution.\n\n\n# This is a supervised encoder similar to TargetEncoder or MEstimateEncoder, but there are some advantages:\n\n# Solid statistical theory behind the technique. Mixed effects models are a mature branch of statistics.\n\n# 2. No hyper-parameters to tune. The amount of shrinkage is automatically determined through the estimation process. In short, the less observations a category has and/or the more the outcome varies for a category then the higher the regularization towards “the prior” or “grand mean”. 3. The technique is applicable for both continuous and binomial targets. If the target is continuous, the encoder returns regularized difference of the observation’s category from the global mean.\n\n# If the target is binomial, the encoder returns regularized log odds per category.\n\n# In comparison to JamesSteinEstimator, this encoder utilizes generalized linear mixed models from statsmodels library.\n\n# Note: This is an alpha implementation. The API of the method may change in the future.\n# ''')\n# v = np.random.randint(0, 10, size=100)\n# v_targets = np.random.randint(0, 10, size=100)\n# v = [str(t) for t in v ]\n# d = pd.DataFrame(); d.index.name = 'category'\n# for randomized in [False, True]:\n# for sigma in [0.01,0.05, 0.1]:\n# enc = ce.GLMMEncoder(randomized = randomized, sigma = sigma)# \n# d['GLMMEncoder sigma' +str(sigma) + ' randomized'+str(randomized)] = enc.fit_transform(v, v_targets) \n# display( d.head(10).round(3) )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.032812Z","iopub.execute_input":"2023-10-24T09:55:52.033585Z","iopub.status.idle":"2023-10-24T09:55:52.042177Z","shell.execute_reply.started":"2023-10-24T09:55:52.033542Z","shell.execute_reply":"2023-10-24T09:55:52.039865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## James-Stein estimator ","metadata":{}},{"cell_type":"code","source":"%%time\nimport category_encoders as ce\n\nprint('''\nJames-Stein estimator.\n\nSee also: Stein's example (paradox) https://en.wikipedia.org/wiki/Stein%27s_example, https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator,\nhttps://mathoverflow.net/questions/93745/james-stein-phenomenon-what-does-it-mean-that-a-james-stein-estimator-beats-lea\n\nTuneable params:\n\nrandomized: bool,\nadds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched).\n\nsigma: float (only used if randomized = True )\nstandard deviation (spread or “width”) of the normal distribution.\n\n\n\nSupported targets: binomial and continuous. For polynomial target support, see PolynomialWrapper.\n\nFor feature value i, James-Stein estimator returns a weighted average of:\n\nThe mean target value for the observed feature value i.\n\nThe mean target value (regardless of the feature value).\n\nThis can be written as:\n\nJS_i = (1-B)*mean(y_i) + B*mean(y)\nThe question is, what should be the weight B? If we put too much weight on the conditional mean value, we will overfit. If we put too much weight on the global mean, we will underfit. The canonical solution in machine learning is to perform cross-validation. However, Charles Stein came with a closed-form solution to the problem. The intuition is: If the estimate of mean(y_i) is unreliable (y_i has high variance), we should put more weight on mean(y). Stein put it into an equation as:\n\nB = var(y_i) / (var(y_i)+var(y))\nThe only remaining issue is that we do not know var(y), let alone var(y_i). Hence, we have to estimate the variances. But how can we reliably estimate the variances, when we already struggle with the estimation of the mean values?! There are multiple solutions:\n\n1. If we have the same count of observations for each feature value i and all y_i are close to each other, we can pretend that all var(y_i) are identical. This is called a pooled model. 2. If the observation counts are not equal, it makes sense to replace the variances with squared standard errors, which penalize small observation counts:\n\nSE^2 = var(y)/count(y)\nThis is called an independent model.\n\nJames-Stein estimator has, however, one practical limitation - it was defined only for normal distributions. If you want to apply it for binary classification, which allows only values {0, 1}, it is better to first convert the mean target value from the bound interval <0,1> into an unbounded interval by replacing mean(y) with log-odds ratio:\n\nlog-odds_ratio_i = log(mean(y_i)/mean(y_not_i))\nThis is called binary model. The estimation of parameters of this model is, however, tricky and sometimes it fails fatally. In these situations, it is better to use beta model, which generally delivers slightly worse accuracy than binary model but does not suffer from fatal failures.\n''')\n\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nfor randomized in [False, True]:\n for sigma in [0.01,0.05, 0.1]:\n enc = ce.JamesSteinEncoder(randomized = randomized, sigma = sigma)# \n d['JamesSteinEncoder sigma' +str(sigma) + ' randomized'+str(randomized)] = enc.fit_transform(v, v_targets) \n \ndisplay( d.head(10).round(3) )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.044175Z","iopub.execute_input":"2023-10-24T09:55:52.044610Z","iopub.status.idle":"2023-10-24T09:55:52.170422Z","shell.execute_reply.started":"2023-10-24T09:55:52.044569Z","shell.execute_reply":"2023-10-24T09:55:52.169576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Catboost Encoder ","metadata":{"execution":{"iopub.status.busy":"2023-10-19T08:56:17.409351Z","iopub.execute_input":"2023-10-19T08:56:17.409978Z","iopub.status.idle":"2023-10-19T08:56:17.416460Z","shell.execute_reply.started":"2023-10-19T08:56:17.409929Z","shell.execute_reply":"2023-10-19T08:56:17.414500Z"}}},{"cell_type":"code","source":"import category_encoders as ce\n\nprint(''' Info on Catboost Category Encoder. \n\nTuneable params:\n\nsigma: float (default = None)\nadds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched). sigma gives the standard deviation (spread or “width”) of the normal distribution.\n\na: float = big values converts to constant (opposite direction to \"smoothing in TE\")\nadditive smoothing (it is the same variable as “m” in m-probability estimate). By default set to 1\n''')\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nfor sigma in [None,0.1, 1,100]:\n for a in [0.001, 1,10, 100]:\n enc = ce.CatBoostEncoder(sigma = sigma , a =a )# smoothing = smoothing)\n enc.fit(v, v_targets)\n d['CatBoostEnc sigma' +str(sigma) + ' a(smoothing)'+str(a)] = enc.transform(v)#, v_targets)\ndisplay( d.head(10).round(3) )\nenc.get_params()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.171537Z","iopub.execute_input":"2023-10-24T09:55:52.172493Z","iopub.status.idle":"2023-10-24T09:55:52.376492Z","shell.execute_reply.started":"2023-10-24T09:55:52.172459Z","shell.execute_reply":"2023-10-24T09:55:52.375255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Leave One Out Encoder ","metadata":{}},{"cell_type":"code","source":"%%time\nimport category_encoders as ce\n\nprint(''' \nLeave one out coding for categorical features.\n\nThis is very similar to target encoding but excludes the current row’s target when calculating the mean target for a level to reduce the effect of outliers.\n\nTune: sigma: float\nadds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched). Sigma gives the standard deviation (spread or “width”) of the normal distribution. The optimal value is commonly between 0.05 and 0.6. The default is to not add noise, but that leads to significantly suboptimal results.\n\n''')\n\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nfor sigma in [None,0.01,0.05,0.1,0.2,0.3,0.4, 0.5,0.6,0.7,0.8,0.9, 1,100]:\n prm_enc_loc = {'sigma':sigma}\n enc = ce.LeaveOneOutEncoder(**prm_enc_loc )\n d['LeaveOneOutEncoder sigma' +str(sigma)] = enc.fit_transform(v, v_targets) \ndisplay( d.head(10).round(3) )\nenc.get_params()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.386812Z","iopub.execute_input":"2023-10-24T09:55:52.387234Z","iopub.status.idle":"2023-10-24T09:55:52.591858Z","shell.execute_reply.started":"2023-10-24T09:55:52.387199Z","shell.execute_reply":"2023-10-24T09:55:52.590470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Backward difference contrast coding","metadata":{}},{"cell_type":"code","source":"%%time\nimport category_encoders as ce\n\nprint(''' Info on Backward difference contrast coding for encoding categorical variables. \n\nNO Tuneable params. \n\nDoes not depend on target \n\nWarning - returns dataframe even for one feature, (not just one column as for TEs)\n''')\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nenc = ce.BackwardDifferenceEncoder( )\nd = enc.fit_transform(v)#, v_targets) \ndisplay( d.head(10).round(3) )\nenc.get_params()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.593702Z","iopub.execute_input":"2023-10-24T09:55:52.594596Z","iopub.status.idle":"2023-10-24T09:55:52.659385Z","shell.execute_reply.started":"2023-10-24T09:55:52.594545Z","shell.execute_reply":"2023-10-24T09:55:52.658259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Count encoding for categorical features.","metadata":{}},{"cell_type":"code","source":"%%time\nimport category_encoders as ce\n\nprint(''' Count encoding for categorical features.\n\nFor a given categorical feature, replace the names of the groups with the group counts. \n\nNO Tuneable params. \n\nDoes not depend on target \n''')\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nenc = ce.CountEncoder( )\nd['CountEncoder'] = enc.fit_transform(v)#, v_targets) \ndisplay( d.head(10).round(3) )\nenc.get_params()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.661452Z","iopub.execute_input":"2023-10-24T09:55:52.661896Z","iopub.status.idle":"2023-10-24T09:55:52.694477Z","shell.execute_reply.started":"2023-10-24T09:55:52.661855Z","shell.execute_reply":"2023-10-24T09:55:52.693210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helmert contrast coding for encoding categorical features.","metadata":{"execution":{"iopub.status.busy":"2023-10-19T11:04:51.373366Z","iopub.execute_input":"2023-10-19T11:04:51.373774Z","iopub.status.idle":"2023-10-19T11:04:51.378730Z","shell.execute_reply.started":"2023-10-19T11:04:51.373745Z","shell.execute_reply":"2023-10-19T11:04:51.377835Z"}}},{"cell_type":"code","source":"%%time\nimport category_encoders as ce\n\nprint(''' Helmert contrast coding for encoding categorical features.\n\nNO Tuneable params. \n\nDoes not depend on target \n''')\nv = np.random.randint(0, 10, size=100)\nv_targets = np.random.randint(0, 10, size=100)\nv = [str(t) for t in v ]\nd = pd.DataFrame(); d.index.name = 'category'\nenc = ce.HelmertEncoder( )\nd = enc.fit_transform(v)#, v_targets) \ndisplay( d.head(10).round(3) )\nenc.get_params()","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.696158Z","iopub.execute_input":"2023-10-24T09:55:52.697168Z","iopub.status.idle":"2023-10-24T09:55:52.760204Z","shell.execute_reply.started":"2023-10-24T09:55:52.697125Z","shell.execute_reply":"2023-10-24T09:55:52.759104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Naive param-by-param Tuner ( tune params sequentially one after another) \n","metadata":{}},{"cell_type":"markdown","source":"\nVary param running through predescribed list, compute scores for each param value, choose the best one, then go to the next param, vary it throught its own list, choose the best, and so on. \n\nIt is quite suboptimal optimization method, but fast - linear complexity in number of params, not exponential as for full search. Thus we avoid curse of dimensionality sacrifying quality. \n\nIt might work better than optuna if we have 2-3 params. Any way it might a good staring point. \n\n Main settings: \n dict_prms_lists - dictionary with lists to change each param\n main_config_model_feature_etc # Initial params \n \n criteria_accept_params = 'all_cv_are_better'\n mode_optimize = 'separate_params_for_each_model(target)' - optimize params for models for i-th target separeately, i.e. might be different params \n 'same_params_for_all_models(targets)' - one model/params for all targets - and these will be tuned\n \n Other params:\n CV_scheme = 'MT' #'AmbrosM' # \n n_components = 30 for dimensional reduction\n \n \n Examples: # List for changing params. \n dict_prms_lists = {'alpha':[0.1, 1,5,10, 100] , 'fit_intercept':[True,False] } \n \n\n More examples:\n # Initial params: \n n_components = 32\n main_config_model_feature_etc = { 'reducer':'tsvd', 'n_components':n_components, 'model':'Ridge', 'alpha':1, 'fit_intercept':True, \n 'features_mode':'onehot', 'list_features_in' : ['sm_name'] }\n # List for changing params. \n dict_prms_lists = {'alpha':[0.01,0.05,0.1,0.5]}# 1,5,10, 100]} # , 'fit_intercept':[True,False] } \n \n \n ","metadata":{}},{"cell_type":"code","source":"%%time\nimport warnings\nimport time\n\nflag_opt_loc = False # True # optimization ON/OFF\n\nverbose = 1000\nn_rounds = 1# Rounds to proceed optimization\nlist_CV_scheme = ['AmbrosM','MT' ] # 'Random_5_42' #'MT'#'AmbrosM' # 'Random_5_42'\ndict_best_res = { CV_scheme:np.inf for CV_scheme in list_CV_scheme }; best_res = [t for t in dict_best_res.values() ]\n\ncriteria_accept_params = 'all_cv_are_better'\nmode_optimize = 'separate_params_for_each_model(target)'\n# mode_optimize = 'same_params_for_all_models(targets)'\n\n\nn_components = 25\n# main_config_model_feature_etc = {'reducer':'tsvd','n_components':n_components, 'model':'LSVR','C':1,'epsilon':1,'features_mode':'onehot','list_features_in':['sm_name'] }\nif mode_optimize == 'same_params_for_all_models(targets)':\n\n # Quantile Transformer Features \n main_config_model_feature_etc = {'model':'KRR', 'alpha':10, 'kernel':'rbf', # 'C':0.001,\n 'a0':1, 'a1':1,\n 'features_mode': 'CatBoostEncoder', 'targets_to_encode':'i_th_target', 'list_features_in': ['cell_type', 'sm_name'],\n 'reducer':'tsvd','n_components':n_components} \n # 402: {'model': 'KRR', 'alpha': 10.0, 'kernel': 'rbf', 'features_mode': 'QuantileEncoder', 'quantile0': 0.75, 'quantile1': 0.6, 'm0': 10000.0, 'm1': 200, 'targets_to_encode': 'i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'reducer': 'tsvd', 'n_components': 25}\n\n dict_prms_lists = {'alpha':[1e-2,1e-1, 1,1e1,1e2, 1e3,1e4,2e4,5e4,1e5,2e5,5e5,1e6,2e6,5e6,1e7,1e8,1e9],\n # 'C':[100,10, 1,1e-1, 1e-2, 1e-3,1e-4,2e-4,5e-4,1e-5,2e-5,5e-5,1e-6,2e-6,5e-6,1e-7],\n 'a0':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'a1':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4]}\n# 'quantile0':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.9],'quantile1':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.9]}\n\n# 0.972636 389EncQuant,1prm-all,Ridge,BigGridA,M,Q # 11 min \n# 0.972094 390EncQuant,1prm-all,Ridge BigGridA,M,Q,2Rounds # 30min - very small update 0.5of4-th digits, moreover alpha became 1e5 - may be if we do not include 2e4 will get it in 1 round\n# Conclusion - 2 round - very small update 0.5of4-th digits\n#391EncQuant,1prm-all,KRRrbf,BigGridA,M,Q,2Rounds\n#392EncQuant,1prm-all,KRRlin,BigGridA,M,Q,2Rounds\n#393EncQuant,1prm-all,Ridge BigGridA,M,Q,3Rounds\n#lb0.657 394EncQuant,1prm-all,Ridge BigGridA,M,Q,3Rounds # Bug corrected \n#395 \n#396 same with linear kernel krr\n#397 sigmoid\n#398 cosine \n# opt 2cv, 3 rounds\n#399 cosine\n#400 sigmoid\n#401 linear\n#402 rbf\n#403 Ridge \n#\n# SVR, op2 cv , 3rounds \n#404 SVR-lin op2-1prm-all quantile tsvd25,3rounds\n#405 same rbf\n#406 same?\n#407same optAmbr\n#408 same svr-linear - opt Amb\n#\n# JamesStein\n# 414 JS, svr-linear,OptAmb,1round\n# 415 js svr-rbf optAmb, 1round\n# 416 js svr rbf opt2 tsvd25 1prm-all\n#417 js svr sigmoid??? opt2 tsvd25 1prm-all\n#418 js svr sigmoid opt2 tsvd25 1prm-all\n# 419 js svr linear opt2 tsvd25 1prm-all\n\n#421LeaveOneOutEncoder svr linear \n# 422 leaveOneOutEncoder svr rbf\n# 423 count encoder\n# 424 catBoostEncoder\n# 425 bug catboost optimize a- smoothing\n# 428 3rnds catb-encoder opt2-smoothing and C svr rbf \n#429 same svr lin\n\n# 431Ridge,CatEnc,op2-sm,al,tsvd30,1prm-all,3rds\n# 432KRR-lin,CatEnc,op2-sm,al,tsvd30,1prm-all,3rds\n# 433KRRrbf,CatEnc,op2-sm,al,tsvd30,1prm-all,3rds\n\n\n\n# # Ridge&Quantile\n# #main_config_model_feature_etc = {'model':'Ridge', 'alpha':1, 'features_mode': 'chembert_mean', 'reducer':'tsvd','n_components':n_components} \n# main_config_model_feature_etc = {'model':'Ridge', 'alpha':1e5, \n# 'features_mode': 'QuantileEncoder', 'quantile0':0.6, 'quantile1':0.6, 'm0':1,'m1':1, # 'sigma0':0.01,'sigma1':0.01, \n# 'targets_to_encode':'i_th_target', 'list_features_in': ['cell_type', 'sm_name'],\n# 'reducer':'tsvd','n_components':n_components} \n# dict_prms_lists = {'alpha':[0.001,0.002,0.005,0.008, 0.01,0.02,0.05,0.08, 0.1,0.2,0.5,0.8, 1,2,5,8,10,20,50,100]}\n# dict_prms_lists = {'alpha':[1e3,1e4,1e5,1e6,1e7,1e8,1e9,1e10],'m':[1e-1,1,1e1,1e2,1e3,1e4], 'quantile':[0.4,0.5,0.6] }\n# dict_prms_lists = {'alpha':[1e5],'m':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'quantile':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75] }\n# dict_prms_lists = {'alpha':[1e5],'m':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'quantile':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75] }\n# dict_prms_lists = {'sigma':[0.01,0.05,0.1,0.2,0.3,0.4,0.5,0.6,0.7]}#,'m':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'quantile':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75] }\n# dict_prms_lists = {'alpha':[1e1, 1e2, 1e3,1e4,2e4,5e4,1e5,2e5,5e5,1e6,2e6,5e6]}# ,'m':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'quantile':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75] }\n# dict_prms_lists = {'sigma0':[1,1e10],'sigma1':[1,1e10]}#,'m':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'quantile':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75] }\n# dict_prms_lists = {'m0':[0.001,1e10],'m1':[0.001,1e10]}#,'m':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'quantile':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75] }\n# dict_prms_lists = {'alpha':[1e1, 1e2, 1e3,1e4,2e4,5e4,1e5,2e5,5e5,1e6,2e6,5e6], 'm0':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'm1':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], \n# 'quantile0':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.9],'quantile1':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.9]}\n# 365tst quantile Ridge Alpha:1e3-10,1prm-all,tsvd30\n#366EncQuant,RidgeAlp:1e3-10,M,Q,1prm-all,tsvd30\n#367EncQuant,RidgeAlp=1e5:1e3-10,M,Q,1prm-all,tsvd30\n#369EncQuant,1prm-all,Ridge,CheckGrid:M,Q\n#370EncQuant,1prm-all,Ridge,CheckGrid:SIGMA\n#371EncQuant,1prm-all,Ridge,CheckGrid:Alpha\n#373EncQuant,1prm-all,Ridge,CheckGrid:SIGMA^2 bug\n#374EncQuant,1prm-all,Ridge,CheckGrid:SIGMA^2 bug\n#375EncQuant,1prm-all,Ridge,CheckGrid:SIGMA^2 'sigma0':[1,10,100],'sigma1':[1,10,100]\n#376EncQuant,1prm-all,Ridge,CheckGrid:SIGMA^2 'sigma0':[1,1e10],'sigma1':[1,1e10]\n#377'm0':[0.001,1e10]^2 bug\n#378'm0':[0.001,1e10]^2\n#379EncQuant,1prm-all,Ridge BigGridA,M,Q\n\n\n\n\n \n# # RFR&TE-ith Try to reproduce simple not bad results with RFR - v258-quick save\n# # LB0653 - 0.990555 2.66748 368 RFR&TE-ith Reproduce 258 - simple, not bad , tsvd25,OptAmbr, 1prm-all\n# main_config_model_feature_etc = {'model': 'RFR','n_estimators':10, 'max_depth':3,\n# 'reducer': 'tsvd', 'n_components': n_components, 'smoothing0':10,'smoothing1':10,\n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']} # \n# dict_prms_lists = {'n_estimators': [5,8,10,12,15,20,25,30,50], 'max_depth': [1,2,3,4,5,6],\n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], } \n \n \n# # Ridge&ChemBert_mean\n# #main_config_model_feature_etc = {'model':'Ridge', 'alpha':1, 'features_mode': 'chembert_mean', 'reducer':'tsvd','n_components':n_components} \n# main_config_model_feature_etc = {'model':'Ridge', 'alpha':1, 'features_mode': 'chembert_cls', 'reducer':'tsvd','n_components':n_components} \n# dict_prms_lists = {'alpha':[0.001,0.002,0.005,0.008, 0.01,0.02,0.05,0.08, 0.1,0.2,0.5,0.8, 1,2,5,8,10,20,50,100]}\n# # LGB&oheDrug\n# main_config_model_feature_etc = {'model': 'LGB','n_estimators':5,'learning_rate':0.1, 'max_depth':3 , 'min_child_samples':5, 'num_leaves':5, 'reg_alpha':0, 'reg_lambda':0, \n# 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# dict_prms_lists = {'n_estimators': [5,8,10,12,15,20,25,30,50,100], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3], 'max_depth': [1,2,3,4,5,6],\n# 'min_child_samples': [ 1,2,3,4,5,10,15,20,50,100, 300], 'num_leaves':[ 2,3,4,5,10,15,20,50,100, 300],'reg_alpha':[ 1e-3,0.1,0.5,1,10], 'reg_lambda':[ 1e-3,0.1,0.5,1,10], }# \n# RFR, ohe\n# main_config_model_feature_etc = {'model': 'RFR','n_estimators':25, 'max_depth':3, 'min_samples_leaf':1 , 'min_samples_split':2, # 'criterion': 'squared_error',\n# 'smoothing0':10,'smoothing1':10, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# dict_prms_lists = {'n_estimators': [5,8,10,15,20,30,50,100], 'max_depth': [1,2,3,4,5,6], 'min_samples_leaf':[1,2,3,5], 'min_samples_split':[2,3,4,5], \n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], }# # 'criterion': ['squared_error','absolute_error','friedman_mse'] # - seems makes worse # ,'poisson' - only for positive\n# RFR, TE-th \n# main_config_model_feature_etc = {'model': 'RFR','n_estimators':25, 'max_depth':3, 'min_samples_leaf':1 , 'min_samples_split':2, # 'criterion': 'squared_error',\n# 'smoothing0':10,'smoothing1':10, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']} # \n# dict_prms_lists = {'n_estimators': [5,8,10,12,15,20,21,22,23,24,25,26,27,28,29,30,50], 'max_depth': [1,2,3,4,5,6], 'min_samples_leaf':[1,2,3,5], 'min_samples_split':[2,3,4,5], \n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], }# # 'criterion': ['squared_error','absolute_error','friedman_mse'] # - seems makes worse # ,'poisson' - only for positive\n \n# LGB \n# main_config_model_feature_etc = {'model': 'LGB','n_estimators':10, 'max_depth':3,'learning_rate':0.1, # 'reg_alpha':1,'reg_lambda':0.5,\n# 'reducer': 'tsvd', 'n_components': n_components, 'smoothing0':10,'smoothing1':10,\n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']} # \n# main_config_model_feature_etc = {'model': 'CATB','iterations':10, 'depth':3,'learning_rate':0.1,'subsample':1, 'colsample_bylevel':1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']} # \n# main_config_model_feature_etc = {'model': 'CATB','iterations':140, 'depth':3,'learning_rate':0.1,'subsample':1, 'colsample_bylevel':1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# main_config_model_feature_etc = {'model': 'LGB','n_estimators':1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# main_config_model_feature_etc = {'model': 'KRR', 'alpha':0.01, 'kernel': 'rbf', 'gamma': None, 'degree': 3, 'coef0': 1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n # ?? [‘additive_chi2’, ‘chi2’, ‘linear’, ‘poly’, ‘polynomial’, ‘rbf’, ‘laplacian’, ‘sigmoid’, ‘cosine’]\n \n# main_config_model_feature_etc = {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'reducer': 'tsvd', 'n_components': n_components, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}\n# main_config_model_feature_etc_tmp = {'reducer':'tsvd','n_components':n_components, 'model':'SVR', 'kernel':'linear','C':0.01 ,'epsilon': 1,'shrinking':True, 'gamma':'scale',\n# 'features_mode':'target_enc_i_th_target', 'list_features_in' : ['cell_type', 'sm_name'] } # \nelse:\n main_config_model_feature_etc = {'reducer':'tsvd','n_components':n_components}\n main_config_model_feature_etc['i_target_cfg']={}\n for i_target in range( n_components ):\n# #429 CatbEnc: best score [0.988689 2.608439] [0.9887 2.6084]\n# main_config_model_feature_etc_tmp = {'model': 'SVR', 'C': 0.01, 'kernel': 'linear', 'a0': 0.1, 'a1': 1, 'features_mode': 'CatBoostEncoder', 'targets_to_encode': 'i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'reducer': 'tsvd', 'n_components': 25}\n# # dict_prms_lists = {'C':[100,10, 1,1e-1, 1e-2, 1e-3,1e-4,1e-5,1e-6,1e-7], \n# # 'a0':[1e-1, 1,1e1,1e2,1e3,1e4], 'a1':[1e-1, 1,1e1,1e2,1e3,1e4]}\n# dict_prms_lists = {'C':[0.01, 1e-1, 1e-3,1e-4,100] } # \n# # 'a0':[1e-1, 1,1e1,1e2,1e3,1e4], 'a1':[1e-1, 1,1e1,1e2,1e3,1e4]}\n# # 443 - analogue of 429 best score [0.9887 2.6084]\n# # 443CatbEnc-ith,svrlin,op2-all-tgs,tsvd25,1r,Ini429 # \n# # 444Catb-ith,SVRl,op2-all.C-3val,tsvd25,1r,Ini429\n# # 445Catb-ith,SVRl,op2-all.C-4val,tsvd25,1r,Ini429\n# # 446Catb-ith,SVRl,op2-all.C-5val,tsvd25,1r,Ini429\n# # 447Catb-ith,SVRl,op2-all.C-4bval,tsvd25,1r,Ini429\n# # 448Catb-ith,SVRl,op2-all.C-5bval,tsvd25,1r,Ini429\n\n\n\n \n# # Quantile & SVR-rbf\n# #404 LB0.619 {'model': 'SVR', 'C': 100, 'alpha': 100000.0, 'kernel': 'rbf', 'features_mode': 'QuantileEncoder', 'quantile0': 0.6, 'quantile1': 0.65, 'm0': 0.1, 'm1': 2, 'targets_to_encode': 'i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'reducer': 'tsvd', 'n_components': 25}\n# main_config_model_feature_etc_tmp = {'model': 'SVR', 'C': 100, 'kernel': 'rbf', 'features_mode': 'QuantileEncoder', 'quantile0': 0.6, 'quantile1': 0.65, 'm0': 0.1, 'm1': 2, 'targets_to_encode': 'i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'reducer': 'tsvd', 'n_components': 25}\n# #dict_prms_lists = {'C':[1e3,1e2,1e1,1,1e-1,1e-2,1e-3,1e-6], 'm0':[0.1,1,1e1,1e2,1e3,1e4], 'm1':[0.1, 1,2,1e1,1e2,2e2,1e3,1e4], \n# # 'quantile0':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95],'quantile1':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95]}\n# dict_prms_lists = {'C':[100,1e3,1e1,1,]} #, 'm0':[0.1,1,1e1,1e2,1e3,1e4], 'm1':[0.1, 1,2,1e1,1e2,2e2,1e3,1e4], \n# # 'quantile0':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95],'quantile1':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95]}\n \n# # #440 - analogue of 4047lb0.619\n# # #440QEnc-ith,svrrbf,op2-all-tgs,tsvd25,1r,Ini404\n# # #441QEnc-ith,svrrbf,op2-all-tgs,tsvd25,1r,Ini404-\n# # #442QEnc-ith,svrrbf,op2-all-tgs,tsvd35,1r,Ini404-\n# # 449Q-ith,SVRr,OP2-a.C8,tsvd25,1r,Ini404-\n# # 450Q-ith,SVRr,OP2-a.C6,tsvd25,1r,Ini404-\n# # 451Q-ith,SVRr,OP2-a.C4,tsvd25,1r,Ini404-\n\n # Quantile & SVR-linear\n main_config_model_feature_etc_tmp = {'model':'SVR', 'C': 0.0002, 'kernel':'linear', \n 'features_mode': 'QuantileEncoder', 'quantile0':0.75, 'quantile1':0.9, 'm0':0.1,'m1':2, # 'sigma0':0.01,'sigma1':0.01, \n 'targets_to_encode':'i_th_target', 'list_features_in': ['cell_type', 'sm_name'],\n 'reducer':'tsvd','n_components':n_components} \n # 408 0.619: best score [0.962011] [0.962]\n # {'model': 'SVR', 'C': 0.0002, 'kernel': 'linear', 'features_mode': 'QuantileEncoder', 'quantile0': 0.75, 'quantile1': 0.9, 'm0': 0.1, 'm1': 2, 'targets_to_encode': 'i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'reducer': 'tsvd', 'n_components': 25}\n # best (mrrmse) score [0.962011] [0.962]\n #dict_prms_lists = {'C':[1,1e-1,1e-2,1e-3,1e-4,2e-4,5e-4,1e-5,1e-6], 'm0':[0.1,1,1e1,1e2,1e3,1e4], 'm1':[0.1, 1,2,1e1,1e2,2e2,1e3,1e4], \n # 'quantile0':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95],'quantile1':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95]}\n dict_prms_lists = {'C':[ 0.0002, 1,1e-1,1e-2,]}# ,2e-4,5e-4,1e-5,1e-6, 'm0':[0.1,1,1e1,1e2,1e3,1e4], 'm1':[0.1, 1,2,1e1,1e2,2e2,1e3,1e4], \n # 'quantile0':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95],'quantile1':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95]}\n# 437 - analogue of 408 lb0.619\n# 437QEnc-ith,svrlin,op2-all-tgs,tsvd25,1r,Ini408 \n# 438QEnc-ith,svrlin,op2-all-tgs,tsvd25,1r,Ini408,+ \n# tsvd35 439QEnc-ith,svrlin,op2-all-tgs,tsvd35,1r,Ini408,+\n# 452Q-ith,SVRl,OP2-a.C6,tsvd25,1r,Ini408\n# 453Q-ith,SVRl,OP2-a.C4,tsvd25,1r,Ini408\n\n\n# # Quantile Transformer Features\n# # 402 (LB0.619): {'model': 'KRR', 'alpha': 10.0, 'kernel': 'rbf', 'features_mode': 'QuantileEncoder', 'quantile0': 0.75, 'quantile1': 0.6, 'm0': 10000.0, 'm1': 200, 'targets_to_encode': 'i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'reducer': 'tsvd', 'n_components': 25}\n# main_config_model_feature_etc_tmp = {'model':'KRR', 'alpha':1e5, 'kernel':'rbf', \n# 'features_mode': 'QuantileEncoder', 'quantile0':0.75, 'quantile1':0.6, 'm0':1e4,'m1':2e2, # 'sigma0':0.01,'sigma1':0.01, \n# 'targets_to_encode':'i_th_target', 'list_features_in': ['cell_type', 'sm_name'],\n# 'reducer':'tsvd','n_components':n_components} \n# dict_prms_lists = {'alpha':[0.1,1,1e1, 1e2, 1e3,1e4,1e5,1e6], 'm0':[1,1e1,1e2,1e3,1e4], 'm1':[1,1e1,1e2,2e2,1e3,1e4], \n# 'quantile0':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9],'quantile1':[0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.85,0.9]}\n# # 434QEnc-ith,krr-rbf,op2-all-tgs,tsvd25,1r \n# # 434 opt-for-all targs analogue of 402:: 0.619\t0.9656\t2.8382\top2cv\t402\t1.182254\t1h46min\tkrr rbf\ttsvd25_KRR_QuantileEncoder_BothEncoded\n# # 435QEnc-ith,krr-rbf,op2-all-tgs,tsvd25,2r\n# # 436QEnc-ith,krr-rbf,op2-all-tgs,tsvd25,1r,Ini402 \n\n\n\n\n# # Quantile Transformer Features\n# main_config_model_feature_etc_tmp = {'model':'Ridge', 'alpha':1e5, # 'kernel':'rbf', \n# 'features_mode': 'QuantileEncoder', 'quantile0':0.6, 'quantile1':0.6, 'm0':1,'m1':1, # 'sigma0':0.01,'sigma1':0.01, \n# 'targets_to_encode':'i_th_target', 'list_features_in': ['cell_type', 'sm_name'],\n# 'reducer':'tsvd','n_components':n_components} \n# dict_prms_lists = {'alpha':[1e1, 1e2, 1e3,1e4,2e4,5e4,1e5,2e5,5e5,1e6,2e6,5e6], 'm0':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], 'm1':[1e-1, 2e-1,5e-1, 1,2,5,1e1,20, 50, 1e2,200,500,1e3,1e4], \n# 'quantile0':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.9],'quantile1':[0.4,0.45, 0.5,0.55, 0.6,0.65,0.7,0.75,0.8,0.9]}\n#387EncQuant,OptAmb-all-tgs,Ridge BigGridA,M,Q\n#385EncQuant,OptAmb-all-tgs,KRRrbf BigGridA,M,Q\n#386EncQuant,OptAmb-all-tgs,KRRlin BigGridA,M,Q\n\n# # SVR & chembert or BackwardDifferenceEncoder or HelmertEncoder\n# list_C_SVR_fromV180 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.02, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02]\n# list_C_SVR_fromV181 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.05, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.05, 0.02, 0.01, 0.01, 0.02, 0.02, 0.02, 0.01, 0.02, 0.05, 0.01, 0.05, 0.05, 0.02, 0.02, 0.01, 0.01, 0.02, 0.02, 0.01]\n# main_config_model_feature_etc_tmp = {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, \n# 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode':'BackwardDifferenceEncoder', 'list_features_in': [ 'sm_name']}# 'cell_type', 'HelmertEncoder' , 'BackwardDifferenceEncoder'}# 'chembert_cls'} \n# #main_config_model_feature_etc_tmp['C'] = list_C_SVR_fromV180[i_target]\n# main_config_model_feature_etc_tmp['C'] = list_C_SVR_fromV181[i_target] \n# dict_prms_lists = {'C': [0.01,0.009,0.011, 0.005, 0.001, 0.02 , 0.05 , 0.1,0.2, 0.5,1,2,5,10,20 ] }# \n\n\n \n# # SVR & chembert or BackwardDifferenceEncoder or HelmertEncoder\n# list_C_SVR_fromV180 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.02, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02]\n# list_C_SVR_fromV181 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.05, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.05, 0.02, 0.01, 0.01, 0.02, 0.02, 0.02, 0.01, 0.02, 0.05, 0.01, 0.05, 0.05, 0.02, 0.02, 0.01, 0.01, 0.02, 0.02, 0.01]\n# main_config_model_feature_etc_tmp = {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, \n# 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode':'BackwardDifferenceEncoder', 'list_features_in': [ 'sm_name']}# 'cell_type', 'HelmertEncoder' , 'BackwardDifferenceEncoder'}# 'chembert_cls'} \n# #main_config_model_feature_etc_tmp['C'] = list_C_SVR_fromV180[i_target]\n# main_config_model_feature_etc_tmp['C'] = list_C_SVR_fromV181[i_target] \n# dict_prms_lists = {'C': [0.01,0.009,0.011, 0.005, 0.001, 0.02 , 0.05 , 0.1,0.2, 0.5,1,2,5,10,20 ] }# \n\n# 0.9702 3.0856 348SVR-l,Opt2-all-tgs,BDE-encBoth,tsvd30 \n# 0.9802 2.9434 349SVR-l,Opt2-all-tgs,Helm-encBoth,tsvd30 \n# 0.9854 2.9475 350SVR-l,Opt2-all-tgs,Helm-encDr,tsvd30 \n# 0.9776 3.138 351SVR-l,Opt2-all-tgs,BDE-encDr,tsvd30 \n\n# # KRR & chembert or BackwardDifferenceEncoder or HelmertEncoder\n# main_config_model_feature_etc_tmp = {'model': 'KRR', 'alpha':1, 'kernel': 'rbf', 'gamma': None, 'degree': 3, 'coef0': 1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'BackwardDifferenceEncoder' , 'list_features_in': [ 'sm_name'] } # 'cell_type', 'BackwardDifferenceEncoder', 'chembert_cls' 'chembert_mean' \n# dict_prms_lists = { 'alpha': [0.01,0.05,0.08,0.09,.1,1.1,1.2,1.3,1.5, 1,10,100 ] } \n# 3-4h: \n#1.6321 2.9923 352KRR-rbf,Opt2-all-tgs,BDE-encBoth,tsvd30 \n#LB0.618 0.9787 2.5917 353KRR-rbf,Opt2-all-tgs,Helm-encBoth,tsvd30 # Not bad LB and CV \n#1.0052 2.6035 354KRR-rbf,Opt2-all-tgs,Helm-encDr,tsvd30 \n#1.0252 2.6293 355KRR-rbf,Opt2-all-tgs,BDE-encDr,tsvd30 \n \n# #CatBoost & chembert or BackwardDifferenceEncoder or HelmertEncoder\n# main_config_model_feature_etc_tmp = {'model': 'CATB','iterations':10, 'depth':3,'learning_rate':0.1,'subsample':1, 'colsample_bylevel':1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'BackwardDifferenceEncoder', 'list_features_in': [ 'sm_name'] } #'HelmertEncoder', 'cell_type', 'BackwardDifferenceEncoder', 'chembert_cls' 'chembert_mean' \n# dict_prms_lists = {'iterations': [5,8,10,12,15,20,25,30,50],'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3,0.4] , 'depth': [1,2,3,4,5,6] }\n# 1.2213 3.113 356CatB,Opt2-all-tgs,BDE-encBoth,tsvd30 # 5.5h \n# 1.0465 2.8551 357CatB,Opt2-all-tgs,Helm-encBoth,tsvd30 #7h9min\n# LB0.625 0.9985 2.609 358CatB,Opt2-all-tgs,Helm-encDr,tsvd30 # 7h48min # Not bad LB and CV , but not so good\n# 1.005 2.6769 359CatB,Opt2-all-tgs,BDE-encDr,tsvd30 # 6h47min\n\n\n\n \n# # RF&chembert \n# main_config_model_feature_etc_tmp = {'model': 'RFR','n_estimators':25, 'max_depth':3, 'min_samples_leaf':1 , 'min_samples_split':2, # 'criterion': 'squared_error',\n# 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'chembert_mean'} # 'cell_type',\n# dict_prms_lists = {'n_estimators': [5,8,10,15,20,25,30,35,50,100], 'max_depth': [1,2,3,4,5,6], 'min_samples_leaf':[1,2,3,5], 'min_samples_split':[2,3,4,5], \n# }# # 'criterion': ['squared_error','absolute_error','friedman_mse'] # - seems makes worse # ,'poisson' - only for positive\n# #LGB&chembert\n# main_config_model_feature_etc_tmp = {'model': 'LGB','n_estimators':5, 'learning_rate':0.1, 'max_depth':3, 'min_child_samples':5, 'num_leaves':5, 'reg_alpha':0, 'reg_lambda':0, \n# 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'chembert_mean'} # 'cell_type',\n# dict_prms_lists = {'n_estimators': [5,8,10,12,15,20,25,30,50,100], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3], 'max_depth': [1,2,3,4,5,6],\n# 'min_child_samples': [ 1,2,3,4,5,10,15,20,50,100, 300], 'num_leaves':[ 2,3,4,5,10,15,20,50,100, 300],'reg_alpha':[ 1e-3,0.1,0.5,1,10], 'reg_lambda':[ 1e-3,0.1,0.5,1,10], }# \n \n# # Ridge&ChemBert_mean\n# main_config_model_feature_etc_tmp = {'model':'Ridge', 'alpha':1, 'features_mode': 'chembert_mean', 'reducer':'tsvd','n_components':n_components} \n# # main_config_model_feature_etc_tmp = {'model':'Ridge', 'alpha':1, 'features_mode': 'chembert_cls', 'reducer':'tsvd','n_components':n_components} \n# dict_prms_lists = {'alpha':[0.001,0.002,0.005,0.008, 0.01,0.02,0.05,0.08, 0.1,0.2,0.5,0.8, 1,2,5,8,10,20,50,100]}\n \n# #LGB & ohe & BackwardDifferenceEncoder & 'HelmertEncoder',\n# main_config_model_feature_etc_tmp = {'model': 'LGB','n_estimators':5, 'learning_rate':0.1, 'max_depth':3, 'min_child_samples':5, 'num_leaves':5, 'reg_alpha':0, 'reg_lambda':0, \n# 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'BackwardDifferenceEncoder', 'list_features_in': [ 'sm_name']} # 'cell_type', 'onehot' 'HelmertEncoder','BackwardDifferenceEncoder',\n# dict_prms_lists = {'n_estimators': [5,8,10,12,15,20,25,30,50,100], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3], 'max_depth': [1,2,3,4,5,6],\n# 'min_child_samples': [ 1,2,3,4,5,10,15,20,50,100, 300], 'num_leaves':[ 2,3,4,5,10,15,20,50,100, 300],'reg_alpha':[ 1e-3,0.1,0.5,1,10], 'reg_lambda':[ 1e-3,0.1,0.5,1,10], }# \n#\n# 1.1421 3.1179 360LGB,Opt2-all-tgs,BDE-encBoth,tsvd30 #10h\n# 1.0581 3.07 361LGB,Opt2-all-tgs,Helm-encBoth,tsvd30 #10h\n# 0.9992 2.6094 362LGB,Opt2-all-tgs,Helm-encDr,tsvd30\n#363LGB,Opt2-all-tgs,BDE-encDr,tsvd30\n\n# RF&oheDrug \n# main_config_model_feature_etc_tmp = {'model': 'RFR','n_estimators':25, 'max_depth':3, 'min_samples_leaf':1 , 'min_samples_split':2, # 'criterion': 'squared_error',\n# 'smoothing0':10,'smoothing1':10, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# dict_prms_lists = {'n_estimators': [5,8,10,15,20,25,30,35,50,100], 'max_depth': [1,2,3,4,5,6], 'min_samples_leaf':[1,2,3,5], 'min_samples_split':[2,3,4,5], \n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], }# # 'criterion': ['squared_error','absolute_error','friedman_mse'] # - seems makes worse # ,'poisson' - only for positive\n \n# LGB \n# main_config_model_feature_etc_tmp = {'model': 'LGB','n_estimators':10, 'max_depth':3,'learning_rate':0.1, # 'reg_alpha':1,'reg_lambda':0.5,\n# 'reducer': 'tsvd', 'n_components': n_components, 'smoothing0':10,'smoothing1':10,\n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']} # \n# dict_prms_lists = {'n_estimators': [5,6,7,8,9,10,12,15,20,25,30,50], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3], 'max_depth': [1,2,3,4,5,6],\n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], }# , 'reg_alpha':[0,1],'reg_lambda':[0,1] } # 'depth': [1,3,6,10 ],\n# #CatBoost&TE-ith:\n# main_config_model_feature_etc_tmp = {'model': 'CATB','iterations':10, 'depth':3,'learning_rate':0.1,'subsample':1, 'colsample_bylevel':1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'smoothing0':10,'smoothing1':10, } # \n# # Good params 254 LB0.608\n# # # Repeat with tsvd40 # 346CatB,TE-ith,Opt2-all-tgs&SM,tsvd40,1R\n# # dict_prms_lists = {'iterations': [5,8,10,12,15,20,25,30,50], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3,0.4] , 'depth': [1,2,3,4,5,6],\n# # 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], }\n# # # Try stricter v347\n# # 347CatB,TE-ith,Opt2-all-tgs&SM,tsvd40,1R,Strict \n# dict_prms_lists = {'iterations': [5,8,10,12,15,20,25,30,50], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3,0.4] , 'depth': [1,2,3,4,5],\n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e3, 0,1], }\n \n# main_config_model_feature_etc_tmp = {'model': 'CATB','iterations':20, 'depth':3,'learning_rate':0.2,'subsample':1, 'colsample_bylevel':1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# main_config_model_feature_etc_tmp = {'model': 'KRR', 'alpha':0.01, 'kernel': 'rbf', 'gamma': None, 'degree': 3, 'coef0': 1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'onehot', 'list_features_in': [ 'sm_name']} # 'cell_type',\n# main_config_model_feature_etc_tmp = {'model': 'KRR', 'alpha':1e3, 'kernel': 'linear', 'gamma': None, 'degree': 3, 'coef0': 1, 'reducer': 'tsvd', 'n_components': n_components, \n# 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name'], 'smoothing0':1, 'smoothing1': 1e4 }\n# main_config_model_feature_etc_tmp['C'] = list_C_SVR_fromV181[i_target]\n# list_C_SVR_fromV180 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.02, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02]\n# list_C_SVR_fromV181 = [0.01, 0.02, 0.01, 0.1, 0.001, 0.01, 0.01, 0.005, 0.05, 0.01, 0.02, 0.05, 0.01, 0.01, 0.01, 0.01, 0.02, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.1, 0.05, 0.01, 0.01, 0.05, 0.02, 0.02, 0.05, 0.02, 0.01, 0.01, 0.02, 0.02, 0.02, 0.01, 0.02, 0.05, 0.01, 0.05, 0.05, 0.02, 0.02, 0.01, 0.01, 0.02, 0.02, 0.01]\n# main_config_model_feature_etc_tmp = {'model': 'KRR', 'alpha':1e6, 'kernel': 'laplacian', 'gamma': None, 'degree': 3, 'coef0': 1, 'reducer': 'tsvd', 'n_components': n_components, \n# main_config_model_feature_etc_tmp = {'model': 'SVR', 'kernel': 'linear', 'C': 0.01, 'epsilon': 1, 'gamma': 0.01, 'shrinking': False, 'features_mode': 'target_enc_i_th_target', 'list_features_in': ['cell_type', 'sm_name']}\n main_config_model_feature_etc['i_target_cfg'][i_target] = main_config_model_feature_etc_tmp.copy()\n\n# ------------------------------------------------------------------------------------------------\n# BackwardDifferenceEncoder\n# ------------------------------------------------------------------------------------------------\n\n# ------------------------------------------------------------------------------------------------\n# ChemBert\n# ------------------------------------------------------------------------------------------------\n# Version 292-299 - preliminary analysis in section \"ChemBert\", seems around 0.1 - best for CLS, \n# 1.0271 2.6712 304RidgeChemBert,Opt2cv-1prm-all 'alpha': 0.02 - we get different from what is by visual inspection \n# 1.0266 2.6714 305RidgeChemBert-CLS,Opt2cv-1prm-all,tsvd25 , 'alpha': 0.2,\n# 1.0299 2.6719 306RidgeChemBert-CLS,Opt2cv-1prm-all,tsvd50 'alpha': 0.2,\n# 1.0304 2.6717 307RidgeChemBert-Mean,Opt2cv-1prm-all,tsvd50 best_prm 0.02\n# 1.0304 2.671712 308RidgeChemBert-Mean,OptAmbr-1prm-all,tsvd50\n# 1.0299 2.6719 309RidgeChemBert-cls,OptAmbr-1prm-all,tsvd50\n# \n# Opt-all-targets\n# To compare benchmark1: 0.615\t0.604\t1.0266\t2.6698\tRidge\tonehot drug\ttsvd30\t12,17\tReproduce MT - onehot (drug only), tsvd30, Ridge0.1, fit_intercept = False\n# 1.0286 2.6311 310RidgeChemBert-mean,Opt2-all-tgs,tsvd50 #8h # \n# 1.0275 2.6557 311RidgeChemBert-cls,Opt2-all-tgs,tsvd50 # 5h34m # \n# 1.0259 2.6717 312RidgeChemBert-cls,OptAmr=all-tgs,tsvd50 # 4.5h # \n# 1.0222 2.6524 313RidgeChemBert-mean,OptAmr-all-tgs,tsvd50 # 6h # Somewhat better benchmark1 in both CV \n#\n# Other models:\n# To compare benchmark1: 0.615\t0.604\t1.0266\t2.6698\tRidge\tonehot drug\ttsvd30\t12,17\tReproduce MT - onehot (drug only), tsvd30, Ridge0.1, fit_intercept = False\n# 0.9934 2.5885 314LGBChemBert-mean,Opt2-all-tgs,tsvd25 #10h45min # may be too good MT score =========== try it ?????????? - may be too good MT\n# 315RF,ChemBert-mean,Opt2-all-tgs,tsvd25 # 12h cancel # \n# 0.9971 2.613 316CatB,ChemBert-mean,Opt2-all-tgs,tsvd25 # 6h46m # # Quite better benchmark1 in both CV \n# 1.0106 2.5728 317KRR-rbf,ChemBert-mean,Opt2-all-tgs,tsvd25 # 2h # # Quite better benchmark1 in both CV\n# 1.0294 2.6281 318KRR-lin,ChemBert-mean,Opt2-all-tgs,tsvd25 #1.5h # \n# 0.9904 3.1711 319SVR-lin,ChemBert-mean,Opt2-all-tgs,tsvd25 # 2h - BAD \n# OptAmb:\n# 0.99 3.172246 320SVR-lin,ChemBert-mean,OptAmb-all-tgs,tsvd25 - AGAIN BAD \n# 1.0238 2.6473 321KRR-lin,ChemBert-mean,OptAmb-all-tgs,tsvd25 # \n# 1.0063 2.6381 322KRR-rbf,ChemBert-mean,OptAmb-all-tgs,tsvd25 # 1h # - not bad - rbf is again better that linear for KRR =========== try it ??????????\n# 0.9953 2.6315 323CatB,ChemBert-mean,OptAmb-all-tgs,tsvd25 #9h\n# CLS:\n# 0.9983 2.631661 324CatB,ChemBert-cls,OptAmb-all-tgs,tsvd25 # 9h20\n# 1.0063 2.6381 325KRR-rbf,ChemBert-cls,OptAmb-all-tgs,tsvd25 #1h37m\n# 1.0238 2.6473 326KRR-lin,ChemBert-cls,OptAmb-all-tgs,tsvd25 # 45min # Better than benchamrk1 in MT\n# 0.9900 3.1722 327SVR-linear,ChemBert-cls,OptAmb-all-tgs,tsvd25 # 1.5h # BAD MT\n# 0.9957 3.2012 328SVR-rbf,ChemBert-cls,OptAmb-all-tgs,tsvd25 # 2hours # BAD MT\n#\n# tsvd50\n# LB0.623 1.0099 2.5706 329KRR,ChemBert-cls,Opt2CV-all-tgs,tsvd50\n# 1.0099 2.5706 330KRR?,ChemBert-mean,Opt2CV-all-tgs,tsvd50\n# 0.9803 3.084694 331Ridge,ChemBert-cls,OptAmb-all-tgs,tsvd50\n# 0.980318 3.084694 332Ridge,ChemBert-mean,OptAmb-all-tgs,tsvd50\n# Below CLS is better than MEAN:\n# 1.005185 2.636429 333KRR-rbf,ChemBert-mean,OptAmb-all-tgs,tsvd50\n# 0.9803 3.084694 334KRR-rbf,ChemBert-cls,OptAmb-all-tgs,tsvd50\n# 0.9805 12h cancel 335SVR-lin,ChemBert-mean???,OptAmb-all-tgs,tsvd50,ExtC\n# 0.9755 12h cancle 336SVR-lin,ChemBert-cls,OptAmb-all-tgs,tsvd50,ExtC\n\n#\n\n# ------------------------------------------------------------------------------------------------\n# Random Forest\n# ------------------------------------------------------------------------------------------------\n# Conclusion: with ohe seems not promising\n#\n# TE-ith, 1prm-all\n# 0.9874 2.5576 258 rfr opAmbr-1prm-all depth,iter, TE-ith,tsvd25, 'n_estimators': 25, 'max_depth': 2,\n# LB0.689 0.9939 2.5496 - huge random variation: 0.985899 - 2.580414 259 rfr op2cv ---------------------------------- TERRIBLE LB for not so bad CVs\n# huge random variation from run to run 'n_estimators': 30, 'max_depth': 2, 'reducer':\n# 1.0008 2.5761 261 rfr op2-1prm-all add criteria, TE-ith,tsvd25 # # 'n_estimators': 21, 'max_depth': 3, 'criterion': 'absolute_error', 'reducer': 'tsvd', 'n_components': 25, 'smoothing0': 10, 'smoothing1': 10,\n# worse results when criteria changed \n# 1.0156-1.021325 2.6193 266 rfr op2-1prm-all,tsvd25,TE-ith,add min_samples_leaf':[1,2,3,5], 'min_samples_split, delete criteria 'n_estimators': 29, 'max_depth': 3, 'min_samples_leaf': 1, 'min_samples_split': 2, 'smoothing0': 10, 'smoothing1': 10, '\n# 0.993978 2.601749 267 rfr opAmbr-1prm-all,tsvd25,TE-ith,add min_samples_leaf':[1,2,3,5], 'min_samples_split, delete criteria\n#\n# tune-every-target-parms:\n# dict_prms_lists = {'n_estimators': [5,8,10,15,20,25,30,35,50,100], 'max_depth': [1,2,3,4,5,6], 'min_samples_leaf':[1,2,3,5], 'min_samples_split':[2,3,4,5], \n# 'smoothing0': [1e1,0,1,1e2,1e4 ] , 'smoothing1': [1e1,1e2, 1e4, 0,1], }# # 'criterion': ['squared_error','absolute_error','friedman_mse'] # - seems makes worse # ,'poisson' - only for positive\n# 1.016613 2.6312 273 rfr opMT-all-tgs,tsvd25,TE-ith #3h9min No so bad ? \n# 1.0256 2.6442 274 rfr op2-all-tgs,tsvd25,TE-ith #2h24min\n# 1.0056 2.7108 275 rfr opAmrb-all-tgs,tsvd25,TE-ith #3h48m\n\n# \n# onehot-Drug - seems not promising\n# 1.011841 2.764828 268 rfr OheDrug, opAmbr-1prm-all,tsvd25, # not promising \n# 1.020964 2.759296 269 rfr OheDrug, op2-1prm-all,tsvd25, # 'n_estimators': 5, 'max_depth': 4, 'min_samples_leaf': 1, 'min_samples_split': 2, 'smoothing0': 0, 'smoothing1': 10,\n# Compare Ridge-MT: 0.615\t\t1.0234\t2.6680\tRidge\tonehot drug\ttsvd30\t19\tMT with fit_intercept = True - CV is better but LB is the same \n# Compare CatB:\n# 1.0085 2.7057 225Catb,oheDr,tsvd25,Op2-1prm-all,Depth,LR, # dict_prms_lists = {'depth': [2,3,4], 'learning_rate':[0.05,0.1,0.15,0.2,0.3,0.5] }\n# 1.0088 2.6985 231CatB,oheDr,tsvd50,Opt2-1prm-all, Iter,LR,dep,2Rnd # 1h7m 'iterations': 200, 'depth': 4, 'learning_rate': 0.1,\n\n\n# ------------------------------------------------------------------------------------------------\n\n# 18 10 2023: \n# CatB&TE-ith&opt2-all-targets - 0.612 - unexpectedly not so bad \n# not so good CatB&ohe: LB0.622 1.0027 2.6498 237CatB,oheDr,tsvd25,OptAmMT-all-tgs,1R\n# not so bad: LGB&TE-ith LB0.616 1.0009 2.6823 246LGB,Opt2-all-tgs,te-ith,tsvd25,2R # 7h7min\n\n# ------------------------------------------------------------------------------------------------\n# LGB\n# ------------------------------------------------------------------------------------------------\n# LGB & TE-ith\n# 1-prm-for-all:\n# 1.017 2.7877 245 LGB, first, 1prm-all, te-ith,tsvd25 'n_estimators': 8, 'max_depth': 3, 'learning_rate': 0.1\n# 0.9976 2.8153 250 LGB,OptA-1prm-all-tgs,te-ith,tsvd25,Smooth 'n_estimators': 8, 'max_depth': 4, 'learning_rate': 0.09, 'reducer': 'tsvd', 'n_components': 25, 'smoothing0': 1, 'smoothing1': 100.0,\n# 1.0144 2.7882 251 LGB,Opt2-1prm-all-tgs,te-ith,tsvd25,Smooth 'n_estimators': 8, 'max_depth': 4, 'learning_rate': 0.1, 'reducer': 'tsvd', 'n_components': 25, 'smoothing0': 10.0, 'smoothing1': 100.0,\n# 1.0712 2.6483 252 LGB,OptMT-1prm-all-tgs,te-ith,tsvd25,Smooth 'n_estimators': 25, 'max_depth': 3, 'learning_rate': 0.1, 'reducer': 'tsvd', 'n_components': 25, 'smoothing0': 10.0, 'smoothing1': 10.0,\n# optimize params for each target separately:\n# LB0.616 1.0009 2.6823 246LGB,Opt2-all-tgs,te-ith,tsvd25,2R # 7h7min\n# 1.0465 2.4793 248LGB,OptMT-all-tgs,te-ith,tsvd25,Smooth #2h50min\n# 0.9749 2.7404 249LGB,OptA-all-tgs,te-ith,tsvd25,Smooth #2h14min\n# \n# LGB&oheDrug - 1prm for all - seems not promosing\n# dict_prms_lists = {'n_estimators': [5,8,10,12,15,20,25,30,50], 'learning_rate':[0.05, 0.08,0.09, 0.1,0.15, 0.2,0.25, 0.3], 'max_depth': [1,2,3,4,5,6],\n# 'min_child_samples': [ 1,2,3,4,5,10,15,20,50,100, 300], 'num_leaves':[ 1,2,3,4,5,10,15,20,50,100, 300],'reg_alpha':[ 1e-3,0.1,0.5,1,10], 'reg_lambda':[ 1e-3,0.1,0.5,1,10], }# \n# 1.0669 2.9469 287 LGB,oheDr,tsvd25,op2-1prm-all, # 'n_estimators': 5, 'learning_rate': 0.1, 'max_depth': 3, 'min_child_samples': 3, 'num_leaves': 300, 'reg_alpha': 0, 'reg_lambda': 0,\n# 1.019299 2.745944 288 LGB,oheDr,tsvd25,opAmb-1prm-all, 'n_estimators': 50, 'learning_rate': 0.08, 'max_depth': 3, 'min_child_samples': 2, 'num_leaves': 300, 'reg_alpha': 10, 'reg_lambda': 0, \n# To compare RidgeMT: 0.615\t\t1.0234\t2.6680\tRidge\tonehot drug\ttsvd30\t19\tMT with fit_intercept = True - CV is better but LB is the same \n# Opt-all-tgs\n# 1.0105 2.71226 289 LGB,oheDr,tsvd25,opAm-all-tgs, # 4.5hours, Not so good scores \n# 1.0573 3.0526 290 LGB,oheDr,tsvd25,op2-all-tgs, #6h Very bad\n\n\n# ------------------------------------------------------------------------------------------------\n# CatBoost\n# ------------------------------------------------------------------------------------------------\n# Conclusions: \n# CatB&oheDr&1prm-for-all - not promosing CV results\n# CatB&oheDr&optprm-all-targes - so-so - both CV&LB: LB0622, 1.0027 2.6498\n# CatB&TE-ith&1prm-for-all - not so good LB&CV LB0.627 0.9908 2.517472 240Catb,TE-ith,OptAm-1prm-all,tsvd25,2R #9min # 'iterations': 10, 'depth': 3, 'learning_rate': 0.2,\n# CatB&TE-ith&&optprm-all-targes - not bad LB, quite good CV especially MT: LB0.612 0.9779 2.4194 244Catb,TE-ith,Opt2-all-tgs,tsvd25,1R\n# to-do:\n# Add smoothing optimizations, try tsvd50, try submit with random and with priors\n\n#CatB&oheDr\n# 1.0085 2.7057 225Catb,oheDr,tsvd25,Op2-1prm-all,Depth,LR, # dict_prms_lists = {'depth': [2,3,4], 'learning_rate':[0.05,0.1,0.15,0.2,0.3,0.5] }\n# 1.0085 2.7057 {'model': 'CATB', 'iterations': 140, 'depth': 4, 'learning_rate': 0.1,\n# 1.0114 2.6873 226Catb ohe iter,LR,depth # 'iterations': 500, 'depth': 3, 'learning_rate': 0.1,\n# 1.0088 2.6992 227Catb ohe iter,LR,depth # 'iterations': 200, 'depth': 4, 'learning_rate': 0.1\n# 1.0095 2.6953 228 iter,LR,depth 'iterations': 250, 'depth': 4, 'learning_rate': 0.1, \n# 1.0095 2.6953 229Carb , iter,LR,depth 'iterations': 250, 'depth': 4, 'learning_rate': 0.1,\n# 1.0095 2.6953 230CatB,oheDr,tsvd25,Opt2-1prm-all, Iter,LR,dep 'iterations': 250, 'depth': 4, 'learning_rate': 0.1 # 24min\n# 1.0088 2.6985 231CatB,oheDr,tsvd50,Opt2-1prm-all, Iter,LR,dep,2Rnd # 1h7m 'iterations': 200, 'depth': 4, 'learning_rate': 0.1,\n# 1.0078 2.6971 232CatB,oheDr,tsvd50,OptA-1prm-all, Iter,LR,dep,2Rnd # 1h7m 'iterations': 140, 'depth': 6, 'learning_rate': 0.1, \n# 1.0318 2.6765 233CatB,oheDr,tsvd50,OptMT-1prm-all,Iter,LR,dep,2R # 2h\n# \n# Opt-all-tgs\n# 1.020577 2.5629 235CatB,oheDr,tsvd25,OptMT-all-tgs,1R\n# 0.9969 2.684496 236CatB,oheDr,tsvd25,OptAm-all-tgs,1R\n# LB0.622 1.0027 2.6498 237CatB,oheDr,tsvd25,OptAmMT-all-tgs,1R\n# 238CatB,oheDr,tsvd25,OptAm-all-tgs,2R #12h cancel#logs almost finished 2rnd, but uplift from the first one is neglible: 0.9969->0.9966\n\n### Catb TE-ith\n# 0.9908 2.517472 239Catb,TE-ith,OptAm-1prm-all,tsvd25 #10min # 'iterations': 10, 'depth': 3, 'learning_rate': 0.2,\n# LB0.627 0.9908 2.517472 240Catb,TE-ith,OptAm-1prm-all,tsvd25,2R #9min # 'iterations': 10, 'depth': 3, 'learning_rate': 0.2,\n# 1.0078 2.4754 241Catb,TE-ith,OptMT-1prm-all,tsvd25,1R 'iterations': 20, 'depth': 2, 'learning_rate': 0.1,\n# \n# Opt-all-tgs\n# LB0.635 1.07448 2.3448 242Catb,TE-ith,OptMT-all-tgs,tsvd25,1R\n# LB0.621 0.9710 2.4709 243Catb,TE-ith,OptAm-all-tgs,tsvd25,1R\n# LB0.612 0.9779 2.4194 244Catb,TE-ith,Opt2-all-tgs,tsvd25,1R, 2h26min\n# 1.015306 2.383358 253CatB,TE-ith,OptMT-all-tgs&SM,tsvd25,1R, #3h42m\n# LB0.608 0.9683 2.4244 254CatB,TE-ith,Opt2-all-tgs&SM,tsvd25,1R, #5h ############## Quite good CV & LB !!!!!!!!!!!!!!!!!!!!!!!!!!!!\n# LB00.633 0.956 2.597715 255CatB,TE-ith,OptAm-all-tgs&SM,tsvd25,1R, #3.5h # == Bad LB, quite good CV =\n# tsvd50:\n#256CatB,TE-ith,OptAm-all-tgs&SM,tsvd50,1R, #12h cancel\n#257CatB,TE-ith,Opt2-all-tgs&SM,tsvd50,1R, #12h cancel\n# tsvd40 instead of 50\n# 338CatB,TE-ith,Opt2-all-tgs&SM,tsvd40,1R,Strict\n# 339CatB,TE-ith,Opt2-all-tgs&SM,tsvd40,1R\n\n# ------------------------------------------------------------------------------------------------\n# SVR\n# ------------------------------------------------------------------------------------------------\n# SVR-l&TE-ith\n# Try to uplift over: \n # LB0.610 0.9721 2.6752 145 OptAm one-for-all lin-SVR tsvd25 TE-ithBoth # 9h \n # Switch to mode_optimize = 'separate_params_for_each_model(target)'\n # 0.969 2.666 179 tsvd30 vary: 'C': [0.01,0.1,0.001 ] 1h13m\n # 0.9684 2.649 180 tsvd30 vary: extend around 0.01 #2h41m\n # 0.9669 2.6458 181 tsvd50 vary: extend around 0.01 #6h\n# As follows: \n# 0.9692 2.6222 199SVR-l,tsvd25,op2-all-tgs,TE-ith,C,Smooth,Rnd1 # 2h \n# 0.9692 2.6222 200SVR-l,tsvd25,op2-all-tgs,TE-ith,C,Smooth,Rnd2 # 3.5h\n# 201SVR-l,tsvd50,op2-all-tgs,TE-ith,C,Smooth,Rnd2 # Cancel 12 hours, but we see from logs that second round is not improving the fist one, so version 202 - is Okay \n# LB0608,with randomization or priors: 0607, 0.9661\t2.6165 202SVR-l,tsvd50,op2-all-tgs,TE-ith,C,Smooth,Rnd1\n\n# dict_prms_lists = { 'C': [0.01,0.02,0.05,0.1,0.001,0.002,0.005,0.008 ]} # , 'epsilon':[0.1,1,2.5,5,10,25, 50, 100],'shrinking':[ True, False] , 'gamma':['scale', 2,1,10,0.1,0.01], }\n# 'gamma':[2,1,10,0.1,0.01], - seems does not influence quality almost at all, but we have not checked default option \"scale\" - will run with it from V162\n# shriking seems also have low effect\n# High values of C are very slow for SVR-linear, BUT low quality :\n# C 10 Valid mrrmse: 1.0042 r2: -0.1374 time: 49.5 count 4 best: [0.9812] CV_scheme AmbrosM\n# C 100 Valid mrrmse: 1.0047 r2: -0.1371 time: 209.0 count 5 best: [0.9812] CV_scheme AmbrosM\n# C 1000 Valid mrrmse: 1.0064 r2: -0.1397 time: 1608.2 count 6 best: [0.9812] CV_scheme AmbrosM\n# C 10000 Valid mrrmse: 1.0271 r2: -0.1511 time: 7513.3 count 7 best: [0.9812] CV_scheme AmbrosM\n \n# 0.9776 2.6801 152 Opt2-all-targ lin-SVR tsvd30 TE-ithBoth # 10h\n# 153 OptAmb-all-targ lin-SVR tsvd25 TE-ithBoth # 12h canceled\n# 154 OptMT-all-targ lin-SVR tsvd25 TE-ithBoth # 12h canceled\n# 0.9776 2.6801 155 Opt2-all-targ lin-SVR tsvd25 TE-ithBoth # 10h \n# 0.9694 3.0762 156 Opt2-all-targ rbf-SVR tsvd25 TE-ithBoth # 6h12m\n# 0.9941 3.1865 157 Opt2-all-targ sig-SVR tsvd25 TE-ithBoth # 6.5h\n# 0.9923 3.1920 158 OptAmbr-all-targ sig-SVR tsvd25 TE-ithBoth #4h13min\n# 0.9667 3.0801 159 OptAmbr-all-targ rbf-SVR tsvd25 TE-ithBoth # 4h38m\n# 1.0854 2.7786 160 OptMT-all-targ rbf-SVR tsvd25 TE-ithBoth # 8.5h\n# 1.1448 3.1293 161 OptMT-all-targ sig-SVR tsvd25 TE-ithBoth # 2.5h \n# Conclusion - other than linear kernel seems to work worse \n\n# Try to improve over the best found - v145 LB0.610 - vary C and n_comp, cut C choices to avoid big, slow, low score\n# LB0.610 0.9721 2.6752 145 OptAm one-for-all lin-SVR tsvd25 TE-ithBoth # 9h \n# might be bug 0.9778 3.145729 166 OptAm one4all l-SVRtsvd50TE-ith,vary Gam\n# might be bug 0.9778 3.145729 167 OptAm one4all l-SVRtsvd50TE-ith # 9min\n# might be bug 0.9778 3.145729 168 OptAm one4all l-SVRtsvd30TE-ith,vary Gam\n# might be bug 0.9778 3.145729 169 OptAm one4all l-SVRtsvd30TE-ith, Ext145C\n# might be bug 170 tsvd100 OptAm one4all l-SVRtsvd100TE-ith\n\n# LB0.610 0.9721 2.6752 145 OptAm one-for-all lin-SVR tsvd25 TE-ithBoth # 9h \n# 0.9671 2.887729 171 params from V145(LB0.610) vary only C got 'C': 0.001 'AmbrosM'\n# 0.9671 2.887729 172 params from V145(LB0.610) vary only C got 'C': 0.001 \n# 0.9669 2.887315 173 params from V145(LB0.610) C-extend C got 'C': 0.001 'AmbrosM'\n# 0.9708 2.6737 174 opt2 params from V145(LB0.610) C-extend 'C': 0.01 'AmbrosM' ,'MT'\n# 0.9708 2.6737 175 opt2 vary all params\n# 0.9708 2.6737 176 opt2 vary all params except Gam\n# 0.9696 2.6713 177 tsvd50 opt2 vary all params --------------------------------- That is quite good LB better than 0.610 above\n# 0.9683 2.6697 178 tsvd100 opt2 vary all params, C-restr --------------------------------- That is quite good LB better than 0.610 above\n# \n# \n# Switch to mode_optimize = 'separate_params_for_each_model(target)'\n# Compare with: 0.61\t\t0.9721\t2.6752\tSVR-lin\tTE-i-th-only\ttsvd25\t145\tOptAm one-for-all lin-SVR tsvd25 TE-ithBoth # 9h \n# 0.969 2.666 179 tsvd30 vary: 'C': [0.01,0.1,0.001 ] 1h13m\n# 0.9684 2.649 180 tsvd30 vary: extend around 0.01 #2h41m --------------------------------- That is quite good LB better than 0.610 above\n# 0.9669 2.6458 181 tsvd50 vary: extend around 0.01 #6h --------------------------------- That is quite good LB better than 0.610 above\n# Compare to LB0608,with randomization or priors: 0607, 0.9661\t2.6165 202SVR-l,tsvd50,op2-all-tgs,TE-ith,C,Smooth,Rnd1\n\n\n\n# Compare to: 0.619\t\t0.9687\t2.7611\tLSVR\tTE-i-th-only\ttsvd25\t83\tLVSR TE-i-th-tsvd optimized2CV both C and smoothing\n# 134-142 - only drug\n# 134 Opt2CV one-for-all linear-SVR tsvd30 TE-ith Drug # 2h10m\n# 0.9725 3.0057 135 Opt2CV one-for-all rbf-SVR tsvd25 TE-ith Drug # 16 min\n# 0.9962 3.2018 136 Opt2CV one-for-all sigm-SVR tsvd25 TE-ith Drug # 18min\n# 0.9963 3.1992 137 OptMT one-for-all sigm-SVR tsvd25 TE-ith Drug # 10min\n# 1.022617 2.7081 138 OptMT one-for-all rbf-SVR tsvd25 TE-ith Drug # 15min\n# 139 OptMT one-for-all lin-SVR tsvd25 TE-ith Drug # 12h cancel\n# 140 OptAm one-for-all lin-SVR tsvd25 TE-ith Drug # 2h6m \n# 0.9680 3.0118 141 OptAm one-for-all rbf-SVR tsvd25 TE-ith Drug 13min\n# 0.9962 3.2018 142 OptAm one-for-all sig-SVR tsvd25 TE-ith Drug 8min\n\n# Include 'cell_type',\n# 0.996 3.2008 143 OptAm one-for-all sig-SVR tsvd25 TE-ithBoth # 16min\n# 0.9692 3.0685 144 OptAm one-for-all rbf-SVR tsvd25 TE-ithBoth # 13min\n# LB0.610 0.9721 2.6752 145 OptAm one-for-all lin-SVR tsvd25 TE-ithBoth # 9h \n# 146 OptMT one-for-all lin-SVR tsvd25 TE-ithBoth # Cancel 12h \n# 0.993851 2.9135 147 OptMT one-for-all rbf-SVR tsvd25 TE-ithBoth # 8min\n# 0.9959 3.1932 148 OptMT one-for-all sig-SVR tsvd25 TE-ithBoth # 13min\n# 0.996 3.2005 149 Opt2 one-for-all sig-SVR tsvd25 TE-ithBoth # 20min\n# 0.9692 3.0685 150 Opt2 one-for-all rbf-SVR tsvd25 TE-ithBoth # 17min\n# LB0.615 0.9812 2.6681 151 Opt2 one-for-all lin-SVR tsvd25 TE-ithBoth # 7h18m\n\n# ------------------------------------------------------------------------------------------------\n# KRR\n# ------------------------------------------------------------------------------------------------\n# KRR&onehot\n# Benchmarks:\n# Ridge: 0.615\t0.604\t1.0266\t2.6698\tRidge\tonehot drug\ttsvd30\t12,17\tReproduce MT - onehot (drug only), tsvd30, Ridge0.1, fit_intercept = False\n# Ridge: 0.613\t0.605\t0.9978\t2.6603\tRidge\tonehot drug\ttsvd30\t20\tImproved MT - fit_intercept = True, alpha = 1 , onehot (drug only) , tsvd30, Ridge\t\n# 1.0227 2.6714 208 oheDrg,KRR-l got alpha=0.1 [1.0227 2.6714]\n# 1.0227 2.6714 'alpha': 0.1, 209oheDrg,KRR-l,tsvd25, Opt2-1prm-4-all,AlphaExt 'alpha': [0.01,0.05,0.08,0.09,.1,1.1,1.2,1.3,1.5, 1,10 ] \n# LB0.614 - reasonable between benchmarks 1.003 2.6579 'alpha': 0.01 210oheDrg,KRR-rbf,tsvd25, Opt2-1prm-4-all,AlphaExt \n# 1.0227 2.6714 'alpha': 0.1, 211 oheDrg,KRR-cos,tsvd25, Opt2-1prm-4-all,AlphaExt \n# 1.001 2.7559 'alpha': 0.01, 212oheDrg,KRR-sig,tsvd25, Opt2-1prm-4-all,AlphaExt \n# 1.0131 2.6601 'alpha': 0.001, 213oheDrg,KRR-sig,tsvd25, Opt2-1prm-4-all,AlphaE+\n# Conclusion: Seems rbf gives better results balancing MT / AmbrosM, so we start looking on it with more details: \n# 1.003 2.6579 'alpha': 0.01 214oheDrg,KRR-rbf,tsvd25, Opt2-1prm-4-all,AlphaE+\n# 1.003 2.6579 'alpha': 0.01, '215oheDrg,KRR-rbf,tsvd25, Opt2-1prm-4-all,AlphaE++ [0.00001, 0.0001,0.001,0.005,0.01,0.02,0.03,0.04,0.05,0.08]\n# 1.003 2.6579 216oheDrg,KRR-rbf,tsvd25, Opt2-1prm-4-all,Al,Ga\n# 1.003 2.6579 217oheDrg,KRR-rbf,tsvd25, Opt2-1prm-4-all,Gam - so Gamma since not improving \n# Check by submit that we get reasonable result LB0.614 - similar to benchmarks \n# Proceed to diffr-prm-for-diffr-targets-optimization\n# 0.9963 2.6277 218oheDrg,KRR-rbf,tsvd25,Opt2-all-tgs,Al # 1hour\n# 0.9954 2.6246 219tsvd50,oheDrg,KRR-rbf,Opt2-all-tgs,Al # 3h17m\n# 0.9953 2.6259 220tsvd30,oheDrg,KRR-rbf,Opt2-all-tgs,Al #1h20min \n# 0.9946 2.6152 221tsvd30,oheDrg,KRR-rbf,Opt2-all-tgs,Al+ [0.01,0.009,0.008,0.005,0.011,0.015,0.02, 0.001,0.0001,0.1,1,10] # 3h9m\n# LB0.615 0.9946 2.6139 222tsvd50,oheDrg,KRR-rbf,Opt2-all-tgs,Al+ # 6h\n# Conclusion - again problem - CV grows, but LB a bit down . The same we have seen on Ridge&Ohe, we see for KernelRidge&Ohe\n\n#dict_prms_lists = { 'alpha': [1e3,1e4,1e5,1e6,1e7,5e7,1e8,1e9], 'smoothing0': [1e1, 0,1e4, 1e2] , 'smoothing1': [1e1, 1e4, 1e2,0] }# , 'gamma':[None, 1, 0.001,1000], } \n# 1.255 2.8559 188 KRR-lin op2-all-targ, Alpha,tsvd25,TE-ith # 1h37min\n# 1.2537 2.8533 189 KRR-lin op2-all-targ, Alpha,tsvd50,TE-ith # 4.5h\n# 1.2832 2.8452 190 KRR-rbf op2-1prm, Alpha 1e3-1e9,tsvd25,TE-ith\n# 1.2832 2.8452 191KRR-rbf op2-1prm,Gam,Alpha1e3-1e9,tsvd25,TE-ith\n# 1.0051 3.2309 192KRR-rbf op2-all-tg,Alpha1e3-1e9,tsvd25,TE-ith # 3h8m\n# 1.0261 2.6547 193KRR-l,Opt2-all-tg,Alp, Smooth, TE-ith #5h12m\n# 1.0261 2.6547 194KRR-rbf,Opt2-all-tg,Alp,Smooth, TE-ith# 6h36\n# fail - no kernel chi2 195KRR-chi2,Opt2-all-tg,Alp,Smooth,TE-ith\n# 0.9938 3.1730 196KRR-sig,Opt2-all-tg,Alp,Smooth,TE-ith # 7.5h\n# 0.9938 3.1730 197KRR-cos,Opt2-all-tg,Alp,Smooth,TE-ith # 6h\n# 0.9943 3.2150 198KRR-Lap,Opt2-all-tg,Alp,Smooth,TE-ith # 5h37min\n#\n# Results from optimization above are not promosing, let us try Ambr,MT only optmizations, and allow smaller Alpha since opt - border\n# 0.988 3.1778 203KRR-cos,OptAmb-all-tgs,Alp,Smooth,TE-ith,tsvd25\n# 1.0227 3.0790 204KRR-cos,OptMT-all-tgs,Alp,Smooth,TE-ith,tsvd25\n# 1.0692 2.5350 205KRR-l,OptMT-all-tgs,Alp,Smooth,TE-ith,tsvd25\n# 1.0251 2.6103 206KRR-l,OptAm-all-tgs,Alp,Smooth,TE-ith,tsvd25\n# 1.0458 2.5556 207KRR-l,Opt2-all-tgs,Alp,Smooth,TE-ith,tsvd25\n# Result 205: 1.0251 2.6103 looks better than 0.615 v12,17 - may be try it later \n#0.615\t0.604\t1.0266\t2.6698\tRidge\tonehot drug\ttsvd30\t12,17\tReproduce MT - onehot (drug only), tsvd30, Ridge0.1, fit_intercept = False\n#0.615\t\t1.0234\t2.6680\tRidge\tonehot drug\ttsvd30\t19\tMT with fit_intercept = True - CV is better but LB is the same \n\n# For KRR-lin results seems do not depend (almost) on component number - probbably that are some means - not so good \n# KRR: for linear KRR gamma does not affect it, as well as coef0 - so alpha is basically the only param \n# alpha 1000.0 Valid mrrmse: 1.3482 r2: -0.5305 time: 23.3 count 1 best: [inf inf] CV_scheme AmbrosM\n# alpha 1000.0 Valid mrrmse: 2.8924 r2: -0.3264 time: 42.0 count 1 best: [inf inf] CV_scheme MT\n# alpha 10000.0 Valid mrrmse: 1.3231 r2: -0.4454 time: 62.4 count 2 best: [1.3482 2.8924] CV_scheme AmbrosM\n# alpha 10000.0 Valid mrrmse: 2.8472 r2: -0.1933 time: 79.1 count 2 best: [1.3482 2.8924] CV_scheme MT\n# alpha 100000.0 Valid mrrmse: 1.2832 r2: -0.3605 time: 101.8 count 3 best: [1.3231 2.8472] CV_scheme AmbrosM\n# Best MT alpha 100000.0 Valid mrrmse: 2.8452 r2: -0.0203 time: 118.6 count 3 best: [1.3231 2.8472] CV_scheme MT\n# alpha 1000000.0 Valid mrrmse: 1.2142 r2: -0.2059 time: 138.8 count 4 best: [1.2832 2.8452] CV_scheme AmbrosM\n# alpha 1000000.0 Valid mrrmse: 2.9924 r2: -0.0266 time: 156.6 count 4 best: [1.2832 2.8452] CV_scheme MT\n# alpha 10000000.0 Valid mrrmse: 1.0192 r2: -0.0425 time: 176.4 count 5 best: [1.2832 2.8452] CV_scheme AmbrosM\n# alpha 10000000.0 Valid mrrmse: 3.114 r2: -0.1344 time: 192.6 count 5 best: [1.2832 2.8452] CV_scheme MT\n# alpha 20000000.0 Valid mrrmse: 0.9964 r2: -0.0389 time: 212.9 count 6 best: [1.2832 2.8452] CV_scheme AmbrosM\n# alpha 20000000.0 Valid mrrmse: 3.1521 r2: -0.1696 time: 229.7 count 6 best: [1.2832 2.8452] CV_scheme MT\n# Best AmbrosM alpha 50000000.0 Valid mrrmse: 0.9945 r2: -0.0508 time: 251.8 count 7 best: [1.2832 2.8452] CV_scheme AmbrosM\n# alpha 50000000.0 Valid mrrmse: 3.1919 r2: -0.2014 time: 268.1 count 7 best: [1.2832 2.8452] CV_scheme MT\n# alpha 80000000.0 Valid mrrmse: 0.9971 r2: -0.0571 time: 289.8 count 8 best: [1.2832 2.8452] CV_scheme AmbrosM\n# alpha 80000000.0 Valid mrrmse: 3.2053 r2: -0.2113 time: 306.5 count 8 best: [1.2832 2.8452] CV_scheme MT\n \n\n\nif flag_opt_loc:\n \n if mode_optimize == 'separate_params_for_each_model(target)':\n list_i_target = list( range(n_components) )\n elif mode_optimize == 'same_params_for_all_models(targets)':\n list_i_target = [0] # Fake list\n\n t00 = time.time() ; i_total_count = 0\n # To suppress all warnings\n warnings.filterwarnings(\"ignore\") \n \n list_best_scores_afer_prm_optimization = []\n for i_round in range(n_rounds): # rounds to repeat the entire search \n for i_prm in range( len( dict_prms_lists )) : # Choose what param will be tuned\n key_loc = list(dict_prms_lists.keys())[i_prm]\n list_prm_values = dict_prms_lists[key_loc] \n for i_target in list_i_target: # choose target --- params for model for THAT target will be tuned . If one model - it is fake list [0],\n #if (i_round == 0) and (i_target <50): continue \n #if (i_round == 1) and (i_target >50): continue \n \n \n if mode_optimize == 'separate_params_for_each_model(target)':\n prm_before_search = main_config_model_feature_etc['i_target_cfg'][i_target][key_loc] \n elif mode_optimize == 'same_params_for_all_models(targets)':\n prm_before_search = main_config_model_feature_etc[key_loc] \n \n \n # Initial prm can be not in the searched list, or skip conditions can lead to that. That is why we need that flag and saving prm_before_search\n flag_achieved_better_or_same_result = False\n\n for prm_value in list_prm_values: # Grid the list of the params \n i_total_count += 1\n # One can set special conditions to adjust for particaular targets \n #if (i_target == 4) and (prm < 1e2): continue\n #if (i_target != 4) and (prm > 1e2): continue\n\n \n if mode_optimize == 'separate_params_for_each_model(target)':\n main_config_model_feature_etc['i_target_cfg'][i_target][key_loc] = prm_value\n elif mode_optimize == 'same_params_for_all_models(targets)':\n main_config_model_feature_etc[key_loc] = prm_value\n \n \n flag_accept_prm = True; dict_res_loc = {}; list_flag_accept_prm =[]\n for CV_scheme in list_CV_scheme:# = ['AmbrosM','MT']\n Y_oof_pred, Y_submit_pred, mrrmse_list, r2_list, dict_optional_res = go_modeling_separate_model_for_each_target( main_config_model_feature_etc, \n CV_scheme = CV_scheme, verbose = 0) \n if verbose >= 100:\n print(key_loc,prm_value, 'Valid mrrmse:', np.round(np.mean( mrrmse_list ),4) , 'r2:', np.round(np.mean(r2_list ),4),\n 'time:', np.round(time.time() - t00,1), 'count',i_total_count, 'best:', np.round(best_res,4), 'CV_scheme', CV_scheme ) \n lres = np.mean(mrrmse_list)\n dict_res_loc[CV_scheme] = lres\n list_flag_accept_prm.append( lres <= best_res)\n if criteria_accept_params == 'all_cv_are_better': # Eearly stopping\n if lres > dict_best_res[CV_scheme]:\n flag_accept_prm = False\n break\n \n if flag_accept_prm:\n dict_best_res = dict_res_loc\n best_prm = prm_value\n flag_achieved_better_or_same_result = True\n best_res = [t for t in dict_best_res.values() ]\n \n\n if flag_achieved_better_or_same_result:\n if mode_optimize == 'separate_params_for_each_model(target)':\n main_config_model_feature_etc['i_target_cfg'][i_target][key_loc] = best_prm\n elif mode_optimize == 'same_params_for_all_models(targets)':\n main_config_model_feature_etc[key_loc] = best_prm\n else:\n if mode_optimize == 'separate_params_for_each_model(target)':\n main_config_model_feature_etc['i_target_cfg'][i_target][key_loc] = prm_before_search\n elif mode_optimize == 'same_params_for_all_models(targets)':\n main_config_model_feature_etc[key_loc] = prm_before_search\n\n list_best_scores_afer_prm_optimization.append( np.round(best_res,6) )\n if verbose >=10:\n print('Finished loop for param',key_loc, 'best value:', best_prm, 'best score', np.round(best_res,6), 'Round',i_round,'i_target' , i_target, 'time:', np.round(time.time() - t00,1), 'count',i_total_count )\n print()\n print(i_round, 'round finsihed')\n print('Round',i_round,'i_target' , i_target, 'prm', key_loc, 'best_prm', best_prm, 'best score', np.round(best_res,6), 'time:', np.round(time.time() - t00,1), 'count',i_total_count )\n print( main_config_model_feature_etc) \n print()\n\n # To re-enable warnings\n warnings.filterwarnings(\"default\") # or \"once\" to show the warning only once\n\n print() \n print(\"naive optimizer finished.\", mode_optimize )\n print( 'best score', np.round(best_res,6), np.round(best_res,4) )\n print( main_config_model_feature_etc) \n print( 'best (mrrmse) score', np.round(best_res,6), np.round(best_res,4) )\n #display(main_config_model_feature_etc )\n \n df_tmp = pd.DataFrame( list_best_scores_afer_prm_optimization ) \n df_tmp.to_csv('report_best_scores_afer_prm_optimization.csv')\n for col in df_tmp.columns:\n plt.figure(figsize = (15,4) )\n plt.plot( df_tmp[col], '*-')\n #plt.plot( list_best_scores_afer_prm_optimization , '*-' )\n plt.title('Best score change after each prm change',fontsize = 20 )\n plt.show()\n ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.762251Z","iopub.execute_input":"2023-10-24T09:55:52.762602Z","iopub.status.idle":"2023-10-24T09:55:52.854066Z","shell.execute_reply.started":"2023-10-24T09:55:52.762572Z","shell.execute_reply":"2023-10-24T09:55:52.852792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nif flag_opt_loc:\n\n for prm_name in dict_prms_lists.keys():\n list_ix = []\n list_val = []\n for i_target in range(500):\n if 'i_target_cfg' in main_config_model_feature_etc.keys():\n if i_target in main_config_model_feature_etc['i_target_cfg'].keys():\n list_ix.append(i_target); list_val.append( main_config_model_feature_etc['i_target_cfg'][i_target][prm_name] )\n plt.figure(figsize = (15,4))\n if (prm_name in ['alpha','C']) or ( 'smoothing' in prm_name):\n plt.plot(list_ix,np.log10( list_val ) , '*-')\n plt.title(' log10 ' + prm_name ,fontsize = 20 )\n else:\n plt.plot(list_ix, ( list_val ) , '*-')\n plt.title(prm_name ,fontsize = 20 )\n plt.xlabel('target index',fontsize = 20)\n plt.grid()\n plt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.855570Z","iopub.execute_input":"2023-10-24T09:55:52.855907Z","iopub.status.idle":"2023-10-24T09:55:52.866808Z","shell.execute_reply.started":"2023-10-24T09:55:52.855878Z","shell.execute_reply":"2023-10-24T09:55:52.865684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if flag_opt_loc:\n print( 'best (mrrmse) score', np.round(best_res,4) , np.round(best_res,6))","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.868185Z","iopub.execute_input":"2023-10-24T09:55:52.868508Z","iopub.status.idle":"2023-10-24T09:55:52.881920Z","shell.execute_reply.started":"2023-10-24T09:55:52.868480Z","shell.execute_reply":"2023-10-24T09:55:52.881003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare,save submissions, oof,etc for found best params (Naive optimization)","metadata":{}},{"cell_type":"code","source":"if flag_opt_loc:\n print( 'best (mrrmse) score', np.round(best_res,4) , np.round(best_res,6))","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.883632Z","iopub.execute_input":"2023-10-24T09:55:52.884751Z","iopub.status.idle":"2023-10-24T09:55:52.892974Z","shell.execute_reply.started":"2023-10-24T09:55:52.884706Z","shell.execute_reply":"2023-10-24T09:55:52.892120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if flag_opt_loc:\n do_modeling_submit_prepare_save_etc( main_config_model_feature_etc ) # , subdirectory_path_postfix = '' ","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.894170Z","iopub.execute_input":"2023-10-24T09:55:52.894860Z","iopub.status.idle":"2023-10-24T09:55:52.904218Z","shell.execute_reply.started":"2023-10-24T09:55:52.894814Z","shell.execute_reply":"2023-10-24T09:55:52.903106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Blend selected submits","metadata":{}},{"cell_type":"code","source":"%%time \n#group by drug and take the mean\ndf_tmp = df_de_train.iloc[:, [1] + list(range(5, df_de_train.shape[1]))] # Take only numeric columns and \"sm_name\"\ndf_aggr = df_tmp.groupby('sm_name').mean().reset_index()\nprint(df_aggr.shape)\ndf_submit_aggr_compound = pd.merge( df_id_map, df_aggr, on='sm_name', how = 'left' ).sort_values('id').drop(columns = ['cell_type', 'sm_name']).set_index('id')\nprint(df_submit_aggr_compound.shape)\ndisplay(df_submit_aggr_compound.head(3))\n\nprint( )\n\ndf_tmp = df_de_train.iloc[:, [0] + list(range(5, df_de_train.shape[1]))] # Take only numeric columns and \"cell_type\"\ndf_aggr = df_tmp.groupby('cell_type').mean().reset_index()\nprint(df_aggr.shape)\ndf_submit_aggr_cell_type = pd.merge( df_id_map, df_aggr, on='cell_type', how = 'left' ).sort_values('id').drop(columns = ['cell_type', 'sm_name']).set_index('id')\nprint(df_submit_aggr_cell_type.shape)\ndisplay(df_submit_aggr_cell_type.head(3))","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:52.905615Z","iopub.execute_input":"2023-10-24T09:55:52.906457Z","iopub.status.idle":"2023-10-24T09:55:53.551827Z","shell.execute_reply.started":"2023-10-24T09:55:52.906409Z","shell.execute_reply":"2023-10-24T09:55:53.550626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndf_submit_aggr_compound.to_csv('df_submit_aggr_compound.csv')\ndf_submit_aggr_cell_type.to_csv('df_submit_aggr_cell_type.csv')","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:55:53.555568Z","iopub.execute_input":"2023-10-24T09:55:53.555909Z","iopub.status.idle":"2023-10-24T09:56:19.368066Z","shell.execute_reply.started":"2023-10-24T09:55:53.555880Z","shell.execute_reply":"2023-10-24T09:56:19.366266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfn = 'submission_Random_20_42_blend_with_priors_0.45_0.1.csv'\nfn = 'submission_simple.csv'\n#/kaggle/input/open-problems-single-cell-perturbations-submitsetc/LB0608_tsvd25_CATB_target_enc_i_th_target_BothEncoded_10-18-13-29_V254/submission_Random_20_42.csv\nfn = 'submission_Random_20_42.csv'\n\nimport os\ni0 = 0\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n #if ('0605' in dirname) or ( '0604' in dirname ):\n if ('0608' in dirname):# or ( '0613' in dirname ) or ( '0615' in dirname ):\n if filename == fn:\n full_fn = os.path.join(dirname, filename)\n print(full_fn)\n if i0 == 0:\n df_submit = pd.read_csv(full_fn, index_col = 0)\n else:\n df_submit += pd.read_csv(full_fn, index_col = 0)\n i0 += 1\nprint(i0) \ndf_submit /= i0\ndisplay( df_submit.head(3) )\ndf_submit_1 = df_submit.copy()\n\n\ni0 = 0\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n #if ('0605' in dirname) or ( '0604' in dirname ):\n if ('0610' in dirname) or ( '0611' in dirname ) or ( '0612' in dirname ) or ( '0612' in dirname ) or ( '0614' in dirname ):\n if filename == fn:\n full_fn = os.path.join(dirname, filename)\n print(full_fn)\n if i0 == 0:\n df_submit = pd.read_csv(full_fn, index_col = 0)\n else:\n df_submit += pd.read_csv(full_fn, index_col = 0)\n i0 += 1\nprint(i0) \ndf_submit /= i0\ndisplay( df_submit.head(3) )\ndf_submit_2 = df_submit.copy()\n\ni0 = 0\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n #if ('0605' in dirname) or ( '0604' in dirname ):\n if ('0615' in dirname):# or ( '0613' in dirname ) or ( '0615' in dirname ):\n if filename == fn:\n full_fn = os.path.join(dirname, filename)\n print(full_fn)\n if i0 == 0:\n df_submit = pd.read_csv(full_fn, index_col = 0)\n else:\n df_submit += pd.read_csv(full_fn, index_col = 0)\n i0 += 1\nprint(i0) \ndf_submit /= i0\ndisplay( df_submit.head(3) )\ndf_submit_3 = df_submit.copy()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:56:19.373699Z","iopub.execute_input":"2023-10-24T09:56:19.374205Z","iopub.status.idle":"2023-10-24T09:57:53.642246Z","shell.execute_reply.started":"2023-10-24T09:56:19.374159Z","shell.execute_reply":"2023-10-24T09:57:53.641065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submit_ensemble1 = 0.8*df_submit_1+0.1*df_submit_2 + 0.1*df_submit_3\ndf_submit = 0.7*(df_submit_ensemble1) + 0.2 * df_submit_aggr_compound + 0.1 * df_submit_aggr_cell_type","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:53.644057Z","iopub.execute_input":"2023-10-24T09:57:53.644481Z","iopub.status.idle":"2023-10-24T09:57:53.828210Z","shell.execute_reply.started":"2023-10-24T09:57:53.644441Z","shell.execute_reply":"2023-10-24T09:57:53.826939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"flag_save_all_submits = False","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:53.830122Z","iopub.execute_input":"2023-10-24T09:57:53.830564Z","iopub.status.idle":"2023-10-24T09:57:53.837640Z","shell.execute_reply.started":"2023-10-24T09:57:53.830521Z","shell.execute_reply":"2023-10-24T09:57:53.836083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nif flag_save_all_submits:\n # LB0602\n df_submit_ensemble1.to_csv('submission_ensemble_08_0608_01_0610_614_01_615.csv')\n display(df_submit_ensemble1)\n\n # LB0606\n df_submit.to_csv('submission_ensemble_08_0608_01_0610_614_01_615_02aggCompound_01_aggCellType.csv')\n display(df_submit)","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:53.839484Z","iopub.execute_input":"2023-10-24T09:57:53.839806Z","iopub.status.idle":"2023-10-24T09:57:53.853557Z","shell.execute_reply.started":"2023-10-24T09:57:53.839778Z","shell.execute_reply":"2023-10-24T09:57:53.852377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# LB0602\ndf_submit_ensemble2 = 0.7*df_submit_1+0.2*df_submit_2 + 0.1*df_submit_3\ndisplay(df_submit_ensemble2.head(5) )\nif flag_save_all_submits:\n df_submit_ensemble2.to_csv('submission_ensemble_07_0608_02_0610_614_01_615.csv')\n\n\n# LB0601\ndf_submit_ensemble3 = 0.7*df_submit_1+0.1*df_submit_2 + 0.2*df_submit_3\ndisplay(df_submit_ensemble2.head(5) )\nif flag_save_all_submits:\n df_submit_ensemble3.to_csv('submission_ensemble_07_0608_01_0610_614_02_615.csv')\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:53.855024Z","iopub.execute_input":"2023-10-24T09:57:53.855368Z","iopub.status.idle":"2023-10-24T09:57:54.093695Z","shell.execute_reply.started":"2023-10-24T09:57:53.855340Z","shell.execute_reply":"2023-10-24T09:57:54.092507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# LB0.601\ndf_submit_ensemble4 = 0.6*df_submit_1+0.1*df_submit_2 + 0.3*df_submit_3\ndisplay(df_submit_ensemble4.head(5) )\nif flag_save_all_submits:\n df_submit_ensemble4.to_csv('submission_ensemble_06_0608_01_0610_614_03_615.csv')\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:54.095223Z","iopub.execute_input":"2023-10-24T09:57:54.095651Z","iopub.status.idle":"2023-10-24T09:57:54.215645Z","shell.execute_reply.started":"2023-10-24T09:57:54.095611Z","shell.execute_reply":"2023-10-24T09:57:54.214398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nif flag_save_all_submits:\n # LB0.601 \n df_submit_ensemble5 = 0.6*df_submit_1+0.0*df_submit_2 + 0.4*df_submit_3\n df_submit_ensemble5.to_csv('submission_ensemble_06_0608_00_0610_614_04_615.csv')\n display(df_submit_ensemble5.head(5) )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:54.217145Z","iopub.execute_input":"2023-10-24T09:57:54.217561Z","iopub.status.idle":"2023-10-24T09:57:54.225367Z","shell.execute_reply.started":"2023-10-24T09:57:54.217518Z","shell.execute_reply":"2023-10-24T09:57:54.224099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# LB0.601\ndf_submit_ensemble6 = 0.5*df_submit_1+0.0*df_submit_2 + 0.5*df_submit_3\ndisplay(df_submit_ensemble6.head(5) )\nif flag_save_all_submits:\n df_submit_ensemble6.to_csv('submission_ensemble_05_0608_00_0610_614_05_615.csv')\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:54.226903Z","iopub.execute_input":"2023-10-24T09:57:54.227305Z","iopub.status.idle":"2023-10-24T09:57:54.354259Z","shell.execute_reply.started":"2023-10-24T09:57:54.227272Z","shell.execute_reply":"2023-10-24T09:57:54.353093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# LB 0.602\ndf_submit_ensemble7 = 0.4*df_submit_1+0.0*df_submit_2 + 0.6*df_submit_3\ndisplay(df_submit_ensemble7.head(5) )\nif flag_save_all_submits:\n df_submit_ensemble7.to_csv('submission_ensemble_04_0608_00_0610_614_06_615.csv')\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:54.355623Z","iopub.execute_input":"2023-10-24T09:57:54.355968Z","iopub.status.idle":"2023-10-24T09:57:54.478112Z","shell.execute_reply.started":"2023-10-24T09:57:54.355939Z","shell.execute_reply":"2023-10-24T09:57:54.477194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Here is some submission which incorporates 0.608 submissions prodeced in that notebook\n# We will blend it with some other submission from here \n# fn = '/kaggle/input/open-problems-single-cell-perturbations-submitsetc/LB0589_submission_top1.csv'\nfn = '/kaggle/input/open-problems-single-cell-perturbations-submitsetc/LB0589_submission_combined09.csv'\n\ndf_submit_others = pd.read_csv(fn, index_col = 0)\ndf_submit_others","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:57:54.479604Z","iopub.execute_input":"2023-10-24T09:57:54.479920Z","iopub.status.idle":"2023-10-24T09:58:00.794512Z","shell.execute_reply.started":"2023-10-24T09:57:54.479893Z","shell.execute_reply":"2023-10-24T09:58:00.793372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndf_submit_combined = 0.92*df_submit_others + 0.07*df_submit_3 + 0.01*df_submit_2\ndf_submit_combined.to_csv('submission_combined_05.csv')\ndisplay(df_submit_combined.head(5) )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:58:00.796286Z","iopub.execute_input":"2023-10-24T09:58:00.797416Z","iopub.status.idle":"2023-10-24T09:58:14.213692Z","shell.execute_reply.started":"2023-10-24T09:58:00.797371Z","shell.execute_reply":"2023-10-24T09:58:14.212513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analysis of submits ","metadata":{}},{"cell_type":"code","source":"%%time\nv = df_submit_ensemble2.abs().mean(axis = 0).sort_values()\nplt.plot(v.values,'*')\nv = df_submit_ensemble4.abs().mean(axis = 0).sort_values()\nplt.plot(v.values,'*')\nplt.show()\nv.describe()\nl = []\nfor k in range(df_submit_ensemble4.shape[1]):\n c = np.corrcoef(df_submit_ensemble4.iloc[:,k], df_submit_ensemble2.iloc[:,k], )[0,1]\n l.append(c)\nplt.plot(np.sort(l))\nplt.show()\nprint( pd.Series(l).describe() )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:58:14.215228Z","iopub.execute_input":"2023-10-24T09:58:14.215663Z","iopub.status.idle":"2023-10-24T09:58:21.143096Z","shell.execute_reply.started":"2023-10-24T09:58:14.215624Z","shell.execute_reply":"2023-10-24T09:58:21.141947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nl = []\nfor k in range(df_submit_ensemble4.shape[1]):\n c = np.corrcoef(df_submit_1.iloc[:,k], df_submit_2.iloc[:,k], )[0,1]\n l.append(c)\nplt.plot(np.sort(l))\nplt.show()\nprint( pd.Series(l).describe() )\n\nl = []\nfor k in range(df_submit_ensemble4.shape[1]):\n c = np.corrcoef(df_submit_1.iloc[:,k], df_submit_3.iloc[:,k], )[0,1]\n l.append(c)\nplt.plot(np.sort(l))\nplt.show()\nprint( pd.Series(l).describe() )\nl = []\nfor k in range(df_submit_ensemble4.shape[1]):\n c = np.corrcoef(df_submit_2.iloc[:,k], df_submit_3.iloc[:,k], )[0,1]\n l.append(c)\nplt.plot(np.sort(l))\nplt.show()\nprint( pd.Series(l).describe() )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:58:21.144652Z","iopub.execute_input":"2023-10-24T09:58:21.145626Z","iopub.status.idle":"2023-10-24T09:58:40.736902Z","shell.execute_reply.started":"2023-10-24T09:58:21.145589Z","shell.execute_reply":"2023-10-24T09:58:40.734875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final timing","metadata":{}},{"cell_type":"code","source":"print('%.1f seconds passed total '%(time.time()-t0start) )\nprint('%.1f minutes passed total '%( (time.time()-t0start)/60) )\nprint('%.2f hours passed total '%( (time.time()-t0start)/3600) )","metadata":{"execution":{"iopub.status.busy":"2023-10-24T09:58:40.738474Z","iopub.execute_input":"2023-10-24T09:58:40.739223Z","iopub.status.idle":"2023-10-24T09:58:40.746403Z","shell.execute_reply.started":"2023-10-24T09:58:40.739187Z","shell.execute_reply":"2023-10-24T09:58:40.745268Z"},"trusted":true},"execution_count":null,"outputs":[]}]}