{"cells":[{"metadata":{},"cell_type":"markdown","source":"### To pick samples for training and validation use the Adversarial Validation, the idea successfully used often across Kaggle (e.g. [Quora Adversarial Validation](https://www.kaggle.com/tunguz/quora-adversarial-validation) and [Adversarial validation](https://www.kaggle.com/konradb/adversarial-validation)) - the idea of training a model that predicts by how much a given sample is different from the test set.\n\n- The public notebook used as a base: Xhlulu @xhlulu - [Jigsaw TPU: XLM-Roberta](https://www.kaggle.com/xhlulu/jigsaw-tpu-xlm-roberta)\n\n- Roberta XML-Base Model - This notebook run both on TPU (for x minutes - this version run on TPU) and Kaggle GPU (for up to 7 hours)\n\n\n### The workflow:\n- Label test data set as 1 and the training set as 0. The training set consists of the randomly sampled Michael Kazachok's [translations](https://www.kaggle.com/miklgr500/jigsaw-train-multilingual-coments-google-api) to 6 languages of train1 (toxic-comment-calssification-training-data), of train2 (unbiased-bias-training-data) and val_8k (validation.csv) that had been encoded in a separate notebook without use of GPU/TPU;\n- Train classification task with pretrained XLM-Roberta-base model (make sure the same sample's translations are in the same fold);\n- Save the model and the oof predictions of the train1, train2, val_8k;\n- Out of this notebook: predict all train1 and train2 samples using  this model (except for those rows used for training)\n- Use those predictions (and the oof predictions of those sampled used for training) to rank samples as to how much they are similar to test set.\n- use threshold 'probability' to sample the desired number of samples for training and adding to validation set\n\n### Observations:\n\n- most of the translations are easily separated from test set - validation set auc = 0.98 => they are very different from test set.\n- there are about only 4k samples in train1 that has predictions ('probability' of being from test set according to classifier) larger than 0.4 which can be added to validation set; most of the samples are well below 0.001\n- selecting samples this way, at least, does not make the performance worse than lucky picks of random sampling\n\n\n","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os, time, gc\n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\n\nimport transformers\nfrom transformers import TFAutoModel, AutoTokenizer\nfrom tqdm.notebook import tqdm\n\nfrom sklearn.model_selection import GroupKFold\n\nosj = os.path.join; osdir = os.listdir","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Configs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Detect hardware, return appropriate distribution strategy\ndef tpu_init():\n    try:\n        # TPU detection. No parameters necessary if TPU_NAME environment variable is\n        # set: this is always the case on Kaggle.\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        tpu = None\n\n    if tpu:\n        tf.config.experimental_connect_to_cluster(tpu)\n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        strategy = tf.distribute.experimental.TPUStrategy(tpu)\n    else:\n        # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n        strategy = tf.distribute.get_strategy()\n\n    print(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\n    return strategy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, max_len=512):\n    \"\"\"\n    https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n    \"\"\"\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    \n    cls_token = sequence_output[:, 0, :]\n    \n    out = Dense(1, activation='sigmoid')(cls_token)\n        \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy',\n                                      metrics=['accuracy', tf.keras.metrics.AUC()])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def tpu_bs(BATCH_SIZE_MULTIPLIER):\n    strategy = tpu_init()\n    bs = BATCH_SIZE_MULTIPLIER * strategy.num_replicas_in_sync\n    \n    return strategy, bs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"debug = False  # True # False\nn_rows = 100_000_000 if not debug else 16*8*2\n\nn_splits = 4\n# load models of folds 0,1,4\nfolds_to_train = [0,1]\nfolds_to_save = [0,1]\nseed_num = 1\nseed_model = 2020\n\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Configuration\nepochs = 2\n\nBATCH_SIZE_MULTIPLIER = 24  # 32  # 24  # 16\nMAX_LEN = 192\nMODEL = 'jplu/tf-xlm-roberta-base'\n\nout_path = './'\nassert os.path.exists(out_path)\n\ndatetime_str = time.strftime(\"%d_%m_time_%H_%M\", time.localtime())\n\nt0 = time.time()\n\ndef keras_seed_everything(seed):\n    # import tensorflow as tf\n    # import os\n    tf.random.set_seed(seed)\n    np.random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nkeras_seed_everything(seed_model)\n\nstrategy, bs = tpu_bs(BATCH_SIZE_MULTIPLIER)\n\ntokenizer = AutoTokenizer.from_pretrained(MODEL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_trans = pd.read_csv(\n    '../input/jig-ds-noneng-raw-begin-data-maxl-192/train_trans_raw_begin_enc_rows_1326360_maxl_192.csv', nrows=n_rows)\n\ntrain_trans = train_trans.sample(n=60_000 if not debug else 60, random_state=seed_num)\n\ntrain_trans = train_trans[train_trans['df_name']=='trans']\n\nval_8k = pd.read_csv(\n    '../input/jig-ds-noneng-raw-begin-data-maxl-192/val_8k_raw_begin_enc_nrows_63812_maxl_192.csv', nrows=n_rows)\n\ntest = pd.read_csv(\n    '../input/jig-ds-noneng-raw-begin-data-maxl-192/test_raw_begin_enc_nrows_63812_maxl_192.csv', nrows=n_rows)\n\ntrain2_trans = pd.read_csv('../input/jig-train2-trans-enc-raw-similar-to-test-preds/train2_trans_similar_test_nrows_838658.csv')\ntrain2_trans = train2_trans.sample(n=176_000 if not debug else 70, random_state=seed_num)\ntrain2_trans.drop('val_fold', axis=1, inplace=True)\ntrain2_trans['toxic'] = (train2_trans['toxic']>0.5).astype(int)\n\ntrain2_trans['df_name'] = 'tr2_trans'\nval_8k['df_name']='val_8k'\ntest['df_name'] = 'test'\n\nenc_cols = [col for col in test.columns if col.startswith('enc_')]\ncols_select = ['id','lang','df_name','toxic'] + enc_cols\ntrain_trans = pd.concat([train_trans[cols_select], train2_trans[cols_select]])\ntrain_trans.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_df_stats(df, df_name='df', text_col = 'comment_text'):\n    print(\"=\"*30)\n    print(f\"\\n{df_name}.shape:\", df.shape)\n    if 'toxic' in df.columns:\n        print(f\"\\n{df_name}['toxic'].value_counts:\\n\", df['toxic'].value_counts())\n    if 'lang' in df.columns:\n        print(f\"\\n{df_name}['lang'].value_counts:\\n\", df['lang'].value_counts())\n    if 'target' in df.columns:\n        print(f\"\\n{df_name}['target'].value_counts:\\n\", df['target'].value_counts())\n\n#print_df_stats(osub, 'osub');\nprint_df_stats(train_trans, 'train_trans'); print_df_stats(val_8k,'val_8k'); print_df_stats(test,'test', text_col = 'content') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"enc_cols = [col for col in test.columns if col.startswith('enc_')]\n\ntrain_trans['target']=0\nval_8k['target']=0\n\ntest['target']=1\n\nsel_cols = ['id','lang','df_name','target']+enc_cols\ntrain = pd.concat([train_trans[sel_cols], val_8k[sel_cols], test[sel_cols]])\n\ntrain = train.sample(frac=1, replace=False, random_state=seed_num)\nprint_df_stats(train, 'train')\ndel train_trans, test, val_8k; _=gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nx_train = train[enc_cols].values.astype('int')[:n_rows]\ny_train = train['target'].values.astype('int')[:n_rows]\n\ntrain.drop(enc_cols, axis=1, inplace=True)\ntrain = train[:n_rows]\n#x_test = test[enc_cols].values.astype('int')\n\nprint(\"x_train.shape\", x_train.shape)\n\ntrain['lang'] = train['lang'].astype('category')\ntrain['target'] = train['target'].astype('int')\ntrain.dtypes\n\nt_preproc = time.time()\nprint(f\"Finished preprocessing in {(t_preproc-t0)/60:.2f} min.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def tpu_bs(BATCH_SIZE_MULTIPLIER):\n    strategy = tpu_init()\n    bs = BATCH_SIZE_MULTIPLIER * strategy.num_replicas_in_sync\n    \n    return strategy, bs\n\nstrategy, bs = tpu_bs(BATCH_SIZE_MULTIPLIER)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Divide train to groups: the same 'id' - one group","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"id_to_group = {k: v for (k,v) in zip(train['id'].unique(), range(len(train['id'].unique())))}\ngroups = train['id'].map(id_to_group).values\ndel id_to_group; _=gc.collect()\ntrain['id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train['preds'] = 999.99\nhist1_ls, hist2_ls = [], []\nt0 = time.time()\n\ngkf = GroupKFold(n_splits=n_splits)\n\nfor fold, (train_idx, valid_idx) in enumerate(gkf.split(x_train, y_train, groups)):\n    \n    if not (fold in folds_to_train):\n        continue\n        \n    t1 = time.time()\n    \n    train_dataset = ( tf.data.Dataset.from_tensor_slices((x_train[train_idx], y_train[train_idx]))\n                                            .repeat().shuffle(2048).batch(bs, drop_remainder=True).prefetch(AUTO)\n                    )\n    valid_dataset = (  tf.data.Dataset.from_tensor_slices((x_train[valid_idx],\n                                                           y_train[valid_idx]))\n                            .batch(bs).prefetch(AUTO) )\n    \n    n_steps = int( max(1, x_train[train_idx].shape[0] // bs) )\n    \n    print(f\"1: Num train samples {len(x_train[train_idx])}, num valid samples = {len(x_train[valid_idx])}\")\n    \n    with strategy.scope():\n        transformer_layer = TFAutoModel.from_pretrained(MODEL)\n        model = build_model(transformer_layer, max_len=MAX_LEN)\n        \n    train_history_1 = model.fit(train_dataset,\n                            steps_per_epoch=n_steps,\n                            validation_data=valid_dataset,\n                            epochs=epochs)\n\n    \n    t2 = time.time()\n    print(f\"\\nTrained fold {fold}, in {(t2-t1)/60:.2f} min.\")\n   \n    \n    hist1_df = pd.DataFrame(train_history_1.history)\n    hist1_ls.append(hist1_df)\n    \n    # save model\n            \n    new_model_filename = f'model_fl{fold}_auc_{hist1_df.iloc[-1,-1]:.5f}_eps_{epochs}.h5'\n    checkpoint_path_fn = os.path.join(out_path, new_model_filename)\n    if fold in folds_to_save:\n        model.save_weights(checkpoint_path_fn)\n        print(f\"Saved fold {fold} weights\")\n    \n    t3 = time.time()\n    \n    train['preds'].iloc[valid_idx]  =  model.predict(valid_dataset, verbose=1).squeeze()\n    train[['id','lang','preds']].to_csv(f'train_preds_after_fold_{fold}.csv', index=False)\n    t4 = time.time()\n    print(f\"Predicted fold {fold} valid_idx - English in {(t4-t3)/60:.2f} min.\")\n    \n    t5 = time.time()\n    print(f\"Predicted fold {fold} valid_idx - English in {(t5-t4)/60:.2f} min.\")\n    \n    print(f\"\\nFOLD {fold}, TOTAL TIME: {(time.time()-t1)/60:.2f} min. \\n =================== END of fold {fold} ====================\\n\")\n    \n    if fold != folds_to_train[-1]:\n        #print(model.summary())\n        del model;  _=gc.collect()\n\n        strategy, bs = tpu_bs(BATCH_SIZE_MULTIPLIER)\n        tf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_df = pd.DataFrame( np.concatenate([h.values for h in hist1_ls]), columns = hist1_ls[0].columns ,\n                      )\nhist_df.to_csv('hist_df.csv', index=False)\nhist_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# leave train with predictions and remove test\ntrain_valid = train.loc[(train['preds']<=1)&(train['target']==0), \n                            ['id', 'lang','df_name','target','preds']]\nprint_df_stats(train_valid, df_name='train_valid')\ntrain_valid.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_valid['df_name'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['df_name']=='test'].sort_values(by='preds').head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_valid['preds'].hist(bins=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"thresholds = [1e-7, 1e-5, 1e-4, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7]\n\nfor prob_thresh in thresholds:\n    train_valid_thresh = train_valid[train_valid['preds']>prob_thresh]\n    #train_valid_thresh.to_csv(f\"train_valid_thresh_{str(prob_thresh).replace('.','_')}_nrows_{train_valid_thresh.shape[0]}.csv\")\n    print(f\"thresh={prob_thresh}: \\tnum samples with preds > thresh : = {train_valid_thresh.shape[0]:,d}\")\n    #      .describe()\n    #print(train_valid_thresh['lang'].train_validue_counts())\n    print(train_valid_thresh['df_name'].value_counts())\n    #print(\"\\nLanguages train_validue_counts in 'train' train_valid_thresh:\")\n    #print(train_valid_thresh.loc[train_valid_thresh['df_name']=='train', 'lang'].train_validue_counts())\n    print(\"=\"*30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save train_valid\ntrain_valid[['id','lang','df_name','target','preds']].to_csv(f'train_valid_folds_{folds_to_train}.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!du -ha ./","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}