{"cells":[{"metadata":{},"cell_type":"markdown","source":"# XLM-R with pseudo labels and knowledge distillation\n\nThis is regular XLM-R trained on 1/6th of the full translated training data as well as on the test data, with pseudo labels blended with the original labels (on the training data). LAMB optimizer and 1 cycle schedule policy.\n\nThe hyper-parameters come from a somewhat extensive search using GCP (Many thanks for google TFRC team and their support!)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Code repo: https://github.com/henrique/jigsaw20","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!git clone https://github.com/henrique/jigsaw20.git\n\nimport sys\nsys.path.append('/kaggle/working/jigsaw20')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom train import train\n\nparams = dict(\n    pooling='first',\n    optimizer='LAMB',\n    batch_size=27,\n    lr=0.000277952,\n    mom_min=0.806579,\n    mom_max=0.922184,\n    div_factor=55.477,\n    final_div_factor=1123.49,\n    weight_decay=7.72285e-06,\n    dropout=0.4,\n    loss_fn='bce',\n    label_smoothing=0.0483175,\n    warm_up=1.2361,\n    epochs=41,\n    steps_per_epoch=250,\n    dataset='../input/jigsaw20-ds-tt6-36/jigsaw20_ds1789117tt6_fold5.npz',\n    path=f'jigsaw',\n    tpu_id=None, gcs=None,\n    seed=1083,\n)\n\n# auc = train(**params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# clean up repo\n!rm -r jigsaw20","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load pre-trained weights and stats","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"!cp /kaggle/input/jigsaw-tt6f5-20200614-091231-val0-965669/* .\n!ls -shS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from matplotlib import pyplot as plt\n_ = pd.read_csv('submission.csv').toxic.hist(bins=100, log=True, alpha=0.6)\n_ = pd.read_csv('valid_oof.csv').groupby('toxic').pred.hist(bins=100, log=True, alpha=0.5)\nplt.legend(['test', 'val0-normal', 'val1-toxic'])\n\npd.read_csv('history.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.read_csv('params0.965669.csv').T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Apply language multipliers\n\nas in https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160980\n\nThanks @christofhenkel this improves our public LB from [0.9475](https://www.kaggle.com/hmendonca/jigsaw20-xlm-r-lb0-9487-singel-model?scriptVersionId=37240126) to [0.9487](https://www.kaggle.com/hmendonca/jigsaw20-xlm-r-lb0-9487-singel-model?scriptVersionId=37241954) !!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('submission.csv')\ntest = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/test.csv')\n\nsub.loc[test[\"lang\"] == \"es\", \"toxic\"] *= 1.06\nsub.loc[test[\"lang\"] == \"fr\", \"toxic\"] *= 1.04\nsub.loc[test[\"lang\"] == \"it\", \"toxic\"] *= 0.97\nsub.loc[test[\"lang\"] == \"pt\", \"toxic\"] *= 0.96\nsub.loc[test[\"lang\"] == \"tr\", \"toxic\"] *= 0.98\n# min-max norm\nsub.toxic -= sub.toxic.min()\nsub.toxic /= sub.toxic.max()\nsub.toxic.hist(bins=100, log=True, alpha=0.6)\n\nsub.to_csv('submission0.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble with best submission\nhttps://www.kaggle.com/xiwuhan/jmtc-2nd-place-solution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ensemble = pd.read_csv('../input/jigsaw20xiwuhanjmtc2ndplacesolution/submission.csv')\n\n# min-max norm\nensemble.toxic -= ensemble.toxic.min()\nensemble.toxic /= ensemble.toxic.max()\n\nensemble.toxic.hist(bins=100, log=True, alpha=0.6)\nsub.toxic.hist(bins=100, log=True, alpha=0.6)\n\nensemble.toxic = ensemble.toxic * 0.8 + sub.toxic * 0.2\nensemble.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}