{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this notebook\n\nJMTC-20 is my first competition. I attended it near the end date and did not get a medal (30 places lower than bronze line).\n\nAfter the end of competition, I read the discussion hold by [1st-place](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160862). their impressive ideas and extensive trys of technique makes me want to learn by reproducing their result, at least part of it.\n\nThey said they would release their code soon, but only a post-processing part is public. Hence, I implement my own multiple mono-lingual models achiving <span style=\"color:red\">lb.9508</span>. Hope this notebook and its previous ones can help other beginers of this JMTC-20 task.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<h3>Public Score milestone</h3>\n\n* [Basic XLM-R model with balanced data](https://www.kaggle.com/mint101/basic-xlm-r-lb-9442-intro?scriptVersionId=39822772):  <span style=\"color:red\">(.942X - .9442)</span>\n* Ensemble of XLM-R model:  <span style=\"color:red\">(.9455)</span>\n* [Pseudo-lableling on XLM-R](https://www.kaggle.com/mint101/example-code-of-pseudo-label-on-xlm-r) for one turn:  <span style=\"color:red\">(.9462)</span>\n* [Transfer to monolinguish models](https://www.kaggle.com/mint101/transfer-to-monolingual-mix):  <span style=\"color:red\">(.9467-.9473)</span>\n* Combine all monolinguish models:  <span style=\"color:red\">(.9500)</span>\n* Adjusting according to [4th-place](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160980):  <span style=\"color:red\">(.9508)</span>\n* Mix with [simple ensemble on public kernels before end date](https://www.kaggle.com/mint101/lb-9482-by-simple-public-result-bf-end-ensemble): <span style=\"color:red\"> (.9514) </span>\n\n<br/>\n    \n * Mix with the public version of [1st](https://www.kaggle.com/rafiko1/1st-place-jigsaw-post-processing-example/output) and [2rd](https://www.kaggle.com/xiwuhan/jmtc-2nd-place-solution?scriptVersionId=37463887) results (just for fun): <span style=\"color:red\">(.9557)</span>\n    \n    \n<u>I just transfer XLM-R result to monolinguish models and combine once. This result can be further used to train XLM-R and further transfer. According to [1st-place](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160862), this pattern is doable and may provide another .001+ boost. I just stop here as I have run out of TPU quota.</u>","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \n\nfrom scipy.special import softmax","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = \"../input/jigsaw-multilingual-toxic-comment-classification/\"\nrecord = \"../input/buffer/\"\n\nbase = record + \"submission-9462.csv\"\nmonos = [\"submission-it-9467.csv\",\n         \"submission-pt-9470.csv\",\n         \"submission-es-9467.csv\",\n         \"submission-tr-9470.csv\",\n         \"submission-fr-9473.csv\",]\n\nget_lang = lambda x: x.split('-')[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv(path + \"test.csv\")\ndic_ids = {k:v.id for k,v in test.groupby([\"lang\"])}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Combine all monolinguish models with base XLM-R","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(base)\nfor m in monos:\n    res = pd.read_csv(record + m)\n    ids = dic_ids[get_lang(m)]\n    sub.loc[ids,\"toxic\"] = res.toxic[ids]\n\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Here, we can obtain <span style=\"color:red\"> lb.9500</span>**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Adjustment according to [4th-place](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160980)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"adj = {\n    \"fr\":1.04,\n    \"es\":1.06,\n    \"pt\":.96,\n    \"it\":.97,\n    \"tr\":.98,\n}\nfor l,v in adj.items():\n    ids = dic_ids[l]\n    sub.loc[ids,\"toxic\"] *= v\n\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We get .0008 bost to <span style=\"color:red\"> lb.9508</span>**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Start Ensemble","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"weight = lambda x: softmax(1/(1-x))\n\ndef mix_result(subs,pbs):\n    toxics = np.array([df.toxic.values for df in subs])\n    w = weight(np.array(pbs))\n    print([\"{:.3f}\".format(i) for i in w])\n    return toxics.T@w","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3>Ensemble with public available kernels before end data</h3>\n\nhttps://www.kaggle.com/mint101/lb-9482-by-simple-public-result-bf-end-ensemble is my simple ensemble of public available kernels before end data. I used the before adjustment version (lb.9473) in the competition.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub1 = pd.read_csv(record+\"submission-public-mix-9482.csv\")\nsub[\"toxic\"] = mix_result([sub,sub1],[.9508,.9482])\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Another.0006 bost to <span style=\"color:red\"> lb.9514</span>**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<h3>Ensemble with public 1st and 2nd kernels</h3>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub1 = pd.read_csv(record+\"submission-1st-place-9550.csv\")\nsub2 = pd.read_csv(record+\"submission-2nd-place-9522.csv\")\nsub[\"toxic\"] = mix_result([sub,sub2,sub1],[.9514,.9522,.9550])\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We can achieve <span style=\"color:red\"> lb.9557</span> here.**\n\n1. Ensemble with 2rd (.9522) alone:  <span style=\"color:red\"> lb.9535</span>\n2. Ensemble with 1st (.9550) alone:  <span style=\"color:red\"> lb.9553</span>\n3. Blend 1st(.9550) with 2rd(.9522):  <span style=\"color:red\"> lb.9556</span>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Further improvement\n\n1. Try variable padding in [4th-place](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160980)\n2. Try augmentation, futher corpus generation with qseudo-labels, in [2nd-place](https://www.kaggle.com/xiwuhan/jmtc-2nd-place-solution?scriptVersionId=37463887)\n3. Try to fine-tune like [Jigsaw20 XLM-R lb0.9487 singel model](https://www.kaggle.com/hmendonca/jigsaw20-xlm-r-lb0-9487-singel-model)\n\n.......","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}