{"cells":[{"metadata":{},"cell_type":"markdown","source":"Jigsaw's API, Perspective, serves toxicity models and others in a growing set of languages (see our documentation for the full list). Over the past year, the field has seen impressive multilingual capabilities from the latest model innovations, including few- and zero-shot learning. We're excited to learn whether these results \"translate\" (pun intended!) to toxicity classification. Your training data will be the English data provided for our previous two competitions and your test data will be Wikipedia talk page comments in several different languages.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I would like to thank my team members, for their awesome contribution in the competition.\n\n1. Ashish Gupta (https://www.kaggle.com/roydatascience)\n2. Mukharbek Organokov (https://www.kaggle.com/muhakabartay)\n3. Firat Gonen (https://www.kaggle.com/frtgnn)\n4. Atharva (https://www.kaggle.com/atharvap329)\n5. Kirill Balakhonov (https://www.kaggle.com/kirill702b) ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Adding Henrique Mendonça's Kernel -.9487 Submission\nhttps://www.kaggle.com/hmendonca/jigsaw20-xlm-r-lb0-9487-singel-model\n\nAdding Seawolfs's Kernel -.9522 Submission\nhttps://www.kaggle.com/xiwuhan/jmtc-2nd-place-solution/output?scriptVersionId=37010822\n\n\nThese Guys deserve much more votes for the innovative approaches. Please vote for his kernel.","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.stats import rankdata\nimport glob\nLABELS = [\"toxic\"]\nall_files = glob.glob(\"../input/jigsaw-data/*.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"outs = [pd.read_csv(f, index_col=0) for f in all_files]\nconcat_sub = pd.concat(outs, axis=1)\ncols = list(map(lambda x: \"m\" + str(x), range(len(concat_sub.columns))))\nconcat_sub.columns = cols\nconcat_sub.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nrank = np.tril(concat_sub.iloc[:,1:].corr().values,-1)\nm = (rank>0).sum()\nm_gmean, s = 0, 0\nfor n in range(min(rank.shape[0],m)):\n    mx = np.unravel_index(rank.argmin(), rank.shape)\n    w = (m-n)/(m+n/10)\n    #print(w)\n    m_gmean += w*(np.log(concat_sub.iloc[:,mx[0]+1])+np.log(concat_sub.iloc[:,mx[1]+1]))/2\n    s += w\n    rank[mx] = 1\nm_gmean = np.exp(m_gmean/s).clip(0.0,1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_list = []\n\n#Public Solution\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9139.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9158.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9189.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9234.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9240.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9242.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9257.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9312.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9322.csv\")[LABELS].values)\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9332.csv\")[LABELS].values)\n\n#Ashish Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9336.csv\")[LABELS].values)\n\n#Kirill Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9338.csv\")[LABELS].values)\n\n#Firat Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9346.csv\")[LABELS].values)\n\n#Atharva Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9358.csv\")[LABELS].values)\n\n#Firat Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9361.csv\")[LABELS].values)\n\n#Atharva Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9368.csv\")[LABELS].values)\n\n#Kirill Solution\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9377.csv\")[LABELS].values)\n\n#Public Solution\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9383.csv\")[LABELS].values)\n\n#Mukharbek Submission\npredict_list.append(pd.read_csv(\"../input/jigsaw-data/submission-.9397.csv\")[LABELS].values)\n\npredict_list_2 = []\n\n# Ashish Submission\npredict_list_2.append(pd.read_csv(\"../input/jigsaw-data/submission-.9414-1.csv\")[LABELS].values)\n\n#Public Submissions\npredict_list_2.append(pd.read_csv(\"../input/jigsaw-data/submission-.9416.csv\")[LABELS].values)\npredict_list_2.append(pd.read_csv(\"../input/jigsaw-data/submission-.9422.csv\")[LABELS].values)\n\n# Ashish Submission\npredict_list_2.append(pd.read_csv(\"../input/jigsaw-data/submission-.9427.csv\")[LABELS].values)\n\n# Public Submission - Henrique Mendonça's .9487 kernel\n# https://www.kaggle.com/hmendonca/jigsaw20-xlm-r-lb0-9487-singel-model\npredict_list_2.append(pd.read_csv(\"../input/jigsaw-data/submission-.9487.csv\")[LABELS].values)\n\n\n# Adding Seawolfs's Kernel -.9522 Submission \n# https://www.kaggle.com/xiwuhan/jmtc-2nd-place-solution/output?scriptVersionId=37010822\npredict_list_2.append(pd.read_csv(\"../input/jigsaw-data/submission-.9522.csv\")[LABELS].values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nprint(\"Rank averaging on \", len(predict_list), \" files\")\npredictions = np.zeros_like(predict_list[0])\nfor predict in predict_list:\n    for i in range(1):\n        predictions[:, i] = np.add(predictions[:, i], rankdata(predict[:, i])/predictions.shape[0])  \n\npredictions = predictions /len(predict_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nprint(\"Rank averaging on \", len(predict_list_2), \" files\")\npredictions_2 = np.zeros_like(predict_list_2[0])\nfor predict in predict_list_2:\n    for i in range(1):\n        predictions_2[:, i] = np.add(predictions_2[:, i], rankdata(predict[:, i])/predictions_2.shape[0])  \n\npredictions_2 = predictions_2 /len(predict_list_2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission_1 = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')\nsubmission_1[LABELS] = predictions\n\nsubmission_2 = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')\nsubmission_2[LABELS] = predictions_2\n\nsubmission = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')\nsubmission[LABELS] = submission_1[LABELS]*.35 + submission_2[LABELS]*.65\n\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}