{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"# NLP- Challange (part 4)\n\n## Jigsaw Multilingual Challange.\n\n\n\n[**Jigsaw**](https://jigsaw.google.com/) is a unit of Google, which is aim towards \"Using technology to make internet safer\"
\n\n\"Disinformation campaigns are becoming more sophisticated, but so are the countermeasures.\" - Jigsaw \n \nJigsaw use forecasting technology to predicts threat in internet and solves it to maintaining sanity of internet.\n\nThis challange by **jigsaw** is aimed towards finding toxic comments which could destabalize internet.
\nWe are given training, testing, validation which contains sentences in different languages and they are classified in one of the
\nfollowing category \n\n1. toxic \n2. severe_toxic\n3. obscence\n4. threat\n5. insult\n6. identiy_threat\n7. none (all columns zero)\n\n\nwe will see the code using tensorflow and then using hugging face.","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAABAAAAAIACAYAAAAczR65AAAgAElEQVR4nO3d/Xkbx7UHYKACwxUYqsBIBYIqECsI4QZM3wqIVCCmAUNuIEwFAiswUwGhCgxVgL1/5EIXgvCxM9iDA5LvPM/7JHFC7G8/gOycnZnt9Xq9BgAAAHjx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwCcRb/fT88AAACJ0gMApMsoDihIAABwZukBAM7u0jrfl5YHAIAXKT0AQIrJZNJcYpvNZunHBgCAFyk9AMBZrZ+2LxaLnR3w1Wp13h7/VlMAAAAgSHoAgLO71Kf/TaMAAABAmPQAAGe3/fQ/+6n/uq1WKwUAAACipAcAOKvr6+vsfv7BpgAAAECQ9AAAZ7X59P9SnvxvNgWAblzKmxWycuzbblSe9edeynGP2L9L2LdLyPCcOF4A30kPAHA2lzz3f90UANhW0omJ+t+WfNZL63Rdyv7U5Nj1N5eyPwCkSA8AcDaXOvd/sykAdOOSOjmRWaKeTF/S8Xuu54bL8lzO9XPJCTxb6QEAzuI5PP1vmudbAHDT2u7YnOs4DQaDZjweNzc3N81sNms+ffrULBaLb4pgy+WyWSwWzXw+b+bzeXN3d9dcX183o9Eo/Zi1NRqNmslk0vzzn/9s7u/vm8fHx2axWDRPT09fC3zr/Xx8fGweHh6au7u75ubmphmPx80PP/yQvg/Q6/kNBc4mPQBAuJ9++um7p/+X2p5jAWDfjeu683kOxzKORqOzZTlmMBiEnIfBYNDc3Nw08/n85OtwsVg0Hz9+bK6urtKvr+19nEwmzWw2a5bL5elfuKZp5vN5c3Nz0wyHw+prPUIX1+x0Oq3a9t3d3Tef8/Hjx7Nf17uO9dXV1UV+9zazlmasPUcAldIDAIS7vb397qZ/e/j/pUwHmD3DAsAuw+HwrAsuHsszHo9Dt1/S2nQ0S7x7966TTv++tlgsmg8fPnSeu8RgMGhub2876/Tva/P5vPn73/++M8O5n9B+/Pixej82v281nd719VTyvZ1MJuHH5P7+/qRjMR6PwzM+PDwU5Xspv/nAs5EeACBMv9//riN66e2l3AzuKrpEtmN5MgoA+zpPXXWkx+Nx8+nTp7Pu0+3t7Vmvo3XH/6+//jprkW6xWHzXWTx3AaCr362aJ8w135f5fB5+TE4tAN3d3YXmGw6HxZnOUZQA2JAeACDUuiO6WCya+/v75u7urplOp810Om1ub2+b6XTazGazZj6fN8vlMn0kwEsoALx58+bsRZdjmd69e3fWPPvaarU6uQAwGAyaDx8+HN1OSaaStlgszrJOQE3xbrVadfod/vDhQ9iw8UO6LFg9PDw0vV5ZAWMwGBR3tlerVeixGo1GJ5/b6CLFb7/9VpRnsVh8/VtrAABnkh4AINRoNCq6KX3z5k0zmUxaDzXtumDwEgoA53763zTfFwC2b6bXHarsAk/TnDYCYDQaNYvF4iL2I3I0wNXVVfhw/7ZtsVgUnbMuOnKz2azTfah5ynx3d9fqszevxd9++63z47H++9LO9b6MkUWKx8fHolzbv/eKAMAZpAcAOKvSp2CTyeTrU8hzrBvw3AsAWVMujuV6CWsAXF9fn9QpPvV63XX9RxQBrq6uTsoZ0fYVAaI6bF0XP17CNID7+/tOfnPfvn0b9ttX2naNpFEEAIKlBwB4FiaTSfP58+eTbz6PtedeAMh4+t80+wsA65vpyAJAaaekpgBwfX0dnrH2f9NlEWA4HF7Mk//ttlgsOn16vK+jt/3K0lMKj5uvQqzJWHMuop6wd3VdbBdDuupw1w7/1+EHziw9AMCzMp1Od96AdzUa4DkXADIXXDyW7TmPADi1839qa3Ntd1UEyF6w89i+tnnCfWqHrmal+zbZI6cBbLb/+Z//6fy3ZTQatdrHNi1qlELptXtzc9PpdQPQUnoAgDSHbrgO/XeRHd3nWgDo9/tpT/+bZncBYPMcPtcCQNsn4m2f3i+Xy2axWDSLxaLoiWqbz69dzXx9nrp68v309NTM5/Pm4eGhmc/nzXw+bx4fH/fub2mnctc8967UDCNv22pWwL+UaQDb18Yp7a+//up8lMK6QFHSMl+rCbxq6QEAnqXhcFi84FOb9twKAOvOW5dFkZqnfMdylnRkIhfYa/sWgPUrLJ+enk7a3nK5bO7u7pqrq6u92x2NRs2vv/769d3vte3UIfKlU2w2h7ff3d014/H46PZHo1EzmUyKvrvb18Nyudy5ndonuJt/12VHd7vVTgPY970+9D3p8vj0evWjIva1U1+9t70vx97Ksd3O8cpEgD3SAwA8ax8/fuz0xvS5FQDW1gsmdqFm2HFJvuvr6063v1qtmpubm9b717aTfMq1tVwum+l0WtwhHw6HX7dbUwipWWyu1yuf5rDOdspr+q6urqqKVqvVau9+njqM+9QizL7jtG41Hd/pdHr0c7db16MkuhxxtVqtvubbdb5Kz2G/3y/ON5lMOrtmAAqlBwB49rosAjzXAkCXaoYdH/vMkpvsmu0fe6pfepO/fhJc0wl/eHj4Js/mttvmqH3l2r6n48fsesJ7bN/XnfBTOlCDwWDnK/eObfvLly8nXeO7Mh8a/r8rz3w+L14Yr+3vy2a+fbkOHaMun3DXDK8/JV/p9dTF8P+a7yhApfQAAM9av99vBoPBSUOKN5sCQEwBIHr7Xc/nrR36XzPPe5/RaNS6g7l5TdeMAuiyI1vTgfrXv/5VtP2m+f5p+qkdt19++aVo+9fX13tHq+z7jakt0NSMTOhqnn3EtIhTCzib/vjjj6JtG/4PJEsPAPAidPX6MgWA11MA2NdhrF31v8vO/1rbkQCbHc71683a7vuxJ6i7OrP7nqDWdsIHg0Hx97d2usM+m0XENgsgDofDr9dqyUiRzeHnbc9TzYiQf/zjH50cl+3RIV2t0fH27dtO8p0y/B8gQXoAgBfj6urq5JtSBYDXUwDYp2a+c5tOd622T383O2Ylc83fv39ftK/rxey6Hiq9a677odbVk9z1go+1225zvWyem5Lc/X7/6yin0tbV8alZhLBN27VOQek1VfNb0fUbCAAKpQcAeFHm8/lJN6YKAK+7ALBv28euqahXivX7/arjUfIu+H1DvPft8+PjY8i+lnZyn56eOtt26cKT19fXX/+2tHDRNHWr9Nf8tp16XdasP9C2dVGgOLb+y2bO1Wrl9x24BOkBAF6UXVMBSm5W3SC+zgLAuvO1az7xsetn85qJWkBsexTAsUz//ve/W3/2sTneu17Dt+vYnarf7xe/2vPU871WOupjfb0dKtAcOkclq/R3OQ2gdjHMtvtV0mpfi3jKeXv//n3I9xOgQHoAgBen5oncuikAvM4CQK9X/gS6y22XXNNtOmAlUxJqFnk79T3u+/z888/NeDxurYviw/b1duz47npyvd0RbfMZpdlr1kk49Sl7yRtWHh8fi/KtVqtmNBp1dt6ObStymg5AgfQAAM/Wvhvo2o5c0ygA9HqvtwBQs4ZE1NP/7c8ajUbNdDot0nZbNQWAx8fHFzOXuvQ1opvD/9e2i45tijT7jt/2ud/8zzVvAzilWFPyhP3Dhw/Fx/Lm5ibsvG2fA7/twIVIDwDwImzfND88PBTdiLpJ/H+vtQCw6330x9q7d+8OXoeX4Fimv/3tbzv37Vgn9vHx8Zuh8JEZIx3r5G4fh10d99JFBJum7i0GNd+N9XZKj3HpPr1//774rRUlU1VKz9t2ixq1AlAoPQDAi1S6sNm6KQC8vgLAumNUOv+8iznMmdb7fcqImab573fmuXau3r1713o/V6vVwQ5r6dP5muH5u6YBHPtNqx36vj0i5th2BoNB8aKB+75Dh4oVJQtjrre7WCwusjgHvErpAQBepO0b5bYLVykAvL4CwFppO+Xp5b7OSFYnpWZo+XZbLBbN/f19c3Nz04zH42cxRaB0yPqud8ivz1nN2iPHCieb18P635e+seDYdnZdc/1+v3j+//pvSwsUNesAlJ43v+vABUkPAPAi9fv9qk6NG8XXWQAYjUbF29xeYf3SHSo6nLJw5qH2559/NrPZrPn1119bd/TOWQQpXVRvXdTYlbFmJMV0Om29v+v/XenT7/V2So9NmxExq9Xqu9fr3d/ff7f9Q39f8kaE9XF4eno6uI3tf76+9owCAC5AegCAF6vmSZkCwOssANQsALjrlWLPtYNxaIX5rl77tm6fPn1q7u7umqurq6Knv10f29Jzfuy3oabouFwudz7lP+avv/76+hltzk/pdJXSofyb34XSdQBKf3NLz5vV/4ELkx4A4MWqWd1cAeB1FgBq3rH+t7/9Lf1c1djXyTw0CqDrIsDmZy6Xy+Zf//pXc319fZZXKq6tn1S3bVdXV0c/8+bmpvg41Kyf0PU0gG2lnezN8zYajYqul+0iyDGlq/+XjjAACJYeAODFqhnWrQDwOgsANUPgT9nmfD5vFovFNz5//vzNv57i8+fPVR2f0oUQ27SSzuBqtWoeHx/DiwGbw/W7fIJeMw3g7u7u69+37QjXfEfu7u5af/66wNDm2PznP//57u+PTa3Y/tySc91m2sbm55+zqATQQnoAgBerdBhr0ygA9HqvswBQuqjYqfu8WCxCnqpvtn3zvg+9Z344HBa/Xm27be5X7T6uVqvmy5cvzWw2a968edP5NV46OqjN78L6ONZMA6i9hqK2s1kIqnlzSunoirbFqtKRCTVvWgAIlh4A4EUrbQoAr68AULrieRf7fGonu02rWfit1/tvEWC9yNoltMVi0VxfX3d6jZV20kuGz9dMJ6mZBhDx1oFer3wUw65zU3oM2v7uln5Pu75uADqQHgDgRSttCgAKAKX7XLM43a4RAF2PCKgtAPR6/+0E1swzr21t9v3333/v5NWC+0YG7Wuli8jVTAOo+d0p3Y+maTcN4JT5/7Xf4TajEw4d133Xz08//dTpbxNAB9IDALxY/X6/6Ca0aRQAer3XVwDo9XKmABxqXRQDTikArI1Go6rXaUa1x8fH4iLAdoc3Yvj/tpppADXFjU+fPhVv59Cx6fXKFhg8VBz58uVLUbZj3+F9523fd8Xwf+BCpQcAeLFqnpApALyeAsBm56erAkDb0QA1BYDSokAXBYDNc1JzjCLa/f39See6dKHD8XhcPMqjZhrAZDIp3q/t7bS5Ro5NAyg5Pod+L0uLE8f2v3RdAcP/gQuVHgDgxdrVkaxZ0Oq1eS0FgE2bndu2He1ThqNnLgJ4iuFw2FxfXzf39/fFT3i7bLWvdose/r82GAy+Wa1+tVodPd81T6xPfetA7eet9+VQp31dnGiz701z+Le3Zj+3v581U3UAAqQHAHixSueyHrsJfS1eYwHg3K8BvORFAEuMRqNmMpk0Hz9+PPkVgqXvjl938Eo6dv/4xz+KMh36PTi23WPTAHbtb01Radd2Dh3LQ/PtS797o9Hom2OxeUxKP+tQseUc0zYAziQ9AMCLVbOImRvH11EA2O681QzZfv/+ffU+PtcRAMcMBoNmPB43Nzc3zf39fUihY33c2owC2D7PpXkODZc/9DrFfr9f3Gltu0/bNp+0n7pfd3d3rT5ntVq1Gh2xOQqiTdv3PS5dU2Hfd9MoAOACpAcAeLFqFi9TAHgdBYBtm6NF2nakfvvtt+oOxf39fTOfz4uVtIwCwC6DwaC5urpqptNp8+nTp+JO4b62PWT+2LkYjUZV21h7eHg4eG4+ffr0zX+uGRHRdhrA5r5uTzdomnZTnXYdr5Jr7N///vfRnA8PD0X7v2tKQem0jaenp/RrHuCA9AAAL1LNnNGmUQDo9V5nAeA5LBhZ2oHtugDQptjRtiAyGo2am5ubr0/ka0dDlAyZv5RFDI+1rqYBHGq73jpQ+pvZZpG90qk1u9YnMPwfeGHSAwC8SKWvjLrEm8fJZNJMp9Nv3N7efvfP1tbzcU/1GgsAvV75cOXtIdDRw4tL17S4lBEAba7zz58/F+3bum0OZT90/Pv9/lnWXeii1Zy30u/MarX67u0G4/G4qBDT5vemJNe+aQWlxY1jbzkASJYeAOA765vCU1Y5P2fOXWrfXX5JBYDSRcROGZK+6bUWAGqGbO/qbEQVAkrXtHguBYBe778jMGo66JtDxg8d95pr6lxt/T1e/2vp1IZeb/c0gGNt+7du39P6Xav4LxaLsFyb/79zaGTOrmJF7VsbAM4oPQBA0+t9f5M5HA6rXkt1CTZvGkuHFl9SAeCPP/4oyj6dTpt+v39yB7R0yO2hVcVrZBUAahaN3O5kd9X53/U5pR3kNgWAwWDQDIfD1iKv95rz3rbIsR7+H73wYlet9Cl2v98vvn7X0wDW11rX8//XSouxm0Wd7d+i5zSCC2CP9AAAO61vmJ/jcMrtub4lN/2XdAN5aM7yrn3qKnubebub2398fOx0v7MKAO/evSve7q651BFqFrBr0zk+9NR3V4v6PVh3QkufFrctwHS18GBE23WszzENoGm+7WyXtJK3FdSsA1BTlGiadtMSAJKlBwD4zps3b77eUD23UQA1i7lttksqABx7Nd12x6Grjvj9/X3RMev6GskqAPzwww9VHcVzDLWvWcCuTa7S1x/WvKauRMSbDt6/f1/0mZfQakfVlF6/6+9u6XeupKNd+tnrYfwlv+VtX0sIcAHSAwB8Z7uz8fbt2/RMbc1ms6Kbze12SQWA0kXfmqb9CuKHhqq36URsFh92rdx9iqwCQK9Xt3ZE9CiA2qJWm87x9jV27iHW29dh6fHfLkjsuq5LC1qX0mpGW9ze3hZtY33tljylLy1O/Pjjjzs/59C1NhgMvhn+32YUV3RxCqAj6QEAvrGrs7FYLC5+QcBe77/zRbcX1Cptl1QAqHmV4ak3wTVFh/fv33/zGafOg88sANQuFnd/fx92HdQuaHmsANDv978Z7dOmRRc7Sp5gr1ar7669Lr5Dl9JqCmu10wAi5v9v/g6UXsPv37//7m/2/aav/3n0GhUAHUkPAPDNjdq+ocaX1DHeZTgcdjLP99L2s3Tht1OH49d0Nru+8c4sAPR69fPFb29vOz//pU90N1vbqQmnzrvPPO/HhqKXLmh5rrbdmd3VuT32pH1foa30bRZtvvOb+WqKjG0XKFxvJ3saEkCg9AAAXx0bahzRwSmx74Z3MBh09o7vSysAbN44tx3VUDsKoG1nafO1YPvWHThlFEDNYnylBYBD+UoXxttsXX5HTun8N035Cvkl7eeff+78Wv/Pf/5zdLub5+Cvv/46+pmlBa3ZbNaMx+POvH37tnn37l0zHo+rjnPNNIDSRffOkSn6NYzX19edX48AQdIDAK/Y+pVx687Qx48fjw6hv7QbrR9++KHq/e372qUVAGpunJfLZfFq2Ifew36o49vV9bDZIW+7z5u5fvrpp52fVaPtu8v3HZcPHz6cvP0PHz60OgaHWtsCQM01tlgsOl1xvWZ/jw1Fb7t2wuZ5jBhGvv6NrTnOm79H+67rXa9wjWpfvnypvqYjm+H/wDOSHgCg6fXKbhrbLLx1DoPBoPnzzz87fbf3pRUAer3/f4p5aD83n8o3zX+LANud833naTweV4+gaHvjPRgMmuvr62Yymey1/u9rnnz/+uuvBz97048//ngwa7/fb7U6/qHzsVgsqoojp5yL7dZmDYD1vz9l2sP6Gqj5HRiPx9VrHByb/186/L/r11nuUnqca9dceHh4+O6zuvid/PTpU/W+d1mo3WyG/wPPTHoAgKbX6zV//PFH0U1X9nSA4XAYckN5iQWAXU8O297Mz+fz5vr6+rtOxGAwOKnzVXqs2r7Hvu1+bRc8SlrbosUpx2bdnp6emru7u2Y8Hu8sPKzPw3Q6/aZzeGzBszatZK7+KdMNlstlM5vNmqurq1bHdjgcNjc3Nycd3zavfNv3+fuO4TlWkW87F36zTSaT4u2Uvt7xUNv8rp1yjGr2vU27tFFpAEekBwCoHjL68ePHlKGXXT4l3W6XWADo9brpjC6Xy2axWHx37KI70utzlt3W+/nTTz/tfFq9ayj1ly9fOs2wXC6bp6enZrFYdLJo5aFWUgDoch2N5XLZ/Pnnn818Pv/Gep9rr7fNvzvW6av5TTvHb1nN96DmCXfUkPua+f9rNW8YadO2i5tZI9IAWkoPALxSbVb+b9NqhzrX+PHHHzuZG32oXWoBoO0T9M3W5dSI7bavc3lomsG5M+5rbTt6/X6/+F3kka20g166Wv8lFGnatDYd4pubm6Ofs3k+zzH8f62m8FMzDaCLouFmO/ZWgmO/AxFFiTZrJABcmPQAwCvX1YJRs9ks9AnaZDIJf2K63o/sc7JPl8N6T2k1naVL6lyWXqeRq6qXFBVKj2HN6/rOeY21eRXedlssFq3OX5tiyamvtqtVMxS+Jl/Xr0B8eHg4ed+7nrZ1dXXV9Hrfdv4VAoALlx4AeOVOefq/q93f33+9KTvVjz/+2Nze3rbu+B97g0GbdskFgF6v23m0NcepbQds23MuAPR6u4sAXczTb9vWnfmav9ml5jWI2a3t2y1qRsuccypT5DSAzfPa9m0WbVtNQSny92tzHQidfuAZSQ8AvGKRr4v6/Pnz14XBSoavjkaj5ubmZucq1udol14A6PXKF2xs07p8+rrLcysA7OpQZI3A2Hz6W9JO6bBdWhGg5LWDpUXNcw7/XzvHNIB+v9/c398Xb2dfe/v27cn73eWohOfwWw2wQ3oA4BVad266fvp/qC2Xy2Y+nzez2ay5u7trptNpM51Om9ls1sxms+bx8fEsQ/yPtUu6qcx6SrurGPD4+HjSU9KMAsC+osYp+zEcDqsXyzv2Gsfttlwuv+t0lbRTn9heXV213tea4fxt23w+LzpnpednvYbJOZ8i1zwJrzmfb9++Ld7OvtbFfne5DsApCxICJEoPALwSu1Y4175vl1QAOKakg3ZK62Lo73MbAXDIYDAIf0K+r9Nb0k45b+vfi+FweNZC4WZbLpdfRz+s8xzrpNdcZxVb4McAAAiQSURBVFlvMilttW8D6KKweur8/83z1sVvVpvXQAJcqPQAwCuVdVNf0865+vpzKgCsTafT4kXP2rTNTuipT0fPXQDYfHf5duuqw3dK53hftvl8fvDJZkm7vb3t7Bpb7+uh62zXPtV8d5fLZTOdTr8OeW+7wFu/3y8+HzWd6q7UdMxrnnqfOu9+tVp1UgTsKk/TPM/faYD/kx4AeIXevHlz8g3YS22XeGPZtvP9yy+/nPzqr8Vi8bXz1eWQ6OFw+HW6xzn9/vvv3/2zmleq9fv9ncej3+83w+GwmUwm1cd+uVw2d3d3Xzt3h457yb53tRjntqurq6PFgJrrbvMY1NqcVpR5jNqYTCbF13Pp8en3+81oNDr5e9R2/YU2xuPxyXkyRm0AdCQ9APAKZT79b/NEMPN967MLLACUGgwGzdXVVTOdTpv7+/tmPp83i8WiWS6XzWq1apbLZbNYLJrHx8fm/v6++ec//9lMJpOdN9WnFAEudWXu6Fxv375tbm5umru7u6/Hf9Ps/9bB2HfML9n2UPzhcPj1WpvNZs18Pm8eHx+/Xm/rtlgsmsVi0Tw9PTXz+bz5/fffm5ubm+JFQp/j9dZF5q5ec5d1jJ7juQEIkh4AeGWy5/5ndu7btEsvAETdSB8bWn1peS9l+9n7d+kyrtfnbHu/nut+vpT9AAiQHgB4BTZvvp7T3P99LbKIcOkFgOjr46Vsr+ttXEoH5hJy7HsavW+KxHM7LpdwjC+J4wHQqfQAwCuxnqusHW6z2cwNLwAAEdIDAC/cS3v6H91e4wgAAADOIj0A8Epc0nvYL7kpAAAAECQ9APBKnPp6uNfSFAAAAAiSHgB4BTz9b98UAAAACJIeAHgFPP1v3xQAAAAIkh4AeIWscg8AAGeXHgCAnqIIAADh0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADx0gMAAAAA8dIDAAAAAPHSAwAAAADB/hfCXRtk1ND2hwAAAABJRU5ErkJggg=="}}},{"metadata":{"trusted":true},"cell_type":"code","source":"import os, time\nimport pandas\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom kaggle_datasets import KaggleDatasets\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\n\n\n# We'll use a tokenizer for the BERT model from the modelling demo notebook.\n!pip install bert-tensorflow\nfrom bert.tokenization import FullTokenizer\n\nprint(tf.version.VERSION)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# LOok inTo tHe dAta"},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = \"../input/jigsaw-multilingual-toxic-comment-classification\"\n\n#herre is am using data from bert_multi dataset which has 'vocab.txt' and other required for bert.\n# https://www.kaggle.com/kivlichangoogle/bert-multi\nBERT_PATH = \"../input/bert-multi/bert_multi_from_tfhub\"\n\n#training data\ntrain1 = pd.read_csv(f\"{PATH}/jigsaw-toxic-comment-train.csv\")\n\n#we will use only train1 data for now \n# train2 = pd.read_csv(f\"{PATH}/jigsaw-unintended-bias-train.csv\")\n\ntest = pd.read_csv(f\"{PATH}/test.csv\")\nvalidation = pd.read_csv(f\"{PATH}/validation.csv\") \n\n#sample submission\nsample = pd.read_csv(f\"{PATH}/sample_submission.csv\")\n\nprint(\"train 1 : \",train1.shape)\nprint(\"test : \",test.shape)\nprint(\"validation: \",validation.shape)\nprint(\"samaple : \", sample.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train1.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are four basic steps performs while making BERT model.\n1. tokenizing the words and converting it to ids.\n2. adding bert layer to keras model\n3. training data\n4. making prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_tokenizer(bert_path = BERT_PATH):\n \n #load bert_layer from BERT_PATH\n bert_layer = tf.saved_model.load(bert_path)\n bert_layer = hub.KerasLayer(bert_layer,trainable=False)\n \n #get the vocab file required for tokenizer\n vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()\n do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()\n tf.gfile = tf.io.gfile\n \n return FullTokenizer(vocab_file=vocab_file, do_lower_case=do_lower_case)\n\ntokenizer = get_tokenizer()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#demo of what tokenizer actually does\nprint(train1.comment_text[0])\n\ntokens = tokenizer.tokenize(train1.comment_text[0])\nprint(tokens)\n\ntoken_ids = tokenizer.convert_tokens_to_ids(tokens)\nprint(token_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#maximum words allowed in single sentence including [CLS] AND [SEP]\nmax_length = 128\n\ndef comment_to_ids(comment,tokenizer=tokenizer,MAX_LEN = max_length):\n \n #tokenize the sentence\n tokens = tokenizer.tokenize(comment)\n \n #truncate the sentence\n if len(tokens) > (MAX_LEN - 2): # -2 because [CLS] ans [SEP] are 2 default tokens to add\n tokens = tokens[:(MAX_LEN -2)]\n \n token_ids = tokenizer.convert_tokens_to_ids([\"[CLS]\"]+tokens+[\"[SEP]\"])\n \n input_mask = [1] * len(token_ids)\n \n #padding sequence\n padding_len = MAX_LEN - len(token_ids)\n \n token_ids.extend([0] * padding_len)\n input_mask.extend([0] * padding_len)\n \n segment_ids = [0] * MAX_LEN\n \n return token_ids, input_mask, segment_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef convert_comments_to_ids(data,MAX_LEN = max_length,text_label=\"comment_text\"):\n \n data[\"input_ids\"], data[\"input_mask\"],data[\"segment_ids\"] = zip(*data[text_label].apply(comment_to_ids))\n \n return data\n\ntrain_ids_df = convert_comments_to_ids(train1)\n\n\n#need some help here \n#as comment_to_ids is returning three values\n#I have written it like this zip(*data[text_label].apply(comment_to_ids))\n# so Is this the best practice or there is any othr way to do it.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# converting various toxic label to one column toxicity\ndef get_toxic_label(comment):\n toxic_labels = ['toxic', 'severe_toxic', 'obscene', 'threat','insult', 'identity_hate']\n \n if comment[toxic_labels].any():\n return 1 \n else:\n return 0\n\ntrain_ids_df[\"Toxicity\"] = train_ids_df.apply(get_toxic_label,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train_ids_df[[\"input_ids\",\"input_mask\",\"segment_ids\"]]\ny = train_ids_df[\"Toxicity\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.2,shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Config"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is\n # set: this is always the case on Kaggle.\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bert Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"BERT_GCS_PATH = KaggleDatasets().get_gcs_path('bert-multi')\nBERT_GCS_PATH_SAVEDMODEL = BERT_GCS_PATH + \"/bert_multi_from_tfhub\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def bert_model(bert_path=BERT_GCS_PATH_SAVEDMODEL,MAX_LEN=max_length,trainable_bert=True):\n \n input_word_ids = tf.keras.layers.Input(shape=(MAX_LEN,),dtype=tf.int32, name=\"input_word_ids\")\n input_mask_ids = tf.keras.layers.Input(shape=(MAX_LEN,),dtype=tf.int32, name=\"input_mask_ids\")\n segment_ids = tf.keras.layers.Input(shape=(MAX_LEN,),dtype=tf.int32, name=\"segment_ids\")\n \n bert_layer = tf.saved_model.load(bert_path)\n bert_layer = hub.KerasLayer(bert_path,trainable=trainable_bert)\n \n pooled_output,_ = bert_layer([input_word_ids,input_mask_ids,segment_ids])\n \n output = tf.keras.layers.Dense(32, activation='relu')(pooled_output)\n output = tf.keras.layers.Dense(1, activation='sigmoid', name='labels')(output)\n \n model = tf.keras.Model(inputs ={\"input_word_ids\":input_word_ids,\n \"input_mask_ids\":input_mask_ids,\n \"segment_ids\":segment_ids},outputs=output)\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Configuration\nEPOCHS = 3\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices(((X_train[\"input_ids\"],X_train[\"input_mask\"],X_train[\"segment_ids\"]),y_train))\n .repeat()\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n )\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices(((X_val[\"input_ids\"],X_val[\"input_mask\"],X_val[\"segment_ids\"]),y_val))\n .repeat()\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n bert_ = bert_model()\n\n # Compile the model. Optimize using stochastic gradient descent.\n bert_.compile(\n loss=tf.keras.losses.BinaryCrossentropy(),\n optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n metrics=[tf.keras.metrics.AUC()])\n\nbert_.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = bert_.fit(\n # Set steps such that the number of examples per epoch is fixed.\n # This makes training on different accelerators more comparable.\n train_dataset, steps_per_epoch=4000/strategy.num_replicas_in_sync,\n epochs=5, verbose=1, validation_data=valid_dataset,\n validation_steps=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Hugging Face"},{"metadata":{"trusted":true},"cell_type":"code","source":"import transformers\nfrom tokenizers import BertWordPieceTokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n\n#this function creates three files given in output\ntokenizer.save_pretrained('.')\n\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt',lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#this code does the tokenizing part and return ids\ndef fast_encode(texts, tokenizer, chunk_size=256, MAX_LEN=max_length):\n \"\"\"\n https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n \"\"\"\n #truncating\n tokenizer.enable_truncation(max_length=MAX_LEN)\n \n #padding\n tokenizer.enable_padding(max_length=MAX_LEN)\n all_ids = []\n \n for i in tqdm(range(0, len(texts), chunk_size)):\n text_chunk = texts[i:i+chunk_size].tolist()\n encs = tokenizer.encode_batch(text_chunk)\n all_ids.extend([enc.ids for enc in encs])\n \n return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nX = fast_encode(train1.comment_text.astype(str), fast_tokenizer, maxlen=max_length)\ny = train1[\"Toxicity\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train,x_valid,y_test,y_valid = train_test_split(X,y,test_size=0.2,shuffle=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n tf.data.Dataset\n .from_tensor_slices((x_train, y_train))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((x_valid, y_valid))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def bert_model_2(transformer, MAX_LEN=max_length):\n \"\"\"\n https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n \"\"\"\n input_word_ids = tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name=\"input_word_ids\")\n sequence_output = transformer(input_word_ids)[0]\n cls_token = sequence_output[:, 0, :]\n \n output = tf.keras.layers.Dense(32, activation='relu')(cls_token)\n\n out = tf.keras.layers.Dense(1, activation='sigmoid')(output)\n \n model = tf.keras.models.Model(inputs=input_word_ids, outputs=out)\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n \n transformer_layer = (\n transformers.TFDistilBertModel\n .from_pretrained('distilbert-base-multilingual-cased')\n )\n \n bert_ = bert_model_2(transformer_layer)\n\n # Compile the model. Optimize using stochastic gradient descent.\n bert_.compile(\n loss=tf.keras.losses.BinaryCrossentropy(),\n optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n metrics=[tf.keras.metrics.AUC()])\n\nbert_.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history2 = bert_.fit(\n # Set steps such that the number of examples per epoch is fixed.\n # This makes training on different accelerators more comparable.\n train_dataset, steps_per_epoch=4000/strategy.num_replicas_in_sync,\n epochs=5, verbose=1, validation_data=valid_dataset,\n validation_steps=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It shows that tokenization in hugging face is superfast and
\nloading and training transformer is quit simple."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}