{"cells":[{"metadata":{},"cell_type":"markdown","source":"Note: While in the earlier editions of Kernel , I was trying to create a basic model , however since I have been using Bokeh for analysis , it requires internet , rendering the modelling useless if done in the same notebook."},{"metadata":{},"cell_type":"markdown","source":"
\n

Table of Contents

\n Introduction to Riids Competition1\n References 2\n Basic EDA and Insights so far3\n "},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"# Introduction to Riids Competition\nThe competition has been quite interesting so far, and there still remains room for more improvement.\nIn this competition , we have been given tabular data , and it seems to be the case of supervised learning problem.\n\n![image.png](attachment:image.png)","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAyAAAAGQCAYAAABWJQQ0AAAgAElEQVR4Ae3dD6xeZX0H8JaSgRYnWrZlmgWJLK4GiYyBY4xVhxoTcIiGDRSGcwYUcFqhZIoE3RjGDnRgJtmGyxgNozItjkEngkNc2wlIHPhnAor8MdiCMApt7d9nOYXS29733t6n95zfec45n5s0lLfnvec9v9/n+fN9733fd8aMGTOSP2rAAAMMMMAAAwwwwAADQQYUOqjQgp6wywADDDDAAAMMMMCAzbcAxgADDDDAAAMMMMAAA4EGFDuw2BKvZz0YYIABBhhggAEGhm5AABFAGGCAAQYYYIABBhhgIMxA2ImGnvRcv2c7GGCAAQYYYIABBhiQ9AQwBhhggAEGGGCAAQYYCDSg2IHFlng968EAAwwwwAADDDAwdAMCiADCAAMMMMAAAwwwwAADYQbCTjT0pOf6PdvBAAMMMMAAAwwwwICkJ4AxwAADDDDAAAMMMMBAoAHFDiy2xOtZDwYYYIABBhhggIGhGxBABBAGGGCAAQYYYIABBhgIMxB2oqEnPdfv2Q4GGGCAAQYYYIABBiQ9AYwBBhhggAEGGGCAAQYCDSh2YLElXs96MMAAAwwwwAADDAzdgAAigDDAAAMMMMAAAwwwwECYgbATDT3puX7PdjDAAAMMMMAAAwwwIOkJYAwwwAADDDDAAAMMMBBoQLEDiy3xetaDAQYYYIABBhhgYOgGBBABhAEGGGCAAQYYYIABBsIMhJ1o6EnP9Xu2gwEGGGCAAQYYYIABSU8AY4ABBhhggAEGGGCAgUADih1YbInXsx4MMMAAAwwwwAADQzcggAggDDDAAAMMMMAAAwwwEGYg7ERDT3qu37MdDDDAAAMMMMAAAwxIegIYAwwwwAADDDDAAAMMBBpQ7MBiS7ye9WCAAQYYYIABBhgYugEBRABhgAEGGGCAAQYYYICBMANhJxp60nP9nu1ggAEGGGCAAQYYYEDSE8AYYIABBhhggAEGGGAg0IBiBxZb4vWsBwMMMMAAAwwwwMDQDQggAggDDDDAAAMMMMAAAwyEGQg70dCTnuv3bAcDDDDAAAMMMMAAA5KeAMYAAwwwwAADDDDAAAOBBhQ7sNgSr2c9GGCAAQYYYIABBoZuQAARQBhggAEGGGCAAQYYYCDMQNiJhp70XL9nOxhggAEGGGCAAQYYkPQEMAYYYIABBhhggAEGGAg0oNiBxZZ4PevBAAMMMMAAAwwwMHQDAogAwgADDDDAAAMMMMAAA2EGwk409KTn+j3bwQADDDDAAAMMMMCApCeAMcAAAwwwwAADDDDAQKABxQ4stsTrWQ8GGGCAAQYYYICBoRsQQAQQBhhggAEGGGCAAQYYCDMQdqKhJz3X79kOBhhggAEGGGCAAQYkPQGMAQYYYIABBhhggAEGAg0odmCxJV7PejDAAAMMMMAAAwwM3YAAIoAwwAADDDDAAAMMMMBAmIGwEw096bl+z3YwwAADDDDAAAMMMCDpCWAMMMAAAwwwwAADDDAQaECxA4st8XrWgwEGGGCAAQYYYGDoBgQQAYQBBhhggAEGGGCAAQbCDISdaOhJz/V7toMBBhhggAEGGGCAAUlPAGOAAQYYYIABBhhggIFAA4odWGyJ17MeDDDAAAMMMMAAA0M3IIAIIAwwwAADDDDAAAMMMBBmIOxEQ096rt+zHQwwwAADDDDAAAMMSHoCGAMMMMAAAwwwwAADDAQaUOzAYku8nvVggAEGGGCAAQYYGLoBAUQAYYABBhhggAEGGGCAgTADYScaetJz/Z7tYIABBhhggAEGGGBA0hPAGGCAAQYYYIABBhhgINCAYgcWW+L1rAcDDDDAAAMMMMDA0A0IIAIIAwwwwAADDDDAAAMMhBkIO9HQk57r92wHAwwwwAADDDDAAAOSngDGAAMMMMAAAwwwwAADgQYUO7DYEq9nPRhggAEGGGCAAQaGbkAAEUAYYIABBhhggAEGGGAgzEDYiYae9Fy/ZzsYYIABBhhggAEGGJD0BDAGGGCAAQYYYIABBhgINKDYgcWWeD3rwQADDDDAAAMMMDB0AwKIAMIAAwwwwAADDDDAAANhBsJONPSk5/o928EAAwwwwAADDDDAgKQngDHAAAMMMMAAAwwwwECgAcUOLLbE61kPBhhggAEGGGCAgaEbEEAEEAYYYIABBhhggAEGGAgzEHaioSc91+/ZDgYYYIABBhhggAEGJD0BjAEGGGCAAQYYYIABBgINKHZgsSVez3owwAADDDDAAAMMDN2AACKAMMAAAwwwwAADDDDAQJiBsBMNPem5fs92MMAAAwwwwAADDDAg6QlgDDDAAAMMMMAAAwwwEGhAsQOLLfF61oMBBhhggAEGGGBg6AYEEAGEAQYYYIABBhhggAEGwgyEnWjoSc/1e7aDAQYYYIABBhhggAFJTwBjgAEGGGCAAQYYYICBQAOKHVhsidezHgwwwAADDDDAAANDNyCACCAMMMAAAwwwwAADDDAQZiDsRENPeq7fsx0MMMAAAwwwwAADDEh6AhgDDDDAAAMMMMAAAwwEGlDswGJLvJ71YIABBhhggAEGGBi6AQFEAGGAAQYYYIABBhhggIEwA2EnGnrSc/2e7WCAAQYYYIABBhhgQNITwBhggAEGGGCAAQYYYCDQgGIHFlvi9awHAwwwwAADDDDAwNANCCACCAMMMMAAAwwwwAADDIQZCDvR0JOe6/dsBwMMMMAAAwwwwAADkp4AxgADDDDAAAMMMMAAA4EGFDuw2BKvZz0YYIABBhhggAEGhm5AABFAGGCAAQYYYIABBhhgIMxA2ImGnvRcv2c7GGCAAQYYYIABBhiQ9AQwBhhggAEGGGCAAQYYCDSg2IHFlng968EAAwwwwAADDDAwdAMCiADCAAMMMMAAAwwwwAADYQbCTjT0pOf6PdvBAAMMMMAAAwwwwICkJ4AxwAADDDDAAAMMMMBAoAHFDiy2xOtZDwYYYIABBhhggIGhGxBABBAGGGCAAQYYYIABBhgIMxB2oqEnPdfv2Q4GGGCAAQYYYIABBiQ9AYwBBhhggAEGGGCAAQYCDSh2YLElXs96MMAAAwwwwAADDAzdgAAigDDAAAMMMMAAAwwwwECYgbATDT3puX7PdjDAAAMMMMAAAwwwIOkJYAwwwAADDDDAAAMMMBBoQLEDiy3xetaDAQYYYIABBhhgYOgGBBABhAEGGGCAAQYYYIABBsIMhJ1o6EnP9Xu2gwEGGGCAAQYYYIABSU8AY4ABBhhggAEGGGCAgUADih1YbIm348963HnnnSnqz4oVK9I3vvGNdOONN6ZFixalyy67LH3iE59IZ511VjrmmGMGbemWW24J60PV7w984AO11vuv//qvQx//ddddV+vjr+bMm266KfQazj333Nqvwdw/8fp//fXXh/b305/+dO39XbZsWeg18DSxJ7VRmxEGFGVEUWqfCJ2jH85SQV9r165N//M//5M++9nPplNOOSX9zu/8zmDcPv7446GdWLhwYa21XbJkSejjv//++2t9/NV8tnLlytBr+Nu//dvar8G8PPG8/MADD4T2two8dfdj3bp1oddQ9+P3/Sb2qTa9qE0vLqL2iQtuLkYZCF3NduNkDz30ULrooovSvvvum2bOnNnbcSGA5OEQQMxno+azyW4TQPLGWHX0ZPX0b8YgA+MMjLvBIOr4rwlB3pzp/CWpnXusX78+rVq1Kl166aW9HM8CSJ4rAaS5OaGv860AkjfGqqP7asF1mT8aMqCwDRXWZNTDIJe/JLV/j9WrV6fbb789HXfccb0xKYDkuRJArHO565wAkjfGqqNza+x443LgBgAYOACTZkZQyl+SyrnHxo0bt75mpHoRe9fNCyB5rgQQ61zumBdA8sZYdXRujR1vXA7cAAADB2DSHEgAGbuc3nHHHem0007rbO8FkLHd3PXfBRDrXO46J4DselztfERujR1vXA7cAAADB9DZTWgbfdt5weny/2/ZsmXrW/y+5S1v6ZwBASRPngBincudLwWQvDFWHZ1bY8cblwM3AMDAAZg0B/gTkLFL65o1a9JHP/rRtOeee3bGggAytoO7/rsAYp3LXecEkF2Pq52PyK2x443LgRsAYOAAOrPpLKFPOy84ffr/e++9N+23336d8CCA5MkTQKxzufOnAJI3xqqjc2vseONy4AYAGDgAk+bAfwIydpl9+umn05lnnlm8CQFkbNd2/XcBxDqXu84JILseVzsfkVtjxxuXAzcAwMABFL/ZLKk/Oy84ffz/6t2yFi9eXLQLASRPngBincudRwWQvDFWHZ1bY8cblwM3AMDAAZg0/QRk5Er7zW9+M+2///5F+hBARrZswhsFEOtc7jongEw4nCb8h9waO964HLgBAAYOoMgNZqk9mXDl6ek/VK8LOfLII4szIoDkgRNArHO5c6oAkjfGqqNza+x443LgBgAYOACTpp+ATLrSrly5Ms2bN68oJwLIpC0b948CiHUud50TQMYNo13ekFtjxxuXAzcAwMABFLWxLL0Xu1yBenpA9Va9Rx11VDFWBJA8aAKIdS53bhVA8sZYdXRujR1vXA7cAAADB2DS9BOQKa20VQh53eteV4QXAWRKLXv+IAHEOpe7zgkgzw+fKf8lt8aONy4HbgCAgQMoYkPZlR5MeSXq6YGPPvpoOuSQQ1o3I4DkARNArHO5c6wAkjfGqqNza+x443LgBgAYOACTpp+AZK2099xzT3rRi17UqhsBJKtlSQCxzuWucwJI3hgTQIyx3DHm+IzNl2IZYEM3kL8k9fMet9xyiwAyjblzyZIloTAEEHN37twtgOQP0dwaO964HLgBAAYOoNWNZNdqn78k9fMeW7ZsSZdffnlrdvwEJM+VAGKdy51rBZC8MVYdnVtjxxuXAzcAwMABmDQznsnOX5L6e49169alt73tba34EUDyXAkg1rncdU4AyRtj1dG5NXa8cTlwAwAMHIBJUwDJX2mfu8djjz2W9tprr3BDAkheywQQ61zuOieA5I2x6ujcGjveuBy4AQAGDsCkKYDkr7Rj7vEf//Ef4YYEkDENmMJfBRDrXO46J4BMYWDtdEhujR1vXA7cAAADBxC+eexyvXdab/xvSmnz5s3p7W9/e6gjASSPngBincuddwWQvDFWHZ1bY8cblwM3AMDAAZg0/QQkf6Xd6R7V54MceOCBYZYEkJ0asIv/FUCsc7nrnACyi0E14p9za+x443LgBgAYOICwTWMf6jxizXHTcxW47LLLwiwJIHnsBBDrXO78K4DkjbHq6NwaO964HLgBAAYOwKTpJyD5K+2Iezz99NPpqKOOCvEkgIxowCQ3CSDWudx1TgCZZEBN8E+5NXa8cTlwAwAMHEDIhrEvNV62bFmK+nP77bene++9N61atSo99dRTacOGDRMse+XcfM0114R4EkDyei6AWOdy52ABJG+MVUfn1tjxxuXADQAwcAAmzYyfgLRt5fzzz08//OEP06ZNm/JXx4B7rF27NuS1IPfcc0965JFHwv6cd955tY6TK664IuyxV3WqQnPddr/97W+HXsOFF15Y+zXUXZM+fT8BJH/C7FP/XYu9cYABRQ4osoWzQ5v80j3sueee6Ygjjkh/8Rd/kaoNf2lfX/ziF3nnnYGOGxBA8mfW0tcOj89+tzADGlJYQyzcHV+4Iz298pWvTAsXLtz6K1r5y2Uz96h+ZSyyBs5lDmegfgMCSP78yGH9DtW01zXt9cXZCNnMD8LAvHnz0s0335y/YjZ0j7/5m78ZRN0tjtaPvhoQQPInx75acF3muYYMKGxDhbUBE37CDXzqU59Ka9asyV85a75H9ToV48rcykB3DQgg+ZMi7931rnet9K6Vk9qc2Jwz0JCBP/iDP0hPPPFE/upZ4z3Wr1+fDj74YD1uqMcWS+tW0wYEkPwJseme+P7Gfc8MaGjPGmrTZ9O31UDbIeTaa69lkUUGOmpAABFA7I3sjxs2oMANF9gC3NEFuOsuZs+e3eqL05988kn22WegowYEEAGk62ugx1/8/r74B2gB6+gCZvC3P7ZOOeWU1j7AsPo1LAbaN6AHerA7BgQQAWR33LiP+SbDgGJlFMuGShjqnIErr7wyfyWt6R4f//jHO1cv84E1gYEZSQDJnwS5MXcwkGUg62CbCRtwBjpm4Bd+4RfS3Xffnb+a1nCP++67j5eOebGAWhMrAwJI/gRo7Bg7DGQZyDrYZsJmgoEOGnjve9+bNm7cmL+iTvMemzdvTr/xG7/BTAfNWEiHvTYKIPmTnzEz7DGj/9n9z76DzYTNBAMdNLB06dL8FXWa99iyZUuq3hbYxGyeZaBbBgSQ/MmP8W4Z16/W+9X6A7A56eBm1sDt3rg56aSTUhUIor9OP/10Y9wYZ6BjBgSQ/JnSuti9dVHPWu1Zqye3KHVsUTJYuz1evv3tb+evqtO8x6JFi4xz45yBjhkQQPInPutjt9dH/QvvX/gJLUQdW4gMyv6MkT/90z8N/ynIPffcY8wb8wx0zIAAIoBY+/uz9hfaSwUutDEW7I4t2F1wdOihh6ZNmzblr6zTuIcPJDTHdmFseIw7OhVA8ic9hnY0pB7qsQsDCrSLAgkCgkBvDMycOTP9+Mc/zl9Zp3GP6nUn3gnLPGue7ZYBASR/0mO8W8b1q/V+tf4AerO5g5mlLhi44oor8lfWad7j2GOPNc4FeQY6ZEAAyZ/0ujD/e4z2KQUZ0IyCmmGB7tAC3VU3hx9+ePjrQE499VS22WagQwYEEAGkq2ucx92ZfX1nHqjFq0OLlwmg7HG1YcOG/NV1Gvf4wAc+YPwavwx0yIAAkj/hWffKXvf0p7j+FPeALFIdWqQM6G6On7Vr1+avrtO4xznnnGNcG9cMdMiAAJI/4VkPu7ke6ltrfWvtxBajDi1GBmi/xskzzzyTv7pO4x7nnnuu8W68MzANAy9+8YvTGWeckS655JJ09dVXp6VLl6avfe1rafny5Y38Wb169TRGfP5dr7/++tp9rFu3Lv+BTOMe1sl+rZP62Xg/Gz9B7ZMKFHrGwPQMPP3009NYZvPv+tGPfrTWeaB6Tcl73/vesD9z586t9fEfd9xxYY+9qtMxxxxT6+Ovxt+73/3u0GsYyjupvexlL0uf+cxn0m233Zbuv//+tGrVqlQ9YbB58+b8gdehewgg05vTrYnq10EDmtbBptW+mVCDYY2Drv8EJDpALViwoNYxd9ddd4VuDf/rv/6r1sdfzRfRv8Y3f/782q+hhHnvoIMOShdffPHWwPGTn/yk90FjIvgCyLDWoBLGnsfQurnWH0AvFxWwuSrZQPSvJnz4wx+udZwLIBNt40bfLoCUNR9VP+VYuHBhWrFiRYoei6OFtH+rAFKW0ZLXL4+tN1Z6cyG1bnAA56LPBqI/Df1973tfreNTAMnbMAog7c9ns2bNSieffHK69tpr0/r16/MaOICjBZD2jfZ5zXNtRfoq8kHVulkBT48Z2G6gev1E9Ne73vWuWse0AJLXQQFku//ouWD27NnphBNOSPfcc09e0wZ2tADSntHoMeF8ev2cAYUwGBgYkoHq3XOiv974xjcKIGPeAclrQPIFdvE1IK997WvTgw8+ONjXdeR0WQCxDg9pHXatW71DDwIDQzHwwhe+MEX/9KB69545c+YIIAJIzn503LFdCiCHH354+u53v5s2btw47jrcMLoCAoh1eCjrsOt83vrzf6l1g6DA6spAeQaOPvro8GdjH3300drnlugQ5V2wxlv2Lljja/LSl740felLX0pr1qwZvct264QVEEDGe7KGqknPDWhwzxtc++ZPvbo7Zqp33on+WrZsWe0GBZC8LnoNSPNj9pxzzkk//vGP8xrj6OcrIIA0b9TarcaFGdCQwhpS+2bN9TFeGfi1X/u19Nhjjz2/4Ef95bLLLqvdtACS1z0BpLk5YM8990z/8i//kjZs2JDXFEfvUAEBpDmj9gBqW6gBjSm0MbVv2lznsK1/5CMf2WHBj/qfU045pXbLAkhe9wSQZsb+vHnztn5aeV43HD2qAgJIM0at++pasAHNKbg5tW/cXOswve+zzz7p3nvvHbXuN3rbli1b0pve9KbaHQsgeW0TQOod9zNnztz6mR5eZJ7ncLKjBZB6jVrr1bMDBjSpA02qfQPnmofl/pJLLpls7W/s36pfS9l3331r9yuA5LVMAKl3vH/yk59M0R/mmdfx7h0tgNRr1Bqvnh0woEkdaFLtGzjXPBz3hxxySHryySdb2ZHcdtttjdgVQPLaKYDUN96/8IUvCB95/KZ0tABSn1Hru1p2xIBGdaRRjWzkXHv//d95551T2gDUfVD161fHH398I24FkLxuCSD1jPMbbrgh/G2s8zrd3aMFkHqMWtPVsUMGNKtDzWpkM+f6+zsGqmdrqyDQxlf1ORFN2RJA8joqgEx/jH/lK19pbSzldbubRwsg0zfa1Hzr++pNQwYUtqHCNrb58niZnYqBiy++uNVPYq4+E2Eqj3N3jhFA8jaZAsjuzxkvetGL0rXXXit85JHLPloA2X2juzOHuo96F2BAEwpoQmMbNdc2PN977713Ou+881r/PfWLLrqoMdcCSN7+TgDZvXmgGkuf+9znhI88brt1tACye0at8erWYQOa1+HmNbbBU5Nujovq7XarD/5r++upp55q1KYAktdhAWT3xvPHPvaxvEI7ercrIIDsnlFrtbp12IDmdbh5jW7y1KVbY+PlL395uuuuu4p4trZ64XuTfgSQvH2eAJI/ls8666xWf4Uxr8PdP1oAyTfa5Bzre+tHgAFFDihyo5sxj5/h6nMJnnjiiSJ2IZs3b06nnnpqo+YFkLxWCyB5c8SRRx6ZHn/88bwiO3paFRBA8oxa99WrBwY0sQdNbHSzpz5ljpHq05jPPffc9P3vf7+otwb9wQ9+0LhHASRvryeA5I3hu+++O6/Ajp52BQSQPKPWZfXqgQFN7EETG9/wqVE54+T1r399+vu///tUbfTbeovdyXYbCxcubNyjADJZB8b/mwAy9fF79dVXjy+gWxqvgAAydaPWY7XqiQGN7EkjG9/0qVPsWHnFK16Rfv/3fz+95z3vSRdccEG66qqr0ve+973W391qsp3IAw88EOJQAJmsC+P/TQCZ2th9xzvekdatWze+gG5pvAICyNSMWofVqUcGNLNHzQzZ/A25Xo2vwh0/QfValAgfAkgeFAFkauvcT37yk7zCOrq2CgggUzMaMb86h14EGVDooEKHbMxcS7Oea1tte/iN7rvvvjDjAkgeIAFk1/PCZz7zmbyiOrrWCggguzZqfVejnhnQ0J41NGwTOMS61bri9uybVW9bGmVCAMnDI4BMvs4dccQRacOGDXlFdXStFRBAJjcaNbc6jz4EGlDswGKHbdBcUzOua11xe/TNbrrpprTXXnuF+RZA8vAIIJPPB1/72tfyCtry0WvWrEkPPfRQ+s53vpPuuOOORv5EjzEBZHKj1nT16aEBTe1hU8M2gkOrXcv7jiJPv3bt2vRbv/VboeaiN0cLFiyo9fqqD4yM/BJAJl7n/viP/ziyFdnn2rRpU6p+vfGLX/xiuvDCC9Oxxx6b9thjj1o9jprHqzeUiPwSQCY2Oqo/blOvHhjQxB40sfHFSI2eHSeRC3IXzlVtjs4+++xwfwJIng4BZOJ1rqpNiV+PPvpouu6669Jxxx0XPr6q+V4AyVdhnZx4nKmN2owwoCgjitLKhO9xlG8xf0nq9z1WrFjRylgRQPJcCSCj55bTTjstr5ABR69cuTJ99rOfTbNmzWplbG1bhwSQ/GZvq53/jh5v6qIuOxlQkJ0K0uqk77GU7TF/ServPaq3LG3LqwCS50oAGT2v3HzzzXmFbPDon//852nRokWtB49tY1oAyW/2ttr57+jxpi7qspMBBdmpIK1tqjyO8i3mL0n9vEe1Wao+JLEtswJInisBZPzccthhhxXzoZ6PPPJIOuCAA1obT6PGsQCSN8aqo0fV0W3jx56aqMlzBhTCYGBgqgbyl6R+3qN6MexUa9bEcQJInisBZPwcV72ou4SvpUuXple96lWtjqdRY1QAydcxqo5uGz/21ERNnjOgEAYDA1M1kL8k9e8eV155ZeubJQEkz5UAMn6Oq37q0PbXP/7jP6bZs2e3Pp5GzX8CSL6OUXV02/ixpyZq8pwBhTAYGJiqgfwlqV/3qN4qM/LzPibqiwCS50oA2XGO+7M/+7O0ZcuWvCLWfPTll19eZPDYNuYEkPyGb6ud/+443tRDPSYwoDATFKboxcFjbsdt/pLUj3tUm7Wvf/3rac6cOUWMCwEkz5UAsuN8sXz58rwC1nj05s2b01VXXVXEOJpsHRFA8ps+WT39245jUD3UYwYEEDAwdQP5S1L371GFj2rDts8++xSzaRJA8lwJIDuO8cceeyyvgDUeXY2lmTNnFjOWJpr/BZD8pk9US7fvOP7UQz2eM6AQBgMDUzWQvyR1+x7Vs7W33HJLcZslASTPlQCyfY77vd/7vdZ+/erhhx8ubixNNPcJIHljrDp6olq6ffv4Uwu1GGNAMcYUwwQyg4fJPOQvSd2+R/WC8z322KO4cSGA5LkSQLbPa//+7/+eV7yajt6wYUOrb1092bw26t8EkPzGj6qj27aPPbVQi50MKMhOBSlus+XxlWM0f0nq5j2qzdJf/uVfFjsWBJA8VwLI9jlk1apVecWr6ejqDRy6NJcLIPmN71J/Pdbtc4JatFaL1k7cqckYUE4qA0P4qt6e9D3veU/R41MAyZMogGyfv9asWZNXvBqOrl5zcvDBBxc9pnZe4wSQ/MbvXEP/v33cqYVajDCgKCOK0qmFwuOPM5y/JHXnHtWLzW+77bZ06KGHFu9fAMlzJYA8O0fMnTs3Va9riv6qPu+ja/O0AJKvpGs99njj9g5qPbLWI2/s3GSpufoYYSB/SerGPZ588sl0ySWXdOLdeao+CyB5roYo75gAABwjSURBVASQZ+fHSy+9NK9wNRxd/cTlsMMO69yaKoDkNz9iDXIOe50eGdDMHjWzc4tc12qfvySVfY/qtR533HFHevGLX9wpOwJInisB5Nl17lvf+lZe4Wo4+rrrruvU2No2Jwsg+c3fVjv/ta9kYEoGpnRQJydQAPS2bgP5S1KZ96h+3eqnP/1pOuWUUzo5tgWQPFcCyIw0a9astHbt2rzC1XD0GWec0ckxJoDkN7/u9cb3s4fpuQEN7nmDO7n4ldqT/CWpvHtUvxJy5plnpl/6pV/qrA0BJM+VADIjHXTQQeGv/1i3bl16yUte0slxJoDkjbHq6FLXLY/LPrdQAxpTaGNMZgV+Jkn+klTePS688MLO2xJA8lwJIDPS0UcfHf4BhDfffHNnx5oAkjfGqqPtJewnGcgykHWwAVbgphj4OMP5S1J596h+DaUL73Q1mWsBJM+VADIjHX/88eEBZP78+Z1dMwWQvDFWHT3ZnOXf4tZpte5MrTvzQA1u4ad1A/lLUpn3qN4NqMuTtACS50oAmZHe+c53hgaQ6nVWVejp6jgTQPLGWHV0V3vtcdsHt2RA4VsqvMmqg4Eqf0kq8x7VBr6Lbw26bawKIHmuBJAZ6U/+5E/yijbNo6vPGznqqKM6O88LIPkAts1P/mtfycCUDEzpoM5OohDob50G8pekcu9xxRVXdHZcCyB5rgSQGencc8/NK9o0j64CyKte9arOjjEBJB9AnWuN72XvMgADmjyAJnd2ESytN9WzqHX8efjhh/NXt5rvUW3iX/nKV3bShgCSh0EAmZE+8YlP5BVtmkdv2rQpveAFL+jk+KrmXQEkH0Bp65XHY39buAENKrxBnV3A1HXisdXGJzKPWk6vueaaTvoSQEZ1c+LbBJAZ6ZOf/OTEBWrgX6q34O3yHCiA5KPocr899onXa7VprDaNfeNOT77AcdGkgeozOH70ox/lr3A13+Opp55KL3/5yzs3VgWQPAgCSHwAWb9+fefG1dg5TwDJG2PV0WPr5+/2EAzs0sAuDzCoOvhiafDLdx39jOxEy+n111/fuTEugEzUzdG3CyAz0vnnnz+6OA3dWv0K1r777tu5sbVt7RBA8mFsq53/lr/+6lERPSriQXR2koaYn901sN9++6UHH3wwf5Wr+R7/93//lw444IBOjUEBJA+BADIjffjDH84r2jSPrl6E/upXv7pT42rsXCaA5AMYWz9/tzdgYJcGdnlAZydQzdfb0g0sWLAgf5Vr4B5Lly7t1DgXQPIQCCAz0rvf/e68ok3z6CqAzJs3r1Pjaux82YcAsnjx4rRkyZKwP2Pr5+/2Hwzs0sAuD+jsBKr5elu6gV/8xV9MP/3pT6e51Zn+3asN/cte9rLOjHUBJK/nAsiMdOKJJ4Z/EOE73vGOzoypnefKPgSQna/J/9sTMFCUgaIeTGcna6g52l0Db33rW0M3RhNtXW+88cbOjD8BZKIujr5dAJmR3va2t4WPs+onnLs7L7R9PwHEmta2QefvvcHeX2BnFwCDbxg299lnn/Too4+O3jkG3lq9a89rXvOaTowXASQPhgAyI73+9a9P1a9FRX41UfeodUEAGcb6E+XJeXgaYUBRRhSlE5swj7s/dk844YRUvWtO219d+SmIAJInpYmN8Nq1a/MexDSPnj9//rTm5blz54YHkCrU//qv//q0Hndb87wA0p/1pS1DzsvQLgwo0C4K1MnFwzV1z/UPf/jDaW7Rpn/3LVu2pOOOO6548wJIXq8FkBlpzz33TNXn3kR/Ve++1cX5WADp3hrSRWce86CdDfriO7kwGLD9NHv66aenjRs3Ru+Pxp3vhhtuKH5cCCDj2jbpDQLIs3PGihUrJq1TE/940003FT+eRq0pAkg/15lRvXabXrdkQOFbKnwnFyW1ana8fPe7321iD5T1Paufgpx00klF+xRAslqaBJBnx20bH/65YcOG9IY3vKHo8TRqXhdAmp3rR9XcbWo+MAMaPrCGd24hHFJ/qp+CRL9QdtRWtvRnbQWQUV2b+DYBZPs618b4uuaaazo37wog280MaQ1yrfoeaECxA4vduUVIbeLHx7e+9a2Jd5KB//L2t7+9WK8CSB4EAWT7OH7mmWfyilfD0dUL9g844IBix9OoeV4A2W5mVH3cpj4MTNvAtL9BpyZVYPS7dAPvf//7a9jyTP9bLFu2rNixLYDk9VcA2T7vtfXBn7feemux42nUnCiAbDczqj5uUx8Gpm1g2t+gU5MqMPrdBQPLly/P22E2cHT1u+ulvhZEAMlruACyfd679tpr84pX09HVePrQhz7UmfVSANlupgtrhseoXx00oGkdbFpnFjG13b3x9b73va+mbc/0vk3162Al9lAAyeurALJ9HB555JGtfebOqlWr0kte8pIix9TYcf7qV786rV69Og/ZNI++/vrri6/L2Br5+/YxpRZqsZsGFG43C2eynMFOk3aqX4Fq+6v6ILUSXwsigOTJEEB2nKva+jWsqmt333331s8kaXLumM73njNnTvrqV7+aB6yGowWQHY1Op4fuq5YdMaBRHWmUwDOwwHPiiSfWsKxP/1t873vfK86eAJLXVwFkx3XuP//zP/MKWPPRS5YsSfvtt19x42qPPfZIV155Zc1XO7VvJ4DsaNS+RD0GYECTB9Dk4hY6NZ/auPvmN785tdW7waOq310//PDDizIkgOQ1XADZcbydfPLJqfq8mza//umf/ilVP20oZS6cNWtW+ru/+7vWSiKA7Gi0FBceh740aEBxGyxuMYuLa+ym80MOOSRVAaDtrwcffDDNnDmzGM8CSJ4IAWT8+P/Rj36UV8QGjr7uuuuKGVNXX311q6FMABlv1LqtJj03oME9b3AxC5w654+16lnJFStWNLD1yfuW1Ye3HXPMMcVYEkDy+ieAjB97V111VV4RGzq6ek3IS1/60lbH1n//93+3/gGoAsh4o9ZMNem5AQ3ueYNbXdjUdvrj6+CDD07VB5m1/XX//fenvffeuwhPAkieBgFk/Disfq2wjQ8lHNW56t2x5s+fHz623vzmN6eHH3541EMKv00AGW/U+qkmPTegwT1vcPiipp71j6kS3hGr2pGcddZZRXgSQPL2hwLI6DFZbXpL+dq4cWOqXhw/d+7cxsfYb/7mb6bq2qtzlvIlgIw2aj1Vlx4b0NweN7fxhUztYsbPYYcdltasWdP6XuH73/9+mj17duuuBJA8CgLI6HH6rne9q7XPBJmog9VPOxcvXpze9KY31T7OjjjiiFS9+P1nP/vZRKdv7XYBZLRRa6y69NiA5va4ubUvYGrV3ni59dZbW9scjD3x+eef37orAWRsR3b9dwFk4nG7dOnSXRewhSOqN5+46aab0gc/+MFp/VTkl3/5l9P73//+9OUvf7mIX+WcqJRNBJCjjz46Vb9mFvXH+jjxOFMbtRlhQFFGFKX1DZbHxOXOBl7zmtekn//85xOt32G3Vz8Fafu1IAJIXrsFkInnk+OPP771F2DvqpvVr0pVLxT/8z//8/TWt741/e7v/m6qPq38V37lV7auVdU71FWfK1L9+lb1Se/HHntsOuecc9LXv/71on7NarLrbCKArFu3brJT1v5vO8/Z/n/icac2ajMDAggY6I6BG264ofZFc3e+4V/91V+1GtIFkLyuCSATj/Fq8169BW1XvqrPL9m0adPWYFH9lGTsnyqoVP9WvWtd174EkImNWqPVpqcGNLanjW11g6imzYyrN7zhDWn9+vWt7y3uu+++Vn0JIHkEBJDJx2P108USPm8nr6v9OloAmdyoNVV9emhAU3vY1FY3h+rZ7Jj6whe+0PrOo3oW9qKLLmrNmQCSR0AA2fWYvOCCCzr5k4M8CeUeLYDs2qi1VY16ZkBDe9bQ1jaF6hgzlt7ylrcU8VOQhx56qDVrAkjeRlIAmdrYrF7f5KudCgggUzNqnVWnHhnQzB41s7UNoRrGjqN//ud/bmeXMOas1U9BPv7xj7diTgAZ04gp/FUAmdr4/O3f/u0UbWsK7RvEIQLI1Ixaa9WpRwY0s0fNbGUzqH7xY+ioo44q4t1tHnvssVbMRW8SFyxYUOt13nXXXaGbSgFk6mP0c5/7nF/FCtX57MkEkKkbteaqVU8MaGRPGlnrBklNyh8Xn//851vYJow/5ZlnnhluTwAZ34fJbhFA8sbz8uXLJyunf2ugAgJInlFrtHr1wIAm9qCJ4RtANWt/3FSfAVB9anLbX9V77b/iFa8INSiA5HVdAMkbr/vvv3+qXuPkK64CAkieUWuwevXAgCb2oImhmz/1KmfMXHnllXE7hEnOFP3p6ALIJM0Y8U8CSP6YPfHEE4v44M8R7ezlTQJIvlFrsZp13IAGdryBwseMYRt+4oknWt+QrF69Oh188MFhFgWQvJYLILs3R5x99tmperMFX81XQADZPaP2L+rWYQOa1+HmhW341KjccbJo0aLmdwdTOMOll14a5lEAmUJDxhwigOz++P3Upz7lReljLDX1VwFk941an9WuowY0rqONC9vsqU/5Y2TVqlVN7Qum/H1/9rOfpde+9rUhLgWQKbdl64ECyPTGcPWGD5s2bcoruqOzKiCATM+odVr9OmhA0zrYtJBNnrp0Z2yU8Lkg1W6j2qhFuBFAsvZ2SQCZ/lj+13/9VyEkj13W0QLI9I1GzL3OoU81GlDMGosZsvnyeJkdZWDlypVZC34TB1evR5kzZ07j40AAyeueAFLPnFEFbK8JybM31aMFkHqMjlob3Ka2hRrQmEIb0/gmznX3y/4//MM/THWtb/S4xYsXN25XAMlroQBS31hfuHChEJLHb0pHCyD1GbW2q2VHDGhURxrV+KZOHbo9Fg488MBUwk9BqteC7LXXXo16FUCmtKd7/iABpN6xfc4553iL3ud11fMXAaReo9Zz9eyAAU3qQJMa3cy5/v6MgUsuuaSe3cA0v8uSJUsaNSuA5DVIAKl/jP/RH/2RDyvMYzjp0QJI/Uat7WpauAENKrxBjW7kXHu//M+dOzdVP4Fo++vJJ59ML3zhCxuzK4DkdVgAaWac/+qv/mpavny5t+nN4zjyaAGkGaPWeHUt2IDmFNycxjZwrrm/7j/ykY+MXOCjb2zypyACSF43BZBmx/vll1+eok3mCSj/aAGkWaPWfPUt0ICmFNgUwWPgn24+HZP7779/EZuhNWvWpBe84AWNWI7e7C1YsKDW67jrrrtCd4QCSPPr3Ote97r0v//7v34aspuyBZDmjU5nXXFf/WnAgKI2UNRaNyseH6O5BqrfTy/h7UKrz07IfexTOV4AydvlCSBxc8h5552XNm7cmNcgRycBJM7oVOZYx+hHgAFFDihyI5swj5vdyQw8/vjjrW9rqk+PnjdvXu3+BZC81gogsXPFIYcckq6++mo/DclgKoDEGp1s7fBvehFkQKGDCl37JszjZncyA2effXZvfwoigGTs7FLySegt/UrnCSeckL7yla/4icgUuAog1rPJ1jP/1ksfvbwom/2WFlyTRFnj6cEHH5zC0t/sIdWvoxx//PG1jkkBJK9nfgLS7rh85zvfmW644Yb0zDPP5DVuQEcLIO0atXarfwsGFL2Fote6GfP4GZ7IwPnnn1/Er4H827/9W63mBZC8nakAUsYcceihh6ZFixalBx54oIhxubOi6nVjjzzySPrSl76UHn744Z3/udH/F0DKMDrRWuJ2/WnAgKI2UNRaN1seH6PTMfCDH/yg0Y3DVL559VqQOn8KIoBMperbjxFAyptDqjeKuPXWW9PKlStb/RWtzZs3p+r1YtXnmZxxxhnPr11VSIr8EkDKMzqddcd99XMKBhRpCkV6flJ2LC9dM3DBBRcU8VqQr371q7WNIwEkb2sogJQ9bx122GFp8eLFadWqVal6++oqsDf1VQWOtWvXbg0dX/7yl1N17lFzmgCS34FRdXRb2WNPf1rtT6snHznxAaEnDNRr4Dvf+U7+alrzPapN1dFHH13LmBdA8pojgNQ7npqan2bOnJlmz56d5syZky688MKtP5VYvXp1XrNHHF2Fmttvvz1dfPHF6cADD0z77LNPqs412XUIICMKuYubJqunf+vGGNSn0D6FnmzSCU/j9YKBZgxUn01QwlddG2EBJK+bddV97PisnkWP/Jo/f/4g14899tgjzZ07N73xjW9M1btqnXrqqem0005LH/rQh9LHPvaxtHDhwq1/qtd7ffCDH0ynn3761mP+8A//ML35zW9OBx10UJo1a1Z27QSQfN1jx4e/N7OWqWuv6tqri8meZGHW/yEY2HvvvdONN96Y7rzzzlb/fOMb36hljN52222h13HyySfX8ri3WateiBzZi89//vO1Pv7qOpYtWxZ6DSeddFLt17CtH/47fh2oXpMRafTTn/507f2NNsrReEdqoiaTGFCcSYpT+4ToXLwxwAADDDDAAAMMDNwAAAMHIGT5zBQGGGCAAQYYYICBSAMCiADCAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwyEGQg7UWSqci4pngEGGGCAAQYYYICBMg0IINIuAwwwwAADDDDAAAMMhBkIO5EEWmYC1Rd9YYABBhhggAEGGIg0IIBIuwwwwAADDDDAAAMMMBBmIOxEkanKuaR4BhhggAEGGGCAAQbKNCCASLsMMMAAAwwwwAADDDAQZiDsRBJomQlUX/SFAQYYYIABBhhgINKAACLtMsAAAwwwwAADDDDAQJiBsBNFpirnkuIZYIABBhhggAEGGCjTgAAi7TLAAAMMMMAAAwwwwECYgbATSaBlJlB90RcGGGCAAQYYYICBSAMCiLTLAAMMMMAAAwwwwAADYQbCThSZqpxLimeAAQYYYIABBhhgoEwDAoi0ywADDDDAAAMMMMAAA2EGwk4kgZaZQPVFXxhggAEGGGCAAQYiDQgg0i4DDDDAAAMMMMAAAwzEGPh/pnjvSmfimRAAAAAASUVORK5CYII="}}},{"metadata":{},"cell_type":"markdown","source":" # References\nFirst and foremost I want to acknowlegde the notebooks which I have taken the inspiration from to create this notebook. If you consider upvoting this notebook , please upvote these as well. I would highly encourage you to read them to get the gist of the problem :\n \n https://www.kaggle.com/erikbruin/riiid-comprehensive-eda-baseline\n \n https://www.kaggle.com/isaienkov/riiid-answer-correctness-prediction-eda-modeling\n \n https://www.kaggle.com/carlmcbrideellis/riiid-eda-and-feature-importance"},{"metadata":{},"cell_type":"markdown","source":"# Basic EDA and Insights so far"},{"metadata":{},"cell_type":"markdown","source":" ## Small Summary of Data \nWe have been given `Train.csv` which contains the data which is in relation to the user , `questions.csv` contains all the data with regards to the questions which are posed to the user , `lectures.csv` contains all the data with regards to lectures , in the `test data` there seems to be the same data as train.csv but it includes two more rows `prior_group_responses` and `prior_group_answers_correct`i.e it contains data of user with regards to his previous performance and how many questions he has responded correctly . Thus the task is to predict how a user would fare to a new question when we have been provided his past data.\n\nFor detailed information of the csvs: https://www.kaggle.com/c/riiid-test-answer-prediction/data"},{"metadata":{},"cell_type":"markdown","source":"We import the required libraries and use the pickle which had been generously created by @Vopani"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install pandas-bokeh","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom bokeh.io import output_file,show,output_notebook,push_notebook\nimport pandas_bokeh\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource,HoverTool,CategoricalColorMapper\nfrom bokeh.layouts import row,column,gridplot\nfrom bokeh.models.widgets import Tabs,Panel,DataTable, TableColumn\nfrom bokeh.palettes import Spectral3\noutput_notebook()\npandas_bokeh.output_notebook()\npd.set_option('plotting.backend', 'pandas_bokeh')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## *Loading Train Data*"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\n\ntrain = pd.read_pickle(\"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\")\n\nprint(\"Train size:\", train.shape)\np=train.head(30)\np.style.set_properties(**{'background-color': 'black',\n 'color': 'lawngreen',\n 'border-color': 'white'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"train['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('bool')\n\ntrain.memory_usage(deep=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My insight on this : \nSo one of thing I have noticed is that the values are in multiples of each other , if you look carefully. When we just look at the compelete dataframe , while unique values might be different, row_id is 2 times user_id , which might show that at two transactions were taken per user atleast.So let's check the different hypothesis which can be made from the same ."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Total unique users are {train.user_id.nunique()}\")\nprint(f\"However the total interactions which seem to be recorded are {len(train.user_id)}\")\nx=len(train.user_id)/train.user_id.nunique()\nprint(\"On average the interactions of a user should be\",x) \nprint(train.user_id.value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data shows how some users have way more interactions while studying , while some users have quite less interactions. The data is skewed , some questions which I am now thinking about are :\n1.) Our predictions will vary with regards to how much user has already interacted with platform , but one hypotheses which should be tested with regards if he faces a new topic or new user interacts with the question to check the prediction should we keep the threshold with regards to how new users performed."},{"metadata":{"trusted":true},"cell_type":"code","source":"Users=pd.DataFrame(train.user_id.value_counts())\nUsers=Users[:10]\nUsers.head()\nUsers=Users.rename(columns={\"user_id\" :\"Interactions\"})\nfrom bokeh.palettes import RdYlGn\nUsers['color'] = RdYlGn[len(Users)]\np_bar = Users.plot_bokeh.bar(\n ylabel=\"No of Interaction\", \n title=\"Top 10 Most Interactive Users\",\n alpha=0.6,fill_color='color',line_color=\"white\",fontsize_title=\"18pt\",fontsize_label=\"16pt\",figsize=(1200,800),fontsize_ticks=\"14pt\",hatch_pattern=\"vertical_wave\",hatch_color=\"white\"\n )\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(sum(train.user_id.value_counts() == 1))\nprint(sum(train.user_id.value_counts() <= 258))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So only 87 users are it seems having only one interaction, and more than 320000 users are having less than what was supposed average amount of interactions , so should all the values above the threshold i.e some 10.000 values be considered as outliers !! "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"x=train.describe()\n# Import seaborn library \ny=x.drop(['count'])\nimport seaborn as sns \ncm = sns.light_palette(\"blue\", as_cmap=True) \ny.style.background_gradient(cmap=cm).set_precision(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at train describe, we can see how almost 63% percent of questions are correctly answered. So , if we can try to keep predictions for unknown target at a baseline of 0.6 ."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n\nquestions = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nexample_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')\nl=example_sample_submission.head()\nl.style.set_properties(**{'background-color': 'black',\n 'color': 'lawngreen',\n 'border-color': 'white'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Checking Question Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"e=questions.head()\ne.style.set_properties(**{'background-color': 'black',\n 'color': 'lawngreen',\n 'border-color': 'white'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"We have {questions.question_id.nunique()} questions for training\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(questions.bundle_id.value_counts())\nprint(f\"We have {questions.question_id.nunique()} questions for training which are divided in {questions.bundle_id.nunique()} bundles\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bundle=pd.DataFrame(questions.bundle_id.value_counts())\nbundle=bundle[:15]\nbundle.head()\nbundle=bundle.rename(columns={\"bundle_id\" :\"Times used\"})\nfrom bokeh.palettes import Category20c\nbundle['color'] = Category20c[len(bundle)]\n\np_bar = bundle.plot_bokeh.bar(\n ylabel=\"No of Interaction\", \n title=\"Top 15 Bundles Used\",\n alpha=0.6,fill_color='color',line_color=\"white\",fontsize_title=\"18pt\",fontsize_label=\"16pt\",figsize=(1200,800),fontsize_ticks=\"14pt\",hatch_pattern=\"vertical_wave\",hatch_color=\"white\"\n )\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Unique bundles of tags {questions.tags.nunique()} which are given to question\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hi, I have taken this code directly from Erik's notebook ."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"questions['tags'] = questions['tags'].astype(str)\n\ntags = [x.split() for x in questions[questions.tags != \"nan\"].tags.values]\ntags = [item for elem in tags for item in elem]\ntags = set(tags)\ntags = list(tags)\nprint(f'There are total {len(tags)} different tags')\nprint","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So there are 1519 different bundles of tags which are made with 188 tags , we can try to find which tags are more related to each other perhaps as use tags correlation to understand the the question. I have to furthur explore this aspect . Just looking at head you can see how 9, 10 are more related and even 131 and 162 seem to be more correlated "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(questions.tags[:50])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note : I still have to also check with regards to Part , while related to TOEIC , it also seems to be important "},{"metadata":{},"cell_type":"markdown","source":"# Checking Lectures Data "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"w=lectures.head(5)\nw.style.set_properties(**{'background-color': 'black',\n 'color': 'lawngreen',\n 'border-color': 'white'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"We have {lectures.lecture_id.nunique()} different lectures\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"lectures['tag'] = lectures['tag'].astype(str)\n\ntag = [x.split() for x in lectures[lectures.tag != \"nan\"].tag.values]\ntag = [item for elem in tag for item in elem]\ntag = set(tag)\ntag = list(tag)\nprint(f'There are total {len(tag)} different tags')\nprint(tag)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thus , it seems that there are 151 lectures tags and 188 different question tags.Moreover the tags on the lectures seem to be unrelated , one lecture seems to have one tag which is very interesting ."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(lectures.tag.value_counts())\ny=len(lectures.tag)/lectures.tag.nunique()\nprint(\"We can check the average to see that how much some lecture are overused\", y )\nprint(sum(lectures.tag.value_counts() == 1))\nprint(sum(lectures.tag.value_counts() >= 2.7))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"z=pd.DataFrame(lectures.type_of.value_counts())\nz.style.background_gradient(cmap='Greens_r').set_precision(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyzing Time-series Data\n \nWell , one of the most interesting findings which can be explored with respect to the competition seems to be analyzing the time-series data and even getting the different properties of the question which can be helped in creating the questions .\n "},{"metadata":{},"cell_type":"markdown","source":"In the description, timestamp is decribed as the time in milliseconds between this user interaction and the first event completion from that user.\nSo perhaps it could mean that user is required to do a event(i.e lecture or answers questions when he first joins the site.)"},{"metadata":{"trusted":true},"cell_type":"code","source":"time=train.timestamp\ntime.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_questions=train[[\"prior_question_elapsed_time\" ,\"prior_question_had_explanation\"]]\nprior_questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_questions.prior_question_had_explanation.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from math import pi\nfrom bokeh.io import output_file, show\nfrom bokeh.palettes import RdYlBu,Viridis\nfrom bokeh.plotting import figure\nfrom bokeh.transform import cumsum\nz = {\n \"Yes\": 89685560,\n \"No\":11544772\n}\n\ndata = pd.Series(z).reset_index(name='value').rename(columns={'index':'country'})\ndata['angle'] = data['value']/data['value'].sum() * 2*pi\ndata['color'] = [\"Green\",\"Darkred\"]\n\np = figure(plot_height=350, title=\"Previous Questions had explanations\", toolbar_location=None,\n tools=\"hover\", tooltips=\"@country: @value\", x_range=(-0.5, 1.0),background_fill_color=\"beige\")\n\np.wedge(x=0, y=1, radius=0.4,\n start_angle=cumsum('angle', include_zero=True), end_angle=cumsum('angle'),\n line_color=\"white\", fill_color='color', legend_field='country', source=data)\n\np.axis.axis_label=None\np.axis.visible=False\np.grid.grid_line_color = None\np.title.text_color = \"midnightblue\"\np.title.text_font = \"times\"\np.title.text_font_style = \"italic\"\np.title.text_font_size = \"18pt\"\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pq=pd.DataFrame(prior_questions.prior_question_elapsed_time.value_counts())\npl=pq.head(15)\npx=pl.sort_index()\npx=px.rename(columns={\"Time_elapsed\" :\"Value\"})\nfrom bokeh.palettes import Category20c\npx['color'] = Category20c[len(px)]\n\np_bar = px.plot_bokeh.bar(\n ylabel=\"No of Interaction\", \n title=\"Top 10 Time Values Used\",\n alpha=0.6,fill_color='color',line_color=\"white\",fontsize_title=\"18pt\",fontsize_label=\"16pt\",figsize=(1200,800),fontsize_ticks=\"14pt\",hatch_pattern=\"vertical_wave\",hatch_color=\"white\"\n )\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One of the interesesting thing about this graphs is that perhpaes on medain people take 17 milliseconds to answer the question , whereas I was expecting a downward sloping curver forthe the questions , however a lot of questions take higher time on average and how data is skewed right shows that ."},{"metadata":{},"cell_type":"markdown","source":"### Fin\n\nWhile this is a quick eda , there still seems to be many areas for inspecting , feature tag can be checked for correlation with question and even with respect to other tags to understand what kind of relations do they have .\nWhile lectures tags are a bit different , these tags seem to be the kind of topics , however lectures having different tags is a bit unexpected . \n### If you liked the notebook, please upvote , but more importantly please drop down suggestions/critiques, I would love to improve on those areas !! "},{"metadata":{},"cell_type":"markdown","source":"Link for my modelling notebook:\nhttps://www.kaggle.com/sahilmaheshwari/making-ensemble-of-catboost-and-lgbm"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}