{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## H&M Personalized Fashion Recommendations\n## Provide product recommendations based on previous purchases\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31254/logos/header.png?t=2021-10-26-22-52-52\")","metadata":{}},{"cell_type":"markdown","source":"## Data Description\nFor this challenge you are given the purchase history of customers across time, along with supporting metadata. Your challenge is to predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customer who did not make any purchase during that time are excluded from the scoring.\n\n## Files\nimages/ - a folder of images corresponding to each article_id; images are placed in subfolders starting with the first three digits of the article_id; note, not all article_id values have a corresponding image.\n\narticles.csv - detailed metadata for each article_id available for purchase\n\ncustomers.csv - metadata for each customer_id in dataset\n\nsample_submission.csv - a sample submission file in the correct format\n\ntransactions_train.csv - the training data, consisting of the purchases each customer for each date, as well as additional information. Duplicate rows correspond to multiple purchases of the same item. Your task is to predict the article_ids each customer will purchase during the 7-day period immediately after the training data period.\n\nNOTE: You must make predictions for all customer_id values found in the sample submission. All customers who made purchases during the test period are scored, regardless of whether they had purchase history in the training data.","metadata":{}},{"cell_type":"markdown","source":"## Evaluation\n\nSubmissions are evaluated according to the Mean Average Precision @ 12 (MAP@12):\n\n![image.png](attachment:b4d47fa4-6914-4c3f-863c-52980d22431e.png)\n\nNotes:\n\nYou will be making purchase predictions for all customer_id values provided, regardless of whether these customers made purchases in the training data.\nCustomer that did not make any purchase during test period are excluded from the scoring.\nThere is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.","metadata":{},"attachments":{"b4d47fa4-6914-4c3f-863c-52980d22431e.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAV8AAABMCAIAAABF4iNXAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAAAASdEVYdFNvZnR3YXJlAEdyZWVuc2hvdF5VCAUAABpvSURBVHhe7Z13VBVH+8d/OfEcT97kjTHGWBM19qhRY6JiNLF37AULSrOjKIJJUCPEgiIg1mDDLkoRUcSu2BHEhhgsKPZCggWxRePvE2Zzz43shYt4917zzucPDzs7uzszO8/3+T6X4v+9kEgkEjWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkgkEnWkOkhenQcPHiQlJQUFBSnHr4nU1NS1a9eePn1aOZaYCakOklfk+fPnFy5c2LhxY3h4uNL0mnj48OGqVatWrFhx7949pUliDqQ6SFTIzMwk5ocPH64cq0GfmJiYhQsXxsXFKU05cunSpZ07d/r7+9vY2CxatCgtLY3GI0eOzJ49e/To0V27drW1tV2+fLnojCXx8vJKSEgQhxKzINXhDSMlJWXs2LGdOnUaOnRodHT0/fv39+zZM2rUKCcnp9WrVyud8g2Rf+rUqQMHDijHaty4cWPJkiXE+c2bN5WmHLl+/fqWLVumTZvWqlWr+fPnC3VAGlxdXVEZZuHg4NC7d2/R+c6dO+jO3LlzxaHELEh1eMPIyMiYN29e9+7dly1bRnw+ffoUe08ULViw4PLly0qn/PHo0SPuTDDzhdKkRmJiop+f35o1a1AopSlHqBewD+vWrWvfvn1gYKBQB+aCcTh27Ni1a9dQva+++kp0Zl6bNm0aMGCAOJSYBakObxjPnj2bPHnywIEDdX7+ypUrwcHBHP7555+iJWcWL148aNCgnj17ent7z5gxo0uXLiNHjkRc+HfcuHGnT58+evQocsNTjh8/Tn9kCJ+CHnl4eJDP+drFxeXMmTP79u2jz/r164WIGOqW9UyF9PT0bdu2dezYUacOJ0+ejI+Px30wCzc3t7Zt24qezGXXrl1t2rQRhxKzINXhDQPvgDS4u7sTTqKFACOBk5bFYa4EBQUJUSC8Y2Nje/Xq9f3331+8eHHOnDmENOoQFha2du1aW1vb/fv305+w9/Hx6dGjBxeePXsWv9C/f/9ff/2VMuHnn3+Oiop68uRJDt2ynqmQXR0EDx482Lx5M5rl6+urNL14sXv37gYNGigHEnMg1eENIyUlhfxMHN69e5fD58+f79y5E7tOoS465Aq5eurUqSNGjOCSw4cP9+nTZ8WKFdj+6dOnk70J7BMnTsyePdve3l5k/qSkJFwGkoT/v3Xr1sSJE4cMGUI5gy54enri/6kCcuiW9UwFVXVAGpKTkydMmLBq1apz586JRkAdrKyslAOJOZDq8IYRExPTqVOn5cuXP378GPvNvyEhIaRxvlB65Ab5nFKfSOYSbELfvn3xCMQtehEQEIA0oBoENlXGtWvX6H/79m1UA3/BIc4Cf/HTTz8lJiaiSniHDRs2CO9gqFvWMxWyqwNVCYqwdOnSyMhITBB2RvQE1KF58+bKgcQcSHV4w1iyZAnufevWrc+ePcM4/P7774sXLz548KCRHzoAWkDwUzsQ1TNnznR2diY+qVP69eu3aNGi0NBQDjEUCxYs4JD+HA4ePBg3gWrs2LGDbrgMhnHkyBE8CP0zMzNz6Jb1zL+gW2pqKlrWsmVLvI/4YSfkg2rCxcVl9erV1CaTJk0Snf/44w/miHKJQ4lZkOqgBXFxcaRHLy+v2rVrY6Ex3sqJvEP4OTk5xcbGog4kf8oE7nz+/HnltBGsX79+2LBh3IHLx48fj0BcvXqVgsXOzs7V1ZUi5dSpUxQIWImIiAj6M3iMAO2UANHR0ba2tggKYX/p0iUsAF+gUDl0Q4wI8m+//Rb5QMgcHR2rVauGfWAduAppqFmzZrFixT755JPy5ctbW1v/NcSscmPlypU6sZCYBakOWrB9+3YSKXu9SJEiHh4eRv6AgCrU+cQ2cUs8Hz16lFod4yA+g9CYe/fuRUVFIRCoidKkxo0bN4hzpEc5No4LFy5MmTKF4kI5lpgDqQ5aQHa9ePEiRXXhwoWpzPOjDphzf3//H3/8ccaMGbNmzSJXX79+HR+hnNYQzD+6EBwcjEYoTWo8ffo0ISHB29tbOTYCSh7uiUvKj8mS5B+pDhpBLFFpf/TRR/lUB4sC1cO5IFLKsRqZmZlbt24NDQ1Vjo2A9Zk7dy5Fk/EfpkhMgVQHjfhXqoPk341UB42Q6iB545DqoBF5VYewsLCvv/763SyqVq1qbW3dv39/u39CS79+/Wxtbfv27du9e/fWrVvXq1evSpUqpUqVKlSoUMGCBStUqLB8+XLjf1BKFcsZiURjpDpoRF7V4cyZMwMHDvz4448LFChAaPn5+cXGxiYZ4NSpU8ePHz98+PCuXbtCQkK8vLw6duxYo0aNsmXLuri48Fzlpq+E5YxEojFSHTQir+rw+PHjffv2kYT/85//fPDBB3369ImPj3/y5EmuH9TxoIyMjJSUlMDAwHbt2nXu3DkyMlI590pYzkgkGiPVQSOePn2aJ3WAu3fv4uorVaqEMy9ZsqSrq6v4lW3ldG48e/YsISGBqyZOnKg0vSqWMxKJlmitDrdv3167dm2LFi1wnv87fxfs/v37ly9f3rFjR+HChQcPHnzs2LHr168r5wxD7r1y5cqUKVM++eST9957jzJ+0aJFt27dev78udIjN0j7cXFx+f+JQ8sZyStw586d3bt3jxs3Tjk2AMPz9/dfs2ZNWlqaWX5+JE9oM6nc1YG8sX79+i5duhQvXrxo0aLOzs5EuHIui0ePHu3du5eKlLMNGjSwtbVVTvwTjCjZBlPq6OhYsWLFdevWiZ/AFYjs1Lt3b2rUUqVKtWzZskePHr169cKONm3a1NPT88iRI6Kn2KkBAQFDhw51cHDgboMGDZozZ06efppYYw4cODB58mSGSmixRKNHj+adKedy5OHDh+fOnXNycmLx33///W+++SY6Ojo9Pd34HwR48ODBS79G/WpYzkjA+N3CsFl8VOngwYOiBeOGfStdujQ+6OzZs6IRfvvtt5iYmPHjx0dERORJ+EwK8cKQyKZly5ZVmjScVO7qQEhjI0NDQ+vVq1e5cuW2bdvq5z02x7Vr17y9vcuUKdOoUSO6JScnK+f+CdLFWClB69SpU6xYsaVLl+r/JBxPYYZIRpMmTXhKSEjI0aNHT506xb/Tp09ndUaOHCm60ThmzJjNmzdj1C9evJiamsoX6BeulZws7gaopuqvNly4cCE4ONjNzY0Q7d+/P49jx+fpZ3W0hOUl6+KzKPuJSSr/vn37Hj582Jiy//ViOSMBI3cLA0PRSCRz584losS1JLNDhw4xi5o1a+r/UQwih429YsUK8p/lGFsyKLNr1qzZl19+KVq0nJRRlQU7IDIysl+/foySYOPxug2RmZnJUz08PChHhw0blpSUpLpXaBQTYJSoDC7jl19+ecld0ycqKor33bhxY/SPhzI3wJCTqXCz9OHRM2fO3LBhA5KEfG7atGn79u3siatXr1KwoAK6Pyhg6Fcb0JEBAwbMmDEDKUGYf/jhB0StT58+4qwFwrJQmCxbtqx69epvv/12iRIlxo4dy3y1d7+WMxIwZrdgM8kEJFX9tAFsGBSEl67bFQK2KO5m4MCBuA92stJqVtBBXBJztLe3Fy1aTip3dRCBTZ1JciZdUBQg0mJDcIoX4+PjQ4pGHWbNmiX+IkB2mCRagNr9/PPPiAjegfjU/XUj4Fa82gULFnz11Vd2dnZIJi2inarp66+/Ll++PF9jB3gK80cOra2tp02bRgslBk6SkbBxuYO4oaFfbUBckFj0gqEyr40bNyJ5lNPirGXCal++fBmDxiJTmxCc8+bNY4JsFKWHVljISIzZLRxSApMw2HKY1qzrFHjppGKSB5URhoiSR/wSOrOjJ3mLe77GP+GbHzIyMihCraysGJVo0XJSuasDd6fMIw5Jzlh6AmnPnj08njeBYK9cuZJ2xvrpp59igVR/3IWe3GHv3r3oC3Pw8/Mj7aBk+n84iKcwHww/3oTlYFFo5EJkhYfWqlWLIhMfRZxv3br1+PHj7M7FixfzOJYgNjaWdIH0oAUODg7ihsCWzf5tAmSle/fuWAa0iZvjjbt27VqoUCFx1mJh7omJiZRCH3/8MXr33XffbdmyhemLqNASSxiJMbuFFrLrkCFDKGbZqFnX/QXdCJVy5crhLrkJKZcEs23bNnFK/FaIra3tjz/+KPpnh6KVDEdCJrsoTVkVK2pF7lGOs0EJTIcOHTrgnadOncodKNLF39HEkWGr2ZPDhw+nJiLbkTvFVeRUWrBI7G3RYqJJqZK7OrAEuEfx67S+vr6kfRwdz6MdE4EaMe7BgwfXqFGD+Ec1lMv0IErPnz/PwiElrCnjLlWq1Et/kpQLb9y40bFjRwonzD+JPS0tjRUPDw9v3rw5E2PalFuoJgXF7NmzS5cufeLECXYJkDo+++wz7MDBgwc7deqk3NGAOvAFF3IrFpcO3BYrUaZMGXHWYuEdEwMxMTE4HbSMsCQ+ExIS9DeoNljCSIzZLXSjAiVbhIaGMmZxIZBakRXMBVdFR0ePHj2aw/r164uz3JnwGzRokP5GeglS8bp16whmNhKzZgdyqzlz5rAtU1JSlE7ZYEhLlixZuHAhqorPZ6jiU1VMEKuHWDDs/fv3ExSEia7URYlIZj179iQGRUt+JtW5c2fRYiS5q8O9e/fIFZQPZOz58+cTbKRfAvLWrVssB5YedWjXrl2rVq2oapiqcpkeJBYsA/6C+6CFVE1EIzNEXJQeWQ6KteZlf/755z/99BPlAzdHYpkkjgNh4pUwAEQKdaDcrVSpkvgAXGxWnAs9EWCMgLghqKoDjSwW0sBL5Z4BAQHsJxZOnLVkGDCiTPWEgSxYsCCbbPz48YSE/i7RBrOPxJjdQjfsJNsMayOuEpAYevToQeRERkZyIcFGPm/atKk4y/a4dOkSG6aB4T94Sx/EiFDHxsfHxxO3fI06sP9ZGaVTNqjNMbksWpEiRSIiItjM9vb2zs7OzAV3gAkiLrgtYcVZ3Z/qJ/u2adOG8ejE10STUiV3dWDEDDcoKIgyngkULVqUN4FNQhQQCyazefNmdom7u7v+h6UCtgvrhc8hsRPS9MTh8BYpT5BDFkjpl6XHq1atIuZZC1wG1SxOiQVFcSkBxGezyMG4cePQVwbwknfg0NXVldW3sbERNwRVdRCgYtg5thEuDqEVqeYlEDVcEhM3BCaIMSu9NYFh8xa8vLyQ13feeYfdQJmN0mkvEOYdiTG7BUhIZGksrTgUkF0YLaVov3792FG4Wra3MOHAniHa2a516tQRLaqw66j/eS6bhypA/KV/rlVOq4GtQEDRMlaMQeL/yZQ4IPIrg6levTqJilngjnEloo5gPenfsmVLpiZuAqabVHZyVwe8Cnq8a9culmPTpk0U/JMmTSIOKfvZH0Qmq1O5cmWsDjqiXPM3DAvrsXz5ctSOKTFtAhXXRGVhbW3NTXReg9GPGDECo8WKUz2qbjJRaiIxOLHixYvrPnfAMjCqUaNGIRzUOEpvw+rAzbkK58LSMxfeBPdRzunBMJgRL9IQjPmlb7vo0z6PKJflBsaH/cQmYF6NGjUKCQkxizqAGUdizG4BEhK7jshRjrMg91apUsXKyorUSkByKxp1d2DPsOGJ9lq1aomWHCA/OTk5EXI8QidJhmBxyPCUD7xr8TENmx8oUphIkyZN2M84C9InIiK+9UjqFT/Ewf11IzT1pPTJRR3QSDSP2EaNCE4USxQFvBXEif1B1DHWatWqsVJkY+WyvyEIKXgQF/EHzpADAhghxBCyHDt27GDcdOMpCF7jxo0RP2oQQy+bpaRs8/f3Jyb5glqG2GZNqXvr1q1LFYrT032cA4bUgWEnJyeziLhBih30W9/F6MinOpgIFoHnMv2GDRuSNvna0HKZGnONxMjdAqg/Fl3kYR3Es9h7VP5kbDwgG1tne0UgEYHYYdGiCnMnL7KTXVxcPD09KazYUQxMOW0AQoBHkyOpJpSmFy/InZRIhO7du3cRBTYnG09ozeHDh1u0aIG9ZRvj1kV/000qO7moA8+gsiIhsxY84+TJk0gUb2XMmDEYB9aIDNy6dWvMD4cvfS4lxkQwr1ixAjlkRQDV3LNnDyIHvFoElZ6ICI0VK1bs0qULjxOXq8JTfH19WQJmjp1Bd7EziA5SigDxIP0agVXOrg408iIXLFiAjiBVqB7SMG3aNHFWHwusLAAJFn/TlfGzGkxHOaE55hqJ8bvFz8+P2p5srBxnlSTNmjWjqmX/UPCWLVsWXSPwdP/LDpuWU2yYHEp0VIC9sXr1avY2AcwWotQlX7KviAilkxrsGWKHXapvNDZs2IC8MlRuKxAfPXAK4aA/ckCNwP1FfxNNSpWc1IHEyOqTYNEqVAClIMjr1asnvolFXmVdSB21a9fmPWHvGSVzE9eiJiwWK8igiTE6o/GMkihFFLgJChcYGIgx4Sn4C55SqFChbt26oa+IqKFVZgtiUigiIiMjGRIDYEgYtqSkJPE5CBZL9GTvqv5qA5f4+Ph079590KBBlM3e3t4oHYMUV71eSAVEDqbmiy++wN3gcRAU3j17i3Xr0KED26JTp05Gfp+JNWFXERhMH9PERHJ1sybCjCPJ024hixBIYWFhOn/B5mHNR48ezWY+e/Ysm5DNQKRR8IsOpCuyBRuGtyNaspOenk5OItuTLMmI7Go2Ib6VvUcQKp3UwDWQ4dmoIikKGAYbY8iQISwj4cCM2MboCKfwJt9++y1XzZo1i6JD9DfRpFTJSR0YAWFftWrVChUqODs7M3rijZYpU6bwNarJ81CjEiVKVKpUqU/WL/bq7AOJ3dXVFU9B0eHh4cGgeYVEbHBwMNkGeStZsiSlgfgPEfr27Yvneeutt5gY9ySl6y/fS/Ay0EUuRLMoelkpbkio4+5YOPyt6IabUP3VBrI9tqVIkSJFixYtlgUjqf/3935eL2gZb4WBsYA9evRAJUUg4R4xNRRcTJxMQvWuXGAYVo93T2CwquK7PyyRNk7+Jcw7kjztFjYkG4Ng03VgW7In0WgGzNvBtLMJeUHsT87SiDEhUbOjyLTikuxwW/FjwfpmmUOyui7Dq8JgCB98FntYacqqc5kUAoHUkqgIlpUrVxLwnGJjMNmmTZtOmDCB0BP98zMphEZcYiS5fyopyQ/YB8oWbDD/knNEFPEve4IXjGDduHHDUOrTh62AF+PFE5NsGraXuJX2WM5IcoW1xZehwjhZpSlHeBH0xNISq6i20mphaDkpqQ6mBSc8fPhw1IHyUme/eWHYKBIvRgaZEI05QASeOXOGOgj/GRsbi+IYH5AkEyo+5SDfWM5IjIEnklHd3NwwOEpTjvAuyLfYZHI1VYPSamFoOSmpDqaF90F5XK9ePb7QWUFijAqIYofiMGfjQOzRmbIWl4GLxkPeuXPHGK8h4Fry/KTX8VcVLGckxsOYMfy4Nn0fbgg637x5E5NP1bB9+/a75vgPhIxBy0lJdTAt0dHRTZo0QSB4SbqPbJFzPDl1JjWkaDGEyLdz584lICMjIyn4jQxItgXXUqxOmTIlr9WmKpYzkjyRkZGxc+dO0mZcXJzSZADs94EDBzw9PYklI8s9c6HZpKQ6mBACY+bMmfXr1/fw8MjMzORQtPN1eHg47+yc3n9Inx363759e8mSJTY2NvRH+HV3yBm6UcXgPEeMGNG8efOAgADlxKtiOSN5BdLS0jZu3JjrN4bQax8fH0w7Oo7TUVotFW0mJdXBhOARcHSoA3GlMw5AdOHPg4KCCDmlSQ12QFhYWLt27QKz/q9KOqcbgKxOZ3YAKZpcwZ2pMxs3blyuXDlra+sdO3Yod3xVLGckEi2R6mBC8HLil2Go0nXJFnd37do1d3d3zCEWUTRmh6oeA8+1FSpU6NKli4ODA5ZelYEDB9rb2/fq1YtnkZ+trKyqV69eunTp995778MPP3Rzc7ug95vyr4DljESiMVIdTMjJkycbNmzYtWvXJL2/yYOhoB1Pcfr0aQpypTUbCQkJxNgHH3xQsGBBouu/huHsu++++84779CzQIECb+tRsWLF4ODg+9l+wj1PWM5IJBoj1cGE7N27t27duo6OjpgFpSnrJ+0iIiLGjh2L/TayepdIzIJUBxOCO8Bg29nZ4agRAqCU2L9/v6en5/bt23MoKyQSS0CqgwnBJuARunXrtmrVql9//TUxMXHTpk1+fn4cpqWlWf4H45L/caQ6mBDMAsWFq6srdfuoUaNGjBjBv8uWLUtJSTHdt9MvXbq0c+dOf39/GxubRYsWIUPKCYkkj0h1+Ldx/fr1LVu2TJs2rVWrVvPnz3+N6rBnzx5HR8eGDRsmG/gvS7IjpeqNRqrDv42HDx8Sk+vWrWvfvn1gYOBrDMizZ8+OGzfOwcEh599T1sd0UiXRAKkOlkt4eDhVib29fWpqanx8vIuLy9KlS5VzOZKenr5t27aOHTu+XnU4efIk0jBjxgzGc/PmTf1vxBjCdFIl0QCpDpYL6oAbt7OzIxQjIyOHDBkSHR2dmJjIvxHZoIPur4yaSB02bNhAkEdFRcXExDAwI38s2kSDkWiAVAfLBSEgUY8cOfLevXuzZs0aNmxYQkICqfjcuXOns5GcnKz7/0FMFJC+vr5WVlYhISHLli1zd3dv0qSJGaVKogFSHSwXVMDNzc3b2xt/PjaLQ4cOmSsgMzMzhw8fXrlyZQqc2NjYuLi43bt3m1GqJBog1cFyOX/+PAHp5+d38eJFe3t7Ly8v/LxyzjCEcWpqKhm+ZcuWPj4+xKpyIn8Q8E5OTgxjwoQJEydOPHHiREpKivQO/26kOlguhB/h7ezsHBQUZGNjQ2Tq/g/hHCCZL1682NHRsVq1asQkwaycyB+bN29GHcLCwqKionr27MkX4eHhyjnDmEiqJNog1UFiFAiTu7v7kSNHjh07hlRR5oSGhirnDGMiqZJog1QHiUSijlQHiUSijlQHiUSijlQHiUSijlQHiUSijlQHiUSijlQHiUSijlQHiUSijlQHiUSixosX/w9zGCsw4VZS+gAAAABJRU5ErkJggg=="}}},{"cell_type":"markdown","source":"## Import packages","metadata":{}},{"cell_type":"code","source":"# Install packages if required","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:23:44.343505Z","iopub.execute_input":"2022-02-13T15:23:44.344644Z","iopub.status.idle":"2022-02-13T15:23:44.365106Z","shell.execute_reply.started":"2022-02-13T15:23:44.344416Z","shell.execute_reply":"2022-02-13T15:23:44.364248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport plotly.express as px\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport plotly as py\nimport plotly.graph_objs as go\ninit_notebook_mode(connected=True)\nfrom matplotlib import pyplot as plt\nfrom plotly.subplots import make_subplots\nimport os\nfrom tqdm import tqdm\nfrom matplotlib_venn import venn2, venn2_circles, venn2_unweighted\nfrom matplotlib_venn import venn3, venn3_circles\nimport calendar\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport torch\nfrom torch.utils import data as torch_data","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:05:50.815796Z","iopub.execute_input":"2022-02-13T16:05:50.816111Z","iopub.status.idle":"2022-02-13T16:05:50.825371Z","shell.execute_reply.started":"2022-02-13T16:05:50.816076Z","shell.execute_reply":"2022-02-13T16:05:50.824672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load data","metadata":{}},{"cell_type":"code","source":"articles_data = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/articles.csv')\ncustomers_data = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/customers.csv')\ntransactions_train = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\nsample_submission = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:23:46.569491Z","iopub.execute_input":"2022-02-13T15:23:46.569837Z","iopub.status.idle":"2022-02-13T15:25:08.398439Z","shell.execute_reply.started":"2022-02-13T15:23:46.569793Z","shell.execute_reply":"2022-02-13T15:25:08.397610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EDA","metadata":{}},{"cell_type":"code","source":"# pd.set_option('display.max_columns', None)\nprint(articles_data.shape, articles_data['article_id'].nunique())\narticles_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:08.399647Z","iopub.execute_input":"2022-02-13T15:25:08.400005Z","iopub.status.idle":"2022-02-13T15:25:08.453890Z","shell.execute_reply.started":"2022-02-13T15:25:08.399975Z","shell.execute_reply":"2022-02-13T15:25:08.453167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_data.columns","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:08.455742Z","iopub.execute_input":"2022-02-13T15:25:08.456527Z","iopub.status.idle":"2022-02-13T15:25:08.463619Z","shell.execute_reply.started":"2022-02-13T15:25:08.456481Z","shell.execute_reply":"2022-02-13T15:25:08.462450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(customers_data.shape, customers_data['customer_id'].nunique())\ncustomers_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:08.465147Z","iopub.execute_input":"2022-02-13T15:25:08.465567Z","iopub.status.idle":"2022-02-13T15:25:09.245177Z","shell.execute_reply.started":"2022-02-13T15:25:08.465485Z","shell.execute_reply":"2022-02-13T15:25:09.244037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(transactions_train.shape, transactions_train['customer_id'].nunique())\ntransactions_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:09.249671Z","iopub.execute_input":"2022-02-13T15:25:09.250230Z","iopub.status.idle":"2022-02-13T15:25:18.125283Z","shell.execute_reply.started":"2022-02-13T15:25:09.250174Z","shell.execute_reply":"2022-02-13T15:25:18.124403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"article_ids_master = set(articles_data['article_id'].unique().tolist())\narticle_ids_train = set(transactions_train['article_id'].unique().tolist())\nprint('# New articles found in transactions train data (not in master articles data):', len(article_ids_train) - len(article_ids_train.intersection(article_ids_master)))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:18.126523Z","iopub.execute_input":"2022-02-13T15:25:18.126775Z","iopub.status.idle":"2022-02-13T15:25:18.641876Z","shell.execute_reply.started":"2022-02-13T15:25:18.126745Z","shell.execute_reply":"2022-02-13T15:25:18.640673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# See transactions for a single customer\ncust_id = customers_data['customer_id'].iloc[0]\ntransactions_train[transactions_train['customer_id']==cust_id]","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:18.643484Z","iopub.execute_input":"2022-02-13T15:25:18.643775Z","iopub.status.idle":"2022-02-13T15:25:24.028057Z","shell.execute_reply.started":"2022-02-13T15:25:18.643743Z","shell.execute_reply":"2022-02-13T15:25:24.026887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sample_submission.shape)\nprint('\\nSample article IDs:', sample_submission['prediction'].iloc[0], '\\n# articles:', len(sample_submission['prediction'].iloc[0].split(' ')))\nsample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:24.029463Z","iopub.execute_input":"2022-02-13T15:25:24.029811Z","iopub.status.idle":"2022-02-13T15:25:24.044329Z","shell.execute_reply.started":"2022-02-13T15:25:24.029764Z","shell.execute_reply":"2022-02-13T15:25:24.043181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_ids_test_data = set(sample_submission['customer_id'].unique().tolist())\ncust_ids_train_data = set(transactions_train['customer_id'].unique().tolist())\nprint('# Customers in test data:',len(cust_ids_test_data))\nprint('# New customers found in test data (not in train data):', len(cust_ids_test_data) - len(cust_ids_test_data.intersection(cust_ids_train_data)))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:24.045842Z","iopub.execute_input":"2022-02-13T15:25:24.046094Z","iopub.status.idle":"2022-02-13T15:25:34.264294Z","shell.execute_reply.started":"2022-02-13T15:25:24.046064Z","shell.execute_reply":"2022-02-13T15:25:34.263307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution by sales and customers - Top products, product types, department, etc.","metadata":{}},{"cell_type":"code","source":"num_customers_by_article = transactions_train.groupby(['article_id']).agg({'customer_id':['count', 'nunique'], 'price':['mean','max','min']}).reset_index()\nnum_customers_by_article.columns = ['article_id', '# Transactions', '# Customers', 'Average price', 'Max. price', 'Min. price']\nnum_customers_by_article = num_customers_by_article.sort_values(by = ['# Customers'], ascending = False)\nnum_customers_by_article = num_customers_by_article.merge(articles_data, how = 'left', on = 'article_id')","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:25:34.265583Z","iopub.execute_input":"2022-02-13T15:25:34.265928Z","iopub.status.idle":"2022-02-13T15:26:03.846674Z","shell.execute_reply.started":"2022-02-13T15:25:34.265884Z","shell.execute_reply":"2022-02-13T15:26:03.845714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_data.shape, num_customers_by_article.shape","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:03.849956Z","iopub.execute_input":"2022-02-13T15:26:03.850248Z","iopub.status.idle":"2022-02-13T15:26:03.856907Z","shell.execute_reply.started":"2022-02-13T15:26:03.850208Z","shell.execute_reply":"2022-02-13T15:26:03.856002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_customers_by_prod_name = num_customers_by_article.groupby(['prod_name']).agg({'# Transactions':'sum', '# Customers':'sum', 'Average price':'mean'}).reset_index()\nnum_customers_by_prod_type = num_customers_by_article.groupby(['product_type_name']).agg({'# Transactions':'sum', '# Customers':'sum', 'Average price':'mean'}).reset_index()\nnum_customers_by_department_name = num_customers_by_article.groupby(['department_name']).agg({'# Transactions':'sum', '# Customers':'sum', 'Average price':'mean'}).reset_index()\nnum_customers_by_section_name = num_customers_by_article.groupby(['section_name']).agg({'# Transactions':'sum', '# Customers':'sum', 'Average price':'mean'}).reset_index()\nnum_customers_by_garment_group_name = num_customers_by_article.groupby(['garment_group_name']).agg({'# Transactions':'sum', '# Customers':'sum', 'Average price':'mean'}).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:03.858251Z","iopub.execute_input":"2022-02-13T15:26:03.858466Z","iopub.status.idle":"2022-02-13T15:26:04.085725Z","shell.execute_reply.started":"2022-02-13T15:26:03.858440Z","shell.execute_reply":"2022-02-13T15:26:04.084667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n# Unique product names:', num_customers_by_prod_name['prod_name'].nunique())\nfig = px.bar(num_customers_by_prod_name.sort_values(by = ['# Transactions'], ascending = False).head(10), x='prod_name', y='# Transactions', title = 'Top product names by # transactions', hover_data = ['# Transactions', '# Customers', 'Average price'])\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_prod_name.sort_values(by = ['# Customers'], ascending = False).head(10), x='prod_name', y='# Customers', title = 'Top product names by # customers', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_prod_name.sort_values(by = ['Average price'], ascending = False).head(10), x='prod_name', y='Average price', title = 'Top product names by average price', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='grey')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:04.086917Z","iopub.execute_input":"2022-02-13T15:26:04.087171Z","iopub.status.idle":"2022-02-13T15:26:06.235489Z","shell.execute_reply.started":"2022-02-13T15:26:04.087141Z","shell.execute_reply":"2022-02-13T15:26:06.234567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n# Unique product types:', num_customers_by_prod_type['product_type_name'].nunique())\nfig = px.bar(num_customers_by_prod_type.sort_values(by = ['# Transactions'], ascending = False).head(10), x='product_type_name', y='# Transactions', title = 'Top product types by # transactions', hover_data = ['# Transactions', '# Customers', 'Average price'])\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_prod_type.sort_values(by = ['# Customers'], ascending = False).head(10), x='product_type_name', y='# Customers', title = 'Top product types by # customers', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_prod_type.sort_values(by = ['Average price'], ascending = False).head(10), x='product_type_name', y='Average price', title = 'Top product types by average price', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='grey')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:06.236792Z","iopub.execute_input":"2022-02-13T15:26:06.237050Z","iopub.status.idle":"2022-02-13T15:26:06.538403Z","shell.execute_reply.started":"2022-02-13T15:26:06.237016Z","shell.execute_reply":"2022-02-13T15:26:06.537547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n# Unique department names:', num_customers_by_department_name['department_name'].nunique())\nfig = px.bar(num_customers_by_department_name.sort_values(by = ['# Transactions'], ascending = False).head(10), x='department_name', y='# Transactions', title = 'Top department names by # transactions', hover_data = ['# Transactions', '# Customers', 'Average price'])\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_department_name.sort_values(by = ['# Customers'], ascending = False).head(10), x='department_name', y='# Customers', title = 'Top department names by # customers', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_department_name.sort_values(by = ['Average price'], ascending = False).head(10), x='department_name', y='Average price', title = 'Top department names by average price', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='grey')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:06.539723Z","iopub.execute_input":"2022-02-13T15:26:06.539972Z","iopub.status.idle":"2022-02-13T15:26:06.836395Z","shell.execute_reply.started":"2022-02-13T15:26:06.539940Z","shell.execute_reply":"2022-02-13T15:26:06.835439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n# Unique section names:', num_customers_by_section_name['section_name'].nunique())\nfig = px.bar(num_customers_by_section_name.sort_values(by = ['# Transactions'], ascending = False).head(10), x='section_name', y='# Transactions', title = 'Top section names by # transactions', hover_data = ['# Transactions', '# Customers', 'Average price'])\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_section_name.sort_values(by = ['# Customers'], ascending = False).head(10), x='section_name', y='# Customers', title = 'Top section names by # customers', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_section_name.sort_values(by = ['Average price'], ascending = False).head(10), x='section_name', y='Average price', title = 'Top section names by average price', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='grey')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:06.837907Z","iopub.execute_input":"2022-02-13T15:26:06.838126Z","iopub.status.idle":"2022-02-13T15:26:07.120503Z","shell.execute_reply.started":"2022-02-13T15:26:06.838099Z","shell.execute_reply":"2022-02-13T15:26:07.119634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n# Unique garment group names:', num_customers_by_garment_group_name['garment_group_name'].nunique())\nfig = px.bar(num_customers_by_garment_group_name.sort_values(by = ['# Transactions'], ascending = False).head(10), x='garment_group_name', y='# Transactions', title = 'Top garment group names by # transactions', hover_data = ['# Transactions', '# Customers', 'Average price'])\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_garment_group_name.sort_values(by = ['# Customers'], ascending = False).head(10), x='garment_group_name', y='# Customers', title = 'Top garment group names by # customers', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)\n\nfig = px.bar(num_customers_by_garment_group_name.sort_values(by = ['Average price'], ascending = False).head(10), x='garment_group_name', y='Average price', title = 'Top garment group names by average price', hover_data = ['# Transactions', '# Customers', 'Average price'])\nfig.update_traces(marker_color='grey')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:07.122273Z","iopub.execute_input":"2022-02-13T15:26:07.122618Z","iopub.status.idle":"2022-02-13T15:26:07.428042Z","shell.execute_reply.started":"2022-02-13T15:26:07.122571Z","shell.execute_reply":"2022-02-13T15:26:07.427091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution of customers - Top customers by clubmembership, fashion news follower, etc.","metadata":{}},{"cell_type":"code","source":"print(transactions_train.shape)\ntransactions_train = transactions_train.merge(customers_data, how = 'left', on=['customer_id'])\nprint(transactions_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:07.429313Z","iopub.execute_input":"2022-02-13T15:26:07.429547Z","iopub.status.idle":"2022-02-13T15:26:33.534929Z","shell.execute_reply.started":"2022-02-13T15:26:07.429516Z","shell.execute_reply":"2022-02-13T15:26:33.534019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bins = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]\nlabels = ['0-9', '10-19', '20-29', '30-39', '40-49', '50-59', '60-69', '70-79', '80-89', '90-100']\ncustomers_data['age group'] = pd.cut(customers_data.age, bins, labels = labels,include_lowest = True)\ntransactions_train['age group'] = pd.cut(transactions_train.age, bins, labels = labels,include_lowest = True)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:33.536274Z","iopub.execute_input":"2022-02-13T15:26:33.536570Z","iopub.status.idle":"2022-02-13T15:26:34.436152Z","shell.execute_reply.started":"2022-02-13T15:26:33.536521Z","shell.execute_reply":"2022-02-13T15:26:34.435117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('# Total customers:', customers_data['customer_id'].nunique())\nprint('# Total customers with transactions:', transactions_train['customer_id'].nunique())\ntransactions_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:34.437362Z","iopub.execute_input":"2022-02-13T15:26:34.437673Z","iopub.status.idle":"2022-02-13T15:26:43.946145Z","shell.execute_reply.started":"2022-02-13T15:26:34.437639Z","shell.execute_reply":"2022-02-13T15:26:43.945102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_customers = transactions_train.groupby(['customer_id']).agg({'article_id':['count','nunique'], 'price':['mean','max','min']}).reset_index()\ntop_customers.columns = ['customer_id', '# Transactions', '# Unique articles bought', 'Average price', 'Max. price', 'Min. price']\ntop_customers = top_customers.merge(customers_data, on = 'customer_id', how = 'left')","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:26:43.947452Z","iopub.execute_input":"2022-02-13T15:26:43.947730Z","iopub.status.idle":"2022-02-13T15:27:19.603443Z","shell.execute_reply.started":"2022-02-13T15:26:43.947699Z","shell.execute_reply":"2022-02-13T15:27:19.602457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_customers.sort_values(by = '# Transactions', ascending = False).head(10)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:19.605179Z","iopub.execute_input":"2022-02-13T15:27:19.605436Z","iopub.status.idle":"2022-02-13T15:27:20.362795Z","shell.execute_reply.started":"2022-02-13T15:27:19.605406Z","shell.execute_reply":"2022-02-13T15:27:20.361627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = top_customers.sort_values(by = ['# Transactions'], ascending = False).head(1000)\nfig = px.scatter(temp, x='# Transactions', y=\"Average price\", hover_data = ['# Unique articles bought', 'Max. price', 'Min. price', 'age group', 'FN', 'Active',\n       'club_member_status', 'fashion_news_frequency'], title = 'Top 1000 customers by # transactions')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:20.364322Z","iopub.execute_input":"2022-02-13T15:27:20.364623Z","iopub.status.idle":"2022-02-13T15:27:21.347725Z","shell.execute_reply.started":"2022-02-13T15:27:20.364579Z","shell.execute_reply":"2022-02-13T15:27:21.346620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = top_customers.sort_values(by = [\"Average price\"], ascending = False).head(1000)\nfig = px.scatter(temp, y='# Transactions', x=\"Average price\", hover_data = ['# Unique articles bought', 'Max. price', 'Min. price', 'age group', 'FN', 'Active',\n       'club_member_status', 'fashion_news_frequency'], title = 'Top 1000 customers by average price')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:21.349246Z","iopub.execute_input":"2022-02-13T15:27:21.349862Z","iopub.status.idle":"2022-02-13T15:27:22.748347Z","shell.execute_reply.started":"2022-02-13T15:27:21.349810Z","shell.execute_reply":"2022-02-13T15:27:22.746707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lst = ['age group', 'club_member_status', 'fashion_news_frequency']\nspecs = [[{'type':'domain'}] * 3] * 1\n\nfor col_name in ['# Transactions', 'customer_id', '# Unique articles bought', 'Average price']:\n    fig = make_subplots(\n            rows=1,\n            cols=3,\n            specs=specs,\n            print_grid=False)\n\n    for i, l in enumerate(lst):\n        # basic math to get col and row\n        row = 1\n        col = i+1\n        # this is the dataframe for every continent\n        d = l\n        temp = top_customers.groupby([d]).agg({'customer_id':'count','# Transactions':'sum', '# Unique articles bought':'sum', 'Average price':'mean'}).reset_index()\n        fig.add_trace(\n            go.Pie(labels=temp[d],\n                   values=temp[col_name],\n                   showlegend=False,\n                   textposition='inside',\n                   textinfo='label+percent',\n                   title='Distribution by ' + d),\n             row=row,\n             col=col\n        )\n    fig.update_layout(title=\"Distribution of \" + col_name + \" by customer metadata\", title_x=0.5)\n    py.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:22.749725Z","iopub.execute_input":"2022-02-13T15:27:22.750009Z","iopub.status.idle":"2022-02-13T15:27:28.018097Z","shell.execute_reply.started":"2022-02-13T15:27:22.749976Z","shell.execute_reply":"2022-02-13T15:27:28.016880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transactions data - summaries based on time","metadata":{}},{"cell_type":"code","source":"transactions_train['Date'] = pd.to_datetime(transactions_train['t_dat'])\ntransactions_train['Year'] = transactions_train['Date'].dt.year\ntransactions_train['Month'] = transactions_train['Date'].dt.month\ntransactions_train['Day'] = transactions_train['Date'].dt.day","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:28.019841Z","iopub.execute_input":"2022-02-13T15:27:28.020122Z","iopub.status.idle":"2022-02-13T15:27:42.740823Z","shell.execute_reply.started":"2022-02-13T15:27:28.020090Z","shell.execute_reply":"2022-02-13T15:27:42.739396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions_train.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:42.742024Z","iopub.execute_input":"2022-02-13T15:27:42.742271Z","iopub.status.idle":"2022-02-13T15:27:42.766020Z","shell.execute_reply.started":"2022-02-13T15:27:42.742241Z","shell.execute_reply":"2022-02-13T15:27:42.764875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = transactions_train.groupby(['Year']).agg({'customer_id':['count', 'nunique']}).reset_index()\ntemp.columns = ['Year', '# Transactions', '# Customers']\ntemp = temp.sort_values(by = ['Year'], ascending = True)\ntemp['Year'] = temp['Year'].astype(str)\nprint('\\nYears:', transactions_train['Year'].min(), 'to', transactions_train['Year'].max())\nfig = px.bar(temp, x='Year', y='# Transactions', title = '# Transactions by year', hover_data = ['# Customers'])\npy.offline.iplot(fig)\n\nfig = px.bar(temp, x='Year', y='# Customers', title = '# Customers by year', hover_data = ['# Transactions'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:27:42.767027Z","iopub.execute_input":"2022-02-13T15:27:42.767286Z","iopub.status.idle":"2022-02-13T15:28:05.418275Z","shell.execute_reply.started":"2022-02-13T15:27:42.767256Z","shell.execute_reply":"2022-02-13T15:28:05.416518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = transactions_train.groupby(['Month']).agg({'customer_id':['count', 'nunique']}).reset_index()\ntemp.columns = ['Month', '# Transactions', '# Customers']\ntemp = temp.sort_values(by = ['Month'], ascending = True)\ntemp['Month'] = temp['Month'].apply(lambda x: calendar.month_name[x])\nfig = px.bar(temp, x='Month', y='# Transactions', title = '# Transactions by month', hover_data = ['# Customers'])\npy.offline.iplot(fig)\n\nfig = px.bar(temp, x='Month', y='# Customers', title = '# Customers by month', hover_data = ['# Transactions'])\nfig.update_traces(marker_color='green')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:28:05.419894Z","iopub.execute_input":"2022-02-13T15:28:05.420527Z","iopub.status.idle":"2022-02-13T15:28:28.544325Z","shell.execute_reply.started":"2022-02-13T15:28:05.420451Z","shell.execute_reply":"2022-02-13T15:28:28.543243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = transactions_train.groupby(['Year','Month']).agg({'customer_id':['count', 'nunique']}).reset_index()\ntemp.columns = ['Year', 'Month', '# Transactions', '# Customers']\ntemp['Year-Month'] = temp['Year'].astype(str) + '-' + temp['Month'].astype(str)\ntemp = temp.sort_values(by = ['Year', 'Month'], ascending = True)\nfig = px.line(temp, x=\"Year-Month\", y=\"# Transactions\", title='# Transactions across time')\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:28:28.545948Z","iopub.execute_input":"2022-02-13T15:28:28.546277Z","iopub.status.idle":"2022-02-13T15:28:53.119848Z","shell.execute_reply.started":"2022-02-13T15:28:28.546235Z","shell.execute_reply":"2022-02-13T15:28:53.118778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customer_id = top_customers.sort_values(by = ['# Transactions'], ascending = False)['customer_id'].iloc[0] # Specify any customer ID\n\ntemp = transactions_train[transactions_train['customer_id']==customer_id]\ntemp = temp.groupby(['Year','Month']).agg({'customer_id':['count']}).reset_index()\ntemp.columns = ['Year', 'Month', '# Transactions']\ntemp['Year-Month'] = temp['Year'].astype(str) + '-' + temp['Month'].astype(str)\ntemp = temp.sort_values(by = ['Year', 'Month'], ascending = True)\nfig = px.line(temp, x=\"Year-Month\", y=\"# Transactions\", title='# Transactions across time for customer ID: '+ str(customer_id))\npy.offline.iplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:28:53.121607Z","iopub.execute_input":"2022-02-13T15:28:53.121880Z","iopub.status.idle":"2022-02-13T15:29:08.230393Z","shell.execute_reply.started":"2022-02-13T15:28:53.121846Z","shell.execute_reply":"2022-02-13T15:29:08.229503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Images of articles","metadata":{}},{"cell_type":"code","source":"articles_data['image_path'] = np.NaN","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:29:08.231889Z","iopub.execute_input":"2022-02-13T15:29:08.232508Z","iopub.status.idle":"2022-02-13T15:29:08.239124Z","shell.execute_reply.started":"2022-02-13T15:29:08.232450Z","shell.execute_reply":"2022-02-13T15:29:08.237935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"file_path = '../input/h-and-m-personalized-fashion-recommendations/images'\nimage_file_paths = []\nfor folder in tqdm(os.listdir(file_path)):\n    images = os.listdir(file_path + '/' + folder)\n    for img in images:\n        article_id = img.split('.jpg')[0]\n        articles_data['image_path'] = np.where(articles_data['article_id']==int(article_id), file_path + '/' + folder + '/' + img, articles_data['image_path'])","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:29:08.240623Z","iopub.execute_input":"2022-02-13T15:29:08.240881Z","iopub.status.idle":"2022-02-13T15:41:01.091537Z","shell.execute_reply.started":"2022-02-13T15:29:08.240852Z","shell.execute_reply":"2022-02-13T15:41:01.090364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"article_ids_master = set(articles_data['article_id'].tolist())\narticle_ids_with_transactions = set(transactions_train[['article_id']].drop_duplicates()['article_id'].tolist())\narticle_ids_with_images = set(articles_data[articles_data['image_path'].notna()]['article_id'].tolist())\nvenn2((article_ids_with_transactions, article_ids_with_images), set_labels = ('Articles with transactions', 'Articles with images'))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:41:01.093465Z","iopub.execute_input":"2022-02-13T15:41:01.093848Z","iopub.status.idle":"2022-02-13T15:41:02.985254Z","shell.execute_reply.started":"2022-02-13T15:41:01.093800Z","shell.execute_reply":"2022-02-13T15:41:02.984035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Almost all articles have images","metadata":{}},{"cell_type":"code","source":"articles_data.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:41:02.993676Z","iopub.execute_input":"2022-02-13T15:41:02.994999Z","iopub.status.idle":"2022-02-13T15:41:03.029121Z","shell.execute_reply.started":"2022-02-13T15:41:02.994922Z","shell.execute_reply":"2022-02-13T15:41:03.028171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\nSample 10 images\\n')\narticle_id = articles_data[articles_data['image_path']!='nan']['article_id'].sample(10).tolist() # Specify list of article IDs\n    \nrows = 2\ncols = 5\naxes=[]\nfig=plt.figure(figsize=(50, 25))\n\ncount = 0\nfor a in range(rows*cols):\n    img_path = articles_data[articles_data['article_id']==article_id[count]]['image_path'].iloc[0]\n    axes.append(fig.add_subplot(rows, cols, a+1))\n    subplot_title=(\"Subplot\"+str(a))\n    axes[-1].set_title(subplot_title)  \n    plt.imshow(plt.imread(img_path))\n    count +=1\n    \nfig.tight_layout()    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:41:03.031644Z","iopub.execute_input":"2022-02-13T15:41:03.032274Z","iopub.status.idle":"2022-02-13T15:41:09.897812Z","shell.execute_reply.started":"2022-02-13T15:41:03.032220Z","shell.execute_reply":"2022-02-13T15:41:09.896760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\nTop 10 images by # transactions\\n')\nnum_customers_by_article = num_customers_by_article.merge(articles_data[['article_id','image_path']], how = 'left', on = 'article_id')\narticle_id = num_customers_by_article[num_customers_by_article['image_path']!='nan'].sort_values(by = ['# Transactions'], ascending = False).head(10)['article_id'].tolist()\n    \nrows = 2\ncols = 5\naxes=[]\nfig=plt.figure(figsize=(50, 25))\n\ncount = 0\nfor a in range(rows*cols):\n    img_path = articles_data[articles_data['article_id']==article_id[count]]['image_path'].iloc[0]\n    axes.append(fig.add_subplot(rows, cols, a+1))\n    subplot_title=(\"Subplot\"+str(a))\n    axes[-1].set_title(subplot_title)  \n    plt.imshow(plt.imread(img_path))\n    count +=1\n    \nfig.tight_layout()    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:41:09.899397Z","iopub.execute_input":"2022-02-13T15:41:09.900245Z","iopub.status.idle":"2022-02-13T15:41:17.161694Z","shell.execute_reply.started":"2022-02-13T15:41:09.900184Z","shell.execute_reply":"2022-02-13T15:41:17.160728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\nTop 10 images by average price\\n')\narticle_id = num_customers_by_article[num_customers_by_article['image_path']!='nan'].sort_values(by = ['Average price'], ascending = False).head(10)['article_id'].tolist()\n    \nrows = 2\ncols = 5\naxes=[]\nfig=plt.figure(figsize=(50, 25))\n\ncount = 0\nfor a in range(rows*cols):\n    img_path = articles_data[articles_data['article_id']==article_id[count]]['image_path'].iloc[0]\n    axes.append(fig.add_subplot(rows, cols, a+1))\n    subplot_title=(\"Subplot\"+str(a))\n    axes[-1].set_title(subplot_title)  \n    plt.imshow(plt.imread(img_path))\n    count +=1\n    \nfig.tight_layout()    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:41:17.163174Z","iopub.execute_input":"2022-02-13T15:41:17.163425Z","iopub.status.idle":"2022-02-13T15:41:23.881766Z","shell.execute_reply.started":"2022-02-13T15:41:17.163394Z","shell.execute_reply":"2022-02-13T15:41:23.880712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelling","metadata":{}},{"cell_type":"markdown","source":"Converting business problem into a supervised problem using the below steps:\n1. Setup model to predict from all customers those that would purchase an article in last 7 days\n2. Setup model to predict from all articles those that would get purchased in last 7 days\n3. Create data at customer and article level using list of customers and articles from the above 2 models\n4. Setup model to predict if a customer would predict a specific article from step 3","metadata":{}},{"cell_type":"code","source":"print('\\nLast day of transaction:\\n')\ntemp = transactions_train[transactions_train['Date']==transactions_train['Date'].max()][['Year', 'Month', 'Day']].drop_duplicates().reset_index(drop = True)\ndisplay(temp)\n# Get day corresponding to last 7 days\nprint('\\n Last 7 days of transaction:\\n')\ntemp = transactions_train[(transactions_train['Year']==temp['Year'].iloc[0]) & (transactions_train['Month']==temp['Month'].iloc[0]) & (transactions_train['Day']>temp['Day'].iloc[0]-7)][['Date','Year', 'Month', 'Day']].drop_duplicates().reset_index(drop = True)\ndisplay(temp)\nlast_7th_date = temp['Date'].iloc[0]\ncust_ids_last_7_days_transactions_data = transactions_train[transactions_train['Date']>=last_7th_date]['customer_id'].unique().tolist()\narticle_ids_last_7_days_transactions_data = transactions_train[transactions_train['Date']>=last_7th_date]['article_id'].unique().tolist()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T15:41:38.560824Z","iopub.execute_input":"2022-02-13T15:41:38.561758Z","iopub.status.idle":"2022-02-13T15:41:39.301131Z","shell.execute_reply.started":"2022-02-13T15:41:38.561682Z","shell.execute_reply":"2022-02-13T15:41:39.300228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tqdm.pandas()\n# customers_data['Flag - Transaction in last 7 days'] = customers_data['customer_id'].progress_apply(lambda x: 1 if x in cust_ids_last_7_days_transactions_data else 0)\ncustomers_data['Flag - Transaction in last 7 days'] = 0\ncustomers_data['Flag - Transaction in last 7 days'][customers_data[customers_data['customer_id'].isin(cust_ids_last_7_days_transactions_data)].index] = 1\nprint('# Customers who made at least 1 transaction in last 7 days:', customers_data['Flag - Transaction in last 7 days'].sum())\nprint('% Customers who made at least 1 transaction in last 7 days:', np.round(customers_data['Flag - Transaction in last 7 days'].sum()/customers_data.shape[0] * 100,2),'%')","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:00:33.910696Z","iopub.execute_input":"2022-02-13T16:00:33.910964Z","iopub.status.idle":"2022-02-13T16:00:34.242228Z","shell.execute_reply.started":"2022-02-13T16:00:33.910936Z","shell.execute_reply":"2022-02-13T16:00:34.241101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# articles_data['Flag - Transaction in last 7 days'] = articles_data['customer_id'].apply(lambda x: 1 if x in article_ids_last_7_days_transactions_data else 0)\narticles_data['Flag - Transaction in last 7 days'] = 0\narticles_data['Flag - Transaction in last 7 days'][articles_data[articles_data['article_id'].isin(article_ids_last_7_days_transactions_data)].index] = 1\nprint('# Articles with at least 1 transaction in last 7 days:', articles_data['Flag - Transaction in last 7 days'].sum())\nprint('% Articles with at least 1 transaction in last 7 days:', np.round(articles_data['Flag - Transaction in last 7 days'].sum()/articles_data.shape[0] * 100,2),'%')","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:00:39.892734Z","iopub.execute_input":"2022-02-13T16:00:39.893370Z","iopub.status.idle":"2022-02-13T16:00:39.924139Z","shell.execute_reply.started":"2022-02-13T16:00:39.893326Z","shell.execute_reply":"2022-02-13T16:00:39.923349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# determine the supported device\ndef get_device():\n    if torch.cuda.is_available():\n        device = torch.device('cuda:0')\n    else:\n        device = torch.device('cpu') # don't have GPU \n    return device\n\n# convert a df to tensor to be used in pytorch\ndef df_to_tensor(df):\n    device = get_device()\n    return torch.from_numpy(df.values).float().to(device)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:04:14.892888Z","iopub.execute_input":"2022-02-13T16:04:14.893186Z","iopub.status.idle":"2022-02-13T16:04:14.900651Z","shell.execute_reply.started":"2022-02-13T16:04:14.893156Z","shell.execute_reply":"2022-02-13T16:04:14.899285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features_for_modelling = ['age'] # Last transaction day difference and other features TBU","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:07:51.446936Z","iopub.execute_input":"2022-02-13T16:07:51.447252Z","iopub.status.idle":"2022-02-13T16:07:51.452482Z","shell.execute_reply.started":"2022-02-13T16:07:51.447218Z","shell.execute_reply":"2022-02-13T16:07:51.451659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DataRetriever_customer_identification(torch_data.Dataset):\n    def __init__(self, paths, targets):\n        self.paths = paths\n        self.targets = targets\n        self.target_flag = True\n        self.train_flag = 'train'\n        if len(targets)==0:\n            self.target_flag = False\n            self.train_flag = 'test'\n          \n    def __len__(self):\n        return len(self.paths)\n    \n    def __getitem__(self, index):\n        _id = self.paths[index]\n        data = customers_data[customers_data['customer_id']==_id][features_for_modelling]\n        data = df_to_tensor(data)\n\n        if (self.target_flag):\n            y = torch.tensor(self.targets[index], dtype=torch.float)\n            return {\"X\": data, \"y\": y}\n        else:\n            return {\"X\": data, \"id\": _id}","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:07:53.175288Z","iopub.execute_input":"2022-02-13T16:07:53.175807Z","iopub.status.idle":"2022-02-13T16:07:53.186670Z","shell.execute_reply.started":"2022-02-13T16:07:53.175773Z","shell.execute_reply":"2022-02-13T16:07:53.185427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_retriever = DataRetriever_customer_identification(\n    customers_data[\"customer_id\"].values, \n    customers_data['Flag - Transaction in last 7 days'].values)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:07:54.478583Z","iopub.execute_input":"2022-02-13T16:07:54.478890Z","iopub.status.idle":"2022-02-13T16:07:54.484132Z","shell.execute_reply.started":"2022-02-13T16:07:54.478856Z","shell.execute_reply":"2022-02-13T16:07:54.483035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_retriever[0]","metadata":{"execution":{"iopub.status.busy":"2022-02-13T16:07:55.279578Z","iopub.execute_input":"2022-02-13T16:07:55.279993Z","iopub.status.idle":"2022-02-13T16:07:55.618333Z","shell.execute_reply.started":"2022-02-13T16:07:55.279961Z","shell.execute_reply":"2022-02-13T16:07:55.617413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://www.pngall.com/wp-content/uploads/2018/04/Under-Construction-PNG-File.png)","metadata":{}}]}