{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"my first notebook\n\nthank you DANIIL KARPOV https://www.kaggle.com/code/vanguarde/h-m-eda-first-look for giving me all the inspiration","metadata":{}},{"cell_type":"markdown","source":"## **<span style=\"color:#008000;font-size:490%\"><center>EDA</center></span><span style=\"color:#008000;font-size:200%\"><center>Exploratory Data Analysis. H&M</center></span>**","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\nFor this challenge you are given the purchase history of customers across time, along with supporting metadata. Your challenge is to predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customer who did not make any purchase during that time are excluded from the scoring.\n\n\nThe dataset contains 4 csv files and one folder with several subfolders, each with a different number of images.\n\nIn this Exploratory Data Analysis Notebook we will look to the data, will analyze the content of each csv file, check for missing data, understand the data distribution, see what are the relations between data in various files.\n\nWe will also explore the image data, understand how images are indexed in the csv files, if there are articles in the dataset without images. We will also explore image additional information, like image width and height.\n\nWe also investigate a very simple baseline model and create an initial submission.\n\n![image.png](attachment:263a5724-ec87-4151-86a2-a85966720b76.png)","metadata":{},"attachments":{"263a5724-ec87-4151-86a2-a85966720b76.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAMwAAACMCAYAAAAwekaeAAAMbGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkJAQIICAlNCbIFIDSAmhBZBeBBshCSSUGBOCir0sKrh2EQEbuiqi2FZA7NiVRbH3xYKKsi7qYkPlTUhA133leyff3PvnzJn/lMzk3gMA/QNPKs1DtQHIlxTIEsKDmaPT0pmkpwCBHzKwA3QeXy5lx8VFAygD97/LuxvQFspVZyXXP+f/q+gKhHI+AMhYiDMFcn4+xMcBwKv4UlkBAESl3mpygVSJZ0OsJ4MBQrxKibNVeLsSZ6rw4X6bpAQOxJcB0KDyeLJsALTuQT2zkJ8NebQ+Q+wqEYglANCHQRzAF/EEECtjH5afP1GJyyG2h/ZSiGE8gJX5HWf23/gzB/l5vOxBrMqrXzRCxHJpHm/q/1ma/y35eYoBH7ZwUEWyiARl/rCGt3InRikxFeIuSWZMrLLWEH8QC1R1BwCliBQRySp71IQv58D6AQOIXQW8kCiITSAOk+TFRKv1mVniMC7EcLegU8QF3CSIDSFeKJSHJqptNsomJqh9ofVZMg5brT/Hk/X7Vfp6oMhNZqv534iEXDU/plUkSkqFmAKxdaE4JQZiLYhd5LmJUWqbkUUiTsyAjUyRoIzfGuIEoSQ8WMWPFWbJwhLU9iX58oF8sY0iMTdGjfcViJIiVPXBTvF5/fHDXLDLQgk7eYBHKB8dPZCLQBgSqsodey6UJCeqeT5IC4ITVGtxijQvTm2PWwrzwpV6S4g95IWJ6rV4SgHcnCp+PEtaEJekihMvyuFFxqniwZeBaMABIYAJFHBkgokgB4hbuxq64DfVTBjgARnIBkLgrNYMrEjtn5HAayIoAn9AJATywXXB/bNCUAj1Xwa1qqszyOqfLexfkQueQpwPokAe/K7oXyUZ9JYCnkCN+B/eeXDwYbx5cCjn/71+QPtNw4aaaLVGMeCRSR+wJIYSQ4gRxDCiA26MB+B+eDS8BsHhhrNwn4E8vtkTnhLaCI8I1wnthNsTxHNlP0Q5CrRD/jB1LTK/rwVuCzk98WDcH7JDZtwANwbOuAf0w8YDoWdPqOWo41ZWhfkD998y+O7XUNuRXckoeQg5iGz/40otRy3PQRZlrb+vjyrWzMF6cwZnfvTP+a76AniP+tESW4jtx85iJ7Dz2GGsATCxY1gj1oIdUeLB3fWkf3cNeEvojycX8oj/4Y+n9qmspNy11rXT9bNqrkA4pUB58DgTpVNl4mxRAZMNnw5CJlfCdxnGdHN1cwNA+axR/X29je9/hiAGLd90834HwP9YX1/foW+6yGMA7PWGx//gN509CwAdTQDOHeQrZIUqHa68EOC/BB2eNCNgBqyAPczHDXgBPxAEQkEkiAVJIA2Mh1UWwX0uA5PBdDAHFINSsAysBhVgA9gMtoNdYB9oAIfBCXAGXASXwXVwF+6eDvASdIN3oBdBEBJCQxiIEWKO2CBOiBvCQgKQUCQaSUDSkAwkG5EgCmQ6Mg8pRVYgFcgmpAbZixxETiDnkTbkNvIQ6UTeIJ9QDKWieqgpaosOR1koG41Ck9BxaDY6CS1C56NL0HK0Gt2J1qMn0IvodbQdfYn2YADTxAwwC8wZY2EcLBZLx7IwGTYTK8HKsGqsDmuCv/NVrB3rwj7iRJyBM3FnuIMj8GScj0/CZ+KL8Qp8O16Pn8Kv4g/xbvwrgUYwITgRfAlcwmhCNmEyoZhQRthKOEA4Dc9SB+EdkUg0INoRveFZTCPmEKcRFxPXEXcTjxPbiI+JPSQSyYjkRPInxZJ4pAJSMWktaSfpGOkKqYP0QUNTw1zDTSNMI11DojFXo0xjh8ZRjSsazzR6ydpkG7IvOZYsIE8lLyVvITeRL5E7yL0UHYodxZ+SRMmhzKGUU+oopyn3KG81NTUtNX004zXFmrM1yzX3aJ7TfKj5kapLdaRyqGOpCuoS6jbqcept6lsajWZLC6Kl0wpoS2g1tJO0B7QPWgwtFy2ulkBrllalVr3WFa1XdDLdhs6mj6cX0cvo++mX6F3aZG1bbY42T3umdqX2Qe2b2j06DJ0ROrE6+TqLdXbonNd5rkvStdUN1RXoztfdrHtS9zEDY1gxOAw+Yx5jC+M0o0OPqGenx9XL0SvV26XXqtetr6vvoZ+iP0W/Uv+IfrsBZmBrwDXIM1hqsM/ghsGnIaZD2EOEQxYNqRtyZch7w6GGQYZCwxLD3YbXDT8ZMY1CjXKNlhs1GN03xo0djeONJxuvNz5t3DVUb6jfUP7QkqH7ht4xQU0cTRJMpplsNmkx6TE1Mw03lZquNT1p2mVmYBZklmO2yuyoWac5wzzAXGy+yvyY+QumPpPNzGOWM08xuy1MLCIsFBabLFotei3tLJMt51rutrxvRbFiWWVZrbJqtuq2NrceZT3dutb6jg3ZhmUjslljc9bmva2dbartAtsG2+d2hnZcuyK7Wrt79jT7QPtJ9tX21xyIDiyHXId1DpcdUUdPR5FjpeMlJ9TJy0nstM6pbRhhmM8wybDqYTedqc5s50LnWueHLgYu0S5zXRpcXg23Hp4+fPnws8O/unq65rlucb07QndE5Ii5I5pGvHFzdOO7Vbpdc6e5h7nPcm90f+3h5CH0WO9xy5PhOcpzgWez5xcvby+ZV51Xp7e1d4Z3lfdNlh4rjrWYdc6H4BPsM8vnsM9HXy/fAt99vn/6Ofvl+u3wez7SbqRw5JaRj/0t/Xn+m/zbA5gBGQEbA9oDLQJ5gdWBj4KsggRBW4OesR3YOeyd7FfBrsGy4APB7zm+nBmc4yFYSHhISUhrqG5ocmhF6IMwy7DssNqw7nDP8GnhxyMIEVERyyNuck25fG4NtzvSO3JG5KkoalRiVEXUo2jHaFl00yh0VOSolaPuxdjESGIaYkEsN3Zl7P04u7hJcYfiifFx8ZXxTxNGJExPOJvISJyQuCPxXVJw0tKku8n2yYrk5hR6ytiUmpT3qSGpK1LbRw8fPWP0xTTjNHFaYzopPSV9a3rPmNAxq8d0jPUcWzz2xji7cVPGnR9vPD5v/JEJ9Am8CfszCBmpGTsyPvNiedW8nkxuZlVmN5/DX8N/KQgSrBJ0Cv2FK4TPsvyzVmQ9z/bPXpndKQoUlYm6xBxxhfh1TkTOhpz3ubG523L78lLzdudr5GfkH5ToSnIlpyaaTZwysU3qJC2Wtk/ynbR6UrcsSrZVjsjHyRsL9OBLfYvCXvGT4mFhQGFl4YfJKZP3T9GZIpnSMtVx6qKpz4rCin6Zhk/jT2uebjF9zvSHM9gzNs1EZmbObJ5lNWv+rI7Z4bO3z6HMyZ3z21zXuSvm/jUvdV7TfNP5s+c//in8p9pirWJZ8c0Ffgs2LMQXihe2LnJftHbR1xJByYVS19Ky0s+L+Ysv/Dzi5/Kf+5ZkLWld6rV0/TLiMsmyG8sDl29fobOiaMXjlaNW1q9iripZ9dfqCavPl3mUbVhDWaNY014eXd641nrtsrWfK0QV1yuDK3dXmVQtqnq/TrDuyvqg9XUbTDeUbvi0Ubzx1qbwTfXVttVlm4mbCzc/3ZKy5ewvrF9qthpvLd36ZZtkW/v2hO2narxranaY7Fhai9Yqajt3jt15eVfIrsY657pNuw12l+4BexR7XuzN2HtjX9S+5v2s/XW/2vxadYBxoKQeqZ9a390gamhvTGtsOxh5sLnJr+nAIZdD2w5bHK48on9k6VHK0flH+44VHes5Lj3edSL7xOPmCc13T44+ee1U/KnW01Gnz50JO3PyLPvssXP+5w6f9z1/8ALrQsNFr4v1LZ4tB37z/O1Aq1dr/SXvS42XfS43tY1sO3ol8MqJqyFXz1zjXrt4PeZ6243kG7dujr3Zfktw6/ntvNuv7xTe6b07+x7hXsl97ftlD0weVP/u8Pvudq/2Iw9DHrY8Snx09zH/8csn8iefO+Y/pT0te2b+rOa52/PDnWGdl1+MedHxUvqyt6v4D50/ql7Zv/r1z6A/W7pHd3e8lr3ue7P4rdHbbX95/NXcE9fz4F3+u973JR+MPmz/yPp49lPqp2e9kz+TPpd/cfjS9DXq672+/L4+KU/G638VwOBAs7IAeLMNAFoaAAzYt1HGqHrBfkFU/Ws/Av8Jq/rFfvECoA6+v8d3wbebmwDs2QLbL8hPh71qHA2AJB+AursPDrXIs9zdVFxU2KcQHvT1vYU9G2klAF+W9fX1Vvf1fdkMg4W943GJqgdVChH2DBu5XzLzM8G/EVV/+l2OP96BMgIP8OP9X9uwkKokUhnxAAAAimVYSWZNTQAqAAAACAAEARoABQAAAAEAAAA+ARsABQAAAAEAAABGASgAAwAAAAEAAgAAh2kABAAAAAEAAABOAAAAAAAAAJAAAAABAAAAkAAAAAEAA5KGAAcAAAASAAAAeKACAAQAAAABAAAAzKADAAQAAAABAAAAjAAAAABBU0NJSQAAAFNjcmVlbnNob3RiGcU2AAAACXBIWXMAABYlAAAWJQFJUiTwAAAB1mlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4xNDA8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MjA0PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+CikFak4AAAAcaURPVAAAAAIAAAAAAAAARgAAACgAAABGAAAARgAACGA+L1ndAAAILElEQVR4AexcsZLdRBA8p44dE5MQ2fwQARR8EBQO+CHOEQkxsWMXkBjPXY1rGHevNDsraSXNq7paabXbM+rp3pWez/fi33/+/vhQn2KgGFjFwIsyzCqealAx8MRAGaaEUAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGaY0UAwEGCjDBMiqocVAGWYHDXx49/vD+7e/PEWS41ff//jp56cdIs8bwnLy8vWbh5evv336mTfj58xuaxhfsNECVnxp/UfE8dWvv/nu3c41Nwkox2LgPQSrcWfkZC35tzSMFOyvH777H0ciYBFN5qOCEAwkCsU+yjCaH8ptq5wk1vPPY5MT4ebrxz+UomnbWxrmzzfffFGQrGHEgEiIXwT61LHnI5nkJI+DS7mNNszauMrP6PiKO7q9nWHev/358/uEJTOzujFMi2+P9zBMVLCjc0KLkuXAH4+O7/FHnd/OMKiQ2WIhzFaBMuZs4eq1qIFlXpYDjS1tZLfVeSPjK+YW7a0Mw4SUKRbDZMXKxGKYtj+aj84dlVdv/Owjsd7H1u2tDMNWvsyKP9vuEs1HBTbKML3xMzXQe9ijvY1h2MqXEQrDZIXb+sU2mo/NM8OD4vTG35oXzW9Ee3vDZFa2qEBGiLJV9Gg+FmtEbr27y4jY9l62PL6NYVAxs4VCmK1iZczZwtVr0Xx0nrTZd4iMWbOx7X1sfXwLw7BiZgzDMFnBMrEYpu2P5mPnynFWtBmzbr2Q+HvNnN/CMKyYmUIxTFaMrCAZrvZnDZPhIhP7TO8vwvXlDcOKmVnxGaaKF7UZQSI83xc1sJ+fyS8TO1MHfw97nN/WMBmBRA2ztSii+XhhZVb5bOytufH3mj2/vGHQ6pctEsJsFSJjzhauXmP/vqTXl9qMYbKxt35UXbr36PVLG4atfhnDMExGfCYWw/T9UQP7+b05RrnwceV868UExcz0XdowTEiZIjFMVoReMTI83z9CtL05ZmNndjbPw17nlzUMK2avOKQgDLNVrIw5W7h6LftIJDg9j0U9XGjO2mZqoRh7t7czTEbAUZHsIYjojocEVoZBrOC+yxoGCSkjYPn/JbKaRz4Zc66JEzUww+zJE/HL8Fl/T1yGtVf/JQ3DhJQxDMNkhcrEYpi+f8TjmGBGhRvlwuet59G4Ou/I9pKGYatfpkAMkxVvD8NEc0K59rx4j4i7Bz/ofrN9lzMMW/0yBWKYLfIz5mzh6rWenHSubaO8oLiC8eHd8h+5yMS1c488vo1hMgKOPvpERdgjgGhOLEY0V7S7CIb+3TUWx/f3fNHgMY44v5xhWEF7/+4YWlGXCpUx5xK2Xkf3KY9XS38dRudrGzEM4kJiCsZsX4jo/Y1uL2UYVFAhLCIKTzDD9OP0PBNLMZZallOPYSIrPTKp3K98ojvMHovKEo891y9lGFRQISVTHIbJyN7DMOxxTGJvJVxmUuGWXTuSIxY7238Zw7CiZQTMMFukZ8zZwrXXkIm3XulZTHnUZQa2OdvjTE0szhHHlzdM5JHDFwCJxI+x53sIgZm4xzBrv1JmMXVxiPKUqYnl+4jjyxgGFW2tIBDxTCRorPbtIQS2mvc8Gq3lB3Ghi8OMvwGh9diivYRhUEGFLC1qD3EMk2GtFR+bv7YfLQx6n8xMDFvnsevSz3jQ3YVdZ5h78cTiZ/svYRgmFC1qD0lImC0cEYIIUD7+q93er7R9PCZOFT7jwePouc7Tc9SimHYeuo5wtM/O1b4ztac3DHskyBQmKoI1BR/xuMYMoQsDu87yW5MTWjg0nuBGY2bqwu5jz/7TG4aJO1MYJJJsUTL5aGyUl8VF13UuapcMg7i18QRzdEyU50x9pzcMK5hdBVuEyw71/PP4NMw/TrXmRq55oUXmylgkXum3uIwLGYc+SxwhPGsytrujWNq3FFPHzdqe2jBrRGSJ38scNqYcj3jRZY8+VoBI4D4Xe27n2n45Rtxac7IxHseej+DB4h1xfEnDiBCOMocvoheZv772HJnBYkdX+yXxLsWTvJGpWvdj822Nm/naqQ2DijoT2aMEwoRp8aOGsXM9Zyye35HYrufx9LwVU8fM3p7SMLp7RH9vas9ijBTHGgGzMeyeW/khLDQ+umDZ9x+W1+z9pzCMGEQ+YhA9nplYJK5MvkiYPgYSeSumn2/Honh+d4nuaILvMWzMsxxPaxgpiO4gZzCJLfhIYTAjeMGzcTYve8xWe4TjYwkOGmfx/fHSO5MfP+v5NIYRUzz/xP6r62zEMiH25smE6U0ZfZ9gea7ZXeReWF7sPpHp2NiZ+w81jO4iZ9tBWEGZCNn4Nf1IwGi1jhrGG05yQSZgQo/GYzhrOJhpzGGGQcWZiZhILiJgEYS0Iz+MIyQ+ZKxWLsgwCAPFElw0thVvi8WkFW+ra4cZJkr4VgRkcLcyiubEDLNW7IrjW7RDRWIJXrR+KGef1xnODzMMK9AZSJPVUj6jdxR/70yUSHxsrMfUvPUe9DqqB9td0FjFQS0yKBp3hr7DDCPkRImfgVAmotG5yXudvCf4D4rPxvq5eo4wkOGQMQUjWjcUT3M5W3uoYZSsaAF03hHtXsVnnKB3gaxhUKzWfaLxrVq0sFrzZrw2hWGEmGgRsmTKY0LPt3Ns1c3m4+ezb6FQ/Ch3XsCR3UXyRON9/vYcmdxeP9PxfwAAAP//XA908gAAB6RJREFU7VyxshxFDHxOHTsmJiHy44cITMEHQeGAH8KOSIiJHbuAxFj3EAiVWjPa2ZuT3mmrYHd2Z1qtlnp2D1x+8deff3x6SHJ8ePvDw4e3P57O5uXrry+Yr958dznz+LfHr0KxaP2rN9/DNR/f//Iv/5evH/+JiedDoM8PLG4oflQ3iWOtlc8tjhY3ax7f+/Ldr3xZ/vwik2FITauAUZXJENSwT+cns2iMI3G++OnnC6bGovHv337zQIbRx6j59HwaI24IC823sOmexLHWyucag3KkXGcPqgHp9lyOdIYhYa0iIsH5bUFFpoPHaD7fj8TgNWinHGF5RmNseUbmOxpfYtO15GO9LVAcWjvKlebIwzOfnFflOqVhSDwqzNMZf6J5hb0sdv5lNYoz/X+7sp4nsZ7eao+f+f/HO7rLSjyO5TUeMhiv1Wc2jNX8XhzCsdZofDke4cm5Fa7TGobFQ58AK4WIFp24IHNqfjxPNn3EMIibl+9Rw0iOrDfz57E+H42lcaqO0xsGNRDvkkeER5gIy2tWyzAa31uvY+q1/Nxr5CNNTLzlW5DizPC0TMYcrbPH25qf/V56w6ACrRQCYaJieY2kDUNzdSNGuFrcRm8oaw3Khe7TZnPEMDpXLwY9G/Eerc/4PLVh0G7rNfBIZITprRs1vNewEa6I2wjDi2/lRfnoNTPNjfhZMejeiDdal/l+ScOMGtgTPPr5MlN0hDnThJIrasjR56dufolpXVNO+i04kyfiZ8WgezOYaG3W+6kNYzXCahEsTK84M/FQI40aXcdF3EYbBFqn8b3xKAatRRsDwo3mj3Ay3U9rGNSEMw2MBEaYaD7dP9pIUZ7o98EMzqphZmKQFtE4M9oRbqUjrWHQbrZSBISJCjbTSBZm9FOM4iMzz3CINrLOd/ZNEIlzRAPNK+M4rWGs4sw0jyeyhenNHzWSZRbGixobYc3gRPNijnSebWz0BpRY8nq1VhIr03VKw6zstkhchInmjxpphMdmo0ajf+jw/uCm1fSzTWetRXnp+7MxRvkexdXrso9TGgY1wMxuiwRHOzia7zXSTPOQ4egPgMr/GoX4IzyPg+SN9JJz0DXipOcjjnoej3nD4PFzOaczDCrMbPOgwkSbymskC4v4kUnImOhAmChnNF/jW3z0HGsc0TS64cxyt3hlvlfGMCs7FmpIVBivkRAWNYj3ne9hWg0/+iSU3K318jm6jmgaiRHhjrhlvZ/OMFZhVgsQ3R295raw5HzrORVfzpHNgEyG5su1fG1pxs/QOappJEaEO+KX9X4qw6Dde7UAkWJTobzPCQuL+FED0u8V/oEvC+7xRzlHdn9kUslBX3uc9FzEUc/jcQSb11Q5lzCM18Ajoc8utmUYj8NoJ0fNHskZYXi8IvhRDSNm9zhmfJbKMFYzru5W0WYaxbM4eoUdGcbCG3HQ8c7OUeNHDRMxo46VfZzGMKgo0ebRglsNqefI8ajY0eYkbLTjnpUz+h0k85LXUU0jOY82CMmj4nV6w4wa2BMdNSRaM9NI0eakWFHDHMk5sjFE8SPYMxoi/SvcT2MYqyir4l/DMFRUD5d2WOJNxuL/aYnyODNnj5NsRMRFzpHX0Q0iii9jVbhOYRhU7FXxrYb0ihLdeSVvNgqd+aDnH9+/u7xh+B6fUSOu5Cz5cBx9juaIeGpcHqO3KT+vfk5hGPSNHC2uLMZM88j5K40qcWavEb+VnCk2wqVnR35feHiEqY9V/hov2ziFYaw3wWoDRwu9Gi9aWGuTONLQVlwLm+YdyTGi41n8rZyy3Lu5YVBBVl/tlgk90XfvjBa/Iw2NcrLwj+SIzGfFPZO/hZ/h3s0NYxXkjJ3Kahgk+O5CX2uTkPnp3x5Hc8yso8x31/VNDaOLykkfLS6vRw3Jz/V59W2m8UZjxO/IG8CLxXFWNqCIYXbr6OV+rWc3NQwXVCe32jjIiDoOj1fjMc7s2Xqrrm4Ss7Gj8yKG2a1jNJcz5qczzFmNM1vos+JFimFxuwWPGc5oU7PWtmEsVU68Z+20Z4k++5a5xWeEzjurWbjUM6bJngPnsnq+6RtGN/XZos8U+iyDRgoh8175fRGJuTp3pOXZtVvle631NzUMJ0UNRI1zjUM2p8a/dZGvmbfO9Yyxp+Ut3tRn5BTFSGGYKOkj8/VnEGHcS5GP6IXWINPc4k2NOF7z/t0Yxir0vRT57AayPs/uRcu7MQw1DZmG/wQx/RVI3t8TdnaTPTc8Mg39wVI67knLuzLMc2vazme/Am2Y/Zp3xMIKtGEKF6+p71egDbNf845YWIE2TOHiNfX9CrRh9mveEQsr0IYpXLymvl+BNsx+zTtiYQXaMIWL19T3K9CG2a95RyysQBumcPGa+n4F2jD7Ne+IhRVowxQuXlPfr0AbZr/mHbGwAm2YwsVr6vsVaMPs17wjFlagDVO4eE19vwJtmP2ad8TCCrRhChevqe9XoA2zX/OOWFiBNkzh4jX1/Qq0YfZr3hELK9CGKVy8pr5fgTbMfs07YmEF2jCFi9fU9yvQhtmveUcsrEAbpnDxmvp+Bdow+zXviIUVaMMULl5T369AG2a/5h2xsAJtmMLFa+r7FWjD7Ne8IxZW4G9QY3wqTDWBMAAAAABJRU5ErkJggg=="}}},{"cell_type":"markdown","source":"# Analysis preparation\n\nWe will include here the required packages for reading, parsing, filtering, processing, visualizing the data, both tabular and image.\n\n![image.png](attachment:58798c57-524a-45c9-b7cc-6f4f6925a0dd.png)\n","metadata":{"execution":{"iopub.status.busy":"2022-05-21T06:21:47.766942Z","iopub.execute_input":"2022-05-21T06:21:47.767448Z","iopub.status.idle":"2022-05-21T06:21:47.796318Z","shell.execute_reply.started":"2022-05-21T06:21:47.767333Z","shell.execute_reply":"2022-05-21T06:21:47.794693Z"}},"attachments":{"58798c57-524a-45c9-b7cc-6f4f6925a0dd.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAcQAAABeCAYAAAC91WSOAAAMbGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkJAQIICAlNCbIFIDSAmhBZBeBBshCSSUGBOCir0sKrh2EQEbuiqi2FZA7NiVRbH3xYKKsi7qYkPlTUhA133leyff3PvnzJn/lMzk3gMA/QNPKs1DtQHIlxTIEsKDmaPT0pmkpwCBHzKwA3QeXy5lx8VFAygD97/LuxvQFspVZyXXP+f/q+gKhHI+AMhYiDMFcn4+xMcBwKv4UlkBAESl3mpygVSJZ0OsJ4MBQrxKibNVeLsSZ6rw4X6bpAQOxJcB0KDyeLJsALTuQT2zkJ8NebQ+Q+wqEYglANCHQRzAF/EEECtjH5afP1GJyyG2h/ZSiGE8gJX5HWf23/gzB/l5vOxBrMqrXzRCxHJpHm/q/1ma/y35eYoBH7ZwUEWyiARl/rCGt3InRikxFeIuSWZMrLLWEH8QC1R1BwCliBQRySp71IQv58D6AQOIXQW8kCiITSAOk+TFRKv1mVniMC7EcLegU8QF3CSIDSFeKJSHJqptNsomJqh9ofVZMg5brT/Hk/X7Vfp6oMhNZqv534iEXDU/plUkSkqFmAKxdaE4JQZiLYhd5LmJUWqbkUUiTsyAjUyRoIzfGuIEoSQ8WMWPFWbJwhLU9iX58oF8sY0iMTdGjfcViJIiVPXBTvF5/fHDXLDLQgk7eYBHKB8dPZCLQBgSqsodey6UJCeqeT5IC4ITVGtxijQvTm2PWwrzwpV6S4g95IWJ6rV4SgHcnCp+PEtaEJekihMvyuFFxqniwZeBaMABIYAJFHBkgokgB4hbuxq64DfVTBjgARnIBkLgrNYMrEjtn5HAayIoAn9AJATywXXB/bNCUAj1Xwa1qqszyOqfLexfkQueQpwPokAe/K7oXyUZ9JYCnkCN+B/eeXDwYbx5cCjn/71+QPtNw4aaaLVGMeCRSR+wJIYSQ4gRxDCiA26MB+B+eDS8BsHhhrNwn4E8vtkTnhLaCI8I1wnthNsTxHNlP0Q5CrRD/jB1LTK/rwVuCzk98WDcH7JDZtwANwbOuAf0w8YDoWdPqOWo41ZWhfkD998y+O7XUNuRXckoeQg5iGz/40otRy3PQRZlrb+vjyrWzMF6cwZnfvTP+a76AniP+tESW4jtx85iJ7Dz2GGsATCxY1gj1oIdUeLB3fWkf3cNeEvojycX8oj/4Y+n9qmspNy11rXT9bNqrkA4pUB58DgTpVNl4mxRAZMNnw5CJlfCdxnGdHN1cwNA+axR/X29je9/hiAGLd90834HwP9YX1/foW+6yGMA7PWGx//gN509CwAdTQDOHeQrZIUqHa68EOC/BB2eNCNgBqyAPczHDXgBPxAEQkEkiAVJIA2Mh1UWwX0uA5PBdDAHFINSsAysBhVgA9gMtoNdYB9oAIfBCXAGXASXwXVwF+6eDvASdIN3oBdBEBJCQxiIEWKO2CBOiBvCQgKQUCQaSUDSkAwkG5EgCmQ6Mg8pRVYgFcgmpAbZixxETiDnkTbkNvIQ6UTeIJ9QDKWieqgpaosOR1koG41Ck9BxaDY6CS1C56NL0HK0Gt2J1qMn0IvodbQdfYn2YADTxAwwC8wZY2EcLBZLx7IwGTYTK8HKsGqsDmuCv/NVrB3rwj7iRJyBM3FnuIMj8GScj0/CZ+KL8Qp8O16Pn8Kv4g/xbvwrgUYwITgRfAlcwmhCNmEyoZhQRthKOEA4Dc9SB+EdkUg0INoRveFZTCPmEKcRFxPXEXcTjxPbiI+JPSQSyYjkRPInxZJ4pAJSMWktaSfpGOkKqYP0QUNTw1zDTSNMI11DojFXo0xjh8ZRjSsazzR6ydpkG7IvOZYsIE8lLyVvITeRL5E7yL0UHYodxZ+SRMmhzKGUU+oopyn3KG81NTUtNX004zXFmrM1yzX3aJ7TfKj5kapLdaRyqGOpCuoS6jbqcept6lsajWZLC6Kl0wpoS2g1tJO0B7QPWgwtFy2ulkBrllalVr3WFa1XdDLdhs6mj6cX0cvo++mX6F3aZG1bbY42T3umdqX2Qe2b2j06DJ0ROrE6+TqLdXbonNd5rkvStdUN1RXoztfdrHtS9zEDY1gxOAw+Yx5jC+M0o0OPqGenx9XL0SvV26XXqtetr6vvoZ+iP0W/Uv+IfrsBZmBrwDXIM1hqsM/ghsGnIaZD2EOEQxYNqRtyZch7w6GGQYZCwxLD3YbXDT8ZMY1CjXKNlhs1GN03xo0djeONJxuvNz5t3DVUb6jfUP7QkqH7ht4xQU0cTRJMpplsNmkx6TE1Mw03lZquNT1p2mVmYBZklmO2yuyoWac5wzzAXGy+yvyY+QumPpPNzGOWM08xuy1MLCIsFBabLFotei3tLJMt51rutrxvRbFiWWVZrbJqtuq2NrceZT3dutb6jg3ZhmUjslljc9bmva2dbartAtsG2+d2hnZcuyK7Wrt79jT7QPtJ9tX21xyIDiyHXId1DpcdUUdPR5FjpeMlJ9TJy0nstM6pbRhhmM8wybDqYTedqc5s50LnWueHLgYu0S5zXRpcXg23Hp4+fPnws8O/unq65rlucb07QndE5Ii5I5pGvHFzdOO7Vbpdc6e5h7nPcm90f+3h5CH0WO9xy5PhOcpzgWez5xcvby+ZV51Xp7e1d4Z3lfdNlh4rjrWYdc6H4BPsM8vnsM9HXy/fAt99vn/6Ofvl+u3wez7SbqRw5JaRj/0t/Xn+m/zbA5gBGQEbA9oDLQJ5gdWBj4KsggRBW4OesR3YOeyd7FfBrsGy4APB7zm+nBmc4yFYSHhISUhrqG5ocmhF6IMwy7DssNqw7nDP8GnhxyMIEVERyyNuck25fG4NtzvSO3JG5KkoalRiVEXUo2jHaFl00yh0VOSolaPuxdjESGIaYkEsN3Zl7P04u7hJcYfiifFx8ZXxTxNGJExPOJvISJyQuCPxXVJw0tKku8n2yYrk5hR6ytiUmpT3qSGpK1LbRw8fPWP0xTTjNHFaYzopPSV9a3rPmNAxq8d0jPUcWzz2xji7cVPGnR9vPD5v/JEJ9Am8CfszCBmpGTsyPvNiedW8nkxuZlVmN5/DX8N/KQgSrBJ0Cv2FK4TPsvyzVmQ9z/bPXpndKQoUlYm6xBxxhfh1TkTOhpz3ubG523L78lLzdudr5GfkH5ToSnIlpyaaTZwysU3qJC2Wtk/ynbR6UrcsSrZVjsjHyRsL9OBLfYvCXvGT4mFhQGFl4YfJKZP3T9GZIpnSMtVx6qKpz4rCin6Zhk/jT2uebjF9zvSHM9gzNs1EZmbObJ5lNWv+rI7Z4bO3z6HMyZ3z21zXuSvm/jUvdV7TfNP5s+c//in8p9pirWJZ8c0Ffgs2LMQXihe2LnJftHbR1xJByYVS19Ky0s+L+Ysv/Dzi5/Kf+5ZkLWld6rV0/TLiMsmyG8sDl29fobOiaMXjlaNW1q9iripZ9dfqCavPl3mUbVhDWaNY014eXd641nrtsrWfK0QV1yuDK3dXmVQtqnq/TrDuyvqg9XUbTDeUbvi0Ubzx1qbwTfXVttVlm4mbCzc/3ZKy5ewvrF9qthpvLd36ZZtkW/v2hO2narxranaY7Fhai9Yqajt3jt15eVfIrsY657pNuw12l+4BexR7XuzN2HtjX9S+5v2s/XW/2vxadYBxoKQeqZ9a390gamhvTGtsOxh5sLnJr+nAIZdD2w5bHK48on9k6VHK0flH+44VHes5Lj3edSL7xOPmCc13T44+ee1U/KnW01Gnz50JO3PyLPvssXP+5w6f9z1/8ALrQsNFr4v1LZ4tB37z/O1Aq1dr/SXvS42XfS43tY1sO3ol8MqJqyFXz1zjXrt4PeZ6243kG7dujr3Zfktw6/ntvNuv7xTe6b07+x7hXsl97ftlD0weVP/u8Pvudq/2Iw9DHrY8Snx09zH/8csn8iefO+Y/pT0te2b+rOa52/PDnWGdl1+MedHxUvqyt6v4D50/ql7Zv/r1z6A/W7pHd3e8lr3ue7P4rdHbbX95/NXcE9fz4F3+u973JR+MPmz/yPp49lPqp2e9kz+TPpd/cfjS9DXq672+/L4+KU/G638VwOBAs7IAeLMNAFoaAAzYt1HGqHrBfkFU/Ws/Av8Jq/rFfvECoA6+v8d3wbebmwDs2QLbL8hPh71qHA2AJB+AursPDrXIs9zdVFxU2KcQHvT1vYU9G2klAF+W9fX1Vvf1fdkMg4W943GJqgdVChH2DBu5XzLzM8G/EVV/+l2OP96BMgIP8OP9X9uwkKokUhnxAAAAimVYSWZNTQAqAAAACAAEARoABQAAAAEAAAA+ARsABQAAAAEAAABGASgAAwAAAAEAAgAAh2kABAAAAAEAAABOAAAAAAAAAJAAAAABAAAAkAAAAAEAA5KGAAcAAAASAAAAeKACAAQAAAABAAABxKADAAQAAAABAAAAXgAAAABBU0NJSQAAAFNjcmVlbnNob3Tfls5TAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB1WlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj45NDwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj40NTI8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpVc2VyQ29tbWVudD5TY3JlZW5zaG90PC9leGlmOlVzZXJDb21tZW50PgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KVcPLJwAAABxpRE9UAAAAAgAAAAAAAAAvAAAAKAAAAC8AAAAvAAAK4eAXefkAAAqtSURBVHgB7F2BkRQ5DDwyggQuFoiAWIiAi4UEIAjy+J/e/y66VLIsz87csrvtqmNmbKkltWRrZ49/Pvz+/fufFw8zYAbMgBkwA0/OwAc3xCevAIdvBsyAGTADFwbcEF0IZsAMmAEzYAY2BtwQXQZmwAyYATNgBjYG3BBdBmbADJgBM2AGNgbcEF0GZsAMmAEzYAY2BtwQXQZmwAyYATNgBjYG3BBdBmbADJgBM2AGNgbcEF0GZsAMmAEzYAY2BtwQXQZmwAyYATNgBjYG3BBdBmbADJgBM2AGNgbcEF0GZsAMmAEzYAY2BtwQXQZmwAyYATNgBjYG3BBdBmbADJgBM2AGNgbcEF0GZsAMmAEzYAY2BtwQXQZmwAyYATNgBjYG3BBdBmbADJgBM2AGNgbcEF0GZsAMmAEzYAY2BtwQXQZmwAyYATNgBjYG3BBdBmbADJgBM2AGNgbcEF0GZqDBwM+fP18+ffrUkPwjAp23t7cXXL98+XL5+bO67w5YHKv+UI9+4fnz58/LcRHHVzPwaAy4IT5aRh3P4Qx8//79BT9oQN++fSvxtdnE5jXTrYCJS8yOL4oX9bl2VKMmnq9m4J4ZcEO85+zZ99MZYDOEoU7zeH19TX3q6GaKaj+u//jxI06lzxXGXr9SQ540A3fOgBvinSfQ7p/LwNevXy9fecLKrHlUjQdvh6tfcVZ48KfTEI/AgC0PM/AMDLghPkOWHeMuBvA1Ixoix6wBjZrP6tebtDd628R6F3PkE23MYqKcr2bgGRhwQ3yGLDvGXQxoM5m9HcLAqIF1dKODajuu4bnTEGcYe/zKfPGcGXgUBtwQHyWTjuNwBrTBzb7yrJrPnrcwtZ0F1mlmlU/A7GBktj1nBh6VATfER82s47qKAW0mnbexUQPb03TiV7VZIDNc9T/Tx9ysyY/0PG8GHpUBN8RHzazjuoqBo/4yzaxxZU52mtkMt4Ox580189dzZuBRGHBDfJRMOo7DGIhvaLPGoc0zOjHTjfJ4Hr1tquzs7W6G0XnrVXu+NwPPwIAb4jNk2TEuMaBvV7M3sdg81dBMV2V5r7Y5l12rhtjB2ONb5ofnzMAjMeCG+EjZdCyHMKBvV1XjgbGq+ex5O6zwNLgKW/1XHb13Q1Q2fG8G/mPADdGVYAaEAW1Ina8VR81nb8MZ4YmLl9tRQ1T/o44+zxq9yvreDDwLA26Iz5Jpx9liQH8fOGtqVfOZ6WbOVHgqXzVqbaiQw1e62Rg11EzWc2bgWRhwQ3yWTDvOKQPx94GzpqHNJ4LPdKM8nrUhVs1s1GxVHzJ4zkbVUDN5z5mBZ2Hgbhoi/xda3X+uZkUeB8evX7+mOZ/ZJs7Hjx9b/9QP5WEYX2HNBmLqYgPrFhwwBsbGNxQcwiu+E+c9r/AZPxijpkN/VJZzvM50KRev2mCrhjbCV300ZH1WWyN9lTnyHjWAfwZrT/5ZR7O9N/L3Gn3qAruzPynfiZOyI787GKpLjjFXcTWzq5gzLJ4vUSd7rnzK5G81dzcNkZu7u5lX5FEk+JmN2e9dFGcmC1sq33mjYEzdT/iU73CmvlQ8dOLCRmEjzLA6GJne2XPkC3ZmPlZ8dXIZY4l4yBnmspHlU/WxjrGin9k5ag61wMNzxqvaVL0sZpUd3TOn3T2jOMppJ6es+44txVab8b7Ll3JV6XTt0o8Ki9xStrruzV+FecaaG+LGqhYJD5OM7GoN8oqD56qYonxnw2kBdgqM8h1Z9b2Ks1qLMeFgwCdDvn3DRuewAM57D42/4yO5jX6CH3y65wBWZ6h9coy5bGR1pf6glhQvYnRqLepc86yHdYdb2tIYwAl54frsqvqQzXirMFS/w9nehpjFBdsYXb6U4ypOyHE/MnY8Yx62tHaxnvlGPfrI53glLuY7/EX9Wzy7IW6srxb+KFGKA5lZMat8p2D00AP+7JCg/EwOWKu+QCcbKzYz/VvNafwzvlS24+9qbnGg4TCBnWxEPPWHvvNw7uhnMkfO6WENXPpY2dCYujoRj7XI+dl+pByv6kPknDJ6JecdOzNs5axqcLS/Kk89XOl3Jy+qV91rfEfiVjaPWHND3FjU5HUKf0S84lCmKgaV79iNGxw2qs1C+coH+rnqC/X0es2mVJxb3JMr2I65QFxoUPjBwPPKiHhRV3nDGuQ1H1E+4qks1zQe1e8c1ip/xH2MD5hV3WI9+t+pYehxKCecw5X86NzoXjE6emwsHY472OSgE7tyPOM2xku/O3ai7uh5xfcRxi3m3RA31jvF2UkOcbAh8NUDnjFGhUZ5yHQ2nBYZsStdlYcP1Vj1ZYRFm51DYYTx3vMaO/zG17z4SyAYq83vovT/H8TCtRpqn7XCQyrqZbySc+pCh3NRX2Xi2lnPeljTRhYH15QPzq36rRhoEOATY6VZKEZnfzJnVWyMZ4atnHViV/mVGOEP/e7Yof/VdRZbpXvrNTfELQNHJZA43BAsNCQ5K1LKY72z4XjIoXDRcLnJaQ84OlQeOtVY9WWEpTEftcFGto6Y14PkCDxijHLCdb0yT5gjZ8qjykZczRtrTOdUV/Hj/JnPyjH8x/PIF/UdXOB5JHtZGPxBTiOfkb+B+mVafensT+asY6PCBj/4QEaeOraVY9ZBFZuu0W9ypWur9xrXEXir9q+Vv7uGuBpwJymaxAy/U+DQI47Ks9iwHguV8ljrFH3c5KqfxRnlYWc0FCuTyfAzOcxpzHhe0YX8mYOHjB44Z9iLuR7Z0IMMMqwD5i7qRS4pp/NVLokfcc981hjBi3If/WE88AdrfNb4Zr5q/MSPPmCPzkaGU+mw7nX/j+QVeySD+W7ce+KjXfrdtUW97LonXxnOrebcEDfmZ8XZKXAkkDgqr4Wq8yqPe25c3I9GVmy0CZ1Y0Jn8CFtxMpmIncnoXIa3iqF419wjB3oIX4PV0V2JU3lSPeYu2lOZka7OR/1OnUWda591D6AhYuAQxtA9oX4zTvLA54vS5I+RDg9+tVlBqT8d3lbwFTvzAT7iWyDE3RmRY+h3B/1e4TjD1pg6fGUYt567u4aIpMW/GpyRyA3XSbImkhs2YnYKjDhxw2mx6hrlYatTQLONDhx9MxnJQy4O9eUaDipcrGn8UfaMZ43rDHzFRGz4/SOu3cHDCPJaq8xdxBnJaP2MdN+be/qu9c/61Lyw3rI9y1g0buJm14irudA15SvDwdyqPHPZ4VmxGb9+aCNPI9/ifMZxlBk90+8uxxmOxnMNTob9nnN32RBB+GysbCRNZmejjGwTJ9sQXIMuC0bnOnarmFjUwOdmquQhp2PVF9Xt3Ct+xk8HY4+M2t2j39FBPvEhTQ/fjh5kmCPcswb0cMO8DuZW42I9UU4xOYdrlNO1M+81HvoPe6xZ8Ab+EBMGecA9Y+n6Tl6AiQ8nOvQ/ZVE/VEbviYU59Ull9J7xdHzNsJWn1T2iup3Y9vqterxX253Yqfc3Xt0Qt6xkxbknWcQZFTM3DLBRtLpBOxuuOhy0KGm/ko/x0XfMd3yJ+p1njf8sG5kfiA1cg6Mzxt5DQDlXDM1l9JeHneoql5Wu2oi4Zz6rT/Qf9nSe9qOPKzUMDMoTb3TlHhmtY179U45HOrQdY8jkR/nT+Q4OsdVX5Zjr1ZX7csWe4lEfcx2eVPdvu/8XAAD///8tmvgAAAfNSURBVO2bi5HUOhREh4wgAWKBCIiFCCAWEoAgyINHD9WP5pb18SwzMuOjqsWydX86kt1rA6++f//+4/IPtLdv316rfP/+/UU/o7bH/tOnTxf9qH358mUUujnuOG/evLl8/Phx0+7Dhw+Xr1+/Xsdk5/5M3tGcnF/BxchzmmGWvjO1bE5ucFFz1fzVxEfzf2RL9qO8ru3du3dXU9e95Vd5mbtse3s168k1yrWo+ZyrtRdmfGvMe5/31r3FwDW15unxPPbmnnbuj/Zgr27HyOOttXpNHSuZjGq0z95a7aej8+UezPFeP5nf4t+LvWLsFYJ4uQqHFlatbs49i+LN0RPE3LgZeybvzA3nGjL2zEZNv5laMn72NT81C0qO/a0cGXNPX7V9/vz5/19C0lf1vn79+vpTa2+tmfwr25yjxnsPNK+n7JJ5jaFxN9nleI2fY/bxMXP42iOOya/Wq/x6IKtprDYzqpyrnc5nbS0AvfvU8R1zZJvcZziP7J1XdczEGzH2fLaO5jHDOP0z517fjHOkPoL4czVGm3N2wRxndPPkRnLsmU3vm2S0+bzBHXtkLzvXrv5MLbKrLWPUnHXOt+aoOW85zzrlv/WQzrjVPseqr9dIe0Bzbu2FjFltcixzqS9umaOKSF17+9ccvv6IY6595TXK77nW/VT9ktlob+2pJ21bNSTzls3eejPvzNql/V7Grn+2ds/FfjofMbfP0Y8I4s8VyptJm6/V9PmsN+44MxvYts41s6FmHw6KmZt1ZqNnPb059hhkDM9Lub99+/bHW9lMPfa/x7HWOWJv7lu1pK8fSuKnt03lac01a6g2OVZzylbjatVP11q1btnK/hHNXJRr78Pa8xnV7/0+svN8Z+PKvq5H3h+am9vMfW/bjJl7yOM6ps1oXi9hvJddrU3nyUTnW63+8rZls/oagvhzBXLj9RZktCkdZ/bGsL1ytm6KrGfPTZw3yKhu5chaMmftj2Ipb+uzpGKN/Gu+e5znXEf1pG2tZcvXa2Tb1rqmXRUJP6AcI4/KqZrUauxc8/RRf6vWanOv86yrznWU05x69Wf8nl3mMuNb7tWM4/5sXtvnvqrraBsdXaf6PXbJoGenOLU5x5452KfG6p335tnze+TYPyOIfgjoN++Z30b22Gsz6S1m1Ea5HWdkl3lcpzbjqNl2Nv6eemw7qmFvbsUTW/nN+o5qeOm4H7KKM7pJxdzca94t33xQtB5MYi07txonY9hGR+17+aptPbz21noN9IA/cm/N7PMsyex7e+eW+Lf4qC7X4+eF/9HVzDMp57Unv3PKv8VvT7ysQ33H7zFu+dTrvfNW7T2fR4/9M4L4aDDke04Cuvn9ANgSlTrrFM8c2/KdjT2yawli5q8iqrGMm7bqb9lXG84hcHYCCOLZd8DJ5p+isSVqiSNt87r61bfa+lOcfnOvbw8psjWOYo8Eccun5+daZEODAATaBBDENhtGnpBAitHoramKXOKovo5r8bMQ6lg/ndpW8WocXbtVEDOu4ri1BNTjHCEAgV8EEER2wmkIpMDNiMSswNS4Aqprbil61VZ11DYSxIyXvrP1pg99CEDgNwEE8TcLek9OIIVmJIgpXBVL9a1xUwyrbcatY86T8XzNx5aP3kTlt9VaArplyzUInJkAgnjm1T/R3KtgjEQihatiqr49Adv7uVS5erlrPNfW86n12ocjBCDwJwEE8U8enD0pgRSM1ltWTn3P58cqtoqjv0usf3+YNfT+octWPMeUIG61jJ3jvTxpRx8CELhcEER2wSkIpMC13rIMoiVIGm+JqQQp/2+ahEjX7KNjilYrztWh2Ppaz6f1ltrzcVyOEIDALwIIIjvh6QmkEM28MaV9hfOSz48pWiNRVt5aRy93Cn7WjCAmDfoQ6BNAEPt8GH0CAiksMwKRwpXTn/FN+9pP0eqJW/q5llHujJ3+M8Kb9vQhcGYCCOKZV/8kc0+xmBGitE9EI1FK29rfK8rpL1/lbrXeJ96Z+bbich0CZyOAIJ5txU84XwvcrKD5rayieom4pCDe461tq+aZz8N1jpxD4MwEEMQzr/5J5i6xUJMQzbYqMLNi2orvt7iXxmnF1/UUXZ3fM5fi0yDwbAQQxGdbUebz1wikKN7jre6vFRqBUhQRxABDFwITBBDECUiYnJfALW+Xq2lZFF/yiXf1HMgPgRUEEMQV1MkJAQhAAAKHI4AgHm5JKAgCEIAABFYQQBBXUCcnBCAAAQgcjgCCeLgloSAIQAACEFhBAEFcQZ2cEIAABCBwOAII4uGWhIIgAAEIQGAFAQRxBXVyQgACEIDA4QggiIdbEgqCAAQgAIEVBBDEFdTJCQEIQAAChyOAIB5uSSgIAhCAAARWEEAQV1AnJwQgAAEIHI4Agni4JaEgCEAAAhBYQQBBXEGdnBCAAAQgcDgCCOLhloSCIAABCEBgBQEEcQV1ckIAAhCAwOEIIIiHWxIKggAEIACBFQQQxBXUyQkBCEAAAocjgCAebkkoCAIQgAAEVhBAEFdQJycEIAABCByOAIJ4uCWhIAhAAAIQWEEAQVxBnZwQgAAEIHA4Agji4ZaEgiAAAQhAYAUBBHEFdXJCAAIQgMDhCCCIh1sSCoIABCAAgRUEEMQV1MkJAQhAAAKHI4AgHm5JKAgCEIAABFYQ+A8Gmwr78P9zUgAAAABJRU5ErkJggg=="}}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n# from wordcloud import WordCloud, STOPWORDS\nfrom datetime import datetime\nfrom PIL import Image\nfrom plotnine import *","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:19:14.505238Z","iopub.execute_input":"2022-06-10T19:19:14.505612Z","iopub.status.idle":"2022-06-10T19:19:18.08454Z","shell.execute_reply.started":"2022-06-10T19:19:14.505501Z","shell.execute_reply":"2022-06-10T19:19:18.083544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ncustomers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ntransactions = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:19:18.086568Z","iopub.execute_input":"2022-06-10T19:19:18.087275Z","iopub.status.idle":"2022-06-10T19:20:27.332052Z","shell.execute_reply.started":"2022-06-10T19:19:18.087226Z","shell.execute_reply":"2022-06-10T19:20:27.33119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### first lets go through the articles that is selling on H&M","metadata":{}},{"cell_type":"code","source":"len(articles.columns)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:27.333276Z","iopub.execute_input":"2022-06-10T19:20:27.333493Z","iopub.status.idle":"2022-06-10T19:20:27.341023Z","shell.execute_reply.started":"2022-06-10T19:20:27.333466Z","shell.execute_reply":"2022-06-10T19:20:27.34031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we have in total of 25 columns. What are they?","metadata":{}},{"cell_type":"code","source":"articles.columns","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:27.343217Z","iopub.execute_input":"2022-06-10T19:20:27.343529Z","iopub.status.idle":"2022-06-10T19:20:27.354069Z","shell.execute_reply.started":"2022-06-10T19:20:27.343487Z","shell.execute_reply":"2022-06-10T19:20:27.353489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"article_id : A unique identifier of every article.\n\nproduct_code, prod_name : A unique identifier of every product and its name (not the same).\n\nproduct_type, product_type_name : The group of product_code and its name\n\ngraphical_appearance_no, graphical_appearance_name : The group of graphics and its name\n\ncolour_group_code, colour_group_name : The group of color and its name\n\nperceived_colour_value_id, perceived_colour_value_name, perceived_colour_master_id, perceived_colour_master_name : The added color info\n\ndepartment_no, department_name: : A unique identifier of every dep and its name\n\nindex_code, index_name: : A unique identifier of every index and its name\n\nindex_group_no, index_group_name: : A group of indeces and its name\n\nsection_no, section_name: : A unique identifier of every section and its name\n\ngarment_group_no, garment_group_name: : A unique identifier of every garment and its name\n\ndetail_desc: : Details","metadata":{}},{"cell_type":"markdown","source":"I notice that all of the data here is categorical / index for each feature. \n\nSo Let's do some question and answer style to think about this data!","metadata":{}},{"cell_type":"markdown","source":"So what product do we have here","metadata":{}},{"cell_type":"code","source":"articles.columns","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:27.35529Z","iopub.execute_input":"2022-06-10T19:20:27.355991Z","iopub.status.idle":"2022-06-10T19:20:27.367241Z","shell.execute_reply.started":"2022-06-10T19:20:27.355948Z","shell.execute_reply":"2022-06-10T19:20:27.365795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_count = pd.DataFrame(articles[['product_code']].value_counts().sort_values(ascending=False),columns = ['count'])\nproduct_count = product_count.reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:27.368842Z","iopub.execute_input":"2022-06-10T19:20:27.369174Z","iopub.status.idle":"2022-06-10T19:20:27.416647Z","shell.execute_reply.started":"2022-06-10T19:20:27.369099Z","shell.execute_reply":"2022-06-10T19:20:27.415648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_count.describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:27.417978Z","iopub.execute_input":"2022-06-10T19:20:27.418223Z","iopub.status.idle":"2022-06-10T19:20:27.458844Z","shell.execute_reply.started":"2022-06-10T19:20:27.418194Z","shell.execute_reply":"2022-06-10T19:20:27.458282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By looking at product code, we can see top one percent of product have multiple different variation. Like \\#1 most count of product is actually kid clothing!\nWhereas the 90% of the product is only average of 1-2 variation. ","metadata":{}},{"cell_type":"markdown","source":"From here we can note that the same product can have variation. ","metadata":{}},{"cell_type":"code","source":"fig = plt.figure()\nax = fig.add_axes([0,0,1,1])\nax.bar(product_count.index[:1000],product_count['count'][:1000])\nplt.ylim([0, 100])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:27.459883Z","iopub.execute_input":"2022-06-10T19:20:27.460247Z","iopub.status.idle":"2022-06-10T19:20:29.229045Z","shell.execute_reply.started":"2022-06-10T19:20:27.460208Z","shell.execute_reply":"2022-06-10T19:20:29.228217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_name = articles[['product_code','prod_name']]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.230277Z","iopub.execute_input":"2022-06-10T19:20:29.230478Z","iopub.status.idle":"2022-06-10T19:20:29.236025Z","shell.execute_reply.started":"2022-06-10T19:20:29.230454Z","shell.execute_reply":"2022-06-10T19:20:29.235217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_name.drop_duplicates()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.238657Z","iopub.execute_input":"2022-06-10T19:20:29.239035Z","iopub.status.idle":"2022-06-10T19:20:29.277832Z","shell.execute_reply.started":"2022-06-10T19:20:29.239006Z","shell.execute_reply":"2022-06-10T19:20:29.277008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_count.merge(product_name, how = \"left\", left_on=\"product_code\",right_on=\"product_code\").sort_values(by='count')","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.280971Z","iopub.execute_input":"2022-06-10T19:20:29.281401Z","iopub.status.idle":"2022-06-10T19:20:29.329256Z","shell.execute_reply.started":"2022-06-10T19:20:29.281364Z","shell.execute_reply":"2022-06-10T19:20:29.328651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I notice the most repeated product is kid clothing!","metadata":{}},{"cell_type":"markdown","source":"prod_name can be really creative, and can definitely contribute later. For now, I may skip this variable. ","metadata":{}},{"cell_type":"markdown","source":"Now I can ask a better question, So what product type do we have here?","metadata":{}},{"cell_type":"code","source":"product_type = articles[['product_code', 'product_type_no','product_type_name']]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.330687Z","iopub.execute_input":"2022-06-10T19:20:29.330904Z","iopub.status.idle":"2022-06-10T19:20:29.33593Z","shell.execute_reply.started":"2022-06-10T19:20:29.330878Z","shell.execute_reply":"2022-06-10T19:20:29.3354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_type_name = articles[['product_type_no','product_type_name']]\nproduct_type_name = product_type_name.drop_duplicates()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.336865Z","iopub.execute_input":"2022-06-10T19:20:29.337164Z","iopub.status.idle":"2022-06-10T19:20:29.35895Z","shell.execute_reply.started":"2022-06-10T19:20:29.337125Z","shell.execute_reply":"2022-06-10T19:20:29.358278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"we have 132 unique type in all our articles. Lets see how they are distributed!","metadata":{}},{"cell_type":"code","source":"product_type = pd.DataFrame(product_type.groupby(by=[\"product_type_no\"])['product_type_name'].count().sort_values(ascending=False))\nproduct_type = product_type.reset_index()\nproduct_type.columns = ['product_type_no', 'product_type_count']","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.360283Z","iopub.execute_input":"2022-06-10T19:20:29.361313Z","iopub.status.idle":"2022-06-10T19:20:29.375404Z","shell.execute_reply.started":"2022-06-10T19:20:29.361267Z","shell.execute_reply":"2022-06-10T19:20:29.374796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_type","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.376838Z","iopub.execute_input":"2022-06-10T19:20:29.377337Z","iopub.status.idle":"2022-06-10T19:20:29.388477Z","shell.execute_reply.started":"2022-06-10T19:20:29.377294Z","shell.execute_reply":"2022-06-10T19:20:29.387645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type_no_to_name = dict(zip(product_type_name['product_type_no'],product_type_name['product_type_name']))","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.389566Z","iopub.execute_input":"2022-06-10T19:20:29.389778Z","iopub.status.idle":"2022-06-10T19:20:29.400545Z","shell.execute_reply.started":"2022-06-10T19:20:29.38975Z","shell.execute_reply":"2022-06-10T19:20:29.399901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_type","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.40187Z","iopub.execute_input":"2022-06-10T19:20:29.402327Z","iopub.status.idle":"2022-06-10T19:20:29.416167Z","shell.execute_reply.started":"2022-06-10T19:20:29.402283Z","shell.execute_reply":"2022-06-10T19:20:29.415389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_type['product_type_name'] = product_type['product_type_no'].apply(lambda x : type_no_to_name[x]) ","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.417239Z","iopub.execute_input":"2022-06-10T19:20:29.417766Z","iopub.status.idle":"2022-06-10T19:20:29.429577Z","shell.execute_reply.started":"2022-06-10T19:20:29.417735Z","shell.execute_reply":"2022-06-10T19:20:29.428717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_type","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.430827Z","iopub.execute_input":"2022-06-10T19:20:29.431133Z","iopub.status.idle":"2022-06-10T19:20:29.44549Z","shell.execute_reply.started":"2022-06-10T19:20:29.431093Z","shell.execute_reply":"2022-06-10T19:20:29.444464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure()\nax = fig.add_axes([0,0,1,1])\nax.bar(product_type['product_type_name'][:20],product_type['product_type_count'][:20])\nplt.xticks(rotation='vertical')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.446645Z","iopub.execute_input":"2022-06-10T19:20:29.447291Z","iopub.status.idle":"2022-06-10T19:20:29.703565Z","shell.execute_reply.started":"2022-06-10T19:20:29.447255Z","shell.execute_reply":"2022-06-10T19:20:29.702669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.pie(product_type['product_type_count'], labels = product_type['product_type_name'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:29.704731Z","iopub.execute_input":"2022-06-10T19:20:29.704933Z","iopub.status.idle":"2022-06-10T19:20:30.847261Z","shell.execute_reply.started":"2022-06-10T19:20:29.704908Z","shell.execute_reply":"2022-06-10T19:20:30.846633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Trouser dress sweater is hily skewed Since they take up more than 40% of articles in H&M","metadata":{"execution":{"iopub.status.busy":"2022-05-20T22:53:51.219181Z","iopub.execute_input":"2022-05-20T22:53:51.219557Z","iopub.status.idle":"2022-05-20T22:53:51.227104Z","shell.execute_reply.started":"2022-05-20T22:53:51.219507Z","shell.execute_reply":"2022-05-20T22:53:51.225024Z"}}},{"cell_type":"code","source":"product_type['product_type_count'].sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:30.848432Z","iopub.execute_input":"2022-06-10T19:20:30.848791Z","iopub.status.idle":"2022-06-10T19:20:30.85457Z","shell.execute_reply.started":"2022-06-10T19:20:30.848749Z","shell.execute_reply":"2022-06-10T19:20:30.853884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate the percentage of top 5 in all the articles\n\nproduct_type[product_type['product_type_name'].isin(['Top','T-shirt','Sweater','Dress','Trousers'])]['product_type_count'].sum() / product_type['product_type_count'].sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:30.856103Z","iopub.execute_input":"2022-06-10T19:20:30.856741Z","iopub.status.idle":"2022-06-10T19:20:30.868501Z","shell.execute_reply.started":"2022-06-10T19:20:30.856606Z","shell.execute_reply":"2022-06-10T19:20:30.867465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above I was exploring what type of articles do we have. And I found that that top 5 most provided type,'Top','T-shirt','Sweater','Dress','Trousers' takes up 40% of the articles sound at H&M. That's crazy! I didn't think it will be highly skewed like this in the articles!","metadata":{}},{"cell_type":"code","source":"articles.columns","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:30.870246Z","iopub.execute_input":"2022-06-10T19:20:30.870572Z","iopub.status.idle":"2022-06-10T19:20:30.879562Z","shell.execute_reply.started":"2022-06-10T19:20:30.870529Z","shell.execute_reply":"2022-06-10T19:20:30.878543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_group = pd.DataFrame(articles[['article_id','product_group_name']].groupby('product_group_name')['article_id'].count().sort_values(ascending=False))\nproduct_group = product_group.reset_index()\nproduct_group.columns = ['product_group_name','count']\nproduct_group","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:30.880854Z","iopub.execute_input":"2022-06-10T19:20:30.881671Z","iopub.status.idle":"2022-06-10T19:20:30.910696Z","shell.execute_reply.started":"2022-06-10T19:20:30.881633Z","shell.execute_reply":"2022-06-10T19:20:30.910084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure()\nax = fig.add_axes([0,0,1,1])\nax.bar(product_group['product_group_name'],product_group['count'])\nplt.xticks(rotation='vertical')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:30.911568Z","iopub.execute_input":"2022-06-10T19:20:30.912468Z","iopub.status.idle":"2022-06-10T19:20:31.169924Z","shell.execute_reply.started":"2022-06-10T19:20:30.912434Z","shell.execute_reply":"2022-06-10T19:20:31.169306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.pie(product_group['count'], labels = product_group['product_group_name'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:31.170815Z","iopub.execute_input":"2022-06-10T19:20:31.171337Z","iopub.status.idle":"2022-06-10T19:20:31.376035Z","shell.execute_reply.started":"2022-06-10T19:20:31.171303Z","shell.execute_reply":"2022-06-10T19:20:31.375109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So far we have looked at data from 1 dimension. Now lets increase the dimension and explore more!","metadata":{}},{"cell_type":"code","source":"articles.groupby(['product_group_name','index_name'])['article_id'].count().sort_values()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:31.382618Z","iopub.execute_input":"2022-06-10T19:20:31.38343Z","iopub.status.idle":"2022-06-10T19:20:31.43117Z","shell.execute_reply.started":"2022-06-10T19:20:31.383361Z","shell.execute_reply":"2022-06-10T19:20:31.430527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=articles, y='product_group_name', color='orange', hue='index_group_name', multiple=\"stack\")\nax.set_xlabel('count by garment group')\nax.set_ylabel('garment group')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:31.43225Z","iopub.execute_input":"2022-06-10T19:20:31.432579Z","iopub.status.idle":"2022-06-10T19:20:32.120037Z","shell.execute_reply.started":"2022-06-10T19:20:31.43255Z","shell.execute_reply":"2022-06-10T19:20:32.118993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I definitley notice *lady and children* is dominating almost all the garment group. ","metadata":{}},{"cell_type":"code","source":"articles.groupby(['product_group_name', 'product_type_name']).count()['article_id'].get(\"Accessories\").sort_values()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:32.121474Z","iopub.execute_input":"2022-06-10T19:20:32.12179Z","iopub.status.idle":"2022-06-10T19:20:32.224616Z","shell.execute_reply.started":"2022-06-10T19:20:32.121749Z","shell.execute_reply":"2022-06-10T19:20:32.223724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since now we have some understanding of Articles, aka most of the stuff are for female and children. \n\nwe can give a look at Customers and check if there is any interesting finding!","metadata":{}},{"cell_type":"code","source":"# missing data percentage\ncustomers.isna().sum() / customers.shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:32.22577Z","iopub.execute_input":"2022-06-10T19:20:32.225983Z","iopub.status.idle":"2022-06-10T19:20:32.508457Z","shell.execute_reply.started":"2022-06-10T19:20:32.225956Z","shell.execute_reply":"2022-06-10T19:20:32.507506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# since its hard to know what age just base on this, we may omit and try guess their age base on what they buy ?\n\nage_na_cus = customers.age.isna()\n\n# kept the one that has age \n\ncustomers_age = customers[age_na_cus.apply(lambda x : not x)]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:32.509685Z","iopub.execute_input":"2022-06-10T19:20:32.509931Z","iopub.status.idle":"2022-06-10T19:20:32.800432Z","shell.execute_reply.started":"2022-06-10T19:20:32.509902Z","shell.execute_reply":"2022-06-10T19:20:32.799444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"lets check out the distribution of age.","metadata":{"execution":{"iopub.status.busy":"2022-05-20T22:55:09.64236Z","iopub.execute_input":"2022-05-20T22:55:09.642749Z","iopub.status.idle":"2022-05-20T22:55:09.648519Z","shell.execute_reply.started":"2022-05-20T22:55:09.642709Z","shell.execute_reply":"2022-05-20T22:55:09.647469Z"}}},{"cell_type":"code","source":"customers_age = customers_age.sort_values(by='age',ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:32.801675Z","iopub.execute_input":"2022-06-10T19:20:32.801892Z","iopub.status.idle":"2022-06-10T19:20:33.357339Z","shell.execute_reply.started":"2022-06-10T19:20:32.801856Z","shell.execute_reply":"2022-06-10T19:20:33.356414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers_age","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:33.358661Z","iopub.execute_input":"2022-06-10T19:20:33.358896Z","iopub.status.idle":"2022-06-10T19:20:33.380259Z","shell.execute_reply.started":"2022-06-10T19:20:33.358867Z","shell.execute_reply":"2022-06-10T19:20:33.379369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.histplot(data=customers_age, x='age', bins=20, color='orange')\nax.set_xlabel('Distribution of the customers age')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:33.381366Z","iopub.execute_input":"2022-06-10T19:20:33.381573Z","iopub.status.idle":"2022-06-10T19:20:33.847312Z","shell.execute_reply.started":"2022-06-10T19:20:33.381548Z","shell.execute_reply":"2022-06-10T19:20:33.846428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not much for customer data besides their age, news, and club member status","metadata":{}},{"cell_type":"markdown","source":"now i want to find out maybe how much the same customer spend by monthly in 2019/9/22 - 2020/9/22. ","metadata":{}},{"cell_type":"code","source":"transactions = transactions.sort_values(['customer_id'])","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:20:33.848433Z","iopub.execute_input":"2022-06-10T19:20:33.848661Z","iopub.status.idle":"2022-06-10T19:21:15.923021Z","shell.execute_reply.started":"2022-06-10T19:20:33.848633Z","shell.execute_reply":"2022-06-10T19:21:15.922208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.t_dat","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:15.924246Z","iopub.execute_input":"2022-06-10T19:21:15.924528Z","iopub.status.idle":"2022-06-10T19:21:15.93356Z","shell.execute_reply.started":"2022-06-10T19:21:15.924499Z","shell.execute_reply":"2022-06-10T19:21:15.932612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:15.934977Z","iopub.execute_input":"2022-06-10T19:21:15.935237Z","iopub.status.idle":"2022-06-10T19:21:20.707484Z","shell.execute_reply.started":"2022-06-10T19:21:15.935206Z","shell.execute_reply":"2022-06-10T19:21:20.706608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.histplot(data=transactions, x='price', bins=100, color='orange')\nax.set_xlabel('Distribution of the customer spending')\nplt.xlim([0,0.2])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:20.708712Z","iopub.execute_input":"2022-06-10T19:21:20.709416Z","iopub.status.idle":"2022-06-10T19:21:32.492797Z","shell.execute_reply.started":"2022-06-10T19:21:20.709373Z","shell.execute_reply":"2022-06-10T19:21:32.492069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.groupby('customer_id')","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:32.493766Z","iopub.execute_input":"2022-06-10T19:21:32.49465Z","iopub.status.idle":"2022-06-10T19:21:32.501225Z","shell.execute_reply.started":"2022-06-10T19:21:32.494598Z","shell.execute_reply":"2022-06-10T19:21:32.500608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"last_year_tran = transactions[transactions['t_dat'] >= '2019-09-22']","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:32.502336Z","iopub.execute_input":"2022-06-10T19:21:32.503118Z","iopub.status.idle":"2022-06-10T19:21:35.58079Z","shell.execute_reply.started":"2022-06-10T19:21:32.503071Z","shell.execute_reply":"2022-06-10T19:21:35.57991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"last_year_tran.t_dat = pd.to_datetime(last_year_tran.t_dat)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:35.582066Z","iopub.execute_input":"2022-06-10T19:21:35.582565Z","iopub.status.idle":"2022-06-10T19:21:38.047211Z","shell.execute_reply.started":"2022-06-10T19:21:35.582523Z","shell.execute_reply":"2022-06-10T19:21:38.046282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"last_year_tran['month'] = last_year_tran.t_dat.apply(lambda x : x.month)\n\n# import cudf\n# train = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\n# train['customer_id'] = train['customer_id'].str[-16:].str.hex_to_int().astype('int64')\n# train['article_id'] = train.article_id.astype('int32')\n# train.t_dat = cudf.to_datetime(train.t_dat)\n# train = train[['t_dat','customer_id','article_id']]\n# train.to_parquet('train.pqt',index=False)\n# print( train.shape )\n# train.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:21:38.048518Z","iopub.execute_input":"2022-06-10T19:21:38.048803Z","iopub.status.idle":"2022-06-10T19:22:36.834453Z","shell.execute_reply.started":"2022-06-10T19:21:38.048769Z","shell.execute_reply":"2022-06-10T19:22:36.833446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How many unique customer made purchase in 2022?","metadata":{}},{"cell_type":"code","source":"last_year_tran['customer_id'].nunique()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:36.835468Z","iopub.execute_input":"2022-06-10T19:22:36.835679Z","iopub.status.idle":"2022-06-10T19:22:40.308768Z","shell.execute_reply.started":"2022-06-10T19:22:36.835654Z","shell.execute_reply":"2022-06-10T19:22:40.307824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**<span style=\"color:#008000;\"> Let's see the general summary about the transaction</span>**","metadata":{}},{"cell_type":"markdown","source":"So I want to see what is the totoal number of item from each customer for last year of transaction. ","metadata":{}},{"cell_type":"code","source":"cus_bought_count = pd.DataFrame(last_year_tran.groupby('customer_id')['article_id'].count().sort_values(ascending=False))","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:40.310174Z","iopub.execute_input":"2022-06-10T19:22:40.310632Z","iopub.status.idle":"2022-06-10T19:22:45.518781Z","shell.execute_reply.started":"2022-06-10T19:22:40.310583Z","shell.execute_reply":"2022-06-10T19:22:45.51794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"last_year_tran","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:45.51987Z","iopub.execute_input":"2022-06-10T19:22:45.520091Z","iopub.status.idle":"2022-06-10T19:22:45.538203Z","shell.execute_reply.started":"2022-06-10T19:22:45.520064Z","shell.execute_reply":"2022-06-10T19:22:45.537443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cus_bought_count = cus_bought_count.reset_index()\ncus_bought_count.columns = ['customer_id','article_count']","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:45.539292Z","iopub.execute_input":"2022-06-10T19:22:45.539536Z","iopub.status.idle":"2022-06-10T19:22:45.609355Z","shell.execute_reply.started":"2022-06-10T19:22:45.539505Z","shell.execute_reply":"2022-06-10T19:22:45.608421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cus_bought_count.describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:45.610258Z","iopub.execute_input":"2022-06-10T19:22:45.610465Z","iopub.status.idle":"2022-06-10T19:22:45.678024Z","shell.execute_reply.started":"2022-06-10T19:22:45.61044Z","shell.execute_reply":"2022-06-10T19:22:45.67747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since I noticed huge discrepency between people who are top 10% of customer and rest of 90%\n\nI may want to consider top 10 to be loyal customer as average of them purchase at least 2-3 item every month.\n\nwhile the rest of 90% make occasional purchases. ","metadata":{}},{"cell_type":"code","source":"cus_bought_count[cus_bought_count['article_count'] >= 36]['article_count'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:45.679006Z","iopub.execute_input":"2022-06-10T19:22:45.67934Z","iopub.status.idle":"2022-06-10T19:22:45.708032Z","shell.execute_reply.started":"2022-06-10T19:22:45.679312Z","shell.execute_reply":"2022-06-10T19:22:45.707451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cus_bought_count[cus_bought_count['article_count'] < 36]['article_count'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:45.708913Z","iopub.execute_input":"2022-06-10T19:22:45.709269Z","iopub.status.idle":"2022-06-10T19:22:45.848287Z","shell.execute_reply.started":"2022-06-10T19:22:45.709234Z","shell.execute_reply":"2022-06-10T19:22:45.847399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I have a business related question, top 10% customer is responsible for how much % of revenue in last year?\n","metadata":{}},{"cell_type":"code","source":"cus_spent = pd.DataFrame(last_year_tran.groupby('customer_id')['price'].sum().sort_values(ascending=False)).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:45.849472Z","iopub.execute_input":"2022-06-10T19:22:45.849689Z","iopub.status.idle":"2022-06-10T19:22:51.179904Z","shell.execute_reply.started":"2022-06-10T19:22:45.849663Z","shell.execute_reply":"2022-06-10T19:22:51.179039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cus_spent.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:51.18144Z","iopub.execute_input":"2022-06-10T19:22:51.181682Z","iopub.status.idle":"2022-06-10T19:22:51.192725Z","shell.execute_reply.started":"2022-06-10T19:22:51.181654Z","shell.execute_reply":"2022-06-10T19:22:51.192073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_10 = cus_bought_count[cus_bought_count['article_count'] >= 36]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:51.193786Z","iopub.execute_input":"2022-06-10T19:22:51.194122Z","iopub.status.idle":"2022-06-10T19:22:51.209923Z","shell.execute_reply.started":"2022-06-10T19:22:51.194093Z","shell.execute_reply":"2022-06-10T19:22:51.209298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_10_count_spent = top_10.merge(cus_spent, how='left', left_on='customer_id',right_on='customer_id')","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:51.211022Z","iopub.execute_input":"2022-06-10T19:22:51.211388Z","iopub.status.idle":"2022-06-10T19:22:52.06094Z","shell.execute_reply.started":"2022-06-10T19:22:51.211354Z","shell.execute_reply":"2022-06-10T19:22:52.060216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_10_count_spent.sample(3)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:52.061997Z","iopub.execute_input":"2022-06-10T19:22:52.06235Z","iopub.status.idle":"2022-06-10T19:22:52.075943Z","shell.execute_reply.started":"2022-06-10T19:22:52.062318Z","shell.execute_reply":"2022-06-10T19:22:52.075044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_10_count_spent.price.sum() / last_year_tran.price.sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:52.077215Z","iopub.execute_input":"2022-06-10T19:22:52.077483Z","iopub.status.idle":"2022-06-10T19:22:52.123559Z","shell.execute_reply.started":"2022-06-10T19:22:52.077453Z","shell.execute_reply":"2022-06-10T19:22:52.122615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After some digging, I find out top 10% of the customer is responsible for 45% of sale in 2022.\n\nIf I was the CEO, I will be really sure those 10% of cusotmer get best possible experience to maintain relationship with them. ","metadata":{}},{"cell_type":"code","source":"plt.boxplot(cus_bought_count['article_count'])","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:52.124754Z","iopub.execute_input":"2022-06-10T19:22:52.125084Z","iopub.status.idle":"2022-06-10T19:22:52.39505Z","shell.execute_reply.started":"2022-06-10T19:22:52.125055Z","shell.execute_reply":"2022-06-10T19:22:52.394495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I want to know what items have the customer bought throughout the entire year. Can be guess about their spending behavior and potentially age and gender?","metadata":{"execution":{"iopub.status.busy":"2022-05-23T19:55:15.959555Z","iopub.execute_input":"2022-05-23T19:55:15.960787Z","iopub.status.idle":"2022-05-23T19:55:17.803111Z","shell.execute_reply.started":"2022-05-23T19:55:15.960734Z","shell.execute_reply":"2022-05-23T19:55:17.801896Z"}}},{"cell_type":"code","source":"#I will start with top 10% customers. Aka loyal customers.\ntop_10.head(1)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:52.39604Z","iopub.execute_input":"2022-06-10T19:22:52.39635Z","iopub.status.idle":"2022-06-10T19:22:52.403376Z","shell.execute_reply.started":"2022-06-10T19:22:52.396323Z","shell.execute_reply":"2022-06-10T19:22:52.40284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am going to explore what has our \\#1 customer has bought for the last year, as this customer bought 1020 items in the last year. ","metadata":{}},{"cell_type":"code","source":"number_1 = last_year_tran[last_year_tran['customer_id'] == top_10.head(1)['customer_id'].values[0]]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:52.404327Z","iopub.execute_input":"2022-06-10T19:22:52.404622Z","iopub.status.idle":"2022-06-10T19:22:54.248713Z","shell.execute_reply.started":"2022-06-10T19:22:52.404591Z","shell.execute_reply":"2022-06-10T19:22:54.247773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.head(3)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.249842Z","iopub.execute_input":"2022-06-10T19:22:54.250088Z","iopub.status.idle":"2022-06-10T19:22:54.274812Z","shell.execute_reply.started":"2022-06-10T19:22:54.250058Z","shell.execute_reply":"2022-06-10T19:22:54.273958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stuff_number1_bought = number_1.merge(articles, how = 'left', left_on = 'article_id', right_on = 'article_id')","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.276268Z","iopub.execute_input":"2022-06-10T19:22:54.276495Z","iopub.status.idle":"2022-06-10T19:22:54.359809Z","shell.execute_reply.started":"2022-06-10T19:22:54.276468Z","shell.execute_reply":"2022-06-10T19:22:54.359036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles['product_group_name'].head(2)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.361127Z","iopub.execute_input":"2022-06-10T19:22:54.361507Z","iopub.status.idle":"2022-06-10T19:22:54.367959Z","shell.execute_reply.started":"2022-06-10T19:22:54.361472Z","shell.execute_reply":"2022-06-10T19:22:54.367187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=stuff_number1_bought, y='product_group_name', color='orange', hue='index_name', multiple=\"stack\")\n# ax.set_xlabel('count by garment group')\n# ax.set_ylabel('garment group')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.369198Z","iopub.execute_input":"2022-06-10T19:22:54.369799Z","iopub.status.idle":"2022-06-10T19:22:54.891426Z","shell.execute_reply.started":"2022-06-10T19:22:54.369753Z","shell.execute_reply":"2022-06-10T19:22:54.890538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From my understanding, Divided is also mainly female clothing. ","metadata":{}},{"cell_type":"code","source":"customers.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.892704Z","iopub.execute_input":"2022-06-10T19:22:54.892931Z","iopub.status.idle":"2022-06-10T19:22:54.906323Z","shell.execute_reply.started":"2022-06-10T19:22:54.892902Z","shell.execute_reply":"2022-06-10T19:22:54.905287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stuff_number1_bought.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.907493Z","iopub.execute_input":"2022-06-10T19:22:54.907725Z","iopub.status.idle":"2022-06-10T19:22:54.931424Z","shell.execute_reply.started":"2022-06-10T19:22:54.907696Z","shell.execute_reply":"2022-06-10T19:22:54.930562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers[customers['customer_id'] == top_10.head(1)['customer_id'].values[0]]","metadata":{"execution":{"iopub.status.busy":"2022-06-10T19:22:54.932731Z","iopub.execute_input":"2022-06-10T19:22:54.932959Z","iopub.status.idle":"2022-06-10T19:22:55.03237Z","shell.execute_reply.started":"2022-06-10T19:22:54.93293Z","shell.execute_reply":"2022-06-10T19:22:55.031608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}