{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Normcore is my style\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Contents\n- [Introduction](#intro)\n- [Setup](#setup)\n- [Memory Reduction](#memory)\n- [Articles of clothing](#articles)\n - [Product type](#ptype)\n - [Accessories](#accessories)\n - [Color](#color)\n- [Customers](#customers)\n- [Transactions](#transactions)\n- [Images](#images)\n- [Another dataset](#dataset2)\n- [Seasonality](#seasonality)\n- [Modelling](#modelling)","metadata":{}},{"cell_type":"code","source":"%%html \n","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:12:18.877356Z","iopub.execute_input":"2022-03-29T18:12:18.877952Z","iopub.status.idle":"2022-03-29T18:12:18.888115Z","shell.execute_reply.started":"2022-03-29T18:12:18.877902Z","shell.execute_reply":"2022-03-29T18:12:18.8872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n# Introduction\n\nThis is my first time participating in a competition like this. Before this, I had never even considered this type of modelling, I was of the mindset of either classification or regression. I am interested in taking part in competitions that force me to learn new things and already this competition has done that, it has broadened my thinking with respect to modelling. Seeing as I am no expert I shall move on and start doing some Exploratory Data Analysis (EDA). Although, I will say that normally I do a small amount of reading and EDA and then proceed to make models. This time around I will flip that and do much more reading/exploration before even thinking about modelling.\n\n#### The thing I have found most useful in this competition so far is going to the discussion and sorting by most-votes. The master's know what they are talking about!","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n# Setup\n\nHere we install any packages we need and take a look at the directories. I like to use Black formatter to increase the legibility of my code. Then there are the Data-science staples of Pandas, Numpy, Matplotlib and Seaborn.","metadata":{}},{"cell_type":"code","source":"# Show the input directories so we can load the data\n!tree -L 2 ../input","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:12:18.890181Z","iopub.execute_input":"2022-03-29T18:12:18.890512Z","iopub.status.idle":"2022-03-29T18:12:19.75089Z","shell.execute_reply.started":"2022-03-29T18:12:18.890471Z","shell.execute_reply":"2022-03-29T18:12:19.750094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install black formatter to adhere to pep8 and make our code more legible\n!pip install nb_black > /dev/null","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:12:19.752653Z","iopub.execute_input":"2022-03-29T18:12:19.75304Z","iopub.status.idle":"2022-03-29T18:12:30.711528Z","shell.execute_reply.started":"2022-03-29T18:12:19.753011Z","shell.execute_reply":"2022-03-29T18:12:30.709649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%load_ext lab_black","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:12:30.71394Z","iopub.execute_input":"2022-03-29T18:12:30.714207Z","iopub.status.idle":"2022-03-29T18:12:30.725797Z","shell.execute_reply.started":"2022-03-29T18:12:30.714178Z","shell.execute_reply":"2022-03-29T18:12:30.724915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import the modules we will use\nimport pandas as pd\nfrom pandasql import sqldf\nimport matplotlib.pylab as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:12:30.728775Z","iopub.execute_input":"2022-03-29T18:12:30.729623Z","iopub.status.idle":"2022-03-29T18:12:30.764372Z","shell.execute_reply.started":"2022-03-29T18:12:30.729564Z","shell.execute_reply":"2022-03-29T18:12:30.762188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles = pd.read_csv(\n \"../input/h-and-m-personalized-fashion-recommendations/articles.csv\"\n)\ncustomers = pd.read_csv(\n \"../input/h-and-m-personalized-fashion-recommendations/customers.csv\"\n)\nsample_submission = pd.read_csv(\n \"../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv\"\n)\ntrain = pd.read_csv(\n \"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\",\n parse_dates=[\"t_dat\"],\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:12:30.769661Z","iopub.execute_input":"2022-03-29T18:12:30.771165Z","iopub.status.idle":"2022-03-29T18:13:41.450981Z","shell.execute_reply.started":"2022-03-29T18:12:30.771004Z","shell.execute_reply":"2022-03-29T18:13:41.450152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n# Memory reduction\n\nMemory reduction is one of those skills that a data scientist needs to learn. Personally I only reduce my data types if I run into issues.","metadata":{}},{"cell_type":"code","source":"# Show the memory utilisation of the train dataframe\ntrain.memory_usage(deep=True)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:13:41.452268Z","iopub.execute_input":"2022-03-29T18:13:41.452521Z","iopub.status.idle":"2022-03-29T18:13:47.729393Z","shell.execute_reply.started":"2022-03-29T18:13:41.452492Z","shell.execute_reply":"2022-03-29T18:13:47.728452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that most of the data is from the `customer_id`. So this will be where we may be able to make the most headway in terms of reducing the data size.\n\nNow apply the the Kagglers favourite memory reduction tecniques. Reduce the datatypes where there is un-necessary space. I have taken the code from this [discussion by Chris Deotte](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635)\n\nAlso there is an issue pointed out by [RDizzle](https://www.kaggle.com/rdizzl3), in this [discussion](https://www.kaggle.com/rdizzl3).","metadata":{}},{"cell_type":"code","source":"train[\"customer_id\"] = (\n train[\"customer_id\"].apply(lambda x: int(x[-16:], 16)).astype(\"int64\")\n)\ntrain[\"article_id\"] = train[\"article_id\"].astype(\"int32\")\n\n# This is an important step as\ntrain[\"article_id\"] = \"0\" + train.article_id.astype(\"str\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:13:47.730559Z","iopub.execute_input":"2022-03-29T18:13:47.730828Z","iopub.status.idle":"2022-03-29T18:15:02.551085Z","shell.execute_reply.started":"2022-03-29T18:13:47.730801Z","shell.execute_reply":"2022-03-29T18:15:02.550307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n# Articles - a.k.a Items of clothing ","metadata":{}},{"cell_type":"code","source":"articles.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:02.55216Z","iopub.execute_input":"2022-03-29T18:15:02.552355Z","iopub.status.idle":"2022-03-29T18:15:02.582374Z","shell.execute_reply.started":"2022-03-29T18:15:02.55233Z","shell.execute_reply":"2022-03-29T18:15:02.581564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"temp = articles.groupby([\"garment_group_name\"])[\"product_type_name\"].nunique()\ndf = pd.DataFrame({\"Garment Group\": temp.index, \"Product Types\": temp.values})\ndf = df.sort_values([\"Product Types\"], ascending=False)\nplt.figure(figsize=(16, 5))\nplt.title(\"Number of Product Types per each Garment Group\")\ns = sns.barplot(x=\"Garment Group\", y=\"Product Types\", data=df, palette=\"cubehelix\")\ns.set_xticklabels(s.get_xticklabels(), rotation=90)\nlocs, labels = plt.xticks()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:02.583505Z","iopub.execute_input":"2022-03-29T18:15:02.583784Z","iopub.status.idle":"2022-03-29T18:15:02.993572Z","shell.execute_reply.started":"2022-03-29T18:15:02.583753Z","shell.execute_reply":"2022-03-29T18:15:02.992561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A lot of unknown product types, I guess it is not too suprising accessories are the largest group. Let's have a look at the accessories.\n\n","metadata":{}},{"cell_type":"code","source":"temp = articles[articles[\"garment_group_name\"] == \"Accessories\"]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:02.997703Z","iopub.execute_input":"2022-03-29T18:15:02.997949Z","iopub.status.idle":"2022-03-29T18:15:03.028888Z","shell.execute_reply.started":"2022-03-29T18:15:02.997922Z","shell.execute_reply":"2022-03-29T18:15:03.027799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f\"There are {len(temp.prod_name.unique())} different accessories\"","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:03.030472Z","iopub.execute_input":"2022-03-29T18:15:03.031Z","iopub.status.idle":"2022-03-29T18:15:03.041131Z","shell.execute_reply.started":"2022-03-29T18:15:03.03095Z","shell.execute_reply":"2022-03-29T18:15:03.040335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp.product_type_name.unique()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:03.042453Z","iopub.execute_input":"2022-03-29T18:15:03.042677Z","iopub.status.idle":"2022-03-29T18:15:03.055664Z","shell.execute_reply.started":"2022-03-29T18:15:03.042651Z","shell.execute_reply":"2022-03-29T18:15:03.054799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These are interesting. I wonder if periods of rain are forcasted things like umbrellas are more likely to be reccomended? If we knew where this data was from I guess we could check to the weather and how umbrella sales correlated.\n\n","metadata":{}},{"cell_type":"code","source":"temp = articles.groupby([\"colour_group_name\"])[\"prod_name\"].nunique()\ndf = pd.DataFrame({\"Colour Name\": temp.index, \"Product names\": temp.values})\ndf = df.sort_values([\"Product names\"], ascending=False)\nplt.figure(figsize=(20, 10))\nplt.title(\n \"Number of Product Names per Colour -- a.k.a is there anything darker than black?\"\n)\ns = sns.barplot(x=\"Colour Name\", y=\"Product names\", data=df, palette=\"cubehelix\")\ns.set_xticklabels(s.get_xticklabels(), rotation=90)\nlocs, labels = plt.xticks()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:03.056565Z","iopub.execute_input":"2022-03-29T18:15:03.056791Z","iopub.status.idle":"2022-03-29T18:15:03.983791Z","shell.execute_reply.started":"2022-03-29T18:15:03.056764Z","shell.execute_reply":"2022-03-29T18:15:03.982848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## What about index_name and index_group_name etc","metadata":{}},{"cell_type":"code","source":"articles.groupby([\"index_group_name\", \"index_group_no\", \"index_name\", \"index_code\"])[\n \"article_id\"\n].count()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:03.985082Z","iopub.execute_input":"2022-03-29T18:15:03.985717Z","iopub.status.idle":"2022-03-29T18:15:04.031909Z","shell.execute_reply.started":"2022-03-29T18:15:03.985683Z","shell.execute_reply":"2022-03-29T18:15:04.030896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So Baby/Children and Women have sub-categories, I don't see that would help us with modelling at this stage. The index code+group, could easily be encoded to uniquely identify each group and name. ","metadata":{}},{"cell_type":"markdown","source":"## A brief interlude to talk about show some SQL\n\nAs all data scientist's know from job advertisement's, SQL is a desireable skill. Often in Kaggle competetions I don't see it as being particularly useful. \n\nHowever after reading [this notebook](https://www.kaggle.com/ludovicocuoghi/h-m-deep-sales-and-customers-analysis) by [Ludo Vicocuoghi](https://www.kaggle.com/ludovicocuoghi) I had to have a look.","metadata":{}},{"cell_type":"code","source":"df_a = sqldf(\n \"\"\"SELECT article_id, prod_name, product_type_name, product_group_name, colour_group_name, index_name\n FROM articles\n \"\"\"\n)\ndf_sold_qty = train[\"article_id\"].value_counts()\n\n\ndf_sold_qty = df_sold_qty.reset_index()\ndf_sold_qty.rename(\n columns={\"article_id\": \"sold_qty\", \"index\": \"article_id\"}, inplace=True\n)\ntop_100_sold = df_sold_qty.iloc[:100]\n\ntop_100_details = sqldf(\n \"\"\"SELECT *\n FROM top_100_sold AS t\n INNER JOIN df_a AS a\n on t.article_id = a.article_id\n \"\"\"\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:04.033522Z","iopub.execute_input":"2022-03-29T18:15:04.034104Z","iopub.status.idle":"2022-03-29T18:15:23.275547Z","shell.execute_reply.started":"2022-03-29T18:15:04.03406Z","shell.execute_reply":"2022-03-29T18:15:23.274731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_100_details","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:23.276666Z","iopub.execute_input":"2022-03-29T18:15:23.276899Z","iopub.status.idle":"2022-03-29T18:15:23.296333Z","shell.execute_reply.started":"2022-03-29T18:15:23.276871Z","shell.execute_reply":"2022-03-29T18:15:23.295751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n#### Gendered items\n\nI am not completely sure if inferring someone's gender is a good idea, for many reasons,but mainly because the person could be purchasing the item for someone else especially if they are a parent.","metadata":{}},{"cell_type":"code","source":"# Gendered Items?\narticles[\"index_group_name\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:23.297505Z","iopub.execute_input":"2022-03-29T18:15:23.298373Z","iopub.status.idle":"2022-03-29T18:15:23.328354Z","shell.execute_reply.started":"2022-03-29T18:15:23.298336Z","shell.execute_reply":"2022-03-29T18:15:23.327745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The most suprising thing about this is that there are almost as many Baby/Children's items as there are Ladieswear. If we knew nothing and were reccomending items randomly I guess we would almost never reccomend sportswear. I think something we should do is find out if sales are related to the number of items in a specific group.","metadata":{}},{"cell_type":"code","source":"t_train = train.copy()\nt_train[\"article_id\"] = t_train[\"article_id\"].astype(\"int64\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:23.32979Z","iopub.execute_input":"2022-03-29T18:15:23.3302Z","iopub.status.idle":"2022-03-29T18:15:31.127292Z","shell.execute_reply.started":"2022-03-29T18:15:23.330171Z","shell.execute_reply":"2022-03-29T18:15:31.12644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"t_train.merge(\n articles[[\"article_id\", \"index_group_name\"]], on=\"article_id\", how=\"inner\"\n)[\"index_group_name\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:31.12872Z","iopub.execute_input":"2022-03-29T18:15:31.128934Z","iopub.status.idle":"2022-03-29T18:15:46.578087Z","shell.execute_reply.started":"2022-03-29T18:15:31.128909Z","shell.execute_reply":"2022-03-29T18:15:46.5773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interesting, no suprise to me that ladieswear has the most sales, but I am suprised by Baby/Children given the number of items. Given that there were far fewer sportswear items in comparison to Baby/Children we can't assume that the number of items in a category is necessarilly reflected in sales.","metadata":{}},{"cell_type":"code","source":"del t_train","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:46.579452Z","iopub.execute_input":"2022-03-29T18:15:46.58033Z","iopub.status.idle":"2022-03-29T18:15:46.589163Z","shell.execute_reply.started":"2022-03-29T18:15:46.580296Z","shell.execute_reply":"2022-03-29T18:15:46.588252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n# Customers - are they always right? ","metadata":{}},{"cell_type":"code","source":"customers.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:46.590207Z","iopub.execute_input":"2022-03-29T18:15:46.590553Z","iopub.status.idle":"2022-03-29T18:15:46.614301Z","shell.execute_reply.started":"2022-03-29T18:15:46.590519Z","shell.execute_reply":"2022-03-29T18:15:46.613752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Some hashed data for id and postal_code","metadata":{}},{"cell_type":"code","source":"customers.fashion_news_frequency.value_counts().plot(kind=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:46.615621Z","iopub.execute_input":"2022-03-29T18:15:46.616663Z","iopub.status.idle":"2022-03-29T18:15:46.975927Z","shell.execute_reply.started":"2022-03-29T18:15:46.616594Z","shell.execute_reply":"2022-03-29T18:15:46.974981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers.fashion_news_frequency.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:46.977806Z","iopub.execute_input":"2022-03-29T18:15:46.978436Z","iopub.status.idle":"2022-03-29T18:15:47.201035Z","shell.execute_reply.started":"2022-03-29T18:15:46.978372Z","shell.execute_reply":"2022-03-29T18:15:47.200255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like there is an error in the database, why would you have NONE and None?","metadata":{}},{"cell_type":"code","source":"customers[customers[\"fashion_news_frequency\"] == \"None\"]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:47.202471Z","iopub.execute_input":"2022-03-29T18:15:47.202926Z","iopub.status.idle":"2022-03-29T18:15:47.433667Z","shell.execute_reply.started":"2022-03-29T18:15:47.202883Z","shell.execute_reply":"2022-03-29T18:15:47.432746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I don't see why NONE would be preffered over None.","metadata":{}},{"cell_type":"code","source":"customers.postal_code.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:47.435037Z","iopub.execute_input":"2022-03-29T18:15:47.435356Z","iopub.status.idle":"2022-03-29T18:15:48.333856Z","shell.execute_reply.started":"2022-03-29T18:15:47.435315Z","shell.execute_reply":"2022-03-29T18:15:48.333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interesting...I wonder if \"2c29ae653a9282cce4151bd87643c907644e09541abc28ae87dea0d1f6603b1c\" is the hash for None? If the encryption applies the same value to each postcode then you wouldn't expect 120k people to live in the same postcode?\n\nI imagine all of these cluster around the store/stores\n\nPresumably the customer ids are integers 1-n","metadata":{}},{"cell_type":"code","source":"customers.customer_id.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:48.335277Z","iopub.execute_input":"2022-03-29T18:15:48.335592Z","iopub.status.idle":"2022-03-29T18:15:50.196489Z","shell.execute_reply.started":"2022-03-29T18:15:48.33555Z","shell.execute_reply":"2022-03-29T18:15:50.195431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Whatever the hashing for this is it seems somewhat predictable? I am sure someone will figure this out before the end and find all the postcodes.","metadata":{}},{"cell_type":"markdown","source":"## Customer ages","metadata":{}},{"cell_type":"code","source":"temp = customers.groupby([\"age\"])[\"customer_id\"].count()\ndf = pd.DataFrame({\"Age\": temp.index, \"Customers\": temp.values})\ndf = df.sort_values([\"Age\"], ascending=False)\nplt.figure(figsize=(20, 10))\nplt.title(f\"Number of Customers by Age\")\ns = sns.barplot(x=\"Age\", y=\"Customers\", data=df, palette=\"cubehelix\")\ns.set_xticklabels(s.get_xticklabels(), rotation=90)\nlocs, labels = plt.xticks()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:50.198083Z","iopub.execute_input":"2022-03-29T18:15:50.198437Z","iopub.status.idle":"2022-03-29T18:15:51.658244Z","shell.execute_reply.started":"2022-03-29T18:15:50.198374Z","shell.execute_reply":"2022-03-29T18:15:51.657392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To me this is the most interesting plot I have seen from the data so far? As someone who is in my earlier thirties and thinks that consumerism and fast-fashion is 'cringe', it is interesting to know that it is not my moral outlook but my age that is driving my opinions. Looks like I have a couple of decades before my \"mid-life\" crisis kicks in.","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n# Customer transactions (Train)","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:51.664422Z","iopub.execute_input":"2022-03-29T18:15:51.664671Z","iopub.status.idle":"2022-03-29T18:15:51.677889Z","shell.execute_reply.started":"2022-03-29T18:15:51.664641Z","shell.execute_reply":"2022-03-29T18:15:51.677074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:51.679291Z","iopub.execute_input":"2022-03-29T18:15:51.67953Z","iopub.status.idle":"2022-03-29T18:15:51.691425Z","shell.execute_reply.started":"2022-03-29T18:15:51.679502Z","shell.execute_reply":"2022-03-29T18:15:51.690849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.article_id.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:15:51.692582Z","iopub.execute_input":"2022-03-29T18:15:51.693178Z","iopub.status.idle":"2022-03-29T18:16:00.175911Z","shell.execute_reply.started":"2022-03-29T18:15:51.693136Z","shell.execute_reply":"2022-03-29T18:16:00.175371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's have a look at the purchases per day.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 9))\ntrain.groupby([\"t_dat\"])[\"article_id\"].count().plot()\n;","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:00.176924Z","iopub.execute_input":"2022-03-29T18:16:00.177496Z","iopub.status.idle":"2022-03-29T18:16:04.458616Z","shell.execute_reply.started":"2022-03-29T18:16:00.177463Z","shell.execute_reply":"2022-03-29T18:16:04.457696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interesting, there are ~8 spikes per year (by eye). I am guessing these are sales. At the moment I can't think of how this will help with modelling. What we really need to know is how the individual products trend over time.","metadata":{}},{"cell_type":"code","source":"temp1 = train.query(\"article_id == '0706016001'\")[[\"t_dat\", \"price\"]]\ntemp2 = train.query(\"article_id == '0706016002'\")[[\"t_dat\", \"price\"]]\ntemp3 = train.query(\"article_id == '0372860001'\")[[\"t_dat\", \"price\"]]\ntemp4 = train.query(\"article_id == '0610776002'\")[[\"t_dat\", \"price\"]]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:04.46032Z","iopub.execute_input":"2022-03-29T18:16:04.460663Z","iopub.status.idle":"2022-03-29T18:16:10.518503Z","shell.execute_reply.started":"2022-03-29T18:16:04.460622Z","shell.execute_reply":"2022-03-29T18:16:10.517511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In [this discussion](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/310496) by [tbierhance](https://www.kaggle.com/tbierhance), the author found that the price scaling is divided by 590, so if you multiply the price by 590 you get the real price.","metadata":{}},{"cell_type":"code","source":"temp1[\"price\"] = temp1[\"price\"] * 590","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:10.520119Z","iopub.execute_input":"2022-03-29T18:16:10.520455Z","iopub.status.idle":"2022-03-29T18:16:10.528714Z","shell.execute_reply.started":"2022-03-29T18:16:10.520388Z","shell.execute_reply":"2022-03-29T18:16:10.528057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp1[\"price\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:10.529915Z","iopub.execute_input":"2022-03-29T18:16:10.53059Z","iopub.status.idle":"2022-03-29T18:16:10.547166Z","shell.execute_reply.started":"2022-03-29T18:16:10.530547Z","shell.execute_reply":"2022-03-29T18:16:10.546455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here you can see the most common prices for the item '0706016001', originally I was unsure what the banding pattern was and asked [here](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308974) why it might be. We concluded that it is the discount applied to the product.","metadata":{}},{"cell_type":"code","source":"top = 19.99\n\nprice_discounts = [round(top - top * (i / 100), 2) for i in list(range(5, 100, 5))]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:10.548563Z","iopub.execute_input":"2022-03-29T18:16:10.549139Z","iopub.status.idle":"2022-03-29T18:16:10.561714Z","shell.execute_reply.started":"2022-03-29T18:16:10.549098Z","shell.execute_reply":"2022-03-29T18:16:10.560911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"price_discounts","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:10.562795Z","iopub.execute_input":"2022-03-29T18:16:10.563155Z","iopub.status.idle":"2022-03-29T18:16:10.576867Z","shell.execute_reply.started":"2022-03-29T18:16:10.563121Z","shell.execute_reply":"2022-03-29T18:16:10.576022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above are the the discounts in 5% increments","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(16, 9))\ntry:\n sns.regplot(x=\"t_dat\", y=\"price\", data=temp1, ax=ax)\nexcept TypeError:\n pass\nplt.axhline(y=18.99, color=\"r\", linestyle=\"-\", label=\"5%\")\nplt.axhline(y=17.99, color=\"g\", linestyle=\"-\", label=\"10%\")\nplt.axhline(y=16.99, color=\"y\", linestyle=\"-\", label=\"15%\")\nplt.axhline(y=15.99, color=\"orange\", linestyle=\"-\", label=\"20%\")\nplt.axhline(y=14.99, color=\"black\", linestyle=\"-\", label=\"25%\")\nplt.legend(title=\"Discount bands\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:10.578189Z","iopub.execute_input":"2022-03-29T18:16:10.578877Z","iopub.status.idle":"2022-03-29T18:16:12.017269Z","shell.execute_reply.started":"2022-03-29T18:16:10.578828Z","shell.execute_reply":"2022-03-29T18:16:12.016603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Problem (my own problem) solved! The banding pattern shows the level of discount","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(16, 9))\ntry:\n sns.regplot(x=\"t_dat\", y=\"price\", data=temp2, ax=ax)\nexcept TypeError:\n pass","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:12.018475Z","iopub.execute_input":"2022-03-29T18:16:12.018704Z","iopub.status.idle":"2022-03-29T18:16:12.504702Z","shell.execute_reply.started":"2022-03-29T18:16:12.018674Z","shell.execute_reply":"2022-03-29T18:16:12.503825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(16, 9))\ntry:\n sns.regplot(x=\"t_dat\", y=\"price\", data=temp4, ax=ax)\nexcept TypeError:\n pass","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:12.505969Z","iopub.execute_input":"2022-03-29T18:16:12.506217Z","iopub.status.idle":"2022-03-29T18:16:12.955532Z","shell.execute_reply.started":"2022-03-29T18:16:12.506189Z","shell.execute_reply":"2022-03-29T18:16:12.954732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(16, 9))\nsns.kdeplot(train.price, ax=ax)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:16:12.956976Z","iopub.execute_input":"2022-03-29T18:16:12.957279Z","iopub.status.idle":"2022-03-29T18:18:00.879186Z","shell.execute_reply.started":"2022-03-29T18:16:12.957247Z","shell.execute_reply":"2022-03-29T18:18:00.878561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Hopefully we can use the prices when we shop online\n\nLet's have a look and see what the spending habits of the customer's are like.\n\nUsed code from this [notebook](https://www.kaggle.com/vanguarde/h-m-eda-first-look) By Daniil Karpov","metadata":{}},{"cell_type":"code","source":"transactions_byid = train.groupby(\"customer_id\").count()\ncustomer_purchases = transactions_byid.sort_values(by=\"price\", ascending=False)[\"price\"]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:00.880345Z","iopub.execute_input":"2022-03-29T18:18:00.881133Z","iopub.status.idle":"2022-03-29T18:18:08.506765Z","shell.execute_reply.started":"2022-03-29T18:18:00.881101Z","shell.execute_reply":"2022-03-29T18:18:08.505793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customer_purchases[:300_000]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:08.508211Z","iopub.execute_input":"2022-03-29T18:18:08.508627Z","iopub.status.idle":"2022-03-29T18:18:08.519238Z","shell.execute_reply.started":"2022-03-29T18:18:08.508576Z","shell.execute_reply":"2022-03-29T18:18:08.518457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 9))\nplt.plot(list(range(300_000)), customer_purchases[:300_000])\nplt.title(\"Purchases per customer\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:08.52051Z","iopub.execute_input":"2022-03-29T18:18:08.520776Z","iopub.status.idle":"2022-03-29T18:18:08.858306Z","shell.execute_reply.started":"2022-03-29T18:18:08.520748Z","shell.execute_reply":"2022-03-29T18:18:08.857449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wow!!! I am not suprised there are whales who purchase 1000's of items over the years, eseentially one item per day over the data set, but that the 300,000 ranked cuttomers in terms of purchases have purchased > 30 items. \n\nI wonder what percentage of customers account for 80 % of the number of items purchase, does the pareto principle hold here?\n\nIncidently, if my loss curve is a smooth as the elbow I will be pleased.","metadata":{}},{"cell_type":"code","source":"total = sum(customer_purchases)\ntotal * 0.8","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:08.859514Z","iopub.execute_input":"2022-03-29T18:18:08.860213Z","iopub.status.idle":"2022-03-29T18:18:09.041826Z","shell.execute_reply.started":"2022-03-29T18:18:08.860177Z","shell.execute_reply":"2022-03-29T18:18:09.040571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sum(customer_purchases[:419_600])","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.043554Z","iopub.execute_input":"2022-03-29T18:18:09.043822Z","iopub.status.idle":"2022-03-29T18:18:09.109131Z","shell.execute_reply.started":"2022-03-29T18:18:09.043783Z","shell.execute_reply":"2022-03-29T18:18:09.108018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"round(((419_600 / len(customer_purchases)) * 100), 2)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.110554Z","iopub.execute_input":"2022-03-29T18:18:09.111428Z","iopub.status.idle":"2022-03-29T18:18:09.120013Z","shell.execute_reply.started":"2022-03-29T18:18:09.111362Z","shell.execute_reply":"2022-03-29T18:18:09.119321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well it is close, in this case, 31 % of customers account for 80 % of items purchased.\n\n
\n\n\n\n# Let's have a look at some images\n","metadata":{}},{"cell_type":"code","source":"import skimage\nfrom skimage.io import imread_collection, imshow_collection, imread, imshow\nimport os","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.12153Z","iopub.execute_input":"2022-03-29T18:18:09.122118Z","iopub.status.idle":"2022-03-29T18:18:09.13077Z","shell.execute_reply.started":"2022-03-29T18:18:09.122077Z","shell.execute_reply":"2022-03-29T18:18:09.130094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"im_dirs = \"../input/h-and-m-personalized-fashion-recommendations/images/\"","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.131902Z","iopub.execute_input":"2022-03-29T18:18:09.132186Z","iopub.status.idle":"2022-03-29T18:18:09.144789Z","shell.execute_reply.started":"2022-03-29T18:18:09.132157Z","shell.execute_reply":"2022-03-29T18:18:09.14373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir(im_dirs)[0:4]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.146315Z","iopub.execute_input":"2022-03-29T18:18:09.146829Z","iopub.status.idle":"2022-03-29T18:18:09.161335Z","shell.execute_reply.started":"2022-03-29T18:18:09.146782Z","shell.execute_reply":"2022-03-29T18:18:09.160347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"load_pattern = os.path.join(\n \"../input/h-and-m-personalized-fashion-recommendations/images/070\", \"*.jpg\"\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.16385Z","iopub.execute_input":"2022-03-29T18:18:09.164609Z","iopub.status.idle":"2022-03-29T18:18:09.171902Z","shell.execute_reply.started":"2022-03-29T18:18:09.164562Z","shell.execute_reply":"2022-03-29T18:18:09.171279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images = imread_collection(\n load_pattern=load_pattern,\n conserve_memory=True,\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.172844Z","iopub.execute_input":"2022-03-29T18:18:09.17352Z","iopub.status.idle":"2022-03-29T18:18:09.33003Z","shell.execute_reply.started":"2022-03-29T18:18:09.17349Z","shell.execute_reply":"2022-03-29T18:18:09.329143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(images.files)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.331485Z","iopub.execute_input":"2022-03-29T18:18:09.331925Z","iopub.status.idle":"2022-03-29T18:18:09.338451Z","shell.execute_reply.started":"2022-03-29T18:18:09.331895Z","shell.execute_reply":"2022-03-29T18:18:09.337704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at the most popular items","metadata":{}},{"cell_type":"code","source":"imshow(*[i for i in images.files if \"0706016001\" in i])","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.339784Z","iopub.execute_input":"2022-03-29T18:18:09.340507Z","iopub.status.idle":"2022-03-29T18:18:09.81147Z","shell.execute_reply.started":"2022-03-29T18:18:09.340466Z","shell.execute_reply":"2022-03-29T18:18:09.810658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imshow(*[i for i in images.files if \"0706016002\" in i])","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:09.813083Z","iopub.execute_input":"2022-03-29T18:18:09.813578Z","iopub.status.idle":"2022-03-29T18:18:10.2459Z","shell.execute_reply.started":"2022-03-29T18:18:09.813536Z","shell.execute_reply":"2022-03-29T18:18:10.244472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No suprise to me that black trousers are the most popular item. Although, I would have also not been suprised if it were black t-shirt.\n\nWe should probably make a helper function to look up any arbitrary image.","metadata":{}},{"cell_type":"code","source":"def image_lookup(g_id):\n im = (\n f\"../input/h-and-m-personalized-fashion-recommendations/images/{g_id[:3]}\"\n + \"/\"\n + g_id\n + \".jpg\"\n )\n imshow(im)\n\n\nimage_lookup(\"0706016001\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.247279Z","iopub.execute_input":"2022-03-29T18:18:10.247743Z","iopub.status.idle":"2022-03-29T18:18:10.635474Z","shell.execute_reply.started":"2022-03-29T18:18:10.247697Z","shell.execute_reply":"2022-03-29T18:18:10.634622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n# More H&M data\n\nI tried to find some more H&M data online and came across another data set on Kaggle [here](https://www.kaggle.com/narayan5259/h-m-dataset)","metadata":{}},{"cell_type":"code","source":"H_M_2 = pd.read_csv(\"../input/h-m-dataset/H M Dataset 2019.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.636752Z","iopub.execute_input":"2022-03-29T18:18:10.637174Z","iopub.status.idle":"2022-03-29T18:18:10.656644Z","shell.execute_reply.started":"2022-03-29T18:18:10.637144Z","shell.execute_reply":"2022-03-29T18:18:10.65571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"H_M_2.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.658081Z","iopub.execute_input":"2022-03-29T18:18:10.658515Z","iopub.status.idle":"2022-03-29T18:18:10.680108Z","shell.execute_reply.started":"2022-03-29T18:18:10.658472Z","shell.execute_reply":"2022-03-29T18:18:10.679301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"H_M_2.groupby([\"Customer ID\"]).count().describe()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.681489Z","iopub.execute_input":"2022-03-29T18:18:10.682354Z","iopub.status.idle":"2022-03-29T18:18:10.741349Z","shell.execute_reply.started":"2022-03-29T18:18:10.682307Z","shell.execute_reply":"2022-03-29T18:18:10.740711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interesting dataset, there are 66 customers who purchased between 11 and 77 items.","metadata":{}},{"cell_type":"code","source":"H_M_2.query(\"`Customer ID` == 'RB-19465'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.742301Z","iopub.execute_input":"2022-03-29T18:18:10.742964Z","iopub.status.idle":"2022-03-29T18:18:10.775278Z","shell.execute_reply.started":"2022-03-29T18:18:10.742935Z","shell.execute_reply":"2022-03-29T18:18:10.774675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, now I see what I should have seen before, there are many duplicates","metadata":{}},{"cell_type":"code","source":"H_M_2.drop_duplicates(inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.777242Z","iopub.execute_input":"2022-03-29T18:18:10.77803Z","iopub.status.idle":"2022-03-29T18:18:10.7926Z","shell.execute_reply.started":"2022-03-29T18:18:10.777971Z","shell.execute_reply":"2022-03-29T18:18:10.791713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"H_M_2.groupby([\"Customer ID\"]).count().describe()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.794496Z","iopub.execute_input":"2022-03-29T18:18:10.795271Z","iopub.status.idle":"2022-03-29T18:18:10.850337Z","shell.execute_reply.started":"2022-03-29T18:18:10.795226Z","shell.execute_reply":"2022-03-29T18:18:10.849325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, so now there are a more reasonable 1-7 purchases per customer.","metadata":{}},{"cell_type":"code","source":"H_M_2.query(\"`Customer ID` == 'RB-19465'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.851666Z","iopub.execute_input":"2022-03-29T18:18:10.851984Z","iopub.status.idle":"2022-03-29T18:18:10.875312Z","shell.execute_reply.started":"2022-03-29T18:18:10.851953Z","shell.execute_reply":"2022-03-29T18:18:10.874485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So 3 belts cost 95 and 3 dresses cost 1.79??? Okay, this does not seem like a legitimate dataset. I wonder what the hell is going on here. \n\nIf this were an official data-set I would be suprised to see profit in there. Maybe, the discount column will help me with the dilema of the banding pattern I noticed in earlier plots.","metadata":{}},{"cell_type":"code","source":"H_M_2[\"Discount\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.876331Z","iopub.execute_input":"2022-03-29T18:18:10.876915Z","iopub.status.idle":"2022-03-29T18:18:10.890162Z","shell.execute_reply.started":"2022-03-29T18:18:10.876881Z","shell.execute_reply":"2022-03-29T18:18:10.889465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So they are in increments of 10 or 5.\n\nI am not convinced about this data set.","metadata":{}},{"cell_type":"code","source":"H_M_2.query(\"`Customer ID` == 'LC-16870'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.891257Z","iopub.execute_input":"2022-03-29T18:18:10.891877Z","iopub.status.idle":"2022-03-29T18:18:10.917875Z","shell.execute_reply.started":"2022-03-29T18:18:10.891827Z","shell.execute_reply":"2022-03-29T18:18:10.91674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"H_M_2.query(\"`Product ID` == 'TEC-AC-10001552'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.919027Z","iopub.execute_input":"2022-03-29T18:18:10.919672Z","iopub.status.idle":"2022-03-29T18:18:10.943836Z","shell.execute_reply.started":"2022-03-29T18:18:10.919568Z","shell.execute_reply":"2022-03-29T18:18:10.942856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"H_M_2[\"Product ID\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.945337Z","iopub.execute_input":"2022-03-29T18:18:10.945635Z","iopub.status.idle":"2022-03-29T18:18:10.956867Z","shell.execute_reply.started":"2022-03-29T18:18:10.945566Z","shell.execute_reply":"2022-03-29T18:18:10.955996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This seems to be largely useless as the dataset is such as small sample and potentially not even legitimate","metadata":{}},{"cell_type":"markdown","source":"\n\n
\n\n## Seasonality\n\n","metadata":{}},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.958372Z","iopub.execute_input":"2022-03-29T18:18:10.959095Z","iopub.status.idle":"2022-03-29T18:18:10.965087Z","shell.execute_reply.started":"2022-03-29T18:18:10.959062Z","shell.execute_reply":"2022-03-29T18:18:10.964565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"t_dat\"]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.966311Z","iopub.execute_input":"2022-03-29T18:18:10.966697Z","iopub.status.idle":"2022-03-29T18:18:10.978228Z","shell.execute_reply.started":"2022-03-29T18:18:10.966668Z","shell.execute_reply":"2022-03-29T18:18:10.97765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_3w = train[train['t_dat'] >= pd.to_datetime('2020-08-31')].copy()\n# train_2w = train[train['t_dat'] >= pd.to_datetime('2020-09-07')].copy()\n# train_1w = train[train['t_dat'] >= pd.to_datetime('2020-09-15')].copy()\ntrain[train[\"t_dat\"] >= pd.to_datetime(\"2020-08-31\")][\"article_id\"].value_counts()[0:12]\npd.to_datetime(\"2020-08-31\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:10.979157Z","iopub.execute_input":"2022-03-29T18:18:10.97974Z","iopub.status.idle":"2022-03-29T18:18:11.283272Z","shell.execute_reply.started":"2022-03-29T18:18:10.979706Z","shell.execute_reply":"2022-03-29T18:18:11.28265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_lookup(\"0924243001\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:11.284452Z","iopub.execute_input":"2022-03-29T18:18:11.284807Z","iopub.status.idle":"2022-03-29T18:18:11.696211Z","shell.execute_reply.started":"2022-03-29T18:18:11.284766Z","shell.execute_reply":"2022-03-29T18:18:11.695308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So it has just occured to me when looking at this jumper, that there should be some \"essential\" items that are essentially always popular - t-shirts perhaps, then there are seasonal items, maybe this jumper although we don't know where the data is from in the world so figuring out what season it is could be important for recommendations. I suspect people who know about clothes will know when the seasons change in-store. And then, after those two, stuff that is just popular becuase of hype. I suppose if we knew the lag or pre-empting of of a season, we could find out what country/hemisphere the data is from.\n\nI am going to Assume US. But will check UK first.","metadata":{}},{"cell_type":"code","source":"spring_UK_19 = pd.date_range(\"2019-03-01\", \"2019-06-21\", freq=\"d\")\nsummer_UK_19 = pd.date_range(\"2019-06-22\", \"2019-09-22\", freq=\"d\")\nautumn_UK_19 = pd.date_range(\"2019-09-23\", \"2019-12-21\", freq=\"d\")\nwinter_UK_19_20 = pd.date_range(\"2019-12-21\", \"2020-2-28\", freq=\"d\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:11.697761Z","iopub.execute_input":"2022-03-29T18:18:11.698247Z","iopub.status.idle":"2022-03-29T18:18:11.714424Z","shell.execute_reply.started":"2022-03-29T18:18:11.698199Z","shell.execute_reply":"2022-03-29T18:18:11.713199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spring_UK_19_top = (\n train[train[\"t_dat\"].isin(spring_UK_19)][\"article_id\"].value_counts()[0:12].index\n)\nsummer_UK_19_top = (\n train[train[\"t_dat\"].isin(summer_UK_19)][\"article_id\"].value_counts()[0:12].index\n)\nautumn_UK_19_top = (\n train[train[\"t_dat\"].isin(autumn_UK_19)][\"article_id\"].value_counts()[0:12].index\n)\nwinter_UK_19_20_top = (\n train[train[\"t_dat\"].isin(winter_UK_19_20)][\"article_id\"].value_counts()[0:12].index\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:11.716073Z","iopub.execute_input":"2022-03-29T18:18:11.716583Z","iopub.status.idle":"2022-03-29T18:18:17.075393Z","shell.execute_reply.started":"2022-03-29T18:18:11.716517Z","shell.execute_reply":"2022-03-29T18:18:17.074465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def image_lookup_path(g_id):\n im = (\n f\"../input/h-and-m-personalized-fashion-recommendations/images/{g_id[:3]}\"\n + \"/\"\n + g_id\n + \".jpg\"\n )\n return im\n\n\nfig, ax = plt.subplots(3, 4, figsize=(15, 10))\nax = ax.flatten()\nfig.suptitle(\"Summer?\", fontsize=22)\nfor i in range(len(summer_UK_19_top)):\n ax[i].title.set_text(summer_UK_19_top[i])\n im = imread(image_lookup_path(summer_UK_19_top[i]))\n ax[i].imshow(im)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:17.077078Z","iopub.execute_input":"2022-03-29T18:18:17.077383Z","iopub.status.idle":"2022-03-29T18:18:21.518886Z","shell.execute_reply.started":"2022-03-29T18:18:17.077339Z","shell.execute_reply":"2022-03-29T18:18:21.518182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ayyyy Looks like summer to me!","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 4, figsize=(15, 10))\nax = ax.flatten()\nfig.suptitle(\"Winter?\", fontsize=22)\n\nfor i in range(len(winter_UK_19_20_top)):\n ax[i].title.set_text(winter_UK_19_20_top[i])\n im = imread(image_lookup_path(winter_UK_19_20_top[i]))\n ax[i].imshow(im)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:21.519988Z","iopub.execute_input":"2022-03-29T18:18:21.520311Z","iopub.status.idle":"2022-03-29T18:18:25.592571Z","shell.execute_reply.started":"2022-03-29T18:18:21.520284Z","shell.execute_reply":"2022-03-29T18:18:25.591671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, so this does not look particularly wintery, but how often do we buy coats/jumpers??","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 4, figsize=(15, 10))\nax = ax.flatten()\nfig.suptitle(\"Autumn?\", fontsize=22)\n\nfor i in range(len(autumn_UK_19_top)):\n ax[i].title.set_text(autumn_UK_19_top[i])\n im = imread(image_lookup_path(autumn_UK_19_top[i]))\n ax[i].imshow(im)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:25.593847Z","iopub.execute_input":"2022-03-29T18:18:25.59459Z","iopub.status.idle":"2022-03-29T18:18:29.969369Z","shell.execute_reply.started":"2022-03-29T18:18:25.594544Z","shell.execute_reply":"2022-03-29T18:18:29.968344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Black work trousers sell all year round","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 4, figsize=(15, 10))\nax = ax.flatten()\nfig.suptitle(\"Spring?\", fontsize=22)\nfor i in range(len(spring_UK_19_top)):\n ax[i].title.set_text(spring_UK_19_top[i])\n try:\n im = imread(image_lookup_path(spring_UK_19_top[i]))\n except:\n pass\n ax[i].imshow(im)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:29.970524Z","iopub.execute_input":"2022-03-29T18:18:29.970752Z","iopub.status.idle":"2022-03-29T18:18:33.930852Z","shell.execute_reply.started":"2022-03-29T18:18:29.970725Z","shell.execute_reply":"2022-03-29T18:18:33.929969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some interesting images!\n\nPeople don't purchase as many socks in the summer! We wear flip-flops in the the summer I guess. Also with the exception of jeans, coloured stuff doesn't seem to be particularly popular!\n\n\n## I wonder what score you would get if you only submitted black trousers!\n\nBefore continuing I want to see what the most popular item is where people only purchased one item at the time of purchase. I realise I said I was going to do some modelling many ideas ago, I will have to refactor this notebook at somepoint.","metadata":{}},{"cell_type":"code","source":"train_dates = train[\"t_dat\"].unique()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:33.932434Z","iopub.execute_input":"2022-03-29T18:18:33.932903Z","iopub.status.idle":"2022-03-29T18:18:34.122763Z","shell.execute_reply.started":"2022-03-29T18:18:33.932864Z","shell.execute_reply":"2022-03-29T18:18:34.121688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All the one-time purchases on the first day of the data","metadata":{}},{"cell_type":"code","source":"train[train[\"t_dat\"] == train_dates[0]].groupby([\"customer_id\"]).filter(\n lambda x: x[\"customer_id\"].shape[0] == 1\n)[\"article_id\"].value_counts().index[0]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:34.124042Z","iopub.execute_input":"2022-03-29T18:18:34.124309Z","iopub.status.idle":"2022-03-29T18:18:35.81613Z","shell.execute_reply.started":"2022-03-29T18:18:34.124277Z","shell.execute_reply":"2022-03-29T18:18:35.815207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_lookup(\"0685687004\")","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:35.817309Z","iopub.execute_input":"2022-03-29T18:18:35.817547Z","iopub.status.idle":"2022-03-29T18:18:36.251592Z","shell.execute_reply.started":"2022-03-29T18:18:35.817514Z","shell.execute_reply":"2022-03-29T18:18:36.250467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I think these one time purchases maybe usefull for modelling so I will compile a df of onetime purchases next time.","metadata":{}},{"cell_type":"code","source":"def get_one_off_items(df):\n purchases = []\n train_dates = df[\"t_dat\"].unique()\n for i in train_dates:\n x = (\n df[df[\"t_dat\"] == i]\n .groupby([\"customer_id\"])\n .filter(lambda x: x[\"customer_id\"].shape[0] == 1)[\"article_id\"]\n .value_counts()\n .index[0]\n )\n purchases.append(x)\n return purchases","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:36.252908Z","iopub.execute_input":"2022-03-29T18:18:36.253135Z","iopub.status.idle":"2022-03-29T18:18:36.271941Z","shell.execute_reply.started":"2022-03-29T18:18:36.253108Z","shell.execute_reply":"2022-03-29T18:18:36.271134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"This takes a very long tome to run, if anyone has any ideas how I can\nspeed this up, please do comment\"\"\"\n\none_off_items = get_one_off_items(train)","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:18:36.273564Z","iopub.execute_input":"2022-03-29T18:18:36.273884Z","iopub.status.idle":"2022-03-29T18:36:35.174123Z","shell.execute_reply.started":"2022-03-29T18:18:36.273841Z","shell.execute_reply":"2022-03-29T18:36:35.171551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"popular_one_time_purchases = pd.Series(one_off_items).value_counts().index[:12]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:36:35.186551Z","iopub.execute_input":"2022-03-29T18:36:35.186881Z","iopub.status.idle":"2022-03-29T18:36:35.196983Z","shell.execute_reply.started":"2022-03-29T18:36:35.186849Z","shell.execute_reply":"2022-03-29T18:36:35.195882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 4, figsize=(15, 10))\nax = ax.flatten()\nfig.suptitle(\"Top 12 one-off purchases\", fontsize=22)\nfor i in range(len(popular_one_time_purchases)):\n ax[i].title.set_text(popular_one_time_purchases[i])\n try:\n im = imread(image_lookup_path(popular_one_time_purchases[i]))\n except:\n pass\n ax[i].imshow(im)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:36:35.19832Z","iopub.execute_input":"2022-03-29T18:36:35.198658Z","iopub.status.idle":"2022-03-29T18:36:39.411936Z","shell.execute_reply.started":"2022-03-29T18:36:35.198616Z","shell.execute_reply":"2022-03-29T18:36:39.410902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Yet again, black trousers are at the top, but the rest of these look like they are more expensive items. This is what I would expect for one-time purchases. Lets check to see if these are relatively expensive items.","metadata":{}},{"cell_type":"code","source":"def price_check(item_id):\n return train.query(f\"article_id == '{item_id}'\")[[\"price\"]].mean()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:36:39.413446Z","iopub.execute_input":"2022-03-29T18:36:39.413716Z","iopub.status.idle":"2022-03-29T18:36:39.423226Z","shell.execute_reply.started":"2022-03-29T18:36:39.413682Z","shell.execute_reply":"2022-03-29T18:36:39.422357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"article_price = pd.DataFrame(\n train.groupby(\"article_id\")[\"price\"].mean() * 590\n).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:36:39.424562Z","iopub.execute_input":"2022-03-29T18:36:39.425273Z","iopub.status.idle":"2022-03-29T18:36:46.210909Z","shell.execute_reply.started":"2022-03-29T18:36:39.4252Z","shell.execute_reply":"2022-03-29T18:36:46.210001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 9))\nplt.scatter(range(len(article_price.price)), article_price.price)\nx_pos = 0\nfor i in range(len(popular_one_time_purchases)):\n plt.text(\n x_pos,\n price_check(popular_one_time_purchases[i]) * 590,\n popular_one_time_purchases[i],\n )\n x_pos += 9000\nplt.axhline(article_price.price.mean(), color=\"red\", label=\"mean price\")\nplt.legend()\n;","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:36:46.212211Z","iopub.execute_input":"2022-03-29T18:36:46.212553Z","iopub.status.idle":"2022-03-29T18:37:08.571755Z","shell.execute_reply.started":"2022-03-29T18:36:46.212513Z","shell.execute_reply":"2022-03-29T18:37:08.571036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, so most of those one off-purchases were above the average price, but barely. The coat and the swimsuit were quite a bit more expensive and the leapord print top and the most popular item were slightly below average.","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n# Modelling?\n\nI have not participated in a competition of this type before, usually it is either classification or regression. \n\nThis will take a lot of reading for me to understand how to tackle this problem. On the surface it feels like feature engineering is where most of the time willbe spent.\n\n\n- You will be making purchase predictions for all customer_id values provided, regardless of whether these customers made purchases in the training data.\n\n- Customer that did not make any purchase during test period are excluded from the scoring.\n - So we still have to make a prediction it just doesn't count?\n\n- There is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.\n\nIt is interesting that by design we should always have 12 items. I wonder if people are more likely to buy something if they are recommended only the thing they would actually like or that thing + 11 other items?\n\nSo essentially, for each customer find the 12 most likely items for them to purchase over the next 7 days.\n\nThe evaluation criteria:\n\n$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k)$\n\nI have seen some submissions that do not use any sort of ML, which is a first for me on Kaggle. For example [this](https://www.kaggle.com/byfone/h-m-trending-products-weekly/notebook?scriptVersionId=88365577) which is currently the top ranked notebook on the leaderboard 1/3/2022. \n\nFrom what I have seen on the \"Code\" tab so far, most models seem to involve taking the most popular products from the last 7 days. Fast fashion? We want the stuff that other people have? Seem's strange.","metadata":{}},{"cell_type":"code","source":"sample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:37:08.572694Z","iopub.execute_input":"2022-03-29T18:37:08.573309Z","iopub.status.idle":"2022-03-29T18:37:08.587934Z","shell.execute_reply.started":"2022-03-29T18:37:08.573275Z","shell.execute_reply":"2022-03-29T18:37:08.586836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission[\"prediction\"][0]","metadata":{"execution":{"iopub.status.busy":"2022-03-29T18:37:08.58949Z","iopub.execute_input":"2022-03-29T18:37:08.589809Z","iopub.status.idle":"2022-03-29T18:37:08.599838Z","shell.execute_reply.started":"2022-03-29T18:37:08.589767Z","shell.execute_reply":"2022-03-29T18:37:08.599121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, so we need to give a string of productids seperated by a space.","metadata":{}},{"cell_type":"markdown","source":"# To do\n- investigate differences in gendered items, for reccomending items this would clearly be important.\n- refactor","metadata":{}},{"cell_type":"markdown","source":"# References and other Notebooks used\n- I used [this](https://www.kaggle.com/chiranjeevbit/h-m-personalized-recommendation-eda-wordcloud/comments) notebook by [Chiranjeev](https://www.kaggle.com/chiranjeevbit) to speed up the exploration.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# ","metadata":{}}]}