{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## **Intro**\n\n**The competition is dedicated to the product recomendations (H&M) **\n\n**Here we have different kinds of data that help us to get good recomendations: **\n\n๐ธ `images` - images of every article_id\n\n๐ `articles` - detailed metadata of every article_id\n\n๐ `customers` - detailed metadata of every customer_id\n\n๐งพ `transactions_train` - purchases with details","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom tqdm.notebook import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:18:42.953662Z","iopub.execute_input":"2022-03-28T02:18:42.954294Z","iopub.status.idle":"2022-03-28T02:18:44.058389Z","shell.execute_reply.started":"2022-03-28T02:18:42.954153Z","shell.execute_reply":"2022-03-28T02:18:44.057254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ncustomers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ntransactions = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:18:44.059920Z","iopub.execute_input":"2022-03-28T02:18:44.060166Z","iopub.status.idle":"2022-03-28T02:19:52.880475Z","shell.execute_reply.started":"2022-03-28T02:18:44.060136Z","shell.execute_reply":"2022-03-28T02:19:52.879286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **2. Articles**","metadata":{}},{"cell_type":"markdown","source":"** This table contains all h&m articles with details such as a type of product, a color, a product group and other features.** \n** Article data description: **\n\n> `article_id` **: A unique identifier of every article.** \n> `product_code`, `prod_name` **: A unique identifier of every product and its name (not the same).** \n> `product_type`, `product_type_name` **: The group of product_code and its name** \n> `graphical_appearance_no`, `graphical_appearance_name` **: The group of graphics and its name** \n> `colour_group_code`, `colour_group_name` **: The group of color and its name** \n> `graphical_appearance_no`, `graphical_appearance_name` **: The group of graphics and its name** \n> `perceived_colour_value_id`, `perceived_colour_value_name`, `perceived_colour_master_id`, `perceived_colour_master_name` **: The added color info** \n> `department_no`, `department_name`: **: A unique identifier of every dep and its name** \n> `index_code`, `index_name`: **: A unique identifier of every index and its name** \n> `index_group_no`, `index_group_name`: **: A group of indeces and its name** \n> `section_no`, `section_name`: **: A unique identifier of every section and its name** \n> `garment_group_no`, `garment_group_name`: **: A unique identifier of every garment and its name** \n> `detail_desc`: **: Details** ","metadata":{}},{"cell_type":"code","source":"articles.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:52.882363Z","iopub.execute_input":"2022-03-28T02:19:52.882730Z","iopub.status.idle":"2022-03-28T02:19:52.934412Z","shell.execute_reply.started":"2022-03-28T02:19:52.882660Z","shell.execute_reply":"2022-03-28T02:19:52.933717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(articles)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:52.936041Z","iopub.execute_input":"2022-03-28T02:19:52.937002Z","iopub.status.idle":"2022-03-28T02:19:52.943418Z","shell.execute_reply.started":"2022-03-28T02:19:52.936957Z","shell.execute_reply":"2022-03-28T02:19:52.942327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"** Customers data description: **\n\n> `customer_id` **: A unique identifier of every customer** \n> `FN` **: 1 or missed ** \n> `Active` **: 1 or missed** \n> `club_member_status` **: Status in club** \n> `fashion_news_frequency` **: How often H&M may send news to customer** \n> `age` **: The current age** \n> `postal_code` **: Postal code of customer** ","metadata":{}},{"cell_type":"code","source":"pd.options.display.max_rows = 50\ncustomers.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:52.945474Z","iopub.execute_input":"2022-03-28T02:19:52.946795Z","iopub.status.idle":"2022-03-28T02:19:52.966088Z","shell.execute_reply.started":"2022-03-28T02:19:52.946738Z","shell.execute_reply":"2022-03-28T02:19:52.965077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#quick_bar_chart('postal_code', (3, 3), False, customers, True)\nlen(customers)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:52.967495Z","iopub.execute_input":"2022-03-28T02:19:52.967892Z","iopub.status.idle":"2022-03-28T02:19:52.979602Z","shell.execute_reply.started":"2022-03-28T02:19:52.967847Z","shell.execute_reply":"2022-03-28T02:19:52.978753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nimport time\n\nimport logging\nimport threading\n\nimport math","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:52.981325Z","iopub.execute_input":"2022-03-28T02:19:52.981596Z","iopub.status.idle":"2022-03-28T02:19:52.991227Z","shell.execute_reply.started":"2022-03-28T02:19:52.981565Z","shell.execute_reply":"2022-03-28T02:19:52.989752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ProgressBar:\n \n def __init__(self, transactions, trans_len = None):\n self.percent_intervals = 3\n self.percent_rounded_decimals = 2\n if trans_len == None:\n self.trans_len = len(transactions)\n else:\n self.trans_len = trans_len\n self.percent = int(self.trans_len / 10**self.percent_intervals)\n self.epsilon = 10**-2\n self.start = time.time()\n\n def check(self, i, chunked=False):\n if i % self.percent == 0 or i + 1 == self.trans_len or chunked:\n end = time.time()\n percent_decimal = i / (self.trans_len - 1)\n percent_current = percent_decimal * 100\n time_elapsed = end - self.start\n time_estimated = time_elapsed / (percent_decimal + self.epsilon)\n sys.stdout.write(\"%6.2f%% time elapsed: %d, estimated: %d\\r\" % (percent_current, time_elapsed, time_estimated))","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:52.993103Z","iopub.execute_input":"2022-03-28T02:19:52.993398Z","iopub.status.idle":"2022-03-28T02:19:53.005063Z","shell.execute_reply.started":"2022-03-28T02:19:52.993361Z","shell.execute_reply":"2022-03-28T02:19:53.003602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_id_hash = {}\npb = ProgressBar(articles)\nfor i in range(len(articles)):\n columns = ['prod_name',\n 'product_type_name',\n 'product_group_name',\n 'graphical_appearance_name',\n 'colour_group_name',\n 'department_name',\n 'index_name',\n 'index_group_name',\n 'section_name',\n 'garment_group_name',\n 'detail_desc']\n row = []\n for column in columns:\n row.append(articles[column][i])\n articles_id = articles['article_id'][i]\n articles_id_hash[articles_id] = row\n pb.check(i)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:19:53.009863Z","iopub.execute_input":"2022-03-28T02:19:53.010630Z","iopub.status.idle":"2022-03-28T02:20:00.526154Z","shell.execute_reply.started":"2022-03-28T02:19:53.010590Z","shell.execute_reply":"2022-03-28T02:20:00.525259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:20:00.527489Z","iopub.execute_input":"2022-03-28T02:20:00.528017Z","iopub.status.idle":"2022-03-28T02:20:00.555233Z","shell.execute_reply.started":"2022-03-28T02:20:00.527973Z","shell.execute_reply":"2022-03-28T02:20:00.554108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers_id_hash = {}\nnan_count = 0\npb = ProgressBar(customers)\nfor i in range(len(customers)):\n customer_age = customers['age'][i]\n customer_id = customers['customer_id'][i]\n customers_id_hash[customer_id] = customer_age\n if math.isnan(customer_age):\n nan_count += 1\n pb.check(i)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:20:00.556648Z","iopub.execute_input":"2022-03-28T02:20:00.558193Z","iopub.status.idle":"2022-03-28T02:20:16.664932Z","shell.execute_reply.started":"2022-03-28T02:20:00.558152Z","shell.execute_reply":"2022-03-28T02:20:16.663873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nan_count / len(customers) * 100","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:20:16.666030Z","iopub.execute_input":"2022-03-28T02:20:16.666252Z","iopub.status.idle":"2022-03-28T02:20:16.672831Z","shell.execute_reply.started":"2022-03-28T02:20:16.666226Z","shell.execute_reply":"2022-03-28T02:20:16.671888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"customers_id_hash['00000dbacae5abe5e23885899a1fa44253a17956c6d1c3d25f88aa139fdfc657']","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:09:03.857219Z","iopub.execute_input":"2022-03-28T02:09:03.857452Z","iopub.status.idle":"2022-03-28T02:09:03.875051Z","shell.execute_reply.started":"2022-03-28T02:09:03.85742Z","shell.execute_reply":"2022-03-28T02:09:03.873773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.max(customers['age']), np.min(customers['age']), np.mean(customers['age']), np.median(customers['age'])","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:09:03.87715Z","iopub.execute_input":"2022-03-28T02:09:03.877411Z","iopub.status.idle":"2022-03-28T02:09:03.92244Z","shell.execute_reply.started":"2022-03-28T02:09:03.877383Z","shell.execute_reply":"2022-03-28T02:09:03.920997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **4. Transactions**","metadata":{}},{"cell_type":"markdown","source":"[**Go to Table of Content**](#Content)","metadata":{}},{"cell_type":"markdown","source":"** Transactions data description: **\n\n> `t_dat` **: A unique identifier of every customer** \n> `customer_id` **: A unique identifier of every customer ** **(in ** `customers` ** table)** \n> `article_id` **: A unique identifier of every article** **(in ** `articles` ** table)** \n> `price` **: Price of purchase** \n> `sales_channel_id` **: 1 or 2** ","metadata":{}},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:09:03.923916Z","iopub.execute_input":"2022-03-28T02:09:03.924129Z","iopub.status.idle":"2022-03-28T02:09:03.938016Z","shell.execute_reply.started":"2022-03-28T02:09:03.924101Z","shell.execute_reply":"2022-03-28T02:09:03.936571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(transactions)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:09:03.94096Z","iopub.execute_input":"2022-03-28T02:09:03.941259Z","iopub.status.idle":"2022-03-28T02:09:03.954938Z","shell.execute_reply.started":"2022-03-28T02:09:03.94123Z","shell.execute_reply":"2022-03-28T02:09:03.954125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"percent_intervals = 3\npercent_rounded_decimals = 2\ntrans_len = len(transactions)\npercent = int(trans_len / 10**percent_intervals)\nepsilon = 10**-2\nstart = time.time()\nfor i in range(0,len(transactions),30):\n transactions['t_dat'][i]\n if i % percent == 0 or i + 1 == trans_len:\n end = time.time()\n percent_decimal = i / (trans_len - 1)\n percent_current = percent_decimal * 100\n time_elapsed = end - start\n time_estimated = time_elapsed / (percent_decimal + epsilon)\n sys.stdout.write(\"%6.2f%% time elapsed: %d, estimated: %d\\r\" % (percent_current, time_elapsed, time_estimated))","metadata":{"execution":{"iopub.status.busy":"2022-03-26T12:43:23.091071Z","iopub.execute_input":"2022-03-26T12:43:23.091892Z","iopub.status.idle":"2022-03-26T12:43:28.802229Z","shell.execute_reply.started":"2022-03-26T12:43:23.091842Z","shell.execute_reply":"2022-03-26T12:43:28.801171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"t_dat = transactions['t_dat']\npb = ProgressBar(transactions)\nfor i in range(0,len(transactions),1):\n t_dat[i]\n pb.check(i)","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:14:57.039117Z","iopub.execute_input":"2022-03-26T13:14:57.039454Z","iopub.status.idle":"2022-03-26T13:15:29.727321Z","shell.execute_reply.started":"2022-03-26T13:14:57.039411Z","shell.execute_reply":"2022-03-26T13:15:29.726253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_list = []\npb = ProgressBar(transactions)\nfor i in range(0,len(transactions),1):\n sub_list = []\n sub_list.append(transactions['t_dat'][i])\n sub_list.append(transactions['customer_id'][i])\n sub_list.append(transactions['article_id'][i])\n sub_list.append(transactions['price'][i])\n speed_list.append(sub_list)\n pb.check(i)","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:26:23.847311Z","iopub.execute_input":"2022-03-26T13:26:23.847592Z","iopub.status.idle":"2022-03-26T13:37:56.911169Z","shell.execute_reply.started":"2022-03-26T13:26:23.847564Z","shell.execute_reply":"2022-03-26T13:37:56.91017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(speed_list)","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:24:33.103522Z","iopub.execute_input":"2022-03-26T13:24:33.104272Z","iopub.status.idle":"2022-03-26T13:24:33.110966Z","shell.execute_reply.started":"2022-03-26T13:24:33.10422Z","shell.execute_reply":"2022-03-26T13:24:33.110117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_list = []\npb = ProgressBar(speed_list)\nfor i in range(0,len(speed_list)):\n new_list.append(speed_list[i][2])\n if i % 1000000 == 0:\n print(speed_list[i][2])\n pb.check(i)","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:44:12.640491Z","iopub.execute_input":"2022-03-26T13:44:12.640789Z","iopub.status.idle":"2022-03-26T13:44:28.967289Z","shell.execute_reply.started":"2022-03-26T13:44:12.640751Z","shell.execute_reply":"2022-03-26T13:44:28.966378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2022-03-27T10:55:02.882922Z","iopub.execute_input":"2022-03-27T10:55:02.883526Z","iopub.status.idle":"2022-03-27T10:55:02.887767Z","shell.execute_reply.started":"2022-03-27T10:55:02.883475Z","shell.execute_reply":"2022-03-27T10:55:02.887159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def thread_function(name):\n print(\"Thread %s: starting\", name)\n time.sleep(3 - name)\n print(\"Thread %s: finishing\", name)\n","metadata":{"execution":{"iopub.status.busy":"2022-03-26T12:56:02.172405Z","iopub.execute_input":"2022-03-26T12:56:02.172981Z","iopub.status.idle":"2022-03-26T12:56:02.176799Z","shell.execute_reply.started":"2022-03-26T12:56:02.172942Z","shell.execute_reply":"2022-03-26T12:56:02.176249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Main : before creating thread\")\nfor i in range(0,2):\n x = threading.Thread(target=thread_function, args=(i,))\n print(\"Main : before running thread\")\n x.start()\nprint(\"Main : wait for the thread to finish\")\n# x.join()\nprint(\"Main : all done\")","metadata":{"execution":{"iopub.status.busy":"2022-03-26T12:56:02.607668Z","iopub.execute_input":"2022-03-26T12:56:02.608199Z","iopub.status.idle":"2022-03-26T12:56:02.615512Z","shell.execute_reply.started":"2022-03-26T12:56:02.608166Z","shell.execute_reply":"2022-03-26T12:56:02.614413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#speed_list = []\n#def thread_function(name):\n# pb = 0\n# if name == 0:\n# pb = ProgressBar(transactions)\n# for i in range(0, len(transactions)):\n# if name == 0 and i % 2 == 0:\n# continue\n# elif name == 1 and i % 2 == 1:\n# continue\n# speed_list.append(transactions['t_dat'][i])\n# if name == 0:\n# pb.check(i)\n# \n# print(\"done\",name)\n# \n#for i in range(0,2):\n# x = threading.Thread(target=thread_function, args=(i,))\n# print(\"Main : before running thread\")\n# x.start()","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:10:48.126131Z","iopub.execute_input":"2022-03-26T13:10:48.12645Z","iopub.status.idle":"2022-03-26T13:10:48.957071Z","shell.execute_reply.started":"2022-03-26T13:10:48.126417Z","shell.execute_reply":"2022-03-26T13:10:48.223527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:48:08.256826Z","iopub.execute_input":"2022-03-26T13:48:08.257599Z","iopub.status.idle":"2022-03-26T13:48:08.273714Z","shell.execute_reply.started":"2022-03-26T13:48:08.257559Z","shell.execute_reply":"2022-03-26T13:48:08.273027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:48:14.393836Z","iopub.execute_input":"2022-03-26T13:48:14.394394Z","iopub.status.idle":"2022-03-26T13:48:14.405738Z","shell.execute_reply.started":"2022-03-26T13:48:14.394319Z","shell.execute_reply":"2022-03-26T13:48:14.405107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions_age_list = []\npb = ProgressBar(transactions)\nfor i in range(0,len(transactions),100):\n customer_id = transactions['customer_id'][i]\n transactions_age_list.append(customers_id_hash[customer_id])\n pb.check(i, True)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T10:58:08.839789Z","iopub.execute_input":"2022-03-27T10:58:08.840114Z","iopub.status.idle":"2022-03-27T10:58:32.394589Z","shell.execute_reply.started":"2022-03-27T10:58:08.840085Z","shell.execute_reply":"2022-03-27T10:58:32.385642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 6), dpi=80)\nn, bins, patches = plt.hist(transactions_age_list, 99-16, density=False, facecolor='#CC071E')#, facecolor='g', alpha=0.75)\nplt.xlabel('Age (Years)')\nplt.ylabel('Count')\nplt.title('Histogram of H&M Customer Ages Transaction')\nplt.xlim(15, 100)\nplt.ylim(0, 20000)\nplt.grid(True)\nplt.xticks(np.arange(15, 105, step=5))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:57:13.306775Z","iopub.execute_input":"2022-03-26T13:57:13.307041Z","iopub.status.idle":"2022-03-26T13:57:14.96858Z","shell.execute_reply.started":"2022-03-26T13:57:13.307012Z","shell.execute_reply":"2022-03-26T13:57:14.967689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2022-03-26T14:09:07.038612Z","iopub.execute_input":"2022-03-26T14:09:07.039413Z","iopub.status.idle":"2022-03-26T14:09:07.045288Z","shell.execute_reply.started":"2022-03-26T14:09:07.039355Z","shell.execute_reply":"2022-03-26T14:09:07.044574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list = {}\npb = ProgressBar(transactions)\nfor i in range(0,len(transactions),1):\n customer_id = transactions['customer_id'][i]\n article_id = transactions['article_id'][i]\n customer_age = customers_id_hash[customer_id]\n if math.isnan(customer_age):\n customer_age = 'nan'\n else:\n customer_age = str(int(customer_age))\n if not customer_age in age_article_id_list:\n age_article_id_list[customer_age] = {}\n if not article_id in age_article_id_list[customer_age]:\n age_article_id_list[customer_age][article_id] = 0\n age_article_id_list[customer_age][article_id] += 1\n \n pb.check(i, True)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T03:03:59.274806Z","iopub.execute_input":"2022-03-28T03:03:59.275521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_list = []\npb = ProgressBar(transactions)\nfor i in range(0,len(transactions),100):\n sub_list = []\n sub_list.append(transactions['t_dat'][i])\n sub_list.append(transactions['customer_id'][i])\n sub_list.append(transactions['article_id'][i])\n sub_list.append(transactions['price'][i])\n speed_list.append(sub_list)\n pb.check(i)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T03:03:05.365292Z","iopub.execute_input":"2022-03-28T03:03:05.365634Z","iopub.status.idle":"2022-03-28T03:03:13.806986Z","shell.execute_reply.started":"2022-03-28T03:03:05.365596Z","shell.execute_reply":"2022-03-28T03:03:13.804434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list = {}\npb = ProgressBar(speed_list)\nfor i in range(0,len(speed_list),1):\n customer_id = speed_list[i][1]\n article_id = speed_list[i][2]\n customer_age = customers_id_hash[customer_id]\n if math.isnan(customer_age):\n customer_age = 'nan'\n else:\n customer_age = str(int(customer_age))\n if not customer_age in age_article_id_list:\n age_article_id_list[customer_age] = {}\n if not article_id in age_article_id_list[customer_age]:\n age_article_id_list[customer_age][article_id] = 0\n age_article_id_list[customer_age][article_id] += 1\n \n pb.check(i, True)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:09:41.545665Z","iopub.execute_input":"2022-03-28T02:09:41.545907Z","iopub.status.idle":"2022-03-28T02:10:02.781071Z","shell.execute_reply.started":"2022-03-28T02:09:41.545878Z","shell.execute_reply":"2022-03-28T02:10:02.765816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list.keys()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:10:10.194625Z","iopub.execute_input":"2022-03-28T02:10:10.195113Z","iopub.status.idle":"2022-03-28T02:10:10.20264Z","shell.execute_reply.started":"2022-03-28T02:10:10.19508Z","shell.execute_reply":"2022-03-28T02:10:10.201959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def sort_obj_by_keys(obj, sort_on_keys=True, reverse=False):\n \n keys = []\n values = []\n for key in obj.keys():\n keys.append(key)\n values.append(obj[key])\n \n ind = None\n if sort_on_keys:\n ind = np.argsort(keys)\n else:\n ind = np.argsort(values)\n \n keys_sort = np.array(keys)[ind]\n values_sort = np.array(values)[ind]\n \n if reverse:\n keys_sort = np.flip(keys_sort)\n values_sort = np.flip(values_sort)\n \n newObj = {}\n newObj['keys'] = keys_sort\n newObj['values'] = values_sort\n \n return newObj","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:21:17.341809Z","iopub.execute_input":"2022-03-28T02:21:17.342757Z","iopub.status.idle":"2022-03-28T02:21:17.349086Z","shell.execute_reply.started":"2022-03-28T02:21:17.342716Z","shell.execute_reply":"2022-03-28T02:21:17.348432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list_sort = sort_obj_by_keys(age_article_id_list)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#age_article_id_list_sort","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"values = age_article_id_list_sort['values']\nnewValues = []\nfor i in range(len(values)):\n value = values[i]\n newValue = sort_obj_by_keys(value, False, True)\n newValues.append(newValue)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list_sort['values'] = newValues","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(age_article_id_list_sort['keys'])):\n age = age_article_id_list_sort['keys'][i]\n articles_per_age = age_article_id_list_sort['values'][i]\n \n print('Age:',age)\n end = 10\n if len(articles_per_age['keys']) < 10:\n end = len(articles_per_age['keys'])\n for j in range(0,end):\n article_id = articles_per_age['keys'][j]\n article_count = articles_per_age['values'][j]\n \n print('Count:',article_count, articles_id_hash[article_id][0])#, articles_id_hash[article_id][10])\n \n print()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions['article_id'][0]","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:40:40.705232Z","iopub.execute_input":"2022-03-28T02:40:40.705647Z","iopub.status.idle":"2022-03-28T02:40:40.713098Z","shell.execute_reply.started":"2022-03-28T02:40:40.705593Z","shell.execute_reply":"2022-03-28T02:40:40.712424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions['article_id']","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:40:49.374649Z","iopub.execute_input":"2022-03-28T02:40:49.374994Z","iopub.status.idle":"2022-03-28T02:40:49.388477Z","shell.execute_reply.started":"2022-03-28T02:40:49.374957Z","shell.execute_reply":"2022-03-28T02:40:49.387207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list = {}\npb = ProgressBar(transactions)\nfor i in range(0,len(transactions),100):\n customer_id = transactions['customer_id'][i]\n article_id = transactions['article_id'][i]\n customer_age = customers_id_hash[customer_id]\n if math.isnan(customer_age):\n customer_age = 'nan'\n else:\n customer_age = str(int(customer_age/5))\n if not customer_age in age_article_id_list:\n age_article_id_list[customer_age] = {}\n if not article_id in age_article_id_list[customer_age]:\n age_article_id_list[customer_age][article_id] = 0\n age_article_id_list[customer_age][article_id] += 1\n \n pb.check(i, True)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:44:33.223366Z","iopub.execute_input":"2022-03-28T02:44:33.223737Z","iopub.status.idle":"2022-03-28T02:45:01.460301Z","shell.execute_reply.started":"2022-03-28T02:44:33.223695Z","shell.execute_reply":"2022-03-28T02:45:01.455115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list_sort = sort_obj_by_keys(age_article_id_list)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:45:01.463131Z","iopub.execute_input":"2022-03-28T02:45:01.463408Z","iopub.status.idle":"2022-03-28T02:45:01.468600Z","shell.execute_reply.started":"2022-03-28T02:45:01.463377Z","shell.execute_reply":"2022-03-28T02:45:01.467440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"values = age_article_id_list_sort['values']\nnewValues = []\nfor i in range(len(values)):\n value = values[i]\n newValue = sort_obj_by_keys(value, False, True)\n newValues.append(newValue)","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:45:01.469873Z","iopub.execute_input":"2022-03-28T02:45:01.470103Z","iopub.status.idle":"2022-03-28T02:45:01.596106Z","shell.execute_reply.started":"2022-03-28T02:45:01.470066Z","shell.execute_reply":"2022-03-28T02:45:01.595422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"age_article_id_list_sort['values'] = newValues","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:45:01.597427Z","iopub.execute_input":"2022-03-28T02:45:01.597808Z","iopub.status.idle":"2022-03-28T02:45:01.603156Z","shell.execute_reply.started":"2022-03-28T02:45:01.597770Z","shell.execute_reply":"2022-03-28T02:45:01.602212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(age_article_id_list_sort['keys'])):\n age = age_article_id_list_sort['keys'][i]\n articles_per_age = age_article_id_list_sort['values'][i]\n \n print('Age:',age)\n end = 10\n if len(articles_per_age['keys']) < 10:\n end = len(articles_per_age['keys'])\n for j in range(0,end):\n article_id = articles_per_age['keys'][j]\n article_count = articles_per_age['values'][j]\n \n print('Count:',article_count, articles_id_hash[article_id][0])#, articles_id_hash[article_id][10])\n \n print()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:45:01.604649Z","iopub.execute_input":"2022-03-28T02:45:01.605172Z","iopub.status.idle":"2022-03-28T02:45:01.656539Z","shell.execute_reply.started":"2022-03-28T02:45:01.605136Z","shell.execute_reply":"2022-03-28T02:45:01.655220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import csv\n\n# open the file in the write mode\nf = open('submission.csv', 'w')\n\n# create the csv writer\nwriter = csv.writer(f)\n\n# write a row to the csv file\nwriter.writerow(['customer_id','prediction'])\n\npb = ProgressBar(customers)\nfor i in range(len(customers['customer_id'])):\n customer_id = customers['customer_id'][i]\n customer_age = customers['age'][i]\n article_id = articles['article_id'][0]\n if math.isnan(customer_age):\n customer_age = 'nan'\n else:\n customer_age = str(int(customer_age/5))\n \n article_ids = ''\n for j in range(0,len(age_article_id_list_sort['keys'])):\n key_age = age_article_id_list_sort['keys'][j]\n if key_age == customer_age:\n values_age = age_article_id_list_sort['values'][j]\n end = 10\n if len(values_age['keys']) < 10:\n end = len(values_age['keys'])\n for k in range(0,end):\n article_id = str(values_age['keys'][k])\n for l in range(len(article_id), 10):\n article_id = '0' + article_id\n article_ids += str(article_id) + ' '\n writer.writerow([customer_id, article_ids])\n pb.check(i)\n# close the file\nf.close()","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:45:17.532054Z","iopub.execute_input":"2022-03-28T02:45:17.532402Z","iopub.status.idle":"2022-03-28T02:46:25.870276Z","shell.execute_reply.started":"2022-03-28T02:45:17.532355Z","shell.execute_reply":"2022-03-28T02:46:25.869199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#age_article_id_list_sort","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:42:40.637664Z","iopub.execute_input":"2022-03-28T02:42:40.640445Z","iopub.status.idle":"2022-03-28T02:42:40.644535Z","shell.execute_reply.started":"2022-03-28T02:42:40.640391Z","shell.execute_reply":"2022-03-28T02:42:40.643869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(age_article_id_list_sort['values'][0]['keys'][0])","metadata":{"execution":{"iopub.status.busy":"2022-03-28T02:43:08.963054Z","iopub.execute_input":"2022-03-28T02:43:08.963814Z","iopub.status.idle":"2022-03-28T02:43:08.970639Z","shell.execute_reply.started":"2022-03-28T02:43:08.963771Z","shell.execute_reply":"2022-03-28T02:43:08.969861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}