{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Data Sets / Structure\n\nI created a notebook that modifies the provided datasets in order to simplify model training afterwards. Additional attributes are created and explained below.\n\n## transactions.pqt\n\n- `t_dat`\n- `customer_id_int` - converted `customer_id`\n- `article_id`\n- `price` - multiplied by 590, units are probably Euro\n- `price_min_a` - minimum price of article up to timepoint `t_dat` (WITHOUT DISCOUNTED MEMBER PRICES)\n- `price_max_a` - maximum price of article up to timepoint `t_dat`\n- `price_min_u` - minimum price the customer has paid up to timepoint `t_dat` **TODO**\n- `price_mean_u` - average price the customer has paid up to timepoint `t_dat` **TODO**\n- `price_max_u` - maximum price the customer has paid up to timepoint `t_dat` **TODO**\n- `weekly_quotient` - number of articles sold in last week of transactions divided by number of articles sold in week of `t_dat`\n- `art_age_weeks` - the article's number of weeks since its first apperance, e.g. 5 means this is the 5th week of availability (min=1)\n- `cust_last_purch_weeks` - the number of weeks between the customers last purchase and the current transaction **TODO**\n\n## transactions_week.pqt\n\n\n\n## customers.pqt\n\n- `customer_id_int` - converted `customer_id`\n- `FN`\n- `Active`\n- `club_member_status`\n- `fashion_news_frequency`\n- `age`\n- `postal_code`\n- `t_dat_min` - date first transaction in transactions.pqt\n- `t_dat_max` - date last transaction in transactions.pqt\n- `ct_carts` - count of different carts in transactions.pqt (==0 -> cold user)\n- `price_min`, `price_mean`, `price_max` - the minimum/average/maximum price the customer has paid in transactions (i.e. for a single article)\n- `cart_value_min`, `cart_value_mean`, `cart_value_max` - the minimum/average/maximum cart value of the customer\n- `index_group_name_*` - quotient how many different articles (not transactions) the customer has bought from each index group name -> proxy for gender\n\n**For model training (e.g. NN), the last attributes of customers suffer from leakage ... need to create rolling attributes also**\n\n## articles.pqt\n\n- ...\n- `hist` - does article show up in transactions.pqt? (==0 -> cold item) **REDUNDANT? -> t_dat_min != NaN**\n- `t_dat_min` - date first appearance in transactions.pqt\n- `t_dat_max` - date last appearance in transactions.pqt\n- `ct_carts` - count of different carts in transactions.pqt, \"How well distributed was this article among different carts?\" **TODO**\n- `price_min` - minimum price in transactions.pqt (WITHOUT DISCOUNTED MEMBER PRICES)\n- `price_max` - maximum price in transactions.pqt\n\n## customers_article_df.pqt\n\nInteraction between customer `c` and article `a`\n\n- `customer_id_int`\n- `article_id`\n- `ct` - number of times customer `c` bought article `a`\n- `ct_carts` - number of carts of customer `c` that contained article `a` \n- `t_dat_min` - first time customer `c` bought article `a`\n- `t_dat_max` - last time customer `c` bought article `a`\n\n\n# Import Packages","metadata":{}},{"cell_type":"code","source":"import cudf\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport json\nimport time\nimport math\n\nPREFIX = 'm_'\n\nprint('RAPIDS version',cudf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:04:22.487889Z","iopub.execute_input":"2022-03-20T00:04:22.488227Z","iopub.status.idle":"2022-03-20T00:04:26.322308Z","shell.execute_reply.started":"2022-03-20T00:04:22.488135Z","shell.execute_reply":"2022-03-20T00:04:26.321532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Data Sets and Convert Data\n\nNote that `transactions_train.csv` ~ **3.49GB** -> use `cudf` to import and handle data manipulation.\n\nWe remove `customer_id` and replace/work with `customer_id_int`.\n\nReference: \n- https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635\n- https://www.kaggle.com/cdeotte/recommend-items-purchased-together-0-021/notebook\n\n## Transactions\n\nPerform memory saving operations and remove `price` & `sales_channel_id`.","metadata":{}},{"cell_type":"code","source":"trans_df = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:04:26.324153Z","iopub.execute_input":"2022-03-20T00:04:26.324433Z","iopub.status.idle":"2022-03-20T00:05:05.980754Z","shell.execute_reply.started":"2022-03-20T00:04:26.324395Z","shell.execute_reply":"2022-03-20T00:05:05.979929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trans_df['customer_id_int'] = trans_df['customer_id'].str[-16:].str.hex_to_int().astype('int64')\ntrans_df['article_id'] = trans_df.article_id.astype('int32')\ntrans_df.t_dat = cudf.to_datetime(trans_df.t_dat)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:05.982085Z","iopub.execute_input":"2022-03-20T00:05:05.982374Z","iopub.status.idle":"2022-03-20T00:05:06.434226Z","shell.execute_reply.started":"2022-03-20T00:05:05.982339Z","shell.execute_reply":"2022-03-20T00:05:06.433324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trans_df['price'] = trans_df['price'].astype('float32')*590 # from discussions on competition\ntrans_df = trans_df.round({'price':2})","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:06.436908Z","iopub.execute_input":"2022-03-20T00:05:06.437204Z","iopub.status.idle":"2022-03-20T00:05:06.501880Z","shell.execute_reply.started":"2022-03-20T00:05:06.437165Z","shell.execute_reply":"2022-03-20T00:05:06.501046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trans_df = trans_df[['t_dat','customer_id_int','article_id', 'price', 'sales_channel_id']]","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:06.505191Z","iopub.execute_input":"2022-03-20T00:05:06.505828Z","iopub.status.idle":"2022-03-20T00:05:06.529083Z","shell.execute_reply.started":"2022-03-20T00:05:06.505795Z","shell.execute_reply":"2022-03-20T00:05:06.528280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trans_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:06.530249Z","iopub.execute_input":"2022-03-20T00:05:06.531227Z","iopub.status.idle":"2022-03-20T00:05:06.564763Z","shell.execute_reply.started":"2022-03-20T00:05:06.531167Z","shell.execute_reply":"2022-03-20T00:05:06.563969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Trending\nReference:\n- https://www.kaggle.com/ebn7amdi/trending/notebook","metadata":{}},{"cell_type":"code","source":"df = trans_df[['t_dat', 'customer_id_int', 'article_id']].to_pandas()\nlast_ts = df['t_dat'].max()\ndf = df.join(df.groupby('t_dat')['t_dat'].max().transform(lambda d: last_ts - (last_ts - d).floor('7D')), on=\"t_dat\", how=\"left\", rsuffix=\"_ldbw\")\ndf = df.rename(columns={'t_dat_ldbw': 'ldbw'})\nweekly_sales = df.drop('customer_id_int', axis=1).groupby(['ldbw', 'article_id']).count()\nweekly_sales = weekly_sales.rename(columns={'t_dat': 'count'})\ndf = df.join(weekly_sales, on=['ldbw', 'article_id'])\nweekly_sales = weekly_sales.reset_index().set_index('article_id')\ndf = df.join(\n    weekly_sales.loc[weekly_sales['ldbw']==last_ts, ['count']],\n    on='article_id', rsuffix=\"_targ\")\ndf['count_targ'].fillna(0, inplace=True)\n# del weekly_sales\ndf['quotient'] = df['count_targ'] / df['count'] # quotient is high when lot of particular article is bought in last week, compared to current week\n\ntrans_df['weekly_quotient'] = df['quotient'].astype('float32')\n\n\n# del df\n\n# e.g. can be used like this\n# trans_df.groupby('article_id')['weekly_quotient'].sum().nlargest(20).index.to_arrow()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:06.565832Z","iopub.execute_input":"2022-03-20T00:05:06.566086Z","iopub.status.idle":"2022-03-20T00:05:24.388536Z","shell.execute_reply.started":"2022-03-20T00:05:06.566049Z","shell.execute_reply":"2022-03-20T00:05:24.387734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Article's age in weeks\ndf = df.merge(df.groupby('article_id')['t_dat'].min(), on='article_id', how='left')\ndf['art_age_weeks'] = (df['t_dat_x'] - df['t_dat_y']).astype('timedelta64[D]').astype('int16').apply(lambda d: 1 + math.floor(d/7))\ndf['art_age_weeks'] = df['art_age_weeks'].astype('int16')\ntrans_df['art_age_weeks'] = df['art_age_weeks']","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:24.389706Z","iopub.execute_input":"2022-03-20T00:05:24.390616Z","iopub.status.idle":"2022-03-20T00:05:57.226291Z","shell.execute_reply.started":"2022-03-20T00:05:24.390584Z","shell.execute_reply":"2022-03-20T00:05:57.225460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Time since customer's last purchase in weeks cust_last_purch_weeks\n#df = df.drop(['count_targ', 'quotient', 't_dat_y', 'art_age_weeks', 'ldbw', 'count'], axis=1)\n#df = df.rename(columns={\"t_dat_x\": \"t_dat\"})\n#df['customer_last_purchase_t_dat'] = df.groupby(['customer_id_int'])['t_dat'].cummax()\n#df['okay'] = df.groupby(['customer_id_int'])['t_dat'].shift()\n#del df\n# df.merge(df.groupby('customer_id_int')['t_dat'].cummax(), on='customer_id_int', how='left')","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:57.227550Z","iopub.execute_input":"2022-03-20T00:05:57.227801Z","iopub.status.idle":"2022-03-20T00:05:57.231833Z","shell.execute_reply.started":"2022-03-20T00:05:57.227767Z","shell.execute_reply":"2022-03-20T00:05:57.231008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trans_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:57.233262Z","iopub.execute_input":"2022-03-20T00:05:57.233756Z","iopub.status.idle":"2022-03-20T00:05:57.251752Z","shell.execute_reply.started":"2022-03-20T00:05:57.233718Z","shell.execute_reply":"2022-03-20T00:05:57.251038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Customers\n\n**From customers.csv**\n\nConvert `customer_id` and fill missing values **note that filling missing values might be questionable, e.g. NaN on FN means we don't know his status -> customer did not actively withdrawn from newsletter**","metadata":{}},{"cell_type":"code","source":"cust_df = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ncust_df['customer_id_int'] = cust_df['customer_id'].apply(lambda x: int(x[-16:],16) ).astype('int64') # needed since transaction also converted\ncust_df[['FN', 'Active']] = cust_df[['FN', 'Active']].fillna(0).astype(np.int8) # Replace NaNs in Newsletter and Active with 0\ncust_df[['club_member_status']] = cust_df[['club_member_status']].fillna('None')\ncust_df['fashion_news_frequency'] = cust_df['fashion_news_frequency'].replace('NONE', 'None')\ndel cust_df['customer_id']","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:05:57.253087Z","iopub.execute_input":"2022-03-20T00:05:57.253561Z","iopub.status.idle":"2022-03-20T00:06:03.501968Z","shell.execute_reply.started":"2022-03-20T00:05:57.253524Z","shell.execute_reply":"2022-03-20T00:06:03.501159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill missing age data\nmean_age = cust_df[\"age\"].mean() # TODO: use other attributes to fill age\ncust_df[\"age\"] = cust_df[\"age\"].fillna(mean_age).astype(np.int8) # age \"intable\"\nprint(f\"Customers age isna after replace with mean ({int(mean_age)}): {sum(cust_df.age.isna())}\")","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:03.503314Z","iopub.execute_input":"2022-03-20T00:06:03.503594Z","iopub.status.idle":"2022-03-20T00:06:03.673416Z","shell.execute_reply.started":"2022-03-20T00:06:03.503558Z","shell.execute_reply":"2022-03-20T00:06:03.672019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get first & last transaction of user, number of carts in transactions (if cold user ct_carts=0)\ncust_df = cust_df.merge(trans_df.to_pandas().groupby('customer_id_int')['t_dat'].agg(\n    t_dat_min='min', t_dat_max='max', ct_carts='nunique'), how=\"left\", left_on=\"customer_id_int\", right_index=True)\ncust_df['ct_carts'] = cust_df['ct_carts'].fillna(0).astype('int8')","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:03.674921Z","iopub.execute_input":"2022-03-20T00:06:03.675189Z","iopub.status.idle":"2022-03-20T00:06:12.835542Z","shell.execute_reply.started":"2022-03-20T00:06:03.675153Z","shell.execute_reply":"2022-03-20T00:06:12.834407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get maximum and mean price customer pays for an article\ncust_df = cust_df.merge(trans_df.to_pandas().groupby('customer_id_int')['price'].agg(price_min='min', price_mean='mean', price_max='max'), on='customer_id_int', how='left')\ncust_df = cust_df.merge(trans_df.to_pandas().groupby(['customer_id_int', 't_dat'])['price'].agg(cart_value='sum').groupby('customer_id_int')['cart_value'].agg(cart_value_min='min', cart_value_mean='mean', cart_value_max='max'), on='customer_id_int', how='left')","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:12.839411Z","iopub.execute_input":"2022-03-20T00:06:12.839702Z","iopub.status.idle":"2022-03-20T00:06:29.178492Z","shell.execute_reply.started":"2022-03-20T00:06:12.839665Z","shell.execute_reply":"2022-03-20T00:06:29.177554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Simplify postal code\npostal_code_map = {pc: pc_simple for pc_simple, pc in enumerate(cust_df.postal_code.unique().tolist())}\ncust_df['postal_code'] = cust_df['postal_code'].map(postal_code_map).astype('int32')","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:29.179956Z","iopub.execute_input":"2022-03-20T00:06:29.180350Z","iopub.status.idle":"2022-03-20T00:06:30.322572Z","shell.execute_reply.started":"2022-03-20T00:06:29.180310Z","shell.execute_reply":"2022-03-20T00:06:30.321738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:30.323744Z","iopub.execute_input":"2022-03-20T00:06:30.324013Z","iopub.status.idle":"2022-03-20T00:06:30.617343Z","shell.execute_reply.started":"2022-03-20T00:06:30.323966Z","shell.execute_reply":"2022-03-20T00:06:30.616545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Transactions Minimum/Maximum Article Price\nGet the rolling maximum & minimum price in transactions (exclude discounted member prices for minimum).","metadata":{}},{"cell_type":"code","source":"# TODO: merge cust_df to get club_member_status and exclude from price counting\ntrans_df['price_min'] = trans_df.groupby('article_id')['price'].cummin()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:30.618628Z","iopub.execute_input":"2022-03-20T00:06:30.618971Z","iopub.status.idle":"2022-03-20T00:06:30.800163Z","shell.execute_reply.started":"2022-03-20T00:06:30.618930Z","shell.execute_reply":"2022-03-20T00:06:30.799379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trans_df['price_max'] = trans_df.groupby('article_id')['price'].cummax()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:30.801493Z","iopub.execute_input":"2022-03-20T00:06:30.801847Z","iopub.status.idle":"2022-03-20T00:06:30.964210Z","shell.execute_reply.started":"2022-03-20T00:06:30.801807Z","shell.execute_reply":"2022-03-20T00:06:30.963385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Articles","metadata":{}},{"cell_type":"code","source":"art_df = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\nart_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:30.965414Z","iopub.execute_input":"2022-03-20T00:06:30.965750Z","iopub.status.idle":"2022-03-20T00:06:32.111610Z","shell.execute_reply.started":"2022-03-20T00:06:30.965711Z","shell.execute_reply":"2022-03-20T00:06:32.110805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"art_df['hist'] = art_df.article_id.isin(trans_df.article_id.to_pandas().unique()).astype(np.int8)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:32.113003Z","iopub.execute_input":"2022-03-20T00:06:32.113481Z","iopub.status.idle":"2022-03-20T00:06:32.371499Z","shell.execute_reply.started":"2022-03-20T00:06:32.113435Z","shell.execute_reply":"2022-03-20T00:06:32.370601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check if article has transaction history, i.e. does show up in transactions (0=cold item)\nart_df['hist'] = art_df.article_id.isin(trans_df.article_id.to_pandas().unique()).astype(np.int8)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:32.373215Z","iopub.execute_input":"2022-03-20T00:06:32.373645Z","iopub.status.idle":"2022-03-20T00:06:32.638416Z","shell.execute_reply.started":"2022-03-20T00:06:32.373579Z","shell.execute_reply":"2022-03-20T00:06:32.637327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"helper = trans_df.to_pandas().groupby('article_id')['t_dat'].agg(t_dat_min='min', t_dat_max='max')\nart_df = art_df.merge(helper, on=\"article_id\", how='left')\ndel helper","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:06:32.640848Z","iopub.execute_input":"2022-03-20T00:06:32.641328Z","iopub.status.idle":"2022-03-20T00:06:35.726172Z","shell.execute_reply.started":"2022-03-20T00:06:32.641284Z","shell.execute_reply":"2022-03-20T00:06:35.725391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Is Article Relevant Next Week?","metadata":{}},{"cell_type":"code","source":"weekly_unique_articles = weekly_sales.reset_index().groupby('ldbw')['article_id'].unique() # 105 rows\nweekly_unique_articles = pd.DataFrame(\n    list(zip(weekly_unique_articles.values, weekly_unique_articles.shift(-1).values)), \n    columns=['articles', 'articles_next'], index=weekly_unique_articles.index)\nweekly_unique_articles.loc['2020-09-22', 'articles_next'] = [] # last week\n\nweekly_unique_articles['res'] = np.empty((len(weekly_unique_articles), 0)).tolist()\nfor index, row in weekly_unique_articles.iterrows():\n    for a in row['articles']:\n        if a in row['articles_next']:\n            weekly_unique_articles.loc[index, 'res'].append((a, 1))\n        else:\n            weekly_unique_articles.loc[index, 'res'].append((a, 0))\n            \nweekly_unique_articles = weekly_unique_articles.apply(lambda x: pd.Series(x['res']),axis=1).stack().reset_index(level=1, drop=True)\nweekly_unique_articles = pd.DataFrame(list(weekly_unique_articles), \n                                      columns=['article_id', 'rel_next_week'],\n                                      index=weekly_unique_articles.index)\nweekly_sales = weekly_sales.merge(weekly_unique_articles, how='left', on=['article_id', 'ldbw'])\nweekly_sales['rel_next_week'] = weekly_sales['rel_next_week'].astype('int8')\nweekly_sales['count'] = weekly_sales['count'].astype('int16')\ndel weekly_unique_articles","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:28:58.505340Z","iopub.execute_input":"2022-03-20T00:28:58.506071Z","iopub.status.idle":"2022-03-20T00:28:58.685047Z","shell.execute_reply.started":"2022-03-20T00:28:58.506030Z","shell.execute_reply":"2022-03-20T00:28:58.684285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weekly_sales.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T00:38:35.680605Z","iopub.execute_input":"2022-03-20T00:38:35.681257Z","iopub.status.idle":"2022-03-20T00:38:35.691110Z","shell.execute_reply.started":"2022-03-20T00:38:35.681214Z","shell.execute_reply":"2022-03-20T00:38:35.690329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Helper Datasets\n\n**Customer/Article Interaction**","metadata":{}},{"cell_type":"code","source":"# Dataframe Index: (cust_id, art_id); Values: how often occured combination\nMIN_PURCHASES = 0 # Cap, ignore combinations where customer only bought article once (REMOVED)\n\ncust_art_df = trans_df.groupby(['customer_id_int', 'article_id'])['t_dat'].agg(['count', 'nunique', 'min', 'max'])\ncust_art_df.columns = ['ct', 'ct_carts', 't_dat_min', 't_dat_max']\ncust_art_df = cust_art_df[cust_art_df.ct>MIN_PURCHASES].sort_values(by=\"ct\", ascending=False).to_pandas()\ncust_art_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-16T19:19:33.473949Z","iopub.execute_input":"2022-03-16T19:19:33.474428Z","iopub.status.idle":"2022-03-16T19:19:36.255923Z","shell.execute_reply.started":"2022-03-16T19:19:33.474393Z","shell.execute_reply":"2022-03-16T19:19:36.255096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_art_df_h = cust_art_df.reset_index().merge(art_df[['article_id', 'index_group_name']], on='article_id', how='left')\ncust_art_df_h = pd.get_dummies(cust_art_df_h, columns=['index_group_name']).groupby(\n    'customer_id_int').sum().filter(regex=(\"index.*\"))\ncust_art_df_h = cust_art_df_h.filter(regex=(\"index.*\")).astype('int16')\n\ncust_art_df_h[\"helper_sum\"] = cust_art_df_h.sum(axis=1).astype('int16')\n\nfor col in ['index_group_name_Baby/Children', 'index_group_name_Divided',\n       'index_group_name_Ladieswear', 'index_group_name_Menswear',\n       'index_group_name_Sport']:\n    cust_art_df_h.loc[:, col] = (cust_art_df_h[col]/cust_art_df_h['helper_sum']).astype('float32')\n\ndel cust_art_df_h['helper_sum']\ncust_df = cust_df.merge(cust_art_df_h, on='customer_id_int', how='left')\n\ndel cust_art_df_h\n\ncust_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-16T19:24:11.537728Z","iopub.execute_input":"2022-03-16T19:24:11.537996Z","iopub.status.idle":"2022-03-16T19:24:27.673187Z","shell.execute_reply.started":"2022-03-16T19:24:11.537958Z","shell.execute_reply":"2022-03-16T19:24:27.671741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_df","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Store results","metadata":{}},{"cell_type":"code","source":"trans_df.to_parquet(f'{PREFIX}transactions.pqt', index=False)\nweekly_sales.to_parquet(f'{PREFIX}weekly_sales.pqt', index=False)\ncust_df.to_parquet(f\"{PREFIX}customers.pqt\", index=False)\nart_df.to_parquet(f\"{PREFIX}articles.pqt\", index=False)\ncust_art_df.to_parquet(f\"{PREFIX}customers_article_df.pqt\")","metadata":{"execution":{"iopub.status.busy":"2022-03-16T18:29:38.451786Z","iopub.execute_input":"2022-03-16T18:29:38.45225Z","iopub.status.idle":"2022-03-16T18:29:42.165537Z","shell.execute_reply.started":"2022-03-16T18:29:38.452209Z","shell.execute_reply":"2022-03-16T18:29:42.164795Z"},"trusted":true},"execution_count":null,"outputs":[]}]}