{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M:CV_Tutorial_FasterTrendingProductsWeekly[JP/EN]","metadata":{}},{"cell_type":"markdown","source":"ノートブックを見ていただきありがとうございます。　　\n\nこのノートブックでは、[FasterTrendingProductsWeekly](https://www.kaggle.com/code/hervind/h-m-faster-trending-products-weekly/notebook)の予測モデルを3つのCV(交差検証)によって結果の比較を行います。\n\n- fold 0\n\n    train <= '2020-09-15'\n    \n    valid >= '2020-09-16'\n    \n- fold 1\n\n    train <= '2020-09-08'\n    \n    valid >= '2020-09-09' & valid <= '2020-09-15' \n\n- fold 2\n\n    train <= '2020-09-01'\n    \n    valid >= '2020-09-02' & valid <= '2020-09-08' \n    \nReference:[How To Setup Local CV](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308919)\n\n\nまた、予測モデルを関数に置き換えているので使い易くしています。\n\nーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーー\n\nThank you for taking a look at the notebook. It was\n\nThis notebook compares the results of a predictive model from with three cross-validations.\n\nIt is also easy to use because it replaces the predictive model with a function.\n\nーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーーー\n\nIf this notebook helps you, please comment and up votes.:)","metadata":{}},{"cell_type":"markdown","source":"## 1. Library\nこのノートブックではGPU（cudf）を使います。\n\nThis notebook uses GPU(cudf).","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport gc\nimport os\nimport time\nimport random\nfrom tqdm.auto import tqdm\nimport cudf","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:36:55.999082Z","iopub.execute_input":"2022-04-07T13:36:55.999943Z","iopub.status.idle":"2022-04-07T13:36:56.003963Z","shell.execute_reply.started":"2022-04-07T13:36:55.999898Z","shell.execute_reply":"2022-04-07T13:36:56.003258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_df(df):\n    print(df.shape)\n    display(df.head())","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:16:20.988621Z","iopub.execute_input":"2022-04-07T13:16:20.989195Z","iopub.status.idle":"2022-04-07T13:16:20.993694Z","shell.execute_reply.started":"2022-04-07T13:16:20.989165Z","shell.execute_reply":"2022-04-07T13:16:20.992889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Function","metadata":{}},{"cell_type":"markdown","source":"- Reference\n\n[H&M : How to calculate MAP@12](https://www.kaggle.com/code/kaerunantoka/h-m-how-to-calculate-map-12)","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007\n# https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n\n\ndef apk(actual, predicted, k=10):\n    \"\"\"\n    Computes the average precision at k.\n    This function computes the average prescision at k between two lists of\n    items.\n    Parameters\n    ----------\n    actual : list\n             A list of elements that are to be predicted (order doesn't matter)\n    predicted : list\n                A list of predicted elements (order does matter)\n    k : int, optional\n        The maximum number of predicted elements\n    Returns\n    -------\n    score : double\n            The average precision at k over the input lists\n    \"\"\"\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    # remove this case in advance\n    # if not actual:\n    #     return 0.0\n\n    return score / min(len(actual), k)\n\n\ndef mapk(actual, predicted, k=10):\n    \"\"\"\n    Computes the mean average precision at k.\n    This function computes the mean average prescision at k between two lists\n    of lists of items.\n    Parameters\n    ----------\n    actual : list\n             A list of lists of elements that are to be predicted \n             (order doesn't matter in the lists)\n    predicted : list\n                A list of lists of predicted elements\n                (order matters in the lists)\n    k : int, optional\n        The maximum number of predicted elements\n    Returns\n    -------\n    score : double\n            The mean average precision at k over the input lists\n    \"\"\"\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])\n\ndef train_valid_split(transactions, nfold=0):\n\n    if nfold==1:\n        train_end_date = '2020-09-08'\n        val_start_date,val_end_date = '2020-09-09', '2020-09-15'\n    elif nfold==2:\n        train_end_date = '2020-09-01'\n        val_start_date,val_end_date = '2020-09-02', '2020-09-08'\n    else:\n        train_end_date = '2020-09-15'\n        val_start_date,val_end_date = '2020-09-16', '2020-09-22'\n    \n    print(f'TrainEndDate:{train_end_date}')\n    print(f'ValidDate:{val_start_date} to {val_end_date}')\n    \n    train_data = transactions.query(f\"t_dat <= '{train_end_date}'\").reset_index(drop=True)\n    valid_data = transactions.query(f\"t_dat >= '{val_start_date}' & t_dat <='{val_end_date}'\").reset_index(drop=True)\n        \n    visualize_df(train_data)\n    visualize_df(valid_data)\n    \n    return train_data, valid_data\n        \n    \n\n    ","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:16:21.08753Z","iopub.execute_input":"2022-04-07T13:16:21.08798Z","iopub.status.idle":"2022-04-07T13:16:21.100883Z","shell.execute_reply.started":"2022-04-07T13:16:21.087949Z","shell.execute_reply":"2022-04-07T13:16:21.100143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Make Dataset","metadata":{}},{"cell_type":"code","source":"!mkdir nfold","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:16:21.171835Z","iopub.execute_input":"2022-04-07T13:16:21.172033Z","iopub.status.idle":"2022-04-07T13:16:22.194193Z","shell.execute_reply.started":"2022-04-07T13:16:21.172009Z","shell.execute_reply":"2022-04-07T13:16:22.193304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\nvisualize_df(transactions)","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:16:22.196311Z","iopub.execute_input":"2022-04-07T13:16:22.196539Z","iopub.status.idle":"2022-04-07T13:17:22.217653Z","shell.execute_reply.started":"2022-04-07T13:16:22.196512Z","shell.execute_reply":"2022-04-07T13:17:22.216886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"folds=[0, 1, 2]\n\nfor nfold in folds:\n    train_data, valid_data = train_valid_split(transactions, nfold=nfold)\n    train_data.to_csv(f'nfold/train_fold{str(nfold)}.csv', index=False)\n    valid_data.to_csv(f'nfold/valid_fold{str(nfold)}.csv', index=False)\n    \n    del train_data, valid_data\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:21:27.009753Z","iopub.execute_input":"2022-04-07T13:21:27.01003Z","iopub.status.idle":"2022-04-07T13:24:53.156256Z","shell.execute_reply.started":"2022-04-07T13:21:27.009998Z","shell.execute_reply":"2022-04-07T13:24:53.15499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Prepare Velidation Data","metadata":{}},{"cell_type":"code","source":"nfold = 0\n\nsub = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv', usecols=['customer_id'])\nvalid_data = pd.read_csv(f'./nfold/valid_fold{str(nfold)}.csv')\n\nvalid_unq = valid_data.groupby('customer_id')['article_id'].apply(list).reset_index()\nvalid_unq['valid_true'] = valid_unq['article_id'].map(lambda x: '0'+' 0'.join(str(x)[1:-1].split(', ')))\n\ndf_mapk = pd.merge(sub, valid_unq[['customer_id', 'valid_true']], on=['customer_id'])\nvisualize_df(df_mapk)","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:27:53.804651Z","iopub.execute_input":"2022-04-07T13:27:53.805253Z","iopub.status.idle":"2022-04-07T13:27:58.718838Z","shell.execute_reply.started":"2022-04-07T13:27:53.805202Z","shell.execute_reply":"2022-04-07T13:27:58.71807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.Prediction","metadata":{}},{"cell_type":"markdown","source":"- Reference\n[H&M: Faster Trending Products Weekly](https://www.kaggle.com/code/hervind/h-m-faster-trending-products-weekly/notebook)","metadata":{}},{"cell_type":"code","source":"def pred_weekly(trainpath, N=12):\n    df = cudf.read_csv(trainpath,\n             usecols=['t_dat', 'customer_id', 'article_id'],\n             dtype={'article_id':'int32', 't_dat':'string', 'customer_id':'string'})\n    \n    df['t_dat'] = cudf.to_datetime(df['t_dat'])\n    df ['customer_id'] = df ['customer_id'].str[-16:].str.hex_to_int().astype('int64')\n    \n    last_ts = df['t_dat'].max()\n    \n    tmp = df[['t_dat']].copy().to_pandas()\n    tmp['dow'] = tmp['t_dat'].dt.dayofweek\n    tmp['ldbw'] = tmp['t_dat'] - pd.TimedeltaIndex(tmp['dow'] - 1, unit='D')\n    tmp.loc[tmp['dow'] >=2 , 'ldbw'] = tmp.loc[tmp['dow'] >=2 , 'ldbw'] + pd.TimedeltaIndex(np.ones(len(tmp.loc[tmp['dow'] >=2])) * 7, unit='D')\n\n    df['ldbw'] = tmp['ldbw'].values\n    \n    \n    weekly_sales = df.drop('customer_id', axis=1).groupby(['ldbw', 'article_id']).count().reset_index()\n    weekly_sales = weekly_sales.rename(columns={'t_dat': 'count'})\n    \n    df = df.merge(weekly_sales, on=['ldbw', 'article_id'], how = 'left')\n    \n    \n    weekly_sales = weekly_sales.reset_index().set_index('article_id')\n\n    df = df.merge(\n        weekly_sales.loc[weekly_sales['ldbw']==last_ts, ['count']],\n        on='article_id', suffixes=(\"\", \"_targ\"))\n\n    df['count_targ'].fillna(0, inplace=True)\n    del weekly_sales\n    \n    df['quotient'] = df['count_targ'] / df['count']\n    \n    target_sales = df.drop('customer_id', axis=1).groupby('article_id')['quotient'].sum()\n    general_pred = target_sales.nlargest(N).index.to_pandas().tolist()\n    general_pred = ['0' + str(article_id) for article_id in general_pred]\n    general_pred_str =  ' '.join(general_pred)\n    del target_sales\n    \n\n    purchase_dict = {}\n\n    tmp = df.copy().to_pandas()\n    tmp['x'] = ((last_ts - tmp['t_dat']) / np.timedelta64(1, 'D')).astype(int)\n    tmp['dummy_1'] = 1 \n    tmp['x'] = tmp[[\"x\", \"dummy_1\"]].max(axis=1)\n\n    a, b, c, d = 2.5e4, 1.5e5, 2e-1, 1e3\n    tmp['y'] = a / np.sqrt(tmp['x']) + b * np.exp(-c*tmp['x']) - d\n\n    tmp['dummy_0'] = 0 \n    tmp['y'] = tmp[[\"y\", \"dummy_0\"]].max(axis=1)\n    tmp['value'] = tmp['quotient'] * tmp['y'] \n\n    tmp = tmp.groupby(['customer_id', 'article_id']).agg({'value': 'sum'})\n    tmp = tmp.reset_index()\n\n    tmp = tmp.loc[tmp['value'] > 100]\n    tmp['rank'] = tmp.groupby(\"customer_id\")[\"value\"].rank(\"dense\", ascending=False)\n    tmp = tmp.loc[tmp['rank'] <= 12]\n\n    # for customer_id in tmp['customer_id'].unique():\n    #     purchase_dict[customer_id] = {} \n\n    # for customer_id, article_id, value in zip(tmp['customer_id'], tmp['article_id'], tmp['value']):\n    #     purchase_dict[customer_id][article_id] = value\n\n    purchase_df = tmp.sort_values(['customer_id', 'value'], ascending = False).reset_index(drop = True)\n    purchase_df['prediction'] = '0' + purchase_df['article_id'].astype(str) + ' '\n    purchase_df = purchase_df.groupby('customer_id').agg({'prediction': sum}).reset_index()\n    purchase_df['prediction'] = purchase_df['prediction'].str.strip()\n    purchase_df = cudf.DataFrame(purchase_df)\n    \n    \n    \n    sub  = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv',\n                            usecols= ['customer_id'], \n                            dtype={'customer_id': 'string'})\n\n    sub['customer_id2'] = sub['customer_id'].str[-16:].str.hex_to_int().astype('int64')\n\n    sub = sub.merge(purchase_df, left_on = 'customer_id2', right_on = 'customer_id', how = 'left', suffixes = ('', '_ignored'))\n\n    sub = sub.to_pandas()\n    sub['prediction'] = sub['prediction'].fillna(general_pred_str)\n    sub['prediction'] = sub['prediction'] + ' ' +  general_pred_str\n    sub['prediction'] = sub['prediction'].str.strip()\n    sub['prediction'] = sub['prediction'].str[:131]\n    sub = sub[['customer_id', 'prediction']]\n    \n    return sub","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-07T13:27:58.721958Z","iopub.execute_input":"2022-04-07T13:27:58.722169Z","iopub.status.idle":"2022-04-07T13:27:58.743989Z","shell.execute_reply.started":"2022-04-07T13:27:58.722143Z","shell.execute_reply":"2022-04-07T13:27:58.743278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainpath = f'./nfold/train_fold{str(nfold)}.csv'\npred_df = pred_weekly(trainpath=trainpath)","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:27:58.745015Z","iopub.execute_input":"2022-04-07T13:27:58.74542Z","iopub.status.idle":"2022-04-07T13:28:38.379792Z","shell.execute_reply.started":"2022-04-07T13:27:58.74538Z","shell.execute_reply":"2022-04-07T13:28:38.378911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_mapk = pd.merge(df_mapk, pred_df, on=['customer_id'], how='left')\nvisualize_df(df_mapk)","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:28:38.383838Z","iopub.execute_input":"2022-04-07T13:28:38.384144Z","iopub.status.idle":"2022-04-07T13:28:39.207687Z","shell.execute_reply.started":"2022-04-07T13:28:38.38408Z","shell.execute_reply":"2022-04-07T13:28:39.206956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tqdm.pandas()\n\nmapk(\n    df_mapk['valid_true'].map(lambda x: x.split()), \n    df_mapk['prediction'].map(lambda x: x.split()), \n    k=12\n)","metadata":{"execution":{"iopub.status.busy":"2022-04-07T13:28:39.209132Z","iopub.execute_input":"2022-04-07T13:28:39.209393Z","iopub.status.idle":"2022-04-07T13:28:39.852964Z","shell.execute_reply.started":"2022-04-07T13:28:39.209357Z","shell.execute_reply":"2022-04-07T13:28:39.852154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.Conclusion","metadata":{}},{"cell_type":"markdown","source":"- Result(CV)\n||fold0|fold1|fold2|\n|:-:|:-:|:-:|:-:|\n|CV|0.0236|0.0233|0.0224|","metadata":{}}]}