Many people would think that user-content-wise data (i.e., mean/sum/count of accuracy/time/explanation/lecture for each user_id and for each content_id) are very effective for prediction.
\nI currently handle these data with nested dict (user-wise dict of content-wise dict) like following.
for row in df:\n aaa_uc_dict[user_id][content_id] = aaa\n bbb_uc_dict[user_id][content_id] = bbb\n ccc_uc_dict[user_id][content_id] = ccc\nI have created 15 dict files of user-content-wise data with my local PC. The size of these dict files become more than 2GB/file, when I saved them in pickle format. I want to use these files in kaggle notebook for inference in prediction loop, but it would easily cause memory error.
\nDoes anyone have ideas for handling these data more efficiently, or I must select two or three parameters?
", "messages": [ { "id": "1110656", "postDate": "12/12/2020 23:42:18", "content": "Many people would think that user-content-wise data (i.e., mean/sum/count of accuracy/time/explanation/lecture for each user_id and for each content_id) are very effective for prediction.
\nI currently handle these data with nested dict (user-wise dict of content-wise dict) like following.
for row in df:\n aaa_uc_dict[user_id][content_id] = aaa\n bbb_uc_dict[user_id][content_id] = bbb\n ccc_uc_dict[user_id][content_id] = ccc\nI have created 15 dict files of user-content-wise data with my local PC. The size of these dict files become more than 2GB/file, when I saved them in pickle format. I want to use these files in kaggle notebook for inference in prediction loop, but it would easily cause memory error.
\nDoes anyone have ideas for handling these data more efficiently, or I must select two or three parameters?
", "rawMarkdown": "Many people would think that user-content-wise data (i.e., mean/sum/count of accuracy/time/explanation/lecture for each user_id and for each content_id) are very effective for prediction.\nI currently handle these data with nested dict (user-wise dict of content-wise dict) like following.\n\n```\nfor row in df:\n aaa_uc_dict[user_id][content_id] = aaa\n bbb_uc_dict[user_id][content_id] = bbb\n ccc_uc_dict[user_id][content_id] = ccc\n```\n\nI have created 15 dict files of user-content-wise data with my local PC. The size of these dict files become more than 2GB/file, when I saved them in pickle format. I want to use these files in kaggle notebook for inference in prediction loop, but it would easily cause memory error. \n\nDoes anyone have ideas for handling these data more efficiently, or I must select two or three parameters?", "votes": null }, { "id": "1111686", "postDate": "12/13/2020 23:13:55", "content": "user_id is int16 and if you use a copy of it in each dictionary it will consume the memory, try to use it once(make dictionary of dictionaries). for example you have a dictionary of users where each user have a dictionary of questions.
\nI used that in my kaggle notebook and it saved me huge amount of memory.
Thank you for your comment.
\nI have created double nested dictionaries (e.g., aaa_uc_dict[user_id][content_id]: user-wise dictionary of content-wise dictionaries for parameter aaa), but it cost me 2GB/file.
\nTriple nested dictionary (e.g., uc_dict[parameter_type][user_id][content_id]: parameter-wise dictionary of user-wise dictionaries of content-wise dictionaries) did not work well, as a size of single file became too huge to save and load.
I .
\nIndices of dict were assigned as np.int in my script. Transforming values into int reduced the size of dict into 1GB for float parameters and 400MB for int parameters.
for cnt row in df[\"user_id\", \"content_id\",\"aaa\"].values:\n user_id=int(row[0])\n content_id=int(row[1])\n aaa=int(row[2])\n aaa_uc_dict[user_id][content_id] = aaa\nAny other ideas are welcome.
", "rawMarkdown": "I ~~self-solved this problem partially~~. \nIndices of dict were assigned as np.int in my script. Transforming values into int reduced the size of dict into 1GB for float parameters and 400MB for int parameters.\n\n```\nfor cnt row in df[\"user_id\", \"content_id\",\"aaa\"].values:\n user_id=int(row[0])\n content_id=int(row[1])\n aaa=int(row[2])\n aaa_uc_dict[user_id][content_id] = aaa\n```\n\nAny other ideas are welcome.", "votes": null }, { "id": "1119876", "postDate": "12/20/2020 12:24:10", "content": "I noticed that my 428 MB dict file (total count of interactions for each user_id and each content_id) consumes 5.9 GB memory (measured with this great script by @higepon).
\nI am still struggling with this issue.
\nAny solutions or comments are highly welcome.
Yes not easy.. I think it should be something where not all the data has to be loaded into memory..
\nI tried:
\nThank you for your comment.
\nWould you explain more about your h5py data structure?
\nFor example, do you create an array of np.zeros(len(questions)) for each user at first?
The issue is that numbers inside dicts are store as int or float objects which consumes aboute 20+ bytes for each number (while in numpy it is maximum 4 bytes!). So if you have dict of dicts, then the number of bytes will be enormous.
\nActually I didn't find any solution for this problem, so I left the whole idea of creating dict of dicts !
\nIf you find any idea please let me know
Still in the process of figuring out the best approach.. but so far: first defined each user as a group member/\"key\" and then created an array (as dataset) for each user with content_id as the first column and the other variables in the other columns. Not sure yet how well it works..
", "rawMarkdown": "Still in the process of figuring out the best approach.. but so far: first defined each user as a group member/\"key\" and then created an array (as dataset) for each user with content_id as the first column and the other variables in the other columns. Not sure yet how well it works..", "votes": null }, { "id": "1121825", "postDate": "12/21/2020 23:59:41", "content": "I had a similar issue and ended up using sqlite3.
\nHere are what I'm doing :) Hope it helps.
I have a nested defaultdict such as foo[user_id][content_id] for feature engineering.
\nThis consumes a lot of memory but which is fine because I'm training on GCP. I just increased VM memory.
\nOnce training is done. I save the nested defaultdict into sqlite3 table.
\nThe table looks like below
CREATE TABLE foot(user_id INTEGER, content_id INTEGER, foo INTEGER, PRIMARY KEY (user_id, content_id))\nYou'll have .db file for the database.
\nBecause Kaggle kernel doesn't have enough memory to have the nested dic in memory I \"SELECT/INSERT/UPDATE\" the foo table whenever it's necessary for submission. The db is on disk so it doesn't consume much memory.
\nSomeone mentioned sqlite3 is slow which is true. But it's fast enough for submission.
\nIf you think sqlite3 is not fast enough you can take a look at the following great article.
\nhttps://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite
Thank you for providing valuable information and code!
\nI will try sqlite3.
I was finding sqlite working quite quickly. This book submits in under three hours with lookup and update of user content. I found that the insert wasn't the problem, it was the merging the selected data with the batch from the test iterator. The bit below, I think is what makes it fast, where you essentially bring the values from the batch into the select query and let sqlite do the merge and then deliver a complete batch ready for prediction, avoiding any pandas merge/join altogether.
\nrecords = df_batch[batch_cols].fillna(0).to_records(index=False)\n\ndef select_state(batch_cols, records):\n return f\"\"\"\n WITH b ({(', ').join(batch_cols)}) AS (\n VALUES {(', ').join(list(map(str, records)))}\n )\n SELECT\n {(', ').join([f'b.{col}' for col in batch_cols])},\n IFNULL(answered_correctly_cumsum, 0) answered_correctly_cumsum, \n...\n FROM b\n LEFT JOIN (\n SELECT user_id, answered_correctly answered_correctly_cumsum,\n answered_incorrectly answered_incorrectly_cumsum\n FROM users\n WHERE {(' OR ').join([f'user_id = {r[0]}' for r in records])}\n ) u ON (u.user_id = b.user_id)\n...\nThe other trick for the update was to use the on conflict clause to update existing values or insert new ones.
", "rawMarkdown": "I was finding sqlite working quite quickly. [This book](https://www.kaggle.com/calebeverett/riiid-submit) submits in under three hours with lookup and update of user content. I found that the insert wasn't the problem, it was the merging the selected data with the batch from the test iterator. The bit below, I think is what makes it fast, where you essentially bring the values from the batch into the select query and let sqlite do the merge and then deliver a complete batch ready for prediction, avoiding any pandas merge/join altogether.\n\n```\nrecords = df_batch[batch_cols].fillna(0).to_records(index=False)\n\ndef select_state(batch_cols, records):\n return f\"\"\"\n WITH b ({(', ').join(batch_cols)}) AS (\n VALUES {(', ').join(list(map(str, records)))}\n )\n SELECT\n {(', ').join([f'b.{col}' for col in batch_cols])},\n IFNULL(answered_correctly_cumsum, 0) answered_correctly_cumsum, \n...\n FROM b\n LEFT JOIN (\n SELECT user_id, answered_correctly answered_correctly_cumsum,\n answered_incorrectly answered_incorrectly_cumsum\n FROM users\n WHERE {(' OR ').join([f'user_id = {r[0]}' for r in records])}\n ) u ON (u.user_id = b.user_id)\n...\n```\n\nThe other trick for the update was to use the on conflict clause to update existing values or insert new ones.", "votes": null }, { "id": "1124519", "postDate": "12/24/2020 02:40:00", "content": "Thank you @calebeverett.
\nI have read your notebooks already, but did not notice its potential (because I could not understand bigquery or sqlite). Now I realize how helpful it is. I will learn many techniques in your notebook.
That would be great if you found something useful there. I think you could load your dataframes into sqlite for the predictions from your current feature engineering methodology without having to spend time understanding the other big query book.
", "rawMarkdown": "That would be great if you found something useful there. I think you could load your dataframes into sqlite for the predictions from your current feature engineering methodology without having to spend time understanding the other big query book.", "votes": null }, { "id": "1125493", "postDate": "12/24/2020 18:53:40", "content": "If you're not afraid of having a bit of noise, there's a way to store data in a restricted amount of memory.
\nChoose a big Dimension value:
\nFor instance D=20_000_000
\nThen create a list or a np.array:
\nMyList=[0]*D
\nMyArray=np.zeros(D)
\nThat's all the amount of memory you'll need.
Then you can store anything inside with a chance of collision accessing items like this :
\nMyArray[abs(hash('any generated key')) % D]
This way you don't have to store index data, and you'll never use more memory than you have.
\nThe problems are :
@higepon @calebeverett Hi! Can you tell me please, if you load .db file to Kaggle dataset? I have an error, when try to do in this way
", "rawMarkdown": "higepon @calebeverett Hi! Can you tell me please, if you load .db file to Kaggle dataset? I have an error, when try to do in this way", "votes": null }, { "id": "1128953", "postDate": "12/27/2020 23:22:38", "content": "The following is working for me. Note that I'm copying the db file to current directory which is required when you write.
\ndata_dir = Path('/kaggle/input/higeponriiid2')\n!cp /kaggle/input/higeponriiid2/riiid.db riiid.db\nconn = sqlite3.connect('riiid.db')\nHope it helps.
", "rawMarkdown": "The following is working for me. Note that I'm copying the db file to current directory which is required when you write.\n\n```\ndata_dir = Path('/kaggle/input/higeponriiid2')\n!cp /kaggle/input/higeponriiid2/riiid.db riiid.db\nconn = sqlite3.connect('riiid.db')\n```\n\nHope it helps.", "votes": null }, { "id": "1128954", "postDate": "12/27/2020 23:24:42", "content": "Forgot to mention but I upload the db to kaggle dataset which is higeponriid2 above.
", "rawMarkdown": "Forgot to mention but I upload the db to kaggle dataset which is higeponriid2 above.", "votes": null }, { "id": "1128955", "postDate": "12/27/2020 23:27:30", "content": "Thank you @higepon
\nI’ll try it!
I was using data frames and loading to db in submit book. They are in the dataset attached to the submit book. The database should be available in the book after you run.
", "rawMarkdown": "I was using data frames and loading to db in submit book. They are in the dataset attached to the submit book. The database should be available in the book after you run.", "votes": null }, { "id": "1129127", "postDate": "12/28/2020 05:23:00", "content": "Try using a dataframe that shares an index. Both speed and memory are fine.
\nIf there is one key, a dict is used, and if there are two keys, a dataframe is used.
Full dataset
\nmemory: ~7gb
\ntime: ~4hour
Thank you @calebeverett
", "rawMarkdown": "Thank you @calebeverett", "votes": null } ], "comments": [ { "id": 1111686, "author_name": "mohamadnawfal", "author_url": "", "post_date": "12/13/2020 23:13:55", "content": "user_id is int16 and if you use a copy of it in each dictionary it will consume the memory, try to use it once(make dictionary of dictionaries). for example you have a dictionary of users where each user have a dictionary of questions.
\nI used that in my kaggle notebook and it saved me huge amount of memory.
Thank you for your comment.
\nI have created double nested dictionaries (e.g., aaa_uc_dict[user_id][content_id]: user-wise dictionary of content-wise dictionaries for parameter aaa), but it cost me 2GB/file.
\nTriple nested dictionary (e.g., uc_dict[parameter_type][user_id][content_id]: parameter-wise dictionary of user-wise dictionaries of content-wise dictionaries) did not work well, as a size of single file became too huge to save and load.
I .
\nIndices of dict were assigned as np.int in my script. Transforming values into int reduced the size of dict into 1GB for float parameters and 400MB for int parameters.
for cnt row in df[\"user_id\", \"content_id\",\"aaa\"].values:\n user_id=int(row[0])\n content_id=int(row[1])\n aaa=int(row[2])\n aaa_uc_dict[user_id][content_id] = aaa\nAny other ideas are welcome.
", "votes": null, "replies": [ { "id": 1119876, "author_name": "tomooinubushi", "author_url": "", "post_date": "12/20/2020 12:24:10", "content": "I noticed that my 428 MB dict file (total count of interactions for each user_id and each content_id) consumes 5.9 GB memory (measured with this great script by @higepon).
\nI am still struggling with this issue.
\nAny solutions or comments are highly welcome.
The issue is that numbers inside dicts are store as int or float objects which consumes aboute 20+ bytes for each number (while in numpy it is maximum 4 bytes!). So if you have dict of dicts, then the number of bytes will be enormous.
\nActually I didn't find any solution for this problem, so I left the whole idea of creating dict of dicts !
\nIf you find any idea please let me know
I had a similar issue and ended up using sqlite3.
\nHere are what I'm doing :) Hope it helps.
I have a nested defaultdict such as foo[user_id][content_id] for feature engineering.
\nThis consumes a lot of memory but which is fine because I'm training on GCP. I just increased VM memory.
\nOnce training is done. I save the nested defaultdict into sqlite3 table.
\nThe table looks like below
CREATE TABLE foot(user_id INTEGER, content_id INTEGER, foo INTEGER, PRIMARY KEY (user_id, content_id))\nYou'll have .db file for the database.
\nBecause Kaggle kernel doesn't have enough memory to have the nested dic in memory I \"SELECT/INSERT/UPDATE\" the foo table whenever it's necessary for submission. The db is on disk so it doesn't consume much memory.
\nSomeone mentioned sqlite3 is slow which is true. But it's fast enough for submission.
\nIf you think sqlite3 is not fast enough you can take a look at the following great article.
\nhttps://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite
Thank you for providing valuable information and code!
\nI will try sqlite3.
I was finding sqlite working quite quickly. This book submits in under three hours with lookup and update of user content. I found that the insert wasn't the problem, it was the merging the selected data with the batch from the test iterator. The bit below, I think is what makes it fast, where you essentially bring the values from the batch into the select query and let sqlite do the merge and then deliver a complete batch ready for prediction, avoiding any pandas merge/join altogether.
\nrecords = df_batch[batch_cols].fillna(0).to_records(index=False)\n\ndef select_state(batch_cols, records):\n return f\"\"\"\n WITH b ({(', ').join(batch_cols)}) AS (\n VALUES {(', ').join(list(map(str, records)))}\n )\n SELECT\n {(', ').join([f'b.{col}' for col in batch_cols])},\n IFNULL(answered_correctly_cumsum, 0) answered_correctly_cumsum, \n...\n FROM b\n LEFT JOIN (\n SELECT user_id, answered_correctly answered_correctly_cumsum,\n answered_incorrectly answered_incorrectly_cumsum\n FROM users\n WHERE {(' OR ').join([f'user_id = {r[0]}' for r in records])}\n ) u ON (u.user_id = b.user_id)\n...\nThe other trick for the update was to use the on conflict clause to update existing values or insert new ones.
", "votes": null, "replies": [] }, { "id": 1124519, "author_name": "tomooinubushi", "author_url": "", "post_date": "12/24/2020 02:40:00", "content": "Thank you @calebeverett.
\nI have read your notebooks already, but did not notice its potential (because I could not understand bigquery or sqlite). Now I realize how helpful it is. I will learn many techniques in your notebook.
That would be great if you found something useful there. I think you could load your dataframes into sqlite for the predictions from your current feature engineering methodology without having to spend time understanding the other big query book.
", "votes": null, "replies": [] }, { "id": 1128936, "author_name": "fredegrec", "author_url": "", "post_date": "12/27/2020 22:42:10", "content": "@higepon @calebeverett Hi! Can you tell me please, if you load .db file to Kaggle dataset? I have an error, when try to do in this way
", "votes": null, "replies": [] }, { "id": 1128953, "author_name": "higepon", "author_url": "", "post_date": "12/27/2020 23:22:38", "content": "The following is working for me. Note that I'm copying the db file to current directory which is required when you write.
\ndata_dir = Path('/kaggle/input/higeponriiid2')\n!cp /kaggle/input/higeponriiid2/riiid.db riiid.db\nconn = sqlite3.connect('riiid.db')\nHope it helps.
", "votes": null, "replies": [] }, { "id": 1128954, "author_name": "higepon", "author_url": "", "post_date": "12/27/2020 23:24:42", "content": "Forgot to mention but I upload the db to kaggle dataset which is higeponriid2 above.
", "votes": null, "replies": [] }, { "id": 1128955, "author_name": "fredegrec", "author_url": "", "post_date": "12/27/2020 23:27:30", "content": "Thank you @higepon
\nI’ll try it!
I was using data frames and loading to db in submit book. They are in the dataset attached to the submit book. The database should be available in the book after you run.
", "votes": null, "replies": [] }, { "id": 1129629, "author_name": "fredegrec", "author_url": "", "post_date": "12/28/2020 13:26:31", "content": "Thank you @calebeverett
", "votes": null, "replies": [] } ] }, { "id": 1120202, "author_name": "hannes82", "author_url": "", "post_date": "12/20/2020 16:57:56", "content": "Yes not easy.. I think it should be something where not all the data has to be loaded into memory..
\nI tried:
\nThank you for your comment.
\nWould you explain more about your h5py data structure?
\nFor example, do you create an array of np.zeros(len(questions)) for each user at first?
Still in the process of figuring out the best approach.. but so far: first defined each user as a group member/\"key\" and then created an array (as dataset) for each user with content_id as the first column and the other variables in the other columns. Not sure yet how well it works..
", "votes": null, "replies": [] } ] }, { "id": 1125493, "author_name": "trottefox", "author_url": "", "post_date": "12/24/2020 18:53:40", "content": "If you're not afraid of having a bit of noise, there's a way to store data in a restricted amount of memory.
\nChoose a big Dimension value:
\nFor instance D=20_000_000
\nThen create a list or a np.array:
\nMyList=[0]*D
\nMyArray=np.zeros(D)
\nThat's all the amount of memory you'll need.
Then you can store anything inside with a chance of collision accessing items like this :
\nMyArray[abs(hash('any generated key')) % D]
This way you don't have to store index data, and you'll never use more memory than you have.
\nThe problems are :
Try using a dataframe that shares an index. Both speed and memory are fine.
\nIf there is one key, a dict is used, and if there are two keys, a dataframe is used.
Full dataset
\nmemory: ~7gb
\ntime: ~4hour