I have seen many calling to ditch Pandas, I thought I'd make it official in a discussion.
\nMy submission time took almost 9 hours, ditching Pandas made scoring take only 20 minutes. Through my findings, for small batches (<30 Rows), looping on numpy arrays and dealing with each row independently is way faster than trying to vectorize the operations. Maybe it'd come obvious for some, but it wasn't obvious for me.
", "messages": [ { "id": "1072792", "postDate": "11/08/2020 17:29:00", "content": "I have seen many calling to ditch Pandas, I thought I'd make it official in a discussion.
\nMy submission time took almost 9 hours, ditching Pandas made scoring take only 20 minutes. Through my findings, for small batches (<30 Rows), looping on numpy arrays and dealing with each row independently is way faster than trying to vectorize the operations. Maybe it'd come obvious for some, but it wasn't obvious for me.
", "rawMarkdown": "I have seen many calling to ditch Pandas, I thought I'd make it official in a discussion.\n\nMy submission time took almost 9 hours, ditching Pandas made scoring take only 20 minutes. Through my findings, for small batches (<30 Rows), looping on numpy arrays and dealing with each row independently is way faster than trying to vectorize the operations. Maybe it'd come obvious for some, but it wasn't obvious for me.", "votes": null }, { "id": "1073751", "postDate": "11/09/2020 22:17:24", "content": "How do you do the merging? You check every key you would need in a pd.merge for every row?
", "rawMarkdown": "How do you do the merging? You check every key you would need in a pd.merge for every row?", "votes": null }, { "id": "1073776", "postDate": "11/09/2020 22:50:13", "content": "I stored the data that I used to use pd.merge on inside python dictionaries, thus lookup time is constant. So while looping, I just retrieve data from those dictionaries and use them to populate my input array.
", "rawMarkdown": "I stored the data that I used to use pd.merge on inside python dictionaries, thus lookup time is constant. So while looping, I just retrieve data from those dictionaries and use them to populate my input array.", "votes": null }, { "id": "1074761", "postDate": "11/11/2020 03:02:45", "content": "9h -> 20min? That's amazing!
\nChange DataFrame to ndarray, loop a dict, concat by hand, and you saved 8h?!
\nMan, that's unbelievable
\nI would like to try
In terms of feature engineering, do you still use pandas?
", "rawMarkdown": "In terms of feature engineering, do you still use pandas?", "votes": null }, { "id": "1075316", "postDate": "11/11/2020 15:19:36", "content": "Do you mean something like a dictionary with user_id as keys and an array with all the features as value? And then a loop over each line of the test_df and concat? I still can't see it. Could you share a couple of lines of code to see how this is done?
", "rawMarkdown": "Do you mean something like a dictionary with user_id as keys and an array with all the features as value? And then a loop over each line of the test_df and concat? I still can't see it. Could you share a couple of lines of code to see how this is done?", "votes": null }, { "id": "1075359", "postDate": "11/11/2020 16:31:34", "content": "The offline features yes, but during inference no.
", "rawMarkdown": "The offline features yes, but during inference no.", "votes": null }, { "id": "1075371", "postDate": "11/11/2020 16:44:45", "content": "It's pretty straightforward. As you said, I have a dictionary with user_id as a key with dictionaries with their features as values; although I have more than three dictionaries of that sort.
\nthis maybe true for small datasets (I haven't tried myself), but for larger datasets we may need new dataframe handling library which is faster than pandas?? Tell if you know any??
", "rawMarkdown": "this maybe true for small datasets (I haven't tried myself), but for larger datasets we may need new dataframe handling library which is faster than pandas?? Tell if you know any??", "votes": null }, { "id": "1075452", "postDate": "11/11/2020 18:25:07", "content": "I guess during the inference phase the dataset are quite small.
", "rawMarkdown": "I guess during the inference phase the dataset are quite small.", "votes": null }, { "id": "1075482", "postDate": "11/11/2020 18:40:43", "content": "You can use CUDF from Rapids, it uses GPU, speeds up alot
", "rawMarkdown": "You can use CUDF from Rapids, it uses GPU, speeds up alot", "votes": null }, { "id": "1075752", "postDate": "11/11/2020 23:43:58", "content": "xarray might be worth a try.
", "rawMarkdown": "xarray might be worth a try.", "votes": null }, { "id": "1075770", "postDate": "11/12/2020 00:21:19", "content": "Seems interesting, does kaggle support it? Or is there a offline dataset to add in the kernel?
", "rawMarkdown": "Seems interesting, does kaggle support it? Or is there a offline dataset to add in the kernel?", "votes": null }, { "id": "1077110", "postDate": "11/13/2020 08:46:16", "content": "I applied same idea, but got small improvement in the execution time
\nfrom 3:00 hrs ---> it became 2:30 hours (0:30 minutes improvement).
\nbelow is how I implemented it:
env = riiideducation.make_env()\niter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n np_test_df = test_df[['user_id', 'content_id']].to_numpy()\n X_submit = np.empty((len(np_test_df), len(model_cols)))\n for i in np.arange(len(X_submit)):\n X_submit[i][0] = users_dict[np_test_df[i][0]]\n X_submit[i][1] = questions_dict[np_test_df[i][1]]\n test_df['answered_correctly'] = model.predict(X_submit)\n env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n@jpmiller
\nI did quick search about xarray, and found below article says that xarray is slow when dealing with small chunks of data. what is your opinion about that?
\nhttps://github.com/pydata/xarray/issues/2799
Nice find. I thought that xarray was speedy but clearly not for these functions. In that case it sounds like Numpy is the way to go. Plain old python dictionaries might also work in some cases.
", "rawMarkdown": "Nice find. I thought that xarray was speedy but clearly not for these functions. In that case it sounds like Numpy is the way to go. Plain old python dictionaries might also work in some cases.", "votes": null }, { "id": "1078414", "postDate": "11/14/2020 18:42:54", "content": "Pandas is great for exploring the dataset as well but for operations numpy is better because its datastructure (arrays) are optmized for compute.
", "rawMarkdown": "Pandas is great for exploring the dataset as well but for operations numpy is better because its datastructure (arrays) are optmized for compute.", "votes": null }, { "id": "1105323", "postDate": "12/07/2020 19:23:38", "content": "Hi @abdessalemboukil, I wanted to ask, if I'll be using numpy array everywhere instead of dataframe, then how will I be able to keep track of the column name, especially while merging the data of different dataframes, as I require the name of the column on which I can perform a merge operation.
", "rawMarkdown": "Hi @abdessalemboukil, I wanted to ask, if I'll be using numpy array everywhere instead of dataframe, then how will I be able to keep track of the column name, especially while merging the data of different dataframes, as I require the name of the column on which I can perform a merge operation.", "votes": null }, { "id": "1105671", "postDate": "12/08/2020 04:32:51", "content": "you don't need names in a matrix, keep track of column index rather.
", "rawMarkdown": "you don't need names in a matrix, keep track of column index rather.", "votes": null }, { "id": "1140847", "postDate": "01/06/2021 10:10:50", "content": "If my submission does not exceed 9h limit in the current public leaderboard, can it be sure that it will not exceed the 9h limit later in the private leaderboard? Thanks.
", "rawMarkdown": "If my submission does not exceed 9h limit in the current public leaderboard, can it be sure that it will not exceed the 9h limit later in the private leaderboard? Thanks.", "votes": null }, { "id": "1140884", "postDate": "01/06/2021 10:46:58", "content": "Your score on the private leaderboard is computed at the same time as your public score, so you have nothing to worry about if your submission already succeeded.
", "rawMarkdown": "Your score on the private leaderboard is computed at the same time as your public score, so you have nothing to worry about if your submission already succeeded.", "votes": null }, { "id": "1141099", "postDate": "01/06/2021 13:53:17", "content": "Thank you very much
", "rawMarkdown": "Thank you very much", "votes": null } ], "comments": [ { "id": 1073751, "author_name": "iuryck", "author_url": "", "post_date": "11/09/2020 22:17:24", "content": "How do you do the merging? You check every key you would need in a pd.merge for every row?
", "votes": null, "replies": [ { "id": 1073776, "author_name": "abdessalemboukil", "author_url": "", "post_date": "11/09/2020 22:50:13", "content": "I stored the data that I used to use pd.merge on inside python dictionaries, thus lookup time is constant. So while looping, I just retrieve data from those dictionaries and use them to populate my input array.
", "votes": null, "replies": [] }, { "id": 1075316, "author_name": "jcesquiveld", "author_url": "", "post_date": "11/11/2020 15:19:36", "content": "Do you mean something like a dictionary with user_id as keys and an array with all the features as value? And then a loop over each line of the test_df and concat? I still can't see it. Could you share a couple of lines of code to see how this is done?
", "votes": null, "replies": [] }, { "id": 1075371, "author_name": "abdessalemboukil", "author_url": "", "post_date": "11/11/2020 16:44:45", "content": "It's pretty straightforward. As you said, I have a dictionary with user_id as a key with dictionaries with their features as values; although I have more than three dictionaries of that sort.
\n9h -> 20min? That's amazing!
\nChange DataFrame to ndarray, loop a dict, concat by hand, and you saved 8h?!
\nMan, that's unbelievable
\nI would like to try
In terms of feature engineering, do you still use pandas?
", "votes": null, "replies": [ { "id": 1075359, "author_name": "abdessalemboukil", "author_url": "", "post_date": "11/11/2020 16:31:34", "content": "The offline features yes, but during inference no.
", "votes": null, "replies": [] }, { "id": 1078414, "author_name": "luukhofman", "author_url": "", "post_date": "11/14/2020 18:42:54", "content": "Pandas is great for exploring the dataset as well but for operations numpy is better because its datastructure (arrays) are optmized for compute.
", "votes": null, "replies": [] } ] }, { "id": 1075447, "author_name": "ujjwalsood07", "author_url": "", "post_date": "11/11/2020 18:19:37", "content": "this maybe true for small datasets (I haven't tried myself), but for larger datasets we may need new dataframe handling library which is faster than pandas?? Tell if you know any??
", "votes": null, "replies": [ { "id": 1075452, "author_name": "ghostskipper", "author_url": "", "post_date": "11/11/2020 18:25:07", "content": "I guess during the inference phase the dataset are quite small.
", "votes": null, "replies": [] }, { "id": 1075482, "author_name": "iuryck", "author_url": "", "post_date": "11/11/2020 18:40:43", "content": "You can use CUDF from Rapids, it uses GPU, speeds up alot
", "votes": null, "replies": [] }, { "id": 1075752, "author_name": "jpmiller", "author_url": "", "post_date": "11/11/2020 23:43:58", "content": "xarray might be worth a try.
", "votes": null, "replies": [] }, { "id": 1075770, "author_name": "iuryck", "author_url": "", "post_date": "11/12/2020 00:21:19", "content": "Seems interesting, does kaggle support it? Or is there a offline dataset to add in the kernel?
", "votes": null, "replies": [] }, { "id": 1077121, "author_name": "mohamadnawfal", "author_url": "", "post_date": "11/13/2020 08:55:12", "content": "@jpmiller
\nI did quick search about xarray, and found below article says that xarray is slow when dealing with small chunks of data. what is your opinion about that?
\nhttps://github.com/pydata/xarray/issues/2799
Nice find. I thought that xarray was speedy but clearly not for these functions. In that case it sounds like Numpy is the way to go. Plain old python dictionaries might also work in some cases.
", "votes": null, "replies": [] } ] }, { "id": 1077110, "author_name": "mohamadnawfal", "author_url": "", "post_date": "11/13/2020 08:46:16", "content": "I applied same idea, but got small improvement in the execution time
\nfrom 3:00 hrs ---> it became 2:30 hours (0:30 minutes improvement).
\nbelow is how I implemented it:
env = riiideducation.make_env()\niter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n np_test_df = test_df[['user_id', 'content_id']].to_numpy()\n X_submit = np.empty((len(np_test_df), len(model_cols)))\n for i in np.arange(len(X_submit)):\n X_submit[i][0] = users_dict[np_test_df[i][0]]\n X_submit[i][1] = questions_dict[np_test_df[i][1]]\n test_df['answered_correctly'] = model.predict(X_submit)\n env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\nHi @abdessalemboukil, I wanted to ask, if I'll be using numpy array everywhere instead of dataframe, then how will I be able to keep track of the column name, especially while merging the data of different dataframes, as I require the name of the column on which I can perform a merge operation.
", "votes": null, "replies": [ { "id": 1105671, "author_name": "adityaecdrid", "author_url": "", "post_date": "12/08/2020 04:32:51", "content": "you don't need names in a matrix, keep track of column index rather.
", "votes": null, "replies": [] } ] }, { "id": 1140847, "author_name": "", "author_url": "", "post_date": "01/06/2021 10:10:50", "content": "If my submission does not exceed 9h limit in the current public leaderboard, can it be sure that it will not exceed the 9h limit later in the private leaderboard? Thanks.
", "votes": null, "replies": [ { "id": 1140884, "author_name": "abdessalemboukil", "author_url": "", "post_date": "01/06/2021 10:46:58", "content": "Your score on the private leaderboard is computed at the same time as your public score, so you have nothing to worry about if your submission already succeeded.
", "votes": null, "replies": [] } ] }, { "id": 1141099, "author_name": "", "author_url": "", "post_date": "01/06/2021 13:53:17", "content": "Thank you very much
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1072792": "I have seen many calling to ditch Pandas, I thought I'd make it official in a discussion.\n\nMy submission time took almost 9 hours, ditching Pandas made scoring take only 20 minutes. Through my findings, for small batches (<30 Rows), looping on numpy arrays and dealing with each row independently is way faster than trying to vectorize the operations. Maybe it'd come obvious for some, but it wasn't obvious for me.", "1073751": "How do you do the merging? You check every key you would need in a pd.merge for every row?", "1073776": "I stored the data that I used to use pd.merge on inside python dictionaries, thus lookup time is constant. So while looping, I just retrieve data from those dictionaries and use them to populate my input array.", "1074761": "9h -> 20min? That's amazing!\nChange DataFrame to ndarray, loop a dict, concat by hand, and you saved 8h?!\nMan, that's unbelievable\nI would like to try", "1074928": "In terms of feature engineering, do you still use pandas?", "1075316": "Do you mean something like a dictionary with user_id as keys and an array with all the features as value? And then a loop over each line of the test_df and concat? I still can't see it. Could you share a couple of lines of code to see how this is done?", "1075359": "The offline features yes, but during inference no.", "1075371": "It's pretty straightforward. As you said, I have a dictionary with user_id as a key with dictionaries with their features as values; although I have more than three dictionaries of that sort.\n1. Create an empty numpy array. Shape: (number of rows, number of feature)\n2. Loop through every row\n3. Populate it with data from these dictionaries and input batch\n4. Proceed to inference and scaling", "1075447": "this maybe true for small datasets (I haven't tried myself), but for larger datasets we may need new dataframe handling library which is faster than pandas?? Tell if you know any??", "1075452": "I guess during the inference phase the dataset are quite small.", "1075482": "You can use CUDF from Rapids, it uses GPU, speeds up alot", "1075752": "xarray might be worth a try.", "1075770": "Seems interesting, does kaggle support it? Or is there a offline dataset to add in the kernel?", "1077110": "I applied same idea, but got small improvement in the execution time\nfrom 3:00 hrs ---> it became 2:30 hours (0:30 minutes improvement).\nbelow is how I implemented it:\n```\nenv = riiideducation.make_env()\niter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n np_test_df = test_df[['user_id', 'content_id']].to_numpy()\n X_submit = np.empty((len(np_test_df), len(model_cols)))\n for i in np.arange(len(X_submit)):\n X_submit[i][0] = users_dict[np_test_df[i][0]]\n X_submit[i][1] = questions_dict[np_test_df[i][1]]\n test_df['answered_correctly'] = model.predict(X_submit)\n env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n```", "1077121": "jpmiller\nI did quick search about xarray, and found below article says that xarray is slow when dealing with small chunks of data. what is your opinion about that?\nhttps://github.com/pydata/xarray/issues/2799", "1077437": "Nice find. I thought that xarray was speedy but clearly not for these functions. In that case it sounds like Numpy is the way to go. Plain old python dictionaries might also work in some cases.", "1078414": "Pandas is great for exploring the dataset as well but for operations numpy is better because its datastructure (arrays) are optmized for compute.", "1105323": "Hi @abdessalemboukil, I wanted to ask, if I'll be using numpy array everywhere instead of dataframe, then how will I be able to keep track of the column name, especially while merging the data of different dataframes, as I require the name of the column on which I can perform a merge operation.", "1105671": "you don't need names in a matrix, keep track of column index rather.", "1140847": "If my submission does not exceed 9h limit in the current public leaderboard, can it be sure that it will not exceed the 9h limit later in the private leaderboard? Thanks.", "1140884": "Your score on the private leaderboard is computed at the same time as your public score, so you have nothing to worry about if your submission already succeeded.", "1141099": "Thank you very much" }, "source": "meta" }