Hi guys I am start doing this competition this week and up until now I still can't make any submission because the runtime exceeded 9 hours. Below are the experiment that I have try :
\n1. Submitting using 10 variable LGBM -- takes ~27 second for dummy test
\nTo preprocess the data I already store the aggregated variable like avg score per user, avg score per question, etc into a dict. For every test batch I am using map function and I also have a seperate dict containing the fillna value for each variable. It basically like this
test_df[var] = test_df[var]['…'].map(dict_value['…']).fillna(dict_fillna[…])
\n2. Submitting using 9 variable LGBM -- takes ~12 second for dummy test
\nIn this attempt I am using join function to simultaneouly map all the aggregated variable at once. Basically like this
test_df = test_df.set_index('…').join([df_value1, df_value2, df_value3, …], how='left').reset_index()
\nUsing this method I manage to reduce the runtime but it still not enough. That's why I want to ask how many variable that you guys used to make succesful submission. What is the dummy test runtime and maybe a tips to further speed up the preprocessing ? Thanks
\nNote :
\nHi guys I am start doing this competition this week and up until now I still can't make any submission because the runtime exceeded 9 hours. Below are the experiment that I have try :
\n1. Submitting using 10 variable LGBM -- takes ~27 second for dummy test
\nTo preprocess the data I already store the aggregated variable like avg score per user, avg score per question, etc into a dict. For every test batch I am using map function and I also have a seperate dict containing the fillna value for each variable. It basically like this
test_df[var] = test_df[var]['…'].map(dict_value['…']).fillna(dict_fillna[…])
\n2. Submitting using 9 variable LGBM -- takes ~12 second for dummy test
\nIn this attempt I am using join function to simultaneouly map all the aggregated variable at once. Basically like this
test_df = test_df.set_index('…').join([df_value1, df_value2, df_value3, …], how='left').reset_index()
\nUsing this method I manage to reduce the runtime but it still not enough. That's why I want to ask how many variable that you guys used to make succesful submission. What is the dummy test runtime and maybe a tips to further speed up the preprocessing ? Thanks
\nNote :
\nCheck this for some benchmarks.
\nThe exact scoring time for example test set should ideally be ~ 1 second (just the scoring not the whole notebook runtime). Because only if you can score 100 rows in a second would you be able to score 2.5M rows of private test set within 9 hours (assuming they are similar in batch size).
", "rawMarkdown": "Check [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192124#1056282) for some benchmarks.\n\nThe exact scoring time for example test set should ideally be ~ 1 second (just the scoring not the whole notebook runtime). Because only if you can score 100 rows in a second would you be able to score 2.5M rows of private test set within 9 hours (assuming they are similar in batch size).", "votes": null }, { "id": "1059985", "postDate": "10/25/2020 16:43:36", "content": "Hmm come to think of it ~1 second is a good measurement. Still got a long way to go but thanks for giving me a goal !!
", "rawMarkdown": "Hmm come to think of it ~1 second is a good measurement. Still got a long way to go but thanks for giving me a **goal** !!", "votes": null }, { "id": "1060271", "postDate": "10/26/2020 02:19:49", "content": "Is there something wrong with subs? My previously successful sub is now taking more than 9 hours yet the notebook is still failing with submission error which doesn't makes sense to me as it cannot be a timeout error.
\nNVM It just got completed after around close to 7 hours
", "rawMarkdown": "Is there something wrong with subs? My previously successful sub is now taking more than 9 hours yet the notebook is still failing with submission error which doesn't makes sense to me as it cannot be a timeout error.\n\nNVM It just got completed after around close to 7 hours", "votes": null }, { "id": "1060340", "postDate": "10/26/2020 04:29:56", "content": "UPDATE : finally able to submit using 4 variable and merge function (idk why but merge is much more faster than join and map). It takes ~600ms for dummy test and ~3 hours for submission so I think the baseline for succesful submission is about ~1.5-1.7 seconds for dummy test
", "rawMarkdown": "UPDATE : finally able to submit using 4 variable and **merge** function (idk why but merge is much more faster than join and map). It takes ~600ms for dummy test and ~3 hours for submission so I think the baseline for succesful submission is about ~1.5-1.7 seconds for dummy test", "votes": null } ], "comments": [ { "id": 1059910, "author_name": "rohanrao", "author_url": "", "post_date": "10/25/2020 15:24:14", "content": "Check this for some benchmarks.
\nThe exact scoring time for example test set should ideally be ~ 1 second (just the scoring not the whole notebook runtime). Because only if you can score 100 rows in a second would you be able to score 2.5M rows of private test set within 9 hours (assuming they are similar in batch size).
", "votes": null, "replies": [ { "id": 1059985, "author_name": "marcellosusanto", "author_url": "", "post_date": "10/25/2020 16:43:36", "content": "Hmm come to think of it ~1 second is a good measurement. Still got a long way to go but thanks for giving me a goal !!
", "votes": null, "replies": [] } ] }, { "id": 1060271, "author_name": "adityaecdrid", "author_url": "", "post_date": "10/26/2020 02:19:49", "content": "Is there something wrong with subs? My previously successful sub is now taking more than 9 hours yet the notebook is still failing with submission error which doesn't makes sense to me as it cannot be a timeout error.
\nNVM It just got completed after around close to 7 hours
", "votes": null, "replies": [] }, { "id": 1060340, "author_name": "marcellosusanto", "author_url": "", "post_date": "10/26/2020 04:29:56", "content": "UPDATE : finally able to submit using 4 variable and merge function (idk why but merge is much more faster than join and map). It takes ~600ms for dummy test and ~3 hours for submission so I think the baseline for succesful submission is about ~1.5-1.7 seconds for dummy test
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1059708": "Hi guys I am start doing this competition this week and up until now I still can't make any submission because the runtime exceeded 9 hours. Below are the experiment that I have try :\n\n**1. Submitting using 10 variable LGBM -- takes ~27 second for dummy test**\nTo preprocess the data I already store the aggregated variable like avg score per user, avg score per question, etc into a dict. For every test batch I am using **map** function and I also have a seperate dict containing the fillna value for each variable. It basically like this\n\n*test_df[var] = test_df[var]['...'].map(dict_value['...']).fillna(dict_fillna[...])*\n\n**2. Submitting using 9 variable LGBM -- takes ~12 second for dummy test**\nIn this attempt I am using **join** function to simultaneouly map all the aggregated variable at once. Basically like this\n\n*test_df = test_df.set_index('...').join([df_value1, df_value2, df_value3, ...], how='left').reset_index()*\n\nUsing this method I manage to reduce the runtime but it still not enough. That's why I want to ask how many variable that you guys used to make **succesful submission**. What is the dummy test runtime and maybe a tips to further speed up the preprocessing ? Thanks\n\nNote :\n - I still haven't tried **merge** function because I read that **join** is much more faster\n - I try calculating the runtime for each phase in the prediction (preprocess, scaling, predict) and 99% of the runtime is because of the preprocessing explained above.", "1059910": "Check [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192124#1056282) for some benchmarks.\n\nThe exact scoring time for example test set should ideally be ~ 1 second (just the scoring not the whole notebook runtime). Because only if you can score 100 rows in a second would you be able to score 2.5M rows of private test set within 9 hours (assuming they are similar in batch size).", "1059985": "Hmm come to think of it ~1 second is a good measurement. Still got a long way to go but thanks for giving me a **goal** !!", "1060271": "Is there something wrong with subs? My previously successful sub is now taking more than 9 hours yet the notebook is still failing with submission error which doesn't makes sense to me as it cannot be a timeout error.\n\nNVM It just got completed after around close to 7 hours", "1060340": "UPDATE : finally able to submit using 4 variable and **merge** function (idk why but merge is much more faster than join and map). It takes ~600ms for dummy test and ~3 hours for submission so I think the baseline for succesful submission is about ~1.5-1.7 seconds for dummy test" }, "source": "meta" }