Mine has been running since last 30mins.
", "messages": [ { "id": "1066247", "postDate": "11/01/2020 14:32:47", "content": "Mine has been running since last 30mins.
", "rawMarkdown": "Mine has been running since last 30mins.", "votes": null }, { "id": "1066311", "postDate": "11/01/2020 15:47:09", "content": "It will take time in hours. Data in test data set is huge
", "rawMarkdown": "It will take time in hours. Data in test data set is huge", "votes": null }, { "id": "1066362", "postDate": "11/01/2020 17:03:35", "content": "Not necessarily. It depends on features and structure of code (eg: I don't use pandas, it is super slow)
\nMine takes ~ 12 minutes for a successful submission.
@rohanrao , Thanks for suggestion. As of now i am using pandas only and it is too slow 😏
", "rawMarkdown": "rohanrao , Thanks for suggestion. As of now i am using pandas only and it is too slow 😏", "votes": null }, { "id": "1066391", "postDate": "11/01/2020 17:41:51", "content": "I'm around 7hrs and I'm using pandas. Currently trying to move to something faster, I haven't figure out what to change to yet.
", "rawMarkdown": "I'm around 7hrs and I'm using pandas. Currently trying to move to something faster, I haven't figure out what to change to yet.", "votes": null }, { "id": "1066683", "postDate": "11/01/2020 23:49:54", "content": "\n\nMine has been running since last 30mins.
\n
Don't worry, I got successful submission after 2 to 3 hours of execution as well depending on models. 😊
", "rawMarkdown": "> Mine has been running since last 30mins.\n\nDon't worry, I got successful submission after 2 to 3 hours of execution as well depending on models. 😊", "votes": null }, { "id": "1066688", "postDate": "11/01/2020 23:55:53", "content": "@rohanrao What would you recommend replacing pandas with to speed it up?
", "rawMarkdown": "rohanrao What would you recommend replacing pandas with to speed it up?", "votes": null }, { "id": "1066724", "postDate": "11/02/2020 00:58:44", "content": "Depending on the type of operations, you could use datatable or rapids or bigquery or plain numpy arrays / dictionaries as @abhimanyud mentioned here.
\nIts also worth exploring C (wrapped in Python) for certain processing to get a speedup (though I feel 9hrs is plenty time if code is structured well).
\nI'm still working on optimizing my own pipeline, my best submission only has 3 engineered features at the moment.
", "rawMarkdown": "Depending on the type of operations, you could use datatable or rapids or bigquery or plain numpy arrays / dictionaries as @abhimanyud mentioned [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193276#1062047).\n\nIts also worth exploring C (wrapped in Python) for certain processing to get a speedup (though I feel 9hrs is plenty time if code is structured well).\n\nI'm still working on optimizing my own pipeline, my best submission only has 3 engineered features at the moment.", "votes": null }, { "id": "1066737", "postDate": "11/02/2020 01:26:04", "content": "@rohanrao Only three engineered features, what kinda of sorcery is this hahahaha
", "rawMarkdown": "rohanrao Only three engineered features, what kinda of sorcery is this hahahaha", "votes": null }, { "id": "1066772", "postDate": "11/02/2020 02:56:16", "content": "Plenty of scores above mine so certainly need to add more 🙂
", "rawMarkdown": "Plenty of scores above mine so certainly need to add more 🙂", "votes": null }, { "id": "1066814", "postDate": "11/02/2020 04:15:12", "content": "\n\nmy best submission only has 3 engineered features at the moment.
\n
OMG and that gets you to 42nd place ATM. 🙏
", "rawMarkdown": ">my best submission only has 3 engineered features at the moment.\n\nOMG and that gets you to 42nd place ATM. :pray:", "votes": null }, { "id": "1067908", "postDate": "11/02/2020 20:18:10", "content": "Did anyone try to simulate in the notebook a submission, in order to estimate how much time it would take on the test set?
", "rawMarkdown": "Did anyone try to simulate in the notebook a submission, in order to estimate how much time it would take on the test set?", "votes": null }, { "id": "1068374", "postDate": "11/03/2020 10:39:35", "content": "@rohanrao do you know of any good tutorials for datatable? I've seen your notebook on reading data in and I'm reading the user guide/API.
\nIt seem very quick, but I keep stumbling into issues.
For example, how do I decrement the values in a column by 1?
\nI started by trying this
train['part'] = train['part'] - 1 \nbut got the following error
\nTypeError: unsupported operand type(s) for -: 'datatable.Frame' and 'int'\nThis is what I have came up with, but it feels like I'm missing something.
\n# Create a dummy column of 1s\ntrain['ones'] = 1\n\n# Decrement a column\ntrain[:, dt.update(part = dt.f.part - dt.f.ones)]\n\n# Drop the unrequired columns\ntrain = train[:, dt.f[:].remove([dt.f.ones])]\nOn my local machine this takes 3ms vs 31ms in pandas (train['part'] = train['part'] - 1), so still alot quicker.
The datatable documentation is not great I admit. But there will be more examples and tutorials with the release of datatable 1.0.0 (still some bugs to resolve but can expect it soon). I'm not aware of any really good and comprehensive tutorial yet but I do plan to write one once 1.0.0 is released.
\nTill then you could use this forum or the Github repo, I'm sure folks will be happy to help, me included.
\ntrain['part'] = train[:, dt.f.part - 1]
The f-expression is used for most of these kind of operations.
", "rawMarkdown": "The datatable documentation is not great I admit. But there will be more examples and tutorials with the release of datatable 1.0.0 (still some bugs to resolve but can expect it soon). I'm not aware of any really good and comprehensive tutorial yet but I do plan to write one once 1.0.0 is released.\n\nTill then you could use this forum or the [Github repo](https://github.com/h2oai/datatable/issues), I'm sure folks will be happy to help, me included.\n\n`train['part'] = train[:, dt.f.part - 1]`\n\nThe [f-expression](https://datatable.readthedocs.io/en/latest/manual/f-expressions.html) is used for most of these kind of operations.", "votes": null }, { "id": "1068383", "postDate": "11/03/2020 10:52:55", "content": "Thank you! I knew that couldn't be the best way!
", "rawMarkdown": "Thank you! I knew that couldn't be the best way!", "votes": null }, { "id": "1068837", "postDate": "11/03/2020 19:35:53", "content": "@rohanrao can I ask you what model you use to get such a huge score without big FE? Btw thanks for the datable trick it speeds up a lot!
", "rawMarkdown": "rohanrao can I ask you what model you use to get such a huge score without big FE? Btw thanks for the datable trick it speeds up a lot!", "votes": null }, { "id": "1069312", "postDate": "11/04/2020 10:02:14", "content": "the loop which make prediction on the hidden set may occupy bigger proportion in running time. In my case,it should cost no more than 5s in the provided 120 samples
", "rawMarkdown": "the loop which make prediction on the hidden set may occupy bigger proportion in running time. In my case,it should cost no more than 5s in the provided 120 samples", "votes": null }, { "id": "1069318", "postDate": "11/04/2020 10:09:11", "content": "agree!hah,it depend more on the time it take to make prediction
", "rawMarkdown": "agree!hah,it depend more on the time it take to make prediction", "votes": null } ], "comments": [ { "id": 1066311, "author_name": "kamalnaithani", "author_url": "", "post_date": "11/01/2020 15:47:09", "content": "It will take time in hours. Data in test data set is huge
", "votes": null, "replies": [ { "id": 1066362, "author_name": "rohanrao", "author_url": "", "post_date": "11/01/2020 17:03:35", "content": "Not necessarily. It depends on features and structure of code (eg: I don't use pandas, it is super slow)
\nMine takes ~ 12 minutes for a successful submission.
@rohanrao , Thanks for suggestion. As of now i am using pandas only and it is too slow 😏
", "votes": null, "replies": [] }, { "id": 1066391, "author_name": "ghostskipper", "author_url": "", "post_date": "11/01/2020 17:41:51", "content": "I'm around 7hrs and I'm using pandas. Currently trying to move to something faster, I haven't figure out what to change to yet.
", "votes": null, "replies": [] }, { "id": 1066688, "author_name": "ghostskipper", "author_url": "", "post_date": "11/01/2020 23:55:53", "content": "@rohanrao What would you recommend replacing pandas with to speed it up?
", "votes": null, "replies": [] }, { "id": 1066724, "author_name": "rohanrao", "author_url": "", "post_date": "11/02/2020 00:58:44", "content": "Depending on the type of operations, you could use datatable or rapids or bigquery or plain numpy arrays / dictionaries as @abhimanyud mentioned here.
\nIts also worth exploring C (wrapped in Python) for certain processing to get a speedup (though I feel 9hrs is plenty time if code is structured well).
\nI'm still working on optimizing my own pipeline, my best submission only has 3 engineered features at the moment.
", "votes": null, "replies": [] }, { "id": 1066737, "author_name": "abdessalemboukil", "author_url": "", "post_date": "11/02/2020 01:26:04", "content": "@rohanrao Only three engineered features, what kinda of sorcery is this hahahaha
", "votes": null, "replies": [] }, { "id": 1066772, "author_name": "rohanrao", "author_url": "", "post_date": "11/02/2020 02:56:16", "content": "Plenty of scores above mine so certainly need to add more 🙂
", "votes": null, "replies": [] }, { "id": 1066814, "author_name": "adityaecdrid", "author_url": "", "post_date": "11/02/2020 04:15:12", "content": "\n\nmy best submission only has 3 engineered features at the moment.
\n
OMG and that gets you to 42nd place ATM. 🙏
", "votes": null, "replies": [] }, { "id": 1068374, "author_name": "ghostskipper", "author_url": "", "post_date": "11/03/2020 10:39:35", "content": "@rohanrao do you know of any good tutorials for datatable? I've seen your notebook on reading data in and I'm reading the user guide/API.
\nIt seem very quick, but I keep stumbling into issues.
For example, how do I decrement the values in a column by 1?
\nI started by trying this
train['part'] = train['part'] - 1 \nbut got the following error
\nTypeError: unsupported operand type(s) for -: 'datatable.Frame' and 'int'\nThis is what I have came up with, but it feels like I'm missing something.
\n# Create a dummy column of 1s\ntrain['ones'] = 1\n\n# Decrement a column\ntrain[:, dt.update(part = dt.f.part - dt.f.ones)]\n\n# Drop the unrequired columns\ntrain = train[:, dt.f[:].remove([dt.f.ones])]\nOn my local machine this takes 3ms vs 31ms in pandas (train['part'] = train['part'] - 1), so still alot quicker.
The datatable documentation is not great I admit. But there will be more examples and tutorials with the release of datatable 1.0.0 (still some bugs to resolve but can expect it soon). I'm not aware of any really good and comprehensive tutorial yet but I do plan to write one once 1.0.0 is released.
\nTill then you could use this forum or the Github repo, I'm sure folks will be happy to help, me included.
\ntrain['part'] = train[:, dt.f.part - 1]
The f-expression is used for most of these kind of operations.
", "votes": null, "replies": [] }, { "id": 1068383, "author_name": "ghostskipper", "author_url": "", "post_date": "11/03/2020 10:52:55", "content": "Thank you! I knew that couldn't be the best way!
", "votes": null, "replies": [] }, { "id": 1068837, "author_name": "matthieuplante", "author_url": "", "post_date": "11/03/2020 19:35:53", "content": "@rohanrao can I ask you what model you use to get such a huge score without big FE? Btw thanks for the datable trick it speeds up a lot!
", "votes": null, "replies": [] } ] }, { "id": 1066683, "author_name": "raghuramr", "author_url": "", "post_date": "11/01/2020 23:49:54", "content": "\n\nMine has been running since last 30mins.
\n
Don't worry, I got successful submission after 2 to 3 hours of execution as well depending on models. 😊
", "votes": null, "replies": [] }, { "id": 1067908, "author_name": "barkay", "author_url": "", "post_date": "11/02/2020 20:18:10", "content": "Did anyone try to simulate in the notebook a submission, in order to estimate how much time it would take on the test set?
", "votes": null, "replies": [ { "id": 1069318, "author_name": "huangtaogan", "author_url": "", "post_date": "11/04/2020 10:09:11", "content": "agree!hah,it depend more on the time it take to make prediction
", "votes": null, "replies": [] } ] }, { "id": 1069312, "author_name": "huangtaogan", "author_url": "", "post_date": "11/04/2020 10:02:14", "content": "the loop which make prediction on the hidden set may occupy bigger proportion in running time. In my case,it should cost no more than 5s in the provided 120 samples
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1066247": "Mine has been running since last 30mins.", "1066311": "It will take time in hours. Data in test data set is huge", "1066362": "Not necessarily. It depends on features and structure of code (eg: I don't use pandas, it is super slow)\nMine takes ~ 12 minutes for a successful submission.", "1066373": "rohanrao , Thanks for suggestion. As of now i am using pandas only and it is too slow 😏", "1066391": "I'm around 7hrs and I'm using pandas. Currently trying to move to something faster, I haven't figure out what to change to yet.", "1066683": "> Mine has been running since last 30mins.\n\nDon't worry, I got successful submission after 2 to 3 hours of execution as well depending on models. 😊", "1066688": "rohanrao What would you recommend replacing pandas with to speed it up?", "1066724": "Depending on the type of operations, you could use datatable or rapids or bigquery or plain numpy arrays / dictionaries as @abhimanyud mentioned [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193276#1062047).\n\nIts also worth exploring C (wrapped in Python) for certain processing to get a speedup (though I feel 9hrs is plenty time if code is structured well).\n\nI'm still working on optimizing my own pipeline, my best submission only has 3 engineered features at the moment.", "1066737": "rohanrao Only three engineered features, what kinda of sorcery is this hahahaha", "1066772": "Plenty of scores above mine so certainly need to add more 🙂", "1066814": ">my best submission only has 3 engineered features at the moment.\n\nOMG and that gets you to 42nd place ATM. :pray:", "1067908": "Did anyone try to simulate in the notebook a submission, in order to estimate how much time it would take on the test set?", "1068374": "rohanrao do you know of any good tutorials for datatable? I've seen your [notebook on reading data in](https://www.kaggle.com/rohanrao/riiid-with-blazing-fast-rid) and I'm reading the [user guide/API](https://datatable.readthedocs.io/en/latest/manual/comparison_with_pandas.html).\nIt seem very quick, but I keep stumbling into issues.\n\nFor example, how do I decrement the values in a column by 1?\nI started by trying this\n```\ntrain['part'] = train['part'] - 1 \n```\nbut got the following error\n```\nTypeError: unsupported operand type(s) for -: 'datatable.Frame' and 'int'\n```\nThis is what I have came up with, but it feels like I'm missing something.\n```\n# Create a dummy column of 1s\ntrain['ones'] = 1\n\n# Decrement a column\ntrain[:, dt.update(part = dt.f.part - dt.f.ones)]\n\n# Drop the unrequired columns\ntrain = train[:, dt.f[:].remove([dt.f.ones])]\n```\nOn my local machine this takes 3ms vs 31ms in pandas (`train['part'] = train['part'] - 1`), so still alot quicker.", "1068381": "The datatable documentation is not great I admit. But there will be more examples and tutorials with the release of datatable 1.0.0 (still some bugs to resolve but can expect it soon). I'm not aware of any really good and comprehensive tutorial yet but I do plan to write one once 1.0.0 is released.\n\nTill then you could use this forum or the [Github repo](https://github.com/h2oai/datatable/issues), I'm sure folks will be happy to help, me included.\n\n`train['part'] = train[:, dt.f.part - 1]`\n\nThe [f-expression](https://datatable.readthedocs.io/en/latest/manual/f-expressions.html) is used for most of these kind of operations.", "1068383": "Thank you! I knew that couldn't be the best way!", "1068837": "rohanrao can I ask you what model you use to get such a huge score without big FE? Btw thanks for the datable trick it speeds up a lot!", "1069312": "the loop which make prediction on the hidden set may occupy bigger proportion in running time. In my case,it should cost no more than 5s in the provided 120 samples", "1069318": "agree!hah,it depend more on the time it take to make prediction" }, "source": "meta" }