\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F436401%2F96523dfe4c26395bad48b255070bf6aa%2FScreenshot%20from%202020-10-14%2002-01-44.png?generation=1602666258375135&alt=media\")
I am doing almost all feature engineering on BigQuery. Maybe, this is a good chance to learn BQ. We should not use Pandas (of course, it's a great tool though) to deal with more than 100M records.
\nP.S. This is not Google Ads.
", "messages": [ { "id": 1045709, "postDate": "2020-10-10T23:39:15.063Z", "content": "I am doing almost all feature engineering on BigQuery. Maybe, this is a good chance to learn BQ. We should not use Pandas (of course, it's a great tool though) to deal with more than 100M records.
\nP.S. This is not Google Ads.
", "rawMarkdown": "I am doing almost all feature engineering on BigQuery. Maybe, this is a good chance to learn BQ. We should not use Pandas (of course, it's a great tool though) to deal with more than 100M records.\n\nP.S. This is not Google Ads.", "votes": 59 }, { "id": 1045922, "postDate": "2020-10-11T06:43:51.893Z", "content": "Your video of using bq in kaggle may be helpful for us : )
\nThank you!!
\nhttps://www.youtube.com/watch?v=oNH_rhf32y0 (Japanese)
Just out of curiosity. Are we allowed to use BQ? If my assumption is correct, we can't get prize.
\ne.g. We are not allowed to use optimization software such as Gurobi in Santa competition. Because it is not free.
https://www.kaggle.com/c/riiid-test-answer-prediction/rules
\nA. COMPETITION-SPECIFIC RULES\nIn addition to the provisions of the General Competition Rules below, you understand and agree to these Competition-Specific Rules required by the Competition Sponsor:\n\n1. WINNER LICENSE.\nUnder Section 11 (Winners Obligations) of the General Rules below, you hereby grant and will grant Competition Sponsor the following license(s) with respect to your Submission if you are a Competition winner:\n\nNon-Exclusive: You hereby grant and will grant to Competition Sponsor and its designees a worldwide, non-exclusive, sub-licensable, transferable, fully paid-up, royalty-free, perpetual, irrevocable right to use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and import your winning Submission and the source code used to generate the Submission, in any media now known or developed in the future, for any purpose whatsoever, commercial or otherwise, without further approval by or payment to you. For generally commercially available software that you used to generate your submission that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, you do not need to grant the license in the preceding sentence for that software.\n\n~~~~~\n\n8. SUBMISSION CODE REQUIREMENTS.\nC. Use of Open Source. Unless otherwise stated in the Specific Competition Rules above, if open source code is used in the model to generate the Submission, then you must only use open source code licensed under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code.\n\n~~~~~\n\n11. WINNERS OBLIGATIONS.\nAs a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:\n\n(a) deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation. The delivered software code should follow these documentation guidelines, must be capable of generating the winning Submission, and contain a description of resources required to build and/or run the executable code successfully. If the final model’s software code includes generally commercially available software that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, then instead of delivering the code for that software to the Competition Sponsor, you must identify that software, method for procuring it, and any parameters or other information necessary to replicate the winning Submission;\nI cannot see any sentences that restrict the use of generally commercially available software. Which sentence are you referring to?
\nbtw, OSS has a limitation that it's needed to be allowed in commercial use.
I tried to list related to license and read carefully, but as you said it seems they would not restrict to use those kind of softwares except this part.
\n\n\nwithout undue expense
\n
Let me confirm if we can use.
\n@jineonjinbaek @addisonhoward @sohier Are we allowed to use BigQuery here?
", "rawMarkdown": "I tried to list related to license and read carefully, but as you said it seems they would not restrict to use those kind of softwares except this part.\n\n> without undue expense\n\nLet me confirm if we can use.\n\n@jineonjinbaek @addisonhoward @sohier Are we allowed to use BigQuery here?", "votes": 2 }, { "id": 1047441, "postDate": "2020-10-12T15:57:04.630Z", "content": "BigQuery use is fine :)
", "rawMarkdown": "BigQuery use is fine :)", "votes": 8 }, { "id": 1049209, "postDate": "2020-10-14T07:52:39.860Z", "content": "If we can't use BQ because it's costly, what about using GCP/AWS for GPU, which are even more costly ? As @addisonhoward said it, BQ should be ok.
", "rawMarkdown": "If we can't use BQ because it's costly, what about using GCP/AWS for GPU, which are even more costly ? As @addisonhoward said it, BQ should be ok.", "votes": 2 }, { "id": 1049269, "postDate": "2020-10-14T09:14:57.160Z", "content": "GPU instance should be significantly expensive than BQ even if you use a spot instance or preemptive instance. The current charge of BQ and GCS is about
\nThe cost mainly comes from storage and the total price is even less than the one-day rent of GPU instances.
\nActually, I have never worried about the BQ cost in the Kaggle competitions because I know the costly cases happen usually in company usage, not in personal usage.
\nYou can try BQ for free. If you create a GCP account, you will get a $300 coupon. You won't be charged more than that as long as you do not agree with an account upgrade.
\nSince the first terabyte of data queried with BQ each month is free, you can get a lot of work on this dataset without charges.
", "rawMarkdown": "Since [the first terabyte of data queried with BQ each month is free](https://cloud.google.com/bigquery/pricing), you can get a lot of work on this dataset without charges. ", "votes": 1 }, { "id": 1049978, "postDate": "2020-10-15T00:23:40.697Z", "content": "@onodera The latest Santa (2019) was a competition that allows Gurobi, personally I myself used Gurobi too, no problem.
", "rawMarkdown": "@onodera The latest Santa (2019) was a competition that allows Gurobi, personally I myself used Gurobi too, no problem.", "votes": 2 }, { "id": 1051346, "postDate": "2020-10-16T12:27:13.967Z", "content": "@sohier is it possible all Kaggle competition datasets can be accessible in BQ directly same way as
\nhttps://www.kaggle.com/datasets?filetype=bigQuery ( at-least tabular data )
By the way, Do you know whether we can use BigQuery not only for training but also for test inference phase?
\n( I cannot find in rule document….)
It might be possible. This competition is allowed to use internet access during inference. if you can submit the kernel with BQ, it should be okay.
\nUsually, internet access is restricted as in https://www.kaggle.com/c/tweet-sentiment-extraction/overview/code-requirements.
\nHowever, BQ advantage is large scale data processing. Pandas might be faster. If you want to just access the tables, uploading the tables as a Kaggle dataset might work better.
", "rawMarkdown": "It might be possible. This competition is allowed to use internet access during inference. if you can submit the kernel with BQ, it should be okay.\n\nUsually, internet access is restricted as in https://www.kaggle.com/c/tweet-sentiment-extraction/overview/code-requirements.\n\nHowever, BQ advantage is large scale data processing. Pandas might be faster. If you want to just access the tables, uploading the tables as a Kaggle dataset might work better." }, { "id": 1052828, "postDate": "2020-10-18T10:24:01.573Z", "content": "Thank you for kind comment.
\nExactly in
\ninference phace Pandas may be better.
I am going to do feature engineering on BigQuery. Is it allowed to import feature data back into Kaggle Input?
", "rawMarkdown": "I am going to do feature engineering on BigQuery. Is it allowed to import feature data back into Kaggle Input?", "replies": [ { "id": 1047165, "postDate": "2020-10-12T10:32:57.770Z", "content": "You have to upload files (e.g., dumped csv from BQ via GCS) as a dataset.
", "rawMarkdown": "You have to upload files (e.g., dumped csv from BQ via GCS) as a dataset.", "votes": 1 }, { "id": 1051338, "postDate": "2020-10-16T12:21:14.507Z", "content": "@tkm2261 @sohier @addisonhoward
\nI have done some feature engineering on bigquery.
\nBut I'm having a hard time on how to load it into kaggle kernel or upload it into kaggle datasets.
\nOr just like you said You have to upload files (e.g., dumped csv from BQ via GCS) as a dataset.
Could you explain it a little more. It would be really helpfull
", "rawMarkdown": "@tkm2261 @sohier @addisonhoward \nI have done some feature engineering on bigquery.\nBut I'm having a hard time on how to load it into kaggle kernel or upload it into kaggle datasets.\nOr just like you said `You have to upload files (e.g., dumped csv from BQ via GCS) as a dataset.`\n\nCould you explain it a little more. It would be really helpfull" } ] }, { "id": 1047744, "postDate": "2020-10-12T22:25:28.827Z", "content": "Congratulation 👍👍
\nthank you for your advices
Your video of using bq in kaggle may be helpful for us : )
\nThank you!!
\nhttps://www.youtube.com/watch?v=oNH_rhf32y0 (Japanese)
Just out of curiosity. Are we allowed to use BQ? If my assumption is correct, we can't get prize.
\ne.g. We are not allowed to use optimization software such as Gurobi in Santa competition. Because it is not free.
https://www.kaggle.com/c/riiid-test-answer-prediction/rules
\nA. COMPETITION-SPECIFIC RULES\nIn addition to the provisions of the General Competition Rules below, you understand and agree to these Competition-Specific Rules required by the Competition Sponsor:\n\n1. WINNER LICENSE.\nUnder Section 11 (Winners Obligations) of the General Rules below, you hereby grant and will grant Competition Sponsor the following license(s) with respect to your Submission if you are a Competition winner:\n\nNon-Exclusive: You hereby grant and will grant to Competition Sponsor and its designees a worldwide, non-exclusive, sub-licensable, transferable, fully paid-up, royalty-free, perpetual, irrevocable right to use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and import your winning Submission and the source code used to generate the Submission, in any media now known or developed in the future, for any purpose whatsoever, commercial or otherwise, without further approval by or payment to you. For generally commercially available software that you used to generate your submission that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, you do not need to grant the license in the preceding sentence for that software.\n\n~~~~~\n\n8. SUBMISSION CODE REQUIREMENTS.\nC. Use of Open Source. Unless otherwise stated in the Specific Competition Rules above, if open source code is used in the model to generate the Submission, then you must only use open source code licensed under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code.\n\n~~~~~\n\n11. WINNERS OBLIGATIONS.\nAs a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:\n\n(a) deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation. The delivered software code should follow these documentation guidelines, must be capable of generating the winning Submission, and contain a description of resources required to build and/or run the executable code successfully. If the final model’s software code includes generally commercially available software that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, then instead of delivering the code for that software to the Competition Sponsor, you must identify that software, method for procuring it, and any parameters or other information necessary to replicate the winning Submission;\nI cannot see any sentences that restrict the use of generally commercially available software. Which sentence are you referring to?
\nbtw, OSS has a limitation that it's needed to be allowed in commercial use.
I tried to list related to license and read carefully, but as you said it seems they would not restrict to use those kind of softwares except this part.
\n\n\nwithout undue expense
\n
Let me confirm if we can use.
\n@jineonjinbaek @addisonhoward @sohier Are we allowed to use BigQuery here?
", "votes": 2, "replies": [] }, { "id": 1047441, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-10-12T15:57:04.630000", "content": "BigQuery use is fine :)
", "votes": 8, "replies": [] }, { "id": 1049209, "author_name": "kkiller", "author_url": "", "post_date": "2020-10-14T07:52:39.860000", "content": "If we can't use BQ because it's costly, what about using GCP/AWS for GPU, which are even more costly ? As @addisonhoward said it, BQ should be ok.
", "votes": 2, "replies": [] }, { "id": 1049269, "author_name": "tkm2261", "author_url": "", "post_date": "2020-10-14T09:14:57.160000", "content": "GPU instance should be significantly expensive than BQ even if you use a spot instance or preemptive instance. The current charge of BQ and GCS is about
\nThe cost mainly comes from storage and the total price is even less than the one-day rent of GPU instances.
\nActually, I have never worried about the BQ cost in the Kaggle competitions because I know the costly cases happen usually in company usage, not in personal usage.
\nYou can try BQ for free. If you create a GCP account, you will get a $300 coupon. You won't be charged more than that as long as you do not agree with an account upgrade.
\nSince the first terabyte of data queried with BQ each month is free, you can get a lot of work on this dataset without charges.
", "votes": 1, "replies": [] }, { "id": 1049978, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-10-15T00:23:40.697000", "content": "@onodera The latest Santa (2019) was a competition that allows Gurobi, personally I myself used Gurobi too, no problem.
", "votes": 2, "replies": [] }, { "id": 1051346, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2020-10-16T12:27:13.967000", "content": "@sohier is it possible all Kaggle competition datasets can be accessible in BQ directly same way as
\nhttps://www.kaggle.com/datasets?filetype=bigQuery ( at-least tabular data )
By the way, Do you know whether we can use BigQuery not only for training but also for test inference phase?
\n( I cannot find in rule document….)
It might be possible. This competition is allowed to use internet access during inference. if you can submit the kernel with BQ, it should be okay.
\nUsually, internet access is restricted as in https://www.kaggle.com/c/tweet-sentiment-extraction/overview/code-requirements.
\nHowever, BQ advantage is large scale data processing. Pandas might be faster. If you want to just access the tables, uploading the tables as a Kaggle dataset might work better.
", "votes": 0, "replies": [] }, { "id": 1052828, "author_name": "pao", "author_url": "", "post_date": "2020-10-18T10:24:01.573000", "content": "Thank you for kind comment.
\nExactly in
\ninference phace Pandas may be better.
I am going to do feature engineering on BigQuery. Is it allowed to import feature data back into Kaggle Input?
", "votes": 0, "replies": [ { "id": 1047165, "author_name": "tkm2261", "author_url": "", "post_date": "2020-10-12T10:32:57.770000", "content": "You have to upload files (e.g., dumped csv from BQ via GCS) as a dataset.
", "votes": 1, "replies": [] }, { "id": 1051338, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-10-16T12:21:14.507000", "content": "@tkm2261 @sohier @addisonhoward
\nI have done some feature engineering on bigquery.
\nBut I'm having a hard time on how to load it into kaggle kernel or upload it into kaggle datasets.
\nOr just like you said You have to upload files (e.g., dumped csv from BQ via GCS) as a dataset.
Could you explain it a little more. It would be really helpfull
", "votes": 0, "replies": [] } ] }, { "id": 1047744, "author_name": "FOUILI Abderrahim", "author_url": "", "post_date": "2020-10-12T22:25:28.827000", "content": "Congratulation 👍👍
\nthank you for your advices