{ "id": 128864, "title": "Setup Google Cloud Instance for DFDC", "url": "/competitions/deepfake-detection-challenge/discussion/128864", "author_name": "Debanga Raj Neog", "post_date": "2020-02-03T21:21:40.294000", "votes": 25, "comment_count": 15, "views": 0, "content": "

Disclaimer: Cheatsheet only

\n\n

Basics

\n\n\n\n

E.g. Device id: sdb, mount dir: dfdc_data

\n\n

With Format

\n\n

\nsudo lsblk && sudo mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb && sudo mkdir -p /mnt/disks/dfdc_data && sudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n

\n\n

Without Format

\n\n

\nsudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n

\n\n

Download data

\n\n\n\n

Mine looks like this:\n\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=[bada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boom]TVZx34pegM3Y%2BnmpP1Lmzlh3blRCm7MpsV9LEEsnXobuP3YwZNEzzF1nh7C%2BG7jtJyuz5JHkJ6pyoHKtHunTw%3D%3D\" -O \"dfdc_train_all.zip\" -c\n\n- Change dfdc_train_all.zip to [YOUR_PERSISTENT_DISK_PATH]/dfdc_train_all.zip

\n\n\n\n

```\nI'm lazy, make a for loop if you wish.

\n\n

Set up Jupyter

\n\n

```

\n\n

Download Anaconda

\n\n

$ wget http://repo.continuum.io/archive/Anaconda3-4.0.0-Linux-x86_64.sh

\n\n

License, path, etc. (Don't forget to say 'yes' to prepending path)

\n\n

$ bash Anaconda3-4.0.0-Linux-x86_64.sh

\n\n

Activate conda

\n\n

$ source .bashrc

\n\n

Quick check

\n\n

$ conda --version

\n\n

Run pip to test

\n\n

pip install torch\npip install torchvision\n\nRunning Jupyter in browser via external IP.\n\n$ jupyter notebook --ip=0.0.0.0 --port=8080 --no-browser &\n\nAvailable at\n\nhttp://[EXTERNAL_IP]:8080\n```

\n\n

Using buckets

\n\n\n\n

You may also want to save any processed data as .zip to the local disk and then export to GCP bucket. The benefit is you can directly download the .zip file from your GCP Bucket page. It also means you can curl the link it to any other machines or instances, e.g. if you are using AWS Sagemaker for GPU based model training.

\n\n

Reference

\n\n", "messages": [ { "id": 736130, "postDate": "2020-02-03T21:21:40.293Z", "content": "

Disclaimer: Cheatsheet only

\n\n

Basics

\n\n\n\n

E.g. Device id: sdb, mount dir: dfdc_data

\n\n

With Format

\n\n

\nsudo lsblk && sudo mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb && sudo mkdir -p /mnt/disks/dfdc_data && sudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n

\n\n

Without Format

\n\n

\nsudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n

\n\n

Download data

\n\n\n\n

Mine looks like this:\n\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=[bada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boom]TVZx34pegM3Y%2BnmpP1Lmzlh3blRCm7MpsV9LEEsnXobuP3YwZNEzzF1nh7C%2BG7jtJyuz5JHkJ6pyoHKtHunTw%3D%3D\" -O \"dfdc_train_all.zip\" -c\n\n- Change dfdc_train_all.zip to [YOUR_PERSISTENT_DISK_PATH]/dfdc_train_all.zip

\n\n\n\n

```\nI'm lazy, make a for loop if you wish.

\n\n

Set up Jupyter

\n\n

```

\n\n

Download Anaconda

\n\n

$ wget http://repo.continuum.io/archive/Anaconda3-4.0.0-Linux-x86_64.sh

\n\n

License, path, etc. (Don't forget to say 'yes' to prepending path)

\n\n

$ bash Anaconda3-4.0.0-Linux-x86_64.sh

\n\n

Activate conda

\n\n

$ source .bashrc

\n\n

Quick check

\n\n

$ conda --version

\n\n

Run pip to test

\n\n

pip install torch\npip install torchvision\n\nRunning Jupyter in browser via external IP.\n\n$ jupyter notebook --ip=0.0.0.0 --port=8080 --no-browser &\n\nAvailable at\n\nhttp://[EXTERNAL_IP]:8080\n```

\n\n

Using buckets

\n\n\n\n

You may also want to save any processed data as .zip to the local disk and then export to GCP bucket. The benefit is you can directly download the .zip file from your GCP Bucket page. It also means you can curl the link it to any other machines or instances, e.g. if you are using AWS Sagemaker for GPU based model training.

\n\n

Reference

\n\n", "rawMarkdown": "## Disclaimer: Cheatsheet only\n\n### Basics\n\n- Create a GCP account at https://cloud.google.com/ and set up billing details\n\n- Create a project\n\n- Create a VM instance\n\n- Setup external IP of the VM to static, set up firewall based on the reference article [1]\n\n- Add a persistent drive to VM, see reference [2]\n\nE.g. Device id: sdb, mount dir: dfdc_data\n\n### With Format\n```\nsudo lsblk && sudo mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb && sudo mkdir -p /mnt/disks/dfdc_data && sudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n```\n### Without Format\n```\nsudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n```\n\n### Download data\n- Use ```CurlWGet``` chrome extension to get data URL (the link of the full training data from the competition website, I'm referring to the single big zip file).\n\nMine looks like this:\n```\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=[bada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boom]TVZx34pegM3Y%2BnmpP1Lmzlh3blRCm7MpsV9LEEsnXobuP3YwZNEzzF1nh7C%2BG7jtJyuz5JHkJ6pyoHKtHunTw%3D%3D\" -O \"dfdc_train_all.zip\" -c\n```\n- Change ```dfdc_train_all.zip``` to ```[YOUR_PERSISTENT_DISK_PATH]/dfdc_train_all.zip```\n\n- Unzip data to your disk\n```\nunzip dfdc_train_all.zip -d /mnt/disks/dfdc_data\n```\n\n- Wait, what? More .zips inside LOL. I am extracting them to a folder called 'data'\n```\nunzip dfdc_train_part_00.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_01.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_02.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_03.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_04.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_05.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_06.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_07.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_08.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_09.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_10.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_11.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_12.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_13.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_14.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_15.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_16.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_17.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_18.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_19.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_20.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_21.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_22.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_23.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_24.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_25.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_26.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_27.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_28.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_29.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_30.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_31.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_32.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_33.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_34.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_35.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_36.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_37.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_38.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_39.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_40.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_41.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_42.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_43.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_44.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_45.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_46.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_47.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_48.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_49.zip -d /mnt/disks/dfdc_data/data/\n\n```\nI'm lazy, make a for loop if you wish.\n\n### Set up Jupyter\n\n```\n# Download Anaconda\n$ wget http://repo.continuum.io/archive/Anaconda3-4.0.0-Linux-x86_64.sh\n\n# License, path, etc. (Don't forget to say 'yes' to prepending path)\n$ bash Anaconda3-4.0.0-Linux-x86_64.sh\n\n# Activate conda\n$ source .bashrc\n\n# Quick check\n$ conda --version \n\n# Run pip to test\npip install torch\npip install torchvision\n```\nRunning Jupyter in browser via external IP.\n```\n$ jupyter notebook --ip=0.0.0.0 --port=8080 --no-browser &\n```\nAvailable at\n```\nhttp://[EXTERNAL_IP]:8080\n```\n\n### Using buckets\n- Create a bucket\n\n- Getting permission\n```\ngsutil config -b\n```\n\n- Copy some data to bucket\n```\ngsutil -m cp -r /mnt/disks/dfdc_data/data gs://[BUCKET_NAME]\n```\n\n- Or copy data from bucket\n```\ngsutil -m cp -r gs://[BUCKET_NAME]/mnt/disks/dfdc_data/data\n```\n### Bonus\nYou may also want to save any processed data as ```.zip``` to the local disk and then export to GCP bucket. The benefit is you can directly download the .zip file from your GCP Bucket page. It also means you can curl the link it to any other machines or instances, e.g. if you are using AWS Sagemaker for GPU based model training.\n\n\n## Reference\n- [1] https://towardsdatascience.com/running-jupyter-notebook-in-google-cloud-platform-in-15-min-61e16da34d52\n- [2] https://cloud.google.com/compute/docs/disks/add-persistent-disk?hl=en_US&_ga=2.190677090.-876535034.1572141372&_gac=1.82478436.1580475048.CjwKCAiA98TxBRBtEiwAVRLqu19jsp_c8qQn3MXIOej2p37h8OknzWikS9e8h5-RjtZsMABdHadmTxoCHb4QAvD_BwE#formatting", "votes": 24 }, { "id": 740180, "postDate": "2020-02-09T01:40:46.693Z", "content": "

BTW you do need to install cuda and cudnn(unless computing on CPU). Please refer to my kernel. Simple copy and paste. link

", "rawMarkdown": "BTW you do need to install cuda and cudnn(unless computing on CPU). Please refer to my kernel. Simple copy and paste. [link](https://www.kaggle.com/unkownhihi/guide-to-setup-a-vm)", "votes": 1, "replies": [ { "id": 745354, "postDate": "2020-02-13T18:29:41.400Z", "content": "

Thanks! I used only for processing in CPU.

", "rawMarkdown": "Thanks! I used only for processing in CPU." } ] }, { "id": 736241, "postDate": "2020-02-04T01:33:20.833Z", "content": "

Very clear! This is a good cheatsheet for someone haven't had experience with GCP ๐Ÿ‘

", "rawMarkdown": "Very clear! This is a good cheatsheet for someone haven't had experience with GCP ๐Ÿ‘ ", "votes": 2 }, { "id": 761343, "postDate": "2020-03-02T12:06:43.913Z", "content": "

@debanga Is there simple way to download some Kaggle data set to GCP VM instance?

", "rawMarkdown": "@debanga Is there simple way to download some Kaggle data set to GCP VM instance?", "replies": [ { "id": 761423, "postDate": "2020-03-02T13:56:49.273Z", "content": "

Use CurlWGet chrome extension. When you press download from Kaggle dataset, the extension will generate a long link. Use that link to โ€œwgetโ€ in VM.

", "rawMarkdown": "Use CurlWGet chrome extension. When you press download from Kaggle dataset, the extension will generate a long link. Use that link to โ€œwgetโ€ in VM.", "votes": 1 } ] }, { "id": 743368, "postDate": "2020-02-12T02:31:09.453Z", "content": "

That was a great and HELPFUL explanation, thanks Zenify

", "rawMarkdown": "That was a great and HELPFUL explanation, thanks Zenify", "replies": [ { "id": 743402, "postDate": "2020-02-12T03:13:36.693Z", "content": "

Welcome :)

", "rawMarkdown": "Welcome :)" } ] }, { "id": 738411, "postDate": "2020-02-06T13:40:21.083Z", "content": "

Thanks for sharing and very useful for the beginners like me to the Google cloud.

\n\n

It is good if someone creates cheatsheet for aws sagemaker and adding data to s3 bucket just like this good cloud cheatsheet .

", "rawMarkdown": "Thanks for sharing and very useful for the beginners like me to the Google cloud. \n\nIt is good if someone creates cheatsheet for ``aws sagemaker`` and adding data to ``s3`` bucket just like this good cloud cheatsheet . " }, { "id": 736917, "postDate": "2020-02-04T17:50:39.413Z", "content": "

Cool! Thanks for sharing!

\n\n

Another pointer- \nYou can use Google's pre-built Deep Learning VMs instead of installing all the dependencies.

\n\n

Link to Deep Learning VM

\n\n

You can choose a TF or PyTorch framework. Add a GPU or a TPU \nJupyterlab is also pre-installed so, you can open a notebook with a click of a button from the AI Hub console.

", "rawMarkdown": "Cool! Thanks for sharing! \n\nAnother pointer- \nYou can use Google's pre-built Deep Learning VMs instead of installing all the dependencies. \n\n[Link to Deep Learning VM](https://console.cloud.google.com/marketplace/details/click-to-deploy-images/deeplearning)\n\nYou can choose a TF or PyTorch framework. Add a GPU or a TPU \nJupyterlab is also pre-installed so, you can open a notebook with a click of a button from the AI Hub console. \n\n", "replies": [ { "id": 736973, "postDate": "2020-02-04T19:00:22.077Z", "content": "

Thanks for the information :))

", "rawMarkdown": "Thanks for the information :))" } ] }, { "id": 736329, "postDate": "2020-02-04T04:19:24.330Z", "content": "

Please replace & with &. I think while I was copying from terminal to the post, it was changed during HTML conversion, and there are simply too many to change in the post :D Maybe I will do sometime later.

", "rawMarkdown": "Please replace ```&``` with ```&```. I think while I was copying from terminal to the post, it was changed during HTML conversion, and there are simply too many to change in the post :D Maybe I will do sometime later.", "replies": [ { "id": 736710, "postDate": "2020-02-04T13:40:54.863Z", "content": "

you can use wildcards (but you need quotes to avoid its expansion in the shell):\nunzip \"dfdc_train_part_??.zip\" -d /mnt/disks/dfdc_data/data/\n๐Ÿ˜‰

", "rawMarkdown": "you can use wildcards (but you need quotes to avoid its expansion in the shell):\n`unzip \"dfdc_train_part_??.zip\" -d /mnt/disks/dfdc_data/data/`\n๐Ÿ˜‰ ", "votes": 3 }, { "id": 736768, "postDate": "2020-02-04T14:33:15.250Z", "content": "

Perfect! Thanks :) It was one of those midnight scripts, when you just want to get things done and brain stops working haha. Will change it here.

", "rawMarkdown": "Perfect! Thanks :) It was one of those midnight scripts, when you just want to get things done and brain stops working haha. Will change it here." } ] }, { "id": 736249, "postDate": "2020-02-04T01:42:44.100Z", "rawMarkdown": "", "isDeleted": true }, { "id": 745547, "postDate": "2020-02-13T23:00:23.423Z", "content": "

@debanga check your email. Thanks!

", "rawMarkdown": "@debanga check your email. Thanks!", "votes": 1 } ], "comments": [ { "id": 740180, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2020-02-09T01:40:46.693000", "content": "

BTW you do need to install cuda and cudnn(unless computing on CPU). Please refer to my kernel. Simple copy and paste. link

", "votes": 1, "replies": [ { "id": 745354, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-02-13T18:29:41.400000", "content": "

Thanks! I used only for processing in CPU.

", "votes": 0, "replies": [] } ] }, { "id": 736241, "author_name": "Hieu Phung", "author_url": "", "post_date": "2020-02-04T01:33:20.833000", "content": "

Very clear! This is a good cheatsheet for someone haven't had experience with GCP ๐Ÿ‘

", "votes": 2, "replies": [] }, { "id": 761343, "author_name": "Darko Androcec", "author_url": "", "post_date": "2020-03-02T12:06:43.913000", "content": "

@debanga Is there simple way to download some Kaggle data set to GCP VM instance?

", "votes": 0, "replies": [ { "id": 761423, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-03-02T13:56:49.273000", "content": "

Use CurlWGet chrome extension. When you press download from Kaggle dataset, the extension will generate a long link. Use that link to โ€œwgetโ€ in VM.

", "votes": 1, "replies": [] } ] }, { "id": 743368, "author_name": "Ricardo", "author_url": "", "post_date": "2020-02-12T02:31:09.453000", "content": "

That was a great and HELPFUL explanation, thanks Zenify

", "votes": 0, "replies": [ { "id": 743402, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-02-12T03:13:36.693000", "content": "

Welcome :)

", "votes": 0, "replies": [] } ] }, { "id": 738411, "author_name": "Sai Srinivas Reddy", "author_url": "", "post_date": "2020-02-06T13:40:21.083000", "content": "

Thanks for sharing and very useful for the beginners like me to the Google cloud.

\n\n

It is good if someone creates cheatsheet for aws sagemaker and adding data to s3 bucket just like this good cloud cheatsheet .

", "votes": 0, "replies": [] }, { "id": 736917, "author_name": "SkyLord", "author_url": "", "post_date": "2020-02-04T17:50:39.413000", "content": "

Cool! Thanks for sharing!

\n\n

Another pointer- \nYou can use Google's pre-built Deep Learning VMs instead of installing all the dependencies.

\n\n

Link to Deep Learning VM

\n\n

You can choose a TF or PyTorch framework. Add a GPU or a TPU \nJupyterlab is also pre-installed so, you can open a notebook with a click of a button from the AI Hub console.

", "votes": 0, "replies": [ { "id": 736973, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-02-04T19:00:22.077000", "content": "

Thanks for the information :))

", "votes": 0, "replies": [] } ] }, { "id": 736329, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-02-04T04:19:24.330000", "content": "

Please replace & with &. I think while I was copying from terminal to the post, it was changed during HTML conversion, and there are simply too many to change in the post :D Maybe I will do sometime later.

", "votes": 0, "replies": [ { "id": 736710, "author_name": "Henrique Mendonรงa", "author_url": "", "post_date": "2020-02-04T13:40:54.863000", "content": "

you can use wildcards (but you need quotes to avoid its expansion in the shell):\nunzip \"dfdc_train_part_??.zip\" -d /mnt/disks/dfdc_data/data/\n๐Ÿ˜‰

", "votes": 3, "replies": [] }, { "id": 736768, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-02-04T14:33:15.250000", "content": "

Perfect! Thanks :) It was one of those midnight scripts, when you just want to get things done and brain stops working haha. Will change it here.

", "votes": 0, "replies": [] } ] }, { "id": 736249, "author_name": "", "author_url": "", "post_date": "2020-02-04T01:42:44.100000", "content": "", "votes": 0, "replies": [] }, { "id": 745547, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2020-02-13T23:00:23.423000", "content": "

@debanga check your email. Thanks!

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "736130": "## Disclaimer: Cheatsheet only\n\n### Basics\n\n- Create a GCP account at https://cloud.google.com/ and set up billing details\n\n- Create a project\n\n- Create a VM instance\n\n- Setup external IP of the VM to static, set up firewall based on the reference article [1]\n\n- Add a persistent drive to VM, see reference [2]\n\nE.g. Device id: sdb, mount dir: dfdc_data\n\n### With Format\n```\nsudo lsblk && sudo mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb && sudo mkdir -p /mnt/disks/dfdc_data && sudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n```\n### Without Format\n```\nsudo mount -o discard,defaults /dev/sdb /mnt/disks/dfdc_data && sudo chmod a+w /mnt/disks/dfdc_data\n```\n\n### Download data\n- Use ```CurlWGet``` chrome extension to get data URL (the link of the full training data from the competition website, I'm referring to the single big zip file).\n\nMine looks like this:\n```\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=[bada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boombada bing bada boom]TVZx34pegM3Y%2BnmpP1Lmzlh3blRCm7MpsV9LEEsnXobuP3YwZNEzzF1nh7C%2BG7jtJyuz5JHkJ6pyoHKtHunTw%3D%3D\" -O \"dfdc_train_all.zip\" -c\n```\n- Change ```dfdc_train_all.zip``` to ```[YOUR_PERSISTENT_DISK_PATH]/dfdc_train_all.zip```\n\n- Unzip data to your disk\n```\nunzip dfdc_train_all.zip -d /mnt/disks/dfdc_data\n```\n\n- Wait, what? More .zips inside LOL. I am extracting them to a folder called 'data'\n```\nunzip dfdc_train_part_00.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_01.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_02.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_03.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_04.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_05.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_06.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_07.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_08.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_09.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_10.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_11.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_12.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_13.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_14.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_15.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_16.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_17.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_18.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_19.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_20.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_21.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_22.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_23.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_24.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_25.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_26.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_27.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_28.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_29.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_30.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_31.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_32.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_33.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_34.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_35.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_36.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_37.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_38.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_39.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_40.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_41.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_42.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_43.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_44.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_45.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_46.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_47.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_48.zip -d /mnt/disks/dfdc_data/data/ && unzip dfdc_train_part_49.zip -d /mnt/disks/dfdc_data/data/\n\n```\nI'm lazy, make a for loop if you wish.\n\n### Set up Jupyter\n\n```\n# Download Anaconda\n$ wget http://repo.continuum.io/archive/Anaconda3-4.0.0-Linux-x86_64.sh\n\n# License, path, etc. (Don't forget to say 'yes' to prepending path)\n$ bash Anaconda3-4.0.0-Linux-x86_64.sh\n\n# Activate conda\n$ source .bashrc\n\n# Quick check\n$ conda --version \n\n# Run pip to test\npip install torch\npip install torchvision\n```\nRunning Jupyter in browser via external IP.\n```\n$ jupyter notebook --ip=0.0.0.0 --port=8080 --no-browser &\n```\nAvailable at\n```\nhttp://[EXTERNAL_IP]:8080\n```\n\n### Using buckets\n- Create a bucket\n\n- Getting permission\n```\ngsutil config -b\n```\n\n- Copy some data to bucket\n```\ngsutil -m cp -r /mnt/disks/dfdc_data/data gs://[BUCKET_NAME]\n```\n\n- Or copy data from bucket\n```\ngsutil -m cp -r gs://[BUCKET_NAME]/mnt/disks/dfdc_data/data\n```\n### Bonus\nYou may also want to save any processed data as ```.zip``` to the local disk and then export to GCP bucket. The benefit is you can directly download the .zip file from your GCP Bucket page. It also means you can curl the link it to any other machines or instances, e.g. if you are using AWS Sagemaker for GPU based model training.\n\n\n## Reference\n- [1] https://towardsdatascience.com/running-jupyter-notebook-in-google-cloud-platform-in-15-min-61e16da34d52\n- [2] https://cloud.google.com/compute/docs/disks/add-persistent-disk?hl=en_US&_ga=2.190677090.-876535034.1572141372&_gac=1.82478436.1580475048.CjwKCAiA98TxBRBtEiwAVRLqu19jsp_c8qQn3MXIOej2p37h8OknzWikS9e8h5-RjtZsMABdHadmTxoCHb4QAvD_BwE#formatting", "740180": "BTW you do need to install cuda and cudnn(unless computing on CPU). Please refer to my kernel. Simple copy and paste. [link](https://www.kaggle.com/unkownhihi/guide-to-setup-a-vm)", "736241": "Very clear! This is a good cheatsheet for someone haven't had experience with GCP ๐Ÿ‘ ", "761343": "@debanga Is there simple way to download some Kaggle data set to GCP VM instance?", "743368": "That was a great and HELPFUL explanation, thanks Zenify", "738411": "Thanks for sharing and very useful for the beginners like me to the Google cloud. \n\nIt is good if someone creates cheatsheet for ``aws sagemaker`` and adding data to ``s3`` bucket just like this good cloud cheatsheet . ", "736917": "Cool! Thanks for sharing! \n\nAnother pointer- \nYou can use Google's pre-built Deep Learning VMs instead of installing all the dependencies. \n\n[Link to Deep Learning VM](https://console.cloud.google.com/marketplace/details/click-to-deploy-images/deeplearning)\n\nYou can choose a TF or PyTorch framework. Add a GPU or a TPU \nJupyterlab is also pre-installed so, you can open a notebook with a click of a button from the AI Hub console. \n\n", "736329": "Please replace ```&``` with ```&```. I think while I was copying from terminal to the post, it was changed during HTML conversion, and there are simply too many to change in the post :D Maybe I will do sometime later.", "736249": "", "745547": "@debanga check your email. Thanks!" } }