{"cells":[{"metadata":{},"cell_type":"markdown","source":"# kaggle-gcs\nHello Kagglers! This notebook presents kaggle-gcs python utility package, for retrieving gcs bucket addresses. You can install from github:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q git+https://github.com/rosawojciech/kaggle-gcs.git\nfrom kagglegcs import kaggle_gcs_client, gcs_info\ngcs_info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Initialize kaggle_gcs_client\nBefore using kaggle_gcs_client please obtain kaggle API key for your account (discussion with organisers [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/172233)). After this you can set up kaggle API manually, or pass arguments username and key:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nkgc = kaggle_gcs_client(username = 'wrrosa', key = UserSecretsClient().get_secret(\"kaggleapikey\")) \n# optional argument \"command\", i.e. on my local machine command = \"/home/wrosa/.local/bin/kaggle\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And simply run:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nkgc.get_gcs_path('cdeotte/melanoma-128x128')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By running it for the second time, it will restore cached path:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nkgc.get_gcs_path('cdeotte/melanoma-128x128')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another example:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nkgc.get_gcs_path('cdeotte/melanoma-256x256')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cache\nNow, mechanism behind this magic, is to push notebook to Kaggle and retrieve results. Therefore, for saving time, it's a good practice to first list all of your needed datasets, and then run get_gcs_path():","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nkgc = kaggle_gcs_client()\nkgc.cache_gcs_paths(['cdeotte/melanoma-128x128','cdeotte/melanoma-256x256','cdeotte/melanoma-384x384','cdeotte/melanoma-512x512'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nprint(kgc.get_gcs_path('cdeotte/melanoma-128x128'))\nprint(kgc.get_gcs_path('cdeotte/melanoma-256x256'))\nprint(kgc.get_gcs_path('cdeotte/melanoma-384x384'))\nprint(kgc.get_gcs_path('cdeotte/melanoma-512x512'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Other functions (soon depreceated)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kagglegcs import gcs_available, get_gcs_path\ngcs_available()[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"melanoma_ds = gcs_available('melanoma-[0-9]+x*')\nmelanoma_ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for fi in melanoma_ds:\n    print(fi+': '+get_gcs_path(fi))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}