Created a simple script that traverses the directory with the data and converts 6 standalone images into a single RGB image. (Using all available cores of your machine). Probably could be helpful for someone to start with the competition. Not sure how efficient it is to join 6 channels of data into 3 RGB channels, though. But a good start point, I believe.
", "messages": [ { "id": "607080", "postDate": "08/24/2019 15:28:03", "content": "Created a simple script that traverses the directory with the data and converts 6 standalone images into a single RGB image. (Using all available cores of your machine). Probably could be helpful for someone to start with the competition. Not sure how efficient it is to join 6 channels of data into 3 RGB channels, though. But a good start point, I believe.
", "rawMarkdown": "Created [a simple script](https://www.kaggle.com/purplejester/joining-channels-into-rgb-images-parallel) that traverses the directory with the data and converts 6 standalone images into a single RGB image. (Using all available cores of your machine). Probably could be helpful for someone to start with the competition. Not sure how efficient it is to join 6 channels of data into 3 RGB channels, though. But a good start point, I believe.", "votes": null }, { "id": "607447", "postDate": "08/25/2019 09:11:02", "content": "What is the advantage over using the script provided by the host?
", "rawMarkdown": "What is the advantage over using the script [provided by the host](https://github.com/recursionpharma/rxrx1-utils/blob/master/rxrx/io.py)?", "votes": null }, { "id": "607677", "postDate": "08/25/2019 17:48:26", "content": "There is also a very good notebook by @xhlulu using the script provided by the host + additional code to perform resizing and saving images to an easily retrievable new path https://www.kaggle.com/xhlulu/recursion-2019-load-resize-and-save-images .
", "rawMarkdown": "There is also a very good notebook by @xhlulu using the script provided by the host + additional code to perform resizing and saving images to an easily retrievable new path https://www.kaggle.com/xhlulu/recursion-2019-load-resize-and-save-images .", "votes": null }, { "id": "608363", "postDate": "08/26/2019 17:03:23", "content": "Probably I've misunderstood, but the host script seems to pull the data from a remote bucket, not from the local files. And this kernel doesn't use multi-processing, as I can see. On my machine, the parallel script gets approximately x12 speed-up. So probably someone could adapt this code for their need. Anyway, decided to share just in case :)
", "rawMarkdown": "Probably I've misunderstood, but the host script seems to pull the data from a remote bucket, not from the local files. And [this kernel](https://www.kaggle.com/xhlulu/recursion-2019-load-resize-and-save-images) doesn't use multi-processing, as I can see. On my machine, the parallel script gets approximately x12 speed-up. So probably someone could adapt this code for their need. Anyway, decided to share just in case :)", "votes": null }, { "id": "609006", "postDate": "08/27/2019 10:26:55", "content": "Updated the script to use only local data without querying remote buckets provided by the host. Before it used rio.combine_metadata. Could be helpful if you run this script on your local machine, it should be capable to finish the processing within 1-2 hours, depending on the number of cores you have. (In my case, it was around ~30 minutes to merge all samples from the train subset).
Probably it is not too useful for interactive kernels, but could be valuable for dataset preparation.
", "rawMarkdown": "Updated the script to use only local data without querying remote buckets provided by the host. Before it used `rio.combine_metadata`. Could be helpful if you run this script on your local machine, it should be capable to finish the processing within 1-2 hours, depending on the number of cores you have. (In my case, it was around ~30 minutes to merge all samples from the train subset).\n\nProbably it is not too useful for interactive kernels, but could be valuable for dataset preparation.", "votes": null }, { "id": "609522", "postDate": "08/27/2019 19:56:15", "content": "If you check rio.combine_metadata in the github repo, you'll find that you can change the base_path for it to be a local one.
\ndef combine_metadata(base_path=DEFAULT_METADATA_BASE_PATH,\n include_controls=True):\n\nDEFAULT_METADATA_BASE_PATH is gs://rxrx1-us-central1/metadata indeed, but you can change it.
source: https://github.com/recursionpharma/rxrx1-utils/blob/master/rxrx/io.py#L233
", "rawMarkdown": "If you check `rio.combine_metadata` in the github repo, you'll find that you can change the base_path for it to be a local one.\n\n```\ndef combine_metadata(base_path=DEFAULT_METADATA_BASE_PATH,\n include_controls=True):\n```\n`DEFAULT_METADATA_BASE_PATH` is `gs://rxrx1-us-central1/metadata` indeed, but you can change it.\n\nsource: https://github.com/recursionpharma/rxrx1-utils/blob/master/rxrx/io.py#L233", "votes": null }, { "id": "609794", "postDate": "08/28/2019 05:35:07", "content": "Ah makes sense! Didn't check it.
", "rawMarkdown": "Ah makes sense! Didn't check it.", "votes": null } ], "comments": [ { "id": 607447, "author_name": "zaharch", "author_url": "", "post_date": "08/25/2019 09:11:02", "content": "What is the advantage over using the script provided by the host?
", "votes": null, "replies": [ { "id": 607677, "author_name": "michelml", "author_url": "", "post_date": "08/25/2019 17:48:26", "content": "There is also a very good notebook by @xhlulu using the script provided by the host + additional code to perform resizing and saving images to an easily retrievable new path https://www.kaggle.com/xhlulu/recursion-2019-load-resize-and-save-images .
", "votes": null, "replies": [] } ] }, { "id": 608363, "author_name": "purplejester", "author_url": "", "post_date": "08/26/2019 17:03:23", "content": "Probably I've misunderstood, but the host script seems to pull the data from a remote bucket, not from the local files. And this kernel doesn't use multi-processing, as I can see. On my machine, the parallel script gets approximately x12 speed-up. So probably someone could adapt this code for their need. Anyway, decided to share just in case :)
", "votes": null, "replies": [] }, { "id": 609006, "author_name": "purplejester", "author_url": "", "post_date": "08/27/2019 10:26:55", "content": "Updated the script to use only local data without querying remote buckets provided by the host. Before it used rio.combine_metadata. Could be helpful if you run this script on your local machine, it should be capable to finish the processing within 1-2 hours, depending on the number of cores you have. (In my case, it was around ~30 minutes to merge all samples from the train subset).
Probably it is not too useful for interactive kernels, but could be valuable for dataset preparation.
", "votes": null, "replies": [ { "id": 609522, "author_name": "michelml", "author_url": "", "post_date": "08/27/2019 19:56:15", "content": "If you check rio.combine_metadata in the github repo, you'll find that you can change the base_path for it to be a local one.
\ndef combine_metadata(base_path=DEFAULT_METADATA_BASE_PATH,\n include_controls=True):\n\nDEFAULT_METADATA_BASE_PATH is gs://rxrx1-us-central1/metadata indeed, but you can change it.
source: https://github.com/recursionpharma/rxrx1-utils/blob/master/rxrx/io.py#L233
", "votes": null, "replies": [] }, { "id": 609794, "author_name": "purplejester", "author_url": "", "post_date": "08/28/2019 05:35:07", "content": "Ah makes sense! Didn't check it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "607080": "Created [a simple script](https://www.kaggle.com/purplejester/joining-channels-into-rgb-images-parallel) that traverses the directory with the data and converts 6 standalone images into a single RGB image. (Using all available cores of your machine). Probably could be helpful for someone to start with the competition. Not sure how efficient it is to join 6 channels of data into 3 RGB channels, though. But a good start point, I believe.", "607447": "What is the advantage over using the script [provided by the host](https://github.com/recursionpharma/rxrx1-utils/blob/master/rxrx/io.py)?", "607677": "There is also a very good notebook by @xhlulu using the script provided by the host + additional code to perform resizing and saving images to an easily retrievable new path https://www.kaggle.com/xhlulu/recursion-2019-load-resize-and-save-images .", "608363": "Probably I've misunderstood, but the host script seems to pull the data from a remote bucket, not from the local files. And [this kernel](https://www.kaggle.com/xhlulu/recursion-2019-load-resize-and-save-images) doesn't use multi-processing, as I can see. On my machine, the parallel script gets approximately x12 speed-up. So probably someone could adapt this code for their need. Anyway, decided to share just in case :)", "609006": "Updated the script to use only local data without querying remote buckets provided by the host. Before it used `rio.combine_metadata`. Could be helpful if you run this script on your local machine, it should be capable to finish the processing within 1-2 hours, depending on the number of cores you have. (In my case, it was around ~30 minutes to merge all samples from the train subset).\n\nProbably it is not too useful for interactive kernels, but could be valuable for dataset preparation.", "609522": "If you check `rio.combine_metadata` in the github repo, you'll find that you can change the base_path for it to be a local one.\n\n```\ndef combine_metadata(base_path=DEFAULT_METADATA_BASE_PATH,\n include_controls=True):\n```\n`DEFAULT_METADATA_BASE_PATH` is `gs://rxrx1-us-central1/metadata` indeed, but you can change it.\n\nsource: https://github.com/recursionpharma/rxrx1-utils/blob/master/rxrx/io.py#L233", "609794": "Ah makes sense! Didn't check it." }, "source": "meta" }