Here's the train_images dataset converted to JPG. The original pixel dimensions are maintained.
\nhttps://www.kaggle.com/datasets/davidbroberts/hubmap-hpa-jpg
\nThis cuts the dataset down from ~8.7G -> ~800MB.
\nThe original TIFF files are 16 bit and could be exported with filters or bit shifts to get more data as well. Let me know if anyone has any ideas and I'll do some other exports.
", "messages": [ { "id": 1829920, "postDate": "2022-06-23T03:43:17.427Z", "content": "Here's the train_images dataset converted to JPG. The original pixel dimensions are maintained.
\nhttps://www.kaggle.com/datasets/davidbroberts/hubmap-hpa-jpg
\nThis cuts the dataset down from ~8.7G -> ~800MB.
\nThe original TIFF files are 16 bit and could be exported with filters or bit shifts to get more data as well. Let me know if anyone has any ideas and I'll do some other exports.
", "rawMarkdown": "Here's the train_images dataset converted to JPG. The original pixel dimensions are maintained.\n\nhttps://www.kaggle.com/datasets/davidbroberts/hubmap-hpa-jpg\n\nThis cuts the dataset down from ~8.7G -> ~800MB.\n\nThe original TIFF files are 16 bit and could be exported with filters or bit shifts to get more data as well. Let me know if anyone has any ideas and I'll do some other exports.", "votes": 28 }, { "id": 1830648, "postDate": "2022-06-23T15:05:21.993Z", "content": "Nice to have jpg to work with, files in jpg format are compressed and will lose some information compared to the tiff format though. Working with png might also be easier than tiff but will not be compressed, e.g no information lost.
", "rawMarkdown": "Nice to have jpg to work with, files in jpg format are compressed and will lose some information compared to the tiff format though. Working with png might also be easier than tiff but will not be compressed, e.g no information lost.", "votes": 3 }, { "id": 1830853, "postDate": "2022-06-23T18:10:32.090Z", "content": "Been there, done that π. Thank you David. I'll be waiting for yours remarkable Notebooks too.
", "rawMarkdown": "Been there, done that π. Thank you David. I'll be waiting for yours remarkable Notebooks too.", "votes": 2 }, { "id": 1831597, "postDate": "2022-06-24T09:40:24.163Z", "content": "Hi David,
\naccording to my EDA, using the tifffile lib the training tiffs are only 8 bit.
Thanks for pointing that out @friedchips, I looked at the file weight and assumed they were 16 bit as most TIFFs in the medical imaging field are. Now I wonder why they use TIFF format if they only needed 8 bits.
", "rawMarkdown": "Thanks for pointing that out @friedchips, I looked at the file weight and assumed they were 16 bit as most TIFFs in the medical imaging field are. Now I wonder why they use TIFF format if they only needed 8 bits." }, { "id": 1835177, "postDate": "2022-06-27T14:47:59.230Z", "content": "I looked into this and a few of the images are indeed 16 bit. The raw pixel values exceed 0-255 range, not by much, but they are bigger than 8 bit.
", "rawMarkdown": "I looked into this and a few of the images are indeed 16 bit. The raw pixel values exceed 0-255 range, not by much, but they are bigger than 8 bit.", "votes": 1 }, { "id": 1851748, "postDate": "2022-07-11T13:51:47.033Z", "content": "I'm using OpenCV for reading tiff files and all of them were 8 bit. I was reading them like this
\nimage = cv2.imread(image_filename, -1)
Any library suggestions for dealing with tiff files?
\nI tried using tifffile package for reading images but the outcome was same as OpenCV's imread. None of the images have pixel values greater than 255. OpenCV and tiffile imread methods return slightly different image arrays but their image mean values were exactly same. Some of the pixel values might deviate by 1 or 2 though.
", "rawMarkdown": "I'm using OpenCV for reading tiff files and all of them were 8 bit. I was reading them like this\n\n`image = cv2.imread(image_filename, -1)`\n\nAny library suggestions for dealing with tiff files?\n\nI tried using tifffile package for reading images but the outcome was same as OpenCV's imread. None of the images have pixel values greater than 255. OpenCV and tiffile imread methods return slightly different image arrays but their image mean values were exactly same. Some of the pixel values might deviate by 1 or 2 though." } ] }, { "id": 1833617, "postDate": "2022-06-26T07:11:28.927Z", "content": "Hi David, Could you share the process of how you did it? or a hint so we it can be used for other datasets too. This is already big help from you.
", "rawMarkdown": "Hi David, Could you share the process of how you did it? or a hint so we it can be used for other datasets too. This is already big help from you.", "replies": [ { "id": 1834475, "postDate": "2022-06-27T01:01:00.263Z", "content": "Sure, here's the code. It's really simple .. PIL does all the work, just open as tiff and save as jpg.
\nimport os\nfrom PIL import Image\n\nfor file in os.listdir(\"train_images\"):\n filename = os.fsdecode(file)\n f = filename[:-4]\n image = Image.open(\"train_images/\" + filename)\n image.save(\"train_images_jpg/\" + f + \"jpg\")\nYou are a genius. Thank you so much for your help.
", "rawMarkdown": "You are a genius. Thank you so much for your help." } ] }, { "id": 1832379, "postDate": "2022-06-25T02:14:50.527Z", "content": "Thank you David, it will help a lot
", "rawMarkdown": "Thank you David, it will help a lot", "isDeleted": true } ], "comments": [ { "id": 1830648, "author_name": "John Scanlan", "author_url": "", "post_date": "2022-06-23T15:05:21.993000", "content": "Nice to have jpg to work with, files in jpg format are compressed and will lose some information compared to the tiff format though. Working with png might also be easier than tiff but will not be compressed, e.g no information lost.
", "votes": 3, "replies": [] }, { "id": 1830853, "author_name": "MarΓlia Prata", "author_url": "", "post_date": "2022-06-23T18:10:32.090000", "content": "Been there, done that π. Thank you David. I'll be waiting for yours remarkable Notebooks too.
", "votes": 2, "replies": [] }, { "id": 1831597, "author_name": "Markus Frank", "author_url": "", "post_date": "2022-06-24T09:40:24.163000", "content": "Hi David,
\naccording to my EDA, using the tifffile lib the training tiffs are only 8 bit.
Thanks for pointing that out @friedchips, I looked at the file weight and assumed they were 16 bit as most TIFFs in the medical imaging field are. Now I wonder why they use TIFF format if they only needed 8 bits.
", "votes": 0, "replies": [] }, { "id": 1835177, "author_name": "David Roberts", "author_url": "", "post_date": "2022-06-27T14:47:59.230000", "content": "I looked into this and a few of the images are indeed 16 bit. The raw pixel values exceed 0-255 range, not by much, but they are bigger than 8 bit.
", "votes": 1, "replies": [] }, { "id": 1851748, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-07-11T13:51:47.033000", "content": "I'm using OpenCV for reading tiff files and all of them were 8 bit. I was reading them like this
\nimage = cv2.imread(image_filename, -1)
Any library suggestions for dealing with tiff files?
\nI tried using tifffile package for reading images but the outcome was same as OpenCV's imread. None of the images have pixel values greater than 255. OpenCV and tiffile imread methods return slightly different image arrays but their image mean values were exactly same. Some of the pixel values might deviate by 1 or 2 though.
", "votes": 0, "replies": [] } ] }, { "id": 1833617, "author_name": "Jims Chacko", "author_url": "", "post_date": "2022-06-26T07:11:28.927000", "content": "Hi David, Could you share the process of how you did it? or a hint so we it can be used for other datasets too. This is already big help from you.
", "votes": 0, "replies": [ { "id": 1834475, "author_name": "David Roberts", "author_url": "", "post_date": "2022-06-27T01:01:00.263000", "content": "Sure, here's the code. It's really simple .. PIL does all the work, just open as tiff and save as jpg.
\nimport os\nfrom PIL import Image\n\nfor file in os.listdir(\"train_images\"):\n filename = os.fsdecode(file)\n f = filename[:-4]\n image = Image.open(\"train_images/\" + filename)\n image.save(\"train_images_jpg/\" + f + \"jpg\")\nYou are a genius. Thank you so much for your help.
", "votes": 0, "replies": [] } ] }, { "id": 1832379, "author_name": "", "author_url": "", "post_date": "2022-06-25T02:14:50.527000", "content": "Thank you David, it will help a lot
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1829920": "Here's the train_images dataset converted to JPG. The original pixel dimensions are maintained.\n\nhttps://www.kaggle.com/datasets/davidbroberts/hubmap-hpa-jpg\n\nThis cuts the dataset down from ~8.7G -> ~800MB.\n\nThe original TIFF files are 16 bit and could be exported with filters or bit shifts to get more data as well. Let me know if anyone has any ideas and I'll do some other exports.", "1830648": "Nice to have jpg to work with, files in jpg format are compressed and will lose some information compared to the tiff format though. Working with png might also be easier than tiff but will not be compressed, e.g no information lost.", "1830853": "Been there, done that π. Thank you David. I'll be waiting for yours remarkable Notebooks too.", "1831597": "Hi David,\naccording to my EDA, using the tifffile lib the training tiffs are only 8 bit.", "1833617": "Hi David, Could you share the process of how you did it? or a hint so we it can be used for other datasets too. This is already big help from you.", "1832379": "Thank you David, it will help a lot" } }