Thanks for providing access to the HPA public data! Can you share what's the relationship between public data and the official train set? Is there any overlap?
\nWe'd need to be more careful adding the public data for training in case of a possible overlap.
", "messages": [ { "id": 1215927, "postDate": "2021-02-24T03:57:48.733Z", "content": "\nThanks for providing access to the HPA public data! Can you share what's the relationship between public data and the official train set? Is there any overlap?
\nWe'd need to be more careful adding the public data for training in case of a possible overlap.
", "rawMarkdown": "@lnhtrang @emmalumpan \n\nThanks for providing access to the [HPA public data](https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg/)! Can you share what's the relationship between public data and the official train set? Is there any overlap? \n\nWe'd need to be more careful adding the public data for training in case of a possible overlap.", "votes": 7 }, { "id": 1218996, "postDate": "2021-02-26T10:57:22.223Z", "content": "There are 20% (of trainset) overlap with public HPA. I will update the list to let you know exactly which images to avoid double download.
", "rawMarkdown": "There are 20% (of trainset) overlap with public HPA. I will update the list to let you know exactly which images to avoid double download. ", "votes": 4, "replies": [ { "id": 1219001, "postDate": "2021-02-26T11:03:33.620Z", "content": "Thank you for the response, this is very much appreciated!
", "rawMarkdown": "Thank you for the response, this is very much appreciated!" }, { "id": 1219213, "postDate": "2021-02-26T14:57:08.463Z", "content": "I have updated.
", "rawMarkdown": "I have updated.", "votes": 3 }, { "id": 1219375, "postDate": "2021-02-26T18:49:07.183Z", "content": "Hosts for this competition have been really great at responding to questions and adding clarification. Keep up the great work - thanks for updating the list.
", "rawMarkdown": "Hosts for this competition have been really great at responding to questions and adding clarification. Keep up the great work - thanks for updating the list." }, { "id": 1219519, "postDate": "2021-02-26T22:57:01.077Z", "content": "Thanks for the update, still 2.4TB but it will help.
", "rawMarkdown": "Thanks for the update, still 2.4TB but it will help." } ] }, { "id": 1218372, "postDate": "2021-02-25T19:12:22.570Z", "content": "Looked on HPA data last night to see how many images might be available for our smallest class, the mitotic spindle. Always a good chance I goofed - but looks like there should be around 200 images. That seems like it gives a huge opportunity for overlap.
", "rawMarkdown": "Looked on HPA data last night to see how many images might be available for our smallest class, the mitotic spindle. Always a good chance I goofed - but looks like there should be around 200 images. That seems like it gives a huge opportunity for overlap.\n\n", "votes": 2, "replies": [ { "id": 1221997, "postDate": "2021-03-01T13:12:01.100Z", "content": "From what I can see, there is legitimately only a single cell (where we can know from a given single class slide level label) in the current training data exhibiting the mitotic spindle class. It's terrific to know that there are many more images available. I know the competition hosts are trimming and converting the public data into a Kaggle dataset, so hopefully, we will see a boost to the Public LB scores.
", "rawMarkdown": "From what I can see, there is legitimately only a single cell (where we can know from a given single class slide level label) in the current training data exhibiting the **mitotic spindle** class. It's terrific to know that there are many more images available. I know the competition hosts are trimming and converting the public data into a Kaggle dataset, so hopefully, we will see a boost to the Public LB scores.", "votes": 1 }, { "id": 1222365, "postDate": "2021-03-01T18:14:07.300Z", "content": "I skipped biology in High School so my eyeball is not very calibrated but I agree that I see only one single cell image that shows mitotic spindle for sure in my set of single cell images. I am currently ignoring any single cells that touch the orginal image edge so there might be a few more but…
\nFor the present I have decided that chasing after improvement for this class may be a very no value added activity. Putting this improvment on the TODO list after I hit LB score of 0.50 or better.
", "rawMarkdown": "I skipped biology in High School so my eyeball is not very calibrated but I agree that I see only one single cell image that shows mitotic spindle for sure in my set of single cell images. I am currently ignoring any single cells that touch the orginal image edge so there might be a few more but...\n\nFor the present I have decided that chasing after improvement for this class may be a very no value added activity. Putting this improvment on the TODO list after I hit LB score of 0.50 or better.", "votes": 1 } ] }, { "id": 1284858, "postDate": "2021-04-26T10:39:12.320Z", "content": "@lnhtrang Thanks for the response
\nWe see that in public data sets the pixel values ranges from 0 to 65535. And when we normalize it to 0 to 255 , we are not able to visualize the image.
what should we do in this case should we clip it to max of 255 or normalize it to 0 to 255??
", "rawMarkdown": "@lnhtrang Thanks for the response\nWe see that in public data sets the pixel values ranges from 0 to 65535. And when we normalize it to 0 to 255 , we are not able to visualize the image.\n\nwhat should we do in this case should we clip it to max of 255 or normalize it to 0 to 255??", "replies": [ { "id": 1284886, "postDate": "2021-04-26T11:22:54.213Z", "content": "You can check the range and type of the data: [0,255] is uint8, or [0,65535] is uint16. Please check when you normalized data to the 255 range, did you also change the data type to uint8? Otherwise Python will just understand that you reduce the signal strength by 256 times.
", "rawMarkdown": "You can check the range and type of the data: [0,255] is uint8, or [0,65535] is uint16. Please check when you normalized data to the 255 range, did you also change the data type to uint8? Otherwise Python will just understand that you reduce the signal strength by 256 times." } ] } ], "comments": [ { "id": 1218996, "author_name": "Trang Le", "author_url": "", "post_date": "2021-02-26T10:57:22.223000", "content": "There are 20% (of trainset) overlap with public HPA. I will update the list to let you know exactly which images to avoid double download.
", "votes": 4, "replies": [ { "id": 1219001, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-02-26T11:03:33.620000", "content": "Thank you for the response, this is very much appreciated!
", "votes": 0, "replies": [] }, { "id": 1219213, "author_name": "Trang Le", "author_url": "", "post_date": "2021-02-26T14:57:08.463000", "content": "I have updated.
", "votes": 3, "replies": [] }, { "id": 1219375, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-26T18:49:07.183000", "content": "Hosts for this competition have been really great at responding to questions and adding clarification. Keep up the great work - thanks for updating the list.
", "votes": 0, "replies": [] }, { "id": 1219519, "author_name": "MPWARE", "author_url": "", "post_date": "2021-02-26T22:57:01.077000", "content": "Thanks for the update, still 2.4TB but it will help.
", "votes": 0, "replies": [] } ] }, { "id": 1218372, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-25T19:12:22.570000", "content": "Looked on HPA data last night to see how many images might be available for our smallest class, the mitotic spindle. Always a good chance I goofed - but looks like there should be around 200 images. That seems like it gives a huge opportunity for overlap.
", "votes": 2, "replies": [ { "id": 1221997, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-01T13:12:01.100000", "content": "From what I can see, there is legitimately only a single cell (where we can know from a given single class slide level label) in the current training data exhibiting the mitotic spindle class. It's terrific to know that there are many more images available. I know the competition hosts are trimming and converting the public data into a Kaggle dataset, so hopefully, we will see a boost to the Public LB scores.
", "votes": 1, "replies": [] }, { "id": 1222365, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-03-01T18:14:07.300000", "content": "I skipped biology in High School so my eyeball is not very calibrated but I agree that I see only one single cell image that shows mitotic spindle for sure in my set of single cell images. I am currently ignoring any single cells that touch the orginal image edge so there might be a few more but…
\nFor the present I have decided that chasing after improvement for this class may be a very no value added activity. Putting this improvment on the TODO list after I hit LB score of 0.50 or better.
", "votes": 1, "replies": [] } ] }, { "id": 1284858, "author_name": "Shubham Joshi", "author_url": "", "post_date": "2021-04-26T10:39:12.320000", "content": "@lnhtrang Thanks for the response
\nWe see that in public data sets the pixel values ranges from 0 to 65535. And when we normalize it to 0 to 255 , we are not able to visualize the image.
what should we do in this case should we clip it to max of 255 or normalize it to 0 to 255??
", "votes": 0, "replies": [ { "id": 1284886, "author_name": "Trang Le", "author_url": "", "post_date": "2021-04-26T11:22:54.213000", "content": "You can check the range and type of the data: [0,255] is uint8, or [0,65535] is uint16. Please check when you normalized data to the 255 range, did you also change the data type to uint8? Otherwise Python will just understand that you reduce the signal strength by 256 times.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1215927": "@lnhtrang @emmalumpan \n\nThanks for providing access to the [HPA public data](https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg/)! Can you share what's the relationship between public data and the official train set? Is there any overlap? \n\nWe'd need to be more careful adding the public data for training in case of a possible overlap.", "1218996": "There are 20% (of trainset) overlap with public HPA. I will update the list to let you know exactly which images to avoid double download. ", "1218372": "Looked on HPA data last night to see how many images might be available for our smallest class, the mitotic spindle. Always a good chance I goofed - but looks like there should be around 200 images. That seems like it gives a huge opportunity for overlap.\n\n", "1284858": "@lnhtrang Thanks for the response\nWe see that in public data sets the pixel values ranges from 0 to 65535. And when we normalize it to 0 to 255 , we are not able to visualize the image.\n\nwhat should we do in this case should we clip it to max of 255 or normalize it to 0 to 255??" } }