Welcome to the HuBMAP: Hacking the Kidney competition!
\nWe are very much looking forward to the innovative solutions teams will generate to identify glomeruli in images of human kidney tissue.
\nTo help you get started, please check out the following notebooks:
\nDataset Details https://www.kaggle.com/leahscherschel/dataset-details
\n(Background Information and Specifications of the Dataset)
Glomeruli Detection Related Works https://www.kaggle.com/leahscherschel/glomeruli-detection-related-works
\n(Prior Work on Glomeruli Detections in Microscopy Images)
Polygon to Pixel Mask Conversion https://www.kaggle.com/leahscherschel/polygon-to-pixel-mask-conversion
\n(How to Convert Polygon Masks to Pixel Masks)
Run-Length Encoding https://www.kaggle.com/leahscherschel/run-length-encoding
\n(How to Perform Run-Length Encoding)
Our team will be monitoring the discussion board and look forward to hearing from you!
\nLeah
\nResearch Assistant
\nIndiana University
Welcome to the HuBMAP: Hacking the Kidney competition!
\nWe are very much looking forward to the innovative solutions teams will generate to identify glomeruli in images of human kidney tissue.
\nTo help you get started, please check out the following notebooks:
\nDataset Details https://www.kaggle.com/leahscherschel/dataset-details
\n(Background Information and Specifications of the Dataset)
Glomeruli Detection Related Works https://www.kaggle.com/leahscherschel/glomeruli-detection-related-works
\n(Prior Work on Glomeruli Detections in Microscopy Images)
Polygon to Pixel Mask Conversion https://www.kaggle.com/leahscherschel/polygon-to-pixel-mask-conversion
\n(How to Convert Polygon Masks to Pixel Masks)
Run-Length Encoding https://www.kaggle.com/leahscherschel/run-length-encoding
\n(How to Perform Run-Length Encoding)
Our team will be monitoring the discussion board and look forward to hearing from you!
\nLeah
\nResearch Assistant
\nIndiana University
Is There anatomical-structure.json file for private dataset? Can we use it in rerun?
", "rawMarkdown": "Is There anatomical-structure.json file for private dataset? Can we use it in rerun?", "votes": 6, "replies": [ { "id": 1089414, "postDate": "2020-11-24T13:30:35.357Z", "content": "@leahscherschel I'd like to know that as well. Could you please confirm if anatomical annotations are available for the private test set? They were provided for the public test set.
", "rawMarkdown": "@leahscherschel I'd like to know that as well. Could you please confirm if anatomical annotations are available for the private test set? They were provided for the public test set.", "votes": 1 }, { "id": 1089479, "postDate": "2020-11-24T14:31:26.017Z", "content": "Anatomical annotation .json files do exist as part of the private test dataset.
", "rawMarkdown": "Anatomical annotation .json files do exist as part of the private test dataset. ", "votes": 6 }, { "id": 1089482, "postDate": "2020-11-24T14:35:29.277Z", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": 2 }, { "id": 1099351, "postDate": "2020-12-02T09:44:27.240Z", "content": "@leahscherschel I did some probe, but it seems that there is no '-anatomical-structure.json' for the private test dataset. Could you confirm it again. Thank you very much.
", "rawMarkdown": "@leahscherschel I did some probe, but it seems that there is no '-anatomical-structure.json' for the private test dataset. Could you confirm it again. Thank you very much.", "votes": 3 } ] }, { "id": 1082168, "postDate": "2020-11-17T16:51:25.293Z", "content": "Hello. I looked over the paper by Kannan et. al, Segmentation of Glomeruli Within Trichrome Images Using Deep Learning. Do I understand it correctly that they did labeling first and then on the labeled images did a treshold to isolate the glomeruli?
", "rawMarkdown": "Hello. I looked over the paper by Kannan et. al, Segmentation of Glomeruli Within Trichrome Images Using Deep Learning. Do I understand it correctly that they did labeling first and then on the labeled images did a treshold to isolate the glomeruli?", "votes": 3, "replies": [ { "id": 1082227, "postDate": "2020-11-17T18:04:20.627Z", "content": "Yes, I believe that to be the essence of the approach. They used the CNN with a sliding window to create a \"heatmap\" of how confident the model was in detecting the presence of a globally sclerosed glomerulus. Then, several processing steps (including Otsu binarization with a manually set threshold value) were implemented to create the rectangular bounding boxes.
", "rawMarkdown": "Yes, I believe that to be the essence of the approach. They used the CNN with a sliding window to create a \"heatmap\" of how confident the model was in detecting the presence of a globally sclerosed glomerulus. Then, several processing steps (including Otsu binarization with a manually set threshold value) were implemented to create the rectangular bounding boxes.", "votes": 4 } ] }, { "id": 1082250, "postDate": "2020-11-17T18:27:04.527Z", "content": "This notebook may also be helpful!
\nHuBMAP: Read Data and Build TFRecords
\nhttps://www.kaggle.com/marcosnovaes/hubmap-read-data-and-build-tfrecords
Reposting this question here:
\n**## Presentation of submission
\n**
Hi Kagglers,
\nHope you can help with a number of queries I have with regards to the presentation for this competition? 😊
\nIn the description provided, it states that \"submissions will have the opportunity to make a presentation to the judge's panel to be evaluated on the below criteria\", which are as follows:
\nMethodology (50 points) [Scientific Prize]
\nInnovation and Applications (30 points) [Innovation Prize]
\nDiversity and Presentation (30 points) [Diversity Prize]
Does this imply we are able to submit three short presentations to cover the respective criteria? Or should we submit a single presentation covering all three criteria?
\nSecondly, is there a preferred format of presentation (example: PowerPoint, poster)?
\nLastly, is there a limit to the number of slides to a presentation or number of pages for a poster presentation?
\nThanks!
@leahscherschel
\nThank you for hosting this competition. I have also enjoyed reading the relevant papers you have posted so far.
By the way, when Kaggle allows external data, it is customary to start an external data thread. Could you please start an external data thread and pin it to the top of Discussion?
Hey Maxwell,
\nWe no longer require an external data thread for competitions that allow external data. However, the qualifications for external data to be permitted are still unchanged. See the rules for more details.
\nThanks!
", "rawMarkdown": "Hey Maxwell,\n\nWe no longer require an external data thread for competitions that allow external data. However, the qualifications for external data to be permitted are still unchanged. See the rules for more details.\n\nThanks!" } ] }, { "id": 1180817, "postDate": "2021-02-01T13:39:46.180Z", "content": "First, I want to thank Google and its partners for providing and supporting Kaggle and the HuBMAP competition; I'm learning a lot both from my mistakes and the excellent work shared by others. However, I would like to register a complaint about the (lack of) feedback from code submissions. In addition to the public score, the only other bit of information is a terse status phrase like \"Submission Timed Out\" or, worst of all, \"Submission Scoring Error\".
\nWhat I've observed in my work is that I'm left doing \"probe\" submissions (19 so far) to try to decipher what aspect of my code is causing the Scoring Error. While I understand that in the real world, you don't get to dictate the constraints of a problem, in this case, I feel that the lack of even minimal feedback, something like \"invalid submission.csv for inputs #2 and #3\", means that I am spending my efforts trying to infer what I'm doing wrong vs. more interesting and useful experimenting with different ML approaches in an effort to best solve the glom-segmentation problem.
\nPerhaps something like the following would help: Provide a function \"report_status\" that could be called anywhere in our application code, which would print into the log a distinctive message like \"STATUS: Virtual Memory Usage xxxx, Virtual CPU Time yyyy\". This would at least help diagnose OOM problems and execution time hotspots. From my own experience, getting a little bit more detail about the \"Submission Scoring Error\" would be very helpful.
\nThanks for your consideration,
\n-- Mark
Try this one:
\nimport os, psutil \n\ndef report_status():\n py = psutil.Process(os.getpid())\n memory_use = py.memory_info()[0] / 2.**30\n print('Status: Memory: {} GByte, CPU load: {}%, Disk use: {}%'.format(str(round(memory_use, 2)), psutil.cpu_percent(), psutil.disk_usage('/kaggle/working').percent))\n\nreport_status()\nGood information, but for a Submission, I don't know how I'd be able to see it. It requires a (small) change in the Submission/Scoring apparatus.
", "rawMarkdown": "Good information, but for a Submission, I don't know how I'd be able to see it. It requires a (small) change in the Submission/Scoring apparatus." } ] }, { "id": 1152826, "postDate": "2021-01-14T13:27:57.823Z", "content": "@leahscherschel @addisonhoward @juliaelliott
\nI have one question. Can we use other external data and/or pre-trained machine learning models that released under CC BY 3.0 or 2.0. In my understanding, they are upward compatible with the 4.0 versions, so treating them the same as 4.0 should be fine.
Hello @leahscherschel,
\nThank you for sharing these resources. Can you please help with adding a team member? I would like to add another person (so far we are only 2 now in the team) but I can't seem to.
\nThank you.
Hi! This would be a question for the Kaggle team.
", "rawMarkdown": "Hi! This would be a question for the Kaggle team.", "votes": 1 }, { "id": 1129957, "postDate": "2020-12-28T17:09:34.327Z", "content": "@easara You can add a team member by first ensuring all team members review & accept the competition's rules. Then, review the instructions on our Competition docs for \"Forming a Team\".
", "rawMarkdown": "@easara You can add a team member by first ensuring all team members review & accept the competition's rules. Then, review the instructions on our Competition docs for [\"Forming a Team\"](https://www.kaggle.com/docs/competitions#forming-a-team).", "votes": 1 }, { "id": 1134935, "postDate": "2021-01-01T17:57:16.113Z", "content": "Thank you @leahscherschel and @juliaelliott .
", "rawMarkdown": "Thank you @leahscherschel and @juliaelliott ." } ] }, { "id": 1121445, "postDate": "2020-12-21T16:38:22.443Z", "content": "In regards to using internet, are we allowed to train using internet? But, we can not submit with internet enabled?
", "rawMarkdown": "In regards to using internet, are we allowed to train using internet? But, we can not submit with internet enabled? ", "replies": [ { "id": 1121601, "postDate": "2020-12-21T19:02:24.440Z", "content": "@dskswu Yes, that's right.
", "rawMarkdown": "@dskswu Yes, that's right." } ] }, { "id": 1096903, "postDate": "2020-11-30T21:32:15.457Z", "content": "Hi, I have a question about how the code competition works. Does my submitted kernel need to train all models used, or is it acceptable to upload a previously trained model (not publicly accessible, trained on my own machine) and use that for evaluation?
", "rawMarkdown": "Hi, I have a question about how the code competition works. Does my submitted kernel need to train all models used, or is it acceptable to upload a previously trained model (not publicly accessible, trained on my own machine) and use that for evaluation?", "replies": [ { "id": 1096911, "postDate": "2020-11-30T21:45:06.730Z", "content": "You may upload a previously trained model for evaluation. However should you win the competition you will be required to provide all models to the host team for validation.
", "rawMarkdown": "You may upload a previously trained model for evaluation. However should you win the competition you will be required to provide all models to the host team for validation." } ] }, { "id": 1090775, "postDate": "2020-11-25T15:22:12.833Z", "content": "Is there a data dictionary for the meta-data file? Or is that something we would have to try research?
", "rawMarkdown": "Is there a data dictionary for the meta-data file? Or is that something we would have to try research?" }, { "id": 1088391, "postDate": "2020-11-23T15:12:45.703Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1283522, "postDate": "2021-04-25T02:36:04.783Z", "content": "Thank you.
", "rawMarkdown": "Thank you." } ], "comments": [ { "id": 1082731, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-11-18T07:12:04.610000", "content": "Is There anatomical-structure.json file for private dataset? Can we use it in rerun?
", "votes": 6, "replies": [ { "id": 1089414, "author_name": "Renan Rodrigues dos Santos", "author_url": "", "post_date": "2020-11-24T13:30:35.357000", "content": "@leahscherschel I'd like to know that as well. Could you please confirm if anatomical annotations are available for the private test set? They were provided for the public test set.
", "votes": 1, "replies": [] }, { "id": 1089479, "author_name": "Leah Godwin", "author_url": "", "post_date": "2020-11-24T14:31:26.017000", "content": "Anatomical annotation .json files do exist as part of the private test dataset.
", "votes": 6, "replies": [] }, { "id": 1089482, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-11-24T14:35:29.277000", "content": "Thank you.
", "votes": 2, "replies": [] }, { "id": 1099351, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-12-02T09:44:27.240000", "content": "@leahscherschel I did some probe, but it seems that there is no '-anatomical-structure.json' for the private test dataset. Could you confirm it again. Thank you very much.
", "votes": 3, "replies": [] } ] }, { "id": 1082168, "author_name": "Adrian Muresan", "author_url": "", "post_date": "2020-11-17T16:51:25.293000", "content": "Hello. I looked over the paper by Kannan et. al, Segmentation of Glomeruli Within Trichrome Images Using Deep Learning. Do I understand it correctly that they did labeling first and then on the labeled images did a treshold to isolate the glomeruli?
", "votes": 3, "replies": [ { "id": 1082227, "author_name": "Leah Godwin", "author_url": "", "post_date": "2020-11-17T18:04:20.627000", "content": "Yes, I believe that to be the essence of the approach. They used the CNN with a sliding window to create a \"heatmap\" of how confident the model was in detecting the presence of a globally sclerosed glomerulus. Then, several processing steps (including Otsu binarization with a manually set threshold value) were implemented to create the rectangular bounding boxes.
", "votes": 4, "replies": [] } ] }, { "id": 1082250, "author_name": "Leah Godwin", "author_url": "", "post_date": "2020-11-17T18:27:04.527000", "content": "This notebook may also be helpful!
\nHuBMAP: Read Data and Build TFRecords
\nhttps://www.kaggle.com/marcosnovaes/hubmap-read-data-and-build-tfrecords
Reposting this question here:
\n**## Presentation of submission
\n**
Hi Kagglers,
\nHope you can help with a number of queries I have with regards to the presentation for this competition? 😊
\nIn the description provided, it states that \"submissions will have the opportunity to make a presentation to the judge's panel to be evaluated on the below criteria\", which are as follows:
\nMethodology (50 points) [Scientific Prize]
\nInnovation and Applications (30 points) [Innovation Prize]
\nDiversity and Presentation (30 points) [Diversity Prize]
Does this imply we are able to submit three short presentations to cover the respective criteria? Or should we submit a single presentation covering all three criteria?
\nSecondly, is there a preferred format of presentation (example: PowerPoint, poster)?
\nLastly, is there a limit to the number of slides to a presentation or number of pages for a poster presentation?
\nThanks!
@leahscherschel
\nThank you for hosting this competition. I have also enjoyed reading the relevant papers you have posted so far.
By the way, when Kaggle allows external data, it is customary to start an external data thread. Could you please start an external data thread and pin it to the top of Discussion?
Hey Maxwell,
\nWe no longer require an external data thread for competitions that allow external data. However, the qualifications for external data to be permitted are still unchanged. See the rules for more details.
\nThanks!
", "votes": 0, "replies": [] } ] }, { "id": 1180817, "author_name": "Mark A Lavin", "author_url": "", "post_date": "2021-02-01T13:39:46.180000", "content": "First, I want to thank Google and its partners for providing and supporting Kaggle and the HuBMAP competition; I'm learning a lot both from my mistakes and the excellent work shared by others. However, I would like to register a complaint about the (lack of) feedback from code submissions. In addition to the public score, the only other bit of information is a terse status phrase like \"Submission Timed Out\" or, worst of all, \"Submission Scoring Error\".
\nWhat I've observed in my work is that I'm left doing \"probe\" submissions (19 so far) to try to decipher what aspect of my code is causing the Scoring Error. While I understand that in the real world, you don't get to dictate the constraints of a problem, in this case, I feel that the lack of even minimal feedback, something like \"invalid submission.csv for inputs #2 and #3\", means that I am spending my efforts trying to infer what I'm doing wrong vs. more interesting and useful experimenting with different ML approaches in an effort to best solve the glom-segmentation problem.
\nPerhaps something like the following would help: Provide a function \"report_status\" that could be called anywhere in our application code, which would print into the log a distinctive message like \"STATUS: Virtual Memory Usage xxxx, Virtual CPU Time yyyy\". This would at least help diagnose OOM problems and execution time hotspots. From my own experience, getting a little bit more detail about the \"Submission Scoring Error\" would be very helpful.
\nThanks for your consideration,
\n-- Mark
Try this one:
\nimport os, psutil \n\ndef report_status():\n py = psutil.Process(os.getpid())\n memory_use = py.memory_info()[0] / 2.**30\n print('Status: Memory: {} GByte, CPU load: {}%, Disk use: {}%'.format(str(round(memory_use, 2)), psutil.cpu_percent(), psutil.disk_usage('/kaggle/working').percent))\n\nreport_status()\nGood information, but for a Submission, I don't know how I'd be able to see it. It requires a (small) change in the Submission/Scoring apparatus.
", "votes": 0, "replies": [] } ] }, { "id": 1152826, "author_name": "hirune924", "author_url": "", "post_date": "2021-01-14T13:27:57.823000", "content": "@leahscherschel @addisonhoward @juliaelliott
\nI have one question. Can we use other external data and/or pre-trained machine learning models that released under CC BY 3.0 or 2.0. In my understanding, they are upward compatible with the 4.0 versions, so treating them the same as 4.0 should be fine.
Hello @leahscherschel,
\nThank you for sharing these resources. Can you please help with adding a team member? I would like to add another person (so far we are only 2 now in the team) but I can't seem to.
\nThank you.
Hi! This would be a question for the Kaggle team.
", "votes": 1, "replies": [] }, { "id": 1129957, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-12-28T17:09:34.327000", "content": "@easara You can add a team member by first ensuring all team members review & accept the competition's rules. Then, review the instructions on our Competition docs for \"Forming a Team\".
", "votes": 1, "replies": [] }, { "id": 1134935, "author_name": "EL-ATEIF Sara", "author_url": "", "post_date": "2021-01-01T17:57:16.113000", "content": "Thank you @leahscherschel and @juliaelliott .
", "votes": 0, "replies": [] } ] }, { "id": 1121445, "author_name": "William Green", "author_url": "", "post_date": "2020-12-21T16:38:22.443000", "content": "In regards to using internet, are we allowed to train using internet? But, we can not submit with internet enabled?
", "votes": 0, "replies": [ { "id": 1121601, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-12-21T19:02:24.440000", "content": "@dskswu Yes, that's right.
", "votes": 0, "replies": [] } ] }, { "id": 1096903, "author_name": "David Hammond", "author_url": "", "post_date": "2020-11-30T21:32:15.457000", "content": "Hi, I have a question about how the code competition works. Does my submitted kernel need to train all models used, or is it acceptable to upload a previously trained model (not publicly accessible, trained on my own machine) and use that for evaluation?
", "votes": 0, "replies": [ { "id": 1096911, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-11-30T21:45:06.730000", "content": "You may upload a previously trained model for evaluation. However should you win the competition you will be required to provide all models to the host team for validation.
", "votes": 0, "replies": [] } ] }, { "id": 1090775, "author_name": "William Green", "author_url": "", "post_date": "2020-11-25T15:22:12.833000", "content": "Is there a data dictionary for the meta-data file? Or is that something we would have to try research?
", "votes": 0, "replies": [] }, { "id": 1088391, "author_name": "", "author_url": "", "post_date": "2020-11-23T15:12:45.703000", "content": "", "votes": 0, "replies": [] }, { "id": 1283522, "author_name": "Elle", "author_url": "", "post_date": "2021-04-25T02:36:04.783000", "content": "Thank you.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1082155": "Welcome to the HuBMAP: Hacking the Kidney competition! \n\nWe are very much looking forward to the innovative solutions teams will generate to identify glomeruli in images of human kidney tissue. \n\nTo help you get started, please check out the following notebooks:\n\nDataset Details https://www.kaggle.com/leahscherschel/dataset-details\n(Background Information and Specifications of the Dataset)\n\nGlomeruli Detection Related Works https://www.kaggle.com/leahscherschel/glomeruli-detection-related-works\n(Prior Work on Glomeruli Detections in Microscopy Images)\n\nPolygon to Pixel Mask Conversion https://www.kaggle.com/leahscherschel/polygon-to-pixel-mask-conversion\n(How to Convert Polygon Masks to Pixel Masks)\n\nRun-Length Encoding https://www.kaggle.com/leahscherschel/run-length-encoding\n(How to Perform Run-Length Encoding)\n\nOur team will be monitoring the discussion board and look forward to hearing from you!\n\nLeah\nResearch Assistant\nIndiana University", "1082731": "Is There anatomical-structure.json file for private dataset? Can we use it in rerun?", "1082168": "Hello. I looked over the paper by Kannan et. al, Segmentation of Glomeruli Within Trichrome Images Using Deep Learning. Do I understand it correctly that they did labeling first and then on the labeled images did a treshold to isolate the glomeruli?", "1082250": "This notebook may also be helpful!\nHuBMAP: Read Data and Build TFRecords\nhttps://www.kaggle.com/marcosnovaes/hubmap-read-data-and-build-tfrecords", "1186179": "Reposting this question here:\n\n**## Presentation of submission\n**\n\nHi Kagglers,\n\nHope you can help with a number of queries I have with regards to the presentation for this competition? 😊\n\nIn the description provided, it states that \"submissions will have the opportunity to make a presentation to the judge's panel to be evaluated on the below criteria\", which are as follows:\nMethodology (50 points) [Scientific Prize]\nInnovation and Applications (30 points) [Innovation Prize]\nDiversity and Presentation (30 points) [Diversity Prize]\n\nDoes this imply we are able to submit three short presentations to cover the respective criteria? Or should we submit a single presentation covering all three criteria?\nSecondly, is there a preferred format of presentation (example: PowerPoint, poster)?\nLastly, is there a limit to the number of slides to a presentation or number of pages for a poster presentation?\nThanks!", "1094649": "@leahscherschel \nThank you for hosting this competition. I have also enjoyed reading the relevant papers you have posted so far.\n\nBy the way, when Kaggle allows external data, it is customary to start **an external data thread**. Could you please start an external data thread and pin it to the top of `Discussion`?\n", "1180817": "First, I want to thank Google and its partners for providing and supporting Kaggle and the HuBMAP competition; I'm learning a lot both from my mistakes and the excellent work shared by others. However, I would like to register a complaint about the (lack of) feedback from code submissions. In addition to the public score, the only other bit of information is a terse status phrase like \"Submission Timed Out\" or, worst of all, \"Submission Scoring Error\".\n\nWhat I've observed in my work is that I'm left doing \"probe\" submissions (19 so far) to try to decipher what aspect of my code is causing the Scoring Error. While I understand that in the real world, you don't get to dictate the constraints of a problem, in this case, I feel that the lack of even minimal feedback, something like \"invalid submission.csv for inputs #2 and #3\", means that I am spending my efforts trying to infer what I'm doing wrong vs. more interesting and useful experimenting with different ML approaches in an effort to best solve the glom-segmentation problem.\n\nPerhaps something like the following would help: Provide a function \"report_status\" that could be called anywhere in our application code, which would print into the log a distinctive message like \"STATUS: Virtual Memory Usage xxxx, Virtual CPU Time yyyy\". This would at least help diagnose OOM problems and execution time hotspots. From my own experience, getting a little bit more detail about the \"Submission Scoring Error\" would be very helpful.\n\nThanks for your consideration,\n-- Mark", "1152826": "@leahscherschel @addisonhoward @juliaelliott \nI have one question. Can we use other external data and/or pre-trained machine learning models that released under CC BY 3.0 or 2.0. In my understanding, they are upward compatible with the 4.0 versions, so treating them the same as 4.0 should be fine. ", "1127735": "Hello @leahscherschel,\nThank you for sharing these resources. Can you please help with adding a team member? I would like to add another person (so far we are only 2 now in the team) but I can't seem to.\nThank you.", "1121445": "In regards to using internet, are we allowed to train using internet? But, we can not submit with internet enabled? ", "1096903": "Hi, I have a question about how the code competition works. Does my submitted kernel need to train all models used, or is it acceptable to upload a previously trained model (not publicly accessible, trained on my own machine) and use that for evaluation?", "1090775": "Is there a data dictionary for the meta-data file? Or is that something we would have to try research?", "1088391": "", "1283522": "Thank you." } }