Firstly I would like to thank all the competitors for their work and interest in this competition, particularly to those who have provided additional annotations or re-annotations and have already made them available to other competitors.
\nThere have been some questions from competitors regarding hand-labelled annotations. To clarify the rules as per section A2:
\n\n\nPublicly, freely available external data is permitted. Entrants may re-annotate images in the training set, however Entrants will (i) ensure the re-annotated data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the re-annotated data for the participants to the official competition forum prior to the Entry Deadline. Entrants may not hand-label predictions in the test data set, including having human observers rate and evaluate the test data set.
\n
This means that competitors may hand-label any images in the training set or in any external dataset including CXR14, as long as any labels used during training are posted in this official forum prior to the Entry deadline of 8th March 2021. This encourages competitors to add to the growing body of data available on this very important topic.
Any information that is used by your model during training that is not derived from the official training dataset that is not already publicly available, will be considered hand-labelled annotation. Any hand-labelled annotation, on external data or otherwise, should be posted in this official forum. This includes annotations that are not directly corresponding to the official labels, such as additional pixel-wise segmentations, or coordinate locations of other important anatomical locations such as the cavoatrial junction or tracheal bifurcation.
While posting in any part of the forum is technically acceptable as per the rules, I would appreciate if competitors are able to post on this thread to centralize this data so other competitors can find it more easily.
Firstly I would like to thank all the competitors for their work and interest in this competition, particularly to those who have provided additional annotations or re-annotations and have already made them available to other competitors.
\nThere have been some questions from competitors regarding hand-labelled annotations. To clarify the rules as per section A2:
\n\n\nPublicly, freely available external data is permitted. Entrants may re-annotate images in the training set, however Entrants will (i) ensure the re-annotated data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the re-annotated data for the participants to the official competition forum prior to the Entry Deadline. Entrants may not hand-label predictions in the test data set, including having human observers rate and evaluate the test data set.
\n
This means that competitors may hand-label any images in the training set or in any external dataset including CXR14, as long as any labels used during training are posted in this official forum prior to the Entry deadline of 8th March 2021. This encourages competitors to add to the growing body of data available on this very important topic.
Any information that is used by your model during training that is not derived from the official training dataset that is not already publicly available, will be considered hand-labelled annotation. Any hand-labelled annotation, on external data or otherwise, should be posted in this official forum. This includes annotations that are not directly corresponding to the official labels, such as additional pixel-wise segmentations, or coordinate locations of other important anatomical locations such as the cavoatrial junction or tracheal bifurcation.
While posting in any part of the forum is technically acceptable as per the rules, I would appreciate if competitors are able to post on this thread to centralize this data so other competitors can find it more easily.
I appreciate this being formally clarified, but I really don't like new data being made available only a few days before the end of the competition. In the future it might be good to set some earlier deadline for data share.
", "rawMarkdown": "I appreciate this being formally clarified, but I really don't like new data being made available only a few days before the end of the competition. In the future it might be good to set some earlier deadline for data share. ", "votes": 11, "replies": [ { "id": 1221119, "postDate": "2021-02-28T17:22:09.077Z", "content": "The old External Data Thread is back! I miss the times when they were mandatory.
\n\n\nbut I really don't like new data being made available only a few days before the end of the competition
\n
Earlier deadline would discourage people from hand labeling. This deadline is a nice compromise between \"whoever hand labels more - wins\" for no disclosure at all and \"why bother to hand label at all, everyone would have plenty of time to include it\" for early disclosure.
\nIf competition host wants to get the best model, then early disclosure is good. If he wants to get the best solution, then late disclosure is probably better.
", "rawMarkdown": "The old External Data Thread is back! I miss the times when they were mandatory.\n\n> but I really don't like new data being made available only a few days before the end of the competition\n\nEarlier deadline would discourage people from hand labeling. This deadline is a nice compromise between \"whoever hand labels more - wins\" for no disclosure at all and \"why bother to hand label at all, everyone would have plenty of time to include it\" for early disclosure.\n\nIf competition host wants to get the best model, then early disclosure is good. If he wants to get the best solution, then late disclosure is probably better.", "votes": 2 } ] }, { "id": 1229627, "postDate": "2021-03-07T13:48:54.807Z", "content": "We have been using available NIH data in our models
\nhttps://www.kaggle.com/nih-chest-xrays/data
also I have collected some images which contain tubes from other open source datasets as well.
\nhttps://www.kaggle.com/raddar/mimic-xray-ett-ngt-cvc
\nhttps://www.kaggle.com/raddar/padchest-tubes
@raddar I think mmic dataset requires an exam which I think goes against publicly available dataset rule.
", "rawMarkdown": "@raddar I think mmic dataset requires an exam which I think goes against publicly available dataset rule.", "votes": 8 }, { "id": 1230292, "postDate": "2021-03-08T01:26:54.310Z", "content": "@jarrelscy
\nIs MIMIC dataset (https://mimic.physionet.org/) allowed in this competition?
\nI personally believe it is in accordance with competition rules.
There were some discussions on this topic before:
\nBut we did not receive any answer from organisers.
\nNow the entry deadline is close, and we all all need a quick response.
\nThank you in advance.
Are you sure you are allowed to host this data here?
\nTaken from: https://mimic.physionet.org/gettingstarted/access/
\n\n\nbut we expect you to take matters of security seriously and to make sure that the data are accessible to no one but yourself.
\n
Don't think this data is openly available, as you also need to take an exam.
", "rawMarkdown": "Are you sure you are allowed to host this data here?\n\nTaken from: https://mimic.physionet.org/gettingstarted/access/\n> but we expect you to take matters of security seriously and to make sure that the data are accessible to no one but yourself.\n\nDon't think this data is openly available, as you also need to take an exam.", "votes": 6 }, { "id": 1231071, "postDate": "2021-03-08T16:39:30.283Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1231172, "postDate": "2021-03-08T18:14:35.527Z", "content": "@raddar
\n1) mimic dataset link u gave belong to which class.
\n2) is it subset of NiH dataset ?
After strong consideration and reading MIMIC licence many dozens of times, I found it vague and subject to interpretation. However, considering the sensitivity of the dataset I decided to remove it's derivative from kaggle datasets.
\nOn top of that I would not entirely agree that MIMIC is not \"publicly available\". Getting access takes less than 2 minutes and is free of charge and in my view complies with competition rules.
\nAll in all, this dataset did have a small impact to our score - not much to be honest; w̶e̶ ̶d̶e̶c̶i̶d̶e̶d̶ ̶n̶o̶t̶ ̶t̶o̶ ̶u̶s̶e̶ ̶i̶t̶ ̶i̶n̶ ̶o̶u̶r̶ ̶s̶u̶b̶m̶i̶s̶s̶i̶o̶n̶s̶.̶
", "rawMarkdown": "After strong consideration and reading MIMIC licence many dozens of times, I found it vague and subject to interpretation. However, considering the sensitivity of the dataset I decided to remove it's derivative from kaggle datasets.\n\nOn top of that I would not entirely agree that MIMIC is not \"publicly available\". Getting access takes less than 2 minutes and is free of charge and in my view complies with competition rules.\n\nAll in all, this dataset did have a small impact to our score - not much to be honest; w̶e̶ ̶d̶e̶c̶i̶d̶e̶d̶ ̶n̶o̶t̶ ̶t̶o̶ ̶u̶s̶e̶ ̶i̶t̶ ̶i̶n̶ ̶o̶u̶r̶ ̶s̶u̶b̶m̶i̶s̶s̶i̶o̶n̶s̶.̶\n\n\n\n", "votes": 7 }, { "id": 1231551, "postDate": "2021-03-09T04:14:08.747Z", "content": "Hello, did u label them by hands?
", "rawMarkdown": "Hello, did u label them by hands?" }, { "id": 1231843, "postDate": "2021-03-09T09:43:20.820Z", "content": "Based on the discussion above, I think it is necessary for organizer to state which external dataset is allowed, in order to make the competition fair. This discussion covers all external datasets discussed in this competition. PLEASE make an official statement for each dataset listed here. @jarrelscy
", "rawMarkdown": "Based on the discussion above, I think it is necessary for organizer to state which external dataset is allowed, in order to make the competition fair. [This discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873) covers all external datasets discussed in this competition. PLEASE make an official statement for each dataset listed [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873). @jarrelscy " }, { "id": 1231865, "postDate": "2021-03-09T10:15:20.990Z", "content": "As per the competition rules, any public external dataset which is freely available (meaning that other competitors don't have to pay money for it and is able to access this), including all datasets on the list by @virilo is allowed for use in this competition.
\nHowever, if you are adding additional hand-labelled annotations to these datasets, you must have made them available to other competitors in this competition.
", "rawMarkdown": "As per the competition rules, any public external dataset which is freely available (meaning that other competitors don't have to pay money for it and is able to access this), including all datasets on the list by @virilo is allowed for use in this competition. \n\nHowever, if you are adding additional hand-labelled annotations to these datasets, you must have made them available to other competitors in this competition. ", "votes": 5 }, { "id": 1231880, "postDate": "2021-03-09T10:27:21.457Z", "content": "@jarrelscy Thank you for clarification!
", "rawMarkdown": "@jarrelscy Thank you for clarification!" }, { "id": 1232102, "postDate": "2021-03-09T13:41:57.783Z", "content": "I think this thread is kind of a joke when competitors used external data which is \"released\" to public at competition's end when nobody can use it actually…
", "rawMarkdown": "I think this thread is kind of a joke when competitors used external data which is \"released\" to public at competition's end when nobody can use it actually...", "votes": -2 }, { "id": 1232939, "postDate": "2021-03-10T03:44:55.127Z", "content": "Is there any duplication between the extra data and the competition data?
", "rawMarkdown": "Is there any duplication between the extra data and the competition data?" }, { "id": 1235262, "postDate": "2021-03-12T02:29:16.913Z", "content": "@raddar how does padchest related mimic dataset link ?
", "rawMarkdown": "@raddar how does padchest related mimic dataset link ?" } ] }, { "id": 1220941, "postDate": "2021-02-28T14:22:24.980Z", "content": "Thank you very much for the prompt response!
", "rawMarkdown": "Thank you very much for the prompt response!", "votes": 1 }, { "id": 1229828, "postDate": "2021-03-07T16:02:57.893Z", "content": "i have been using some images from NIH data set pseudo label them with my best model then retrain
", "rawMarkdown": "i have been using some images from NIH data set pseudo label them with my best model then retrain", "votes": 2 }, { "id": 1222828, "postDate": "2021-03-02T07:28:00.880Z", "content": "@jarrelscy - could you ask organisers to have this thread pinned so it stays at the top of the discussion posts and so it is more visible for everyone? Thanks!
", "rawMarkdown": "@jarrelscy - could you ask organisers to have this thread pinned so it stays at the top of the discussion posts and so it is more visible for everyone? Thanks! ", "votes": 1 }, { "id": 1235963, "postDate": "2021-03-12T16:17:54.427Z", "content": "Hi Everyone and @jarrelscy ,
\nI just have a small misunderstanding on the external data
I dont know whether i am right or wrong.
\nMIMIC Dataset is not allowed and Padchest and Chest is allowed
but what makes me misunderstanding is that
\nIn the other ongoing competition Named as VinBigData Chest X-ray Abnormalities Detection in the discussion
both MIMIC and Chexpert are allowed .
\nPlease clarify @jarrelscy or anyone on this panel
@jarrelscy said, all external dataset listed here, including MIMIC is allowed, https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865
", "rawMarkdown": "@jarrelscy said, all external dataset listed [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873 ), including MIMIC is allowed, https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865", "votes": 1 } ] }, { "id": 1221416, "postDate": "2021-03-01T00:40:27.920Z", "content": "Thank you very much for your kind support.
\nThanks to your post, my question has been resolved.
I appreciate this being formally clarified, but I really don't like new data being made available only a few days before the end of the competition. In the future it might be good to set some earlier deadline for data share.
", "votes": 11, "replies": [ { "id": 1221119, "author_name": "fffrrt", "author_url": "", "post_date": "2021-02-28T17:22:09.077000", "content": "The old External Data Thread is back! I miss the times when they were mandatory.
\n\n\nbut I really don't like new data being made available only a few days before the end of the competition
\n
Earlier deadline would discourage people from hand labeling. This deadline is a nice compromise between \"whoever hand labels more - wins\" for no disclosure at all and \"why bother to hand label at all, everyone would have plenty of time to include it\" for early disclosure.
\nIf competition host wants to get the best model, then early disclosure is good. If he wants to get the best solution, then late disclosure is probably better.
", "votes": 2, "replies": [] } ] }, { "id": 1229627, "author_name": "raddar", "author_url": "", "post_date": "2021-03-07T13:48:54.807000", "content": "We have been using available NIH data in our models
\nhttps://www.kaggle.com/nih-chest-xrays/data
also I have collected some images which contain tubes from other open source datasets as well.
\nhttps://www.kaggle.com/raddar/mimic-xray-ett-ngt-cvc
\nhttps://www.kaggle.com/raddar/padchest-tubes
@raddar I think mmic dataset requires an exam which I think goes against publicly available dataset rule.
", "votes": 8, "replies": [] }, { "id": 1230292, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-03-08T01:26:54.310000", "content": "@jarrelscy
\nIs MIMIC dataset (https://mimic.physionet.org/) allowed in this competition?
\nI personally believe it is in accordance with competition rules.
There were some discussions on this topic before:
\nBut we did not receive any answer from organisers.
\nNow the entry deadline is close, and we all all need a quick response.
\nThank you in advance.
Are you sure you are allowed to host this data here?
\nTaken from: https://mimic.physionet.org/gettingstarted/access/
\n\n\nbut we expect you to take matters of security seriously and to make sure that the data are accessible to no one but yourself.
\n
Don't think this data is openly available, as you also need to take an exam.
", "votes": 6, "replies": [] }, { "id": 1231071, "author_name": "", "author_url": "", "post_date": "2021-03-08T16:39:30.283000", "content": "", "votes": 0, "replies": [] }, { "id": 1231172, "author_name": "Jaideep", "author_url": "", "post_date": "2021-03-08T18:14:35.527000", "content": "@raddar
\n1) mimic dataset link u gave belong to which class.
\n2) is it subset of NiH dataset ?
After strong consideration and reading MIMIC licence many dozens of times, I found it vague and subject to interpretation. However, considering the sensitivity of the dataset I decided to remove it's derivative from kaggle datasets.
\nOn top of that I would not entirely agree that MIMIC is not \"publicly available\". Getting access takes less than 2 minutes and is free of charge and in my view complies with competition rules.
\nAll in all, this dataset did have a small impact to our score - not much to be honest; w̶e̶ ̶d̶e̶c̶i̶d̶e̶d̶ ̶n̶o̶t̶ ̶t̶o̶ ̶u̶s̶e̶ ̶i̶t̶ ̶i̶n̶ ̶o̶u̶r̶ ̶s̶u̶b̶m̶i̶s̶s̶i̶o̶n̶s̶.̶
", "votes": 7, "replies": [] }, { "id": 1231551, "author_name": "Zekun", "author_url": "", "post_date": "2021-03-09T04:14:08.747000", "content": "Hello, did u label them by hands?
", "votes": 0, "replies": [] }, { "id": 1231843, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-03-09T09:43:20.820000", "content": "Based on the discussion above, I think it is necessary for organizer to state which external dataset is allowed, in order to make the competition fair. This discussion covers all external datasets discussed in this competition. PLEASE make an official statement for each dataset listed here. @jarrelscy
", "votes": 0, "replies": [] }, { "id": 1231865, "author_name": "Jarrel Seah", "author_url": "", "post_date": "2021-03-09T10:15:20.990000", "content": "As per the competition rules, any public external dataset which is freely available (meaning that other competitors don't have to pay money for it and is able to access this), including all datasets on the list by @virilo is allowed for use in this competition.
\nHowever, if you are adding additional hand-labelled annotations to these datasets, you must have made them available to other competitors in this competition.
", "votes": 5, "replies": [] }, { "id": 1231880, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-03-09T10:27:21.457000", "content": "@jarrelscy Thank you for clarification!
", "votes": 0, "replies": [] }, { "id": 1232102, "author_name": "AmorfEvo", "author_url": "", "post_date": "2021-03-09T13:41:57.783000", "content": "I think this thread is kind of a joke when competitors used external data which is \"released\" to public at competition's end when nobody can use it actually…
", "votes": -2, "replies": [] }, { "id": 1232939, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2021-03-10T03:44:55.127000", "content": "Is there any duplication between the extra data and the competition data?
", "votes": 0, "replies": [] }, { "id": 1235262, "author_name": "Jaideep", "author_url": "", "post_date": "2021-03-12T02:29:16.913000", "content": "@raddar how does padchest related mimic dataset link ?
", "votes": 0, "replies": [] } ] }, { "id": 1220941, "author_name": "rikein12", "author_url": "", "post_date": "2021-02-28T14:22:24.980000", "content": "Thank you very much for the prompt response!
", "votes": 1, "replies": [] }, { "id": 1229828, "author_name": "Mohamed abdelrazik", "author_url": "", "post_date": "2021-03-07T16:02:57.893000", "content": "i have been using some images from NIH data set pseudo label them with my best model then retrain
", "votes": 2, "replies": [] }, { "id": 1222828, "author_name": "something4kag", "author_url": "", "post_date": "2021-03-02T07:28:00.880000", "content": "@jarrelscy - could you ask organisers to have this thread pinned so it stays at the top of the discussion posts and so it is more visible for everyone? Thanks!
", "votes": 1, "replies": [] }, { "id": 1235963, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-03-12T16:17:54.427000", "content": "Hi Everyone and @jarrelscy ,
\nI just have a small misunderstanding on the external data
I dont know whether i am right or wrong.
\nMIMIC Dataset is not allowed and Padchest and Chest is allowed
but what makes me misunderstanding is that
\nIn the other ongoing competition Named as VinBigData Chest X-ray Abnormalities Detection in the discussion
both MIMIC and Chexpert are allowed .
\nPlease clarify @jarrelscy or anyone on this panel
@jarrelscy said, all external dataset listed here, including MIMIC is allowed, https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865
", "votes": 1, "replies": [] } ] }, { "id": 1221416, "author_name": "YYama", "author_url": "", "post_date": "2021-03-01T00:40:27.920000", "content": "Thank you very much for your kind support.
\nThanks to your post, my question has been resolved.