![\"Screen-Shot-2021-05-21-at-6-32-22-PM\"](\"https://i.ibb.co/3mVP2Md/Screen-Shot-2021-05-21-at-6-32-22-PM.png\")
Hi there.
\nI have created a notebook to develop my understanding of studies and how the image level labels interact with these study level labels. This notebook is simple. It captures the studies with more than one image and plots them. It also calls out which images have bounding boxes and which don't.
\nGoing forward when I refer to the image level dataset, I am referring to the images for which we have to generate bounding box predictions to localize opacities. When I refer to the study level dataset, I am referring to the images for which we have to generate a classification (one of the following… atypical, typical, indeterminate, negative… remember that the first three labels listed indicate the patient is presenting with covid while the final label indicates that the patient is negative for pneumonia… i.e. no-covid)
My takeaway understanding at a high level is as follows:
\nSome Observed Defects/Weirdness Is Shown Below
\nEXAMPLE A - NEARLY IDENTICAL IMAGES w/ SMALL DIFFERENCES
\n
EXAMPLE B - IMAGES ARE DIFFERENT (images often appear to be the same patient but at different angles, times or positions)
\n![\"Screen-Shot-2021-05-21-at-6-35-14-PM\"](\"https://i.ibb.co/qFjd8c3/Screen-Shot-2021-05-21-at-6-35-14-PM.png\")
EXAMPLE C - NEARLY IDENTICAL IMAGES EXAMPLE SHOWING HORIZONTAL FLIP
\n![\"Screen-Shot-2021-05-21-at-6-34-57-PM\"](\"https://i.ibb.co/dGzTX34/Screen-Shot-2021-05-21-at-6-34-57-PM.png\")
EXAMPLE D - IMAGES SHOWING PERSPECTIVE TRANSFORM DEFECT (and dramatic difference in focus/sharpness)
\n![\"Screen-Shot-2021-05-21-at-6-34-46-PM\"](\"https://i.ibb.co/3NVQVmP/Screen-Shot-2021-05-21-at-6-34-46-PM.png\")
EXAMPLE E - IMAGES SHOWING EXTRA BLURRINESS/SHARPNESS/FOCUS DEFECT
\n![\"Screen-Shot-2021-05-21-at-6-34-17-PM\"](\"https://i.ibb.co/ys90B1g/Screen-Shot-2021-05-21-at-6-34-17-PM.png\")
EXAMPLE F - IMAGES SHOWING CROPPING DEFECT
\n![\"Screen-Shot-2021-05-21-at-6-33-14-PM\"](\"https://i.ibb.co/FVvR7qg/Screen-Shot-2021-05-21-at-6-33-14-PM.png\")
EXAMPLE G - IMAGES SHOWING POOR IMAGE QUALITY (and different angles? or something…)
\n![\"Screen-Shot-2021-05-21-at-6-33-03-PM\"](\"https://i.ibb.co/BL9sxbQ/Screen-Shot-2021-05-21-at-6-33-03-PM.png\")
EXAMPLE H - IMAGES SHOWING CONTRAST/EXPOSURE/BRIGHTNESS DEFECT & POOR IMAGE QUALITY
\n![\"Screen-Shot-2021-05-21-at-6-32-58-PM\"](\"https://i.ibb.co/kqpwg5h/Screen-Shot-2021-05-21-at-6-32-58-PM.png\")
EXAMPLE I - SHOWS THAT A STUDY MAY CONTAIN UP TO A SINGLE IMAGE w/ BOUNDING BOXES
\n![\"Screen-Shot-2021-05-21-at-6-32-55-PM\"](\"https://i.ibb.co/HhmG8tz/Screen-Shot-2021-05-21-at-6-32-55-PM.png\")
EXAMPLE J - SHOWS THAT A STUDY MAY NOT CONTAIN ANY BOUNDING BOXES
\n![\"Screen-Shot-2021-05-21-at-6-32-30-PM\"](\"https://i.ibb.co/c6DwGq0/Screen-Shot-2021-05-21-at-6-32-30-PM.png\")
I know I was struggling to understand aspects of this competition… I hope this will help others!
", "messages": [ { "id": 1318021, "postDate": "2021-05-21T22:51:35.797Z", "content": "Hi there.
\nI have created a notebook to develop my understanding of studies and how the image level labels interact with these study level labels. This notebook is simple. It captures the studies with more than one image and plots them. It also calls out which images have bounding boxes and which don't.
\nGoing forward when I refer to the image level dataset, I am referring to the images for which we have to generate bounding box predictions to localize opacities. When I refer to the study level dataset, I am referring to the images for which we have to generate a classification (one of the following… atypical, typical, indeterminate, negative… remember that the first three labels listed indicate the patient is presenting with covid while the final label indicates that the patient is negative for pneumonia… i.e. no-covid)
My takeaway understanding at a high level is as follows:
\nSome Observed Defects/Weirdness Is Shown Below
\nEXAMPLE A - NEARLY IDENTICAL IMAGES w/ SMALL DIFFERENCES
\n
EXAMPLE B - IMAGES ARE DIFFERENT (images often appear to be the same patient but at different angles, times or positions)
\n
EXAMPLE C - NEARLY IDENTICAL IMAGES EXAMPLE SHOWING HORIZONTAL FLIP
\n
EXAMPLE D - IMAGES SHOWING PERSPECTIVE TRANSFORM DEFECT (and dramatic difference in focus/sharpness)
\n
EXAMPLE E - IMAGES SHOWING EXTRA BLURRINESS/SHARPNESS/FOCUS DEFECT
\n
EXAMPLE F - IMAGES SHOWING CROPPING DEFECT
\n
EXAMPLE G - IMAGES SHOWING POOR IMAGE QUALITY (and different angles? or something…)
\n
EXAMPLE H - IMAGES SHOWING CONTRAST/EXPOSURE/BRIGHTNESS DEFECT & POOR IMAGE QUALITY
\n
EXAMPLE I - SHOWS THAT A STUDY MAY CONTAIN UP TO A SINGLE IMAGE w/ BOUNDING BOXES
\n
EXAMPLE J - SHOWS THAT A STUDY MAY NOT CONTAIN ANY BOUNDING BOXES
\n
I know I was struggling to understand aspects of this competition… I hope this will help others!
", "rawMarkdown": "Hi there. \n\n---\n\nI have [**created a notebook**](https://www.kaggle.com/dschettler8845/covid-detection-studies-with-multiple-images-viz) to develop my understanding of **studies** and how the **image** level labels interact with these **study** level labels. This notebook is simple. It captures the studies with more than one image and plots them. It also calls out which images have bounding boxes and which don't.\n\nGoing forward when I refer to the **image** level dataset, I am referring to the images for which we have to generate bounding box predictions to localize opacities. When I refer to the **study** level dataset, I am referring to the images for which we have to generate a classification (one of the following... **`atypical`**, **`typical`**, **`indeterminate`**, **`negative`**... remember that the first three labels listed indicate the patient is presenting with **covid** while the final label indicates that the patient is **negative for pneumonia... i.e. no-covid**)\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\nThanks. I have not looked at the data yet. Your post is very helpful. Why is there only maximum 1 image per study with bbox when the study images are so similar? If a study is positive, shouldn't all the images in the study have bbox?
", "rawMarkdown": "Thanks. I have not looked at the data yet. Your post is very helpful. Why is there only maximum 1 image per study with bbox when the study images are so similar? If a study is positive, shouldn't all the images in the study have bbox?", "votes": 3, "replies": [ { "id": 1370132, "postDate": "2021-06-30T00:44:03.083Z", "content": "@cdeotte you can look this thread. According to this thread, These other duplicate/similar images were likely not looked at by the annotators as we were unaware of them during the initial annotation process
Thanks a lot, this is very helpful.
\nNote that, despite what you have wrote, atypical class means probably non-covid cases, or in the competition paper words:
\n", "rawMarkdown": "Thanks a lot, this is very helpful.\nNote that, despite what you have wrote, `atypical` class means probably non-covid cases, or in [the competition paper](https://journals.lww.com/thoracicimaging/Fulltext/2020/11000/Review_of_Chest_Radiograph_Findings_of_COVID_19.4.aspx) words:\n>“Findings atypical or uncommonly reported for COVID-19 pneumonia. Consider alternative diagnoses”", "votes": 3, "replies": [ { "id": 1321497, "postDate": "2021-05-24T18:00:13.800Z", "content": "“Findings atypical or uncommonly reported for COVID-19 pneumonia. Consider alternative diagnoses”
\n
I did not know this! Thank you
", "rawMarkdown": "I did not know this! Thank you" } ] }, { "id": 1318778, "postDate": "2021-05-22T15:11:26.737Z", "content": "this is helpful . Hope the organizers can clarify about how the test images BBox are annotated . if its same as training images (one image with BBox while other are not) it will get tricky on which images should we submit with bbox
", "rawMarkdown": "this is helpful . Hope the organizers can clarify about how the test images BBox are annotated . if its same as training images (one image with BBox while other are not) it will get tricky on which images should we submit with bbox ", "votes": 4, "replies": [ { "id": 1318847, "postDate": "2021-05-22T16:00:07.660Z", "content": "100% agree. This is very confusing currently. I will update this post if the competition organizers clarify things.
", "rawMarkdown": "100% agree. This is very confusing currently. I will update this post if the competition organizers clarify things.", "votes": 3 }, { "id": 1319647, "postDate": "2021-05-23T12:13:35.463Z", "content": "Quite silent organizers…
", "rawMarkdown": "Quite silent organizers...", "votes": 3 }, { "id": 1319656, "postDate": "2021-05-23T12:20:23.533Z", "content": "Yeah I’m not sure how we’re supposed to proceed without more communication…
\nI guess I’ll train a study level model? Maybe probe the lb to identify what percentage of image level labels have boxes?
", "rawMarkdown": "Yeah I’m not sure how we’re supposed to proceed without more communication...\n\nI guess I’ll train a study level model? Maybe probe the lb to identify what percentage of image level labels have boxes?", "votes": 2 }, { "id": 1319897, "postDate": "2021-05-23T15:21:22.047Z", "content": "I've already submitted my detection model and visually confirmed that it was acceptable, but the score was almost 0.
", "rawMarkdown": "I've already submitted my detection model and visually confirmed that it was acceptable, but the score was almost 0.", "votes": 3 }, { "id": 1321414, "postDate": "2021-05-24T17:07:40.743Z", "content": "It is nice Sir.
", "rawMarkdown": "It is nice Sir. ", "votes": 1 } ] }, { "id": 1371834, "postDate": "2021-07-01T09:12:06.157Z", "content": "Thanks a lot for this discussion and the detailed takeaways! Helped me understand why my bounding boxes weren't stacking properly :D
", "rawMarkdown": "Thanks a lot for this discussion and the detailed takeaways! Helped me understand why my bounding boxes weren't stacking properly :D", "votes": 1 }, { "id": 1353768, "postDate": "2021-06-17T08:31:28.290Z", "content": "@dschettler8845 way we take _ve for pneuomonia is slightly different if understand from Annotations method posted by organizers, -ve for pneumonia might mean normal pneumonia not really covid pneumonia . Opactity in Xray is covid based. It might mean no opacity but still Not negative for pneumonia
", "rawMarkdown": "@dschettler8845 way we take _ve for pneuomonia is slightly different if understand from Annotations method posted by organizers, -ve for pneumonia might mean normal pneumonia not really covid pneumonia . Opactity in Xray is covid based. It might mean no opacity but still Not negative for pneumonia", "votes": 1 }, { "id": 1322913, "postDate": "2021-05-25T19:30:08.417Z", "content": "Hi there, thanks for the super useful discussion.
\nSo if I understand correct - your example A contributes 9 images to the training set, but they are effectively all the same image with the same label?
\nAny idea of how frequently this occurs throughout the dataset?
Hi sorry for the delay, see below!
\n# number of images in a study : frequency of occurrence in training data\n1 : 2649\n2 : 3049\n3 : 304\n4 : 40\n5 : 6\n6 : 2\n7 : 1\n8 : 2\n9 : 1\nThis is helpful. Thanks!
", "rawMarkdown": "This is helpful. Thanks!", "votes": 1 } ] }, { "id": 1321398, "postDate": "2021-05-24T17:02:06.420Z", "content": "Such a great topic I watched Ever. Thanks for it Sir. upvoted for you hardwork.
", "rawMarkdown": "Such a great topic I watched Ever. Thanks for it Sir. upvoted for you hardwork. ", "votes": 1 }, { "id": 1856135, "postDate": "2022-07-15T06:31:56.040Z", "content": "Sorry to say but I am not able to understand this.
", "rawMarkdown": "Sorry to say but I am not able to understand this.", "replies": [ { "id": 1857163, "postDate": "2022-07-15T23:05:36.053Z", "content": "Anything I can do to help clarify things?
", "rawMarkdown": "Anything I can do to help clarify things?" } ] }, { "id": 1374344, "postDate": "2021-07-03T08:29:36.970Z", "content": "Wow…
\nIt is not enough only one Upvote
\nThanks for your sharing👍
Thanks! This is post is a competition-saver.
", "rawMarkdown": "Thanks! This is post is a competition-saver." }, { "id": 1369489, "postDate": "2021-06-29T11:57:14.487Z", "content": "Thank you for your clarification! This competition is indeed confusing. I wonder if you could point me in the right direction regarding my submission.csv file. It looks perfectly in line with the sample_submission.csv to me, but I can't pass the \"Submission Scoring Error\" no matter what I do. I kind of run out of ideas.
", "rawMarkdown": "Thank you for your clarification! This competition is indeed confusing. I wonder if you could point me in the right direction regarding my submission.csv file. It looks perfectly in line with the sample_submission.csv to me, but I can't pass the \"Submission Scoring Error\" no matter what I do. I kind of run out of ideas. " }, { "id": 1365848, "postDate": "2021-06-26T07:26:07.557Z", "content": "Nice sharing!
", "rawMarkdown": "Nice sharing!" }, { "id": 1327372, "postDate": "2021-05-29T08:04:56.463Z", "content": "thanks. i'ts good hint for me. i try to use it
", "rawMarkdown": "thanks. i'ts good hint for me. i try to use it" }, { "id": 1327341, "postDate": "2021-05-29T07:24:55.540Z", "content": "Very useful topic.I don't understand the differrence between image level and study level until read your topic.Good luck.
", "rawMarkdown": "Very useful topic.I don't understand the differrence between image level and study level until read your topic.Good luck." }, { "id": 1326819, "postDate": "2021-05-28T17:50:51.690Z", "content": "Such an amazing illustration of the topic. Thank you for this. Just upvoted for your work…… All the best!!!
", "rawMarkdown": "Such an amazing illustration of the topic. Thank you for this. Just upvoted for your work...... All the best!!!", "replies": [ { "id": 1326930, "postDate": "2021-05-28T18:16:33.180Z", "content": "Thank you for your kind words. I’m glad it helped.
", "rawMarkdown": "Thank you for your kind words. I’m glad it helped.", "votes": 1 }, { "id": 1327004, "postDate": "2021-05-28T19:27:24.477Z", "content": "your welcome 😊
", "rawMarkdown": "your welcome 😊" } ] }, { "id": 1321488, "postDate": "2021-05-24T17:56:51.270Z", "content": "This is helpful.
", "rawMarkdown": "This is helpful." }, { "id": 1323166, "postDate": "2021-05-26T03:58:20.320Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1327931, "postDate": "2021-05-29T18:39:23.983Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 1325552, "postDate": "2021-05-27T20:33:37.410Z", "content": "Thanks a lot, this is very helpful.👍
", "rawMarkdown": "Thanks a lot, this is very helpful.👍" }, { "id": 1323161, "postDate": "2021-05-26T03:54:05.210Z", "content": "Thanks a lot!! This topic is helpful.
", "rawMarkdown": "Thanks a lot!! This topic is helpful." } ], "comments": [ { "id": 1370122, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-06-30T00:33:41.977000", "content": "Thanks. I have not looked at the data yet. Your post is very helpful. Why is there only maximum 1 image per study with bbox when the study images are so similar? If a study is positive, shouldn't all the images in the study have bbox?
", "votes": 3, "replies": [ { "id": 1370132, "author_name": "Awsaf", "author_url": "", "post_date": "2021-06-30T00:44:03.083000", "content": "@cdeotte you can look this thread. According to this thread, These other duplicate/similar images were likely not looked at by the annotators as we were unaware of them during the initial annotation process
Thanks a lot, this is very helpful.
\nNote that, despite what you have wrote, atypical class means probably non-covid cases, or in the competition paper words:
\n", "votes": 3, "replies": [ { "id": 1321497, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-05-24T18:00:13.800000", "content": "“Findings atypical or uncommonly reported for COVID-19 pneumonia. Consider alternative diagnoses”
\n
I did not know this! Thank you
", "votes": 0, "replies": [] } ] }, { "id": 1318778, "author_name": "yuvaramsingh", "author_url": "", "post_date": "2021-05-22T15:11:26.737000", "content": "this is helpful . Hope the organizers can clarify about how the test images BBox are annotated . if its same as training images (one image with BBox while other are not) it will get tricky on which images should we submit with bbox
", "votes": 4, "replies": [ { "id": 1318847, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-05-22T16:00:07.660000", "content": "100% agree. This is very confusing currently. I will update this post if the competition organizers clarify things.
", "votes": 3, "replies": [] }, { "id": 1319647, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-05-23T12:13:35.463000", "content": "Quite silent organizers…
", "votes": 3, "replies": [] }, { "id": 1319656, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-05-23T12:20:23.533000", "content": "Yeah I’m not sure how we’re supposed to proceed without more communication…
\nI guess I’ll train a study level model? Maybe probe the lb to identify what percentage of image level labels have boxes?
", "votes": 2, "replies": [] }, { "id": 1319897, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-05-23T15:21:22.047000", "content": "I've already submitted my detection model and visually confirmed that it was acceptable, but the score was almost 0.
", "votes": 3, "replies": [] }, { "id": 1321414, "author_name": "YashSharma", "author_url": "", "post_date": "2021-05-24T17:07:40.743000", "content": "It is nice Sir.
", "votes": 1, "replies": [] } ] }, { "id": 1371834, "author_name": "GVG", "author_url": "", "post_date": "2021-07-01T09:12:06.157000", "content": "Thanks a lot for this discussion and the detailed takeaways! Helped me understand why my bounding boxes weren't stacking properly :D
", "votes": 1, "replies": [] }, { "id": 1353768, "author_name": "Jaideep", "author_url": "", "post_date": "2021-06-17T08:31:28.290000", "content": "@dschettler8845 way we take _ve for pneuomonia is slightly different if understand from Annotations method posted by organizers, -ve for pneumonia might mean normal pneumonia not really covid pneumonia . Opactity in Xray is covid based. It might mean no opacity but still Not negative for pneumonia
", "votes": 1, "replies": [] }, { "id": 1322913, "author_name": "Reuben Schmidt", "author_url": "", "post_date": "2021-05-25T19:30:08.417000", "content": "Hi there, thanks for the super useful discussion.
\nSo if I understand correct - your example A contributes 9 images to the training set, but they are effectively all the same image with the same label?
\nAny idea of how frequently this occurs throughout the dataset?
Hi sorry for the delay, see below!
\n# number of images in a study : frequency of occurrence in training data\n1 : 2649\n2 : 3049\n3 : 304\n4 : 40\n5 : 6\n6 : 2\n7 : 1\n8 : 2\n9 : 1\nThis is helpful. Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 1321398, "author_name": "YashSharma", "author_url": "", "post_date": "2021-05-24T17:02:06.420000", "content": "Such a great topic I watched Ever. Thanks for it Sir. upvoted for you hardwork.
", "votes": 1, "replies": [] }, { "id": 1856135, "author_name": "Hafiz Sayyed Ali Hamdani", "author_url": "", "post_date": "2022-07-15T06:31:56.040000", "content": "Sorry to say but I am not able to understand this.
", "votes": 0, "replies": [ { "id": 1857163, "author_name": "Darien Schettler", "author_url": "", "post_date": "2022-07-15T23:05:36.053000", "content": "Anything I can do to help clarify things?
", "votes": 0, "replies": [] } ] }, { "id": 1374344, "author_name": "J.H.Lee", "author_url": "", "post_date": "2021-07-03T08:29:36.970000", "content": "Wow…
\nIt is not enough only one Upvote
\nThanks for your sharing👍
Thanks! This is post is a competition-saver.
", "votes": 0, "replies": [] }, { "id": 1369489, "author_name": "Kostiantyn Perun", "author_url": "", "post_date": "2021-06-29T11:57:14.487000", "content": "Thank you for your clarification! This competition is indeed confusing. I wonder if you could point me in the right direction regarding my submission.csv file. It looks perfectly in line with the sample_submission.csv to me, but I can't pass the \"Submission Scoring Error\" no matter what I do. I kind of run out of ideas.
", "votes": 0, "replies": [] }, { "id": 1365848, "author_name": "zZammm", "author_url": "", "post_date": "2021-06-26T07:26:07.557000", "content": "Nice sharing!
", "votes": 0, "replies": [] }, { "id": 1327372, "author_name": "tensor choko", "author_url": "", "post_date": "2021-05-29T08:04:56.463000", "content": "thanks. i'ts good hint for me. i try to use it
", "votes": 0, "replies": [] }, { "id": 1327341, "author_name": "hujianxin", "author_url": "", "post_date": "2021-05-29T07:24:55.540000", "content": "Very useful topic.I don't understand the differrence between image level and study level until read your topic.Good luck.
", "votes": 0, "replies": [] }, { "id": 1326819, "author_name": "Ali Mashood", "author_url": "", "post_date": "2021-05-28T17:50:51.690000", "content": "Such an amazing illustration of the topic. Thank you for this. Just upvoted for your work…… All the best!!!
", "votes": 0, "replies": [ { "id": 1326930, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-05-28T18:16:33.180000", "content": "Thank you for your kind words. I’m glad it helped.
", "votes": 1, "replies": [] }, { "id": 1327004, "author_name": "Ali Mashood", "author_url": "", "post_date": "2021-05-28T19:27:24.477000", "content": "your welcome 😊
", "votes": 0, "replies": [] } ] }, { "id": 1321488, "author_name": "Alan Rozario", "author_url": "", "post_date": "2021-05-24T17:56:51.270000", "content": "This is helpful.
", "votes": 0, "replies": [] }, { "id": 1323166, "author_name": "", "author_url": "", "post_date": "2021-05-26T03:58:20.320000", "content": "", "votes": 0, "replies": [] }, { "id": 1327931, "author_name": "Alokdas05", "author_url": "", "post_date": "2021-05-29T18:39:23.983000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 1325552, "author_name": "Mohamed Bakrey Mahmoud", "author_url": "", "post_date": "2021-05-27T20:33:37.410000", "content": "Thanks a lot, this is very helpful.👍
", "votes": 0, "replies": [] }, { "id": 1323161, "author_name": "Yurumin", "author_url": "", "post_date": "2021-05-26T03:54:05.210000", "content": "Thanks a lot!! This topic is helpful.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1318021": "Hi there. \n\n---\n\nI have [**created a notebook**](https://www.kaggle.com/dschettler8845/covid-detection-studies-with-multiple-images-viz) to develop my understanding of **studies** and how the **image** level labels interact with these **study** level labels. This notebook is simple. It captures the studies with more than one image and plots them. It also calls out which images have bounding boxes and which don't.\n\nGoing forward when I refer to the **image** level dataset, I am referring to the images for which we have to generate bounding box predictions to localize opacities. When I refer to the **study** level dataset, I am referring to the images for which we have to generate a classification (one of the following... **`atypical`**, **`typical`**, **`indeterminate`**, **`negative`**... remember that the first three labels listed indicate the patient is presenting with **covid** while the final label indicates that the patient is **negative for pneumonia... i.e. no-covid**)\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n\n\n---\n\n