{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction to the Sartorius Data Challenge ##\n\nThis notebook is intended for a new user to get acquainted with the Kaggle challenge\ntitled \"[Sartorius - Cell Instance Segmentation][k_challenge]\". It will not do the\nwork necessary to submit a prediction file, but it will help to get started.\n(I am not doing any predictions in this notebook.)\n\nThe challenge involves segmenting individual neuronal cells in microscopic images.\nCurrent solutions using computer vision have limited accuracy, depending on the\ncell lines and shapes in the images. The goal of this challenge will be to find\nan efficient solution that will save time and effort.\n\nThe scoring is based on assigning pixels correctly, as signal or background. That challenge is not just distinguishing how many cell bodies are in the image, but to determine the borders of the bodies for irregular shapes.\n\n(The issue of assigning pixels when two cells overlap needs to be dealt with, because the submission requires assigning non-overlapping cells. In the discussion forum, the [competition host posted a comment][comment_challenge] that mentioned \"the amount of overlap in the evaluation dataset is very low (lower than the training dataset)\".)\n\nThe [challenge data directory][d_challenge] consists of:\n- train.csv, which will be used to describe the images to be used for the training set\n- train directory, which has 606 PNG images which are roughly 250 kB in size\n- test directory, which has 3 PNG images. The hidden test set of roughly 240 images can only be accessed by your notebook when you submit.\n- train_semi_supervised and LIVECell_dataset_2021 directories, for additional information and testing. Those will not be examined here.\n\n[k_challenge]: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/overview\n[d_challenge]: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/data\n[comment_challenge]: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/280250#1571081","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as img\n\n%matplotlib inline\n\n# Set to false if using local copies\nif (True):\n    dataDir = '/kaggle/input/sartorius-cell-instance-segmentation/'\nelse:\n    dataDir = ''\n\ntrain_file = dataDir + 'train.csv'\norg_df = pd.read_csv(train_file)","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:22:42.954166Z","iopub.execute_input":"2021-11-05T22:22:42.95457Z","iopub.status.idle":"2021-11-05T22:22:43.66773Z","shell.execute_reply.started":"2021-11-05T22:22:42.954451Z","shell.execute_reply":"2021-11-05T22:22:43.666747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### General stats for the CSV file ###\n\nThe training CSV file has 606 file identifiers, which matches the number of image files\nin the train folder.\nThe average number of cells annotated for each image is 120, with as few as 4 and as many\nas 790.\n\nThe image files are all the same size (704 x 520 pixels), but it will be useful to keep\nthe image size flexible, in case we want to use other images while developing the code.\n\nThe 3 cell types are called 'shsy5y', 'astro', and 'cort'.\n\n(I have not explored the other information yet, and will not use them in this notebook.)","metadata":{}},{"cell_type":"code","source":"print(org_df.info(show_counts=True))\nprint(org_df.nunique())\n\nnum_ids = org_df['id'].nunique()\nannot_list = np.ndarray(shape=(num_ids),dtype=int)\n#                        'int16')\nfor count,id in enumerate(org_df['id'].unique()):\n    annot_list[count] = org_df[org_df['id'] == id].shape[0]\n\nprint(\"Unique image files: \", annot_list.size)\nprint('Average number of annotations per file: ',  \\\n      np.average(annot_list))\nprint('Min number of annotations per file: ',  \\\n      np.min(annot_list))\nprint('Max number of annotations per file: ',  \\\n      np.max(annot_list))\nprint(\"Image width : \",org_df['width'].unique())\nprint(\"Image height: \",org_df['height'].unique())\nprint(\"Cell types: \",org_df['cell_type'].unique())","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:22:43.669412Z","iopub.execute_input":"2021-11-05T22:22:43.669675Z","iopub.status.idle":"2021-11-05T22:22:51.30685Z","shell.execute_reply.started":"2021-11-05T22:22:43.669644Z","shell.execute_reply":"2021-11-05T22:22:51.305711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load an image ###\n\nTo start, I downloaded (manually) the first 5 PNG files that I saw in the Kaggle Data Explorer. In this case, I want to use the 5th one, named \"029e5b3b89c7.png\".\n\nAt first, I used the pypng library to load the image, where I learned that the image was\nin greyscale. I decided to drop that library, and just use matplotlib (and assume that\nall images are greyscale). The dimensions of the image are what we expected, and\nthe pixels range in value from 0 to 1 (float).\n\nI found it difficult to see the details in the plot without making the image larger.\nTo help with that, I made another image where all of the pizels with values > 0.5\nwere assigned to 1. This made the background bright, and helps me to see the\nstructures better.","metadata":{}},{"cell_type":"code","source":"image_list = [\n    \"01ae5a43a2ab\",\n    \"026b3c2c4b32\",\n    \"029e5b3b89c7\",\n    \"0030fd0e6378\",\n    \"0140b3c8f445\"\n]\nid_image = 2\n\n# reading png image file\npngFile = \"{}train/{}.png\".format(dataDir, image_list[id_image])\nim = img.imread(pngFile)\n\nim2 = im.copy()\nim2[im2 > 0.5] = 1.0\n\n# show image(s)\nfig, axs = plt.subplots(1, 2)\nfig.set_size_inches(14.0, 8.0)\naxs[0].set_title('normal')\naxs[0].imshow(im, cmap='gray')\naxs[1].set_title('altered')\naxs[1].imshow(im2, cmap='gray')\nplt.show()\n\nprint(\"file size \",im.shape)\nprint(\"min pixel value = \",np.min(im))\nprint(\"max pixel value = \",np.max(im))","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:22:51.31007Z","iopub.execute_input":"2021-11-05T22:22:51.310326Z","iopub.status.idle":"2021-11-05T22:22:51.868138Z","shell.execute_reply.started":"2021-11-05T22:22:51.310289Z","shell.execute_reply":"2021-11-05T22:22:51.86709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Annotated cells ###\n\nNext I wanted to see where the identified cells were in the image.\n\nThe annotation format is under [Overview/Evaluation][a_challenge], under the subheading\n\"Submission File\"\n\nEach line in the train.CSV file is a cell in an image. The \"annotation\" is a string of\nnumber pairs. The first number is the starting pixel index, and the second number is\nthe run length.\n\nFor this initial scan of the data, I didn't want to worry about coloring the plots\nor dealing with alpha transparency values. I made a black & white mask showing the\ncell locations, and put that next to the image for comparison.\n\n[a_challenge]: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/overview/evaluation","metadata":{}},{"cell_type":"code","source":"temp_df = org_df[org_df['id'] == image_list[id_image]]\n\ndef annot_mask(masker, annot_string, val= 0.0):\n    an_arr = np.fromstring(annot_string, dtype=int, sep=' ')\n    for ind in range(0, an_arr.size, 2):\n        ix = an_arr[ind]\n        iy = ix + an_arr[ind+1]\n        masker[ix:iy] = val\n    return masker\n\nmasker = np.ones(im.shape[0]*im.shape[1])\nfor an_string in temp_df['annotation']:\n    masker = annot_mask(masker, an_string)\n\nmasker = masker.reshape(im.shape)\n\n# show image(s)\nfig, axs = plt.subplots(1, 2)\nfig.set_size_inches(14.0, 8.0)\naxs[0].imshow(im2, cmap='gray')\naxs[0].set_title('altered')\naxs[1].imshow(masker, cmap='gray')\naxs[1].set_title('annotated cells')\nplt.show()\n\nprint(\"Number of annotations: \",temp_df.shape[0])","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:22:51.870337Z","iopub.execute_input":"2021-11-05T22:22:51.871377Z","iopub.status.idle":"2021-11-05T22:22:52.398684Z","shell.execute_reply.started":"2021-11-05T22:22:51.87131Z","shell.execute_reply":"2021-11-05T22:22:52.39755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion ###\n\nThere is much to do before a submission can be made, but sometimes you need to tread\nwater in the shallow part of the pool before swimming in the deep end. Good luck\nwith your algorithms!","metadata":{}}]}