{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## STEP 0 - COMPUTER VISION EDA - INTERESTING CASES IN ULTRA-MNIST DATASET\n\nIn computer vision the most important part is to look visually into data (this is FIRST STEP) ... so let's look into \"interesting\" cases in dataset ...\n\nEDA in computer vision is different then in tabular data. We must work visually on each step (even visual validation of our model performance). My CV workflow for data:\n\n* look into data - spend 80% working with images (I always download whole dataset and watching on photos)\n* visualize images (or … visualize using videos)\n* visualize prediction (classification result, bboxes, segmentations etc.) - score is not enough - we sholud see prediction and well and find, understad anomalies to improve model.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"<div class=\"alert alert-success\">\n  <strong>Other my work in this competiton</strong>\n    <ul>\n        <li>STEP 0 - COMPUTER VISION EDA - INTERESTING CASES IN ULTRA-MNIST DATASET -> this notebook</li>\n        <li><a href=\"https://www.kaggle.com/c/ultra-mnist/discussion/312003\">STEP 1 - Simplify problem - new DATASET - PLAIN BACKGROUND without checkboard</a></li>\n        <li><a href=\"https://www.kaggle.com/remekkinas/step-2-find-numbers-no-model-required/edit\">STEP 2 - Find Numbers - NO MODEL REQUIRED ... only IMAGE MANIPULATIONS</a></li>\n    </ul>\n</div>","metadata":{}},{"cell_type":"code","source":"import cv2\nfrom PIL import Image as Img\nfrom IPython.display import Image\n\nimport pandas as pd","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:39:37.797178Z","iopub.execute_input":"2022-03-09T20:39:37.797576Z","iopub.status.idle":"2022-03-09T20:39:37.80182Z","shell.execute_reply.started":"2022-03-09T20:39:37.797546Z","shell.execute_reply":"2022-03-09T20:39:37.800912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/ultra-mnist/train.csv\")\n\nIMG_PATH = \"../input/ultra-mnist/train/\"","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:39:39.098826Z","iopub.execute_input":"2022-03-09T20:39:39.099426Z","iopub.status.idle":"2022-03-09T20:39:39.136736Z","shell.execute_reply.started":"2022-03-09T20:39:39.09936Z","shell.execute_reply":"2022-03-09T20:39:39.136107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## INTRO\nIn this dataset you can see the beauty and at the same time the typical problems found in MNIST. The contest consists in adding up the numbers that are placed on a checkered background. Typical pipeline can look like:\n\n1. Simplify - normalize background - [Digit Cleaner Concept](https://www.kaggle.com/lukaszborecki/digit-cleaner-concept) from @lukaszborecki \n\n<div style=\"text-align:center;\">\n<h3>Oryginal</h3> </div>\n<div align=\"center\"><img src=\"https://i.ibb.co/zrgchbt/ultra-mnist-000a.jpg\" width=640/></div>\n\n<h3>&nbsp;</h3>\n<div style=\"text-align:center;\">\n<h3>Simplified</h3></div>\n<div align=\"center\"><img src=\"https://i.ibb.co/0DZYLQG/ultra-mnist.jpg\" width=640/></div>\n\n<div align=\"center\"><img src=\"https://i.ibb.co/GWQD8Yj/ultra-mnist-002.jpg\" width=640/></div>\n\n2. Detect numbers (find ROI - Region of Interest and classify) - object detector (yolo 5 will be great) - 10 classes ...+ SAHI - it should work perfectly in this case (detect on whole image and then on slices 4x4 - small numbers - build in feature) -> https://www.kaggle.com/remekkinas/sahi-slicing-aided-hyper-inference-yv5-and-yx\n3. Add up numbers \n4. Submit","metadata":{}},{"cell_type":"markdown","source":"## ONE DAY SMALL AND BIG NUMBERS MET ON CHECKED BOARD AT ONCE ... AND ...\n\nObservations:\n- background could be challanging in this competition - since we have regullar rectangle we can simplify workflow and remove checkboard at first step - see below proposition","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'aaofnmvaxy.jpeg', width = 480, height = 480)","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.443757Z","iopub.execute_input":"2022-03-09T20:11:00.444047Z","iopub.status.idle":"2022-03-09T20:11:00.486203Z","shell.execute_reply.started":"2022-03-09T20:11:00.444013Z","shell.execute_reply":"2022-03-09T20:11:00.485293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... EVEN SMALLER CAME AS WELL ... \n\nObservations:\n- numbers are different in size - it could be challanging for object detector to see BIG numbers and very small at the same time - proposition -> SAHI (Slicing Aided Hyper Inference) or tiling","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'acfyvsrlck.jpeg', width = 480, height = 480)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:11:00.487472Z","iopub.execute_input":"2022-03-09T20:11:00.488004Z","iopub.status.idle":"2022-03-09T20:11:00.505997Z","shell.execute_reply.started":"2022-03-09T20:11:00.487962Z","shell.execute_reply":"2022-03-09T20:11:00.505384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... OR EVEN INVISIBLE .... ","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'acybfuinqq.jpeg', width = 480, height = 480)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.507099Z","iopub.execute_input":"2022-03-09T20:11:00.507466Z","iopub.status.idle":"2022-03-09T20:11:00.522589Z","shell.execute_reply.started":"2022-03-09T20:11:00.507437Z","shell.execute_reply":"2022-03-09T20:11:00.521885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BUT WHEN WE SLICE THEM ... LOOKS GREAT ...\n\n","metadata":{}},{"cell_type":"code","source":"img = cv2.imread(IMG_PATH + 'acybfuinqq.jpeg', cv2.IMREAD_GRAYSCALE)\n\ndisplay(Img.fromarray(img[1000:2000, 0:1000]).resize((480,480)))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:22:03.979932Z","iopub.execute_input":"2022-03-09T20:22:03.980445Z","iopub.status.idle":"2022-03-09T20:22:04.043869Z","shell.execute_reply.started":"2022-03-09T20:22:03.980392Z","shell.execute_reply":"2022-03-09T20:22:04.042898Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## THEY HAVE A PROBLEM WITH THEMSELVES ... THEY CAN'T DECIDE IF THEY ARE SIX OR NINE ...\n\n- For sure it will be challanging to distinguish between 6 and 9 - we have to deal with this in inference part (some rules during number summary)\n- During training FILP augumentation will be helpfull eg. below you can find 9/6 flipped/mirrored","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'afljkeleeq.jpeg', width = 480, height = 480)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.524234Z","iopub.execute_input":"2022-03-09T20:11:00.524656Z","iopub.status.idle":"2022-03-09T20:11:00.544867Z","shell.execute_reply.started":"2022-03-09T20:11:00.524623Z","shell.execute_reply":"2022-03-09T20:11:00.54392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.imread(IMG_PATH + 'afljkeleeq.jpeg', cv2.IMREAD_GRAYSCALE)\n\ndisplay(Img.fromarray(img[1000:2000, 3000:4000]).resize((480,480)))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:35:13.744671Z","iopub.execute_input":"2022-03-09T20:35:13.74497Z","iopub.status.idle":"2022-03-09T20:35:13.810663Z","shell.execute_reply.started":"2022-03-09T20:35:13.744939Z","shell.execute_reply":"2022-03-09T20:35:13.809683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... TWO ... OR MAYBE FIVE ... OR MAYBE TWO OR ...EIGHT ????\n\nObservations:\n- Some digits are unknown - below you can recognize it as a 5 or 2 (flipped)","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'abgmdxywua.jpeg', width = 480, height = 480)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:11:00.546337Z","iopub.execute_input":"2022-03-09T20:11:00.5472Z","iopub.status.idle":"2022-03-09T20:11:00.565039Z","shell.execute_reply.started":"2022-03-09T20:11:00.547149Z","shell.execute_reply":"2022-03-09T20:11:00.564202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.imread(IMG_PATH + 'abgmdxywua.jpeg', cv2.IMREAD_GRAYSCALE)\n\ndisplay(Img.fromarray(img[0:1000, 3000:4000]).resize((480,480)))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:24:28.132631Z","iopub.execute_input":"2022-03-09T20:24:28.132924Z","iopub.status.idle":"2022-03-09T20:24:28.194033Z","shell.execute_reply.started":"2022-03-09T20:24:28.132885Z","shell.execute_reply":"2022-03-09T20:24:28.193151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SOME OF THEM MASK WELL ... WANT TO HIDE WHAT NUMBER THEY ARE ...\n\nObservations:\n- Some numbers ar clipped - augumentations CUTOUT could be helpfull","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'akscpxgert.jpeg', width = 480, height = 480)","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.566677Z","iopub.execute_input":"2022-03-09T20:11:00.567205Z","iopub.status.idle":"2022-03-09T20:11:00.585547Z","shell.execute_reply.started":"2022-03-09T20:11:00.567149Z","shell.execute_reply":"2022-03-09T20:11:00.584653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.imread(IMG_PATH + 'akscpxgert.jpeg', cv2.IMREAD_GRAYSCALE)\n\ndisplay(Img.fromarray(img[3000:4000, 0:1000]).resize((480,480)))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:36:27.304479Z","iopub.execute_input":"2022-03-09T20:36:27.304796Z","iopub.status.idle":"2022-03-09T20:36:27.365754Z","shell.execute_reply.started":"2022-03-09T20:36:27.304759Z","shell.execute_reply":"2022-03-09T20:36:27.365181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... OR EVEN WORSE ... THEY TRY TO IMITATE OTHER NUMBERS ...\n\nObservations:\n- it could be a number or ... just noise ... and it does not count ... ","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'adkwmebvfp.jpeg', width = 480, height = 480)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:11:00.586764Z","iopub.execute_input":"2022-03-09T20:11:00.587453Z","iopub.status.idle":"2022-03-09T20:11:00.604211Z","shell.execute_reply.started":"2022-03-09T20:11:00.587416Z","shell.execute_reply":"2022-03-09T20:11:00.603427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.imread(IMG_PATH + 'adkwmebvfp.jpeg', cv2.IMREAD_GRAYSCALE)\n\ndisplay(Img.fromarray(img[0:1000, 3000:4000]).resize((480,480)))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:25:57.230223Z","iopub.execute_input":"2022-03-09T20:25:57.230641Z","iopub.status.idle":"2022-03-09T20:25:57.294215Z","shell.execute_reply.started":"2022-03-09T20:25:57.23061Z","shell.execute_reply":"2022-03-09T20:25:57.29319Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SOME OF THEM ARE BOLD OR ... THIN ... ","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'abnlhghfrc.jpeg', width = 480, height = 480)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.605495Z","iopub.execute_input":"2022-03-09T20:11:00.605931Z","iopub.status.idle":"2022-03-09T20:11:00.623085Z","shell.execute_reply.started":"2022-03-09T20:11:00.605895Z","shell.execute_reply":"2022-03-09T20:11:00.622368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SOME ... EVEN DO NOT KNOW ... WHAT NUMBERS ARE ... \n\nObservations:\n- on image below it is probably 2 ... - we can check this using dataset sum","metadata":{}},{"cell_type":"code","source":"train_df.query(\"id == 'asyflfhieg'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:40:20.394284Z","iopub.execute_input":"2022-03-09T20:40:20.394557Z","iopub.status.idle":"2022-03-09T20:40:20.416198Z","shell.execute_reply.started":"2022-03-09T20:40:20.394529Z","shell.execute_reply":"2022-03-09T20:40:20.414864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I can see: \n- for sure 8 + 7 + 8 = 23\n- and ... two numbers .... 1 ... and ... 2?\n\nDo you see the numbers? :)","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'asyflfhieg.jpeg', width = 480, height = 480) ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.656338Z","iopub.execute_input":"2022-03-09T20:11:00.656648Z","iopub.status.idle":"2022-03-09T20:11:00.672546Z","shell.execute_reply.started":"2022-03-09T20:11:00.656615Z","shell.execute_reply":"2022-03-09T20:11:00.671896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is 2 :)","metadata":{}},{"cell_type":"code","source":"img = cv2.imread(IMG_PATH + 'asyflfhieg.jpeg', cv2.IMREAD_GRAYSCALE)\n\ndisplay(Img.fromarray(img[0:1600, 1100:2100]).resize((480,480)))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:27:59.299466Z","iopub.execute_input":"2022-03-09T20:27:59.299926Z","iopub.status.idle":"2022-03-09T20:27:59.367115Z","shell.execute_reply.started":"2022-03-09T20:27:59.299857Z","shell.execute_reply":"2022-03-09T20:27:59.366503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.query(\"id == 'atmerkpirp'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:47:25.186546Z","iopub.execute_input":"2022-03-09T20:47:25.186834Z","iopub.status.idle":"2022-03-09T20:47:25.20056Z","shell.execute_reply.started":"2022-03-09T20:47:25.1868Z","shell.execute_reply":"2022-03-09T20:47:25.199848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below you can find different case:\n- 9 + 9 + 4 + 2 = 24\n- and ... unknown number is ... 2 :(","metadata":{}},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'atmerkpirp.jpeg', width = 480, height = 480) ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.673948Z","iopub.execute_input":"2022-03-09T20:11:00.674704Z","iopub.status.idle":"2022-03-09T20:11:00.692694Z","shell.execute_reply.started":"2022-03-09T20:11:00.674663Z","shell.execute_reply":"2022-03-09T20:11:00.691993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.query(\"id == 'awmmyedwwr'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:50:16.511416Z","iopub.execute_input":"2022-03-09T20:50:16.511715Z","iopub.status.idle":"2022-03-09T20:50:16.526496Z","shell.execute_reply.started":"2022-03-09T20:50:16.511682Z","shell.execute_reply":"2022-03-09T20:50:16.525206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'awmmyedwwr.jpeg', width = 480, height = 480)  ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T20:11:00.693853Z","iopub.execute_input":"2022-03-09T20:11:00.694661Z","iopub.status.idle":"2022-03-09T20:11:00.713511Z","shell.execute_reply.started":"2022-03-09T20:11:00.694623Z","shell.execute_reply":"2022-03-09T20:11:00.712526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## LET'S SUM THE NUMBERS ....\n\n* 2 + 6 + 6 + 5 = 19\n* 2 + 6 + 9 + 5 = 22\n* 2 + 9 + 9 + 5 = 25","metadata":{}},{"cell_type":"code","source":"train_df.query(\"id == 'azuthanjmr'\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:51:54.079039Z","iopub.execute_input":"2022-03-09T20:51:54.07935Z","iopub.status.idle":"2022-03-09T20:51:54.093203Z","shell.execute_reply.started":"2022-03-09T20:51:54.079319Z","shell.execute_reply":"2022-03-09T20:51:54.09252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Image(filename = IMG_PATH + 'azuthanjmr.jpeg', width = 480, height = 480)  ","metadata":{"execution":{"iopub.status.busy":"2022-03-09T20:11:00.714689Z","iopub.execute_input":"2022-03-09T20:11:00.714906Z","iopub.status.idle":"2022-03-09T20:11:00.749518Z","shell.execute_reply.started":"2022-03-09T20:11:00.714879Z","shell.execute_reply":"2022-03-09T20:11:00.748709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## WHAT TO DO NEXT?","metadata":{}},{"cell_type":"markdown","source":"### STEP 1. SIMPLIFY PROBLEM AND MAKE PLAIN BACKGROUND\nDataset: https://www.kaggle.com/c/ultra-mnist/discussion/312003\n<div align=\"center\"><img src=\"https://i.ibb.co/0DZYLQG/ultra-mnist.jpg\"/></div>\n","metadata":{}},{"cell_type":"markdown","source":"### STEP 2. Find Numbers - NO MODEL REQUIRED ... only IMAGE MANIPULATIONS\n\nNotebook: https://www.kaggle.com/remekkinas/step-2-find-numbers-no-model-required\n\n<div align=\"center\"><img src=\"https://i.ibb.co/w44cdZK/aaa.png\"/></div>","metadata":{}},{"cell_type":"markdown","source":"### STEP 3. MNIST number recognition","metadata":{}}]}