{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## COMPUTER VISION EDA - INTERESTING CASES IN ULTRA-MNIST DATASET\n\nIn computer vision the most important part is to look visually into data (this is FIRST STEP) ... so let's look into \"interesting\" cases in dataset ...\n\nEDA in computer vision is different then in tabular data. We must work visually on each step (even visual validation of our model performance). My CV workflow for data:\n\n* look into data - spend 80% working with images (I always download whole dataset and watching on photos)\n* visualize images (or … visualize using videos)\n* visualize prediction (classification result, bboxes, segmentations etc.) - score is not enough - we should see prediction and well and find, understad anomalies to improve model.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import cv2\nfrom PIL import Image as PILImage\nfrom IPython.display import Image\n\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:40:15.933957Z","iopub.execute_input":"2022-03-09T22:40:15.934360Z","iopub.status.idle":"2022-03-09T22:40:16.381801Z","shell.execute_reply.started":"2022-03-09T22:40:15.934261Z","shell.execute_reply":"2022-03-09T22:40:16.380872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls ../input","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:41:30.432453Z","iopub.execute_input":"2022-03-09T22:41:30.433290Z","iopub.status.idle":"2022-03-09T22:41:31.188061Z","shell.execute_reply.started":"2022-03-09T22:41:30.433235Z","shell.execute_reply":"2022-03-09T22:41:31.186936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/ultra-mnist/train.csv\")\nIMG_PATH = \"../input/ultra-mnist/train/\"","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:46:37.696228Z","iopub.execute_input":"2022-03-09T22:46:37.697193Z","iopub.status.idle":"2022-03-09T22:46:37.726166Z","shell.execute_reply.started":"2022-03-09T22:46:37.697136Z","shell.execute_reply":"2022-03-09T22:46:37.725500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:46:39.748057Z","iopub.execute_input":"2022-03-09T22:46:39.748534Z","iopub.status.idle":"2022-03-09T22:46:39.757926Z","shell.execute_reply.started":"2022-03-09T22:46:39.748500Z","shell.execute_reply":"2022-03-09T22:46:39.757063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ONE DAY SMALL AND BIG NUMBERS MET ON CHECKED BOARD AT ONCE ... AND ...\n\nObservations:\n- background could be challanging in this competition.\n- Since we have regular rectangle we can\n    - simplify workflow and \n    - remove checkboard at first step\n    \nsee below proposition","metadata":{}},{"cell_type":"code","source":"type(IMG_PATH)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:47:40.668237Z","iopub.execute_input":"2022-03-09T22:47:40.668542Z","iopub.status.idle":"2022-03-09T22:47:40.674533Z","shell.execute_reply.started":"2022-03-09T22:47:40.668506Z","shell.execute_reply":"2022-03-09T22:47:40.673626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_image(id=\"aaofnmvaxy\", IMG_PATH=IMG_PATH, train_df=train_df):\n    fname = IMG_PATH + f\"{id}.jpeg\"\n    print(train_df[train_df[\"id\"]==id])\n    print(\"---\"*20)\n    return Image(filename=fname, width=480, height=480)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:56:12.716364Z","iopub.execute_input":"2022-03-09T22:56:12.716846Z","iopub.status.idle":"2022-03-09T22:56:12.722776Z","shell.execute_reply.started":"2022-03-09T22:56:12.716799Z","shell.execute_reply":"2022-03-09T22:56:12.721956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_image(id=\"aaofnmvaxy\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:56:13.725478Z","iopub.execute_input":"2022-03-09T22:56:13.725745Z","iopub.status.idle":"2022-03-09T22:56:13.747294Z","shell.execute_reply.started":"2022-03-09T22:56:13.725717Z","shell.execute_reply":"2022-03-09T22:56:13.746496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... EVEN SMALLER CAME AS WELL ... \n\nObservations:\n- numbers are different in size \n- it could be challanging for object detector to see BIG numbers and very small at the same time \n- proposition -> `SAHI` (Slicing Aided Hyper Inference) or tiling","metadata":{}},{"cell_type":"code","source":"show_image(id=\"acfyvsrlck\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:56:46.245109Z","iopub.execute_input":"2022-03-09T22:56:46.246074Z","iopub.status.idle":"2022-03-09T22:56:46.275329Z","shell.execute_reply.started":"2022-03-09T22:56:46.246030Z","shell.execute_reply":"2022-03-09T22:56:46.274407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... OR EVEN INVISIBLE .... ","metadata":{}},{"cell_type":"code","source":"show_image(id=\"acybfuinqq\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T22:57:30.331272Z","iopub.execute_input":"2022-03-09T22:57:30.332178Z","iopub.status.idle":"2022-03-09T22:57:30.365122Z","shell.execute_reply.started":"2022-03-09T22:57:30.332109Z","shell.execute_reply":"2022-03-09T22:57:30.364350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BUT WHEN WE SLICE THEM ... IT LOOKS GREAT ...\n\n","metadata":{}},{"cell_type":"code","source":"def show_slice(id, x0, x1, y0, y1, IMG_PATH=IMG_PATH):\n    fname = IMG_PATH + f\"{id}.jpeg\"\n    img = cv2.imread(fname, cv2.IMREAD_GRAYSCALE)\n    return display(PILImage.fromarray(img[x0:x1, y0:y1]).resize((480, 480)))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:04:59.171267Z","iopub.execute_input":"2022-03-09T23:04:59.172049Z","iopub.status.idle":"2022-03-09T23:04:59.177803Z","shell.execute_reply.started":"2022-03-09T23:04:59.171994Z","shell.execute_reply":"2022-03-09T23:04:59.176686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_slice(id=\"acybfuinqq\", x0=1000, x1=2000, y0=0, y1=1000)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:05:27.906154Z","iopub.execute_input":"2022-03-09T23:05:27.906942Z","iopub.status.idle":"2022-03-09T23:05:27.982477Z","shell.execute_reply.started":"2022-03-09T23:05:27.906893Z","shell.execute_reply":"2022-03-09T23:05:27.981587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## THEY HAVE A PROBLEM WITH THEMSELVES ... THEY CAN'T DECIDE IF THEY ARE SIX OR NINE ...\n\n- For sure it will be challanging to distinguish between 6 and 9 - we have to deal with this in inference part (some rules during number summary)\n- During training **FLIP augumentation** will be helpfull eg. below you can find 9/6 flipped/mirrored","metadata":{}},{"cell_type":"code","source":"show_image(id=\"afljkeleeq\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:08:50.857393Z","iopub.execute_input":"2022-03-09T23:08:50.857731Z","iopub.status.idle":"2022-03-09T23:08:50.880551Z","shell.execute_reply.started":"2022-03-09T23:08:50.857698Z","shell.execute_reply":"2022-03-09T23:08:50.878967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_slice(\"afljkeleeq\", 1000, 2000, 3000, 4000)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:10:09.210002Z","iopub.execute_input":"2022-03-09T23:10:09.210309Z","iopub.status.idle":"2022-03-09T23:10:09.286061Z","shell.execute_reply.started":"2022-03-09T23:10:09.210272Z","shell.execute_reply":"2022-03-09T23:10:09.285245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... TWO ... OR MAYBE FIVE ... OR MAYBE TWO OR ...EIGHT ????\n\nObservations:\n- Some digits are unknown - below you can recognize it as a 5 or 2 (flipped)","metadata":{}},{"cell_type":"code","source":"show_image(id=\"abgmdxywua\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:15:00.201737Z","iopub.execute_input":"2022-03-09T23:15:00.203045Z","iopub.status.idle":"2022-03-09T23:15:00.238900Z","shell.execute_reply.started":"2022-03-09T23:15:00.202985Z","shell.execute_reply":"2022-03-09T23:15:00.237219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_slice(\"abgmdxywua\", 0, 1000, 3000, 4000)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:16:02.323270Z","iopub.execute_input":"2022-03-09T23:16:02.323616Z","iopub.status.idle":"2022-03-09T23:16:02.402022Z","shell.execute_reply.started":"2022-03-09T23:16:02.323580Z","shell.execute_reply":"2022-03-09T23:16:02.400804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SOME OF THEM MASK WELL ... WANT TO HIDE WHAT NUMBER THEY ARE ...\n\nObservations:\n- Some numbers ar clipped - augumentations CUTOUT could be helpfull","metadata":{}},{"cell_type":"code","source":"show_image(\"akscpxgert\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:16:49.969639Z","iopub.execute_input":"2022-03-09T23:16:49.970017Z","iopub.status.idle":"2022-03-09T23:16:50.002031Z","shell.execute_reply.started":"2022-03-09T23:16:49.969982Z","shell.execute_reply":"2022-03-09T23:16:50.000851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_slice(\"akscpxgert\", 3000, 4000, 0, 1000)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:17:14.747478Z","iopub.execute_input":"2022-03-09T23:17:14.747948Z","iopub.status.idle":"2022-03-09T23:17:14.837353Z","shell.execute_reply.started":"2022-03-09T23:17:14.747916Z","shell.execute_reply":"2022-03-09T23:17:14.836341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ... OR EVEN WORSE ... THEY TRY TO IMITATE OTHER NUMBERS ...\n\nObservations:\n- it could be a number or ... just noise ... and it does not count ... ","metadata":{}},{"cell_type":"code","source":"show_image(\"adkwmebvfp\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:17:49.805782Z","iopub.execute_input":"2022-03-09T23:17:49.806482Z","iopub.status.idle":"2022-03-09T23:17:49.841245Z","shell.execute_reply.started":"2022-03-09T23:17:49.806410Z","shell.execute_reply":"2022-03-09T23:17:49.840302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_slice(\"adkwmebvfp\", 0, 1000, 3000, 4000)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:18:12.464803Z","iopub.execute_input":"2022-03-09T23:18:12.465096Z","iopub.status.idle":"2022-03-09T23:18:12.542193Z","shell.execute_reply.started":"2022-03-09T23:18:12.465068Z","shell.execute_reply":"2022-03-09T23:18:12.541514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SOME OF THEM ARE BOLD OR ... THIN ... ","metadata":{}},{"cell_type":"code","source":"show_image(\"abnlhghfrc\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:19:01.524636Z","iopub.execute_input":"2022-03-09T23:19:01.524950Z","iopub.status.idle":"2022-03-09T23:19:01.555206Z","shell.execute_reply.started":"2022-03-09T23:19:01.524915Z","shell.execute_reply":"2022-03-09T23:19:01.554328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SOME ... EVEN DO NOT KNOW ... WHAT NUMBERS ARE ... \n\nObservations:\n- on image below it is probably 2 ... - we can check this using dataset sum","metadata":{}},{"cell_type":"code","source":"show_image(id=\"asyflfhieg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:20:10.516247Z","iopub.execute_input":"2022-03-09T23:20:10.516816Z","iopub.status.idle":"2022-03-09T23:20:10.547861Z","shell.execute_reply.started":"2022-03-09T23:20:10.516760Z","shell.execute_reply":"2022-03-09T23:20:10.546697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I can see: \n- for sure 8 + 7 + 8 = 23\n- and ... two numbers .... 1 ... and ... 2?\n\nDo you see the numbers? :)","metadata":{}},{"cell_type":"markdown","source":"This is 2 :)","metadata":{}},{"cell_type":"code","source":"show_slice(\"asyflfhieg\", 0, 1600, 1100, 2100)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:21:13.517570Z","iopub.execute_input":"2022-03-09T23:21:13.518824Z","iopub.status.idle":"2022-03-09T23:21:13.590461Z","shell.execute_reply.started":"2022-03-09T23:21:13.518768Z","shell.execute_reply":"2022-03-09T23:21:13.589500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-----------------------------------","metadata":{}},{"cell_type":"code","source":"show_image(\"atmerkpirp\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:21:42.209708Z","iopub.execute_input":"2022-03-09T23:21:42.210014Z","iopub.status.idle":"2022-03-09T23:21:42.239693Z","shell.execute_reply.started":"2022-03-09T23:21:42.209979Z","shell.execute_reply":"2022-03-09T23:21:42.238798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below you can find different case:\n- 9 + 9 + 4 + 2 = 24\n- and ... unknown number is ... 2 :(","metadata":{}},{"cell_type":"markdown","source":"--------------------------------------","metadata":{}},{"cell_type":"code","source":"show_image(\"awmmyedwwr\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:23:19.577330Z","iopub.execute_input":"2022-03-09T23:23:19.578216Z","iopub.status.idle":"2022-03-09T23:23:19.608107Z","shell.execute_reply.started":"2022-03-09T23:23:19.578160Z","shell.execute_reply":"2022-03-09T23:23:19.607035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------------------------","metadata":{}},{"cell_type":"markdown","source":"## LET'S SUM THE NUMBERS ....\n\n* 2 + 6 + 6 + 5 = 19\n* 2 + 6 + 9 + 5 = 22\n* 2 + 9 + 9 + 5 = 25","metadata":{}},{"cell_type":"code","source":"show_image(\"azuthanjmr\")","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:24:15.290119Z","iopub.execute_input":"2022-03-09T23:24:15.290737Z","iopub.status.idle":"2022-03-09T23:24:15.330411Z","shell.execute_reply.started":"2022-03-09T23:24:15.290696Z","shell.execute_reply":"2022-03-09T23:24:15.329492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## WHAT TO DO?","metadata":{}},{"cell_type":"markdown","source":"### STEP 1. SIMPLIFY PROBLEM AND MAKE PLAIN BACKGROUND\n\n<div align=\"center\"><img src=\"https://i.ibb.co/0DZYLQG/ultra-mnist.jpg\"/></div>","metadata":{}},{"cell_type":"markdown","source":"### STEP 2. IN PROGRESS","metadata":{}}]}