{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on March 15, 2023. By Marília Prata, mpwolke","metadata":{"_kg_hide-input":false}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-15T19:40:00.332033Z","iopub.execute_input":"2023-03-15T19:40:00.332782Z","iopub.status.idle":"2023-03-15T19:40:00.490089Z","shell.execute_reply.started":"2023-03-15T19:40:00.332733Z","shell.execute_reply":"2023-03-15T19:40:00.488989Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition Citation\n\n@misc{vesuvius-challenge-ink-detection,\n\n    author = {Christy Chapman, Daniel Havir, Ian Janicki, Jonny Hyman, JP Posma, Nat \n    \n    Friedman, Ryan Holbrook, Stephen Parsons, Will Cukierski},\n    \n    title = {Vesuvius Challenge - Ink Detection},\n    \n    publisher = {Kaggle},\n    \n    year = {2023},\n    \n    url = {https://kaggle.com/competitions/vesuvius-challenge-ink-detection}\n}","metadata":{}},{"cell_type":"markdown","source":"![](https://perunrace.com/wp-content/uploads/2022/01/Vesuvius-Volcano-virtual-challenge-1-1.jpg)https://perunrace.com/product/vesuvius-virtual-run-virtual-challenge/","metadata":{}},{"cell_type":"markdown","source":"<center style=\"font-family:verdana;\"><h1 style=\"font-size:200%; padding: 10px; background: Sienna;\"><b style=\"color:#DAA520;\">Vesuvius Challenge</b></h1></center>\n\n\"Resurrect an ancient library from the ashes of a volcano. The Vesuvius Challenge is a machine learning and computer vision competition to read the Herculaneum Papyri.\"\n\n79 AD - MOUNT VESUVIUS ERUPTS.\n\n\"In Herculaneum, twenty meters of hot mud and ash bury an enormous villa once owned by the father-in-law of Julius Caesar. Inside, there is a vast library of papyrus scrolls.\"\n\n\"The scrolls are carbonized by the heat of the volcanic debris. But they are also preserved. For centuries, as virtually every ancient text exposed to the air decays and disappears, the library of the Villa of the Papyri waits underground, intact.\"\n\n1750 AD - A FARMER DISCOVERS THE BURIED VILLA.\n\n\"While digging a well, an Italian farmworker encounters a marble pavement. Excavations unearth beautiful statues and frescoes – and hundreds of scrolls. Carbonized and ashen, they are extremely fragile. But the temptation to open them is great; if read, they would more than double the corpus of literature we have from antiquity.\"\n\n\"Early attempts to open the scrolls unfortunately destroy many of them. A few are painstakingly unrolled by an Italian monk over several decades, and they are found to contain philosophical texts written in Greek. More than six hundred remain unopened and unreadable.\"\n\n2015 AD - DR. BRENT SEALES PIONEERS VIRTUAL UNWRAPPING.\n\n\"Using X-ray tomography and computer vision, a team led by Dr. Brent Seales at the University of Kentucky reads the En-Gedi scroll without opening it. Discovered in the Dead Sea region of Israel, the scroll is found to contain text from the book of Leviticus.\"\n\n\"This achievement shows that a carbonized scroll can be digitally unrolled and read without physically opening it. Virtual unwrapping has since emerged as a growing field with multiple successes.\" \n\n\"But the Herculaneum Papyri prove more challenging: unlike the denser inks used in the En-Gedi scroll, the Herculaneum ink is carbon-based, affording no X-ray contrast against the underlying carbon-based papyrus.\"\n\n2019 AD - ENTER THE PARTICLE ACCELERATOR.\n\n\"Determined to apply virtual unwrapping to the Herculaneum Papyri, Dr. Seales and his team set out to test a new idea. Under infrared light, some detached fragments of the papyri are readable, and it seems possible that these can be used as ground truth data for a machine learning model that could detect otherwise invisible ink from X-rays.\"\n\nhttps://scrollprize.org/","metadata":{}},{"cell_type":"code","source":"!pip install gdal","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-15T20:28:19.347828Z","iopub.execute_input":"2023-03-15T20:28:19.348331Z","iopub.status.idle":"2023-03-15T20:28:23.421277Z","shell.execute_reply.started":"2023-03-15T20:28:19.348291Z","shell.execute_reply":"2023-03-15T20:28:23.419741Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#That Gdal above I didn't use it","metadata":{}},{"cell_type":"code","source":"import os, sys, random, gc, math, glob, time, pathlib\nimport numpy as np\nimport pandas as pd\nimport io, timeit, os, gc, pickle, psutil\nimport warnings\nimport cv2\n#import gdal\n#import osgeo\nimport json\nimport rasterio\nfrom rasterio.windows import Window\n\nimport seaborn as sns\nsns.set()\nsns.set_context(\"paper\", font_scale=1.2) \n\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport matplotlib.colors as cols\n\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2023-03-15T21:20:44.131065Z","iopub.execute_input":"2023-03-15T21:20:44.131498Z","iopub.status.idle":"2023-03-15T21:20:45.432386Z","shell.execute_reply.started":"2023-03-15T21:20:44.131455Z","shell.execute_reply":"2023-03-15T21:20:45.431189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by MPWARE https://www.kaggle.com/code/mpware/masks-quick-eda-updated-data/notebook\n\nprint('Python        : ' + sys.version.split('\\n')[0])\nprint('Numpy         : ' + np.__version__)\nprint('Pandas        : ' + pd.__version__)\nprint('Rasterio      : ' + rasterio.__version__)\n#print('GDal          : ' + osgeo.gdal.__version__)\nprint('OpenCV        : ' + cv2.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-03-15T20:29:17.023283Z","iopub.execute_input":"2023-03-15T20:29:17.024615Z","iopub.status.idle":"2023-03-15T20:29:17.032125Z","shell.execute_reply.started":"2023-03-15T20:29:17.024562Z","shell.execute_reply":"2023-03-15T20:29:17.030824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\nimport time\nimport datetime\nimport glob\n\nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sn\n\nimport json\nimport numpy as np\nimport os\nimport pandas as pd\nimport tifffile as tiff\nfrom matplotlib import colors\nfrom matplotlib import pyplot as plt\nfrom matplotlib.lines import Line2D\nfrom matplotlib_venn import venn2_unweighted","metadata":{"execution":{"iopub.status.busy":"2023-03-15T21:20:51.706341Z","iopub.execute_input":"2023-03-15T21:20:51.707220Z","iopub.status.idle":"2023-03-15T21:20:51.876466Z","shell.execute_reply.started":"2023-03-15T21:20:51.707170Z","shell.execute_reply":"2023-03-15T21:20:51.875079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Georgii Kostiuchik https://www.kaggle.com/code/georgiikostiuchik/hubmap-exploratory-data-analysis\n\ntrain_images = glob.glob('/kaggle/input/vesuvius-challenge-ink-detection/train/*/surface_volume/*.tiff')\ntest_images = glob.glob('/kaggle/input/vesuvius-challenge-ink-detection/test/*/surface_volume/*.tiff')\n\ntrain_images = list(map(lambda x: os.path.basename(x), train_images))\ntest_images = list(map(lambda x: os.path.basename(x), test_images))","metadata":{"execution":{"iopub.status.busy":"2023-03-15T20:35:08.275262Z","iopub.execute_input":"2023-03-15T20:35:08.276881Z","iopub.status.idle":"2023-03-15T20:35:08.312138Z","shell.execute_reply.started":"2023-03-15T20:35:08.276818Z","shell.execute_reply":"2023-03-15T20:35:08.311040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Georgii Kostiuchik https://www.kaggle.com/code/georgiikostiuchik/hubmap-exploratory-data-analysis\n\n# open and resize image\nimage = cv2.imread('/kaggle/input/vesuvius-challenge-ink-detection/train/1/surface_volume/04.tif')\nimage_resize = cv2.resize(image,(image.shape[1]//10,image.shape[0]//10), interpolation = cv2.INTER_CUBIC)","metadata":{"execution":{"iopub.status.busy":"2023-03-15T20:38:30.923050Z","iopub.execute_input":"2023-03-15T20:38:30.924268Z","iopub.status.idle":"2023-03-15T20:38:33.048034Z","shell.execute_reply.started":"2023-03-15T20:38:30.924211Z","shell.execute_reply":"2023-03-15T20:38:33.046657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Georgii Kostiuchik https://www.kaggle.com/code/georgiikostiuchik/hubmap-exploratory-data-analysis\n\n# calculate colors\npixel_colors = image_resize.reshape((np.shape(image_resize)[0]*np.shape(image_resize)[1], 3))\nnorm = colors.Normalize(vmin=-1.,vmax=1.)\nnorm.autoscale(pixel_colors)\npixel_colors = norm(pixel_colors).tolist()\n\n# split channels\nb, g, r = cv2.split(image_resize)\n\n# scatter plot\nfig = plt.figure()\naxis = fig.add_subplot(1, 1, 1, projection='3d')\naxis.scatter(r.flatten(), g.flatten(), b.flatten(), facecolors=pixel_colors, marker='.')\naxis.set_xlabel('Red')\naxis.set_ylabel('Green')\naxis.set_zlabel('Blue')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-15T20:39:01.815880Z","iopub.execute_input":"2023-03-15T20:39:01.816696Z","iopub.status.idle":"2023-03-15T20:39:11.095190Z","shell.execute_reply.started":"2023-03-15T20:39:01.816655Z","shell.execute_reply":"2023-03-15T20:39:11.093908Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Georgii Kostiuchik https://www.kaggle.com/code/georgiikostiuchik/hubmap-exploratory-data-analysis\n\n# convert to hsv\nhsv_image = cv2.cvtColor(image_resize, cv2.COLOR_BGR2HSV)\nh, s, v = cv2.split(hsv_image)\n\n# scatter plot\nfig = plt.figure()\naxis = fig.add_subplot(1, 1, 1, projection='3d')\naxis.scatter(s.flatten(), h.flatten(), v.flatten(), facecolors=pixel_colors, marker='.')\naxis.set_xlabel('Saturation')\naxis.set_ylabel('Hue')\naxis.set_zlabel('Value')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-15T20:39:33.452726Z","iopub.execute_input":"2023-03-15T20:39:33.453907Z","iopub.status.idle":"2023-03-15T20:39:41.916156Z","shell.execute_reply.started":"2023-03-15T20:39:33.453851Z","shell.execute_reply":"2023-03-15T20:39:41.914922Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#not exactly what I expected though ancient papyrus scrolls are Not the same of the kidney ","metadata":{}},{"cell_type":"code","source":"DATA_PATH = \"../input/vesuvius-challenge-ink-detection/\"","metadata":{"execution":{"iopub.status.busy":"2023-03-15T21:01:30.183686Z","iopub.execute_input":"2023-03-15T21:01:30.184142Z","iopub.status.idle":"2023-03-15T21:01:30.189916Z","shell.execute_reply.started":"2023-03-15T21:01:30.184107Z","shell.execute_reply":"2023-03-15T21:01:30.188368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_paths = sorted(glob.glob(os.path.join(DATA_PATH, 'train/*/surface_volume/*.tif')))","metadata":{"execution":{"iopub.status.busy":"2023-03-15T21:01:35.028670Z","iopub.execute_input":"2023-03-15T21:01:35.029126Z","iopub.status.idle":"2023-03-15T21:01:35.074070Z","shell.execute_reply.started":"2023-03-15T21:01:35.029086Z","shell.execute_reply":"2023-03-15T21:01:35.072736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Number of images: {len(image_paths)}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-15T20:43:49.450515Z","iopub.execute_input":"2023-03-15T20:43:49.451594Z","iopub.status.idle":"2023-03-15T20:43:49.456410Z","shell.execute_reply.started":"2023-03-15T20:43:49.451545Z","shell.execute_reply":"2023-03-15T20:43:49.455439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Ihelon https://www.kaggle.com/code/ihelon/illustrations-kumapi390-eda\n\nimage_shapes = {}\n\nfor ind, image_path in enumerate(image_paths):\n    image = cv2.imread(image_path)\n    image_shapes[image.shape] = image_shapes.get(image.shape, 0) + 1\n    \nimage_shapes","metadata":{"execution":{"iopub.status.busy":"2023-03-15T21:01:44.110748Z","iopub.execute_input":"2023-03-15T21:01:44.112184Z","iopub.status.idle":"2023-03-15T21:09:56.723533Z","shell.execute_reply.started":"2023-03-15T21:01:44.112133Z","shell.execute_reply":"2023-03-15T21:09:56.722002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Ihelon https://www.kaggle.com/code/ihelon/illustrations-kumapi390-eda\n\nn_cols = 4\n\nfor ind, image_path in enumerate(image_paths):\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    \n    if ind % n_cols == 0:\n        plt.figure(figsize=(16, 5))\n    plt.subplot(1, n_cols, ind % n_cols + 1)\n    plt.imshow(image)\n    plt.axis(\"off\")\n    if ind % n_cols == n_cols - 1:\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-15T21:11:59.017065Z","iopub.execute_input":"2023-03-15T21:11:59.017564Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#I have stopped the code above since it was taking too long. Then I tried other snippets that didn't work.","metadata":{}},{"cell_type":"markdown","source":"#Conclusion\n\nSince all the codes I read on past competitions involves csv files I'll wait to learn how to show images with RLE functions without csv files.","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-15T21:36:50.448996Z","iopub.execute_input":"2023-03-15T21:36:50.449524Z","iopub.status.idle":"2023-03-15T21:36:50.458875Z","shell.execute_reply.started":"2023-03-15T21:36:50.449472Z","shell.execute_reply":"2023-03-15T21:36:50.457756Z"}}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nIhelon https://www.kaggle.com/code/ihelon/illustrations-kumapi390-eda\n\nMPWARE https://www.kaggle.com/code/mpware/masks-quick-eda-updated-data/notebook\n\nGeorgii Kostiuchik https://www.kaggle.com/code/georgiikostiuchik/hubmap-exploratory-data-analysis","metadata":{}}]}