{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84969,"databundleVersionId":10033515,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on November 07, 2024. By Marília Prata, mpwolke","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport plotly.graph_objs as go\nimport plotly.offline as py\nimport plotly.express as px\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-11-07T03:13:51.743048Z","iopub.execute_input":"2024-11-07T03:13:51.743799Z","iopub.status.idle":"2024-11-07T03:13:56.453766Z","shell.execute_reply.started":"2024-11-07T03:13:51.743703Z","shell.execute_reply":"2024-11-07T03:13:56.452584Z"},"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Open Microscopy Environment (OME). Zarr format\n\nOME-Zarr: a cloud-optimized bioimaging file format with international community support\n\r\nCitation: Moore, J., Basurto-Lozada, D., Besson, S. et al. OME-Zarr: a cloud-optimized bioimaging file format with international community support. Histochem Cell Biol 160, 223–251 (2023). https://doi.org/10.1007/s00418-023-02209-1\r\n\r\n\"A growing community is constructing a next-generation file format (NGFF) for bioimaging to overcome problems of scalability and heterogeneity. Organized by the Open Microscopy Environment (OME), individuals and institutes across diverse modalities facing these problems have designed a format specification process (OME-NGFF) to address these needs. \"\r\n\r\n\"Over the last few years, a new data format, Zarr, has been developed for the storage of large N-dimensional typed arrays in the cloud. The Zarr format is now heavily adopted across many scientific communities from genomics to astrophysics . Zarr stores associated metadata in JSON and binary data in individually referenceable “chunk”-files, providing a flexible, scalable method for storing multidimensional data. In 2021, OME published the first specification and example uses of a “next-generation file format” (NGFF) in bioimaging using the Zarr format. The first versions of this format, OME-Zarr, focused on developing functionality that tests and demonstrates the utility of the format in bioimaging domains that routinely generate large, metadata-rich datasets—high content screening, digital pathology, electron microscopy, and light sheet imaging.\"\r\n\r\nLibraries\r\n\r\n\"Behind most of the visualization tools above and many other applications are OME-Zarr capable libraries that can be used in a wide variety of situations. Workflow systems like Nextflow or Snakemake can use them to read or write OME-Zarr data, and the same is true of machine learning pipelines like Tensorflow and PyTorch. Where dedicated widgets like ITKWidgets are not available, these libraries can make use of existing software stacks like Dask and NumPy to visualize the data in Jupyter Notebooks or to perform parallel analysis.\"","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/czii-cryo-et-object-identification/sample_submission.csv')\nsub.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:16:21.220015Z","iopub.execute_input":"2024-11-07T03:16:21.220386Z","iopub.status.idle":"2024-11-07T03:16:21.256841Z","shell.execute_reply.started":"2024-11-07T03:16:21.220353Z","shell.execute_reply":"2024-11-07T03:16:21.255446Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Code by Anmorgul https://www.kaggle.com/anmorgul/strange-pattern-cottonwood-willow\n#https://www.kaggle.com/code/mpwolke/roosevelt-forest-of-northern-colorado-charts\n\n#Two lines Required to Plot Plotly\n\nimport plotly.io as pio\npio.renderers.default = 'iframe'\n\n\nfor i in range(4,5):\n    fig = px.scatter_3d(sub, x='x', y='y', z='z',\n                  color='particle_type', size_max=8, width=800, height=800, opacity=0.9, template=\"plotly_dark\", title='3D Scatter of Particle Types')\n    fig.update_layout(\n        font_size=8,\n        legend_font_size=16,)\n    fig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:24:31.516339Z","iopub.execute_input":"2024-11-07T04:24:31.517119Z","iopub.status.idle":"2024-11-07T04:24:31.631851Z","shell.execute_reply.started":"2024-11-07T04:24:31.517076Z","shell.execute_reply":"2024-11-07T04:24:31.630658Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### I prefer not to comment my chart above. It wasn't what I expected.  ","metadata":{}},{"cell_type":"markdown","source":"## Six particle types with different difficulty levels of prediction.\n\n* apo-ferritin (easy)\n  \n* beta-amylase (impossible, not scored)\n \n* beta-galactosidase (hard)\n \n* ribosome (easy)\n  \n* thyroglobulin (hard)\n \n* virus-like-particle (easy)\n\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/data","metadata":{}},{"cell_type":"markdown","source":"### virus_like_particle json file","metadata":{}},{"cell_type":"code","source":"#JSON\nimport json\n\nvirus_like_particle = pd.read_json(path_or_buf='/kaggle/input/czii-cryo-et-object-identification/train/overlay/ExperimentRuns/TS_5_4/Picks/virus-like-particle.json')\n\nvirus_like_particle.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:26:01.213872Z","iopub.execute_input":"2024-11-07T03:26:01.214310Z","iopub.status.idle":"2024-11-07T03:26:01.239897Z","shell.execute_reply.started":"2024-11-07T03:26:01.214265Z","shell.execute_reply":"2024-11-07T03:26:01.238777Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Apoferritin file\n\nApoferritin-Based Nanoparticles as Imaging Agents\n\r\"\nApoferritin is a protein commonly present in the intestinal mucosa membrane. The important biological function of apoferritin is its ability to bind and store iron, by combining with a ferric hydroxide–phosphate compound to form ferritin\"\n\nhttps://www.sciencedirect.com/topics/chemistry/apoferritin.","metadata":{}},{"cell_type":"code","source":"apo_ferritin = pd.read_json(path_or_buf='/kaggle/input/czii-cryo-et-object-identification/train/overlay/ExperimentRuns/TS_5_4/Picks/apo-ferritin.json')\n\napo_ferritin.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:27:29.406190Z","iopub.execute_input":"2024-11-07T03:27:29.407422Z","iopub.status.idle":"2024-11-07T03:27:29.441098Z","shell.execute_reply.started":"2024-11-07T03:27:29.407366Z","shell.execute_reply":"2024-11-07T03:27:29.440044Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sub[\"y_\"] = sub[\"y\"] * -1\nexample_frame = sub.query(\"experiment == 'TS_69_2' and particle_type== 'apo-ferritin'\")\npx.scatter(example_frame, x=\"x\", y=\"y_\", color=\"particle_type\", title='Tiny Apoferritin' )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:21:06.257388Z","iopub.execute_input":"2024-11-07T04:21:06.257828Z","iopub.status.idle":"2024-11-07T04:21:06.356412Z","shell.execute_reply.started":"2024-11-07T04:21:06.257789Z","shell.execute_reply":"2024-11-07T04:21:06.355107Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### beta-amylase file\n\n\"Beta Amylase is an enzyme with a 3D structure consisting of large and small domains. It has a pocket-like catalytic site with specific amino acids serving as nucleophile and proton donor. This enzyme is unique in its exo-acting style and holds substrates tightly during catalysis.\"\r\n\r\nAI generated definition based on: \r\nEncyclopedia of Microbiology (Third Edition), 2\n\nhttps://www.sciencedirect.com/topics/neuroscience/beta-amylase009","metadata":{}},{"cell_type":"code","source":"beta_amylase = pd.read_json(path_or_buf='/kaggle/input/czii-cryo-et-object-identification/train/overlay/ExperimentRuns/TS_5_4/Picks/beta-amylase.json')\n\nbeta_amylase.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:31:33.143377Z","iopub.execute_input":"2024-11-07T03:31:33.143829Z","iopub.status.idle":"2024-11-07T03:31:33.173179Z","shell.execute_reply.started":"2024-11-07T03:31:33.143792Z","shell.execute_reply":"2024-11-07T03:31:33.171786Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### beta-galactosidase file\n\n\"β-Galactosidase (EC 3.2.1.23, beta-gal or β-gal; systematic name β-D-galactoside galactohydrolase) is a glycoside hydrolase enzyme that catalyzes hydrolysis of terminal non-reducing β-D-galactose residues in β-D-galactosides. (This enzyme digests many β-Galactosides, not just lactose.\"\n\nhttps://en.wikipedia.org/wiki/%CE%92-Galactosidase","metadata":{}},{"cell_type":"code","source":"beta_galactosidase = pd.read_json(path_or_buf='/kaggle/input/czii-cryo-et-object-identification/train/overlay/ExperimentRuns/TS_5_4/Picks/beta-galactosidase.json')\n\nbeta_galactosidase.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:35:23.098566Z","iopub.execute_input":"2024-11-07T03:35:23.100044Z","iopub.status.idle":"2024-11-07T03:35:23.134829Z","shell.execute_reply.started":"2024-11-07T03:35:23.099977Z","shell.execute_reply":"2024-11-07T03:35:23.133218Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"example_experiment = sub.query(\"experiment == 'TS_5_4'\")\npx.scatter_3d(example_experiment, x=\"x\", y=\"y\", z=\"z\", color=\"particle_type\", template=\"plotly_dark\", title=\"Beta-Galactosidase 3D Scatter\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:10:45.896913Z","iopub.execute_input":"2024-11-07T04:10:45.897370Z","iopub.status.idle":"2024-11-07T04:10:46.018556Z","shell.execute_reply.started":"2024-11-07T04:10:45.897324Z","shell.execute_reply":"2024-11-07T04:10:46.017343Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ribosome file\n\n\"A ribosome is an intercellular structure made of both RNA and protein, and it is the site of protein synthesis in the cell. The ribosome reads the messenger RNA (mRNA) sequence and translates that genetic code into a specified string of amino acids, which grow into long chains that fold to form proteins.\"\n\nhttps://www.genome.gov/genetics-glossary/Ribosome#:~:text=A%20ribosome%20is%20an%20intercellular,that%20fold%20to%20form%20proteins.","metadata":{}},{"cell_type":"code","source":"ribosome = pd.read_json(path_or_buf='/kaggle/input/czii-cryo-et-object-identification/train/overlay/ExperimentRuns/TS_5_4/Picks/ribosome.json')\n\nribosome.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:38:13.296871Z","iopub.execute_input":"2024-11-07T03:38:13.297302Z","iopub.status.idle":"2024-11-07T03:38:13.330692Z","shell.execute_reply.started":"2024-11-07T03:38:13.297263Z","shell.execute_reply":"2024-11-07T03:38:13.329692Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"example_experiment = sub.query(\"experiment == 'TS_6_4'\")\npx.scatter_3d(example_experiment, x=\"x\", y=\"y\", z=\"z\", color=\"particle_type\", template=\"plotly_dark\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:05:33.573202Z","iopub.execute_input":"2024-11-07T04:05:33.573647Z","iopub.status.idle":"2024-11-07T04:05:33.673116Z","shell.execute_reply.started":"2024-11-07T04:05:33.573611Z","shell.execute_reply":"2024-11-07T04:05:33.671766Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### thyroglobulin file\n\n\"Thyroglobulin (Tg) is a glycoprotein homodimer produced predominantly by the thryroid gland. It acts as a substrate for the synthesis of thyroxine and triiodothyronine as well as the storage of the inactive forms of thyroid hormone and iodine.\"\n\nhttps://www.ncbi.nlm.nih.gov/gene/7038#:~:text=Thyroglobulin%20(Tg)%20is%20a%20glycoprotein,of%20thyroid%20hormone%20and%20iodine.","metadata":{}},{"cell_type":"code","source":"thyroglobulin = pd.read_json(path_or_buf='/kaggle/input/czii-cryo-et-object-identification/train/overlay/ExperimentRuns/TS_5_4/Picks/thyroglobulin.json')\n\nthyroglobulin.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T03:40:38.192838Z","iopub.execute_input":"2024-11-07T03:40:38.193295Z","iopub.status.idle":"2024-11-07T03:40:38.226101Z","shell.execute_reply.started":"2024-11-07T03:40:38.193257Z","shell.execute_reply":"2024-11-07T03:40:38.224703Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!pip install zarr","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:29:46.242150Z","iopub.execute_input":"2024-11-07T04:29:46.242704Z","iopub.status.idle":"2024-11-07T04:30:02.101032Z","shell.execute_reply.started":"2024-11-07T04:29:46.242666Z","shell.execute_reply":"2024-11-07T04:30:02.099149Z"},"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ctfdeconvolved zarr file","metadata":{}},{"cell_type":"code","source":"import zarr\nctfdeconvolved = zarr.open('/kaggle/input/czii-cryo-et-object-identification/train/static/ExperimentRuns/TS_69_2/VoxelSpacing10.000/ctfdeconvolved.zarr', mode='r')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:34:16.699140Z","iopub.execute_input":"2024-11-07T04:34:16.699600Z","iopub.status.idle":"2024-11-07T04:34:16.898177Z","shell.execute_reply.started":"2024-11-07T04:34:16.699548Z","shell.execute_reply":"2024-11-07T04:34:16.897118Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#By David List https://www.kaggle.com/code/davidlist/experiment-ts-6-4-visualization\n\nprint(ctfdeconvolved)\nprint(ctfdeconvolved[0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:34:32.317513Z","iopub.execute_input":"2024-11-07T04:34:32.318478Z","iopub.status.idle":"2024-11-07T04:34:32.330525Z","shell.execute_reply.started":"2024-11-07T04:34:32.318432Z","shell.execute_reply":"2024-11-07T04:34:32.328549Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### wbp (weighted back projection) zarr file","metadata":{}},{"cell_type":"code","source":"import zarr\nwbp = zarr.open('/kaggle/input/czii-cryo-et-object-identification/train/static/ExperimentRuns/TS_5_4/VoxelSpacing10.000/wbp.zarr', mode='r')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:41:30.697911Z","iopub.execute_input":"2024-11-07T04:41:30.698975Z","iopub.status.idle":"2024-11-07T04:41:30.712703Z","shell.execute_reply.started":"2024-11-07T04:41:30.698926Z","shell.execute_reply":"2024-11-07T04:41:30.711251Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#By David List https://www.kaggle.com/code/davidlist/experiment-ts-6-4-visualization\n\nprint(wbp)\nprint(wbp[0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:42:28.789136Z","iopub.execute_input":"2024-11-07T04:42:28.789516Z","iopub.status.idle":"2024-11-07T04:42:28.801381Z","shell.execute_reply.started":"2024-11-07T04:42:28.789482Z","shell.execute_reply":"2024-11-07T04:42:28.799645Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Since wbp[0][1] and ctfdeconvolved[0] [1] are quite the same. I left only [2]","metadata":{"execution":{"iopub.status.busy":"2024-11-07T04:46:40.617231Z","iopub.execute_input":"2024-11-07T04:46:40.617623Z","iopub.status.idle":"2024-11-07T04:46:45.663285Z","shell.execute_reply.started":"2024-11-07T04:46:40.617581Z","shell.execute_reply":"2024-11-07T04:46:45.662101Z"}}},{"cell_type":"code","source":"#By David List https://www.kaggle.com/code/davidlist/experiment-ts-6-4-visualization\n\nfig = plt.figure(figsize=(10,10))\nfor i in range(25):\n    ax = plt.subplot(5, 5, i + 1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.imshow(ctfdeconvolved[2][i])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:36:35.551503Z","iopub.execute_input":"2024-11-07T04:36:35.552802Z","iopub.status.idle":"2024-11-07T04:36:37.717645Z","shell.execute_reply.started":"2024-11-07T04:36:35.552744Z","shell.execute_reply":"2024-11-07T04:36:37.716434Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### wbp[2]","metadata":{}},{"cell_type":"code","source":"#By David List https://www.kaggle.com/code/davidlist/experiment-ts-6-4-visualization\n\nfig = plt.figure(figsize=(10,10))\nfor i in range(25):\n    ax = plt.subplot(5, 5, i + 1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.imshow(wbp[2][i])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-07T04:44:45.580868Z","iopub.execute_input":"2024-11-07T04:44:45.581247Z","iopub.status.idle":"2024-11-07T04:44:47.769623Z","shell.execute_reply.started":"2024-11-07T04:44:45.581214Z","shell.execute_reply":"2024-11-07T04:44:47.768454Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### I tried also to open the Zarr with Dask and xarray, unfortunately I wasn't well succeeded. ","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nDavid List https://www.kaggle.com/code/davidlist/experiment-ts-6-4-visualization","metadata":{}}]}