{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.14"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84969,"databundleVersionId":10033515,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":192.965049,"end_time":"2024-11-12T15:04:20.012449","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-11-12T15:01:07.047400","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"1b7e3316","cell_type":"markdown","source":"# ZARR - A BREIF INTRO ","metadata":{"papermill":{"duration":0.019225,"end_time":"2024-11-12T15:01:10.553460","exception":false,"start_time":"2024-11-12T15:01:10.534235","status":"completed"},"tags":[]}},{"id":"acfb1b05","cell_type":"markdown","source":"I have come across .zarr data type frequently these days ,\n\nso lets understand how to use it , meanwhile will also see why its important.\n\n\n\nZarr is an open-source Python library designed for the efficient storage and manipulation of large, N-dimensional arrays. It enables the creation of chunked and compressed arrays, facilitating scalable data processing, especially in parallel computing environments.","metadata":{"papermill":{"duration":0.018769,"end_time":"2024-11-12T15:01:10.590673","exception":false,"start_time":"2024-11-12T15:01:10.571904","status":"completed"},"tags":[]}},{"id":"edb6baf2","cell_type":"code","source":"pip install zarr","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:10.629896Z","iopub.status.busy":"2024-11-12T15:01:10.629422Z","iopub.status.idle":"2024-11-12T15:01:31.487046Z","shell.execute_reply":"2024-11-12T15:01:31.485587Z"},"papermill":{"duration":20.881955,"end_time":"2024-11-12T15:01:31.490486","exception":false,"start_time":"2024-11-12T15:01:10.608531","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"e783f949","cell_type":"markdown","source":"# ZARR CREATION OPTIONS","metadata":{"papermill":{"duration":0.020111,"end_time":"2024-11-12T15:01:31.535077","exception":false,"start_time":"2024-11-12T15:01:31.514966","status":"completed"},"tags":[]}},{"id":"ce7d1663","cell_type":"code","source":"import zarr\n\nimport numpy as np \n\n\n\n\n\n# Maping  each option to the corresponding Zarr function\n\n# why? - becz its clean and ez to understand \n\n\n\noptions = {\n\n    \"create\": zarr.create,            # Initialize new array with shape, chunks, and dtype\n\n    \"array\": zarr.array,              # Create array from existing data\n\n    \"ones\": zarr.ones,                # Create array filled with ones\n\n    \"zeros\": zarr.zeros,              # Create array filled with zeros\n\n    \"empty\": zarr.empty,              # Create uninitialized array\n\n    \"full\": zarr.full,                # Create array filled with specified value\n\n    \"ones_like\": zarr.ones_like,      # Create ones array matching shape/dtype of existing array\n\n    \"zeros_like\": zarr.zeros_like,    # Create zeros array matching shape/dtype of existing array\n\n    \"empty_like\": zarr.empty_like,    # Create uninitialized array matching shape/dtype of existing array\n\n    \"full_like\": zarr.full_like,      # Create array with specified value, matching shape/dtype\n\n    \"open_array\": zarr.open_array,    # Open or create array with specified parameters\n\n    \"open_like\": zarr.open_like,      # Open array matching shape/dtype of existing array\n\n}\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:31.583302Z","iopub.status.busy":"2024-11-12T15:01:31.582817Z","iopub.status.idle":"2024-11-12T15:01:31.723750Z","shell.execute_reply":"2024-11-12T15:01:31.722297Z"},"papermill":{"duration":0.171772,"end_time":"2024-11-12T15:01:31.726782","exception":false,"start_time":"2024-11-12T15:01:31.555010","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"2d33989a","cell_type":"markdown","source":"\n\ntheir are some zarr functionalities and then there some like functionalities as well\n\nlike functionalities relies on the data passed as argument \n\nbecause the reuslting zarr array will have same dtype and shape\n","metadata":{"papermill":{"duration":0.028443,"end_time":"2024-11-12T15:01:31.775469","exception":false,"start_time":"2024-11-12T15:01:31.747026","status":"completed"},"tags":[]}},{"id":"1bcb6d4e","cell_type":"code","source":"data = np.random.rand(10**4,10**4)\n\nz = zarr.array(data,chunks=(1000,1000))  #zarr.array creates an array out of numpy array\n\nz[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:31.830249Z","iopub.status.busy":"2024-11-12T15:01:31.828511Z","iopub.status.idle":"2024-11-12T15:01:34.614494Z","shell.execute_reply":"2024-11-12T15:01:34.613096Z"},"papermill":{"duration":2.816181,"end_time":"2024-11-12T15:01:34.618006","exception":false,"start_time":"2024-11-12T15:01:31.801825","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"bc56d97d","cell_type":"code","source":"#The options - create , ones , zeros, empty have similar syntax\n\nz1 = options['create'](shape=(100, 100), chunks=(10, 10), dtype='f8')\n\n\n\n\n\n#The optioins ones_like, zeros_like , empty_like , array have similar syntax \n\nz2 = options['zeros_like'](data, chunks=(10, 10))","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:34.663229Z","iopub.status.busy":"2024-11-12T15:01:34.662767Z","iopub.status.idle":"2024-11-12T15:01:34.669954Z","shell.execute_reply":"2024-11-12T15:01:34.668744Z"},"papermill":{"duration":0.032445,"end_time":"2024-11-12T15:01:34.672546","exception":false,"start_time":"2024-11-12T15:01:34.640101","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"f0a8d121","cell_type":"code","source":"#then there is full which goes for  initialising the array by filling it with a value \n\nz3 = zarr.full(shape=(100, 100), fill_value=42, chunks=(10, 10), dtype='f8')\n\n\n\n\n\n#full_like as the name goes full like a data shape and its dtype \n\nz4 = zarr.full_like(data, fill_value=42, chunks=(10, 10))","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:34.716230Z","iopub.status.busy":"2024-11-12T15:01:34.715780Z","iopub.status.idle":"2024-11-12T15:01:34.723713Z","shell.execute_reply":"2024-11-12T15:01:34.722395Z"},"papermill":{"duration":0.033234,"end_time":"2024-11-12T15:01:34.726596","exception":false,"start_time":"2024-11-12T15:01:34.693362","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"bc598995","cell_type":"code","source":"#open_array\n\nz11 = zarr.open_array('data/example.zarr', mode='w', shape=(100, 100), chunks=(10, 10), dtype='f8')\n\n\n\n#open_like the datat provided , by using its shape and dtype\n\nz12 = zarr.open_like(data, 'data/example_like.zarr', mode='w', chunks=(10, 10))","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:34.771809Z","iopub.status.busy":"2024-11-12T15:01:34.771395Z","iopub.status.idle":"2024-11-12T15:01:34.780791Z","shell.execute_reply":"2024-11-12T15:01:34.779361Z"},"papermill":{"duration":0.034929,"end_time":"2024-11-12T15:01:34.784056","exception":false,"start_time":"2024-11-12T15:01:34.749127","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"6cbd2577","cell_type":"code","source":"zarr.save('/kaggle/working/data.zarr',data,path='1st_data')","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:34.829866Z","iopub.status.busy":"2024-11-12T15:01:34.829005Z","iopub.status.idle":"2024-11-12T15:01:36.398059Z","shell.execute_reply":"2024-11-12T15:01:36.396776Z"},"papermill":{"duration":1.594827,"end_time":"2024-11-12T15:01:36.401086","exception":false,"start_time":"2024-11-12T15:01:34.806259","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"1f99f529","cell_type":"code","source":"'''\n\nso elements are float64 , which means that each element accquired 8 bytes\n\nso 10**8 elements would be requiring 800 mb accquired space but it settled in 669.3 mb \n\n'''\n\ndata[0][0].dtype \n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:36.444097Z","iopub.status.busy":"2024-11-12T15:01:36.443664Z","iopub.status.idle":"2024-11-12T15:01:36.452731Z","shell.execute_reply":"2024-11-12T15:01:36.451276Z"},"papermill":{"duration":0.033397,"end_time":"2024-11-12T15:01:36.455425","exception":false,"start_time":"2024-11-12T15:01:36.422028","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"af5a72b8","cell_type":"code","source":"saved  =  ((z.nbytes-z.nbytes_stored)/z.nbytes)*100\n\nprint(f\"Memory saved by {saved:.2f} %\")","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:36.500873Z","iopub.status.busy":"2024-11-12T15:01:36.500391Z","iopub.status.idle":"2024-11-12T15:01:36.508629Z","shell.execute_reply":"2024-11-12T15:01:36.507158Z"},"papermill":{"duration":0.035346,"end_time":"2024-11-12T15:01:36.511669","exception":false,"start_time":"2024-11-12T15:01:36.476323","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"a2458309","cell_type":"code","source":"#lets see if we get after loading th same zarr array\n\nz5 = zarr.load(store = '/kaggle/working/data.zarr',path = '1st_data')\n\nassert np.array_equal(z[:], z5[:]) , \"they are not equal\"","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:36.556956Z","iopub.status.busy":"2024-11-12T15:01:36.556455Z","iopub.status.idle":"2024-11-12T15:01:38.491018Z","shell.execute_reply":"2024-11-12T15:01:38.489768Z"},"papermill":{"duration":1.961477,"end_time":"2024-11-12T15:01:38.494258","exception":false,"start_time":"2024-11-12T15:01:36.532781","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"c24a59fe","cell_type":"code","source":"'''\n\nPersistence arrays \n\n Zarr arrays can  be stored on a main memory and also on file system, cloud  \n\n enabling persistence of data between sessions. \n\n'''\n\nz6 = zarr.open(store ='/kaggle/working/data.zarr',mode='r',path='1st_data',chunks=(10,10))\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:38.537722Z","iopub.status.busy":"2024-11-12T15:01:38.537242Z","iopub.status.idle":"2024-11-12T15:01:38.544458Z","shell.execute_reply":"2024-11-12T15:01:38.543241Z"},"papermill":{"duration":0.032403,"end_time":"2024-11-12T15:01:38.547106","exception":false,"start_time":"2024-11-12T15:01:38.514703","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"094247a7","cell_type":"code","source":"z_  = z #for storing purpose \n\na = np.random.rand(10000, 10000)\n\nz.append(a,axis=0)\n\nz.shape","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:38.591577Z","iopub.status.busy":"2024-11-12T15:01:38.591093Z","iopub.status.idle":"2024-11-12T15:01:40.685049Z","shell.execute_reply":"2024-11-12T15:01:40.683912Z"},"papermill":{"duration":2.119318,"end_time":"2024-11-12T15:01:40.688042","exception":false,"start_time":"2024-11-12T15:01:38.568724","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"65dc733a","cell_type":"code","source":"np.vstack([a,a]).shape","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:40.732710Z","iopub.status.busy":"2024-11-12T15:01:40.732256Z","iopub.status.idle":"2024-11-12T15:01:41.527251Z","shell.execute_reply":"2024-11-12T15:01:41.525934Z"},"papermill":{"duration":0.820464,"end_time":"2024-11-12T15:01:41.530312","exception":false,"start_time":"2024-11-12T15:01:40.709848","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"db15af53","cell_type":"code","source":"z.append(np.vstack([a,a]),axis=1)","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:41.578215Z","iopub.status.busy":"2024-11-12T15:01:41.577735Z","iopub.status.idle":"2024-11-12T15:01:44.311935Z","shell.execute_reply":"2024-11-12T15:01:44.310770Z"},"papermill":{"duration":2.761806,"end_time":"2024-11-12T15:01:44.314836","exception":false,"start_time":"2024-11-12T15:01:41.553030","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"112ad9c9","cell_type":"code","source":"#well i want to keep only 10000 x 10000 things so lets resize\n\n\n\nz.resize(10000,10000)\n\nz.shape","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:44.359847Z","iopub.status.busy":"2024-11-12T15:01:44.359289Z","iopub.status.idle":"2024-11-12T15:01:44.508244Z","shell.execute_reply":"2024-11-12T15:01:44.507148Z"},"papermill":{"duration":0.173969,"end_time":"2024-11-12T15:01:44.510871","exception":false,"start_time":"2024-11-12T15:01:44.336902","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"2ae953a5","cell_type":"markdown","source":"# COMPRESSOR","metadata":{"papermill":{"duration":0.02155,"end_time":"2024-11-12T15:01:44.554198","exception":false,"start_time":"2024-11-12T15:01:44.532648","status":"completed"},"tags":[]}},{"id":"96f3e090","cell_type":"code","source":"#Compressor \n\nfrom numcodecs import Zstd\n\n\n\ncompressor = Zstd(level=4,checksum=True)\n\ndata2 = np.random.normal(loc=0,scale=1,size =(1000,1000))\n\nzarr_array = zarr.array(data2,compressor=compressor,chunks=(100,100))\n\nzarr_array[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:44.598450Z","iopub.status.busy":"2024-11-12T15:01:44.597996Z","iopub.status.idle":"2024-11-12T15:01:44.695365Z","shell.execute_reply":"2024-11-12T15:01:44.693996Z"},"papermill":{"duration":0.123056,"end_time":"2024-11-12T15:01:44.698643","exception":false,"start_time":"2024-11-12T15:01:44.575587","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"359bbb5f","cell_type":"code","source":"#lets check level and compression efficiency\n\nfrom tqdm import tqdm \n\neff = []\n\nfor i in tqdm(range(1,22)):\n\n    compressor = Zstd(level=i,checksum=True)\n\n    array = zarr.array(data2,compressor=compressor,chunks=(100,100))\n\n    efficency = ((array.nbytes-array.nbytes_stored)/array.nbytes)*100\n\n    eff.append(efficency)\n\n    ","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:44.745889Z","iopub.status.busy":"2024-11-12T15:01:44.745439Z","iopub.status.idle":"2024-11-12T15:01:53.387732Z","shell.execute_reply":"2024-11-12T15:01:53.386123Z"},"papermill":{"duration":8.670284,"end_time":"2024-11-12T15:01:53.391778","exception":false,"start_time":"2024-11-12T15:01:44.721494","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"3cf8f6dc","cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.plot(range(len(eff)),eff,marker='o')\n\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:53.438933Z","iopub.status.busy":"2024-11-12T15:01:53.438465Z","iopub.status.idle":"2024-11-12T15:01:53.768019Z","shell.execute_reply":"2024-11-12T15:01:53.766671Z"},"papermill":{"duration":0.356597,"end_time":"2024-11-12T15:01:53.771026","exception":false,"start_time":"2024-11-12T15:01:53.414429","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"47ca0c3e","cell_type":"code","source":"#4.14 is the highest which is less than the default compressor of zarr array (Blosc)\n\n# intresting as levels increase the comrpession efficency goes down ","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:53.819972Z","iopub.status.busy":"2024-11-12T15:01:53.819530Z","iopub.status.idle":"2024-11-12T15:01:53.824531Z","shell.execute_reply":"2024-11-12T15:01:53.823272Z"},"papermill":{"duration":0.032295,"end_time":"2024-11-12T15:01:53.827194","exception":false,"start_time":"2024-11-12T15:01:53.794899","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"9199ed7d","cell_type":"code","source":"array.info #some info can be displayed ","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:53.876156Z","iopub.status.busy":"2024-11-12T15:01:53.875148Z","iopub.status.idle":"2024-11-12T15:01:53.884625Z","shell.execute_reply":"2024-11-12T15:01:53.883454Z"},"papermill":{"duration":0.036948,"end_time":"2024-11-12T15:01:53.887399","exception":false,"start_time":"2024-11-12T15:01:53.850451","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"4629befa","cell_type":"markdown","source":"# FILTERS","metadata":{"papermill":{"duration":0.022892,"end_time":"2024-11-12T15:01:53.933787","exception":false,"start_time":"2024-11-12T15:01:53.910895","status":"completed"},"tags":[]}},{"id":"4b1e088e","cell_type":"code","source":"'''\n\n filters are data transformation mechanisms applied to array chunks before compression. \n\n They preprocess data to enhance compression\n\n efficiency or adapt data for specific storage requirements.\n\n'''\n\n\n\nfrom numcodecs import Delta,Quantize\n\nfilter1 = Delta(dtype='f8',astype='f8')\n\nfilter2 = Quantize(digits=1,dtype='f8',astype='f8')\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:53.981981Z","iopub.status.busy":"2024-11-12T15:01:53.981112Z","iopub.status.idle":"2024-11-12T15:01:53.987755Z","shell.execute_reply":"2024-11-12T15:01:53.986412Z"},"papermill":{"duration":0.034156,"end_time":"2024-11-12T15:01:53.990899","exception":false,"start_time":"2024-11-12T15:01:53.956743","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"a7fd7396","cell_type":"code","source":"'''\n\nlets take Zstd compression only and apply filters in different permutations\n\n\n\noption1 - filter1\n\noption2 - filter2\n\noption3 - filter1,filter2\n\noption4 - filter2,filter1\n\n\n\n'''\n\neff2 = []\n\noptions ={\n\n\n\n'option1' :  [filter1],\n\n'option2' :  [filter2],\n\n'option3' :  [filter1,filter2],\n\n'option4'  : [filter2,filter1]\n\n    \n\n}\n\nfor option in options.values():\n\n eff1=[]\n\n for i in tqdm(range(1,22)):\n\n    compressor = Zstd(level=i,checksum=True)\n\n    array = zarr.array(data2,compressor=compressor,chunks=(100,100),filters=option)\n\n    efficency = ((array.nbytes-array.nbytes_stored)/array.nbytes)*100\n\n    eff1.append(efficency)\n\n eff2.append(eff1)","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:01:54.039794Z","iopub.status.busy":"2024-11-12T15:01:54.038767Z","iopub.status.idle":"2024-11-12T15:04:16.306616Z","shell.execute_reply":"2024-11-12T15:04:16.305041Z"},"papermill":{"duration":142.295757,"end_time":"2024-11-12T15:04:16.309798","exception":false,"start_time":"2024-11-12T15:01:54.014041","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"682a38db","cell_type":"code","source":"\n\nplt.plot(range(len(eff)),eff,marker='o',label='baseline')\n\nplt.plot(range(len(eff2[0])),eff2[0],marker='o',label='option1')\n\nplt.plot(range(len(eff2[0])),eff2[1],marker='*',label='option2')\n\nplt.plot(range(len(eff2[0])),eff2[2],marker='^',label='option3')\n\nplt.plot(range(len(eff2[0])),eff2[3],marker='p',label='option4')\n\n\n\nplt.legend()\n\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:16.372002Z","iopub.status.busy":"2024-11-12T15:04:16.371442Z","iopub.status.idle":"2024-11-12T15:04:16.710681Z","shell.execute_reply":"2024-11-12T15:04:16.709268Z"},"papermill":{"duration":0.373584,"end_time":"2024-11-12T15:04:16.713503","exception":false,"start_time":"2024-11-12T15:04:16.339919","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"a6d054be","cell_type":"code","source":"#Option 2 surpasses them all -> quantize\n\nfilter2 = Quantize(digits=1,dtype='f8',astype='f8')\n\ncompressor = Zstd(level=4,checksum=True)\n\nzarr_array = zarr.array(data2,compressor = compressor, filters = [filter2])\n\nnp.all(zarr_array==data2)\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:16.778810Z","iopub.status.busy":"2024-11-12T15:04:16.777932Z","iopub.status.idle":"2024-11-12T15:04:16.871148Z","shell.execute_reply":"2024-11-12T15:04:16.869798Z"},"papermill":{"duration":0.128584,"end_time":"2024-11-12T15:04:16.874044","exception":false,"start_time":"2024-11-12T15:04:16.745460","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"4dc16571","cell_type":"code","source":"zarr_array.info","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:16.941739Z","iopub.status.busy":"2024-11-12T15:04:16.941164Z","iopub.status.idle":"2024-11-12T15:04:16.949961Z","shell.execute_reply":"2024-11-12T15:04:16.948673Z"},"papermill":{"duration":0.04663,"end_time":"2024-11-12T15:04:16.952470","exception":false,"start_time":"2024-11-12T15:04:16.905840","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"d817d70a","cell_type":"code","source":"#the graph is self explantory that option 2 is the best \n\n#one of the potential reasons is that digits = 1 means i am setting it to low precision\n\n#hence high compression ","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.020128Z","iopub.status.busy":"2024-11-12T15:04:17.019657Z","iopub.status.idle":"2024-11-12T15:04:17.024861Z","shell.execute_reply":"2024-11-12T15:04:17.023485Z"},"papermill":{"duration":0.042141,"end_time":"2024-11-12T15:04:17.027735","exception":false,"start_time":"2024-11-12T15:04:16.985594","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"ccd5f137","cell_type":"markdown","source":"# GROUPS","metadata":{"papermill":{"duration":0.031172,"end_time":"2024-11-12T15:04:17.091151","exception":false,"start_time":"2024-11-12T15:04:17.059979","status":"completed"},"tags":[]}},{"id":"338b95cf","cell_type":"code","source":"#Zarr Groups \n\n\n\n\n\n'''\n\nso what's the idea of zarr groups \n\n\n\nHierarchical Organization: Groups can contain multiple arrays and sub-groups, \n\nenabling the creation of nested structures for complex data organization.\n\n\n\n'''\n\n#lets create one!\n\n\n\nroot = zarr.group()\n\n\n\nfoo = root.create_group('foo')\n\nbar = foo.create_group('bar')\n\nbar_array = bar.full('bar_array',fill_value=10,shape=(100,100),dtype='i2',chunks=(10,10))\n\nboo_array = foo.full('boo_array',fill_value=10,shape=(100,100),dtype='i2',chunks=(10,10))\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.155865Z","iopub.status.busy":"2024-11-12T15:04:17.155342Z","iopub.status.idle":"2024-11-12T15:04:17.166494Z","shell.execute_reply":"2024-11-12T15:04:17.165131Z"},"papermill":{"duration":0.046904,"end_time":"2024-11-12T15:04:17.169565","exception":false,"start_time":"2024-11-12T15:04:17.122661","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"a12f046b","cell_type":"code","source":"#members can be acessed also like\n\nroot['foo/bar']","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.236027Z","iopub.status.busy":"2024-11-12T15:04:17.235548Z","iopub.status.idle":"2024-11-12T15:04:17.243938Z","shell.execute_reply":"2024-11-12T15:04:17.242549Z"},"papermill":{"duration":0.044513,"end_time":"2024-11-12T15:04:17.246372","exception":false,"start_time":"2024-11-12T15:04:17.201859","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"bc71d82f","cell_type":"code","source":"z = root.create('foo/bar/rrr',shape=(1000,1000),chunks=(10,10),dtype='i4')\n\nz","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.312273Z","iopub.status.busy":"2024-11-12T15:04:17.311810Z","iopub.status.idle":"2024-11-12T15:04:17.320740Z","shell.execute_reply":"2024-11-12T15:04:17.319509Z"},"papermill":{"duration":0.045388,"end_time":"2024-11-12T15:04:17.323383","exception":false,"start_time":"2024-11-12T15:04:17.277995","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"2432f1dd","cell_type":"code","source":"try:\n\n    print(root.tree())\n\nexcept AttributeError:\n\n    pass \n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.390416Z","iopub.status.busy":"2024-11-12T15:04:17.389954Z","iopub.status.idle":"2024-11-12T15:04:17.398001Z","shell.execute_reply":"2024-11-12T15:04:17.396758Z"},"papermill":{"duration":0.04588,"end_time":"2024-11-12T15:04:17.401643","exception":false,"start_time":"2024-11-12T15:04:17.355763","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"d8f412e4","cell_type":"code","source":"#lets gather some info about our groups, sub groups and arrays \n\nprint(root.info)\n\nprint(foo.info)\n\nprint(bar.info)\n\n# we can easily verify the information with the tree structure ","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.466844Z","iopub.status.busy":"2024-11-12T15:04:17.466404Z","iopub.status.idle":"2024-11-12T15:04:17.474079Z","shell.execute_reply":"2024-11-12T15:04:17.472761Z"},"papermill":{"duration":0.043801,"end_time":"2024-11-12T15:04:17.477401","exception":false,"start_time":"2024-11-12T15:04:17.433600","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"0b343e74","cell_type":"markdown","source":"# USER DEFINED ATTRIBUTES FOR METADATA","metadata":{"papermill":{"duration":0.031212,"end_time":"2024-11-12T15:04:17.540551","exception":false,"start_time":"2024-11-12T15:04:17.509339","status":"completed"},"tags":[]}},{"id":"1869cd81","cell_type":"code","source":"#for storing application specific metadata we can define attributes \n\nroot.attrs['parent'] ='father  of foo'\n\nfoo.attrs['parent'] ='father of boo_array and bar'\n\nfoo.attrs['child'] ='child of root'\n\nbar.attrs['parent'] ='father of bar_array and rrr'\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.607536Z","iopub.status.busy":"2024-11-12T15:04:17.607079Z","iopub.status.idle":"2024-11-12T15:04:17.614111Z","shell.execute_reply":"2024-11-12T15:04:17.612686Z"},"papermill":{"duration":0.04448,"end_time":"2024-11-12T15:04:17.616686","exception":false,"start_time":"2024-11-12T15:04:17.572206","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"ba4a5d6e","cell_type":"code","source":"for attrs in foo.attrs:\n\n print(foo.attrs[attrs]) #printing foo metadata ","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.683623Z","iopub.status.busy":"2024-11-12T15:04:17.683136Z","iopub.status.idle":"2024-11-12T15:04:17.689900Z","shell.execute_reply":"2024-11-12T15:04:17.688574Z"},"papermill":{"duration":0.045327,"end_time":"2024-11-12T15:04:17.693598","exception":false,"start_time":"2024-11-12T15:04:17.648271","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"5a97d74e","cell_type":"markdown","source":"# ADVANCED INDEXING ","metadata":{"papermill":{"duration":0.032498,"end_time":"2024-11-12T15:04:17.758453","exception":false,"start_time":"2024-11-12T15:04:17.725955","status":"completed"},"tags":[]}},{"id":"e42eac73","cell_type":"markdown","source":"**COORDINATE INDEXING**","metadata":{"papermill":{"duration":0.031558,"end_time":"2024-11-12T15:04:17.821901","exception":false,"start_time":"2024-11-12T15:04:17.790343","status":"completed"},"tags":[]}},{"id":"676de0ca","cell_type":"code","source":"#Advanced indexing using zarr\n\n#using v.index\n\n\n\n\n\nrows = np.array([0, 2, 4])\n\ncols = np.array([1, 3, 5])\n\n\n\n# Access elements at specified indices\n\nselected_elements = zarr_array.vindex[rows, cols]\n\nselected_elements","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.892685Z","iopub.status.busy":"2024-11-12T15:04:17.892253Z","iopub.status.idle":"2024-11-12T15:04:17.903674Z","shell.execute_reply":"2024-11-12T15:04:17.902070Z"},"papermill":{"duration":0.05042,"end_time":"2024-11-12T15:04:17.906447","exception":false,"start_time":"2024-11-12T15:04:17.856027","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"c632a4b8","cell_type":"code","source":"z =  zarr.array(np.arange(5)**3)\n\nelements = z.get_coordinate_selection([1,4])\n\nprint(z[:],'\\n',elements)","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:17.974207Z","iopub.status.busy":"2024-11-12T15:04:17.973123Z","iopub.status.idle":"2024-11-12T15:04:17.981907Z","shell.execute_reply":"2024-11-12T15:04:17.980528Z"},"papermill":{"duration":0.045598,"end_time":"2024-11-12T15:04:17.985348","exception":false,"start_time":"2024-11-12T15:04:17.939750","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"dd0e0d89","cell_type":"code","source":"z.set_coordinate_selection([1, 4], [23,24]) #modified the values at 1,4 index \n\nz[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.052857Z","iopub.status.busy":"2024-11-12T15:04:18.052335Z","iopub.status.idle":"2024-11-12T15:04:18.062159Z","shell.execute_reply":"2024-11-12T15:04:18.060650Z"},"papermill":{"duration":0.047277,"end_time":"2024-11-12T15:04:18.065055","exception":false,"start_time":"2024-11-12T15:04:18.017778","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"ef3a0d50","cell_type":"code","source":"z = zarr.array(np.arange(12).reshape(4,3))\n\nz[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.131152Z","iopub.status.busy":"2024-11-12T15:04:18.130737Z","iopub.status.idle":"2024-11-12T15:04:18.140326Z","shell.execute_reply":"2024-11-12T15:04:18.138977Z"},"papermill":{"duration":0.045681,"end_time":"2024-11-12T15:04:18.142895","exception":false,"start_time":"2024-11-12T15:04:18.097214","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"2457a152","cell_type":"code","source":"z[[0,2],[0,2]] #it means i want to acess the (0,0) and (2,2) element in matrix","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.211580Z","iopub.status.busy":"2024-11-12T15:04:18.211117Z","iopub.status.idle":"2024-11-12T15:04:18.220275Z","shell.execute_reply":"2024-11-12T15:04:18.219122Z"},"papermill":{"duration":0.046721,"end_time":"2024-11-12T15:04:18.223150","exception":false,"start_time":"2024-11-12T15:04:18.176429","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"a34640ae","cell_type":"code","source":"#When the indexing arrays have different shapes, they are broadcast together.\n\n#which means \n\nz[0,[0,2]] # it works the same as z[[0,0],[0,2]]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.291140Z","iopub.status.busy":"2024-11-12T15:04:18.290719Z","iopub.status.idle":"2024-11-12T15:04:18.299424Z","shell.execute_reply":"2024-11-12T15:04:18.298241Z"},"papermill":{"duration":0.045516,"end_time":"2024-11-12T15:04:18.302278","exception":false,"start_time":"2024-11-12T15:04:18.256762","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"3974eb0a","cell_type":"markdown","source":"**MASK INDEXING**","metadata":{"papermill":{"duration":0.033651,"end_time":"2024-11-12T15:04:18.370089","exception":false,"start_time":"2024-11-12T15:04:18.336438","status":"completed"},"tags":[]}},{"id":"b2b792fa","cell_type":"code","source":"z = zarr.array(np.arange(10) ** 2)\n\nprint(z[:])\n\nsel = np.zeros_like(z, dtype=bool)\n\nsel[1]=True \n\nsel[3]=True\n\n\n\n#nowm lets display only masked ones\n\nz.get_mask_selection(sel)","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.438269Z","iopub.status.busy":"2024-11-12T15:04:18.437748Z","iopub.status.idle":"2024-11-12T15:04:18.450244Z","shell.execute_reply":"2024-11-12T15:04:18.449043Z"},"papermill":{"duration":0.049948,"end_time":"2024-11-12T15:04:18.452802","exception":false,"start_time":"2024-11-12T15:04:18.402854","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"7410b3dc","cell_type":"code","source":"#at masked values we can also specify other values \n\nz.set_mask_selection(sel,[34,67])\n\nz[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.521298Z","iopub.status.busy":"2024-11-12T15:04:18.520835Z","iopub.status.idle":"2024-11-12T15:04:18.530387Z","shell.execute_reply":"2024-11-12T15:04:18.529199Z"},"papermill":{"duration":0.047111,"end_time":"2024-11-12T15:04:18.533071","exception":false,"start_time":"2024-11-12T15:04:18.485960","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"771ae291","cell_type":"markdown","source":"both mask and coordinate indexing  allow selecting arbitrary items from an array, also known as point selection.\n\n\n\nAlos Zarr arrays also support methods for orthogonal indexing, which allows selections to be made along each dimension of an array independently","metadata":{"papermill":{"duration":0.033519,"end_time":"2024-11-12T15:04:18.600732","exception":false,"start_time":"2024-11-12T15:04:18.567213","status":"completed"},"tags":[]}},{"id":"2c720857","cell_type":"markdown","source":"**ORTHOGONAL INDEXING**","metadata":{"papermill":{"duration":0.032834,"end_time":"2024-11-12T15:04:18.667585","exception":false,"start_time":"2024-11-12T15:04:18.634751","status":"completed"},"tags":[]}},{"id":"d14857b8","cell_type":"code","source":"#Orthogonal indexing \n\nz = zarr.array(np.arange(12).reshape(4,3))\n\nz.get_orthogonal_selection((slice(None),[0,2])) #gets the 1st and 3rd column\n\nz.get_orthogonal_selection(([0,2],slice(None))) #gets the 1st and 3rd row\n","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.736328Z","iopub.status.busy":"2024-11-12T15:04:18.735833Z","iopub.status.idle":"2024-11-12T15:04:18.747791Z","shell.execute_reply":"2024-11-12T15:04:18.746411Z"},"papermill":{"duration":0.050609,"end_time":"2024-11-12T15:04:18.751384","exception":false,"start_time":"2024-11-12T15:04:18.700775","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"4f752b28","cell_type":"code","source":"sel = np.zeros_like(z,dtype=bool)\n\nsel[1,1] = True\n\nsel[2,2] = True\n\nz.vindex[sel] = [100,100]\n\nz[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.820311Z","iopub.status.busy":"2024-11-12T15:04:18.819850Z","iopub.status.idle":"2024-11-12T15:04:18.830636Z","shell.execute_reply":"2024-11-12T15:04:18.829349Z"},"papermill":{"duration":0.048298,"end_time":"2024-11-12T15:04:18.833243","exception":false,"start_time":"2024-11-12T15:04:18.784945","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"c7324b83","cell_type":"code","source":"#orthogonal indexing functionality is also available via the oindex property\n\nz.oindex[:,[2]] =[[1],[2],[3],[4]]\n\nz[:]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:18.904755Z","iopub.status.busy":"2024-11-12T15:04:18.904326Z","iopub.status.idle":"2024-11-12T15:04:18.914102Z","shell.execute_reply":"2024-11-12T15:04:18.912863Z"},"papermill":{"duration":0.048824,"end_time":"2024-11-12T15:04:18.917048","exception":false,"start_time":"2024-11-12T15:04:18.868224","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"11cf66b2","cell_type":"markdown","source":"**Block indexing - Get the chunk out directly**","metadata":{"papermill":{"duration":0.033642,"end_time":"2024-11-12T15:04:18.984451","exception":false,"start_time":"2024-11-12T15:04:18.950809","status":"completed"},"tags":[]}},{"id":"9efb2323","cell_type":"code","source":"z= zarr.array(z,chunks=(1,3))\n\nz.get_block_selection(0)","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:19.055437Z","iopub.status.busy":"2024-11-12T15:04:19.054997Z","iopub.status.idle":"2024-11-12T15:04:19.067995Z","shell.execute_reply":"2024-11-12T15:04:19.066880Z"},"papermill":{"duration":0.050923,"end_time":"2024-11-12T15:04:19.070432","exception":false,"start_time":"2024-11-12T15:04:19.019509","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"fc853ade","cell_type":"code","source":"z.blocks[3,0]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:19.139777Z","iopub.status.busy":"2024-11-12T15:04:19.139313Z","iopub.status.idle":"2024-11-12T15:04:19.147549Z","shell.execute_reply":"2024-11-12T15:04:19.146412Z"},"papermill":{"duration":0.046186,"end_time":"2024-11-12T15:04:19.150207","exception":false,"start_time":"2024-11-12T15:04:19.104021","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"a6635447","cell_type":"code","source":"z[...]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:19.221347Z","iopub.status.busy":"2024-11-12T15:04:19.220874Z","iopub.status.idle":"2024-11-12T15:04:19.229808Z","shell.execute_reply":"2024-11-12T15:04:19.228512Z"},"papermill":{"duration":0.047681,"end_time":"2024-11-12T15:04:19.232558","exception":false,"start_time":"2024-11-12T15:04:19.184877","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"38c5eea5","cell_type":"code","source":"z.set_block_selection((3,0),1) #the chunk at index 3,0 will be replaced or filled with value 1","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:19.364962Z","iopub.status.busy":"2024-11-12T15:04:19.364540Z","iopub.status.idle":"2024-11-12T15:04:19.370996Z","shell.execute_reply":"2024-11-12T15:04:19.369430Z"},"papermill":{"duration":0.045084,"end_time":"2024-11-12T15:04:19.374082","exception":false,"start_time":"2024-11-12T15:04:19.328998","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"d4363954","cell_type":"code","source":"z[...]","metadata":{"execution":{"iopub.execute_input":"2024-11-12T15:04:19.444346Z","iopub.status.busy":"2024-11-12T15:04:19.443895Z","iopub.status.idle":"2024-11-12T15:04:19.452239Z","shell.execute_reply":"2024-11-12T15:04:19.451025Z"},"papermill":{"duration":0.046074,"end_time":"2024-11-12T15:04:19.454745","exception":false,"start_time":"2024-11-12T15:04:19.408671","status":"completed"},"tags":[]},"outputs":[],"execution_count":null}]}