Skip to content

Arbitrary chunking of uncompressed files (e.g. netCDF3) #86

@TomNicholas

Description

@TomNicholas

@rabernat made the interesting point to me that uncompressed files (e.g. netCDF3 files) have no specific chunking, as you can start reading bytes from any point in the file immediately, with no minimum unit of decompression to do first.

I'm not totally sure what this implies for VirtualiZarr generating references from netCDF3 files, as it's still meaningful to talk about one chunk per file concatenated together in a manifest.

Perhaps this is something that should ultimately be taken account of in zarr readers: that reading bytes from an uncompressed array does not require loading an entire chunk into memory first.

Metadata

Metadata

Assignees

No one assigned

    Labels

    KerchunkRelating to the kerchunk library / specification itselfenhancementNew feature or requestzarr-pythonRelevant to zarr-python upstream

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions