Skip to content

ODD PI 25.4 Objective 7: 🧊 Zarr/Icechunk for Level 2 data storage and search #207

@abarciauskas-bgse

Description

@abarciauskas-bgse

Motivation

Teams across the spectrum at NASA have asked for new more efficient approaches to store and manage Level 2 or “swath” data. Level 2 datasets from platforms such as MODIS and VIIRS are often stored in arrays with irregular or non-gridded footprints which makes them incompatible with many of the storage and analysis tools used in the community which only work with regularly gridded “Level 3” data.

Zarr provides the best modern option for array storage with Icechunk adding transactional versioning and virtualization support on top of the Zarr model.

Description

We plan to create a middleware library which allows us to commit array data and its tabular metadata (like STAC or UMM) concurrently as a single transaction to an Icechunk store. Users can then use a SQL like query API (similar to STAC and CMR filtering) to select data based on criteria and then have direct access to the underlying array data. This has several advantages, the first being that the underlying data are not stored separately from the metadata as we do in the current STAC and CMR approaches which can lead to data/metadata inconsistency. The biggest advantage is being able to efficiently and seamlessly query and access Level 2 / swath based array data which can then be used easily for aligned swaths in analysis packages like xarray. This is one of the biggest requests we’ve seen from the NASA data user community recently and will become even more important with the upcoming release of SWOT and NISAR data

Acceptance Criteria

  • Middleware library which allows us to commit array data and its tabular metadata (like STAC or UMM) concurrently as a single transaction to an Icechunk store

Sub-tasks

  • example 1
  • example 2

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions