Generate Rust code representing GRIB2 tables from GDAL CSVs

[JackKelly]

In this comment @mpiannucci said:

I... was going to codegen [a Rust implementation of the GRIB code tables] but never had the time

We could use this issue to capture some ideas & design requirements for the codegen. I'll get started below but please shout if I'm off track 🙂.

Input to the codegen

GDAL's CSV representation of the GRIB tables. The README for that directory is here: gdal/frmts/grib/degrib/README.TXT

Output from the codegen

Rust code!

• Rust Enums which represent the GRIB code tables.
• Make it super-easy and super-fast to look up the appropriate enum variant by generating the definitions for at least 2 phf::Maps (perfect hash functions generated at compile time):
  1. Map from abbreviation String to enum variant. This will be useful for parsing .idx files (Add idx metadata file handling #41).
  2. Map from numeric GRIB code to enum variant.

Questions

UPDATE: I've removed the questions for now because I'm working on a code sketch which I hope answers the main questions... I'll post a comment with the code sketch soon...

Development timeline

Hopefully I'll be able to help write the PR which implements this in a few months... But, of course, I'm more than happy for other people to do it!

Related

• "Introduction to Code Generation in Rust" by Chip Reed at CrabNebula, November 2023. (Discusses very useful crates for us: quote, walkdir, and phf)

[JackKelly]

UPDATE: I've removed the questions from the post above because I'm working on a code sketch which I hope answers the main questions... I'll post a comment with the code sketch soon...

[JackKelly]

Huh. I've just stumbled across the grib-rs project (another GRIB reader implemented in Rust 🙂). grib-rs includes some codegen. Some notable parts of grib-rs:

• gen/src/grib2_codeflag_csv.rs: Parse WMO CSV files into a vector of Wgrib2TableEntry structs.
• codegen/src/param_codes.rs: Generate enum variants from a vector of Wgrib2TableEntry structs. I think the generated code is never materialised to disk. I think it's just generated on-the-fly during compilation.
• codegen/tests/01-normal-case.rs: An example of a generated enum.

(note that, in the Wgrib2TableEntry struct, what gribberish would call the abbrev is called the name).

AFAICT, the representation of the GRIB tables in grib-rs is "flat": grib-rs doesn't attempt to represent the hierarchy "Discipline > Category > Product > Parameter".

I don't think grib-rs currently has any ability to read .idx files. So you'd have to use a HashMap<String, Wgrib2TableEntry> to find the parameter associated with each "abbreviation" string in an .idx file. Which might be fine, from a performance perspective.

I must admit I'm leaning towards the idea that it would be useful to have a common "grib_tables" crate which "just" represents the GRIB tables as Rust code (and the auto-generated code would be materialised to disk). UPDATE see my comment below.

[mpiannucci]

Yeah i know about grib-rs. Its similar but there is no python support, and because of the way it defines code tables its limited to WMO tables which was not sufficient for me

[JackKelly]

I'll post a comment with the code sketch soon

I've been back-and-forth on this design.

TL;DR: I think I'm gonna implement a simple stand-alone crate that loads the WMO and GDAL CSVs at runtime and allows folks to look up parameters by their numeric ID or by their abbreviation. More details here: https://github.com/JackKelly/hypergrib/tree/main/crates/grib_tables

I had previously started work on some code that tries to faithfully capture the GRIB tables hierarchy (Discipline > Category > Parameter). But, for hypergrib, this feels overkill. It might be slightly faster because we can create a perfect hash at compile time. But then I realised that we only need to look things up in the GRIB table whilst indexing a new GRIB dataset. When "normal users" use the dataset via hypergrib, they can just read the metadata that we create for the dataset.

[emfdavid]

Heads up eccodes just changed the variable names for NOAA grib2 files.
dswrf -> sdswrf
ulwrf -> sulwrf
etc...
Still trying to work out if this is intentional or a bug.

[williamhobbs]

@emfdavid, did you figure out if those variable name changes in eccodes were intentional?