regression testing requires BIT_STREAM_WORD_TYPE=uint64

[pythonmobile]

Any particular reason why regression testing does not support BIT_STREAM_WORD_TYPE=uint8?

[lindstro]

@pythonmobile You raise a valid question. The simple answer is that each new test (as currently written) requires manually recording the result of a checksum, and zfp has hundreds of these tests that depend on and have been designed for the default word size of 64 bits. For portability across big and little and endian platforms, the H5Z-ZFP filter requires an 8-bit word size, and we really ought to include checksums for 8-bit words. We're moving toward a more automated solution that will facilitate this, but are not quite there yet. If you have a use case other than HDF5 I/O that requires 8-bit words, we'd appreciate sharing it with us.

[leventt]

just as a note... i came across this while searching for endianness for an emscripten build...

BIT_STREAM_WORD_TYPE=uint64
is necessary for emscripten build as well, i feel like.

Has there been any changes on the HEAD, regarding this and would you know a best revision to build little endian compatible zfp?

[leventt]

latest release 0.5.5 works great so far for the build \o/

here is my script to build if it helps someone:

cd zfp
git checkout tags/0.5.5
git pull
mkdir build
cd build
emcmake cmake .. -DBIT_STREAM_WORD_TYPE=uint64
cmake --build . --config Release

[lindstro]

BIT_STREAM_WORD_TYPE=uint64 is required only by the tests, as we have not yet added checksums for other word sizes. I'm not familiar with emscripten and so cannot comment on why using other word sizes would be problematic.

I will say, however, that zfp streams generated using different word sizes are not generally compatible. Although streams generated on little endian machines with a word size larger than 8 bits will match 8-bit streams in terms of the sequence of bits, such streams have different alignment requirements. This will cause issues when writing multiple fields to the same stream or when decompressing an 8-bit stream using a larger word size, as that may cause reads beyond the end of the stream.

I'm not sure if this answers your question about "little endian compatible zfp."

[leventt]

I don't think I need little endian compatibility tbh. It was just a build error :)
Emscripten, I guess is compensating for old machines online but I think I should be fine, since I will only need it on client side machines and my host machine is using the python wrappers for zfp. Let me find out and I can make a note here if you like if it works or not.

Thanks for the insight and the amazing code \o/

[picca]

Hello, I am facing this issue when packaging zfp and the h5z-zfp plugin.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1060807

Until a certain amount of time the zfp library provided by Debian was compiled with uint8.

In order to run the test suite, we switched from uint8 to uint64.

The hdf5 plugin used the version of zfp(uint8) in order to generate compressed dataset.

so, if we compile now zfp with uint64, could we open the older generated files ?

I think that this is not possible (because the h5z unit test is failling with uint64).

The solution in order to run the suite of zfp is to compile zfp with unit64 run the test and then re-compile it with uint8, but If zfp (uint8 and unit64) are not compatible, what is from your point of view the best way to deal with these incompatibilities especially in the case of long conservation of hdf5 data files.

thanks for considering

[lindstro]

@picca It is true that H5Z-ZFP requires uint8 words. This is a hard requirement, not just one related to potentially failing tests.

Meanwhile, zfp's tests have been written only for uint64 and will fail with other word sizes. This basically comes down to us not having had time to add checksums and stream lengths for other word sizes. Admittedly, this is a significant weakness of our tests, especially for interoperability with H5Z-ZFP.

We really should bite the bullet and generate checksums for all supported word sizes. There are, however, similar challenges with other compile-time macros such as ZFP_ROUNDING_MODE, ZFP_WITH_TIGHT_ERROR, and ZFP_WITH_DAZ, which all alter the generated bit stream. With 4 word sizes, 3 rounding modes, and 2 settings each for TIGHT_ERROR and DAZ, there are 48 possible configurations, each involving 432 + 16 checksums, for a grand total of over 20,000 checksums that need recording. This does not include another 100 or so checksums per configuration that we're adding to the zfp CLI to allow for some basic testing of the installed binaries. While the checksum generation and capture is quasi automatic, it's still a lot of manual work to handle this properly, which is why it has not happened yet.

To further complicate matters, the CUDA and HIP backends currently support only uint64, so those would fail if zfp is built with uint8.

I believe we should commit to addressing this in the 1.1.0 release (still months away). We should also store in libzfp sufficient information to determine the build configuration. Until then, however, you're essentially forced to build with uint64 for testing and then to switch to uint8 for H5Z-ZFP compatibility.

so, if we compile now zfp with uint64, could we open the older generated files ?

Not reliably. A big-endian machine will not process 64-bit words properly (though I believe the H5Z-ZFP filter handles this correctly). And although little-endian machine will, a uint8 generated stream could end at any byte offset, so a 64-bit read could access memory beyond the end of the stream. Also, stream concatenation will not work properly as each compressed stream is word-aligned. I don't know whether that's an issue with H5Z-ZFP.

what is from your point of view the best way to deal with these incompatibilities

There is no great solution here. You need uint8 for H5Z-ZFP and for complete portability. You want uint64 when performance matters and for applications (such as in-memory computations) that do not persist zfp files. Likely the best short term solution is to manually test with uint64 first, then build and package uint8. That is, until we fix this, hopefully in 1.1.0. But even then, uint8 will be your best bet to cater to all applications.

We expect to have a more robust strategy in place for zfp 2, where alignment issues go away. I believe H5Z-ZFP already handles endianness properly; applications that write raw zfp streams (outside of HDF5) would have to be cognizant of endianness issues, as would any application that writes raw floats and ints.

Perhaps not the answer you were hoping for, but this is the best we can do at the moment. I'm also roping in @markcmiller86 who might answer any issues with H5Z-ZFP. Mark, I assume all errors in the H5Z-ZFP tests stem from ZFP lib not compiled with -DBIT_STREAM_WORD_TYPE=uint8.

[markcmiller86]

With 4 word sizes, 3 rounding modes, and 2 settings each for TIGHT_ERROR and DAZ, there are 48 possible configurations, each involving 432 + 16 checksums, for a grand total of over 20,000 checksums that need recording.

Is total coverage actually necessary here or is just a sample along various dimensions of the configuration space sufficient?

A big-endian machine will not process 64-bit words properly (though I believe the H5Z-ZFP filter handles this correctly).

Right, H5Z-ZFP does have tests now that test reads/writes in a cross-endian context. They work! There is some performance issue with how HDF5 library needs to handle it though.

There is no great solution here. You need uint8 for H5Z-ZFP and for complete portability. You want uint64 when performance matters and for applications (such as in-memory computations) that do not persist zfp files.

For my own edification, how big of a performance issue are we talking about here? 2x, 10x, what? In the context of H5Z-ZFP, we're already doing I/O, so performance higher than disk I/O bandwidth is not too relevant. Of course, with SSDs, that situation is a bit muddy but still supportable I think.

Mark, I assume all errors in the H5Z-ZFP tests stem from ZFP lib not compiled with -DBIT_STREAM_WORD_TYPE=uint8.

Right, H5Z-ZFP has a run-time check for whether the ZFP library it is linked with has been configured with -DBIT_STREAM_WORD_TYPE=uint8 and it will fail the compression attempt. I can't recall whether we can adjust that to silently ignore the compression attempt and let H5Dwrite just write uncompressed. I know the caller can select that behavior by telling HDF5 library whether they want the filter treated as mandatory or optional.

Now, it just occurred to me, H5Z-ZFP offers both a generic interface and a properties interface and in the generic interface, the caller explicitly decides whether the filter should be H5Z_FLAG_MANDATORY or H5Z_FLAG_OPTIONAL with the flags arg in the H5Pset_filter() call which the caller must make itself. I dunno how this is affected in the properties case.

[markcmiller86]

Now, it just occurred to me, H5Z-ZFP offers both a generic interface and a properties interface and in the generic interface, the caller explicitly decides whether the filter should be H5Z_FLAG_MANDATORY or H5Z_FLAG_OPTIONAL with the flags arg in the H5Pset_filter() call which the caller must make itself. I dunno how this is affected in the properties case.

Ok, so I just looked and it looks like the properties interface assumes behavior should be mandatory...

https://github.com/LLNL/H5Z-ZFP/blob/092190c38d5760b3212ce4d96cced767b90aa189/src/H5Zzfp_props.c#L93C76-L94C1

We should improve that a bit in H5Z-ZFP so a caller using properties interface can nonetheless still allow for the filter to be optional.

[lindstro]

Is total coverage actually necessary here or is just a sample along various dimensions of the configuration space sufficient?

Well, for any given build configuration, you likely want to run all the tests (but not tests for other build configurations). I don't think there's much point in running tests for a configuration other than the chosen one, if that's what you're suggesting.

how big of a performance issue are we talking about here?

Enough that you don't want to give it up for in-memory compressed-array operations. It's true that this performance difference is of no consequence to I/O applications, but that's really a secondary use case. :)

We should improve that a bit in H5Z-ZFP so a caller using properties interface can nonetheless still allow for the filter to be optional.

Perhaps. Users would likely be very disappointed in zfp if it silently fails and writes the data without compression... Would zfp configuration be checked at load time? For example, suppose zfp is installed with the default uint64 word size. An unsuspecting user then decides to use H5Z-ZFP and gets very crummy compression without knowing why. I'm not sure I like that idea.

[markcmiller86]

Enough that you don't want to give it up for in-memory compressed-array operations

Ah, ok. Well, in that situation I can understand. You're potentially taking a hit for every array item reference. That could get onerously bad.

An unsuspecting user then decides to use H5Z-ZFP and gets very crummy compression without knowing why. I'm not sure I like that idea.

Well, just remember, in the generic interface, this is already true for any caller that uses H5Z_FLAGS_OPTIONAL for the flags parameter...I think...now I am puzzling...because we error out of the filter's can_apply() method...which I think is called during H5Dcreate().

So, in the optional case, the dataset will wind up being created but without the filter the caller requested. Subsequent H5Dwrites() will bypass the filter. This is already possible and is intended for cases where callers may be writing a wide variety of different kinds of datasets and not wanting to individually fiddle with dataset creation properties for each kind of dataset. The filters not relevant to a given dataset are just ignored and a consistent dataset creation properties is passed in for all of them.

Now, HDF5 supports a return type of h5tri_t for the set_local() and I am honestly not sure if we're returning a simple false answer for those calls or an error condition. If an error condition is returned, dataset creation will fail, regardless of whether the caller specified the filter was optional. Ok, so its obvious we need to understand this better.

[lindstro]

I'm not sufficiently familiar with HDF5 and how such flags might impact the average user, but I suspect many (most?) zfp binary installations--and there are lots of them in the form of prebuilt Linux, Anaconda, and Spack packages--will use the default 64-bit setting. If the user has to explicitly request filter bypass, then I think allowing that would be fine. But is there any chance that filter failure might occur at load time unbeknownst to the user?

[markcmiller86]

But is there any chance that filter failure might occur at load time unbeknownst to the user?

So, I am not sure I know what you mean by "load time"? Do you mean when the HDF5 library dlopen()'s the filter plugin? No, there is no chance of this check effecting that. These checks occur at H5Dcreate() time. So, before any actual data has been written.

As I mention in review of set_local() and can_apply(), we do currently error when ZFP is found to have been compiled incorrectly. This means the condition is in the user's face and not easily overlooked, which I think we both agree is appropriate and different from, for example, silently skipping ZFP compression for an array of char or short or maybe struct {int a; char b; float c}.