mmap-backed persistent ZFP arrays

[jvo203]

It would be nice to be able to save to/restore compressed ZFP arrays from mmapped files. At present a crude workaround has been applied to zfparray.h:

added three extra variables to the array class:
std::string storage; //persistent storage via mmap
bool is_mmapped;
size_t storage_size;

and then alloc() / free() have been changed too:

// allocate memory for compressed data
void alloc(bool clear = true)
{
bytes = blocks * blkbits / CHAR_BIT;
printf("zfp::array3::alloc<clear:%d, storage:%s, bytes:%zu>\n", clear, storage.c_str(), bytes);
if (bytes > 1 && storage != "")
{
//try mmap first
int fd = open(storage.c_str(), O_RDWR | O_CREAT, (mode_t)0600);
if (fd != -1)
{
int stat = ftruncate64(fd, bytes);
if (!stat)
{
void *buffer = mmap(0, bytes, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

      if (buffer != MAP_FAILED)
      {
        data = (uchar *)buffer;
        is_mmapped = true;
        storage_size = bytes;
        printf("zfp::array3::alloc<mmap OK>.\n");
      }
      else
        perror("mmap");
    }
    else
    {
      perror("ftruncate64");
    }
    close(fd);
  }
}

if (!is_mmapped)
{
  reallocate(data, bytes, 0x100u);
  if (clear)
    std::fill(data, data + bytes, 0);
}
stream_close(zfp->stream);
zfp_stream_set_bit_stream(zfp, stream_open(data, bytes));
clear_cache();

}

// free memory associated with compressed data
void free()
{
nx = ny = nz = 0;
bx = by = bz = 0;
blocks = 0;
stream_close(zfp->stream);
zfp_stream_set_bit_stream(zfp, 0);
bytes = 0;
if (!is_mmapped)
deallocate(data);
else
{
if (data != NULL)
{
if (munmap(data, storage_size) == -1)
perror("un-mapping error");
else
printf("zfp::array3::free.\n");

    storage_size = 0;
    is_mmapped = false;
  };
};
data = 0;
deallocate(shape);
shape = 0;

}

This is a "quick&dirty" workaround. A more elegant ZFP-wide solution would be much nicer.

the array3 constructor in zfparray3.h has been modified slightly too:
array3(uint nx, uint ny, uint nz, double rate, const Scalar* p = 0, size_t csize = 0, std::string _storage = std::string("")) :
array(3, Codec::type),
cache(lines(csize, nx, ny, nz))
{
storage = _storage;
is_mmapped = false;
storage_size = 0;
set_rate(rate);
resize(nx, ny, nz, p == 0);
if (p)
set(p);
}

[lindstro]

@jvo203 Thanks for the suggestion and code. I think this capability is not mmap specific but would be generally useful when the caller has already allocated and initialized a buffer with compressed data. I think a constructor that takes such a pre-allocated buffer would be the right approach. I'd rather have the caller than zfp deal with memory mapping, which requires platform specific code.

This ought to be a simple fix that we can implement on a feature branch.

[jvo203]

One thing to bear in mind: the reason for putting mmap inside the array class is the need to know the exact size of the buffer that needs to be allocated via mmap. At present this is calculated inside array::alloc():
bytes = blocks * blkbits / CHAR_BIT;
once blocks and blkbits are known.

To work-around C++ allocating/deallocating zero-sized ZFP arrays during some C++-specific internal copy operations (copy constructors etc), mmap is only used if the requested size > 0 bytes.

Since even after ZFP compression there is no guarantee that a compressed ZFP stream will fit entirely in RAM, I have been forced to mmap-allocate the data buffers even during the initial creattion of ZFP compressed arrays. The initial compressing of data already uses mmapped RAM. The Linux kernel is then left to manage how much data (and when) is shifted from mmapped-RAM to the underlying disk file, depending on the actual pressure on RAM. And the persistent disk file helps avoid having to compress data again upon future read-only accesses to the data.

I guess different people would have different use scenarios in mind... Mine is just one example. After some careful consideration an elegant solution can surely be found!

[lindstro]

@jvo203 You make a good point. We could add a static function that returns the compressed size of an array of given dimensions, scalar type, and rate. Another option is to add an optional Boolean to the constructors (or as a template parameter) to tell zfp not to allocate memory. Yet another option is to template the zfp arrays on an allocator class. Or to make the alloc function virtual.

We'll consider these options and others to support the more general case of allocating the compressed data outside of zfp. We have to consider what to do when resize is called and how external allocation will interact with zfp's views, etc. I'm hesitant to include mmap and MapViewOfFile solutions directly in zfp if there are reasonable alternatives, as there appear to be.

[jvo203]

@lindstro At the very least making all the memory-related functions like alloc/free virtual would be most welcome (and easiest for you!). At first I tried to extend zfp by adding a custom class with the mmap logic hidden inside but because alloc()/free() were protected but not virtual, C++ would not let me override these crucial methods. So making alloc()/free() override-friendly would be at the top of my personal wish list! Many thanks.