Have zipwrapper understand compressed tar files.

[brugger1]

Is your feature request related to a problem?

Robert Casperson from LLNL called and was trying to open a compressed multipart silo file and this didn't work. He was compressing the files because he wanted to save disk space. The problem is that only the root file got decompressed and any files that were referenced by the root file weren't found. I e-mailed Mark about this and he suggested enhancing the zipwrapper reader to understand compressed tar files. This way uncompressing the file would uncrompress all the files and it would be in the proper form (assuming the files didn't have absolute paths, which would is generally a bad idea anyway).

The compressed tar file name would need to be the same as the root file name. For example, if the files were:

abc.root.silo
abc/abc.0.silo
abc/abc.1.silo

Then the compressed tar file name would be "abc.root.silo.tar.gz" consisting of "abc.root.silo" and the directory "abc".

Impact

• High - User productivity significantly improved

[markcmiller86]

@casperson1 I'll be recording some notes here as I begin work. Since you are new to GitHub, I suggest having a quick look over our docs on managing notifications. NOTE: that link will be active only for a while because its part of a pull request. If it fails, try this one

[markcmiller86]

If possible, I'd like to make the solution we develop generally useful and not require a special file naming convention. That said, one basic requirement is that each tarball (and companion root file) represents the file(s) needed for a single timestep. But, I think that can pan out in a few different ways...

Case 1A all files, including root file are in the tarball

• User asks VisIt to open foo_0000.tar.gz or foo_0001.tar.gz, etc...
• ZipWrapper uncompresses and untars foo_0000.tar.gz producing a directory, foo_0000 with a root file, foo_0000/foo.root, and domain level files, foo_0000/part000000.silo, foo_0000/part000001.silo, ..., in the directory
• So, stem.tar.gz maps to stem dir in the file system

Case 1B all files, including root file are in the tarball but root expands at same level as tarball...

• User asks VisIt to open foo_0000.tar.gz or foo_0001.tar.gz, etc...
• ZipWrapper uncompresses and untars foo_0000.tar.gz producing a root file at the same level as the tarfile foo0000.root and a directory, foo_0000 with domain level files, foo_0000/part000000.silo, foo_0000/part000001.silo, ..., in the directory
• stem.tar.gz maps to stem.<ext> and stem dir in the file system (a pair of things)

Case 2 root file and tarball are companion files

• User asks VisIt to open foo_0000.root.gz
• ZipWrapper uncompresses the root file
• VisIt checks for companion foo_0000.tar.gz and if present VisIt untars and uncompresses the companion tarball.
• stem.<grouping>.gz uncompresses to stem.<grouping> and a stem.tar.gz gets uncompressed too.

[markcmiller86]

From @casperson1

$ ls
xxxx00000.root
xxxx00000 <- silo files in here

$ zip -r xxxx00000.root.zip xxxx00000*
(Adds xxxx00000.root and all the files in the xxxx00000 directory)

Ideally when I open xxxx00000.root.zip in visit with ZipWrapper it would extract in the tmp directory, and know to open xxxx00000.root. That file depends on the silo files in xxxx00000 which I believe would extract along with xxxx00000.root.zip. It almost seems like visit knows to do this, but I may be misunderstanding. The point I got stuck today is it didn’t actually unzip the file and seemed to try to do something with sh, leading to permissions errors.

[markcmiller86]

Ideally when I open xxxx00000.root.zip in visit with ZipWrapper it would extract in the tmp directory, and know to open xxxx00000.root.

Ok, so this is captured in Case 1B...the user asks VisIt to open a compressed archive and the root file expands as the same level as the archive and all the domain files are in a subdir.