avtResampleFilter - parallel distribution not correct. Error in avtImagePartition::EstablishPartitionBoundaries

[ARSanderson]

I have a simple four domain data set that I resampled and the distributed on to four ranks. I was expecting that each rank would have data. They did not. Only three of the four ranks had data. That would be okay but the resampling left gaps in the data which is also not correct. Theses gaps are clearly seen in the last three images, when using 4, 5, and 6 ranks. The first two images use 2 and 3 ranks.

When using 6 ranks only 4 ranks have data - but there are gaps
When using 5 ranks only 4 ranks have data - but there are gaps
When using 4 ranks only 3 ranks have data - but there are gaps

Note: the gaps are NOT at the patch/domain boundaries.

I can consistently reproduce these results. The correct result would be for every rank to have data.

Patch mesh boundaries:

Each cell for two patches:

Volume rendering 2 ranks:

Volume rendering 3 ranks:

Volume rendering 4 ranks:

Volume rendering 5 ranks:

Volume rendering 6 ranks:

[markcmiller86]

@ARSanderson how thick (in terms of numbers of elements) is the dataset in the Z dimension? It looks like it could be just one element thick making it a somewhat unusual case that maybe resample operator isn't yet designed for.

[ARSanderson]

@markcmiller86 - correct, see second image (just added). I would not think that a dimension of 1 would cause any issues. One can not subdivide. To me is it more of modulo issue in the Y axis. But then again with 4 ranks where 3 data set were produced I would expect 3 relatively evenly sized samples in Y. That is not the case. Which is also bothersome.

Regardless though the dimension of 1 is on the original data. The dimensions of the resampled data is 128 1024 8.

[brugger1]

@ARSanderson can you attach the data set?

[ARSanderson]

Datasets that should show the issue when running on four ranks, only three get data.

VolumeMapperData.tar.gz

[ARSanderson]

I looked at avtImagePartition::EstablishPartitionBoundaries which where the issue lies. The case it fails on is the simple case where the number of samples is the same across each scanline (bin). The problem is the equal height histogram problem. Instead of having the bin width be constant with a variable number of samples in each bin one wants the bin width to be variable with number of samples in each bin be as close to being the “same" as possible.

To give some more context the rank that gets none:

Say you have 34 bins each with 1 item in the bin. If there are four ranks each rank ideally would have ~8 items (equal depth). However, there are bounds of 1x to 1.5x of the average in the code of what each rank should get. Thus the goal becomes 8-12 with a prefernce towards the max. So the first rank gets 12, the second gets 12, and the third gets 10 (the balance) and the last rank gets none.

I have done a quick search and the proper way to solve the issue is much more complex. I have not thought how we might do a simple solution, perhaps an iterative solution. Reduce the max until all ranks have data and the variance is minimized.

12 - 12 - 10 - 0
11 - 11 - 11 - 1
10 - 10 - 10 - 4
9 - 9 - 9 - 7
8 - 8 - 8 - 10 - stop

Then look at the variance and pick the best.