[pooma-dev] Problem with Field::makeOwnCopy() and Relations

[John H. Hall]

Richard:
> Well, it didnt compile, so I doubt this. But anyway - its getting 
> better
> everytime I spend a few hours hunting for another bug...
>
We abandoned makeOwnCopy a long time ago, so maybe something bad 
happened to it later in the serial case also.
>
> Uh, while I dont use dependend Field feature (yet), I cannot see how
> the centering can make a difference here.
>
It been a while since I used POOMA so some of my cobwebs need to be 
shaken out. In moving from POOMA R1 to later versions of POOMA R2 we 
removed centering as a template argument, so you are correct, the real 
issue is that the current instantiation of relations inside POOMA 
requires fields with the same number of subFields because of the way 
Scott wrote the loops. So when I referred to centerings, I didn't mean 
Vert and Cell, I meant Edge and Cell would be incompatible since they 
have a differing number of subFields.
> ...
> Yes, but program (or better, expression) flow would be very hard to 
> follow
> then. So from a maintainance point of view I doubt this would be 
> useful.
> I see it can be useful for sort of CSE, where you set up a dependent 
> field
> for the expression.
>
Since we were trying to investigate modern computer science we 
developed tools allow us to do full round-trip engineering. So the 
dependencies were actually quite simple to track and we worked in a 
graphical environment in maintaining them using a customized version of 
a technology called "Describe" (originally GDPro) from Embarcadero 
Technologies. We had to introduce and work through a lot of concepts, 
like "state" to make things work. Also, the way we implemented 
relations you had to think of them as having views of the things they 
were dependent upon which always led back to the same data, otherwise 
you would get divergent unexpected copies of your data floating around. 
We had some additional structure to support this.

One of the major contributors to code fragility is implicit contracts 
between subroutines which are violated when an unaware subroutine is 
placed between them that violates the "contract". Computers were meant 
to track these things and the use of relations led to a significant 
reduction in user-supplied code. As with all technologies, there are 
pathological cases that would eventually result in trouble if not 
designed around, but, they were fairly straightforward to avoid. Since 
most of them led to infinite loops, that also made it simpler to figure 
out when you had fallen into a trap. We also defined a Macro which was 
placed at the beginning of every method by our automatic code generator 
that would allow us to conditionally turn on the call chain to allow us 
to see if we had any cascades or unnecessary loop updates going on as a 
result of having our dependencies set up incorrectly.

A question we never had to ask was "Has this variable been updated?" 
When you are integrating a large set of physics packages, it becomes 
important to have these contracts explicitly spelled out or remove them 
with a scheme such as ours. Avoiding unintended side-effects becomes 
the name of the game.

> As you are (were?) a POOMA user, did you have tricks to overcome the
> overly simplistic handing of the inner guards and their exchange? I.e.
> the fact that only a simple flag is kept for the state of the internal
> guards, so you cannot optimize f.i. directional splitted CFD and 
> instead
> do tree times the communication you need to?
No, we talked about it a lot. Our hardware was not bandwidth limited, 
it instead had a large latency, so we bundled all the communications 
into as few messages as possible, but, we really didn't care what the 
message size was. We spent much more time thinking about the external 
guard cell fills and while we came up with some ideas for optimizing 
these, most of them have never been implemented.

Our team did come up with the deferred guard cell update idea which was 
wonderfully implemented by Scott Haney (that's the flag you are 
referring to). Before this the guard cells were updated at the end of 
every assign statement, which was truly problematic for performance. 
His code in R1 would only require updates if the RHS expression were 
using guard cells. I believe the R2 version just always requires an 
update if the field is on the RHS of an expression (stencil or no). A 
major driver for recognizing the difference between Dependent and 
Independent Fields was that Dependent Fields would almost never need to 
communicate to fill their guard cells (only if they stenciled across 
the Independent Field). We didn't get far enough along to take 
advantage of that optimization.
>
> Btw. - what is/was the application you were using POOMA on? Is it
> available somewhere, so people can learn from it?
>
There were a bunch of CFD codes all developed under a project at Los 
Alamos National Laboratory known as Blanca. AFAIK none of these codes 
are available to the public and most of them were export controlled so 
they probably never will be.

In the current version you do lose some optimizations if you stick to 
the strict data parallel syntax (e.g. loop fusion, directional guard 
cell fills for operator split algorithms). But it is still the most 
advanced array/field abstraction developed to date and I hope to get 
back to moving it forward again soon.
> Thanks,
>
> Richard.