From rguenth at tat.physik.uni-tuebingen.de  Fri May  9 07:44:10 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Fri, 9 May 2003 09:44:10 +0200 (CEST)
Subject: [PATCH] Fix Engine<..., MultiPatch<..., Remote<Brick> > >::makeOwnCopy
Message-ID: <Pine.LNX.4.44.0305090940550.640-100000@goofy>

Hi!

The following patch fixes makeOwnCopy of remote multipatch engines by
moving the ElementProperties specialization for remote engines contained
in the Engine/tests/makeOwnCopy.cpp test (doh!) to the Engine/RemoteEngine.h
file.

Tested by testing the makeOwnCopy testcase and my personal Field based
testcase that failed previously. Ok?

2003 May 09  Richard Guenther <richard.guenther at uni-tuebingen.de>

	* src/Engine/tests/makeOwnCopy.cpp: move ElementProperties
	specialization ...
	src/Engine/RemoteEngine.h: ... here.

Index: src/Engine/RemoteEngine.h
===================================================================
RCS file: /home/pooma/Repository/r2/src/Engine/RemoteEngine.h,v
retrieving revision 1.35
diff -u -r1.35 RemoteEngine.h
--- src/Engine/RemoteEngine.h	5 Mar 2002 16:14:38 -0000	1.35
+++ src/Engine/RemoteEngine.h	9 May 2003 07:40:14 -0000
@@ -2147,6 +2147,16 @@
   }
 };

+//-----------------------------------------------------------------------------
+// Traits class telling RefCountedBlockPointer that this class has
+// shallow semantics and a makeOwnCopy method.
+//-----------------------------------------------------------------------------
+
+template <int Dim, class T, class Eng>
+struct ElementProperties<Engine<Dim, T, Remote<Eng> > >
+  : public MakeOwnCopyProperties<Engine<Dim, T, Remote<Eng> > >
+{ };
+
 // } // namespace Pooma
 ///////////////////////////////////////////////////////////////////////////////

Index: src/Engine/tests/makeOwnCopy.cpp
===================================================================
RCS file: /home/pooma/Repository/r2/src/Engine/tests/makeOwnCopy.cpp,v
retrieving revision 1.1
diff -u -r1.1 makeOwnCopy.cpp
--- src/Engine/tests/makeOwnCopy.cpp	16 May 2001 21:21:07 -0000	1.1
+++ src/Engine/tests/makeOwnCopy.cpp	9 May 2003 07:40:15 -0000
@@ -127,16 +127,6 @@
   Pooma::finalize();
   return ret;
 }
-
-//-----------------------------------------------------------------------------
-// Traits class telling RefCountedBlockPointer that this class has
-// shallow semantics and a makeOwnCopy method.
-//-----------------------------------------------------------------------------
-
-template <int Dim, class T, class Eng>
-struct ElementProperties<Engine<Dim, T, Remote<Eng> > >
-  : public MakeOwnCopyProperties<Engine<Dim, T, Remote<Eng> > >
-{ };

 // ACL:rcsinfo
 // ----------------------------------------------------------------------



From rguenth at tat.physik.uni-tuebingen.de  Wed May 21 15:02:57 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Wed, 21 May 2003 17:02:57 +0200 (CEST)
Subject: Problem with Field::makeOwnCopy() and Relations
Message-ID: <Pine.LNX.4.44.0305211658450.1790-100000@bellatrix.tat.physik.uni-tuebingen.de>

Hi!

We have a problem with Field::makeOwnCopy() and Relations in case of
a Field with multiple subfields. Consider

  Field_t f(canonicalCentering(FaceType, Continuous), ...);
  Pooma::addAllConstantFaceBC(f, 0.0);
  Field_t g(f);
  g.makeOwnCopy();

now in FieldEngine::makeOwnCopy() we get passed g as target s and do

   ...
      data(m, c).relations().makeOwnCopy(s);
   ...

i.e. we retarget all the subfields relation to the base field g which
the relation later chokes on with an assert.

Duh.

Any ideas?

I'll try to pass s.subField(m, c) here, but I dont think this will work
- will it?

Richard.

--
Richard Guenther <richard dot guenther at uni-tuebingen dot de>
WWW: http://www.tat.physik.uni-tuebingen.de/~rguenth/



From rguenth at tat.physik.uni-tuebingen.de  Wed May 21 15:12:48 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Wed, 21 May 2003 17:12:48 +0200 (CEST)
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations
 tests=USER_AGENT_PINE,X_AUTH_WARNING version=2.54
In-Reply-To: <Pine.LNX.4.44.0305211658450.1790-100000@bellatrix.tat.physik.uni-tuebingen.de>
Message-ID: <Pine.LNX.4.44.0305211711470.1790-100000@bellatrix.tat.physik.uni-tuebingen.de>

On Wed, 21 May 2003, Richard Guenther wrote:

> Hi!
>
> We have a problem with Field::makeOwnCopy() and Relations in case of
> a Field with multiple subfields. Consider
>
>   Field_t f(canonicalCentering(FaceType, Continuous), ...);
>   Pooma::addAllConstantFaceBC(f, 0.0);
>   Field_t g(f);
>   g.makeOwnCopy();
>
> now in FieldEngine::makeOwnCopy() we get passed g as target s and do
>
>    ...
>       data(m, c).relations().makeOwnCopy(s);
>    ...
>
> i.e. we retarget all the subfields relation to the base field g which
> the relation later chokes on with an assert.
>
> I'll try to pass s.subField(m, c) here, but I dont think this will work
> - will it?

It seems it does. Tested with Fields, Ok?

Richard.

===== FieldEngine.h 1.4 vs edited =====
--- 1.4/r2/src/Field/FieldEngine/FieldEngine.h	Thu Jan 30 12:11:07 2003
+++ edited/FieldEngine.h	Wed May 21 17:04:26 2003
@@ -606,7 +606,7 @@
       {
         data(m, c) = model[m*stride_m + c];
         data(m, c).engine().makeOwnCopy();
-        data(m, c).relations().makeOwnCopy(s);
+        data(m, c).relations().makeOwnCopy(s.subField(m, c));
       }
     }
   }



From jxyh at lanl.gov  Wed May 21 16:07:14 2003
From: jxyh at lanl.gov (John H. Hall)
Date: Wed, 21 May 2003 10:07:14 -0600
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations tests=USER_AGENT_PINE,X_AUTH_WARNING version=2.54
In-Reply-To: <Pine.LNX.4.44.0305211711470.1790-100000@bellatrix.tat.physik.uni-tuebingen.de>
Message-ID: <495BCD23-8BA6-11D7-B49D-000A27AF286E@lanl.gov>

Richard:
I think I ran into this one. There is an inconsistency between the 
general field interface and the subField interface regarding relations. 
The relations only exist at the lowest subField layer. This becomes 
complex when you begin talking about "multi-material fields" with 
complex centerings. I would like to have it so that if you have a 
relation that involves all the subFields, it would exist at the level 
above the lowest layer of subFields that it depends upon. Right now we 
build some complex relations that have to be called for each centering 
or material, but, they could be called only once as a driver and walk 
through all the subFields themselves. Part of this is my fault, during 
the rewrite to add multi-material fields and the relations concepts we 
were in a hurry and I agreed to have the relations and expressions only 
make sense at the lowest subField level as a practical concession.

The reason you are having so many problems in this one area is that we 
decided to stop using makeOwnCopy and instead came up with some other 
workarounds at the application level (I was on a team of users, not 
developers of POOMA). So we never relied on makeOwnCopy and therefore 
it never got debugged in parallel, it seemed to work fine in serial.

Has anyone explained why we changed the name from Boundary Conditions 
to relations. POOMA has within it a rudimentary implementation of the 
concept of Independent and Dependent fields. Unfortunately, the 
built-in set of relation routines has a defect that doesn't allow a 
Dependent field to be dependent on another field with a different 
centering. We fixed this dilemma in our integration layer with POOMA, 
so the fix never became a part of POOMA itself. Anyhow, when relations 
are working properly, you can extend the concept of automatic updating 
of the Boundary Conditions to include automatic updating of fields that 
are dependent upon other fields. So when an Independent Field (no 
dependencies on other fields) gets changed, it not only marks its own 
Boundaries as dirty, but, it also calls a tree which marks all of its 
dependent fields as dirty also. Therefore, when a dependent field is on 
the RHS of an equation, it triggers a set of relations that result in 
its being current. This is done using deferred evaluation, so it only 
happens if the dependent field is used. Another name for these 
dependent fields would be a cached evaluation. Once the evaluation has 
been triggered, the field is marked as clean and in all further uses it 
acts as a normal field. If you know the calculation will only be done 
once, or that it is extremely cheap to perform, you should probably use 
a calculational engine that doesn't cache the result.

Another feature which we didn't have time to play with, is that when a 
dependent field is marked as dirty, you may as well set it to zero and 
free its data. Since establishing the relation tree is much more 
onerous than managing data, we got to the point where we almost never 
got rid of a field, we just set it to zero (manually instead of 
automatically) and poof its memory was essentially gone.

If you tried hard enough, you should be able to write a code in which 
no field calculations (besides setting up the independent fields) occur 
until a final print statement. At that point all of the relations would 
trigger and all of the work would be done (deferred evaluation).

John Hall


On Wednesday, May 21, 2003, at 09:12  AM, Richard Guenther wrote:

> On Wed, 21 May 2003, Richard Guenther wrote:
>
>> Hi!
>>
>> We have a problem with Field::makeOwnCopy() and Relations in case of
>> a Field with multiple subfields. Consider
>>
>>   Field_t f(canonicalCentering(FaceType, Continuous), ...);
>>   Pooma::addAllConstantFaceBC(f, 0.0);
>>   Field_t g(f);
>>   g.makeOwnCopy();
>>
>> now in FieldEngine::makeOwnCopy() we get passed g as target s and do
>>
>>    ...
>>       data(m, c).relations().makeOwnCopy(s);
>>    ...
>>
>> i.e. we retarget all the subfields relation to the base field g which
>> the relation later chokes on with an assert.
>>
>> I'll try to pass s.subField(m, c) here, but I dont think this will 
>> work
>> - will it?
>
> It seems it does. Tested with Fields, Ok?
>
> Richard.
>
> ===== FieldEngine.h 1.4 vs edited =====
> --- 1.4/r2/src/Field/FieldEngine/FieldEngine.h	Thu Jan 30 12:11:07 2003
> +++ edited/FieldEngine.h	Wed May 21 17:04:26 2003
> @@ -606,7 +606,7 @@
>        {
>          data(m, c) = model[m*stride_m + c];
>          data(m, c).engine().makeOwnCopy();
> -        data(m, c).relations().makeOwnCopy(s);
> +        data(m, c).relations().makeOwnCopy(s.subField(m, c));
>        }
>      }
>    }



From rguenth at tat.physik.uni-tuebingen.de  Wed May 21 17:45:20 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Wed, 21 May 2003 19:45:20 +0200 (CEST)
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations
In-Reply-To: <495BCD23-8BA6-11D7-B49D-000A27AF286E@lanl.gov>
Message-ID: <Pine.LNX.4.44.0305211937040.626-100000@goofy>

On Wed, 21 May 2003, John H. Hall wrote:

> The reason you are having so many problems in this one area is that we
> decided to stop using makeOwnCopy and instead came up with some other
> workarounds at the application level (I was on a team of users, not
> developers of POOMA). So we never relied on makeOwnCopy and therefore
> it never got debugged in parallel, it seemed to work fine in serial.

Well, it didnt compile, so I doubt this. But anyway - its getting better
everytime I spend a few hours hunting for another bug...

> Has anyone explained why we changed the name from Boundary Conditions
> to relations. POOMA has within it a rudimentary implementation of the
> concept of Independent and Dependent fields. Unfortunately, the
> built-in set of relation routines has a defect that doesn't allow a
> Dependent field to be dependent on another field with a different
> centering.

Uh, while I dont use dependend Field feature (yet), I cannot see how
the centering can make a difference here.

> We fixed this dilemma in our integration layer with POOMA,
> so the fix never became a part of POOMA itself. Anyhow, when relations
> are working properly, you can extend the concept of automatic updating
> of the Boundary Conditions to include automatic updating of fields that
> are dependent upon other fields. So when an Independent Field (no
> dependencies on other fields) gets changed, it not only marks its own
> Boundaries as dirty, but, it also calls a tree which marks all of its
> dependent fields as dirty also. Therefore, when a dependent field is on
> the RHS of an equation, it triggers a set of relations that result in
> its being current. This is done using deferred evaluation, so it only
> happens if the dependent field is used. Another name for these
> dependent fields would be a cached evaluation. Once the evaluation has
> been triggered, the field is marked as clean and in all further uses it
> acts as a normal field. If you know the calculation will only be done
> once, or that it is extremely cheap to perform, you should probably use
> a calculational engine that doesn't cache the result.
>
> Another feature which we didn't have time to play with, is that when a
> dependent field is marked as dirty, you may as well set it to zero and
> free its data. Since establishing the relation tree is much more
> onerous than managing data, we got to the point where we almost never
> got rid of a field, we just set it to zero (manually instead of
> automatically) and poof its memory was essentially gone.
>
> If you tried hard enough, you should be able to write a code in which
> no field calculations (besides setting up the independent fields) occur
> until a final print statement. At that point all of the relations would
> trigger and all of the work would be done (deferred evaluation).

Yes, but program (or better, expression) flow would be very hard to follow
then. So from a maintainance point of view I doubt this would be useful.
I see it can be useful for sort of CSE, where you set up a dependent field
for the expression.

As you are (were?) a POOMA user, did you have tricks to overcome the
overly simplistic handing of the inner guards and their exchange? I.e.
the fact that only a simple flag is kept for the state of the internal
guards, so you cannot optimize f.i. directional splitted CFD and instead
do tree times the communication you need to?

Btw. - what is/was the application you were using POOMA on? Is it
available somewhere, so people can learn from it?

Thanks,

Richard.



From rguenth at tat.physik.uni-tuebingen.de  Wed May 21 18:43:52 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Wed, 21 May 2003 20:43:52 +0200 (CEST)
Subject: [Q] Using PETSc from within POOMA applications
Message-ID: <Pine.LNX.4.44.0305212041030.626-100000@goofy>

Hi!

Has anyone tried or been successful using the PETSc library to solve
linear systems with the RHS coming from POOMA Arrays? Or are there any
other preferable ways to solve f.i. a poisson equation with a parallel
setup in POOMA?

Thanks,

Richard.



From jxyh at lanl.gov  Wed May 21 18:57:27 2003
From: jxyh at lanl.gov (John H. Hall)
Date: Wed, 21 May 2003 12:57:27 -0600
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations
In-Reply-To: <Pine.LNX.4.44.0305211937040.626-100000@goofy>
Message-ID: <110314DC-8BBE-11D7-B49D-000A27AF286E@lanl.gov>

Richard:
> Well, it didnt compile, so I doubt this. But anyway - its getting 
> better
> everytime I spend a few hours hunting for another bug...
>
We abandoned makeOwnCopy a long time ago, so maybe something bad 
happened to it later in the serial case also.
>
> Uh, while I dont use dependend Field feature (yet), I cannot see how
> the centering can make a difference here.
>
It been a while since I used POOMA so some of my cobwebs need to be 
shaken out. In moving from POOMA R1 to later versions of POOMA R2 we 
removed centering as a template argument, so you are correct, the real 
issue is that the current instantiation of relations inside POOMA 
requires fields with the same number of subFields because of the way 
Scott wrote the loops. So when I referred to centerings, I didn't mean 
Vert and Cell, I meant Edge and Cell would be incompatible since they 
have a differing number of subFields.
> ...
> Yes, but program (or better, expression) flow would be very hard to 
> follow
> then. So from a maintainance point of view I doubt this would be 
> useful.
> I see it can be useful for sort of CSE, where you set up a dependent 
> field
> for the expression.
>
Since we were trying to investigate modern computer science we 
developed tools allow us to do full round-trip engineering. So the 
dependencies were actually quite simple to track and we worked in a 
graphical environment in maintaining them using a customized version of 
a technology called "Describe" (originally GDPro) from Embarcadero 
Technologies. We had to introduce and work through a lot of concepts, 
like "state" to make things work. Also, the way we implemented 
relations you had to think of them as having views of the things they 
were dependent upon which always led back to the same data, otherwise 
you would get divergent unexpected copies of your data floating around. 
We had some additional structure to support this.

One of the major contributors to code fragility is implicit contracts 
between subroutines which are violated when an unaware subroutine is 
placed between them that violates the "contract". Computers were meant 
to track these things and the use of relations led to a significant 
reduction in user-supplied code. As with all technologies, there are 
pathological cases that would eventually result in trouble if not 
designed around, but, they were fairly straightforward to avoid. Since 
most of them led to infinite loops, that also made it simpler to figure 
out when you had fallen into a trap. We also defined a Macro which was 
placed at the beginning of every method by our automatic code generator 
that would allow us to conditionally turn on the call chain to allow us 
to see if we had any cascades or unnecessary loop updates going on as a 
result of having our dependencies set up incorrectly.

A question we never had to ask was "Has this variable been updated?" 
When you are integrating a large set of physics packages, it becomes 
important to have these contracts explicitly spelled out or remove them 
with a scheme such as ours. Avoiding unintended side-effects becomes 
the name of the game.

> As you are (were?) a POOMA user, did you have tricks to overcome the
> overly simplistic handing of the inner guards and their exchange? I.e.
> the fact that only a simple flag is kept for the state of the internal
> guards, so you cannot optimize f.i. directional splitted CFD and 
> instead
> do tree times the communication you need to?
No, we talked about it a lot. Our hardware was not bandwidth limited, 
it instead had a large latency, so we bundled all the communications 
into as few messages as possible, but, we really didn't care what the 
message size was. We spent much more time thinking about the external 
guard cell fills and while we came up with some ideas for optimizing 
these, most of them have never been implemented.

Our team did come up with the deferred guard cell update idea which was 
wonderfully implemented by Scott Haney (that's the flag you are 
referring to). Before this the guard cells were updated at the end of 
every assign statement, which was truly problematic for performance. 
His code in R1 would only require updates if the RHS expression were 
using guard cells. I believe the R2 version just always requires an 
update if the field is on the RHS of an expression (stencil or no). A 
major driver for recognizing the difference between Dependent and 
Independent Fields was that Dependent Fields would almost never need to 
communicate to fill their guard cells (only if they stenciled across 
the Independent Field). We didn't get far enough along to take 
advantage of that optimization.
>
> Btw. - what is/was the application you were using POOMA on? Is it
> available somewhere, so people can learn from it?
>
There were a bunch of CFD codes all developed under a project at Los 
Alamos National Laboratory known as Blanca. AFAIK none of these codes 
are available to the public and most of them were export controlled so 
they probably never will be.

In the current version you do lose some optimizations if you stick to 
the strict data parallel syntax (e.g. loop fusion, directional guard 
cell fills for operator split algorithms). But it is still the most 
advanced array/field abstraction developed to date and I hope to get 
back to moving it forward again soon.
> Thanks,
>
> Richard.



From rguenth at tat.physik.uni-tuebingen.de  Wed May 21 19:41:27 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Wed, 21 May 2003 21:41:27 +0200 (CEST)
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations
In-Reply-To: <110314DC-8BBE-11D7-B49D-000A27AF286E@lanl.gov>
Message-ID: <Pine.LNX.4.44.0305212135440.626-100000@goofy>

On Wed, 21 May 2003, John H. Hall wrote:

> Richard:
> >
> > Uh, while I dont use dependend Field feature (yet), I cannot see how
> > the centering can make a difference here.
> >
> It been a while since I used POOMA so some of my cobwebs need to be
> shaken out. In moving from POOMA R1 to later versions of POOMA R2 we
> removed centering as a template argument, so you are correct, the real
> issue is that the current instantiation of relations inside POOMA
> requires fields with the same number of subFields because of the way
> Scott wrote the loops. So when I referred to centerings, I didn't mean
> Vert and Cell, I meant Edge and Cell would be incompatible since they
> have a differing number of subFields.

Ah ok, I see what you are talking about. But these expressions involving
multiple subfields are error prone and sometimes quite surprising in the
outcome. Not only because you cannot take views of fields with >1
subfields and calling field.physicalDomain() is "undefined" for fields
with >1 subfield.

> > ...
> > Yes, but program (or better, expression) flow would be very hard to
> > follow
> > then. So from a maintainance point of view I doubt this would be
> > useful.
> > I see it can be useful for sort of CSE, where you set up a dependent
> > field
> > for the expression.
> >
> Since we were trying to investigate modern computer science we
> developed tools allow us to do full round-trip engineering. So the
> dependencies were actually quite simple to track and we worked in a
> graphical environment in maintaining them using a customized version of
> a technology called "Describe" (originally GDPro) from Embarcadero
> Technologies. We had to introduce and work through a lot of concepts,
> like "state" to make things work. Also, the way we implemented
> relations you had to think of them as having views of the things they
> were dependent upon which always led back to the same data, otherwise
> you would get divergent unexpected copies of your data floating around.
> We had some additional structure to support this.
>
> One of the major contributors to code fragility is implicit contracts
> between subroutines which are violated when an unaware subroutine is
> placed between them that violates the "contract". Computers were meant
> to track these things and the use of relations led to a significant
> reduction in user-supplied code. As with all technologies, there are
> pathological cases that would eventually result in trouble if not
> designed around, but, they were fairly straightforward to avoid. Since
> most of them led to infinite loops, that also made it simpler to figure
> out when you had fallen into a trap. We also defined a Macro which was
> placed at the beginning of every method by our automatic code generator
> that would allow us to conditionally turn on the call chain to allow us
> to see if we had any cascades or unnecessary loop updates going on as a
> result of having our dependencies set up incorrectly.
>
> A question we never had to ask was "Has this variable been updated?"
> When you are integrating a large set of physics packages, it becomes
> important to have these contracts explicitly spelled out or remove them
> with a scheme such as ours. Avoiding unintended side-effects becomes
> the name of the game.

Yes, I see. Some highlevel tools to at least view the dependence graph
would be useful. Should be extractable at runtime and displayable by
f.i. the dot package ... sounds like a project ;)

> > As you are (were?) a POOMA user, did you have tricks to overcome the
> > overly simplistic handing of the inner guards and their exchange? I.e.
> > the fact that only a simple flag is kept for the state of the internal
> > guards, so you cannot optimize f.i. directional splitted CFD and
> > instead
> > do tree times the communication you need to?
> No, we talked about it a lot. Our hardware was not bandwidth limited,
> it instead had a large latency, so we bundled all the communications
> into as few messages as possible, but, we really didn't care what the
> message size was. We spent much more time thinking about the external
> guard cell fills and while we came up with some ideas for optimizing
> these, most of them have never been implemented.
>
> Our team did come up with the deferred guard cell update idea which was
> wonderfully implemented by Scott Haney (that's the flag you are
> referring to). Before this the guard cells were updated at the end of
> every assign statement, which was truly problematic for performance.
> His code in R1 would only require updates if the RHS expression were
> using guard cells. I believe the R2 version just always requires an
> update if the field is on the RHS of an expression (stencil or no). A
> major driver for recognizing the difference between Dependent and
> Independent Fields was that Dependent Fields would almost never need to
> communicate to fill their guard cells (only if they stenciled across
> the Independent Field). We didn't get far enough along to take
> advantage of that optimization.

Yes. If I have enough time I'll look into these issues, but a one-day
try had no success sofar.

> >
> > Btw. - what is/was the application you were using POOMA on? Is it
> > available somewhere, so people can learn from it?
> >
> There were a bunch of CFD codes all developed under a project at Los
> Alamos National Laboratory known as Blanca. AFAIK none of these codes
> are available to the public and most of them were export controlled so
> they probably never will be.
>
> In the current version you do lose some optimizations if you stick to
> the strict data parallel syntax (e.g. loop fusion, directional guard
> cell fills for operator split algorithms). But it is still the most
> advanced array/field abstraction developed to date and I hope to get
> back to moving it forward again soon.

I hope so, too. POOMA is really the best designed package for CFD I
know, and I'm glad to have stumbled over it.

Richard.



From jxyh at lanl.gov  Wed May 21 20:49:34 2003
From: jxyh at lanl.gov (John H. Hall)
Date: Wed, 21 May 2003 14:49:34 -0600
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations
In-Reply-To: <Pine.LNX.4.44.0305212135440.626-100000@goofy>
Message-ID: <BA47B030-8BCD-11D7-B49D-000A27AF286E@lanl.gov>


On Wednesday, May 21, 2003, at 01:41  PM, Richard Guenther wrote:

> Not only because you cannot take views of fields with >1
> subfields and calling field.physicalDomain() is "undefined" for fields
> with >1 subfield.
>

I remember the reason for the field.physicalDomain() being defined only 
at the innermost subFields (what would it mean at higher levels since 
it is different for each direction and centering?) But, I don't 
remember a restriction on taking views of edge-centered fields for 
example. We took views of these things all over the place. In fact our 
field object factory was in the unoptimized case up to the second view 
(third object) by the time we returned from its create function call. I 
think you will have to be more specific in what you mean here.

We even had multi-material fields with edge centering, so that they 
would have a layer of subFields for each material and another layer of 
subFields for each direction in the centering and we took views of 
those all the time.

Here is a routine from our object factory:

// =====================================================================
// This method now handles AllFace, AllEdge, and ReplicatedSubFields 
(MMFields)
// TecMesh::newField(const centering& Center, const int numFields)
// =====================================================================

template<class field>
field TecMesh::newField( const CenteringType& center, const int 
numFields, bool forceIt )
{
	typedef field::T_t T;
	typedef field::Layout_t Layout_t;
	enum { Dim = field::dimensions };

	Vector<Dim,Real> origin;
	LoadPoomaVector(origin,stdOrigin); //Copy STL vector (stdOrigin) into 
POOMA vector (origin)
	Vector<Dim,Real> spacings;
	LoadPoomaVector(spacings,stdSpacing);
	Ptr<Layout_t> pLayout = unwrap<Ptr<Layout_t> >(*getLayout());

          Centering<Dim> Center = canonicalCentering<Dim>(center, 
Continuous, AllDim);

	if(( numFields > 1 ) || (forceIt==true)) { // forceIt : e.g. A 
1-material MM_Field needs a subField
                                                                         
      //for compatibility with general multimaterial case
		field tField( numFields, Center,
			*pLayout, origin, spacings ); // Actually create field
                   ... // Initialize Boundary conditions, etc.
		return tField; // return view of field
	}
	else {
		field tField( Center, *pLayout, origin, spacings ); // Actually 
create field
                   ... // Initialize Boundary conditions, etc.
		return tField; // return view of field
	}
}

Notice, we are returning fields as views, not references to fields 
(which would be referring back to deleted views). And for a 
multi-material field with edge centering in 3-D with 10 materials there 
would be 30 subFields at the leaf object level.

We even violated the sanctity of the public interface for POOMA and set 
it up where a Multi-Material field was pointing to the engine for a 
single subField so that we could use the same data some times in a 
single material expression and other times in a multi-material 
expression (actually, a loop over materials to form expressions since 
the POOMA team didn't have time to write the MMField loop expressions)

So something like:
       for(int n=0;n<MM_IntEnergy.numMaterials();++n) {
           MM_IntEnergy.material(n)  = where( MM_Mass.material(n) > eps,
                                     Old.MM_IntEnergy.material(n) - 
(TmpWorkTerm * Dt)/MM_Mass.material(n),
                                     0.0 );
       }
should become:
           MM_IntEnergy  = where( MM_Mass > eps,
                                     Old.MM_IntEnergy - (TmpWorkTerm * 
Dt)/MM_Mass,
                                     0.0 );
This would mean that POOMA could move all of the RHS updating outside 
the loop so that no blocking would occur from material to material 
during the loop.

We could also refer to MM_IntEnergy.material(n) as simply IntEnergy for 
the nth material (we have a bag-type generic object collection known as 
the DataDirectory and the Material objects were derived from this 
hierarchical container and therefore each material could have a field 
known as "IntEnergy"). So that allowed expressions like IntEnergy*=0.5. 
If the single material model got called for each material, then the 
effect would be the same as MM_IntEnergy *= 0.5 which of course right 
now requires a loop over materials.

I am really curious what you mean here. Has something really 
significant changed?
John Hall



From rguenth at tat.physik.uni-tuebingen.de  Thu May 22 07:58:43 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Thu, 22 May 2003 09:58:43 +0200 (CEST)
Subject: [pooma-dev] Problem with Field::makeOwnCopy() and Relations
In-Reply-To: <BA47B030-8BCD-11D7-B49D-000A27AF286E@lanl.gov>
Message-ID: <Pine.LNX.4.44.0305220947540.1790-100000@bellatrix.tat.physik.uni-tuebingen.de>

On Wed, 21 May 2003, John H. Hall wrote:

>
> On Wednesday, May 21, 2003, at 01:41  PM, Richard Guenther wrote:
>
> > Not only because you cannot take views of fields with >1
> > subfields and calling field.physicalDomain() is "undefined" for fields
> > with >1 subfield.
> >
>
> I remember the reason for the field.physicalDomain() being defined only
> at the innermost subFields (what would it mean at higher levels since
> it is different for each direction and centering?)

I agree a generic physicalDomain() on a multi-subfield Field does not make
sense very much, but

Domain_t FieldEngine::physicalDomain() const
{
  if (centeringSize() == 1)
     return cellDomainToCenteringDomain(physicalCellDomain_m, centering_m, 0);
  else
     return physicalCellDomain_m;
}

will probably lead to unexpected results for the "unexperienced" user.
Note that FieldEngine::physicalDomain(int centering) does the right thing
here.

Either I'd expect the above to always return the physical domain of
centering zero, or to abort in the multi-subfield case.


> But, I don't
> remember a restriction on taking views of edge-centered fields for
> example. We took views of these things all over the place. In fact our
> field object factory was in the unoptimized case up to the second view
> (third object) by the time we returned from its create function call. I
> think you will have to be more specific in what you mean here.

Hmm - I refer to

Field.h:
struct View1Implementation<Field<Mesh, T, EngineTag>, Domain,
true>::make

where I can read

   PAssert(f.numSubFields() == 0);

and I stumbled over it yesterday (just noticed when I had -DNOPAssert off,
it _did_ seem to work without - but I'm not sure). Ah I see - this is for
Loc/int views only, so this seems to make sense. Hmm - maybe I need to go
back and look where I got this failure.

Btw. I'm coming across all these limitations/bugs while playing with
ScalarCode and Fields.

Richard.

--
Richard Guenther <richard dot guenther at uni-tuebingen dot de>
WWW: http://www.tat.physik.uni-tuebingen.de/~rguenth/



From rguenth at tat.physik.uni-tuebingen.de  Wed May 28 13:58:30 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Wed, 28 May 2003 15:58:30 +0200 (CEST)
Subject: Brick engine and pointer aliasing
Message-ID: <Pine.LNX.4.44.0305281551470.17960-100000@bellatrix.tat.physik.uni-tuebingen.de>

Hi!

Currently the data members of the Brick and BrickView engines are
_not_ marked restrict, i.e. they're T *data_m. While strictly
speaking this is correct it harms performance on vector computers
quite a lot.

For dataparallel statements in POOMA the result is undefined, if
iterations depend on each other, which is equivalent to that the
compiler may assume restrictness of all data_m pointers, here?
[note the question mark]

For non-dataparallel statements, the situation is more complicated.
While under the restrict assumption a loop like

for (i=0; i<4; ++i)
  A(i) = A(i-1);

is the same as non-restricted(?), if we have two views to the same
Array, things get messed up, as in

  for (i=0; i<4; ++i)
    A(Interval1)(i) = A(Interval2)(i-1);

as now the iterations can be executed in parallel if BrickViews
have restricted data members.

The question now is, do we actually "support" such non-dataparallel
statements involving different views of the same Brick engine? Can
we specify such uses as undefined behavior? Can we mark Brick and
BrickView engine data_m members restrict?

Any thoughts on these issues?

Richard.

--
Richard Guenther <richard dot guenther at uni-tuebingen dot de>
WWW: http://www.tat.physik.uni-tuebingen.de/~rguenth/



From rguenth at tat.physik.uni-tuebingen.de  Thu May 29 22:18:17 2003
From: rguenth at tat.physik.uni-tuebingen.de (Richard Guenther)
Date: Fri, 30 May 2003 00:18:17 +0200 (CEST)
Subject: [PATCH-PING] Re: [pooma-dev] Problem with Field::makeOwnCopy() and
 Relations
In-Reply-To: <Pine.LNX.4.44.0305211711470.1790-100000@bellatrix.tat.physik.uni-tuebingen.de>
Message-ID: <Pine.LNX.4.44.0305300016470.557-100000@goofy>


What about this patch? It fixes the problem with makeOwnCopy() on
multi-centering fields for me.

Richard.

On Wed, 21 May 2003, Richard Guenther wrote:

> On Wed, 21 May 2003, Richard Guenther wrote:
>
> > Hi!
> >
> > We have a problem with Field::makeOwnCopy() and Relations in case of
> > a Field with multiple subfields. Consider
> >
> >   Field_t f(canonicalCentering(FaceType, Continuous), ...);
> >   Pooma::addAllConstantFaceBC(f, 0.0);
> >   Field_t g(f);
> >   g.makeOwnCopy();
> >
> > now in FieldEngine::makeOwnCopy() we get passed g as target s and do
> >
> >    ...
> >       data(m, c).relations().makeOwnCopy(s);
> >    ...
> >
> > i.e. we retarget all the subfields relation to the base field g which
> > the relation later chokes on with an assert.
>
> It seems it does. Tested with Fields, Ok?
>
> Richard.
>
> ===== FieldEngine.h 1.4 vs edited =====
> --- 1.4/r2/src/Field/FieldEngine/FieldEngine.h	Thu Jan 30 12:11:07 2003
> +++ edited/FieldEngine.h	Wed May 21 17:04:26 2003
> @@ -606,7 +606,7 @@
>        {
>          data(m, c) = model[m*stride_m + c];
>          data(m, c).engine().makeOwnCopy();
> -        data(m, c).relations().makeOwnCopy(s);
> +        data(m, c).relations().makeOwnCopy(s.subField(m, c));
>        }
>      }
>    }
>



From oldham at codesourcery.com  Sat May 31 04:05:29 2003
From: oldham at codesourcery.com (Jeffrey D. Oldham)
Date: Fri, 30 May 2003 21:05:29 -0700
Subject: [pooma-dev] Brick engine and pointer aliasing
In-Reply-To: <Pine.LNX.4.44.0305281551470.17960-100000@bellatrix.tat.physik.uni-tuebingen.de>
References: <Pine.LNX.4.44.0305281551470.17960-100000@bellatrix.tat.physik.uni-tuebingen.de>
Message-ID: <3ED82A09.5020808@codesourcery.com>

Richard Guenther wrote:
> Hi!
> 
> Currently the data members of the Brick and BrickView engines are
> _not_ marked restrict, i.e. they're T *data_m. While strictly
> speaking this is correct it harms performance on vector computers
> quite a lot.
> 
> For dataparallel statements in POOMA the result is undefined, if
> iterations depend on each other, which is equivalent to that the
> compiler may assume restrictness of all data_m pointers, here?
> [note the question mark]
> 
> For non-dataparallel statements, the situation is more complicated.
> While under the restrict assumption a loop like
> 
> for (i=0; i<4; ++i)
>   A(i) = A(i-1);
> 
> is the same as non-restricted(?), if we have two views to the same
> Array, things get messed up, as in
> 
>   for (i=0; i<4; ++i)
>     A(Interval1)(i) = A(Interval2)(i-1);
> 
> as now the iterations can be executed in parallel if BrickViews
> have restricted data members.
> 
> The question now is, do we actually "support" such non-dataparallel
> statements involving different views of the same Brick engine? Can
> we specify such uses as undefined behavior? Can we mark Brick and
> BrickView engine data_m members restrict?
> 
> Any thoughts on these issues?

I assume that the rule is that, for a data-parallel assignment, the code 
is only correct if the iterations can occur in any order without any 
change in the result.  Thus, I assume that restrict is acceptable.

Jeffrey D. Oldham
oldham at codesourcery.com