Parallel File I/O

[Arno Candel]

Hi,

Is there a clever way to handle large distributed Array I/O to disk? I 
don't want all contexts to block each other while reading/writing.


A straight-forward reader implementation like



Array<3, double, MultiPatch<GridTag,Remote<Brick> >  A;
A.initialize(Domain, Partition, DistributedTag());

for i=A.domain()[0].first() to A.domain()[0].last()
 for j=A.domain()[1].first() to A.domain()[1].last()
  for k=A.domain()[2].first() to A.domain()[2].last()
    {
      my_ifstream >> value;
      A(i,j,k) = value;
    }




scales miserably when reading a small 18 MB input ASCII file due to 
blocking and communication:


1 node: 15.5176 s
2 nodes: 84.5347 s
4 nodes: 212.838 s
6 nodes: 798.704 s
12 nodes: 1538.78 s


I would appreciate any suggestions for using exisiting (binary) parallel 
I/O tools provided by Pooma r2.4.0

"IO/FileSetReader.h" and "IO/FileSetWriter.h" seem to have problems with 
the latest release.


Thanks,
Arno Candel