how to initialize a parallel vector from one process?

[Andrew Colombi]

PETSc will automatically distribute data to the appropriate processor.
 When you use VecSetValues it will cache values that belong to other
processors and distribute when you call
VecAssemblyBegin/VecAssemblyEnd.

So you can do this:

Loop over y:
    if rank == 0:
        read slice y
        VecSetValues( y )
  VecAssemblyBegin()
  VecAssemblyEnd()

Note that you can run out of memory on rank 0 process if you try to
cache too many values before calling the Assembly methods.

-Andrew

On Sun, Jul 6, 2008 at 5:01 AM, Tonellot, Thierry-Laurent D
<thierrylaurent.tonellot at aramco.com> wrote:
>
> Thank you Jed,
>
> I'm in the same situation where I cannot assume that the data can fit in one process memory space.
> I will certainly end by implementing it in the dirty way.
> It would be neat if a function could take care of distributing the data to each process based on the global indices range. Maybe such a function already exists...
>
> Another solution would be to write a new viewer similar to binary viewer... Except that instead of reading in a file one would perform a request to a database.
> Did you consider this option while implementing your own initialization?
> Do you know if there is any documentation on how to implement a new viewer?
>
> Thierry
>
> -----Original Message-----
> From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Jed Brown
> Sent: Sunday, July 06, 2008 12:27 PM
> To: petsc-users at mcs.anl.gov
> Subject: Re: how to initialize a parallel vector from one process?
>
> I've done something similar when reading and writing netCDF in parallel where
> the files are too large to be stored on a single processor.  NetCDF-4 makes this
> obsolete, but here's the idea:
>
> * The new parallel job makes a PETSc DA and uses PETSC_DECIDE to for partitioning.
>
> * Process zero reads the header and broadcasts the dimensions.
>
> * Each process determines the index range it needs to interpolate the file data
>  onto the locally owned computational grid.  Send this index range to rank
>  zero.
>
> * Rank zero reads each of the blocks sequentially (the netCDF API has a read
>  (imin,imax)x(jmin,jmax)x(kmin,kmax)) and sends it to the appropriate process.
>
> * Each process does the necessary interpolation locally.
>
> I've found that this performs just fine for many GiB of state and hundreds of
> processors.  You have to get your hands a bit dirty with MPI.  Of course, there
> are simpler (pure PETSc solutions) if you can fit the whole state on process
> zero or if you can use the PETSc binary format.
>
> Jed
>
>
> On Sun 2008-07-06 11:22, Tonellot, Thierry-Laurent D wrote:
>> Hi,
>>
>>
>>
>> At the very beginning of my application I need to read data in a database to
>> initialize a 3D vector distributed over several processes.
>>
>> The database I'm using can only be accessed by one process (for instance the
>> process 0). Moreover, due to performance issues, we need to limit the request
>> to the database. Therefore the data need to be read by slices, for instance
>> (z,x) slices.
>>
>> A typical initialization would then consist in the following pseudo code:
>>
>>
>>
>> Loop over y
>>
>>             If (rank=0)
>>
>>                         Read slice y
>>
>>                         Send to each process the appropriate part of the data
>> slice
>>
>> Else
>>
>>             Receive data
>>
>> Endif
>>
>> End loop
>>
>>
>>
>> This process is quite heavy and its performances will probably depend on the
>> way it is implemented.
>>
>>
>>
>> I'm wondering if there is any way to perform this initialization efficiently
>> using Petsc?
>>
>>
>>
>> I'm also considering other packages to handle distributed arrays and I'm
>> wondering how a package like global arrays compares with petsc/DA?
>>
>>
>>
>> For instance global arrays seem to have a feature which is partly solving my
>> problem above using the function "ga_fill_patch" which fills only a region of
>> the parallel vector and can be called by any process...
>>
>>
>>
>> Thank you in advance,
>>
>>
>>
>> Thierry
>>
>
>