[petsc-users] local Vec to global Vec and global 3-D array

[Barry Smith]

On May 15, 2013, at 12:41 PM, Roc Wang <pengxwang at hotmail.com> wrote:

> Thanks, please take a look my further questions. 
> 
> > Subject: Re: [petsc-users] local Vec to global Vec and global 3-D array
> > From: bsmith at mcs.anl.gov
> > Date: Wed, 15 May 2013 11:40:03 -0500
> > CC: petsc-users at mcs.anl.gov
> > To: pengxwang at hotmail.com
> > 
> > 
> > On May 15, 2013, at 10:19 AM, Roc Wang <pengxwang at hotmail.com> wrote:
> > 
> > > Hello, 
> > > 
> > > 1. I am trying to save the solution of a PDE for a 3-D geometry domain. I used DM to manage the matrix and vector.
> > > The solution vector was converted to local 3D arrays on each process successfully. But there was errors when the functions DMLocalToGlobalBegin() and DMLocalToGlobalEnd() were called.
> > > 
> > > I built the code by following the example in /petsc-3.3-p6/src/dm/examples/tutorials/ex3.c. The codes for this are as followings :
> > > 
> > > /*Get the solution vec */
> > > ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr);
> > > ierr = KSPGetSolution(ksp,&x);CHKERRQ(ierr);
> > > 
> > > /* local 3d arrays */ 
> > > ierr = DMDAVecGetArray(da, x, &localArray3d ); CHKERRQ(ierr);
> > > 
> > > /* global 3d array */ 
> > > ierr = DMCreateGlobalVector(da,&gsol3d);CHKERRQ(ierr);
> > > ierr = DMLocalToGlobalBegin(da,x,INSERT_VALUES,gsol3d);CHKERRQ(ierr);
> > > ierr = DMLocalToGlobalEnd(da,x,INSERT_VALUES,gsol3d);CHKERRQ(ierr);
> > 
> > The x here must not have been obtained from this da. You need to have obtained the x somewhere before with DMCreateLocalVector(da,&x) or with a VecDuplicate from such a vector. Likely the vector put into the KSP is not from this da.
> 
> !***************************************
>   The procedure of creating x is like this:
>   
>   call KSPCreate() to build ksp;
>   call DMDACreate3d() to build da;
>   ierr = KSPSetDM(ksp,da); // associate da with ksp;
>   
>   Then after ksp was solved:
>   
>   ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr);
>   ierr = KSPGetSolution(ksp,&x);CHKERRQ(ierr);
>   ierr = DMDAVecGetArray(da, x, &localArray3d ); CHKERRQ(ierr);
> 
>  Here, da was associated with the ksp by calling ierr = KSPSetDM(ksp,da).
>  The x here was obtained by calling the KSPGetSolution(ksp,&x), before this I didn't call DMCreateLocalVector(da,&x).  It seems Vec x is associated with da since there is no errors when DMDAVecGetArray(da, x, &localArray3d ) was called.  

   The x from KSPGetSolution() IS a global vector, it is not a global vector. You cannot do a local to global from the x global vector.

> Do you mean I should call DMCreateLocalVector(da,&x) explicitly before ierr = KSPGetSolution(ksp,&x)?
> !***************************************
> 
> > > 
> > > 
> > > 2. I am trying to visualize the solution in a 3-D domain by using some software such as Tecplot. To my understand, I need to output the solution in the format of 3-D array. The vectors of KSP solution in PETSc are local vectors on each process. So they have to be assembled to a global vector and then converted to a 3-D global array. I am not sure if my approach is a good way and if there is some functions in PETSc to output the global solutions directly?
> > 
> > Yes. VecView() on DMDA vectors automatically reorders the entries on the file to use the natural ordering. There are variety of possible viewers you can use including binary, ASCII (no good for anything but tiny problems), VTK, HDF5 
> 
> !***************************************
>  VecView only outputs the vector itself. How to output the indexes of nodes in 3 dimemsions (i,j,k) for each element of the vector, such as:
>  
>  i   j  k  x
>  0 0 0  1.0
>  0 0 1  2.0
>  ...
>  m n p  10.0
> !***************************************

   If you want some ASCII format like that, then write a STAND ALONE sequential program that reads from the binary file with VecLoad() and then outputs the format you want. Never Never ever try to do a parallel output of data in this kind of format, it will be slow and hard to write and there is no reason to write it.

   Barry

> > 
> > Barry
> > 
> > > 
> > > Part of the error information when DMLocalToGlobalBegin() and DMLocalToGlobalEnd() were called 
> > > 
> > > [0]PETSC ERROR: [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message ------------------------------------
> > > [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> > > [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> > > Nonconforming object sizes!
> > > Nonconforming object sizes!
> > > [2]PETSC ERROR: [3]PETSC ERROR: Vector wrong size 30 for scatter 60 (scatter forward and vector from != ctx from size)!
> > > Vector wrong size 20 for scatter 45 (scatter forward and vector from != ctx from size)!
> >