[petsc-users] VecAssembly gets stuck

[Stefano Zampini]

> On Sep 13, 2019, at 11:37 PM, José Lorenzo via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> I'm using PETSc 3.10.2, I guess it is the master branch but I do not know for sure as I didn't install it myself.
> 
> You are right, each processor provides data for all the boundary entries.
> 
> I have carried out a few more tests and apparently it gets stuck during VecAssemblyBegin. I don't know whether I can be more preciseabout this.
> 
> El vie., 13 sept. 2019 a las 20:14, Smith, Barry F. (<bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>) escribió:
> 
>   What version of PETSc is this? The master branch? 
> 
> > call VecSetValues(H, nedge_all, edgappglo(diredg_app) - 1, Hself, ADD_VALUES, ierr)
> 
> So each process is providing all data for all the boundary entries in the vector?
> 
> I don't think there is anything wrong with what you are doing but the mechanism that does the communication inside the VecAssembly cannot know about the structure of the communication and so will do it inefficiently. It would be useful to know where it is "stuck" that might help us improve the assembly process for your case.
> 
> But I think it would better for you to just use MPI directly to put the data where it is needed. 
> 
> 
> > On Sep 13, 2019, at 11:30 AM, José Lorenzo via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > 
> > Hello,
> > 
> > I am solving a finite element problem with Dirichlet boundary conditions using PETSC. In the boundary conditions there are two terms: a first one that is known before hand (normally zero) and a second term that depends linearly on the unknown variable itself in the whole domain. Therefore, at every time step I need to iterate as the boundary condition depends on the field and the latter depends on the BC. Moreover, the problem is nonlinear and I use a ghosted vector to represent the field.
> > 
> > Every processor manages a portion of the domain and a portion of the boundary (if not interior). At every Newton iteration within the time loop the way I set the boundary conditions is as follows:
> > 
> > First, each processor computes the known term of the BC (first term) and inserts the values into the vector
> > 
> > call VecSetValues(H, nedge_own, edglocglo(diredg_loc) - 1, Hdir, INSERT_VALUES, ierr)
> > call VecAssemblyBegin(H, ierr)
> > call VecAssemblyEnd(H, ierr)
> > 
> > As far as I understand, at this stage VecAssembly will not need to communicate to other processors as each processor only sets values to components that belong to it.
> > 
> > Then, each processor computes its own contribution to the field-dependent term of the BC for the whole domain boundary as 
> > 
> > call VecSetValues(H, nedge_all, edgappglo(diredg_app) - 1, Hself, ADD_VALUES, ierr)
> > call VecAssemblyBegin(H, ierr)
> > call VecAssemblyEnd(H, ierr)
> > 
> > In this case communication will be needed as each processor will add values to vector components that are not stored by it, and I guess it might get very busy as all the processors will need to communicate with each other.
> > 
> > When using this strategy I don't find any issue for problems using a small amount of processors, but recently I've been solving using 90 processors and the simulation always hangs at the second VecSetValues at some random time step. It works fine for some time steps but at some point it just gets stuck and I have to cancel the simulation. 
> > 

The words “hangs” and “gets stuck” 99% percent of the time indicates some memory issue with your code.
First; are all processes calling VecAssemblyBegin/End or only a subset of them? VecAssemblyBegin/End must be called by all processes
Second: run with valgrind, even a smaller case http://www.valgrind.org/ <http://www.valgrind.org/>
> > I have managed to overcome this by making each processor contribute to its own components using first MPI_Reduce and then doing
> > 
> > call VecSetValues(H, nedge_own, edgappglo(diredg_app_loc), Hself_own, ADD_VALUES, ierr)
> > call VecAssemblyBegin(H, ierr)
> > call VecAssemblyEnd(H, ierr)
> > 
> > However I would like to understand whether there is something wrong in the code above.
> > 
> > Thank you.
> > 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190913/2ad23e2b/attachment.html>