[petsc-users] VecAssembly gets stuck

[José Lorenzo]

Hello,

I am solving a finite element problem with Dirichlet boundary conditions
using PETSC. In the boundary conditions there are two terms: a first one
that is known before hand (normally zero) and a second term that depends
linearly on the unknown variable itself in the whole domain. Therefore, at
every time step I need to iterate as the boundary condition depends on the
field and the latter depends on the BC. Moreover, the problem is nonlinear
and I use a ghosted vector to represent the field.

Every processor manages a portion of the domain and a portion of the
boundary (if not interior). At every Newton iteration within the time loop
the way I set the boundary conditions is as follows:

First, each processor computes the known term of the BC (first term) and
inserts the values into the vector

call VecSetValues(H, nedge_own, edglocglo(diredg_loc) - 1, Hdir,
INSERT_VALUES, ierr)
call VecAssemblyBegin(H, ierr)
call VecAssemblyEnd(H, ierr)

As far as I understand, at this stage VecAssembly will not need to
communicate to other processors as each processor only sets values to
components that belong to it.

Then, each processor computes its own contribution to the field-dependent
term of the BC for the whole domain boundary as

call VecSetValues(H, nedge_all, edgappglo(diredg_app) - 1, Hself,
ADD_VALUES, ierr)
call VecAssemblyBegin(H, ierr)
call VecAssemblyEnd(H, ierr)

In this case communication will be needed as each processor will add values
to vector components that are not stored by it, and I guess it might get
very busy as all the processors will need to communicate with each other.

When using this strategy I don't find any issue for problems using a small
amount of processors, but recently I've been solving using 90 processors
and the simulation always hangs at the second VecSetValues at some random
time step. It works fine for some time steps but at some point it just gets
stuck and I have to cancel the simulation.

I have managed to overcome this by making each processor contribute to its
own components using first MPI_Reduce and then doing

call VecSetValues(H, nedge_own, edgappglo(diredg_app_loc), Hself_own,
ADD_VALUES, ierr)
call VecAssemblyBegin(H, ierr)
call VecAssemblyEnd(H, ierr)

However I would like to understand whether there is something wrong in the
code above.

Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190913/8793b843/attachment.html>