[petsc-users] VecAssembly gets stuck

[José Lorenzo]

I thought so and ran the code with the option -malloc_dump but everything
looked fine, no warnings were displayed in a smaller case. Perhaps this is
not enough?

Yes, all the processes call both VecAssemblyBegin/End.

I will try valgrind.

El vie., 13 sept. 2019 a las 22:45, Stefano Zampini (<
stefano.zampini at gmail.com>) escribió:

>
>
> On Sep 13, 2019, at 11:37 PM, José Lorenzo via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> I'm using PETSc 3.10.2, I guess it is the master branch but I do not know
> for sure as I didn't install it myself.
>
> You are right, each processor provides data for all the boundary entries.
>
> I have carried out a few more tests and apparently it gets stuck during
> VecAssemblyBegin. I don't know whether I can be more preciseabout this.
>
> El vie., 13 sept. 2019 a las 20:14, Smith, Barry F. (<bsmith at mcs.anl.gov>)
> escribió:
>
>>
>>   What version of PETSc is this? The master branch?
>>
>> > call VecSetValues(H, nedge_all, edgappglo(diredg_app) - 1, Hself,
>> ADD_VALUES, ierr)
>>
>> So each process is providing all data for all the boundary entries in the
>> vector?
>>
>> I don't think there is anything wrong with what you are doing but the
>> mechanism that does the communication inside the VecAssembly cannot know
>> about the structure of the communication and so will do it inefficiently.
>> It would be useful to know where it is "stuck" that might help us improve
>> the assembly process for your case.
>>
>> But I think it would better for you to just use MPI directly to put the
>> data where it is needed.
>>
>>
>> > On Sep 13, 2019, at 11:30 AM, José Lorenzo via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Hello,
>> >
>> > I am solving a finite element problem with Dirichlet boundary
>> conditions using PETSC. In the boundary conditions there are two terms: a
>> first one that is known before hand (normally zero) and a second term that
>> depends linearly on the unknown variable itself in the whole domain.
>> Therefore, at every time step I need to iterate as the boundary condition
>> depends on the field and the latter depends on the BC. Moreover, the
>> problem is nonlinear and I use a ghosted vector to represent the field.
>> >
>> > Every processor manages a portion of the domain and a portion of the
>> boundary (if not interior). At every Newton iteration within the time loop
>> the way I set the boundary conditions is as follows:
>> >
>> > First, each processor computes the known term of the BC (first term)
>> and inserts the values into the vector
>> >
>> > call VecSetValues(H, nedge_own, edglocglo(diredg_loc) - 1, Hdir,
>> INSERT_VALUES, ierr)
>> > call VecAssemblyBegin(H, ierr)
>> > call VecAssemblyEnd(H, ierr)
>> >
>> > As far as I understand, at this stage VecAssembly will not need to
>> communicate to other processors as each processor only sets values to
>> components that belong to it.
>> >
>> > Then, each processor computes its own contribution to the
>> field-dependent term of the BC for the whole domain boundary as
>> >
>> > call VecSetValues(H, nedge_all, edgappglo(diredg_app) - 1, Hself,
>> ADD_VALUES, ierr)
>> > call VecAssemblyBegin(H, ierr)
>> > call VecAssemblyEnd(H, ierr)
>> >
>> > In this case communication will be needed as each processor will add
>> values to vector components that are not stored by it, and I guess it might
>> get very busy as all the processors will need to communicate with each
>> other.
>> >
>> > When using this strategy I don't find any issue for problems using a
>> small amount of processors, but recently I've been solving using 90
>> processors and the simulation always hangs at the second VecSetValues at
>> some random time step. It works fine for some time steps but at some point
>> it just gets stuck and I have to cancel the simulation.
>> >
>>
>
> The words “hangs” and “gets stuck” 99% percent of the time indicates some
> memory issue with your code.
> First; are all processes calling VecAssemblyBegin/End or only a subset of
> them? VecAssemblyBegin/End must be called by all processes
> Second: run with valgrind, even a smaller case http://www.valgrind.org/
>
> > I have managed to overcome this by making each processor contribute to
>> its own components using first MPI_Reduce and then doing
>> >
>> > call VecSetValues(H, nedge_own, edgappglo(diredg_app_loc), Hself_own,
>> ADD_VALUES, ierr)
>> > call VecAssemblyBegin(H, ierr)
>> > call VecAssemblyEnd(H, ierr)
>> >
>> > However I would like to understand whether there is something wrong in
>> the code above.
>> >
>> > Thank you.
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190913/8d92e095/attachment-0001.html>