[petsc-users] Matrix assembly takes too long

[Barry Smith]

Call MatSetOption(mat,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);

Before calling MatZeroRows() the first time, insert the periodic condition extra column value into the matrix for each periodic row so the matrix "has room" for that extra value.

With this change, there should never be any mallocs, and the process should be pretty fast.

  Barry



> On Jul 15, 2024, at 5:19 AM, Alex Sp <alex.sp2408 at gmail.com> wrote:
> 
> I am using versions 3.19 and 3.20.
> 
> Indeed I get extra mallocs during MatSetValues(). The problem is somewhere in the imposition of periodicity. Since the condition for any unknown u is: u(z=-W) - u(z=+W) = 0, what I do is to store in all processes the global node numbers in these boundaries. Then I call MatZeroRows() for the rows corresponding to (z=-W) and set 1 in the diagonal. Finally, I call MatSetValues() for the same rows and the columns corresponding to (z=+W) to insert the -1 value. I suspect that this creates the problem but why does it keep adding mallocs after the first assembly?
> Here is part of the -info output:
> 
> [0] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374781
> [0] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778
> [0] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778
> [0] <vec> VecAssemblyBegin_MPI_BTS(): Stash has 11980 entries, uses 3 mallocs.
> [0] <vec> VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs.
> [0] <mat> MatAssemblyBegin_MPIAIJ(): Stash has 5529600 entries, uses 0 mallocs.
> [0] <vec> VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs.
> [0] <vec> VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs.
> [0] <mat> MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 13835 X 36985; storage space: 1990 unneeded,2343675 used
> [0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 20339
> [0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 330
> [0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 13835) < 0.6. Do not use CompressedRow routines.
> 
> On Mon, 15 Jul 2024 at 01:36, Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>> 
>> 
>>> On Jul 14, 2024, at 6:04 PM, Alex Sp <alex.sp2408 at gmail.com <mailto:alex.sp2408 at gmail.com>> wrote:
>>> 
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>> Hello,
>>> 
>>> I am trying to solve the problem of the 3D fluid flow around a cylinder using an in-house finite element Fortran code. I want to apply periodic boundary conditions in the neutral direction (the z direction which coincides with the axis of the cylinder). The method works fine and it gives correct results. However the assembly of the matrix for the Newton Raphson method takes too long. The whole process takes about 10 minutes for a single Newton iteration. For the same system size it takes only 10 seconds if instead of periodic conditions I set wall boundaries (no slip, no penetration). What is also strange is that the very first matrix assembly is quite normal (takes about 30 seconds). 
>> 
>>    Does the matrix non-zero structure change between the first and later matrix assemblies? Your behavior is generally a result of new non-zero locations being entered into the matrix after the first assembly is done. 
>> 
>>    What version of PETSc are you using?
>> 
>>    You can run with the option -info and grep for mallocs to see if additional mallocs are needed
>> 
>> 
>> 
>>> What could be the reason for this? It seems like some matrix entries are allocated anew every time. However this happens only when I apply periodicity. I do not do any matrix preallocation.
>>> 
>>> Thanks in advance,
>>> Alex
>> 
> 
> 
> --
> Alexandros Spyridakis
> PhD Candidate
> Department of Chemical Engineering, University of Patras, Greece
> Laboratory of Fluid Mechanics and Rheology

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240715/1ab090d7/attachment.html>