[petsc-users] About Preallocation for MPI matrix

[Matthew Knepley]

On Mon, Aug 1, 2022 at 11:45 AM E. Ekici <ee331 at cam.ac.uk> wrote:

> Hi,
>
> I would like to preallocate my highly sparse matrices to get speed up
> during assembling.
>
> Up to 70000x70000, it works fine in parallel.
>
> But when I exceed around 70000 rows/columns, I receive following error at
> preallocation line;
>
> petsc4py.PETSc.Error: error code 4031
> [0] MatMPIAIJSetPreallocation() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:4039
> [0] MatMPIAIJSetPreallocation_MPIAIJ() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:2845
> [0] MatSeqAIJSetPreallocation() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:3963
> [0] MatSeqAIJSetPreallocation_SeqAIJ() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:4031
> [0] PetscMallocA() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:401
> [0] PetscMallocAlign() at
> /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:49
> [0] MatSeqAIJSetPreallocation_SeqAIJ
>
> I have attached my sparsity pattern as *CSR.png*.
>
> I use petsc4py for matrix generation, and here are the commands that I am
> using to generate matrix;
>
> row # global row numbers
> col = # global column numbers
> val = # flattened 2D nonzero array
>
> mat = PETSc.Mat().create(PETSc.COMM_WORLD)
> mat.setSizes([(local_size, global_size), (local_size, global_size)])
> mat.setType('mpiaij')
> ONNZ = len(row)*np.ones(local_size,dtype=np.int32)
>

Is len(row) == local_size? This looks like it is allocating a dense
diagonal block and a semi-dense off-diagonal block.
Is that true? If so, you are probably running out of memory.

  Thanks,

      Matt


> mat.setPreallocationNNZ([ONNZ,ONNZ]) # This line throws the error above
> mat.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
> mat.setUp()
> mat.setValues(row, col, val, addv=PETSc.InsertMode.ADD_VALUES)
> mat.assemblyBegin()
> mat.assemblyEnd()
>
> Many thanks for your response in advance,
>
> Kind regards
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220801/84a4f283/attachment.html>