[petsc-users] MatCreateBAIJ, SNES, Preallocation...

[William Coirier]

Folks:

I'm developing an application using the SNES, and overall it's working great, as many of our other PETSc-based projects. But, I'm having a problem related to (presumably) pre-allocation, block matrices and SNES.

Without going into details about the actual problem we are solving, here are the symptoms/characteristics/behavior.

  *   For the SNES Jacobian, I'm using MatCreateBAIJ for a block size=3, and letting "PETSC_DECIDE" the partitioning. Actual call is:

     *   ierr = MatCreateBAIJ(PETSC_COMM_WORLD, bs, PETSC_DECIDE, PETSC_DECIDE, (int)3 * numNodesSAM, (int)3 * numNodesSAM, PETSC_DEFAULT, NULL, PETSC_DEFAULT, NULL, &J);

  *   When registering the SNES jacobian function, I set the B and J matrices to be the same.

     *       ierr = SNESSetJacobian(snes, J, J, SAMformSNESJ, (void *)this); CHKERRQ(ierr);

  *   I can either let PETSc figure out the allocation structure:

     *   ierr = MatMPIBAIJSetPreallocation(J, bs, PETSC_DEFAULT, NULL,PETSC_DEFAULT, NULL);

  *    or, do it myself, since I know the fill pattern,

     *   ierr = MatMPIBAIJSetPreallocation(J, bs, d_nz_dum,&d_nnz[0],o_nz_dum,&o_nnz[0]);
The symptoms/problems are as follows:

  *   Whether I do preallocation or not, the "setup" time is pretty long. It might take 2 minutes before SNES starts doing its thing. After this setup, convergence and speed is great. But this first phase takes a long time. I'm assuming this has to be related to some poor preallocation setup so it's doing tons of mallocs where it's not needed.
  *   If I don't call my Jacobian formulation before calling SNESSolve, I get a segmentation violation in a PETSc routine.  (If I DO call my Jacobian first, things work great, although slow for the setup phase.) Here's a snippet of the traceback:
0  0x00000000009649fc in MatMultAdd_SeqBAIJ_3 (A=<optimized out>,
    xx=0x3a525b0, yy=0x3a531b0, zz=0x3a531b0)
    at /home/jstutts/Downloads/petsc-3.11.1/src/mat/impls/baij/seq/baij2.c:1424
#1  0x00000000006444cb in MatMult_MPIBAIJ (A=0x15da340, xx=0x3a542a0,
    yy=0x3a531b0)
    at /home/jstutts/Downloads/petsc-3.11.1/src/mat/impls/baij/mpi/mpibaij.c:1380
#2  0x00000000005b2c0f in MatMult (mat=0x15da340, x=x at entry=0x3a542a0,
    y=y at entry=0x3a531b0)
    at /home/jstutts/Downloads/petsc-3.11.1/src/mat/interface/matrix.c:2396
#3  0x0000000000c61f2e in PCApplyBAorAB (pc=0x1ce78c0, side=PC_LEFT,
    x=0x3a542a0, y=y at entry=0x3a548a0, work=0x3a531b0)
    at /home/jstutts/Downloads/petsc-3.11.1/src/ksp/pc/interface/precon.c:690
#4  0x0000000000ccb36b in KSP_PCApplyBAorAB (w=<optimized out>, y=0x3a548a0,
    x=<optimized out>, ksp=0x1d44d50)
    at /home/jstutts/Downloads/petsc-3.11.1/include/petsc/private/kspimpl.h:309
#5  KSPGMRESCycle (itcount=itcount at entry=0x7fffffffc02c,
    ksp=ksp at entry=0x1d44d50)
    at /home/jstutts/Downloads/petsc-3.11.1/src/ksp/ksp/impls/gmres/gmres.c:152
#6  0x0000000000ccbf6f in KSPSolve_GMRES (ksp=0x1d44d50)
    at /home/jstutts/Downloads/petsc-3.11.1/src/ksp/ksp/impls/gmres/gmres.c:237
#7  0x00000000007dc193 in KSPSolve (ksp=0x1d44d50, b=b at entry=0x1d41c70,
    x=x at entry=0x1cebf40)

I apologize if I've missed something in the documentation or examples, but I can't seem to figure this one out. The "setup" seems to take too long, and from my previous experiences with PETSc, this is due to a poor preallocation strategy.
Any and all help is appreciated!
-----------------------------------------------------------------------
William J. Coirier, Ph.D.
Director, Aerosciences and Engineering Analysis Branch
Advanced Concepts Development and Test Division
Kratos Defense and Rocket Support Services
4904 Research Drive
Huntsville, AL 35805
256-327-8170
256-327-8120 (fax)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190516/862a1346/attachment.html>