[petsc-dev] Errors in MatSetValuesBlockedLocal() when using AIJ, but not BAIJ!

[Smith, Barry F.]

> On Feb 20, 2019, at 12:50 PM, Mills, Richard Tran via petsc-dev <petsc-dev at mcs.anl.gov> wrote:
> 
> Folks,
> 
> I'm working with some PFLOTRAN examples, and I've been scratching my head for a while at errors that are being generated in MatSetValuesBlockedLocal(), related to new nonzeros requiring a malloc. E.g.,
> 
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (2,0) caused a malloc
> 
> The strange thing is that all of these examples run fine when I use the PFLOTRAN default BAIJ matrix type (with blocks column-oriented) -- I only see the issue when I switch to using AIJ. If I set the matrix option MAT_NEW_NONZERO_ALLOCATION_ERR to PETSC_FALSE to allow things to run with AIJ, everything seems to work perfectly. (And once the Jacobian matrix is constructed the first time, the memory usage is not growing according to what I see from -malloc_log.)
> 
> I've spent some time poking around with a debugger in the variants of MatSetValues() and MatXAIJSetPreallocation(), and I haven't been able make much progress in determining why I'm seeing this in AIJ vs. BAIJ -- it's easy to get confused looking at this code, and it's going to take me more time to follow what is going on than I've had to spare so far. So, I've got a few questions. First, should it even be possible to see what I am seeing? That is, if the MatSetValuesBlocked() routine is not causing new allocations when using BAIJ, should this be possible with AIJ?


    Sure, because the preallocation for BAIJ matrix is by block, while for AIJ it is by point (even if you provide a block size to AIJ). So long as the new entry is within an allocated block it will not trigger an error with BAIJ but may with AIJ.

   Barry

> (Clearly it *is* possible, but should it be?) I'm still trying to figure out if there is something wrong with what PFLOTRAN is doing, vs. something going wrong somewhere inside PETSc.
> 
> Any hints about what to look at from someone with more familiarity with this code would be appreciated.
> 
> --Richard