[petsc-dev] MatMPIAIJGetLocalMat problem with GPUs

[Mark Adams]

We have a bug in the AMGx test snes_tests-ex13_amgx in parallel.
Matt Martineau found that MatMPIAIJGetLocalMat worked in the first pass in
the code below, where the local matrix is created (INITIAL), but in the
next pass, when "REUSE" is used, he sees an invalid pointer.
Matt found that it does have offloadmask == CPU.
Maybe it is missing logic to put the output in same state as the input?

Any ideas on this or should I just dig into it?

Thanks,

bool partial_setup_allowed = (pc->setupcalled && pc->flag !=
DIFFERENT_NONZERO_PATTERN);
199   if (amgx->nranks > 1) {
200     if (partial_setup_allowed) {
202       PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX,
&amgx->localA)); // This path seems doesn't work by the time we reach
AmgX API
203     } else {
205       PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX,
&amgx->localA)); // This path works
206     }
207   } else {
208     amgx->localA = Pmat;
209   }
210
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220623/c14bbd7f/attachment.html>