<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> It looks like the current code copies the nonzero values to the CPU from the MPI matrix (with the calls PetscCall(MatSeqAIJGetArrayRead(mpimat->A,&aav));<br class=""> PetscCall(MatSeqAIJGetArrayRead(mpimat->B,&bav));, then copies them into the CPU memory of the Seq matrix. When the matrix entries are next accessed on the GPU it should automatically copy them down to the GPU. So the code looks ok even for GPUs. We'll need to see the full error message with what the "invalid pointer" is.<div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""></div><div class=""> Yes this routine is terribly inefficient for GPU matrices, it needs to be specialized to not use the GPU memory but that is a separate issue from there being bugs in the current code.</div><div class=""><br class=""></div><div class=""> The code also seems to implicitly assume the parallel matrix has the same nonzero pattern with a reuse. This should be checked with each use by stashing the nonzero state of the matrix into the sequential matrix and making sure the parallel matrix has that same stashed value each time. Currently if one changes the nonzero matrix of the parallel matrix one is likely to get random confusing crashes due to memory corruption. But likely not the problem here.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 23, 2022, at 2:23 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">We have a bug in the AMGx test snes_tests-ex13_amgx in parallel.</div><div class="">Matt Martineau found that MatMPIAIJGetLocalMat worked in the first pass in the code below, where the local matrix is created (INITIAL), but in the next pass, when "REUSE" is used, he sees an invalid pointer.</div><div class="">Matt found that it does have offloadmask == CPU. </div><div class="">Maybe it is missing logic to put the output in same state as the input?</div><div class=""><br class=""></div><div class="">Any ideas on this or should I just dig into it?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">bool partial_setup_allowed = (pc->setupcalled && pc->flag != DIFFERENT_NONZERO_PATTERN);
199 if (amgx->nranks > 1) {
200 if (partial_setup_allowed) {
202 PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_REUSE_MATRIX, &amgx->localA)); // This path seems doesn't work by the time we reach AmgX API
203 } else {
205 PetscCall(MatMPIAIJGetLocalMat(Pmat, MAT_INITIAL_MATRIX, &amgx->localA)); // This path works
206 }
207 } else {
208 amgx->localA = Pmat;
209 }
210</pre></div></div>
</div></blockquote></div><br class=""></div></div></body></html>