[petsc-dev] MatMPIAIJGetLocalMat problem with GPUs

[Mark Adams]

It looks like the problem is with a commit: ** | | | | | e24d7920346 -
Fixing issue with PCReset_AMGX (5 days ago) <Matt Martineau> *
(see at end of log below)
Somehow this got on a different "branch".
I manually merged this commit and the rest of the rebase went well.

* 3d182f284fb - (HEAD -> adams/pcamgx, origin/adams/pcamgx) Adding some
debug statements for current issue with MPI replace_coeffs. Added dense LU
rows parameter. Fixed bug with is_dev_ptrs check on localA. (4 days ago)
<Matt Martineau>
* 1eddad9f41b - Fixed missing header for AmgX. (5 days ago) <Matt Martineau>
*   f21c19d509c - Merging changes. (5 days ago) <Matt Martineau>
|\
| * 9588175158c - Formatting. (11 days ago) <Matt Martineau>
| * d2177216795 - added example of AMGx get resource handle (13 days ago)
<markadams4>
| * 39ddb1e5c9f - added example of AMGx get resource handle (13 days ago)
<markadams4>
| * b910459cea6 - Apply 1 suggestion(s) to 1 file(s) (13 days ago) <Junchao
Zhang>
| * c7aba6013ac - added man page for PCAmgXGetResources (13 days ago)
<markadams4>
| * 9614b39f079 - format (3 weeks ago) <markadams4>
| * 2e4e38bb34f - format (3 weeks ago) <markadams4>
| * 43a573f96cd - format (3 weeks ago) <markadams4>
| * 51d8249942b - move amgx test to exx13, a Laplacian, and added Matt's
parameters (3 weeks ago) <Mark Adams>
| * 091b47e09a4 - Add PC interface to AmgX (3 weeks ago) <Matt Martineau>
| *   1973fd480b2 - Merge branch
'jacobf/2022-06-07/ci-fixup-ifort-warn-flag' into 'main' (3 weeks ago)
<Satish Balay>
| |\
| | * 3ef015ba504 - (origin/jacobf/2022-06-07/ci-fixup-ifort-warn-flag)
ifort does not support Werror, it uses -warn instead (3 weeks ago) <Jacob
Faibussowitsch>
| |/
| *   af8607ca2c7 - Merge branch 'psanan/dmstag-docs' into 'main' (3 weeks
ago) <Satish Balay>
| |\
| | * dbcff43011e - DMStag: add test ex51 to use an example in the manual
(3 weeks ago) <Patrick Sanan>
| | * c1733020a21 - DMStag: clean up man pages (3 weeks ago) <Patrick Sanan>
| | * 6dcdbf3e3f6 - Docs: DMStag manual chapter (3 weeks ago) <Patrick
Sanan>
| | * 3d36c81e509 - DMStag ex6: clean up comment (3 weeks ago) <Patrick
Sanan>
| * |   52de7263196 - Merge branch 'jczhang/add-matmult-benchmark' into
'main' (3 weeks ago) <Satish Balay>
| |\ \
| | |/
| |/|
| | * 25f1f01c597 - Mat: add a simple test to benchmark MatMult() (3 weeks
ago) <Junchao Zhang>
| * |   18a8fff0f58 - Merge remote-tracking branch 'origin/release' (3
weeks ago) <Satish Balay>
| |\ \
| | * \   1dd2f923321 - Merge branch 'relax-mpi-check' into 'release' (3
weeks ago) <Satish Balay>
| | |\ \
| | | * | 1c2847649e9 - allow more compatible mpi versions (6 weeks ago)
<Min RK>
| | * | |   0fc06cf4238 - Merge branch 'knepley/fix-schurm-getpmat' into
'release' (3 weeks ago) <Satish Balay>
| | |\ \ \
| | | * | | ec892a4cfec - Mat: Initialize argument correctly Reported-by:
Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no> (3 weeks ago)
<Matthew G. Knepley>
| | * | | |   8c18d935a40 - Merge branch
'barry/2022-05-27/avoid-cmake-findmpi-bug-mpi-c-compiler/release' into
'release' (3 weeks ago) <Satish Balay>
| | |\ \ \ \
| | | |/ / /
| | |/| | |
| | | * | | 5bc5884e1b9 -
(origin/barry/2022-05-27/avoid-cmake-findmpi-bug-mpi-c-compiler/release)
Here is the relevant part of CMake's findMPI() (4 weeks ago) <Barry Smith>
| * | | | |   01fc38526f9 - Merge branch 'jczhang/fix-kokkos-libdl' into
'main' (3 weeks ago) <Satish Balay>
| |\ \ \ \ \
| | * | | | | 73ad7384285 - (origin/jczhang/fix-kokkos-libdl) Config: the
-DKokkos_ENABLE_LIBDL=OFF workaround is only need for CUDA. (3 weeks ago)
<Junchao Zhang>
| * | | | | |   8520412e16e - Merge branch 'DMLabel' into 'main' (3 weeks
ago) <Satish Balay>
| |\ \ \ \ \ \
| | * | | | | | ca1f07610f2 - Wrap most of DMLabel (3 weeks ago) <Romain
Beucher>
| | | |_|_|_|/
| | |/| | | |
| * | | | | |   29b033dba35 - Merge branch 'balay/buildshared_with_at_file'
into 'main' (3 weeks ago) <Satish Balay>
| |\ \ \ \ \ \
| | |_|/ / / /
| |/| | | | |
| | * | | | | f16d4c0b3d5 - build: use @file linker option in the shared
library target - to avoid a very long link command (3 weeks ago) <Satish
Balay>
| |/ / / / /
| * | | | |   7554fc37512 - Merge branch 'knepley/fix-plex-extrude-dim'
into 'main' (3 weeks ago) <Satish Balay>
| |\ \ \ \ \
| | |/ / / /
| |/| | | |
| | * | | | b1b17bd547d - Plex: Fix dimension when extruding from submesh
(3 weeks ago) <Matthew G. Knepley>
| * | | | |   a1c3d047ac1 - Merge branch 'release' (3 weeks ago) <Satish
Balay>
| |\ \ \ \ \
| | |/ / / /
| |/| / / /
| | |/ / /
| | * | | 32a6fd01d7f - (tag: v3.17.2) Increase patchlevel to 3.17.2 (3
weeks ago) <Satish Balay>
| | * | |   efd2351b6b4 - Merge branch 'balay/fix-theta' into 'release' (3
weeks ago) <Satish Balay>
| | |\ \ \
| | | * | | 65b5c8351be - (origin/balay/fix-theta) configure: fix CI build
error on theta (4 weeks ago) <Satish Balay>
| | |/ / /
| * | | |   540c0c32db4 - Merge branch
'barry/2022-05-24/add-petsccall-fortran-examples' into 'main' (4 weeks ago)
<Satish Balay>
| |\ \ \ \
| | * | | | d8606c274c0 -
(origin/barry/2022-05-24/add-petsccall-fortran-examples) Add modern error
checking to rest of Fortran examples (4 weeks ago) <Barry Smith>
| | * | | | f84028053d1 - Add modern error checking to sys/tests and
tutorial Fortran examples (5 weeks ago) <Barry Smith>
| | * | | | a28b22e78e5 - Fix for bug in PetscCallMPI() from Fortran (5
weeks ago) <Barry Smith>
| | * | | | 7f4bc9a7294 - Error immediately from configure if standard math
functions not found (5 weeks ago) <Barry Smith>
| | * | | | ab02228480e - making fortran stubs should not cause
regeneration of test harness scripts (5 weeks ago) <Barry Smith>
| * | | | |   dae471e4997 - Merge remote-tracking branch 'origin/release'
(4 weeks ago) <Satish Balay>
| |\ \ \ \ \
| | | |/ / /
| | |/| | |
| | * | | |   44b0fa11aac - Merge branch 'balay/dll-ifort' into 'release'
(4 weeks ago) <Satish Balay>
| | |\ \ \ \
| | | * | | | eabb67cc616 - (origin/balay/dll-ifort) CI: reorder windows
stage2/3 jobs (4 weeks ago) <Satish Balay>
| | | * | | | 7f5817409f5 - CI: windows short paths are not portable - so
first evaluate, and the use. (4 weeks ago) <Satish Balay>
| | | * | | | 58b343d3ad5 - CI: enable windows/ifort test with shared
libraries (4 weeks ago) <Satish Balay>
| | | * | | | 96c4ff540a1 - Fix compile issues with win32/ifort and
--with-shared-libraries (4 weeks ago) <Satish Balay>
| | |/ / / /
| | * | | |   4f99c01180d - Merge branch
'haplav/fix-dmplexcreatepartitionergraph' into 'release' (4 weeks ago)
<Satish Balay>
| | |\ \ \ \
| | | * | | | b68380d8d6b - DMPlexCreatePartitionerGraph_{Overlap,Native}:
fix indexing for cell start > 0 (4 weeks ago) <Vaclav Hapla>
| | |/ / / /
** | | | | | e24d7920346 - Fixing issue with PCReset_AMGX (5 days ago)
<Matt Martineau>*

On Sun, Jun 26, 2022 at 10:16 AM Mark Adams <mfadams at lbl.gov> wrote:

>
>
> On Sat, Jun 25, 2022 at 9:39 AM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>>   Does AMGX require sorted column indices? (Python indentation notation
>> below)
>>
>>   If not
>>      just use MatMPIAIJGetLocalMatMerge instead of MatMPIAIJGetLocalMat.
>>
>>
> Ugh, I worked on this this AM without rebasing over main and lost my work
> ... there are conflicts in amgx.cxx that I don't understand.
> amgx.cxx should not even be in main so I am puzzled.
>
> Anyway, I added  MatMPIAIJGetLocalMatMerge (and lost it), but I'm not sure
> how to resolve the new ordering of the data.
> MatMPIAIJGetLocalMatMerge returns an index set (IS) with the global
> columns of the diagonal block columns.
> I am thinking that we could simply clobber the first 'n' column indices in
> the MatMPIAIJGetLocalMatMerge output matrix with this IS.
> This would make it what Matt wants I think.
> (Maybe this clobber should go into MatMPIAIJGetLocalMatMerge? Maybe I am
> missing something...)
>
> I see:
> PetscCall(MatGetRowIJ(amgx->localA, 0, PETSC_FALSE, PETSC_FALSE,
> &amgx->nLocalRows, &rowOffsets, &colIndices, &done));
>
> Maybe something like this needs to be called next to get AMGx the new
> indices?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220627/2f0894d2/attachment-0001.html>