[petsc-users] unsorted local columns in 3.8?
Hong
hzhang at mcs.anl.gov
Sun Nov 5 21:03:06 CST 2017
Mark:
Bug is fixed in branch hzhang/fix-submat_samerowdist
https://bitbucket.org/petsc/petsc/branch/hzhang/fix-submat_samerowdist
I also add the test runex56. Please test it and let me know if there is a
problem.
Hong
Also, I have been using -petscpartition_type but now I see
> -pc_gamg_mat_partitioning_type. Is -petscpartition_type depreciated for
> GAMG?
>
> Is this some sort of auto generated portmanteau? I can not find
> pc_gamg_mat_partitioning_type in the source.
>
> On Thu, Nov 2, 2017 at 6:44 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> Great, thanks,
>>
>> And could you please add these parameters to a regression test? As I
>> recall we have with-parmetis regression test.
>>
>> On Thu, Nov 2, 2017 at 6:35 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> Mark:
>>> I used petsc/src/ksp/ksp/examples/tutorials/ex56.c :-(
>>> Now testing src/snes/examples/tutorials/ex56.c with your options, I can
>>> reproduce the error.
>>> I'll fix it.
>>>
>>> Hong
>>>
>>> Hong,
>>>>
>>>> I've tested with master and I get the same error. Maybe the
>>>> partitioning parameters are wrong. -pc_gamg_mat_partitioning_type is new to
>>>> me.
>>>>
>>>> Can you run this (snes ex56) w/o the error?
>>>>
>>>>
>>>> 17:33 *master* *= ~*/Codes/petsc/src/snes/examples/tutorials*$ make
>>>> runex
>>>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpiexec -n 4
>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2
>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason
>>>> -snes_monitor_short -ksp_monitor_short -snes_converged_reason
>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -matrap 0
>>>> -matptap_scalable -ex56_dm_view -run_type 1 -pc_gamg_repartition true
>>>> [0] 27 global equations, 9 vertices
>>>> [0] 27 equations in vector, 9 vertices
>>>> 0 SNES Function norm 122.396
>>>> 0 KSP Residual norm 122.396
>>>> <snip>
>>>> depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))
>>>> [0] 4725 global equations, 1575 vertices
>>>> [0] 4725 equations in vector, 1575 vertices
>>>> 0 SNES Function norm 17.9091
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>> [0]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>>
>>>>
>>>> On Thu, Nov 2, 2017 at 1:35 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>
>>>>> Mark :
>>>>> I realize that using maint or master branch, I cannot reproduce the
>>>>> same error.
>>>>> For this example, you must use a parallel partitioner, e.g.,'current'
>>>>> gives me following error:
>>>>> [0]PETSC ERROR: This is the DEFAULT NO-OP partitioner, it currently
>>>>> only supports one domain per processor
>>>>> use -pc_gamg_mat_partitioning_type parmetis or chaco or ptscotch for
>>>>> more than one subdomain per processor
>>>>>
>>>>> Please rebase your branch with maint or master, then see if you still
>>>>> have problem.
>>>>>
>>>>> Hong
>>>>>
>>>>>
>>>>>>
>>>>>> On Thu, Nov 2, 2017 at 11:07 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Mark,
>>>>>>> I can reproduce this in an old branch, but not in current maint and
>>>>>>> master.
>>>>>>> Which branch are you using to produce this error?
>>>>>>>
>>>>>>
>>>>>> I am using a branch from Matt. Let me try to merge it with master.
>>>>>>
>>>>>>
>>>>>>> Hong
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 2, 2017 at 9:28 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>
>>>>>>>> I am able to reproduce this with snes ex56 with 2 processors and
>>>>>>>> adding -pc_gamg_repartition true
>>>>>>>>
>>>>>>>> I'm not sure how to fix it.
>>>>>>>>
>>>>>>>> 10:26 1 knepley/feature-plex-boxmesh-create *=
>>>>>>>> ~/Codes/petsc/src/snes/examples/tutorials$ make
>>>>>>>> PETSC_DIR=/Users/markadams/Codes/petsc
>>>>>>>> PETSC_ARCH=arch-macosx-gnu-g runex
>>>>>>>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpiexec -n 2
>>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2
>>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason
>>>>>>>> -snes_monitor_short -ksp_monitor_short -snes_converged_reason
>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>>>> jacobi -petscpartitioner_type simple -mat_block_size 3 -matrap 0
>>>>>>>> -matptap_scalable -ex56_dm_view -run_type 1 -pc_gamg_repartition true
>>>>>>>> [0] 27 global equations, 9 vertices
>>>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>>>> 0 SNES Function norm 122.396
>>>>>>>> 0 KSP Residual norm 122.396
>>>>>>>> 1 KSP Residual norm 20.4696
>>>>>>>> 2 KSP Residual norm 3.95009
>>>>>>>> 3 KSP Residual norm 0.176181
>>>>>>>> 4 KSP Residual norm 0.0208781
>>>>>>>> 5 KSP Residual norm 0.00278873
>>>>>>>> 6 KSP Residual norm 0.000482741
>>>>>>>> 7 KSP Residual norm 4.68085e-05
>>>>>>>> 8 KSP Residual norm 5.42381e-06
>>>>>>>> 9 KSP Residual norm 5.12785e-07
>>>>>>>> 10 KSP Residual norm 2.60389e-08
>>>>>>>> 11 KSP Residual norm 4.96201e-09
>>>>>>>> 12 KSP Residual norm 1.989e-10
>>>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 12
>>>>>>>> 1 SNES Function norm 1.990e-10
>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>> iterations 1
>>>>>>>> DM Object: Mesh (ex56_) 2 MPI processes
>>>>>>>> type: plex
>>>>>>>> Mesh in 3 dimensions:
>>>>>>>> 0-cells: 12 12
>>>>>>>> 1-cells: 20 20
>>>>>>>> 2-cells: 11 11
>>>>>>>> 3-cells: 2 2
>>>>>>>> Labels:
>>>>>>>> boundary: 1 strata with value/size (1 (39))
>>>>>>>> Face Sets: 5 strata with value/size (1 (2), 2 (2), 3 (2), 5 (1),
>>>>>>>> 6 (1))
>>>>>>>> marker: 1 strata with value/size (1 (27))
>>>>>>>> depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2))
>>>>>>>> [0] 441 global equations, 147 vertices
>>>>>>>> [0] 441 equations in vector, 147 vertices
>>>>>>>> 0 SNES Function norm 49.7106
>>>>>>>> 0 KSP Residual norm 49.7106
>>>>>>>> 1 KSP Residual norm 12.9252
>>>>>>>> 2 KSP Residual norm 2.38019
>>>>>>>> 3 KSP Residual norm 0.426307
>>>>>>>> 4 KSP Residual norm 0.0692155
>>>>>>>> 5 KSP Residual norm 0.0123092
>>>>>>>> 6 KSP Residual norm 0.00184874
>>>>>>>> 7 KSP Residual norm 0.000320761
>>>>>>>> 8 KSP Residual norm 5.48957e-05
>>>>>>>> 9 KSP Residual norm 9.90089e-06
>>>>>>>> 10 KSP Residual norm 1.5127e-06
>>>>>>>> 11 KSP Residual norm 2.82192e-07
>>>>>>>> 12 KSP Residual norm 4.62364e-08
>>>>>>>> 13 KSP Residual norm 7.99573e-09
>>>>>>>> 14 KSP Residual norm 1.3028e-09
>>>>>>>> 15 KSP Residual norm 2.174e-10
>>>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 15
>>>>>>>> 1 SNES Function norm 2.174e-10
>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>> iterations 1
>>>>>>>> DM Object: Mesh (ex56_) 2 MPI processes
>>>>>>>> type: plex
>>>>>>>> Mesh in 3 dimensions:
>>>>>>>> 0-cells: 45 45
>>>>>>>> 1-cells: 96 96
>>>>>>>> 2-cells: 68 68
>>>>>>>> 3-cells: 16 16
>>>>>>>> Labels:
>>>>>>>> marker: 1 strata with value/size (1 (129))
>>>>>>>> Face Sets: 5 strata with value/size (1 (18), 2 (18), 3 (18), 5
>>>>>>>> (9), 6 (9))
>>>>>>>> boundary: 1 strata with value/size (1 (141))
>>>>>>>> depth: 4 strata with value/size (0 (45), 1 (96), 2 (68), 3 (16))
>>>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>>>> 0 SNES Function norm 17.9091
>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>> [0]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>> [1]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [1]PETSC ERROR: No support for this operation for this object type
>>>>>>>> [1]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 2, 2017 at 9:51 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 1, 2017 at 9:36 PM, Randy Michael Churchill <
>>>>>>>>> rchurchi at pppl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Doing some additional testing, the issue goes away when removing
>>>>>>>>>> the gamg preconditioner line from the petsc.rc:
>>>>>>>>>> -pc_type gamg
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yea, this is GAMG setup.
>>>>>>>>>
>>>>>>>>> This is the code. findices is create with ISCreateStride, so it
>>>>>>>>> is sorted ...
>>>>>>>>>
>>>>>>>>> Michael is repartitioning the coarse grids. Maybe we don't have a
>>>>>>>>> regression test with this...
>>>>>>>>>
>>>>>>>>> I will try to reproduce this.
>>>>>>>>>
>>>>>>>>> Michael: you can use hypre for now, or turn repartitioning off
>>>>>>>>> (eg, -fsa_fieldsplit_lambda_upper_pc_gamg_repartition false), but
>>>>>>>>> I'm not sure this will fix this.
>>>>>>>>>
>>>>>>>>> You don't have hypre parameters for all of your all of your
>>>>>>>>> solvers. I think 'boomeramg' is the default pc_hypre_type. That should be
>>>>>>>>> good enough for you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> IS findices;
>>>>>>>>> PetscInt Istart,Iend;
>>>>>>>>> Mat Pnew;
>>>>>>>>>
>>>>>>>>> ierr = MatGetOwnershipRange(Pold, &Istart,
>>>>>>>>> &Iend);CHKERRQ(ierr);
>>>>>>>>> #if defined PETSC_GAMG_USE_LOG
>>>>>>>>> ierr = PetscLogEventBegin(petsc_gamg_
>>>>>>>>> setup_events[SET15],0,0,0,0);CHKERRQ(ierr);
>>>>>>>>> #endif
>>>>>>>>> ierr = ISCreateStride(comm,Iend-Istar
>>>>>>>>> t,Istart,1,&findices);CHKERRQ(ierr);
>>>>>>>>> ierr = ISSetBlockSize(findices,f_bs);CHKERRQ(ierr);
>>>>>>>>> ierr = MatCreateSubMatrix(Pold, findices, new_eq_indices,
>>>>>>>>> MAT_INITIAL_MATRIX, &Pnew);CHKERRQ(ierr);
>>>>>>>>> ierr = ISDestroy(&findices);CHKERRQ(ierr);
>>>>>>>>>
>>>>>>>>> #if defined PETSC_GAMG_USE_LOG
>>>>>>>>> ierr = PetscLogEventEnd(petsc_gamg_se
>>>>>>>>> tup_events[SET15],0,0,0,0);CHKERRQ(ierr);
>>>>>>>>> #endif
>>>>>>>>> ierr = MatDestroy(a_P_inout);CHKERRQ(ierr);
>>>>>>>>>
>>>>>>>>> /* output - repartitioned */
>>>>>>>>> *a_P_inout = Pnew;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 1, 2017 at 8:23 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Randy:
>>>>>>>>>>> Thanks, I'll check it tomorrow.
>>>>>>>>>>> Hong
>>>>>>>>>>>
>>>>>>>>>>> OK, this might not be completely satisfactory, because it
>>>>>>>>>>>> doesn't show the partitioning or how the matrix is created, but this
>>>>>>>>>>>> reproduces the problem. I wrote out my matrix, Amat, from the larger
>>>>>>>>>>>> simulation, and load it in this script. This must be run with MPI rank
>>>>>>>>>>>> greater than 1. This may be some combination of my petsc.rc, because when I
>>>>>>>>>>>> use the PetscInitialize with it, it throws the error, but when using
>>>>>>>>>>>> default (PETSC_NULL_CHARACTER) it runs fine.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Oct 31, 2017 at 9:58 AM, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Randy:
>>>>>>>>>>>>> It could be a bug or a missing feature in our new
>>>>>>>>>>>>> MatCreateSubMatrix_MPIAIJ_SameRowDist().
>>>>>>>>>>>>> It would be helpful if you can provide us a simple example
>>>>>>>>>>>>> that produces this example.
>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm running a Fortran code that was just changed over to using
>>>>>>>>>>>>>> petsc 3.8 (previously petsc 3.7.6). An error was thrown during a KSPSetUp()
>>>>>>>>>>>>>> call. The error is "unsorted iscol_local is not implemented yet" (see full
>>>>>>>>>>>>>> error below). I tried to trace down the difference in the source files, but
>>>>>>>>>>>>>> where the error occurs (MatCreateSubMatrix_MPIAIJ_SameRowDist())
>>>>>>>>>>>>>> doesn't seem to have existed in v3.7.6, so I'm unsure how to compare. It
>>>>>>>>>>>>>> seems the error is that the order of the columns locally are unsorted,
>>>>>>>>>>>>>> though I don't think I specify a column order in the creation of the matrix:
>>>>>>>>>>>>>> call MatCreate(this%comm,AA,ierr)
>>>>>>>>>>>>>> call MatSetSizes(AA,npetscloc,npets
>>>>>>>>>>>>>> cloc,nreal,nreal,ierr)
>>>>>>>>>>>>>> call MatSetType(AA,MATAIJ,ierr)
>>>>>>>>>>>>>> call MatSetup(AA,ierr)
>>>>>>>>>>>>>> call MatGetOwnershipRange(AA,low,high,ierr)
>>>>>>>>>>>>>> allocate(d_nnz(npetscloc),o_nnz(npetscloc))
>>>>>>>>>>>>>> call getNNZ(grid,npetscloc,low,high
>>>>>>>>>>>>>> ,d_nnz,o_nnz,this%xgc_petsc,nreal,ierr)
>>>>>>>>>>>>>> call MatSeqAIJSetPreallocation(AA,P
>>>>>>>>>>>>>> ETSC_NULL_INTEGER,d_nnz,ierr)
>>>>>>>>>>>>>> call MatMPIAIJSetPreallocation(AA,P
>>>>>>>>>>>>>> ETSC_NULL_INTEGER,d_nnz,PETSC_NULL_INTEGER,o_nnz,ierr)
>>>>>>>>>>>>>> deallocate(d_nnz,o_nnz)
>>>>>>>>>>>>>> call MatSetOption(AA,MAT_IGNORE_OFF
>>>>>>>>>>>>>> _PROC_ENTRIES,PETSC_TRUE,ierr)
>>>>>>>>>>>>>> call MatSetOption(AA,MAT_KEEP_NONZE
>>>>>>>>>>>>>> RO_PATTERN,PETSC_TRUE,ierr)
>>>>>>>>>>>>>> call MatSetup(AA,ierr)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [62]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> [62]PETSC ERROR: No support for this operation for this
>>>>>>>>>>>>>> object type
>>>>>>>>>>>>>> [62]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>>>>>> [62]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting.
>>>>>>>>>>>>>> [62]PETSC ERROR: Petsc Release Version 3.8.0,
>>>>>>>>>>>>>> unknown[62]PETSC ERROR: #1 MatCreateSubMatrix_MPIAIJ_SameRowDist()
>>>>>>>>>>>>>> line 3418 in /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>> 8.0/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>>>>>>> [62]PETSC ERROR: #2 MatCreateSubMatrix_MPIAIJ() line 3247 in
>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [62]PETSC ERROR: #3 MatCreateSubMatrix() line 7872 in
>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/mat/interface/matrix.c
>>>>>>>>>>>>>> [62]PETSC ERROR: #4 PCGAMGCreateLevel_GAMG() line 383 in
>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [62]PETSC ERROR: #5 PCSetUp_GAMG() line 561 in
>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [62]PETSC ERROR: #6 PCSetUp() line 924 in
>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/ksp/pc/interface/precon.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [62]PETSC ERROR: #7 KSPSetUp() line 378 in
>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/ksp/ksp/interface/itfu
>>>>>>>>>>>>>> nc.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171105/7e01e6b7/attachment-0001.html>
More information about the petsc-users
mailing list