[petsc-users] unsorted local columns in 3.8?
Hong
hzhang at mcs.anl.gov
Wed Nov 8 11:24:02 CST 2017
mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order
2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11
-ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg
-pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
-pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
-pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason
-use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
-mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type jacobi
-pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1
==30976== Invalid read of size 16
==30976== at 0x8550946: dswap_k_NEHALEM (in
/usr/lib/openblas-base/libblas.so.3)
==30976== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas.so.3)
==30976== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.3.0)
==30976== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)
==30976== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
==30976== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
==30976== by 0x41056E: main (ex56.c:395)
==30976== Address 0xdc650d0 is 52,480 bytes inside a block of size 52,488
alloc'd
==30976== at 0x4C2D110: memalign (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30976== by 0x51590F6: PetscMallocAlign (mal.c:39)
==30976== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)
==30976== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
==30976== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
==30976== by 0x41056E: main (ex56.c:395)
You can fix it on branch hzhang/fix-submat_samerowdist.
Hong
On Wed, Nov 8, 2017 at 11:01 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
>
> On Wed, Nov 8, 2017 at 11:09 AM, Hong <hzhang at mcs.anl.gov> wrote:
>
>> Mark:
>>
>>> Hong, is
>>> > 0-cells: 12 12 0 0
>>> > 1-cells: 20 20 0 0
>>> > 2-cells: 11 11 0 0
>>> > 3-cells: 2 2 0 0
>>>
>>> from the old version?
>>>
>> In O-build on my macPro, I get the above. In g-build, I get
>>
>>>
>>> 0-cells: 8 8 8 8
>>> 1-cells: 12 12 12 12
>>> 2-cells: 6 6 6 6
>>> 3-cells: 1 1 1 1
>>>
>> I get this on linux machine.
>> Do you know why?
>>
>
> I can not reproduce your O output. I will look at it later.
>
> Valgrind is failing on me right now. I will look into it but can you
> valgrind it?
>
>
>>
>> Hong
>>
>>>
>>> On Tue, Nov 7, 2017 at 10:13 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>
>>>> Mark:
>>>> I removed option '-ex56_dm_view'.
>>>> Hong
>>>>
>>>> Humm, this looks a little odd, but it may be OK. Is this this diffing
>>>>> with the old non-repartition data? (more below)
>>>>>
>>>>> On Tue, Nov 7, 2017 at 11:45 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>
>>>>>> Mark,
>>>>>> The fix is merged to next branch for tests which show diff as
>>>>>>
>>>>>> ******* Testing: testexamples_PARMETIS *******
>>>>>> 5c5
>>>>>> < 1 SNES Function norm 1.983e-10
>>>>>> ---
>>>>>> > 1 SNES Function norm 1.990e-10
>>>>>> 10,13c10,13
>>>>>> < 0-cells: 8 8 8 8
>>>>>> < 1-cells: 12 12 12 12
>>>>>> < 2-cells: 6 6 6 6
>>>>>> < 3-cells: 1 1 1 1
>>>>>>
>>>>>>
>>>>> I assume this is the old.
>>>>>
>>>>>
>>>>>> ---
>>>>>> > 0-cells: 12 12 0 0
>>>>>> > 1-cells: 20 20 0 0
>>>>>> > 2-cells: 11 11 0 0
>>>>>> > 3-cells: 2 2 0 0
>>>>>> 15,18c15,18
>>>>>>
>>>>>>
>>>>> and this is the new.
>>>>>
>>>>> This is funny because the processors are not fully populated. This can
>>>>> happen on coarse grids and indeed it should happen in a test with good
>>>>> coverage.
>>>>>
>>>>> I assume these diffs are views from coarse grids? That is, in the raw
>>>>> output files do you see fully populated fine grids, with no diffs, and then
>>>>> the diffs come on coarse grids.
>>>>>
>>>>> Repartitioning the coarse grids can change the coarsening, It is
>>>>> possible that repartitioning causes faster coarsening (it does a little)
>>>>> and this faster coarsening is tripping the aggregation switch, which gives
>>>>> us empty processors.
>>>>>
>>>>> Am I understanding this correctly ...
>>>>>
>>>>> Thanks,
>>>>> Mark
>>>>>
>>>>>
>>>>>> < boundary: 1 strata with value/size (1 (23))
>>>>>> < Face Sets: 4 strata with value/size (1 (1), 2 (1), 4 (1), 6 (1))
>>>>>> < marker: 1 strata with value/size (1 (15))
>>>>>> < depth: 4 strata with value/size (0 (8), 1 (12), 2 (6), 3 (1))
>>>>>> ---
>>>>>> > boundary: 1 strata with value/size (1 (39))
>>>>>> > Face Sets: 5 strata with value/size (1 (2), 2 (2), 4 (2), 5 (1), 6 (1))
>>>>>> > marker: 1 strata with value/size (1 (27))
>>>>>> > depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2))
>>>>>>
>>>>>> see http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/07/examples_full_next-tmp.log
>>>>>>
>>>>>> I guess parmetis produces random partition on different machines (I made output file for ex56_1 on my imac). Please take a look at the differences. If the outputs are correct, I will remove option '-ex56_dm_view'
>>>>>>
>>>>>> Hong
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 5, 2017 at 9:03 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Mark:
>>>>>>> Bug is fixed in branch hzhang/fix-submat_samerowdist
>>>>>>> https://bitbucket.org/petsc/petsc/branch/hzhang/fix-submat_s
>>>>>>> amerowdist
>>>>>>>
>>>>>>> I also add the test runex56. Please test it and let me know if there
>>>>>>> is a problem.
>>>>>>> Hong
>>>>>>>
>>>>>>> Also, I have been using -petscpartition_type but now I see
>>>>>>>> -pc_gamg_mat_partitioning_type. Is -petscpartition_type
>>>>>>>> depreciated for GAMG?
>>>>>>>>
>>>>>>>> Is this some sort of auto generated portmanteau? I can not find
>>>>>>>> pc_gamg_mat_partitioning_type in the source.
>>>>>>>>
>>>>>>>> On Thu, Nov 2, 2017 at 6:44 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>>
>>>>>>>>> Great, thanks,
>>>>>>>>>
>>>>>>>>> And could you please add these parameters to a regression test? As
>>>>>>>>> I recall we have with-parmetis regression test.
>>>>>>>>>
>>>>>>>>> On Thu, Nov 2, 2017 at 6:35 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Mark:
>>>>>>>>>> I used petsc/src/ksp/ksp/examples/tutorials/ex56.c :-(
>>>>>>>>>> Now testing src/snes/examples/tutorials/ex56.c with your
>>>>>>>>>> options, I can reproduce the error.
>>>>>>>>>> I'll fix it.
>>>>>>>>>>
>>>>>>>>>> Hong
>>>>>>>>>>
>>>>>>>>>> Hong,
>>>>>>>>>>>
>>>>>>>>>>> I've tested with master and I get the same error. Maybe the
>>>>>>>>>>> partitioning parameters are wrong. -pc_gamg_mat_partitioning_type is new to
>>>>>>>>>>> me.
>>>>>>>>>>>
>>>>>>>>>>> Can you run this (snes ex56) w/o the error?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 17:33 *master* *= ~*/Codes/petsc/src/snes/examples/tutorials*$
>>>>>>>>>>> make runex
>>>>>>>>>>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpiexec -n 4
>>>>>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2
>>>>>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>>>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>>>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason
>>>>>>>>>>> -snes_monitor_short -ksp_monitor_short -snes_converged_reason
>>>>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
>>>>>>>>>>> -mg_levels_pc_type jacobi -pc_gamg_mat_partitioning_type parmetis
>>>>>>>>>>> -mat_block_size 3 -matrap 0 -matptap_scalable -ex56_dm_view -run_type 1
>>>>>>>>>>> -pc_gamg_repartition true
>>>>>>>>>>> [0] 27 global equations, 9 vertices
>>>>>>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>>>>>>> 0 SNES Function norm 122.396
>>>>>>>>>>> 0 KSP Residual norm 122.396
>>>>>>>>>>> <snip>
>>>>>>>>>>> depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))
>>>>>>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>>>>>>> 0 SNES Function norm 17.9091
>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object
>>>>>>>>>>> type
>>>>>>>>>>> [0]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>>>>>>>>>>> ocumentation/faq.html for trouble shooting.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 2, 2017 at 1:35 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Mark :
>>>>>>>>>>>> I realize that using maint or master branch, I cannot reproduce
>>>>>>>>>>>> the same error.
>>>>>>>>>>>> For this example, you must use a parallel partitioner,
>>>>>>>>>>>> e.g.,'current' gives me following error:
>>>>>>>>>>>> [0]PETSC ERROR: This is the DEFAULT NO-OP partitioner, it
>>>>>>>>>>>> currently only supports one domain per processor
>>>>>>>>>>>> use -pc_gamg_mat_partitioning_type parmetis or chaco or
>>>>>>>>>>>> ptscotch for more than one subdomain per processor
>>>>>>>>>>>>
>>>>>>>>>>>> Please rebase your branch with maint or master, then see if you
>>>>>>>>>>>> still have problem.
>>>>>>>>>>>>
>>>>>>>>>>>> Hong
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 2, 2017 at 11:07 AM, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mark,
>>>>>>>>>>>>>> I can reproduce this in an old branch, but not in current
>>>>>>>>>>>>>> maint and master.
>>>>>>>>>>>>>> Which branch are you using to produce this error?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using a branch from Matt. Let me try to merge it with
>>>>>>>>>>>>> master.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 2, 2017 at 9:28 AM, Mark Adams <mfadams at lbl.gov>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am able to reproduce this with snes ex56 with 2 processors
>>>>>>>>>>>>>>> and adding -pc_gamg_repartition true
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not sure how to fix it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 10:26 1 knepley/feature-plex-boxmesh-create *=
>>>>>>>>>>>>>>> ~/Codes/petsc/src/snes/examples/tutorials$ make
>>>>>>>>>>>>>>> PETSC_DIR=/Users/markadams/Codes/petsc
>>>>>>>>>>>>>>> PETSC_ARCH=arch-macosx-gnu-g runex
>>>>>>>>>>>>>>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpiexec
>>>>>>>>>>>>>>> -n 2 ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it
>>>>>>>>>>>>>>> 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>>>>>>>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>>>>>>>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason
>>>>>>>>>>>>>>> -snes_monitor_short -ksp_monitor_short -snes_converged_reason
>>>>>>>>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
>>>>>>>>>>>>>>> -mg_levels_pc_type jacobi -petscpartitioner_type simple -mat_block_size 3
>>>>>>>>>>>>>>> -matrap 0 -matptap_scalable -ex56_dm_view -run_type 1 -pc_gamg_repartition
>>>>>>>>>>>>>>> true
>>>>>>>>>>>>>>> [0] 27 global equations, 9 vertices
>>>>>>>>>>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>>>>>>>>>>> 0 SNES Function norm 122.396
>>>>>>>>>>>>>>> 0 KSP Residual norm 122.396
>>>>>>>>>>>>>>> 1 KSP Residual norm 20.4696
>>>>>>>>>>>>>>> 2 KSP Residual norm 3.95009
>>>>>>>>>>>>>>> 3 KSP Residual norm 0.176181
>>>>>>>>>>>>>>> 4 KSP Residual norm 0.0208781
>>>>>>>>>>>>>>> 5 KSP Residual norm 0.00278873
>>>>>>>>>>>>>>> 6 KSP Residual norm 0.000482741
>>>>>>>>>>>>>>> 7 KSP Residual norm 4.68085e-05
>>>>>>>>>>>>>>> 8 KSP Residual norm 5.42381e-06
>>>>>>>>>>>>>>> 9 KSP Residual norm 5.12785e-07
>>>>>>>>>>>>>>> 10 KSP Residual norm 2.60389e-08
>>>>>>>>>>>>>>> 11 KSP Residual norm 4.96201e-09
>>>>>>>>>>>>>>> 12 KSP Residual norm 1.989e-10
>>>>>>>>>>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 12
>>>>>>>>>>>>>>> 1 SNES Function norm 1.990e-10
>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>>>>>>>>> iterations 1
>>>>>>>>>>>>>>> DM Object: Mesh (ex56_) 2 MPI processes
>>>>>>>>>>>>>>> type: plex
>>>>>>>>>>>>>>> Mesh in 3 dimensions:
>>>>>>>>>>>>>>> 0-cells: 12 12
>>>>>>>>>>>>>>> 1-cells: 20 20
>>>>>>>>>>>>>>> 2-cells: 11 11
>>>>>>>>>>>>>>> 3-cells: 2 2
>>>>>>>>>>>>>>> Labels:
>>>>>>>>>>>>>>> boundary: 1 strata with value/size (1 (39))
>>>>>>>>>>>>>>> Face Sets: 5 strata with value/size (1 (2), 2 (2), 3 (2),
>>>>>>>>>>>>>>> 5 (1), 6 (1))
>>>>>>>>>>>>>>> marker: 1 strata with value/size (1 (27))
>>>>>>>>>>>>>>> depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3
>>>>>>>>>>>>>>> (2))
>>>>>>>>>>>>>>> [0] 441 global equations, 147 vertices
>>>>>>>>>>>>>>> [0] 441 equations in vector, 147 vertices
>>>>>>>>>>>>>>> 0 SNES Function norm 49.7106
>>>>>>>>>>>>>>> 0 KSP Residual norm 49.7106
>>>>>>>>>>>>>>> 1 KSP Residual norm 12.9252
>>>>>>>>>>>>>>> 2 KSP Residual norm 2.38019
>>>>>>>>>>>>>>> 3 KSP Residual norm 0.426307
>>>>>>>>>>>>>>> 4 KSP Residual norm 0.0692155
>>>>>>>>>>>>>>> 5 KSP Residual norm 0.0123092
>>>>>>>>>>>>>>> 6 KSP Residual norm 0.00184874
>>>>>>>>>>>>>>> 7 KSP Residual norm 0.000320761
>>>>>>>>>>>>>>> 8 KSP Residual norm 5.48957e-05
>>>>>>>>>>>>>>> 9 KSP Residual norm 9.90089e-06
>>>>>>>>>>>>>>> 10 KSP Residual norm 1.5127e-06
>>>>>>>>>>>>>>> 11 KSP Residual norm 2.82192e-07
>>>>>>>>>>>>>>> 12 KSP Residual norm 4.62364e-08
>>>>>>>>>>>>>>> 13 KSP Residual norm 7.99573e-09
>>>>>>>>>>>>>>> 14 KSP Residual norm 1.3028e-09
>>>>>>>>>>>>>>> 15 KSP Residual norm 2.174e-10
>>>>>>>>>>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 15
>>>>>>>>>>>>>>> 1 SNES Function norm 2.174e-10
>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>>>>>>>>> iterations 1
>>>>>>>>>>>>>>> DM Object: Mesh (ex56_) 2 MPI processes
>>>>>>>>>>>>>>> type: plex
>>>>>>>>>>>>>>> Mesh in 3 dimensions:
>>>>>>>>>>>>>>> 0-cells: 45 45
>>>>>>>>>>>>>>> 1-cells: 96 96
>>>>>>>>>>>>>>> 2-cells: 68 68
>>>>>>>>>>>>>>> 3-cells: 16 16
>>>>>>>>>>>>>>> Labels:
>>>>>>>>>>>>>>> marker: 1 strata with value/size (1 (129))
>>>>>>>>>>>>>>> Face Sets: 5 strata with value/size (1 (18), 2 (18), 3
>>>>>>>>>>>>>>> (18), 5 (9), 6 (9))
>>>>>>>>>>>>>>> boundary: 1 strata with value/size (1 (141))
>>>>>>>>>>>>>>> depth: 4 strata with value/size (0 (45), 1 (96), 2 (68), 3
>>>>>>>>>>>>>>> (16))
>>>>>>>>>>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>>>>>>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>>>>>>>>>>> 0 SNES Function norm 17.9091
>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this
>>>>>>>>>>>>>>> object type
>>>>>>>>>>>>>>> [0]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> [1]PETSC ERROR: No support for this operation for this
>>>>>>>>>>>>>>> object type
>>>>>>>>>>>>>>> [1]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Nov 2, 2017 at 9:51 AM, Mark Adams <mfadams at lbl.gov>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 9:36 PM, Randy Michael Churchill <
>>>>>>>>>>>>>>>> rchurchi at pppl.gov> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Doing some additional testing, the issue goes away when
>>>>>>>>>>>>>>>>> removing the gamg preconditioner line from the petsc.rc:
>>>>>>>>>>>>>>>>> -pc_type gamg
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yea, this is GAMG setup.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the code. findices is create with ISCreateStride,
>>>>>>>>>>>>>>>> so it is sorted ...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Michael is repartitioning the coarse grids. Maybe we don't
>>>>>>>>>>>>>>>> have a regression test with this...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will try to reproduce this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Michael: you can use hypre for now, or turn repartitioning
>>>>>>>>>>>>>>>> off (eg, -fsa_fieldsplit_lambda_upper_pc_gamg_repartition
>>>>>>>>>>>>>>>> false), but I'm not sure this will fix this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You don't have hypre parameters for all of your all of your
>>>>>>>>>>>>>>>> solvers. I think 'boomeramg' is the default pc_hypre_type. That should be
>>>>>>>>>>>>>>>> good enough for you.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> IS findices;
>>>>>>>>>>>>>>>> PetscInt Istart,Iend;
>>>>>>>>>>>>>>>> Mat Pnew;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ierr = MatGetOwnershipRange(Pold, &Istart,
>>>>>>>>>>>>>>>> &Iend);CHKERRQ(ierr);
>>>>>>>>>>>>>>>> #if defined PETSC_GAMG_USE_LOG
>>>>>>>>>>>>>>>> ierr = PetscLogEventBegin(petsc_gamg_
>>>>>>>>>>>>>>>> setup_events[SET15],0,0,0,0);CHKERRQ(ierr);
>>>>>>>>>>>>>>>> #endif
>>>>>>>>>>>>>>>> ierr = ISCreateStride(comm,Iend-Istar
>>>>>>>>>>>>>>>> t,Istart,1,&findices);CHKERRQ(ierr);
>>>>>>>>>>>>>>>> ierr = ISSetBlockSize(findices,f_bs);CHKERRQ(ierr);
>>>>>>>>>>>>>>>> ierr = MatCreateSubMatrix(Pold, findices,
>>>>>>>>>>>>>>>> new_eq_indices, MAT_INITIAL_MATRIX, &Pnew);CHKERRQ(ierr);
>>>>>>>>>>>>>>>> ierr = ISDestroy(&findices);CHKERRQ(ierr);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> #if defined PETSC_GAMG_USE_LOG
>>>>>>>>>>>>>>>> ierr = PetscLogEventEnd(petsc_gamg_se
>>>>>>>>>>>>>>>> tup_events[SET15],0,0,0,0);CHKERRQ(ierr);
>>>>>>>>>>>>>>>> #endif
>>>>>>>>>>>>>>>> ierr = MatDestroy(a_P_inout);CHKERRQ(ierr);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /* output - repartitioned */
>>>>>>>>>>>>>>>> *a_P_inout = Pnew;
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 8:23 PM, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Randy:
>>>>>>>>>>>>>>>>>> Thanks, I'll check it tomorrow.
>>>>>>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> OK, this might not be completely satisfactory, because it
>>>>>>>>>>>>>>>>>>> doesn't show the partitioning or how the matrix is created, but this
>>>>>>>>>>>>>>>>>>> reproduces the problem. I wrote out my matrix, Amat, from the larger
>>>>>>>>>>>>>>>>>>> simulation, and load it in this script. This must be run with MPI rank
>>>>>>>>>>>>>>>>>>> greater than 1. This may be some combination of my petsc.rc, because when I
>>>>>>>>>>>>>>>>>>> use the PetscInitialize with it, it throws the error, but when using
>>>>>>>>>>>>>>>>>>> default (PETSC_NULL_CHARACTER) it runs fine.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Oct 31, 2017 at 9:58 AM, Hong <
>>>>>>>>>>>>>>>>>>> hzhang at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Randy:
>>>>>>>>>>>>>>>>>>>> It could be a bug or a missing feature in our new
>>>>>>>>>>>>>>>>>>>> MatCreateSubMatrix_MPIAIJ_SameRowDist().
>>>>>>>>>>>>>>>>>>>> It would be helpful if you can provide us a simple
>>>>>>>>>>>>>>>>>>>> example that produces this example.
>>>>>>>>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm running a Fortran code that was just changed over
>>>>>>>>>>>>>>>>>>>>> to using petsc 3.8 (previously petsc 3.7.6). An error was thrown during a
>>>>>>>>>>>>>>>>>>>>> KSPSetUp() call. The error is "unsorted iscol_local is not implemented yet"
>>>>>>>>>>>>>>>>>>>>> (see full error below). I tried to trace down the difference in the source
>>>>>>>>>>>>>>>>>>>>> files, but where the error occurs (MatCreateSubMatrix_MPIAIJ_SameRowDist())
>>>>>>>>>>>>>>>>>>>>> doesn't seem to have existed in v3.7.6, so I'm unsure how to compare. It
>>>>>>>>>>>>>>>>>>>>> seems the error is that the order of the columns locally are unsorted,
>>>>>>>>>>>>>>>>>>>>> though I don't think I specify a column order in the creation of the matrix:
>>>>>>>>>>>>>>>>>>>>> call MatCreate(this%comm,AA,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatSetSizes(AA,npetscloc,npets
>>>>>>>>>>>>>>>>>>>>> cloc,nreal,nreal,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatSetType(AA,MATAIJ,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatSetup(AA,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatGetOwnershipRange(AA,low,high,ierr)
>>>>>>>>>>>>>>>>>>>>> allocate(d_nnz(npetscloc),o_nnz(npetscloc))
>>>>>>>>>>>>>>>>>>>>> call getNNZ(grid,npetscloc,low,high
>>>>>>>>>>>>>>>>>>>>> ,d_nnz,o_nnz,this%xgc_petsc,nreal,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatSeqAIJSetPreallocation(AA,P
>>>>>>>>>>>>>>>>>>>>> ETSC_NULL_INTEGER,d_nnz,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatMPIAIJSetPreallocation(AA,P
>>>>>>>>>>>>>>>>>>>>> ETSC_NULL_INTEGER,d_nnz,PETSC_NULL_INTEGER,o_nnz,ierr)
>>>>>>>>>>>>>>>>>>>>> deallocate(d_nnz,o_nnz)
>>>>>>>>>>>>>>>>>>>>> call MatSetOption(AA,MAT_IGNORE_OFF
>>>>>>>>>>>>>>>>>>>>> _PROC_ENTRIES,PETSC_TRUE,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatSetOption(AA,MAT_KEEP_NONZE
>>>>>>>>>>>>>>>>>>>>> RO_PATTERN,PETSC_TRUE,ierr)
>>>>>>>>>>>>>>>>>>>>> call MatSetup(AA,ierr)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>> --------------------------------
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: No support for this operation for
>>>>>>>>>>>>>>>>>>>>> this object type
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: unsorted iscol_local is not
>>>>>>>>>>>>>>>>>>>>> implemented yet
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>>>>>>>>>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: Petsc Release Version 3.8.0,
>>>>>>>>>>>>>>>>>>>>> unknown[62]PETSC ERROR: #1 MatCreateSubMatrix_MPIAIJ_SameRowDist()
>>>>>>>>>>>>>>>>>>>>> line 3418 in /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>>>>>>> 8.0/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: #2 MatCreateSubMatrix_MPIAIJ() line
>>>>>>>>>>>>>>>>>>>>> 3247 in /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>>>>>>> 8.0/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: #3 MatCreateSubMatrix() line 7872 in
>>>>>>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/mat/interface/matrix.c
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: #4 PCGAMGCreateLevel_GAMG() line 383
>>>>>>>>>>>>>>>>>>>>> in /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>>>>>>> 8.0/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: #5 PCSetUp_GAMG() line 561 in
>>>>>>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>>>>>>> 8.0/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: #6 PCSetUp() line 924 in
>>>>>>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>>>>>>> 8.0/src/ksp/pc/interface/precon.c
>>>>>>>>>>>>>>>>>>>>> [62]PETSC ERROR: #7 KSPSetUp() line 378 in
>>>>>>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>>>>>>> 8.0/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171108/b13d560a/attachment-0001.html>
More information about the petsc-users
mailing list