[petsc-users] unsorted local columns in 3.8?
Mark Adams
mfadams at lbl.gov
Tue Nov 7 19:37:01 CST 2017
Humm, this looks a little odd, but it may be OK. Is this this diffing with
the old non-repartition data? (more below)
On Tue, Nov 7, 2017 at 11:45 AM, Hong <hzhang at mcs.anl.gov> wrote:
> Mark,
> The fix is merged to next branch for tests which show diff as
>
> ******* Testing: testexamples_PARMETIS *******
> 5c5
> < 1 SNES Function norm 1.983e-10
> ---
> > 1 SNES Function norm 1.990e-10
> 10,13c10,13
> < 0-cells: 8 8 8 8
> < 1-cells: 12 12 12 12
> < 2-cells: 6 6 6 6
> < 3-cells: 1 1 1 1
>
>
I assume this is the old.
> ---
> > 0-cells: 12 12 0 0
> > 1-cells: 20 20 0 0
> > 2-cells: 11 11 0 0
> > 3-cells: 2 2 0 0
> 15,18c15,18
>
>
and this is the new.
This is funny because the processors are not fully populated. This can
happen on coarse grids and indeed it should happen in a test with good
coverage.
I assume these diffs are views from coarse grids? That is, in the raw
output files do you see fully populated fine grids, with no diffs, and then
the diffs come on coarse grids.
Repartitioning the coarse grids can change the coarsening, It is possible
that repartitioning causes faster coarsening (it does a little) and this
faster coarsening is tripping the aggregation switch, which gives us empty
processors.
Am I understanding this correctly ...
Thanks,
Mark
> < boundary: 1 strata with value/size (1 (23))
> < Face Sets: 4 strata with value/size (1 (1), 2 (1), 4 (1), 6 (1))
> < marker: 1 strata with value/size (1 (15))
> < depth: 4 strata with value/size (0 (8), 1 (12), 2 (6), 3 (1))
> ---
> > boundary: 1 strata with value/size (1 (39))
> > Face Sets: 5 strata with value/size (1 (2), 2 (2), 4 (2), 5 (1), 6 (1))
> > marker: 1 strata with value/size (1 (27))
> > depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2))
>
> see http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/07/examples_full_next-tmp.log
>
> I guess parmetis produces random partition on different machines (I made output file for ex56_1 on my imac). Please take a look at the differences. If the outputs are correct, I will remove option '-ex56_dm_view'
>
> Hong
>
>
> On Sun, Nov 5, 2017 at 9:03 PM, Hong <hzhang at mcs.anl.gov> wrote:
>
>> Mark:
>> Bug is fixed in branch hzhang/fix-submat_samerowdist
>> https://bitbucket.org/petsc/petsc/branch/hzhang/fix-submat_samerowdist
>>
>> I also add the test runex56. Please test it and let me know if there is a
>> problem.
>> Hong
>>
>> Also, I have been using -petscpartition_type but now I see
>>> -pc_gamg_mat_partitioning_type. Is -petscpartition_type depreciated for
>>> GAMG?
>>>
>>> Is this some sort of auto generated portmanteau? I can not find
>>> pc_gamg_mat_partitioning_type in the source.
>>>
>>> On Thu, Nov 2, 2017 at 6:44 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> Great, thanks,
>>>>
>>>> And could you please add these parameters to a regression test? As I
>>>> recall we have with-parmetis regression test.
>>>>
>>>> On Thu, Nov 2, 2017 at 6:35 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>
>>>>> Mark:
>>>>> I used petsc/src/ksp/ksp/examples/tutorials/ex56.c :-(
>>>>> Now testing src/snes/examples/tutorials/ex56.c with your options, I
>>>>> can reproduce the error.
>>>>> I'll fix it.
>>>>>
>>>>> Hong
>>>>>
>>>>> Hong,
>>>>>>
>>>>>> I've tested with master and I get the same error. Maybe the
>>>>>> partitioning parameters are wrong. -pc_gamg_mat_partitioning_type is new to
>>>>>> me.
>>>>>>
>>>>>> Can you run this (snes ex56) w/o the error?
>>>>>>
>>>>>>
>>>>>> 17:33 *master* *= ~*/Codes/petsc/src/snes/examples/tutorials*$ make
>>>>>> runex
>>>>>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpiexec -n 4
>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2
>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason
>>>>>> -snes_monitor_short -ksp_monitor_short -snes_converged_reason
>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -matrap 0
>>>>>> -matptap_scalable -ex56_dm_view -run_type 1 -pc_gamg_repartition true
>>>>>> [0] 27 global equations, 9 vertices
>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>> 0 SNES Function norm 122.396
>>>>>> 0 KSP Residual norm 122.396
>>>>>> <snip>
>>>>>> depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))
>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>> 0 SNES Function norm 17.9091
>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>> --------------------------------------------------------------
>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>> [0]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>>>>>> ocumentation/faq.html for trouble shooting.
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 2, 2017 at 1:35 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Mark :
>>>>>>> I realize that using maint or master branch, I cannot reproduce the
>>>>>>> same error.
>>>>>>> For this example, you must use a parallel partitioner,
>>>>>>> e.g.,'current' gives me following error:
>>>>>>> [0]PETSC ERROR: This is the DEFAULT NO-OP partitioner, it currently
>>>>>>> only supports one domain per processor
>>>>>>> use -pc_gamg_mat_partitioning_type parmetis or chaco or ptscotch for
>>>>>>> more than one subdomain per processor
>>>>>>>
>>>>>>> Please rebase your branch with maint or master, then see if you
>>>>>>> still have problem.
>>>>>>>
>>>>>>> Hong
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 2, 2017 at 11:07 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Mark,
>>>>>>>>> I can reproduce this in an old branch, but not in current maint
>>>>>>>>> and master.
>>>>>>>>> Which branch are you using to produce this error?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I am using a branch from Matt. Let me try to merge it with master.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 2, 2017 at 9:28 AM, Mark Adams <mfadams at lbl.gov>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I am able to reproduce this with snes ex56 with 2 processors and
>>>>>>>>>> adding -pc_gamg_repartition true
>>>>>>>>>>
>>>>>>>>>> I'm not sure how to fix it.
>>>>>>>>>>
>>>>>>>>>> 10:26 1 knepley/feature-plex-boxmesh-create *=
>>>>>>>>>> ~/Codes/petsc/src/snes/examples/tutorials$ make
>>>>>>>>>> PETSC_DIR=/Users/markadams/Codes/petsc
>>>>>>>>>> PETSC_ARCH=arch-macosx-gnu-g runex
>>>>>>>>>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpiexec -n 2
>>>>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2
>>>>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason
>>>>>>>>>> -snes_monitor_short -ksp_monitor_short -snes_converged_reason
>>>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>>>>>> jacobi -petscpartitioner_type simple -mat_block_size 3 -matrap 0
>>>>>>>>>> -matptap_scalable -ex56_dm_view -run_type 1 -pc_gamg_repartition true
>>>>>>>>>> [0] 27 global equations, 9 vertices
>>>>>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>>>>>> 0 SNES Function norm 122.396
>>>>>>>>>> 0 KSP Residual norm 122.396
>>>>>>>>>> 1 KSP Residual norm 20.4696
>>>>>>>>>> 2 KSP Residual norm 3.95009
>>>>>>>>>> 3 KSP Residual norm 0.176181
>>>>>>>>>> 4 KSP Residual norm 0.0208781
>>>>>>>>>> 5 KSP Residual norm 0.00278873
>>>>>>>>>> 6 KSP Residual norm 0.000482741
>>>>>>>>>> 7 KSP Residual norm 4.68085e-05
>>>>>>>>>> 8 KSP Residual norm 5.42381e-06
>>>>>>>>>> 9 KSP Residual norm 5.12785e-07
>>>>>>>>>> 10 KSP Residual norm 2.60389e-08
>>>>>>>>>> 11 KSP Residual norm 4.96201e-09
>>>>>>>>>> 12 KSP Residual norm 1.989e-10
>>>>>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 12
>>>>>>>>>> 1 SNES Function norm 1.990e-10
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>>>> iterations 1
>>>>>>>>>> DM Object: Mesh (ex56_) 2 MPI processes
>>>>>>>>>> type: plex
>>>>>>>>>> Mesh in 3 dimensions:
>>>>>>>>>> 0-cells: 12 12
>>>>>>>>>> 1-cells: 20 20
>>>>>>>>>> 2-cells: 11 11
>>>>>>>>>> 3-cells: 2 2
>>>>>>>>>> Labels:
>>>>>>>>>> boundary: 1 strata with value/size (1 (39))
>>>>>>>>>> Face Sets: 5 strata with value/size (1 (2), 2 (2), 3 (2), 5
>>>>>>>>>> (1), 6 (1))
>>>>>>>>>> marker: 1 strata with value/size (1 (27))
>>>>>>>>>> depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2))
>>>>>>>>>> [0] 441 global equations, 147 vertices
>>>>>>>>>> [0] 441 equations in vector, 147 vertices
>>>>>>>>>> 0 SNES Function norm 49.7106
>>>>>>>>>> 0 KSP Residual norm 49.7106
>>>>>>>>>> 1 KSP Residual norm 12.9252
>>>>>>>>>> 2 KSP Residual norm 2.38019
>>>>>>>>>> 3 KSP Residual norm 0.426307
>>>>>>>>>> 4 KSP Residual norm 0.0692155
>>>>>>>>>> 5 KSP Residual norm 0.0123092
>>>>>>>>>> 6 KSP Residual norm 0.00184874
>>>>>>>>>> 7 KSP Residual norm 0.000320761
>>>>>>>>>> 8 KSP Residual norm 5.48957e-05
>>>>>>>>>> 9 KSP Residual norm 9.90089e-06
>>>>>>>>>> 10 KSP Residual norm 1.5127e-06
>>>>>>>>>> 11 KSP Residual norm 2.82192e-07
>>>>>>>>>> 12 KSP Residual norm 4.62364e-08
>>>>>>>>>> 13 KSP Residual norm 7.99573e-09
>>>>>>>>>> 14 KSP Residual norm 1.3028e-09
>>>>>>>>>> 15 KSP Residual norm 2.174e-10
>>>>>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 15
>>>>>>>>>> 1 SNES Function norm 2.174e-10
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>>>> iterations 1
>>>>>>>>>> DM Object: Mesh (ex56_) 2 MPI processes
>>>>>>>>>> type: plex
>>>>>>>>>> Mesh in 3 dimensions:
>>>>>>>>>> 0-cells: 45 45
>>>>>>>>>> 1-cells: 96 96
>>>>>>>>>> 2-cells: 68 68
>>>>>>>>>> 3-cells: 16 16
>>>>>>>>>> Labels:
>>>>>>>>>> marker: 1 strata with value/size (1 (129))
>>>>>>>>>> Face Sets: 5 strata with value/size (1 (18), 2 (18), 3 (18), 5
>>>>>>>>>> (9), 6 (9))
>>>>>>>>>> boundary: 1 strata with value/size (1 (141))
>>>>>>>>>> depth: 4 strata with value/size (0 (45), 1 (96), 2 (68), 3 (16))
>>>>>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>>>>>> 0 SNES Function norm 17.9091
>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>> [0]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message
>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>> [1]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>> [1]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 2, 2017 at 9:51 AM, Mark Adams <mfadams at lbl.gov>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 1, 2017 at 9:36 PM, Randy Michael Churchill <
>>>>>>>>>>> rchurchi at pppl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Doing some additional testing, the issue goes away when
>>>>>>>>>>>> removing the gamg preconditioner line from the petsc.rc:
>>>>>>>>>>>> -pc_type gamg
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yea, this is GAMG setup.
>>>>>>>>>>>
>>>>>>>>>>> This is the code. findices is create with ISCreateStride, so it
>>>>>>>>>>> is sorted ...
>>>>>>>>>>>
>>>>>>>>>>> Michael is repartitioning the coarse grids. Maybe we don't have
>>>>>>>>>>> a regression test with this...
>>>>>>>>>>>
>>>>>>>>>>> I will try to reproduce this.
>>>>>>>>>>>
>>>>>>>>>>> Michael: you can use hypre for now, or turn repartitioning off
>>>>>>>>>>> (eg, -fsa_fieldsplit_lambda_upper_pc_gamg_repartition false),
>>>>>>>>>>> but I'm not sure this will fix this.
>>>>>>>>>>>
>>>>>>>>>>> You don't have hypre parameters for all of your all of your
>>>>>>>>>>> solvers. I think 'boomeramg' is the default pc_hypre_type. That should be
>>>>>>>>>>> good enough for you.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> {
>>>>>>>>>>> IS findices;
>>>>>>>>>>> PetscInt Istart,Iend;
>>>>>>>>>>> Mat Pnew;
>>>>>>>>>>>
>>>>>>>>>>> ierr = MatGetOwnershipRange(Pold, &Istart,
>>>>>>>>>>> &Iend);CHKERRQ(ierr);
>>>>>>>>>>> #if defined PETSC_GAMG_USE_LOG
>>>>>>>>>>> ierr = PetscLogEventBegin(petsc_gamg_
>>>>>>>>>>> setup_events[SET15],0,0,0,0);CHKERRQ(ierr);
>>>>>>>>>>> #endif
>>>>>>>>>>> ierr = ISCreateStride(comm,Iend-Istar
>>>>>>>>>>> t,Istart,1,&findices);CHKERRQ(ierr);
>>>>>>>>>>> ierr = ISSetBlockSize(findices,f_bs);CHKERRQ(ierr);
>>>>>>>>>>> ierr = MatCreateSubMatrix(Pold, findices, new_eq_indices,
>>>>>>>>>>> MAT_INITIAL_MATRIX, &Pnew);CHKERRQ(ierr);
>>>>>>>>>>> ierr = ISDestroy(&findices);CHKERRQ(ierr);
>>>>>>>>>>>
>>>>>>>>>>> #if defined PETSC_GAMG_USE_LOG
>>>>>>>>>>> ierr = PetscLogEventEnd(petsc_gamg_se
>>>>>>>>>>> tup_events[SET15],0,0,0,0);CHKERRQ(ierr);
>>>>>>>>>>> #endif
>>>>>>>>>>> ierr = MatDestroy(a_P_inout);CHKERRQ(ierr);
>>>>>>>>>>>
>>>>>>>>>>> /* output - repartitioned */
>>>>>>>>>>> *a_P_inout = Pnew;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 1, 2017 at 8:23 PM, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Randy:
>>>>>>>>>>>>> Thanks, I'll check it tomorrow.
>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>
>>>>>>>>>>>>> OK, this might not be completely satisfactory, because it
>>>>>>>>>>>>>> doesn't show the partitioning or how the matrix is created, but this
>>>>>>>>>>>>>> reproduces the problem. I wrote out my matrix, Amat, from the larger
>>>>>>>>>>>>>> simulation, and load it in this script. This must be run with MPI rank
>>>>>>>>>>>>>> greater than 1. This may be some combination of my petsc.rc, because when I
>>>>>>>>>>>>>> use the PetscInitialize with it, it throws the error, but when using
>>>>>>>>>>>>>> default (PETSC_NULL_CHARACTER) it runs fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Oct 31, 2017 at 9:58 AM, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Randy:
>>>>>>>>>>>>>>> It could be a bug or a missing feature in our new
>>>>>>>>>>>>>>> MatCreateSubMatrix_MPIAIJ_SameRowDist().
>>>>>>>>>>>>>>> It would be helpful if you can provide us a simple example
>>>>>>>>>>>>>>> that produces this example.
>>>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm running a Fortran code that was just changed over to
>>>>>>>>>>>>>>>> using petsc 3.8 (previously petsc 3.7.6). An error was thrown during a
>>>>>>>>>>>>>>>> KSPSetUp() call. The error is "unsorted iscol_local is not implemented yet"
>>>>>>>>>>>>>>>> (see full error below). I tried to trace down the difference in the source
>>>>>>>>>>>>>>>> files, but where the error occurs (MatCreateSubMatrix_MPIAIJ_SameRowDist())
>>>>>>>>>>>>>>>> doesn't seem to have existed in v3.7.6, so I'm unsure how to compare. It
>>>>>>>>>>>>>>>> seems the error is that the order of the columns locally are unsorted,
>>>>>>>>>>>>>>>> though I don't think I specify a column order in the creation of the matrix:
>>>>>>>>>>>>>>>> call MatCreate(this%comm,AA,ierr)
>>>>>>>>>>>>>>>> call MatSetSizes(AA,npetscloc,npets
>>>>>>>>>>>>>>>> cloc,nreal,nreal,ierr)
>>>>>>>>>>>>>>>> call MatSetType(AA,MATAIJ,ierr)
>>>>>>>>>>>>>>>> call MatSetup(AA,ierr)
>>>>>>>>>>>>>>>> call MatGetOwnershipRange(AA,low,high,ierr)
>>>>>>>>>>>>>>>> allocate(d_nnz(npetscloc),o_nnz(npetscloc))
>>>>>>>>>>>>>>>> call getNNZ(grid,npetscloc,low,high
>>>>>>>>>>>>>>>> ,d_nnz,o_nnz,this%xgc_petsc,nreal,ierr)
>>>>>>>>>>>>>>>> call MatSeqAIJSetPreallocation(AA,P
>>>>>>>>>>>>>>>> ETSC_NULL_INTEGER,d_nnz,ierr)
>>>>>>>>>>>>>>>> call MatMPIAIJSetPreallocation(AA,P
>>>>>>>>>>>>>>>> ETSC_NULL_INTEGER,d_nnz,PETSC_NULL_INTEGER,o_nnz,ierr)
>>>>>>>>>>>>>>>> deallocate(d_nnz,o_nnz)
>>>>>>>>>>>>>>>> call MatSetOption(AA,MAT_IGNORE_OFF
>>>>>>>>>>>>>>>> _PROC_ENTRIES,PETSC_TRUE,ierr)
>>>>>>>>>>>>>>>> call MatSetOption(AA,MAT_KEEP_NONZE
>>>>>>>>>>>>>>>> RO_PATTERN,PETSC_TRUE,ierr)
>>>>>>>>>>>>>>>> call MatSetup(AA,ierr)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [62]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>> --------------------------------
>>>>>>>>>>>>>>>> [62]PETSC ERROR: No support for this operation for this
>>>>>>>>>>>>>>>> object type
>>>>>>>>>>>>>>>> [62]PETSC ERROR: unsorted iscol_local is not implemented yet
>>>>>>>>>>>>>>>> [62]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>>>>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>> [62]PETSC ERROR: Petsc Release Version 3.8.0,
>>>>>>>>>>>>>>>> unknown[62]PETSC ERROR: #1 MatCreateSubMatrix_MPIAIJ_SameRowDist()
>>>>>>>>>>>>>>>> line 3418 in /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>> 8.0/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>>>>>>>>> [62]PETSC ERROR: #2 MatCreateSubMatrix_MPIAIJ() line 3247
>>>>>>>>>>>>>>>> in /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>> 8.0/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>>>>>>>>> [62]PETSC ERROR: #3 MatCreateSubMatrix() line 7872 in
>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.8.0/src/mat/interface/matrix.c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [62]PETSC ERROR: #4 PCGAMGCreateLevel_GAMG() line 383 in
>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>> 8.0/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>>> [62]PETSC ERROR: #5 PCSetUp_GAMG() line 561 in
>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>> 8.0/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>>> [62]PETSC ERROR: #6 PCSetUp() line 924 in
>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>> 8.0/src/ksp/pc/interface/precon.c
>>>>>>>>>>>>>>>> [62]PETSC ERROR: #7 KSPSetUp() line 378 in
>>>>>>>>>>>>>>>> /global/u1/r/rchurchi/petsc/3.
>>>>>>>>>>>>>>>> 8.0/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> R. Michael Churchill
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171107/ddca8a81/attachment-0001.html>
More information about the petsc-users
mailing list