[petsc-dev] memory leak in fieldsplit

Sat Dec 1 06:19:40 CST 2012

On Sat, Dec 1, 2012 at 12:47 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Did someone fix this?. I can't reproduce.

I think Barry did. kspupper was not being destroyed on restart.

   Matt

> On Fri, Nov 23, 2012 at 11:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>   The nightly builds
>>
>> Possible problem with with ex19_fieldsplit_4, diffs above
>> =========================================
>>
>> has an apparent memory leak.  Output below. Thus must be some peculiar to
>> a particular usage of field split. Anybody take responsibility?
>>
>> > [0]Total space allocated 12816 bytes
>> > [ 0]16 bytes PetscStrallocpy() line 187 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>> >       [0]  PCCreate_LU() line 283 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c
>> >       [0]  PCSetType() line 59 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  MatSetFromOptions_SchurComplement() line 123 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>> >       [0]  MatSetFromOptions() line 174 in
>> > /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>> >       [0]  PCSetUp_FieldSplit() line 405 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetUp() line 810 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPSetUp() line 182 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSolve() line 350 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  SNES_KSPSolve() line 4434 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscStrallocpy() line 187 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>> >       [0]  PCCreate_LU() line 283 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c
>> >       [0]  PCSetType() line 59 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  MatSetFromOptions_SchurComplement() line 123 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>> >       [0]  MatSetFromOptions() line 174 in
>> > /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>> >       [0]  PCSetUp_FieldSplit() line 405 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetUp() line 810 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPSetUp() line 182 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSolve() line 350 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  SNES_KSPSolve() line 4434 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]176 bytes PCCreate_LU() line 284 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c
>> >       [0]  PCSetType() line 59 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  MatSetFromOptions_SchurComplement() line 123 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>> >       [0]  MatSetFromOptions() line 174 in
>> > /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>> >       [0]  PCSetUp_FieldSplit() line 405 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetUp() line 810 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPSetUp() line 182 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSolve() line 350 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  SNES_KSPSolve() line 4434 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscStrallocpy() line 187 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>> >       [0]  PetscObjectChangeTypeName() line 129 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pname.c
>> >       [0]  PCSetType() line 59 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  MatSetFromOptions_SchurComplement() line 123 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>> >       [0]  MatSetFromOptions() line 174 in
>> > /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>> >       [0]  PCSetUp_FieldSplit() line 405 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetUp() line 810 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPSetUp() line 182 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSolve() line 350 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  SNES_KSPSolve() line 4434 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]128 bytes PetscSplitReductionCreate() line 99 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  PetscSplitReductionGet() line 416 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  VecNormBegin() line 622 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]256 bytes PetscSplitReductionCreate() line 96 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  PetscSplitReductionGet() line 416 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  VecNormBegin() line 622 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]512 bytes PetscSplitReductionCreate() line 95 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  PetscSplitReductionGet() line 416 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  VecNormBegin() line 622 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]512 bytes PetscSplitReductionCreate() line 94 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  PetscSplitReductionGet() line 416 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  VecNormBegin() line 622 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]64 bytes PetscSplitReductionCreate() line 89 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  PetscSplitReductionGet() line 416 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  VecNormBegin() line 622 in
>> > /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>> >       [0]  SNESSolve_LS() line 144 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>> >       [0]  SNESSolve() line 3605 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscStrallocpy() line 187 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>> >       [0]  PetscObjectSetOptionsPrefix() line 29 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/prefix.c
>> >       [0]  KSPSetOptionsPrefix() line 49 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscStrallocpy() line 187 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>> >       [0]  PetscObjectSetOptionsPrefix() line 29 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/prefix.c
>> >       [0]  PCSetOptionsPrefix() line 1267 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPSetOptionsPrefix() line 49 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]144 bytes PCCreate() line 328 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPGetPC() line 1448 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSetOptionsPrefix() line 49 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]64 bytes PCCreate() line 328 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPGetPC() line 1448 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSetOptionsPrefix() line 49 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]576 bytes PCCreate() line 328 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>> >       [0]  KSPGetPC() line 1448 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> >       [0]  KSPSetOptionsPrefix() line 49 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscStrallocpy() line 187 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>> >       [0]  PetscObjectChangeTypeName() line 129 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pname.c
>> >       [0]  KSPSetType() line 722 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes KSPDefaultConvergedCreate() line 569 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/iterativ.c
>> >       [0]  KSPCreate() line 628 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]96 bytes KSPCreate() line 635 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]64 bytes KSPCreate() line 635 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]944 bytes KSPCreate() line 635 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>> >       [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetFields() line 1232 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions_FieldSplit() line 998 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> >       [0]  PCSetFromOptions() line 175 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>> >       [0]  KSPSetFromOptions() line 347 in
>> > /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>> >       [0]  SNESSetFromOptions() line 612 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscCommDuplicate() line 154 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/tagm.c
>> >       [0]  PetscHeaderCreate_Private() line 30 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/inherit.c
>> >       [0]  SNESCreate() line 1396 in
>> > /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>> > [ 0]16 bytes PetscThreadCommReductionCreate() line 466 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>> >       [0]  PetscThreadCommInitialize() line 1223 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>> >       [0]  PetscThreadCommInitializePackage() line 51 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>> >       [0]  PetscInitialize() line 651 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>> > [ 0]16 bytes PetscThreadCommReductionCreate() line 462 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>> >       [0]  PetscThreadCommInitialize() line 1223 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>> >       [0]  PetscThreadCommInitializePackage() line 51 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>> >       [0]  PetscInitialize() line 651 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>> > [ 0]16 bytes PetscThreadCommReductionCreate() line 458 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>> >       [0]  PetscThreadCommInitialize() line 1223 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>> >       [0]  PetscThreadCommInitializePackage() line 51 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>> >       [0]  PetscInitialize() line 651 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>> > [ 0]48 bytes PetscThreadCommReductionCreate() line 456 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>> >       [0]  PetscThreadCommInitialize() line 1223 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>> >       [0]  PetscThreadCommInitializePackage() line 51 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>> >       [0]  PetscInitialize() line 651 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>> > [ 0]16 bytes PetscThreadCommReductionCreate() line 462 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>> >       [0]  PetscThreadCommInitialize() line 1223 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>> >       [0]  PetscThreadCommInitializePackage() line 51 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>> >       [0]  PetscInitialize() line 651 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>> > [ 0]16 bytes PetscThreadCommReductionCreate() line 458 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>> >       [0]  PetscThreadCommInitialize() line 1223 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>> >       [0]  PetscThreadCommInitializePackage() line 51 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>> >       [0]  PetscInitialize() line 651 in
>> > /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>
>

--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener