[petsc-dev] memory leak in fieldsplit

Sat Dec 1 08:18:40 CST 2012

On Dec 1, 2012, at 6:19 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Dec 1, 2012 at 12:47 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> Did someone fix this?. I can't reproduce.
> 
> I think Barry did. kspupper was not being destroyed on restart.

   Yes I did  (fix was partially hidden within some other changes maybe)

> 
>   Matt
> 
>> On Fri, Nov 23, 2012 at 11:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>  The nightly builds
>>> 
>>> Possible problem with with ex19_fieldsplit_4, diffs above
>>> =========================================
>>> 
>>> has an apparent memory leak.  Output below. Thus must be some peculiar to
>>> a particular usage of field split. Anybody take responsibility?
>>> 
>>>> [0]Total space allocated 12816 bytes
>>>> [ 0]16 bytes PetscStrallocpy() line 187 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>>>>      [0]  PCCreate_LU() line 283 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c
>>>>      [0]  PCSetType() line 59 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  MatSetFromOptions_SchurComplement() line 123 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>>>>      [0]  MatSetFromOptions() line 174 in
>>>> /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>>>>      [0]  PCSetUp_FieldSplit() line 405 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetUp() line 810 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPSetUp() line 182 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSolve() line 350 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  SNES_KSPSolve() line 4434 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscStrallocpy() line 187 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>>>>      [0]  PCCreate_LU() line 283 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c
>>>>      [0]  PCSetType() line 59 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  MatSetFromOptions_SchurComplement() line 123 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>>>>      [0]  MatSetFromOptions() line 174 in
>>>> /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>>>>      [0]  PCSetUp_FieldSplit() line 405 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetUp() line 810 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPSetUp() line 182 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSolve() line 350 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  SNES_KSPSolve() line 4434 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]176 bytes PCCreate_LU() line 284 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c
>>>>      [0]  PCSetType() line 59 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  MatSetFromOptions_SchurComplement() line 123 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>>>>      [0]  MatSetFromOptions() line 174 in
>>>> /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>>>>      [0]  PCSetUp_FieldSplit() line 405 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetUp() line 810 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPSetUp() line 182 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSolve() line 350 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  SNES_KSPSolve() line 4434 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscStrallocpy() line 187 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>>>>      [0]  PetscObjectChangeTypeName() line 129 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pname.c
>>>>      [0]  PCSetType() line 59 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  MatSetFromOptions_SchurComplement() line 123 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/utils/schurm.c
>>>>      [0]  MatSetFromOptions() line 174 in
>>>> /Users/barrysmith/Src/petsc-dev/src/mat/utils/gcreate.c
>>>>      [0]  PCSetUp_FieldSplit() line 405 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetUp() line 810 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPSetUp() line 182 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSolve() line 350 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  SNES_KSPSolve() line 4434 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]128 bytes PetscSplitReductionCreate() line 99 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  PetscSplitReductionGet() line 416 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  VecNormBegin() line 622 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]256 bytes PetscSplitReductionCreate() line 96 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  PetscSplitReductionGet() line 416 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  VecNormBegin() line 622 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]512 bytes PetscSplitReductionCreate() line 95 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  PetscSplitReductionGet() line 416 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  VecNormBegin() line 622 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]512 bytes PetscSplitReductionCreate() line 94 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  PetscSplitReductionGet() line 416 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  VecNormBegin() line 622 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]64 bytes PetscSplitReductionCreate() line 89 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  PetscSplitReductionGet() line 416 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  VecNormBegin() line 622 in
>>>> /Users/barrysmith/Src/petsc-dev/src/vec/vec/utils/comb.c
>>>>      [0]  SNESSolve_LS() line 144 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c
>>>>      [0]  SNESSolve() line 3605 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscStrallocpy() line 187 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>>>>      [0]  PetscObjectSetOptionsPrefix() line 29 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/prefix.c
>>>>      [0]  KSPSetOptionsPrefix() line 49 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscStrallocpy() line 187 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>>>>      [0]  PetscObjectSetOptionsPrefix() line 29 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/prefix.c
>>>>      [0]  PCSetOptionsPrefix() line 1267 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPSetOptionsPrefix() line 49 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]144 bytes PCCreate() line 328 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPGetPC() line 1448 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSetOptionsPrefix() line 49 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]64 bytes PCCreate() line 328 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPGetPC() line 1448 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSetOptionsPrefix() line 49 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]576 bytes PCCreate() line 328 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c
>>>>      [0]  KSPGetPC() line 1448 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>>      [0]  KSPSetOptionsPrefix() line 49 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscStrallocpy() line 187 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/utils/str.c
>>>>      [0]  PetscObjectChangeTypeName() line 129 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pname.c
>>>>      [0]  KSPSetType() line 722 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes KSPDefaultConvergedCreate() line 569 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/iterativ.c
>>>>      [0]  KSPCreate() line 628 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]96 bytes KSPCreate() line 635 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]64 bytes KSPCreate() line 635 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]944 bytes KSPCreate() line 635 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcreate.c
>>>>      [0]  PCFieldSplitSetFields_FieldSplit() line 1059 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetFields() line 1232 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCFieldSplitSetRuntimeSplits_Private() line 213 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions_FieldSplit() line 998 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>>>      [0]  PCSetFromOptions() line 175 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/pcset.c
>>>>      [0]  KSPSetFromOptions() line 347 in
>>>> /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itcl.c
>>>>      [0]  SNESSetFromOptions() line 612 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscCommDuplicate() line 154 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/tagm.c
>>>>      [0]  PetscHeaderCreate_Private() line 30 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/inherit.c
>>>>      [0]  SNESCreate() line 1396 in
>>>> /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c
>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 466 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>>>>      [0]  PetscThreadCommInitialize() line 1223 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>>>>      [0]  PetscThreadCommInitializePackage() line 51 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>>>>      [0]  PetscInitialize() line 651 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 462 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>>>>      [0]  PetscThreadCommInitialize() line 1223 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>>>>      [0]  PetscThreadCommInitializePackage() line 51 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>>>>      [0]  PetscInitialize() line 651 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 458 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>>>>      [0]  PetscThreadCommInitialize() line 1223 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>>>>      [0]  PetscThreadCommInitializePackage() line 51 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>>>>      [0]  PetscInitialize() line 651 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>>> [ 0]48 bytes PetscThreadCommReductionCreate() line 456 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>>>>      [0]  PetscThreadCommInitialize() line 1223 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>>>>      [0]  PetscThreadCommInitializePackage() line 51 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>>>>      [0]  PetscInitialize() line 651 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 462 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>>>>      [0]  PetscThreadCommInitialize() line 1223 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>>>>      [0]  PetscThreadCommInitializePackage() line 51 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>>>>      [0]  PetscInitialize() line 651 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 458 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c
>>>>      [0]  PetscThreadCommInitialize() line 1223 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c
>>>>      [0]  PetscThreadCommInitializePackage() line 51 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c
>>>>      [0]  PetscInitialize() line 651 in
>>>> /Users/barrysmith/Src/petsc-dev/src/sys/objects/pinit.c
>>> 
>> 
> 
> 
> 
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener