[Petsc-trilinos-discussion] Scope and requirements

[Bartlett, Roscoe A.]

> > In Trilinos, the basic concept is that all objects should print to an
> > arbitrary Teuchos::FancyOStream object (which takes any arbitrary
> > std::ostream object).  The underlying std::ostream object has an
> > abstract stream buffer object that can be overridden to send output
> > anywhere.  The class Teuchos::FancyOStream contains some utilities to
> > improve the formatting of the output (like adding indentation in a
> > nice way, adding prefixes for the procID, etc.) and has other little
> > features (but has no real dependence on MPI or anything else).
> 
> Where does it get the procID without MPI?  How does it collate output
> From multiple ranks without MPI?

[Bartlett, Roscoe A.] 

By default it gets the procID it from Teuchos::GlobalMPISession::getRank() which gets it from MPI_rank(MPI_COMM_WORLD ...) by but you can just set it manually to override that.  That is the only indirect dependence on MPI and that class returns the obvious values when MPI is not enabled.  That is the only MPI dependence on this but that takes care of 95%  or more use cases.  CASL VERA Tiamat already has FancyOStreams that override the underlying ostreams and the procIDs.  Currently MPI is not used for output coordination (but we could and we might do that for CASL VERA Tiamat).

>From the detailed responses below, it sounds like PETSc should have the facilities needed to redirect output as would be needed.  The challenge would then be to get all the APP codes to allow us to drill down through the to PETSc objects.  That is still likely a good bit of work.

-Ross

> > Does PETSc allow the user to provide a polymorphic object
> > (i.e. a struct with void* and a set of function pointers) to allow the
> > user to send output anywhere?
> 
> That's what PetscViewer objects are for.
> 
> > You would provided a virtual version of printf(...) basically for
> > PETSc object to use.  In standard use case would just be a
> > fall-through call to printf(...).
> 
> Users can also override PetscVFPrintf(), but that is lower level and has
> discarded the object association.  It is common for users to call
> PetscPrintf() directly, but PETSc library code never does it.
> 
> > The hard part is getting the std::ostream object into the objects that
> > you want and telling them how verbose you want them to be.
> 
> Instead of a verbosity "level", we use named monitors for each type of
> diagnostic.  Those diagnostics can go to the same place or different
> places.  Depending on how the user would like to multiplex, they might
> need to add the monitor from code (i.e., make a function call) instead
> of just run-time options, but most can be configured with run-time
> options.
> 
> > For a little more info on this approach, see GCG 16 "Always send
> > output to some general std::ostream object; Never send output directly
> > to std::cout or std::cerr; Never print output with print(...) or
> > printf(...)" in:
> >
> >     http://web.ornl.gov/~8vt/TrilinosCodingDocGuidelines.pdf
> 
> All output from the library is performed via PetscViewer objects.  Most
> diagnostics are provided by monitors, each of which has a
> (reference-counted, possibly-shared) PetscViewer.  Monitors are not part
> of the solver objects themselves, and you can register any number of
> monitors.
> 
> > It would be a huge amount of work to find a way to create useful
> > comprehensible output from a multi-physics code send directly to one
> > stream (i.e. STDOUT).  In the case of CASL VERA Tiamat, the different
> > physics codes actually run in parallel of each other in a block Jacobi
> > black-box solve so even the output from the root rank of each physics
> > would be (and currently is) jumbled together.
> 
> Each physics component should set a different prefix.  For linear
> solvers, this might look like:
> 
>   KSPSetOptionsPrefix(ksp_phys1,"-phys1_");
> 
>   KSPSetOptionsPrefix(ksp_phys2,"-phys2_");
> 
> The monitors can then be directed to separate files, e.g.,
> 
>   -phys1_ksp_monitor phys1.log -phys2_ksp_monitor_true_residual
> phys2.log
> 
> I wrote this in my earlier email, so maybe something wasn't clear there?
> If you want to do something much more flexible, the caller can register
> their own monitors.
> 
> > I suspect that for complex multi-physics APPs like CASL VERA Tiamat,
> > the best overall approach would be to get each physics module to send
> > all of its output to independent files and then just print a nice
> > short summary/table to STDOUT.  That is, separate files for COBRA-TF,
> > MPACT, Insilico, and MOOSE/Peregrine (and in the future MAMBA) would
> > be used that will be written to on the root process of the cluster for
> > each of these modules.  Even this will be hard to implement because,
> > for example, one would need to set up MOOSE/Peregrine to redirect all
> > of its objects and code that it calls to output to a single
> > std::ostream object which is given to it by Tiamat (which in this case
> > will actually be an std::ofstsream object).  This std::ostream object
> > needs to be created by the driver Tiamat and passed into
> > MOOSE/Peregrine and then MOOSE/Peregrine needs to find a way to
> make
> > *all* output created on its behalf send to that std::ostream object,
> > including all PETSc objects it creates and calls.
> 
> If you want this coarser-grained approach, the easiest way is to use
> PetscViewerASCIIGetStdout(comm_physicsX,&viewer) and configure the
> viewer to send output wherever you like.  This viewer is set as an
> attribute on the communicator, which provides a default PetscViewer when
> one is not provided explicitly.
> 
> > That means that PETSc needs to allow users to provide an arbitrary
> > output object that could in turn send its output to the right
> > std::ostream object.  The same goes for Fortran code creating output
> > (but would be harder to set up obviously).
> 
> Fortran callers can use the same functions, though
> PetscViewerASCIIPrintf() is not variadic so the caller must create the
> formatted string first.
> 
> > Just getting these Fortran codes to redirect their output to a general
> > function that we could override to send output to the underlying
> > std::ostream object for their physics code will be a challenge in
> > itself.  But since we control a big part of the funding all the
> > Fortran codes, that would be possible.  MOOSE would be a different
> > story because we have no control over MOOSE.  Therefore, MOOSE might
> > be a lost cause and we might just have to turn off all its outputting
> > if we want a clean STDOUT.
> 
> MOOSE and libMesh are developed by responsible people so there is no
> need to control their funding, just explain the situation rationally.
> 
> > 1) A C++ driver code sets up std::ostream objects and all Trilinos and
> > PETSc objects that are created redirect output to those objects.
> 
> Create a PetscViewer that behaves in this way or (more crudely) set
> PetscVFPrintf.
> 
> > 2) A C driver code that calls Trilinos and PETSc (perhaps indirectly)
> > redirects all output through printf(...), sprintf(...), fprintf(...)
> > or some other function defined on the C side.
> 
> Does Trilinos have a C interface for this?
> 
> > 3) A Fortran code that calls Trilinos and PETSc (perhaps indirectly)
> > redirects all output to some arbitrary Fortran print facilities.
> 
> This currently requires a C stub to call into the Fortran, though a
> native Fortran interface could be added if someone seriously wants to
> manage this from Fortran.
> 
> > However, unless someone is willing to set up and support the
> > infrastructure to maintain the above examples with at least the
> > main-line development versions of PETSc and Trilinos, there is no
> > point in doing anything because it will be lost.  That is the first
> > issue to address.  DOE program managers say that they may want better
> > coordination but are they willing to pay for it?  CASL can get by as
> > described above with the status quo (which is a mess) and my guess is
> > that they would not want to invest a lot of money in Trilinos PETSc
> > compatibility, even just outputting clean-up.  Who is going to
> > actually pay for this which includes the infrastructure to maintain
> > it?  Again, if it is not worth maintaining and providing a reasonable
> > delivery mechanism to users, it is not worth implementing in the first
> > place.
> 
> It sounds like everything or nearly everything is already in-place, so
> output may not need specialized funding.  Testing is a concern, but that
> is a matter of either having a --download-teuchos for one of our build
> machines and/or mocking the other interface.
> 
> I agree that any larger interoperability effort will need funding, since
> it is clearly not "basic research".
> 
> > That is where the lifecycle issues including regulated backward
> > compatibility become critical.
> 
> I don't think the relevant interfaces have changed significantly since
> the 1990s.