[mpich-discuss] MPE tracing

[chan at mcs.anl.gov]

----- "Georges Markomanolis" <georges.markomanolis at ens-lyon.fr> wrote:

> If I understand right the trick with the printf works when the output
> of the traces is the stdout. Moreover I have changed it also, I save one
> file per process and I am buffering the data in order to avoiding 
> writing to the hard disk many times. The overhead is decreased enough
> with this way. I tried to test also the MPI_Pcontrol in my application
> but it didn't work because it should be implemented in the library. 
> Maybe I will try it more if I have time. I don't know/think that it is
> easy to convert the output of the logging to my format. Of course with
> tracing libraries I can do it directly by changing the trace_mpi_core.c 
> and it is really simple. Is there any way for the logging also?
> For example in the tracing libraries instead of having: [1] Starting 
> MPI_Send with count = 131220, dest = 0, tag = 1..
> I save to the trace: p1 send to p0 131220*sizeof(datatype)

What I have in mind is essentially using the TRACE_PRINTF macro (which
you've modified to print to a local file instead of stdout) to do the printf
trick.  Also, the MPI_Pcontrol support I have in mind is essentially
adding a switch in TRACE_PRINTF macro so the printing can be turn on
or off by MPI_Pcontrol.  This should be easily done.

MPE logging records trace into a local memory buffer which is periodically
flushed to a local file.  The local file from each process is then merged
into the final clog2 file during MPI_Finalize().  You could turn off the
merging by setting the environment variable, MPE_DELETE_LOCALFILE, to "no"
(The details is in MPE's README), then you will see local clog2 files.  

> Moreover I am interested in the time for acquiring the traces and with
> this way the overheads seems to be less than 3%. Finally for the cases
> that the trace files are really big, I am trying to avoid having only
> one trace file like in logging. So if it is possible to have one
> logging file per process and moreover to be able to save it with my format 
> before writing the normal logging files then it should be nice but
> maybe 
> it needs a lot of time compared to the implementation of MPI_Pcontrol
> 
> for tracing libraries.
> The programs that I profile use only the MPI_COMM_WORLD but it would
> be 
> nice to have the patch for any case.
> 
> Best regards,
> Georges
> 
> 
> chan at mcs.anl.gov wrote:
> > The equivalent of customized logging for tracing is to add a 
> > printf("Starting of block A...") before the selected block of codes
> > and do another printf("Ending of block A...") after the block of
> codes.
> > But that still produces a lot of output.  I think what should be
> done
> > is to have MPI_Pcontrol() implemented in the tracing library, so
> that
> > one can turn on or off tracing with MPI_Pcontrol.  I think the
> feature
> > can be easily added to the trace_mpi_core.c ...  But it is not there
> yet.
> >
> > Given the tracing library does not have MPI_Pcontrol support and you
> may
> > want to use logging library with MPI_Pcontrol instead.  BTW, are
> the
> > MPI programs being profiled uses communicator besides MPI_COMM_WORLD
> ?
> > If the answer is yes, your version of logging library may contain
> > bugs in MPI_Pcontrol when used with MPI communicator functions. 
> Let
> > me know, I will give you a patch...
> >
> > A.Chan
> >
> > ----- "Georges Markomanolis" <george at markomanolis.com> wrote:
> >
> >   
> >> Thank you all for your answers,
> >>
> >> I used clog2print because I wanted to see only the traces, not the
> 
> >> visualization as I need to convert the MPE traces into another
> format
> >>
> >> that I need. Moreover I changed the output of the tracing library
> in 
> >> order to have the format that I mentioned and it seems to work very
> 
> >> nice. One more question. Is it possible to have a selective 
> >> instrumentation with the tracing library? I think something
> similar
> >> with 
> >> the customized logging but for tracing, I just need to know which
> MPI
> >>
> >> calls are executed by a block of code.
> >>
> >> Thanks a lot,
> >> Best regards,
> >> Georges
> >>
> >> chan at mcs.anl.gov wrote:
> >>     
> >>> ----- "Georges Markomanolis" <georges.markomanolis at ens-lyon.fr>
> >>>       
> >> wrote:
> >>     
> >>>   
> >>>       
> >>>> Dear all,
> >>>>
> >>>> I want to ask you about the MPE's libraries. While I use the
> >>>>         
> >> logging 
> >>     
> >>>> library for an application, I dump the clog file with the command
> 
> >>>> clog2print (is the right way to see the content of the files?).
> >>>>     
> >>>>         
> >>> No, you should use the command jumpshot to "view" clog2 file.
> >>>
> >>>   
> >>>       
> >>>> Although 
> >>>> my application executes MPI_Bcast commands I can't see them.
> From
> >>>>         
> >> the
> >>     
> >>>> manual I read that nested calls are not taken into account, like
> 
> >>>> MPI_Bcast. Is this why I can't see any MPI_Bcast? 
> >>>>     
> >>>>         
> >>> No. nested call probably is irrelevant here.
> >>>
> >>>   
> >>>       
> >>>> Moreover when I link 
> >>>> with the tracing library I can see the MPI_Bcast commands. So in
> >>>>         
> >> the 
> >>     
> >>>> case that I want to know about all the MPI commands I should use
> >>>> tracing 
> >>>> not logging? I don't need the timestamps and the visualization.
> >>>>     
> >>>>         
> >>> If you have access to the source code and you don't care about
> the
> >>>       
> >> order
> >>     
> >>> of MPI calls being executed, you can simply use "grep" of string
> >>>       
> >> that
> >>     
> >>> is prefixed with MPI_, e.g.
> >>>
> >>> grep "MPI_[A-Za-z_]*(" *.c
> >>>
> >>> If you still need to do logging, you can send me how you
> >>>       
> >> linked/compiled
> >>     
> >>> to enable MPE logging (I suspect something wrong there).
> >>>
> >>> A.Chan
> >>>   
> >>>       
> >>>> Thanks a lot,
> >>>> Best regards,
> >>>> Georges
> >>>> _______________________________________________
> >>>> mpich-discuss mailing list
> >>>> mpich-discuss at mcs.anl.gov
> >>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> >>>>     
> >>>>         
> >>> _______________________________________________
> >>> mpich-discuss mailing list
> >>> mpich-discuss at mcs.anl.gov
> >>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> >>>
> >>>       
> >
> >