[mpich-discuss] Trouble in getting the logging interface to work
Krishna Chaitanya
kris.c1986 at gmail.com
Tue Mar 25 15:20:23 CDT 2008
>I would think that you'll make MPE calls in your PERUSE's callback
function. In other
words, you should compile everything as "mpicc -mpe=log" NOT "mpicc
-mpe=mpilog"....
I was trying to have the PERUSE function make the MPE call
before it invokes the user defined call-back function. Will try this
out.
Thanks,
Krishna Chaitanya K
On 3/25/08, Anthony Chan <chan at mcs.anl.gov> wrote:
>
> MPE profiling libraries are written based on PMPI so it is different
> from PERUSE. I'm not familar with PERUSE's event-based model. Wouldn't
> it mean that you need to implement an event-based callback mechanism in
> your modified MPICH2 libraries ? If that is the case, I would think that
> you'll make MPE calls in your PERUSE's callback function. In other
> words, you should compile everything as "mpicc -mpe=log"
> NOT "mpicc -mpe=mpilog"....
>
> On Tue, 25 Mar 2008, Krishna Chaitanya wrote:
>
> >> The 2 MPI_Init definitions are defined in 2 different libraries as
> > you have found.
> > True. But, when the MPI application is being compiled, we notice
> > the order "-lmpe -lmpich". But, how is it that a conflict is not
> > reported?
> >
> > I am using PERUSE to profile the MPICH library. So, as of now, I am
> > having PERUSE_TRACE_COMM_EVENT defined in libmpich.a. However, that is
> > currently generating text output. I wish to graphically represent the
> > PERUSE events in conjunction with the other MPI events.
> >
> > Looking at the way the MPE library has been built, I understood that I
> > need to have PERUSE_TRACE_COMM_EVENT defined in liblmpe.a. This way,
> > when I compile the PERUSE-MPICH application with the -mpe=mpilog
> > switch, the function is invoked in the mpe library is invoked, the
> > occurence is logged, and then the actual PERUSE function defined in
> > libmpich.a is invoked.
> > Have I got that right?
> >
> >
> > Thanks,
> > Krishna Chaitanya
> >
> >
> > On 3/25/08, Anthony Chan <chan at mcs.anl.gov> wrote:
> >>
> >>
> >> On Mon, 24 Mar 2008, Krishna Chaitanya wrote:
> >>
> >>> That answers a lot of my questions.
> >>> Sticking to MPI_Init for this discussion, the function is defined at two
> >>> places.So,there will be a reference to MPI_Init in lmpich and also a
> >>> reference in lmpe. But, how is it that when an MPI application is being
> >>> compiled, mpicc doesnt complain that there were two references to
> >> MPI_Init?
> >>> What exactly has been done to solve this?
> >>
> >> log_mpi_core.c uses the PMPI interface defined in MPI standard. The 2
> >> MPI_Init definitions are defined in 2 different libraries as you have
> >> found.
> >>
> >>> I am actually facing this problem with the PERUSE function that i have
> >>> defined at :
> >>> 1 > /src/peruse/peruse.c and
> >>> 2 > src/mpe2/src/wrappers/src/log_mpi_core.c.
> >>>
> >>> The first one goes into lmpich and the second one is a part of lmpe.
> >>> However, when I compile my MPI program, I get the following message :
> >>> /home/kc/mpich-install//lib/libmpich.a(peruse.o): In function
> >>> `PERUSE_TRACE_COMM_EVENT':
> >>> /home/kc/mpich-src/src/peruse/peruse.c:324: multiple definition of
> >>> `PERUSE_TRACE_COMM_EVENT'
> >>>
> >>
> /home/kc/mpich-install//lib/liblmpe.a(log_mpi_core.o):/home/kc/mpich-src/src/mpe2/src/wrappers/src/log_mpi_core.c:6830:
> >>> first defined here
> >>> collect2: ld returned 1 exit status
> >>
> >> Why did you define PERUSE_TRACE_COMM_EVENT twice ? One in libmpich.a
> >> and one in liblmpe.a. Are you trying to use MPE to profile PERUSE or to
> >> use PERUSE to profile MPE ?
> >>
> >> A.Chan
> >>
> >>>
> >>>
> >>> Thanks for your time,
> >>> Krishna Chaitanya K
> >>>
> >>>
> >>> On Mon, Mar 24, 2008 at 10:22 AM, Anthony Chan <chan at mcs.anl.gov> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Mon, 24 Mar 2008, Krishna Chaitanya wrote:
> >>>>
> >>>>> Sorry for re-posting.
> >>>>>> I took a look at the documentation at src/util/multichannel/mpi.c
> >>>>> Guess this is only for windows.
> >>>>> It would be of great help if someone could point me to the
> >>>>> function that takes care of mapping MPI_Init to its wrapper, defined
> in
> >>>>> src/mpe2/src/wrappers/src/log_mpi_core.c, when the library is compiled
> >>>> with
> >>>>> the --enable-mpe switch., instead of the function defined in
> >>>>> src/mpi/init/init.c
> >>>>
> >>>> The function in src/mpi/init/init.c is defined when linked with
> -lmpich,
> >>>> i.e. mpicc. The functions in log_mpi_core.c is defined when linked with
> >>>> "mpicc -mpe=mpilog". Try do "mpicc .... -show" will show you what
> >>>> libraries and their link order.
> >>>>
> >>>> A.Chan
> >>>>
> >>>>>
> >>>>> Krishna Chaitanya K
> >>>>>
> >>>>> On Sun, Mar 23, 2008 at 2:29 PM, Krishna Chaitanya
> <kris.c1986 at gmail.com
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>>> See section "CUSTOMIZING LOGFILES" in mpich2-xxx/src/mpe2/README.
> >>>>>> Correct me if I am wrong :
> >>>>>> Since I am dealing with PERUSE events, whenever such an event occurs,
> a
> >>>>>> PERUSE function, defined in <mpich-dir>/src/peruse/peruse.c, is
> invoked
> >>>> by
> >>>>>> the MPI library. I am trying to get this event displayed in the
> >>>> jumpshot
> >>>>>> output. For this to be done, I need to define a wrapper function
> which
> >>>> gets
> >>>>>> invoked when a PERUSE event occurs, to log the event and then to call
> >>>> the
> >>>>>> actuall peruse function, which is similar to the way the wrapper
> >>>> function at
> >>>>>> log_mpi_core.c is called, when MPI_Init is called.
> >>>>>>
> >>>>>> Could you please clarify on the dynamic mapping?
> >>>>>> I took a look at the documentation at src/util/multichannel/mpi.c. I
> >>>>>> think, I understood what is going on in LoadFunctions() and the way
> the
> >>>>>> function pointers are assigned addresses depending the dll that is
> >>>> being
> >>>>>> used.
> >>>>>>
> >>>>>> Krishna Chaitanya K
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Mar 23, 2008 at 12:59 PM, Anthony Chan <chan at mcs.anl.gov>
> >>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> See section "CUSTOMIZING LOGFILES" in mpich2-xxx/src/mpe2/README.
> >>>>>>> You don't need to modify MPE libraries.
> >>>>>>>
> >>>>>>> A.Chan
> >>>>>>>
> >>>>>>> On Sun, 23 Mar 2008, Krishna Chaitanya wrote:
> >>>>>>>
> >>>>>>>> I have modified the mpe library to log the events that I am
> >>>> interested
> >>>>>>> in
> >>>>>>>> monitoring. But, I am bit hazy about how a function like MPI_Init
> is
> >>>>>>>> actually linked to the MPI_Init routine in the file log_mpi_core.c
> >>>>>>> when we
> >>>>>>>> compile the MPI application with the -mpe=mpilog switch. Could
> >>>> someone
> >>>>>>> point
> >>>>>>>> me to the routine that takes care of such a mapping?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Krishna Chaitanya K
> >>>>>>>>
> >>>>>>>> On Sat, Mar 22, 2008 at 3:01 AM, Krishna Chaitanya <
> >>>>>>> kris.c1986 at gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks a lot. I installed the latest jdk version and I am now able
> >>>> to
> >>>>>>> look
> >>>>>>>>> at the jumpshot output.
> >>>>>>>>>
> >>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sat, Mar 22, 2008 at 1:45 AM, Anthony Chan <chan at mcs.anl.gov>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> The error that you showed earlier does not suggest the problem is
> >>>>>>> with
> >>>>>>>>>> running jumpshot on your machine with limited memory. If your
> >>>> clog2
> >>>>>>>>>> file
> >>>>>>>>>> isn't too bad, send it to me.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 21 Mar 2008, Krishna Chaitanya wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I resolved that issue.
> >>>>>>>>>>> My comp ( Intel centrino 32 bit , 256 MB RAM - Dated, I agree)
> >>>>>>> hangs
> >>>>>>>>>> each
> >>>>>>>>>>> time I launch jumpshot with the slogfile. Since this is an
> >>>>>>> independent
> >>>>>>>>>>> project, I am constrained when it comes to the availability of
> >>>>>>>>>> machines.
> >>>>>>>>>>> Would you recommend that I give it a try on a 64bit AMD, 512MB
> >>>> RAM?
> >>>>>>> (
> >>>>>>>>>> Will
> >>>>>>>>>>> have to start from installing linux on this machine. Is it worth
> >>>>>>> the
> >>>>>>>>>> effort
> >>>>>>>>>>> ?) If it requires higher configuration, would you please suggest
> a
> >>>>>>>>>> lighter
> >>>>>>>>>>> graphical tool that I can use to present the occurrence of
> events
> >>>>>>> and
> >>>>>>>>>> the
> >>>>>>>>>>> corresponding times?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Mar 21, 2008 at 8:23 PM, Anthony Chan <chan at mcs.anl.gov>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, 21 Mar 2008, Krishna Chaitanya wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The file block pointer to the Tree Directory is NOT
> >>>> initialized!,
> >>>>>>>>>> can't
> >>>>>>>>>>>> read
> >>>>>>>>>>>>> it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> That means the slog2 file isn't generated completely.
> Something
> >>>>>>> went
> >>>>>>>>>>>> wrong in the convertion process (assuming your clog2 file is
> >>>>>>>>>> complete).
> >>>>>>>>>>>> If your MPI program doesn't finish MPI_Finalize normally, your
> >>>>>>> clog2
> >>>>>>>>>>>> file will be incomplete.
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> IS there any environment variable that needs to be
> >>>>>>>>>> initialsed?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nothing needs to be initialized by hand.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A.Chan
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Mar 20, 2008 at 4:56 PM, Dave Goodell <
> >>>>>>> goodell at mcs.anl.gov>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> It's pretty hard to debug this issue via email. However, you
> >>>>>>> could
> >>>>>>>>>>>>>> try running valgrind on your modified MPICH2 to see if any
> >>>>>>> obvious
> >>>>>>>>>>>>>> bugs pop out. When you do, make sure that you configure with
> >>>>>>> "--
> >>>>>>>>>>>>>> enable-g=dbg,meminit" in order to avoid spurious warnings and
> >>>> to
> >>>>>>> be
> >>>>>>>>>>>>>> able to see stack traces.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mar 19, 2008, at 1:05 PM, Krishna Chaitanya wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The problem seems to be with the communicator in MPI_Bcast()
> >>>>>>>>>> (/src/
> >>>>>>>>>>>>>>> mpi/coll/bcast.c).
> >>>>>>>>>>>>>>> The comm_ptr is initialized to NULL and after a call to
> >>>>>>>>>>>>>>> MPID_Comm_get_ptr( comm, comm_ptr ); , the comm_ptr points
> to
> >>>>>>> the
> >>>>>>>>>>>>>>> communicator object which was created throught MPI_Init().
> >>>>>>>>>>>>>>> However, MPID_Comm_valid_ptr( comm_ptr, mpi_errno ) returns
> >>>>>>> with
> >>>>>>>>>> a
> >>>>>>>>>>>>>>> value other than MPI_SUCCESS.
> >>>>>>>>>>>>>>> During some traces, it used to crash at this point itself.
> On
> >>>>>>> some
> >>>>>>>>>>>>>>> other traces, it used to go into the progress engine as I
> >>>>>>>>>> described
> >>>>>>>>>>>>>>> in my previous mails.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> What could be the reason? Hope someone chips in. I havent
> been
> >>>>>>>>>> able
> >>>>>>>>>>>>>>> to figure this out for sometime now.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Mar 19, 2008 at 8:44 AM, Krishna Chaitanya
> >>>>>>>>>>>>>>> <kris.c1986 at gmail.com> wrote:
> >>>>>>>>>>>>>>> This might help :
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In the MPID_Comm structure, I have included the following
> line
> >>>>>>> for
> >>>>>>>>>>>>>>> the peruse place-holder :
> >>>>>>>>>>>>>>> struct mpich_peruse_handle_t** c_peruse_handles;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> And in the function, MPID_Init_thread(), i have the line
> >>>>>>>>>>>>>>> MPIR_Process.comm_world->c_peruse_handles = NULL;
> >>>>>>>>>>>>>>> when the rest of the members of the comm_world structure
> are
> >>>>>>>>>> being
> >>>>>>>>>>>>>>> populated.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Mar 19, 2008 at 8:19 AM, Krishna Chaitanya
> >>>>>>>>>>>>>>> <kris.c1986 at gmail.com> wrote:
> >>>>>>>>>>>>>>> Thanks for the help. I am facing an weird problem right now.
> >>>> To
> >>>>>>>>>>>>>>> incorporate the PERUSE component, I have modified the
> >>>>>>> communicator
> >>>>>>>>>>>>>>> data structure to incude the PERUSE handles. The program
> >>>>>>> executes
> >>>>>>>>>>>>>>> as expected when compiled without the "mpe=mpilog" flag.When
> I
> >>>>>>>>>>>>>>> compile it with the mpe component, the program gives this
> >>>>>>> output :
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Fatal error in MPI_Bcast: Invalid communicator, error stack:
> >>>>>>>>>>>>>>> MPI_Bcast(784): MPI_Bcast(buf=0x9260f98, count=1, MPI_INT,
> >>>>>>> root=0,
> >>>>>>>>>>>>>>> MPI_COMM_WORLD) failed
> >>>>>>>>>>>>>>> MPI_Bcast(717): Invalid communicator
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On tracing further, I understood this :
> >>>>>>>>>>>>>>> MPI_Init () ( log_mpi_core.c )
> >>>>>>>>>>>>>>> -- > PMPI_Init ( the communicator object is created here )
> >>>>>>>>>>>>>>> -- > MPE_Init_log ()
> >>>>>>>>>>>>>>> -- > CLOG_Local_init()
> >>>>>>>>>>>>>>> -- > CLOG_Buffer_init4write ()
> >>>>>>>>>>>>>>> -- > CLOG_Preamble_env_init()
> >>>>>>>>>>>>>>> -- > MPI_Bcast () (bcast.c)
> >>>>>>>>>>>>>>> -- > MPIR_Bcast ()
> >>>>>>>>>>>>>>> -- > MPIC_Recv ()
> /
> >>>>>>>>>>>>>>> MPIC_Send()
> >>>>>>>>>>>>>>> -- > MPIC_Wait()
> >>>>>>>>>>>>>>> < Program crashes >
> >>>>>>>>>>>>>>> The MPIC_Wait function is invoking the progress engine,
> >>>>>>> which
> >>>>>>>>>>>>>>> works properly without the mpe component.
> >>>>>>>>>>>>>>> Even within the progress engine, MPIDU_Sock_wait() and
> >>>>>>>>>>>>>>> MPIDI_CH3I_Progress_handle_sock_event() are executed a
> couple
> >>>>>>> of
> >>>>>>>>>>>>>>> times before the program crashes in the
> >>>>>>> MPIDU_Socki_handle_read()
> >>>>>>>>>>>>>>> or the MPIDU_Socki_handle_write() functions. ( The read()
> and
> >>>>>>> the
> >>>>>>>>>>>>>>> write() functions work two times, I think)
> >>>>>>>>>>>>>>> I am finding it very hard to reason why the program
> >>>>>>> crashes
> >>>>>>>>>>>>>>> with mpe. Could you please suggest where I need to look at
> to
> >>>>>>> sort
> >>>>>>>>>>>>>>> this issue out?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Mar 19, 2008 at 2:20 AM, Anthony Chan <
> >>>> chan at mcs.anl.gov
> >>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, 19 Mar 2008, Krishna Chaitanya wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>> I tried configuring MPICH2 by doing :
> >>>>>>>>>>>>>>>> ./configure --prefix=/home/kc/mpich-install/ --enable-mpe
> >>>>>>>>>>>>>>>> --with-logging=SLOG CC=gcc CFLAGS=-g && make && make
> >>>>>>> install
> >>>>>>>>>>>>>>>> It flashed an error messaage saying :
> >>>>>>>>>>>>>>>> onfigure: error: ./src/util/logging/SLOG does not exist.
> >>>>>>>>>>>>>>> Configure aborted
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The --with-logging is for MPICH2's internal logging, not
> MPE's
> >>>>>>>>>>>>>>> logging.
> >>>>>>>>>>>>>>> As what you did below is fine is fine.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> After that, I tried :
> >>>>>>>>>>>>>>>> ./configure --prefix=/home/kc/mpich-install/ --enable-mpe
> >>>>>>> CC=gcc
> >>>>>>>>>>>>>>> CFLAGS=-g
> >>>>>>>>>>>>>>>> && make && make install
> >>>>>>>>>>>>>>>> The installation was normal, when I tried compiling
> >>>> an
> >>>>>>>>>>>>>>> example
> >>>>>>>>>>>>>>>> program by doing :
> >>>>>>>>>>>>>>>> mpicc -mpilog -o sample sample.c
> >>>>>>>>>>>>>>>> cc1: error: unrecognized command line option "-mpilog"
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do "mpicc -mpe=mpilog -o sample sample.c" instead. For more
> >>>>>>>>>> details,
> >>>>>>>>>>>>>>> see "mpicc -mpe=help" and see mpich2/src/mpe2/README.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> A.Chan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can anyone please tell me what needs to be done to
> >>>>>>> use
> >>>>>>>>>>>>>>> the SLOG
> >>>>>>>>>>>>>>>> logging format?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Krishna Chaitanya K
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> In the middle of difficulty, lies opportunity
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> In the middle of difficulty, lies opportunity
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> In the middle of difficulty, lies opportunity
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> In the middle of difficulty, lies opportunity
> >>>
> >>
> >>
> >
> >
> > --
> > In the middle of difficulty, lies opportunity
> >
> >
>
>
--
In the middle of difficulty, lies opportunity
More information about the mpich-discuss
mailing list