[mpich-discuss] mpich compilation issue
Gus Correa
gus at ldeo.columbia.edu
Wed Sep 14 12:16:22 CDT 2011
akshar bhosale wrote:
> No, it is not bug because the same code is running in other machine.
Hi Akshar
The devilish thing with MPI programs is that success in one machine,
or with a certain number of processes, or in a specific network,
doesn't mean that the program is bug free.
It may still fail in a different environment.
[Actually, even sequential programs are like that.
In one computer, with a certain compiler, etc,
say, a segmentation fault may not appear, may not break
at run time, but in another machine/OS/compiler it may.]
"It works" is a necessary but not sufficient condition for
"being bug-free"
The error message you've got is pretty clear.
There is a deadlock.
See Pavan's suggestion in his email.
I hope this helps,
Gus Correa
>
>
> Can something be done so that even if mpif77 does not get created,
> no issues; but mpif90 (mpich-1.2.7p1, gfortran) should get created
> and fortran helloworld should run without any issues.
>
> Thank you
> Akshar.
>
> On Tue, Sep 13, 2011 at 10:39 PM, Gus Correa <gus at ldeo.columbia.edu
> <mailto:gus at ldeo.columbia.edu>> wrote:
> Hi Akshar
>
> It sounds as a bug in the program,
> which causes a deadlock, not a problem with MPICH2.
> Your error message:
>
> > MPID_Send(57): DEADLOCK: attempting to send a message to the local
> > process without a prior matching receive
> If you post the code, the list may help you.
>
>
> Gus Correa
>
>
> akshar bhosale wrote:
>
> sorry for not answering why not mpich2.
> We have tried to compile our code using mpich2, it compiles but
> while running the executable generated, it runs and in between
> it comes out giving some mpi send error. The code compiled with
> mpif90 (mpich-1.2.7p1) using gfortran on other machine runs fine.
> Error is as follows:
>
> #######
> 3 aft MPI_SEND 2 7 2 0
> 3 call MPI_SEND 2 8 2
> 0
> 3 aft MPI_SEND 2 8 2
> 0
> 3 call MPI_SEND 2 9 2
> 0
> 3 aft MPI_SEND 2 9 2
> 0
> 3 call MPI_SEND 2 10 2
> 0
> 3 aft MPI_SEND 2 10 2
> 0
> 3 call MPI_SEND 3 1 3
> 0
> [cli_3]: aborting job:
> Fatal error in MPI_Send: Other MPI error, error stack:
> MPI_Send(175): MPI_Send(buf=0x2aaab2af2018, count=1,
> dtype=USER<vector>, dest=3, tag=30003, MPI_COMM_WORLD) failed
> MPID_Send(57): DEADLOCK: attempting to send a message to the
> local process without a prior matching receive
> ###############
>
> -akshar
>
> On Tue, Sep 13, 2011 at 10:02 PM, akshar bhosale
> <akshar.bhosale at gmail.com <mailto:akshar.bhosale at gmail.com>
> <mailto:akshar.bhosale at gmail.__com
> <mailto:akshar.bhosale at gmail.com>>> wrote:
>
> thanks for reply.
>
> Goal is to generate mpif90 using mpich-1.2.7p1 using gfortran
> which
> will generate executable from our fortran code.
>
> source files to be compiled are .f and .f90 files.
>
> f2c flag is given because mp[if90 was not getting generated. no
> other reason. we want mpif90.
>
> Now we have tried giving only FC=gfortran and no other flag of
> option except prefix, but still it is not generating mpif90
> nor mpif77.
>
> mpif90 generated by using [export F90=fortran;F90FLAGS="-ff2c".]
> options if tries to compile our code, it throws error
>
>
> <path to mpif90>/mpif90 -f90=gfortran -Wl -o ../output_code.x
> set.o
> inp.o otp.o bvsol.o user_kg.o mpi_util.o mpi_communication.o
> mpi_memalloc.o driver.o rxn.o advdi_1d_finitediff_mf.o
> aq_lib.o \
> -L/ioapi/Linux2_x86_64gfort -lioapi
> -L/np_cs/netcdf/INSTALL/lib -lnetcdf -L/usr/lib64/
> -lgomp /np_cs/lapack/lapack-3.2.1/__lapack_LINUX.a
> /np_cs/lapack/lapack-3.2.1/__blas_LINUX.a
>
> /build/lib/libmpich.a(farg.o): In function `mpir_iargc__':
> farg.f:(.text+0x6): undefined reference to `f__xargc'
> /build/lib/libmpich.a(farg.o): In function `mpir_getarg__':
> farg.f:(.text+0x38): undefined reference to `getarg_'
> collect2: ld returned 1 exit status
> make: *** [../output_code.x] Error 1
>
> -akshar
>
>
>
>
>
>
> On Tue, Sep 13, 2011 at 8:52 PM, Gus Correa
> <gus at ldeo.columbia.edu <mailto:gus at ldeo.columbia.edu>
> <mailto:gus at ldeo.columbia.edu
> <mailto:gus at ldeo.columbia.edu>>__> wrote:
>
> Hi Pavan and Akshar
>
> I see, it's MPICH-1.
> May I ask why?
>
> Would a workaround perhaps be to build only mpif77 (no
> mpif90),
> replace 'use mpi' by '#include "mpif.h"' in the
> application code,
> and recompile it with mpif77?
>
> I wonder if -ff2c works at all with Fortran90.
> (See man page detail below.)
>
> Anyway, I remember that there were complaints of runtime
> errors (P4 errors) with mpich-1,
> reported on this and other mailing lists,
> when running on current Linux kernels.
> Even with simple codes like the cpi example.
> Using MPICH-1 may be an uphill battle.
>
> I hope this helps,
> Gus Correa
>
> >From man gfortran (4.1.2):
>
> -ff2c
> Generate code designed to be compatible with code
> generated by g77 and f2c.
>
> The calling conventions used by g77 (originally
> implemented in f2c) require functions that return type
> default
> "REAL" to actually return the C type
> "double", and functions that return type
> "COMPLEX" to
> return the values via an extra argument in the calling
> sequence
> that points to where to store the
> return value. Under the default GNU calling
> conventions, such functions simply return their results
> as they
> would in GNU C -- default "REAL" functions
> return the C type "float", and "COMPLEX" functions
> return the GNU C type "complex". Additionally, this option
> implies the -fsecond-underscore option,
> unless -fno-second-underscore is explicitly
> requested.
>
> This does not affect the generation of code that
> interfaces with the libgfortran library.
>
> Caution: It is not a good idea to mix Fortran code
> compiled with "-ff2c" with code compiled with the default
> "-fno-f2c" calling conventions as, calling
> "COMPLEX" or default "REAL" functions between
> program
> parts which were compiled with different calling conventions
> will break at execution time.
>
> Caution: This will break code which passes
> intrinsic
> functions of type default "REAL" or "COMPLEX" as actual
> arguments, as the library implementations
> use the -fno-f2c calling conventions.
>
>
>
>
> Pavan Balaji wrote:
>
>
> Gus: He doesn't want to use MPICH2. He's using MPICH-1.
>
> -- Pavan
>
> On 09/13/2011 09:51 AM, Gus Correa wrote:
>
> Hi Akshar
>
> Maybe the F90FLAGS="-ff2c" flag is causing the
> problem?
> What if you remove it and use gfortran for F77
> and FC?
>
>
> Also, my 'configure -help' (a bit old, MPICH2
> 1.3.2p1)
> mentions FC, not F90, as the environment variable
> to be set.
>
> For what it is worth, MPICH2 builds fine for us, with
> gcc, g++ and gfortran (for F77 and FC).
>
> My two cents,
> Gus Correa
>
> akshar bhosale wrote:
>
> Hi,
>
> If i specify F77 anf F90 as gfortran, it
> doesn't
> create mpif90 and
> mpif77 at all.
> Please see attached config log
> If i specify F90=gfortran and
> F90FLAGS="-ff2c", the
> it creates mpif90
> and mpif77 but is generating below issue.
> Kindly help.
> -akshar
>
> On Tue, Sep 13, 2011 at 9:46 AM, Rajeev
> Thakur<thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov> <mailto:thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov>>
> <mailto:thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov>
> <mailto:thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov>>>> wrote:
>
> In the configure step, if you specify
> both F77
> and F90 as gfortran,
> it should work.
>
> Rajeev
>
> On Sep 12, 2011, at 9:49 PM, akshar
> bhosale wrote:
>
> > Hi,
> > thanks.
> > but, in case of mpicc, it can be ignored,
> but in case of mpif90,
> it is still an issue.
> > e.g. hello_mpi.f90 program
> > ####################
> > program main
> > use mpi
> > !
> > ! implicit none
> > !
> > integer error
> > integer id
> > integer p
> > real ( kind = 8 ) wtime
> > !
> > ! Initialize MPI.
> > !
> > call MPI_Init ( error )
> > !
> > ! Get the number of processes.
> > !
> > call MPI_Comm_size ( MPI_COMM_WORLD, p,
> error )
> >
> > !
> > ! Get the individual process ID.
> > !
> > call MPI_Comm_rank (
> MPI_COMM_WORLD, id,
> error )
> > !
> > ! Print a message.
> > !
> > if ( id == 0 ) then
> >
> > wtime = MPI_Wtime ( )
> >
> > write ( *, '(a)' ) ' '
> > write ( *, '(a)' ) 'HELLO_MPI -
> Master
> process:'
> > write ( *, '(a)' ) '
> FORTRAN90/MPI version'
> > write ( *, '(a)' ) ' '
> > write ( *, '(a)' ) ' An MPI test
> program.'
> > write ( *, '(a)' ) ' '
> > write ( *, '(a,i8)' ) ' The
> number of
> processes is ', p
> > write ( *, '(a)' ) ' '
> >
> > end if
> >
> > write ( *, '(a)' ) ' '
> > write ( *, '(a,i8,a)' ) ' Process
> ', id,
> ' says "Hello, world!"'
> >
> > if ( id == 0 ) then
> >
> > write ( *, '(a)' ) ' '
> > write ( *, '(a)' ) 'HELLO_MPI -
> Master
> process:'
> > write ( *, '(a)' ) ' Normal end of
> execution: "Goodbye,
> world!".'
> >
> > wtime = MPI_Wtime ( ) - wtime
> > write ( *, '(a)' ) ' '
> > write ( *, '(a,g14.6,a)' ) ' Elapsed
> wall clock time = ',
> wtime, ' seconds.'
> >
> > end if
> > !
> > ! Shut down MPI.
> > !
> > call MPI_Finalize ( error )
> >
> > stop
> > end
> > ############################
> >
> > $ ../../../build/bin/mpif90 hello_mpi.f90
> >
>
> /scratch1/np_cs/trial1/build/____lib/libmpich.a(farg.o):
> In function
> `mpir_iargc__':
> > farg.f:(.text+0x6): undefined
> reference to
> `f__xargc'
> >
>
> /scratch1/np_cs/trial1/build/____lib/libmpich.a(farg.o):
> In function
> `mpir_getarg__':
> > farg.f:(.text+0x38): undefined
> reference to
> `getarg_'
> > collect2: ld returned 1 exit status
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 12, 2011 at 9:42 AM,
> Rajeev Thakur
> <thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov>
> <mailto:thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov>><__mailto:tha__kur at mcs.anl.gov
> <mailto:tha__kur at mcs.anl.gov>
>
> <mailto:thakur at mcs.anl.gov
> <mailto:thakur at mcs.anl.gov>>>> wrote:
> > Ignore that. Compile a real program
> as mpicc
> foo.c.
> >
> > On Sep 11, 2011, at 8:38 PM, akshar
> bhosale
> wrote:
> >
> > > Hi,
> > > thank you for reply.
> > >
> > > I tried to install it make it using
> following options now and i
> get following error with undefined
> reference to
> main. options given
> while configuring the same are :
> > > export F90=fortran;F90FLAGS="-ff2c".
> > > but error of undefined reference
> to main
> is observed in every
> executable like :
> > >
> > > ./mpicc -v
> > > mpicc for 1.2.7 (release) of :
> 2005/06/22
> 16:33:49
> > > Using built-in specs.
> > > Target: x86_64-redhat-linux
> > > Configured with: ../configure
> --prefix=/usr
> --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared
> --enable-threads=posix
> --enable-checking=release
> --with-system-zlib
> --enable-__cxa_atexit
> --disable-libunwind-exceptions
> --enable-libgcj-multifile
>
> --enable-languages=c,c++,objc,____obj-c++,java,fortran,ada
> --enable-java-awt=gtk --disable-dssi
> --disable-plugin
>
> --with-java-home=/usr/lib/jvm/____java-1.4.2-gcj-1.4.2.0/jre
> --with-cpu=generic --host=x86_64-redhat-linux
> > > Thread model: posix
> > > gcc version 4.1.2 20080704 (Red
> Hat 4.1.2-50)
> > >
> /usr/libexec/gcc/x86_64-____redhat-linux/4.1.2/collect2
> --eh-frame-hdr -m elf_x86_64 --hash-style=gnu
> -dynamic-linker
> /lib64/ld-linux-x86-64.so.2
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crt1.o
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crti.o
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/crtbegin.o
> -L/scratch1/np_cs/trial2/____build/lib
> -L/opt/intel/mkl/10.2.2.025/____lib/em64t
> <http://10.2.2.025/__lib/em64t>
>
> <http://10.2.2.025/lib/em64t><__http://10.2.2.025/__lib/em64t
>
> <http://10.2.2.025/lib/em64t>>
> -L/opt/intel/mkl/10.2.2.025/____lib/em64t
> <http://10.2.2.025/__lib/em64t>
>
> <http://10.2.2.025/lib/em64t><__http://10.2.2.025/__lib/em64t
>
> <http://10.2.2.025/lib/em64t>>
> -L/usr/lib/gcc/x86_64-redhat-____linux/4.1.2
> -L/usr/lib/gcc/x86_64-redhat-____linux/4.1.2
>
> -L/usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64
> -L/lib/../lib64 -L/usr/lib/../lib64 -lmpich
> -lpthread -lrt -lgcc
> --as-needed -lgcc_s --no-as-needed -lc -lgcc
> --as-needed -lgcc_s
> --no-as-needed
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/crtend.o
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crtn.o
> > >
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crt1.o:
> In
> function `_start':
> > > (.text+0x20): undefined reference
> to `main'
> > > collect2: ld returned 1 exit status
> > >
> > > ./mpif90 -v
> > > mpif90 for 1.2.7 (release) of :
> 2005/06/22 16:33:49
> > > Driving: gfortran
> -L/scratch1/np_cs/trial2/____build/lib -v
> -lmpichf90 -lmpich -lpthread -lrt
> -lgfortranbegin -lgfortran -lm
> -shared-libgcc
> > > Using built-in specs.
> > > Target: x86_64-redhat-linux
> > > Configured with: ../configure
> --prefix=/usr
> --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared
> --enable-threads=posix
> --enable-checking=release
> --with-system-zlib
> --enable-__cxa_atexit
> --disable-libunwind-exceptions
> --enable-libgcj-multifile
>
> --enable-languages=c,c++,objc,____obj-c++,java,fortran,ada
> --enable-java-awt=gtk --disable-dssi
> --disable-plugin
>
> --with-java-home=/usr/lib/jvm/____java-1.4.2-gcj-1.4.2.0/jre
> --with-cpu=generic --host=x86_64-redhat-linux
> > > Thread model: posix
> > > gcc version 4.1.2 20080704 (Red
> Hat 4.1.2-50)
> > >
> /usr/libexec/gcc/x86_64-____redhat-linux/4.1.2/collect2
> --eh-frame-hdr -m elf_x86_64 --hash-style=gnu
> -dynamic-linker
> /lib64/ld-linux-x86-64.so.2
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crt1.o
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crti.o
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/crtbegin.o
> -L/scratch1/np_cs/trial2/____build/lib
> -L/opt/intel/mkl/10.2.2.025/____lib/em64t
> <http://10.2.2.025/__lib/em64t>
>
> <http://10.2.2.025/lib/em64t><__http://10.2.2.025/__lib/em64t
>
> <http://10.2.2.025/lib/em64t>>
> -L/opt/intel/mkl/10.2.2.025/____lib/em64t
> <http://10.2.2.025/__lib/em64t>
>
> <http://10.2.2.025/lib/em64t><__http://10.2.2.025/__lib/em64t
>
> <http://10.2.2.025/lib/em64t>>
> -L/usr/lib/gcc/x86_64-redhat-____linux/4.1.2
> -L/usr/lib/gcc/x86_64-redhat-____linux/4.1.2
>
> -L/usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64
> -L/lib/../lib64 -L/usr/lib/../lib64
> -lmpichf90
> -lmpich -lpthread
> -lrt -lgfortranbegin -lgfortran -lm -lgcc_s
> -lgcc -lc -lgcc_s -lgcc
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/crtend.o
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/../../../../lib64/____crtn.o
> > >
>
> /usr/lib/gcc/x86_64-redhat-____linux/4.1.2/libgfortranbegin.____a(fmain.o):
> In function `main':
> > > (.text+0xa): undefined reference
> to `MAIN__'
> > > collect2: ld returned 1 exit status
> > >
> > > Am i missing somethin?
> > > thank you
> > > -akshar
> > >
> > > On Mon, Sep 12, 2011 at 1:24 AM,
> William
> Gropp
> <wgropp at illinois.edu
> <mailto:wgropp at illinois.edu>
> <mailto:wgropp at illinois.edu
> <mailto:wgropp at illinois.edu>><__mailto:wg__ropp at illinois.edu
> <mailto:wg__ropp at illinois.edu>
>
> <mailto:wgropp at illinois.edu
> <mailto:wgropp at illinois.edu>>>> wrote:
> > > You can use gfortran for F77 and
> F90. If
> your requirement is
> to use gfortran for F77 and a different
> fortran
> for F90, then they
> will need to use consistent linker names,
> calling conventions, data
> sizes, and libraries. The config.log
> file will
> have details on why
> configure decided that the Fortran 77 and
> Fortran 90 compilers were
> not compatible.
> > >
> > > Bill
> > >
> > >
> > > On Sep 9, 2011, at 4:28 PM, akshar
> bhosale wrote:
> > >
> > > hi,
> > > we have requirement of mpich-1.2.7
> with
> gfortran. We have fiven
> flags as :
> CC=gcc;F90=fortran;F77=____gfortran;
> and tried to configure,
> it throws error as configure: error:
> Fortran 90
> and Fortran 77
> compilers are not compatible..
> > > We have to do it using gfortran
> only. os
> is rhel5.1 64 bit
> > > what extran flags settings we have
> to do?
> > >
> > > -akshar
> > >
>
> ___________________________________________________
> > > mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> <mailto:mpich-discuss at mcs.anl.
> <mailto:mpich-discuss at mcs.anl.>____gov
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>>
> > > To manage subscription options or
> unsubscribe:
> > >
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
> > >
> > > William Gropp
> > > Director, Parallel Computing Institute
> > > Deputy Director for Research
> > > Institute for Advanced Computing
> Applications and Technologies
> > > Paul and Cynthia Saylor Professor of
> Computer Science
> > > University of Illinois
> Urbana-Champaign
> > >
> > >
> > >
> > >
> > >
>
> ___________________________________________________
> > > mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> <mailto:mpich-discuss at mcs.anl.
> <mailto:mpich-discuss at mcs.anl.>____gov
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>>
> > > To manage subscription options or
> unsubscribe:
> > >
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
> >
> >
>
> ___________________________________________________
> > mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> <mailto:mpich-discuss at mcs.anl.
> <mailto:mpich-discuss at mcs.anl.>____gov
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>>
> > To manage subscription options or
> unsubscribe:
> >
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
> >
> >
>
> ___________________________________________________
> > mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> <mailto:mpich-discuss at mcs.anl.
> <mailto:mpich-discuss at mcs.anl.>____gov
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>>
> > To manage subscription options or
> unsubscribe:
> >
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
>
>
> ___________________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> <mailto:mpich-discuss at mcs.anl.
> <mailto:mpich-discuss at mcs.anl.>____gov
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>>
> To manage subscription options or
> unsubscribe:
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
>
>
>
>
> ------------------------------____----------------------------__--__------------
>
>
> ___________________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> To manage subscription options or unsubscribe:
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
>
>
> ___________________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> To manage subscription options or unsubscribe:
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
>
>
>
> ___________________________________________________
> mpich-discuss mailing list mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> <mailto:mpich-discuss at mcs.anl.__gov
> <mailto:mpich-discuss at mcs.anl.gov>>
> To manage subscription options or unsubscribe:
>
> https://lists.mcs.anl.gov/____mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>
> <https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>__>
>
>
>
>
> ------------------------------__------------------------------__------------
>
> _________________________________________________
> mpich-discuss mailing list mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>
>
> _________________________________________________
> mpich-discuss mailing list mpich-discuss at mcs.anl.gov
> <mailto:mpich-discuss at mcs.anl.gov>
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> mpich-discuss mailing list mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
More information about the mpich-discuss
mailing list