[mpich-discuss] mpich compilation issue

akshar bhosale akshar.bhosale at gmail.com
Tue Sep 13 21:41:27 CDT 2011


No, it is not bug because the same code is running in other machine.



> Can something be done so that even if mpif77 does not get created, no
> issues; but mpif90 (mpich-1.2.7p1, gfortran) should get created and fortran
> helloworld should run without any issues.
>
Thank you
Akshar.

On Tue, Sep 13, 2011 at 10:39 PM, Gus Correa <gus at ldeo.columbia.edu> wrote:
 Hi Akshar

> It sounds as a bug in the program,
> which causes a deadlock, not a problem with MPICH2.
> Your error message:
>
> > MPID_Send(57): DEADLOCK: attempting to send a message to the local
> > process without a prior matching receive
> If you post the code, the list may help you.
>
>
> Gus Correa
>
>
> akshar bhosale wrote:
>
>> sorry for not answering why not mpich2.
>> We have tried to compile our code using mpich2, it compiles but while
>> running the executable generated, it runs and in between it comes out giving
>> some mpi send error. The code compiled with mpif90 (mpich-1.2.7p1) using
>> gfortran on other machine runs fine.
>> Error is as follows:
>>
>> #######
>>  3  aft MPI_SEND           2           7           2           0
>>           3  call MPI_SEND           2           8           2           0
>>           3  aft MPI_SEND           2           8           2           0
>>           3  call MPI_SEND           2           9           2           0
>>           3  aft MPI_SEND           2           9           2           0
>>           3  call MPI_SEND           2          10           2           0
>>           3  aft MPI_SEND           2          10           2           0
>>           3  call MPI_SEND           3           1           3           0
>> [cli_3]: aborting job:
>> Fatal error in MPI_Send: Other MPI error, error stack:
>> MPI_Send(175): MPI_Send(buf=0x2aaab2af2018, count=1, dtype=USER<vector>,
>> dest=3, tag=30003, MPI_COMM_WORLD) failed
>> MPID_Send(57): DEADLOCK: attempting to send a message to the local process
>> without a prior matching receive
>> ###############
>>
>> -akshar
>>
>> On Tue, Sep 13, 2011 at 10:02 PM, akshar bhosale <
>> akshar.bhosale at gmail.com <mailto:akshar.bhosale at gmail.**com<akshar.bhosale at gmail.com>>>
>> wrote:
>>
>>    thanks for reply.
>>
>>    Goal is to generate mpif90 using mpich-1.2.7p1 using gfortran which
>>    will generate executable from our fortran code.
>>
>>    source files to be compiled are .f and .f90 files.
>>
>>    f2c flag is given because mp[if90 was not getting generated. no
>>    other reason. we want mpif90.
>>
>>    Now we have tried giving only FC=gfortran and no other flag of
>>    option except prefix, but still it is not generating mpif90 nor mpif77.
>>
>>    mpif90 generated  by using [export F90=fortran;F90FLAGS="-ff2c".]
>>    options if tries to compile our code, it throws error
>>
>>
>>    <path to mpif90>/mpif90 -f90=gfortran -Wl -o ../output_code.x set.o
>>    inp.o otp.o bvsol.o user_kg.o mpi_util.o mpi_communication.o
>>    mpi_memalloc.o driver.o rxn.o advdi_1d_finitediff_mf.o  aq_lib.o \
>>            -L/ioapi/Linux2_x86_64gfort -lioapi
>>    -L/np_cs/netcdf/INSTALL/lib -lnetcdf -L/usr/lib64/
>>     -lgomp /np_cs/lapack/lapack-3.2.1/**lapack_LINUX.a
>>    /np_cs/lapack/lapack-3.2.1/**blas_LINUX.a
>>
>>    /build/lib/libmpich.a(farg.o): In function `mpir_iargc__':
>>    farg.f:(.text+0x6): undefined reference to `f__xargc'
>>    /build/lib/libmpich.a(farg.o): In function `mpir_getarg__':
>>    farg.f:(.text+0x38): undefined reference to `getarg_'
>>    collect2: ld returned 1 exit status
>>    make: *** [../output_code.x] Error 1
>>
>>    -akshar
>>
>>
>>
>>
>>
>>
>>    On Tue, Sep 13, 2011 at 8:52 PM, Gus Correa <gus at ldeo.columbia.edu
>>    <mailto:gus at ldeo.columbia.edu>**> wrote:
>>
>>        Hi Pavan and Akshar
>>
>>        I see, it's MPICH-1.
>>        May I ask why?
>>
>>        Would a workaround perhaps be to build only mpif77 (no mpif90),
>>        replace 'use mpi' by '#include "mpif.h"' in the application code,
>>        and recompile it with mpif77?
>>
>>        I wonder if -ff2c works at all with Fortran90.
>>        (See man page detail below.)
>>
>>        Anyway, I remember that there were complaints of runtime
>>        errors (P4 errors) with mpich-1,
>>        reported on this and other mailing lists,
>>        when running on current Linux kernels.
>>        Even with simple codes like the cpi example.
>>        Using MPICH-1 may be an uphill battle.
>>
>>        I hope this helps,
>>        Gus Correa
>>
>>         >From man gfortran (4.1.2):
>>
>>              -ff2c
>>                  Generate code designed to be compatible with code
>>        generated by g77 and f2c.
>>
>>                  The calling conventions used by g77 (originally
>>        implemented in f2c) require functions that return type default
>>        "REAL" to actually return the C type
>>                  "double", and functions that return type "COMPLEX" to
>>        return the values via an extra argument in the calling sequence
>>        that points to where to store the
>>                  return value.  Under the default GNU calling
>>        conventions, such functions simply return their results as they
>>        would in GNU C -- default "REAL" functions
>>                  return the C type "float", and "COMPLEX" functions
>>        return the GNU C type "complex".  Additionally, this option
>>        implies the -fsecond-underscore option,
>>                  unless -fno-second-underscore is explicitly requested.
>>
>>                  This does not affect the generation of code that
>>        interfaces with the libgfortran library.
>>
>>                  Caution: It is not a good idea to mix Fortran code
>>        compiled with "-ff2c" with code compiled with the default
>>        "-fno-f2c" calling conventions as, calling
>>                  "COMPLEX" or default "REAL" functions between program
>>        parts which were compiled with different calling conventions
>>        will break at execution time.
>>
>>                  Caution: This will break code which passes intrinsic
>>        functions of type default "REAL" or "COMPLEX" as actual
>>        arguments, as the library implementations
>>                  use the -fno-f2c calling conventions.
>>
>>
>>
>>
>>        Pavan Balaji wrote:
>>
>>
>>            Gus: He doesn't want to use MPICH2. He's using MPICH-1.
>>
>>                   -- Pavan
>>
>>            On 09/13/2011 09:51 AM, Gus Correa wrote:
>>
>>                Hi Akshar
>>
>>                Maybe the F90FLAGS="-ff2c" flag is causing the problem?
>>                What if you remove it and use gfortran for F77 and FC?
>>
>>
>>                Also, my 'configure -help' (a bit old, MPICH2 1.3.2p1)
>>                mentions FC, not F90, as the environment variable to be
>> set.
>>
>>                For what it is worth, MPICH2 builds fine for us, with
>>                gcc, g++ and gfortran (for F77 and FC).
>>
>>                My two cents,
>>                Gus Correa
>>
>>                akshar bhosale wrote:
>>
>>                    Hi,
>>
>>                       If i specify F77 anf F90 as gfortran, it doesn't
>>                    create mpif90 and
>>                    mpif77 at all.
>>                    Please see attached config log
>>                    If i specify F90=gfortran and F90FLAGS="-ff2c", the
>>                    it creates mpif90
>>                    and mpif77 but is generating below issue.
>>                    Kindly help.
>>                    -akshar
>>
>>                    On Tue, Sep 13, 2011 at 9:46 AM, Rajeev
>>                    Thakur<thakur at mcs.anl.gov <mailto:thakur at mcs.anl.gov>
>>                    <mailto:thakur at mcs.anl.gov
>>                    <mailto:thakur at mcs.anl.gov>>>  wrote:
>>
>>                        In the configure step, if you specify both F77
>>                    and F90 as gfortran,
>>                        it should work.
>>
>>                        Rajeev
>>
>>                        On Sep 12, 2011, at 9:49 PM, akshar bhosale wrote:
>>
>>                         >  Hi,
>>                         >  thanks.
>>                         >  but, in case of mpicc, it can be ignored,
>>                    but in case of mpif90,
>>                        it is still an issue.
>>                         >  e.g. hello_mpi.f90 program
>>                         >  ####################
>>                         >  program main
>>                         >     use mpi
>>                         >  !
>>                         >  ! implicit none
>>                         >  !
>>                         >    integer error
>>                         >    integer id
>>                         >    integer p
>>                         >    real ( kind = 8 ) wtime
>>                         >  !
>>                         >  !  Initialize MPI.
>>                         >  !
>>                         >    call MPI_Init ( error )
>>                         >  !
>>                         >  !  Get the number of processes.
>>                         >  !
>>                         >    call MPI_Comm_size ( MPI_COMM_WORLD, p,
>>                    error )
>>                         >
>>                         >  !
>>                         >  !  Get the individual process ID.
>>                         >  !
>>                         >    call MPI_Comm_rank ( MPI_COMM_WORLD, id,
>>                    error )
>>                         >  !
>>                         >  !  Print a message.
>>                         >  !
>>                         >    if ( id == 0 ) then
>>                         >
>>                         >      wtime = MPI_Wtime ( )
>>                         >
>>                         >      write ( *, '(a)' ) ' '
>>                         >      write ( *, '(a)' ) 'HELLO_MPI - Master
>>                    process:'
>>                         >      write ( *, '(a)' ) '  FORTRAN90/MPI
>> version'
>>                         >      write ( *, '(a)' ) ' '
>>                         >      write ( *, '(a)' ) '  An MPI test program.'
>>                         >      write ( *, '(a)' ) ' '
>>                         >      write ( *, '(a,i8)' ) '  The number of
>>                    processes is ', p
>>                         >      write ( *, '(a)' ) ' '
>>                         >
>>                         >    end if
>>                         >
>>                         >    write ( *, '(a)' ) ' '
>>                         >    write ( *, '(a,i8,a)' ) '  Process ', id,
>>                    ' says "Hello, world!"'
>>                         >
>>                         >    if ( id == 0 ) then
>>                         >
>>                         >      write ( *, '(a)' ) ' '
>>                         >      write ( *, '(a)' ) 'HELLO_MPI - Master
>>                    process:'
>>                         >      write ( *, '(a)' ) '  Normal end of
>>                    execution: "Goodbye,
>>                        world!".'
>>                         >
>>                         >      wtime = MPI_Wtime ( ) - wtime
>>                         >      write ( *, '(a)' ) ' '
>>                         >      write ( *, '(a,g14.6,a)' ) '  Elapsed
>>                    wall clock time = ',
>>                        wtime, ' seconds.'
>>                         >
>>                         >     end if
>>                         >   !
>>                         >   !  Shut down MPI.
>>                         >   !
>>                         >    call MPI_Finalize ( error )
>>                         >
>>                         >    stop
>>                         >    end
>>                         >  ############################
>>                         >
>>                         >  $ ../../../build/bin/mpif90 hello_mpi.f90
>>                         >
>>                     /scratch1/np_cs/trial1/build/_**
>> _lib/libmpich.a(farg.o):
>>                    In function
>>                        `mpir_iargc__':
>>                         >  farg.f:(.text+0x6): undefined reference to
>>                    `f__xargc'
>>                         >
>>                     /scratch1/np_cs/trial1/build/_**
>> _lib/libmpich.a(farg.o):
>>                    In function
>>                        `mpir_getarg__':
>>                         >  farg.f:(.text+0x38): undefined reference to
>>                    `getarg_'
>>                         >  collect2: ld returned 1 exit status
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >
>>                         >  On Mon, Sep 12, 2011 at 9:42 AM, Rajeev Thakur
>>                        <thakur at mcs.anl.gov
>>                    <mailto:thakur at mcs.anl.gov><**mailto:
>> tha__kur at mcs.anl.gov
>>
>>                    <mailto:thakur at mcs.anl.gov>>>  wrote:
>>                         >  Ignore that. Compile a real program as mpicc
>>                    foo.c.
>>                         >
>>                         >  On Sep 11, 2011, at 8:38 PM, akshar bhosale
>>                    wrote:
>>                         >
>>                         >  >  Hi,
>>                         >  >  thank you for reply.
>>                         >  >
>>                         >  >  I tried to install it make it using
>>                    following options now and i
>>                        get following error with undefined reference to
>>                    main. options given
>>                        while configuring the same are :
>>                         >  >  export F90=fortran;F90FLAGS="-ff2c".
>>                         >  >  but error of undefined reference to main
>>                    is observed in every
>>                        executable like :
>>                         >  >
>>                         >  >  ./mpicc -v
>>                         >  >  mpicc for 1.2.7 (release) of : 2005/06/22
>>                    16:33:49
>>                         >  >  Using built-in specs.
>>                         >  >  Target: x86_64-redhat-linux
>>                         >  >  Configured with: ../configure --prefix=/usr
>>                        --mandir=/usr/share/man
>>                    --infodir=/usr/share/info --enable-shared
>>                        --enable-threads=posix --enable-checking=release
>>                    --with-system-zlib
>>                        --enable-__cxa_atexit
>> --disable-libunwind-exceptions
>>                        --enable-libgcj-multifile
>>                                           --enable-languages=c,c++,objc,*
>> *__obj-c++,java,fortran,ada
>>                        --enable-java-awt=gtk --disable-dssi
>>                    --disable-plugin
>>                                           --with-java-home=/usr/lib/jvm/*
>> *__java-1.4.2-gcj-1.4.2.0/jre
>>                        --with-cpu=generic --host=x86_64-redhat-linux
>>                         >  >  Thread model: posix
>>                         >  >  gcc version 4.1.2 20080704 (Red Hat
>> 4.1.2-50)
>>                         >  >
>>  /usr/libexec/gcc/x86_64-__**redhat-linux/4.1.2/collect2
>>                        --eh-frame-hdr -m elf_x86_64 --hash-style=gnu
>>                    -dynamic-linker
>>                        /lib64/ld-linux-x86-64.so.2
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crt1.o
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crti.o
>>                        /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/crtbegin.o
>>                        -L/scratch1/np_cs/trial2/__**build/lib
>>                        -L/opt/intel/mkl/10.2.2.025/__**lib/em64t<http://10.2.2.025/__lib/em64t>
>>                    <http://10.2.2.025/lib/em64t><**
>> http://10.2.2.025/__lib/em64t <http://10.2.2.025/__lib/em64t>
>>
>>                    <http://10.2.2.025/lib/em64t>>
>>                        -L/opt/intel/mkl/10.2.2.025/__**lib/em64t<http://10.2.2.025/__lib/em64t>
>>                    <http://10.2.2.025/lib/em64t><**
>> http://10.2.2.025/__lib/em64t <http://10.2.2.025/__lib/em64t>
>>
>>                    <http://10.2.2.025/lib/em64t>>
>>                        -L/usr/lib/gcc/x86_64-redhat-_**_linux/4.1.2
>>                        -L/usr/lib/gcc/x86_64-redhat-_**_linux/4.1.2
>>                                           -L/usr/lib/gcc/x86_64-redhat-_*
>> *_linux/4.1.2/../../../../lib64
>>                        -L/lib/../lib64 -L/usr/lib/../lib64 -lmpich
>>                    -lpthread -lrt -lgcc
>>                        --as-needed -lgcc_s --no-as-needed -lc -lgcc
>>                    --as-needed -lgcc_s
>>                        --no-as-needed
>>                    /usr/lib/gcc/x86_64-redhat-__**linux/4.1.2/crtend.o
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crtn.o
>>                         >  >
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crt1.o:
>>                    In
>>                        function `_start':
>>                         >  >  (.text+0x20): undefined reference to `main'
>>                         >  >  collect2: ld returned 1 exit status
>>                         >  >
>>                         >  >  ./mpif90 -v
>>                         >  >  mpif90 for 1.2.7 (release) of :
>>                    2005/06/22 16:33:49
>>                         >  >  Driving: gfortran
>>                    -L/scratch1/np_cs/trial2/__**build/lib -v
>>                        -lmpichf90 -lmpich -lpthread -lrt
>>                    -lgfortranbegin -lgfortran -lm
>>                        -shared-libgcc
>>                         >  >  Using built-in specs.
>>                         >  >  Target: x86_64-redhat-linux
>>                         >  >  Configured with: ../configure --prefix=/usr
>>                        --mandir=/usr/share/man
>>                    --infodir=/usr/share/info --enable-shared
>>                        --enable-threads=posix --enable-checking=release
>>                    --with-system-zlib
>>                        --enable-__cxa_atexit
>> --disable-libunwind-exceptions
>>                        --enable-libgcj-multifile
>>                                           --enable-languages=c,c++,objc,*
>> *__obj-c++,java,fortran,ada
>>                        --enable-java-awt=gtk --disable-dssi
>>                    --disable-plugin
>>                                           --with-java-home=/usr/lib/jvm/*
>> *__java-1.4.2-gcj-1.4.2.0/jre
>>                        --with-cpu=generic --host=x86_64-redhat-linux
>>                         >  >  Thread model: posix
>>                         >  >  gcc version 4.1.2 20080704 (Red Hat
>> 4.1.2-50)
>>                         >  >
>>  /usr/libexec/gcc/x86_64-__**redhat-linux/4.1.2/collect2
>>                        --eh-frame-hdr -m elf_x86_64 --hash-style=gnu
>>                    -dynamic-linker
>>                        /lib64/ld-linux-x86-64.so.2
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crt1.o
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crti.o
>>                        /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/crtbegin.o
>>                        -L/scratch1/np_cs/trial2/__**build/lib
>>                        -L/opt/intel/mkl/10.2.2.025/__**lib/em64t<http://10.2.2.025/__lib/em64t>
>>                    <http://10.2.2.025/lib/em64t><**
>> http://10.2.2.025/__lib/em64t <http://10.2.2.025/__lib/em64t>
>>
>>                    <http://10.2.2.025/lib/em64t>>
>>                        -L/opt/intel/mkl/10.2.2.025/__**lib/em64t<http://10.2.2.025/__lib/em64t>
>>                    <http://10.2.2.025/lib/em64t><**
>> http://10.2.2.025/__lib/em64t <http://10.2.2.025/__lib/em64t>
>>
>>                    <http://10.2.2.025/lib/em64t>>
>>                        -L/usr/lib/gcc/x86_64-redhat-_**_linux/4.1.2
>>                        -L/usr/lib/gcc/x86_64-redhat-_**_linux/4.1.2
>>                                           -L/usr/lib/gcc/x86_64-redhat-_*
>> *_linux/4.1.2/../../../../lib64
>>                        -L/lib/../lib64 -L/usr/lib/../lib64 -lmpichf90
>>                    -lmpich -lpthread
>>                        -lrt -lgfortranbegin -lgfortran -lm -lgcc_s
>>                    -lgcc -lc -lgcc_s -lgcc
>>                        /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/crtend.o
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/../../../../lib64/**__crtn.o
>>                         >  >
>>                                           /usr/lib/gcc/x86_64-redhat-__**
>> linux/4.1.2/libgfortranbegin._**_a(fmain.o):
>>                        In function `main':
>>                         >  >  (.text+0xa): undefined reference to `MAIN__'
>>                         >  >  collect2: ld returned 1 exit status
>>                         >  >
>>                         >  >  Am i missing somethin?
>>                         >  >  thank you
>>                         >  >  -akshar
>>                         >  >
>>                         >  >  On Mon, Sep 12, 2011 at 1:24 AM, William
>>                    Gropp
>>                        <wgropp at illinois.edu
>>                    <mailto:wgropp at illinois.edu><**mailto:
>> wg__ropp at illinois.edu
>>
>>                    <mailto:wgropp at illinois.edu>>>  wrote:
>>                         >  >  You can use gfortran for F77 and F90.  If
>>                    your requirement is
>>                        to use gfortran for F77 and a different fortran
>>                    for F90, then they
>>                        will need to use consistent linker names,
>>                    calling conventions, data
>>                        sizes, and libraries.  The config.log file will
>>                    have details on why
>>                        configure decided that the Fortran 77 and
>>                    Fortran 90 compilers were
>>                        not compatible.
>>                         >  >
>>                         >  >  Bill
>>                         >  >
>>                         >  >
>>                         >  >  On Sep 9, 2011, at 4:28 PM, akshar
>>                    bhosale wrote:
>>                         >  >
>>                         >  >  hi,
>>                         >  >  we have requirement of mpich-1.2.7 with
>>                    gfortran. We have fiven
>>                        flags as : CC=gcc;F90=fortran;F77=__**gfortran;
>>                    and tried to configure,
>>                        it throws error as configure: error: Fortran 90
>>                    and Fortran 77
>>                        compilers are not compatible..
>>                         >  >  We have to do it using gfortran only. os
>>                    is rhel5.1 64 bit
>>                         >  >  what extran flags settings we have to do?
>>                         >  >
>>                         >  >  -akshar
>>                         >  >
>>                     ______________________________**___________________
>>                         >  >  mpich-discuss mailing list
>>      mpich-discuss at mcs.anl.gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                        <mailto:mpich-discuss at mcs.anl.**__gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >>
>>                         >  >  To manage subscription options or
>>                    unsubscribe:
>>                         >  >
>>                     https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                    <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>                         >  >
>>                         >  >  William Gropp
>>                         >  >  Director, Parallel Computing Institute
>>                         >  >  Deputy Director for Research
>>                         >  >  Institute for Advanced Computing
>>                    Applications and Technologies
>>                         >  >  Paul and Cynthia Saylor Professor of
>>                    Computer Science
>>                         >  >  University of Illinois Urbana-Champaign
>>                         >  >
>>                         >  >
>>                         >  >
>>                         >  >
>>                         >  >
>>                     ______________________________**___________________
>>                         >  >  mpich-discuss mailing list
>>      mpich-discuss at mcs.anl.gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                        <mailto:mpich-discuss at mcs.anl.**__gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >>
>>                         >  >  To manage subscription options or
>>                    unsubscribe:
>>                         >  >
>>                     https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                    <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>                         >
>>                         >
>>                     ______________________________**___________________
>>                         >  mpich-discuss mailing list
>>    mpich-discuss at mcs.anl.gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                        <mailto:mpich-discuss at mcs.anl.**__gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >>
>>                         >  To manage subscription options or unsubscribe:
>>                         >
>>                     https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                    <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>                         >
>>                         >
>>                     ______________________________**___________________
>>                         >  mpich-discuss mailing list
>>    mpich-discuss at mcs.anl.gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                        <mailto:mpich-discuss at mcs.anl.**__gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >>
>>                         >  To manage subscription options or unsubscribe:
>>                         >
>>                     https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                    <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>
>>                        ______________________________**
>> ___________________
>>                        mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                        <mailto:mpich-discuss at mcs.anl.**__gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >>
>>                        To manage subscription options or unsubscribe:
>>                                           https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                    <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>
>>
>>
>>                    ------------------------------**
>> __----------------------------**--__------------
>>
>>                    ______________________________**___________________
>>                    mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>>                    <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                    To manage subscription options or unsubscribe:
>>                    https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                    <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>
>>
>>                ______________________________**___________________
>>                mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>>                <mailto:mpich-discuss at mcs.anl.**gov<mpich-discuss at mcs.anl.gov>
>> >
>>                To manage subscription options or unsubscribe:
>>                https://lists.mcs.anl.gov/__**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>                <https://lists.mcs.anl.gov/**
>> mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>
>>
>>
>>        ______________________________**___________________
>>        mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>>        <mailto:mpich-discuss at mcs.anl.**gov <mpich-discuss at mcs.anl.gov>>
>>        To manage subscription options or unsubscribe:
>>        https://lists.mcs.anl.gov/__**mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss>
>>        <https://lists.mcs.anl.gov/**mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>> **>
>>
>>
>>
>>
>> ------------------------------**------------------------------**
>> ------------
>>
>> ______________________________**_________________
>> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
>> To manage subscription options or unsubscribe:
>> https://lists.mcs.anl.gov/**mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>>
>
> ______________________________**_________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/**mailman/listinfo/mpich-discuss<https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20110914/6ed3df7f/attachment-0001.htm>


More information about the mpich-discuss mailing list