[mpich-discuss] undefined symbol: MPIR_Thread

Anthony Chan chan at mcs.anl.gov
Thu Feb 3 09:55:50 CST 2011 

In your ccsm build log, you have this command and output:

/usr/local/mpich2-1.3.1/bin/mpif90 -o /usr/workdir/cesm1_0_2/scripts/test1/run/ccsm.exe ccsm_driver.o map_atmatm_mct.o map_atmice_mct.o map_atmlnd_mct.o map_atmocn_mct.o map_glcglc_mct.o map_iceice_mct.o map_iceocn_mct.o map_lndlnd_mct.o map_ocnocn_mct.o map_rofocn_mct.o map_rofrof_mct.o map_snoglc_mct.o map_snosno_mct.o mrg_x2a_mct.o mrg_x2g_mct.o mrg_x2i_mct.o mrg_x2l_mct.o mrg_x2o_mct.o mrg_x2s_mct.o seq_avdata_mod.o seq_diag_mct.o seq_domain_mct.o seq_flux_mct.o seq_frac_mct.o seq_hist_mod.o seq_rearr_mod.o seq_rest_mod.o -L/usr/workdir/cesm1_0_2/scripts/test1/lib -latm -llnd -lice -locn -lglc -L/usr/workdir/cesm1_0_2/scripts/test1/lib -lcsm_share -lmct -lmpeu -lpio  -L/usr/local/netcdf-4.1.1/lib -lnetcdf -lnetcdff -L/usr/local/hdf5/lib -lhdf5_hl -lhdf5  -L/usr/local/zlib/lib -lz -lm -lnetcdf   -L/usr/local/lib -lcurl 
/usr/bin/ld: Warning: alignment 16 of symbol `mpifcmb4_' in /usr/local/lib/libmpich.a(setbot.o) is smaller than 32 in ccsm_driver.o
/usr/bin/ld: Warning: alignment 16 of symbol `mpifcmb6_' in /usr/local/lib/libmpich.a(setbot.o) is smaller than 32 in ccsm_driver.o
/usr/bin/ld: Warning: alignment 16 of symbol `mpifcmb8_' in /usr/local/lib/libmpich.a(setbot.o) is smaller than 32 in ccsm_driver.o

Notice that the warning is about /usr/local/lib/libmpich.a which is mpich2 from ubuntu repository (I think
mpich2-1.3.1's libmpich.a is in /usr/local/mpich2-1.3.1/lib/libmpich.a), right.  Since you need libcurl.a(.so)
from /usr/local/lib, you HAVE TO remove /usr/local/lib/libmpich.a, otherwise mpif90 will use libmpich.a from
your /usr/local/lib first..

A.Chan

----- Original Message -----
> yes, ccsm.exe is created again, as a new folder is created each time I
> run the building process (meaning that the old one is removed).
> So once the compilation starts, all is stored in folder /test1/....
> 
> In attach the log file for creating the ccsm executable, as mct is not
> the module that creates the ccsm.exe file.
> 
> On 02/03/2011 04:34 PM, Anthony Chan wrote:
> > Your MCT build log does not contain the link command that creates
> > ccsm.exe,
> > so I am not sure if ccsm.exe has actually been recreated with the
> > mpich2-1.3.1,
> > i.e. you could be still use the old ccsm.exe that was linked with
> > ubuntu mpich.
> >
> > So you need to remove all .o, .a (or .so) and ccsm.exe (or all .exe)
> > files before
> > compiling again to make sure everything is compiled and linked
> > correctly with
> > mpich2-1.3.1...
> >
> > A.Chan
> >
> > ----- Original Message -----
> >> Ok!
> >>
> >> I realised that in first instance I installed mpich via the ubuntu
> >> repository, and afterwards i built it myself.
> >> That was the reason why I had another MPI installation present.
> >>
> >> So I un-installed the first version, and built the other one again,
> >> to
> >> be sure.
> >> I have set the references as you said:
> >>
> >> FC := /usr/local/mpich2-1.3.1/bin/mpif90
> >> CC := /usr/local/mpich2-1.3.1/bin/mpicc
> >> F90 := /usr/local/mpich2-1.3.1/bin/mpif90
> >> MPICC := /usr/local/mpich2-1.3.1/bin/mpicc
> >> MPIF90 := /usr/local/mpich2-1.3.1/bin/mpif90
> >>
> >> and compiled the model again. This went ok, an executable was
> >> created
> >> (see attach for example of one of the modules)
> >> Running the executable unfortunately gave me the same error:
> >> ./ccsm.exe: symbol lookup error: ./ccsm.exe: undefined symbol:
> >> MPIR_Thread
> >>
> >> So back to square one?
> >>
> >> Any new ideas?
> >> Matthias
> >>
> >>
> >> On 02/02/2011 07:08 PM, Anthony Chan wrote:
> >>> You indicated mpich2-1.3.1 was installed in
> >>> /usr/local/mpich2-1.3.1
> >>> so /usr/local/bin/mpif90 refers to mpif90 from other MPI
> >>> implementation.
> >>> So you MUST specify fullpath to mpif90 in all your build scripts
> >>> and/or
> >>> makefiles. I also notice your bldlog uses compiler variables
> >>> besides
> >>> FC and CC,
> >>> i.e. MPICC, MPIF90, F90..., and some of these fortran
> >>>
> >>> CC = $(MPICH_PATH)/bin/mpicc
> >>> MPICC = $(MPICH_PATH)/bin/mpicc
> >>> FC = $(MPICH_PATH)/bin/mpif90
> >>> F90 = $(MPICH_PATH)/bin/mpif90
> >>> MPIF90 = $(MPICH_PATH)/bin/mpif90
> >>>
> >>> and leave LIB_MPI and INC_MPI blank (that would help you locate
> >>> which script
> >>> and/or makefile that set FC to "gfortran" or "mpif90").
> >>>
> >>> The other option is to simply prefix your PATH with
> >>> /usr/local/mpich2-1.3.1 to
> >>> avoid /usr/local/bin's MPI compiler wrappers being used in CCSM
> >>> build system,
> >>> but then you have to check your environment every time running
> >>> CCSM
> >>> build system,
> >>> and your build output won't show correct MPI implementation is
> >>> being
> >>> used or not.
> >>> So I think you should follow the 1st approach that was suggested.
> >>>
> >>> A.Chan
> >>>
> >>> ----- Original Message -----
> >>>> The files that I send were indeed with the LIB and INC
> >>>> definitions
> >>>> included.
> >>>> But I did the test as well without these indications (and setting
> >>>> FC
> >>>> and
> >>>> CC to MPICH2/etc/etc/...), and the results were the same?
> >>>> see example built log in attach.
> >>>>
> >>>> The "which mpif90" statement gives me:
> >>>> /usr/local/bin/mpif90
> >>>>
> >>>> Matthias
> >>>>
> >>>> On 02/02/2011 04:48 PM, Anthony Chan wrote:
> >>>>> All your bldlog files indicate that you set FC to mpif90 and
> >>>>> NOT $(MPICH_PATH)/bin/mpif90 as I suggested earlier. You should
> >>>>> set both LIB_MPI and INC_MPI to empty strings. Do you have other
> >>>>> MPI implementations (older MPICH2 or other MPI implementations)
> >>>>> installed on your system besides MPICH2-1.3.1 ? Do a "which
> >>>>> mpif90"
> >>>>> to find out.
> >>>>>
> >>>>> A.Chan
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> Dear Anthony!
> >>>>>>
> >>>>>> I have now done what you suggested below:
> >>>>>> Set FC and CC to mpif90 and mpicc and removing the inc_ and
> >>>>>> lib_
> >>>>>> references.
> >>>>>> Again, I could built the executable, but running the latter
> >>>>>> results
> >>>>>> in
> >>>>>> ./ccsm.exe: undefined symbol: MPIR_Thread
> >>>>>>
> >>>>>> Therefore, I set everything back to before and added this to
> >>>>>> the
> >>>>>> LIB_MPI
> >>>>>> LIB_MPI := $(MPICH_PATH)/lib -lmpichf90 -lmpichf90 -lmpich
> >>>>>> -lopa
> >>>>>> -lmpl -lrt -lpthread
> >>>>>>
> >>>>>> As the last arguments were shown by mpif90 -show. But again,
> >>>>>> same
> >>>>>> error.
> >>>>>>
> >>>>>> For every module of the model, a log file is saved. I attached
> >>>>>> the
> >>>>>> 3
> >>>>>> most important ones:
> >>>>>> - pio: parallel IO
> >>>>>> - lnd: the land module that I would like to test
> >>>>>> - ccsm: the log for building the executable
> >>>>>>
> >>>>>> As far as I can see, there could be something in pio, but I'm
> >>>>>> not
> >>>>>> sure,
> >>>>>> and it doesn't seem to be enough to stop the compilation
> >>>>>> process.
> >>>>>>
> >>>>>> Wouldn't there be any other way to deal with this thread
> >>>>>> problem?
> >>>>>> Kind regards,
> >>>>>>
> >>>>>> Matthias
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 02/02/2011 01:12 AM, Anthony Chan wrote:
> >>>>>>> I took a quick look at the scripts that you sent but they
> >>>>>>> ain't
> >>>>>>> very
> >>>>>>> helpful
> >>>>>>> as these files source some other files which ain't included in
> >>>>>>> your
> >>>>>>> attachment.
> >>>>>>> In any case, my knowledge on CCSM is limited. Since the build
> >>>>>>> problem is CCSM
> >>>>>>> specific, you may want to contact CCSM developers to see what
> >>>>>>> it
> >>>>>>> should be done.
> >>>>>>>
> >>>>>>> Given your final executable (or one of the executables) has
> >>>>>>> unresolved symbol
> >>>>>>> MPIR_Thread, so it seems it didn't link in MPICH2 libraries
> >>>>>>> correctly.
> >>>>>>> I would first set FC=to $(MPICH_PATH)/bin/mpif90 and
> >>>>>>> CC=$(MPICH_PATH)/bin/mpicc
> >>>>>>> and leave LIB_MPI and INC_MPI empty since mpif90 and mpicc
> >>>>>>> should
> >>>>>>> take care of
> >>>>>>> LIB_MPI and INC_MPI. You may want to save the compile and
> >>>>>>> linking
> >>>>>>> output of
> >>>>>>> whole build process to see what goes wrong.
> >>>>>>>
> >>>>>>> A.Chan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>>> mmm, I am not sure if I can answer this question.
> >>>>>>>> It is a global climate model, with a lot of packages, and in
> >>>>>>>> order
> >>>>>>>> to
> >>>>>>>> compile it I follow the installation guide of the model.
> >>>>>>>> So I am not completely sure what I should provide you with?
> >>>>>>>>
> >>>>>>>> There is a seperate Macros files in which I have to adapt
> >>>>>>>> libs
> >>>>>>>> and
> >>>>>>>> flags
> >>>>>>>> for my system. The contents are as follows:
> >>>>>>>> # Note that CPPDEFS is set in Macros.cppdefs
> >>>>>>>> CPPDEFS += -DLINUX -DSEQ_$(FRAMEWORK) -DFORTRANUNDERSCORE
> >>>>>>>> -DNO_R16
> >>>>>>>> -DNO_SHR_VMATH
> >>>>>>>>
> >>>>>>>> ifeq ($(compile_threaded), true)
> >>>>>>>> CPPDEFS += -DTHREADED_OMP
> >>>>>>>> endif
> >>>>>>>>
> >>>>>>>> FC := mpif90
> >>>>>>>> CC := mpicc
> >>>>>>>>
> >>>>>>>> HDF5_PATH := /usr/local/hdf5
> >>>>>>>> HDF5_LIB := $(HDF5_PATH)/lib -lhdf5 -lhdf5_hl
> >>>>>>>> NETCDF_PATH := /usr/local/netcdf-4.1.1
> >>>>>>>> INC_NETCDF := $(NETCDF_PATH)/include
> >>>>>>>> LIB_NETCDF := $(NETCDF_PATH)/lib -lnetcdf
> >>>>>>>> -L/usr/local/hdf5/lib
> >>>>>>>> -lhdf5_hl -lhdf5 -L/usr/local/zlib/lib -lz -lm
> >>>>>>>> MOD_NETCDF := $(NETCDF_PATH)/include
> >>>>>>>> MPICH_PATH := /usr/local/mpich2-1.3.1
> >>>>>>>> INC_MPI := $(MPICH_PATH)/include
> >>>>>>>> LIB_MPI := $(MPICH_PATH)/lib
> >>>>>>>> MPI_LIB_NAME := mpich
> >>>>>>>> PNETCDF_PATH :=
> >>>>>>>> INC_PNETCDF :=
> >>>>>>>> LIB_PNETCDF :=
> >>>>>>>> LAPACK_LIBDIR :=
> >>>>>>>> #=====================================================
> >>>>>>>>
> >>>>>>>> CFLAGS := $(CPPDEFS)
> >>>>>>>> FIXEDFLAGS := -132
> >>>>>>>> FREEFLAGS := -FR
> >>>>>>>> ##FFLAGS := $(CPPDEFS) -g -fp-model precise -convert
> >>>>>>>> big_endian
> >>>>>>>> -assume byterecl -ftz -traceback
> >>>>>>>> FFLAGS := $(CPPDEFS)-g -O2 -I/usr/local/include
> >>>>>>>> FFLAGS_NOOPT := $(FFLAGS) -O0
> >>>>>>>> FFLAGS_OPT := -O2
> >>>>>>>> LDFLAGS :=
> >>>>>>>> AR := ar
> >>>>>>>> MOD_SUFFIX := mod
> >>>>>>>> CONFIG_SHELL :=
> >>>>>>>>
> >>>>>>>> So maybe something is wrong here???
> >>>>>>>> In attach I also sent you the configure file, and the build
> >>>>>>>> file
> >>>>>>>> to
> >>>>>>>> compile the model.
> >>>>>>>> That works all fine. Only when running the obtained
> >>>>>>>> executable,
> >>>>>>>> I
> >>>>>>>> get
> >>>>>>>> the error MPIR_Thread...
> >>>>>>>> to run the .exe, I use the following:
> >>>>>>>> mpirun ./ccsm.exe>&! ccsm.log.$LID
> >>>>>>>>
> >>>>>>>> as is mentioned in the test2.Latitude_E6400.run file in
> >>>>>>>> attach.
> >>>>>>>>
> >>>>>>>> I do not know if you can do something with this, if not,
> >>>>>>>> please
> >>>>>>>> let
> >>>>>>>> me
> >>>>>>>> know....
> >>>>>>>>
> >>>>>>>> Matthias
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 02/01/2011 09:17 PM, Anthony Chan wrote:
> >>>>>>>>> ----- Original Message -----
> >>>>>>>>>
> >>>>>>>>>> I can also perform the test as in the installation guide:
> >>>>>>>>>> mpiexec -n 5 ./examples/cpi
> >>>>>>>>>>
> >>>>>>>>>> Process 1 of 5 is on MaffieLE6400
> >>>>>>>>>> Process 3 of 5 is on MaffieLE6400
> >>>>>>>>>> Process 0 of 5 is on MaffieLE6400
> >>>>>>>>>> Process 2 of 5 is on MaffieLE6400
> >>>>>>>>>> Process 4 of 5 is on MaffieLE6400
> >>>>>>>>>> pi is approximately 3.1415926544231230, Error is
> >>>>>>>>>> 0.0000000008333298
> >>>>>>>>>> wall clock time = 0.000435
> >>>>>>>>>>
> >>>>>>>>>> as far as I can see all works fine?
> >>>>>>>>> Yes.
> >>>>>>>>>
> >>>>>>>>>> The reason I configured with the --with-thread-package=none
> >>>>>>>>>> the
> >>>>>>>>>> second
> >>>>>>>>>> time was to deal with the MPIR_Thread error...
> >>>>>>>>>> So I wonder how I can deal with this error?
> >>>>>>>>> Let's just focus on using the mpich2 install that was
> >>>>>>>>> configured
> >>>>>>>>> without
> >>>>>>>>> --with-thread-package. How did you compile your application
> >>>>>>>>> ?
> >>>>>>>>>
> >>>>>>>>>> Thanks for your help!
> >>>>>>>>>>
> >>>>>>>>>> Matthias
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 02/01/2011 08:24 PM, Anthony Chan wrote:
> >>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>> Dear Anthony,
> >>>>>>>>>>>>
> >>>>>>>>>>>> My apologies for posting to the wrong email list. I will
> >>>>>>>>>>>> register
> >>>>>>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>> other one so that in the future I can put my questions
> >>>>>>>>>>>> there.
> >>>>>>>>>>>> For now i'll post this directly to you, hope that is
> >>>>>>>>>>>> fine??
> >>>>>>>>>>> Yes. I cc'ed my reply to mpich-discuss as someone else may
> >>>>>>>>>>> have
> >>>>>>>>>>> things to add.
> >>>>>>>>>>>
> >>>>>>>>>>>> I am not sure if I understand what you mean with:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Did you use mpif90/mpif77 to link your code or just use
> >>>>>>>>>>>> above
> >>>>>>>>>>>> link
> >>>>>>>>>>>> command which seems incomplete ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> I used the following line to check how mpich2 was built,
> >>>>>>>>>>>> as
> >>>>>>>>>>>> I
> >>>>>>>>>>>> first
> >>>>>>>>>>>> got
> >>>>>>>>>>>> an error with f90 vs f95.
> >>>>>>>>>>>> So with
> >>>>>>>>>>>> /usr/local/mpich2-1.3.1/bin/mpif90 -show
> >>>>>>>>>>>>
> >>>>>>>>>>>> I get finally got:
> >>>>>>>>>>>> gfortran -O2 -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>> -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>> -L/usr/local/mpich2-1.3.1/lib
> >>>>>>>>>>>> -lmpichf90 -lmpichf90 -lmpich -lopa -lmpl -lrt -lpthread
> >>>>>>>>>>>>
> >>>>>>>>>>>> Do you mean that there is something missing here? And
> >>>>>>>>>>>> should
> >>>>>>>>>>>> I
> >>>>>>>>>>>> add
> >>>>>>>>>>>> it
> >>>>>>>>>>>> than in my definition of INC_MPI and LIB_MPI?
> >>>>>>>>>>> Let's focus if your mpich2 installation is correct first.
> >>>>>>>>>>>> One other question. I was thinking that it could be
> >>>>>>>>>>>> useful
> >>>>>>>>>>>> to
> >>>>>>>>>>>> built
> >>>>>>>>>>>> mpich2 without threading, so I used:
> >>>>>>>>>>>> ./configure -prefix=/usr/local/mpich2-1.3.1
> >>>>>>>>>>>> --with-thread-package=none
> >>>>>>>>>>> Try "./configure --prefix=/usr/local/mpich2-1.3.1",
> >>>>>>>>>>> without
> >>>>>>>>>>> --with-thread-package.
> >>>>>>>>>>>
> >>>>>>>>>>> A.Chan
> >>>>>>>>>>>
> >>>>>>>>>>>> But then I get a whole list of errors, e.g.:
> >>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getn.o):
> >>>>>>>>>>>> In
> >>>>>>>>>>>> function `MPIU_Thread_CS_exit_lockname_recursive_impl_':
> >>>>>>>>>>>> info_getn.c:(.text+0x124): undefined reference to
> >>>>>>>>>>>> `MPIU_Thread_mutex_unlock'
> >>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getnth.o):
> >>>>>>>>>>>> In
> >>>>>>>>>>>> function `MPIU_Thread_CS_enter_lockname_recursive_impl_':
> >>>>>>>>>>>> info_getnth.c:(.text+0x32): undefined reference to
> >>>>>>>>>>>> `MPIU_Thread_mutex_lock'
> >>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getnth.o):
> >>>>>>>>>>>> In
> >>>>>>>>>>>> function `MPIU_Thread_CS_exit_lockname_recursive_impl_':
> >>>>>>>>>>>> info_getnth.c:(.text+0xf4): undefined reference to
> >>>>>>>>>>>> `MPIU_Thread_mutex_unlock'
> >>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getvallen.o):
> >>>>>>>>>>>> In function
> >>>>>>>>>>>> `MPIU_Thread_CS_enter_lockname_recursive_impl_':
> >>>>>>>>>>>> info_getvallen.c:(.text+0x32): undefined reference to
> >>>>>>>>>>>> `MPIU_Thread_mutex_lock'
> >>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getvallen.o):
> >>>>>>>>>>>> In function
> >>>>>>>>>>>> `MPIU_Thread_CS_exit_lockname_recursive_impl_':
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Any ideas here?
> >>>>>>>>>>>> Matthias
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> -------- Original Message --------
> >>>>>>>>>>>> Subject: Re: [mpich2-dev] undefined symbol: MPIR_Thread
> >>>>>>>>>>>> Date: Tue, 1 Feb 2011 19:41:54 +0100
> >>>>>>>>>>>> From: Anthony Chan<chan at mcs.anl.gov>
> >>>>>>>>>>>> Reply-To: Anthony Chan<chan at mcs.anl.gov>,
> >>>>>>>>>>>> "mpich2-dev at mcs.anl.gov"
> >>>>>>>>>>>> <mpich2-dev at mcs.anl.gov>
> >>>>>>>>>>>> To: mpich2-dev at mcs.anl.gov<mpich2-dev at mcs.anl.gov>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> mpich2-dev for mpich2 development not general MPICH usage
> >>>>>>>>>>>> questions,
> >>>>>>>>>>>> please send your question to mpich-discuss at mcs.anl.gov.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>        All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        when running a model with mpirun ./exe
> >>>>>>>>>>>>>        I get the following error:
> >>>>>>>>>>>>>        ./exe: symbol lookup error: ./exe: undefined
> >>>>>>>>>>>>>        symbol:
> >>>>>>>>>>>>>        MPIR_Thread
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        I have no idea how to deal with this. I use
> >>>>>>>>>>>>>        mpich2-1.3.1,
> >>>>>>>>>>>>>        compiled
> >>>>>>>>>>>>>        with
> >>>>>>>>>>>>>        gfortran, with the following libraries:
> >>>>>>>>>>>>>        gfortran -O2 -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>>>        -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>>>        -L/usr/local/mpich2-1.3.1/lib
> >>>>>>>>>>>>>        -lmpichf90 -lmpichf90 -lmpich -lopa -lmpl -lrt
> >>>>>>>>>>>>>        -lpthread
> >>>>>>>>>>>> Did you use mpif90/mpif77 to link your code or just use
> >>>>>>>>>>>> above
> >>>>>>>>>>>> link
> >>>>>>>>>>>> command which seems incomplete ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> A.Chan
> >>>>>>>>>>>>
> >>>>>>>>>>>>>        Could anybody help me with this?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        Regards,
> >>>>>>>>>>>>>        Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        --
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        -------------------------------------------------------------
> >>>>>>>>>>>>>        FWO Post-doctoral Research Fellow
> >>>>>>>>>>>>>        Department of Earth& Environmental Sciences
> >>>>>>>>>>>>>        Physical and Regional Geography Research Group
> >>>>>>>>>>>>>        Regional climate studies
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        Celestijnenlaan 200E
> >>>>>>>>>>>>>        3001 Heverlee (Leuven)
> >>>>>>>>>>>>>        BELGIUM
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        Tel: + 32 16 326424
> >>>>>>>>>>>>>        Fax: + 32 16 322980
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        http://geo.kuleuven.be/aow/
> >>>>>>>>>>>>>        www.kuleuven.be/geography
> >>>>>>>>>>>>>        -------------------------------------------------------------
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -------------------------------------------------------------
> >>>>>>>>>> FWO Post-doctoral Research Fellow
> >>>>>>>>>> Department of Earth& Environmental Sciences
> >>>>>>>>>> Physical and Regional Geography Research Group
> >>>>>>>>>> Regional climate studies
> >>>>>>>>>>
> >>>>>>>>>> Celestijnenlaan 200E
> >>>>>>>>>> 3001 Heverlee (Leuven)
> >>>>>>>>>> BELGIUM
> >>>>>>>>>>
> >>>>>>>>>> Tel: + 32 16 326424
> >>>>>>>>>> Fax: + 32 16 322980
> >>>>>>>>>>
> >>>>>>>>>> http://geo.kuleuven.be/aow/
> >>>>>>>>>> www.kuleuven.be/geography
> >>>>>>>>>> -------------------------------------------------------------
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -------------------------------------------------------------
> >>>>>>>> FWO Post-doctoral Research Fellow
> >>>>>>>> Department of Earth& Environmental Sciences
> >>>>>>>> Physical and Regional Geography Research Group
> >>>>>>>> Regional climate studies
> >>>>>>>>
> >>>>>>>> Celestijnenlaan 200E
> >>>>>>>> 3001 Heverlee (Leuven)
> >>>>>>>> BELGIUM
> >>>>>>>>
> >>>>>>>> Tel: + 32 16 326424
> >>>>>>>> Fax: + 32 16 322980
> >>>>>>>>
> >>>>>>>> http://geo.kuleuven.be/aow/
> >>>>>>>> www.kuleuven.be/geography
> >>>>>>>> -------------------------------------------------------------
> >>>>>> --
> >>>>>>
> >>>>>>
> >>>>>> -------------------------------------------------------------
> >>>>>> FWO Post-doctoral Research Fellow
> >>>>>> Department of Earth& Environmental Sciences
> >>>>>> Physical and Regional Geography Research Group
> >>>>>> Regional climate studies
> >>>>>>
> >>>>>> Celestijnenlaan 200E
> >>>>>> 3001 Heverlee (Leuven)
> >>>>>> BELGIUM
> >>>>>>
> >>>>>> Tel: + 32 16 326424
> >>>>>> Fax: + 32 16 322980
> >>>>>>
> >>>>>> http://geo.kuleuven.be/aow/
> >>>>>> www.kuleuven.be/geography
> >>>>>> -------------------------------------------------------------
> >>>> --
> >>>>
> >>>>
> >>>> -------------------------------------------------------------
> >>>> FWO Post-doctoral Research Fellow
> >>>> Department of Earth& Environmental Sciences
> >>>> Physical and Regional Geography Research Group
> >>>> Regional climate studies
> >>>>
> >>>> Celestijnenlaan 200E
> >>>> 3001 Heverlee (Leuven)
> >>>> BELGIUM
> >>>>
> >>>> Tel: + 32 16 326424
> >>>> Fax: + 32 16 322980
> >>>>
> >>>> http://geo.kuleuven.be/aow/
> >>>> www.kuleuven.be/geography
> >>>> -------------------------------------------------------------
> >> --
> >>
> >>
> >> -------------------------------------------------------------
> >> FWO Post-doctoral Research Fellow
> >> Department of Earth& Environmental Sciences
> >> Physical and Regional Geography Research Group
> >> Regional climate studies
> >>
> >> Celestijnenlaan 200E
> >> 3001 Heverlee (Leuven)
> >> BELGIUM
> >>
> >> Tel: + 32 16 326424
> >> Fax: + 32 16 322980
> >>
> >> http://geo.kuleuven.be/aow/
> >> www.kuleuven.be/geography
> >> -------------------------------------------------------------
> 
> --
> 
> 
> -------------------------------------------------------------
> FWO Post-doctoral Research Fellow
> Department of Earth& Environmental Sciences
> Physical and Regional Geography Research Group
> Regional climate studies
> 
> Celestijnenlaan 200E
> 3001 Heverlee (Leuven)
> BELGIUM
> 
> Tel: + 32 16 326424
> Fax: + 32 16 322980
> 
> http://geo.kuleuven.be/aow/
> www.kuleuven.be/geography
> -------------------------------------------------------------

More information about the mpich-discuss mailing list