[mpich-discuss] undefined symbol: MPIR_Thread

Anthony Chan chan at mcs.anl.gov
Thu Feb 3 14:38:16 CST 2011 


----- Original Message -----
> I attached the .run script, but end the end this is the command that
> is
> used:
> mpiexec ./ccsm.exe
> 
> or if I just start it in my shell session directly:
> /usr/local/mpich2-1.3.1/bin/mpiexec ./ccsm.exe

Does that work now ?

> 
> 
> 
> 
> 
> On 02/03/2011 07:10 PM, Anthony Chan wrote:
> > What is the exact command you used to launch ccsm.exe ?
> >
> > ----- Original Message -----
> >> MPICH2-1.3.1's default process manager is hydra not mpd,
> >> so you don't need to start mpd... Older mpich2 requires
> >> mpd but not 1.3.1 MPIR_Thread is defined in libmpich.a (AFAICT)
> >> I don't know why your ccsm.exe still says it can't find
> >> MPIR_Thread.
> >>
> >> Is everything in CCSM statically linked ?
> >>
> >>
> >> ----- Original Message -----
> >>> I was wondering whether this has also something to do with the
> >>> problem.
> >>> On a website of a person installing ccsm on his laptop (also using
> >>> mpich2) he states that before running the executable, he has to
> >>> start
> >>> mpi first
> >>>> mpd&
> >>> Now, if I do that, then I get:
> >>> /usr/workdir/cesm1_0_2/scripts$ config: problems opening file
> >>> /etc/mpd.conf for reading: Permission denied
> >>>
> >>> or as root
> >>> listen: Failed to listen on localhost (line 69): Address already
> >>> in
> >>> use:
> >>>
> >>> Does this mean it is already on, or that there is an error?
> >>>
> >>>
> >>> On 02/02/2011 07:08 PM, Anthony Chan wrote:
> >>>> You indicated mpich2-1.3.1 was installed in
> >>>> /usr/local/mpich2-1.3.1
> >>>> so /usr/local/bin/mpif90 refers to mpif90 from other MPI
> >>>> implementation.
> >>>> So you MUST specify fullpath to mpif90 in all your build scripts
> >>>> and/or
> >>>> makefiles. I also notice your bldlog uses compiler variables
> >>>> besides
> >>>> FC and CC,
> >>>> i.e. MPICC, MPIF90, F90..., and some of these fortran
> >>>>
> >>>> CC = $(MPICH_PATH)/bin/mpicc
> >>>> MPICC = $(MPICH_PATH)/bin/mpicc
> >>>> FC = $(MPICH_PATH)/bin/mpif90
> >>>> F90 = $(MPICH_PATH)/bin/mpif90
> >>>> MPIF90 = $(MPICH_PATH)/bin/mpif90
> >>>>
> >>>> and leave LIB_MPI and INC_MPI blank (that would help you locate
> >>>> which script
> >>>> and/or makefile that set FC to "gfortran" or "mpif90").
> >>>>
> >>>> The other option is to simply prefix your PATH with
> >>>> /usr/local/mpich2-1.3.1 to
> >>>> avoid /usr/local/bin's MPI compiler wrappers being used in CCSM
> >>>> build system,
> >>>> but then you have to check your environment every time running
> >>>> CCSM
> >>>> build system,
> >>>> and your build output won't show correct MPI implementation is
> >>>> being
> >>>> used or not.
> >>>> So I think you should follow the 1st approach that was suggested.
> >>>>
> >>>> A.Chan
> >>>>
> >>>> ----- Original Message -----
> >>>>> The files that I send were indeed with the LIB and INC
> >>>>> definitions
> >>>>> included.
> >>>>> But I did the test as well without these indications (and
> >>>>> setting
> >>>>> FC
> >>>>> and
> >>>>> CC to MPICH2/etc/etc/...), and the results were the same?
> >>>>> see example built log in attach.
> >>>>>
> >>>>> The "which mpif90" statement gives me:
> >>>>> /usr/local/bin/mpif90
> >>>>>
> >>>>> Matthias
> >>>>>
> >>>>> On 02/02/2011 04:48 PM, Anthony Chan wrote:
> >>>>>> All your bldlog files indicate that you set FC to mpif90 and
> >>>>>> NOT $(MPICH_PATH)/bin/mpif90 as I suggested earlier. You should
> >>>>>> set both LIB_MPI and INC_MPI to empty strings. Do you have
> >>>>>> other
> >>>>>> MPI implementations (older MPICH2 or other MPI implementations)
> >>>>>> installed on your system besides MPICH2-1.3.1 ? Do a "which
> >>>>>> mpif90"
> >>>>>> to find out.
> >>>>>>
> >>>>>> A.Chan
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> Dear Anthony!
> >>>>>>>
> >>>>>>> I have now done what you suggested below:
> >>>>>>> Set FC and CC to mpif90 and mpicc and removing the inc_ and
> >>>>>>> lib_
> >>>>>>> references.
> >>>>>>> Again, I could built the executable, but running the latter
> >>>>>>> results
> >>>>>>> in
> >>>>>>> ./ccsm.exe: undefined symbol: MPIR_Thread
> >>>>>>>
> >>>>>>> Therefore, I set everything back to before and added this to
> >>>>>>> the
> >>>>>>> LIB_MPI
> >>>>>>> LIB_MPI := $(MPICH_PATH)/lib -lmpichf90 -lmpichf90 -lmpich
> >>>>>>> -lopa
> >>>>>>> -lmpl -lrt -lpthread
> >>>>>>>
> >>>>>>> As the last arguments were shown by mpif90 -show. But again,
> >>>>>>> same
> >>>>>>> error.
> >>>>>>>
> >>>>>>> For every module of the model, a log file is saved. I attached
> >>>>>>> the
> >>>>>>> 3
> >>>>>>> most important ones:
> >>>>>>> - pio: parallel IO
> >>>>>>> - lnd: the land module that I would like to test
> >>>>>>> - ccsm: the log for building the executable
> >>>>>>>
> >>>>>>> As far as I can see, there could be something in pio, but I'm
> >>>>>>> not
> >>>>>>> sure,
> >>>>>>> and it doesn't seem to be enough to stop the compilation
> >>>>>>> process.
> >>>>>>>
> >>>>>>> Wouldn't there be any other way to deal with this thread
> >>>>>>> problem?
> >>>>>>> Kind regards,
> >>>>>>>
> >>>>>>> Matthias
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 02/02/2011 01:12 AM, Anthony Chan wrote:
> >>>>>>>> I took a quick look at the scripts that you sent but they
> >>>>>>>> ain't
> >>>>>>>> very
> >>>>>>>> helpful
> >>>>>>>> as these files source some other files which ain't included
> >>>>>>>> in
> >>>>>>>> your
> >>>>>>>> attachment.
> >>>>>>>> In any case, my knowledge on CCSM is limited. Since the build
> >>>>>>>> problem is CCSM
> >>>>>>>> specific, you may want to contact CCSM developers to see what
> >>>>>>>> it
> >>>>>>>> should be done.
> >>>>>>>>
> >>>>>>>> Given your final executable (or one of the executables) has
> >>>>>>>> unresolved symbol
> >>>>>>>> MPIR_Thread, so it seems it didn't link in MPICH2 libraries
> >>>>>>>> correctly.
> >>>>>>>> I would first set FC=to $(MPICH_PATH)/bin/mpif90 and
> >>>>>>>> CC=$(MPICH_PATH)/bin/mpicc
> >>>>>>>> and leave LIB_MPI and INC_MPI empty since mpif90 and mpicc
> >>>>>>>> should
> >>>>>>>> take care of
> >>>>>>>> LIB_MPI and INC_MPI. You may want to save the compile and
> >>>>>>>> linking
> >>>>>>>> output of
> >>>>>>>> whole build process to see what goes wrong.
> >>>>>>>>
> >>>>>>>> A.Chan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> mmm, I am not sure if I can answer this question.
> >>>>>>>>> It is a global climate model, with a lot of packages, and in
> >>>>>>>>> order
> >>>>>>>>> to
> >>>>>>>>> compile it I follow the installation guide of the model.
> >>>>>>>>> So I am not completely sure what I should provide you with?
> >>>>>>>>>
> >>>>>>>>> There is a seperate Macros files in which I have to adapt
> >>>>>>>>> libs
> >>>>>>>>> and
> >>>>>>>>> flags
> >>>>>>>>> for my system. The contents are as follows:
> >>>>>>>>> # Note that CPPDEFS is set in Macros.cppdefs
> >>>>>>>>> CPPDEFS += -DLINUX -DSEQ_$(FRAMEWORK) -DFORTRANUNDERSCORE
> >>>>>>>>> -DNO_R16
> >>>>>>>>> -DNO_SHR_VMATH
> >>>>>>>>>
> >>>>>>>>> ifeq ($(compile_threaded), true)
> >>>>>>>>> CPPDEFS += -DTHREADED_OMP
> >>>>>>>>> endif
> >>>>>>>>>
> >>>>>>>>> FC := mpif90
> >>>>>>>>> CC := mpicc
> >>>>>>>>>
> >>>>>>>>> HDF5_PATH := /usr/local/hdf5
> >>>>>>>>> HDF5_LIB := $(HDF5_PATH)/lib -lhdf5 -lhdf5_hl
> >>>>>>>>> NETCDF_PATH := /usr/local/netcdf-4.1.1
> >>>>>>>>> INC_NETCDF := $(NETCDF_PATH)/include
> >>>>>>>>> LIB_NETCDF := $(NETCDF_PATH)/lib -lnetcdf
> >>>>>>>>> -L/usr/local/hdf5/lib
> >>>>>>>>> -lhdf5_hl -lhdf5 -L/usr/local/zlib/lib -lz -lm
> >>>>>>>>> MOD_NETCDF := $(NETCDF_PATH)/include
> >>>>>>>>> MPICH_PATH := /usr/local/mpich2-1.3.1
> >>>>>>>>> INC_MPI := $(MPICH_PATH)/include
> >>>>>>>>> LIB_MPI := $(MPICH_PATH)/lib
> >>>>>>>>> MPI_LIB_NAME := mpich
> >>>>>>>>> PNETCDF_PATH :=
> >>>>>>>>> INC_PNETCDF :=
> >>>>>>>>> LIB_PNETCDF :=
> >>>>>>>>> LAPACK_LIBDIR :=
> >>>>>>>>> #=====================================================
> >>>>>>>>>
> >>>>>>>>> CFLAGS := $(CPPDEFS)
> >>>>>>>>> FIXEDFLAGS := -132
> >>>>>>>>> FREEFLAGS := -FR
> >>>>>>>>> ##FFLAGS := $(CPPDEFS) -g -fp-model precise -convert
> >>>>>>>>> big_endian
> >>>>>>>>> -assume byterecl -ftz -traceback
> >>>>>>>>> FFLAGS := $(CPPDEFS)-g -O2 -I/usr/local/include
> >>>>>>>>> FFLAGS_NOOPT := $(FFLAGS) -O0
> >>>>>>>>> FFLAGS_OPT := -O2
> >>>>>>>>> LDFLAGS :=
> >>>>>>>>> AR := ar
> >>>>>>>>> MOD_SUFFIX := mod
> >>>>>>>>> CONFIG_SHELL :=
> >>>>>>>>>
> >>>>>>>>> So maybe something is wrong here???
> >>>>>>>>> In attach I also sent you the configure file, and the build
> >>>>>>>>> file
> >>>>>>>>> to
> >>>>>>>>> compile the model.
> >>>>>>>>> That works all fine. Only when running the obtained
> >>>>>>>>> executable,
> >>>>>>>>> I
> >>>>>>>>> get
> >>>>>>>>> the error MPIR_Thread...
> >>>>>>>>> to run the .exe, I use the following:
> >>>>>>>>> mpirun ./ccsm.exe>&! ccsm.log.$LID
> >>>>>>>>>
> >>>>>>>>> as is mentioned in the test2.Latitude_E6400.run file in
> >>>>>>>>> attach.
> >>>>>>>>>
> >>>>>>>>> I do not know if you can do something with this, if not,
> >>>>>>>>> please
> >>>>>>>>> let
> >>>>>>>>> me
> >>>>>>>>> know....
> >>>>>>>>>
> >>>>>>>>> Matthias
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 02/01/2011 09:17 PM, Anthony Chan wrote:
> >>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>
> >>>>>>>>>>> I can also perform the test as in the installation guide:
> >>>>>>>>>>> mpiexec -n 5 ./examples/cpi
> >>>>>>>>>>>
> >>>>>>>>>>> Process 1 of 5 is on MaffieLE6400
> >>>>>>>>>>> Process 3 of 5 is on MaffieLE6400
> >>>>>>>>>>> Process 0 of 5 is on MaffieLE6400
> >>>>>>>>>>> Process 2 of 5 is on MaffieLE6400
> >>>>>>>>>>> Process 4 of 5 is on MaffieLE6400
> >>>>>>>>>>> pi is approximately 3.1415926544231230, Error is
> >>>>>>>>>>> 0.0000000008333298
> >>>>>>>>>>> wall clock time = 0.000435
> >>>>>>>>>>>
> >>>>>>>>>>> as far as I can see all works fine?
> >>>>>>>>>> Yes.
> >>>>>>>>>>
> >>>>>>>>>>> The reason I configured with the
> >>>>>>>>>>> --with-thread-package=none
> >>>>>>>>>>> the
> >>>>>>>>>>> second
> >>>>>>>>>>> time was to deal with the MPIR_Thread error...
> >>>>>>>>>>> So I wonder how I can deal with this error?
> >>>>>>>>>> Let's just focus on using the mpich2 install that was
> >>>>>>>>>> configured
> >>>>>>>>>> without
> >>>>>>>>>> --with-thread-package. How did you compile your application
> >>>>>>>>>> ?
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for your help!
> >>>>>>>>>>>
> >>>>>>>>>>> Matthias
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 02/01/2011 08:24 PM, Anthony Chan wrote:
> >>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>> Dear Anthony,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> My apologies for posting to the wrong email list. I will
> >>>>>>>>>>>>> register
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>> other one so that in the future I can put my questions
> >>>>>>>>>>>>> there.
> >>>>>>>>>>>>> For now i'll post this directly to you, hope that is
> >>>>>>>>>>>>> fine??
> >>>>>>>>>>>> Yes. I cc'ed my reply to mpich-discuss as someone else
> >>>>>>>>>>>> may
> >>>>>>>>>>>> have
> >>>>>>>>>>>> things to add.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I am not sure if I understand what you mean with:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Did you use mpif90/mpif77 to link your code or just use
> >>>>>>>>>>>>> above
> >>>>>>>>>>>>> link
> >>>>>>>>>>>>> command which seems incomplete ?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I used the following line to check how mpich2 was built,
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>> I
> >>>>>>>>>>>>> first
> >>>>>>>>>>>>> got
> >>>>>>>>>>>>> an error with f90 vs f95.
> >>>>>>>>>>>>> So with
> >>>>>>>>>>>>> /usr/local/mpich2-1.3.1/bin/mpif90 -show
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I get finally got:
> >>>>>>>>>>>>> gfortran -O2 -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>>> -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>>> -L/usr/local/mpich2-1.3.1/lib
> >>>>>>>>>>>>> -lmpichf90 -lmpichf90 -lmpich -lopa -lmpl -lrt -lpthread
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do you mean that there is something missing here? And
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>> I
> >>>>>>>>>>>>> add
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>> than in my definition of INC_MPI and LIB_MPI?
> >>>>>>>>>>>> Let's focus if your mpich2 installation is correct first.
> >>>>>>>>>>>>> One other question. I was thinking that it could be
> >>>>>>>>>>>>> useful
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> built
> >>>>>>>>>>>>> mpich2 without threading, so I used:
> >>>>>>>>>>>>> ./configure -prefix=/usr/local/mpich2-1.3.1
> >>>>>>>>>>>>> --with-thread-package=none
> >>>>>>>>>>>> Try "./configure --prefix=/usr/local/mpich2-1.3.1",
> >>>>>>>>>>>> without
> >>>>>>>>>>>> --with-thread-package.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A.Chan
> >>>>>>>>>>>>
> >>>>>>>>>>>>> But then I get a whole list of errors, e.g.:
> >>>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getn.o):
> >>>>>>>>>>>>> In
> >>>>>>>>>>>>> function `MPIU_Thread_CS_exit_lockname_recursive_impl_':
> >>>>>>>>>>>>> info_getn.c:(.text+0x124): undefined reference to
> >>>>>>>>>>>>> `MPIU_Thread_mutex_unlock'
> >>>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getnth.o):
> >>>>>>>>>>>>> In
> >>>>>>>>>>>>> function
> >>>>>>>>>>>>> `MPIU_Thread_CS_enter_lockname_recursive_impl_':
> >>>>>>>>>>>>> info_getnth.c:(.text+0x32): undefined reference to
> >>>>>>>>>>>>> `MPIU_Thread_mutex_lock'
> >>>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getnth.o):
> >>>>>>>>>>>>> In
> >>>>>>>>>>>>> function `MPIU_Thread_CS_exit_lockname_recursive_impl_':
> >>>>>>>>>>>>> info_getnth.c:(.text+0xf4): undefined reference to
> >>>>>>>>>>>>> `MPIU_Thread_mutex_unlock'
> >>>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getvallen.o):
> >>>>>>>>>>>>> In function
> >>>>>>>>>>>>> `MPIU_Thread_CS_enter_lockname_recursive_impl_':
> >>>>>>>>>>>>> info_getvallen.c:(.text+0x32): undefined reference to
> >>>>>>>>>>>>> `MPIU_Thread_mutex_lock'
> >>>>>>>>>>>>> /home/matthias/Downloads/mpich2-1.3.1/lib/libmpich.a(info_getvallen.o):
> >>>>>>>>>>>>> In function
> >>>>>>>>>>>>> `MPIU_Thread_CS_exit_lockname_recursive_impl_':
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Any ideas here?
> >>>>>>>>>>>>> Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -------- Original Message --------
> >>>>>>>>>>>>> Subject: Re: [mpich2-dev] undefined symbol: MPIR_Thread
> >>>>>>>>>>>>> Date: Tue, 1 Feb 2011 19:41:54 +0100
> >>>>>>>>>>>>> From: Anthony Chan<chan at mcs.anl.gov>
> >>>>>>>>>>>>> Reply-To: Anthony Chan<chan at mcs.anl.gov>,
> >>>>>>>>>>>>> "mpich2-dev at mcs.anl.gov"
> >>>>>>>>>>>>> <mpich2-dev at mcs.anl.gov>
> >>>>>>>>>>>>> To: mpich2-dev at mcs.anl.gov<mpich2-dev at mcs.anl.gov>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> mpich2-dev for mpich2 development not general MPICH
> >>>>>>>>>>>>> usage
> >>>>>>>>>>>>> questions,
> >>>>>>>>>>>>> please send your question to mpich-discuss at mcs.anl.gov.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>        All,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        when running a model with mpirun ./exe
> >>>>>>>>>>>>>>        I get the following error:
> >>>>>>>>>>>>>>        ./exe: symbol lookup error: ./exe: undefined
> >>>>>>>>>>>>>>        symbol:
> >>>>>>>>>>>>>>        MPIR_Thread
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        I have no idea how to deal with this. I use
> >>>>>>>>>>>>>>        mpich2-1.3.1,
> >>>>>>>>>>>>>>        compiled
> >>>>>>>>>>>>>>        with
> >>>>>>>>>>>>>>        gfortran, with the following libraries:
> >>>>>>>>>>>>>>        gfortran -O2 -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>>>>        -I/usr/local/mpich2-1.3.1/include
> >>>>>>>>>>>>>>        -L/usr/local/mpich2-1.3.1/lib
> >>>>>>>>>>>>>>        -lmpichf90 -lmpichf90 -lmpich -lopa -lmpl -lrt
> >>>>>>>>>>>>>>        -lpthread
> >>>>>>>>>>>>> Did you use mpif90/mpif77 to link your code or just use
> >>>>>>>>>>>>> above
> >>>>>>>>>>>>> link
> >>>>>>>>>>>>> command which seems incomplete ?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> A.Chan
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>        Could anybody help me with this?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        Regards,
> >>>>>>>>>>>>>>        Matthias
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        --
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        -------------------------------------------------------------
> >>>>>>>>>>>>>>        FWO Post-doctoral Research Fellow
> >>>>>>>>>>>>>>        Department of Earth& Environmental Sciences
> >>>>>>>>>>>>>>        Physical and Regional Geography Research Group
> >>>>>>>>>>>>>>        Regional climate studies
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        Celestijnenlaan 200E
> >>>>>>>>>>>>>>        3001 Heverlee (Leuven)
> >>>>>>>>>>>>>>        BELGIUM
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        Tel: + 32 16 326424
> >>>>>>>>>>>>>>        Fax: + 32 16 322980
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        http://geo.kuleuven.be/aow/
> >>>>>>>>>>>>>>        www.kuleuven.be/geography
> >>>>>>>>>>>>>>        -------------------------------------------------------------
> >>>>>>>>>>> --
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -------------------------------------------------------------
> >>>>>>>>>>> FWO Post-doctoral Research Fellow
> >>>>>>>>>>> Department of Earth& Environmental Sciences
> >>>>>>>>>>> Physical and Regional Geography Research Group
> >>>>>>>>>>> Regional climate studies
> >>>>>>>>>>>
> >>>>>>>>>>> Celestijnenlaan 200E
> >>>>>>>>>>> 3001 Heverlee (Leuven)
> >>>>>>>>>>> BELGIUM
> >>>>>>>>>>>
> >>>>>>>>>>> Tel: + 32 16 326424
> >>>>>>>>>>> Fax: + 32 16 322980
> >>>>>>>>>>>
> >>>>>>>>>>> http://geo.kuleuven.be/aow/
> >>>>>>>>>>> www.kuleuven.be/geography
> >>>>>>>>>>> -------------------------------------------------------------
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -------------------------------------------------------------
> >>>>>>>>> FWO Post-doctoral Research Fellow
> >>>>>>>>> Department of Earth& Environmental Sciences
> >>>>>>>>> Physical and Regional Geography Research Group
> >>>>>>>>> Regional climate studies
> >>>>>>>>>
> >>>>>>>>> Celestijnenlaan 200E
> >>>>>>>>> 3001 Heverlee (Leuven)
> >>>>>>>>> BELGIUM
> >>>>>>>>>
> >>>>>>>>> Tel: + 32 16 326424
> >>>>>>>>> Fax: + 32 16 322980
> >>>>>>>>>
> >>>>>>>>> http://geo.kuleuven.be/aow/
> >>>>>>>>> www.kuleuven.be/geography
> >>>>>>>>> -------------------------------------------------------------
> >>>>>>> --
> >>>>>>>
> >>>>>>>
> >>>>>>> -------------------------------------------------------------
> >>>>>>> FWO Post-doctoral Research Fellow
> >>>>>>> Department of Earth& Environmental Sciences
> >>>>>>> Physical and Regional Geography Research Group
> >>>>>>> Regional climate studies
> >>>>>>>
> >>>>>>> Celestijnenlaan 200E
> >>>>>>> 3001 Heverlee (Leuven)
> >>>>>>> BELGIUM
> >>>>>>>
> >>>>>>> Tel: + 32 16 326424
> >>>>>>> Fax: + 32 16 322980
> >>>>>>>
> >>>>>>> http://geo.kuleuven.be/aow/
> >>>>>>> www.kuleuven.be/geography
> >>>>>>> -------------------------------------------------------------
> >>>>> --
> >>>>>
> >>>>>
> >>>>> -------------------------------------------------------------
> >>>>> FWO Post-doctoral Research Fellow
> >>>>> Department of Earth& Environmental Sciences
> >>>>> Physical and Regional Geography Research Group
> >>>>> Regional climate studies
> >>>>>
> >>>>> Celestijnenlaan 200E
> >>>>> 3001 Heverlee (Leuven)
> >>>>> BELGIUM
> >>>>>
> >>>>> Tel: + 32 16 326424
> >>>>> Fax: + 32 16 322980
> >>>>>
> >>>>> http://geo.kuleuven.be/aow/
> >>>>> www.kuleuven.be/geography
> >>>>> -------------------------------------------------------------
> >>> --
> >>>
> >>>
> >>> -------------------------------------------------------------
> >>> FWO Post-doctoral Research Fellow
> >>> Department of Earth& Environmental Sciences
> >>> Physical and Regional Geography Research Group
> >>> Regional climate studies
> >>>
> >>> Celestijnenlaan 200E
> >>> 3001 Heverlee (Leuven)
> >>> BELGIUM
> >>>
> >>> Tel: + 32 16 326424
> >>> Fax: + 32 16 322980
> >>>
> >>> http://geo.kuleuven.be/aow/
> >>> www.kuleuven.be/geography
> >>> -------------------------------------------------------------
> 
> --
> 
> 
> -------------------------------------------------------------
> FWO Post-doctoral Research Fellow
> Department of Earth& Environmental Sciences
> Physical and Regional Geography Research Group
> Regional climate studies
> 
> Celestijnenlaan 200E
> 3001 Heverlee (Leuven)
> BELGIUM
> 
> Tel: + 32 16 326424
> Fax: + 32 16 322980
> 
> http://geo.kuleuven.be/aow/
> www.kuleuven.be/geography
> -------------------------------------------------------------

More information about the mpich-discuss mailing list