[mpich2-dev] build issues with MPE on Windows(32) and MinGW

Wed Nov 4 09:36:43 CST 2009

On Wed, Nov 4, 2009 at 1:17 PM, Jayesh Krishna <jayesh at mcs.anl.gov> wrote:
>  Hmmm... It looks like the newer version of gnu tools don't require dlltool magic to use the windows dll import libraries. So you don't need a gcc-compatible libmpe.a afterall...

Ah!! Now I more or less understand your concerns... I've start testing
on Win just a few month ago, so I've been using recent versions of all
the stuff... I just do not know about any limitations MinGW or other
GNU tools had in the past.

>  Regarding the patch, let me do some testing locally (The right place to make the change is in the windows configure script) and get back to you. However this change won't make it to the next (1.2.1) release (too close to the release date).

OK. I've fixed my code to #include these headers (and likely work with
older MPICH2 releases), so no hurry from my side...

>
> Regards,
> Jayesh
> ----- Original Message -----
> From: "Lisandro Dalcin" <dalcinl at gmail.com>
> To: mpich2-dev at mcs.anl.gov
> Cc: jayesh at mcs.anl.gov
> Sent: Monday, November 2, 2009 4:49:03 PM GMT -06:00 US/Canada Central
> Subject: Re: [mpich2-dev] build issues with MPE on Windows(32) and MinGW
>
> On Mon, Nov 2, 2009 at 8:21 PM,  <jayesh at mcs.anl.gov> wrote:
>> Hi,
>>  Where do you have libmpe.a (It is not installed with MPICH2 on windows and you seem to be linking your code with "-lmpe")?
>>
>
> Well, I have installed a "mpe.lib" (import lib?) in
> %ProgramFiles%\MPICH2\lib, and a "mpe.dll" in c:\windows\system32 ...
> So my code is likely reading stuff from mpe.lib and linking with
> mpe.dll, right? I think libmpe.a is not required at all... BTW, all
> this is on Windows XP.
>
> Remember, I've manually removed any trace of other MPI/MPE installs,
> so I got these libs after re-installing the latest MPICH2 release
> (moreover, the dates of the installed files are all the same). So I
> think the MPE libs are from my last MPICH2 install...
>
>>
>> (PS: The problem with multiple installations of MPICH2 was a result of us not upgrading the product versions correctly for MPICH2. We expected users to uninstall their previous version of MPICH2 before installing a new version. This will not happen for future releases.)
>>
>
> OK... Good to know you are going to fix this, hunting for files left
> around after a broken uninstall is not a pleasure :-) ...
>
>
>> Regards,
>> Jayesh
>> ----- Original Message -----
>> From: "Lisandro Dalcin" <dalcinl at gmail.com>
>> To: mpich2-dev at mcs.anl.gov
>> Cc: jayesh at mcs.anl.gov
>> Sent: Monday, November 2, 2009 4:10:38 PM GMT -06:00 US/Canada Central
>> Subject: Re: [mpich2-dev] build issues with MPE on Windows(32) and MinGW
>>
>> On Mon, Nov 2, 2009 at 7:41 PM,  <jayesh at mcs.anl.gov> wrote:
>>> Hi,
>>>  Each MPICH2 release on windows is independent of each other (You shouldn't have to install a version of MPICH2 before installing another). Let us know the problem you faced and we can sort it out.
>>
>> Well, at this point I'm not sure what happened. As I got confused with
>> your previous mail, I decided to unistall all the MPI's I had in my
>> Windows system. I went to "Add/Remove Programs" (not sure if it is the
>> exact translation, using a Spanish version of Windows here) and had
>> three MPICH2 entries; I've uninstalled the latest one with success,
>> but I was not able to uninstall the other two, a box "missing program
>> required for uninstall" or something like that appeared. Then I had no
>> other choice to manually search for and remove any file (mostly DLL's
>> in c:\windows\system32 ) let out (and use some utilities to remove the
>> MPICH2 entries for the installed program lists...)
>>
>> Perhaps all this nightmare was just my fault... I really do not have
>> any Windows experience, I'm just making sure that my Python bindings
>> can be built and used on Windows against MPICH2, DeinoMPI and MS HPC
>> Cluster Pack 08, with either MSVC and MinGW...
>>
>>
>>>
>>>  Which library did you use to compile/link your code (Please provide us with the path/name of the libraries)? Is your code written in C/python/C++/fortran ? Please provide us more details regarding your program.
>>>
>>
>> You have attached the whole output I get when building mpi4py on Win32
>> with MPICH2 and MinGW.
>>
>> I pasted below the most relevant info which appear at the end of the
>> log, i.e, the part where MPE is being actually used (this output is
>> from some monkeypatched Python's distutils in order to make it more
>> featured):
>>
>> building 'mpi4py.MPE' extension
>> C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -DHAVE_MPE=1
>> -IC:\Python26\include -IC:\Python26\PC "-IC:\Archivos de
>> programa\MPICH2\include" -c src\MPE.c -o
>> build\temp.win32-2.6\Release\src\mpe.o
>> writing build\temp.win32-2.6\Release\src\MPE.def
>> C:\MinGW\bin\gcc.exe -mno-cygwin -shared -s
>> build\temp.win32-2.6\Release\src\mpe.o
>> build\temp.win32-2.6\Release\src\MPE.def -LC:\Python26\libs
>> -LC:\Python26\PCbuild "-LC:\Archivos de programa\MPICH2\lib" -lmpe
>> -lpython26 -lmsvcr90 -lmpi -o build\lib.win32-2.6\mpi4py\MPE.pyd
>>
>> After that, I'm able to "from mpi4py import MPE", and the
>> MPE-instrumented script here:
>> http://code.google.com/p/mpi4py/source/browse/trunk/demo/mpe-logging/cpilog.py
>> works just fine, generating a "cpilog.clog2" file at the end of the
>> run. No MPI calls are logged (because of the lack of RTLD_GLOBAL
>> semantics on Win), just the user-defined states and events.
>>
>> So this makes me think that using MinGW and linking against MPE does
>> actually work. However, I point it again, these two includes (stdint.h
>> and intypes.h) are required to workaround build failures. Again, I
>> paste to my mpi4py pach where you can see these two headers included
>> right above including mpe.h
>>
>> http://code.google.com/p/mpi4py/source/diff?spec=svn522&r=522&format=side&path=/trunk/src/MPE/mpe-log.c.
>>
>>
>>>
>>> (PS: You should be able to do stuff similar to dlopen() with LoadLibrary() on windows...)
>>>
>>
>> Do you know of any way of getting RTLD_GLOBAL semantics? I know about
>> LoadLibrary(), but AFAIK there is not equivalent do dlopen() with
>> mode=RTLD_GLOBAL... Perhaps I'm plain wrong, as I have no experience
>> on Windows... Any point on this will be appreciated.
>>
>>
>>> -Jayesh
>>>
>>> ----- Original Message -----
>>> From: "Lisandro Dalcin" <dalcinl at gmail.com>
>>> To: mpich2-dev at mcs.anl.gov
>>> Cc: "Jayesh Krishna" <jayesh at mcs.anl.gov>
>>> Sent: Monday, November 2, 2009 3:20:50 PM GMT -06:00 US/Canada Central
>>> Subject: Re: [mpich2-dev] build issues with MPE on Windows(32) and MinGW
>>>
>>> On Mon, Nov 2, 2009 at 5:35 PM,  <jayesh at mcs.anl.gov> wrote:
>>>> Hi,
>>>
>>> Sorry for taking so long to get back... I have little Windows
>>> experience, and your mail REALLY confused me. I had to start from
>>> scratch and uninstall DeinoMPI, MPICH2 and MS HPC Cluster Pack from my
>>> Windows system to make sure I was not being bitten by the infamous
>>> "DLL Hell" :-) ... In the process, I've discovered that for upgrading
>>> MPICH2, fist I have to uninstall the previous version :-( (BTW, is
>>> this right?)... But I think I was able to get things cleaned-up and
>>> then re-installed latest MPICH2-1.2 using the MSI installer.
>>>
>>>>
>>>>  We currently don't provide a mingw compatible library for MPE on windows.
>>>>
>>>
>>> What that means? This is the part that confused me...
>>>
>>> I've just built a Python extension module linked against MPE...
>>> because of the way my code works and Windows limitations (no
>>> equivalent to dlopen() with RTLD_GLOBAL, AFAIK), I cannot log MPI
>>> calls, but the logging of user-defined event and states is definitely
>>> working (to be honest, just a sequential try, no mpiexec involved).
>>>
>>> So I think you are wrong... MPE (at least some parts) do work with
>>> MinGW on Windows(32) ... Why did you said that this is not supported??
>>>
>>>>
>>>> Why are you using the MPE header files provided with MPICH2 on windows with mingw?
>>>>
>>>
>>> Because I want my code (i.e, mpi4py) to work with MinGW. No more, no
>>> less... And part of MPE (I mean, at least the logging of user-defined
>>> states and events) DO WORK with MinGW after using the fix commented in
>>> my previous mail (I mean, #include stdint.h and inttypes.h)
>>>
>>>>
>>>> (PS: The CLOG types are defined using pre-defined windows types. There is no need to include <stdint.h> or <inttypes.h> on windows.)
>>>>
>>>
>>> But if you want to use MinGW, it seems you have to include these two
>>> headers to make GCC "understand" these pre-defined windows types.
>>>
>>>
>>> PS: Am I being clear enough?
>>>
>>>
>>>> Regards,
>>>> Jayesh
>>>> ----- Original Message -----
>>>> From: "Lisandro Dalcin" <dalcinl at gmail.com>
>>>> To: mpich2-dev at mcs.anl.gov
>>>> Sent: Monday, November 2, 2009 1:02:46 PM GMT -06:00 US/Canada Central
>>>> Subject: [mpich2-dev] build issues with MPE on Windows(32) and MinGW
>>>>
>>>> Anthony, this is unrelated to our previous discussions about -fPIC, so
>>>> I'm reporting the issue here. I hope it is the right place. If not,
>>>> let me know (perhaps mpich-discuss ??).
>>>>
>>>> This diff at Google Code will be clear enough, I think:
>>>>
>>>> http://code.google.com/p/mpi4py/source/diff?spec=svn522&r=522&format=side&path=/trunk/src/MPE/mpe-log.c
>>>>
>>>> Without these two headers included, my Python ext module fails bad to
>>>> build with MinGW. Interestingly, "clog_inttypes.h" has these two
>>>> includes commented out on Windows (using the binary installer here).
>>>> So perhaps protecting the inclusion of these two headers as shown in
>>>> the link above will do the trick? To put it clear, I'm talking about
>>>> using this:
>>>>
>>>>   #if defined(_WIN32) && defined(__GNUC__)
>>>>     #include <stdint.h>
>>>>     #include <inttypes.h>
>>>>   #endif
>>>>
>>>>
>>>> --
>>>> Lisandro Dalcín
>>>> ---------------
>>>> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
>>>> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
>>>> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
>>>> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
>>>> Tel/Fax: +54-(0)342-451.1594
>>>>
>>>
>>>
>>>
>>> --
>>> Lisandro Dalcín
>>> ---------------
>>> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
>>> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
>>> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
>>> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>>
>>
>>
>>
>> --
>> Lisandro Dalcín
>> ---------------
>> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
>> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
>> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
>> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>>
>
>
>
> --
> Lisandro Dalcín
> ---------------
> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>

-- 
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594