[petsc-users] Confusion/failures about the tests involved in including Hypre

Barry Smith bsmith at petsc.dev
Thu Jul 20 18:49:44 CDT 2023 

  Satish,

   I think hypre.py in main needs minversion = 2.29 instead of 2.14



> On Jul 20, 2023, at 11:05 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> Can check config/BuildSystem/config/packages/hypre.py
> 
> petsc-3.19 (or release branch) is compatible with hypre 2.28.0, petsc 'main' branch with 2.29.0
> 
> Satish
> 
> On Thu, 20 Jul 2023, Barry Smith via petsc-users wrote:
> 
>> 
>>  You cannot use this version of PETSc, 3.19, with the version of hypre you installed. In hypre they recently changed hypre_Error from an integer to a struct which completely breaks compatibility with previous versions of hypre (and hence previous versions of PETSc). You must use the main git branch of PETSc with the version of hypre you installed.
>> 
>>  Barry
>> 
>> 
>>> On Jul 20, 2023, at 5:10 AM, Daniel Stone <daniel.stone at opengosim.com> wrote:
>>> 
>>> Hi All,
>>> 
>>> Many thanks for the detailed explainations and ideas!
>>> 
>>> I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly:
>>> 
>>> --------------------------------
>>> 
>>>         FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o
>>>         CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o
>>>         CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o
>>>         CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o
>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int'
>>>        hypre__global_error = 0;
>>>                            ^ ~
>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid'
>>>      __VA_ARGS__; \
>>>      ^~~~~~~~~~~
>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int'
>>>        hypre__global_error = 0;
>>>                            ^ ~
>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid'
>>>      __VA_ARGS__; \
>>>      ^~~~~~~~~~~
>>> 2 errors generated.
>>> make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1
>>> make[3]: *** Waiting for unfinished jobs....
>>>         FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o
>>>         CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o
>>>         CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o
>>> make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1'
>>> make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2
>>> make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1'
>>> **************************ERROR*************************************
>>>  Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log
>>>  Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov <mailto:petsc-maint at mcs.anl.gov>
>>> ********************************************************************
>>> Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100
>>> 
>>> ----------------------------------------
>>> 
>>> But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" 
>>> cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied
>>> into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one.
>>> 
>>> 
>>> On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default,
>>> which I didn't change:
>>> 
>>> # Configuration options
>>> option(HYPRE_ENABLE_SHARED           "Build a shared library" OFF)
>>> option(HYPRE_ENABLE_BIGINT           "Use long long int for HYPRE_Int" OFF)
>>> option(HYPRE_ENABLE_MIXEDINT         "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF)
>>> [....]
>>> 
>>> 
>>> checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon.
>>> 
>>> 
>>> 
>>> Thanks again,
>>> 
>>> Daniel
>>> 
>>> 
>>> 
>>> On Wed, Jul 19, 2023 at 4:58 PM Satish Balay via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>> I think it should work with static libraries and 64bit compilers.
>>>> 
>>>> That's how I think --download-f2cblaslapack [etc] work.
>>>> 
>>>> Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow..
>>>> 
>>>> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64
>>>> $ nm -Ao msmpi.lib |grep " MPI_Init"
>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init
>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread
>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized
>>>> 
>>>> However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail.
>>>> 
>>>> Checking PETSc dll build:
>>>> 
>>>> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib
>>>> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays
>>>> libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays
>>>> libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays
>>>> 
>>>> It also has the unmangled symbol - so I guess this mode can work generally with dlls.
>>>> 
>>>> Satish
>>>> 
>>>> 
>>>> On Wed, 19 Jul 2023, Barry Smith wrote:
>>>> 
>>>>> 
>>>>>  Satish,
>>>>> 
>>>>>   So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail).
>>>>> 
>>>>>  Barry
>>>>> 
>>>>> 
>>>>>> On Jul 19, 2023, at 11:40 AM, Satish Balay <balay at mcs.anl.gov <mailto:balay at mcs.anl.gov>> wrote:
>>>>>> 
>>>>>> BTW: Some explanation of configure:
>>>>>> 
>>>>>> It attempts the following on linux:
>>>>>> 
>>>>>>>>>>>> 
>>>>>> Source:
>>>>>> #include "confdefs.h"
>>>>>> #include "conffix.h"
>>>>>> /* Override any gcc2 internal prototype to avoid an error. */
>>>>>> char HYPRE_IJMatrixCreate();
>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); }
>>>>>> 
>>>>>> int main(void) {
>>>>>> _check_HYPRE_IJMatrixCreate();
>>>>>> return 0;
>>>>>> }
>>>>>> <<<<<<<
>>>>>> 
>>>>>> Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate();
>>>>>> 
>>>>>> Compiling it manually:
>>>>>> 
>>>>>>>>>> 
>>>>>> [balay at pj01 petsc]$ cat conftest.c
>>>>>> char HYPRE_IJMatrixCreate();
>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); }
>>>>>> 
>>>>>> int main(void) {
>>>>>> _check_HYPRE_IJMatrixCreate();
>>>>>> return 0;
>>>>>> }
>>>>>> [balay at pj01 petsc]$ gcc -c conftest.c
>>>>>> [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate
>>>>>> conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate
>>>>>> conftest.o:                 U HYPRE_IJMatrixCreate
>>>>>> [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate
>>>>>> arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate
>>>>>> [balay at pj01 petsc]$ 
>>>>>> <<<<
>>>>>> 
>>>>>> Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds!
>>>>>> 
>>>>>>>>>>>> 
>>>>>> [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so
>>>>>> [balay at pj01 petsc]$ echo $?
>>>>>> 0
>>>>>> <<<<<
>>>>>> 
>>>>>> On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures.
>>>>>> 
>>>>>> Satish
>>>>>> 
>>>>>> 
>>>>>> On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote:
>>>>>> 
>>>>>>> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works.
>>>>>>> 
>>>>>>> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py
>>>>>>> index 5bc88322aa2..2d6c7932e17 100644
>>>>>>> --- a/config/BuildSystem/config/packages/hypre.py
>>>>>>> +++ b/config/BuildSystem/config/packages/hypre.py
>>>>>>> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage):
>>>>>>>    self.requiresversion = 1
>>>>>>>    self.gitcommit       = 'v'+self.version
>>>>>>>    self.download        = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz']
>>>>>>> -    self.functions       = ['HYPRE_IJMatrixCreate']
>>>>>>> +    self.functions       = []
>>>>>>>    self.includes        = ['HYPRE.h']
>>>>>>>    self.liblist         = [['libHYPRE.a']]
>>>>>>>    self.buildLanguages  = ['C','Cxx']
>>>>>>> 
>>>>>>> 
>>>>>>> Satish
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, 19 Jul 2023, Barry Smith wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared
>>>>>>>> 
>>>>>>>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem.
>>>>>>>> 
>>>>>>>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone <daniel.stone at opengosim.com <mailto:daniel.stone at opengosim.com>> wrote:
>>>>>>>>> 
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works
>>>>>>>>> by setting up a VS project to run some of the example programs.
>>>>>>>>> 
>>>>>>>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications
>>>>>>>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc
>>>>>>>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script).
>>>>>>>>> 
>>>>>>>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like:
>>>>>>>>> 
>>>>>>>>> #include "confdefs.h"
>>>>>>>>> #include "conffix.h"
>>>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */
>>>>>>>>> 
>>>>>>>>> #include "HYPRE.h"
>>>>>>>>> 
>>>>>>>>> char HYPRE_IJMatrixCreate();
>>>>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); }
>>>>>>>>> 
>>>>>>>>> int main() {
>>>>>>>>> _check_HYPRE_IJMatrixCreate();;
>>>>>>>>> return 0;
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that
>>>>>>>>> include PTScotch.
>>>>>>>>> 
>>>>>>>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately:
>>>>>>>>> 
>>>>>>>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" ....
>>>>>>>>> 
>>>>>>>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more 
>>>>>>>>> nontrivial Hypre example programs, I get the same error:
>>>>>>>>> 
>>>>>>>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1
>>>>>>>>> 
>>>>>>>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but
>>>>>>>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. 
>>>>>>>>> 
>>>>>>>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case?
>>>>>>>>> 
>>>>>>>>> Many thanks,
>>>>>>>>> 
>>>>>>>>> Daniel
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>>

More information about the petsc-users mailing list