[petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build

[Barry Smith]

   Likely you can fix the problem by adding

#if defined(PETSC_HAVE_MKL_INTEL_ILP64)
  #define MKL_ILP64
#endif

before the #include <mkl_spblas.h> in

src/mat/impls/aij/seq/aijmkl/aijmkl.c  and 
src/mat/impls/baij/seq/baijmkl/baijmkl.c

  Please let us know if this resolves the problem.

  Barry
> On Sep 25, 2022, at 12:37 AM, Bro H <broheimhelldawg29 at gmail.com> wrote:
> 
> Barry, thank you for answering. I did some further testing. My MKL
> version is 20220002 as detected by PETSc. I tried to compile one of
> the examples distributed with MKL
> (${MKLROOT}/latest/examples/examples_core_c.tgz/c/sparse_blas/source/sparse_csr.c)
> that contains calls to mkl_sparse_d_mv function from Sparse BLAS,
> among others. It compiled and ran without errors with 64-bit indices.
> 
> I used the following command to compile this example on Debian 11:
> 
> gcc `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts`
> sparse_csr.c `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -libs`
> 
> mkl_link_tool is bundled with MKL. It returns:
> 
> `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = -DMKL_ILP64
> -m64  -I"${MKLROOT}/include"
> 
> `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` =
> -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64
> -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl
> 
> So in theory it should work when PetscInt is 64 bits, am I right? The
> only problem is with the build scripts or source code in PETSc?
> 
> Per your suggestion, I have tried to change requires32bitint to 0 in
> mkl_sparse.py and mkl_sparse_optimize.py routines. PETSc used 64-bit
> indices, but configuration also set MKL to 32-bit indices, so I added
> --with-64-bit-blas-indices. I then ran into "Cannot use mkl_sparse
> with 64-bit BLAS/Lapack indices" error, and so I tried to change
> requires32bitintblas to 0 in config/BuildSystem/config/package.py.
> Configuration was successful with both PETSc and MKL set to 64-bit
> indices, however, it looks like PETSc was still using 32-bit indices
> during compilation of aijmkl implementation, with warnings such as:
> 
> petsc-3.17.4/src/mat/impls/aij/seq/aijmkl/aijmkl.c:282:98: warning:
> passing argument 7 of ‘mkl_sparse_d_export_csr’ from incompatible
> pointer type
> 
> Are there any other modifications required to make PETSc work with
> 64-bit MKL Sparse BLAS?
> 
> On Sun, Sep 25, 2022 at 3:54 AM Barry Smith <bsmith at petsc.dev> wrote:
>> 
>> 
>>  It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to.
>> 
>>  You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers.
>> 
>> 
>>> On Sep 24, 2022, at 11:04 AM, Bro H <broheimhelldawg29 at gmail.com> wrote:
>>> 
>>> Hello.
>>> 
>>> I would like to build PETSc with support for MKL Sparse BLAS, so that
>>> I can use MATAIJMKL for improved performance, but I also need to use
>>> 64 bit indices. I'm trying to build using the following parameters:
>>> 
>>> ./configure --force --prefix="/opt/libs/petsc" --with-precision=double
>>> --with-64-bit-indices
>>> --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/
>>> --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0
>>> --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1
>>> --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1
>>> 
>>> Configuration fails when using --with-64-bit-indices simultaneously
>>> with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the
>>> following possible errors:
>>> 
>>> "Cannot use mkl_sparse with 64 bit integers, it is not coded for this
>>> capability"
>>> "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded
>>> for this capability"
>>> 
>>> But doesn't the latest version of MKL Sparse BLAS support 64 bit
>>> indices? It is using MKL_INT in mkl_spblas.h.
>>> 
>>> Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS?
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220925/9c98287e/attachment-0001.html>