[petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build

[Bro H]

Barry, thanks, it appears to be working correctly after following your
latest suggestion. I used the following configuration command:

./configure --force --prefix=/opt/libs/petsc COPTFLAGS="-O3"
CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" --with-precision=double
--with-64-bit-indices --with-64-bit-blas-indices
--known-64-bit-blas-indices=1
--with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/
--with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0
--with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1
--with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1

However, I still get warnings during build, with the exception that
MKL Sparse BLAS now expects long long int instead of int for MKL_INT:

warning: passing argument 7 of ‘mkl_sparse_d_export_csr’ from
incompatible pointer type [-Wincompatible-pointer-types]
note: expected ‘long long int **’ but argument is of type ‘PetscInt
**’ {aka ‘long int **’}

Not a big problem for me right now since they are the same size in my
case. But these warnings should probably be fixed in the future.

Could you or other PETSc devs please add official support for MKL
Sparse BLAS with 64-bit indices in an upcoming PETSc release?

On Sun, Sep 25, 2022 at 9:50 PM Barry Smith <bsmith at petsc.dev> wrote:
>
>
>    Likely you can fix the problem by adding
>
> #if defined(PETSC_HAVE_MKL_INTEL_ILP64)
>   #define MKL_ILP64
> #endif
>
> before the #include <mkl_spblas.h> in
>
> src/mat/impls/aij/seq/aijmkl/aijmkl.c  and
> src/mat/impls/baij/seq/baijmkl/baijmkl.c
>
>   Please let us know if this resolves the problem.
>
>   Barry
>
> On Sep 25, 2022, at 12:37 AM, Bro H <broheimhelldawg29 at gmail.com> wrote:
>
> Barry, thank you for answering. I did some further testing. My MKL
> version is 20220002 as detected by PETSc. I tried to compile one of
> the examples distributed with MKL
> (${MKLROOT}/latest/examples/examples_core_c.tgz/c/sparse_blas/source/sparse_csr.c)
> that contains calls to mkl_sparse_d_mv function from Sparse BLAS,
> among others. It compiled and ran without errors with 64-bit indices.
>
> I used the following command to compile this example on Debian 11:
>
> gcc `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts`
> sparse_csr.c `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -libs`
>
> mkl_link_tool is bundled with MKL. It returns:
>
> `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = -DMKL_ILP64
> -m64  -I"${MKLROOT}/include"
>
> `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` =
> -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64
> -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl
>
> So in theory it should work when PetscInt is 64 bits, am I right? The
> only problem is with the build scripts or source code in PETSc?
>
> Per your suggestion, I have tried to change requires32bitint to 0 in
> mkl_sparse.py and mkl_sparse_optimize.py routines. PETSc used 64-bit
> indices, but configuration also set MKL to 32-bit indices, so I added
> --with-64-bit-blas-indices. I then ran into "Cannot use mkl_sparse
> with 64-bit BLAS/Lapack indices" error, and so I tried to change
> requires32bitintblas to 0 in config/BuildSystem/config/package.py.
> Configuration was successful with both PETSc and MKL set to 64-bit
> indices, however, it looks like PETSc was still using 32-bit indices
> during compilation of aijmkl implementation, with warnings such as:
>
> petsc-3.17.4/src/mat/impls/aij/seq/aijmkl/aijmkl.c:282:98: warning:
> passing argument 7 of ‘mkl_sparse_d_export_csr’ from incompatible
> pointer type
>
> Are there any other modifications required to make PETSc work with
> 64-bit MKL Sparse BLAS?
>
> On Sun, Sep 25, 2022 at 3:54 AM Barry Smith <bsmith at petsc.dev> wrote:
>
>
>
>  It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to.
>
>  You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers.
>
>
> On Sep 24, 2022, at 11:04 AM, Bro H <broheimhelldawg29 at gmail.com> wrote:
>
> Hello.
>
> I would like to build PETSc with support for MKL Sparse BLAS, so that
> I can use MATAIJMKL for improved performance, but I also need to use
> 64 bit indices. I'm trying to build using the following parameters:
>
> ./configure --force --prefix="/opt/libs/petsc" --with-precision=double
> --with-64-bit-indices
> --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/
> --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0
> --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1
> --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1
>
> Configuration fails when using --with-64-bit-indices simultaneously
> with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the
> following possible errors:
>
> "Cannot use mkl_sparse with 64 bit integers, it is not coded for this
> capability"
> "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded
> for this capability"
>
> But doesn't the latest version of MKL Sparse BLAS support 64 bit
> indices? It is using MKL_INT in mkl_spblas.h.
>
> Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS?
>
>
>