[petsc-users] PETSc with 64 bit indices and MKL Sparse BLAS fails to build

[Bro H]

Sorry, forgot to send a copy to petsc-users mailing list when I first
replied. My first reply is below.

On Sun, Sep 25, 2022 at 11:37 AM Bro H <broheimhelldawg29 at gmail.com> wrote:
>
> Barry, thank you for answering. I did some further testing. My MKL
> version is 20220002 as detected by PETSc. I tried to compile one of
> the examples distributed with MKL
> (${MKLROOT}/latest/examples/examples_core_c.tgz/c/sparse_blas/source/sparse_csr.c)
> that contains calls to mkl_sparse_d_mv function from Sparse BLAS,
> among others. It compiled and ran without errors with 64-bit indices.
>
> I used the following command to compile this example on Debian 11:
>
> gcc `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts`
> sparse_csr.c `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -libs`
>
> mkl_link_tool is bundled with MKL. It returns:
>
> `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` = -DMKL_ILP64
> -m64  -I"${MKLROOT}/include"
>
> `mkl_link_tool --quiet -c gnu_c -o gomp -i ilp64 -opts` =
> -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64
> -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl
>
> So in theory it should work when PetscInt is 64 bits, am I right? The
> only problem is with the build scripts or source code in PETSc?
>
> Per your suggestion, I have tried to change requires32bitint to 0 in
> mkl_sparse.py and mkl_sparse_optimize.py routines. PETSc used 64-bit
> indices, but configuration also set MKL to 32-bit indices, so I added
> --with-64-bit-blas-indices. I then ran into "Cannot use mkl_sparse
> with 64-bit BLAS/Lapack indices" error, and so I tried to change
> requires32bitintblas to 0 in config/BuildSystem/config/package.py.
> Configuration was successful with both PETSc and MKL set to 64-bit
> indices, however, it looks like PETSc was still using 32-bit indices
> during compilation of aijmkl implementation, with warnings such as:
>
> petsc-3.17.4/src/mat/impls/aij/seq/aijmkl/aijmkl.c:282:98: warning:
> passing argument 7 of ‘mkl_sparse_d_export_csr’ from incompatible
> pointer type
>
> Are there any other modifications required to make PETSc work with
> 64-bit MKL Sparse BLAS?
>
> On Sun, Sep 25, 2022 at 3:54 AM Barry Smith <bsmith at petsc.dev> wrote:
> >
> >
> >   It is possible they recently added support for using it with 64 bit integers. You would need to through their documents to see how to get mkl_spblas.h. to use 64 bit integers and if the library for 64 bit integer has a different name that would need to be linked to.
> >
> >   You would need to remove the requires32bitint = 1 from the various config/BuildSystem/config/packages/mkl_*.py routines to get configure to accept MKL sparse with 64 bit integers.
> >
> >
> > > On Sep 24, 2022, at 11:04 AM, Bro H <broheimhelldawg29 at gmail.com> wrote:
> > >
> > > Hello.
> > >
> > > I would like to build PETSc with support for MKL Sparse BLAS, so that
> > > I can use MATAIJMKL for improved performance, but I also need to use
> > > 64 bit indices. I'm trying to build using the following parameters:
> > >
> > > ./configure --force --prefix="/opt/libs/petsc" --with-precision=double
> > > --with-64-bit-indices
> > > --with-blas-lapack-dir=/opt/intel/oneapi/mkl/latest/
> > > --with-mkl_sparse=1 --with-mkl_sparse_optimize=1 --with-debugging=0
> > > --with-shared-libraries=1 --with-cxx=0 --with-mpi=1 --with-hdf5=1
> > > --with-hdf5-dir="${HDF5_ROOT}" --with-openmp=1
> > >
> > > Configuration fails when using --with-64-bit-indices simultaneously
> > > with --with-mkl_sparse=1 and/or --with-mkl_sparse_optimize=1 with the
> > > following possible errors:
> > >
> > > "Cannot use mkl_sparse with 64 bit integers, it is not coded for this
> > > capability"
> > > "Cannot use mkl_sparse_optimize with 64 bit integers, it is not coded
> > > for this capability"
> > >
> > > But doesn't the latest version of MKL Sparse BLAS support 64 bit
> > > indices? It is using MKL_INT in mkl_spblas.h.
> > >
> > > Is it possible to build PETSc with 64 bit indices and MKL Sparse BLAS?
> >