<div dir="ltr">I have put the Fieldsplit additive solver loop in an OpenMP loop and it seems to work with CPU solvers. The VecScatters seem to work and the KSP solver seems to work on the CPU. THis is an MPI serial code.<div><br></div><div>It fails with a cuSparse Jacobi solver with the error below, on each thread (two threads/blocks in this example).<div><div>I am not following this stack trace completely. eg, is the error in VecInitializePackage perhaps? </div><div>I don't understand the thread safety mechanism. (eg, I'm not sure why the CPU solvers worked), but I am thinking that this mechanism did not get used properly in cuSparse. Understandable. <br></div><div><br></div><div>Can anyone shed any light on what might be going wrong here?</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div><div>======= Backtrace: =========<br>/lib64/libc.so.6(cfree+0x4a0)[0x200021839be0]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(PetscFreeAlign+0x4c)[0x2000002a368c]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(PetscOptionsEnd_Private+0xf4)[0x2000002e53f0]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x7c6c28)[0x2000008b6c28]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecCreate_SeqCUDA+0x11c)[0x20000052c510]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecSetType+0x670)[0x200000549664]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecCreateSeqCUDA+0x150)[0x20000052c0b0]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x43c198)[0x20000052c198]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecDuplicate+0x44)[0x200000542168]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecDuplicateVecs_Default+0x148)[0x200000543820]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecDuplicateVecs+0x54)[0x2000005425f4]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPCreateVecs+0x4b4)[0x2000016f0aec]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x16b8b84)[0x2000017a8b84]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPSetUp+0x918)[0x2000016d8368]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x15ec754)[0x2000016dc754]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPSolve+0x44)[0x2000016df100]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x14ff5a8)[0x2000015ef5a8]<br>/sw/summit/gcc/6.4.0/lib64/libgomp.so.1(GOMP_parallel+0x74)[0x200021711074]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x14efb28)[0x2000015dfb28]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(PCApply+0x588)[0x200001685c70]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x16b6374)[0x2000017a6374]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x16b6738)[0x2000017a6738]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x15ed594)[0x2000016dd594]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPSolve+0x44)[0x2000016df100]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(SNESSolve_NEWTONLS+0x11f4)[0x20000189dcd8]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(SNESSolve+0x1b54)[0x2000018401a8]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x18b0d3c)[0x2000019a0d3c]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(TSStep+0x408)[0x2000019284a0]<br>/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(TSSolve+0x15cc)[0x20000192a334]<br>./ex2[0x1000a348]<br>/lib64/libc.so.6(+0x25200)[0x2000217c5200]<br>/lib64/libc.so.6(__libc_start_main+0xc4)[0x2000217c53f4]<br></div></div></div></div>