[petsc-users] MatSetValues error with ViennaCL types
Matthew Knepley
knepley at gmail.com
Thu Aug 30 05:10:18 CDT 2018
On Wed, Aug 29, 2018 at 6:21 PM Manuel Valera <mvalera-w at sdsu.edu> wrote:
> I understand that Matthew, thanks,
>
> I am trying to see if i get any speedup in this configuration, otherwise i
> will be waiting for the MPIAIJ support on PETSc,
>
> Is there any other gpu preconditioner that works on parallel matrices
> directly?
>
Not that I know of . Such a hypothetical preconditioner would need to pass
information back and forth, across the PCI bus, and
then through MPI (even if you imagine direct GPU-GPU communication, it
still hits the PCI bus). This channel has small bandwidth,
but even more limiting is the high latency. Its unclear what kind of
algorithm would do well in this situation.
Thanks,
Matt
> Thanks again,
>
>
>
> On Wed, Aug 29, 2018 at 3:05 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Wed, Aug 29, 2018 at 5:49 PM Manuel Valera <mvalera-w at sdsu.edu> wrote:
>>
>>> Update:
>>>
>>> I made it work like you suggested Barry, i had to comment the code line
>>> to set up the pc_type saviennacl, that way i am getting as ksp_view:
>>>
>>> KSP Object: 2 MPI processes
>>> type: cg
>>> maximum iterations=10000, initial guess is zero
>>> tolerances: relative=1e-07, absolute=1e-50, divergence=10000.
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 2 MPI processes
>>> type: bjacobi
>>> number of blocks = 2
>>> Local solve is same for all blocks, in the following KSP and PC
>>> objects:
>>> KSP Object: (sub_) 1 MPI processes
>>> type: preonly
>>> maximum iterations=10000, initial guess is zero
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
>>> left preconditioning
>>> using NONE norm type for convergence test
>>> PC Object: (sub_) 1 MPI processes
>>> type: ilu
>>> out-of-place factorization
>>> 0 levels of fill
>>> tolerance for zero pivot 2.22045e-14
>>> matrix ordering: natural
>>> factor fill ratio given 1., needed 1.
>>> Factored matrix follows:
>>> Mat Object: 1 MPI processes
>>> type: seqaij
>>> rows=75, cols=75
>>> package used to perform factorization: petsc
>>> total: nonzeros=927, allocated nonzeros=927
>>> total number of mallocs used during MatSetValues calls =0
>>> not using I-node routines
>>> linear system matrix = precond matrix:
>>> Mat Object: 1 MPI processes
>>> type: seqaijviennacl
>>> rows=75, cols=75
>>> total: nonzeros=927, allocated nonzeros=1335
>>> total number of mallocs used during MatSetValues calls =84
>>> not using I-node routines
>>> linear system matrix = precond matrix:
>>> Mat Object: 2 MPI processes
>>> type: mpiaijviennacl
>>> rows=125, cols=125
>>> total: nonzeros=1685, allocated nonzeros=2885
>>> total number of mallocs used during MatSetValues calls =184
>>> has attached near null space
>>> Press_Max: 1.9995807630564922E-004
>>>
>>>
>>> This is what we want, right?
>>>
>>> Still if i try using pc_type saviennacl directly it breaks,
>>>
>>
>> You are missing the point of the error message.
>>
>> We do not support SAVIENNACL on parallel matrices (MPIAIJ). We do support
>> it
>> on serial matrices (SEQAIJ). When you are setting in directly, you are
>> applying it
>> to a parallel matrix. If you use Block-Jacobi, you are applying it to
>> each submatrix,
>> and those submatrices are all serial.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Thanks for your help i'll try my speedup like this,
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 29, 2018 at 2:21 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>> wrote:
>>>
>>>> Ok, executing with:
>>>>
>>>> mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/
>>>> jid=tiny_cuda_test_n1 -pc_type bjacobi -pc_sub_type saviennacl -ksp_view
>>>>
>>>>
>>>> I get:
>>>>
>>>>
>>>> SETTING GPU TYPES
>>>> Matrix type: mpiaijviennacl
>>>>
>>>> Of sizes: 125 x 125
>>>> Matrix type: mpiaijviennacl
>>>>
>>>> Of sizes: 125 x 125
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.3-1264-g6fb0cd4
>>>> GIT Date: 2018-08-29 12:42:36 -0600
>>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed
>>>> Aug 29 14:20:42 2018
>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-cc=mpicc
>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>>>> --download-viennacl --download-cusp
>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>> [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>> [1]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>>> probably memory access out of range
>>>> [1]PETSC ERROR: Try option -start_in_debugger or
>>>> -on_error_attach_debugger
>>>> [1]PETSC ERROR: or see
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>>> OS X to find memory corruption errors
>>>> [1]PETSC ERROR: likely location of problem given in stack below
>>>> [1]PETSC ERROR: --------------------- Stack Frames
>>>> ------------------------------------
>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>> available,
>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the
>>>> function
>>>> [1]PETSC ERROR: is given.
>>>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>> [1]PETSC ERROR: [1] PetscError line 352
>>>> /home/valera/petsc/src/sys/error/err.c
>>>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>> [1]PETSC ERROR: [1] PCSetUp line 894
>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>> [1]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [1]PETSC ERROR: Signal received
>>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.3-1264-g6fb0cd4
>>>> GIT Date: 2018-08-29 12:42:36 -0600
>>>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed
>>>> Aug 29 14:20:42 2018
>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-cc=mpicc
>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>>>> --download-viennacl --download-cusp
>>>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
>>>>
>>>> --------------------------------------------------------------------------
>>>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>> with errorcode 59.
>>>>
>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>> You may or may not see output from other processes, depending on
>>>> exactly when Open MPI kills them.
>>>>
>>>> --------------------------------------------------------------------------
>>>> [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>>> batch system) has told this process to end
>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>> -on_error_attach_debugger
>>>> [0]PETSC ERROR: or see
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>>> OS X to find memory corruption errors
>>>> [0]PETSC ERROR: likely location of problem given in stack below
>>>> [0]PETSC ERROR: --------------------- Stack Frames
>>>> ------------------------------------
>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>> available,
>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the
>>>> function
>>>> [0]PETSC ERROR: is given.
>>>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>>> /home/valera/petsc/src/sys/objects/tagm.c
>>>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 41
>>>> /home/valera/petsc/src/sys/objects/inherit.c
>>>> [0]PETSC ERROR: [0] ISCreate line 35
>>>> /home/valera/petsc/src/vec/is/is/interface/isreg.c
>>>> [0]PETSC ERROR: [0] ISCreateGeneral line 658
>>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>> [0]PETSC ERROR: [0] PCSetUp line 894
>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [0]PETSC ERROR: Signal received
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.3-1264-g6fb0cd4
>>>> GIT Date: 2018-08-29 12:42:36 -0600
>>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed
>>>> Aug 29 14:20:42 2018
>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-cc=mpicc
>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>>>> --download-viennacl --download-cusp
>>>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file
>>>> [node50:77836] 1 more process has sent help message help-mpi-api.txt /
>>>> mpi-abort
>>>> [node50:77836] Set MCA parameter "orte_base_help_aggregate" to 0 to see
>>>> all help / error messages
>>>>
>>>>
>>>>
>>>> On Wed, Aug 29, 2018 at 2:02 PM, Smith, Barry F. <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>> Please send complete error message
>>>>>
>>>>>
>>>>> > On Aug 29, 2018, at 3:40 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>> wrote:
>>>>> >
>>>>> > Yeah, no sorry, i get the same error with -pc_type bjacobi
>>>>> -sub_pc_type SAVIENNACL : "Currently only handles ViennaCL matrices"
>>>>> >
>>>>> >
>>>>> > Thanks and let me know of any progress on this issue,
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Aug 29, 2018 at 1:37 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>> wrote:
>>>>> > Awesome, thanks!
>>>>> >
>>>>> > On Wed, Aug 29, 2018 at 1:29 PM, Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>> >
>>>>> >
>>>>> > > On Aug 29, 2018, at 3:26 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>> wrote:
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > You may need to use just plain PCBJACOBI or PCASM for parallelism
>>>>> and then SAVIENNACL sequentially on each block.
>>>>> > >
>>>>> > >
>>>>> > > Can you elaborate a little in how to do this? do you mean i can
>>>>> only use viennacl in serial at the moment or this is a suggestion of a
>>>>> workaround on how to make it work?
>>>>> >
>>>>> > -pc_type bjacobi -sub_pc_type SAVIENNACL
>>>>> >
>>>>> > This will run in parallel and utilize the GPU for each block of
>>>>> the preconditioner.
>>>>> >
>>>>> > Barry
>>>>> >
>>>>> > >
>>>>> > >
>>>>> > > Thanks,
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > > On Aug 29, 2018, at 1:50 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>> wrote:
>>>>> > > >
>>>>> > > > Hi everyone,
>>>>> > > >
>>>>> > > > Thanks for your responses, i understand communicating on this
>>>>> way to this level of technicality can be hard, i still think we can work
>>>>> ways to solve this problem though,
>>>>> > > >
>>>>> > > > I can say the following at this point:
>>>>> > > >
>>>>> > > > • Program works without issues in any other non-gpu
>>>>> preconditioner, just by calling -pc_type ### .
>>>>> > > > • Program works correctly with ViennaCL preconditioners
>>>>> with mpirun -n 1, that is on one processor.
>>>>> > > > • Program breaks with every ViennaCL preconditioner when
>>>>> attempting any more than 1 processors.
>>>>> > > > • I haven't tried other GPU preconditioners because
>>>>> ViennaCL gave the best performance in 1 processor before.
>>>>> > > > • I have made sure the matrix type is mpiaijviennacl as it
>>>>> is printed right before the error log.
>>>>> > > > • I am attaching what i think are the backtrace logs of
>>>>> the run with two cores, as i have limited experience with this kind of
>>>>> debugging, and i cannot make sense of what they say, please let me know if
>>>>> you need me to do something else.
>>>>> > > >
>>>>> > > > Current options i can think of are giving you permission to
>>>>> clone my model repo, or for you to guide me trough the process remotely,
>>>>> > > >
>>>>> > > > Thanks,
>>>>> > > >
>>>>> > > > .-.-.-.-
>>>>> > > >
>>>>> > > >
>>>>> > > > The error i get is still:
>>>>> > > >
>>>>> > > > Matrix type: mpiaijviennacl
>>>>>
>>>>> > > > Of sizes: 125 x 125
>>>>> > > > Matrix type: mpiaijviennacl
>>>>>
>>>>> > > > Of sizes: 125 x 125
>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR: No support for this operation for this object
>>>>> type
>>>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>> > > > [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>> valera Wed Aug 29 11:43:25 2018
>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> > > > [1]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>> Violation, probably memory access out of range
>>>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> > > > [1]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>> Apple Mac OS X to find memory corruption errors
>>>>> > > > [1]PETSC ERROR: likely location of problem given in stack below
>>>>> > > > [1]PETSC ERROR: --------------------- Stack Frames
>>>>> ------------------------------------
>>>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>> not available,
>>>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start of
>>>>> the function
>>>>> > > > [1]PETSC ERROR: is given.
>>>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>> > > > [1]PETSC ERROR: [1] PetscError line 352
>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [1]PETSC ERROR: [1] KSPSetUp line 294
>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> > > > [1]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > > > [1]PETSC ERROR: Signal received
>>>>> > > > [1]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>> valera Wed Aug 29 11:43:25 2018
>>>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown
>>>>> file
>>>>> > > >
>>>>> --------------------------------------------------------------------------
>>>>> > > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>>> > > > with errorcode 59.
>>>>> > > >
>>>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>> processes.
>>>>> > > > You may or may not see output from other processes, depending on
>>>>> > > > exactly when Open MPI kills them.
>>>>> > > >
>>>>> --------------------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>> (or the batch system) has told this process to end
>>>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> > > > [0]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>> Apple Mac OS X to find memory corruption errors
>>>>> > > > [0]PETSC ERROR: likely location of problem given in stack below
>>>>> > > > [0]PETSC ERROR: --------------------- Stack Frames
>>>>> ------------------------------------
>>>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>> not available,
>>>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start of
>>>>> the function
>>>>> > > > [0]PETSC ERROR: is given.
>>>>> > > > [0]PETSC ERROR: [0] MatSetErrorIfFailure line 116
>>>>> /home/valera/petsc/src/mat/utils/gcreate.c
>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [0]PETSC ERROR: [0] KSPSetUp line 294
>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR: Signal received
>>>>> > > > [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>> valera Wed Aug 29 11:43:25 2018
>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > [0]PETSC ERROR: #4 User provided function() line 0 in unknown
>>>>> file
>>>>> > > > [node50:32783] 1 more process has sent help message
>>>>> help-mpi-api.txt / mpi-abort
>>>>> > > > [node50:32783] Set MCA parameter "orte_base_help_aggregate" to 0
>>>>> to see all help / error messages
>>>>> > > >
>>>>> > > >
>>>>> > > > On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <
>>>>> rupp at iue.tuwien.ac.at> wrote:
>>>>> > > > Hi Manuel,
>>>>> > > >
>>>>> > > > as Barry said, it is hard for us to provide any help without
>>>>> having a more complete picture of what is going on.
>>>>> > > >
>>>>> > > > The error you report seems to come from the AMG preconditioner
>>>>> in ViennaCL. This can have many origins. Do other preconditioners run
>>>>> without error? Can you run in a debugger and provide a complete backtrace?
>>>>> > > >
>>>>> > > > Thanks and best regards,
>>>>> > > > Karli
>>>>> > > >
>>>>> > > >
>>>>> > > > On 08/29/2018 01:33 AM, Manuel Valera wrote:
>>>>> > > > Talked too fast,
>>>>> > > >
>>>>> > > > After fixing that problem, i tried more than one mpi processor
>>>>> and got the following:
>>>>> > > >
>>>>> > > > Matrix type: mpiaijviennacl
>>>>> > > > Of sizes: 125 x 125
>>>>> > > > Matrix type: mpiaijviennacl
>>>>> > > > Of sizes: 125 x 125
>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR: No support for this operation for this object
>>>>> type
>>>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>> > > > [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>> valera Tue Aug 28 16:30:02 2018
>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>>>>> http://saviennacl.cu>
>>>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [1]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>> Violation, probably memory access out of range
>>>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> > > > [1]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>> Apple Mac OS X to find memory corruption errors
>>>>> > > > [1]PETSC ERROR: likely location of problem given in stack below
>>>>> > > > [1]PETSC ERROR: --------------------- Stack Frames
>>>>> ------------------------------------
>>>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>> not available,
>>>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start of
>>>>> the function
>>>>> > > > [1]PETSC ERROR: is given.
>>>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>> > > > [1]PETSC ERROR: [1] PetscError line 352
>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu<
>>>>> http://saviennacl.cu>
>>>>> > > >
>>>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [1]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > > > [1]PETSC ERROR: Signal received
>>>>> > > > [1]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>> valera Tue Aug 28 16:30:02 2018
>>>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown
>>>>> file
>>>>> > > >
>>>>> --------------------------------------------------------------------------
>>>>> > > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>>> > > > with errorcode 59.
>>>>> > > >
>>>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>> processes.
>>>>> > > > You may or may not see output from other processes, depending on
>>>>> > > > exactly when Open MPI kills them.
>>>>> > > >
>>>>> --------------------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>> (or the batch system) has told this process to end
>>>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> > > > [0]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>> Apple Mac OS X to find memory corruption errors
>>>>> > > > [0]PETSC ERROR: likely location of problem given in stack below
>>>>> > > > [0]PETSC ERROR: --------------------- Stack Frames
>>>>> ------------------------------------
>>>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>> not available,
>>>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start of
>>>>> the function
>>>>> > > > [0]PETSC ERROR: is given.
>>>>> > > > [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>>>> /home/valera/petsc/src/sys/objects/tagm.c
>>>>> > > > [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34
>>>>> /home/valera/petsc/src/sys/objects/inherit.c
>>>>> > > > [0]PETSC ERROR: [0] ISCreate line 35
>>>>> /home/valera/petsc/src/vec/is/is/interface/isreg.c
>>>>> > > > [0]PETSC ERROR: [0] ISCreateGeneral line 668
>>>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu<
>>>>> http://saviennacl.cu>
>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > > > [0]PETSC ERROR: Signal received
>>>>> > > > [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>> valera Tue Aug 28 16:30:02 2018
>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > [0]PETSC ERROR: #3 User provided function() line 0 in unknown
>>>>> file
>>>>> > > > [node50:30582] 1 more process has sent help message
>>>>> help-mpi-api.txt / mpi-abort
>>>>> > > > [node50:30582] Set MCA parameter "orte_base_help_aggregate" to 0
>>>>> to see all help / error messages
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > It is currently running in 1 mpi processor + GPU but i would
>>>>> like to call at least 16 mpi processors + GPU to do the rest of the data
>>>>> management who is not part of the main laplacian on the mpi and the
>>>>> laplacian solution on the GPU, is this currently possible?
>>>>> > > >
>>>>> > > > Thanks for your help,
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera <
>>>>> mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > >
>>>>> > > > Ok, i found the culprit and we can close this thread,
>>>>> > > >
>>>>> > > > The problem was a missing variable for setting the maximum
>>>>> columns,
>>>>> > > > which i deleted at some point without realizing. The error
>>>>> message
>>>>> > > > was too ambiguous to catch this so i had to compare with a
>>>>> previous
>>>>> > > > working version of the arguments of MatSetValues, it was
>>>>> evident then.
>>>>> > > >
>>>>> > > > Good news is that i can now set the values with the viennacl
>>>>> types too,
>>>>> > > >
>>>>> > > > Thanks for your kind help,
>>>>> > > >
>>>>> > > > Manuel
>>>>> > > >
>>>>> > > > On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F.
>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>> > > >
>>>>> > > >
>>>>> > > > 1) PetscMalloc() is never valid or needed in Fortran
>>>>> > > >
>>>>> > > > 2) there is no reason to use
>>>>> DMSetMatrixPreallocateOnly()
>>>>> > > > just use DMCreateMatrix() assuming that using a DM (DMDA,
>>>>> > > > DMPLEX, etc) is suitable for your problem.
>>>>> > > >
>>>>> > > > At this end we are totally guessing at what you are
>>>>> doing
>>>>> > > > and so have little help we can provide. A simple,
>>>>> nonworking
>>>>> > > > code that tries to do what you would need would help us
>>>>> a great
>>>>> > > > deal in understanding that you are trying to do.
>>>>> > > >
>>>>> > > > Barry
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > > On Aug 28, 2018, at 1:18 PM, Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > >
>>>>> > > > > Matthew, PetscMalloc gives the same error,
>>>>> > > > >
>>>>> > > > > Barry, it would be very hard for me to get the code
>>>>> to a
>>>>> > > > minimum working example, i guess all i need to
>>>>> understand is how
>>>>> > > > to setup a DM matrix with DMSetMatrixPreallocateOnly()
>>>>> instead
>>>>> > > > of MatMPIAIJSetPreallocation() as we were doing before,
>>>>> is there
>>>>> > > > a simple example who does this in Fortran?
>>>>> > > > >
>>>>> > > > > Is the PetscMalloc call needed? is 'call
>>>>> > > > PetscMalloc(1,row,ierr)' a valid, compilable call to
>>>>> > > > PetscMalloc? what other reason may there be for this
>>>>> error to
>>>>> > > > happen ?
>>>>> > > > >
>>>>> > > > > Just remembering, that trying to setup the matrix
>>>>> with the
>>>>> > > > MatAIJSetPreallocation() brings up an error to
>>>>> acknowledge the
>>>>> > > > viennacl datatypes and that's why i'm trying to make
>>>>> this change
>>>>> > > > on your recommendation,
>>>>> > > > >
>>>>> > > > > Thanks for your help,
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F.
>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>> > > > >
>>>>> > > > > Send your code in a way we can compile and run it;
>>>>> it must
>>>>> > > > be some simple issue that is hard to communicate in
>>>>> email.
>>>>> > > > >
>>>>> > > > > Barry
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > > On Aug 27, 2018, at 5:51 PM, Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > > >
>>>>> > > > > > Hello everyone,
>>>>> > > > > >
>>>>> > > > > > I just had time to work on this again, and checked
>>>>> the code
>>>>> > > > for errors on the matrix entries, this is the exact code
>>>>> i was
>>>>> > > > using for creating the matrix without
>>>>> > > > DMSetMatrixPreallocateOnly, using
>>>>> MatMPIAIJSetPreallocation and
>>>>> > > > it worked that way, but trying this way i get the same
>>>>> 'Column
>>>>> > > > too large' error using any number at the column position
>>>>> of
>>>>> > > > MatSetValues,
>>>>> > > > > >
>>>>> > > > > > I have set up my code to print the column argument
>>>>> (n) of
>>>>> > > > MatSetValues and in this case is 7 (lower than 124), it
>>>>> still
>>>>> > > > gives error, even entering a specific number in the
>>>>> MatSetValues
>>>>> > > > column argument position gives the same error.
>>>>> > > > > >
>>>>> > > > > > So next i went back to ex.50 here:
>>>>> > > >
>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex50.c.html
>>>>> > > > <
>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex50.c.html
>>>>> >
>>>>> > > > and it has a very similar structure except the
>>>>> PetscMalloc1()
>>>>> > > > call, so i tried adding that and got:
>>>>> > > > > >
>>>>> > > > > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114:
>>>>> undefined
>>>>> > > > reference to `petscmalloc1_'
>>>>> > > > > >
>>>>> > > > > > Any ideas on this behaviour?
>>>>> > > > > >
>>>>> > > > > > Thanks so much,
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F.
>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>> > > > > >
>>>>> > > > > > Column too large: col 10980 max 124
>>>>> > > > > >
>>>>> > > > > > You need to check the code that is generating
>>>>> the matrix
>>>>> > > > entries. The matrix has 124 columns but you are
>>>>> attempting to
>>>>> > > > put a value at column 10980
>>>>> > > > > >
>>>>> > > > > > Barry
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > > > >
>>>>> > > > > > > Thanks Matthew and Barry,
>>>>> > > > > > >
>>>>> > > > > > > Now my code looks like:
>>>>> > > > > > >
>>>>> > > > > > > call
>>>>> DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>>>>> > > > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>> > > > > > > call MatSetFromOptions(A,ierr)
>>>>> > > > > > > call MatSetUp(A,ierr)
>>>>> > > > > > > [...]
>>>>> > > > > > > call
>>>>> > > >
>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>> > > > > > > [...]
>>>>> > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr)
>>>>> > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr)
>>>>> > > > > > >
>>>>> > > > > > > And i get a different error, now is:
>>>>> > > > > > >
>>>>> > > > > > > [0]PETSC ERROR: --------------------- Error
>>>>> Message
>>>>> > > >
>>>>> --------------------------------------------------------------
>>>>> > > > > > > [0]PETSC ERROR: Argument out of range
>>>>> > > > > > > [0]PETSC ERROR: Column too large: col 10980 max
>>>>> 124
>>>>> > > > > > > [0]PETSC ERROR: See
>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>> for
>>>>> > > > trouble shooting.
>>>>> > > > > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>> named node50
>>>>> > > > by valera Wed Aug 15 19:40:00 2018
>>>>> > > > > > > [0]PETSC ERROR: Configure options
>>>>> PETSC_ARCH=cuda-debug
>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>> --with-shared-libraries=1
>>>>> > > > --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 442
>>>>> in
>>>>> > > > /home/valera/petsc/src/mat/impls/aij/seq/aij.c
>>>>> > > > > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in
>>>>> > > > /home/valera/petsc/src/mat/interface/matrix.c
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > Thanks again,
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry F.
>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>> > > > > > >
>>>>> > > > > > > Should be
>>>>> > > > > > >
>>>>> > > > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>> > > > > > >
>>>>> > > > > > > and remove the rest. You need to set the type
>>>>> of Mat
>>>>> > > > you want the DM to return BEFORE you create the matrix.
>>>>> > > > > > >
>>>>> > > > > > > Barry
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > > > > >
>>>>> > > > > > > > Ok thanks for clarifying that, i wasn't sure if
>>>>> there
>>>>> > > > were different types,
>>>>> > > > > > > >
>>>>> > > > > > > > Here is a stripped down version of my code, it
>>>>> seems
>>>>> > > > like the preallocation is working now since the matrix
>>>>> > > > population part is working without problem, but here it
>>>>> is for
>>>>> > > > illustration purposes:
>>>>> > > > > > > >
>>>>> > > > > > > > call
>>>>> DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>>>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>> > > > > > > > call MatSetFromOptions(A,ierr)
>>>>> > > > > > > > call
>>>>> DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>> > > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>> > > > > > > > call
>>>>> > > >
>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr)
>>>>> > > > > > > > call MatSetUp(A,ierr)
>>>>> > > > > > > > [...]
>>>>> > > > > > > > call
>>>>> > > >
>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>> > > > > > > > [...]
>>>>> > > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY,
>>>>> ierr)
>>>>> > > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr)
>>>>> > > > > > > >
>>>>> > > > > > > > Adding the first line there did the trick,
>>>>> > > > > > > >
>>>>> > > > > > > > Now the problem seems to be the program is not
>>>>> > > > recognizing the matrix as ViennaCL type when i try with
>>>>> more
>>>>> > > > than one processor, i get now:
>>>>> > > > > > > >
>>>>> > > > > > > > [0]PETSC ERROR: --------------------- Error
>>>>> Message
>>>>> > > >
>>>>> --------------------------------------------------------------
>>>>> > > > > > > > [0]PETSC ERROR: No support for this operation
>>>>> for this
>>>>> > > > object type
>>>>> > > > > > > > [0]PETSC ERROR: Currently only handles ViennaCL
>>>>> matrices
>>>>> > > > > > > > [0]PETSC ERROR: See
>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>> for
>>>>> > > > trouble shooting.
>>>>> > > > > > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>> named
>>>>> > > > node50 by valera Wed Aug 15 14:44:22 2018
>>>>> > > > > > > > [0]PETSC ERROR: Configure options
>>>>> PETSC_ARCH=cuda-debug
>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>> --with-shared-libraries=1
>>>>> > > > --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47
>>>>> in
>>>>> > > > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/
>>>>> saviennacl.cu
>>>>> > > > <http://saviennacl.cu>
>>>>> > > > > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>> > > > /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>> > > > > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>>>> > > > /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> > > > > > > >
>>>>> > > > > > > > When running with:
>>>>> > > > > > > >
>>>>> > > > > > > > mpirun -n 1 ./gcmLEP.GPU
>>>>> > > > tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2
>>>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type
>>>>> aijviennacl
>>>>> > > > -pc_type saviennacl -log_view
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > Thanks,
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley
>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>> > > > > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > > > > > It seems to be resumed on: I do not know how to
>>>>> > > > preallocate a DM Matrix correctly.
>>>>> > > > > > > >
>>>>> > > > > > > > There is only one matrix type, Mat. There are no
>>>>> > > > separate DM matrices. A DM can create a matrix for you
>>>>> > > > > > > > using DMCreateMatrix(), but that is a Mat and
>>>>> it is
>>>>> > > > preallocated correctly. I am not sure what you are doing.
>>>>> > > > > > > >
>>>>> > > > > > > > Thanks,
>>>>> > > > > > > >
>>>>> > > > > > > > Matt
>>>>> > > > > > > >
>>>>> > > > > > > > The interesting part is that it only breaks
>>>>> when i need
>>>>> > > > to populate a GPU matrix from MPI, so kudos on that, but
>>>>> it
>>>>> > > > seems i need to do better on my code to get this setup
>>>>> working,
>>>>> > > > > > > >
>>>>> > > > > > > > Any help would be appreciated,
>>>>> > > > > > > >
>>>>> > > > > > > > Thanks,
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley
>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>> > > > > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > > > > > Thanks Matthew,
>>>>> > > > > > > >
>>>>> > > > > > > > I try to do that when calling:
>>>>> > > > > > > >
>>>>> > > > > > > > call
>>>>> > > >
>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr)
>>>>> > > > > > > >
>>>>> > > > > > > > But i am not aware on how to do this for the DM
>>>>> if it
>>>>> > > > needs something more specific/different,
>>>>> > > > > > > >
>>>>> > > > > > > > The error says that your preallocation is wrong
>>>>> for the
>>>>> > > > values you are putting in. The DM does not control
>>>>> either,
>>>>> > > > > > > > so I do not understand your email.
>>>>> > > > > > > >
>>>>> > > > > > > > Thanks,
>>>>> > > > > > > >
>>>>> > > > > > > > Matt
>>>>> > > > > > > >
>>>>> > > > > > > > Thanks,
>>>>> > > > > > > >
>>>>> > > > > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley
>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>> > > > > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera
>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>> > > > > > > > Hello PETSc devs,
>>>>> > > > > > > >
>>>>> > > > > > > > I am running into an error when trying to use
>>>>> the
>>>>> > > > MATMPIAIJVIENNACL Matrix type in MPI calls, the same
>>>>> code runs
>>>>> > > > for MATSEQAIJVIENNACL type in one processor. The error
>>>>> happens
>>>>> > > > when calling MatSetValues for this specific
>>>>> configuration. It
>>>>> > > > does not occur when using MPI DMMatrix types only.
>>>>> > > > > > > >
>>>>> > > > > > > > The DM properly preallocates the matrix. I am
>>>>> assuming
>>>>> > > > you do not here.
>>>>> > > > > > > >
>>>>> > > > > > > > Matt
>>>>> > > > > > > >
>>>>> > > > > > > > Any help will be appreciated,
>>>>> > > > > > > >
>>>>> > > > > > > > Thanks,
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > My program call:
>>>>> > > > > > > >
>>>>> > > > > > > > mpirun -n 2 ./gcmLEP.GPU
>>>>> > > > tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2
>>>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type
>>>>> aijviennacl
>>>>> > > > -pc_type saviennacl -log_view
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > The error (repeats after each MatSetValues
>>>>> call):
>>>>> > > > > > > >
>>>>> > > > > > > > [1]PETSC ERROR: --------------------- Error
>>>>> Message
>>>>> > > >
>>>>> --------------------------------------------------------------
>>>>> > > > > > > > [1]PETSC ERROR: Argument out of range
>>>>> > > > > > > > [1]PETSC ERROR: Inserting a new nonzero at
>>>>> global
>>>>> > > > row/column (75, 50) into matrix
>>>>> > > > > > > > [1]PETSC ERROR: See
>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>> for
>>>>> > > > trouble shooting.
>>>>> > > > > > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>> > > > > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>> named
>>>>> > > > node50 by valera Wed Aug 15 13:10:44 2018
>>>>> > > > > > > > [1]PETSC ERROR: Configure options
>>>>> PETSC_ARCH=cuda-debug
>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>> --with-shared-libraries=1
>>>>> > > > --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>> > > > > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line
>>>>> 608 in
>>>>> > > > /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>>>> > > > > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339 in
>>>>> > > > /home/valera/petsc/src/mat/interface/matrix.c
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > My Code structure:
>>>>> > > > > > > >
>>>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>> > > > > > > > call MatSetFromOptions(A,ierr)
>>>>> > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD, numprocs,
>>>>> ierr)
>>>>> > > > > > > > if (numprocs > 1) then ! set matrix type
>>>>> parallel
>>>>> > > > > > > > ! Get local size
>>>>> > > > > > > > call
>>>>> DMDACreateNaturalVector(daDummy,Tmpnat,ierr)
>>>>> > > > > > > > call VecGetLocalSize(Tmpnat,locsize,ierr)
>>>>> > > > > > > > call VecDestroy(Tmpnat,ierr)
>>>>> > > > > > > > ! Set matrix
>>>>> > > > > > > > #ifdef GPU
>>>>> > > > > > > > call MatSetType(A,MATAIJVIENNACL,ierr)
>>>>> > > > > > > > call
>>>>> DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>> > > > > > > > call
>>>>> DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>> > > > > > > > print*,'SETTING GPU TYPES'
>>>>> > > > > > > > #else
>>>>> > > > > > > > call DMSetMatType(daDummy,MATMPIAIJ,ierr)
>>>>> > > > > > > > call DMSetMatType(daDummy,VECMPI,ierr)
>>>>> > > > > > > > call MatSetType(A,MATMPIAIJ,ierr)!
>>>>> > > > > > > > #endif
>>>>> > > > > > > > call
>>>>> > > >
>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr)
>>>>> > > > > > > > else ! set matrix type
>>>>> sequential
>>>>> > > > > > > > #ifdef GPU
>>>>> > > > > > > > call
>>>>> DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr)
>>>>> > > > > > > > call
>>>>> DMSetVecType(daDummy,VECSEQVIENNACL,ierr)
>>>>> > > > > > > > call MatSetType(A,MATSEQAIJVIENNACL,ierr)
>>>>> > > > > > > > print*,'SETTING GPU TYPES'
>>>>> > > > > > > > #else
>>>>> > > > > > > > call DMSetMatType(daDummy,MATSEQAIJ,ierr)
>>>>> > > > > > > > call DMSetMatType(daDummy,VECSEQ,ierr)
>>>>> > > > > > > > call MatSetType(A,MATSEQAIJ,ierr)
>>>>> > > > > > > > #endif
>>>>> > > > > > > > call MatSetUp(A,ierr)
>>>>> > > > > > > > call
>>>>> > > > getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend)
>>>>> > > > > > > >
>>>>> > > > > > > > do k=zstart,zend-1
>>>>> > > > > > > > do j=ystart,yend-1
>>>>> > > > > > > > do i=xstart,xend-1
>>>>> > > > > > > > [..]
>>>>> > > > > > > > call
>>>>> > > >
>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>> > > > > > > > [..]
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > --
>>>>> > > > > > > > What most experimenters take for granted before
>>>>> they
>>>>> > > > begin their experiments is infinitely more interesting
>>>>> than any
>>>>> > > > results to which their experiments lead.
>>>>> > > > > > > > -- Norbert Wiener
>>>>> > > > > > > >
>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > --
>>>>> > > > > > > > What most experimenters take for granted before
>>>>> they
>>>>> > > > begin their experiments is infinitely more interesting
>>>>> than any
>>>>> > > > results to which their experiments lead.
>>>>> > > > > > > > -- Norbert Wiener
>>>>> > > > > > > >
>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > >
>>>>> > > > > > > > --
>>>>> > > > > > > > What most experimenters take for granted before
>>>>> they
>>>>> > > > begin their experiments is infinitely more interesting
>>>>> than any
>>>>> > > > results to which their experiments lead.
>>>>> > > > > > > > -- Norbert Wiener
>>>>> > > > > > > >
>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>> > > > > > > >
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > <backt-2.log><backt.log>
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180830/d4dfa823/attachment-0001.html>
More information about the petsc-users
mailing list