[petsc-users] MatSetValues error with ViennaCL types
Manuel Valera
mvalera-w at sdsu.edu
Thu Aug 30 15:41:13 CDT 2018
Matthew, and anyone who wants to chip in on this,
Just to close this thread then, can you suggest a roadmap to get this
MPI+GPU hibrydization done? you mentioned the ASM PCs, i may look into
that, but is not clear to me now if this is doable now in PETSc or if i
have to wait for more functionality to be added,
I have two available machines, one with 20 cores and a P100 GPU, and other
with 8 V100 GPUs and MPI for communication, how would you guys do this?
My model is fully distributed with DMDAs except for output, and the linear
solver takes around 56% of the computation time,
Thanks,
On Thu, Aug 30, 2018 at 8:25 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Thu, Aug 30, 2018 at 10:52 AM Manuel Valera <mvalera-w at sdsu.edu> wrote:
>
>> I thought newer NVIDIA cards had integrated circuits, not using PCI
>> anymore, still the same case?
>>
>> How would the GPU clusters work, then? Let's say i have a cluster of GPUs
>> communicated by an MPI node, would PETSc work on this configuration at some
>> point? This would be my next step if I could make the MPI matrices work
>> with GPU, we have such a cluster on campus.
>>
>
> That works now, in that you just use more loosely coupled preconditioners,
> like ASM. That was my point.
> Its unlikely that tightly coupled PCs would be useful in this regime.
>
>
>> Are you familiar with the Magma library? It was recommended recently to
>> tackle this problem,
>>
>
> It does not tackle this problem.
>
> Thanks,
>
> Matt
>
>
>> I am most interested in your insight,
>>
>> Thanks,
>>
>> On Thu, Aug 30, 2018, 3:10 AM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Wed, Aug 29, 2018 at 6:21 PM Manuel Valera <mvalera-w at sdsu.edu>
>>> wrote:
>>>
>>>> I understand that Matthew, thanks,
>>>>
>>>> I am trying to see if i get any speedup in this configuration,
>>>> otherwise i will be waiting for the MPIAIJ support on PETSc,
>>>>
>>>> Is there any other gpu preconditioner that works on parallel matrices
>>>> directly?
>>>>
>>>
>>> Not that I know of . Such a hypothetical preconditioner would need to
>>> pass information back and forth, across the PCI bus, and
>>> then through MPI (even if you imagine direct GPU-GPU communication, it
>>> still hits the PCI bus). This channel has small bandwidth,
>>> but even more limiting is the high latency. Its unclear what kind of
>>> algorithm would do well in this situation.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Thanks again,
>>>>
>>>>
>>>>
>>>> On Wed, Aug 29, 2018 at 3:05 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, Aug 29, 2018 at 5:49 PM Manuel Valera <mvalera-w at sdsu.edu>
>>>>> wrote:
>>>>>
>>>>>> Update:
>>>>>>
>>>>>> I made it work like you suggested Barry, i had to comment the code
>>>>>> line to set up the pc_type saviennacl, that way i am getting as ksp_view:
>>>>>>
>>>>>> KSP Object: 2 MPI processes
>>>>>> type: cg
>>>>>> maximum iterations=10000, initial guess is zero
>>>>>> tolerances: relative=1e-07, absolute=1e-50, divergence=10000.
>>>>>> left preconditioning
>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>> PC Object: 2 MPI processes
>>>>>> type: bjacobi
>>>>>> number of blocks = 2
>>>>>> Local solve is same for all blocks, in the following KSP and PC
>>>>>> objects:
>>>>>> KSP Object: (sub_) 1 MPI processes
>>>>>> type: preonly
>>>>>> maximum iterations=10000, initial guess is zero
>>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
>>>>>> left preconditioning
>>>>>> using NONE norm type for convergence test
>>>>>> PC Object: (sub_) 1 MPI processes
>>>>>> type: ilu
>>>>>> out-of-place factorization
>>>>>> 0 levels of fill
>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>> matrix ordering: natural
>>>>>> factor fill ratio given 1., needed 1.
>>>>>> Factored matrix follows:
>>>>>> Mat Object: 1 MPI processes
>>>>>> type: seqaij
>>>>>> rows=75, cols=75
>>>>>> package used to perform factorization: petsc
>>>>>> total: nonzeros=927, allocated nonzeros=927
>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>> not using I-node routines
>>>>>> linear system matrix = precond matrix:
>>>>>> Mat Object: 1 MPI processes
>>>>>> type: seqaijviennacl
>>>>>> rows=75, cols=75
>>>>>> total: nonzeros=927, allocated nonzeros=1335
>>>>>> total number of mallocs used during MatSetValues calls =84
>>>>>> not using I-node routines
>>>>>> linear system matrix = precond matrix:
>>>>>> Mat Object: 2 MPI processes
>>>>>> type: mpiaijviennacl
>>>>>> rows=125, cols=125
>>>>>> total: nonzeros=1685, allocated nonzeros=2885
>>>>>> total number of mallocs used during MatSetValues calls =184
>>>>>> has attached near null space
>>>>>> Press_Max: 1.9995807630564922E-004
>>>>>>
>>>>>>
>>>>>> This is what we want, right?
>>>>>>
>>>>>> Still if i try using pc_type saviennacl directly it breaks,
>>>>>>
>>>>>
>>>>> You are missing the point of the error message.
>>>>>
>>>>> We do not support SAVIENNACL on parallel matrices (MPIAIJ). We do
>>>>> support it
>>>>> on serial matrices (SEQAIJ). When you are setting in directly, you are
>>>>> applying it
>>>>> to a parallel matrix. If you use Block-Jacobi, you are applying it to
>>>>> each submatrix,
>>>>> and those submatrices are all serial.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> Thanks for your help i'll try my speedup like this,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 29, 2018 at 2:21 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Ok, executing with:
>>>>>>>
>>>>>>> mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/
>>>>>>> jid=tiny_cuda_test_n1 -pc_type bjacobi -pc_sub_type saviennacl -ksp_view
>>>>>>>
>>>>>>>
>>>>>>> I get:
>>>>>>>
>>>>>>>
>>>>>>> SETTING GPU TYPES
>>>>>>> Matrix type: mpiaijviennacl
>>>>>>>
>>>>>>> Of sizes: 125 x 125
>>>>>>> Matrix type: mpiaijviennacl
>>>>>>>
>>>>>>> Of sizes: 125 x 125
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>> v3.9.3-1264-g6fb0cd4 GIT Date: 2018-08-29 12:42:36 -0600
>>>>>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera
>>>>>>> Wed Aug 29 14:20:42 2018
>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2
>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl --download-cusp
>>>>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>> [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>> [1]PETSC ERROR: ------------------------------
>>>>>>> ------------------------------------------
>>>>>>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>> Violation, probably memory access out of range
>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger or
>>>>>>> -on_error_attach_debugger
>>>>>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
>>>>>>> documentation/faq.html#valgrind
>>>>>>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>> Mac OS X to find memory corruption errors
>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below
>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames
>>>>>>> ------------------------------------
>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>> available,
>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the
>>>>>>> function
>>>>>>> [1]PETSC ERROR: is given.
>>>>>>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>>>> [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/
>>>>>>> error/err.c
>>>>>>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>> [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
>>>>>>> interface/precon.c
>>>>>>> [1]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [1]PETSC ERROR: Signal received
>>>>>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>> [1]PETSC ERROR: Petsc Development GIT revision:
>>>>>>> v3.9.3-1264-g6fb0cd4 GIT Date: 2018-08-29 12:42:36 -0600
>>>>>>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera
>>>>>>> Wed Aug 29 14:20:42 2018
>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2
>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl --download-cusp
>>>>>>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
>>>>>>> ------------------------------------------------------------
>>>>>>> --------------
>>>>>>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>>>>> with errorcode 59.
>>>>>>>
>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>>>> You may or may not see output from other processes, depending on
>>>>>>> exactly when Open MPI kills them.
>>>>>>> ------------------------------------------------------------
>>>>>>> --------------
>>>>>>> [0]PETSC ERROR: ------------------------------
>>>>>>> ------------------------------------------
>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>> the batch system) has told this process to end
>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>> -on_error_attach_debugger
>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
>>>>>>> documentation/faq.html#valgrind
>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>> Mac OS X to find memory corruption errors
>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below
>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames
>>>>>>> ------------------------------------
>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>> available,
>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the
>>>>>>> function
>>>>>>> [0]PETSC ERROR: is given.
>>>>>>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>>>>>> /home/valera/petsc/src/sys/objects/tagm.c
>>>>>>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 41
>>>>>>> /home/valera/petsc/src/sys/objects/inherit.c
>>>>>>> [0]PETSC ERROR: [0] ISCreate line 35 /home/valera/petsc/src/vec/is/
>>>>>>> is/interface/isreg.c
>>>>>>> [0]PETSC ERROR: [0] ISCreateGeneral line 658
>>>>>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>>>>>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
>>>>>>> interface/precon.c
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>> v3.9.3-1264-g6fb0cd4 GIT Date: 2018-08-29 12:42:36 -0600
>>>>>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera
>>>>>>> Wed Aug 29 14:20:42 2018
>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2
>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl --download-cusp
>>>>>>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file
>>>>>>> [node50:77836] 1 more process has sent help message help-mpi-api.txt
>>>>>>> / mpi-abort
>>>>>>> [node50:77836] Set MCA parameter "orte_base_help_aggregate" to 0 to
>>>>>>> see all help / error messages
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 29, 2018 at 2:02 PM, Smith, Barry F. <bsmith at mcs.anl.gov
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Please send complete error message
>>>>>>>>
>>>>>>>>
>>>>>>>> > On Aug 29, 2018, at 3:40 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > Yeah, no sorry, i get the same error with -pc_type bjacobi
>>>>>>>> -sub_pc_type SAVIENNACL : "Currently only handles ViennaCL matrices"
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Thanks and let me know of any progress on this issue,
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Aug 29, 2018 at 1:37 PM, Manuel Valera <
>>>>>>>> mvalera-w at sdsu.edu> wrote:
>>>>>>>> > Awesome, thanks!
>>>>>>>> >
>>>>>>>> > On Wed, Aug 29, 2018 at 1:29 PM, Smith, Barry F. <
>>>>>>>> bsmith at mcs.anl.gov> wrote:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > > On Aug 29, 2018, at 3:26 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>>>>> wrote:
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > You may need to use just plain PCBJACOBI or PCASM for
>>>>>>>> parallelism and then SAVIENNACL sequentially on each block.
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > Can you elaborate a little in how to do this? do you mean i can
>>>>>>>> only use viennacl in serial at the moment or this is a suggestion of a
>>>>>>>> workaround on how to make it work?
>>>>>>>> >
>>>>>>>> > -pc_type bjacobi -sub_pc_type SAVIENNACL
>>>>>>>> >
>>>>>>>> > This will run in parallel and utilize the GPU for each block
>>>>>>>> of the preconditioner.
>>>>>>>> >
>>>>>>>> > Barry
>>>>>>>> >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > Thanks,
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > > On Aug 29, 2018, at 1:50 PM, Manuel Valera <
>>>>>>>> mvalera-w at sdsu.edu> wrote:
>>>>>>>> > > >
>>>>>>>> > > > Hi everyone,
>>>>>>>> > > >
>>>>>>>> > > > Thanks for your responses, i understand communicating on this
>>>>>>>> way to this level of technicality can be hard, i still think we can work
>>>>>>>> ways to solve this problem though,
>>>>>>>> > > >
>>>>>>>> > > > I can say the following at this point:
>>>>>>>> > > >
>>>>>>>> > > > • Program works without issues in any other non-gpu
>>>>>>>> preconditioner, just by calling -pc_type ### .
>>>>>>>> > > > • Program works correctly with ViennaCL preconditioners
>>>>>>>> with mpirun -n 1, that is on one processor.
>>>>>>>> > > > • Program breaks with every ViennaCL preconditioner
>>>>>>>> when attempting any more than 1 processors.
>>>>>>>> > > > • I haven't tried other GPU preconditioners because
>>>>>>>> ViennaCL gave the best performance in 1 processor before.
>>>>>>>> > > > • I have made sure the matrix type is mpiaijviennacl as
>>>>>>>> it is printed right before the error log.
>>>>>>>> > > > • I am attaching what i think are the backtrace logs of
>>>>>>>> the run with two cores, as i have limited experience with this kind of
>>>>>>>> debugging, and i cannot make sense of what they say, please let me know if
>>>>>>>> you need me to do something else.
>>>>>>>> > > >
>>>>>>>> > > > Current options i can think of are giving you permission to
>>>>>>>> clone my model repo, or for you to guide me trough the process remotely,
>>>>>>>> > > >
>>>>>>>> > > > Thanks,
>>>>>>>> > > >
>>>>>>>> > > > .-.-.-.-
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > The error i get is still:
>>>>>>>> > > >
>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>>
>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>>
>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: No support for this operation for this object
>>>>>>>> type
>>>>>>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>>>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>> valera Wed Aug 29 11:43:25 2018
>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>> > > > [1]PETSC ERROR: ------------------------------
>>>>>>>> ------------------------------------------
>>>>>>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>> Violation, probably memory access out of range
>>>>>>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> > > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html#valgrind
>>>>>>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>> > > > [1]PETSC ERROR: likely location of problem given in stack
>>>>>>>> below
>>>>>>>> > > > [1]PETSC ERROR: --------------------- Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>> not available,
>>>>>>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start of
>>>>>>>> the function
>>>>>>>> > > > [1]PETSC ERROR: is given.
>>>>>>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>>>>> > > > [1]PETSC ERROR: [1] PetscError line 352
>>>>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [1]PETSC ERROR: [1] KSPSetUp line 294
>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>> > > > [1]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > > > [1]PETSC ERROR: Signal received
>>>>>>>> > > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>>> > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>> valera Wed Aug 29 11:43:25 2018
>>>>>>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in
>>>>>>>> unknown file
>>>>>>>> > > > ------------------------------------------------------------
>>>>>>>> --------------
>>>>>>>> > > > MPI_ABORT was invoked on rank 1 in communicator
>>>>>>>> MPI_COMM_WORLD
>>>>>>>> > > > with errorcode 59.
>>>>>>>> > > >
>>>>>>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>> processes.
>>>>>>>> > > > You may or may not see output from other processes, depending
>>>>>>>> on
>>>>>>>> > > > exactly when Open MPI kills them.
>>>>>>>> > > > ------------------------------------------------------------
>>>>>>>> --------------
>>>>>>>> > > > [0]PETSC ERROR: ------------------------------
>>>>>>>> ------------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> > > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html#valgrind
>>>>>>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>> > > > [0]PETSC ERROR: likely location of problem given in stack
>>>>>>>> below
>>>>>>>> > > > [0]PETSC ERROR: --------------------- Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>> not available,
>>>>>>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start of
>>>>>>>> the function
>>>>>>>> > > > [0]PETSC ERROR: is given.
>>>>>>>> > > > [0]PETSC ERROR: [0] MatSetErrorIfFailure line 116
>>>>>>>> /home/valera/petsc/src/mat/utils/gcreate.c
>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [0]PETSC ERROR: [0] KSPSetUp line 294
>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: Signal received
>>>>>>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>> valera Wed Aug 29 11:43:25 2018
>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > [0]PETSC ERROR: #4 User provided function() line 0 in
>>>>>>>> unknown file
>>>>>>>> > > > [node50:32783] 1 more process has sent help message
>>>>>>>> help-mpi-api.txt / mpi-abort
>>>>>>>> > > > [node50:32783] Set MCA parameter "orte_base_help_aggregate"
>>>>>>>> to 0 to see all help / error messages
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <
>>>>>>>> rupp at iue.tuwien.ac.at> wrote:
>>>>>>>> > > > Hi Manuel,
>>>>>>>> > > >
>>>>>>>> > > > as Barry said, it is hard for us to provide any help without
>>>>>>>> having a more complete picture of what is going on.
>>>>>>>> > > >
>>>>>>>> > > > The error you report seems to come from the AMG
>>>>>>>> preconditioner in ViennaCL. This can have many origins. Do other
>>>>>>>> preconditioners run without error? Can you run in a debugger and provide a
>>>>>>>> complete backtrace?
>>>>>>>> > > >
>>>>>>>> > > > Thanks and best regards,
>>>>>>>> > > > Karli
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > On 08/29/2018 01:33 AM, Manuel Valera wrote:
>>>>>>>> > > > Talked too fast,
>>>>>>>> > > >
>>>>>>>> > > > After fixing that problem, i tried more than one mpi
>>>>>>>> processor and got the following:
>>>>>>>> > > >
>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: No support for this operation for this object
>>>>>>>> type
>>>>>>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>>>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>> valera Tue Aug 28 16:30:02 2018
>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>>>>>>>> http://saviennacl.cu>
>>>>>>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [1]PETSC ERROR: ------------------------------
>>>>>>>> ------------------------------------------
>>>>>>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>> Violation, probably memory access out of range
>>>>>>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> > > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html#valgrind
>>>>>>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>> > > > [1]PETSC ERROR: likely location of problem given in stack
>>>>>>>> below
>>>>>>>> > > > [1]PETSC ERROR: --------------------- Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>> not available,
>>>>>>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start of
>>>>>>>> the function
>>>>>>>> > > > [1]PETSC ERROR: is given.
>>>>>>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>>>>> > > > [1]PETSC ERROR: [1] PetscError line 352
>>>>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu<
>>>>>>>> http://saviennacl.cu>
>>>>>>>> > > >
>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [1]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > > > [1]PETSC ERROR: Signal received
>>>>>>>> > > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>>> > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>> valera Tue Aug 28 16:30:02 2018
>>>>>>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in
>>>>>>>> unknown file
>>>>>>>> > > > ------------------------------------------------------------
>>>>>>>> --------------
>>>>>>>> > > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>>>>>> > > > with errorcode 59.
>>>>>>>> > > >
>>>>>>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>> processes.
>>>>>>>> > > > You may or may not see output from other processes, depending
>>>>>>>> on
>>>>>>>> > > > exactly when Open MPI kills them.
>>>>>>>> > > > ------------------------------------------------------------
>>>>>>>> --------------
>>>>>>>> > > > [0]PETSC ERROR: ------------------------------
>>>>>>>> ------------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> > > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html#valgrind
>>>>>>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>> > > > [0]PETSC ERROR: likely location of problem given in stack
>>>>>>>> below
>>>>>>>> > > > [0]PETSC ERROR: --------------------- Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>> not available,
>>>>>>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start of
>>>>>>>> the function
>>>>>>>> > > > [0]PETSC ERROR: is given.
>>>>>>>> > > > [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>>>>>>> /home/valera/petsc/src/sys/objects/tagm.c
>>>>>>>> > > > [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34
>>>>>>>> /home/valera/petsc/src/sys/objects/inherit.c
>>>>>>>> > > > [0]PETSC ERROR: [0] ISCreate line 35
>>>>>>>> /home/valera/petsc/src/vec/is/is/interface/isreg.c
>>>>>>>> > > > [0]PETSC ERROR: [0] ISCreateGeneral line 668
>>>>>>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu<
>>>>>>>> http://saviennacl.cu>
>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > > > [0]PETSC ERROR: Signal received
>>>>>>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/
>>>>>>>> documentation/faq.html for trouble shooting.
>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>> valera Tue Aug 28 16:30:02 2018
>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > [0]PETSC ERROR: #3 User provided function() line 0 in
>>>>>>>> unknown file
>>>>>>>> > > > [node50:30582] 1 more process has sent help message
>>>>>>>> help-mpi-api.txt / mpi-abort
>>>>>>>> > > > [node50:30582] Set MCA parameter "orte_base_help_aggregate"
>>>>>>>> to 0 to see all help / error messages
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > It is currently running in 1 mpi processor + GPU but i would
>>>>>>>> like to call at least 16 mpi processors + GPU to do the rest of the data
>>>>>>>> management who is not part of the main laplacian on the mpi and the
>>>>>>>> laplacian solution on the GPU, is this currently possible?
>>>>>>>> > > >
>>>>>>>> > > > Thanks for your help,
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera <
>>>>>>>> mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>>>>> > > >
>>>>>>>> > > > Ok, i found the culprit and we can close this thread,
>>>>>>>> > > >
>>>>>>>> > > > The problem was a missing variable for setting the
>>>>>>>> maximum columns,
>>>>>>>> > > > which i deleted at some point without realizing. The
>>>>>>>> error message
>>>>>>>> > > > was too ambiguous to catch this so i had to compare with
>>>>>>>> a previous
>>>>>>>> > > > working version of the arguments of MatSetValues, it was
>>>>>>>> evident then.
>>>>>>>> > > >
>>>>>>>> > > > Good news is that i can now set the values with the
>>>>>>>> viennacl types too,
>>>>>>>> > > >
>>>>>>>> > > > Thanks for your kind help,
>>>>>>>> > > >
>>>>>>>> > > > Manuel
>>>>>>>> > > >
>>>>>>>> > > > On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F.
>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > 1) PetscMalloc() is never valid or needed in
>>>>>>>> Fortran
>>>>>>>> > > >
>>>>>>>> > > > 2) there is no reason to use
>>>>>>>> DMSetMatrixPreallocateOnly()
>>>>>>>> > > > just use DMCreateMatrix() assuming that using a DM
>>>>>>>> (DMDA,
>>>>>>>> > > > DMPLEX, etc) is suitable for your problem.
>>>>>>>> > > >
>>>>>>>> > > > At this end we are totally guessing at what you
>>>>>>>> are doing
>>>>>>>> > > > and so have little help we can provide. A simple,
>>>>>>>> nonworking
>>>>>>>> > > > code that tries to do what you would need would help
>>>>>>>> us a great
>>>>>>>> > > > deal in understanding that you are trying to do.
>>>>>>>> > > >
>>>>>>>> > > > Barry
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > > On Aug 28, 2018, at 1:18 PM, Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > >
>>>>>>>> > > > > Matthew, PetscMalloc gives the same error,
>>>>>>>> > > > >
>>>>>>>> > > > > Barry, it would be very hard for me to get the
>>>>>>>> code to a
>>>>>>>> > > > minimum working example, i guess all i need to
>>>>>>>> understand is how
>>>>>>>> > > > to setup a DM matrix with
>>>>>>>> DMSetMatrixPreallocateOnly() instead
>>>>>>>> > > > of MatMPIAIJSetPreallocation() as we were doing
>>>>>>>> before, is there
>>>>>>>> > > > a simple example who does this in Fortran?
>>>>>>>> > > > >
>>>>>>>> > > > > Is the PetscMalloc call needed? is 'call
>>>>>>>> > > > PetscMalloc(1,row,ierr)' a valid, compilable call to
>>>>>>>> > > > PetscMalloc? what other reason may there be for this
>>>>>>>> error to
>>>>>>>> > > > happen ?
>>>>>>>> > > > >
>>>>>>>> > > > > Just remembering, that trying to setup the matrix
>>>>>>>> with the
>>>>>>>> > > > MatAIJSetPreallocation() brings up an error to
>>>>>>>> acknowledge the
>>>>>>>> > > > viennacl datatypes and that's why i'm trying to make
>>>>>>>> this change
>>>>>>>> > > > on your recommendation,
>>>>>>>> > > > >
>>>>>>>> > > > > Thanks for your help,
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F.
>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>
>>>>>>>> wrote:
>>>>>>>> > > > >
>>>>>>>> > > > > Send your code in a way we can compile and run
>>>>>>>> it; it must
>>>>>>>> > > > be some simple issue that is hard to communicate in
>>>>>>>> email.
>>>>>>>> > > > >
>>>>>>>> > > > > Barry
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > > > On Aug 27, 2018, at 5:51 PM, Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > > >
>>>>>>>> > > > > > Hello everyone,
>>>>>>>> > > > > >
>>>>>>>> > > > > > I just had time to work on this again, and
>>>>>>>> checked the code
>>>>>>>> > > > for errors on the matrix entries, this is the exact
>>>>>>>> code i was
>>>>>>>> > > > using for creating the matrix without
>>>>>>>> > > > DMSetMatrixPreallocateOnly, using
>>>>>>>> MatMPIAIJSetPreallocation and
>>>>>>>> > > > it worked that way, but trying this way i get the
>>>>>>>> same 'Column
>>>>>>>> > > > too large' error using any number at the column
>>>>>>>> position of
>>>>>>>> > > > MatSetValues,
>>>>>>>> > > > > >
>>>>>>>> > > > > > I have set up my code to print the column
>>>>>>>> argument (n) of
>>>>>>>> > > > MatSetValues and in this case is 7 (lower than 124),
>>>>>>>> it still
>>>>>>>> > > > gives error, even entering a specific number in the
>>>>>>>> MatSetValues
>>>>>>>> > > > column argument position gives the same error.
>>>>>>>> > > > > >
>>>>>>>> > > > > > So next i went back to ex.50 here:
>>>>>>>> > > > http://www.mcs.anl.gov/petsc/
>>>>>>>> petsc-current/src/ts/examples/tutorials/ex50.c.html
>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/petsc-current/src/ts/
>>>>>>>> examples/tutorials/ex50.c.html>
>>>>>>>> > > > and it has a very similar structure except the
>>>>>>>> PetscMalloc1()
>>>>>>>> > > > call, so i tried adding that and got:
>>>>>>>> > > > > >
>>>>>>>> > > > > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114:
>>>>>>>> undefined
>>>>>>>> > > > reference to `petscmalloc1_'
>>>>>>>> > > > > >
>>>>>>>> > > > > > Any ideas on this behaviour?
>>>>>>>> > > > > >
>>>>>>>> > > > > > Thanks so much,
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F.
>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>
>>>>>>>> wrote:
>>>>>>>> > > > > >
>>>>>>>> > > > > > Column too large: col 10980 max 124
>>>>>>>> > > > > >
>>>>>>>> > > > > > You need to check the code that is generating
>>>>>>>> the matrix
>>>>>>>> > > > entries. The matrix has 124 columns but you are
>>>>>>>> attempting to
>>>>>>>> > > > put a value at column 10980
>>>>>>>> > > > > >
>>>>>>>> > > > > > Barry
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > Thanks Matthew and Barry,
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > Now my code looks like:
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > call DMSetMatrixPreallocateOnly(
>>>>>>>> daDummy,PETSC_TRUE,ierr)
>>>>>>>> > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATMPIAIJVIENNACL,ierr)
>>>>>>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>>>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>> > > > > > > call MatSetFromOptions(A,ierr)
>>>>>>>> > > > > > > call MatSetUp(A,ierr)
>>>>>>>> > > > > > > [...]
>>>>>>>> > > > > > > call
>>>>>>>> > > > MatSetValues(A,1,row,sumpos,
>>>>>>>> pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>>>>> > > > > > > [...]
>>>>>>>> > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY,
>>>>>>>> ierr)
>>>>>>>> > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY,
>>>>>>>> ierr)
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > And i get a different error, now is:
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > [0]PETSC ERROR: --------------------- Error
>>>>>>>> Message
>>>>>>>> > > > -----------------------------
>>>>>>>> ---------------------------------
>>>>>>>> > > > > > > [0]PETSC ERROR: Argument out of range
>>>>>>>> > > > > > > [0]PETSC ERROR: Column too large: col 10980
>>>>>>>> max 124
>>>>>>>> > > > > > > [0]PETSC ERROR: See
>>>>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>>>>> for
>>>>>>>> > > > trouble shooting.
>>>>>>>> > > > > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13
>>>>>>>> +0300
>>>>>>>> > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>>>>> named node50
>>>>>>>> > > > by valera Wed Aug 15 19:40:00 2018
>>>>>>>> > > > > > > [0]PETSC ERROR: Configure options
>>>>>>>> PETSC_ARCH=cuda-debug
>>>>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>>>>> --with-shared-libraries=1
>>>>>>>> > > > --with-debugging=1 --with-cuda=1
>>>>>>>> --CUDAFLAGS=-arch=sm_60
>>>>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line
>>>>>>>> 442 in
>>>>>>>> > > > /home/valera/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>> > > > > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in
>>>>>>>> > > > /home/valera/petsc/src/mat/interface/matrix.c
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > Thanks again,
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry
>>>>>>>> F.
>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>
>>>>>>>> wrote:
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > Should be
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATMPIAIJVIENNACL,ierr)
>>>>>>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>>>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > and remove the rest. You need to set the
>>>>>>>> type of Mat
>>>>>>>> > > > you want the DM to return BEFORE you create the
>>>>>>>> matrix.
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > Barry
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Ok thanks for clarifying that, i wasn't sure
>>>>>>>> if there
>>>>>>>> > > > were different types,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Here is a stripped down version of my code,
>>>>>>>> it seems
>>>>>>>> > > > like the preallocation is working now since the matrix
>>>>>>>> > > > population part is working without problem, but here
>>>>>>>> it is for
>>>>>>>> > > > illustration purposes:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > call DMSetMatrixPreallocateOnly(
>>>>>>>> daDummy,PETSC_TRUE,ierr)
>>>>>>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>> > > > > > > > call MatSetFromOptions(A,ierr)
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATMPIAIJVIENNACL,ierr)
>>>>>>>> > > > > > > > call DMSetVecType(daDummy,
>>>>>>>> VECMPIVIENNACL,ierr)
>>>>>>>> > > > > > > > call
>>>>>>>> > > > MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,
>>>>>>>> PETSC_NULL_INTEGER,ierr)
>>>>>>>> > > > > > > > call MatSetUp(A,ierr)
>>>>>>>> > > > > > > > [...]
>>>>>>>> > > > > > > > call
>>>>>>>> > > > MatSetValues(A,1,row,sumpos,
>>>>>>>> pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>>>>> > > > > > > > [...]
>>>>>>>> > > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY,
>>>>>>>> ierr)
>>>>>>>> > > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY,
>>>>>>>> ierr)
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Adding the first line there did the trick,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Now the problem seems to be the program is
>>>>>>>> not
>>>>>>>> > > > recognizing the matrix as ViennaCL type when i try
>>>>>>>> with more
>>>>>>>> > > > than one processor, i get now:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > [0]PETSC ERROR: --------------------- Error
>>>>>>>> Message
>>>>>>>> > > > -----------------------------
>>>>>>>> ---------------------------------
>>>>>>>> > > > > > > > [0]PETSC ERROR: No support for this
>>>>>>>> operation for this
>>>>>>>> > > > object type
>>>>>>>> > > > > > > > [0]PETSC ERROR: Currently only handles
>>>>>>>> ViennaCL matrices
>>>>>>>> > > > > > > > [0]PETSC ERROR: See
>>>>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>>>>> for
>>>>>>>> > > > trouble shooting.
>>>>>>>> > > > > > > > [0]PETSC ERROR: Petsc Development GIT
>>>>>>>> revision:
>>>>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13
>>>>>>>> +0300
>>>>>>>> > > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>>>>> named
>>>>>>>> > > > node50 by valera Wed Aug 15 14:44:22 2018
>>>>>>>> > > > > > > > [0]PETSC ERROR: Configure options
>>>>>>>> PETSC_ARCH=cuda-debug
>>>>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>>>>> --with-shared-libraries=1
>>>>>>>> > > > --with-debugging=1 --with-cuda=1
>>>>>>>> --CUDAFLAGS=-arch=sm_60
>>>>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line
>>>>>>>> 47 in
>>>>>>>> > > > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/
>>>>>>>> saviennacl.cu
>>>>>>>> > > > <http://saviennacl.cu>
>>>>>>>> > > > > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>> > > > /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> > > > > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>>>>>>> > > > /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > When running with:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > mpirun -n 1 ./gcmLEP.GPU
>>>>>>>> > > > tc=TestCases/LockRelease/LE_6x6x6/
>>>>>>>> jid=tiny_cuda_test_n2
>>>>>>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type
>>>>>>>> aijviennacl
>>>>>>>> > > > -pc_type saviennacl -log_view
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Thanks,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew
>>>>>>>> Knepley
>>>>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > > > > > It seems to be resumed on: I do not know how
>>>>>>>> to
>>>>>>>> > > > preallocate a DM Matrix correctly.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > There is only one matrix type, Mat. There
>>>>>>>> are no
>>>>>>>> > > > separate DM matrices. A DM can create a matrix for you
>>>>>>>> > > > > > > > using DMCreateMatrix(), but that is a Mat
>>>>>>>> and it is
>>>>>>>> > > > preallocated correctly. I am not sure what you are
>>>>>>>> doing.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Thanks,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Matt
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > The interesting part is that it only breaks
>>>>>>>> when i need
>>>>>>>> > > > to populate a GPU matrix from MPI, so kudos on that,
>>>>>>>> but it
>>>>>>>> > > > seems i need to do better on my code to get this
>>>>>>>> setup working,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Any help would be appreciated,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Thanks,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew
>>>>>>>> Knepley
>>>>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > > > > > Thanks Matthew,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > I try to do that when calling:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > call
>>>>>>>> > > > MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,
>>>>>>>> PETSC_NULL_INTEGER,ierr)
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > But i am not aware on how to do this for the
>>>>>>>> DM if it
>>>>>>>> > > > needs something more specific/different,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > The error says that your preallocation is
>>>>>>>> wrong for the
>>>>>>>> > > > values you are putting in. The DM does not control
>>>>>>>> either,
>>>>>>>> > > > > > > > so I do not understand your email.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Thanks,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Matt
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Thanks,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew
>>>>>>>> Knepley
>>>>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera
>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>> wrote:
>>>>>>>> > > > > > > > Hello PETSc devs,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > I am running into an error when trying to
>>>>>>>> use the
>>>>>>>> > > > MATMPIAIJVIENNACL Matrix type in MPI calls, the same
>>>>>>>> code runs
>>>>>>>> > > > for MATSEQAIJVIENNACL type in one processor. The
>>>>>>>> error happens
>>>>>>>> > > > when calling MatSetValues for this specific
>>>>>>>> configuration. It
>>>>>>>> > > > does not occur when using MPI DMMatrix types only.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > The DM properly preallocates the matrix. I
>>>>>>>> am assuming
>>>>>>>> > > > you do not here.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Matt
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Any help will be appreciated,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > Thanks,
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > My program call:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > mpirun -n 2 ./gcmLEP.GPU
>>>>>>>> > > > tc=TestCases/LockRelease/LE_6x6x6/
>>>>>>>> jid=tiny_cuda_test_n2
>>>>>>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type
>>>>>>>> aijviennacl
>>>>>>>> > > > -pc_type saviennacl -log_view
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > The error (repeats after each MatSetValues
>>>>>>>> call):
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > [1]PETSC ERROR: --------------------- Error
>>>>>>>> Message
>>>>>>>> > > > -----------------------------
>>>>>>>> ---------------------------------
>>>>>>>> > > > > > > > [1]PETSC ERROR: Argument out of range
>>>>>>>> > > > > > > > [1]PETSC ERROR: Inserting a new nonzero at
>>>>>>>> global
>>>>>>>> > > > row/column (75, 50) into matrix
>>>>>>>> > > > > > > > [1]PETSC ERROR: See
>>>>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>>>>> for
>>>>>>>> > > > trouble shooting.
>>>>>>>> > > > > > > > [1]PETSC ERROR: Petsc Development GIT
>>>>>>>> revision:
>>>>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13
>>>>>>>> +0300
>>>>>>>> > > > > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>>>>> named
>>>>>>>> > > > node50 by valera Wed Aug 15 13:10:44 2018
>>>>>>>> > > > > > > > [1]PETSC ERROR: Configure options
>>>>>>>> PETSC_ARCH=cuda-debug
>>>>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>>>>> --with-shared-libraries=1
>>>>>>>> > > > --with-debugging=1 --with-cuda=1
>>>>>>>> --CUDAFLAGS=-arch=sm_60
>>>>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>> > > > > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ()
>>>>>>>> line 608 in
>>>>>>>> > > > /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>> > > > > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339
>>>>>>>> in
>>>>>>>> > > > /home/valera/petsc/src/mat/interface/matrix.c
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > My Code structure:
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>> > > > > > > > call MatSetFromOptions(A,ierr)
>>>>>>>> > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,
>>>>>>>> numprocs, ierr)
>>>>>>>> > > > > > > > if (numprocs > 1) then ! set matrix type
>>>>>>>> parallel
>>>>>>>> > > > > > > > ! Get local size
>>>>>>>> > > > > > > > call DMDACreateNaturalVector(
>>>>>>>> daDummy,Tmpnat,ierr)
>>>>>>>> > > > > > > > call VecGetLocalSize(Tmpnat,
>>>>>>>> locsize,ierr)
>>>>>>>> > > > > > > > call VecDestroy(Tmpnat,ierr)
>>>>>>>> > > > > > > > ! Set matrix
>>>>>>>> > > > > > > > #ifdef GPU
>>>>>>>> > > > > > > > call MatSetType(A,MATAIJVIENNACL,ierr)
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATMPIAIJVIENNACL,ierr)
>>>>>>>> > > > > > > > call DMSetVecType(daDummy,
>>>>>>>> VECMPIVIENNACL,ierr)
>>>>>>>> > > > > > > > print*,'SETTING GPU TYPES'
>>>>>>>> > > > > > > > #else
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATMPIAIJ,ierr)
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,VECMPI,ierr)
>>>>>>>> > > > > > > > call MatSetType(A,MATMPIAIJ,ierr)!
>>>>>>>> > > > > > > > #endif
>>>>>>>> > > > > > > > call
>>>>>>>> > > > MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,
>>>>>>>> PETSC_NULL_INTEGER,ierr)
>>>>>>>> > > > > > > > else ! set matrix type
>>>>>>>> sequential
>>>>>>>> > > > > > > > #ifdef GPU
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATSEQAIJVIENNACL,ierr)
>>>>>>>> > > > > > > > call DMSetVecType(daDummy,
>>>>>>>> VECSEQVIENNACL,ierr)
>>>>>>>> > > > > > > > call MatSetType(A,
>>>>>>>> MATSEQAIJVIENNACL,ierr)
>>>>>>>> > > > > > > > print*,'SETTING GPU TYPES'
>>>>>>>> > > > > > > > #else
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,
>>>>>>>> MATSEQAIJ,ierr)
>>>>>>>> > > > > > > > call DMSetMatType(daDummy,VECSEQ,ierr)
>>>>>>>> > > > > > > > call MatSetType(A,MATSEQAIJ,ierr)
>>>>>>>> > > > > > > > #endif
>>>>>>>> > > > > > > > call MatSetUp(A,ierr)
>>>>>>>> > > > > > > > call
>>>>>>>> > > > getCenterInfo(daGrid,xstart,
>>>>>>>> ystart,zstart,xend,yend,zend)
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > do k=zstart,zend-1
>>>>>>>> > > > > > > > do j=ystart,yend-1
>>>>>>>> > > > > > > > do i=xstart,xend-1
>>>>>>>> > > > > > > > [..]
>>>>>>>> > > > > > > > call
>>>>>>>> > > > MatSetValues(A,1,row,sumpos,
>>>>>>>> pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>>>>> > > > > > > > [..]
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > --
>>>>>>>> > > > > > > > What most experimenters take for granted
>>>>>>>> before they
>>>>>>>> > > > begin their experiments is infinitely more
>>>>>>>> interesting than any
>>>>>>>> > > > results to which their experiments lead.
>>>>>>>> > > > > > > > -- Norbert Wiener
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > --
>>>>>>>> > > > > > > > What most experimenters take for granted
>>>>>>>> before they
>>>>>>>> > > > begin their experiments is infinitely more
>>>>>>>> interesting than any
>>>>>>>> > > > results to which their experiments lead.
>>>>>>>> > > > > > > > -- Norbert Wiener
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > --
>>>>>>>> > > > > > > > What most experimenters take for granted
>>>>>>>> before they
>>>>>>>> > > > begin their experiments is infinitely more
>>>>>>>> interesting than any
>>>>>>>> > > > results to which their experiments lead.
>>>>>>>> > > > > > > > -- Norbert Wiener
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > <backt-2.log><backt.log>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180830/9b44bdb4/attachment-0001.html>
More information about the petsc-users
mailing list