[petsc-users] MatSetValues error with ViennaCL types
Matthew Knepley
knepley at gmail.com
Thu Aug 30 15:45:20 CDT 2018
On Thu, Aug 30, 2018 at 4:41 PM Manuel Valera <mvalera-w at sdsu.edu> wrote:
> Matthew, and anyone who wants to chip in on this,
>
> Just to close this thread then, can you suggest a roadmap to get this
> MPI+GPU hibrydization done? you mentioned the ASM PCs, i may look into
> that, but is not clear to me now if this is doable now in PETSc or if i
> have to wait for more functionality to be added,
>
This is doable now. First select ASM (-pc_type asm). The serial submatrices
should be of the correct type. Then select
the subsolver to be your favorite GPU thing (-sub_pc_type saviennacl).
> I have two available machines, one with 20 cores and a P100 GPU, and other
> with 8 V100 GPUs and MPI for communication, how would you guys do this?
>
The above is appropriate for this setup.
Thanks,
Matt
> My model is fully distributed with DMDAs except for output, and the linear
> solver takes around 56% of the computation time,
>
> Thanks,
>
>
>
>
>
> On Thu, Aug 30, 2018 at 8:25 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Thu, Aug 30, 2018 at 10:52 AM Manuel Valera <mvalera-w at sdsu.edu>
>> wrote:
>>
>>> I thought newer NVIDIA cards had integrated circuits, not using PCI
>>> anymore, still the same case?
>>>
>>> How would the GPU clusters work, then? Let's say i have a cluster of
>>> GPUs communicated by an MPI node, would PETSc work on this configuration at
>>> some point? This would be my next step if I could make the MPI matrices
>>> work with GPU, we have such a cluster on campus.
>>>
>>
>> That works now, in that you just use more loosely coupled
>> preconditioners, like ASM. That was my point.
>> Its unlikely that tightly coupled PCs would be useful in this regime.
>>
>>
>>> Are you familiar with the Magma library? It was recommended recently to
>>> tackle this problem,
>>>
>>
>> It does not tackle this problem.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> I am most interested in your insight,
>>>
>>> Thanks,
>>>
>>> On Thu, Aug 30, 2018, 3:10 AM Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>>> On Wed, Aug 29, 2018 at 6:21 PM Manuel Valera <mvalera-w at sdsu.edu>
>>>> wrote:
>>>>
>>>>> I understand that Matthew, thanks,
>>>>>
>>>>> I am trying to see if i get any speedup in this configuration,
>>>>> otherwise i will be waiting for the MPIAIJ support on PETSc,
>>>>>
>>>>> Is there any other gpu preconditioner that works on parallel matrices
>>>>> directly?
>>>>>
>>>>
>>>> Not that I know of . Such a hypothetical preconditioner would need to
>>>> pass information back and forth, across the PCI bus, and
>>>> then through MPI (even if you imagine direct GPU-GPU communication, it
>>>> still hits the PCI bus). This channel has small bandwidth,
>>>> but even more limiting is the high latency. Its unclear what kind of
>>>> algorithm would do well in this situation.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thanks again,
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 29, 2018 at 3:05 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Aug 29, 2018 at 5:49 PM Manuel Valera <mvalera-w at sdsu.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Update:
>>>>>>>
>>>>>>> I made it work like you suggested Barry, i had to comment the code
>>>>>>> line to set up the pc_type saviennacl, that way i am getting as ksp_view:
>>>>>>>
>>>>>>> KSP Object: 2 MPI processes
>>>>>>> type: cg
>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>> tolerances: relative=1e-07, absolute=1e-50, divergence=10000.
>>>>>>> left preconditioning
>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>> PC Object: 2 MPI processes
>>>>>>> type: bjacobi
>>>>>>> number of blocks = 2
>>>>>>> Local solve is same for all blocks, in the following KSP and PC
>>>>>>> objects:
>>>>>>> KSP Object: (sub_) 1 MPI processes
>>>>>>> type: preonly
>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
>>>>>>> left preconditioning
>>>>>>> using NONE norm type for convergence test
>>>>>>> PC Object: (sub_) 1 MPI processes
>>>>>>> type: ilu
>>>>>>> out-of-place factorization
>>>>>>> 0 levels of fill
>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>> matrix ordering: natural
>>>>>>> factor fill ratio given 1., needed 1.
>>>>>>> Factored matrix follows:
>>>>>>> Mat Object: 1 MPI processes
>>>>>>> type: seqaij
>>>>>>> rows=75, cols=75
>>>>>>> package used to perform factorization: petsc
>>>>>>> total: nonzeros=927, allocated nonzeros=927
>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>> not using I-node routines
>>>>>>> linear system matrix = precond matrix:
>>>>>>> Mat Object: 1 MPI processes
>>>>>>> type: seqaijviennacl
>>>>>>> rows=75, cols=75
>>>>>>> total: nonzeros=927, allocated nonzeros=1335
>>>>>>> total number of mallocs used during MatSetValues calls =84
>>>>>>> not using I-node routines
>>>>>>> linear system matrix = precond matrix:
>>>>>>> Mat Object: 2 MPI processes
>>>>>>> type: mpiaijviennacl
>>>>>>> rows=125, cols=125
>>>>>>> total: nonzeros=1685, allocated nonzeros=2885
>>>>>>> total number of mallocs used during MatSetValues calls =184
>>>>>>> has attached near null space
>>>>>>> Press_Max: 1.9995807630564922E-004
>>>>>>>
>>>>>>>
>>>>>>> This is what we want, right?
>>>>>>>
>>>>>>> Still if i try using pc_type saviennacl directly it breaks,
>>>>>>>
>>>>>>
>>>>>> You are missing the point of the error message.
>>>>>>
>>>>>> We do not support SAVIENNACL on parallel matrices (MPIAIJ). We do
>>>>>> support it
>>>>>> on serial matrices (SEQAIJ). When you are setting in directly, you
>>>>>> are applying it
>>>>>> to a parallel matrix. If you use Block-Jacobi, you are applying it to
>>>>>> each submatrix,
>>>>>> and those submatrices are all serial.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> Thanks for your help i'll try my speedup like this,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 29, 2018 at 2:21 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ok, executing with:
>>>>>>>>
>>>>>>>> mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/
>>>>>>>> jid=tiny_cuda_test_n1 -pc_type bjacobi -pc_sub_type saviennacl -ksp_view
>>>>>>>>
>>>>>>>>
>>>>>>>> I get:
>>>>>>>>
>>>>>>>>
>>>>>>>> SETTING GPU TYPES
>>>>>>>> Matrix type: mpiaijviennacl
>>>>>>>>
>>>>>>>> Of sizes: 125 x 125
>>>>>>>> Matrix type: mpiaijviennacl
>>>>>>>>
>>>>>>>> Of sizes: 125 x 125
>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>>>>> [0]PETSC ERROR: See
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>> shooting.
>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.3-1264-g6fb0cd4 GIT Date: 2018-08-29 12:42:36 -0600
>>>>>>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera
>>>>>>>> Wed Aug 29 14:20:42 2018
>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl --download-cusp
>>>>>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>> [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> [1]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>> Violation, probably memory access out of range
>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> [1]PETSC ERROR: or see
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>>> Mac OS X to find memory corruption errors
>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below
>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>>> available,
>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the
>>>>>>>> function
>>>>>>>> [1]PETSC ERROR: is given.
>>>>>>>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>>>>> [1]PETSC ERROR: [1] PetscError line 352
>>>>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>>>>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>> [1]PETSC ERROR: [1] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> [1]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [1]PETSC ERROR: Signal received
>>>>>>>> [1]PETSC ERROR: See
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>> shooting.
>>>>>>>> [1]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.3-1264-g6fb0cd4 GIT Date: 2018-08-29 12:42:36 -0600
>>>>>>>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera
>>>>>>>> Wed Aug 29 14:20:42 2018
>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl --download-cusp
>>>>>>>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>>>>>> with errorcode 59.
>>>>>>>>
>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>>>>> You may or may not see output from other processes, depending on
>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>>> the batch system) has told this process to end
>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>>> Mac OS X to find memory corruption errors
>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below
>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>>> available,
>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the
>>>>>>>> function
>>>>>>>> [0]PETSC ERROR: is given.
>>>>>>>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>>>>>>> /home/valera/petsc/src/sys/objects/tagm.c
>>>>>>>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 41
>>>>>>>> /home/valera/petsc/src/sys/objects/inherit.c
>>>>>>>> [0]PETSC ERROR: [0] ISCreate line 35
>>>>>>>> /home/valera/petsc/src/vec/is/is/interface/isreg.c
>>>>>>>> [0]PETSC ERROR: [0] ISCreateGeneral line 658
>>>>>>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>>>>>>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>> [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>> [0]PETSC ERROR: See
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>> shooting.
>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>> v3.9.3-1264-g6fb0cd4 GIT Date: 2018-08-29 12:42:36 -0600
>>>>>>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera
>>>>>>>> Wed Aug 29 14:20:42 2018
>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2
>>>>>>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>>>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>>>>>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl --download-cusp
>>>>>>>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file
>>>>>>>> [node50:77836] 1 more process has sent help message
>>>>>>>> help-mpi-api.txt / mpi-abort
>>>>>>>> [node50:77836] Set MCA parameter "orte_base_help_aggregate" to 0 to
>>>>>>>> see all help / error messages
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 29, 2018 at 2:02 PM, Smith, Barry F. <
>>>>>>>> bsmith at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Please send complete error message
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > On Aug 29, 2018, at 3:40 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > Yeah, no sorry, i get the same error with -pc_type bjacobi
>>>>>>>>> -sub_pc_type SAVIENNACL : "Currently only handles ViennaCL matrices"
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Thanks and let me know of any progress on this issue,
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Wed, Aug 29, 2018 at 1:37 PM, Manuel Valera <
>>>>>>>>> mvalera-w at sdsu.edu> wrote:
>>>>>>>>> > Awesome, thanks!
>>>>>>>>> >
>>>>>>>>> > On Wed, Aug 29, 2018 at 1:29 PM, Smith, Barry F. <
>>>>>>>>> bsmith at mcs.anl.gov> wrote:
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > > On Aug 29, 2018, at 3:26 PM, Manuel Valera <mvalera-w at sdsu.edu>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > You may need to use just plain PCBJACOBI or PCASM for
>>>>>>>>> parallelism and then SAVIENNACL sequentially on each block.
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > Can you elaborate a little in how to do this? do you mean i
>>>>>>>>> can only use viennacl in serial at the moment or this is a suggestion of a
>>>>>>>>> workaround on how to make it work?
>>>>>>>>> >
>>>>>>>>> > -pc_type bjacobi -sub_pc_type SAVIENNACL
>>>>>>>>> >
>>>>>>>>> > This will run in parallel and utilize the GPU for each block
>>>>>>>>> of the preconditioner.
>>>>>>>>> >
>>>>>>>>> > Barry
>>>>>>>>> >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > Thanks,
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > > On Aug 29, 2018, at 1:50 PM, Manuel Valera <
>>>>>>>>> mvalera-w at sdsu.edu> wrote:
>>>>>>>>> > > >
>>>>>>>>> > > > Hi everyone,
>>>>>>>>> > > >
>>>>>>>>> > > > Thanks for your responses, i understand communicating on
>>>>>>>>> this way to this level of technicality can be hard, i still think we can
>>>>>>>>> work ways to solve this problem though,
>>>>>>>>> > > >
>>>>>>>>> > > > I can say the following at this point:
>>>>>>>>> > > >
>>>>>>>>> > > > • Program works without issues in any other non-gpu
>>>>>>>>> preconditioner, just by calling -pc_type ### .
>>>>>>>>> > > > • Program works correctly with ViennaCL
>>>>>>>>> preconditioners with mpirun -n 1, that is on one processor.
>>>>>>>>> > > > • Program breaks with every ViennaCL preconditioner
>>>>>>>>> when attempting any more than 1 processors.
>>>>>>>>> > > > • I haven't tried other GPU preconditioners because
>>>>>>>>> ViennaCL gave the best performance in 1 processor before.
>>>>>>>>> > > > • I have made sure the matrix type is mpiaijviennacl
>>>>>>>>> as it is printed right before the error log.
>>>>>>>>> > > > • I am attaching what i think are the backtrace logs
>>>>>>>>> of the run with two cores, as i have limited experience with this kind of
>>>>>>>>> debugging, and i cannot make sense of what they say, please let me know if
>>>>>>>>> you need me to do something else.
>>>>>>>>> > > >
>>>>>>>>> > > > Current options i can think of are giving you permission to
>>>>>>>>> clone my model repo, or for you to guide me trough the process remotely,
>>>>>>>>> > > >
>>>>>>>>> > > > Thanks,
>>>>>>>>> > > >
>>>>>>>>> > > > .-.-.-.-
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > The error i get is still:
>>>>>>>>> > > >
>>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>>>
>>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>>>
>>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: No support for this operation for this
>>>>>>>>> object type
>>>>>>>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>>>>>> > > > [0]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>>> valera Wed Aug 29 11:43:25 2018
>>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>> > > > [1]PETSC ERROR:
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>> -on_error_attach_debugger
>>>>>>>>> > > > [1]PETSC ERROR: or see
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>> > > > [1]PETSC ERROR: likely location of problem given in stack
>>>>>>>>> below
>>>>>>>>> > > > [1]PETSC ERROR: --------------------- Stack Frames
>>>>>>>>> ------------------------------------
>>>>>>>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack
>>>>>>>>> are not available,
>>>>>>>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start
>>>>>>>>> of the function
>>>>>>>>> > > > [1]PETSC ERROR: is given.
>>>>>>>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>>>>>> > > > [1]PETSC ERROR: [1] PetscError line 352
>>>>>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [1]PETSC ERROR: [1] KSPSetUp line 294
>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>> > > > [1]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > [1]PETSC ERROR: Signal received
>>>>>>>>> > > > [1]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>>> valera Wed Aug 29 11:43:25 2018
>>>>>>>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in
>>>>>>>>> unknown file
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> > > > MPI_ABORT was invoked on rank 1 in communicator
>>>>>>>>> MPI_COMM_WORLD
>>>>>>>>> > > > with errorcode 59.
>>>>>>>>> > > >
>>>>>>>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>> processes.
>>>>>>>>> > > > You may or may not see output from other processes,
>>>>>>>>> depending on
>>>>>>>>> > > > exactly when Open MPI kills them.
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR:
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>> -on_error_attach_debugger
>>>>>>>>> > > > [0]PETSC ERROR: or see
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>> > > > [0]PETSC ERROR: likely location of problem given in stack
>>>>>>>>> below
>>>>>>>>> > > > [0]PETSC ERROR: --------------------- Stack Frames
>>>>>>>>> ------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack
>>>>>>>>> are not available,
>>>>>>>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start
>>>>>>>>> of the function
>>>>>>>>> > > > [0]PETSC ERROR: is given.
>>>>>>>>> > > > [0]PETSC ERROR: [0] MatSetErrorIfFailure line 116
>>>>>>>>> /home/valera/petsc/src/mat/utils/gcreate.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] KSPSetUp line 294
>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: Signal received
>>>>>>>>> > > > [0]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>>> valera Wed Aug 29 11:43:25 2018
>>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > [0]PETSC ERROR: #4 User provided function() line 0 in
>>>>>>>>> unknown file
>>>>>>>>> > > > [node50:32783] 1 more process has sent help message
>>>>>>>>> help-mpi-api.txt / mpi-abort
>>>>>>>>> > > > [node50:32783] Set MCA parameter "orte_base_help_aggregate"
>>>>>>>>> to 0 to see all help / error messages
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <
>>>>>>>>> rupp at iue.tuwien.ac.at> wrote:
>>>>>>>>> > > > Hi Manuel,
>>>>>>>>> > > >
>>>>>>>>> > > > as Barry said, it is hard for us to provide any help without
>>>>>>>>> having a more complete picture of what is going on.
>>>>>>>>> > > >
>>>>>>>>> > > > The error you report seems to come from the AMG
>>>>>>>>> preconditioner in ViennaCL. This can have many origins. Do other
>>>>>>>>> preconditioners run without error? Can you run in a debugger and provide a
>>>>>>>>> complete backtrace?
>>>>>>>>> > > >
>>>>>>>>> > > > Thanks and best regards,
>>>>>>>>> > > > Karli
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > On 08/29/2018 01:33 AM, Manuel Valera wrote:
>>>>>>>>> > > > Talked too fast,
>>>>>>>>> > > >
>>>>>>>>> > > > After fixing that problem, i tried more than one mpi
>>>>>>>>> processor and got the following:
>>>>>>>>> > > >
>>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>>> > > > Matrix type: mpiaijviennacl
>>>>>>>>> > > > Of sizes: 125 x 125
>>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: No support for this operation for this
>>>>>>>>> object type
>>>>>>>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>>>>>>>> > > > [0]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>>> valera Tue Aug 28 16:30:02 2018
>>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>>>>>>>>> http://saviennacl.cu>
>>>>>>>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [1]PETSC ERROR:
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>> -on_error_attach_debugger
>>>>>>>>> > > > [1]PETSC ERROR: or see
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>> > > > [1]PETSC ERROR: likely location of problem given in stack
>>>>>>>>> below
>>>>>>>>> > > > [1]PETSC ERROR: --------------------- Stack Frames
>>>>>>>>> ------------------------------------
>>>>>>>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack
>>>>>>>>> are not available,
>>>>>>>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start
>>>>>>>>> of the function
>>>>>>>>> > > > [1]PETSC ERROR: is given.
>>>>>>>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>>>>>>>> /home/valera/petsc/src/sys/error/errtrace.c
>>>>>>>>> > > > [1]PETSC ERROR: [1] PetscError line 352
>>>>>>>>> /home/valera/petsc/src/sys/error/err.c
>>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu<
>>>>>>>>> http://saviennacl.cu>
>>>>>>>>> > > >
>>>>>>>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [1]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > [1]PETSC ERROR: Signal received
>>>>>>>>> > > > [1]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>>> valera Tue Aug 28 16:30:02 2018
>>>>>>>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in
>>>>>>>>> unknown file
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> > > > MPI_ABORT was invoked on rank 1 in communicator
>>>>>>>>> MPI_COMM_WORLD
>>>>>>>>> > > > with errorcode 59.
>>>>>>>>> > > >
>>>>>>>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>> processes.
>>>>>>>>> > > > You may or may not see output from other processes,
>>>>>>>>> depending on
>>>>>>>>> > > > exactly when Open MPI kills them.
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR:
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>> -on_error_attach_debugger
>>>>>>>>> > > > [0]PETSC ERROR: or see
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>> > > > [0]PETSC ERROR: likely location of problem given in stack
>>>>>>>>> below
>>>>>>>>> > > > [0]PETSC ERROR: --------------------- Stack Frames
>>>>>>>>> ------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack
>>>>>>>>> are not available,
>>>>>>>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start
>>>>>>>>> of the function
>>>>>>>>> > > > [0]PETSC ERROR: is given.
>>>>>>>>> > > > [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>>>>>>>> /home/valera/petsc/src/sys/objects/tagm.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34
>>>>>>>>> /home/valera/petsc/src/sys/objects/inherit.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] ISCreate line 35
>>>>>>>>> /home/valera/petsc/src/vec/is/is/interface/isreg.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] ISCreateGeneral line 668
>>>>>>>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu<
>>>>>>>>> http://saviennacl.cu>
>>>>>>>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894
>>>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > [0]PETSC ERROR: Signal received
>>>>>>>>> > > > [0]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>>>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>>>>>>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by
>>>>>>>>> valera Tue Aug 28 16:30:02 2018
>>>>>>>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>>>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > [0]PETSC ERROR: #3 User provided function() line 0 in
>>>>>>>>> unknown file
>>>>>>>>> > > > [node50:30582] 1 more process has sent help message
>>>>>>>>> help-mpi-api.txt / mpi-abort
>>>>>>>>> > > > [node50:30582] Set MCA parameter "orte_base_help_aggregate"
>>>>>>>>> to 0 to see all help / error messages
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > It is currently running in 1 mpi processor + GPU but i would
>>>>>>>>> like to call at least 16 mpi processors + GPU to do the rest of the data
>>>>>>>>> management who is not part of the main laplacian on the mpi and the
>>>>>>>>> laplacian solution on the GPU, is this currently possible?
>>>>>>>>> > > >
>>>>>>>>> > > > Thanks for your help,
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera <
>>>>>>>>> mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>>>>>>>> > > >
>>>>>>>>> > > > Ok, i found the culprit and we can close this thread,
>>>>>>>>> > > >
>>>>>>>>> > > > The problem was a missing variable for setting the
>>>>>>>>> maximum columns,
>>>>>>>>> > > > which i deleted at some point without realizing. The
>>>>>>>>> error message
>>>>>>>>> > > > was too ambiguous to catch this so i had to compare with
>>>>>>>>> a previous
>>>>>>>>> > > > working version of the arguments of MatSetValues, it was
>>>>>>>>> evident then.
>>>>>>>>> > > >
>>>>>>>>> > > > Good news is that i can now set the values with the
>>>>>>>>> viennacl types too,
>>>>>>>>> > > >
>>>>>>>>> > > > Thanks for your kind help,
>>>>>>>>> > > >
>>>>>>>>> > > > Manuel
>>>>>>>>> > > >
>>>>>>>>> > > > On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F.
>>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > 1) PetscMalloc() is never valid or needed in
>>>>>>>>> Fortran
>>>>>>>>> > > >
>>>>>>>>> > > > 2) there is no reason to use
>>>>>>>>> DMSetMatrixPreallocateOnly()
>>>>>>>>> > > > just use DMCreateMatrix() assuming that using a DM
>>>>>>>>> (DMDA,
>>>>>>>>> > > > DMPLEX, etc) is suitable for your problem.
>>>>>>>>> > > >
>>>>>>>>> > > > At this end we are totally guessing at what you
>>>>>>>>> are doing
>>>>>>>>> > > > and so have little help we can provide. A simple,
>>>>>>>>> nonworking
>>>>>>>>> > > > code that tries to do what you would need would help
>>>>>>>>> us a great
>>>>>>>>> > > > deal in understanding that you are trying to do.
>>>>>>>>> > > >
>>>>>>>>> > > > Barry
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > > On Aug 28, 2018, at 1:18 PM, Manuel Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > >
>>>>>>>>> > > > > Matthew, PetscMalloc gives the same error,
>>>>>>>>> > > > >
>>>>>>>>> > > > > Barry, it would be very hard for me to get the
>>>>>>>>> code to a
>>>>>>>>> > > > minimum working example, i guess all i need to
>>>>>>>>> understand is how
>>>>>>>>> > > > to setup a DM matrix with
>>>>>>>>> DMSetMatrixPreallocateOnly() instead
>>>>>>>>> > > > of MatMPIAIJSetPreallocation() as we were doing
>>>>>>>>> before, is there
>>>>>>>>> > > > a simple example who does this in Fortran?
>>>>>>>>> > > > >
>>>>>>>>> > > > > Is the PetscMalloc call needed? is 'call
>>>>>>>>> > > > PetscMalloc(1,row,ierr)' a valid, compilable call to
>>>>>>>>> > > > PetscMalloc? what other reason may there be for this
>>>>>>>>> error to
>>>>>>>>> > > > happen ?
>>>>>>>>> > > > >
>>>>>>>>> > > > > Just remembering, that trying to setup the matrix
>>>>>>>>> with the
>>>>>>>>> > > > MatAIJSetPreallocation() brings up an error to
>>>>>>>>> acknowledge the
>>>>>>>>> > > > viennacl datatypes and that's why i'm trying to make
>>>>>>>>> this change
>>>>>>>>> > > > on your recommendation,
>>>>>>>>> > > > >
>>>>>>>>> > > > > Thanks for your help,
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F.
>>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>
>>>>>>>>> wrote:
>>>>>>>>> > > > >
>>>>>>>>> > > > > Send your code in a way we can compile and run
>>>>>>>>> it; it must
>>>>>>>>> > > > be some simple issue that is hard to communicate in
>>>>>>>>> email.
>>>>>>>>> > > > >
>>>>>>>>> > > > > Barry
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > > > On Aug 27, 2018, at 5:51 PM, Manuel Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Hello everyone,
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > I just had time to work on this again, and
>>>>>>>>> checked the code
>>>>>>>>> > > > for errors on the matrix entries, this is the exact
>>>>>>>>> code i was
>>>>>>>>> > > > using for creating the matrix without
>>>>>>>>> > > > DMSetMatrixPreallocateOnly, using
>>>>>>>>> MatMPIAIJSetPreallocation and
>>>>>>>>> > > > it worked that way, but trying this way i get the
>>>>>>>>> same 'Column
>>>>>>>>> > > > too large' error using any number at the column
>>>>>>>>> position of
>>>>>>>>> > > > MatSetValues,
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > I have set up my code to print the column
>>>>>>>>> argument (n) of
>>>>>>>>> > > > MatSetValues and in this case is 7 (lower than 124),
>>>>>>>>> it still
>>>>>>>>> > > > gives error, even entering a specific number in the
>>>>>>>>> MatSetValues
>>>>>>>>> > > > column argument position gives the same error.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > So next i went back to ex.50 here:
>>>>>>>>> > > >
>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex50.c.html
>>>>>>>>> > > > <
>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex50.c.html
>>>>>>>>> >
>>>>>>>>> > > > and it has a very similar structure except the
>>>>>>>>> PetscMalloc1()
>>>>>>>>> > > > call, so i tried adding that and got:
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114: undefined
>>>>>>>>> > > > reference to `petscmalloc1_'
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Any ideas on this behaviour?
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Thanks so much,
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry
>>>>>>>>> F.
>>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Column too large: col 10980 max 124
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > You need to check the code that is
>>>>>>>>> generating the matrix
>>>>>>>>> > > > entries. The matrix has 124 columns but you are
>>>>>>>>> attempting to
>>>>>>>>> > > > put a value at column 10980
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Barry
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > Thanks Matthew and Barry,
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > Now my code looks like:
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > call
>>>>>>>>> DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>>>>>>>>> > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>>>>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>>> > > > > > > call MatSetFromOptions(A,ierr)
>>>>>>>>> > > > > > > call MatSetUp(A,ierr)
>>>>>>>>> > > > > > > [...]
>>>>>>>>> > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>>>>>> > > > > > > [...]
>>>>>>>>> > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY,
>>>>>>>>> ierr)
>>>>>>>>> > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY,
>>>>>>>>> ierr)
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > And i get a different error, now is:
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > [0]PETSC ERROR: --------------------- Error
>>>>>>>>> Message
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > > > > [0]PETSC ERROR: Argument out of range
>>>>>>>>> > > > > > > [0]PETSC ERROR: Column too large: col 10980
>>>>>>>>> max 124
>>>>>>>>> > > > > > > [0]PETSC ERROR: See
>>>>>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>>>>>> for
>>>>>>>>> > > > trouble shooting.
>>>>>>>>> > > > > > > [0]PETSC ERROR: Petsc Development GIT
>>>>>>>>> revision:
>>>>>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13
>>>>>>>>> +0300
>>>>>>>>> > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug
>>>>>>>>> named node50
>>>>>>>>> > > > by valera Wed Aug 15 19:40:00 2018
>>>>>>>>> > > > > > > [0]PETSC ERROR: Configure options
>>>>>>>>> PETSC_ARCH=cuda-debug
>>>>>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>>>>>> --with-shared-libraries=1
>>>>>>>>> > > > --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60
>>>>>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line
>>>>>>>>> 442 in
>>>>>>>>> > > > /home/valera/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>> > > > > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in
>>>>>>>>> > > > /home/valera/petsc/src/mat/interface/matrix.c
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > Thanks again,
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry
>>>>>>>>> F.
>>>>>>>>> > > > <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > Should be
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>>>>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > and remove the rest. You need to set the
>>>>>>>>> type of Mat
>>>>>>>>> > > > you want the DM to return BEFORE you create the
>>>>>>>>> matrix.
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > Barry
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Ok thanks for clarifying that, i wasn't
>>>>>>>>> sure if there
>>>>>>>>> > > > were different types,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Here is a stripped down version of my code,
>>>>>>>>> it seems
>>>>>>>>> > > > like the preallocation is working now since the
>>>>>>>>> matrix
>>>>>>>>> > > > population part is working without problem, but here
>>>>>>>>> it is for
>>>>>>>>> > > > illustration purposes:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>>>>>>>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>>> > > > > > > > call MatSetFromOptions(A,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr)
>>>>>>>>> > > > > > > > call MatSetUp(A,ierr)
>>>>>>>>> > > > > > > > [...]
>>>>>>>>> > > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>>>>>> > > > > > > > [...]
>>>>>>>>> > > > > > > > call MatAssemblyBegin(A,
>>>>>>>>> MAT_FINAL_ASSEMBLY, ierr)
>>>>>>>>> > > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY,
>>>>>>>>> ierr)
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Adding the first line there did the trick,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Now the problem seems to be the program is
>>>>>>>>> not
>>>>>>>>> > > > recognizing the matrix as ViennaCL type when i try
>>>>>>>>> with more
>>>>>>>>> > > > than one processor, i get now:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > [0]PETSC ERROR: --------------------- Error
>>>>>>>>> Message
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > > > > > [0]PETSC ERROR: No support for this
>>>>>>>>> operation for this
>>>>>>>>> > > > object type
>>>>>>>>> > > > > > > > [0]PETSC ERROR: Currently only handles
>>>>>>>>> ViennaCL matrices
>>>>>>>>> > > > > > > > [0]PETSC ERROR: See
>>>>>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>>>>>> for
>>>>>>>>> > > > trouble shooting.
>>>>>>>>> > > > > > > > [0]PETSC ERROR: Petsc Development GIT
>>>>>>>>> revision:
>>>>>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13
>>>>>>>>> +0300
>>>>>>>>> > > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a
>>>>>>>>> cuda-debug named
>>>>>>>>> > > > node50 by valera Wed Aug 15 14:44:22 2018
>>>>>>>>> > > > > > > > [0]PETSC ERROR: Configure options
>>>>>>>>> PETSC_ARCH=cuda-debug
>>>>>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>>>>>> --with-shared-libraries=1
>>>>>>>>> > > > --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60
>>>>>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL()
>>>>>>>>> line 47 in
>>>>>>>>> > > > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/
>>>>>>>>> saviennacl.cu
>>>>>>>>> > > > <http://saviennacl.cu>
>>>>>>>>> > > > > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>>>>>>>> > > > /home/valera/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> > > > > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>>>>>>>> > > > /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > When running with:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > mpirun -n 1 ./gcmLEP.GPU
>>>>>>>>> > > > tc=TestCases/LockRelease/LE_6x6x6/
>>>>>>>>> jid=tiny_cuda_test_n2
>>>>>>>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type
>>>>>>>>> aijviennacl
>>>>>>>>> > > > -pc_type saviennacl -log_view
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Thanks,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew
>>>>>>>>> Knepley
>>>>>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel
>>>>>>>>> Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > > It seems to be resumed on: I do not know
>>>>>>>>> how to
>>>>>>>>> > > > preallocate a DM Matrix correctly.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > There is only one matrix type, Mat. There
>>>>>>>>> are no
>>>>>>>>> > > > separate DM matrices. A DM can create a matrix for
>>>>>>>>> you
>>>>>>>>> > > > > > > > using DMCreateMatrix(), but that is a Mat
>>>>>>>>> and it is
>>>>>>>>> > > > preallocated correctly. I am not sure what you are
>>>>>>>>> doing.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Thanks,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Matt
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > The interesting part is that it only breaks
>>>>>>>>> when i need
>>>>>>>>> > > > to populate a GPU matrix from MPI, so kudos on that,
>>>>>>>>> but it
>>>>>>>>> > > > seems i need to do better on my code to get this
>>>>>>>>> setup working,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Any help would be appreciated,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Thanks,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew
>>>>>>>>> Knepley
>>>>>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel
>>>>>>>>> Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > > Thanks Matthew,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > I try to do that when calling:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr)
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > But i am not aware on how to do this for
>>>>>>>>> the DM if it
>>>>>>>>> > > > needs something more specific/different,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > The error says that your preallocation is
>>>>>>>>> wrong for the
>>>>>>>>> > > > values you are putting in. The DM does not control
>>>>>>>>> either,
>>>>>>>>> > > > > > > > so I do not understand your email.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Thanks,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Matt
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Thanks,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew
>>>>>>>>> Knepley
>>>>>>>>> > > > <knepley at gmail.com <mailto:knepley at gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel
>>>>>>>>> Valera
>>>>>>>>> > > > <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>>
>>>>>>>>> wrote:
>>>>>>>>> > > > > > > > Hello PETSc devs,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > I am running into an error when trying to
>>>>>>>>> use the
>>>>>>>>> > > > MATMPIAIJVIENNACL Matrix type in MPI calls, the same
>>>>>>>>> code runs
>>>>>>>>> > > > for MATSEQAIJVIENNACL type in one processor. The
>>>>>>>>> error happens
>>>>>>>>> > > > when calling MatSetValues for this specific
>>>>>>>>> configuration. It
>>>>>>>>> > > > does not occur when using MPI DMMatrix types only.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > The DM properly preallocates the matrix. I
>>>>>>>>> am assuming
>>>>>>>>> > > > you do not here.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Matt
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Any help will be appreciated,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > Thanks,
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > My program call:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > mpirun -n 2 ./gcmLEP.GPU
>>>>>>>>> > > > tc=TestCases/LockRelease/LE_6x6x6/
>>>>>>>>> jid=tiny_cuda_test_n2
>>>>>>>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type
>>>>>>>>> aijviennacl
>>>>>>>>> > > > -pc_type saviennacl -log_view
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > The error (repeats after each MatSetValues
>>>>>>>>> call):
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > [1]PETSC ERROR: --------------------- Error
>>>>>>>>> Message
>>>>>>>>> > > >
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > > > > > > > [1]PETSC ERROR: Argument out of range
>>>>>>>>> > > > > > > > [1]PETSC ERROR: Inserting a new nonzero at
>>>>>>>>> global
>>>>>>>>> > > > row/column (75, 50) into matrix
>>>>>>>>> > > > > > > > [1]PETSC ERROR: See
>>>>>>>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>>>>>>>>> for
>>>>>>>>> > > > trouble shooting.
>>>>>>>>> > > > > > > > [1]PETSC ERROR: Petsc Development GIT
>>>>>>>>> revision:
>>>>>>>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13
>>>>>>>>> +0300
>>>>>>>>> > > > > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a
>>>>>>>>> cuda-debug named
>>>>>>>>> > > > node50 by valera Wed Aug 15 13:10:44 2018
>>>>>>>>> > > > > > > > [1]PETSC ERROR: Configure options
>>>>>>>>> PETSC_ARCH=cuda-debug
>>>>>>>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>>>>>>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2
>>>>>>>>> --with-shared-libraries=1
>>>>>>>>> > > > --with-debugging=1 --with-cuda=1
>>>>>>>>> --CUDAFLAGS=-arch=sm_60
>>>>>>>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>>>>>>>> > > > > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ()
>>>>>>>>> line 608 in
>>>>>>>>> > > > /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>> > > > > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339
>>>>>>>>> in
>>>>>>>>> > > > /home/valera/petsc/src/mat/interface/matrix.c
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > My Code structure:
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr)
>>>>>>>>> > > > > > > > call MatSetFromOptions(A,ierr)
>>>>>>>>> > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,
>>>>>>>>> numprocs, ierr)
>>>>>>>>> > > > > > > > if (numprocs > 1) then ! set matrix type
>>>>>>>>> parallel
>>>>>>>>> > > > > > > > ! Get local size
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMDACreateNaturalVector(daDummy,Tmpnat,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> VecGetLocalSize(Tmpnat,locsize,ierr)
>>>>>>>>> > > > > > > > call VecDestroy(Tmpnat,ierr)
>>>>>>>>> > > > > > > > ! Set matrix
>>>>>>>>> > > > > > > > #ifdef GPU
>>>>>>>>> > > > > > > > call MatSetType(A,MATAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>>>>>>>> > > > > > > > print*,'SETTING GPU TYPES'
>>>>>>>>> > > > > > > > #else
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATMPIAIJ,ierr)
>>>>>>>>> > > > > > > > call DMSetMatType(daDummy,VECMPI,ierr)
>>>>>>>>> > > > > > > > call MatSetType(A,MATMPIAIJ,ierr)!
>>>>>>>>> > > > > > > > #endif
>>>>>>>>> > > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr)
>>>>>>>>> > > > > > > > else ! set matrix type
>>>>>>>>> sequential
>>>>>>>>> > > > > > > > #ifdef GPU
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetVecType(daDummy,VECSEQVIENNACL,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> MatSetType(A,MATSEQAIJVIENNACL,ierr)
>>>>>>>>> > > > > > > > print*,'SETTING GPU TYPES'
>>>>>>>>> > > > > > > > #else
>>>>>>>>> > > > > > > > call
>>>>>>>>> DMSetMatType(daDummy,MATSEQAIJ,ierr)
>>>>>>>>> > > > > > > > call DMSetMatType(daDummy,VECSEQ,ierr)
>>>>>>>>> > > > > > > > call MatSetType(A,MATSEQAIJ,ierr)
>>>>>>>>> > > > > > > > #endif
>>>>>>>>> > > > > > > > call MatSetUp(A,ierr)
>>>>>>>>> > > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend)
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > do k=zstart,zend-1
>>>>>>>>> > > > > > > > do j=ystart,yend-1
>>>>>>>>> > > > > > > > do i=xstart,xend-1
>>>>>>>>> > > > > > > > [..]
>>>>>>>>> > > > > > > > call
>>>>>>>>> > > >
>>>>>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr)
>>>>>>>>> > > > > > > > [..]
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > --
>>>>>>>>> > > > > > > > What most experimenters take for granted
>>>>>>>>> before they
>>>>>>>>> > > > begin their experiments is infinitely more
>>>>>>>>> interesting than any
>>>>>>>>> > > > results to which their experiments lead.
>>>>>>>>> > > > > > > > -- Norbert Wiener
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > --
>>>>>>>>> > > > > > > > What most experimenters take for granted
>>>>>>>>> before they
>>>>>>>>> > > > begin their experiments is infinitely more
>>>>>>>>> interesting than any
>>>>>>>>> > > > results to which their experiments lead.
>>>>>>>>> > > > > > > > -- Norbert Wiener
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > --
>>>>>>>>> > > > > > > > What most experimenters take for granted
>>>>>>>>> before they
>>>>>>>>> > > > begin their experiments is infinitely more
>>>>>>>>> interesting than any
>>>>>>>>> > > > results to which their experiments lead.
>>>>>>>>> > > > > > > > -- Norbert Wiener
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/>
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > <backt-2.log><backt.log>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180830/91977be1/attachment-0001.html>
More information about the petsc-users
mailing list