[petsc-users] MatSetValues error with ViennaCL types
Manuel Valera
mvalera-w at sdsu.edu
Wed Aug 29 15:15:30 CDT 2018
By the way, one more piece of the puzzle that i feel may contribute,
If i try setting the Preconditioner in the program as either one of:
call PCSetType(pc,PCSAVIENNACL ,ierr)
call PCSetType(pc,PCROWSCALINGVIENNACL,ierr)
call PCSetType(pc,PCCHOWILUVIENNACL,ierr)
It will give an error at compile time of the like:
call PCSetType(pc,PCCHOWILUVIENNACL,ierr)
1
Error: Symbol 'pcchowiluviennacl' at (1) has no IMPLICIT type
If i instead use:
call
PetscOptionsSetValue(PETSC_NULL_OPTIONS,'-pc_type','saviennacl',ierr)
Or give it that option at execute time as a command line argument, it
setups the matrix as intended and it works for one processor,
Let me know if this rings a bell, i just tried updating the petsc repo and
reinstalling but it didn't work,
Thanks,
Manuel
On Wed, Aug 29, 2018 at 11:50 AM, Manuel Valera <mvalera-w at sdsu.edu> wrote:
> Hi everyone,
>
> Thanks for your responses, i understand communicating on this way to this
> level of technicality can be hard, i still think we can work ways to solve
> this problem though,
>
> I can say the following at this point:
>
>
> - Program works without issues in any other non-gpu preconditioner,
> just by calling -pc_type ### .
> - Program works correctly with ViennaCL preconditioners with mpirun -n
> 1, that is on one processor.
> - Program breaks with every ViennaCL preconditioner when attempting
> any more than 1 processors.
> - I haven't tried other GPU preconditioners because ViennaCL gave the
> best performance in 1 processor before.
> - I have made sure the matrix type is mpiaijviennacl as it is printed
> right before the error log.
> - I am attaching what i think are the backtrace logs of the run with
> two cores, as i have limited experience with this kind of debugging, and i
> cannot make sense of what they say, please let me know if you need me to do
> something else.
>
>
> Current options i can think of are giving you permission to clone my model
> repo, or for you to guide me trough the process remotely,
>
> Thanks,
>
> .-.-.-.-
>
>
> The error i get is still:
>
> Matrix type: mpiaijviennacl
>
> Of sizes: 125 x 125
> Matrix type: mpiaijviennacl
>
> Of sizes: 125 x 125
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Currently only handles ViennaCL matrices
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
> Date: 2018-05-31 17:31:13 +0300
> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed
> Aug 29 11:43:25 2018
> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
> --download-viennacl
> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
> [0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/
> interface/precon.c
> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/
> ksp/interface/itfunc.c
> [1]PETSC ERROR: ------------------------------
> ------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [1]PETSC ERROR: INSTEAD the line number of the start of the function
> [1]PETSC ERROR: is given.
> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
> /home/valera/petsc/src/sys/error/errtrace.c
> [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/
> error/err.c
> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
> [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
> interface/precon.c
> [1]PETSC ERROR: [1] KSPSetUp line 294 /home/valera/petsc/src/ksp/
> ksp/interface/itfunc.c
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
> Date: 2018-05-31 17:31:13 +0300
> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed
> Aug 29 11:43:25 2018
> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
> --download-viennacl
> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR: ------------------------------
> ------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: INSTEAD the line number of the start of the function
> [0]PETSC ERROR: is given.
> [0]PETSC ERROR: [0] MatSetErrorIfFailure line 116
> /home/valera/petsc/src/mat/utils/gcreate.c
> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
> interface/precon.c
> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
> interface/precon.c
> [0]PETSC ERROR: [0] KSPSetUp line 294 /home/valera/petsc/src/ksp/
> ksp/interface/itfunc.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
> Date: 2018-05-31 17:31:13 +0300
> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed
> Aug 29 11:43:25 2018
> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
> --download-viennacl
> [0]PETSC ERROR: #4 User provided function() line 0 in unknown file
> [node50:32783] 1 more process has sent help message help-mpi-api.txt /
> mpi-abort
> [node50:32783] Set MCA parameter "orte_base_help_aggregate" to 0 to see
> all help / error messages
>
>
> On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <rupp at iue.tuwien.ac.at> wrote:
>
>> Hi Manuel,
>>
>> as Barry said, it is hard for us to provide any help without having a
>> more complete picture of what is going on.
>>
>> The error you report seems to come from the AMG preconditioner in
>> ViennaCL. This can have many origins. Do other preconditioners run without
>> error? Can you run in a debugger and provide a complete backtrace?
>>
>> Thanks and best regards,
>> Karli
>>
>>
>> On 08/29/2018 01:33 AM, Manuel Valera wrote:
>>
>>> Talked too fast,
>>>
>>> After fixing that problem, i tried more than one mpi processor and got
>>> the following:
>>>
>>> Matrix type: mpiaijviennacl
>>> Of sizes: 125 x 125
>>> Matrix type: mpiaijviennacl
>>> Of sizes: 125 x 125
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
>>> Date: 2018-05-31 17:31:13 +0300
>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue
>>> Aug 28 16:30:02 2018
>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>>> --download-viennacl
>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>>> http://saviennacl.cu>
>>> [0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/
>>> interface/precon.c
>>> [1]PETSC ERROR: ------------------------------
>>> ------------------------------------------
>>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [1]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d
>>> ocumentation/faq.html#valgrind
>>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [1]PETSC ERROR: likely location of problem given in stack below
>>> [1]PETSC ERROR: --------------------- Stack Frames
>>> ------------------------------------
>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [1]PETSC ERROR: INSTEAD the line number of the start of the
>>> function
>>> [1]PETSC ERROR: is given.
>>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>>> /home/valera/petsc/src/sys/error/errtrace.c
>>> [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/err
>>> or/err.c
>>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>>> http://saviennacl.cu>
>>>
>>> [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
>>> interface/precon.c
>>> [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [1]PETSC ERROR: Signal received
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
>>> Date: 2018-05-31 17:31:13 +0300
>>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue
>>> Aug 28 16:30:02 2018
>>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>>> --download-viennacl
>>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
>>> ------------------------------------------------------------
>>> --------------
>>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>> with errorcode 59.
>>>
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>> ------------------------------------------------------------
>>> --------------
>>> [0]PETSC ERROR: ------------------------------
>>> ------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d
>>> ocumentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: --------------------- Stack Frames
>>> ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [0]PETSC ERROR: INSTEAD the line number of the start of the
>>> function
>>> [0]PETSC ERROR: is given.
>>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>>> /home/valera/petsc/src/sys/objects/tagm.c
>>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34
>>> /home/valera/petsc/src/sys/objects/inherit.c
>>> [0]PETSC ERROR: [0] ISCreate line 35 /home/valera/petsc/src/vec/is/
>>> is/interface/isreg.c
>>> [0]PETSC ERROR: [0] ISCreateGeneral line 668
>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>>> http://saviennacl.cu>
>>> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
>>> interface/precon.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
>>> Date: 2018-05-31 17:31:13 +0300
>>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue
>>> Aug 28 16:30:02 2018
>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>>> --download-viennacl
>>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file
>>> [node50:30582] 1 more process has sent help message help-mpi-api.txt /
>>> mpi-abort
>>> [node50:30582] Set MCA parameter "orte_base_help_aggregate" to 0 to see
>>> all help / error messages
>>>
>>>
>>>
>>> It is currently running in 1 mpi processor + GPU but i would like to
>>> call at least 16 mpi processors + GPU to do the rest of the data management
>>> who is not part of the main laplacian on the mpi and the laplacian solution
>>> on the GPU, is this currently possible?
>>>
>>> Thanks for your help,
>>>
>>>
>>>
>>> On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera <mvalera-w at sdsu.edu
>>> <mailto:mvalera-w at sdsu.edu>> wrote:
>>>
>>> Ok, i found the culprit and we can close this thread,
>>>
>>> The problem was a missing variable for setting the maximum columns,
>>> which i deleted at some point without realizing. The error message
>>> was too ambiguous to catch this so i had to compare with a previous
>>> working version of the arguments of MatSetValues, it was evident
>>> then.
>>>
>>> Good news is that i can now set the values with the viennacl types
>>> too,
>>>
>>> Thanks for your kind help,
>>>
>>> Manuel
>>>
>>> On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F.
>>> <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>>
>>>
>>> 1) PetscMalloc() is never valid or needed in Fortran
>>>
>>> 2) there is no reason to use DMSetMatrixPreallocateOnly()
>>> just use DMCreateMatrix() assuming that using a DM (DMDA,
>>> DMPLEX, etc) is suitable for your problem.
>>>
>>> At this end we are totally guessing at what you are doing
>>> and so have little help we can provide. A simple, nonworking
>>> code that tries to do what you would need would help us a great
>>> deal in understanding that you are trying to do.
>>>
>>> Barry
>>>
>>>
>>>
>>>
>>>
>>> > On Aug 28, 2018, at 1:18 PM, Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> >
>>> > Matthew, PetscMalloc gives the same error,
>>> >
>>> > Barry, it would be very hard for me to get the code to a
>>> minimum working example, i guess all i need to understand is how
>>> to setup a DM matrix with DMSetMatrixPreallocateOnly() instead
>>> of MatMPIAIJSetPreallocation() as we were doing before, is there
>>> a simple example who does this in Fortran?
>>> >
>>> > Is the PetscMalloc call needed? is 'call
>>> PetscMalloc(1,row,ierr)' a valid, compilable call to
>>> PetscMalloc? what other reason may there be for this error to
>>> happen ?
>>> >
>>> > Just remembering, that trying to setup the matrix with the
>>> MatAIJSetPreallocation() brings up an error to acknowledge the
>>> viennacl datatypes and that's why i'm trying to make this change
>>> on your recommendation,
>>> >
>>> > Thanks for your help,
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F.
>>> <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>> >
>>> > Send your code in a way we can compile and run it; it must
>>> be some simple issue that is hard to communicate in email.
>>> >
>>> > Barry
>>> >
>>> >
>>> > > On Aug 27, 2018, at 5:51 PM, Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> > >
>>> > > Hello everyone,
>>> > >
>>> > > I just had time to work on this again, and checked the code
>>> for errors on the matrix entries, this is the exact code i was
>>> using for creating the matrix without
>>> DMSetMatrixPreallocateOnly, using MatMPIAIJSetPreallocation and
>>> it worked that way, but trying this way i get the same 'Column
>>> too large' error using any number at the column position of
>>> MatSetValues,
>>> > >
>>> > > I have set up my code to print the column argument (n) of
>>> MatSetValues and in this case is 7 (lower than 124), it still
>>> gives error, even entering a specific number in the MatSetValues
>>> column argument position gives the same error.
>>> > >
>>> > > So next i went back to ex.50 here:
>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/t
>>> utorials/ex50.c.html
>>> <http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/
>>> tutorials/ex50.c.html>
>>> and it has a very similar structure except the PetscMalloc1()
>>> call, so i tried adding that and got:
>>> > >
>>> > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114: undefined
>>> reference to `petscmalloc1_'
>>> > >
>>> > > Any ideas on this behaviour?
>>> > >
>>> > > Thanks so much,
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F.
>>> <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>> > >
>>> > > Column too large: col 10980 max 124
>>> > >
>>> > > You need to check the code that is generating the matrix
>>> entries. The matrix has 124 columns but you are attempting to
>>> put a value at column 10980
>>> > >
>>> > > Barry
>>> > >
>>> > >
>>> > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> > > >
>>> > > > Thanks Matthew and Barry,
>>> > > >
>>> > > > Now my code looks like:
>>> > > >
>>> > > > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>>> > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>> > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>> > > > call DMCreateMatrix(daDummy,A,ierr)
>>> > > > call MatSetFromOptions(A,ierr)
>>> > > > call MatSetUp(A,ierr)
>>> > > > [...]
>>> > > > call
>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS
>>> ERT_VALUES,ierr)
>>> > > > [...]
>>> > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr)
>>> > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr)
>>> > > >
>>> > > > And i get a different error, now is:
>>> > > >
>>> > > > [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > > [0]PETSC ERROR: Argument out of range
>>> > > > [0]PETSC ERROR: Column too large: col 10980 max 124
>>> > > > [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for
>>> trouble shooting.
>>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50
>>> by valera Wed Aug 15 19:40:00 2018
>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>> > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 442 in
>>> /home/valera/petsc/src/mat/impls/aij/seq/aij.c
>>> > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in
>>> /home/valera/petsc/src/mat/interface/matrix.c
>>> > > >
>>> > > >
>>> > > > Thanks again,
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry F.
>>> <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>> > > >
>>> > > > Should be
>>> > > >
>>> > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>> > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>> > > > call DMCreateMatrix(daDummy,A,ierr)
>>> > > >
>>> > > > and remove the rest. You need to set the type of Mat
>>> you want the DM to return BEFORE you create the matrix.
>>> > > >
>>> > > > Barry
>>> > > >
>>> > > >
>>> > > >
>>> > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> > > > >
>>> > > > > Ok thanks for clarifying that, i wasn't sure if there
>>> were different types,
>>> > > > >
>>> > > > > Here is a stripped down version of my code, it seems
>>> like the preallocation is working now since the matrix
>>> population part is working without problem, but here it is for
>>> illustration purposes:
>>> > > > >
>>> > > > > call DMSetMatrixPreallocateOnly(daD
>>> ummy,PETSC_TRUE,ierr)
>>> > > > > call DMCreateMatrix(daDummy,A,ierr)
>>> > > > > call MatSetFromOptions(A,ierr)
>>> > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>> > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>> > > > > call
>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_N
>>> ULL_INTEGER,ierr)
>>> > > > > call MatSetUp(A,ierr)
>>> > > > > [...]
>>> > > > > call
>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS
>>> ERT_VALUES,ierr)
>>> > > > > [...]
>>> > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr)
>>> > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr)
>>> > > > >
>>> > > > > Adding the first line there did the trick,
>>> > > > >
>>> > > > > Now the problem seems to be the program is not
>>> recognizing the matrix as ViennaCL type when i try with more
>>> than one processor, i get now:
>>> > > > >
>>> > > > > [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > > > [0]PETSC ERROR: No support for this operation for this
>>> object type
>>> > > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>>> > > > > [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for
>>> trouble shooting.
>>> > > > > [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>> > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named
>>> node50 by valera Wed Aug 15 14:44:22 2018
>>> > > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>> > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>>> <http://saviennacl.cu>
>>> > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>>> > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>>> > > > >
>>> > > > > When running with:
>>> > > > >
>>> > > > > mpirun -n 1 ./gcmLEP.GPU
>>> tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2
>>> -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl
>>> -pc_type saviennacl -log_view
>>> > > > >
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley
>>> <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>> > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> > > > > It seems to be resumed on: I do not know how to
>>> preallocate a DM Matrix correctly.
>>> > > > >
>>> > > > > There is only one matrix type, Mat. There are no
>>> separate DM matrices. A DM can create a matrix for you
>>> > > > > using DMCreateMatrix(), but that is a Mat and it is
>>> preallocated correctly. I am not sure what you are doing.
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > > Matt
>>> > > > >
>>> > > > > The interesting part is that it only breaks when i need
>>> to populate a GPU matrix from MPI, so kudos on that, but it
>>> seems i need to do better on my code to get this setup working,
>>> > > > >
>>> > > > > Any help would be appreciated,
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley
>>> <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>> > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> > > > > Thanks Matthew,
>>> > > > >
>>> > > > > I try to do that when calling:
>>> > > > >
>>> > > > > call
>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_N
>>> ULL_INTEGER,ierr)
>>> > > > >
>>> > > > > But i am not aware on how to do this for the DM if it
>>> needs something more specific/different,
>>> > > > >
>>> > > > > The error says that your preallocation is wrong for the
>>> values you are putting in. The DM does not control either,
>>> > > > > so I do not understand your email.
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > > Matt
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley
>>> <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>> > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera
>>> <mvalera-w at sdsu.edu <mailto:mvalera-w at sdsu.edu>> wrote:
>>> > > > > Hello PETSc devs,
>>> > > > >
>>> > > > > I am running into an error when trying to use the
>>> MATMPIAIJVIENNACL Matrix type in MPI calls, the same code runs
>>> for MATSEQAIJVIENNACL type in one processor. The error happens
>>> when calling MatSetValues for this specific configuration. It
>>> does not occur when using MPI DMMatrix types only.
>>> > > > >
>>> > > > > The DM properly preallocates the matrix. I am assuming
>>> you do not here.
>>> > > > >
>>> > > > > Matt
>>> > > > >
>>> > > > > Any help will be appreciated,
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > My program call:
>>> > > > >
>>> > > > > mpirun -n 2 ./gcmLEP.GPU
>>> tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2
>>> -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl
>>> -pc_type saviennacl -log_view
>>> > > > >
>>> > > > >
>>> > > > > The error (repeats after each MatSetValues call):
>>> > > > >
>>> > > > > [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > > > [1]PETSC ERROR: Argument out of range
>>> > > > > [1]PETSC ERROR: Inserting a new nonzero at global
>>> row/column (75, 50) into matrix
>>> > > > > [1]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for
>>> trouble shooting.
>>> > > > > [1]PETSC ERROR: Petsc Development GIT revision:
>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>>> > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named
>>> node50 by valera Wed Aug 15 13:10:44 2018
>>> > > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>>> > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in
>>> /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>> > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339 in
>>> /home/valera/petsc/src/mat/interface/matrix.c
>>> > > > >
>>> > > > >
>>> > > > > My Code structure:
>>> > > > >
>>> > > > > call DMCreateMatrix(daDummy,A,ierr)
>>> > > > > call MatSetFromOptions(A,ierr)
>>> > > > > call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr)
>>> > > > > if (numprocs > 1) then ! set matrix type parallel
>>> > > > > ! Get local size
>>> > > > > call DMDACreateNaturalVector(daDummy,Tmpnat,ierr)
>>> > > > > call VecGetLocalSize(Tmpnat,locsize,ierr)
>>> > > > > call VecDestroy(Tmpnat,ierr)
>>> > > > > ! Set matrix
>>> > > > > #ifdef GPU
>>> > > > > call MatSetType(A,MATAIJVIENNACL,ierr)
>>> > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>>> > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>>> > > > > print*,'SETTING GPU TYPES'
>>> > > > > #else
>>> > > > > call DMSetMatType(daDummy,MATMPIAIJ,ierr)
>>> > > > > call DMSetMatType(daDummy,VECMPI,ierr)
>>> > > > > call MatSetType(A,MATMPIAIJ,ierr)!
>>> > > > > #endif
>>> > > > > call
>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_N
>>> ULL_INTEGER,ierr)
>>> > > > > else ! set matrix type sequential
>>> > > > > #ifdef GPU
>>> > > > > call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr)
>>> > > > > call DMSetVecType(daDummy,VECSEQVIENNACL,ierr)
>>> > > > > call MatSetType(A,MATSEQAIJVIENNACL,ierr)
>>> > > > > print*,'SETTING GPU TYPES'
>>> > > > > #else
>>> > > > > call DMSetMatType(daDummy,MATSEQAIJ,ierr)
>>> > > > > call DMSetMatType(daDummy,VECSEQ,ierr)
>>> > > > > call MatSetType(A,MATSEQAIJ,ierr)
>>> > > > > #endif
>>> > > > > call MatSetUp(A,ierr)
>>> > > > > call
>>> getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend)
>>> > > > >
>>> > > > > do k=zstart,zend-1
>>> > > > > do j=ystart,yend-1
>>> > > > > do i=xstart,xend-1
>>> > > > > [..]
>>> > > > > call
>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS
>>> ERT_VALUES,ierr)
>>> > > > > [..]
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > What most experimenters take for granted before they
>>> begin their experiments is infinitely more interesting than any
>>> results to which their experiments lead.
>>> > > > > -- Norbert Wiener
>>> > > > >
>>> > > > > https://www.cse.buffalo.edu/~knepley/
>>> <https://www.cse.buffalo.edu/%7Eknepley/>
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > What most experimenters take for granted before they
>>> begin their experiments is infinitely more interesting than any
>>> results to which their experiments lead.
>>> > > > > -- Norbert Wiener
>>> > > > >
>>> > > > > https://www.cse.buffalo.edu/~knepley/
>>> <https://www.cse.buffalo.edu/%7Eknepley/>
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > What most experimenters take for granted before they
>>> begin their experiments is infinitely more interesting than any
>>> results to which their experiments lead.
>>> > > > > -- Norbert Wiener
>>> > > > >
>>> > > > > https://www.cse.buffalo.edu/~knepley/
>>> <https://www.cse.buffalo.edu/%7Eknepley/>
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> >
>>> >
>>>
>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180829/97adbdc5/attachment-0001.html>
More information about the petsc-users
mailing list