[petsc-users] How to extract array portions
FRANCAVILLA MATTEO ALESSANDRO
d019598 at polito.it
Wed Apr 6 07:12:56 CDT 2016
Thanks Dave, that's exactly what I was looking for, and it (partially)
solved my problem.
The MatMult operation now works fine, unfortunately not for every number of
MPI processes (while the MatMult involving each submatrix works fine for any
number of processes):
....
matArray(1) = L_discretized
matArray(2) = K_discretized
matArray(3) = L_MR
matArray(4) = K_MR
CALL
MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matArray,TheMatrix,IERR)
CALL MatSetFromOptions(TheMatrix,IERR)
!
CALL VecCreate(PETSC_COMM_WORLD,x,IERR)
CALL VecSetSizes(x,PETSC_DECIDE,2*NumberOfUnknowns,IERR)
CALL VecSetFromOptions(x,IERR)
CALL VecSet(x,0.+0.*PETSC_i,IERR)
CALL VecSetValues(x,1,0,1.+0.*PETSC_i,INSERT_VALUES,IERR)
CALL VecAssemblyBegin(x,IERR)
CALL VecAssemblyEnd(x,IERR)
CALL VecCreate(PETSC_COMM_WORLD,y,IERR)
CALL VecSetSizes(y,PETSC_DECIDE,NumberOfUnknowns+NumberOfMeasures,IERR)
CALL VecSetFromOptions(y,IERR)
CALL MatMult(TheMatrix,x,y,IERR)
....
I guess there is something wrong with the distribution of vectors between
the MPI processes. With the specific sizes of my problem everything works
fine with 1-2-3-6 processes, otherwise I get "Nonconforming object sizes"
error, e.g. with 8 processes:
[2]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: Nonconforming object sizes
[2]PETSC ERROR: Mat mat,Vec y: local dim 383 382
[2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[2]PETSC ERROR: Nonconforming object sizes
./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by
alessandro Wed Apr 6 13:31:45 2016
[2]PETSC ERROR: Configure options
--with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
--with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
--with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
--with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
--with-scalar-type=complex
[2]PETSC ERROR: #1 MatMult() line 2216 in
/home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
...
[6]PETSC ERROR: Mat mat,Vec y: local dim 381 382
...
[7]PETSC ERROR: Mat mat,Vec y: local dim 381 382
...
[1]PETSC ERROR: Mat mat,Vec y: local dim 383 382
...
I noticed that the code works (runs and provides correct results) when the
number of processes divides the length of y2 (number of rows of A21 and
A22); however when I apply MatMult separately on A21 and A22 everything
works fine with any number of processes
Any idea where I'm messing up here?
Thanks,
Alessandro
On Wed, 6 Apr 2016 11:16:43 +0200
Dave May <dave.mayhem23 at gmail.com> wrote:
> On 6 April 2016 at 11:08, FRANCAVILLA MATTEO ALESSANDRO
><d019598 at polito.it>
> wrote:
>
>> Hi,
>>
>> I'm trying to set a linear problem with a 2x2 block
>>matrix (say A=[A11
>> A12; A21 A22] in matlab notation). Obviously I could
>>generate a single
>> matrix, but I would like to keep it as a 2x2 block
>>matrix with the 4 blocks
>> generated and used independently (e.g., because at some
>>point I need to
>> change some of those blocks to matrix-free). My plan was
>>then to generate a
>> MatShell and set a MATOP_MULT shell operation to
>>explicitly compute y=A*x
>> by splitting x=[x1;x2] and y=[y1;y2] and carrying out
>>the single
>> matrix-vector multiplications, and reassembling the
>>final result.
>
>
> Take a look at MatNest - it does exactly what you want.
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATNEST.html#MATNEST
>
> Thanks,
> Dave
>
>
>> My problem is that my efforts in extracting x1 and x2
>>from the original x
>> vector have failed. The following Fortran code is what
>>seemed to me the
>> most reasonable thing to do to set the matrix-vector
>>multiplication, but it
>> does not work
>>
>>
>> !======================================================
>> subroutine MyMult(A,x,y,IERR)
>>
>> implicit none
>>
>> #include <petsc/finclude/petscsys.h>
>> #include <petsc/finclude/petscvec.h>
>> #include <petsc/finclude/petscvec.h90>
>> #include <petsc/finclude/petscmat.h>
>>
>> Mat A
>> Vec x,y, x1, x2, y1, y2
>> IS :: ix1, ix2, iy1, iy2
>> PetscErrorCode ierr
>> PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures
>> PetscInt, ALLOCATABLE :: INDICES(:)
>> PetscMPIInt :: RANK
>>
>> CALL VecGetSize(x,N,IERR)
>> NumberOfUnknowns = N / 2
>> CALL VecGetSize(y,N,IERR)
>> NumberOfMeasures = N - NumberOfUnknowns
>>
>> N = NumberOfUnknowns +
>>MAX(NumberOfUnknowns,NumberOfMeasures)
>> ALLOCATE(INDICES(N))
>> INDICES = (/ (ii, ii=0, N-1) /)
>>
>> CALL
>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR)
>> CALL VecGetSubVector(x,ix1,x1,IERR)
>> CALL
>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR)
>> CALL VecGetSubVector(x,ix2,x2,IERR)
>> CALL
>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR)
>> CALL VecGetSubVector(y,iy1,y1,IERR)
>> CALL
>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR)
>> CALL VecGetSubVector(y,iy2,y2,IERR)
>>
>> CALL MatMult(L_discretized,x1,y1,IERR)
>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR)
>> CALL MatMult(L_MR,x1,y2,IERR)
>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR)
>>
>> CALL VecRestoreSubVector(y,iy1,y1,IERR)
>> CALL VecRestoreSubVector(y,iy2,y2,IERR)
>>
>> return
>> end subroutine MyMult
>> !======================================================
>>
>>
>> Obviously the sequence of calls to ISCreateGeneral and
>>VecGetSubVector
>> does not do what I expect, as the errors I'm getting are
>>in the following
>> MatMult multiplications (the global sizes of x1 and x2
>>SHOULD BE both 771,
>> while y1 and y2 should be 771 and 2286):
>>
>> 1) executed with 1 MPI process:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Nonconforming object sizes
>> [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771
>> [0]PETSC ERROR: See
>>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03,
>>2015
>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named
>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6
>>10:54:03 2016
>> [0]PETSC ERROR: Configure options
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
>> --with-scalar-type=complex
>> [0]PETSC ERROR: #1 MatMult() line 2215 in
>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Nonconforming object sizes
>> [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771
>> [0]PETSC ERROR: See
>>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03,
>>2015
>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named
>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6
>>10:54:03 2016
>> [0]PETSC ERROR: Configure options
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
>> --with-scalar-type=complex
>> [0]PETSC ERROR: #2 MatMultAdd() line 2396 in
>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
>>
>>
>>
>> 2) executed with 4 MPI processes (I'm copying the output
>>of process 0
>> only, they're all the same):
>>
>> ....
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Nonconforming object sizes
>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084
>> [0]PETSC ERROR: See
>>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03,
>>2015
>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named
>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6
>>10:49:13 2016
>> [0]PETSC ERROR: Configure options
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
>> --with-scalar-type=complex
>> [0]PETSC ERROR: #1 MatMult() line 2214 in
>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Nonconforming object sizes
>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084
>> [0]PETSC ERROR: See
>>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03,
>>2015
>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named
>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6
>>10:49:13 2016
>> [0]PETSC ERROR: Configure options
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
>> --with-scalar-type=complex
>> [0]PETSC ERROR: #2 MatMultAdd() line 2393 in
>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Nonconforming object sizes
>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084
>> [0]PETSC ERROR: See
>>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03,
>>2015
>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named
>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6
>>10:49:13 2016
>> [0]PETSC ERROR: Configure options
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
>> --with-scalar-type=complex
>> [0]PETSC ERROR: #3 MatMult() line 2214 in
>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Nonconforming object sizes
>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084
>> [0]PETSC ERROR: See
>>http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03,
>>2015
>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named
>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6
>>10:49:13 2016
>> [0]PETSC ERROR: Configure options
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl
>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc
>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90
>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec
>> --with-scalar-type=complex
>> [0]PETSC ERROR: #4 MatMultAdd() line 2393 in
>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c
>> ....
>>
>>
>> Any idea? I would like to keep the flexibility of having
>>a sequence of
>> mat-vec multiplications as in
>>
>> CALL MatMult(L_discretized,x1,y1,IERR)
>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR)
>> CALL MatMult(L_MR,x1,y2,IERR)
>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR)
>>
>> but for doing that I need to be able to extract x1 and
>>x2, and to
>> reassemble the partial results into y.
>>
>> Thanks in advance,
>> Alessandro
>>
___________________________________
Matteo Alessandro Francavilla, PhD
Antenna and EMC Lab (LACE)
Istituto Superiore Mario Boella (ISMB)
Torino, Italy
phone +39 011 2276 711
More information about the petsc-users
mailing list