[petsc-users] problem in using MatCreateMPIAIJWithArrays routine
Barry Smith
bsmith at mcs.anl.gov
Tue Aug 12 20:20:49 CDT 2014
On each process you should only pass the row and column indices for the rows you want on that process. In other words do not pass the entire matrix on each process. Take a look at the example matrix in the manual page http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html
Barry
On Aug 12, 2014, at 8:02 PM, Priyank Patel <ppate024 at odu.edu> wrote:
> Hello,
> I am trying to do sparse - dense matrix multiplication and for that I am using a CSR Matrix(that are three array) in PETSC using the function MatCreateMPIAIJWithArrays() to create a petsc matrix.
> My call to routine is as below
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> //Initialize PETSC
> MPI_Init(&argc, &argv);
> PetscInitialize(&argc, &argv, (char*) 0, help);
>
> MPI_Comm_rank(PETSC_COMM_WORLD, &taskid);
> MPI_Comm_size(PETSC_COMM_WORLD, &numtasks);
>
> //Create A Sparse Matrix fro the files
> ierr = MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, m, m, PETSC_DETERMINE, PETSC_DETERMINE, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);
>
> //ierr = MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, m, m, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);
>
> MatView(A,PETSC_VIEWER_STDOUT_WORLD);
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> My make file is as below
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> PETSC_DIR=/export/software/petsc-3.3
>
> INTEL_ROOT=/export/software/intel/composerxe-2011.4.191/
> MKL_LIB_PATH_64=/export/software/intel/mkl/lib/intel64/
> INTEL_LIB_PATH_64=/export/software/intel/composerxe-2011.4.191/compiler/lib/intel64/
> MKL_INC_PATH=/export/software/intel/mkl/include/
> SCALAPACK_LIB_PATH=/export/software/scalapack-2.0.2/lib/
> SCALAPACK_INC_PATH=/export/software/scalapack-2.0.2/
> GENERAL_LIB_PATH=/export/software/lib/
>
> CLINKER=mpicc
>
> PKG_INC = -I${MKL_INC_PATH}
> PKG_LIB = -Wl,-rpath,${MKL_LIB_PATH_64} -L${MKL_LIB_PATH_64} -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -Wl,-rpath,${INTEL_LIB_PATH_64} -L${INTEL_LIB_PATH_64} -liomp5 -Wl,-rpath,${SCALAPACK_LIB_PATH} -L${SCALAPACK_LIB_PATH} -lscalapack -llapack -lblas
>
> CFLAGS =
> FFLAGS =
> CPPFLAGS = ${PKG_INC}
> FPPFLAGS =
> CLEANFILES =
>
> include ${PETSC_DIR}/conf/variables
> include ${PETSC_DIR}/conf/rules
>
> blockCG_1: blockCG_1.o chkopts
> -${CLINKER} -o blockCG_1 blockCG_1.o ${PKG_LIB} ${PETSC_LIB}
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> And I tried to run the program with following command
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> /opt/openmpi/bin/mpirun -np 3 ./blockCG_1 /home/ppatel/MPI/LF10_2
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> but the routine is not distributing the matrix and give me the result as below.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> Matrix Object: 1 MPI processes
> type: mpiaij
> row 0: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 1: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 2: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 3: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
> row 4: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 5: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
> row 6: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 7: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
> row 8: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
> row 9: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
> row 10: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
> row 11: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
> row 12: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
> row 13: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
> row 14: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
> row 15: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 16: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 17: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 18: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 19: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 20: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 21: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
> row 22: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 23: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
> row 24: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 25: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
> row 26: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
> row 27: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
> row 28: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
> row 29: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
> row 30: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
> row 31: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
> row 32: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
> row 33: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 34: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 35: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 36: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 37: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 38: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 39: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
> row 40: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 41: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
> row 42: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 43: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
> row 44: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
> row 45: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
> row 46: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
> row 47: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
> row 48: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
> row 49: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
> row 50: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
> row 51: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 52: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 53: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 54: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 55: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 56: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 57: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
> row 58: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 59: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
> row 60: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 61: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
> row 62: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
> row 63: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
> row 64: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
> row 65: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
> row 66: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
> row 67: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
> row 68: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
> row 69: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 70: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 71: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 72: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 73: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 74: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 75: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
> row 76: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 77: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
> row 78: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 79: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
> row 80: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
> row 81: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
> row 82: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
> row 83: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
> row 84: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
> row 85: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
> row 86: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
> row 87: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 88: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 89: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 90: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 91: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 92: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 93: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
> row 94: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 95: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
> row 96: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 97: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
> row 98: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
> row 99: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
> row 100: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
> row 101: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
> row 102: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
> row 103: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
> row 104: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
> row 105: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 106: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 107: (15, 477.155) (16, 1.76724) (17, 3.53448)
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> that is creating a matrix with rows repeated 6 times instead of distributed over 6 process.
>
> Can you please tell me where I am doing wrong??
>
> Thanks for your time
>
> Priyank Patel
> ppate024 at odu.edu
>
>
>
>
>
More information about the petsc-users
mailing list