[petsc-users] problem in using MatCreateMPIAIJWithArrays routine
hong at aspiritech.org
hong at aspiritech.org
Thu Aug 14 09:55:41 CDT 2014
Priyank :
>
> ---------------------------------------------------------------------------------------------------------------------------------------------
> --> the i and j array should be 0 based.
> if I want to add one row in P2 as below then what would be the i and j array
> for that?
> 1 0 0
> 2 0 3 P0
> -------
> 4 5 6 P1
> -------
> 0 0 7 P2
> Because the start position for the non-zero value is 2.
For P2:
i = { 0,1}
j = { 2}
v = {7}
Hong
>
> On 13-Aug-2014, at 6:11 pm, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Usually the matrix is generated in parallel so each process has part of the
> matrix. If the matrix is generated sequentially then that will limit the
> parallel performance you can get.
>
> The MPI model is not particularly friendly to “some parts of my code are
> purely sequential, but I want other parts to be parallel”
>
> Barry
>
> On Aug 13, 2014, at 3:52 PM, Priyank Patel <ppate024 at odu.edu> wrote:
>
> Thank you Barry,
>
> But what if I have a matrix that is 1000000 X 1000000 and with 32
> processors? I would be a very time consuming task to indicate column and row
> indices for all of them I guess. Do you think there is another way to do it?
> I just want to do parallel matrix matrix multiplication.
>
> Priyank Patel
> ppate024 at odu.edu
>
>
>
>
> On 12-Aug-2014, at 9:20 pm, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On each process you should only pass the row and column indices for the rows
> you want on that process. In other words do not pass the entire matrix on
> each process. Take a look at the example matrix in the manual page
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html
>
> Barry
>
> On Aug 12, 2014, at 8:02 PM, Priyank Patel <ppate024 at odu.edu> wrote:
>
> Hello,
> I am trying to do sparse - dense matrix multiplication and for that I am
> using a CSR Matrix(that are three array) in PETSC using the function
> MatCreateMPIAIJWithArrays() to create a petsc matrix.
> My call to routine is as below
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> //Initialize PETSC
> MPI_Init(&argc, &argv);
> PetscInitialize(&argc, &argv, (char*) 0, help);
>
> MPI_Comm_rank(PETSC_COMM_WORLD, &taskid);
> MPI_Comm_size(PETSC_COMM_WORLD, &numtasks);
>
> //Create A Sparse Matrix fro the files
> ierr = MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, m, m, PETSC_DETERMINE,
> PETSC_DETERMINE, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);
>
> //ierr = MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, m, m, h_iA, h_jA,
> h_valA, &A);CHKERRQ(ierr);
>
> MatView(A,PETSC_VIEWER_STDOUT_WORLD);
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> My make file is as below
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> PETSC_DIR=/export/software/petsc-3.3
>
> INTEL_ROOT=/export/software/intel/composerxe-2011.4.191/
> MKL_LIB_PATH_64=/export/software/intel/mkl/lib/intel64/
> INTEL_LIB_PATH_64=/export/software/intel/composerxe-2011.4.191/compiler/lib/intel64/
> MKL_INC_PATH=/export/software/intel/mkl/include/
> SCALAPACK_LIB_PATH=/export/software/scalapack-2.0.2/lib/
> SCALAPACK_INC_PATH=/export/software/scalapack-2.0.2/
> GENERAL_LIB_PATH=/export/software/lib/
>
> CLINKER=mpicc
>
> PKG_INC = -I${MKL_INC_PATH}
> PKG_LIB = -Wl,-rpath,${MKL_LIB_PATH_64} -L${MKL_LIB_PATH_64}
> -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -Wl,-rpath,${INTEL_LIB_PATH_64}
> -L${INTEL_LIB_PATH_64} -liomp5 -Wl,-rpath,${SCALAPACK_LIB_PATH}
> -L${SCALAPACK_LIB_PATH} -lscalapack -llapack -lblas
>
> CFLAGS =
> FFLAGS =
> CPPFLAGS = ${PKG_INC}
> FPPFLAGS =
> CLEANFILES =
>
> include ${PETSC_DIR}/conf/variables
> include ${PETSC_DIR}/conf/rules
>
> blockCG_1: blockCG_1.o chkopts
> -${CLINKER} -o blockCG_1 blockCG_1.o ${PKG_LIB} ${PETSC_LIB}
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> And I tried to run the program with following command
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> /opt/openmpi/bin/mpirun -np 3 ./blockCG_1 /home/ppatel/MPI/LF10_2
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> but the routine is not distributing the matrix and give me the result as
> below.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> loading matrix from : /home/ppatel/MPI/LF10_2
> nRows: 18, nCoefs: 82.
> Matrix Object: 1 MPI processes
> type: mpiaij
> row 0: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 1: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 2: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 3: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
> 477.155)
> row 4: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
> row 5: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
> 477.155)
> row 6: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
> row 7: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
> 477.155)
> row 8: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
> 1.76724)
> row 9: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
> 477.155)
> row 10: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
> 1.76724)
> row 11: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
> 477.155)
> row 12: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
> 1.76724)
> row 13: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
> 477.155)
> row 14: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
> 1.76724)
> row 15: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 16: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 17: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 18: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 19: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 20: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 21: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
> 477.155)
> row 22: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
> 1.76724)
> row 23: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
> 477.155)
> row 24: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
> 1.76724)
> row 25: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
> 477.155)
> row 26: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
> 1.76724)
> row 27: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
> 477.155)
> row 28: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
> 1.76724)
> row 29: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
> 477.155)
> row 30: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
> 1.76724)
> row 31: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
> 477.155)
> row 32: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
> 1.76724)
> row 33: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 34: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 35: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 36: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 37: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 38: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 39: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
> 477.155)
> row 40: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
> 1.76724)
> row 41: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
> 477.155)
> row 42: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
> 1.76724)
> row 43: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
> 477.155)
> row 44: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
> 1.76724)
> row 45: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
> 477.155)
> row 46: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
> 1.76724)
> row 47: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
> 477.155)
> row 48: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
> 1.76724)
> row 49: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
> 477.155)
> row 50: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
> 1.76724)
> row 51: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 52: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 53: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 54: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 55: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 56: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 57: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
> 477.155)
> row 58: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
> 1.76724)
> row 59: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
> 477.155)
> row 60: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
> 1.76724)
> row 61: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
> 477.155)
> row 62: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
> 1.76724)
> row 63: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
> 477.155)
> row 64: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
> 1.76724)
> row 65: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
> 477.155)
> row 66: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
> 1.76724)
> row 67: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
> 477.155)
> row 68: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
> 1.76724)
> row 69: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 70: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 71: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 72: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 73: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 74: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 75: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
> 477.155)
> row 76: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
> 1.76724)
> row 77: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
> 477.155)
> row 78: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
> 1.76724)
> row 79: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
> 477.155)
> row 80: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
> 1.76724)
> row 81: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
> 477.155)
> row 82: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
> 1.76724)
> row 83: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
> 477.155)
> row 84: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
> 1.76724)
> row 85: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
> 477.155)
> row 86: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
> 1.76724)
> row 87: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 88: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 89: (15, 477.155) (16, 1.76724) (17, 3.53448)
> row 90: (0, 3.53448) (1, -477.155) (2, 1.76724)
> row 91: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
> row 92: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
> row 93: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
> 477.155)
> row 94: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
> 1.76724)
> row 95: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
> 477.155)
> row 96: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
> 1.76724)
> row 97: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
> 477.155)
> row 98: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
> 1.76724)
> row 99: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
> 477.155)
> row 100: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
> 1.76724)
> row 101: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
> 477.155)
> row 102: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
> 1.76724)
> row 103: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
> 477.155)
> row 104: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
> 1.76724)
> row 105: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
> row 106: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
> row 107: (15, 477.155) (16, 1.76724) (17, 3.53448)
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> that is creating a matrix with rows repeated 6 times instead of distributed
> over 6 process.
>
> Can you please tell me where I am doing wrong??
>
> Thanks for your time
>
> Priyank Patel
> ppate024 at odu.edu
>
>
>
>
>
>
>
>
> --
>
>
>
>
>
> --
> BEGIN-ANTISPAM-VOTING-LINKS
> ------------------------------------------------------
>
> Teach CanIt if this mail (ID 03MCmbAxj) is spam:
> Spam:
> https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=s
> Not spam:
> https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=n
> Forget vote:
> https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=f
> ------------------------------------------------------
> END-ANTISPAM-VOTING-LINKS
>
>
More information about the petsc-users
mailing list