[petsc-users] problem in using MatCreateMPIAIJWithArrays routine
Priyank Patel
ppate024 at odu.edu
Thu Aug 14 06:52:02 CDT 2014
Hello,
As per your advice I am trying to pass the row and column indices to MatCreateMPIAIJWithArrays(),
but there is a confusion, if we follow the matrix given in the manual.
---------------------------------------------------------------------------------------------------------------------------------------------
The i and j indices are 0 based, and i indices are indices corresponding to the local j array.
3990: The format which is used for the sparse matrix input, is equivalent to a
3991: row-major ordering.. i.e for the following matrix, the input data expected is
3992: as shown:
3994: 1 0 0
3995: 2 0 3 P0
3996: -------
3997: 4 5 6 P1
3999: Process0 [P0]: rows_owned=[0,1]
4000: i = {0,1,3} [size = nrow+1 = 2+1]
4001: j = {0,0,2} [size = nz = 6]
4002: v = {1,2,3} [size = nz = 6]
4004: Process1 [P1]: rows_owned=[2]
4005: i = {0,3} [size = nrow+1 = 1+1]
4006: j = {0,1,2} [size = nz = 6]
4007: v = {4,5,6} [size = nz = 6]
---------------------------------------------------------------------------------------------------------------------------------------------
--> the i and j array should be 0 based.
if I want to add one row in P2 as below then what would be the i and j array for that?
1 0 0
2 0 3 P0
-------
4 5 6 P1
-------
0 0 7 P2
Because the start position for the non-zero value is 2.
Thanks
Priyank
On 13-Aug-2014, at 6:11 pm, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> Usually the matrix is generated in parallel so each process has part of the matrix. If the matrix is generated sequentially then that will limit the parallel performance you can get.
>
> The MPI model is not particularly friendly to “some parts of my code are purely sequential, but I want other parts to be parallel”
>
> Barry
>
> On Aug 13, 2014, at 3:52 PM, Priyank Patel <ppate024 at odu.edu> wrote:
>
>> Thank you Barry,
>>
>> But what if I have a matrix that is 1000000 X 1000000 and with 32 processors? I would be a very time consuming task to indicate column and row indices for all of them I guess. Do you think there is another way to do it? I just want to do parallel matrix matrix multiplication.
>>
>> Priyank Patel
>> ppate024 at odu.edu
>>
>>
>>
>>
>> On 12-Aug-2014, at 9:20 pm, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>> On each process you should only pass the row and column indices for the rows you want on that process. In other words do not pass the entire matrix on each process. Take a look at the example matrix in the manual page http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html
>>>
>>> Barry
>>>
>>> On Aug 12, 2014, at 8:02 PM, Priyank Patel <ppate024 at odu.edu> wrote:
>>>
>>>> Hello,
>>>> I am trying to do sparse - dense matrix multiplication and for that I am using a CSR Matrix(that are three array) in PETSC using the function MatCreateMPIAIJWithArrays() to create a petsc matrix.
>>>> My call to routine is as below
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> //Initialize PETSC
>>>> MPI_Init(&argc, &argv);
>>>> PetscInitialize(&argc, &argv, (char*) 0, help);
>>>>
>>>> MPI_Comm_rank(PETSC_COMM_WORLD, &taskid);
>>>> MPI_Comm_size(PETSC_COMM_WORLD, &numtasks);
>>>>
>>>> //Create A Sparse Matrix fro the files
>>>> ierr = MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, m, m, PETSC_DETERMINE, PETSC_DETERMINE, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);
>>>>
>>>> //ierr = MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, m, m, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);
>>>>
>>>> MatView(A,PETSC_VIEWER_STDOUT_WORLD);
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> My make file is as below
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> PETSC_DIR=/export/software/petsc-3.3
>>>>
>>>> INTEL_ROOT=/export/software/intel/composerxe-2011.4.191/
>>>> MKL_LIB_PATH_64=/export/software/intel/mkl/lib/intel64/
>>>> INTEL_LIB_PATH_64=/export/software/intel/composerxe-2011.4.191/compiler/lib/intel64/
>>>> MKL_INC_PATH=/export/software/intel/mkl/include/
>>>> SCALAPACK_LIB_PATH=/export/software/scalapack-2.0.2/lib/
>>>> SCALAPACK_INC_PATH=/export/software/scalapack-2.0.2/
>>>> GENERAL_LIB_PATH=/export/software/lib/
>>>>
>>>> CLINKER=mpicc
>>>>
>>>> PKG_INC = -I${MKL_INC_PATH}
>>>> PKG_LIB = -Wl,-rpath,${MKL_LIB_PATH_64} -L${MKL_LIB_PATH_64} -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -Wl,-rpath,${INTEL_LIB_PATH_64} -L${INTEL_LIB_PATH_64} -liomp5 -Wl,-rpath,${SCALAPACK_LIB_PATH} -L${SCALAPACK_LIB_PATH} -lscalapack -llapack -lblas
>>>>
>>>> CFLAGS =
>>>> FFLAGS =
>>>> CPPFLAGS = ${PKG_INC}
>>>> FPPFLAGS =
>>>> CLEANFILES =
>>>>
>>>> include ${PETSC_DIR}/conf/variables
>>>> include ${PETSC_DIR}/conf/rules
>>>>
>>>> blockCG_1: blockCG_1.o chkopts
>>>> -${CLINKER} -o blockCG_1 blockCG_1.o ${PKG_LIB} ${PETSC_LIB}
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> And I tried to run the program with following command
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> /opt/openmpi/bin/mpirun -np 3 ./blockCG_1 /home/ppatel/MPI/LF10_2
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> but the routine is not distributing the matrix and give me the result as below.
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> Matrix Object: 1 MPI processes
>>>> type: mpiaij
>>>> row 0: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 1: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 2: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 3: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
>>>> row 4: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
>>>> row 5: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
>>>> row 6: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
>>>> row 7: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
>>>> row 8: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
>>>> row 9: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
>>>> row 10: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
>>>> row 11: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
>>>> row 12: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
>>>> row 13: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
>>>> row 14: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
>>>> row 15: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 16: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 17: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 18: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 19: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 20: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 21: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
>>>> row 22: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
>>>> row 23: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
>>>> row 24: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
>>>> row 25: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
>>>> row 26: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
>>>> row 27: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
>>>> row 28: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
>>>> row 29: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
>>>> row 30: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
>>>> row 31: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
>>>> row 32: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
>>>> row 33: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 34: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 35: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 36: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 37: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 38: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 39: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
>>>> row 40: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
>>>> row 41: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
>>>> row 42: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
>>>> row 43: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
>>>> row 44: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
>>>> row 45: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
>>>> row 46: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
>>>> row 47: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
>>>> row 48: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
>>>> row 49: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
>>>> row 50: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
>>>> row 51: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 52: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 53: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 54: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 55: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 56: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 57: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
>>>> row 58: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
>>>> row 59: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
>>>> row 60: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
>>>> row 61: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
>>>> row 62: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
>>>> row 63: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
>>>> row 64: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
>>>> row 65: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
>>>> row 66: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
>>>> row 67: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
>>>> row 68: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
>>>> row 69: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 70: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 71: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 72: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 73: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 74: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 75: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
>>>> row 76: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
>>>> row 77: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
>>>> row 78: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
>>>> row 79: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
>>>> row 80: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
>>>> row 81: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
>>>> row 82: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
>>>> row 83: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
>>>> row 84: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
>>>> row 85: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
>>>> row 86: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
>>>> row 87: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 88: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 89: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 90: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 91: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 92: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 93: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)
>>>> row 94: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)
>>>> row 95: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)
>>>> row 96: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)
>>>> row 97: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)
>>>> row 98: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)
>>>> row 99: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)
>>>> row 100: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)
>>>> row 101: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)
>>>> row 102: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)
>>>> row 103: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)
>>>> row 104: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)
>>>> row 105: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 106: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 107: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> that is creating a matrix with rows repeated 6 times instead of distributed over 6 process.
>>>>
>>>> Can you please tell me where I am doing wrong??
>>>>
>>>> Thanks for your time
>>>>
>>>> Priyank Patel
>>>> ppate024 at odu.edu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>
>
>
>
> --
> BEGIN-ANTISPAM-VOTING-LINKS
> ------------------------------------------------------
>
> Teach CanIt if this mail (ID 03MCmbAxj) is spam:
> Spam: https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=s
> Not spam: https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=n
> Forget vote: https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=f
> ------------------------------------------------------
> END-ANTISPAM-VOTING-LINKS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140814/352c5c0f/attachment-0001.html>
More information about the petsc-users
mailing list