<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hello,<div>I am trying to do sparse - dense matrix multiplication and for that I am using a CSR Matrix(that are three array) in PETSC using the function <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html#MatCreateMPIAIJWithArrays">MatCreateMPIAIJWithArrays</a>() to create a petsc matrix.</div><div>My call to routine is as below</div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><div><span class="Apple-tab-span" style="color: rgb(150, 211, 95); white-space: pre;"> </span><font color="#77bb41">//Initialize PETSC</font></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>MPI_Init(&argc, &argv);</font></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>PetscInitialize(&argc, &argv, (char*) 0, help);</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font color="#77bb41"> </font></span></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>MPI_Comm_rank(PETSC_COMM_WORLD, &taskid);</font></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>MPI_Comm_size(PETSC_COMM_WORLD, &numtasks);</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font color="#77bb41"> </font></span></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>//Create A Sparse Matrix fro the files</font></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>ierr = MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, m, m, PETSC_DETERMINE, PETSC_DETERMINE, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font color="#77bb41"> </font></span></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>//ierr = MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, m, m, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font color="#77bb41"> </font></span></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space: pre;"> </span>MatView(A,PETSC_VIEWER_STDOUT_WORLD);</font></div></div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>My make file is as below</div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><div><font color="#77bb41">PETSC_DIR=/export/software/petsc-3.3</font></div><div><font color="#77bb41"><br></font></div><div><font color="#77bb41">INTEL_ROOT=/export/software/intel/composerxe-2011.4.191/</font></div><div><font color="#77bb41">MKL_LIB_PATH_64=/export/software/intel/mkl/lib/intel64/</font></div><div><font color="#77bb41">INTEL_LIB_PATH_64=/export/software/intel/composerxe-2011.4.191/compiler/lib/intel64/</font></div><div><font color="#77bb41">MKL_INC_PATH=/export/software/intel/mkl/include/</font></div><div><font color="#77bb41">SCALAPACK_LIB_PATH=/export/software/scalapack-2.0.2/lib/</font></div><div><font color="#77bb41">SCALAPACK_INC_PATH=/export/software/scalapack-2.0.2/</font></div><div><font color="#77bb41">GENERAL_LIB_PATH=/export/software/lib/</font></div><div><font color="#77bb41"><br></font></div><div><font color="#77bb41">CLINKER=mpicc</font></div><div><font color="#77bb41"><br></font></div><div><font color="#77bb41">PKG_INC = -I${MKL_INC_PATH}</font></div><div><font color="#77bb41">PKG_LIB = -Wl,-rpath,${MKL_LIB_PATH_64} -L${MKL_LIB_PATH_64} -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -Wl,-rpath,${INTEL_LIB_PATH_64} -L${INTEL_LIB_PATH_64} -liomp5 -Wl,-rpath,${SCALAPACK_LIB_PATH} -L${SCALAPACK_LIB_PATH} -lscalapack -llapack -lblas</font></div><div><font color="#77bb41"><br></font></div><div><font color="#77bb41">CFLAGS =</font></div><div><font color="#77bb41">FFLAGS =</font></div><div><font color="#77bb41">CPPFLAGS = ${PKG_INC}</font></div><div><font color="#77bb41">FPPFLAGS =</font></div><div><font color="#77bb41">CLEANFILES =</font></div><div><font color="#77bb41"><br></font></div><div><font color="#77bb41">include ${PETSC_DIR}/conf/variables</font></div><div><font color="#77bb41">include ${PETSC_DIR}/conf/rules</font></div><div><font color="#77bb41"><br></font></div><div><font color="#77bb41">blockCG_1: blockCG_1.o chkopts</font></div><div><font color="#77bb41"><span class="Apple-tab-span" style="white-space:pre"> </span>-${CLINKER} -o blockCG_1 blockCG_1.o ${PKG_LIB} ${PETSC_LIB}</font></div></div><div><font color="#77bb41"><br></font></div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>And I tried to run the program with following command</div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><font color="#77bb41">/opt/openmpi/bin/mpirun -np 3 ./blockCG_1 /home/ppatel/MPI/LF10_2</font></div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>but the routine is not distributing the matrix and give me the result as below.</div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><div><font color="#77bb41">loading matrix from : /home/ppatel/MPI/LF10_2</font></div><div><font color="#77bb41">nRows: 18, nCoefs: 82.</font></div><div><font color="#77bb41">loading matrix from : /home/ppatel/MPI/LF10_2</font></div><div><font color="#77bb41">nRows: 18, nCoefs: 82.</font></div><div><font color="#77bb41">loading matrix from : /home/ppatel/MPI/LF10_2</font></div><div><font color="#77bb41">nRows: 18, nCoefs: 82.</font></div><div><font color="#77bb41">loading matrix from : /home/ppatel/MPI/LF10_2</font></div><div><font color="#77bb41">nRows: 18, nCoefs: 82.</font></div><div><font color="#77bb41">loading matrix from : /home/ppatel/MPI/LF10_2</font></div><div><font color="#77bb41">nRows: 18, nCoefs: 82.</font></div><div><font color="#77bb41">loading matrix from : /home/ppatel/MPI/LF10_2</font></div><div><font color="#77bb41">nRows: 18, nCoefs: 82.</font></div><div><font color="#77bb41">Matrix Object: 1 MPI processes</font></div><div><font color="#77bb41"> type: mpiaij</font></div><div><font color="#77bb41">row 0: (0, 3.53448) (1, -477.155) (2, 1.76724)</font></div><div><font color="#77bb41">row 1: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)</font></div><div><font color="#77bb41">row 2: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)</font></div><div><font color="#77bb41">row 3: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)</font></div><div><font color="#77bb41">row 4: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)</font></div><div><font color="#77bb41">row 5: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)</font></div><div><font color="#77bb41">row 6: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)</font></div><div><font color="#77bb41">row 7: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)</font></div><div><font color="#77bb41">row 8: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)</font></div><div><font color="#77bb41">row 9: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)</font></div><div><font color="#77bb41">row 10: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)</font></div><div><font color="#77bb41">row 11: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)</font></div><div><font color="#77bb41">row 12: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)</font></div><div><font color="#77bb41">row 13: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)</font></div><div><font color="#77bb41">row 14: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)</font></div><div><font color="#77bb41">row 15: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)</font></div><div><font color="#77bb41">row 16: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)</font></div><div><font color="#77bb41">row 17: (15, 477.155) (16, 1.76724) (17, 3.53448)</font></div><div><font color="#77bb41">row 18: (0, 3.53448) (1, -477.155) (2, 1.76724)</font></div><div><font color="#77bb41">row 19: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)</font></div><div><font color="#77bb41">row 20: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)</font></div><div><font color="#77bb41">row 21: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)</font></div><div><font color="#77bb41">row 22: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)</font></div><div><font color="#77bb41">row 23: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)</font></div><div><font color="#77bb41">row 24: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)</font></div><div><font color="#77bb41">row 25: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)</font></div><div><font color="#77bb41">row 26: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)</font></div><div><font color="#77bb41">row 27: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)</font></div><div><font color="#77bb41">row 28: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)</font></div><div><font color="#77bb41">row 29: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)</font></div><div><font color="#77bb41">row 30: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)</font></div><div><font color="#77bb41">row 31: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)</font></div><div><font color="#77bb41">row 32: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)</font></div><div><font color="#77bb41">row 33: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)</font></div><div><font color="#77bb41">row 34: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)</font></div><div><font color="#77bb41">row 35: (15, 477.155) (16, 1.76724) (17, 3.53448)</font></div><div><font color="#77bb41">row 36: (0, 3.53448) (1, -477.155) (2, 1.76724)</font></div><div><font color="#77bb41">row 37: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)</font></div><div><font color="#77bb41">row 38: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)</font></div><div><font color="#77bb41">row 39: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)</font></div><div><font color="#77bb41">row 40: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)</font></div><div><font color="#77bb41">row 41: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)</font></div><div><font color="#77bb41">row 42: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)</font></div><div><font color="#77bb41">row 43: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)</font></div><div><font color="#77bb41">row 44: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)</font></div><div><font color="#77bb41">row 45: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)</font></div><div><font color="#77bb41">row 46: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)</font></div><div><font color="#77bb41">row 47: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)</font></div><div><font color="#77bb41">row 48: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)</font></div><div><font color="#77bb41">row 49: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)</font></div><div><font color="#77bb41">row 50: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)</font></div><div><font color="#77bb41">row 51: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)</font></div><div><font color="#77bb41">row 52: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)</font></div><div><font color="#77bb41">row 53: (15, 477.155) (16, 1.76724) (17, 3.53448)</font></div><div><font color="#77bb41">row 54: (0, 3.53448) (1, -477.155) (2, 1.76724)</font></div><div><font color="#77bb41">row 55: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)</font></div><div><font color="#77bb41">row 56: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)</font></div><div><font color="#77bb41">row 57: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)</font></div><div><font color="#77bb41">row 58: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)</font></div><div><font color="#77bb41">row 59: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)</font></div><div><font color="#77bb41">row 60: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)</font></div><div><font color="#77bb41">row 61: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)</font></div><div><font color="#77bb41">row 62: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)</font></div><div><font color="#77bb41">row 63: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)</font></div><div><font color="#77bb41">row 64: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)</font></div><div><font color="#77bb41">row 65: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)</font></div><div><font color="#77bb41">row 66: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)</font></div><div><font color="#77bb41">row 67: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)</font></div><div><font color="#77bb41">row 68: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)</font></div><div><font color="#77bb41">row 69: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)</font></div><div><font color="#77bb41">row 70: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)</font></div><div><font color="#77bb41">row 71: (15, 477.155) (16, 1.76724) (17, 3.53448)</font></div><div><font color="#77bb41">row 72: (0, 3.53448) (1, -477.155) (2, 1.76724)</font></div><div><font color="#77bb41">row 73: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)</font></div><div><font color="#77bb41">row 74: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)</font></div><div><font color="#77bb41">row 75: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)</font></div><div><font color="#77bb41">row 76: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)</font></div><div><font color="#77bb41">row 77: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)</font></div><div><font color="#77bb41">row 78: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)</font></div><div><font color="#77bb41">row 79: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)</font></div><div><font color="#77bb41">row 80: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)</font></div><div><font color="#77bb41">row 81: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)</font></div><div><font color="#77bb41">row 82: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)</font></div><div><font color="#77bb41">row 83: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)</font></div><div><font color="#77bb41">row 84: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)</font></div><div><font color="#77bb41">row 85: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)</font></div><div><font color="#77bb41">row 86: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)</font></div><div><font color="#77bb41">row 87: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)</font></div><div><font color="#77bb41">row 88: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)</font></div><div><font color="#77bb41">row 89: (15, 477.155) (16, 1.76724) (17, 3.53448)</font></div><div><font color="#77bb41">row 90: (0, 3.53448) (1, -477.155) (2, 1.76724)</font></div><div><font color="#77bb41">row 91: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)</font></div><div><font color="#77bb41">row 92: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)</font></div><div><font color="#77bb41">row 93: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6, 477.155)</font></div><div><font color="#77bb41">row 94: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6, 1.76724)</font></div><div><font color="#77bb41">row 95: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8, 477.155)</font></div><div><font color="#77bb41">row 96: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8, 1.76724)</font></div><div><font color="#77bb41">row 97: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10, 477.155)</font></div><div><font color="#77bb41">row 98: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10, 1.76724)</font></div><div><font color="#77bb41">row 99: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12, 477.155)</font></div><div><font color="#77bb41">row 100: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12, 1.76724)</font></div><div><font color="#77bb41">row 101: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14, 477.155)</font></div><div><font color="#77bb41">row 102: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14, 1.76724)</font></div><div><font color="#77bb41">row 103: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16, 477.155)</font></div><div><font color="#77bb41">row 104: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16, 1.76724)</font></div><div><font color="#77bb41">row 105: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)</font></div><div><font color="#77bb41">row 106: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)</font></div><div><font color="#77bb41">row 107: (15, 477.155) (16, 1.76724) (17, 3.53448)</font></div></div><div><br></div><div>------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div>that is creating a matrix with rows repeated 6 times instead of distributed over 6 process.</div><div><br></div><div>Can you please tell me where I am doing wrong??</div><div><br></div><div>Thanks for your time</div><div><br><div apple-content-edited="true">
<div>Priyank Patel</div><div><a href="mailto:ppate024@odu.edu">ppate024@odu.edu</a></div><div><br></div><div><br></div><br class="Apple-interchange-newline">
</div>
<pre width="80"><br></pre><div><br></div></div></body></html>