<div dir="ltr"><div>Hello, I am working on converting a serial code to use Petsc. To do this it starts in serial with the old code, creates some Petsc matrices/vectors and does the most critical calculations in Petsc, and then returns to serial to return to the old code. After this I was planning to expand the area of the code which uses Petsc, but starting with a small chunk seemed appropriate. To return to the serial code, I need to have every process receive a full copy of a vector. I believe VecScatters are intended for this, but I cannot get it to work. I have produced a minimal example which starts with a serial double array, creates a Vec which is successfully distributed, and then tries to use a VecScatter to aggregate the vector from each process. After the VecScatter step, each process only has the parts of the vector which were available to the process in the distributed vector. If you could please show how to aggregate the vector, it would be greatly appreciated.</div><div><br></div><div>Use with: mpiexec -n 2 ./test:</div><div><br></div><div><div> #include <petscvec.h></div><div><br></div><div> double primes[] = {2,3,5,7,11,13,17};</div><div> int nprimes = 7;</div><div><br></div><div> int main(int argc,char **argv)</div><div> {</div><div><span style="white-space:pre"> </span> PetscInitialize(&argc,&argv, NULL,NULL);</div><div><span style="white-space:pre"> </span></div><div> <span style="white-space:pre"> </span>MPI_Comm comm=MPI_COMM_WORLD;</div><div><span style="white-space:pre"> </span> Vec xpar,xseq;</div><div><span style="white-space:pre"> </span> PetscInt low,high;</div><div><span style="white-space:pre"> </span> IS index_set_global, index_set_local;</div><div> <span style="white-space:pre"> </span>const PetscInt *indices;</div><div><span style="white-space:pre"> </span> VecScatter vc;</div><div><span style="white-space:pre"> </span> PetscErrorCode ierr;</div><div><span style="white-space:pre"> </span></div><div><span style="white-space:pre"> </span> //Set up parallel vector</div><div><span style="white-space:pre"> </span> ierr = VecCreateMPI(comm, PETSC_DETERMINE, nprimes, &xpar); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecGetOwnershipRange(xpar, &low, &high); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = ISCreateStride(comm, high - low, low, 1, &index_set_global); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = ISGetIndices(index_set_global, &indices); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = ISView(index_set_global, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecSetValues(xpar, high - low, indices, primes + low, INSERT_VALUES);CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecAssemblyBegin(xpar); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecAssemblyEnd(xpar); CHKERRQ(ierr);</div><div> <span style="white-space:pre"> </span>ierr = VecView(xpar, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span></div><div><span style="white-space:pre"> </span> //Scatter parallel vector so all processes have full vector</div><div><span style="white-space:pre"> </span> ierr = VecCreateSeq(PETSC_COMM_SELF, nprimes, &xseq); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> //ierr = VecCreateMPI(comm, high - low, nprimes, &xseq); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = ISCreateStride(comm, high - low, 0, 1, &index_set_local); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecScatterCreate(xpar, index_set_local, xseq, index_set_global, &vc); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecScatterBegin(vc, xpar, xseq, ADD_VALUES, SCATTER_FORWARD); CHKERRQ(ierr);</div><div> ierr = VecScatterEnd(vc, xpar, xseq, ADD_VALUES, SCATTER_FORWARD); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = PetscPrintf(PETSC_COMM_SELF, "\nPrinting out scattered vector\n"); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> ierr = VecView(xseq, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr);</div><div><span style="white-space:pre"> </span> </div><div><span style="white-space:pre"> </span> PetscFinalize();</div><div> }</div></div><div><br></div><div><br></div><div>Output:</div><div><br></div><div><div> IS Object: 2 MPI processes</div><div> type: stride</div><div> [0] Index set is permutation</div><div> [0] Number of indices in (stride) set 4</div><div> [0] 0 0</div><div> [0] 1 1</div><div> [0] 2 2</div><div> [0] 3 3</div><div> [1] Number of indices in (stride) set 3</div><div> [1] 0 4</div><div> [1] 1 5</div><div> [1] 2 6</div><div> Vec Object: 2 MPI processes</div><div> type: mpi</div><div> Process [0]</div><div> 2.</div><div> 3.</div><div> 5.</div><div> 7.</div><div> Process [1]</div><div> 11.</div><div> 13.</div><div> 17.</div><div><br></div><div> Printing out scattered vector</div><div><br></div><div> Printing out scattered vector</div><div> Vec Object: 1 MPI processes</div><div> type: seq</div><div> 2.</div><div> 3.</div><div> 5.</div><div> 7.</div><div> 0.</div><div> 0.</div><div> 0.</div></div><div><br></div><div><br></div><div><br></div><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div dir="ltr">Anthony Ruth</div><div dir="ltr"><div><div><div>Condensed Matter Theory</div><div>University of Notre Dame</div></div></div></div></div></div></div>
</div>