The completed code is:<div><br><div><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span style="color: #aa0d91">void</span> PETSc(<span style="color: #aa0d91">double</span> mat[],<span style="color: #aa0d91">double</span> vec[],<span style="color: #aa0d91">double</span> sol[],<span style="color: #aa0d91">long</span> N,<span style="color: #aa0d91">double</span> tol) {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>Vec x,b;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>Mat A;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>KSP ksp;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>PC pc;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>PetscInt i,j,col[N];</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>PetscScalar value[N];</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>PetscScalar val;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecCreate(PETSC_COMM_WORLD,&x);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecSetSizes(x,PETSC_DECIDE,N);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecSetFromOptions(x);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecDuplicate(x,&b);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>MatCreate(PETSC_COMM_WORLD,&A);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>MatSetFromOptions(A);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span><span style="color: #aa0d91">for</span> (i=<span style="color: #1c00cf">0</span>; i<N; i++) {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span><span style="color: #aa0d91">for</span> (j=<span style="color: #1c00cf">0</span>;j<N;j++) {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>value[j]=mat[i*N+j];</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>col[j]=j;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>}</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>MatSetValues(A,<span style="color: #1c00cf">1</span>,&i,N,col,value,INSERT_VALUES);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>}</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"> </p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span><span style="color: #aa0d91">for</span> (i=<span style="color: #1c00cf">0</span>; i<N; i++) {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>val=vec[i];</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecSetValues(b,<span style="color: #1c00cf">1</span>,&i,&val,INSERT_VALUES);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>val=sol[i];</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecSetValues(x,<span style="color: #1c00cf">1</span>,&i,&val,INSERT_VALUES);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>}</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecAssemblyBegin(b);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>VecAssemblyEnd(b);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>KSPCreate(PETSC_COMM_WORLD,&ksp);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><span class="Apple-tab-span" style="white-space:pre"> </span>KSPGetPC(ksp,&pc);</p><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"></p><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco">
KSPSetType(ksp, KSPGMRES);</p><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> PCSetType(pc, PCBJACOBI);</p><p></p><div><span class="Apple-style-span" style="font-family: Monaco; font-size: 10px;"><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco">
KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> KSPSetFromOptions(ksp);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> KSPSolve(ksp,b,x);</p><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"><br></p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> <span style="color: #aa0d91">for</span> (i=<span style="color: #1c00cf">0</span>;i<N;i++) {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> VecGetValues(x,<span style="color: #1c00cf">1</span>,&i,&sol[i]);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> }</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco; min-height: 14.0px"> </p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> VecDestroy(x);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> VecDestroy(b);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> MatDestroy(A);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> KSPDestroy(ksp);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco"> PetscFinalize();</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Monaco">}</p><div><br></div></span></div>Once this function is call in a loop, it reports error.</div><div><br><div class="gmail_quote">On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng <span dir="ltr"><<a href="mailto:yfeng1@tigers.lsu.edu">yfeng1@tigers.lsu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">I put PETSc solver in one function and use another function to call that.<div>This problem only appears when I put the function with PETSc solver in a loop,</div>
<div>it works well at first step, and reports error "<span style="border-collapse:collapse;white-space:pre">An error occurred in MPI_Comm_rank after MPI was finalized<span style="border-collapse:separate;white-space:normal">"</span></span></div>
<div>at second time. The program is designed to support only one processor like:</div><div><br></div><div><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>Vec x,b;</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>Mat A;</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>KSP ksp;</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>PC pc;</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>PetscInt i,j,col[N];</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>PetscScalar value[N];</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>PetscScalar val;</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco;min-height:14.0px"><br></p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>VecCreate(PETSC_COMM_WORLD,&x);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>VecSetSizes(x,PETSC_DECIDE,N);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>VecSetFromOptions(x);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>VecDuplicate(x,&b);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>MatCreate(PETSC_COMM_WORLD,&A);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>MatSetFromOptions(A);</p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> ........</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> ........</p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"></p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>KSPCreate(PETSC_COMM_WORLD,&ksp);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><span style="white-space:pre"> </span>KSPGetPC(ksp,&pc);</p><p></p><div><span style="font-family:Monaco;font-size:10px"> ................</span></div>
<div><span style="font-family:Monaco;font-size:10px"> ...............</span></div><div><span style="font-family:Monaco;font-size:10px"><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">
KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> KSPSetFromOptions(ksp);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> KSPSolve(ksp,b,x);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco;color:#007400"><span style="color:rgb(0, 0, 0)"><br></span></p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> VecDestroy(x);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> VecDestroy(b);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> MatDestroy(A);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> KSPDestroy(ksp);</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"> PetscFinalize();</p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"><br></p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">
Any one has ideal about this? </p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">The detailed error description is:</p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco"></p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">
An error occurred in MPI_Comm_rank</p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">*** after MPI was finalized</p><p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">*** MPI_ERRORS_ARE_FATAL (goodbye)</p>
<p style="margin:0.0px 0.0px 0.0px 0.0px;font:10.0px Monaco">Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!</p><div><br></div><div>Thank you so much in advance!</div>
<div><br></div><p></p></span></div></div>
</blockquote></div><br></div></div>