<div>Thank you soooooo much, Dr. Brown. It fixed my problem. I really appreciate your time and help. :)</div><div> </div><div>thank you again,</div><div>Alan</div><div><br><br> </div><div class="gmail_quote">On Fri, Sep 7, 2012 at 9:20 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div> if(!localBC.tBC) {</div><div> MPI_Isend(&_TestV[0][_Index.tcEnd-1][0], 1, columntype, NbrRank.t, SendTag.T1st, PETSC_COMM_WORLD, &request);</div>
<div> MPI_Isend(&_TestV[0][_Index.tcEnd][0], 1, columntype, NbrRank.t, SendTag.T2nd, PETSC_COMM_WORLD, &request);</div>
<div> }</div><div><br></div><div> if(!localBC.bBC) {</div><div> MPI_Isend(&_TestV[0][_Index.bcStr+1][0], 1, columntype, NbrRank.b, SendTag.B1st, PETSC_COMM_WORLD, &request);</div><div> MPI_Isend(&_TestV[0][_Index.bcStr][0], 1, columntype, NbrRank.b, SendTag.B2nd, PETSC_COMM_WORLD, &request);</div>
<div> }</div><div><br></div><div> MPI_Barrier(PETSC_COMM_WORLD);</div><div> printf("Rank = %d finished sending!!!\n", rank);</div><div><br></div><div> if(!localBC.tBC) {</div><div> MPI_Irecv(&_TestV[0][_Index.tbStr+1][0], 1, columntype, NbrRank.t, RecvTag.T2nd, PETSC_COMM_WORLD, &request);</div>
<div> MPI_Irecv(&_TestV[0][_Index.tbStr][0], 1, columntype, NbrRank.t, RecvTag.T1st, PETSC_COMM_WORLD, &request);</div><div> }</div><div><br></div><div> if(!localBC.bBC) {</div><div> MPI_Irecv(&_TestV[0][_Index.bbEnd-1][0], 1, columntype, NbrRank.b, RecvTag.B2nd, PETSC_COMM_WORLD, &request);</div>
<div> MPI_Irecv(&_TestV[0][_Index.bbEnd][0], 1, columntype, NbrRank.b, RecvTag.B1st, PETSC_COMM_WORLD, &request);</div><div> }</div><div><br></div><div> MPI_Wait(&request, &status);</div><div><br>
</div><div><br></div><div>You are creating far more requests than you are waiting on. You need to keep track of <b>every</b> request and eventually wait on all of them.</div><div><br></div><div>It is generally better for performance to post the receives first, then post the sends, then MPI_Waitall() on all the requests.</div>
<div class="HOEnZb"><div class="h5">
<br><div class="gmail_quote">On Fri, Sep 7, 2012 at 4:50 PM, Zhenglun (Alan) Wei <span dir="ltr"><<a href="mailto:zhenglun.wei@gmail.com" target="_blank">zhenglun.wei@gmail.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div text="#000000" bgcolor="#FFFFFF">
<div>Dear folks, <br>
I did more tests, since I want to make sure where I'm wrong.
<br>
As Dr. Smith suggested, I tested my code using OpenMPI and
MPICH. Both of them have the memory accumulation problem.
Therefore, I suppose there is a bug in my code. I went into the
code, and changed the non-blocking MPI communication to blocking
one. The memory accumulation problem is just gone by itself.
However, I have to change it back since the blocking MPI
communication does not allow me to do massive data communication.
Now, I'm searching for related topics on non-blocking MPI
communication. <br>
Here I cut off those unrelated part of my code and attach the
communication part here. Could anyone help me to briefly check if
there is any obvious mistake I made in the program? After unzip
the file, './AlanRun' will execute the program. <br>
<br>
I really appreciate your help :)<span><font color="#888888"><br>
Alan</font></span><div><div><br>
<br>
<br>
On 9/6/2012 5:56 PM, Jed Brown wrote:<br>
</div></div></div><div><div>
<blockquote type="cite">
<p>Numeric data that the solver sees should be stored in Vecs. You
can put other scalars in Vecs if you like.</p>
<div class="gmail_quote">On Sep 6, 2012 5:48 PM, "Zhenglun (Alan)
Wei" <<a href="mailto:zhenglun.wei@gmail.com" target="_blank">zhenglun.wei@gmail.com</a>>
wrote:<br type="attribution">
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div text="#000000" bgcolor="#FFFFFF">
<div>Dear Dr. Brown,<br>
I'm not quite familiar with VecScatter. I just read
its explanation; it seems requires that my data is stored
as a form of vectors (is it the vector in PETSc?).
However, my data are stored as arrays in C program. <br>
Is that any problem in MPI or it is likely a problem
of my code?<br>
<br>
thanks,<br>
Alan <br>
On 9/6/2012 5:44 PM, Jed Brown wrote:<br>
</div>
<blockquote type="cite">
<p>Are you familiar with VecScatter?</p>
<div class="gmail_quote">On Sep 6, 2012 5:38 PM, "Zhenglun
(Alan) Wei" <<a href="mailto:zhenglun.wei@gmail.com" target="_blank">zhenglun.wei@gmail.com</a>>
wrote:<br type="attribution">
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
Dear All,<br>
I hope you're having a nice day.<br>
I met a memory problem for MPI data
communication. I guess here is a good place to ask
this question since you guys are experts and may
experienced the same problem before.<br>
I used the MPI derived data type
(MPI_Type_contiguous, MPI_Type_vector and
MPI_Type_indexed) to communicate data for a simulation
of 3D problem. The communication is fine, as I checked
every single data it sent and received. However, the
problem is that the memory keeps increasing while
communication. Therefore, I tested each of these three
types. MPI_Type_contiguous does not have any problem;
while MPI_Type_vector and MPI_Type_indexed have
problem of memory accumulation. I tried to use
MPI_Type_free, but it does not help. Have anyone
experienced this problem before?<br>
Would this be related to the non-blocking MPI
communication (MPI_Isend and MPI_Irecv). I have to use
this non-blocking communication since the blocking
communication is extremely slow when it has a lot of
data involved in the communication.<br>
Is there any alternative way in PETSc that could
do the similar work of MPI derived types?<br>
<br>
thanks,<br>
Alan<br>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br>
</div></div></blockquote></div><br>