<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Thank you so much, Dr. Knepley.<br>
    <br>
    Alan<br>
    <br>
    On 4/19/2012 9:51 AM, Matthew Knepley wrote:
    <blockquote
cite="mid:CAMYG4GnuYfynEgPb1Zt7ZHAQ1OO8CO6HVNBEcAbaL2zF3O1Pug@mail.gmail.com"
      type="cite">On Thu, Apr 19, 2012 at 10:49 AM, Zhenglun (Alan) Wei
      <span dir="ltr">&lt;<a moz-do-not-send="true"
          href="mailto:zhenglun.wei@gmail.com">zhenglun.wei@gmail.com</a>&gt;</span>
      wrote:<br>
      <div class="gmail_quote">
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div bgcolor="#FFFFFF" text="#000000"> Dear Dr. Knepley, <br>
            &nbsp;&nbsp;&nbsp;&nbsp; It is very nice to hear that. I will read the manual.
            Do we have any examples showing its functions?<br>
          </div>
        </blockquote>
        <div><br>
        </div>
        <div>Lots of examples. Start with SNES ex5 and ex19.</div>
        <div><br>
        </div>
        <div>&nbsp; &nbsp;Matt</div>
        <div>&nbsp;</div>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div bgcolor="#FFFFFF" text="#000000"> thank you so much,<br>
            Alan<br>
            On 4/19/2012 9:35 AM, Matthew Knepley wrote:
            <blockquote type="cite">On Thu, Apr 19, 2012 at 10:18 AM,
              Zhenglun (Alan) Wei <span dir="ltr">&lt;<a
                  moz-do-not-send="true"
                  href="mailto:zhenglun.wei@gmail.com" target="_blank">zhenglun.wei@gmail.com</a>&gt;</span>
              wrote:<br>
              <div class="gmail_quote">
                <blockquote class="gmail_quote" style="margin:0 0 0
                  .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  <div bgcolor="#FFFFFF" text="#000000">
                    <div>
                      <div>
                        <div>"</div>
                        <div>&nbsp; &nbsp; TESTVAR ***a, ***b, ***c;</div>
                        <div>&nbsp; &nbsp; TESTVAR **aa, **bb, **cc;&nbsp;</div>
                        <div>&nbsp; &nbsp; TESTVAR *arraya, *arrayb, *arrayc;</div>
                      </div>
                      <div><br>
                      </div>
                      <div>&nbsp; &nbsp; arraya = (TESTVAR*)
                        calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));</div>
                      <div>&nbsp; &nbsp; arrayb = (TESTVAR*)
                        calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));</div>
                      <div>&nbsp; &nbsp; arrayc = (TESTVAR*)
                        calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));</div>
                      <div><br>
                      </div>
                      <div>&nbsp; &nbsp; aa =(TESTVAR**) calloc(SIZE*SIZE,
                        sizeof(TESTVAR*));</div>
                      <div>&nbsp; &nbsp; bb =(TESTVAR**) calloc(SIZE*SIZE,
                        sizeof(TESTVAR*));</div>
                      <div>&nbsp; &nbsp; cc =(TESTVAR**) calloc(SIZE*SIZE,
                        sizeof(TESTVAR*));</div>
                      <div>&nbsp; &nbsp;&nbsp;</div>
                      <div>&nbsp; &nbsp; for(i = 0; i &lt; SIZE*SIZE; i++) {</div>
                      <div>&nbsp; &nbsp; &nbsp; aa[i] = &amp;arraya[i*SIZE];</div>
                      <div>&nbsp; &nbsp; &nbsp; bb[i] = &amp;arrayb[i*SIZE];</div>
                      <div>&nbsp; &nbsp; &nbsp; cc[i] = &amp;arrayc[i*SIZE];&nbsp;</div>
                      <div>&nbsp; &nbsp; }</div>
                      <div><br>
                      </div>
                      <div>&nbsp; &nbsp; a =(TESTVAR***) calloc(SIZE*SIZE,
                        sizeof(TESTVAR**));</div>
                      <div>&nbsp; &nbsp; b =(TESTVAR***) calloc(SIZE*SIZE,
                        sizeof(TESTVAR**));</div>
                      <div>&nbsp; &nbsp; c =(TESTVAR***) calloc(SIZE*SIZE,
                        sizeof(TESTVAR**));</div>
                      <div>&nbsp; &nbsp;&nbsp;</div>
                      <div>&nbsp; &nbsp; for(i = 0; i &lt; SIZE; i++) {</div>
                      <div>&nbsp; &nbsp; &nbsp; a[i] = &amp;aa[i*SIZE];</div>
                      <div>&nbsp; &nbsp; &nbsp; b[i] = &amp;bb[i*SIZE];</div>
                      <div>&nbsp; &nbsp; &nbsp; c[i] = &amp;cc[i*SIZE];</div>
                      <div>&nbsp; &nbsp; }</div>
                      <div>"</div>
                      <div>&nbsp; It works. However, I wonder if there is any
                        other good ideas for 3D problem other than this
                        kinda of 'two-layer' approach.</div>
                    </div>
                    <div><br>
                    </div>
                    <div><b><u>What is the reason for not using DMDA?</u><br>
                      </b>In 2D, I established a 2D array for data
                      communication between nodes by using MPI derived
                      data type. It allows me to easily communicate both
                      contiguous (i.e. MPI_TYPE_CONTIGUOUS) and
                      non-contiguous (i.e. MPI_TYPE_VECTOR) data. That
                      is why I use this similar approach in 3D, though
                      an additional data type, i.e. MPI_TYPE_INDEXED,
                      need to be used. Does DMDA have those type of
                      function or derived data type?</div>
                  </div>
                </blockquote>
                <div><br>
                </div>
                <div>It definitely does communication between the local
                  pieces. Do you want something else?</div>
                <div>&nbsp;</div>
                <blockquote class="gmail_quote" style="margin:0 0 0
                  .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  <div bgcolor="#FFFFFF" text="#000000">
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div>
                        <div>"2, I have a little question on PETSc about
                          3D processor ordering. Does PETSc have any
                          function giving me the nodes/rank number of
                          neighboring nodes/ranks? Are those
                          'Application Ordering' functions applicable
                          for my case?"</div>
                      </div>
                    </blockquote>
                    <div><br>
                    </div>
                    <div><u><b>What do you mean by neighboring? If it is
                          jsut stencil neighbors, then use a local
                          vector.</b></u></div>
                    <div>When I send and receive data with MPI_Send and
                      MPI_RECV, I need provide the 'destination' (in
                      MPI_Send refer to'<a moz-do-not-send="true"
href="http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html"
                        target="_blank">http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html</a>')


                      and 'source' (in MPI_RECV refer to'<a
                        moz-do-not-send="true"
href="http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html"
                        target="_blank">http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html</a>').


                      In a 2D problem with Cartesian grid, 4 processes
                      divide the whole domain to 4 sub-domain. <br>
                      ---------------------------- <br>
                      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br>
                      ----------------------------<br>
                      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br>
                      ---------------------------<br>
                      &nbsp;Then, for node 1, the neighboring nodes are '0'
                      and '3', which '0' is the left node and '3' is the
                      top node. I wonder if PETSc has any function that
                      I can call to obtain those neighboring nodes so
                      that I do not need to construct my function. <br>
                    </div>
                  </div>
                </blockquote>
                <div><br>
                </div>
                <div>Yes, it looks like you should just use a DMDA. See
                  the manual section.</div>
                <div><br>
                </div>
                <div>&nbsp; &nbsp;Matt</div>
                <div>&nbsp;</div>
                <blockquote class="gmail_quote" style="margin:0 0 0
                  .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  <div bgcolor="#FFFFFF" text="#000000">
                    <div> I'm sorry for confusing you. <br>
                      <br>
                      thanks in advance,<br>
                      Alan <br>
                    </div>
                    <br>
                    On 4/19/2012 4:52 AM, Matthew Knepley wrote:
                    <blockquote type="cite">On Wed, Apr 18, 2012 at 3:52
                      PM, Alan Wei <span dir="ltr">&lt;<a
                          moz-do-not-send="true"
                          href="mailto:zhenglun.wei@gmail.com"
                          target="_blank">zhenglun.wei@gmail.com</a>&gt;</span>
                      wrote:<br>
                      <div class="gmail_quote">
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex"> Dear all,
                          <div>&nbsp; &nbsp; I hope you're having a nice day. I
                            have a further question on this issue in 3D.</div>
                          <div>1, Following the idea of Dr. Brown and
                            Dr. Knepley, I finished a 2D test, which
                            works very fine. Here, I did it in 3D by</div>
                          <div>
                            <div>
                              <div>"</div>
                              <div>&nbsp; &nbsp; TESTVAR ***a, ***b, ***c;</div>
                              <div>&nbsp; &nbsp; TESTVAR **aa, **bb, **cc;&nbsp;</div>
                              <div>&nbsp; &nbsp; TESTVAR *arraya, *arrayb,
                                *arrayc;</div>
                            </div>
                            <div><br>
                            </div>
                            <div>&nbsp; &nbsp; arraya = (TESTVAR*)
                              calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));</div>
                            <div>&nbsp; &nbsp; arrayb = (TESTVAR*)
                              calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));</div>
                            <div>&nbsp; &nbsp; arrayc = (TESTVAR*)
                              calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));</div>
                            <div><br>
                            </div>
                            <div>&nbsp; &nbsp; aa =(TESTVAR**) calloc(SIZE*SIZE,
                              sizeof(TESTVAR*));</div>
                            <div>&nbsp; &nbsp; bb =(TESTVAR**) calloc(SIZE*SIZE,
                              sizeof(TESTVAR*));</div>
                            <div>&nbsp; &nbsp; cc =(TESTVAR**) calloc(SIZE*SIZE,
                              sizeof(TESTVAR*));</div>
                            <div>&nbsp; &nbsp;&nbsp;</div>
                            <div>&nbsp; &nbsp; for(i = 0; i &lt; SIZE*SIZE; i++) {</div>
                            <div>&nbsp; &nbsp; &nbsp; aa[i] = &amp;arraya[i*SIZE];</div>
                            <div>&nbsp; &nbsp; &nbsp; bb[i] = &amp;arrayb[i*SIZE];</div>
                            <div>&nbsp; &nbsp; &nbsp; cc[i] = &amp;arrayc[i*SIZE];&nbsp;</div>
                            <div>&nbsp; &nbsp; }</div>
                            <div><br>
                            </div>
                            <div>&nbsp; &nbsp; a =(TESTVAR***) calloc(SIZE*SIZE,
                              sizeof(TESTVAR**));</div>
                            <div>&nbsp; &nbsp; b =(TESTVAR***) calloc(SIZE*SIZE,
                              sizeof(TESTVAR**));</div>
                            <div>&nbsp; &nbsp; c =(TESTVAR***) calloc(SIZE*SIZE,
                              sizeof(TESTVAR**));</div>
                            <div>&nbsp; &nbsp;&nbsp;</div>
                            <div>&nbsp; &nbsp; for(i = 0; i &lt; SIZE; i++) {</div>
                            <div>&nbsp; &nbsp; &nbsp; a[i] = &amp;aa[i*SIZE];</div>
                            <div>&nbsp; &nbsp; &nbsp; b[i] = &amp;bb[i*SIZE];</div>
                            <div>&nbsp; &nbsp; &nbsp; c[i] = &amp;cc[i*SIZE];</div>
                            <div>&nbsp; &nbsp; }</div>
                            <div>"</div>
                            <div>&nbsp; It works. However, I wonder if there
                              is any other good ideas for 3D problem
                              other than this kinda of 'two-layer'
                              approach.</div>
                          </div>
                        </blockquote>
                        <div><br>
                        </div>
                        <div>What is the reason for not using DMDA?</div>
                        <div>&nbsp;</div>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div>
                            <div>2, I have a little question on PETSc
                              about 3D processor ordering. Does PETSc
                              have any function giving me the nodes/rank
                              number of neighboring nodes/ranks? Are
                              those 'Application Ordering' functions
                              applicable for my case?</div>
                          </div>
                        </blockquote>
                        <div><br>
                        </div>
                        <div>What do you mean by neighboring? If it is
                          jsut stencil neighbors, then use a local
                          vector.</div>
                        <div><br>
                        </div>
                        <div>&nbsp; &nbsp;Matt</div>
                        <div>&nbsp;</div>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div>
                            <div>thanks,</div>
                            <div>Alan</div>
                            <br>
                            <div class="gmail_quote">On Fri, Apr 13,
                              2012 at 5:41 PM, Jed Brown <span
                                dir="ltr">&lt;<a moz-do-not-send="true"
                                  href="mailto:jedbrown@mcs.anl.gov"
                                  target="_blank">jedbrown@mcs.anl.gov</a>&gt;</span>
                              wrote:<br>
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div>
                                  <div class="gmail_quote">On Fri, Apr
                                    13, 2012 at 17:38, Zhenglun (Alan)
                                    Wei <span dir="ltr">&lt;<a
                                        moz-do-not-send="true"
                                        href="mailto:zhenglun.wei@gmail.com"
                                        target="_blank">zhenglun.wei@gmail.com</a>&gt;</span>
                                    wrote:<br>
                                    <blockquote class="gmail_quote"
                                      style="margin:0 0 0
                                      .8ex;border-left:1px #ccc
                                      solid;padding-left:1ex"> &nbsp;&nbsp;&nbsp; I
                                      have a final question on it. Is it
                                      taken a lot of memory for doing
                                      this? As I understand, pointers
                                      won't occupy many memories and it
                                      works like an alias. It will not,
                                      to my limit knowledge, take much
                                      extra memory by doing this. </blockquote>
                                  </div>
                                  <br>
                                </div>
                                <div>A pointer takes about as much space
                                  as a floating point value, so that
                                  array of pointers costs about 1*N
                                  compared to the N*N matrix.</div>
                              </blockquote>
                            </div>
                            <br>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                      <br clear="all">
                      <span class="HOEnZb"><font color="#888888"> <span><font
                              color="#888888">
                              <div><br>
                              </div>
                              -- <br>
                              What most experimenters take for granted
                              before they begin their experiments is
                              infinitely more interesting than any
                              results to which their experiments lead.<br>
                              -- Norbert Wiener<br>
                            </font></span></font></span></blockquote>
                    <span class="HOEnZb"><font color="#888888"> <br>
                      </font></span></div>
                  <span class="HOEnZb"><font color="#888888"> </font></span></blockquote>
                <span class="HOEnZb"><font color="#888888"> </font></span></div>
              <span class="HOEnZb"><font color="#888888"> <br>
                  <br clear="all">
                  <div><br>
                  </div>
                  -- <br>
                  What most experimenters take for granted before they
                  begin their experiments is infinitely more interesting
                  than any results to which their experiments lead.<br>
                  -- Norbert Wiener<br>
                </font></span></blockquote>
            <br>
          </div>
        </blockquote>
      </div>
      <br>
      <br clear="all">
      <div><br>
      </div>
      -- <br>
      What most experimenters take for granted before they begin their
      experiments is infinitely more interesting than any results to
      which their experiments lead.<br>
      -- Norbert Wiener<br>
    </blockquote>
    <br>
  </body>
</html>