<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Yeeee-ha!&nbsp; <br>
    <br>
    VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was
    missing.<br>
    <br>
    Thank you Mohammad, thank you all who helped me today!<br>
    <br>
    <br>
    &nbsp;&nbsp;&nbsp; Cheers<br>
    <br>
    <br>
    &nbsp;&nbsp;&nbsp; Bojan<br>
    <br>
    <br>
    <br>
    On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
    <blockquote
cite="mid:CAKsPKwkGzZ13_Jwn=L3=9BPgO6T2j0fjpeO7brXwh2tS9pZLEg@mail.gmail.com"
      type="cite">
      <div dir="ltr">You also need calls to
        VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update
        the ghost values if you change them in the global
        representation. See Petsc Manual 3.2 pp 55-56
        <div><br>
        </div>
        <div>Mohammad<br>
          <br>
          <div class="gmail_quote">On Thu, Feb 23, 2012 at 3:18 PM,
            Bojan Niceno <span dir="ltr">&lt;<a moz-do-not-send="true"
                href="mailto:bojan.niceno@psi.ch">bojan.niceno@psi.ch</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <div class="im"> On 2/23/2012 11:49 PM, Mohammad
                  Mirzadeh wrote:
                  <blockquote type="cite">
                    <div dir="ltr">based on,
                      <div><br>
                      </div>
                      <div><span>VecCreateGhost(PETSC_COMM_</span><span>WORLD,
                          n, PETSC_DECIDE, nghost, &amp;ghosts[0],
                          &amp;x);</span></div>
                      <div><font color="#222222" face="monospace"
                          size="1"><br>
                        </font></div>
                      <div><font color="#222222" face="monospace"
                          size="1"><br>
                        </font></div>
                      <div>it seems to me that x is your actual ghosted
                        vector. If this is true, then you need to get
                        its "local" form via&nbsp;<a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm"
                          target="_blank">VecGhostGetLocalForm</a>().
                        Once you have done, you should be able to access
                        the ghosted nodes. Are you calling this function
                        anywhere?</div>
                    </div>
                  </blockquote>
                  <br>
                </div>
                I tried that before.&nbsp; I did:<br>
                <br>
                Vec lx;<br>
                VecGhostGetLocalForm(x, &amp;lx)<br>
                <br>
                then I copied "lx" to my variable, like<br>
                <br>
                for(Int i=1; i&lt;N; i++)&nbsp;&nbsp;&nbsp; /* N includes buffer cells
                */<br>
                &nbsp; unk[i] = lx[i]<br>
                <br>
                but ghost values were also zero.&nbsp;&nbsp; I am thinking that
                PETSc somehow clears the ghost values after a call to
                KSP.&nbsp; Is it the case?<br>
                <br>
                <br>
                &nbsp;&nbsp;&nbsp; Kind regards,<br>
                <br>
                <br>
                &nbsp;&nbsp;&nbsp; Bojan
                <div>
                  <div class="h5"><br>
                    <br>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div><br>
                          <div class="gmail_quote">On Thu, Feb 23, 2012
                            at 2:32 PM, Bojan Niceno <span dir="ltr">&lt;<a
                                moz-do-not-send="true"
                                href="mailto:bojan.niceno@psi.ch"
                                target="_blank">bojan.niceno@psi.ch</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote"
                              style="margin:0 0 0 .8ex;border-left:1px
                              #ccc solid;padding-left:1ex">
                              <div bgcolor="#FFFFFF" text="#000000">
                                Dear Mohammad,<br>
                                <br>
                                <br>
                                it doesn't help me, or I did not
                                understand your explanation.<br>
                                <br>
                                If I do this:<br>
                                <br>
                                <tt>&nbsp; /* copy internal values (THIS
                                  WORKS, BUT COPIES NO BUFFER VALUES) */
                                  <div><br>
                                    &nbsp; for(Int i=0; i&lt;n; i++) {<br>
                                    &nbsp;&nbsp;&nbsp; Int gi =
                                    mesh.nodes[i].global_number;<br>
                                    &nbsp;&nbsp;&nbsp; VecGetValues(x, 1, &amp;gi,
                                    &amp;unk[i]);<br>
                                    &nbsp; }<br>
                                    <br>
                                  </div>
                                  &nbsp; /* copy ghost values (CREATES MANY
                                  WARNINGS */
                                  <div><br>
                                    &nbsp; for(Int i=n; i&lt;N; i++) {<br>
                                  </div>
                                  &nbsp;&nbsp;&nbsp; VecGetValues(x, 1, &amp;i,
                                  &amp;unk[i]);<br>
                                  &nbsp; }</tt><br>
                                <br>
                                I get arnings are like this.<br>
                                <br>
                                <tt>
                                  <div>[0]PETSC ERROR:
                                    --------------------- Error Message
                                    ------------------------------------<br>
                                    [0]PETSC ERROR: Argument out of
                                    range!<br>
                                  </div>
                                  [0]PETSC ERROR: Can only get local
                                  values, trying 3518!<br>
                                  [3]PETSC ERROR: ---------------------
                                  Error Message
                                  ------------------------------------<br>
                                  [3]PETSC ERROR: Argument out of range!<br>
                                  [3]PETSC ERROR: Can only get local
                                  values, trying 3511!<br>
                                  [3]PETSC ERROR:
                                  ------------------------------------------------------------------------<br>
                                </tt><br>
                                What am I doing wrong here?<br>
                                <br>
                                <br>
                                &nbsp;&nbsp;&nbsp; Cheers,<br>
                                <br>
                                <br>
                                &nbsp;&nbsp;&nbsp; Bojan
                                <div>
                                  <div><br>
                                    <br>
                                    <br>
                                    On 2/23/2012 11:23 PM, Mohammad
                                    Mirzadeh wrote:
                                    <blockquote type="cite">
                                      <p>just index x with the local
                                        numberings. if you have 'n'
                                        local nodes and 'g' ghost nodes
                                        in x, ghost nodes indecies run
                                        from 'n' to 'n+g-1'</p>
                                      <div class="gmail_quote">On Feb
                                        23, 2012 1:16 PM, "Bojan Niceno"
                                        &lt;<a moz-do-not-send="true"
                                          href="mailto:bojan.niceno@psi.ch"
                                          target="_blank">bojan.niceno@psi.ch</a>&gt;

                                        wrote:<br type="attribution">
                                        <blockquote class="gmail_quote"
                                          style="margin:0 0 0
                                          .8ex;border-left:1px #ccc
                                          solid;padding-left:1ex">
                                          <div bgcolor="#FFFFFF"
                                            text="#000000"> Dear Jed,<br>
                                            <br>
                                            thanks.<br>
                                            <br>
                                            Now I have the following:<br>
                                            <br>
                                            - Array unk, which should
                                            hold values inside the
                                            partition and in ghost
                                            cells.&nbsp; It is big enough to
                                            hold both<br>
                                            - Vec x, created by command
                                            VecCreateGhost, with proper
                                            padding for ghost cells<br>
                                            - Successful call to linear
                                            solver in parallel.<br>
                                            <br>
                                            But I need to copy ghost
                                            values from x to my array
                                            unk. <br>
                                            <br>
                                            How can I do it?<br>
                                            <br>
                                            <br>
                                            &nbsp;&nbsp;&nbsp; Kind regards,<br>
                                            <br>
                                            <br>
                                            &nbsp;&nbsp;&nbsp; Bojan<br>
                                            <br>
                                            On 2/23/2012 10:10 PM, Jed
                                            Brown wrote:
                                            <blockquote type="cite">
                                              <div class="gmail_quote">On
                                                Thu, Feb 23, 2012 at
                                                15:05, Bojan Niceno <span
                                                  dir="ltr">&lt;<a
                                                    moz-do-not-send="true"
href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>&gt;</span>
                                                wrote:<br>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">
                                                  <div bgcolor="#FFFFFF"
                                                    text="#000000">No, I
                                                    use global.<br>
                                                  </div>
                                                </blockquote>
                                                <div><br>
                                                </div>
                                                <div>The local form is
                                                  just a local vector.
                                                  It doesn't even know
                                                  that a global problem
                                                  exists. You can't
                                                  index into it using
                                                  global indices. (In
                                                  general, there is no
                                                  efficient way to look
                                                  up information in the
                                                  local vector (includes
                                                  ghost points) using
                                                  global indices.)</div>
                                                <div>&nbsp;</div>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">
                                                  <div bgcolor="#FFFFFF"
                                                    text="#000000"> <br>
                                                    <tt>&nbsp; for(Int i=0;
                                                      i&lt;n; i++) {<br>
                                                      &nbsp;&nbsp;&nbsp; Int gi =
                                                      mesh.nodes[i].global_number;<br>
                                                      &nbsp;&nbsp;&nbsp;
                                                      VecGetValues(x, 1,
                                                      &amp;gi,
                                                      &amp;unk[i]);<br>
                                                      &nbsp; }<br>
                                                    </tt><br>
                                                    "n" is defined as
                                                    the number of cells
                                                    inside, i.e. without
                                                    buffers.&nbsp; "unk" is
                                                    my external array.&nbsp;
                                                    If I try to access
                                                    buffer values, I
                                                    use:<br>
                                                    <br>
                                                    <tt>&nbsp; for(Int i=0;
                                                      i&lt;N; i++) {<br>
                                                      &nbsp;&nbsp;&nbsp; Int gi =
                                                      mesh.nodes[i].global_number;<br>
                                                      &nbsp;&nbsp;&nbsp;
                                                      VecGetValues(x, 1,
                                                      &amp;gi,
                                                      &amp;unk[i]);<br>
                                                      &nbsp; }<br>
                                                    </tt><br>
                                                    But then I end up
                                                    with tons of
                                                    warnings, presumably
                                                    because I am going
                                                    beyond "n".&nbsp; Vector
                                                    x was created with
                                                    VecCreateGhost.</div>
                                                </blockquote>
                                              </div>
                                              <br>
                                            </blockquote>
                                            <br>
                                            <br>
                                            <div>-- <br>
                                              <img
                                                src="cid:part1.01060801.06050602@psi.ch"
                                                border="0"></div>
                                          </div>
                                        </blockquote>
                                      </div>
                                    </blockquote>
                                    <br>
                                    <br>
                                  </div>
                                </div>
                                <span><font color="#888888">
                                    <div>-- <br>
                                      <img
                                        src="cid:part2.07030405.09070507@psi.ch"
                                        border="0"></div>
                                  </font></span></div>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                    <br>
                  </div>
                </div>
                <span class="HOEnZb"><font color="#888888">
                    <div>-- <br>
                      <img src="cid:part3.03040106.01020303@psi.ch"
                        border="0"></div>
                  </font></span></div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
    <br>
    <div class="moz-signature">-- <br>
      <img src="cid:part4.01060500.04040109@psi.ch" border="0"></div>
  </body>
</html>