<div dir="ltr">Are you using local numbering when accessing the local part of ghost nodes?<br><br><div class="gmail_quote">On Thu, Feb 23, 2012 at 12:43 PM, Bojan Niceno <span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch">bojan.niceno@psi.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Dear Matt,<br>
<br>
<br>
I have a new insight, although is not the full resolution. If I
change my code in PETScSolver.cpp from:<br>
<br>
<tt><br>
</tt><tt>/*-------------------------------------------------+<br>
| Make necessary PETSc intializations for vetors |<br>
+-------------------------------------------------*/<br>
Int nghost = N - n;<br>
Int * ghosts = new Int(nghost);<br>
for(Int n=0; n<M.mesh.nodes.size(); n++) {<br>
assert( M.mesh.nodes[n].global_number >= 0);<br>
assert( M.mesh.nodes[n].global_number < 14065);<br>
}<br>
for(Int i=n; i<N; i++) {<br>
assert( M.mesh.nodes[i].global_number >= 0);<br>
assert( M.mesh.nodes[i].global_number < 14065);<br>
assert( ! (M.mesh.nodes[i].global_number >= n_start
&&<br>
M.mesh.nodes[i].global_number < n_end) );<br>
ghosts[i] = M.mesh.nodes[i].global_number;<br>
}<br>
<br>
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost,
&ghosts[0], &x);<br>
</tt><br>
to:<br>
<br>
<tt> /*-------------------------------------------------+<br>
| Make necessary PETSc intializations for vetors |<br>
+-------------------------------------------------*/<br>
Int nghost = N - n;<br>
Indices ghosts; //
<---= NEW!<br>
for(Int n=0; n<M.mesh.nodes.size(); n++) {<br>
assert( M.mesh.nodes[n].global_number >= 0);<br>
assert( M.mesh.nodes[n].global_number < 14065);<br>
}<br>
for(Int i=n; i<N; i++) {<br>
assert( M.mesh.nodes[i].global_number >= 0);<br>
assert( M.mesh.nodes[i].global_number < 14065);<br>
assert( ! (M.mesh.nodes[i].global_number >= n_start
&&<br>
M.mesh.nodes[i].global_number < n_end) );<br>
ghosts.push_back( M.mesh.nodes[i].global_number );</tt><tt>
// <---= NEW!</tt><tt> <br>
}<br>
assert( ghosts.size() == nghost ); </tt><tt>//
<---= NEW!</tt><br>
<tt><br>
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost,
&ghosts[0], &x);</tt><br>
<br>
I pass the VecCreateGhost phase. "Indices" is an STL container of
integers. It seems it works better than classical C array for this
case.<br>
<br>
<br>
However, I still do not see the ghost values, i.e. I get the
following error:<br>
<br>
<tt>[1]PETSC ERROR: --------------------- Error Message
------------------------------------<br>
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error
Message ------------------------------------<br>
[0]PETSC ERROR: Argument out of range!<br>
[0]PETSC ERROR: Can only get local values, trying 3529!<br>
[0]PETSC ERROR:
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Argument out of range!<br>
[1]PETSC ERROR: Can only get local values, trying 22!<br>
[1]PETSC ERROR:
------------------------------------------------------------------------<br>
[1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
09:28:45 CST 2012<br>
[1]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[1]PETSC ERROR: See docs/faq.html for hints about trouble
shooting.<br>
[1]PETSC ERROR: See docs/index.html for manual pages.<br>
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error
Message ------------------------------------<br>
[2]PETSC ERROR: Argument out of range!<br>
[2]PETSC ERROR: Can only get local values, trying 86!<br>
[2]PETSC ERROR:
------------------------------------------------------------------------<br>
[2]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
09:28:45 CST 2012</tt><br>
<br>
when I am trying to access values in ghost cells. What do I have to
use to see them ghosts? I reckon VecGhostGetLocalForm should be
used, right?<br>
<br>
<br>
Kind regards,<br>
<br>
<br>
Bojan<div><div class="h5"><br>
<br>
<br>
On 2/23/2012 8:36 PM, Matthew Knepley wrote:
<blockquote type="cite">On Thu, Feb 23, 2012 at 1:33 PM, Bojan Niceno <span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>></span>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Dear Matt,<br>
<br>
<br>
I sent the code as an attached tarball. I sent it with case
I run, so is 2 MB big. It is now in the cue for moderator's
approval. <br>
</div>
</blockquote>
<div><br>
</div>
<div>No, you HAVE to send it to <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>,
as I said last time, for exactly this reason.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Thanks.<br>
<br>
<br>
Kind regards,<br>
<br>
<br>
Bojan<br>
<br>
<br>
On 2/23/2012 8:04 PM, Matthew Knepley wrote:
<blockquote type="cite">On Thu, Feb 23, 2012 at 12:51 PM,
Bojan Niceno <span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>></span>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Dear Matt, <br>
<br>
<br>
are you sure? It is almost 4000 lines long! Shall
I send only the function which bother me?<br>
<br>
If the entire code is what you need, shall I make a
tarball and attach it?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Send something the builds and runs. Don't care how
long it is.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Kind
regards,<br>
<br>
<br>
Bojan<br>
<br>
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
<blockquote type="cite">On Thu, Feb 23, 2012 at
12:28 PM, Bojan Niceno <span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>></span>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> On
2/23/2012 7:24 PM, Matthew Knepley wrote:
<blockquote type="cite">On Thu, Feb 23, 2012
at 12:05 PM, Bojan Niceno <span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>></span>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Dear Matthew,<br>
<br>
<br>
thank you for your response. When I
use VecCreateGhost, I get the
following:<br>
</div>
</blockquote>
<div><br>
</div>
<div>It appears that you passed a bad
communicator. Did you not initialize a
'comm' variable?</div>
</div>
</blockquote>
<br>
I pass PETSC_COMM_WORLD to VecCreateGhost.
<br>
<br>
I don't know what you mean by 'comm'
variable :-( I called all the routines to
initialize PETSc.</div>
</blockquote>
<div><br>
</div>
<div>Send your code to <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Cheers,<br>
<br>
<br>
Bojan<br>
<br>
<blockquote type="cite">
<div class="gmail_quote">
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
[0]PETSC ERROR:
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Caught signal number
11 SEGV: Segmentation Violation,
probably memory access out of range<br>
[0]PETSC ERROR: Try option
-start_in_debugger or
-on_error_attach_debugger<br>
[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind</a>[0]PETSC
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to
find memory corruption errors<br>
[0]PETSC ERROR: likely location of
problem given in stack below<br>
[0]PETSC ERROR:
--------------------- Stack Frames
------------------------------------<br>
[0]PETSC ERROR: Note: The EXACT line
numbers in the stack are not
available,<br>
[0]PETSC ERROR: INSTEAD the
line number of the start of the
function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0]
PetscCommDuplicate line 140
src/sys/objects/tagm.c<br>
[0]PETSC ERROR: [0]
PetscHeaderCreate_Private line 30
src/sys/objects/inherit.c<br>
[0]PETSC ERROR: [0] VecCreate line
32 src/vec/vec/interface/veccreate.c<br>
[0]PETSC ERROR: [0]
VecCreateGhostWithArray line 567
src/vec/vec/impls/mpi/pbvec.c<br>
[0]PETSC ERROR: [0] VecCreateGhost
line 647
src/vec/vec/impls/mpi/pbvec.c<br>
[0]PETSC ERROR:
--------------------- Error Message
------------------------------------<br>
[0]PETSC ERROR: Signal received!<br>
[0]PETSC ERROR:
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Petsc Release
Version 3.2.0, Patch 6, Wed Jan 11
09:28:45 CST 2012<br>
[0]PETSC ERROR: See
docs/changes/index.html for recent
updates.<br>
[0]PETSC ERROR: See docs/faq.html
for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html
for manual pages.<br>
[0]PETSC ERROR:
------------------------------------------------------------------------<br>
[0]PETSC ERROR: ./PSI-Flow on a
arch-linu named lccfd06 by niceno
Thu Feb 23 19:02:45 2012<br>
[0]PETSC ERROR: Libraries linked
from
/homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib<br>
[0]PETSC ERROR: Configure run at Fri
Feb 10 10:24:13 2012<br>
[0]PETSC ERROR: Configure options<br>
[0]PETSC ERROR:
------------------------------------------------------------------------<br>
[0]PETSC ERROR: User provided
function() line 0 in unknown
directory unknown file<br>
<br>
I don't understand what could be
causing it. I took very good care
to match the global numbers of ghost
cells when calling VecCreateGhost<br>
<br>
<br>
Kind regards,<br>
<br>
<br>
Bojan<br>
<br>
<br>
On 2/23/2012 5:53 PM, Matthew
Knepley wrote:
<blockquote type="cite">On Thu, Feb
23, 2012 at 10:46 AM, Bojan Niceno
<span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>></span>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hi
all,<br>
<br>
I've never used a mailing list
before, so I hope this message
will reach PETSc users and
experts and someone might be
willing to help me. I am also
novice in PETSc.<br>
<br>
I have developed an
unstructured finite volume
solver on top of PETSc
libraries. In sequential, it
works like a charm. For the
parallel version, I do domain
decomposition externally with
Metis, and work out local and
global numberings, as well as
communication patterns between
processor. (The latter don't
seem to be needed for PETSc,
though.) When I run my
program in parallel, it also
works, but I miss values in
vectors' ghost points.<br>
<br>
I create vectors with command:
VecCreate(PETSC_COMM_WORLD,
&x);<br>
<br>
Is it possible to get the
ghost values if a vector is
created like this?<br>
</blockquote>
<div><br>
</div>
<div>I do not understand this
question. By definition,
"ghost values" are those not
stored in the global vector.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I
have tried to use
VecCreateGhost, but for some
reason which is beyond my
comprehension, PETSc goes
berserk when it reaches the
command:
VecCreateGhost(PETSC_COMM_WORLD,
n, PETSC_DECIDE, nghost,
ifrom, &x)<br>
</blockquote>
<div><br>
</div>
<div>I think you can understand
that "berserk" tells me
absolutely nothing. Error
message? Stack trace? Did you
try to run an</div>
<div>example which uses
VecGhost?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Can
anyone help me? Either how to
reach ghost values for vector
created by VecCreate, or how
to use VecCreateGhost
properly?<br>
<br>
<br>
Kind regards,<br>
<br>
Bojan<br>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most experimenters take
for granted before they begin
their experiments is
infinitely more interesting
than any results to which
their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></blockquote>
<span><font color="#888888"> <br>
<br>
<div>-- <br>
<img src="cid:part1.09040006.05010708@psi.ch" border="0"></div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></blockquote>
<span><font color="#888888"> <br>
<br>
<div>-- <br>
<img src="cid:part2.02010208.01060905@psi.ch" border="0"></div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></blockquote>
<span><font color="#888888"> <br>
<br>
<div>-- <br>
<img src="cid:part3.06090106.05090301@psi.ch" border="0"></div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they
begin their experiments is infinitely more interesting
than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></blockquote>
<span><font color="#888888"> <br>
<br>
<div>-- <br>
<img src="cid:part4.04020400.02050807@psi.ch" border="0"></div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener<br>
</blockquote>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888"><div>-- <br>
<img src="cid:part5.03030009.04030304@psi.ch" border="0"></div>
</font></span></div>
</blockquote></div><br></div>