<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi Matt,<br>
<br>
Attached is a simple test program where I'm loading the simpleblock-100.exo and trying to set up a grid with closed boundaries on all the outer faces.
<br>
<br>
We'd like to do it this way so we can use the same sort of architecture as in PetscFVM to skip ghost faces when calculating fluxes (like you suggested in your response yesterday)<br>
<br>
It works fine on a single processor but when we run it on 2 processors there's an indexing error when trying to create a matrix. There's obviously something wrong with the way we're setting up the dm but I don't know what it is? Or perrhaps we're not allowed
to use DMPlexConstructGhostCells in this way? Also I was surprised by the values given for end_interior_cell on each processor as it seems to include the overlap cells which is not what I expected?<br>
<br>
One last thing, the FORTRAN interface for DMPlexCreateLabel was missing so I added it into zplexlabel.c which works fine but I'm not sure if this was the right way to do it. My version is attached.<br>
<br>
The full output from the program for two processors is below.<br>
<br>
Cheers<br>
John<br>
<br>
(ps Is there a way to make DMView output the Labels information for all processors?)<br>
<br>
$ mpirun -n 2 simpleTest2 <br>
DM Object:Parallel Mesh 2 MPI processes<br>
type: plex<br>
Parallel Mesh in 3 dimensions:<br>
0-cells: 12 16<br>
1-cells: 20 28<br>
2-cells: 11 16<br>
3-cells: 2 3<br>
Labels:<br>
ClosedBoundaries: 0 strata of sizes ()<br>
Face Sets: 3 strata of sizes (2, 2, 5)<br>
Cell Sets: 1 strata of sizes (2)<br>
depth: 4 strata of sizes (12, 20, 11, 2)<br>
DM Object: 2 MPI processes<br>
type: plex<br>
DM_0x7f91c1d27ff0_0 in 3 dimensions:<br>
0-cells: 12 16<br>
1-cells: 20 28<br>
2-cells: 11 16<br>
3-cells: 2 (0) 3 (0)<br>
Labels:<br>
ghost: 2 strata of sizes (1, 10)<br>
vtk: 1 strata of sizes (1)<br>
Cell Sets: 1 strata of sizes (2)<br>
Face Sets: 3 strata of sizes (2, 2, 5)<br>
ClosedBoundaries: 0 strata of sizes ()<br>
0 3 3 19 35<br>
depth: 4 strata of sizes (12, 20, 11, 2)<br>
0 2 2 14 25<br>
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[1]PETSC ERROR: Argument out of range<br>
[1]PETSC ERROR: Section point -1 should be in [1, 3)<br>
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[0]PETSC ERROR: Argument out of range<br>
[0]PETSC ERROR: Section point -3 should be in [0, 1)<br>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2945-g6900bc9 GIT Date: 2015-03-10 18:15:26 -0500<br>
[0]PETSC ERROR: simpleTest2 on a arch-darwin-c-debug named Johns-MacBook-Air.local by jposunz Thu Mar 12 22:01:26 2015<br>
[0]PETSC ERROR: Configure options --download-fblaslapack --useThreads=0 --download-triangle --download-netcdf --download-exodusii --download-hdf5 --download-ptscotch --download-chaco<br>
[0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in /Users/jposunz/projects/src/petsc/src/vec/is/utils/vsectionis.c<br>
[0]PETSC ERROR: #2 DMPlexUpdateAllocation_Static() line 609 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[0]PETSC ERROR: #3 DMPlexPreallocateOperator() line 763 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[0]PETSC ERROR: #4 DMCreateMatrix_Plex() line 746 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plex.c<br>
[0]PETSC ERROR: [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.<br>
[1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2945-g6900bc9 GIT Date: 2015-03-10 18:15:26 -0500<br>
[1]PETSC ERROR: simpleTest2 on a arch-darwin-c-debug named Johns-MacBook-Air.local by jposunz Thu Mar 12 22:01:26 2015<br>
#5 DMCreateMatrix() line 958 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
[1]PETSC ERROR: Configure options --download-fblaslapack --useThreads=0 --download-triangle --download-netcdf --download-exodusii --download-hdf5 --download-ptscotch --download-chaco<br>
[1]PETSC ERROR: #1 PetscSectionGetDof() line 499 in /Users/jposunz/projects/src/petsc/src/vec/is/utils/vsectionis.c<br>
[1]PETSC ERROR: #2 DMPlexUpdateAllocation_Static() line 609 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[1]PETSC ERROR: #3 DMPlexPreallocateOperator() line 763 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[1]PETSC ERROR: #4 DMCreateMatrix_Plex() line 746 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plex.c<br>
[1]PETSC ERROR: #5 DMCreateMatrix() line 958 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
--------------------------------------------------------------------------<br>
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD <br>
with errorcode 63.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br>
--------------------------------------------------------------------------<br>
<br>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<pre>--<br>Dr John O'Sullivan
Lecturer
Department of Engineering Science
University of Auckland, New Zealand
email: <a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev">jp.osullivan at auckland.ac.nz</a>
tel: +64 (0)9 923 85353</pre>
</div>
</div>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF618442"><font color="#000000" size="2" face="Tahoma"><b>From:</b> Matthew Knepley [knepley@gmail.com]<br>
<b>Sent:</b> Thursday, 12 March 2015 3:06 a.m.<br>
<b>To:</b> John O'Sullivan<br>
<b>Cc:</b> petsc-dev@mcs.anl.gov<br>
<b>Subject:</b> Re: [petsc-dev] DMPlex, Finite Volume, overlap and ghost cells...<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Mar 11, 2015 at 3:34 AM, John O'Sullivan <span dir="ltr">
<<a href="mailto:jp.osullivan@auckland.ac.nz" target="_blank">jp.osullivan@auckland.ac.nz</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-color:rgb(204,204,204); border-left-style:solid; padding-left:1ex">
<div>
<div style="direction:ltr; font-family:Tahoma; color:rgb(0,0,0); font-size:10pt">
Hi all,<br>
<br>
I've managed to get myself very confused about how to use DMPlex correctly for a distributed Finite Volume grid...<br>
<br>
My understanding was that along partition boundaries the ghost cells are used store cell information from neighbouring partitions so that the fluxes can be calculated.<br>
<br>
Though debugging through ex11 it seems like overlap is set equal to 1?<br>
<br>
I'm solving a simple pressure-diffusion equation on a 1D column (from an exo grid) which works fine on a single processor but not in parallel. I'm certainly not setting things up right or labeling correctly...<br>
<br>
Could someone please explain the most appropriate way to set up and label the DM, whether the overlap should be 0 or 1 and whether ghost cells should be placed on internal partition boundaries.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, for FV the partition overlap should be 1, as it is in ex11. This means that when the partition happens, we will</div>
<div>put a layer of cells on the other side of partition boundaries,</div>
<div><br>
</div>
<div>When DMPlexConstructGhostCells() is called, it will put ghost cells on the other side of the true boundary. Both</div>
<div>kinds of cells will be separated from interior cells by the DMPlexGetHybridBounds(&cMax), where cMax is the</div>
<div>first ghost cell.</div>
<div><br>
</div>
<div>Now you still need a way to get rid of faces between ghost cells (which only occur in cells across a partition boundary).</div>
<div>To do this, you use the "ghost" label we make during partitioning:</div>
<div><br>
</div>
<div> ierr = DMPlexGetLabel(dm, "ghost", &ghostLabel);CHKERRQ(ierr);</div>
<div> ierr = DMLabelGetValue(ghostLabel, face, &ghost);CHKERRQ(ierr);</div>
<div> if (ghost >= 0) continue;</div>
<div><br>
</div>
<div>What exactly is going wrong?</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-color:rgb(204,204,204); border-left-style:solid; padding-left:1ex">
<div>
<div style="direction:ltr; font-family:Tahoma; color:rgb(0,0,0); font-size:10pt">
Thanks!<br>
John<br>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<pre>--<br>Dr John O'Sullivan
Lecturer
Department of Engineering Science
University of Auckland, New Zealand
email: <a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev" target="_blank">jp.osullivan at auckland.ac.nz</a>
tel: +64 (0)9 923 85353</pre>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>