<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Mar 13, 2015 at 2:09 PM, John O'Sullivan <span dir="ltr"><<a href="mailto:jp.osullivan@auckland.ac.nz" target="_blank">jp.osullivan@auckland.ac.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">Hi Matt,<br>
<br>
I think we'd prefer a Fortran wrapper for DMPlexGetGlobalOffset_Internal(). We've got our heads around offsets now and are happy working with them.<br></div></div></blockquote><div><br></div><div>I apologize for taking so long. I got caught up in the SIAM Meeting. I have pushed</div><div><br></div><div> PetscErrorCode DMPlexGetPointGlobalField(DM dm, PetscInt point, PetscInt field, PetscInt *start, PetscInt *end)</div><div><br></div><div>to next. This corresponds with the public interface naming scheme Jed started.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
In a relate query, when we're setting up a new vector with a different dof then we follow the approach you use in plexgeometry.c (see example code snippet below). But I don't understand why when we run this in parallel we DON'T need to check if the cell is
off-process before calling PetscSectionSetDof. In fact when I did check (using the ghost label approach) it then gave the error: "Global dof 1 for point 1 is not the unconstrained 0" (see full error log below)?<br></div></div></blockquote><div><br></div><div>I am not sure you want to exclude off-process cells, since then you cannot communicate ghost results. The</div><div>error here comes from disagreement between two procs over what dofs live in a cell. If you want cells that</div><div>do not communicate, you should use a different PetscSF.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
Thanks!<br>
<br>
call DMPlexGetHeightStratum(dm, 0, start_cell, end_cell, ierr); CHKERRQ(ierr)<br>
call DMPlexGetHybridBounds(dm, end_interior_cell, &<br>
PETSC_NULL_INTEGER, PETSC_NULL_INTEGER, PETSC_NULL_INTEGER, ierr); CHKERRQ(ierr)<br>
<br>
call DMClone(dm, dmRocks, ierr); CHKERRQ(ierr)<br>
call PetscSectionCreate(PETSC_COMM_WORLD, rocks_section, ierr); CHKERRQ(ierr)<br>
call PetscSectionSetChart(rocks_section, start_cell, end_interior_cell, ierr); CHKERRQ(ierr)<span class=""><br>
do c = start_cell, end_interior_cell-1<br></span>
! Want a new vector with 4 dofs<br>
! Doesn't work if we exclude cells that are off-process<br>
call PetscSectionSetDof(rocks_section, c, 4, ierr); CHKERRQ(ierr)<br>
end do<br>
<br>
call PetscSectionSetUp(rocks_section, ierr); CHKERRQ(ierr)<br>
call DMSetDefaultSection(dmRocks, rocks_section, ierr); CHKERRQ(ierr)<br>
call DMCreateGlobalVector(dmRocks, rockproperties, ierr); CHKERRQ(ierr)<br>
<br>
<div>The full error log if we exclude off-process cells during PetscSectionSetDof:<span class=""><br>
<br>
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br></span>
[0]PETSC ERROR: Invalid argument<br>
[0]PETSC ERROR: Global dof 1 for point 1 is not the unconstrained 0<span class=""><br>
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br></span>
[1]PETSC ERROR: Invalid argument<br>
[1]PETSC ERROR: Global dof 1 for point 2 is not the unconstrained 0<span class=""><br>
[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br></span>
[1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2966-g3546a49 GIT Date: 2015-03-12 10:43:22 -0500<br>
[1]PETSC ERROR: darcy on a arch-darwin-c-debug named Johns-MacBook-Air.local by jposunz Sat Mar 14 08:07:25 2015<span class=""><br>
[1]PETSC ERROR: Configure options --download-fblaslapack --useThreads=0 --download-triangle --download-netcdf --download-exodusii --download-hdf5 --download-ptscotch --download-chaco<br></span><span class="">
[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br></span>
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2966-g3546a49 GIT Date: 2015-03-12 10:43:22 -0500<br>
[0]PETSC ERROR: [1]PETSC ERROR: #1 PetscSectionCreateGlobalSection() line 1010 in /Users/jposunz/projects/src/petsc/src/vec/is/utils/vsectionis.c<br>
[1]PETSC ERROR: #2 DMGetDefaultGlobalSection() line 3269 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
[1]PETSC ERROR: #3 DMCreateGlobalVector_Section_Private() line 13 in /Users/jposunz/projects/src/petsc/src/dm/interface/dmi.c<br>
[1]PETSC ERROR: #4 DMCreateGlobalVector_Plex() line 1170 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexcreate.c<br>
[1]PETSC ERROR: #5 DMCreateGlobalVector() line 698 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
darcy on a arch-darwin-c-debug named Johns-MacBook-Air.local by jposunz Sat Mar 14 08:07:25 2015<span class=""><br>
[0]PETSC ERROR: Configure options --download-fblaslapack --useThreads=0 --download-triangle --download-netcdf --download-exodusii --download-hdf5 --download-ptscotch --download-chaco<br></span>
[0]PETSC ERROR: #1 PetscSectionCreateGlobalSection() line 1010 in /Users/jposunz/projects/src/petsc/src/vec/is/utils/vsectionis.c<br>
[0]PETSC ERROR: #2 DMGetDefaultGlobalSection() line 3269 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
[0]PETSC ERROR: #3 DMCreateGlobalVector_Section_Private() line 13 in /Users/jposunz/projects/src/petsc/src/dm/interface/dmi.c<br>
[0]PETSC ERROR: #4 DMCreateGlobalVector_Plex() line 1170 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexcreate.c<br>
[0]PETSC ERROR: #5 DMCreateGlobalVector() line 698 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<span class=""><br>
--------------------------------------------------------------------------<br>
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD <br></span>
with errorcode 62.<span class=""><br>
<br>
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<pre>--<br>Dr John O'Sullivan
Lecturer
Department of Engineering Science
University of Auckland, New Zealand
email: <a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev" target="_blank">jp.osullivan at auckland.ac.nz</a>
tel: +64 (0)9 923 85353</pre>
</div>
</div>
</div>
</span></div>
<div style="font-family:'Times New Roman';color:rgb(0,0,0);font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma"><b>From:</b> Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br>
<b>Sent:</b> Friday, 13 March 2015 11:57 p.m.<span class=""><br>
<b>To:</b> John O'Sullivan<br>
<b>Cc:</b> <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a><br>
</span><b>Subject:</b> Re: More on FV overlap cells and Fortran<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Fri, Mar 13, 2015 at 4:56 AM, John O'Sullivan <span dir="ltr">
<<a href="mailto:jp.osullivan@auckland.ac.nz" target="_blank">jp.osullivan@auckland.ac.nz</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">Hi Matt,<br>
<br>
We're getting there with our parallel Fortran FV code. But here's another question about the best way to do things. Another simpleTest2 code is attached to demonstrate the question and below is a copy of the relevant part:<br>
<br>
<span style="font-size:10pt;font-family:Tahoma,sans-serif" lang="EN-US"><span><a href="mailto:petsc-dev@mcs.anl.gov" target="_blank"></a></span></span>! We worked out its important not to write to the parts of arrays that correspond to overlap cells.
<br>
! Following ex11 it would be done something like this using C:<br>
!<br>
! ierr = DMPlexPointGlobalRef(dm,c,x,&xc);CHKERRQ(ierr);<br>
! if (xc) {ierr = (*mod->solution)(mod,0.0,cg->centroid,xc,mod->solutionctx);CHKERRQ(ierr);}<br>
! <br>
! And according to the documentation on DMPlexPointGlobalRef xc would be returned as null for
<br>
! the overlap blocks that don't belong to the processor<br>
!<br>
! We can achieve the same using the ghost labels on the overlap cells but is this a good<br>
! idea? We'd prefer to follow the standard approach ie in ex11 but I don't think there's a<br>
! clean way to do this in fortran as we can't pass pointers to structures...<br>
! Could it be done with DMPlexPointGlobalRef? If so, any ideas?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I can do two things:</div>
<div><br>
</div>
<div> a) Make a Fortran version of DMPlexPointGlobalRef(), which returns an F90 array of size dof</div>
<div><br>
</div>
<div>or</div>
<div><br>
</div>
<div> b) Make a Fortran wrapper for DMPlexGetGlobalOffset_Internal(), which returns start/end, and</div>
<div> start < 0 if its off-process</div>
<div><br>
</div>
<div>Tell me what you think is better. Also, you can see here that we are experimenting with different</div>
<div>interfaces. Jed likes getting a pointer back. I like getting offsets back.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt"> do c = start_cell, end_interior_cell-1
<br>
! ??? call DMPlexPointGlobalRef(dm, c, localx, someref)???<br>
call DMLabelGetValue(label, c, ghost, ierr); CHKERRQ(ierr)<br>
if (ghost < 0) then<br>
call PetscSectionGetOffset(solution_section, c, cell_offset, ierr); CHKERRQ(ierr)<br>
cell_offset = cell_offset + 1<br>
localx(cell_offset) = 100.0<br>
end if<br>
end do<br>
<br>
<div>Cheers!<br>
John<br>
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<pre>--<br>Dr John O'Sullivan
Lecturer
Department of Engineering Science
University of Auckland, New Zealand
email: <a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev" target="_blank">jp.osullivan at auckland.ac.nz</a>
tel: +64 (0)9 923 85353</pre>
</div>
</div>
</div>
</div>
<div style="font-family:'Times New Roman';color:rgb(0,0,0);font-size:16px">
<hr>
<span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br>
</span>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-NZ">
<div><span></span>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif" lang="EN-US">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif" lang="EN-US"> Matthew Knepley [mailto:<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]
<br>
<b>Sent:</b> Friday, 13 March 2015 4:44 a.m.<span><br>
<b>To:</b> John O'Sullivan<br>
<b>Cc:</b> <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a><br>
</span><b>Subject:</b> Re: Simple Test example not working in parallel for DMPlex/FV/overlap/ghost cells...<u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Mar 12, 2015 at 4:19 AM, John O'Sullivan <<a href="mailto:jp.osullivan@auckland.ac.nz" target="_blank">jp.osullivan@auckland.ac.nz</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black">Hi Matt,<br>
<br>
Attached is a simple test program where I'm loading the simpleblock-100.exo and trying to set up a grid with closed boundaries on all the outer faces.
<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Just so I understand what you want. You do not want any ghost cells made around the outer boundary. What you have will do that.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black">We'd like to do it this way so we can use the same sort of architecture as in PetscFVM to skip ghost faces when calculating fluxes (like you suggested in your
response yesterday)<u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Note that now, it will also mark the outer boundary faces as ghosts because they have no partner cell.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black">It works fine on a single processor but when we run it on 2 processors there's an indexing error when trying to create a matrix. There's obviously something
wrong with the way we're setting up the dm but I don't know what it is? Or perrhaps we're not allowed to use DMPlexConstructGhostCells in this way? Also I was surprised by the values given for end_interior_cell on each processor as it seems to include the
overlap cells which is not what I expected?<u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This is a bug in my latest preallocation push. Thanks for finding it. I pushed a fix for this, and for many Fortran things<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">to next. I am including my modified source so that everything is deallocated correctly (run with -malloc_test).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It seems to work for me now. Let me know if you have problems.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black">One last thing, the FORTRAN interface for DMPlexCreateLabel was missing so I added it into zplexlabel.c which works fine but I'm
not sure if this was the right way to do it. My version is attached.<br>
<br>
The full output from the program for two processors is below.<br>
<br>
Cheers<br>
John<br>
<br>
(ps Is there a way to make DMView output the Labels information for all processors?)<br>
<br>
$ mpirun -n 2 simpleTest2 <br>
DM Object:Parallel Mesh 2 MPI processes<br>
type: plex<br>
Parallel Mesh in 3 dimensions:<br>
0-cells: 12 16<br>
1-cells: 20 28<br>
2-cells: 11 16<br>
3-cells: 2 3<br>
Labels:<br>
ClosedBoundaries: 0 strata of sizes ()<br>
Face Sets: 3 strata of sizes (2, 2, 5)<br>
Cell Sets: 1 strata of sizes (2)<br>
depth: 4 strata of sizes (12, 20, 11, 2)<br>
DM Object: 2 MPI processes<br>
type: plex<br>
DM_0x7f91c1d27ff0_0 in 3 dimensions:<br>
0-cells: 12 16<br>
1-cells: 20 28<br>
2-cells: 11 16<br>
3-cells: 2 (0) 3 (0)<br>
Labels:<br>
ghost: 2 strata of sizes (1, 10)<br>
vtk: 1 strata of sizes (1)<br>
Cell Sets: 1 strata of sizes (2)<br>
Face Sets: 3 strata of sizes (2, 2, 5)<br>
ClosedBoundaries: 0 strata of sizes ()<br>
0 3 3 19 35<br>
depth: 4 strata of sizes (12, 20, 11, 2)<br>
0 2 2 14 25<br>
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[1]PETSC ERROR: Argument out of range<br>
[1]PETSC ERROR: Section point -1 should be in [1, 3)<br>
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[0]PETSC ERROR: Argument out of range<br>
[0]PETSC ERROR: Section point -3 should be in [0, 1)<br>
[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2945-g6900bc9 GIT Date: 2015-03-10 18:15:26 -0500<br>
[0]PETSC ERROR: simpleTest2 on a arch-darwin-c-debug named Johns-MacBook-Air.local by jposunz Thu Mar 12 22:01:26 2015<br>
[0]PETSC ERROR: Configure options --download-fblaslapack --useThreads=0 --download-triangle --download-netcdf --download-exodusii --download-hdf5 --download-ptscotch --download-chaco<br>
[0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in /Users/jposunz/projects/src/petsc/src/vec/is/utils/vsectionis.c<br>
[0]PETSC ERROR: #2 DMPlexUpdateAllocation_Static() line 609 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[0]PETSC ERROR: #3 DMPlexPreallocateOperator() line 763 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[0]PETSC ERROR: #4 DMCreateMatrix_Plex() line 746 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plex.c<br>
[0]PETSC ERROR: [1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
[1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2945-g6900bc9 GIT Date: 2015-03-10 18:15:26 -0500<br>
[1]PETSC ERROR: simpleTest2 on a arch-darwin-c-debug named Johns-MacBook-Air.local by jposunz Thu Mar 12 22:01:26 2015<br>
#5 DMCreateMatrix() line 958 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
[1]PETSC ERROR: Configure options --download-fblaslapack --useThreads=0 --download-triangle --download-netcdf --download-exodusii --download-hdf5 --download-ptscotch --download-chaco<br>
[1]PETSC ERROR: #1 PetscSectionGetDof() line 499 in /Users/jposunz/projects/src/petsc/src/vec/is/utils/vsectionis.c<br>
[1]PETSC ERROR: #2 DMPlexUpdateAllocation_Static() line 609 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[1]PETSC ERROR: #3 DMPlexPreallocateOperator() line 763 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plexpreallocate.c<br>
[1]PETSC ERROR: #4 DMCreateMatrix_Plex() line 746 in /Users/jposunz/projects/src/petsc/src/dm/impls/plex/plex.c<br>
[1]PETSC ERROR: #5 DMCreateMatrix() line 958 in /Users/jposunz/projects/src/petsc/src/dm/interface/dm.c<br>
--------------------------------------------------------------------------<br>
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD <br>
with errorcode 63.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br>
--------------------------------------------------------------------------<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black"><u></u> <u></u></span></p>
<div>
<div>
<div>
<pre><span style="color:black">--<br>Dr John O'Sullivan<u></u><u></u></span></pre>
<pre><span style="color:black">Lecturer<u></u><u></u></span></pre>
<pre><span style="color:black">Department of Engineering Science<u></u><u></u></span></pre>
<pre><span style="color:black">University of Auckland, New Zealand<u></u><u></u></span></pre>
<pre><span style="color:black">email: <a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev" target="_blank">jp.osullivan at auckland.ac.nz</a><u></u><u></u></span></pre>
<pre><span style="color:black">tel: +64 (0)9 923 85353<u></u><u></u></span></pre>
</div>
</div>
</div>
</div>
<div>
<div class="MsoNormal" style="text-align:center" align="center"><span style="color:black">
<hr size="2" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-family:Tahoma,sans-serif;color:black">From:</span></b><span style="font-family:Tahoma,sans-serif;color:black"> Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br>
<b>Sent:</b> Thursday, 12 March 2015 3:06 a.m.<br>
<b>To:</b> John O'Sullivan<br>
<b>Cc:</b> <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-dev] DMPlex, Finite Volume, overlap and ghost cells...</span><span style="color:black"><u></u><u></u></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">On Wed, Mar 11, 2015 at 3:34 AM, John O'Sullivan <<a href="mailto:jp.osullivan@auckland.ac.nz" target="_blank">jp.osullivan@auckland.ac.nz</a>> wrote:<u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black">Hi all,<br>
<br>
I've managed to get myself very confused about how to use DMPlex correctly for a distributed Finite Volume grid...<br>
<br>
My understanding was that along partition boundaries the ghost cells are used store cell information from neighbouring partitions so that the fluxes can be calculated.<br>
<br>
Though debugging through ex11 it seems like overlap is set equal to 1?<br>
<br>
I'm solving a simple pressure-diffusion equation on a 1D column (from an exo grid) which works fine on a single processor but not in parallel. I'm certainly not setting things up right or labeling correctly...<br>
<br>
Could someone please explain the most appropriate way to set up and label the DM, whether the overlap should be 0 or 1 and whether ghost cells should be placed on internal partition boundaries.<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Yes, for FV the partition overlap should be 1, as it is in ex11. This means that when the partition happens, we will<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">put a layer of cells on the other side of partition boundaries,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">When DMPlexConstructGhostCells() is called, it will put ghost cells on the other side of the true boundary. Both<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">kinds of cells will be separated from interior cells by the DMPlexGetHybridBounds(&cMax), where cMax is the<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">first ghost cell.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Now you still need a way to get rid of faces between ghost cells (which only occur in cells across a partition boundary).<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">To do this, you use the "ghost" label we make during partitioning:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> ierr = DMPlexGetLabel(dm, "ghost", &ghostLabel);CHKERRQ(ierr);<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> ierr = DMLabelGetValue(ghostLabel, face, &ghost);CHKERRQ(ierr);<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> if (ghost >= 0) continue;<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">What exactly is going wrong?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> Matt<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:black">Thanks!<br>
John</span><span><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:rgb(136,136,136)"><u></u><u></u></span></span></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<pre><span style="color:rgb(136,136,136)">--<br>Dr John O'Sullivan<u></u><u></u></span></pre>
<pre><span style="color:rgb(136,136,136)">Lecturer<u></u><u></u></span></pre>
<pre><span style="color:rgb(136,136,136)">Department of Engineering Science<u></u><u></u></span></pre>
<pre><span style="color:rgb(136,136,136)">University of Auckland, New Zealand<u></u><u></u></span></pre>
<pre><span style="color:rgb(136,136,136)">email: <a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev" target="_blank">jp.osullivan at auckland.ac.nz</a><u></u><u></u></span></pre>
<pre><span style="color:rgb(136,136,136)">tel: +64 (0)9 923 85353<u></u><u></u></span></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br>
<br clear="all">
</span><span><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span><span style="color:rgb(136,136,136)">-- </span><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <span><font color="#888888"><u></u></font></span></p>
<span><font color="#888888"></font></span></div>
<span><font color="#888888">
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
</font></span></div>
<span><font color="#888888"></font></span></div>
<span><font color="#888888"></font></span></div>
<span><font color="#888888"></font></span></div>
<span><font color="#888888"></font></span></div>
<span><font color="#888888"></font></span></div>
<span><font color="#888888"></font></span></blockquote>
<span><font color="#888888"></font></span></div>
<span><font color="#888888"><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>