<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang <span dir="ltr"><<a href="mailto:jchang27@uh.edu" target="_blank">jchang27@uh.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>1) I did what you had suggested and it works very well thanks.<br></div></div></div></div></blockquote><div><br></div><div>Cool.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div></div>2) In my case, calling uninterpolate does make a difference, just not necessarily in the solver step (I am using Tao's BLMVM). I attached two output files, one containing the timings and summaries with uninterpolating and one without uninterpolating. And the source code if you wanted to verify the "correctness" of my implementation. Namely, the biggest difference I see is in setting up the PetscSection. When uninterpolating the data this steps takes 8.64039 seconds but when I do not uninterpolate the data it takes 61.2977 seconds. Is this "significant" enough, or is it unimportant given the fact that this step occurs only once during any simulation?<br></div></div></div></blockquote><div><br></div><div>That is significant. Most of that time is in the operator preallocation routine. It was cleaner to first gather the point</div><div>adjacency information, and then push the dof information on top. If I aggressively merged the dof data, I could have</div><div>pruned a bunch of the work here and gotten much closer to the first time, at the expense of more complicated code.</div><div>I guess this is the right pattern if you know that you will only have unknowns on cells and vertices.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div></div>3) Also, about DMPlexDistribute(...). Right now it still seems extremely slow, hence why I am distributing a coarse mesh and having each processor refine its local portion of the mesh. In my current implementation, I have rank 0 distribute the mesh via ParMETIS. Will there eventually be a methodology to do parallel I/O then redistribution for load balancing or does it already exist?<br></div></div></blockquote><div><br></div><div>Redistribution for load balancing already exists. The stumbling block right now is parallel mesh import. However,</div><div>the group at ICL (Michael Lange and Gerard Gorman) is working on this, and we expect to have it working by</div><div>the end of summer. Until then, I recommend exactly what you are doing.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div>Thanks,<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div>On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang <span dir="ltr"><<a href="mailto:jchang27@uh.edu" target="_blank">jchang27@uh.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi all,<br><br></div>So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh information for a cell-vertex mesh only:<br><br>DM Object: 1 MPI processes<br> type: plex<br>DM_0x84000000_0 in 3 dimensions:<br> 0-cells: 159<br> 3-cells: 592<br>Labels:<br> marker: 1 strata of sizes (100)<br> depth: 2 strata of sizes (159, 592)<br><br></div>I am interpolating the mesh with the following lines:<br><br> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr);<br> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr);<br> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr);<br> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr);<br> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr);<br> ierr = DMDestroy(dm);CHKERRQ(ierr);<br> *dm = idm;<br><br></div>And the resulting mesh:<br><br>DM Object: 1 MPI processes<br> type: plex<br>DM_0x84000000_1 in 3 dimensions:<br> 0-cells: 159<br> 1-cells: 845<br> 2-cells: 1280<br> 3-cells: 592<br>Labels:<br> marker: 1 strata of sizes (292)<br> depth: 4 strata of sizes (159, 845, 1280, 592)<br><br></div>The reason I did this is so that incase I decide to refine the mesh with "-dm_refine <number>" the DM will be sure to include the "marker" points in the refinement process. Now, if I decide to uninterpolate the mesh with the following lines:<br><br> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr);<br> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr);<br> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr);<br> ierr = DMDestroy(dm);CHKERRQ(ierr);<br> *dm = idm;<br><br></div>I end up with this mesh:<br><br>DM Object: 1 MPI processes<br> type: plex<br>DM_0x84000000_2 in 3 dimensions:<br> 0-cells: 159<br> 3-cells: 592<br>Labels:<br> marker: 1 strata of sizes (292)<br> depth: 2 strata of sizes (159, 592)<br><br><div><div><div>The problem is that although the cells and vertices have gone back to the original number, the "marker" label still includes face/edge points. This gives me an error once I invoke DMCreateGobalVector(...).<br><br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Argument out of range<br>[0]PETSC ERROR: Section point 755 should be in [0, 751)<br>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500<br>[0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named pacotaco by justin Tue Apr 7 19:09:12 2015<br>[0]PETSC ERROR: Configure options --download-chaco --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-mpich --download-mumps --download-netcdf --download-parmetis --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug<br>[0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c<br>[0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in /home/justin/petsc-dev/src/dm/impls/plex/plex.c<br>[0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in /home/justin/petsc-dev/src/dm/impls/plex/plex.c<br>[0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c<br>[0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in /home/justin/petsc-dev/src/dm/impls/plex/plex.c<br>[0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in /home/justin/petsc-dev/src/dm/interface/dm.c<br>[0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in /home/justin/petsc-dev/src/dm/interface/dm.c<br>[0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in /home/justin/petsc-dev/src/dm/interface/dmi.c<br>[0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c<br>[0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in /home/justin/petsc-dev/src/dm/interface/dm.c<br>[0]PETSC ERROR: #11 main() line 363 in /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c<br><br><br></div><div>From this error, it seems it's trying to access sieve points that non longer exist. And I think it has to do with the label "marker" still contains data from the interpolated mesh. Is there a way to "uninterpolate" or remove such points? Or is there a better way of approaching this whole thing?<br></div></div></div></div></blockquote><div><br></div></div></div><div>1) It would be easy to write a small function to throw out the points in the label that do not exist in the DM. You</div><div> would just extract each stratumIS and then Clear each invalid point.</div><div><br></div><div>2) However, I do not understand the rationale for this above. You could just call MarkBoundaryFaces on the final mesh.</div><div> If you are really trying to preserve a label across regular refinement AND you really do not want an interpolated mesh,</div><div> then code up 1). I have yet to see a case where the extra memory for edges and faces makes a difference, but it may exist.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div></div><div>Thanks,<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div><br><br></div></font></span></div></div><span><font color="#888888"><br>-- <br><div><div dir="ltr"><div><div><div>Justin Chang<br></div>PhD Candidate, Civil Engineering - Computational Sciences<br></div>University of Houston, Department of Civil and Environmental Engineering<br></div>Houston, TX 77004<br><a href="tel:%28512%29%20963-3262" value="+15129633262" target="_blank">(512) 963-3262</a><br></div></div></font></span></div>
</blockquote></span></div><span><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><br clear="all"><br>-- <br><div><div dir="ltr"><div><div><div>Justin Chang<br></div>PhD Candidate, Civil Engineering - Computational Sciences<br></div>University of Houston, Department of Civil and Environmental Engineering<br></div>Houston, TX 77004<br><a href="tel:%28512%29%20963-3262" value="+15129633262" target="_blank">(512) 963-3262</a><br></div></div>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>